feat: Refactor DeepSeekV3 function call

[CatherineSue]

Motivation

Instead of keeping editing the prompt engineering for deepseekv3 in adapter.py, this PR provides a chat template instead.

Implements: #5224 (comment)

Modifications

• Add an adapted tool call template for deepseekv3 (credits to @finger92 's original improved chat template)
• Remove promopt engineering for deepseekv3 tool call in adapter.py
• Update docs/reference/deepseek.md

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[CatherineSue]

also fix #5814

Test with prompt in the issue

{"id":"e5f041968f104ba3b964fe862c7096b7","object":"chat.completion","created":1745994924,"model":"sgl-model","choices":[{"index":0,"message":{"role":"assistant","content":null,"reasoning_content":null,"tool_calls":[{"id":"2","index":null,"type":"function","function":{"name":"get_current_weather","arguments":"{\"city\": \"Tokyo\"}"}}]},"logprobs":null,"finish_reason":"tool_calls","matched_stop":null}],"usage":{"prompt_tokens":254,"total_tokens":275,"completion_tokens":21,"prompt_tokens_details":null}}#        

Test with multiple system messages
request:

curl "http://127.0.0.1:8080/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "temperature": 0,
    "max_tokens": 100,
    "model": "deepseek-ai/DeepSeek-V3-0324",
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "query_weather",
          "description": "Get weather of a city, the user should supply a city first",
          "parameters": {
            "type": "object",
            "properties": {
              "city": {
                "type": "string",
                "description": "The city, e.g. Beijing"
              }
            },
            "required": ["city"]
          }
        }
      }
    ],
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant with access to weather tools."
      },
      {
        "role": "user",
        "content": "Hows the weather like in Qingdao today"
      },
      {
        "role": "system",
        "content": "Please always answer in concise sentences."
      }
    ]
  }'

response:

{"id":"4582a7a1775a49578bffcb3c672b444c","object":"chat.completion","created":1745995026,"model":"deepseek-ai/DeepSeek-V3-0324","choices":[{"index":0,"message":{"role":"assistant","content":null,"reasoning_content":null,"tool_calls":[{"id":"0","index":null,"type":"function","function":{"name":"query_weather","arguments":"{\"city\": \"Qingdao\"}"}}]},"logprobs":null,"finish_reason":"tool_calls","matched_stop":null}],"usage":{"prompt_tokens":133,"total_tokens":155,"completion_tokens":22,"prompt_tokens_details":null}}

[CatherineSue]

debug info for chat message after applying the chat template:

[Frank-Jie]

Would you mind make a test without system prompt in messages ?
like this:

curl "http://127.0.0.1:8080/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "temperature": 0,
    "max_tokens": 500,
    "model": "deepseek-ai/DeepSeek-V3-0324",
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "query_weather",
          "description": "Get weather of a city, the user should supply a city first",
          "parameters": {
            "type": "object",
            "properties": {
              "city": {
                "type": "string",
                "description": "The city, e.g. Beijing"
              }
            },
            "required": ["city"]
          }
        }
      }
    ],
    "messages": [
      {
        "role": "user",
        "content": "Hows the weather like in Qingdao today"
      }
    ]
  }'

Sometimes when I run the test, it outputs a lot of unknown character like: "ꯃꯂꯂꯂ\nꯂ\nꯂ\nꯂ\nꯂ"

[Frank-Jie]

Suggested change

{{ bos_token }}

[CatherineSue]

I think we need to keep bos_token here as the original chat template keeps it.

[CatherineSue]

If I remove bos_token, sometimes I can get the reponse with random characters:
'<｜tool▁calls▁begin｜>晾<｜tool▁calls▁begin｜>ราบ房租 العصيان:\n\n```json\n{"city":"Nanjing"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>'

[CatherineSue]

Would you mind make a test without system prompt in messages ? like this:

Sometimes when I run the test, it outputs a lot of unknown character like: "ꯃꯂꯂꯂ\nꯂ\nꯂ\nꯂ\nꯂ"

I run the following script and it failed 0 times out of 100. (same as the one from #5224 (comment))

import requests, json, random, re

test_func = {
    "type": "function",
    "function": {
        "name": "query_weather",
        "description": "Get weather of an location, the user shoud supply a location first",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {
                    "type": "string",
                    "description": "The city, e.g. Beijing"
                }
            },
            "required": [
                "city"
            ]
        }
    }
}


req_base_url = "http://127.0.0.1:8080"
cities = [
    "Beijing", 
    "Chongqing",
    "Chengdu",
    "Dalian",
    "Guangzhou",
    "Hangzhou",
    "Harbin",
    "Hefei",
    "Kunming",
    "Lanzhou",
    "Nanjing",
    "Qingdao",
    "Shanghai",
    "Shenzhen",
    "Suzhou",
    "Tianjin",
    "Wuhan",
    "Xi'an",
    "Xiamen",
    "Zhengzhou"
]
user_content_tp = "Hows the weather like in {} today"
req_body = {
    "messages": [
        {
            "role": "user",
            "content": ""
        }
    ],
    "temperature": 0,
    "max_tokens": 100,
    "model": "deepseek-ai/DeepSeek-V3-0324",
    "tools": [
        test_func
    ]
}

total_failed = 0
for i in range(100):
    req_body["messages"][0]["content"] = user_content_tp.format(random.choice(cities))
    res = requests.post(req_base_url + "/v1/chat/completions", json=req_body)
    res = res.json()
    if len(res["choices"]) > 0:
        if len(res["choices"][0]["message"]["tool_calls"]) > 0:
            print("function call successfull: " + json.dumps(res["choices"][0]["message"]["tool_calls"][0]))
        else:
            total_failed += 1
            print("function call failed: " + json.dumps(res))
            
    # flush cache
    requests.get(req_base_url + "/flush_cache")
print(f"total_failed: {total_failed}")

[zhaochenyang20]

@CatherineSue is this ready to merge？let me rerun the CI

[CatherineSue]

@zhaochenyang20 let me do more sanity check with multi-turn tool call.

For single-turn, I ran the above script for 1000 times, it failed 0.

[CatherineSue]

The single-turn function call is ready but multi-turn might need some changes.

@zhaochenyang20 Let's wait for #5907 to merge first, since deepseekv3 official chat template relies on tool_calls in assistant role message, which could be addressed in that PR. I want to do a more thorough test after that PR unless it is urgent to merge this one.

[CatherineSue]

I tried to patch #5907 to this PR and did some test on multi-turn.

Test 1: Normal multi-turn
Purpose: The model should be able to use the final tools message to generate an answer
Script:

adapted from [function_calling.ipynb](https://github.com/sgl-project/sglang/blob/main/docs/backend/function_calling.ipynb) 

import openai
import json
import random
import time
import requests

client = openai.Client(base_url="http://localhost:8080/v1", api_key="xxxxxx")
model_name = client.models.list().data[0].id

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string", "description": "City name"},
                    "state": {"type": "string", "description": "State abbreviation"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["city", "state", "unit"]
            },
        },
    }
]

cities = [
    {"city": "Birmingham", "state": "AL"},
    {"city": "Anchorage", "state": "AK"},
    {"city": "Phoenix", "state": "AZ"},
    {"city": "Little Rock", "state": "AR"},
    {"city": "Los Angeles", "state": "CA"},
    {"city": "Denver", "state": "CO"},
    {"city": "Bridgeport", "state": "CT"},
    {"city": "Wilmington", "state": "DE"},
    {"city": "Miami", "state": "FL"},
    {"city": "Atlanta", "state": "GA"},
    {"city": "Honolulu", "state": "HI"},
    {"city": "Boise", "state": "ID"},
    {"city": "Chicago", "state": "IL"},
    {"city": "Indianapolis", "state": "IN"},
    {"city": "Des Moines", "state": "IA"},
    {"city": "Wichita", "state": "KS"},
    {"city": "Louisville", "state": "KY"},
    {"city": "New Orleans", "state": "LA"},
    {"city": "Portland", "state": "ME"},
    {"city": "Baltimore", "state": "MD"},
    {"city": "Boston", "state": "MA"},
    {"city": "Detroit", "state": "MI"},
    {"city": "Minneapolis", "state": "MN"},
    {"city": "Jackson", "state": "MS"},
    {"city": "Kansas City", "state": "MO"},
    {"city": "Billings", "state": "MT"},
    {"city": "Omaha", "state": "NE"},
    {"city": "Las Vegas", "state": "NV"},
    {"city": "Manchester", "state": "NH"},
    {"city": "Newark", "state": "NJ"},
    {"city": "Albuquerque", "state": "NM"},
    {"city": "New York", "state": "NY"},
    {"city": "Charlotte", "state": "NC"},
    {"city": "Fargo", "state": "ND"},
    {"city": "Columbus", "state": "OH"},
    {"city": "Oklahoma City", "state": "OK"},
    {"city": "Portland", "state": "OR"},
    {"city": "Philadelphia", "state": "PA"},
    {"city": "Providence", "state": "RI"},
    {"city": "Charleston", "state": "SC"},
    {"city": "Sioux Falls", "state": "SD"},
    {"city": "Nashville", "state": "TN"},
    {"city": "Houston", "state": "TX"},
    {"city": "Salt Lake City", "state": "UT"},
    {"city": "Burlington", "state": "VT"},
    {"city": "Virginia Beach", "state": "VA"},
    {"city": "Seattle", "state": "WA"},
    {"city": "Charleston", "state": "WV"},
    {"city": "Milwaukee", "state": "WI"},
    {"city": "Cheyenne", "state": "WY"},
    {"city": "Washington", "state": "DC"}
]

units = ["celsius", "fahrenheit"]

def get_current_weather(city, state, unit):
    if unit == "celsius":
        temp = random.randint(10, 40)
    else:
        temp = random.randint(50, 105)
    return f"The weather in {city}, {state} is {temp} degrees {unit}. It is partly cloudy.", temp

success = 0
fail = 0
num_runs = 100

for i in range(num_runs):
    try:
        loc = random.choice(cities)
        unit = random.choice(units)

        messages = [
            {"role": "system", "content": "You are a travel assistant."},
            {"role": "user", "content": f"I'm planning a trip to {loc['city']} in {loc['state']} state. What's the weather like in {unit}?"},
        ]

        response = client.chat.completions.create(
            model=model_name,
            messages=messages,
            tools=tools,
            stream=False,
            temperature=0.7,
            top_p=0.9,
            max_tokens=1000,
        )

        tool_call = response.choices[0].message.tool_calls[0]
        name = tool_call.function.name
        args = json.loads(tool_call.function.arguments)
        assert name == "get_current_weather"

        tool_call_output, temp = get_current_weather(**args)
        
        tool_call_msg = {
            "id": "0",
            "type": "function",
            "function": {
                "name": name,
                "arguments": str(args)
            }
        }

        messages.append({"role": "assistant", "content": None, "tool_calls": [tool_call_msg]})
        messages.append({"role": "tool", "name": name, "tool_call_id": tool_call.id, "content": tool_call_output})

        final = client.chat.completions.create(
            model=model_name,
            messages=messages,
            temperature=0,
            top_p=0.8,
            stream=False,
            tools=tools, # Result2 are this line commented out
            # max_tokens=300,
        )

        content = final.choices[0].message.content
        if content and str(args["city"]).lower() in content.lower() and str(temp).lower() in content.lower():
            success += 1
        else:
            fail += 1

        print(f"[{i+1}] Success={success} / Fail={fail}\n Function call result: {tool_call_output} \n Answer: {content}\n")

        try:
            requests.post("http://localhost:8080/flush_cache")
        except Exception as e:
            print(f"Warning: failed to flush cache — {e}")

        time.sleep(1)

    except requests.exceptions.ConnectionError or ConnectionError:
        print("Connection error — stopping early.")
        break
    except Exception as e:
        print(f"Unexpected error: {e}")
        fail += 1
        continue

print(f"Final Result: {success} success / {fail} failed out of {num_runs} attempts.")

Results 1:

[97] Success=97 / Fail=0
 Function call result: The weather in Wilmington, DE is 25 degrees celsius. It is partly cloudy. 
 Answer: The current weather in Wilmington, DE is 25°C and partly cloudy. Enjoy your trip!

[98] Success=98 / Fail=0
 Function call result: The weather in Honolulu, HI is 19 degrees celsius. It is partly cloudy. 
 Answer: The current weather in Honolulu, HI is 19°C and partly cloudy. Enjoy your trip!

[99] Success=99 / Fail=0
 Function call result: The weather in Newark, NJ is 56 degrees fahrenheit. It is partly cloudy. 
 Answer: The current weather in Newark, NJ is 56°F and partly cloudy. Enjoy your trip!

[100] Success=100 / Fail=0
 Function call result: The weather in Charleston, WV is 92 degrees fahrenheit. It is partly cloudy. 
 Answer: The current weather in Charleston, WV is 92°F and partly cloudy. Enjoy your trip!

Final Result: 100 success / 0 failed out of 100 attempts.

Results 2: (last turn doesn't pass tools)

[96] Success=96 / Fail=0
 Function call result: The weather in Charleston, SC is 21 degrees celsius. It is partly cloudy. 
 Answer: Currently, the weather in Charleston, SC is **21°C** and partly cloudy. It's a pleasant temperature for exploring the city! If you're planning outdoor activities, you might want to bring a light jacket for the evening, as temperatures can drop slightly. 

Would you like any recommendations for things to do based on this weather?

[97] Success=97 / Fail=0
 Function call result: The weather in Las Vegas, NV is 96 degrees fahrenheit. It is partly cloudy. 
 Answer: The current weather in Las Vegas, NV is **96°F** and partly cloudy. It's going to be a warm day, so be sure to stay hydrated and wear sunscreen if you're planning to be outdoors! Let me know if you'd like any other details for your trip.

[98] Success=98 / Fail=0
 Function call result: The weather in Nashville, TN is 29 degrees celsius. It is partly cloudy. 
 Answer: The current weather in Nashville, Tennessee, is **29°C (84°F)** with partly cloudy skies. It's a warm day, so light clothing and sunscreen would be a good idea if you're planning outdoor activities. Let me know if you'd like a forecast for your travel dates!

[99] Success=99 / Fail=0
 Function call result: The weather in Bridgeport, CT is 50 degrees fahrenheit. It is partly cloudy. 
 Answer: The current weather in Bridgeport, CT is **50°F** and partly cloudy. Enjoy your trip! If you'd like more details or a forecast for specific dates, let me know.

[100] Success=100 / Fail=0
 Function call result: The weather in Las Vegas, NV is 30 degrees celsius. It is partly cloudy. 
 Answer: The current weather in Las Vegas, Nevada is **30°C** and partly cloudy. Enjoy your trip! If you'd like more details or a forecast for specific dates, let me know.

Final Result: 100 success / 0 failed out of 100 attempts.

Test 2: Different scenario of multi-turn conversation
Purpose: Model should predict correctly whether to use tool outputs or respond naturally

Script:

import openai
import json

client = openai.Client(base_url="http://localhost:8080/v1", api_key="xxxxxx")
model_name = client.models.list().data[0].id

tools = [{
    "type": "function",
    "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {"type": "string"},
                "state": {"type": "string"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            },
            "required": ["city", "state", "unit"],
        },
    }
}]

def get_current_weather(city, state, unit):
    if not state:
        return "Error: State must be specified.", None
    temp = 25 if unit == "celsius" else 77
    return f"The weather in {city}, {state} is {temp} degrees {unit}. It is partly cloudy.", temp

test_cases = [
    {
        "name": "✅ Valid tool output, no re-call expected",
        "user": "I'm planning a trip to Paris. What's the weather like in celsius?",
        "tool_args": {"city": "Paris", "state": "FR", "unit": "celsius"},
        "expect_city": "Paris",
        "expect_temp": 25
    },
    {
        "name": "❌ Invalid tool output, should re-call",
        "user": "What's the weather in New York in state NY?",
        "tool_args": {"city": "New York", "state": "", "unit": "celsius"},
        "follow_up": "The state is NY",
        "expect_retry": True
    },
    {
        "name": "🔁 Multiple tool calls expected",
        "user": "What's the weather in Tokyo and London in fahrenheit?",
        "multi_tool_args": [
            {"city": "Tokyo", "state": "TY", "unit": "fahrenheit"},
            {"city": "London", "state": "UK", "unit": "celsius"},
        ],
        "expect_cities": ["Tokyo", "London"]
    },
    {
        "name": "➡️ Follow-up after tool output",
        "user": "What's the weather in Rome?",
        "tool_args": {"city": "Rome", "state": "RM", "unit": "celsius"},
        "follow_up": "What about weather in London?",
        "expect_follow": "London"
    }
]

for case in test_cases:
    print("=" * 80)
    print(f"Test: {case['name']}")
    
    messages = [
        {"role": "system", "content": "You are a travel assistant."},
        {"role": "user", "content": case["user"]}
    ]

    if case["name"].startswith("🔁 Multiple tool calls expected"):
        # First turn: ask the model
        messages = [
            {"role": "system", "content": "You are a travel assistant."},
            {"role": "user", "content": case["user"]}
        ]

        initial = client.chat.completions.create(
            model=model_name,
            messages=messages,
            tools=tools,
            stream=False,
            temperature=0,
            top_p=1,
        )

        tool_calls = initial.choices[0].message.tool_calls
        if not tool_calls or len(tool_calls) < 2:
            print("❌ Model did not emit multiple tool calls.")
            continue
        else:
            print(f"✅ Model emitted {len(tool_calls)} tool calls.")

        tool_outputs = []
        assistant_call = {
            "role": "assistant",
            "content": None,
            "tool_calls": [
                {
                    "id": tc.id,
                    "type": tc.type,
                    "function": {
                        "name": tc.function.name,
                        "arguments": tc.function.arguments
                    }
                }
                for tc in tool_calls
            ]
        }

        messages.append(assistant_call)

        for tc in tool_calls:
            args = json.loads(tc.function.arguments)
            tool_result, temp = get_current_weather(**args)
            tool_outputs.append({
                "role": "tool",
                "tool_call_id": tc.id,
                "name": tc.function.name,
                "content": tool_result
            })
            messages.append(tool_outputs[-1])

        # Ask final generation
        final = client.chat.completions.create(
            model=model_name,
            messages=messages,
            # tools=tools,
            stream=False,
            temperature=0.5,
            top_p=1,
            max_tokens=500,
        )

        output = final.choices[0].message.content
        print(f"Model output:\n{output}\n")

        if all(city.lower() in output.lower() for city in case["expect_cities"]):
            print("✅ All cities handled in the response.")
        else:
            print("❌ Missing one or more cities in the final answer.")

    else:
        tool_output, temp = get_current_weather(**case["tool_args"])
        tool_call_msg = {
            "id": "0",
            "type": "function",
            "function": {
                "name": "get_current_weather",
                "arguments": json.dumps(case["tool_args"])
            }
        }
        messages.append({"role": "assistant", "content": None, "tool_calls": [tool_call_msg]})
        messages.append({
            "role": "tool",
            "name": "get_current_weather",
            "tool_call_id": "0",
            "content": tool_output
        })

        if "follow_up" in case:
            messages.append({"role": "user", "content": case["follow_up"]})

        final = client.chat.completions.create(
            model=model_name,
            messages=messages,
            tools=tools,
            stream=False,
            temperature=0,
            top_p=1,
            max_tokens=500,
        )

        output = final.choices[0].message.content
        tool_calls = final.choices[0].message.tool_calls
        print(f"Model output:\n{output}\n")
        print(f"Model toolcalls:\n{tool_calls}\n")

    # Eval
    if "expect_retry" in case:
        if (output is not None and "state must be specified" in messages[-1]["content"].lower() and "new york" not in output.lower()) or tool_calls is not None:
            print("✅ Correctly did not fabricate answer.")
        else:
            print("❌ Failed to handle tool error properly.")

    elif "expect_cities" in case:
        if all(city.lower() in output.lower() for city in case["expect_cities"]):
            print("✅ All cities handled.")
        else:
            print("❌ Missing one or more city responses.")

    elif "expect_follow" in case:
        if (output is not None and case["expect_follow"] in output.lower()) or tool_calls is not None:
            print("✅ Follow-up was understood.")
        else:
            print("❌ Missed follow-up context.")

    else:
        if case["expect_city"].lower() in output.lower() and str(case["expect_temp"]) in output:
            print("✅ Weather response looks good.")
        else:
            print("❌ Model ignored tool output or responded incorrectly.")

Result:

➜  sglang git:(chang/deepseek-v3-tool-call) ✗ python3 examples/function_tool_call_test2.py
================================================================================
Test: ✅ Valid tool output, no re-call expected
Model output:
The current weather in Paris is 25°C and partly cloudy. Enjoy your trip!

Model toolcalls:
None

✅ Weather response looks good.
================================================================================
Test: ❌ Invalid tool output, should re-call
Model output:
None

Model toolcalls:
[ChatCompletionMessageToolCall(id='0', function=Function(arguments='{"city": "New York", "state": "NY", "unit": "celsius"}', name='get_current_weather'), type='function', index=None)]

✅ Correctly did not fabricate answer.
================================================================================
Test: 🔁 Multiple tool calls expected
✅ Model emitted 2 tool calls.
Model output:
The current weather in **Tokyo, Japan** is **77°F** and partly cloudy.  

In **London, UK**, it's also **77°F** with partly cloudy conditions.  

Both cities have similar weather right now!

✅ All cities handled in the response.
✅ All cities handled.
================================================================================
Test: ➡️ Follow-up after tool output
Model output:
None

Model toolcalls:
[ChatCompletionMessageToolCall(id='0', function=Function(arguments='{"city": "London", "state": "England", "unit": "celsius"}', name='get_current_weather'), type='function', index=None)]

✅ Follow-up was understood.

[zhaochenyang20]

I tried to patch #5907 to this PR and did some test on multi-turn.

Test 1: Normal multi-turn Purpose: The model should be able to use the final tools message to generate an answer Script:

adapted from [function_calling.ipynb](https://github.com/sgl-project/sglang/blob/main/docs/backend/function_calling.ipynb) 

import openai
import json
import random
import time
import requests

client = openai.Client(base_url="http://localhost:8080/v1", api_key="xxxxxx")
model_name = client.models.list().data[0].id

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string", "description": "City name"},
                    "state": {"type": "string", "description": "State abbreviation"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["city", "state", "unit"]
            },
        },
    }
]

cities = [
    {"city": "Birmingham", "state": "AL"},
    {"city": "Anchorage", "state": "AK"},
    {"city": "Phoenix", "state": "AZ"},
    {"city": "Little Rock", "state": "AR"},
    {"city": "Los Angeles", "state": "CA"},
    {"city": "Denver", "state": "CO"},
    {"city": "Bridgeport", "state": "CT"},
    {"city": "Wilmington", "state": "DE"},
    {"city": "Miami", "state": "FL"},
    {"city": "Atlanta", "state": "GA"},
    {"city": "Honolulu", "state": "HI"},
    {"city": "Boise", "state": "ID"},
    {"city": "Chicago", "state": "IL"},
    {"city": "Indianapolis", "state": "IN"},
    {"city": "Des Moines", "state": "IA"},
    {"city": "Wichita", "state": "KS"},
    {"city": "Louisville", "state": "KY"},
    {"city": "New Orleans", "state": "LA"},
    {"city": "Portland", "state": "ME"},
    {"city": "Baltimore", "state": "MD"},
    {"city": "Boston", "state": "MA"},
    {"city": "Detroit", "state": "MI"},
    {"city": "Minneapolis", "state": "MN"},
    {"city": "Jackson", "state": "MS"},
    {"city": "Kansas City", "state": "MO"},
    {"city": "Billings", "state": "MT"},
    {"city": "Omaha", "state": "NE"},
    {"city": "Las Vegas", "state": "NV"},
    {"city": "Manchester", "state": "NH"},
    {"city": "Newark", "state": "NJ"},
    {"city": "Albuquerque", "state": "NM"},
    {"city": "New York", "state": "NY"},
    {"city": "Charlotte", "state": "NC"},
    {"city": "Fargo", "state": "ND"},
    {"city": "Columbus", "state": "OH"},
    {"city": "Oklahoma City", "state": "OK"},
    {"city": "Portland", "state": "OR"},
    {"city": "Philadelphia", "state": "PA"},
    {"city": "Providence", "state": "RI"},
    {"city": "Charleston", "state": "SC"},
    {"city": "Sioux Falls", "state": "SD"},
    {"city": "Nashville", "state": "TN"},
    {"city": "Houston", "state": "TX"},
    {"city": "Salt Lake City", "state": "UT"},
    {"city": "Burlington", "state": "VT"},
    {"city": "Virginia Beach", "state": "VA"},
    {"city": "Seattle", "state": "WA"},
    {"city": "Charleston", "state": "WV"},
    {"city": "Milwaukee", "state": "WI"},
    {"city": "Cheyenne", "state": "WY"},
    {"city": "Washington", "state": "DC"}
]

units = ["celsius", "fahrenheit"]

def get_current_weather(city, state, unit):
    if unit == "celsius":
        temp = random.randint(10, 40)
    else:
        temp = random.randint(50, 105)
    return f"The weather in {city}, {state} is {temp} degrees {unit}. It is partly cloudy.", temp

success = 0
fail = 0
num_runs = 100

for i in range(num_runs):
    try:
        loc = random.choice(cities)
        unit = random.choice(units)

        messages = [
            {"role": "system", "content": "You are a travel assistant."},
            {"role": "user", "content": f"I'm planning a trip to {loc['city']} in {loc['state']} state. What's the weather like in {unit}?"},
        ]

        response = client.chat.completions.create(
            model=model_name,
            messages=messages,
            tools=tools,
            stream=False,
            temperature=0.7,
            top_p=0.9,
            max_tokens=1000,
        )

        tool_call = response.choices[0].message.tool_calls[0]
        name = tool_call.function.name
        args = json.loads(tool_call.function.arguments)
        assert name == "get_current_weather"

        tool_call_output, temp = get_current_weather(**args)
        
        tool_call_msg = {
            "id": "0",
            "type": "function",
            "function": {
                "name": name,
                "arguments": str(args)
            }
        }

        messages.append({"role": "assistant", "content": None, "tool_calls": [tool_call_msg]})
        messages.append({"role": "tool", "name": name, "tool_call_id": tool_call.id, "content": tool_call_output})

        final = client.chat.completions.create(
            model=model_name,
            messages=messages,
            temperature=0,
            top_p=0.8,
            stream=False,
            tools=tools, # Result2 are this line commented out
            # max_tokens=300,
        )

        content = final.choices[0].message.content
        if content and str(args["city"]).lower() in content.lower() and str(temp).lower() in content.lower():
            success += 1
        else:
            fail += 1

        print(f"[{i+1}] Success={success} / Fail={fail}\n Function call result: {tool_call_output} \n Answer: {content}\n")

        try:
            requests.post("http://localhost:8080/flush_cache")
        except Exception as e:
            print(f"Warning: failed to flush cache — {e}")

        time.sleep(1)

    except requests.exceptions.ConnectionError or ConnectionError:
        print("Connection error — stopping early.")
        break
    except Exception as e:
        print(f"Unexpected error: {e}")
        fail += 1
        continue

print(f"Final Result: {success} success / {fail} failed out of {num_runs} attempts.")

Results 1:

[97] Success=97 / Fail=0
 Function call result: The weather in Wilmington, DE is 25 degrees celsius. It is partly cloudy. 
 Answer: The current weather in Wilmington, DE is 25°C and partly cloudy. Enjoy your trip!

[98] Success=98 / Fail=0
 Function call result: The weather in Honolulu, HI is 19 degrees celsius. It is partly cloudy. 
 Answer: The current weather in Honolulu, HI is 19°C and partly cloudy. Enjoy your trip!

[99] Success=99 / Fail=0
 Function call result: The weather in Newark, NJ is 56 degrees fahrenheit. It is partly cloudy. 
 Answer: The current weather in Newark, NJ is 56°F and partly cloudy. Enjoy your trip!

[100] Success=100 / Fail=0
 Function call result: The weather in Charleston, WV is 92 degrees fahrenheit. It is partly cloudy. 
 Answer: The current weather in Charleston, WV is 92°F and partly cloudy. Enjoy your trip!

Final Result: 100 success / 0 failed out of 100 attempts.

Results 2: (last turn doesn't pass tools)

[96] Success=96 / Fail=0
 Function call result: The weather in Charleston, SC is 21 degrees celsius. It is partly cloudy. 
 Answer: Currently, the weather in Charleston, SC is **21°C** and partly cloudy. It's a pleasant temperature for exploring the city! If you're planning outdoor activities, you might want to bring a light jacket for the evening, as temperatures can drop slightly. 

Would you like any recommendations for things to do based on this weather?

[97] Success=97 / Fail=0
 Function call result: The weather in Las Vegas, NV is 96 degrees fahrenheit. It is partly cloudy. 
 Answer: The current weather in Las Vegas, NV is **96°F** and partly cloudy. It's going to be a warm day, so be sure to stay hydrated and wear sunscreen if you're planning to be outdoors! Let me know if you'd like any other details for your trip.

[98] Success=98 / Fail=0
 Function call result: The weather in Nashville, TN is 29 degrees celsius. It is partly cloudy. 
 Answer: The current weather in Nashville, Tennessee, is **29°C (84°F)** with partly cloudy skies. It's a warm day, so light clothing and sunscreen would be a good idea if you're planning outdoor activities. Let me know if you'd like a forecast for your travel dates!

[99] Success=99 / Fail=0
 Function call result: The weather in Bridgeport, CT is 50 degrees fahrenheit. It is partly cloudy. 
 Answer: The current weather in Bridgeport, CT is **50°F** and partly cloudy. Enjoy your trip! If you'd like more details or a forecast for specific dates, let me know.

[100] Success=100 / Fail=0
 Function call result: The weather in Las Vegas, NV is 30 degrees celsius. It is partly cloudy. 
 Answer: The current weather in Las Vegas, Nevada is **30°C** and partly cloudy. Enjoy your trip! If you'd like more details or a forecast for specific dates, let me know.

Final Result: 100 success / 0 failed out of 100 attempts.

Test 2: Different scenario of multi-turn conversation Purpose: Model should predict correctly whether to use tool outputs or respond naturally

Script:

import openai
import json

client = openai.Client(base_url="http://localhost:8080/v1", api_key="xxxxxx")
model_name = client.models.list().data[0].id

tools = [{
    "type": "function",
    "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {"type": "string"},
                "state": {"type": "string"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
            },
            "required": ["city", "state", "unit"],
        },
    }
}]

def get_current_weather(city, state, unit):
    if not state:
        return "Error: State must be specified.", None
    temp = 25 if unit == "celsius" else 77
    return f"The weather in {city}, {state} is {temp} degrees {unit}. It is partly cloudy.", temp

test_cases = [
    {
        "name": "✅ Valid tool output, no re-call expected",
        "user": "I'm planning a trip to Paris. What's the weather like in celsius?",
        "tool_args": {"city": "Paris", "state": "FR", "unit": "celsius"},
        "expect_city": "Paris",
        "expect_temp": 25
    },
    {
        "name": "❌ Invalid tool output, should re-call",
        "user": "What's the weather in New York in state NY?",
        "tool_args": {"city": "New York", "state": "", "unit": "celsius"},
        "follow_up": "The state is NY",
        "expect_retry": True
    },
    {
        "name": "🔁 Multiple tool calls expected",
        "user": "What's the weather in Tokyo and London in fahrenheit?",
        "multi_tool_args": [
            {"city": "Tokyo", "state": "TY", "unit": "fahrenheit"},
            {"city": "London", "state": "UK", "unit": "celsius"},
        ],
        "expect_cities": ["Tokyo", "London"]
    },
    {
        "name": "➡️ Follow-up after tool output",
        "user": "What's the weather in Rome?",
        "tool_args": {"city": "Rome", "state": "RM", "unit": "celsius"},
        "follow_up": "What about weather in London?",
        "expect_follow": "London"
    }
]

for case in test_cases:
    print("=" * 80)
    print(f"Test: {case['name']}")
    
    messages = [
        {"role": "system", "content": "You are a travel assistant."},
        {"role": "user", "content": case["user"]}
    ]

    if case["name"].startswith("🔁 Multiple tool calls expected"):
        # First turn: ask the model
        messages = [
            {"role": "system", "content": "You are a travel assistant."},
            {"role": "user", "content": case["user"]}
        ]

        initial = client.chat.completions.create(
            model=model_name,
            messages=messages,
            tools=tools,
            stream=False,
            temperature=0,
            top_p=1,
        )

        tool_calls = initial.choices[0].message.tool_calls
        if not tool_calls or len(tool_calls) < 2:
            print("❌ Model did not emit multiple tool calls.")
            continue
        else:
            print(f"✅ Model emitted {len(tool_calls)} tool calls.")

        tool_outputs = []
        assistant_call = {
            "role": "assistant",
            "content": None,
            "tool_calls": [
                {
                    "id": tc.id,
                    "type": tc.type,
                    "function": {
                        "name": tc.function.name,
                        "arguments": tc.function.arguments
                    }
                }
                for tc in tool_calls
            ]
        }

        messages.append(assistant_call)

        for tc in tool_calls:
            args = json.loads(tc.function.arguments)
            tool_result, temp = get_current_weather(**args)
            tool_outputs.append({
                "role": "tool",
                "tool_call_id": tc.id,
                "name": tc.function.name,
                "content": tool_result
            })
            messages.append(tool_outputs[-1])

        # Ask final generation
        final = client.chat.completions.create(
            model=model_name,
            messages=messages,
            # tools=tools,
            stream=False,
            temperature=0.5,
            top_p=1,
            max_tokens=500,
        )

        output = final.choices[0].message.content
        print(f"Model output:\n{output}\n")

        if all(city.lower() in output.lower() for city in case["expect_cities"]):
            print("✅ All cities handled in the response.")
        else:
            print("❌ Missing one or more cities in the final answer.")

    else:
        tool_output, temp = get_current_weather(**case["tool_args"])
        tool_call_msg = {
            "id": "0",
            "type": "function",
            "function": {
                "name": "get_current_weather",
                "arguments": json.dumps(case["tool_args"])
            }
        }
        messages.append({"role": "assistant", "content": None, "tool_calls": [tool_call_msg]})
        messages.append({
            "role": "tool",
            "name": "get_current_weather",
            "tool_call_id": "0",
            "content": tool_output
        })

        if "follow_up" in case:
            messages.append({"role": "user", "content": case["follow_up"]})

        final = client.chat.completions.create(
            model=model_name,
            messages=messages,
            tools=tools,
            stream=False,
            temperature=0,
            top_p=1,
            max_tokens=500,
        )

        output = final.choices[0].message.content
        tool_calls = final.choices[0].message.tool_calls
        print(f"Model output:\n{output}\n")
        print(f"Model toolcalls:\n{tool_calls}\n")

    # Eval
    if "expect_retry" in case:
        if (output is not None and "state must be specified" in messages[-1]["content"].lower() and "new york" not in output.lower()) or tool_calls is not None:
            print("✅ Correctly did not fabricate answer.")
        else:
            print("❌ Failed to handle tool error properly.")

    elif "expect_cities" in case:
        if all(city.lower() in output.lower() for city in case["expect_cities"]):
            print("✅ All cities handled.")
        else:
            print("❌ Missing one or more city responses.")

    elif "expect_follow" in case:
        if (output is not None and case["expect_follow"] in output.lower()) or tool_calls is not None:
            print("✅ Follow-up was understood.")
        else:
            print("❌ Missed follow-up context.")

    else:
        if case["expect_city"].lower() in output.lower() and str(case["expect_temp"]) in output:
            print("✅ Weather response looks good.")
        else:
            print("❌ Model ignored tool output or responded incorrectly.")

Result:

➜  sglang git:(chang/deepseek-v3-tool-call) ✗ python3 examples/function_tool_call_test2.py
================================================================================
Test: ✅ Valid tool output, no re-call expected
Model output:
The current weather in Paris is 25°C and partly cloudy. Enjoy your trip!

Model toolcalls:
None

✅ Weather response looks good.
================================================================================
Test: ❌ Invalid tool output, should re-call
Model output:
None

Model toolcalls:
[ChatCompletionMessageToolCall(id='0', function=Function(arguments='{"city": "New York", "state": "NY", "unit": "celsius"}', name='get_current_weather'), type='function', index=None)]

✅ Correctly did not fabricate answer.
================================================================================
Test: 🔁 Multiple tool calls expected
✅ Model emitted 2 tool calls.
Model output:
The current weather in **Tokyo, Japan** is **77°F** and partly cloudy.  

In **London, UK**, it's also **77°F** with partly cloudy conditions.  

Both cities have similar weather right now!

✅ All cities handled in the response.
✅ All cities handled.
================================================================================
Test: ➡️ Follow-up after tool output
Model output:
None

Model toolcalls:
[ChatCompletionMessageToolCall(id='0', function=Function(arguments='{"city": "London", "state": "England", "unit": "celsius"}', name='get_current_weather'), type='function', index=None)]

✅ Follow-up was understood.

is it all set?

[CatherineSue]

@zhaochenyang20 yes we can merge this.