[RFE] Support for tools in streaming mode

[jesuino]

Feature request description

I can't use tools with streaming, it always fails with the following error:

{"code":500,"message":"Cannot use tools with stream","type":"server_error"}

Steps to reproduce:

• Pull some model that supports tool (e.g. Qwen3:8b)
• Serve the model
• Use a client with tools in streaming mode (I used specifically the Java Langchain4j OpenAPI client)

Suggest potential solution

No response

Have you considered any alternatives?

No response

Additional context

No response

[rhatdan]

Could you show precisely the command you are executing?

[ericcurtin]

We need this merged in llama.cpp first:

ggml-org/llama.cpp#12379

[jesuino]

Thanks, I was suspecting that it was at llama.cpp level

@rhatdan I am actually using the serve mode. Here's my HTTP request:

2025-05-22 11:33:44,113 DEBUG [dev.lan.htt.cli.log.LoggingHttpClient] (JavaFX Application Thread) HTTP request:
- method: POST
- url: http://localhost:8080/chat/completions
- headers: [Authorization: Beare...r ], [User-Agent: langchain4j-openai], [Content-Type: application/json]
- body: {
  "model" : "qwen/Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf:latest",
  "messages" : [ {
    "role" : "assistant",
    "content" : ""
  }, {
    "role" : "user",
    "content" : "What day is today"
  } ],
  "stream" : true,
  "stream_options" : {
    "include_usage" : true
  },
  "tools" : [ {
    "type" : "function",
    "function" : {
      "name" : "currentDateTimeZone",
      "description" : "Returns the current date and time with timezone",
      "parameters" : {
        "type" : "object",
        "properties" : { },
        "required" : [ ]
      }
    }
  }, {
    "type" : "function",
    "function" : {
      "name" : "currentTime",
      "description" : "Returns the current time. Responds to question: what time is it?",
      "parameters" : {
        "type" : "object",
        "properties" : { },
        "required" : [ ]
      }
    }
  }, {
    "type" : "function",
    "function" : {
      "name" : "currentDateTime",
      "description" : "Returns the current date and time",
      "parameters" : {
        "type" : "object",
        "properties" : { },
        "required" : [ ]
      }
    }
  }, {
    "type" : "function",
    "function" : {
      "name" : "currentYear",
      "description" : "Returns the current year. Use this if user asks for the current year or mention 'this year'",
      "parameters" : {
        "type" : "object",
        "properties" : { },
        "required" : [ ]
      }
    }
  }, {
    "type" : "function",
    "function" : {
      "name" : "currentDate",
      "description" : "Returns the current date including the year, month and day. It is the equivalent of user asking for today's date or current year",
      "parameters" : {
        "type" : "object",
        "properties" : { },
        "required" : [ ]
      }
    }
  } ]
}

[cgruver]

@jesuino I am keeping a llama.cpp branch here - https://github.com/cgruver/llama.cpp/tree/tools

It is the code in ggml-org/llama.cpp#12379 that I am trying to keep merged with upstream https://github.com/ggml-org/llama.cpp

If you want, you can build your own Ramalama with my fork to test it out.

I'm currently using it to test AI code assistants.