Misc. bug: server provides strutured output for response_format: json_object, but not for response_format: json_schema

[andysalerno]

Name and Version

on latest commit ce8784b

but have been noticing it for several days now

Operating systems

No response

Which llama.cpp modules do you know to be affected?

No response

Problem description & steps to reproduce

I have been trying to follow the steps for structured json output using json_schema in server here

however, I was not able to get any combination of json_schema to work. I was able to get json_object to work, doing effectively the same thing, but since this differs from the OpenAI API (which I suppose server is striving for) I suppose it's a bug. The official server docs also mention json_schema is supported (see the above link)

Does not work, does not apply any structured json schema at all, and does not indicate any failure or warning (simply returns unstructured output), taken directly from OpenAI's docs (click "curl" for the curl version)

    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "math_reasoning",
        "schema": {
          "type": "object",
          "properties": {
            "steps": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "explanation": {
                    "type": "string"
                  },
                  "output": {
                    "type": "string"
                  }
                },
                "required": [
                  "explanation",
                  "output"
                ],
                "additionalProperties": false
              }
            },
            "final_answer": {
              "type": "string"
            }
          },
          "required": [
            "steps",
            "final_answer"
          ],
          "additionalProperties": false
        },
        "strict": true
      }
    }

however, "json_object" works as expected

 "response_format": {
    "type": "json_object",
    "schema": {
        "type": "object",
        "properties": {
            "steps": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "explanation": {
                            "type": "string"
                        },
                        "output": {
                            "type": "string"
                        }
                    },
                    "required": [
                        "explanation",
                        "output"
                    ],
                    "additionalProperties": false
                }
            },
            "final_answer": {
                "type": "string"
            }
        },
        "required": [
            "steps",
            "final_answer"
        ],
        "additionalProperties": false
    }
}

First Bad Commit

No response

Relevant log output

No response

[ggerganov]

Should be an easy fix - please submit a patch

[VJHack]

@andysalerno This issue was already addressed in #9527. Here are the commands of a basic example and outputs when I run it for both json_schema and json_object.

Command for json_schema:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer no-key" -d '{
    "model": "tinyllama",
    "messages": [
        {"role": "system", "content": "Extract the event information."},
        {"role": "user", "content": "Alice and Bob are going to a science fair on Friday."}
    ],
   "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "calendar event",
        "schema": {
          "type": "object",
          "properties": {
            "participants": {
              "type": "array",
              "items": {
                "type": "string",
                "additionalProperties": false
              }
            },
            "name":{
              "type": "string"
            },
            "date":{
              "type": "string"
            }
          },
          "required": ["date", "name", "participants"],
          "additionalProperties": false
        },
        "strict": true
      }
    }
}' | jq ".choices[0].message.content" | jq -r "." | jq .

Output:

{
  "participants": [
    "Alice",
    "Bob"
  ],
  "name": "Science fair",
  "date": "Friday, March 14, 2021 at 03:00 PM (UTC)"
}

Command for json_object:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer no-key" -d '{
    "model": "tinyllama",
    "messages": [
        {"role": "system", "content": "Extract the event information."},
        {"role": "user", "content": "Alice and Bob are going to a science fair on Friday."}
    ],
    "response_format": {
        "type": "json_object",
        "schema": {
            "type": "object",
            "properties": {
                "participants": {
                    "type": "array",
                    "items": {
                        "type": "string",
                        "additionalProperties": false
                    }
                },
                "name": {
                    "type": "string"
                },
                "date": {
                    "type": "string"
                } 
          },
            "required": [
                "participants",
                "name",
                "date"
            ],
            "additionalProperties": false
        }
    }
}' | jq ".choices[0].message.content" | jq -r "." | jq .

Output:

{
  "participants": [
    "Alice",
    "Bob"
  ],
  "name": "Science Fair",
  "date": "18-11-2021"
}

Both examples here follow the schema defined in the request. Please let me know if the docs need to be updated. It would help if you could share some terminal output or logs.

[betweenus]

In b4739 json_schema not working

[arnfaldur]

It does indeed seem to not be working.

This response_format:

{
  "type": "json_object",
  "schema": { ... },
}

works fine. But I could not get "type": "json_schema" to control the output.

[betweenus]

This behavior also described in #11988