Skip to content

Can SGL generate list of json? #53

Closed

Description

@CSWellesSun

I want to generate the following format, that is, list of jsons:
[
{"name": "Alice", "age": 1},
{"name": "Bob", "age": 2},
]
The number of the objects in the list is random depending on the output of LLM. So can SGL support such format?

Activity

  1. hnyls2002 commented on Jan 19, 2024

    @hnyls2002
    Collaborator

    Of course, you can. Just specify the regex of the json list. Here is a simple example code:

    from sglang import function, gen, set_default_backend, Runtime
    
    
    JSON_ITEM_REGEX = r"{\"name\": \"\w+\", \"age\": \d+}"
    
    JSON_LIST_REGEX = r"\[\n\s*(" + JSON_ITEM_REGEX + r",?\n\s*" + ")*" + r"\]"
    
    
    @function
    def regex_gen(s):
        s += "Q: Show me a example of a list of JSON items, where each item has a name and an age.\n"
        s += "Here is an example:\n"
        s += """[
        {"name": "Alice", "age": 1},
        {"name": "Bob", "age": 2},
    ]\n\n
    """
        s += "A: \n" + gen(
            "answer",
            temperature=0,
            regex=JSON_LIST_REGEX,
            max_tokens=100,
        )
    
    
    runtime = Runtime(model_path="meta-llama/Llama-2-7b-chat-hf")
    set_default_backend(runtime)
    
    state = regex_gen.run()
    
    print(state.text())
    
    runtime.shutdown()

    And the result output is:

    Q: Show me a example of a list of JSON items, where each item has a name and an age.
    Here is an example:
    [
        {"name": "Alice", "age": 1},
        {"name": "Bob", "age": 2},
    ]
    
    
    A: 
    [
        {"name": "Alice", "age": 1},
        {"name": "Bob", "age": 2},
        {"name": "Charlie", "age": 3},
    ]
  2. hnyls2002 commented on Jan 19, 2024

    @hnyls2002
    Collaborator

    However, letting LLM control the generated number of JSON items may be unreliable. By calling sgl.gen() with only the JSON item regex multiple times, you can decode whatever number you want.

  3. CSWellesSun commented on Jan 19, 2024

    @CSWellesSun
    ContributorAuthor

    Guidance provides better primitives such as zero_or_more and one_or_more, so will sgl supports such primitives like this: guidance-ai/guidance#402 (comment)

  4. hnyls2002 commented on Jan 19, 2024

    @hnyls2002
    Collaborator

    The primitive zero_or_more is more like a syntax sugar. We now only support constraint decoding with raw regex format; the higher-level primitives may be supported in the future, not only the primitives in this issue; you can see #39.

    The guidance implements this with its recursively select: https://github.com/guidance-ai/guidance/blob/d36601b62096311988fbba1ba15ae4126fb695df/guidance/library/_one_or_more.py#L5

    As for now, you can write it like this in our sglang frontend:

    JSON_ITEM_REGEX = r'(\s)*{\"name\": \"\w+\", \"age\": \d+},\n'
    
    def one_or_more_wrapper(regex_string):
        return r'(' + regex_string + r')+'
    
    ...
        s += (
            "A: \n[\n"
            + gen(
                "answer",
                temperature=0,
                regex=one_or_more_wrapper(JSON_ITEM_REGEX),
                max_tokens=100,
            )
            + "\n]"
        )
    ...

    If you are worried that the generated JSON item would be truncated due to the length limit, maybe in the future, we can have a mechanism to guarantee the integrity for some simple primitives like one_or_more, zero_or_more and ends_with.

  5. Vvvvhvh commented on Jan 19, 2024

    @Vvvvhvh

    Hi

  6. CSWellesSun commented on Jan 23, 2024

    @CSWellesSun
    ContributorAuthor

    The primitive zero_or_more is more like a syntax sugar. We now only support constraint decoding with raw regex format; the higher-level primitives may be supported in the future, not only the primitives in this issue; you can see #39.

    The guidance implements this with its recursively select: https://github.com/guidance-ai/guidance/blob/d36601b62096311988fbba1ba15ae4126fb695df/guidance/library/_one_or_more.py#L5

    As for now, you can write it like this in our sglang frontend:

    JSON_ITEM_REGEX = r'(\s)*{\"name\": \"\w+\", \"age\": \d+},\n'
    
    def one_or_more_wrapper(regex_string):
        return r'(' + regex_string + r')+'
    
    ...
        s += (
            "A: \n[\n"
            + gen(
                "answer",
                temperature=0,
                regex=one_or_more_wrapper(JSON_ITEM_REGEX),
                max_tokens=100,
            )
            + "\n]"
        )
    ...

    If you are worried that the generated JSON item would be truncated due to the length limit, maybe in the future, we can have a mechanism to guarantee the integrity for some simple primitives like one_or_more, zero_or_more and ends_with.

    I hope to realize list of jsons and for every single json, I need to get its value during generation. But if I write the whole structure in one gen, I'm not able to get the value. I hope to realize the following structure:

    @sgl.function
    def single(s):
        s += "{"
        s += "'name': " + sgl.gen("name") + ","
        if s["name"] == "A":
            s += "'age': " + sgl.gen("age")
        else:
            s += "'phone': " + sgl.gen("phone")
        s += "}"
    
    @sgl.function
    def multiple(s):
        s += "["
        s += zero_or_more(single(s))
        s += "]"

    Besides, using regular expression has the following drawbacks:

    1. Regular expression can not save tokens during generation. For example, 'age:' in the above example don't need to be generated, which can save some time.
    2. The above regular expression will lead to a wrong format of list of jsons, that is, the last json of the list still ends with a comma, which is incorrect.
  7. github-actions commented on Jul 25, 2024

    @github-actions
    Contributor

    This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.

  8. Adenialzz commented on Nov 20, 2024

    @Adenialzz

    can this be achieved using json schema like this ?

    json_schema = json.dumps(
        {
            "type": "object",
            "properties": {
                "name": {"type": "string", "pattern": "^[\\w]+$"},
                "population": {"type": "integer"},
            },
            "required": ["name", "population"],
        }
    )
  9. 6 remaining items

  10. added 2 commits that reference this issue on Jan 9, 2026
  11. added a commit that references this issue on Apr 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions