Skip to content

About parameter max_tokens #360

Closed
Closed

Description

@for-just-we

May sglang support automatically deduce output token num based on context length. It seems that I need to manually specify max_tokens cause I always get incomplete responses without this parameter. However, in my scenario, it's hard to figure out the max_token I need. It would be much more convenient if sglang could automatically deduce it.

Activity

  1. deoxynet commented on Apr 11, 2024

    @deoxynet

    Usually specifying a punctuation stop token is sufficient. I've found that leading the model with an ' is enough to make it want to close the quote and works well.

    For example:

    s += 'Given the context above, the the most relevant section is "' + sgl.gen("SECTION", stop_token="'", max_token=99999)
    s += "'."

    Other punctuation I've found works well: [], {} .\n.

  2. for-just-we commented on Apr 12, 2024

    @for-just-we
    ContributorAuthor

    Usually specifying a punctuation stop token is sufficient. I've found that leading the model with an ' is enough to make it want to close the quote and works well.

    For example:

    s += 'Given the context above, the the most relevant section is "' + sgl.gen("SECTION", stop_token="'", max_token=99999)
    s += "'."

    Other punctuation I've found works well: [], {} .\n.

    If I simply specify max_token=99999 without stop_token, would generation stop if it reaches max context length?

  3. deoxynet commented on Apr 12, 2024

    @deoxynet

    Depends on the model I think. Sometimes if you have no idea how long it is, you could try starting the prompt off in XML tags, and in the instructions asking the model to close it by providing an example. Something like this?

    s += "<your instructions here>"
    s += "<your examples here>"
    s  += "<context here>"
    ---
    s += 'Summarize the context above:\n<summary>\n'
    s += sgl.gen("SUMMARY", stop_token="\n</summary>", max_token=99999)
    s += "\n</summary>"
  4. merrymercy commented on Jul 9, 2024

    @merrymercy
    Collaborator

    The correct usage is to specify a large enough max_tokens and stop condition. Some chat-tuned models will also stop by themselves with EOS` tokens.

  5. added a commit that references this issue on Jan 9, 2026
  6. added 3 commits that reference this issue on Jul 31, 2026
  7. added a commit that references this issue on Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions