You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
May sglang support automatically deduce output token num based on context length. It seems that I need to manually specify max_tokens cause I always get incomplete responses without this parameter. However, in my scenario, it's hard to figure out the max_token I need. It would be much more convenient if sglang could automatically deduce it.
Usually specifying a punctuation stop token is sufficient. I've found that leading the model with an ' is enough to make it want to close the quote and works well.
For example:
s+='Given the context above, the the most relevant section is "'+sgl.gen("SECTION", stop_token="'", max_token=99999)
s+="'."
Other punctuation I've found works well: [], {}.\n.
Usually specifying a punctuation stop token is sufficient. I've found that leading the model with an ' is enough to make it want to close the quote and works well.
For example:
s+='Given the context above, the the most relevant section is "'+sgl.gen("SECTION", stop_token="'", max_token=99999)
s+="'."
Other punctuation I've found works well: [], {}.\n.
If I simply specify max_token=99999 without stop_token, would generation stop if it reaches max context length?
Depends on the model I think. Sometimes if you have no idea how long it is, you could try starting the prompt off in XML tags, and in the instructions asking the model to close it by providing an example. Something like this?
Activity
deoxynet commentedon Apr 11, 2024
Usually specifying a punctuation stop token is sufficient. I've found that leading the model with an
'is enough to make it want to close the quote and works well.For example:
Other punctuation I've found works well:
[],{}.\n.for-just-we commentedon Apr 12, 2024
If I simply specify
max_token=99999withoutstop_token, would generation stop if it reaches max context length?deoxynet commentedon Apr 12, 2024
Depends on the model I think. Sometimes if you have no idea how long it is, you could try starting the prompt off in XML tags, and in the instructions asking the model to close it by providing an example. Something like this?
merrymercy commentedon Jul 9, 2024
The correct usage is to specify a large enough
max_tokensandstopcondition. Some chat-tuned models will also stop by themselves withEOS` tokens.Revert "add: w4a8 group gemm (sgl-project#353)" (sgl-project#360)
Support grok2 (sgl-project#360)
Merge fix/int8-quality-360: graded quality harness, INT8-W8A8 passes …
sgl-project#108: cause verdicts -- (1) and (2) read clean, drift is t…
sgl-project#108: the rule is two-sided -- above kv<tp, replicated is …
[sgl-project#584] Window verdict: the rates are measured, and the gov…