Fix #236: treat max_tokens=0 as empty string - #356
Conversation
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Fixes compilation/runtime behavior for Lark rules configured with max_tokens=0, aligning it with “empty rule” semantics and preventing non-terminating matcher output.
Changes:
- Add a regression test asserting
max_tokens=0behaves like an explicitly empty rule. - Update the compiler to treat
max_tokens=0rules as matching the empty string (compile to"").
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| parser/tests/test_ll.rs | Adds regression coverage for max_tokens=0 producing an empty match. |
| parser/src/lark/compiler.rs | Implements the max_tokens=0 special-case by compiling it as an empty string. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // max_tokens=0 means the rule is allowed to emit no tokens at all, | ||
| // i.e. it matches only the empty string. Compiling it as a |
There was a problem hiding this comment.
Ok. reworded to make the cap semantics explicit: max_tokens=N caps a rule at N emitted tokens, so max_tokens=0 forces exactly zero (epsilon) rather than being optional.
| fn test_ll_max_tokens_zero() { | ||
| // max_tokens=0 means the rule emits no tokens, i.e. it matches the empty | ||
| // string -- so `name` contributes nothing and we go straight to "xy". | ||
| // See issue #236. |
There was a problem hiding this comment.
done. switched to the full GitHub URL to match the compiler-side comment.
A rule with max_tokens=0 should be allowed to emit no tokens, i.e. match only the empty string. It was instead compiled to a token-limited lexeme that never terminates at runtime (emits an opening token then loops). Special-case max_tokens=0 to compile to "", uniformly across terminal, subgrammar and json bodies. Adds a regression test. Fixes guidance-ai#236
c6f1a03 to
e17ba90
Compare
|
Thank you for the fix! Looks clean to me. Running CI |
max_tokens=0was compiled into a token-limited lexeme that never terminates at runtime (the{-then-garbage symptom in #236). Since a rule capped at zero tokens can only match the empty string, this special-cases it to compile to"", placed before the stop/terminal/subgrammar dispatch so it's handled uniformly for all body types. Includes a regression test (test_ll_max_tokens_zero) with the issue's repro plus aname: ""equivalence baseline. Full parser suite + fmt + clippy clean.