Commit 2eca43c
committed
Fix critical oversight in lexer buffer refilling
Since the lexer buffer wraps, the refilling gets handled in two steps:
First, iff the buffer would wrap, the buffer is refilled until its end.
Then, if more characters are requested, that amount is refilled too.
An important detail is that `read()` may not return as many characters as
requested; for this reason, the first step checks if its `read()` was
"full", and skips the second step otherwise.
This is also where a bug lied.
After a *lot* of trying, I eventually managed to reproduce the bug on an
OpenBSD VM, and after adding a couple of `assert`s in `peekInternal`, this
is what happened, starting at line 724:
0. `lexerState->nbChars` is 0, `lexerState->index` is 19;
1. We end up with `target` = 42, and `writeIndex` = 19;
2. 42 + 19 is greater than `LEXER_BUF_SIZE` (= 42), so the `if` is entered;
3. Within the first `readChars`, **`read` only returns 16 bytes**,
advancing `writeIndex` to 35 and `target` to 26;
4. Within the second `readChars`, a `read(26)` is issued, overflowing the
buffer.
The bug should be clear now: **the check at line 750 failed to work!** Why?
Because `readChars` modifies `writeIndex`.
The fix is simply to cache the number of characters expected, and use that.1 parent c246942 commit 2eca43c
1 file changed
+6
-2
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
729 | 729 | | |
730 | 730 | | |
731 | 731 | | |
| 732 | + | |
| 733 | + | |
732 | 734 | | |
733 | 735 | | |
734 | 736 | | |
| |||
741 | 743 | | |
742 | 744 | | |
743 | 745 | | |
744 | | - | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
745 | 749 | | |
746 | | - | |
| 750 | + | |
747 | 751 | | |
748 | 752 | | |
749 | 753 | | |
| |||
0 commit comments