Fix incomplete_tag generation with trailing whitespace
#518
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously we relied on the last SyntaxNode consuming all trailing whitespace when detecting incomplete syntax. However this assumption was broken by #397 and is generally fragile with respect to any extra bump_trivia() calls. Fix this by just comparing to the stream position before any bumping of remaining trivia.
c42f
force-pushed
the
caf/fix-incomplete-tag-with-whitespace
branch
from
e5e19c1 to
9fcc903
Compare
|
The additional test cases I had were "CompletionFoo.test(1, 1, " (which is the one that base depends on so good to CI test here) and "[x)". They both look good here so LGTM. |
|
Ah now I see. The But the JuliaSyntax tests are drawn from Base tests in test/client.jl (plus additional cases I constructed). |
c42f
added a commit
that referenced
this pull request
Dec 8, 2024
Previously we relied on the last SyntaxNode consuming all trailing whitespace when detecting incomplete syntax. However this assumption was broken by #397 and is generally fragile with respect to any extra bump_trivia() calls. Fix this by just comparing to the stream position before any bumping of remaining trivia.
c42f
added a commit
that referenced
this pull request
Feb 13, 2025
Rework incomplete expression detection so that trailing whitespace is always ignored, regardless of how the parser itself decides to attach it to other nodes of the tree. To do this, we walk back from the end of the parse stream and look for the byte offset of the last non-whitespace token. We then use that to determine whether the error node is "at the end of the parse". Improve testing by * Extracting the incomplete expressions which are part of the REPL stdlib tests and ensuring these match the incomplete tag generation of the flisp parser. Fix some divergences for `var""` syntax and invalid escape sequences in strings. * Ensuring that we test both `:statement` and `:all` level parsing - the REPL uses `:all` to allow parsing of multiple top level statements, so we need to test this. Also fix a minor bug where `enable_in_core!(false)` would result in the flisp parser being used, regardless of whether `VERSION` ships with JuliaSyntax enabled by default. Fixes #519. See also #518.
c42f
added a commit
that referenced
this pull request
Feb 13, 2025
Rework incomplete expression detection so that trailing whitespace is always ignored, regardless of how the parser itself decides to attach it to other nodes of the tree. To do this, we walk back from the end of the parse stream and look for the byte offset of the last non-whitespace token. We then use that to determine whether the error node is "at the end of the parse". Improve testing by * Extracting the incomplete expressions which are part of the REPL stdlib tests and ensuring these match the incomplete tag generation of the flisp parser. Fix some divergences for `var""` syntax and invalid escape sequences in strings. * Ensuring that we test both `:statement` and `:all` level parsing - the REPL uses `:all` to allow parsing of multiple top level statements, so we need to test this. Also fix a minor bug where `enable_in_core!(false)` would result in the flisp parser being used, regardless of whether `VERSION` ships with JuliaSyntax enabled by default. Fixes #519. See also #518.
c42f
added a commit
that referenced
this pull request
Feb 13, 2025
Rework incomplete expression detection so that trailing whitespace is always ignored, regardless of how the parser itself decides to attach it to other nodes of the tree. To do this, we walk back from the end of the parse stream and look for the byte offset of the last non-whitespace token. We then use that to determine whether the error node is "at the end of the parse". Improve testing by * Extracting the incomplete expressions which are part of the REPL stdlib tests and ensuring these match the incomplete tag generation of the flisp parser. Fix some divergences for `var""` syntax and invalid escape sequences in strings. * Ensuring that we test both `:statement` and `:all` level parsing - the REPL uses `:all` to allow parsing of multiple top level statements, so we need to test this. Also fix a minor bug where `enable_in_core!(false)` would result in the flisp parser being used, regardless of whether `VERSION` ships with JuliaSyntax enabled by default. Fixes #519. See also #518.
c42f
added a commit
that referenced
this pull request
Feb 13, 2025
Rework incomplete expression detection so that trailing whitespace is always ignored, regardless of how the parser itself decides to attach it to other nodes of the tree. To do this, we walk back from the end of the parse stream and look for the byte offset of the last non-whitespace token. We then use that to determine whether the error node is "at the end of the parse". Improve testing by * Extracting the incomplete expressions which are part of the REPL stdlib tests and ensuring these match the incomplete tag generation of the flisp parser. Fix some divergences for `var""` syntax and invalid escape sequences in strings. * Ensuring that we test both `:statement` and `:all` level parsing - the REPL uses `:all` to allow parsing of multiple top level statements, so we need to test this. Also fix a minor bug where `enable_in_core!(false)` would result in the flisp parser being used, regardless of whether `VERSION` ships with JuliaSyntax enabled by default. Fixes #519. See also #518.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously we relied on the last SyntaxNode consuming all trailing whitespace when detecting incomplete syntax. However this assumption was broken by #397 and is generally fragile with respect to any extra bump_trivia() calls. Fix this by just comparing to the stream position before any bumping of remaining trivia.
Should fix #512