rustc: Fix joint-ness of stringified token-streams #50838
Conversation
|
r? @eddyb (rust_highfive has picked a reviewer for you, use r? to override) |
rust-highfive
added
the
S-waiting-on-review
|
@bors r+ |
|
📌 Commit dda8797 has been approved by |
bors
added
S-waiting-on-bors
|
☔ The latest upstream changes (presumably #50566) made this pull request unmergeable. Please resolve the merge conflicts. |
bors
added
S-waiting-on-author
This commit fixes `StringReader`'s parsing of tokens which have been stringified through procedural macros. Whether or not a token tree is joint is defined by span information, but when working with procedural macros these spans are often dummy and/or overridden which means that they end up considering all operators joint if they can! The fix here is to track the raw source span as opposed to the overridden span. With this information we can more accurately classify `Punct` structs as either joint or not. Closes rust-lang#50700
|
@bors: r=eddyb |
|
📌 Commit 0ee031a has been approved by |
bors
added
S-waiting-on-bors
rustc: Fix joint-ness of stringified token-streams This commit fixes `StringReader`'s parsing of tokens which have been stringified through procedural macros. Whether or not a token tree is joint is defined by span information, but when working with procedural macros these spans are often dummy and/or overridden which means that they end up considering all operators joint if they can! The fix here is to track the raw source span as opposed to the overridden span. With this information we can more accurately classify `Punct` structs as either joint or not. Closes #50700
|
☀️ Test successful - status-appveyor, status-travis |
| @@ -121,6 +132,7 @@ impl<'a> StringReader<'a> { | |||
| sp: self.peek_span, | |||
| }; | |||
| self.advance_token()?; | |||
| self.span_src_raw = self.peek_span_src_raw; | |||
There was a problem hiding this comment.
Heh, I've just spend like an hour perusing this line, and I came to the conclusion that:
- this code is buggy
- this code is observationaly correct
The code is buggy because, while span and token point to this token, span_src points to the next token. In other words, span_src_raw and peek_span_src_raw are just equal all the time, because, first advance_token sets peek_span_src_raw, and then we assign peek_span_src_raw to span_src_raw.
However, this bug does not show up, because we only use peek span to check for jointness. And if spans A and B are adjacent, than spans following A and B are adjacent as well.
In other words, for ++ we don't check that first + is adjacent to the next, we check that the token following the first + (the second +) is adjacent to the token, following the second + (whitespace).
Similarly, for + + we actually check that whitespace following + is not adjacent.
This commit fixes
StringReader's parsing of tokens which have been stringifiedthrough procedural macros. Whether or not a token tree is joint is defined by
span information, but when working with procedural macros these spans are often
dummy and/or overridden which means that they end up considering all operators
joint if they can!
The fix here is to track the raw source span as opposed to the overridden span.
With this information we can more accurately classify
Punctstructs as eitherjoint or not.
Closes #50700