Chuck does not remove preceding space

[zmcgohan2]

"chuck ice"

[pokey]

Thanks for filing this one. We do need to make our space removal a touch more aggressive. The original implementation was too aggressive, and resulted in merged tokens, but I think we ended up pulling back a bit too far.

The policy I was thinking is as follows:

1. If there are spaces on either side of the token, always remove one of them
2. If the token is at start / end of a line (ignoring leading white space) and there is a space on the other side of it, remove the space
3. Otherwise, if there is space on just one side of the token, remove it if and only if doing so will not cause the newly adjacent tokens to become a single token. For example, we don't want to remove the space if it ends up separating two alphanumeric tokens

Make sense? Cc/ @AndreasArvidsson

[AndreasArvidsson]

One and two is already implemented. What is missing is actually to check the tokenizer if we can remove the white space between two tokens without them becoming single. It not as easy as just check for alphanumerics because we have many more cases where two tokens can become one if you remove the space.
Easier solution is probably just pass the suggested change to the tokenizer and just see the result

[pokey]

Yep that's what I was thinking too 👍

One other thought is that we may want to do the check a bit further down the pipeline, because today's space removal can end up merging tokens if eg you say "chuck past air" and your cursor is sitting at the end of a token. For example, in the string big apple pie, if your cursor is at the end of big and you say "chuck past air", it will end up as bigpie

[pokey]

This already works today; was fixed by #1028