`bump_glue` `next_byte` disregard `num_tokens`

[sunxd3]

MWE:

julia> ps = ParseStream("a + b - c * d"); peek(ps);

julia> bump_glue(ps, K"Identifier", EMPTY_FLAGS, 4)
JuliaSyntax.ParseStreamPosition(0x00000002, 0x00000000)

julia> ps.tokens[end] |> dump
JuliaSyntax.SyntaxToken
  head: JuliaSyntax.SyntaxHead
    kind: JuliaSyntax.Kind K"Identifier"
    flags: UInt16 0x0000
  orig_kind: JuliaSyntax.Kind K"Identifier"
  preceding_whitespace: Bool false
  next_byte: UInt32 0x00000003

If I understand correctly, I would expect the next_byte to be 0x00000006 as,

julia> JuliaSyntax.peek_token(ps)
Identifier      |6

Maybe at

JuliaSyntax.jl/src/parse_stream.jl

Line 738 in fc572f9

stream.lookahead[i+1].next_byte))

should be

                                     stream.lookahead[i+num_tokens].next_byte))

and maybe bound checking?

This effects build_tree as

JuliaSyntax.jl/src/parse_stream.jl

Line 1075 in fc572f9

stream.tokens[end].next_byte - 1)

[c42f]

bump_glue is a bit of an unfortunate anomaly that I wish I could remove; it only needs to exist to resolve a lexing ambiguity with signed numbers:

• -2 means the literal integer -2
• -2*x means the infix call (call-i -2 x)
• However -2^x means (call-pre - (call-i 2 ^ x)) due to precedence (See #julialang/julia/18851)

Anyway, so it seems the easiest way forward here is to just remove the num_tokens parameter entirely, as we only ever use bump_glue with num_tokens == 2.

[sunxd3]

That sounds reasonable to me. PR #338