Fix multiple unnecessary calls to strnom::skip_whitespace

[zzau13]

#126

fn main() {
    let path = "../rust/src/librustc_ast/util/parser.rs";
    let content = std::fs::read_to_string(path).unwrap();
    for _ in 0..1000 {
        content.parse::<proc_macro2::TokenStream>().unwrap();
    }
}

cargo build --release  
valgrind --tool=callgrind target/release/bench_proc
callgrind_annotate --auto=yes callgrind.out.*

This one step, it eat about 300M ir.

I can improve it much more but it needs a refactor, the alt! macros they are not efficient and the common logic is repeated many times, the most called skip_whitespace but there are also others like fallback::token_kind, fallback::op_char ...

If you like, I continue.

[dtolnay]

Thanks!

Some notes below.

[dtolnay]

Is this needed? It looks like input.is_empty() would only be true at most once in the entire parse so I don't see a way that this would affect performance. If you omit it, the next loop iteration will fail to parse a token_tree and break, which is still correct behavior.

[dtolnay]

Help me understand this change: This causes all whitespace and comments to be parsed twice as many times, doesn't it? The previous behavior made 3 calls to skip_whitespace but only 1 of them would do any work. The new behavior makes 2 calls to skip_whitespace where the second one redoes all the work already done by the first one. Seems like which one is better would depend heavily on the characteristics of the input.

Instead we might want to split some of these functions like what we have for symbol_leading_ws vs symbol below, and always call the right one depending on whether whitespace has already been processed at the call site.

[zzau13]

Yes, it should go where it was.
I want to remove the call to skip_whitespace at spanned. I just removed it to make sure it worked since the logic is complex.

[dtolnay]

This function should be deleted if it no longer does anything different from symbol.

[zzau13]

Even though the use of whitespace is still very exaggerated, I don't want to change too much at once to avoid breaking anything. I have left it in a single place in token_stream. The results:

Old

Ir
--------------------------------------------------------------------------------
4,866,615,165  PROGRAM TOTALS

--------------------------------------------------------------------------------
Ir             file:function
--------------------------------------------------------------------------------
2,004,232,000  ???:proc_macro2::strnom::whitespace

New

Ir
--------------------------------------------------------------------------------
4,334,090,565  PROGRAM TOTALS

--------------------------------------------------------------------------------
Ir             file:function
--------------------------------------------------------------------------------
1,541,298,000  ???:proc_macro2::strnom::whitespace

[dtolnay]

Thank you!