proc_macro_attribute literals are tokenized as errors

[eidloi]

rust-analyzer version: v0.3.1791 (stable), v0.4.1793 (Pre-release)

rustc version: rustc 1.74.1 (a28077b28 2023-12-04)

relevant settings: n/a

Imagine a macro in the format of:

#[simple_interaction_for("test")]
struct Test {
    pub test: &'static str,
}

Where the macro implementation is printing the attribute tokens for simplicity:

#[proc_macro_attribute]
pub fn simple_interaction_for(attr: TokenStream, item: TokenStream) -> TokenStream {
    eprintln!("input: {:#?}", attr);
 
    item
}

Expected output:

This is from cargo expand, it generates the expected output and the project builds without errors.

input: TokenStream [
    Literal {
        kind: Str,
        symbol: "test",
        suffix: None,
        span: #0 bytes(159..165),
    },
]

Actual output:

Printed in the Output of VSCode (Rust Analyzer Language Server)

input: TokenStream [
    Literal {
        kind: Err,
        symbol: "\"test\"",
        suffix: None,
        span: 16,
    },
]

The same happens with other literals as well.

Note:

Weirdly enough initially it worked correctly on a few tries (just changing the literal in the annotation prints it again and again), but then got into the error state and never recovered. Re-installing the plugin or switching to the pre-release version didn't help.

[Veykril]

The kind is currently stubbed out in the proc-macro server, hence the Err variant.

[Tyrubias]

@Veykril I'm interested in fixing this issue. I noticed in both the current proc macro server backend and the legacy TokenId backend there's the comment // FIXME: keep track of LitKind and Suffix. Is this where I should start? What kind of tracking do we need to do?

[Veykril]

The comment is a bit confusingly written, rather keeping track of what is meant is we need to parse the string and figure out the kind and suffix from that. Note that this needs to be done both in the leagcy server and the rust_analyzer_span one.Though thats most likely just gonna be duplicated code, there shouldn't be any differences there. Here is what rustc is doing https://github.com/rust-lang/rust/blob/5113ed28ea1451a13eae3a05dca0dbabfd56f587/compiler/rustc_expand/src/proc_macro_server.rs#L444