[FRAME core] ☂ Benchmark syntax V2 follow-ups

[ggwpez]

After paritytech/substrate#12924 is done, some things to consider (cc @sam0x17):

• Remove code duplication with old macro expansion code.
• migrate new benchmarking syntax from frame_support::benchmarking to frame_benchmarking::v2 substrate#13235
• #13277
• benchmarking v2 syntax is confusingly lax about return types, leading to confusion substrate#13278
• benchmarking v2 syntax does not issue warnings on unused benchmark params substrate#13303
• generation of real benchmark functions for benchmarking v2 substrate#13224
• add _(force_origin) support to benchmarking v2 substrate#13721
• FRAME: Avoid boilerplate weight macros #205
• bench syntax v2: run test in single functions #376
• Testing
  • Migrate more pallets so we can be sure that we cover everything with the new syntax.
  • Check for helpful error messages, maybe add help/tutorial links.
  • Pallet UI tests, especially if we add new V2-only syntax.
• In-code docs
  • Allow /// comments on benchmarks. substrate#13383
  • Carrying over benchmark comments into the metadata and output them into the WeightInfo trait.
  • Expose benchmark functions to rust doc. I don think there is a good way of doing this though, since we want them private… maybe having a benchmarks::doc crate which only contains the docs could work.
• Syntax extensions
  • Try if the #[must_use] can work by using Clippy #[must_use] for benchmarking components #285 or manually.
  • Modern version of impl_benchmark_test_suite!.
  • Consider multi-char component names
  • Add #[pov_mode] attribute from Add Proof Size to Weight Output substrate#11637
• External docs
  • Either extend pallet-basic-examples or create a new examples pallet to show off all the V2 syntax as reference.
  • Check all README docs, substrate docs and docs.rs stuff
  • Update Tutorials
• Announce to Community (should do this once we are sure not to have breaking syntax changes)

[sam0x17]

@ggwpez as part of this, do we also want callable benchmark functions based on the existing function defs? I was contemplating just taking the function defs, replacing the Linear components with things like x: u32, and adding a verify boolean to the end of the signature

So this:

#[benchmark]
fn my_benchmark(x: Linear<1, 100>) {
    let caller = whitelisted_caller();
    for i in 0..x {
        do_something();
    }

    #[extrinsic_call]
    _(RawOrigin::Signed(caller.clone()), something, something);

    assert_eq!(something, something);
}

would expand to something like this (in addition to all the traits and structs it already expands to):

#[benchmark]
fn my_benchmark(x: u32, verify: bool) {
    let caller = whitelisted_caller();
    for i in 0..x {
        do_something();
    }

    my_benchmark(RawOrigin::Signed(caller.clone()), something, something);

    if verify {
        assert_eq!(something, something);
    }
}

Then the functions would be acessible directly from the enclosing module like:

benchmarks::my_benchmark(33, true);

I don't know if this would work in all cases. One of the trait impls already basically does this, but I figured people are going to be confused if they realize these function defs don't actually become function defs, so this would fix that

We could also preserve doc comments on these function defs (in addition to putting them on weightinfo, etc)

[ggwpez]

I don't know if this would work in all cases. One of the trait impls already basically does this, but I figured people are going to be confused if they realize these function defs don't actually become function defs, so this would fix that

Not sure how useful this would be, since nobody should call these functions manually anyway. Or in which case would you want to call them directly?

We could also preserve doc comments on these function defs (in addition to putting them on weightinfo, etc)

Yea we should somehow expose the docs i think, but its rather low priority. We could also do some additional doc crate for that since the benches are private.

[sam0x17]

Ok this makes sense. Will focus on the other items first then 👍🏻

[jasl]

Although

#[extrinsic_call]
_(RawOrigin::Signed(caller.clone()), something, something);

is valid syntax, but it looks very rare, and Clion warns #[extrinsic_call] has syntax error (expects '(', ';', , '[' or '}'，but got '_', I shall submit a ticket for them)

parhaps use macro here is more readdable and IDE-friendly?

[sam0x17]

Is that just for the _() version?

…

On Wed, Jan 25, 2023 at 5:49 AM Jun Jiang ***@***.***> wrote: Although #[extrinsic_call] _(RawOrigin::Signed(caller.clone()), something, something); is valid syntax, but it looks very rare, and Clion warns #[extrinsic_call] has syntax error (expects '(', ';', , '[' or '}'，but got '_') parhaps use macro here is more readdable and friendly for IDE? — Reply to this email directly, view it on GitHub <#380>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOE4LLIIE7FMHK5GIKTMK3WUEAKHANCNFSM6AAAAAATZKNEKM> . You are receiving this because you were assigned.Message ID: ***@***.***>

[jasl]

Is that just for the _() version?
…
On Wed, Jan 25, 2023 at 5:49 AM Jun Jiang @.> wrote: Although #[extrinsic_call] (RawOrigin::Signed(caller.clone()), something, something); is valid syntax, but it looks very rare, and Clion warns #[extrinsic_call] has syntax error (expects '(', ';', , '[' or '}'，but got '') parhaps use macro here is more readdable and friendly for IDE? — Reply to this email directly, view it on GitHub <#13132 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOE4LLIIE7FMHK5GIKTMK3WUEAKHANCNFSM6AAAAAATZKNEKM . You are receiving this because you were assigned.Message ID: @.>

yes, given full name will resovle the syntax error.

[sam0x17]

yes, given full name will resovle the syntax error.

@ggwpez what do you think of this? I'm not particularly attached to the _() shortcut so I'd be fine with removing it, but I know you like it.

On the flipside, there are a number of places in the codebase already where things like rust-analyzer break.

[jasl]

Another issue is blank benchmark will make compiling error

#[benchmarks]
mod benchmarks {
	use super::*;

	impl_benchmark_test_suite!(Pallet, crate::tests::new_test_ext(), crate::tests::Test);
}

should I open an issue for this?

[ggwpez]

parhaps use macro here is more readdable and IDE-friendly?

On the flipside, there are a number of places in the codebase already where things like rust-analyzer break.

Yea, I dont think we should try to work around a the bugs of downstream tooling. This would only hinder our progress, but in this case the syntax is not very good to begin with.

I'm not particularly attached to the _() shortcut so I'd be fine with removing it, but I know you like it.

TBH I dont like it that much either… any better idea? Since we have the chance to throw out the garbage we should try to get the best syntax possible.

#[extrinsic_call(x, y, z)]

could also work, but maybe confusing as to what it is doing since otherwise its used as attribute macro.
We could also try self instead of _, or can we leave the _ away and just have the (args...)?
(cc @shawntabrizi in case you are reading this)

[ggwpez]

Another issue is blank benchmark will make compiling error

#[benchmarks]
mod benchmarks {
	use super::*;

	impl_benchmark_test_suite!(Pallet, crate::tests::new_test_ext(), crate::tests::Test);
}

should I open an issue for this?

What should that code do? It cannot implement tests without a benchmark. Guess we could fix that, but I dont see how it would be useful.

[sam0x17]

Yeah regarding the empty benchmark thing, it will correctly issue no errors in the case that you just do:

#[benchmarks]
mod benchmarks {
}

Beyond that, of course impl_benchmark_test_suite! won't work if there are no benchmarks...