[nexus] Refactor test-utilities to helper crate, add test benchmarks

[smklein]

Results are visible by running:

$ cargo bench -p omicron-nexus --bench setup_benchmark

Test Setup/do_full_setup                                                                          
                        time:   [8.3949 s 8.4655 s 8.5444 s]
                        change: [+2.0754% +4.8573% +7.9552%] (p = 0.00 < 0.05)

Test Setup/do_crdb_setup                                                                          
                        time:   [7.7077 s 7.7553 s 7.8020 s]
                        change: [+2.6847% +6.6535% +11.453%] (p = 0.00 < 0.05)

Test Setup/do_clickhouse_setup                                                                          
                        time:   [517.82 ms 520.83 ms 524.71 ms]
                        change: [-0.6317% +0.3463% +1.2429%] (p = 0.53 > 0.05)

This implies:

• Running a "null" Nexus test takes about 8.5 seconds
• 90% of that time is setting up + tearing down CRDB.

[davepacheco]

Neat. Is there an easy way to get on-CPU time vs. blocked time? I wonder how much of this latency is hidden by the parallelism.

[davepacheco]

Oh, one question before you land this: should we not add this crate to the workspace?

question about workspace toml

[davepacheco]

Output from my test machine (lots of noise trimmed out):

$ cargo bench -p omicron-nexus --bench setup_benchmark
...
                                                                                     Test Setup/do_full_setup                        
                        time:   [5.4927 s 5.5361 s 5.5889 s]
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild
...
Test Setup/do_crdb_setup                        
                        time:   [4.7564 s 4.7928 s 4.8355 s]
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild
...
Test Setup/do_clickhouse_setup                        
                        time:   [523.34 ms 525.28 ms 527.28 ms]

This is quite a bit faster than you saw. This is ivanova, which is a Matisse with 16 threads and 64 GiB of DRAM.

[smklein]

Neat. Is there an easy way to get on-CPU time vs. blocked time? I wonder how much of this latency is hidden by the parallelism.

Certainly would be possible to add profiling, but that doesn't seem to exist out-of-the-box with criterion. They do provide hooks if we have a profiler we wanna use.

Oh, one question before you land this: should we not add this crate to the workspace?

Sure, added. I don't really have a clear idea when this is necessary at the top-level, vs just being an implicit dependency -- regardless of status in the top-level workspace file, this crate gets built when nexus is tested, since it's a dev-dependency. By adding it to the workspace, it'll be built (I think) even when we aren't trying to build tests. Maybe that's fine?

This is quite a bit faster than you saw. This is ivanova, which is a Matisse with 16 threads and 64 GiB of DRAM.

I'm running on a Linux laptop with 16 threads, but 32 GB DRAM. TBH, I don't think absolute performance matters as much as relative performance. But you bring up a good point regarding parallelism.

[david-crespo]

whatever enables #493 is good