Future-proof Benchmarking ⏱

[p-shahi]

eta: 2023Q2

• perf protocol
  • spec Introduce perf spec specs#478
  • feat(perf): implement libp2p perf protocol rust-libp2p#3508 (rust-libp2p)
  • go-libp2p feat(perf): add (automation, provision, build, run) tooling #184
  • feat(@libp2p/protocol-perf): Implement perf protocol js-libp2p#1604 (js-libp2p)
  • perf: throughput test (TCP, QUIC, libp2p, but not iperf) never exits slow start #261
  • Add vanilla Go http in order to measure overhead of TLS #204
• Create performance tooling to run perf protocol implementation on cloud instances
  • First iteration feat(perf): add (automation, provision, build, run) tooling #184
  • Only use a single seconds measurement. Don't split in connection establishment, upload and download. These are difficult to differentiate. refactor(perf): expose single latency measurement  #207
  • Add js-libp2p feat(@libp2p/protocol-perf): Implement perf protocol js-libp2p#1604 feat: add js-libp2p perf tests #244
  • Add back zig-libp2p
  • Run iperf on both UDP and TCP
  • Requests per second (for some request/response protocol)
  • Handshakes per second (useful to identify overhead in connection initialization).
  • Support testing other security protocols, i.e. Noise
  • Resource utilization benchmark?
  • Don't blindly trust server and clients ssh key (mitm attack)
• Automation see Automate performance benchmarking #183
• Visualize data emitted by above performance tooling Create a performance dashboard #156
  • ideally mirror advanced filtering in the URL for better sharing
• Create test to show libp2p's performance compared to http #27
• Benchmark js-libp2p transfer performance #65
• blog about performance benchmarking in libp2p blog#74
• Diverse benchmarks e.g. multiple streams or multiple connections
• Write documentation

[BigLep]

@mxinden : I know you're doing blog drafting at https://observablehq.com/@mxinden-workspace/libp2p-perf - thanks!

I did leave some comments inline.

A couple of general things that came to mind while looking at it:

1. Is this just where we're drafting, or something we expect to point users to?
2. If pointing users to it, can we create a libp2p Observable namespace?
3. What is going to be the presence on the libp2p blog about this?

(no pressure to respond now - I don't mean to be catching you in-progress. If this has been though through and you just haven't gotten to any of this yet, that is fine. I am just leaving notes now while it's coming to mind.)

[mxinden]

Status Update

#184 and thus a first iteration of the libp2p performance benchmark setup is merged. Follow-up / future work in tracked in the description of this issue.

[mxinden]

Crossposting a Slack message here, for those following through GitHub only.

First iteration of the libp2p performance benchmarking setup merged

More specifically #184.

Let's try it out. We recently released a new version of rust-libp2p, v0.52.0. Now we can add it to the benchmarking setup, see how well it performs.

1. Create a pull request against libp2p/test-plans adding the new version fix(perf/rust-libp2p): update to latest v0.52.0 release #202. (Note that in this particular case we are updating an existing version, as we have been already testing a pre-release of v0.52.0.)
2. Trigger the libp2p perf test GitHub workflow for the branch of the pull request https://github.com/libp2p/test-plans/actions/workflows/perf.yml
3. Follow the workflow run, which eventually pushes the benchmark results to the branch https://github.com/libp2p/test-plans/actions/runs/5351995316/jobs/9706612810
4. See the new commit on your pull request fix(perf/rust-libp2p): update to latest v0.52.0 release #202 b94960f
5. Visualize the benchmark results via our ObservableHQ dashboard. Just make sure to choose the right branch, in this case perf-rust-libp2p-v0.52 in the Branch input https://observablehq.com/@mxinden-workspace/libp2p-performance-dashboard

Note that this is the first iteration only. There are lots of follow-ups, especially around the dashboard. Future work tracked on #63.

Thanks to Marco, Piotr and Marten for all the help on the project!

https://filecoinproject.slack.com/archives/C03K82MU486/p1687492998525109