Benchmarking Roadmap #432
Description
Expanded Benchmarking Suite & Analysis
To ensure that ReadySet is performant in a wide array of use cases, we'll invest in upgrading our benchmarking infrastructure: apps, tests, and analysis and tackle any potential areas for improvement that are surfaced by this work.
In addition to measuring cache performance, these new benchmarks will stress test our networking/proxying, protocol emulation, serialization, and any other components on the hot path.
This project aims to be more consistent and systematic about creating and executing benchmarks, collecting data, as well as analyzing the outputs of our existing benchmarking suites (benchmarks crate, system-benchmarks crate).
Most of the foreseeable work will be exploratory: research and building scripts/tools for benchmarking and then analyzing the results. If any deficiencies are identified, we'll update the milestones to reflect planned improvements.
Plan
Since the combinatorics around all of the various benchmarking parameters are quite large, the current project will prioritize the following:
Comparisons
- MySQL vs Postgres protocol emulations (pure warm cache reads)
- ReadySet warm cache reads vs going directly to the primary database
- ReadySet warm cache reads vs using a look-aside cache like Memcached or Redis
Queries
- Simple point reads by PK/unique index lookups
- Simple queries with a full materialization (like “select count(1) from table” or “select * from table”)
- Moderate joins/aggregation queries
The same set of schema/queries will be used for all the benchmarking variants to keep things simple.
Issues
- Allow custom var-overrides of data generated SQL fields #370
- Add benchmark test for noria lobste.rs sample #337
- benchmarks: Allow queries with no parameters #327
- Add benchmark for select all rows from a table #326
- Measure row count effect on hot read path #371
- Measure row size effect on hot read path #372
- Improve message coalescing in the PG extended query protocol path #367
- Understand why Postgres performs worse than MySQL when serving warm reads #368
- system-benchmarks: fix news_app tests for PG #339
- New benchmark to compare readyset vs memcached vs direct to upstream #212