Skip to content

chore: compare to v0.6.7 #326

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 21 commits into
base: main
Choose a base branch
from
Draft

chore: compare to v0.6.7 #326

wants to merge 21 commits into from

Conversation

maksymar
Copy link
Contributor

@maksymar maksymar commented May 28, 2025

DO NOT SUBMIT!

This PR is an empty commit to run canbench with up-to-date dependencies to see if benchmark results changed.

@maksymar maksymar changed the title chore: empty commit chore: benchmark baseline May 28, 2025
Copy link

github-actions bot commented May 28, 2025

canbench 🏋 (dir: ./benchmarks/btreemap) 90fe1d5 2025-06-12 11:52:20 UTC

./benchmarks/btreemap/canbench_results.yml is up to date
📦 canbench_results_btreemap.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes 👍
    counts:   [total 285 | regressed 0 | improved 0 | new 0 | unchanged 285]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  heap_increase:
    status:   No significant changes 👍
    counts:   [total 285 | regressed 0 | improved 0 | new 0 | unchanged 285]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 285 | regressed 0 | improved 0 | new 0 | unchanged 285]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------
CSV results saved to canbench_results.csv

Copy link

github-actions bot commented May 28, 2025

canbench 🏋 (dir: ./benchmarks/compare) 90fe1d5 2025-06-12 11:51:32 UTC

./benchmarks/compare/canbench_results.yml is up to date
📦 canbench_results_compare.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes 👍
    counts:   [total 18 | regressed 0 | improved 0 | new 0 | unchanged 18]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  heap_increase:
    status:   No significant changes 👍
    counts:   [total 18 | regressed 0 | improved 0 | new 0 | unchanged 18]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 18 | regressed 0 | improved 0 | new 0 | unchanged 18]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------
CSV results saved to canbench_results.csv

Copy link

github-actions bot commented May 28, 2025

canbench 🏋 (dir: ./benchmarks/vec) 90fe1d5 2025-06-12 11:50:43 UTC

./benchmarks/vec/canbench_results.yml is up to date
📦 canbench_results_vec.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes 👍
    counts:   [total 16 | regressed 0 | improved 0 | new 0 | unchanged 16]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  heap_increase:
    status:   No significant changes 👍
    counts:   [total 16 | regressed 0 | improved 0 | new 0 | unchanged 16]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 16 | regressed 0 | improved 0 | new 0 | unchanged 16]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------
CSV results saved to canbench_results.csv

Copy link

github-actions bot commented May 28, 2025

canbench 🏋 (dir: ./benchmarks/memory_manager) 90fe1d5 2025-06-12 11:50:31 UTC

./benchmarks/memory_manager/canbench_results.yml is up to date
📦 canbench_results_memory-manager.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes 👍
    counts:   [total 3 | regressed 0 | improved 0 | new 0 | unchanged 3]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  heap_increase:
    status:   No significant changes 👍
    counts:   [total 3 | regressed 0 | improved 0 | new 0 | unchanged 3]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 3 | regressed 0 | improved 0 | new 0 | unchanged 3]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------
CSV results saved to canbench_results.csv

@maksymar maksymar changed the title chore: benchmark baseline chore: compare to v0.6.7 Jun 6, 2025
Copy link

github-actions bot commented Jun 12, 2025

canbench 🏋 (dir: ./benchmarks/btreeset) 90fe1d5 2025-06-12 11:50:50 UTC

./benchmarks/btreeset/canbench_results.yml is up to date
📦 canbench_results_btreeset.csv available in artifacts

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes 👍
    counts:   [total 100 | regressed 0 | improved 0 | new 0 | unchanged 100]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  heap_increase:
    status:   No significant changes 👍
    counts:   [total 100 | regressed 0 | improved 0 | new 0 | unchanged 100]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 100 | regressed 0 | improved 0 | new 0 | unchanged 100]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------
CSV results saved to canbench_results.csv

@maksymar
Copy link
Contributor Author

maksymar commented Jun 12, 2025

Compare v0.6.9 vs v0.6.7

btreemap

---------------------------------------------------

Summary:
  instructions:
    status:   Regressions and improvements 🔴🟢
    counts:   [total 285 | regressed 50 | improved 134 | new 0 | unchanged 101]
    change:   [max +899.01M | p75 +4.24M | median -8.50M | p25 -99.48M | min -1.78B]
    change %: [max +26.15% | p75 +0.94% | median -1.49% | p25 -9.79% | min -88.41%]

  heap_increase:
    status:   No significant changes 👍
    counts:   [total 285 | regressed 0 | improved 0 | new 0 | unchanged 285]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 285 | regressed 0 | improved 0 | new 0 | unchanged 285]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------

Only significant changes:
| status | name                                        | calls |     ins |  ins Δ% | HI |  HI Δ% | SMI |  SMI Δ% |
|--------|---------------------------------------------|-------|---------|---------|----|--------|-----|---------|
|   +    | btreemap_v2_scan_keys_1k_0b                 |       |   1.19M | +26.15% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_scan_keys_rev_1k_0b             |       |   1.19M | +23.88% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_remove_blob_1024_128            |       |   7.38B | +13.87% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_remove_blob_512_128             |       |   4.09B | +13.32% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_mem_manager_remove_blob512_u64  |       |   4.35B | +12.87% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_remove_blob_256_128             |       |   2.44B | +10.74% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_scan_keys_1k_10kib              |       |   2.60M | +10.32% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_scan_keys_rev_1k_10kib          |       |   2.58M |  +9.59% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_remove_blob_128_128             |       |   1.61B |  +9.15% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_range_count_1k_10kib            |       |   2.65M |  +8.67% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_insert_blob_1024_128            |       |   5.50B |  +7.70% |  0 |  0.00% | 196 |   0.00% |
|   +    | btreemap_v2_insert_blob_512_128             |       |   3.05B |  +6.63% |  0 |  0.00% | 111 |   0.00% |
|   +    | btreemap_v2_insert_blob_256_128             |       |   1.80B |  +6.50% |  0 |  0.00% |  67 |   0.00% |
|   +    | btreemap_v2_mem_manager_insert_blob512_u64  |       |   3.15B |  +6.23% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_remove_vec_32_64                |       | 981.77M |  +5.58% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_remove_vec_32_16                |       | 885.75M |  +5.38% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_insert_blob_128_128             |       |   1.19B |  +4.62% |  0 |  0.00% |  46 |   0.00% |
|   +    | btreemap_v2_remove_blob_32_4                |       | 712.64M |  +3.93% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_insert_blob_32_64               |       | 543.10M |  +3.87% |  0 |  0.00% |  18 |   0.00% |
|   +    | btreemap_v2_get_vec_16_128                  |       | 456.73M |  +3.83% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_remove_blob_32_64               |       | 754.17M |  +3.81% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_remove_blob_32_8                |       | 712.28M |  +3.80% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_insert_blob_32_32               |       | 536.93M |  +3.77% |  0 |  0.00% |  13 |   0.00% |
|   +    | btreemap_v2_remove_vec_32_32                |       | 879.00M |  +3.77% |  0 |  0.00% |   0 |   0.00% |
|   +    | btreemap_v2_insert_blob_32_128              |       | 551.94M |  +3.66% |  0 |  0.00% |  28 |   0.00% |
|  ...   | ... 134 rows omitted ...                    |       |         |         |    |        |     |         |
|   -    | btreemap_v2_insert_vec_1024_128             |       |   2.73B | -17.75% |  0 |  0.00% | 193 |   0.00% |
|   -    | btreemap_v2_get_vec_128_128                 |       | 583.90M | -17.80% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_pop_last_vec_128_128            |       |   1.51B | -17.85% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_contains_vec_32_256             |       | 441.72M | -18.09% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_contains_vec_128_128            |       | 574.28M | -19.15% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_pop_last_vec_256_128            |       |   2.06B | -20.30% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_pop_first_vec_256_128           |       |   2.00B | -21.02% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_pop_first_blob_64_128           |       |   1.05B | -21.19% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_pop_last_blob_64_128            |       |   1.02B | -22.30% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_pop_last_vec_512_128            |       |   2.79B | -25.20% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_pop_first_vec_512_128           |       |   2.69B | -25.40% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_get_vec_256_128                 |       | 912.19M | -26.01% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_contains_vec_256_128            |       | 902.32M | -26.81% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_get_blob_64_128                 |       | 436.96M | -27.38% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_mem_manager_get_vec512_u64      |       |   1.27B | -29.03% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_pop_last_vec_1024_128           |       |   4.24B | -29.62% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_pop_first_vec_1024_128          |       |   4.02B | -30.38% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_get_vec_512_128                 |       |   1.26B | -30.83% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_contains_blob_64_128            |       | 414.36M | -31.14% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_mem_manager_contains_vec512_u64 |       |   1.23B | -31.31% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_contains_vec_512_128            |       |   1.25B | -31.38% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_get_vec_1024_128                |       |   1.84B | -36.97% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_contains_vec_1024_128           |       |   1.82B | -37.54% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_get_10mib_values                |       | 388.59M | -68.34% |  0 |  0.00% |   0 |   0.00% |
|   -    | btreemap_v2_contains_10mib_values           |       | 142.21M | -88.41% |  0 |  0.00% |   0 |   0.00% |

ins = instructions, HI = heap_increase, SMI = stable_memory_increase, Δ% = percent change

---------------------------------------------------

compare

---------------------------------------------------

Summary:
  instructions:
    status:   Improvements detected 🟢
    counts:   [total 18 | regressed 0 | improved 3 | new 0 | unchanged 15]
    change:   [max +11 | p75 0 | median -2 | p25 -245.41K | min -2.89B]
    change %: [max 0.00% | p75 0.00% | median -0.00% | p25 -0.00% | min -25.60%]

  heap_increase:
    status:   No significant changes 👍
    counts:   [total 18 | regressed 0 | improved 0 | new 0 | unchanged 18]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 18 | regressed 0 | improved 0 | new 0 | unchanged 18]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------

Only significant changes:
| status | name                    | calls |     ins |  ins Δ% |    HI |  HI Δ% |   SMI |  SMI Δ% |
|--------|-------------------------|-------|---------|---------|-------|--------|-------|---------|
|   -    | read_chunks_btreemap_1m |       | 134.60B |  -2.10% | 1.89K |  0.00% | 3.20K |   0.00% |
|   -    | read_chunks_btreemap_1k |       |   5.42B |  -7.21% | 1.60K |  0.00% | 1.67K |   0.00% |
|   -    | read_chunks_btreemap_1  |       |   1.22B | -25.60% | 3.23K |  0.00% | 1.67K |   0.00% |

ins = instructions, HI = heap_increase, SMI = stable_memory_increase, Δ% = percent change

---------------------------------------------------

memory_manager

---------------------------------------------------

Summary:
  instructions:
    status:   No significant changes 👍
    counts:   [total 3 | regressed 0 | improved 0 | new 0 | unchanged 3]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  heap_increase:
    status:   No significant changes 👍
    counts:   [total 3 | regressed 0 | improved 0 | new 0 | unchanged 3]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 3 | regressed 0 | improved 0 | new 0 | unchanged 3]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------

vec

---------------------------------------------------

Summary:
  instructions:
    status:   Improvements detected 🟢
    counts:   [total 16 | regressed 0 | improved 1 | new 0 | unchanged 15]
    change:   [max +102.33K | p75 +20.00K | median 0 | p25 0 | min -480.00K]
    change %: [max +1.82% | p75 +0.31% | median 0.00% | p25 0.00% | min -9.11%]

  heap_increase:
    status:   No significant changes 👍
    counts:   [total 16 | regressed 0 | improved 0 | new 0 | unchanged 16]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   No significant changes 👍
    counts:   [total 16 | regressed 0 | improved 0 | new 0 | unchanged 16]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------

Only significant changes:
| status | name        | calls |   ins |  ins Δ% | HI |  HI Δ% | SMI |  SMI Δ% |
|--------|-------------|-------|-------|---------|----|--------|-----|---------|
|   -    | vec_get_u64 |       | 4.79M |  -9.11% |  0 |  0.00% |   0 |   0.00% |

ins = instructions, HI = heap_increase, SMI = stable_memory_increase, Δ% = percent change

---------------------------------------------------

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant