A full run takes too long

[krausest]

The full run for chrome 141 took 14 hours just for the keyed results. Non-keyed are faster, but I'm already skipping non-keyed for odd chrome versions.
Due to the long duration I'm already delaying performing the run. The goal should be that the benchmark completes the run in 8-10 hours for keyed implementations such that it fits in a night.

The question is how to reduce duration:

• Retire old frameworks
• Remove some benchmarks that add little value

[n-ce]

Have you thought of running the benchmark with Bun?

[n-ce]

here's the list of frameworks that are still getting maintained :

• React
• Vue
• Svelte
• Angular
• Solid
• Preact
• Inferno
• Mithril
• Leptos
• Dioxus
• uhtml
• ripple
• lit
• Vanjs
• Alpine
• ivi
• omi
• ember
• karyon
• imba
• qwik
• blazor
• stenciljs
• michijs
• targetjs

with regards to your 'removing less value' frameworks, I think a middle ground is to run them only once every two tests.

[krausest]

Have you thought of running the benchmark with Bun?

I'm pretty sure that most time is spent by puppeteer executing the test in the browser, so I don't see that bun can improve the run time.

[krausest]

Here's a pair plot for the memory benchmarks. Seems like having both 22_run-memory and 23_update5-memory adds little value (they have a correlation of 0.9981931902570598),

[krausest]

Here's the pair plot for the CPU benchmarks:

The highest correlation here is between 01_run1k and 08_create1k-after1k_x2: 0.9589454583199726
Maybe also a candidate for removal.

[leeoniya]

maybe just re-use the prior results from the deep-in-the-red frameworks unless they update or someone reaches out to indicate significant optimizations? i bet they burn 20%+ of the total runtime without having moved the needle in months or years.

[syduki]

The correlation thing is insightful, with some degree of tolerance you may shave all scenarios except 01_run1k, and maybe that is the intention if taking in consideration the weights assigned recently to each scenario. It's a kind of "optimization" like comparing how many plums go into the factory with how much jam comes out — if there's correlation, why bother measuring the jam.

And how about the "mechanics" between 01_run1k and 08_create1k-after1k_x2? While it creates rows as others, 08_create1k-after1k_x2 is a completely different thing, as it implies running reconcile algorithm (in proper frameworks). Comparing it to 01_run1k and 02_replace1k is not very useful, since the latter ones just add rows to an empty container.

More logically is to get rid of redundant/useless scenarios.
The first of these is 07_create10k, it was mentioned several times on different threads that it is an absurd thing - basically a magnification of 01_run1k multiplied by browser side effects.
The second is 02_replace1k which is basically 09_clear1k_x8 + 01_run1k
I guess that removing these two would cut the running time by half.

And about controversial "Retire old frameworks", it also raises doubts about the intent of this benchmark, because some full-fledged real frameworks like Knockout may end up in the trash to make space for some hot-rod/gelded/whatever-you-name-it ones conceived especially for this "competition". Maybe it's time to define a compliance level for the "participation" — something like a proper "swap N random rows" instead of a lazy swap(a, b) or worse.