chrome 124: weird results

[krausest]

Management Summary:

• Chrome 124 results don't look right
• The test runner yields results that differ from manual testing for create 1k rows (see comment below)
• Frameworks that use RAF might be reported as faster than frameworks without RAF (see imba below). I didn't look into that issue in detail. The create 1k rows issue is currently more important to me.

Details:

A first run for the keyed implementations show results that don't look good.
The fastest frameworks score now 1.05. In the history this has been a bad sign that something might be odd with the measurement.
The fastest implementation for clear row was imba:

Imba uses RAF to schedule the rendering. In the screenshot above we'd compute 14.95 msecs for the duration (from click event to the end of the first commit event). Please note that there's a separate hit test and commit event later in the trace that doesn't count for our computation at least at the moment.

Vanillajs has only one commit event and looks more straight forward:

So here we'd take 17.38 msecs for vanillajs.

For chrome 123 we measured 15.3 for vanillajs and 16.7 for imba.
With chrome 124 the results look different:

The boxplot looked like that for chrome 123:

[krausest]

If I add a RAF to vanillajs I get results pretty close to imba and I get a second commit event.

Maybe it was a wrong decision to count to the first commit and ignore the second commit in this case.
It seems like that second commit is a consequence of the RAF call (and not something spurious) and should be included in the measurement.

[krausest]

Ran the benchmark for the keyed frameworks again last night. At least results are very consistent ☹️
Here's a screenshot:

or here if you want to look at the table.
It'll take some time for further analysis.

[krausest]

Here's a look at create rows:
A rather normal vanillajs trace:

The fastest ivi trace:


This wasn't the case with chrome 123:

With manual testing I haven't managed to get faster results for ivi, it looks more like chrome 123.

Using puppeteer as the test runner reports 34.1 msecs for ivi and 37.3 msecs for vanillajs.
Webdrivercdp reports 33.9 msecs for ivi and 33.8 for vanillajs, which is more consistent with manual testing.

Other tests (so far it was the same browser windows, but a new tab per run):
puppeteer with a new browser window per run: 33.4 for ivi and 36.2 for vanillajs.
puppeteer running the create bench in one tab: 33.6 for ivi and 34.3 for vanillajs

Manual testing gives me 38.5 msecs for ivi and 36.8 for vanillajs. Something is wrong here...

[antonmak1]

@krausest Maybe this has something to do with the HTML and DOM changes in version 124? Specifically, "Document render-blocking" or "setHTMLUnsafe and parseHTMLUnsafe" or something like that?

[krausest]

I took a closer look at the create 1k rows issue - but I have no good news:
I ran a manual test for create 1k rows with 8 runs for ivi, solid, doohtml and vanillajs. This is what a boxplot looks like for this manual test. I'd consider that to be the ground truth:

Please note that there's a suspicious outlier for vanillajs with 34.29 msecs. I kept a screenshot of that trace.
There's a quite clear ordering between the other three: doohtml < ivi < solid

What's interesting is that the chart above is remarkably similar to the puppeteer chrome 123 results:

However the results for all testdrivers are disappointing. I performed three runs for each of them:

1. Puppeteer: "clearly ivi is fastest"

At least results are repeatable though not similar to the manual results.

2. Playwright: "solid is fastest"

3. Webdrivercdp: "where is my mind?"
   
   The first run fits to "doo < ivi < solidjs" though vanillajs is closer to the outlier from the manual testing above, but run two and three just look random.

Currently I'm out of ideas. I don't see how I could publish chrome 124 results soon.

[antonmak1]

@krausest Perhaps, if the old code works somehow wrong, then first of all, it needs to be tested together with the new code, which needs to be written based on version 124. It is clear that some kind of nonsense is coming out, that all the results now start from 1.05 and those frameworks and libraries that were plus or minus in one place are now 10 positions ahead, then 10 positions behind - this is nonsense. How can 7+ releases be adequate, but then everything breaks with the new one? This means that there is definitely a new error somewhere in the code that was not an error before or was not considered one.

[mksunny1]

I will add my mind here. See if I can come up with something. I am not too familiar with this yet.

[krausest]

There's not much code that was updated: 30c247a
I also tried to update all the dependencies, but it doesn't change anything so I rolled them back. So I really think the factor that makes the difference is chrome 124.

I tried if windows reports better results, but it doesn't look like it does.

[krausest]

Just when I finished my plan what I'd do after quitting this project I tried one last thing 😄
What if I add a sleep call of a second before each interaction?
Turns out that this results in the following charts for three independent runs for puppeteer:

Those charts look much closer to manual testing (and of course as old as I'm feeling that second between clicks seems realistic).

Now I have to find out:

1. How does the sleep impact the chart of the other testdrivers?
2. Where do we need the sleep calls? (hopefully not at many places)
3. What's the minimum duration for the sleep?

[krausest]

1: I don't see that the delay helps the other test drivers.
playwright with a delay of 1 second:

Looks different from the chart without delays, but not close to the manual testing result.

Nor does webdrivercdp:

So we'll stick with puppeteer the other drivers do not report values closer to manual testing.

[krausest]

2. Chrome 124 is bizarre. I tried first adding sleeps between all interactions and then trying to filter out where they are actually needed.

I got that chart with puppeteer:

Look how bad those numbers are: > 50 msecs. That's ridiculous.
Here's a trace for one such bad run:

And here's the trace file:
doohtml-keyed_01_run1k_5.json
The trace looks right, the computation of the duration is OK, there's no RAF, no GC, nothing suspicious. But scripting 5 ms and rendering 48 ms is just incredibly bad.

It turns out that one sleep call causes that bad performance:
Before runBenchmark we call forceGC. If we sleep after that we get the bad performance:

Uncommenting the line (which just sleeps for a second via setTimeout in a promise) causes the bad performance.
Without that sleep:

Just 37 msecs.
doohtml-keyed_01_run1k_2.json
The script duration is now 2 msecs and the rendering duration 37 mecs (!).

Found out something: This can be resolved by adding the trace category "disabled-by-default-v8.cpu_profiler". When enables were below 40 msecs with the sleep after forceGC().