[Question] Performance issues on Firefox?

[mxschmitt]

It feels like Firefox with Playwright has some big performance issues compared to chromium or webkit. Is this already known?

Example:

const playwright = require("playwright");

const TEST_COUNT = 5;

(async () => {
  for (const browserType of ['chromium', 'firefox', 'webkit']) {
    const durations = [];
    for (const count in Array.from(Array(TEST_COUNT))) {
      const startTime = new Date().getTime()
      const browser = await playwright[browserType].launch();
      const context = await browser.newContext();
      const page = await context.newPage();
      await page.goto('https://www.amazon.com');
      await browser.close();
      const neededTime = new Date().getTime() - startTime
      durations.push(neededTime)
    }
    const average = durations.reduce((prev, curr) => prev + curr) / TEST_COUNT
    console.log(`Browser ${browserType} took on average ${average} ms`)
  }
})();

Execution output on my 2019 MacBook Pro (native on the Host):

Browser chromium took on average 1477.8 ms
Browser firefox took on average 6373.4 ms
Browser webkit took on average 1897.6 ms

And output on my try.play.wright.tech instance (Docker based):

Browser chromium took on average 1579 ms
Browser firefox took on average 8287.4 ms
Browser webkit took on average 1802 ms

[mxschmitt]

I've made some further research by extending it with this:

const playwright = require("playwright");

const TEST_COUNT = 5;

(async () => {
  for (const browserType of ['chromium', 'firefox']) {
    const durationsLaunch = [];
    const durationsNewContext = [];
    const durationsNewPage = [];
    const durationsGoto = [];
    const durationsClose = [];
    for (const count in Array.from(Array(TEST_COUNT))) {
      var startTime = new Date().getTime()
      const browser = await playwright[browserType].launch();
      var neededTime = new Date().getTime() - startTime
      durationsLaunch.push(neededTime)
      startTime = new Date().getTime()
      const context = await browser.newContext();
      neededTime = new Date().getTime() - startTime
      durationsNewContext.push(neededTime)
      startTime = new Date().getTime()
      const page = await context.newPage();
      neededTime = new Date().getTime() - startTime
      durationsNewPage.push(neededTime)
      startTime = new Date().getTime()
      await page.goto('https://www.amazon.com');
      neededTime = new Date().getTime() - startTime
      durationsGoto.push(neededTime)
      startTime = new Date().getTime()
      await browser.close();
      neededTime = new Date().getTime() - startTime
      durationsClose.push(neededTime)
    }
    const averageLaunch = durationsLaunch.reduce((prev, curr) => prev + curr) / TEST_COUNT
    const averageNewContext = durationsNewContext.reduce((prev, curr) => prev + curr) / TEST_COUNT
    const averageNewPage = durationsNewPage.reduce((prev, curr) => prev + curr) / TEST_COUNT
    const averageGoto = durationsGoto.reduce((prev, curr) => prev + curr) / TEST_COUNT
    const averageClose = durationsClose.reduce((prev, curr) => prev + curr) / TEST_COUNT
    console.log(`Browser ${browserType}`)
    console.log(JSON.stringify({
      averageLaunch,
      averageNewContext,
      averageNewPage,
      averageGoto,
      averageClose
    }))
    console.log("-----------------------")
  }
})();

Output on try.playwright.tech

Browser chromium
{"averageLaunch":118.6,"averageNewContext":1.6,"averageNewPage":52.2,"averageGoto":1389,"averageClose":10.8}
-----------------------
Browser firefox
{"averageLaunch":1984.2,"averageNewContext":19.2,"averageNewPage":246.2,"averageGoto":1538,"averageClose":4969}
-----------------------

Closing Firefox seems to take a very long time. That needs to be improved in my mind. The launch which takes around 2 seconds on average is kinda reasonable. But 2 seconds is also very long. idk

// Edit: by starting it with the DEBUG=pw:launcher environment variable set, it's also reproducible.

[pavelfeldman]

Thanks for the stats! Yes, we know that Firefox is significantly slower. But that's what we are getting out of the box - not some kind of lag introduced by the Playwright harness. We need to take the profiler in hand and figure it out at some point. Close being the culprit is a great start, so thank you for that.

[pavelfeldman]

I modified your script to have TEST_COUNT = 2, but I also create contexts in a loop, 10 contexts per launch. Pasting the results below.

TL/DR is that launching and closing the browser is much slower, but that is not supposed to happen often - only when creating test runner workers and/or browser crashes. The hot operations are creating and closing the contexts and those seem to be within reason on Firefox. 2 seconds to close it still on the high side, but not critical for batch operation.

Browser chromium
{
 "averageLaunch":199,
 "averageNewContext":1.3,
 "averageNewPage":78.25,
 "averageGoto":137.45,
 "averageCloseContext":4.1,
 "averageClose":2.5
}
-----------------------
Browser firefox
{
 "averageLaunch":1960,
 "averageNewContext":10.3,
 "averageNewPage":97.95,
 "averageGoto":155.2,
 "averageCloseContext":13.55,
 "averageClose":2131.5
}
-----------------------
Browser webkit
{
 "averageLaunch":13,
 "averageNewContext":17.45,
 "averageNewPage":131.55,
 "averageGoto":98.65,
 "averageCloseContext":0.95,
 "averageClose":9
}

[mxschmitt]

Yep true, normally you should not start the browser very often in a row. But nevertheless it would be nice to try to optimise that, I also already tried to investigate a bit, but came to no direct conclusion.

Primary goal is to optimise the close process first, because 2+ seconds for just closing is not reasonable in my mind.

[mxschmitt]

Small update regarding that, seems like Firefox is now even worse. From 2.1 seconds up to 3,8 seconds.
Version: 0.14.0
Browser chromium

{
    "averageLaunch": 108.5,
    "averageNewContext": 5,
    "averageNewPage": 51.5,
    "averageGoto": 894.5,
    "averageClose": 35
}

---

Browser firefox

{
    "averageLaunch": 1100.5,
    "averageNewContext": 3,
    "averageNewPage": 886,
    "averageGoto": 1276.5,
    "averageClose": 3872
}

---

Browser webkit

{
    "averageLaunch": 4.5,
    "averageNewContext": 73.5,
    "averageNewPage": 105,
    "averageGoto": 1223,
    "averageClose": 14.5
}

---

[whimboo]

@pavelfeldman or @JoelEinbinder could one of you please run that script with the pure CDP implementation for Firefox Nightly if possible? I wonder how much those number differ compared to your custom build.

We are aware of poor performance during connection and closing, but the numbers aren't that huge as listed here.

[pavelfeldman]

@whimboo : most of the time spent is due to Firefox's unfolding profile data and precompiling scripts into it upon the first profile run (I vaguely remember it being 25Mb of scripts). It happens independently of any protocol calls into the browser, upon launch. If you use persistent profile where this has already happened, there will be no delay whatsoever.

The solution would be to cache the profile template upon installation. We tried that, but it made the installation process brittle due to permissions and other issues.

[pavelfeldman]

@whimboo I see you are from Mozilla, do you have a hint for us on how we could fix it together? It would affect CDP and Juggler, so seems like something everyone would benefit from.

[whimboo]

@pavelfeldman it would be great if one of you could send a Gecko profile to us. Basically you would have pass the environment variable MOZ_PROFILER_STARTUP=1 to the Firefox process. Once Firefox is up, navigate to http://profiler.firefox.com. Enable the toolbar button, and capture the profile immediately. After publishing it please give us the link here.

[aslushnikov]

@whimboo thank you for the instructions! Followed them, here's the profile I got: published launch profile.

For the record, this is profile with Playwright v1.1.1 that I captured with the following script:

const { firefox } = require('playwright');

(async () => {
  const browser = await firefox.launch({
    headless: false,
    env: {
      ...process.env,
      MOZ_PROFILER_STARTUP: 1,
    },
  });
  const page = await browser.newPage();
  await page.goto('http://profiler.firefox.com');
})();

[aslushnikov]

mid-august 2020: Performance issues are still there for Firefox, but these are not actionable on the Playwright side.

I'll close this for now; we'll reopen if it's more pressing.