[REGRESSION]: Very slow on v1.1

[jinjor]

Context:

• GOOD Playwright Version: v1.0.2
• BAD Playwright Version: v1.1.0
• Operating System: Mac
• Extra: Both local (mac) and CI (ubuntu) environment, using chromium for tests

Code Snippet

Sorry, no snippet to reproduce this. I have a huge proprietary code and have not find the suspicious code in it. I will share it whenever I find it.

Describe the bug

I'm trying to upgrade playwright v0.11.1 to v1.1.1, but found a huge performance loss between v1.0.2 and v1.1.0. Our test becomes about 4x slower.

On local (mac):

• v0.11.1: npm run page-test 430.59s user 64.33s system 133% cpu 6:11.71 total
• v1.0.2: npm run page-test 396.36s user 53.51s system 106% cpu 7:00.52 total
• v1.1.0: npm run page-test 2600.17s user 224.96s system 199% cpu 23:38.11 total
• v1.1.1: npm run page-test 2621.26s user 223.12s system 200% cpu 23:40.09 total

On CircleCI (ubuntu):

• v0.11.1: 7m30s
• v1.0.2: 7m45s
• v1.1.1: 28m42s

It seems the new version uses CPU much more than before. Has something internally changed? Is there anyone who has run into similar situation?

For more context:

• Tests are written in mocha framework (not in parallel), they are written in one large file.
• Playwright is launched only once, context, page object is reused through all tests.
• Proxy is not used.

[pavelfeldman]

Thank you for the report. If we fail repoducing it locally, can you help us bisecting it? We will give you a range of Playwright packages and/or Chromium revisions and ask you to run your tests against this range to find the culprit.

[pavelfeldman]

The first would be ruling out the upstream (Chromium change). You would point 1.0.2 to Chromium that 1.1.1 uses. It should work and if it is slow, we would know the regression is in Chromium. If it is fast, we would know that the regression is in Playwright. Depending on the result, we would ask you to bisect the range of given Chromium revisions or a range of Playwright revisions.

[pavelfeldman]

@jinjor: could you join our slack and ping me in the #general channel?

[pavelfeldman]

@jinjor I'll start typing instructions here, but in case you show up on Slack we can continue there.

• in your 1.0.2 setup that works fine, edit file node_modules/playwright/browsers.json and replace chromium's revision 764964 with 775089:

    {
      "name": "chromium",
      "revision": "775089"
    }

• run node ./node_modules/playwright/install.js

This should install newer Chromium like this:

Downloading chromium v775089 - 125.5 Mb [====================] 100% 0.0s 
chromium v775089 downloaded to /Users/pfeldman/Library/Caches/ms-playwright/chromium-775089

Now you have 1.0.2 that runs against new Chromium. Run your tests and tell us if that is slow. If it is, we'll give you Chromium revisions to put there.

[jinjor]

Thank you for the instruction 😃

@jinjor: could you join our slack and ping me in the #general channel?

Okay, I'll join it when I get ready.

[pavelfeldman]

Also, even though you can't share a repro snippet, if you could tell us what your tests do, it might help. For example, if they take a lot of fullPage screenshots, etc.

[pavelfeldman]

Another question is whether you are using launch or launchPersistent

[jinjor]

Okay.

Also, even though you can't share a repro snippet, if you could tell us what your tests do, it might help. For example, if they take a lot of fullPage screenshots, etc.

• The test is like an E2E test, but the server is just a mock running in the local machine (so the cost of server side logic is very cheap).
• It visits a page, and then click buttons, input text, submit, etc. The app is a SPA, so there are a lot of JSON requests.
• "console", "dialog", "pageerror", "response" hook is used for logging and assertion.
• The first test case takes some screenshots (~100), but it does not take so long.

Another question is whether you are using launch or launchPersistent

launch

[jinjor]

• 1.0.2 - 775089 => fast
  • 410.55s user 56.31s system 105% cpu 7:23.87 total
• 1.1.0 - 764964 => slow (some tests are skipped because of timeout)
  • 1757.85s user 140.82s system 188% cpu 16:46.77 total

1.0.2 with the newer chromium(775089) was fast, so the slowdown should not be caused by chromium.

[pavelfeldman]

Awesome, thank you! Let us prepare a bisect script for you. You probably don't need to run all the tests all the time. With the 4x difference, you could find a test or two that are now much slower and only take tens of seconds to run. That way bisect process will take few minutes altogether.

[pavelfeldman]

@aslushnikov ^^

[jinjor]

I found a workaround. If Page object is created for each test case, the result gets much better.
Here are some results (all on v1.1.1).

Call context.newPage() and page.close() in every test case:

npm run page-test  536.37s user 84.96s system 105% cpu 9:50.80 total

Reuse Page object through all test cases:

npm run page-test  2591.17s user 223.55s system 200% cpu 23:23.48 total

[pavelfeldman]

Oh, you should create entire contexts, not even the pages for each test! @jinjor