[BUG] Page.goto() flakiness - seeing TimeoutError sometimes

[dialupdev]

Context:

• Playwright Version: v1.2.1
• Operating System: Docker on Linux
• Node version: v12.18.2
• Browser: Chromium

Describe the bug

Hey team. I hope all is well. I'm seeing occasional timeouts during page.goto(). I haven't found a way to repro this constistently - it only happens about 5% 20% of the time. The strange thing is that in these cases, the navigation actually did succeed. I can tell by looking at screenshots and videos. It's just that Playwright doesn't recognize the load event.

TimeoutError: Timeout 30000ms exceeded during page.goto.
====================== page.goto logs ======================
[api] navigating to "https://portfolio.adobe.com", waiting until "load"
[api] navigated to "https://portfolio.adobe.com/"
[api] "domcontentloaded" event fired
============================================================
Note: use DEBUG=pw:api environment variable and rerun to capture Playwright logs.

  147 |   // we have an environment to insert the session into
  148 |   if (page.url().indexOf(CONFIG.baseUrl) === -1) {
> 149 |     await page.goto(CONFIG.baseUrl);
      |                ^
  150 |     await page.waitForSelector(Home.CTAHeader);
  151 |   }
  152 | }

  at ProgressController.run (node_modules/playwright-chromium/lib/progress.js:73:30)
  at runNavigationTask (node_modules/playwright-chromium/lib/frames.js:961:23)
  at Frame.goto (node_modules/playwright-chromium/lib/frames.js:284:16)
  at Frame.goto (node_modules/playwright-chromium/lib/helper.js:79:31)
  at node_modules/playwright-chromium/lib/page.js:203:61
  at Page._attributeToPage (node_modules/playwright-chromium/lib/page.js:356:20)
  at Page.goto (node_modules/playwright-chromium/lib/page.js:203:21)
  at Page.apply (node_modules/playwright-chromium/lib/helper.js:79:31)
    -- ASYNC --
  at Frame.goto (node_modules/playwright-chromium/lib/helper.js:78:23)
  at node_modules/playwright-chromium/lib/page.js:203:61
  at Page._attributeToPage (node_modules/playwright-chromium/lib/page.js:356:20)
  at Page.goto (node_modules/playwright-chromium/lib/page.js:203:21)
  at Page.apply (node_modules/playwright-chromium/lib/helper.js:79:31)
  at _maybeGoToHomepage (helpers/authHelper.ts:149:16)
  at command (helpers/authHelper.ts:206:24)
    -- ASYNC --
  at Page.apply (node_modules/playwright-chromium/lib/helper.js:78:23)
  at _maybeGoToHomepage (helpers/authHelper.ts:149:16)
  at command (helpers/authHelper.ts:206:24)
  at signupAndLogin (helpers/authHelper.ts:210:10)
  at Object.<anonymous> (specs/features/onboarding.critical.ts:26:11)

Have you seen this at all before? Do you have any idea what could be causing this or how I could repro this issue consistently? As a potential workaround, is there any way to change the default event to domcontentloaded for all page.goto() calls?

Thank you for taking a look.

[dgozman]

Hi Steve! I noticed that https://portfolio.adobe.com has a doubleclick iframe. Could you please try to repro while blocking that iframe, so that we know whether the issue is in the main frame or the iframe?

await page.route(/doubleclick/, route => route.abort());

[dialupdev]

Thank you for the suggestion @dgozman. I'm still seeing pretty frequent failures though after making that change. Any other potential causes to look out for, or ideas about how to isolate the cause?

[dgozman]

@celeryclub No more ideas just yet. Does this always happen on "https://portfolio.adobe.com" ? If so, I can try stress-testing locally.

[dgozman]

I tried running locally with the above url, reusing a single page and creating a new page, but so far never had a timeout. More details/help would be appreciated 😄

[dialupdev]

Thanks so much for trying this. I'm going to try to collect some more info about how to repro.

[dialupdev]

@dgozman Would you be able to share with me the script you used to run stress tests?

[dgozman]

Sorry, it was just a part of our test. Here is something I'd try:

async function loop(reusePage) {
  let page = await browser.newPage();
  while (true) {
    if (!reusePage) {
      await page.close();
      page = await browser.newPage();
    }
    await page.goto('....');
  }
}

// 20 loops in parallel
for (let i = 0; i < 20; i++)
  loop(i < 10);

[dialupdev]

Thank you! I'll give it a go.

[dialupdev]

I tried this script and also didn't see any failures, though I'm still seeing them occasionally in my Jenkins job. Do you think there's a way to find out what resources are pending whenever this error happens? Like something to put in a catch.

[dgozman]

The best way would be to run with DEBUG=pw:protocol and send the log to us. Note that it contains a lot of information like cookies, so make sure you are comfortable sharing that.

Otherwise, perhaps logging all requests would help. Try adding the following and sharing the output.

page.on('request', r => console.log('request', r.url()));
page.on('requestfailed', r => console.log('requestfailed', r.url()));
page.on('requestfinished', r => console.log('requestfinished', r.url()));

[aslushnikov]

Closing since the discussion has stalled here.