False positives failures in CI but not local since updating CodeceptJS

[sethvanwykJET]

What are you trying to achieve?

Running tests in the pipeline to give the same results as the local environment.

What do you get instead?

So for many tests we've needed to disable them in the CI as the test is unable to find the xpath with the value when trying to see the data on screen.

Provide test source code if related

  I.waitForElement(`//div[@data-qa='delivery-costs-indicator']//div[contains(text(),'Free')]`);
  I.seeElement(`//div[@data-qa='delivery-costs-indicator']//div[contains(text(),'Free')]`);

For this sample above, the wait for element passes correctly, but it's not able to see the same element it just waited for. The allure report mentions it's undefined.

It's also inconsistent. It might pass in one test and then fail in another. Checking the screenshot, the element is clearly present and it always passes in the local environment.

Example from another test

The content it was looking for was present in the screenshot

Details

• CodeceptJS version: 3.4.1 (From 3.2.2)
• NodeJS Version: 16
• Operating System: Uncertain (docker image running in the cloud)
• puppeteer

We recently also updated the docker image running in our CI and since then the problems have started.

# To debug build process locally run the command below in the project root directory.
# `docker build -f ./tools/docker/end2end-tests/Dockerfile -t debug --progress=plain --build-arg WORKDIR=$(pwd) .`

# Download Puppeteer image along with its dependencies
FROM ghcr.io/puppeteer/puppeteer:19.8.3

# Set working directory to current terminal path when initialized with the right build argument.
ARG WORKDIR
WORKDIR $WORKDIR

# Elevated privileges to install packages and run commands
USER root

# Add group first to make sure the ID is assigned consistently
RUN groupadd --system nightmare && useradd --system --create-home --gid nightmare nightmare

# Installing the pre-required packages and libraries
RUN apt-get update
RUN apt-get install -y libgtk2.0-0 libgconf-2-4 \
      libasound2 libxtst6 libxss1 libnss3 xvfb \
      libcairo2-dev libgif-dev libjpeg-dev libpango1.0-dev \
      libpng-dev libgbm-dev curl google-chrome-unstable \
      inetutils-ping default-jdk

RUN export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/

# Install dumb-init to help prevent zombie chrome processes.
ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64 /usr/local/bin/dumb-init
RUN chmod +x /usr/local/bin/dumb-init

# Install Node.js 16
RUN curl -fsSL https://deb.nodesource.com/setup_16.x | bash - \
    && apt-get install -y nodejs

# Install Yarn
RUN curl -sL https://dl.yarnpkg.com/debian/pubkey.gpg | gpg --dearmor | tee /usr/share/keyrings/yarnkey.gpg >/dev/null \
    echo "deb [signed-by=/usr/share/keyrings/yarnkey.gpg] https://dl.yarnpkg.com/debian stable main" | tee /etc/apt/sources.list.d/yarn.list \
    apt-get update && apt-get install yarn -y

# Clean up
RUN  apt-get clean \
  && rm -rf /var/lib/apt/lists/ \
  && rm -rf /var/cache/oracle-jdk8-installer

# Set yarn version needed for repo
RUN yarn set version 3.2.2

# Copy Repo into the image
COPY . /codecept

# Give run permissions to the entrypoint
RUN chmod -R +x /codecept/tools/docker/end2end-tests

# Install dependencies
RUN yarn --cwd /codecept

RUN ln -s /codecept/node_modules/.bin/codeceptjs /usr/local/bin/codeceptjs
RUN ln -s /codecept/node_modules/.bin/allure /usr/local/bin/allure
RUN ln -s /codecept/node_modules/.bin/cross-env /usr/local/bin/cross-env
RUN ln -s /codecept/node_modules/.bin/electron /usr/local/bin/electron

# Create a directory for the test runner
RUN mkdir /tests
WORKDIR /tests

# Allows us to pass arguments to CodeceptJS run via environment variables
ENV CODECEPT_ARGS=""
ENV RUN_MULTIPLE=false
ENV NO_OF_WORKERS=""

# Set HOST ENV variable for Selenium Server
ENV HOST=selenium

# Set the entrypoint for Nightmare and dumb-init.
ENTRYPOINT ["dumb-init", "--", "/codecept/tools/docker/end2end-tests/entrypoint"]

# Initialize the container with bash
CMD ["bash"]

[sethvanwykJET]

I've also tried replacing

RUN ln -s /codecept/node_modules/.bin/codeceptjs /usr/local/bin/codeceptjs

with

RUN ln -s /codecept/node_modules/codeceptjs/bin/codecept.js /usr/local/bin/codeceptjs

But the results were the same, so I don't think it's the difference in how the environment is mounted.

[sethvanwykJET]

I see that between the upgrades that the assertions that were taking about 155ms to validate is immediately failing 4ms in.

waitForAction is currently set to 1500, and waitForTimeout is 10000.

But these 2 don't seem to have an impact on what's happening here.

I tried to look into assertion timeouts but couldn't find anything in the documentation. Am I missing something?

[sethvanwykJET]

Another example of passing locally, but failing in the CI

In this example puppeteer gave a different error:

Execution context was destroyed, most likely because of a navigation.

But the icon was still clearly visible in the screenshot

[sethvanwykJET]

CI behaving strangely was due to this not being configured

https://codecept.io/plugins/#steptimeout

Context being destroyed was because of click() being used instead of clickLink() when navigation takes place.