SeleniumBase v4.34.13+ Misses Responses when using CDP Events

[jdholtz]

Hello. I've been running into an issue with SeleniumBase for a while where the driver doesn't always receive responses in the CDP listener function (which Is why I have been stuck using an old SB version). There have been a few peculiarities that have made this difficult for me to diagnose what exactly the issue is:

• I only see this issue in Docker
• It only happens on driver initializations after the first (the first driver initialization works perfectly fine)
• It only happens when I driver.get() two URLs--the responses are not received for the second URL in this case but are always received for the first URL

I've been able to work this out into a minimal example to reproduce (provided below). I narrowed it down to be added in this commit: 902929a (upgrading to the newest version with this code removed makes the code work perfectly fine--no missed responses). My driver version is 142, so this code path does run.

The script.py file I provided explains the process. The issue I am seeing is that, on the first driver initialization, all responses are received. However, when I quit the driver and then reinitialize it at a later point, I add the CDP listener to the new driver, and driver.get the two URLs again. With this new driver it doesn't see many URLs anymore in the received responses (specifically, a lot of the https://seleniumbase.io/ URLs, which is the second URL used with driver.get). In SB v4.34.12, the URL discrepancy is only a few trivial resources at most (like CSS files).

Do you have any ideas on what I could be doing wrong here with SeleniumBase? Perhaps quitting the driver without doing other cleanup steps (the examples really only show driver.quit being called though)? Sorry if the explanation is somewhat unclear, it is not clear to me exactly what is happening but is definitely affected by the commit I referenced above. I'm happy to provide more details or do more debugging.

script.py

import time

from seleniumbase import Driver

urls1 = set()
urls2 = set()
run = 0


def get_driver():
    driver = Driver(
        undetectable=True, uc_cdp_events=True, driver_version="keep", headless1=True, incognito=True
    )
    driver.add_cdp_listener("Network.responseReceived", response_received)
    driver.get("https://example.com")
    return driver


def response_received(data):
    url = data["params"]["response"]["url"]
    if run == 0:
        urls1.add(url)
    else:
        urls2.add(url)


print("--------------------- Initializing driver ---------------------")
driver = get_driver()
driver.get("https://seleniumbase.io/")
driver.quit()

time.sleep(5)
run = 1

print("--------------------- Initializing driver again ---------------------")
driver = get_driver()
driver.get("https://seleniumbase.io/")
driver.quit()

print()
print("URL comparison")
only_in_first = urls1 - urls2
only_in_second = urls2 - urls1
print(f"URLs only in first run ({len(only_in_first)}):")
for url in only_in_first:
    print(f"  {url}")
print(f"URLs only in second run ({len(only_in_second)}):")
for url in only_in_second:
    print(f"  {url}")

requirements.txt

seleniumbase==4.45.7
# seleniumbase==4.34.12

Dockerfile

FROM python:3.14-alpine3.22

WORKDIR /app

RUN apk add --update --no-cache chromium chromium-chromedriver uv

RUN adduser -D testuser -h /app
RUN chown -R testuser:testuser /app
USER testuser

COPY requirements.txt ./
RUN uv venv /app/.venv \
  && uv pip install --python /app/.venv/bin/python --no-cache -r requirements.txt
# Make sure the Python virtual environment is used
ENV PATH="/app/.venv/bin:$PATH"

COPY script.py script.py

ENTRYPOINT ["python3", "-u", "script.py"]

[mdmintz]

Interesting, when I removed the part specific to 133+, it picked up several internal URLs that weren't part of the websites visited:

  chrome://new-tab-page/calendar_data.mojom-webui.js
  chrome://resources/cr_components/page_image_service/page_image_service.mojom-webui.js
  chrome://new-tab-page/microsoft_auth.mojom-webui.js
  chrome://new-tab-page/microsoft_files.mojom-webui.js
  chrome://new-tab-page/outlook_calendar.mojom-webui.js
  chrome://new-tab-page/lazy_load.js
  chrome://new-tab-page/tab_group_types.mojom-webui.js
  chrome://new-tab-page/file_suggestion.mojom-webui.js
  chrome://new-tab-page/google_calendar.mojom-webui.js
  chrome://new-tab-page/drive_suggestion.mojom-webui.js
  chrome://new-tab-page/most_relevant_tab_resumption.mojom-webui.js
  chrome://new-tab-page/url_visit_types.mojom-webui.js
  chrome://new-tab-page/tab_groups.mojom-webui.js

I don't think I can remove it without chrome extensions appearing as window_handles in driver.window_handles, or internal chrome URLs appearing from "Network.responseReceived". The current code correctly filters out unwanted URLs. It's interesting that you're only seeing your specific issue in Docker.

From what I can tell, the missed responses are only the internal chrome URLs not related to the URLs I'm actually visiting, which is what I would be interested in. Do you have an example where responses were missed that weren't internal chrome URLs?

[jdholtz]

I did try some variations using Xvfb, removing the driver version parameter, using headed=True, etc. but didn't see any different result. However, in the code in the commit I referenced, I did comment out the driver.connect() call and that did fix the issue (I now see seleniumbase.io responses for both instantiations). Maybe that is platform-specific or operates differently based on the chromium driver the Linux distro has? I'm not familiar enough with the SB codebase to know what other effects of removing the driver.connect() call could have though.

[mdmintz]

In the next release, I'll create a way to skip the connect() call if using the following code before launching the browser:

from seleniumbase import config as sb_config
sb_config.skip_133_patch = True

[jdholtz]

That sounds great, thank you!

[mdmintz]

The update is ready. Upgrade to 4.45.8 and use the workaround as needed.

[jdholtz]

That new version with that config works great. Thanks very much for helping me debug and the quick workaround!