feat: add an internal HttpClient to be used in send_request for PlaywrightCrawler using APIRequestContext bound to the browser context

[Mantisus]

Description

• This PR, adds an internal HttpClient implementation for PlaywrightCrawler using the APIRequestContext from Playwright. This ensures that HTTP requests use the same proxies as the browser context.

Issues

• Closes: Implement a playwright-based HttpClient implementation #928

[Mantisus]

After discussions with @janbuchar in Slack, we came to the decision that this approach is best for PlaywrightCrawler. The requests will be closer to the browser context, and will use the same proxies that were set when the context was opened.

However, there is a problem. Playwright does not propagate the headers set with set_extra_http_headers to the request context. This results in the request context not receiving fingerprint headers.

It relevant #1055

Refactoring would be required to pass the headers set when opening the browser context to the crawling context. Or wait for Playwright to do something about it on their end

[Copilot]

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

tests/unit/crawlers/_playwright/test_playwright_crawler.py:486

• Ensure that send_request_response.read() is decoded (e.g. using .decode('utf-8')) before passing it to json.loads to avoid potential type errors when parsing bytes.

check_data['send_request'] = dict(json.loads(send_request_response.read()))

[janbuchar]

Nice, just a couple of nits 🙂

[vdusek]

LGTM