Crawling very slow and timeout error #354

Jourdelune · 2024-07-24T12:38:57Z

Hello, I'm experiencing performance issues with my web crawler after approximately 1.5 to 2 hours of runtime. The crawling speed significantly decreases to about one site per minute or less, and I'm encountering numerous timeout errors.

Questions:

What could be causing the performance degradation over time (maybe related to the queue size) ?
Is this behavior correct?

Here is the code I use:

import asyncio

from crawlee.beautifulsoup_crawler import (
    BeautifulSoupCrawler,
    BeautifulSoupCrawlingContext,
)


async def main() -> None:
    crawler = BeautifulSoupCrawler()

    @crawler.router.default_handler
    async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
        url = context.request.url
        context.log.info(f"Processing page: {url}")
        await context.enqueue_links(strategy="all")

    # Start the crawler with the provided URLs
    await crawler.run(["https://crawlee.dev/"])


if __name__ == "__main__":
    asyncio.run(main())

The logs and errors:

[crawlee.beautifulsoup_crawler.beautifulsoup_crawler] ERROR An exception occurred during handling of a request. This places the crawler and its underlying storages into an unknown state and crawling will be terminated.
      Traceback (most recent call last):
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/crawlee/basic_crawler/basic_crawler.py", line 872, in __run_request_handler
          await self._context_pipeline(crawling_context, self.router)
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/crawlee/basic_crawler/context_pipeline.py", line 62, in __call__
          result = await middleware_instance.__anext__()
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/crawlee/beautifulsoup_crawler/beautifulsoup_crawler.py", line 69, in _make_http_request
          result = await self._http_client.crawl(
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/crawlee/http_clients/httpx_client.py", line 98, in crawl
          response = await client.send(http_request, follow_redirects=True)
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/httpx/_client.py", line 1675, in send
          raise exc
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/httpx/_client.py", line 1669, in send
          await response.aread()
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/httpx/_models.py", line 911, in aread
          self._content = b"".join([part async for part in self.aiter_bytes()])
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/httpx/_models.py", line 911, in <listcomp>
          self._content = b"".join([part async for part in self.aiter_bytes()])
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/httpx/_models.py", line 929, in aiter_bytes
          async for raw_bytes in self.aiter_raw():
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/httpx/_models.py", line 987, in aiter_raw
          async for raw_stream_bytes in self.stream:
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/httpx/_client.py", line 149, in __aiter__
          async for chunk in self._stream:
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 254, in __aiter__
          async for part in self._httpcore_stream:
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 367, in __aiter__
          raise exc from None
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 363, in __aiter__
          async for part in self._stream:
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 349, in __aiter__
          raise exc
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 341, in __aiter__
          async for chunk in self._connection._receive_response_body(**kwargs):
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 210, in _receive_response_body
          event = await self._receive_event(timeout=timeout)
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/httpcore/_async/http11.py", line 224, in _receive_event
          data = await self._network_stream.read(
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/httpcore/_backends/anyio.py", line 35, in read
          return await self._stream.receive(max_bytes=max_bytes)
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/anyio/streams/tls.py", line 205, in receive
          data = await self._call_sslobject_method(self._ssl_object.read, max_bytes)
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/anyio/streams/tls.py", line 147, in _call_sslobject_method
          data = await self.transport_stream.receive()
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1145, in receive
          await AsyncIOBackend.checkpoint()
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2050, in checkpoint
          await sleep(0)
        File "/home/debian/.pyenv/versions/3.10.4/lib/python3.10/asyncio/tasks.py", line 596, in sleep
          await __sleep0()
        File "/home/debian/.pyenv/versions/3.10.4/lib/python3.10/asyncio/tasks.py", line 590, in __sleep0
          yield
      asyncio.exceptions.CancelledError

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/home/debian/.pyenv/versions/3.10.4/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
          return fut.result()
      asyncio.exceptions.CancelledError

      The above exception was the direct cause of the following exception:

      Traceback (most recent call last):
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/crawlee/_utils/wait.py", line 37, in wait_for
          return await asyncio.wait_for(operation(), timeout.total_seconds())
        File "/home/debian/.pyenv/versions/3.10.4/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
          raise exceptions.TimeoutError() from exc
      asyncio.exceptions.TimeoutError

      The above exception was the direct cause of the following exception:

      Traceback (most recent call last):
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/crawlee/basic_crawler/basic_crawler.py", line 782, in __run_task_function
          await wait_for(
        File "/home/debian/Crawler/venv/lib/python3.10/site-packages/crawlee/_utils/wait.py", line 39, in wait_for
          raise asyncio.TimeoutError(timeout_message) from ex
      asyncio.exceptions.TimeoutError: Request handler timed out after 60.0 seconds
[crawlee.autoscaling.autoscaled_pool] WARN  Task timed out after *not set* seconds
d[crawlee.statistics.statistics] INFO  crawlee.beautifulsoup_crawler.beautifulsoup_crawler request statistics {
        "requests_finished": 5622,
        "requests_failed": 1,
        "retry_histogram": [
          5622,
          0,
          1
        ],
        "request_avg_failed_duration": 0.859078,
        "request_avg_finished_duration": 50.458161,
        "requests_finished_per_minute": 127,
        "requests_failed_per_minute": 0,
        "request_total_duration": 283676.640522,
        "requests_total": 5623,
        "crawler_runtime": 2646.37724
      }
[crawlee.autoscaling.autoscaled_pool] INFO  current_concurrency = 197; desired_concurrency = 161; cpu = 0.0; mem = 0.0; ev

The text was updated successfully, but these errors were encountered:

Jourdelune · 2024-07-24T17:33:39Z

It use a lot of ram also (after 4h-6h of crawl):

[crawlee.autoscaling.autoscaled_pool] INFO  current_concurrency = 183; desired_concurrency = 173; cpu = 0.0; mem = 0.0; event_loop = 0.252; client_info = 0.0
[crawlee.autoscaling.snapshotter] WARN  Memory is critically overloaded. Using 7.04 GB of 7.81 GB (90%). Consider increasing available memory.
[crawlee.statistics.statistics] INFO  crawlee.beautifulsoup_crawler.beautifulsoup_crawler request statistics {
        "requests_finished": 30381,
        "requests_failed": 7,
        "retry_histogram": [
          30374,
          7,
          7
        ],
        "request_avg_failed_duration": 1.340926,
        "request_avg_finished_duration": 120.59418,
        "requests_finished_per_minute": 87,
        "requests_failed_per_minute": 0,
        "request_total_duration": 3663781.171706,
        "requests_total": 30388,
        "crawler_runtime": 20939.883378
      }

It's the user that have to limit the number of url added in the queue or the lib manage that? (hard limit etc)

janbuchar · 2024-07-24T19:59:13Z

Interesting. What is your total available memory?

Jourdelune · 2024-07-24T20:29:33Z

32 Giga is available on my system.

Jourdelune · 2024-07-29T08:03:19Z

I export my storage on Google Drive so you can test that: https://drive.google.com/file/d/1P8AgbgbVLmujiceYRtMIKK91zn9GVjen/view?usp=sharing

CRAWLEE_PURGE_ON_START=0 python test.py

When there is a lot of pending requests, crawlee is very very slow.

marisancans · 2024-08-09T19:52:32Z

I'm seeing slow scraping too. About 200 requests per minute. I even self host the webpage to scrape. There are numerous times when scraper literally does nothing and waits for something.

janbuchar · 2024-08-12T09:20:09Z

@marisancans Would you mind sharing your scraper code as well? It might help us debug.

black7375 · 2024-08-14T09:10:35Z

I also have the warning Consider increasing available memory., is there any way for the user to control the memory allocation?

janbuchar · 2024-08-15T10:53:17Z

I also have the warning Consider increasing available memory., is there any way for the user to control the memory allocation?

Unless you're limiting the memory usage knowingly, no, there isn't, at least without digging deep in Crawlee's internals. Of course, if you're working with a cloud platform such as Apify, you can configure the available memory there.

black7375 · 2024-08-15T13:06:47Z

Thank you for your response :D

- closes #423 - closes #174 - closes #203 - related to #354 - let's investigate this, but I believe that it won't go away until #433 is resolved - closes #83 Locking in memory storage was not implemented - see #433 --------- Co-authored-by: Vlada Dusek <[email protected]>

khorovodnykov-vladyslav · 2025-01-30T09:54:38Z

I'm seeing slow scraping too. About 200 requests per minute. I even self host the webpage to scrape. There are numerous times when scraper literally does nothing and waits for something.

What is your PC setup (RAM, CPU)? I try to increase speed and can't go over 25 req/min. Maybe you can advise something to increase speed, some parameters, I use concurrency set.: ConcurrencySettings(min_concurrency=10, max_concurrency=200, max_tasks_per_minute=200, desired_concurrency=110)

vdusek added t-tooling Issues with this label are in the ownership of the tooling team. bug Something isn't working. labels Jul 25, 2024

janbuchar self-assigned this Jul 30, 2024

janbuchar mentioned this issue Aug 13, 2024

refactor!: RequestQueue and service management rehaul #429

Merged

janbuchar mentioned this issue Dec 16, 2024

How to economize on RAM when starting a crawl with a large list of urls? #816

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crawling very slow and timeout error #354

Crawling very slow and timeout error #354

Jourdelune commented Jul 24, 2024 •

edited

Loading

Jourdelune commented Jul 24, 2024 •

edited

Loading

janbuchar commented Jul 24, 2024

Jourdelune commented Jul 24, 2024

Jourdelune commented Jul 29, 2024

marisancans commented Aug 9, 2024

janbuchar commented Aug 12, 2024

black7375 commented Aug 14, 2024 •

edited

Loading

janbuchar commented Aug 15, 2024

black7375 commented Aug 15, 2024

khorovodnykov-vladyslav commented Jan 30, 2025

Crawling very slow and timeout error #354

Crawling very slow and timeout error #354

Comments

Jourdelune commented Jul 24, 2024 • edited Loading

Jourdelune commented Jul 24, 2024 • edited Loading

janbuchar commented Jul 24, 2024

Jourdelune commented Jul 24, 2024

Jourdelune commented Jul 29, 2024

marisancans commented Aug 9, 2024

janbuchar commented Aug 12, 2024

black7375 commented Aug 14, 2024 • edited Loading

janbuchar commented Aug 15, 2024

black7375 commented Aug 15, 2024

khorovodnykov-vladyslav commented Jan 30, 2025

Jourdelune commented Jul 24, 2024 •

edited

Loading

Jourdelune commented Jul 24, 2024 •

edited

Loading

black7375 commented Aug 14, 2024 •

edited

Loading