Skip to content

Commit 59bb39d

Browse files
Add note about chunksize blocking the event loop in aio cursors
When chunksize is set, execute() creates a lazy reader instead of loading all data. Subsequent iteration triggers blocking S3 reads not wrapped in asyncio.to_thread(). Document this limitation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent ea55fbc commit 59bb39d

File tree

3 files changed

+24
-0
lines changed

3 files changed

+24
-0
lines changed

docs/aio.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,4 +201,14 @@ row = await cursor.fetchone() # Await required — reads from S3
201201
rows = await cursor.fetchall() # Await required
202202
```
203203

204+
```{note}
205+
When using AioPandasCursor or AioPolarsCursor with the `chunksize` option,
206+
`execute()` creates a lazy reader (e.g., pandas `TextFileReader`) instead of
207+
loading all data at once. Subsequent iteration via `as_pandas()`, `fetchone()`,
208+
or `async for` triggers chunk-by-chunk S3 reads that are **not** wrapped in
209+
`asyncio.to_thread()` and will block the event loop. If you need chunked
210+
processing in an async application, consider wrapping the iteration in
211+
`asyncio.to_thread()` yourself, or use the default non-chunked mode.
212+
```
213+
204214
See each cursor's documentation page for detailed usage examples.

docs/pandas.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -832,3 +832,10 @@ async with await aconnect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/",
832832
await cursor.execute("SELECT * FROM many_rows")
833833
df = cursor.as_pandas()
834834
```
835+
836+
```{note}
837+
When using AioPandasCursor with the `chunksize` option, `execute()` creates a lazy
838+
`TextFileReader` instead of loading all data at once. Subsequent iteration via
839+
`as_pandas()`, `fetchone()`, or `async for` triggers chunk-by-chunk S3 reads that
840+
are not wrapped in `asyncio.to_thread()` and will block the event loop.
841+
```

docs/polars.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -652,3 +652,10 @@ async with await aconnect(s3_staging_dir="s3://YOUR_S3_BUCKET/path/to/",
652652
await cursor.execute("SELECT * FROM many_rows")
653653
df = cursor.as_polars()
654654
```
655+
656+
```{note}
657+
When using AioPolarsCursor with the `chunksize` option, `execute()` creates a lazy
658+
reader instead of loading all data at once. Subsequent iteration via `as_polars()`,
659+
`fetchone()`, or `async for` triggers chunk-by-chunk S3 reads that are not wrapped
660+
in `asyncio.to_thread()` and will block the event loop.
661+
```

0 commit comments

Comments
 (0)