fix: 3145 add read_parquet(use_arrow: bool)

[zilto]

Attempt to fix #3145

Changes

• the main change is a single line: yield batch.to_pylist() -> yield batch if use_pyarrow else batch.to_pylist()
• add basic tests for all readers (previously untested)
• fixed typing for arg items from the incorrect Iterator to Iterable

Future work

• unify reader signatures
• implement CSV reader with pyarrow instead of pandas given the former is a required dependency of dlt
• set default read_parquet(use_pyarrow=True) instead of False because it should be faster and pyarrow is a required deps.
• add a read_parquet_duckdb() reader OR automatically use duckdb inside read_parquet() if duckdb is available

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	docs	22b8645	Commit Preview URL Branch Preview URL	Sep 30 2025, 04:10 PM

[rudolfix]

LGTM! common tests are not passing on pendulum 2.0 - there's no Timezone object there.

[zilto]

@rudolfix any objection to setting use_pyarrow = True as default? User reported a 20x performance improvement from 10min to <1min loading big parquet files

[zilto]

@rudolfix any objection to setting use_pyarrow = True as default? User reported a 20x performance improvement from 10min to <1min loading big parquet files

After discussion, we decided to keep use_pyarrow=False as default for full backwards compatibility.

There could be edge cases that differ between Python and Pyarrow normalization and type inference. This could evolve the schema or break pipelines.

TODO

Once we evaluate some of these edge cases, we should set a deprecation warning akin to

In dlt==X.Y.Z, we're introduced use_pyarrow: bool with default False. Set to True for better performance. In dlt==X.Y.Z, default will be use_pyarrow=True. This should cause no error, but you can try use_pyarrow=True right now to validate it.