Skip to content

feat: CSV Reader Configs #466

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 26 commits into from
Jun 7, 2025
Merged

feat: CSV Reader Configs #466

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
5b18e50
[spark tests]
shehabgamin May 21, 2025
245e7f7
Merge branch 'main' of github.com:lakehq/sail into 1-spark-cov
shehabgamin May 27, 2025
2e02a58
save progress
shehabgamin May 27, 2025
dbcb003
Merge branch 'main' of github.com:lakehq/sail into 1-spark-cov
shehabgamin May 28, 2025
fb43e25
Save
shehabgamin May 29, 2025
397d269
Save
shehabgamin May 29, 2025
efe7c15
save
shehabgamin May 29, 2025
a09e738
Merge branch 'main' of github.com:lakehq/sail into 1-spark-cov
shehabgamin May 31, 2025
63555ba
Save progress
shehabgamin May 31, 2025
a69d749
fix merge conflict
shehabgamin Jun 2, 2025
f66ca6e
csv read options big pain
shehabgamin Jun 3, 2025
5287676
Save
shehabgamin Jun 3, 2025
7c3b4e9
save
shehabgamin Jun 3, 2025
fb2871c
Save
shehabgamin Jun 3, 2025
b47fdf2
fix merge
shehabgamin Jun 6, 2025
25625f0
csv read options progress
shehabgamin Jun 6, 2025
ed8fc14
save
shehabgamin Jun 6, 2025
919dc92
save
shehabgamin Jun 6, 2025
d419519
[spark tests]
shehabgamin Jun 6, 2025
daf46b5
[spark tests]
shehabgamin Jun 6, 2025
918918d
update defaults
shehabgamin Jun 7, 2025
ce521d9
cleanup
shehabgamin Jun 7, 2025
9b1a0ab
[spark tests]
shehabgamin Jun 7, 2025
0adfb78
quote
shehabgamin Jun 7, 2025
8e3bf1c
[spark tests]
shehabgamin Jun 7, 2025
8cca250
[spark tests]
shehabgamin Jun 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 56 additions & 65 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions crates/sail-common/src/config/application.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@

- key: cluster.worker_max_idle_time_secs
type: number
default: "180"
default: "120"
description: The maximum idle time in seconds before a worker is removed.

- key: cluster.worker_heartbeat_interval_secs
Expand Down Expand Up @@ -202,7 +202,7 @@

- key: execution.batch_size
type: number
default: "8192"
default: "16384"
description: The batch size for physical plan execution.
experimental: true

Expand Down Expand Up @@ -456,7 +456,7 @@

- key: parquet.maximum_buffered_record_batches_per_stream
type: number
default: "16"
default: "32"
description: |
(Writing) The maximum number of buffered record batches per stream for the Parquet writer.

Expand Down
Loading