feat: export the data from a table to parquet files by fengjiachun · Pull Request #1000 · GreptimeTeam/greptimedb

fengjiachun · 2023-02-14T13:34:03Z

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

To export the data from a table to a Parquet file, use the COPY statement.

COPY tbl TO '/xxx/xxx/output.parquet' WITH (FORMAT = 'parquet');

If the data is too large, it will be automatically split into multiple files, one file per 5 million rows.

Checklist

I have written the necessary rustdoc comments.
I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

codecov · 2023-02-15T04:03:29Z

Codecov Report

Merging #1000 (11692f2) into develop (a9c8584) will decrease coverage by 0.37%.
The diff coverage is 48.24%.

@@             Coverage Diff             @@
##           develop    #1000      +/-   ##
===========================================
- Coverage    86.18%   85.82%   -0.37%     
===========================================
  Files          439      444       +5     
  Lines        63866    64459     +593     
===========================================
+ Hits         55044    55321     +277     
- Misses        8822     9138     +316

Flag	Coverage Δ
rust	`85.82% <48.24%> (-0.37%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/datanode/src/sql/copy_table.rs	`0.00% <0.00%> (ø)`
src/query/src/datafusion/planner.rs	`68.85% <0.00%> (ø)`
src/sql/src/statements.rs	`90.54% <ø> (ø)`
src/sql/src/statements/statement.rs	`50.00% <ø> (ø)`
src/table/src/requests.rs	`61.53% <0.00%> (-5.13%)`	⬇️
src/frontend/src/instance.rs	`79.36% <33.33%> (-0.21%)`	⬇️
src/datanode/src/sql.rs	`79.12% <41.66%> (ø)`
src/datanode/src/instance/sql.rs	`76.39% <45.94%> (-3.96%)`	⬇️
src/sql/src/parsers/copy_parser.rs	`85.33% <85.33%> (ø)`
src/sql/src/statements/copy.rs	`95.65% <95.65%> (ø)`
... and 33 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

fengjiachun · 2023-02-17T06:23:59Z

PTAL @MichaelScofield

MichaelScofield

there're conflicts

killme2008 · 2023-02-18T09:32:06Z

@fengjiachun Maybe we can impl COPY tbl From file too. Just like duckdb https://duckdb.org/docs/sql/statements/copy.html

fengjiachun · 2023-02-19T04:29:08Z

@fengjiachun Maybe we can impl COPY tbl From file too. Just like duckdb https://duckdb.org/docs/sql/statements/copy.html

Good suggestion, I can implement it in a new PR

fengjiachun · 2023-02-20T01:53:44Z

PTAL @MichaelScofield @killme2008

evenyag · 2023-02-20T02:57:15Z

one file per million rows

This might be too small. 10 ~ 50 million should be better choices.

fengjiachun · 2023-02-20T03:32:33Z

one file per million rows

This might be too small. 10 ~ 50 million should be better choices.

I was worried about the high memory usage, I adjusted the default value to 5M, and in the future I think we should add an Option to configure it.

v0y4g3r · 2023-02-20T03:40:20Z

Maybe it's time for us to find a way to support stream writes of SST files. Let me create an issue for this.

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

Co-authored-by: fys <40801205+Fengys123@users.noreply.github.com>

v0y4g3r

LGTM

* feat: copy table parser * feat: coopy table * chore: minor fix * chore: give stmt a more clearer name * chore: unified naming * chore: minor change * chore: add a todo * chore: end up with an empty file when occur an empty table * feat: format with copy table * feat: with options * chore: by cr * chore: default 5M rows per segment * Update src/datanode/src/sql/copy_table.rs Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> * Update src/datanode/src/sql/copy_table.rs Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> * Update src/datanode/src/error.rs Co-authored-by: fys <40801205+Fengys123@users.noreply.github.com> --------- Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: fys <40801205+Fengys123@users.noreply.github.com>

fengjiachun changed the title ~~feat: copy table~~ feat: export the data from a table to a parquet file Feb 14, 2023

fengjiachun marked this pull request as draft February 14, 2023 13:36

fengjiachun force-pushed the feat/copy-table branch 5 times, most recently from 9a54859 to 3fe9f2f Compare February 15, 2023 03:44

fengjiachun marked this pull request as ready for review February 15, 2023 03:45

fengjiachun force-pushed the feat/copy-table branch 3 times, most recently from ed893da to ab8258e Compare February 15, 2023 07:23

fengjiachun changed the title ~~feat: export the data from a table to a parquet file~~ feat: export the data from a table to parquet files Feb 15, 2023

fengjiachun added this to the v0.1 milestone Feb 15, 2023

sunng87 added docs-required This change requires docs update. cloud followup required labels Feb 15, 2023

sunng87 mentioned this pull request Feb 15, 2023

Update docs for feat: export the data from a table to parquet files GreptimeTeam/docs#188

Closed

fengjiachun requested review from MichaelScofield, killme2008 and waynexia February 15, 2023 11:21

MichaelScofield reviewed Feb 16, 2023

View reviewed changes

Comment thread src/datanode/src/sql/copy_table.rs

MichaelScofield reviewed Feb 17, 2023

View reviewed changes

Comment thread src/datanode/src/sql/copy_table.rs Outdated

Comment thread src/sql/src/parsers/copy_parser.rs

fengjiachun force-pushed the feat/copy-table branch from 212ad30 to bbfe8db Compare February 17, 2023 06:21

fengjiachun force-pushed the feat/copy-table branch from 923e71e to 4076880 Compare February 17, 2023 07:58

MichaelScofield reviewed Feb 17, 2023

View reviewed changes

Comment thread src/sql/src/error.rs Outdated

feat: copy table parser

5f207db

feat: coopy table

41b5c21

fengjiachun added 8 commits February 19, 2023 12:32

chore: minor fix

dd28f3d

chore: give stmt a more clearer name

569ac66

chore: unified naming

d37190e

chore: minor change

8b0d217

chore: add a todo

cd1556f

chore: end up with an empty file when occur an empty table

ad3cc60

feat: format with copy table

660370a

feat: with options

1d93620

fengjiachun force-pushed the feat/copy-table branch from 4076880 to f9c91c2 Compare February 19, 2023 04:49

chore: by cr

ca4f564

fengjiachun force-pushed the feat/copy-table branch from f9c91c2 to ca4f564 Compare February 19, 2023 04:53

fengjiachun mentioned this pull request Feb 19, 2023

Copy table from file #1038

Closed

MichaelScofield approved these changes Feb 20, 2023

View reviewed changes

chore: default 5M rows per segment

0fa9aff

v0y4g3r reviewed Feb 20, 2023

View reviewed changes

Comment thread src/datanode/src/sql/copy_table.rs Outdated

Comment thread src/table/src/requests.rs

Comment thread src/datanode/src/sql/copy_table.rs

Comment thread src/datanode/src/sql/copy_table.rs Outdated

Comment thread src/datanode/src/sql/copy_table.rs Outdated

Update src/datanode/src/sql/copy_table.rs

d1c3cce

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

fengys1996 reviewed Feb 20, 2023

View reviewed changes

Comment thread src/datanode/src/error.rs Outdated

fengjiachun and others added 2 commits February 20, 2023 11:50

Update src/datanode/src/sql/copy_table.rs

fee1dec

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

Update src/datanode/src/error.rs

11692f2

Co-authored-by: fys <40801205+Fengys123@users.noreply.github.com>

killme2008 reviewed Feb 20, 2023

View reviewed changes

Comment thread src/datanode/src/sql/copy_table.rs

v0y4g3r approved these changes Feb 20, 2023

View reviewed changes

v0y4g3r merged commit 9161796 into GreptimeTeam:develop Feb 20, 2023

fengjiachun deleted the feat/copy-table branch February 20, 2023 08:45

killme2008 mentioned this pull request May 8, 2023

Export persisted data #604

Closed

Conversation

fengjiachun commented Feb 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's changed and what's your intention?

Checklist

Refer to a related PR or issue link (optional)

Uh oh!

codecov bot commented Feb 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fengjiachun commented Feb 17, 2023

Uh oh!

MichaelScofield left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

killme2008 commented Feb 18, 2023

Uh oh!

fengjiachun commented Feb 19, 2023

Uh oh!

fengjiachun commented Feb 20, 2023

Uh oh!

evenyag commented Feb 20, 2023

Uh oh!

fengjiachun commented Feb 20, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

v0y4g3r commented Feb 20, 2023

Uh oh!

Uh oh!

Uh oh!

v0y4g3r left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

fengjiachun commented Feb 14, 2023 •

edited

Loading

codecov bot commented Feb 15, 2023 •

edited

Loading