feat: add DictionaryVector DataType by waynexia · Pull Request #1061 · GreptimeTeam/greptimedb

waynexia · 2023-02-22T09:22:46Z

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

Add a new datatype Dictionary to our type system. But only type definition and conversion are included, without the actual implementation. This is to make schema conversion with arrow work.

This patch also fixes RangeManipulate plan's behavior. It will now included the original timestamp column, and add the "manipulated" timestamp range vector as a new column.

Checklist

I have written the necessary rustdoc comments.
I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

killme2008 · 2023-02-22T14:06:13Z

Test failed:

thread 'tests::promql_test::sql_insert_tql_query_ceil' panicked at 'assertion failed: `(left == right)`
  left: `"+---------------------+-----------+--------------+-------+\\n+---------------------+-----------+--------------+-------+\\n+---------------------+-----------+--------------+-------+\\n| 1970-01-01T00:00:00 | 67        | 1024         | host1 |\\n| 1970-01-01T00:00:10 | 100       | 20480        | host1 |\\n| 1970-01-01T00:00:20 | 100       | 20480        | host1 |\\n| 1970-01-01T00:00:30 | 32        | 8192         | host1 |\\n| 1970-01-01T00:00:40 | 96        | 334          | host1 |\\n| 1970-01-01T00:00:50 | 12424     | 1334         | host1 |\\n| 1970-01-01T00:01:00 | 12424     | 1334         | host1 |\\n| 1970-01-01T00:01:10 | 12424     | 1334         | host1 |\\n| 1970-01-01T00:01:20 | 0         | 2334         | host1 |\\n| 1970-01-01T00:01:30 | 0         | 2334         | host1 |\\n| 1970-01-01T00:01:40 | 49        | 3334         | host1 |\\n| ts                  | ceil(cpu) | ceil(memory) | host  |"`,
 right: `"+---------------------+-----------+--------------+-------+\\n+---------------------+-----------+--------------+-------+\\n+---------------------+-----------+--------------+-------+\\n| 1970-01-01T00:00:00 | 67        | 1024         | host1 |\\n| 1970-01-01T00:00:10 | 100       | 2048         | host1 |\\n| 1970-01-01T00:00:20 | 100       | 2048         | host1 |\\n| 1970-01-01T00:00:30 | 32        | 8192         | host1 |\\n| 1970-01-01T00:00:40 | 96        | 334          | host1 |\\n| 1970-01-01T00:00:50 | 1242      | 1334         | host1 |\\n| 1970-01-01T00:01:00 | 1242      | 1334         | host1 |\\n| 1970-01-01T00:01:10 | 1242      | 1334         | host1 |\\n| 1970-01-01T00:01:20 | 0         | 2334         | host1 |\\n| 1970-01-01T00:01:30 | 0         | 2334         | host1 |\\n| 1970-01-01T00:01:40 | 49        | 3334         | host1 |\\n| ts                  | ceil(cpu) | ceil(memory) | host  |"`', src/datanode/src/tests/test_util.rs:179:5
stack backtrace:

evenyag

Mostly LGTM

evenyag · 2023-02-23T03:35:03Z

Test failed:

thread 'tests::promql_test::sql_insert_tql_query_ceil' panicked at 'assertion failed: `(left == right)`
  left: `"+---------------------+-----------+--------------+-------+\\n+---------------------+-----------+--------------+-------+\\n+---------------------+-----------+--------------+-------+\\n| 1970-01-01T00:00:00 | 67        | 1024         | host1 |\\n| 1970-01-01T00:00:10 | 100       | 20480        | host1 |\\n| 1970-01-01T00:00:20 | 100       | 20480        | host1 |\\n| 1970-01-01T00:00:30 | 32        | 8192         | host1 |\\n| 1970-01-01T00:00:40 | 96        | 334          | host1 |\\n| 1970-01-01T00:00:50 | 12424     | 1334         | host1 |\\n| 1970-01-01T00:01:00 | 12424     | 1334         | host1 |\\n| 1970-01-01T00:01:10 | 12424     | 1334         | host1 |\\n| 1970-01-01T00:01:20 | 0         | 2334         | host1 |\\n| 1970-01-01T00:01:30 | 0         | 2334         | host1 |\\n| 1970-01-01T00:01:40 | 49        | 3334         | host1 |\\n| ts                  | ceil(cpu) | ceil(memory) | host  |"`,
 right: `"+---------------------+-----------+--------------+-------+\\n+---------------------+-----------+--------------+-------+\\n+---------------------+-----------+--------------+-------+\\n| 1970-01-01T00:00:00 | 67        | 1024         | host1 |\\n| 1970-01-01T00:00:10 | 100       | 2048         | host1 |\\n| 1970-01-01T00:00:20 | 100       | 2048         | host1 |\\n| 1970-01-01T00:00:30 | 32        | 8192         | host1 |\\n| 1970-01-01T00:00:40 | 96        | 334          | host1 |\\n| 1970-01-01T00:00:50 | 1242      | 1334         | host1 |\\n| 1970-01-01T00:01:00 | 1242      | 1334         | host1 |\\n| 1970-01-01T00:01:10 | 1242      | 1334         | host1 |\\n| 1970-01-01T00:01:20 | 0         | 2334         | host1 |\\n| 1970-01-01T00:01:30 | 0         | 2334         | host1 |\\n| 1970-01-01T00:01:40 | 49        | 3334         | host1 |\\n| ts                  | ceil(cpu) | ceil(memory) | host  |"`', src/datanode/src/tests/test_util.rs:179:5
stack backtrace:

But this test case is failed
https://github.com/GreptimeTeam/greptimedb/actions/runs/4241443815/jobs/7371668796

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

codecov · 2023-02-23T03:58:05Z

Codecov Report

Merging #1061 (9f698cd) into develop (51641db) will increase coverage by 0.09%.
The diff coverage is 71.67%.

❗ Current head 9f698cd differs from pull request most recent head 39fff84. Consider uploading reports for the commit 39fff84 to get more accurate results

@@             Coverage Diff             @@
##           develop    #1061      +/-   ##
===========================================
+ Coverage    85.70%   85.80%   +0.09%     
===========================================
  Files          450      451       +1     
  Lines        65828    65990     +162     
===========================================
+ Hits         56418    56620     +202     
+ Misses        9410     9370      -40

Flag	Coverage Δ
rust	`85.80% <71.67%> (+0.09%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/api/src/helper.rs	`91.48% <ø> (ø)`
src/common/grpc-expr/src/insert.rs	`81.28% <0.00%> (+0.14%)`	⬆️
src/common/grpc/src/select.rs	`87.42% <ø> (ø)`
src/common/substrait/src/types.rs	`51.92% <0.00%> (ø)`
src/datatypes/src/data_type.rs	`94.96% <0.00%> (-3.30%)`	⬇️
src/datatypes/src/type_id.rs	`38.70% <0.00%> (-5.74%)`	⬇️
src/datatypes/src/types/dictionary_type.rs	`0.00% <0.00%> (ø)`
src/datatypes/src/value.rs	`94.77% <0.00%> (-0.17%)`	⬇️
src/datatypes/src/vectors/eq.rs	`99.32% <ø> (ø)`
src/script/src/python/vector.rs	`70.45% <0.00%> (-0.09%)`	⬇️
... and 15 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

… prom-scalar-udf

v0y4g3r

LGTM

* fix stddev and stdvar. try build range function expr Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * feat: add dictionary data type Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * preserve timestamp column in range manipulator Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * plan range functions Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update test result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix typo Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update test result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update test result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * resolve CR comments Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * resolve CR comments Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

waynexia added 6 commits February 21, 2023 16:04

fix stddev and stdvar. try build range function expr

e10f5dc

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Merge branch 'dev' into prom-scalar-udf

f156a31

feat: add dictionary data type

a634cd2

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

preserve timestamp column in range manipulator

8f1a06a

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

plan range functions

16be440

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

update test result

b3c0862

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

waynexia requested review from evenyag, killme2008 and v0y4g3r February 22, 2023 09:22

waynexia added 3 commits February 22, 2023 17:24

fix typo

50ab9a5

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Merge branch 'dev' into prom-scalar-udf

c950118

update test result

9b2e0b4

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

waynexia mentioned this pull request Feb 22, 2023

feat: add PyO3(Hence CPython as a Optional Backend #976

Merged

2 tasks

evenyag approved these changes Feb 23, 2023

View reviewed changes

Comment thread src/common/grpc/src/select.rs Outdated

Comment thread src/datatypes/src/types/dictionary_type.rs Outdated

Comment thread src/promql/src/planner.rs Outdated

waynexia added 2 commits February 23, 2023 11:35

update test result

a82c6e1

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

resolve CR comments

956ddab

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Merge branch 'develop' into prom-scalar-udf

9f698cd

killme2008 reviewed Feb 23, 2023

View reviewed changes

Comment thread src/datatypes/src/value.rs Outdated

Comment thread src/promql/src/planner.rs Outdated

waynexia added 2 commits February 23, 2023 19:30

resolve CR comments

93cccdb

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Merge branch 'prom-scalar-udf' of github.com:waynexia/greptimedb into…

39fff84

… prom-scalar-udf

waynexia requested a review from killme2008 February 23, 2023 12:21

v0y4g3r approved these changes Feb 23, 2023

View reviewed changes

v0y4g3r merged commit 4cc3ac3 into GreptimeTeam:develop Feb 23, 2023

waynexia deleted the prom-scalar-udf branch February 23, 2023 12:37

waynexia mentioned this pull request Feb 23, 2023

Tracking Issue: Improve PromQL compliance #1042

Closed

35 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add DictionaryVector DataType#1061

feat: add DictionaryVector DataType#1061
v0y4g3r merged 14 commits intoGreptimeTeam:developfrom
waynexia:prom-scalar-udf

waynexia commented Feb 22, 2023 •

edited

Loading

Uh oh!

killme2008 commented Feb 22, 2023

Uh oh!

evenyag left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

evenyag commented Feb 23, 2023

Uh oh!

codecov bot commented Feb 23, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

v0y4g3r left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

waynexia commented Feb 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's changed and what's your intention?

Checklist

Refer to a related PR or issue link (optional)

Uh oh!

killme2008 commented Feb 22, 2023

Uh oh!

evenyag left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

evenyag commented Feb 23, 2023

Uh oh!

codecov bot commented Feb 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

v0y4g3r left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

waynexia commented Feb 22, 2023 •

edited

Loading

codecov bot commented Feb 23, 2023 •

edited

Loading