Skip to content

feat: use Python Script as UDF in SQL#839

Merged
killme2008 merged 9 commits intodevelopfrom
py_in_sql
Jan 13, 2023
Merged

feat: use Python Script as UDF in SQL#839
killme2008 merged 9 commits intodevelopfrom
py_in_sql

Conversation

@discord9
Copy link
Copy Markdown
Contributor

@discord9 discord9 commented Jan 6, 2023

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

So now you can use sql query like this:

mysql> select system_status(cpu_util, memory_util, disk_util) from system_metrics;
+--------------------------------------------------------------------------------------------+
| system_status(system_metrics.cpu_util,system_metrics.memory_util,system_metrics.disk_util) |
+--------------------------------------------------------------------------------------------+
| green                                                                                      |
| yellow                                                                                     |
| red                                                                                        |
+--------------------------------------------------------------------------------------------+
3 rows in set (0.01 sec)

where system_status is this python script:

@coprocessor(args=["cpu_util", "memory_util", "disk_util"],
             returns = ["status"])
def system_status(cpus, memories, disks)->vector[str]:
    statuses = []
    for cpu, memory, disk in zip(cpus, memories, disks):
        if cpu > 80 or memory > 80 or disk > 80:
            statuses.append("red")
            continue

        status = cpu * 0.4 + memory * 0.4 + disk * 0.2

        if status > 80:
            statuses.append("yellow")
        elif status > 50:
            statuses.append("yellow")
        else:
            statuses.append("green")

    return statuses
  • Now every time PyScript get insert&compile it will insert a UDF with same name
    the relevant type is PyUdf

  • Need to write explict return type in python script i.e. above -> vector[str], so UDF know its' type

  • Can only return one value due to limitation of UDF(Even more vectors are returned, it will be ignore by UDF's eval() function

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link

#675 however still havn't support this feature of creating UDF in SQL(Might need to extend sql parser)
Also fix #232 with Settings to not let Python VM set SIGINT handler(so our tokio can take ctrl_c handler)

@waynexia
Copy link
Copy Markdown
Member

waynexia commented Jan 6, 2023

Coooool!

@codecov
Copy link
Copy Markdown

codecov bot commented Jan 6, 2023

Codecov Report

Merging #839 (2eab28c) into develop (7762873) will increase coverage by 1.03%.
The diff coverage is 77.67%.

@@             Coverage Diff             @@
##           develop     #839      +/-   ##
===========================================
+ Coverage    85.26%   86.29%   +1.03%     
===========================================
  Files          418      422       +4     
  Lines        55285    55885     +600     
===========================================
+ Hits         47140    48228    +1088     
+ Misses        8145     7657     -488     
Flag Coverage Δ
rust 86.29% <77.67%> (+1.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/common/query/src/error.rs 88.04% <ø> (ø)
src/script/src/python/engine.rs 77.02% <70.58%> (-4.00%) ⬇️
src/script/src/manager.rs 94.84% <100.00%> (-0.99%) ⬇️
src/script/src/python/coprocessor.rs 86.32% <100.00%> (+0.27%) ⬆️
src/script/src/python/coprocessor/parse.rs 78.04% <100.00%> (+0.59%) ⬆️
src/script/src/python/test.rs 84.48% <100.00%> (ø)
src/common/query/src/lib.rs 62.50% <0.00%> (-37.50%) ⬇️
src/table/src/table.rs 25.00% <0.00%> (-25.00%) ⬇️
src/servers/src/error.rs 34.54% <0.00%> (-15.46%) ⬇️
src/table/src/requests.rs 55.55% <0.00%> (-6.95%) ⬇️
... and 147 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Copy link
Copy Markdown
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

license-eye has totally checked 645 files.

Valid Invalid Ignored Fixed
539 1 105 0
Click to see the invalid file list
  • src/servers/tests/py_script/mod.rs

Comment thread src/servers/tests/py_script/mod.rs
waynexia and others added 2 commits January 10, 2023 14:18
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Comment thread src/common/query/src/error.rs
Comment thread src/script/src/python/engine.rs
Comment thread src/servers/tests/py_script/mod.rs
@waynexia waynexia requested a review from killme2008 January 10, 2023 09:14
Comment thread src/common/query/src/error.rs
Comment thread src/script/Cargo.toml
Comment thread src/script/src/python/coprocessor.rs
Comment thread src/script/src/python/engine.rs Outdated
Comment thread src/script/src/python/engine.rs Outdated
Comment thread src/script/src/python/engine.rs Outdated
Comment thread src/script/src/python/engine.rs Outdated
Comment thread src/script/src/python/engine.rs Outdated
Copy link
Copy Markdown
Member

@killme2008 killme2008 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread src/script/src/python/engine.rs
@killme2008 killme2008 merged commit e428a84 into develop Jan 13, 2023
@killme2008 killme2008 deleted the py_in_sql branch January 13, 2023 06:35
@killme2008 killme2008 mentioned this pull request Jan 13, 2023
2 tasks
paomian pushed a commit to paomian/greptimedb that referenced this pull request Oct 19, 2023
* feat: reg PyScript as UDF

* refactor: use `ConcreteDataType` instead

* fix: accept `str` data type

* fix: allow binary to capture SIGINT

* test: add test for py udf

* Update src/servers/tests/py_script/mod.rs

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* style: clippy problem

* style: add newline

* chore: PR advices

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unexpected behavior when calling a script

3 participants