Skip to content

fix: store flow query ctx on creation#5963

Merged
discord9 merged 6 commits intoGreptimeTeam:mainfrom
discord9:flow_info_schema
Apr 23, 2025
Merged

fix: store flow query ctx on creation#5963
discord9 merged 6 commits intoGreptimeTeam:mainfrom
discord9:flow_info_schema

Conversation

@discord9
Copy link
Copy Markdown
Contributor

@discord9 discord9 commented Apr 23, 2025

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

#5959

What's changed and what's your intention?

flow need to parse raw_sql, hence if schema!=public, on recover flows it will failed to find correct table if raw_sql ref to tables not in public schema, fix is to store schema for flows, and when recover flows using it
hence when recovering flow using raw_sql, need the same QueryContext so that the sql can be parsed as the same logical plan

  • local test confirm upgrade metadata is compatible(the new query context field is #[serde(default)])

PR Checklist

Please convert it to a draft if some of the following conditions are not met.

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR requires documentation updates.
  • API changes are backward compatible.
  • Schema or data changes are backward compatible.

@github-actions github-actions Bot added the docs-not-required This change does not impact docs. label Apr 23, 2025
@discord9 discord9 marked this pull request as draft April 23, 2025 08:13
Copy link
Copy Markdown
Collaborator

@fengjiachun fengjiachun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@discord9 discord9 changed the title fix: store flow schema on creation fix: store flow query ctx on creation Apr 23, 2025
@discord9 discord9 marked this pull request as ready for review April 23, 2025 08:55
@discord9 discord9 added this pull request to the merge queue Apr 23, 2025
Merged via the queue into GreptimeTeam:main with commit 79ed7bb Apr 23, 2025
38 checks passed
@discord9 discord9 deleted the flow_info_schema branch April 23, 2025 10:23
discord9 added a commit to discord9/greptimedb that referenced this pull request Apr 23, 2025
github-merge-queue Bot pushed a commit that referenced this pull request Apr 23, 2025
* feat: use flow batching engine

broken: try using logical plan

fix: use dummy catalog for logical plan

fix: insert plan exec&sqlness grpc addr

feat: use frontend instance in flownode in standalone

feat: flow type in metasrv&fix: flush flow out of sync& column name alias

tests: sqlness update

tests: sqlness flow rebuild udpate

chore: per review

refactor: keep chnl mgr

refactor: use catalog mgr for get table

tests: use valid sql

fix: add more check

refactor: put flow type determine to frontend

* chore: update proto

* chore: update proto to main branch

* fix: add locks for create/drop flow&docs: update docs

* feat: flush_flow flush all ranges now

* test: add align time window test

* docs: explain `nodeid` use in check task

* refactor: AddAutoColumnRewriter check for Projection

* refactor: per review

* fix: query without time window also clean dirty time window

* chore: better logging

* chore: add comments per review

* refactor: per review

* chore: per review

* chore: per review rename args

* refactor: per review partially

* chore: update docs

* chore: use better error variant

* chore: better error variant

* refactor: rename FlowWorkerManager to FlowStreamingEngine

* rename again

* refactor: per review

* chore: rebase after #5963 merged

* refactor: rename all flow_worker_manager occurs

* docs: rm resolved TODO
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-not-required This change does not impact docs.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants