feat: track flow source tables for TQL and info schema

[waynexia]

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

Fill the source_table column for TQL flow tasks in batching mode

PR Checklist

Please convert it to a draft if some of the following conditions are not met.

• I have written the necessary rustdoc comments.
• I have added the necessary unit tests and integration tests.
• This PR requires documentation updates.
• API changes are backward compatible.
• Schema or data changes are backward compatible.

[gemini-code-assist]

Summary of Changes

Hello @waynexia, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces the capability to track and expose the source tables used within TQL (Time-series Query Language) flow tasks. By integrating a PromQL parser, the system can now accurately identify the underlying tables referenced in TQL expressions, making this crucial metadata available in the information_schema.flows table. This enhancement improves observability and manageability of data flows by providing clear visibility into their dependencies.

Highlights

• PromQL Parsing Integration: Integrated the promql-parser library to enable parsing of TQL expressions and extracting source table information.
• TQL Source Table Extraction: Implemented new utility functions to recursively traverse PromQL abstract syntax trees and identify source table names, including handling __schema__ and __database__ label matchers.
• Information Schema Update: Modified the information_schema.flows builder to correctly populate the source_table_names column for TQL flow tasks by utilizing the newly implemented PromQL parsing logic.
• Improved Source Table Retrieval: Refactored the source table retrieval mechanism in information_schema.flows to directly use table_info_by_id for efficiency.
• Enhanced Test Coverage: Added comprehensive unit and integration tests to validate the accurate extraction of source table names from various TQL query formats and their correct display in the information schema.

Changelog

• Cargo.lock
  • Added promql-parser dependency.
• src/catalog/src/system_schema/information_schema/flows.rs
  • Updated the logic to retrieve source table names for flows, now using table_info_by_id and the new TQL parsing capabilities.
• src/operator/src/expr_helper.rs
  • Modified existing tests and added new ones to verify the correct extraction of source table names from TQL expressions, including those with schema/database matchers.
• src/sql/Cargo.toml
  • Added promql-parser as a dependency.
• src/sql/src/util.rs
  • Introduced new functions for parsing PromQL expressions to extract source table names, including handling schema and database label matchers, and integrated these into the extract_tables_from_query function.
• tests/cases/distributed/flow-tql/flow_tql.result
  • Updated expected test results to include source_table_names output for TQL flows.
• tests/cases/standalone/flow-tql/flow_tql.result
  • Updated expected test results to include source_table_names output for TQL flows.
• tests/cases/standalone/flow-tql/flow_tql.sql
  • Added SELECT source_table_names queries to test scripts to verify the new functionality.

Activity

• Author confirmed necessary rustdoc comments have been written.
• Author confirmed necessary unit and integration tests have been added.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[waynexia]

@codex review

[gemini-code-assist]

Code Review

This pull request introduces a great feature to track source tables for TQL flows. The implementation correctly parses PromQL queries to extract metric names, which correspond to table names, and also handles schema/database overrides via labels. The refactoring in information_schema/flows.rs to improve the efficiency of fetching source table names is also a good improvement.

My review includes a couple of suggestions to make the TQL parsing logic more robust and predictable, especially regarding how different types of matchers for __name__ are handled and the precedence between __schema__ and __database__ labels.

Overall, this is a solid contribution. Please take a look at my comments.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Can't wait for the next one!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".