AI-Assisted Rules Generation Improvements#925
Merged
Conversation
* Allow input data to be a path * Minor docs update
Contributor
|
✅ 417/417 passed, 35 skipped, 3h3m24s total Running from acceptance #3112 |
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR enhances the AI-assisted rules generation functionality by allowing input data to be specified as either a table name or a file path, adds support for generating rules with filters, and updates documentation with clearer examples and a link to the AI-assisted rules generation demo.
Key Changes
- Extended
generate_dq_rules_ai_assistedto acceptInputConfiginstead of a simple table name string - Added filter support in generated rules with updated training examples and LLM signature
- Updated documentation examples to use
InputConfigand replaced generic reference names like "ref_view" with "ref_df_key"
Reviewed Changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/test_utils.py | Removed test_column_metadata test as the function moved to llm_utils |
| tests/unit/test_llm_utils.py | Added parametrized test for get_column_metadata covering both table and path inputs |
| tests/integration/test_ai_rules_generator.py | Updated tests to use InputConfig and added new test for path-based input |
| src/databricks/labs/dqx/utils.py | Removed get_column_metadata function (moved to llm_utils) |
| src/databricks/labs/dqx/profiler/profiler_runner.py | Updated calls to use InputConfig parameter |
| src/databricks/labs/dqx/profiler/generator.py | Changed generate_dq_rules_ai_assisted signature to accept InputConfig |
| src/databricks/labs/dqx/llm/resources/training_examples.yml | Added filter example to training data |
| src/databricks/labs/dqx/llm/llm_utils.py | Added get_column_metadata function accepting InputConfig |
| src/databricks/labs/dqx/llm/llm_core.py | Updated LLM signature to include filter field guidance |
| docs/dqx/docs/reference/quality_checks.mdx | Changed reference examples from "ref_view" to "ref_df_key" |
| docs/dqx/docs/guide/ai_assisted_quality_checks_generation.mdx | Updated examples to use InputConfig and improved documentation clarity |
| docs/dqx/docs/demos.mdx | Added link to AI-assisted checks generation demo |
| demos/dqx_demo_ai_assisted_checks_generation.py | Updated demo to use InputConfig |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
souravg-db2
approved these changes
Nov 11, 2025
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #925 +/- ##
==========================================
+ Coverage 89.90% 89.93% +0.02%
==========================================
Files 60 60
Lines 5181 5184 +3
==========================================
+ Hits 4658 4662 +4
+ Misses 523 522 -1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
mwojtyczka
added a commit
that referenced
this pull request
Dec 1, 2025
* Generationg of DQX rules from ODCS Data Contracts ([#932](#932)). The Data Contract Quality Rules Generation feature has been introduced, enabling users to generate data quality rules directly from data contracts following the Open Data Contract Standard (ODCS). This feature supports three types of rule generation: predefined rules derived from schema properties and constraints, explicit DQX rules embedded in the contract, and text-based rules defined in natural language and processed by a Large Language Model (LLM) to generate appropriate checks. The feature provides rich metadata tracing generated rules back to the source contract for lineage and governance, and it can be used to implement federated data governance, standardize data contracts, and maintain version-controlled quality rules alongside schema definitions. * AI-Assisted Primary Key Detection and Uniqueness Rules Generation ([#934](#934)). Introduced AI-assisted primary key detection and uniqueness rules generation capabilities, leveraging Large Language Models (LLMs) to analyze table schema and metadata. This feature analyzes table schemas and metadata to intelligently detect single or composite primary keys, and performs validation by checking for duplicate values. The `DQProfiler` class now includes a `detect_primary_keys_with_llm` method, which returns a dictionary containing the primary key detection result, including the table name, success status, detected primary key columns, confidence level, reasoning, and error message if any. The `DQGenerator` class has been extended to utilize uniqueness profiles from the profiler for AI-assisted uniqueness rules generation. Various updates have been made to the configuration options, including the addition of an `llm_primary_key_detection` option, which allows users to control whether AI-assisted primary key detection is enabled or disabled. * AI-Assisted Rules Generation Improvements ([#925](#925)). The AI-Assisted Rules Generation feature has been enhanced to handle input as a path in addition to a table, and to generate rules with a filter. The `generate_dq_rules_ai_assisted` method now accepts an `InputConfig` object, which allows users to specify the location and format of the input data, enabling more flexible input handling and filtering capabilities. The feature includes test cases to verify its functionality, including manual tests, unit tests, and integration tests, and the documentation has been updated with minor changes to reflect the new functionality. Additionally, the code has been modified to capitalize keywords to stabilize integration tests, and the `DQGenerator` class has been updated to accommodate the changes, allowing users to generate data quality rules from a variety of input sources. The `InputConfig` class provides a flexible way to configure the input data, including its location and format, and the `get_column_metadata` function has been introduced to retrieve column metadata from a given location. Overall, these updates aim to enhance the functionality and usability of the AI-assisted rules generation feature, providing more flexibility and accuracy in generating data quality rules. * Added case-insensitive comparison support to is_in_list and is_not_null_and_is_in_list checks ([#673](#673)). The `is_in_list` and `is_not_null_and_is_in_list` check functions have been enhanced to support case-insensitive comparison, allowing users to choose between case-sensitive and case-insensitive comparisons via an optional `case_sensitive` boolean flag that defaults to True. These checks verify if values in a specified column are present in a list of allowed values, with the `is_not_null_and_is_in_list` check also requiring the values to be non-null. The updated checks provide more flexibility in data validation, enabling users to configure parameters such as the column to check, the list of allowed values, and the case sensitivity flag. However, it is recommended to use the `foreign_key` dataset-level check for large lists of allowed values or for columns of type `MapType` or `StructType`, as these checks are not suitable for such scenarios. * Added documentation for using DQX in streaming scenarios with foreach batch ([#948](#948)). Documentation and example code snippets were added to demonstrate how to apply checks in foreachBatch structured streaming function. * Added telemetry to track count of input tables ([#954](#954)). Added additional telemetry for better trakcing of DQX usage to help improve the product. * Added support for installing DQX from private PYPI repositories ([#930](#930)). The DQX library has been enhanced with support for installing DQX using a company-hosted PyPI mirror, which is necessary for enterprises that block the public PyPI index. The documentation has been added to describe the feature. The tool installation code has been modified to include new functionality for automatically upload dependencies to a workspace when internet access is blocked. * Support Custom Folder Installation for CLI Commands ([#942](#942)). The command-line interface (CLI) has been enhanced to support custom installation folders, providing users with greater flexibility when working with the library. A new `--install-folder` argument has been introduced, allowing users to specify a custom installation folder when running various CLI commands, such as opening dashboards, workflows, logs, and profiles. This argument override the default installation location to support scenarios where the user installs DQX in a custom location. The library's dependency on sqlalchemy has also been updated to require a version greater than or equal to 2.0 and less than 3.0 to avoid dependency issues in older DBRs. * Enhancement to end to end tests ([#921](#921)). The e2e tests has been enhanced to test integration with dbt transformation framework. Additionally, the documentation for contributing to the project and testing has been updated to simplify the setup process for running tests locally. BREAKING CHANGES! * Renamed `level` parameter to `criticality` in `generate_dq_rules` method of `DQGenerator` for consistency. * Replaced `table: str` parameter with `input_config: InputConfig` in `profile_table` method of `DQProfiler` for greater flexibility. * Replaced `table_name: str` parameter with `input_config: InputConfig` in `generate_dq_rules_ai_assisted` method of `DQGenerator` for greater flexibility.
Merged
mwojtyczka
added a commit
that referenced
this pull request
Dec 1, 2025
* Generationg of DQX rules from ODCS Data Contracts ([#932](#932)). The Data Contract Quality Rules Generation feature has been introduced, enabling users to generate data quality rules directly from data contracts following the Open Data Contract Standard (ODCS). This feature supports three types of rule generation: predefined rules derived from schema properties and constraints, explicit DQX rules embedded in the contract, and text-based rules defined in natural language and processed by a Large Language Model (LLM) to generate appropriate checks. The feature provides rich metadata tracing generated rules back to the source contract for lineage and governance, and it can be used to implement federated data governance, standardize data contracts, and maintain version-controlled quality rules alongside schema definitions. * AI-Assisted Primary Key Detection and Uniqueness Rules Generation ([#934](#934)). Introduced AI-assisted primary key detection and uniqueness rules generation capabilities, leveraging Large Language Models (LLMs) to analyze table schema and metadata. This feature analyzes table schemas and metadata to intelligently detect single or composite primary keys, and performs validation by checking for duplicate values. The `DQProfiler` class now includes a `detect_primary_keys_with_llm` method, which returns a dictionary containing the primary key detection result, including the table name, success status, detected primary key columns, confidence level, reasoning, and error message if any. The `DQGenerator` class has been extended to utilize uniqueness profiles from the profiler for AI-assisted uniqueness rules generation. Various updates have been made to the configuration options, including the addition of an `llm_primary_key_detection` option, which allows users to control whether AI-assisted primary key detection is enabled or disabled. * AI-Assisted Rules Generation Improvements ([#925](#925)). The AI-Assisted Rules Generation feature has been enhanced to handle input as a path in addition to a table, and to generate rules with a filter. The `generate_dq_rules_ai_assisted` method now accepts an `InputConfig` object, which allows users to specify the location and format of the input data, enabling more flexible input handling and filtering capabilities. The feature includes test cases to verify its functionality, including manual tests, unit tests, and integration tests, and the documentation has been updated with minor changes to reflect the new functionality. Additionally, the code has been modified to capitalize keywords to stabilize integration tests, and the `DQGenerator` class has been updated to accommodate the changes, allowing users to generate data quality rules from a variety of input sources. The `InputConfig` class provides a flexible way to configure the input data, including its location and format, and the `get_column_metadata` function has been introduced to retrieve column metadata from a given location. Overall, these updates aim to enhance the functionality and usability of the AI-assisted rules generation feature, providing more flexibility and accuracy in generating data quality rules. * Added case-insensitive comparison support to is_in_list and is_not_null_and_is_in_list checks ([#673](#673)). The `is_in_list` and `is_not_null_and_is_in_list` check functions have been enhanced to support case-insensitive comparison, allowing users to choose between case-sensitive and case-insensitive comparisons via an optional `case_sensitive` boolean flag that defaults to True. These checks verify if values in a specified column are present in a list of allowed values, with the `is_not_null_and_is_in_list` check also requiring the values to be non-null. The updated checks provide more flexibility in data validation, enabling users to configure parameters such as the column to check, the list of allowed values, and the case sensitivity flag. However, it is recommended to use the `foreign_key` dataset-level check for large lists of allowed values or for columns of type `MapType` or `StructType`, as these checks are not suitable for such scenarios. * Added documentation for using DQX in streaming scenarios with foreach batch ([#948](#948)). Documentation and example code snippets were added to demonstrate how to apply checks in foreachBatch structured streaming function. * Added telemetry to track count of input tables ([#954](#954)). Added additional telemetry for better trakcing of DQX usage to help improve the product. * Added support for installing DQX from private PYPI repositories ([#930](#930)). The DQX library has been enhanced with support for installing DQX using a company-hosted PyPI mirror, which is necessary for enterprises that block the public PyPI index. The documentation has been added to describe the feature. The tool installation code has been modified to include new functionality for automatically upload dependencies to a workspace when internet access is blocked. * Support Custom Folder Installation for CLI Commands ([#942](#942)). The command-line interface (CLI) has been enhanced to support custom installation folders, providing users with greater flexibility when working with the library. A new `--install-folder` argument has been introduced, allowing users to specify a custom installation folder when running various CLI commands, such as opening dashboards, workflows, logs, and profiles. This argument override the default installation location to support scenarios where the user installs DQX in a custom location. The library's dependency on sqlalchemy has also been updated to require a version greater than or equal to 2.0 and less than 3.0 to avoid dependency issues in older DBRs. * Enhancement to end to end tests ([#921](#921)). The e2e tests has been enhanced to test integration with dbt transformation framework. Additionally, the documentation for contributing to the project and testing has been updated to simplify the setup process for running tests locally. BREAKING CHANGES! * Renamed `level` parameter to `criticality` in `generate_dq_rules` method of `DQGenerator` for consistency. * Replaced `table: str` parameter with `input_config: InputConfig` in `profile_table` method of `DQProfiler` for greater flexibility. * Replaced `table_name: str` parameter with `input_config: InputConfig` in `generate_dq_rules_ai_assisted` method of `DQGenerator` for greater flexibility.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
sql_querycheckTests