Feature/dbt e2e test#921
Merged
Merged
Conversation
Contributor
|
✅ 413/413 passed, 1 flaky, 35 skipped, 3h4m2s total Flaky tests:
Running from acceptance #3092 |
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds end-to-end testing support for the DBT demo by implementing a new test function and updating documentation to reflect the configuration requirements.
- Adds
test_dbt_demofunction to execute DBT projects in E2E tests - Updates documentation to correct
DATABRICKS_HOSTformat and addTEST_DEFAULT_WAREHOUSE_HTTP_PATHconfiguration - Adds dbt-core and dbt-databricks installation to the GitHub Actions workflow
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| tests/e2e/test_run_demos.py | Implements new E2E test for DBT demo with OAuth/token authentication support |
| docs/dqx/docs/reference/testing.mdx | Updates environment variable documentation with corrected host format and warehouse path |
| docs/dqx/docs/dev/contributing.mdx | Updates contributor documentation with corrected host format and warehouse path |
| .github/workflows/acceptance.yml | Adds dbt-core and dbt-databricks package installation step |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #921 +/- ##
=======================================
Coverage 89.93% 89.93%
=======================================
Files 60 60
Lines 5166 5166
=======================================
Hits 4646 4646
Misses 520 520 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
mwojtyczka
added a commit
that referenced
this pull request
Dec 1, 2025
* Generationg of DQX rules from ODCS Data Contracts ([#932](#932)). The Data Contract Quality Rules Generation feature has been introduced, enabling users to generate data quality rules directly from data contracts following the Open Data Contract Standard (ODCS). This feature supports three types of rule generation: predefined rules derived from schema properties and constraints, explicit DQX rules embedded in the contract, and text-based rules defined in natural language and processed by a Large Language Model (LLM) to generate appropriate checks. The feature provides rich metadata tracing generated rules back to the source contract for lineage and governance, and it can be used to implement federated data governance, standardize data contracts, and maintain version-controlled quality rules alongside schema definitions. * AI-Assisted Primary Key Detection and Uniqueness Rules Generation ([#934](#934)). Introduced AI-assisted primary key detection and uniqueness rules generation capabilities, leveraging Large Language Models (LLMs) to analyze table schema and metadata. This feature analyzes table schemas and metadata to intelligently detect single or composite primary keys, and performs validation by checking for duplicate values. The `DQProfiler` class now includes a `detect_primary_keys_with_llm` method, which returns a dictionary containing the primary key detection result, including the table name, success status, detected primary key columns, confidence level, reasoning, and error message if any. The `DQGenerator` class has been extended to utilize uniqueness profiles from the profiler for AI-assisted uniqueness rules generation. Various updates have been made to the configuration options, including the addition of an `llm_primary_key_detection` option, which allows users to control whether AI-assisted primary key detection is enabled or disabled. * AI-Assisted Rules Generation Improvements ([#925](#925)). The AI-Assisted Rules Generation feature has been enhanced to handle input as a path in addition to a table, and to generate rules with a filter. The `generate_dq_rules_ai_assisted` method now accepts an `InputConfig` object, which allows users to specify the location and format of the input data, enabling more flexible input handling and filtering capabilities. The feature includes test cases to verify its functionality, including manual tests, unit tests, and integration tests, and the documentation has been updated with minor changes to reflect the new functionality. Additionally, the code has been modified to capitalize keywords to stabilize integration tests, and the `DQGenerator` class has been updated to accommodate the changes, allowing users to generate data quality rules from a variety of input sources. The `InputConfig` class provides a flexible way to configure the input data, including its location and format, and the `get_column_metadata` function has been introduced to retrieve column metadata from a given location. Overall, these updates aim to enhance the functionality and usability of the AI-assisted rules generation feature, providing more flexibility and accuracy in generating data quality rules. * Added case-insensitive comparison support to is_in_list and is_not_null_and_is_in_list checks ([#673](#673)). The `is_in_list` and `is_not_null_and_is_in_list` check functions have been enhanced to support case-insensitive comparison, allowing users to choose between case-sensitive and case-insensitive comparisons via an optional `case_sensitive` boolean flag that defaults to True. These checks verify if values in a specified column are present in a list of allowed values, with the `is_not_null_and_is_in_list` check also requiring the values to be non-null. The updated checks provide more flexibility in data validation, enabling users to configure parameters such as the column to check, the list of allowed values, and the case sensitivity flag. However, it is recommended to use the `foreign_key` dataset-level check for large lists of allowed values or for columns of type `MapType` or `StructType`, as these checks are not suitable for such scenarios. * Added documentation for using DQX in streaming scenarios with foreach batch ([#948](#948)). Documentation and example code snippets were added to demonstrate how to apply checks in foreachBatch structured streaming function. * Added telemetry to track count of input tables ([#954](#954)). Added additional telemetry for better trakcing of DQX usage to help improve the product. * Added support for installing DQX from private PYPI repositories ([#930](#930)). The DQX library has been enhanced with support for installing DQX using a company-hosted PyPI mirror, which is necessary for enterprises that block the public PyPI index. The documentation has been added to describe the feature. The tool installation code has been modified to include new functionality for automatically upload dependencies to a workspace when internet access is blocked. * Support Custom Folder Installation for CLI Commands ([#942](#942)). The command-line interface (CLI) has been enhanced to support custom installation folders, providing users with greater flexibility when working with the library. A new `--install-folder` argument has been introduced, allowing users to specify a custom installation folder when running various CLI commands, such as opening dashboards, workflows, logs, and profiles. This argument override the default installation location to support scenarios where the user installs DQX in a custom location. The library's dependency on sqlalchemy has also been updated to require a version greater than or equal to 2.0 and less than 3.0 to avoid dependency issues in older DBRs. * Enhancement to end to end tests ([#921](#921)). The e2e tests has been enhanced to test integration with dbt transformation framework. Additionally, the documentation for contributing to the project and testing has been updated to simplify the setup process for running tests locally. BREAKING CHANGES! * Renamed `level` parameter to `criticality` in `generate_dq_rules` method of `DQGenerator` for consistency. * Replaced `table: str` parameter with `input_config: InputConfig` in `profile_table` method of `DQProfiler` for greater flexibility. * Replaced `table_name: str` parameter with `input_config: InputConfig` in `generate_dq_rules_ai_assisted` method of `DQGenerator` for greater flexibility.
Merged
mwojtyczka
added a commit
that referenced
this pull request
Dec 1, 2025
* Generationg of DQX rules from ODCS Data Contracts ([#932](#932)). The Data Contract Quality Rules Generation feature has been introduced, enabling users to generate data quality rules directly from data contracts following the Open Data Contract Standard (ODCS). This feature supports three types of rule generation: predefined rules derived from schema properties and constraints, explicit DQX rules embedded in the contract, and text-based rules defined in natural language and processed by a Large Language Model (LLM) to generate appropriate checks. The feature provides rich metadata tracing generated rules back to the source contract for lineage and governance, and it can be used to implement federated data governance, standardize data contracts, and maintain version-controlled quality rules alongside schema definitions. * AI-Assisted Primary Key Detection and Uniqueness Rules Generation ([#934](#934)). Introduced AI-assisted primary key detection and uniqueness rules generation capabilities, leveraging Large Language Models (LLMs) to analyze table schema and metadata. This feature analyzes table schemas and metadata to intelligently detect single or composite primary keys, and performs validation by checking for duplicate values. The `DQProfiler` class now includes a `detect_primary_keys_with_llm` method, which returns a dictionary containing the primary key detection result, including the table name, success status, detected primary key columns, confidence level, reasoning, and error message if any. The `DQGenerator` class has been extended to utilize uniqueness profiles from the profiler for AI-assisted uniqueness rules generation. Various updates have been made to the configuration options, including the addition of an `llm_primary_key_detection` option, which allows users to control whether AI-assisted primary key detection is enabled or disabled. * AI-Assisted Rules Generation Improvements ([#925](#925)). The AI-Assisted Rules Generation feature has been enhanced to handle input as a path in addition to a table, and to generate rules with a filter. The `generate_dq_rules_ai_assisted` method now accepts an `InputConfig` object, which allows users to specify the location and format of the input data, enabling more flexible input handling and filtering capabilities. The feature includes test cases to verify its functionality, including manual tests, unit tests, and integration tests, and the documentation has been updated with minor changes to reflect the new functionality. Additionally, the code has been modified to capitalize keywords to stabilize integration tests, and the `DQGenerator` class has been updated to accommodate the changes, allowing users to generate data quality rules from a variety of input sources. The `InputConfig` class provides a flexible way to configure the input data, including its location and format, and the `get_column_metadata` function has been introduced to retrieve column metadata from a given location. Overall, these updates aim to enhance the functionality and usability of the AI-assisted rules generation feature, providing more flexibility and accuracy in generating data quality rules. * Added case-insensitive comparison support to is_in_list and is_not_null_and_is_in_list checks ([#673](#673)). The `is_in_list` and `is_not_null_and_is_in_list` check functions have been enhanced to support case-insensitive comparison, allowing users to choose between case-sensitive and case-insensitive comparisons via an optional `case_sensitive` boolean flag that defaults to True. These checks verify if values in a specified column are present in a list of allowed values, with the `is_not_null_and_is_in_list` check also requiring the values to be non-null. The updated checks provide more flexibility in data validation, enabling users to configure parameters such as the column to check, the list of allowed values, and the case sensitivity flag. However, it is recommended to use the `foreign_key` dataset-level check for large lists of allowed values or for columns of type `MapType` or `StructType`, as these checks are not suitable for such scenarios. * Added documentation for using DQX in streaming scenarios with foreach batch ([#948](#948)). Documentation and example code snippets were added to demonstrate how to apply checks in foreachBatch structured streaming function. * Added telemetry to track count of input tables ([#954](#954)). Added additional telemetry for better trakcing of DQX usage to help improve the product. * Added support for installing DQX from private PYPI repositories ([#930](#930)). The DQX library has been enhanced with support for installing DQX using a company-hosted PyPI mirror, which is necessary for enterprises that block the public PyPI index. The documentation has been added to describe the feature. The tool installation code has been modified to include new functionality for automatically upload dependencies to a workspace when internet access is blocked. * Support Custom Folder Installation for CLI Commands ([#942](#942)). The command-line interface (CLI) has been enhanced to support custom installation folders, providing users with greater flexibility when working with the library. A new `--install-folder` argument has been introduced, allowing users to specify a custom installation folder when running various CLI commands, such as opening dashboards, workflows, logs, and profiles. This argument override the default installation location to support scenarios where the user installs DQX in a custom location. The library's dependency on sqlalchemy has also been updated to require a version greater than or equal to 2.0 and less than 3.0 to avoid dependency issues in older DBRs. * Enhancement to end to end tests ([#921](#921)). The e2e tests has been enhanced to test integration with dbt transformation framework. Additionally, the documentation for contributing to the project and testing has been updated to simplify the setup process for running tests locally. BREAKING CHANGES! * Renamed `level` parameter to `criticality` in `generate_dq_rules` method of `DQGenerator` for consistency. * Replaced `table: str` parameter with `input_config: InputConfig` in `profile_table` method of `DQProfiler` for greater flexibility. * Replaced `table_name: str` parameter with `input_config: InputConfig` in `generate_dq_rules_ai_assisted` method of `DQGenerator` for greater flexibility.
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Updated End-to-End tests with DBT demo
Linked issues
This PR addresses the [FEATURE]: dbt demo to e2e tests #496
Tests