Add documentation for using DQEngine in foreachBatch#948
Merged
Conversation
Contributor
|
✅ 427/427 passed, 1 flaky, 41 skipped, 2h55m39s total Flaky tests:
Running from acceptance #3268 |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #948 +/- ##
==========================================
- Coverage 90.35% 90.30% -0.06%
==========================================
Files 62 62
Lines 5569 5569
==========================================
- Hits 5032 5029 -3
- Misses 537 540 +3 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
ghanse
changed the title
mwojtyczka
approved these changes
Nov 27, 2025
| Spark's `foreachBatch` allows users to define an arbitrary processing function run for each streaming batch. When using DQX in a `foreachBatch` function, create a `DQEngine` instance with a mock `WorkspaceClient`. | ||
|
|
||
| <Admonition type="warning" title="Engine functionality with mocked WorkspaceClient"> | ||
| Some DQX engine methods require a valid Databricks `WorkspaceClient`. Methods must support local execution to be run in a `foreachBatch` function with a mocked `WorkspaceClient`. Check the **Supports local execution** column [here](/docs/reference/engine/#dqx-engine-methods) to verify that DQX engine methods can be executed with a mocked client. |
Contributor
There was a problem hiding this comment.
maybe add info that workspace client authentication is not supported by spark inside the foreach batch.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.
|
|
||
| # Read the data and apply the `foreachBatch` function | ||
| input_data = spark.readStream.format("rate-micro-batch").option("rowsPerBatch", 100).load() | ||
| input_data.writeStream.foreachBatch(validate_and_write_micro_batch_data).start() |
There was a problem hiding this comment.
The example code is missing .trigger() configuration and .awaitTermination() to properly manage the streaming query lifecycle. Consider adding these for a complete working example, such as .trigger(availableNow=True).foreachBatch(validate_and_write_micro_batch_data).start().awaitTermination().
Suggested change
| input_data.writeStream.foreachBatch(validate_and_write_micro_batch_data).start() | |
| query = input_data.writeStream.trigger(availableNow=True).foreachBatch(validate_and_write_micro_batch_data).start() | |
| query.awaitTermination() |
mwojtyczka
added a commit
that referenced
this pull request
Dec 1, 2025
* Generationg of DQX rules from ODCS Data Contracts ([#932](#932)). The Data Contract Quality Rules Generation feature has been introduced, enabling users to generate data quality rules directly from data contracts following the Open Data Contract Standard (ODCS). This feature supports three types of rule generation: predefined rules derived from schema properties and constraints, explicit DQX rules embedded in the contract, and text-based rules defined in natural language and processed by a Large Language Model (LLM) to generate appropriate checks. The feature provides rich metadata tracing generated rules back to the source contract for lineage and governance, and it can be used to implement federated data governance, standardize data contracts, and maintain version-controlled quality rules alongside schema definitions. * AI-Assisted Primary Key Detection and Uniqueness Rules Generation ([#934](#934)). Introduced AI-assisted primary key detection and uniqueness rules generation capabilities, leveraging Large Language Models (LLMs) to analyze table schema and metadata. This feature analyzes table schemas and metadata to intelligently detect single or composite primary keys, and performs validation by checking for duplicate values. The `DQProfiler` class now includes a `detect_primary_keys_with_llm` method, which returns a dictionary containing the primary key detection result, including the table name, success status, detected primary key columns, confidence level, reasoning, and error message if any. The `DQGenerator` class has been extended to utilize uniqueness profiles from the profiler for AI-assisted uniqueness rules generation. Various updates have been made to the configuration options, including the addition of an `llm_primary_key_detection` option, which allows users to control whether AI-assisted primary key detection is enabled or disabled. * AI-Assisted Rules Generation Improvements ([#925](#925)). The AI-Assisted Rules Generation feature has been enhanced to handle input as a path in addition to a table, and to generate rules with a filter. The `generate_dq_rules_ai_assisted` method now accepts an `InputConfig` object, which allows users to specify the location and format of the input data, enabling more flexible input handling and filtering capabilities. The feature includes test cases to verify its functionality, including manual tests, unit tests, and integration tests, and the documentation has been updated with minor changes to reflect the new functionality. Additionally, the code has been modified to capitalize keywords to stabilize integration tests, and the `DQGenerator` class has been updated to accommodate the changes, allowing users to generate data quality rules from a variety of input sources. The `InputConfig` class provides a flexible way to configure the input data, including its location and format, and the `get_column_metadata` function has been introduced to retrieve column metadata from a given location. Overall, these updates aim to enhance the functionality and usability of the AI-assisted rules generation feature, providing more flexibility and accuracy in generating data quality rules. * Added case-insensitive comparison support to is_in_list and is_not_null_and_is_in_list checks ([#673](#673)). The `is_in_list` and `is_not_null_and_is_in_list` check functions have been enhanced to support case-insensitive comparison, allowing users to choose between case-sensitive and case-insensitive comparisons via an optional `case_sensitive` boolean flag that defaults to True. These checks verify if values in a specified column are present in a list of allowed values, with the `is_not_null_and_is_in_list` check also requiring the values to be non-null. The updated checks provide more flexibility in data validation, enabling users to configure parameters such as the column to check, the list of allowed values, and the case sensitivity flag. However, it is recommended to use the `foreign_key` dataset-level check for large lists of allowed values or for columns of type `MapType` or `StructType`, as these checks are not suitable for such scenarios. * Added documentation for using DQX in streaming scenarios with foreach batch ([#948](#948)). Documentation and example code snippets were added to demonstrate how to apply checks in foreachBatch structured streaming function. * Added telemetry to track count of input tables ([#954](#954)). Added additional telemetry for better trakcing of DQX usage to help improve the product. * Added support for installing DQX from private PYPI repositories ([#930](#930)). The DQX library has been enhanced with support for installing DQX using a company-hosted PyPI mirror, which is necessary for enterprises that block the public PyPI index. The documentation has been added to describe the feature. The tool installation code has been modified to include new functionality for automatically upload dependencies to a workspace when internet access is blocked. * Support Custom Folder Installation for CLI Commands ([#942](#942)). The command-line interface (CLI) has been enhanced to support custom installation folders, providing users with greater flexibility when working with the library. A new `--install-folder` argument has been introduced, allowing users to specify a custom installation folder when running various CLI commands, such as opening dashboards, workflows, logs, and profiles. This argument override the default installation location to support scenarios where the user installs DQX in a custom location. The library's dependency on sqlalchemy has also been updated to require a version greater than or equal to 2.0 and less than 3.0 to avoid dependency issues in older DBRs. * Enhancement to end to end tests ([#921](#921)). The e2e tests has been enhanced to test integration with dbt transformation framework. Additionally, the documentation for contributing to the project and testing has been updated to simplify the setup process for running tests locally. BREAKING CHANGES! * Renamed `level` parameter to `criticality` in `generate_dq_rules` method of `DQGenerator` for consistency. * Replaced `table: str` parameter with `input_config: InputConfig` in `profile_table` method of `DQProfiler` for greater flexibility. * Replaced `table_name: str` parameter with `input_config: InputConfig` in `generate_dq_rules_ai_assisted` method of `DQGenerator` for greater flexibility.
Merged
mwojtyczka
added a commit
that referenced
this pull request
Dec 1, 2025
* Generationg of DQX rules from ODCS Data Contracts ([#932](#932)). The Data Contract Quality Rules Generation feature has been introduced, enabling users to generate data quality rules directly from data contracts following the Open Data Contract Standard (ODCS). This feature supports three types of rule generation: predefined rules derived from schema properties and constraints, explicit DQX rules embedded in the contract, and text-based rules defined in natural language and processed by a Large Language Model (LLM) to generate appropriate checks. The feature provides rich metadata tracing generated rules back to the source contract for lineage and governance, and it can be used to implement federated data governance, standardize data contracts, and maintain version-controlled quality rules alongside schema definitions. * AI-Assisted Primary Key Detection and Uniqueness Rules Generation ([#934](#934)). Introduced AI-assisted primary key detection and uniqueness rules generation capabilities, leveraging Large Language Models (LLMs) to analyze table schema and metadata. This feature analyzes table schemas and metadata to intelligently detect single or composite primary keys, and performs validation by checking for duplicate values. The `DQProfiler` class now includes a `detect_primary_keys_with_llm` method, which returns a dictionary containing the primary key detection result, including the table name, success status, detected primary key columns, confidence level, reasoning, and error message if any. The `DQGenerator` class has been extended to utilize uniqueness profiles from the profiler for AI-assisted uniqueness rules generation. Various updates have been made to the configuration options, including the addition of an `llm_primary_key_detection` option, which allows users to control whether AI-assisted primary key detection is enabled or disabled. * AI-Assisted Rules Generation Improvements ([#925](#925)). The AI-Assisted Rules Generation feature has been enhanced to handle input as a path in addition to a table, and to generate rules with a filter. The `generate_dq_rules_ai_assisted` method now accepts an `InputConfig` object, which allows users to specify the location and format of the input data, enabling more flexible input handling and filtering capabilities. The feature includes test cases to verify its functionality, including manual tests, unit tests, and integration tests, and the documentation has been updated with minor changes to reflect the new functionality. Additionally, the code has been modified to capitalize keywords to stabilize integration tests, and the `DQGenerator` class has been updated to accommodate the changes, allowing users to generate data quality rules from a variety of input sources. The `InputConfig` class provides a flexible way to configure the input data, including its location and format, and the `get_column_metadata` function has been introduced to retrieve column metadata from a given location. Overall, these updates aim to enhance the functionality and usability of the AI-assisted rules generation feature, providing more flexibility and accuracy in generating data quality rules. * Added case-insensitive comparison support to is_in_list and is_not_null_and_is_in_list checks ([#673](#673)). The `is_in_list` and `is_not_null_and_is_in_list` check functions have been enhanced to support case-insensitive comparison, allowing users to choose between case-sensitive and case-insensitive comparisons via an optional `case_sensitive` boolean flag that defaults to True. These checks verify if values in a specified column are present in a list of allowed values, with the `is_not_null_and_is_in_list` check also requiring the values to be non-null. The updated checks provide more flexibility in data validation, enabling users to configure parameters such as the column to check, the list of allowed values, and the case sensitivity flag. However, it is recommended to use the `foreign_key` dataset-level check for large lists of allowed values or for columns of type `MapType` or `StructType`, as these checks are not suitable for such scenarios. * Added documentation for using DQX in streaming scenarios with foreach batch ([#948](#948)). Documentation and example code snippets were added to demonstrate how to apply checks in foreachBatch structured streaming function. * Added telemetry to track count of input tables ([#954](#954)). Added additional telemetry for better trakcing of DQX usage to help improve the product. * Added support for installing DQX from private PYPI repositories ([#930](#930)). The DQX library has been enhanced with support for installing DQX using a company-hosted PyPI mirror, which is necessary for enterprises that block the public PyPI index. The documentation has been added to describe the feature. The tool installation code has been modified to include new functionality for automatically upload dependencies to a workspace when internet access is blocked. * Support Custom Folder Installation for CLI Commands ([#942](#942)). The command-line interface (CLI) has been enhanced to support custom installation folders, providing users with greater flexibility when working with the library. A new `--install-folder` argument has been introduced, allowing users to specify a custom installation folder when running various CLI commands, such as opening dashboards, workflows, logs, and profiles. This argument override the default installation location to support scenarios where the user installs DQX in a custom location. The library's dependency on sqlalchemy has also been updated to require a version greater than or equal to 2.0 and less than 3.0 to avoid dependency issues in older DBRs. * Enhancement to end to end tests ([#921](#921)). The e2e tests has been enhanced to test integration with dbt transformation framework. Additionally, the documentation for contributing to the project and testing has been updated to simplify the setup process for running tests locally. BREAKING CHANGES! * Renamed `level` parameter to `criticality` in `generate_dq_rules` method of `DQGenerator` for consistency. * Replaced `table: str` parameter with `input_config: InputConfig` in `profile_table` method of `DQProfiler` for greater flexibility. * Replaced `table_name: str` parameter with `input_config: InputConfig` in `generate_dq_rules_ai_assisted` method of `DQGenerator` for greater flexibility.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
This PR adds documentation for using DQX with
foreachBatch.Linked issues
Resolves #922
Tests
N/A as this is a docs-only PR.