Adding equality row-level checks#535
Merged
Merged
Conversation
AdityaMandiwal
requested review from
tombonfert
and removed request for
a team
Contributor
|
✅ 10/10 passed, 1h22m42s total Running from acceptance #1924 |
updated examples and tests
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds two new row-level equality checks to the data quality framework: is_not_equal_to and is_equal_to. These functions enable validation that column values are or are not equal to specified comparison values.
- Added
is_equal_toandis_not_equal_tofunctions with support for numeric, string, date, and timestamp values - Updated test files and documentation with comprehensive examples
- Added validation to ensure the comparison value parameter is provided
Reviewed Changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
src/databricks/labs/dqx/check_funcs.py |
Implements the core equality check functions |
tests/unit/test_row_checks.py |
Adds unit tests for validation error handling |
tests/integration/test_row_checks.py |
Adds comprehensive integration tests with various data types |
tests/resources/all_row_checks.yaml |
Adds YAML configuration examples for the new checks |
tests/integration/test_apply_checks.py |
Updates integration tests with new column data and check configurations |
src/databricks/labs/dqx/llm/resources/yaml_checks_examples.yml |
Adds YAML examples for LLM resources |
docs/dqx/docs/reference/quality_rules.mdx |
Updates documentation with function descriptions and usage examples |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
mwojtyczka
added a commit
that referenced
this pull request
Aug 22, 2025
* Added quality checker and end to end workflows ([#519](#519)). This release introduces no-code solution for applying checks. The following workflows were added: quality-checker (apply checks and save results to tables) and end-to-end (e2e) workflows (profile input data, generate quality checks, apply the checks, save results to tables). The workflows enable quality checking for data at-rest without the need for code-level integration. It supports reference data for checks using tables (e.g., required by foreign key or compare datasets checks) as well as custom python check functions (mapping of custom check funciton to the module path in the workspace or Unity Catalog volume containing the function definition). The workflows handle one run config for each job run. Future release will introduce functionality to execute this across multiple tables. In addition, CLI commands have been added to execute the workflows. Additionaly, DQX workflows are configured now to execute using serverless clusters, with an option to use standards clusters as well. InstallationChecksStorageHandler now support absolute workspace path locations. * Added built-in row-level check for PII detection ([#486](#486)). Introduced a new built-in check for Personally Identifiable Information (PII) detection, which utilizes the Presidio framework and can be configured using various parameters, such as NLP entity recognition configuration. This check can be defined using the `does_not_contain_pii` check function and can be customized to suit specific use cases. The check requires `pii` extras to be installed: `pip install databricks-labs-dqx[pii]`. Furthermore, a new enum class `NLPEngineConfig` has been introduced to define various NLP engine configurations for PII detection. Overall, these updates aim to provide more robust and customizable quality checking capabilities for detecting PII data. * Added equality row-level checks ([#535](#535)). Two new row-level checks, `is_equal_to` and `is_not_equal_to`, have been introduced to enable equality checks on column values, allowing users to verify whether the values in a specified column are equal to or not equal to a given value, which can be a numeric literal, column expression, string literal, date literal, or timestamp literal. * Added demo for Spark Structured Streaming ([#518](#518)). Added demo to showcase usage of DQX with Spark Structured Streaming for in-transit data quality checking. The demo is available as Databricks notebook, and can be run on any Databricks workspace. * Added clarification to profiler summary statistics ([#523](#523)). Added new section on understanding summary statistics, which explains how these statistics are computed on a sampled subset of the data and provides a reference for the various summary statistics fields. * Fixed rounding datetimes in the checks generator ([#517](#517)). The generator has been enhanced to correctly handle midnight values when rounding "up", ensuring that datetime values already at midnight remain unchanged, whereas previously they were rounded to the next day. * Added API Docs ([#520](#520)). The DQX API documentation is generated automatically using docstrings. As part of this change the library's documentation has been updated to follow Google style. * Improved test automation by adding end-to-end test for the asset bundles demo ([#533](#533)). BREAKING CHANGES! * `ExtraParams` was moved from `databricks.labs.dqx.rule` module to `databricks.labs.dqx.config`
Merged
mwojtyczka
added a commit
that referenced
this pull request
Aug 23, 2025
* Added quality checker and end to end workflows ([#519](#519)). This release introduces no-code solution for applying checks. The following workflows were added: quality-checker (apply checks and save results to tables) and end-to-end (e2e) workflows (profile input data, generate quality checks, apply the checks, save results to tables). The workflows enable quality checking for data at-rest without the need for code-level integration. It supports reference data for checks using tables (e.g., required by foreign key or compare datasets checks) as well as custom python check functions (mapping of custom check funciton to the module path in the workspace or Unity Catalog volume containing the function definition). The workflows handle one run config for each job run. Future release will introduce functionality to execute this across multiple tables. In addition, CLI commands have been added to execute the workflows. Additionaly, DQX workflows are configured now to execute using serverless clusters, with an option to use standards clusters as well. InstallationChecksStorageHandler now support absolute workspace path locations. * Added built-in row-level check for PII detection ([#486](#486)). Introduced a new built-in check for Personally Identifiable Information (PII) detection, which utilizes the Presidio framework and can be configured using various parameters, such as NLP entity recognition configuration. This check can be defined using the `does_not_contain_pii` check function and can be customized to suit specific use cases. The check requires `pii` extras to be installed: `pip install databricks-labs-dqx[pii]`. Furthermore, a new enum class `NLPEngineConfig` has been introduced to define various NLP engine configurations for PII detection. Overall, these updates aim to provide more robust and customizable quality checking capabilities for detecting PII data. * Added equality row-level checks ([#535](#535)). Two new row-level checks, `is_equal_to` and `is_not_equal_to`, have been introduced to enable equality checks on column values, allowing users to verify whether the values in a specified column are equal to or not equal to a given value, which can be a numeric literal, column expression, string literal, date literal, or timestamp literal. * Added demo for Spark Structured Streaming ([#518](#518)). Added demo to showcase usage of DQX with Spark Structured Streaming for in-transit data quality checking. The demo is available as Databricks notebook, and can be run on any Databricks workspace. * Added clarification to profiler summary statistics ([#523](#523)). Added new section on understanding summary statistics, which explains how these statistics are computed on a sampled subset of the data and provides a reference for the various summary statistics fields. * Fixed rounding datetimes in the checks generator ([#517](#517)). The generator has been enhanced to correctly handle midnight values when rounding "up", ensuring that datetime values already at midnight remain unchanged, whereas previously they were rounded to the next day. * Added API Docs ([#520](#520)). The DQX API documentation is generated automatically using docstrings. As part of this change the library's documentation has been updated to follow Google style. * Improved test automation by adding end-to-end test for the asset bundles demo ([#533](#533)). BREAKING CHANGES! * `ExtraParams` was moved from `databricks.labs.dqx.rule` module to `databricks.labs.dqx.config`
mwojtyczka
added a commit
that referenced
this pull request
Aug 25, 2025
## 0.9.1 * Added quality checker and end to end workflows ([#519](#519)). This release introduces no-code solution for applying checks. The following workflows were added: quality-checker (apply checks and save results to tables) and end-to-end (e2e) workflows (profile input data, generate quality checks, apply the checks, save results to tables). The workflows enable quality checking for data at-rest without the need for code-level integration. It supports reference data for checks using tables (e.g., required by foreign key or compare datasets checks) as well as custom python check functions (mapping of custom check funciton to the module path in the workspace or Unity Catalog volume containing the function definition). The workflows handle one run config for each job run. Future release will introduce functionality to execute this across multiple tables. In addition, CLI commands have been added to execute the workflows. Additionaly, DQX workflows are configured now to execute using serverless clusters, with an option to use standards clusters as well. InstallationChecksStorageHandler now support absolute workspace path locations. * Added built-in row-level check for PII detection ([#486](#486)). Introduced a new built-in check for Personally Identifiable Information (PII) detection, which utilizes the Presidio framework and can be configured using various parameters, such as NLP entity recognition configuration. This check can be defined using the `does_not_contain_pii` check function and can be customized to suit specific use cases. The check requires `pii` extras to be installed: `pip install databricks-labs-dqx[pii]`. Furthermore, a new enum class `NLPEngineConfig` has been introduced to define various NLP engine configurations for PII detection. Overall, these updates aim to provide more robust and customizable quality checking capabilities for detecting PII data. * Added equality row-level checks ([#535](#535)). Two new row-level checks, `is_equal_to` and `is_not_equal_to`, have been introduced to enable equality checks on column values, allowing users to verify whether the values in a specified column are equal to or not equal to a given value, which can be a numeric literal, column expression, string literal, date literal, or timestamp literal. * Added demo for Spark Structured Streaming ([#518](#518)). Added demo to showcase usage of DQX with Spark Structured Streaming for in-transit data quality checking. The demo is available as Databricks notebook, and can be run on any Databricks workspace. * Added clarification to profiler summary statistics ([#523](#523)). Added new section on understanding summary statistics, which explains how these statistics are computed on a sampled subset of the data and provides a reference for the various summary statistics fields. * Fixed rounding datetimes in the checks generator ([#517](#517)). The generator has been enhanced to correctly handle midnight values when rounding "up", ensuring that datetime values already at midnight remain unchanged, whereas previously they were rounded to the next day. * Added API Docs ([#520](#520)). The DQX API documentation is generated automatically using docstrings. As part of this change the library's documentation has been updated to follow Google style. * Improved test automation by adding end-to-end test for the asset bundles demo ([#533](#533)). BREAKING CHANGES! * `ExtraParams` was moved from `databricks.labs.dqx.rule` module to `databricks.labs.dqx.config`
Merged
mwojtyczka
added a commit
that referenced
this pull request
Aug 25, 2025
The release replaces v0.9.0, which were missing PyPI package and will be removed. ## 0.9.1 * Added quality checker and end to end workflows ([#519](#519)). This release introduces no-code solution for applying checks. The following workflows were added: quality-checker (apply checks and save results to tables) and end-to-end (e2e) workflows (profile input data, generate quality checks, apply the checks, save results to tables). The workflows enable quality checking for data at-rest without the need for code-level integration. It supports reference data for checks using tables (e.g., required by foreign key or compare datasets checks) as well as custom python check functions (mapping of custom check funciton to the module path in the workspace or Unity Catalog volume containing the function definition). The workflows handle one run config for each job run. Future release will introduce functionality to execute this across multiple tables. In addition, CLI commands have been added to execute the workflows. Additionaly, DQX workflows are configured now to execute using serverless clusters, with an option to use standards clusters as well. InstallationChecksStorageHandler now support absolute workspace path locations. * Added built-in row-level check for PII detection ([#486](#486)). Introduced a new built-in check for Personally Identifiable Information (PII) detection, which utilizes the Presidio framework and can be configured using various parameters, such as NLP entity recognition configuration. This check can be defined using the `does_not_contain_pii` check function and can be customized to suit specific use cases. The check requires `pii` extras to be installed: `pip install databricks-labs-dqx[pii]`. Furthermore, a new enum class `NLPEngineConfig` has been introduced to define various NLP engine configurations for PII detection. Overall, these updates aim to provide more robust and customizable quality checking capabilities for detecting PII data. * Added equality row-level checks ([#535](#535)). Two new row-level checks, `is_equal_to` and `is_not_equal_to`, have been introduced to enable equality checks on column values, allowing users to verify whether the values in a specified column are equal to or not equal to a given value, which can be a numeric literal, column expression, string literal, date literal, or timestamp literal. * Added demo for Spark Structured Streaming ([#518](#518)). Added demo to showcase usage of DQX with Spark Structured Streaming for in-transit data quality checking. The demo is available as Databricks notebook, and can be run on any Databricks workspace. * Added clarification to profiler summary statistics ([#523](#523)). Added new section on understanding summary statistics, which explains how these statistics are computed on a sampled subset of the data and provides a reference for the various summary statistics fields. * Fixed rounding datetimes in the checks generator ([#517](#517)). The generator has been enhanced to correctly handle midnight values when rounding "up", ensuring that datetime values already at midnight remain unchanged, whereas previously they were rounded to the next day. * Added API Docs ([#520](#520)). The DQX API documentation is generated automatically using docstrings. As part of this change the library's documentation has been updated to follow Google style. * Improved test automation by adding end-to-end test for the asset bundles demo ([#533](#533)). BREAKING CHANGES! * `ExtraParams` was moved from `databricks.labs.dqx.rule` module to `databricks.labs.dqx.config`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Added new row-level checks:
is_not_equal_tois_equal_toLinked issues
Resolves #413
Tests