Equality safe row matching in compare datasets check#473
Merged
Conversation
* Updated docs * Check all pk columns for comparison
mwojtyczka
changed the title
Contributor
|
✅ 243/243 passed, 1 skipped, 42m55s total Running from acceptance #1310 |
mwojtyczka
changed the title
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR enhances the compare_datasets check with configurable null-safe matching behavior for both row matching and column value comparison. It introduces two new boolean parameters that default to True, ensuring backwards compatibility while providing more precise control over how NULL values are handled during dataset comparisons.
- Added
null_safe_row_matchingandnull_safe_column_value_matchingparameters with True defaults - Improved row matching logic to check all primary key columns instead of just the first one
- Updated test cases to reflect corrected behavior for missing/extra row detection
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
src/databricks/labs/dqx/check_funcs.py |
Core implementation of null-safe matching parameters and refactored row matching logic |
tests/integration/test_dataset_checks.py |
Updated existing tests and added new test cases for null-safe matching scenarios |
docs/dqx/docs/reference/quality_rules.mdx |
Enhanced documentation with detailed parameter descriptions and usage examples |
demos/dqx_demo_library.py |
Updated demo showcasing the new null-safe matching options |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
mwojtyczka
added a commit
that referenced
this pull request
Aug 6, 2025
* Added new row-level freshness check ([#495](#495)). A new data quality check function, `is_data_fresh`, has been introduced to identify stale data resulting from delayed pipelines, enabling early detection of upstream issues. This function assesses whether the values in a specified timestamp column are within a specified number of minutes from a base timestamp column. The function takes three parameters: the column to check, the maximum age in minutes before data is considered stale, and an optional base timestamp column, defaulting to the current timestamp if not provided. * Added new dataset-level freshess check ([#499](#499)). A new dataset-level check function, `is_data_fresh_per_time_window`, has been added to validate whether at least a specified minimum number of records arrive within every specified time window, ensuring data freshness. This function is customizable, allowing users to define the time window, minimum records per window, and lookback period. * Improvements have been made to the performance of aggregation check functions, and the check message format has been updated for better readability. * Created llm util function to get check functions details ([#469](#469)). A new utility function has been introduced to provide definitions of all check functions, enabling the generation of prompts for Large Language Models (LLMs) to create check functions. * Added equality safe row and column matching in compare datasets check ([#473](#473)). The compare datasets check functionality has been enhanced to handle null values during row matching and column value comparisons, improving its robustness and flexibility. Two new optional parameters, `null_safe_row_matching` and `null_safe_column_value_matching`, have been introduced to control how null values are handled, both defaulting to True. These parameters allow for null-safe primary key matching and column value matching, ensuring accurate comparison results even when null values are present in the data. The check now excludes specific columns from value comparison using the `exclude_columns` parameter while still considering them for row matching. * Fixed datetime rounding logic in profiler ([#483](#483)). The datetime rounding logic has been improved in profiler to respect the `round=False` option, which was previously ignored. The code now handles the `OverflowError` that occurs when rounding up the maximum datetime value by capping the result and logging a warning. * Added loading and saving checks from file in Unity Catalog Volume ([#512](#512)). This change introduces support for storing quality checks in a Unity Catalog Volume, in addition to existing storage types such as tables, files, and workspace files. The storage location of quality checks has been unified into a single configuration field called `checks_location`, replacing the previous `checks_file` and `checks_table` fields, to simplify the configuration and remove ambiguity by ensuring only one storage location can be defined per run configuration. The `checks_location` field can point to a file in the local path, workspace, installation folder, or Unity Catalog Volume, providing users with more flexibility and clarity when managing their quality checks. * Refactored methods for loading and saving checks ([#487](#487)). The `DQEngine` class has undergone significant changes to improve modularity and maintainability, including the unification of methods for loading and saving checks under the `load_checks` and `save_checks` methods, which take a `config` parameter to determine the storage type, such as `FileChecksStorageConfig`, `WorkspaceFileChecksStorageConfig`, `TableChecksStorageConfig`, or `InstallationChecksStorageConfig`. * Storing checks using dqx classes ([#474](#474)). The data quality engine has been enhanced with methods to convert quality checks between `DQRule` objects and Python dictionaries, allowing for flexibility in check definition and usage. The `serialize_checks` method converts a list of `DQRule` instances into a dictionary representation, while the `deserialize_checks` method performs the reverse operation, converting a dictionary representation back into a list of `DQRule` instances. Additionally, the `DQRule` class now includes a `to_dict` method to convert a `DQRule` instance into a structured dictionary, providing a standardized representation of the rule's metadata. These changes enable users to work with checks in both formats, store and retrieve checks easily, and improve the overall management and storage of data quality checks. The conversion process supports local execution and handles non-complex column expressions, although complex PySpark expressions or Python functions may not be fully reconstructable when converting from class to metadata format. * Added llm utility funciton to extract checks examples in yaml from docs ([#506](#506)). This is achieved through a new Python script that extracts YAML examples from MDX documentation files and creates a combined YAML file with all the extracted examples. The script utilizes regular expressions to extract YAML code blocks from MDX content, validates each YAML block, and combines all valid blocks into a single list. The combined YAML file is then created in the LLM resources directory for use in language model processing. BREAKING CHANGES! * The `checks_file` and `checks_table` fields have been removed from the installation run configuration. They are now consolidated into the single `checks_location` field. This change simplifies the configuration and clearly defines where checks are stored. * The `load_run_config` method has been moved to `config_loader.RunConfigLoader`, as it is not intended for direct use and falls outside the `DQEngine` core responsibilities. DEPRECIATION CHANGES! If you are loading or saving checks from a storage (file, workspace file, table, installation), you are affected. We are deprecating the below methods. We are keeping the methods in the `DQEngine` but you should update your code as these methods will be removed in future versions. * Loading checks to storage has been unified under `load_checks` method. The following methods have been removed from the `DQEngine`: `load_checks_from_local_file`, `load_checks_from_workspace_file`, `load_checks_from_installation`, `load_checks_from_table`. * Saving checks in storage has been unified under `load_checks` method. The following methods have been removed from the `DQEngine`: `save_checks_in_local_file`, `save_checks_in_workspace_file`, `save_checks_in_installation`, `save_checks_in_table`. The `save_checks` and `load_checks` take `config` as a parameter, which determines the storage types used. The following storage configs are currently supported: * `FileChecksStorageConfig`: file in the local filesystem (YAML or JSON) * `WorkspaceFileChecksStorageConfig`: file in the workspace (YAML or JSON) * `TableChecksStorageConfig`: a table * `InstallationChecksStorageConfig`: storage defined in the installation context, using either the `checks_table` or `checks_file` field from the run configuration.
Merged
mwojtyczka
added a commit
that referenced
this pull request
Aug 6, 2025
* Added new row-level freshness check ([#495](#495)). A new data quality check function, `is_data_fresh`, has been introduced to identify stale data resulting from delayed pipelines, enabling early detection of upstream issues. This function assesses whether the values in a specified timestamp column are within a specified number of minutes from a base timestamp column. The function takes three parameters: the column to check, the maximum age in minutes before data is considered stale, and an optional base timestamp column, defaulting to the current timestamp if not provided. * Added new dataset-level freshess check ([#499](#499)). A new dataset-level check function, `is_data_fresh_per_time_window`, has been added to validate whether at least a specified minimum number of records arrive within every specified time window, ensuring data freshness. This function is customizable, allowing users to define the time window, minimum records per window, and lookback period. * Improvements have been made to the performance of aggregation check functions, and the check message format has been updated for better readability. * Created llm util function to get check functions details ([#469](#469)). A new utility function has been introduced to provide definitions of all check functions, enabling the generation of prompts for Large Language Models (LLMs) to create check functions. * Added equality safe row and column matching in compare datasets check ([#473](#473)). The compare datasets check functionality has been enhanced to handle null values during row matching and column value comparisons, improving its robustness and flexibility. Two new optional parameters, `null_safe_row_matching` and `null_safe_column_value_matching`, have been introduced to control how null values are handled, both defaulting to True. These parameters allow for null-safe primary key matching and column value matching, ensuring accurate comparison results even when null values are present in the data. The check now excludes specific columns from value comparison using the `exclude_columns` parameter while still considering them for row matching. * Fixed datetime rounding logic in profiler ([#483](#483)). The datetime rounding logic has been improved in profiler to respect the `round=False` option, which was previously ignored. The code now handles the `OverflowError` that occurs when rounding up the maximum datetime value by capping the result and logging a warning. * Added loading and saving checks from file in Unity Catalog Volume ([#512](#512)). This change introduces support for storing quality checks in a Unity Catalog Volume, in addition to existing storage types such as tables, files, and workspace files. The storage location of quality checks has been unified into a single configuration field called `checks_location`, replacing the previous `checks_file` and `checks_table` fields, to simplify the configuration and remove ambiguity by ensuring only one storage location can be defined per run configuration. The `checks_location` field can point to a file in the local path, workspace, installation folder, or Unity Catalog Volume, providing users with more flexibility and clarity when managing their quality checks. * Refactored methods for loading and saving checks ([#487](#487)). The `DQEngine` class has undergone significant changes to improve modularity and maintainability, including the unification of methods for loading and saving checks under the `load_checks` and `save_checks` methods, which take a `config` parameter to determine the storage type, such as `FileChecksStorageConfig`, `WorkspaceFileChecksStorageConfig`, `TableChecksStorageConfig`, or `InstallationChecksStorageConfig`. * Storing checks using dqx classes ([#474](#474)). The data quality engine has been enhanced with methods to convert quality checks between `DQRule` objects and Python dictionaries, allowing for flexibility in check definition and usage. The `serialize_checks` method converts a list of `DQRule` instances into a dictionary representation, while the `deserialize_checks` method performs the reverse operation, converting a dictionary representation back into a list of `DQRule` instances. Additionally, the `DQRule` class now includes a `to_dict` method to convert a `DQRule` instance into a structured dictionary, providing a standardized representation of the rule's metadata. These changes enable users to work with checks in both formats, store and retrieve checks easily, and improve the overall management and storage of data quality checks. The conversion process supports local execution and handles non-complex column expressions, although complex PySpark expressions or Python functions may not be fully reconstructable when converting from class to metadata format. * Added llm utility funciton to extract checks examples in yaml from docs ([#506](#506)). This is achieved through a new Python script that extracts YAML examples from MDX documentation files and creates a combined YAML file with all the extracted examples. The script utilizes regular expressions to extract YAML code blocks from MDX content, validates each YAML block, and combines all valid blocks into a single list. The combined YAML file is then created in the LLM resources directory for use in language model processing. BREAKING CHANGES! * The `checks_file` and `checks_table` fields have been removed from the installation run configuration. They are now consolidated into the single `checks_location` field. This change simplifies the configuration and clearly defines where checks are stored. * The `load_run_config` method has been moved to `config_loader.RunConfigLoader`, as it is not intended for direct use and falls outside the `DQEngine` core responsibilities. DEPRECIATION CHANGES! If you are loading or saving checks from a storage (file, workspace file, table, installation), you are affected. We are deprecating the below methods. We are keeping the methods in the `DQEngine` but you should update your code as these methods will be removed in future versions. * Loading checks to storage has been unified under `load_checks` method. The following methods have been removed from the `DQEngine`: `load_checks_from_local_file`, `load_checks_from_workspace_file`, `load_checks_from_installation`, `load_checks_from_table`. * Saving checks in storage has been unified under `load_checks` method. The following methods have been removed from the `DQEngine`: `save_checks_in_local_file`, `save_checks_in_workspace_file`, `save_checks_in_installation`, `save_checks_in_table`. The `save_checks` and `load_checks` take `config` as a parameter, which determines the storage types used. The following storage configs are currently supported: * `FileChecksStorageConfig`: file in the local filesystem (YAML or JSON) * `WorkspaceFileChecksStorageConfig`: file in the workspace (YAML or JSON) * `TableChecksStorageConfig`: a table * `InstallationChecksStorageConfig`: storage defined in the installation context, using either the `checks_table` or `checks_file` field from the run configuration.
AdityaMandiwal
pushed a commit
that referenced
this pull request
Aug 21, 2025
## Changes
<!-- Summary of your changes that are easy to understand. Add
screenshots when necessary -->
* Added `null_safe_row_matching` and `null_safe_column_value_matching`
parameters both defaulted to True. This is to ensure that by default row
and column value matching are done by treating NULLs as equal. In data
validation tools, null-safe primary key matching and column value
matching is important to avoid treating (NULL, NULL) keys as
non-matching rows. Without this, rows with joint null PKs will be
incorrectly marked as missing or extra.
In the below example rows should match:
```
df:
id | id2 | name
1 | Null | "val"
ref_df:
id | id2 | name
1 | Null | Null
(1, Null) - row match, Null treated as equal when `null_safe_row_matching` is enabled
("val", Null) - no column match since Null is treated as unknown when `null_safe_column_value_matching` is disabled
columns=["id", "id2"]
checks = [
DQDatasetRule(
criticality="error",
check_func=check_funcs.compare_datasets,
columns=columns,
check_func_kwargs={
"ref_columns": columns,
"ref_df_name": "ref_df",
"null_safe_row_matching": True, # treat NULLs equal when matching rows, default True
"null_safe_column_value_matching": False, # don't treat NULLs equal when comparing column values, default True
},
)
]
```
* Check all pk columns for missing or extra rows
* Updated docs
* Updated demo to showcase different options
### Linked issues
<!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes,
fixed, resolve, resolves, resolved. See
https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword
-->
Resolves #..
### Tests
<!-- How is this tested? Please see the checklist below and also
describe any other relevant tests -->
- [x] manually tested
- [] added unit tests
- [x] added integration tests
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
AdityaMandiwal
pushed a commit
that referenced
this pull request
Aug 21, 2025
* Added new row-level freshness check ([#495](#495)). A new data quality check function, `is_data_fresh`, has been introduced to identify stale data resulting from delayed pipelines, enabling early detection of upstream issues. This function assesses whether the values in a specified timestamp column are within a specified number of minutes from a base timestamp column. The function takes three parameters: the column to check, the maximum age in minutes before data is considered stale, and an optional base timestamp column, defaulting to the current timestamp if not provided. * Added new dataset-level freshess check ([#499](#499)). A new dataset-level check function, `is_data_fresh_per_time_window`, has been added to validate whether at least a specified minimum number of records arrive within every specified time window, ensuring data freshness. This function is customizable, allowing users to define the time window, minimum records per window, and lookback period. * Improvements have been made to the performance of aggregation check functions, and the check message format has been updated for better readability. * Created llm util function to get check functions details ([#469](#469)). A new utility function has been introduced to provide definitions of all check functions, enabling the generation of prompts for Large Language Models (LLMs) to create check functions. * Added equality safe row and column matching in compare datasets check ([#473](#473)). The compare datasets check functionality has been enhanced to handle null values during row matching and column value comparisons, improving its robustness and flexibility. Two new optional parameters, `null_safe_row_matching` and `null_safe_column_value_matching`, have been introduced to control how null values are handled, both defaulting to True. These parameters allow for null-safe primary key matching and column value matching, ensuring accurate comparison results even when null values are present in the data. The check now excludes specific columns from value comparison using the `exclude_columns` parameter while still considering them for row matching. * Fixed datetime rounding logic in profiler ([#483](#483)). The datetime rounding logic has been improved in profiler to respect the `round=False` option, which was previously ignored. The code now handles the `OverflowError` that occurs when rounding up the maximum datetime value by capping the result and logging a warning. * Added loading and saving checks from file in Unity Catalog Volume ([#512](#512)). This change introduces support for storing quality checks in a Unity Catalog Volume, in addition to existing storage types such as tables, files, and workspace files. The storage location of quality checks has been unified into a single configuration field called `checks_location`, replacing the previous `checks_file` and `checks_table` fields, to simplify the configuration and remove ambiguity by ensuring only one storage location can be defined per run configuration. The `checks_location` field can point to a file in the local path, workspace, installation folder, or Unity Catalog Volume, providing users with more flexibility and clarity when managing their quality checks. * Refactored methods for loading and saving checks ([#487](#487)). The `DQEngine` class has undergone significant changes to improve modularity and maintainability, including the unification of methods for loading and saving checks under the `load_checks` and `save_checks` methods, which take a `config` parameter to determine the storage type, such as `FileChecksStorageConfig`, `WorkspaceFileChecksStorageConfig`, `TableChecksStorageConfig`, or `InstallationChecksStorageConfig`. * Storing checks using dqx classes ([#474](#474)). The data quality engine has been enhanced with methods to convert quality checks between `DQRule` objects and Python dictionaries, allowing for flexibility in check definition and usage. The `serialize_checks` method converts a list of `DQRule` instances into a dictionary representation, while the `deserialize_checks` method performs the reverse operation, converting a dictionary representation back into a list of `DQRule` instances. Additionally, the `DQRule` class now includes a `to_dict` method to convert a `DQRule` instance into a structured dictionary, providing a standardized representation of the rule's metadata. These changes enable users to work with checks in both formats, store and retrieve checks easily, and improve the overall management and storage of data quality checks. The conversion process supports local execution and handles non-complex column expressions, although complex PySpark expressions or Python functions may not be fully reconstructable when converting from class to metadata format. * Added llm utility funciton to extract checks examples in yaml from docs ([#506](#506)). This is achieved through a new Python script that extracts YAML examples from MDX documentation files and creates a combined YAML file with all the extracted examples. The script utilizes regular expressions to extract YAML code blocks from MDX content, validates each YAML block, and combines all valid blocks into a single list. The combined YAML file is then created in the LLM resources directory for use in language model processing. BREAKING CHANGES! * The `checks_file` and `checks_table` fields have been removed from the installation run configuration. They are now consolidated into the single `checks_location` field. This change simplifies the configuration and clearly defines where checks are stored. * The `load_run_config` method has been moved to `config_loader.RunConfigLoader`, as it is not intended for direct use and falls outside the `DQEngine` core responsibilities. DEPRECIATION CHANGES! If you are loading or saving checks from a storage (file, workspace file, table, installation), you are affected. We are deprecating the below methods. We are keeping the methods in the `DQEngine` but you should update your code as these methods will be removed in future versions. * Loading checks to storage has been unified under `load_checks` method. The following methods have been removed from the `DQEngine`: `load_checks_from_local_file`, `load_checks_from_workspace_file`, `load_checks_from_installation`, `load_checks_from_table`. * Saving checks in storage has been unified under `load_checks` method. The following methods have been removed from the `DQEngine`: `save_checks_in_local_file`, `save_checks_in_workspace_file`, `save_checks_in_installation`, `save_checks_in_table`. The `save_checks` and `load_checks` take `config` as a parameter, which determines the storage types used. The following storage configs are currently supported: * `FileChecksStorageConfig`: file in the local filesystem (YAML or JSON) * `WorkspaceFileChecksStorageConfig`: file in the workspace (YAML or JSON) * `TableChecksStorageConfig`: a table * `InstallationChecksStorageConfig`: storage defined in the installation context, using either the `checks_table` or `checks_file` field from the run configuration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
null_safe_row_matchingandnull_safe_column_value_matchingparameters both defaulted to True. This is to ensure that by default row and column value matching are done by treating NULLs as equal. In data validation tools, null-safe primary key matching and column value matching is important to avoid treating (NULL, NULL) keys as non-matching rows. Without this, rows with joint null PKs will be incorrectly marked as missing or extra.In the below example rows should match:
Linked issues
Resolves #..
Tests