Fix/round datetime midnight#517
Merged
mwojtyczka merged 7 commits intoAug 8, 2025
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR fixes the _round_datetime method to preserve midnight values during "up" rounding. Previously, midnight timestamps were always rounded to the next day, which could produce unexpected results when generating min/max rules for timestamp fields. The fix ensures that if a datetime is already at midnight, it remains unchanged when rounded "up".
Key changes:
- Modified
_round_datetimemethod to check if value is already at midnight before rounding up - Added clearer error handling with ValueError for invalid directions
- Added integration test to verify the behavior within data quality profiling logic
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
src/databricks/labs/dqx/profiler/profiler.py |
Fixed _round_datetime method to preserve midnight values during "up" rounding |
tests/integration/test_profiler.py |
Added integration test to verify midnight rounding behavior in profiling context |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
mwojtyczka
reviewed
Aug 8, 2025
mwojtyczka
reviewed
Aug 8, 2025
mwojtyczka
reviewed
Aug 8, 2025
mwojtyczka
reviewed
Aug 8, 2025
AdityaMandiwal
pushed a commit
that referenced
this pull request
Aug 21, 2025
## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary --> This PR improves the `_round_datetime` method by preserving midnight values during "up" rounding. Previously, even midnight timestamps were rounded to the next day, which could produce unexpected results in min/max rule generation for timestamp fields. Now, if the input datetime is already at midnight, it remains unchanged when rounded "up". This update also adds an integration test (`test_profiler_rounding_midnight_behavior`) to verify this behavior within the data quality profiling logic. ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> Resolves [#516](#516) ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [x] manually tested - [ ] added unit tests - [x] added integration tests - [ ] added end-to-end tests --------- Co-authored-by: Marcin Wojtyczka <marcin.wojtyczka@databricks.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
mwojtyczka
added a commit
that referenced
this pull request
Aug 22, 2025
* Added quality checker and end to end workflows ([#519](#519)). This release introduces no-code solution for applying checks. The following workflows were added: quality-checker (apply checks and save results to tables) and end-to-end (e2e) workflows (profile input data, generate quality checks, apply the checks, save results to tables). The workflows enable quality checking for data at-rest without the need for code-level integration. It supports reference data for checks using tables (e.g., required by foreign key or compare datasets checks) as well as custom python check functions (mapping of custom check funciton to the module path in the workspace or Unity Catalog volume containing the function definition). The workflows handle one run config for each job run. Future release will introduce functionality to execute this across multiple tables. In addition, CLI commands have been added to execute the workflows. Additionaly, DQX workflows are configured now to execute using serverless clusters, with an option to use standards clusters as well. InstallationChecksStorageHandler now support absolute workspace path locations. * Added built-in row-level check for PII detection ([#486](#486)). Introduced a new built-in check for Personally Identifiable Information (PII) detection, which utilizes the Presidio framework and can be configured using various parameters, such as NLP entity recognition configuration. This check can be defined using the `does_not_contain_pii` check function and can be customized to suit specific use cases. The check requires `pii` extras to be installed: `pip install databricks-labs-dqx[pii]`. Furthermore, a new enum class `NLPEngineConfig` has been introduced to define various NLP engine configurations for PII detection. Overall, these updates aim to provide more robust and customizable quality checking capabilities for detecting PII data. * Added equality row-level checks ([#535](#535)). Two new row-level checks, `is_equal_to` and `is_not_equal_to`, have been introduced to enable equality checks on column values, allowing users to verify whether the values in a specified column are equal to or not equal to a given value, which can be a numeric literal, column expression, string literal, date literal, or timestamp literal. * Added demo for Spark Structured Streaming ([#518](#518)). Added demo to showcase usage of DQX with Spark Structured Streaming for in-transit data quality checking. The demo is available as Databricks notebook, and can be run on any Databricks workspace. * Added clarification to profiler summary statistics ([#523](#523)). Added new section on understanding summary statistics, which explains how these statistics are computed on a sampled subset of the data and provides a reference for the various summary statistics fields. * Fixed rounding datetimes in the checks generator ([#517](#517)). The generator has been enhanced to correctly handle midnight values when rounding "up", ensuring that datetime values already at midnight remain unchanged, whereas previously they were rounded to the next day. * Added API Docs ([#520](#520)). The DQX API documentation is generated automatically using docstrings. As part of this change the library's documentation has been updated to follow Google style. * Improved test automation by adding end-to-end test for the asset bundles demo ([#533](#533)). BREAKING CHANGES! * `ExtraParams` was moved from `databricks.labs.dqx.rule` module to `databricks.labs.dqx.config`
Merged
mwojtyczka
added a commit
that referenced
this pull request
Aug 23, 2025
* Added quality checker and end to end workflows ([#519](#519)). This release introduces no-code solution for applying checks. The following workflows were added: quality-checker (apply checks and save results to tables) and end-to-end (e2e) workflows (profile input data, generate quality checks, apply the checks, save results to tables). The workflows enable quality checking for data at-rest without the need for code-level integration. It supports reference data for checks using tables (e.g., required by foreign key or compare datasets checks) as well as custom python check functions (mapping of custom check funciton to the module path in the workspace or Unity Catalog volume containing the function definition). The workflows handle one run config for each job run. Future release will introduce functionality to execute this across multiple tables. In addition, CLI commands have been added to execute the workflows. Additionaly, DQX workflows are configured now to execute using serverless clusters, with an option to use standards clusters as well. InstallationChecksStorageHandler now support absolute workspace path locations. * Added built-in row-level check for PII detection ([#486](#486)). Introduced a new built-in check for Personally Identifiable Information (PII) detection, which utilizes the Presidio framework and can be configured using various parameters, such as NLP entity recognition configuration. This check can be defined using the `does_not_contain_pii` check function and can be customized to suit specific use cases. The check requires `pii` extras to be installed: `pip install databricks-labs-dqx[pii]`. Furthermore, a new enum class `NLPEngineConfig` has been introduced to define various NLP engine configurations for PII detection. Overall, these updates aim to provide more robust and customizable quality checking capabilities for detecting PII data. * Added equality row-level checks ([#535](#535)). Two new row-level checks, `is_equal_to` and `is_not_equal_to`, have been introduced to enable equality checks on column values, allowing users to verify whether the values in a specified column are equal to or not equal to a given value, which can be a numeric literal, column expression, string literal, date literal, or timestamp literal. * Added demo for Spark Structured Streaming ([#518](#518)). Added demo to showcase usage of DQX with Spark Structured Streaming for in-transit data quality checking. The demo is available as Databricks notebook, and can be run on any Databricks workspace. * Added clarification to profiler summary statistics ([#523](#523)). Added new section on understanding summary statistics, which explains how these statistics are computed on a sampled subset of the data and provides a reference for the various summary statistics fields. * Fixed rounding datetimes in the checks generator ([#517](#517)). The generator has been enhanced to correctly handle midnight values when rounding "up", ensuring that datetime values already at midnight remain unchanged, whereas previously they were rounded to the next day. * Added API Docs ([#520](#520)). The DQX API documentation is generated automatically using docstrings. As part of this change the library's documentation has been updated to follow Google style. * Improved test automation by adding end-to-end test for the asset bundles demo ([#533](#533)). BREAKING CHANGES! * `ExtraParams` was moved from `databricks.labs.dqx.rule` module to `databricks.labs.dqx.config`
mwojtyczka
added a commit
that referenced
this pull request
Aug 25, 2025
## 0.9.1 * Added quality checker and end to end workflows ([#519](#519)). This release introduces no-code solution for applying checks. The following workflows were added: quality-checker (apply checks and save results to tables) and end-to-end (e2e) workflows (profile input data, generate quality checks, apply the checks, save results to tables). The workflows enable quality checking for data at-rest without the need for code-level integration. It supports reference data for checks using tables (e.g., required by foreign key or compare datasets checks) as well as custom python check functions (mapping of custom check funciton to the module path in the workspace or Unity Catalog volume containing the function definition). The workflows handle one run config for each job run. Future release will introduce functionality to execute this across multiple tables. In addition, CLI commands have been added to execute the workflows. Additionaly, DQX workflows are configured now to execute using serverless clusters, with an option to use standards clusters as well. InstallationChecksStorageHandler now support absolute workspace path locations. * Added built-in row-level check for PII detection ([#486](#486)). Introduced a new built-in check for Personally Identifiable Information (PII) detection, which utilizes the Presidio framework and can be configured using various parameters, such as NLP entity recognition configuration. This check can be defined using the `does_not_contain_pii` check function and can be customized to suit specific use cases. The check requires `pii` extras to be installed: `pip install databricks-labs-dqx[pii]`. Furthermore, a new enum class `NLPEngineConfig` has been introduced to define various NLP engine configurations for PII detection. Overall, these updates aim to provide more robust and customizable quality checking capabilities for detecting PII data. * Added equality row-level checks ([#535](#535)). Two new row-level checks, `is_equal_to` and `is_not_equal_to`, have been introduced to enable equality checks on column values, allowing users to verify whether the values in a specified column are equal to or not equal to a given value, which can be a numeric literal, column expression, string literal, date literal, or timestamp literal. * Added demo for Spark Structured Streaming ([#518](#518)). Added demo to showcase usage of DQX with Spark Structured Streaming for in-transit data quality checking. The demo is available as Databricks notebook, and can be run on any Databricks workspace. * Added clarification to profiler summary statistics ([#523](#523)). Added new section on understanding summary statistics, which explains how these statistics are computed on a sampled subset of the data and provides a reference for the various summary statistics fields. * Fixed rounding datetimes in the checks generator ([#517](#517)). The generator has been enhanced to correctly handle midnight values when rounding "up", ensuring that datetime values already at midnight remain unchanged, whereas previously they were rounded to the next day. * Added API Docs ([#520](#520)). The DQX API documentation is generated automatically using docstrings. As part of this change the library's documentation has been updated to follow Google style. * Improved test automation by adding end-to-end test for the asset bundles demo ([#533](#533)). BREAKING CHANGES! * `ExtraParams` was moved from `databricks.labs.dqx.rule` module to `databricks.labs.dqx.config`
Merged
mwojtyczka
added a commit
that referenced
this pull request
Aug 25, 2025
The release replaces v0.9.0, which were missing PyPI package and will be removed. ## 0.9.1 * Added quality checker and end to end workflows ([#519](#519)). This release introduces no-code solution for applying checks. The following workflows were added: quality-checker (apply checks and save results to tables) and end-to-end (e2e) workflows (profile input data, generate quality checks, apply the checks, save results to tables). The workflows enable quality checking for data at-rest without the need for code-level integration. It supports reference data for checks using tables (e.g., required by foreign key or compare datasets checks) as well as custom python check functions (mapping of custom check funciton to the module path in the workspace or Unity Catalog volume containing the function definition). The workflows handle one run config for each job run. Future release will introduce functionality to execute this across multiple tables. In addition, CLI commands have been added to execute the workflows. Additionaly, DQX workflows are configured now to execute using serverless clusters, with an option to use standards clusters as well. InstallationChecksStorageHandler now support absolute workspace path locations. * Added built-in row-level check for PII detection ([#486](#486)). Introduced a new built-in check for Personally Identifiable Information (PII) detection, which utilizes the Presidio framework and can be configured using various parameters, such as NLP entity recognition configuration. This check can be defined using the `does_not_contain_pii` check function and can be customized to suit specific use cases. The check requires `pii` extras to be installed: `pip install databricks-labs-dqx[pii]`. Furthermore, a new enum class `NLPEngineConfig` has been introduced to define various NLP engine configurations for PII detection. Overall, these updates aim to provide more robust and customizable quality checking capabilities for detecting PII data. * Added equality row-level checks ([#535](#535)). Two new row-level checks, `is_equal_to` and `is_not_equal_to`, have been introduced to enable equality checks on column values, allowing users to verify whether the values in a specified column are equal to or not equal to a given value, which can be a numeric literal, column expression, string literal, date literal, or timestamp literal. * Added demo for Spark Structured Streaming ([#518](#518)). Added demo to showcase usage of DQX with Spark Structured Streaming for in-transit data quality checking. The demo is available as Databricks notebook, and can be run on any Databricks workspace. * Added clarification to profiler summary statistics ([#523](#523)). Added new section on understanding summary statistics, which explains how these statistics are computed on a sampled subset of the data and provides a reference for the various summary statistics fields. * Fixed rounding datetimes in the checks generator ([#517](#517)). The generator has been enhanced to correctly handle midnight values when rounding "up", ensuring that datetime values already at midnight remain unchanged, whereas previously they were rounded to the next day. * Added API Docs ([#520](#520)). The DQX API documentation is generated automatically using docstrings. As part of this change the library's documentation has been updated to follow Google style. * Improved test automation by adding end-to-end test for the asset bundles demo ([#533](#533)). BREAKING CHANGES! * `ExtraParams` was moved from `databricks.labs.dqx.rule` module to `databricks.labs.dqx.config`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
This PR improves the
_round_datetimemethod by preserving midnight values during "up" rounding. Previously, even midnight timestamps were rounded to the next day, which could produce unexpected results in min/max rule generation for timestamp fields. Now, if the input datetime is already at midnight, it remains unchanged when rounded "up".This update also adds an integration test (
test_profiler_rounding_midnight_behavior) to verify this behavior within the data quality profiling logic.Linked issues
Resolves #516
Tests