Added documentation for best practices and fix datetime limit handling#997
Merged
Conversation
mwojtyczka
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds comprehensive best practices documentation for using DQX in production environments and reorganizes documentation sidebar positions to improve navigation. The changes include practical guidance on managing data quality rules, usage patterns, and deployment strategies.
Changes:
- Added a new best practices guide covering production usage, rule management, and deployment strategies
- Updated sidebar positions across all documentation files to use a clearer numbering scheme (100s, 200s, 300s, etc.)
- Enhanced test case description to be more specific about SQL expression usage
Reviewed changes
Copilot reviewed 28 out of 28 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/dqx/docs/guide/best_practices.mdx | New comprehensive best practices documentation with production guidance for DQX |
| tests/perf/generate_md_report.py | Updated benchmarks sidebar position from 13 to 507 |
| tests/integration/test_ai_rules_generator.py | Enhanced test input description to specify 'NOT LIKE' SQL operator |
| docs/dqx/docs/reference/testing.mdx | Updated sidebar position from 4 to 504 |
| docs/dqx/docs/reference/quality_checks.mdx | Updated sidebar position from 3 to 501 |
| docs/dqx/docs/reference/profiler.mdx | Updated sidebar position from 2 to 503 |
| docs/dqx/docs/reference/index.mdx | Updated sidebar position from 11 to 500 |
| docs/dqx/docs/reference/engine.mdx | Updated sidebar position from 1 to 502 |
| docs/dqx/docs/reference/cli.mdx | Updated sidebar position from 11 to 505 |
| docs/dqx/docs/reference/benchmarks.mdx | Updated sidebar position from 13 to 507 |
| docs/dqx/docs/reference/api/index.mdx | Updated sidebar position from 12 to 506 |
| docs/dqx/docs/motivation.mdx | Updated sidebar position from 1 to 100 |
| docs/dqx/docs/installation.mdx | Updated sidebar position from 2 to 200 |
| docs/dqx/docs/guide/summary_metrics.mdx | Updated sidebar position from 8 to 308 |
| docs/dqx/docs/guide/quality_dashboard.mdx | Updated sidebar position from 9 to 309 |
| docs/dqx/docs/guide/quality_checks_storage.mdx | Updated sidebar position from 6 to 305 |
| docs/dqx/docs/guide/quality_checks_definition.mdx | Updated sidebar position from 7 to 306 |
| docs/dqx/docs/guide/quality_checks_apply.mdx | Updated sidebar position from 8 to 307 |
| docs/dqx/docs/guide/index.mdx | Updated sidebar position from 3 to 300 |
| docs/dqx/docs/guide/data_profiling.mdx | Updated sidebar position from 4 to 301 |
| docs/dqx/docs/guide/data_contract_quality_rules_generation.mdx | Updated sidebar position from 6 to 304 |
| docs/dqx/docs/guide/ai_assisted_quality_checks_generation.mdx | Updated sidebar position from 5 to 302 |
| docs/dqx/docs/guide/ai_assisted_primary_key_detection.mdx | Updated sidebar position from 6 to 303 |
| docs/dqx/docs/guide/additional_configuration.mdx | Updated sidebar position from 10 to 310 |
| docs/dqx/docs/dev/index.mdx | Updated sidebar position from 12 to 600 |
| docs/dqx/docs/dev/docs_authoring.mdx | Added sidebar position 602 (previously missing) |
| docs/dqx/docs/dev/contributing.mdx | Added sidebar position 601 (previously missing) |
| docs/dqx/docs/demos.mdx | Updated sidebar position from 10 to 400 |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
|
✅ 510/510 passed, 3 flaky, 41 skipped, 4h21m3s total Flaky tests:
Running from acceptance #3608 |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #997 +/- ##
==========================================
- Coverage 90.75% 90.55% -0.21%
==========================================
Files 64 64
Lines 6509 6519 +10
==========================================
- Hits 5907 5903 -4
- Misses 602 616 +14 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 30 out of 30 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
mwojtyczka
changed the title
ghanse
requested changes
Jan 19, 2026
ghanse
approved these changes
Jan 20, 2026
mwojtyczka
added a commit
that referenced
this pull request
Feb 9, 2026
* New DQX Data Quality Dashboard ([#1019](#1019)). The data quality dashboard has been significantly enhanced to provide a centralized view of data quality metrics across all tables, allowing users to monitor and track data quality issues with greater ease. The dashboard now consists of three tabs - Data Quality Summary, Data Quality by Table (Time Series), and Data Quality by Table (Full Snapshot) - each catering to different monitoring scenarios, and offers customizable parameters for reporting column names and filtering tables with data quality issues. Additionally, the installation process for the dashboard has been simplified, with options to import it directly to a Workspace or deploy it automatically using the Databricks CLI. * DQX App Skeleton ([#982](#982)). The DQX application (frontend and backend) has been built with a core set of features, including configuration management and AI-assisted rule generation based on natural-language input from users. A comprehensive README documents the application architecture as well as development and deployment workflows. Future versions of DQX will introduce additional functionality (loading/saving rules, rules authoring in graphical form) and provide a streamlined, user-friendly way to deploy the application directly into a Databricks workspace. * Added Decimal support to check functions and to min_max generator ([#1013](#1013)) ([#1017](#1017)). The data quality checks have been enhanced to support Python's Decimal type, in addition to int and float, for min/max validation checks, enabling proper data quality checks for decimal-precise financial and scientific data where floating-point precision issues would cause false positives. * Added DQX produciton best practices and fix datetime limit handling ([#997](#997)). Practical guidance and best practices for using DQX in production have been added, covering aspects such as storing checks in Delta tables, enforcing access controls, and optimizing rules for performance and scalability. Fixes have also been implemented to address issues related to handling date and datetime limits, particularly when provided as strings. * Added new row-level check functions: is_null, is_empty, and is_null_or_empty ([#1015](#1015)). DQX now includes three new check functions, `is_null`, `is_empty`, and `is_null_or_empty`, which enable verification of column values as null, empty strings, or both, complementing existing checks like `is_not_null`, `is_not_empty`, and `is_not_null_and_not_empty`. The functions also support optional arguments, like `trim_strings` to trim spaces from strings. * Added tolerance to equality and non-equality check functions ([#1011](#1011)). The library's quality check functionality has been enhanced to support absolute and relative tolerance parameters for numeric value comparisons in `is_equal_to`, `is_not_equal_to`, `is_aggr_equal` and `is_aggr_not_equal` checks, allowing for more flexible and precise control over data validation. The introduction of tolerance logic, which checks for absolute and relative differences within specified thresholds via `abs_tolerance` and `rel_tolerance` parameters, provides more nuanced comparisons for numeric data. * Allow new lines in sql expression checks ([#1009](#1009)). SQL expression check function (`sql_expression`) has been updated to support new lines in its expression argument, allowing for more complex and formatted SQL expressions. * Allow summary metrics with SparkConnect sessions ([#1000](#1000)). The library now supports writing summary metrics directly to a table with SparkConnect sessions, eliminating the need for a classic compute cluster in Dedicated access mode. This change lifts the previous restriction and enables generatic summary metrics using Serverless and all standard clusters with Databricks Runtime 17.3LTS or higeher. * Fixed loading checks from a delta table with special characters ([#992](#992)). The loading checks functionality from a delta table has been fixed to handle special characters in the fully qualified table. * Fixed resolution of pii detection check function ([#1003](#1003)). The PII detection check function resolution has been enhanced to support the application of checks defined as metadata (YAML). * Fixed serialization/deserialization of row filter parameter for dataset-level rules ([#1021](#1021)). The `filter` field in checks definition now correctly pushes down the `filter` condition defined at the check-level as `row_filter` to the check function, allowing checks to operate on the relevant subset of rows before aggregation. The documentation has been updated to advice users to use op-level `filter` condition for consistency instead of `row_filter` parameter. Overall, these changes aim to enhance the overall user experience. * Improved Lakeflow Declarative Pipeline tests ([#1010](#1010)). The Lakeflow Declarative Pipeline (LDP) tests have been enhanced to utilize full Unity Catalog mode, enabling support for writing to arbitrary catalogs and schemas, and performing additional checks to prevent certain operations. * Updated Lakebase authentication method ([#975](#975)). The Lakebase authentication method has been updated to utilize a client ID instead of a username, simplifying its use in the context of a Databricks App. The `lakebase_user` parameter has been replaced with `lakebase_client_id`, an optional service principal client ID used to connect to Lakebase, defaulting to the caller's identity if not provided. This change enhances the security and reliability of the authentication process, making it easier to work with Lakebase as a checks storage. * Updated handling of metadata columns during schema validation ([#1002](#1002)). The `has_valid_schema` check has been enhanced to provide more flexibility in schema validation by introducing an optional `exclude_columns` parameter, allowing users to specify columns to ignore during validation. This parameter can be used to exclude metadata columns or other columns not relevant to schema validation, and it takes precedence over the `columns` list. * Updated product info when missing in config while verifying workspace client ([#987](#987)). The workspace client configuration has been enhanced to default product information to `dqx` with the current version when it is missing, ensuring that product information is always set for telemetry purposes. * Updated profiler and generator documentation ([#1026](#1026)). The data profiling and quality checks generation feature has been enhanced with updated documentation, providing reference information for data quality profile types and associated rules. * Added filter attribute in rules generated from ODCS ([#978](#978)). The rules generation process has been enhanced with the introduction of a filter attribute in rules generated from Open Data Contract Standard (ODCS), allowing for more flexible and targeted rules creation.
Merged
mwojtyczka
added a commit
that referenced
this pull request
Feb 9, 2026
Change Log for New Release: * New DQX Data Quality Dashboard ([#1019](#1019)). The data quality dashboard has been significantly enhanced to provide a centralized view of data quality metrics across all tables, allowing users to monitor and track data quality issues with greater ease. The dashboard now consists of three tabs - Data Quality Summary, Data Quality by Table (Time Series), and Data Quality by Table (Full Snapshot) - each catering to different monitoring scenarios, and offers customizable parameters for reporting column names and filtering tables with data quality issues. Additionally, the installation process for the dashboard has been simplified, with options to import it directly to a Workspace or deploy it automatically using the Databricks CLI. * DQX App Skeleton ([#982](#982)). The DQX application (frontend and backend) has been built with a core set of features, including configuration management and AI-assisted rule generation based on natural-language input from users. A comprehensive README documents the application architecture as well as development and deployment workflows. Future versions of DQX will introduce additional functionality (loading/saving rules, rules authoring in graphical form) and provide a streamlined, user-friendly way to deploy the application directly into a Databricks workspace. * Added Decimal support to check functions and to min_max generator ([#1013](#1013)) ([#1017](#1017)). The data quality checks have been enhanced to support Python's Decimal type, in addition to int and float, for min/max validation checks, enabling proper data quality checks for decimal-precise financial and scientific data where floating-point precision issues would cause false positives. * Added DQX produciton best practices and fix datetime limit handling ([#997](#997)). Practical guidance and best practices for using DQX in production have been added, covering aspects such as storing checks in Delta tables, enforcing access controls, and optimizing rules for performance and scalability. Fixes have also been implemented to address issues related to handling date and datetime limits, particularly when provided as strings. * Added new row-level check functions: is_null, is_empty, and is_null_or_empty ([#1015](#1015)). DQX now includes three new check functions, `is_null`, `is_empty`, and `is_null_or_empty`, which enable verification of column values as null, empty strings, or both, complementing existing checks like `is_not_null`, `is_not_empty`, and `is_not_null_and_not_empty`. The functions also support optional arguments, like `trim_strings` to trim spaces from strings. * Added tolerance to equality and non-equality check functions ([#1011](#1011)). The library's quality check functionality has been enhanced to support absolute and relative tolerance parameters for numeric value comparisons in `is_equal_to`, `is_not_equal_to`, `is_aggr_equal` and `is_aggr_not_equal` checks, allowing for more flexible and precise control over data validation. The introduction of tolerance logic, which checks for absolute and relative differences within specified thresholds via `abs_tolerance` and `rel_tolerance` parameters, provides more nuanced comparisons for numeric data. * Allow new lines in sql expression checks ([#1009](#1009)). SQL expression check function (`sql_expression`) has been updated to support new lines in its expression argument, allowing for more complex and formatted SQL expressions. * Allow summary metrics with SparkConnect sessions ([#1000](#1000)). The library now supports writing summary metrics directly to a table with SparkConnect sessions, eliminating the need for a classic compute cluster in Dedicated access mode. This change lifts the previous restriction and enables generatic summary metrics using Serverless and all standard clusters with Databricks Runtime 17.3LTS or higeher. * Fixed loading checks from a delta table with special characters ([#992](#992)). The loading checks functionality from a delta table has been fixed to handle special characters in the fully qualified table. * Fixed resolution of pii detection check function ([#1003](#1003)). The PII detection check function resolution has been enhanced to support the application of checks defined as metadata (YAML). * Fixed serialization/deserialization of row filter parameter for dataset-level rules ([#1021](#1021)). The `filter` field in checks definition now correctly pushes down the `filter` condition defined at the check-level as `row_filter` to the check function, allowing checks to operate on the relevant subset of rows before aggregation. The documentation has been updated to advice users to use op-level `filter` condition for consistency instead of `row_filter` parameter. Overall, these changes aim to enhance the overall user experience. * Improved Lakeflow Declarative Pipeline tests ([#1010](#1010)). The Lakeflow Declarative Pipeline (LDP) tests have been enhanced to utilize full Unity Catalog mode, enabling support for writing to arbitrary catalogs and schemas, and performing additional checks to prevent certain operations. * Updated Lakebase authentication method ([#975](#975)). The Lakebase authentication method has been updated to utilize a client ID instead of a username, simplifying its use in the context of a Databricks App. The `lakebase_user` parameter has been replaced with `lakebase_client_id`, an optional service principal client ID used to connect to Lakebase, defaulting to the caller's identity if not provided. This change enhances the security and reliability of the authentication process, making it easier to work with Lakebase as a checks storage. * Updated handling of metadata columns during schema validation ([#1002](#1002)). The `has_valid_schema` check has been enhanced to provide more flexibility in schema validation by introducing an optional `exclude_columns` parameter, allowing users to specify columns to ignore during validation. This parameter can be used to exclude metadata columns or other columns not relevant to schema validation, and it takes precedence over the `columns` list. * Updated product info when missing in config while verifying workspace client ([#987](#987)). The workspace client configuration has been enhanced to default product information to `dqx` with the current version when it is missing, ensuring that product information is always set for telemetry purposes. * Updated profiler and generator documentation ([#1026](#1026)). The data profiling and quality checks generation feature has been enhanced with updated documentation, providing reference information for data quality profile types and associated rules. * Added filter attribute in rules generated from ODCS ([#978](#978)). The rules generation process has been enhanced with the introduction of a filter attribute in rules generated from Open Data Contract Standard (ODCS), allowing for more flexible and targeted rules creation. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Linked issues
Resolves #590
Tests