Added quality checker and end to end workflows#519
Merged
Merged
Conversation
This reverts commit f4a9901.
# Conflicts: # docs/dqx/docs/guide/quality_checks.mdx # docs/dqx/docs/reference/engine.mdx # src/databricks/labs/dqx/profiler/runner.py # src/databricks/labs/dqx/profiler/workflow.py # tests/integration/conftest.py
* added support for absolute paths to checks location in installation checks storage handler * set product correctly when creating workspace client in workflow context * use string for run_time filed in extra params to simplify serialization * added integration test for quality checker * removed unused connect field from config * removed unused properties from contexts
added integration tests for custom functions
mwojtyczka
added
the
do-not-merge
updated tool demo updated docs
updated existing docs
ghanse
requested changes
Aug 21, 2025
| The "default" run config will be used if the run config is not provided. The run config is used to select specific run configuration from the 'config.yml'. | ||
|
|
||
| The following DQX configuration from 'config.yml' is used by the profiler workflow: | ||
| - 'input_config': configuration for the input data. |
Collaborator
There was a problem hiding this comment.
Would be nice to have a link here to the config documentation.
Contributor
Author
There was a problem hiding this comment.
it is already linked: configuration file
or do you mean to create a separate page for the config?
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds comprehensive data quality checking functionality and end-to-end workflows to DQX. It introduces a new quality checker job/workflow that can validate data at rest independently of code-level integration, supports custom check functions and reference data validation, and adds CLI commands for applying checks and running end-to-end workflows.
Key Changes
- Added quality checker workflow and end-to-end workflow capabilities
- Introduced support for custom check functions and reference tables
- Added serverless cluster support by default for all workflows
- Refactored installer architecture with cleaner separation of concerns
Reviewed Changes
Copilot reviewed 82 out of 85 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/test_workflow_task.py | Updated test assertions for job_cluster default value change |
| tests/unit/test_workflow_instaler.py | Fixed import path typo and quote style consistency |
| tests/unit/test_utils.py | Added test for new get_reference_dataframes utility function |
| src/databricks/labs/dqx/workflows_runner.py | Renamed from runtime.py and added quality checker/e2e workflows |
| src/databricks/labs/dqx/utils.py | Added get_reference_dataframes function for reference table handling |
| src/databricks/labs/dqx/quality_checker/ | New module containing quality checker workflow and runner implementations |
| src/databricks/labs/dqx/profiler/ | Refactored profiler components with updated naming and storage handling |
| src/databricks/labs/dqx/installer/ | Major refactoring with new warehouse installer, version checker, and config provider |
| src/databricks/labs/dqx/config.py | Moved ExtraParams from rule module and added new configuration fields |
| src/databricks/labs/dqx/cli.py | Added new CLI commands for apply-checks and e2e workflows |
| src/databricks/labs/dqx/checks_resolver.py | Added support for resolving custom check functions from file paths |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
# Conflicts: # docs/dqx/docs/reference/quality_checks.mdx
dinbab1984
approved these changes
Aug 22, 2025
dinbab1984
left a comment
dinbab1984
left a comment
Contributor
There was a problem hiding this comment.
LGTM, tested the DQX tooling demo. Everything works as expected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
databricks labs dqx apply-checksdatabricks labs dqx e2eInstallationChecksStorageHandlerserverless_cluster=True). Applicable to profiler and quality checker.connectfield from heconfigExtraParamsfromrulemodule toconfigto be able to set additional params for the quality checker. Also it is not used in rules and don't really belongs there.run_timeinExtraParamsas string in iso format to be able to serialize the configtimeout-minutesparameter to the profiler and apply checks cli commandsTODO:
FOLLOW UPs:
BREAKING CHANGES!
ExtraParamsmoved fromdatabricks.labs.dqx.ruletodatabricks.labs.dqx.configLinked issues
Resolves #173
Tests