Adding support for running checks on multiple tables

[mwojtyczka]

Changes

Added new execution methods to the engine and Workflows to support running quality checks on multiple tables including running for all tables/views matching wildcard patterns.

DETAILS:

• Added new methods to the engine:
  • apply_checks_and_save_in_tables - run quality checking and save results for multiple run configs.
  • apply_checks_and_save_in_tables_for_patterns - run quality checking and save results for all tables matching wildcard patterns.
• Renamed profile_tables to profile_tables_for_patterns to use common naming convention.
• Extended workflows (profiler, quality checker and e2e) to support execution of workflows for all run configs and for tables matching wildcard patterns. Conventions for pattern based execution:
  • Run config is used as a template for all relevant fields except location
  • Input table location is derived from the patterns (patterns job parameter)
  • Optionally user can provide exclude patterns (exclude_patterns job parameter)
  • Output and quarantine tables are constructed as follows: <input_table><output_table_suffix>, <input_table><quarantine_table_suffix>.
  • output_table_suffix job parameter uses "_dq_output" by default.
  • quarantine_table_suffix job parameter uses "_dq_quarantine" by default.
  • For table-based checks location, checks are saved/loaded from the delta table. For file-based checks location, the file name is replaced with "<input_table>.yml". In addition, if the location is specified as a relative path, it is prefixed with the workspace installation folder.
  • Quarantine is performed only if the run config has quarantine_config defined.
• Workflows are deployed by default with empty run config name (run_config_name) and patterns (patterns) so that they run for all defined run configs in the configuration file by default. User can run workflows for specific run config by providing run_config_name, and for tables matching wildcard patterns by providing patterns parameter when running the workflows.
• Updated the following CLI commands to make run-config parameter optional and by default execute all run configs: validate-checks, profile, apply-checks
• Added --patterns, --exclude-patterns, --output-table-suffix, and --quarantine-table-suffix parameters to the CLI commands for pattern based matching execution of workflows
• Added demo to showcase how to use profiler and apply checks across multiple tables
• Corrected doc examples on how to load checks from a storage
• Corrected docs for sql expression

BREAKING CHANGES!

• Workflows run by default for all run configs from configuration file. Previously, the default behaviour was to run them for a specific run config only.
• The following depreciated methods are removed from the DQEngine: load_checks_from_local_file, load_checks_from_workspace_file, load_checks_from_table, load_checks_from_installation, save_checks_in_local_file, save_checks_in_workspace_file, save_checks_in_table,, save_checks_in_installation, load_run_config. For loading and saving checks, users are advised to use load_checks and save_checks of the DQEngine described here, which support various storage types.

Linked issues

Resolves #488 #586

Tests

• manually tested
• added unit tests
• added integration tests
• added end-to-end tests

[Copilot]

Pull Request Overview

This PR adds support for running data quality checks on multiple tables simultaneously, extending the existing single-table functionality. The implementation includes a new apply_checks_and_save_in_tables method that enables end-to-end checking for multiple input sources with configurable parallelism.

Key changes include:

• Added multi-table processing capability with configurable parallelism
• Refactored I/O operations into a separate module for better organization
• Enhanced storage factory to support dynamic handler creation based on location patterns

Reviewed Changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/databricks/labs/dqx/engine.py	Added apply_checks_and_save_in_tables method and helper functions for multi-table processing
src/databricks/labs/dqx/io.py	New module containing refactored I/O operations moved from utils.py
src/databricks/labs/dqx/utils.py	Added list_tables function for table discovery and removed I/O functions
src/databricks/labs/dqx/checks_storage.py	Enhanced factory with create_for_location method for dynamic handler creation
tests/integration/test_apply_checks_and_save_in_table.py	New comprehensive test suite for multi-table functionality
demos/dqx_multi_table_demo.py	Demo notebook showcasing multi-table processing capabilities

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[github-actions]

✅ 402/402 passed, 3 flaky, 1 skipped, 4h14m30s total

Flaky tests:

• 🤪 test_e2e_workflow (13m11.591s)
• 🤪 test_col_is_not_null (1.173s)
• 🤪 test_e2e_workflow_with_custom_install_folder (15m25.76s)

_{Running from acceptance #2685}

[dinbab1984]

LGTM

[Copilot]

Pull Request Overview

Copilot reviewed 63 out of 66 changed files in this pull request and generated 6 comments.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Pull Request Overview

Copilot reviewed 63 out of 66 changed files in this pull request and generated 9 comments.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

Pull Request Overview

Copilot reviewed 68 out of 71 changed files in this pull request and generated 4 comments.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}