Support Custom Folder Installation for CLI Commands#942
Merged
Conversation
Contributor
|
✅ 427/427 passed, 2 flaky, 41 skipped, 2h34m47s total Flaky tests:
Running from acceptance #3234 |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #942 +/- ##
==========================================
+ Coverage 90.17% 90.21% +0.04%
==========================================
Files 62 62
Lines 5566 5569 +3
==========================================
+ Hits 5019 5024 +5
+ Misses 547 545 -2 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
mwojtyczka
requested changes
Nov 24, 2025
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds support for custom installation folders to DQX CLI commands by introducing an optional --install-folder argument. This enables users to specify non-default installation locations when running CLI commands, with the custom folder path being passed through the context initialization chain.
Key changes:
- Added
install_folderparameter to CLI commands (open-remote-config,open-dashboards,validate-checks,profile,apply-checks,e2e,logs) - Updated context classes (
GlobalContext,WorkspaceContext,WorkspaceInstaller) to accept and propagate theinstall_folderparameter - Added comprehensive integration tests for custom folder functionality
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/databricks/labs/dqx/cli.py | Added install_folder parameter to CLI commands and updated WorkspaceContext initialization |
| src/databricks/labs/dqx/contexts/global_context.py | Added install_folder field and conditional Installation initialization logic |
| src/databricks/labs/dqx/contexts/workspace_context.py | Added install_folder parameter to constructor and passed to parent class |
| src/databricks/labs/dqx/installer/install.py | Updated WorkspaceInstaller to pass install_folder to parent class |
| tests/integration/test_cli.py | Added integration tests covering all CLI commands with custom folder support |
| tests/integration/conftest.py | Added serverless compute skip condition to custom folder test setup |
| labs.yml | Added install-folder flag documentation to all relevant commands |
| pyproject.toml | Updated dependency versions (databricks-sdk, sqlalchemy) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
mwojtyczka
reviewed
Nov 24, 2025
mwojtyczka
reviewed
Nov 24, 2025
mwojtyczka
approved these changes
Nov 24, 2025
mwojtyczka
reviewed
Nov 24, 2025
vb-dbrks
pushed a commit
that referenced
this pull request
Nov 25, 2025
## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary --> This PR adds `--install-folder` arguments for the DQX CLI commands. This allows users to run the CLI commands when DQX is installed in a custom folder. When users install DQX in a custom folder, they should add arguments to run DQX CLI commands: ``` databricks labs dqx open-dashboards --install-folder "/Workspace/..." ``` ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> Resolves #671 ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [x] manually tested - [ ] added unit tests - [x] added integration tests - [ ] added end-to-end tests - [ ] added performance tests
mwojtyczka
added a commit
that referenced
this pull request
Dec 1, 2025
* Generationg of DQX rules from ODCS Data Contracts ([#932](#932)). The Data Contract Quality Rules Generation feature has been introduced, enabling users to generate data quality rules directly from data contracts following the Open Data Contract Standard (ODCS). This feature supports three types of rule generation: predefined rules derived from schema properties and constraints, explicit DQX rules embedded in the contract, and text-based rules defined in natural language and processed by a Large Language Model (LLM) to generate appropriate checks. The feature provides rich metadata tracing generated rules back to the source contract for lineage and governance, and it can be used to implement federated data governance, standardize data contracts, and maintain version-controlled quality rules alongside schema definitions. * AI-Assisted Primary Key Detection and Uniqueness Rules Generation ([#934](#934)). Introduced AI-assisted primary key detection and uniqueness rules generation capabilities, leveraging Large Language Models (LLMs) to analyze table schema and metadata. This feature analyzes table schemas and metadata to intelligently detect single or composite primary keys, and performs validation by checking for duplicate values. The `DQProfiler` class now includes a `detect_primary_keys_with_llm` method, which returns a dictionary containing the primary key detection result, including the table name, success status, detected primary key columns, confidence level, reasoning, and error message if any. The `DQGenerator` class has been extended to utilize uniqueness profiles from the profiler for AI-assisted uniqueness rules generation. Various updates have been made to the configuration options, including the addition of an `llm_primary_key_detection` option, which allows users to control whether AI-assisted primary key detection is enabled or disabled. * AI-Assisted Rules Generation Improvements ([#925](#925)). The AI-Assisted Rules Generation feature has been enhanced to handle input as a path in addition to a table, and to generate rules with a filter. The `generate_dq_rules_ai_assisted` method now accepts an `InputConfig` object, which allows users to specify the location and format of the input data, enabling more flexible input handling and filtering capabilities. The feature includes test cases to verify its functionality, including manual tests, unit tests, and integration tests, and the documentation has been updated with minor changes to reflect the new functionality. Additionally, the code has been modified to capitalize keywords to stabilize integration tests, and the `DQGenerator` class has been updated to accommodate the changes, allowing users to generate data quality rules from a variety of input sources. The `InputConfig` class provides a flexible way to configure the input data, including its location and format, and the `get_column_metadata` function has been introduced to retrieve column metadata from a given location. Overall, these updates aim to enhance the functionality and usability of the AI-assisted rules generation feature, providing more flexibility and accuracy in generating data quality rules. * Added case-insensitive comparison support to is_in_list and is_not_null_and_is_in_list checks ([#673](#673)). The `is_in_list` and `is_not_null_and_is_in_list` check functions have been enhanced to support case-insensitive comparison, allowing users to choose between case-sensitive and case-insensitive comparisons via an optional `case_sensitive` boolean flag that defaults to True. These checks verify if values in a specified column are present in a list of allowed values, with the `is_not_null_and_is_in_list` check also requiring the values to be non-null. The updated checks provide more flexibility in data validation, enabling users to configure parameters such as the column to check, the list of allowed values, and the case sensitivity flag. However, it is recommended to use the `foreign_key` dataset-level check for large lists of allowed values or for columns of type `MapType` or `StructType`, as these checks are not suitable for such scenarios. * Added documentation for using DQX in streaming scenarios with foreach batch ([#948](#948)). Documentation and example code snippets were added to demonstrate how to apply checks in foreachBatch structured streaming function. * Added telemetry to track count of input tables ([#954](#954)). Added additional telemetry for better trakcing of DQX usage to help improve the product. * Added support for installing DQX from private PYPI repositories ([#930](#930)). The DQX library has been enhanced with support for installing DQX using a company-hosted PyPI mirror, which is necessary for enterprises that block the public PyPI index. The documentation has been added to describe the feature. The tool installation code has been modified to include new functionality for automatically upload dependencies to a workspace when internet access is blocked. * Support Custom Folder Installation for CLI Commands ([#942](#942)). The command-line interface (CLI) has been enhanced to support custom installation folders, providing users with greater flexibility when working with the library. A new `--install-folder` argument has been introduced, allowing users to specify a custom installation folder when running various CLI commands, such as opening dashboards, workflows, logs, and profiles. This argument override the default installation location to support scenarios where the user installs DQX in a custom location. The library's dependency on sqlalchemy has also been updated to require a version greater than or equal to 2.0 and less than 3.0 to avoid dependency issues in older DBRs. * Enhancement to end to end tests ([#921](#921)). The e2e tests has been enhanced to test integration with dbt transformation framework. Additionally, the documentation for contributing to the project and testing has been updated to simplify the setup process for running tests locally. BREAKING CHANGES! * Renamed `level` parameter to `criticality` in `generate_dq_rules` method of `DQGenerator` for consistency. * Replaced `table: str` parameter with `input_config: InputConfig` in `profile_table` method of `DQProfiler` for greater flexibility. * Replaced `table_name: str` parameter with `input_config: InputConfig` in `generate_dq_rules_ai_assisted` method of `DQGenerator` for greater flexibility.
Merged
mwojtyczka
added a commit
that referenced
this pull request
Dec 1, 2025
* Generationg of DQX rules from ODCS Data Contracts ([#932](#932)). The Data Contract Quality Rules Generation feature has been introduced, enabling users to generate data quality rules directly from data contracts following the Open Data Contract Standard (ODCS). This feature supports three types of rule generation: predefined rules derived from schema properties and constraints, explicit DQX rules embedded in the contract, and text-based rules defined in natural language and processed by a Large Language Model (LLM) to generate appropriate checks. The feature provides rich metadata tracing generated rules back to the source contract for lineage and governance, and it can be used to implement federated data governance, standardize data contracts, and maintain version-controlled quality rules alongside schema definitions. * AI-Assisted Primary Key Detection and Uniqueness Rules Generation ([#934](#934)). Introduced AI-assisted primary key detection and uniqueness rules generation capabilities, leveraging Large Language Models (LLMs) to analyze table schema and metadata. This feature analyzes table schemas and metadata to intelligently detect single or composite primary keys, and performs validation by checking for duplicate values. The `DQProfiler` class now includes a `detect_primary_keys_with_llm` method, which returns a dictionary containing the primary key detection result, including the table name, success status, detected primary key columns, confidence level, reasoning, and error message if any. The `DQGenerator` class has been extended to utilize uniqueness profiles from the profiler for AI-assisted uniqueness rules generation. Various updates have been made to the configuration options, including the addition of an `llm_primary_key_detection` option, which allows users to control whether AI-assisted primary key detection is enabled or disabled. * AI-Assisted Rules Generation Improvements ([#925](#925)). The AI-Assisted Rules Generation feature has been enhanced to handle input as a path in addition to a table, and to generate rules with a filter. The `generate_dq_rules_ai_assisted` method now accepts an `InputConfig` object, which allows users to specify the location and format of the input data, enabling more flexible input handling and filtering capabilities. The feature includes test cases to verify its functionality, including manual tests, unit tests, and integration tests, and the documentation has been updated with minor changes to reflect the new functionality. Additionally, the code has been modified to capitalize keywords to stabilize integration tests, and the `DQGenerator` class has been updated to accommodate the changes, allowing users to generate data quality rules from a variety of input sources. The `InputConfig` class provides a flexible way to configure the input data, including its location and format, and the `get_column_metadata` function has been introduced to retrieve column metadata from a given location. Overall, these updates aim to enhance the functionality and usability of the AI-assisted rules generation feature, providing more flexibility and accuracy in generating data quality rules. * Added case-insensitive comparison support to is_in_list and is_not_null_and_is_in_list checks ([#673](#673)). The `is_in_list` and `is_not_null_and_is_in_list` check functions have been enhanced to support case-insensitive comparison, allowing users to choose between case-sensitive and case-insensitive comparisons via an optional `case_sensitive` boolean flag that defaults to True. These checks verify if values in a specified column are present in a list of allowed values, with the `is_not_null_and_is_in_list` check also requiring the values to be non-null. The updated checks provide more flexibility in data validation, enabling users to configure parameters such as the column to check, the list of allowed values, and the case sensitivity flag. However, it is recommended to use the `foreign_key` dataset-level check for large lists of allowed values or for columns of type `MapType` or `StructType`, as these checks are not suitable for such scenarios. * Added documentation for using DQX in streaming scenarios with foreach batch ([#948](#948)). Documentation and example code snippets were added to demonstrate how to apply checks in foreachBatch structured streaming function. * Added telemetry to track count of input tables ([#954](#954)). Added additional telemetry for better trakcing of DQX usage to help improve the product. * Added support for installing DQX from private PYPI repositories ([#930](#930)). The DQX library has been enhanced with support for installing DQX using a company-hosted PyPI mirror, which is necessary for enterprises that block the public PyPI index. The documentation has been added to describe the feature. The tool installation code has been modified to include new functionality for automatically upload dependencies to a workspace when internet access is blocked. * Support Custom Folder Installation for CLI Commands ([#942](#942)). The command-line interface (CLI) has been enhanced to support custom installation folders, providing users with greater flexibility when working with the library. A new `--install-folder` argument has been introduced, allowing users to specify a custom installation folder when running various CLI commands, such as opening dashboards, workflows, logs, and profiles. This argument override the default installation location to support scenarios where the user installs DQX in a custom location. The library's dependency on sqlalchemy has also been updated to require a version greater than or equal to 2.0 and less than 3.0 to avoid dependency issues in older DBRs. * Enhancement to end to end tests ([#921](#921)). The e2e tests has been enhanced to test integration with dbt transformation framework. Additionally, the documentation for contributing to the project and testing has been updated to simplify the setup process for running tests locally. BREAKING CHANGES! * Renamed `level` parameter to `criticality` in `generate_dq_rules` method of `DQGenerator` for consistency. * Replaced `table: str` parameter with `input_config: InputConfig` in `profile_table` method of `DQProfiler` for greater flexibility. * Replaced `table_name: str` parameter with `input_config: InputConfig` in `generate_dq_rules_ai_assisted` method of `DQGenerator` for greater flexibility.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
This PR adds
--install-folderarguments for the DQX CLI commands. This allows users to run the CLI commands when DQX is installed in a custom folder.When users install DQX in a custom folder, they should add arguments to run DQX CLI commands:
Linked issues
Resolves #671
Tests