Allow installation in a custom folder

.github/workflows/acceptance.yml

@@ -41,7 +41,7 @@ jobs:
           python-version: '3.12'
 
       - name: Install hatch
-        run: pip install hatch==1.9.4
+        run: pip install "hatch==1.13.0" "click<8.3"
 
       - name: Run unit tests and generate test coverage report
         run: make test
@@ -93,7 +93,7 @@ jobs:
           python-version: '3.12'
 
       - name: Install hatch
-        run: pip install hatch==1.9.4
+        run: pip install "hatch==1.13.0" "click<8.3"
 
       - name: Run integration tests on serverless cluster
         uses: databrickslabs/sandbox/acceptance@acceptance/v0.4.4
@@ -125,7 +125,7 @@ jobs:
           python-version: '3.12'
 
       - name: Install hatch
-        run: pip install hatch==1.9.4
+        run: pip install "hatch==1.13.0" "click<8.3"
 
       - name: Install Databricks CLI
         run: |
@@ -177,7 +177,7 @@ jobs:
           python-version: '3.12'
 
       - name: Install hatch
-        run: pip install hatch==1.9.4
+        run: pip install "hatch==1.13.0" "click<8.3"
 
       - name: Install Databricks CLI
         run: |

.github/workflows/docs-release.yml

@@ -28,7 +28,7 @@ jobs:
 
       - name: Install Hatch
         run: |
-          pip install hatch==1.9.4
+          pip install "hatch==1.13.0" "click<8.3"
 
       - uses: actions/setup-node@v4
         with:

.github/workflows/downstreams.yml

@@ -43,7 +43,7 @@ jobs:
 
       - name: Install toolchain
         run: |
-          pip install hatch==1.9.4
+          pip install "hatch==1.13.0" "click<8.3"
 
       - name: Check downstream compatibility
         uses: databrickslabs/sandbox/downstreams@downstreams/v0.0.1

.github/workflows/nightly.yml

@@ -32,7 +32,7 @@ jobs:
           python-version: '3.12'
 
       - name: Install hatch
-        run: pip install hatch==1.9.4
+        run: pip install "hatch==1.13.0" "click<8.3"
 
       - name: Run unit tests and generate test coverage report
         run: make test
@@ -81,7 +81,7 @@ jobs:
           python-version: '3.12'
 
       - name: Install hatch
-        run: pip install hatch==1.9.4
+        run: pip install "hatch==1.13.0" "click<8.3"
 
       - name: Run integration tests on serverless cluster
         uses: databrickslabs/sandbox/acceptance@acceptance/v0.4.4
@@ -97,7 +97,6 @@ jobs:
           DATABRICKS_SERVERLESS_COMPUTE_ID: ${{ env.DATABRICKS_SERVERLESS_COMPUTE_ID }}
 
   e2e:
-    if: github.event_name == 'pull_request' && !github.event.pull_request.draft && !github.event.pull_request.head.repo.fork
     environment: tool
     runs-on: larger
     steps:
@@ -114,7 +113,7 @@ jobs:
           python-version: '3.12'
 
       - name: Install hatch
-        run: pip install hatch==1.9.4
+        run: pip install "hatch==1.13.0" "click<8.3"
 
       - name: Install Databricks CLI
         run: |
@@ -147,7 +146,6 @@ jobs:
           ARM_TENANT_ID: ${{ secrets.ARM_TENANT_ID }}
 
   e2e_serverless:
-    if: github.event_name == 'pull_request' && !github.event.pull_request.draft && !github.event.pull_request.head.repo.fork
     environment: tool
     runs-on: larger
     env:
@@ -166,7 +164,7 @@ jobs:
           python-version: '3.12'
 
       - name: Install hatch
-        run: pip install hatch==1.9.4
+        run: pip install "hatch==1.13.0" "click<8.3"
 
       - name: Install Databricks CLI
         run: |
@@ -219,7 +217,7 @@ jobs:
           python-version: '3.12'
 
       - name: Install hatch
-        run: pip install hatch==1.9.4
+        run: pip install "hatch==1.13.0" "click<8.3"
 
       - name: Login to Azure for azure-cli authentication
         uses: azure/login@v2

.github/workflows/performance.yml

@@ -43,7 +43,7 @@ jobs:
           cache-dependency-path: '**/pyproject.toml'
 
       - name: Install hatch
-        run: pip install hatch==1.9.4
+        run: pip install "hatch==1.13.0" "click<8.3"
 
       - name: Login to Azure for azure-cli authentication
         uses: azure/login@v2

.github/workflows/push.yml

@@ -37,7 +37,7 @@ jobs:
 
       - name: Run unit tests
         run: |
-          pip install hatch==1.9.4
+          pip install "hatch==1.13.0" "click<8.3"
           make test
 
   fmt:

.github/workflows/release.yml

@@ -27,7 +27,7 @@ jobs:
 
       - name: Build wheels
         run: |
-          pip install hatch==1.9.4
+          pip install "hatch==1.13.0" "click<8.3"
           hatch build
 
       - name: Github release

docs/dqx/docs/guide/quality_checks_storage.mdx

@@ -12,11 +12,22 @@ import TabItem from '@theme/TabItem';
 DQX provides flexible methods to load and save quality checks (rules) defined as metadata (a list of dictionaries) from different storage backends, making it easier to manage, share, and reuse checks across workflows and environments.
 
 Saving and loading methods accept a storage backend configuration as input. The following backend configuration are currently supported:
-- `FileChecksStorageConfig`: local files (JSON/YAML), or workspace files if invoked from Databricks notebook or job
-- `WorkspaceFileChecksStorageConfig`: workspace files (JSON/YAML) using absolute paths
-- `VolumeFileChecksStorageConfig`: Unity Catalog volumes (JSON/YAML file)
-- `TableChecksStorageConfig`: Unity Catalog tables
-- `InstallationChecksStorageConfig`: installation-managed location from the run config, ignores location and infers it from `checks_location` in the run config
+* `FileChecksStorageConfig`: local files (JSON/YAML), or workspace files if invoked from Databricks notebook or job. Containing fields:
+    * `location`: absolute or relative file path in the local filesystem (JSON or YAML); also works with absolute or relative workspace file paths if invoked from Databricks notebook or job.
+* `WorkspaceFileChecksStorageConfig`: workspace files (JSON/YAML) using absolute paths. Containing fields:
+    * `location`: absolute workspace file path (JSON or YAML).
+* `TableChecksStorageConfig`: Unity Catalog tables. Containing fields:
+    * `location`: table fully qualified name.
+    * `run_config_name`: (optional) run configuration name to load (use "default" if not provided).
+    * `mode`: (optional) write mode for saving checks (`overwrite` or `append`, default is `overwrite`). The `overwrite` mode will only replace checks for the specific run config and not all checks in the table.
+* `VolumeFileChecksStorageConfig`: Unity Catalog volumes (JSON/YAML file). Containing fields:
+    * `location`: Unity Catalog Volume file path (JSON or YAML).
+* `InstallationChecksStorageConfig`: installation-managed location from the run config, ignores location and infers it from `checks_location` in the run config. Containing fields:
+    * `location` (optional): automatically set based on the `checks_location` field from the run configuration.
+    * `install_folder`: (optional) installation folder where DQX is installed, only required when custom installation folder is used.
+    * `run_config_name` (optional) - run configuration name to load (use "default" if not provided).
+    * `product_name`: (optional) name of the product (use "dqx" if not provided).
+    * `assume_user`: (optional) if True, assume user installation, otherwise global installation (skipped if `install_folder` is provided).
 
 You can find details on how to define checks [here](/docs/guide/quality_checks_definition).
 

docs/dqx/docs/installation.mdx

@@ -82,8 +82,7 @@ Install a specific version of DQX in your Databricks workspace via Databricks CL
 databricks labs install dqx@v0.8.0
 ```
 
-You'll be prompted to select a [configuration profile](https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication) created by `databricks auth login` command,
-and other configuration options.
+You'll be prompted to select a [configuration profile](https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication) created by `databricks auth login` command, and other configuration options.
 
 The cli command will install the following components in the workspace installation folder:
 - A Python [wheel file](https://peps.python.org/pep-0427/) with the library packaged.
@@ -98,19 +97,38 @@ It is recommended to use serverless clusters for the workflows, as it allows for
 If serverless clusters are not used, a default cluster configuration will be used for the workflows.
 Alternatively, you can override the cluster configuration for each workflow in the `config.yml` file after the installation, to provide existing clusters to use.
 
-#### User vs Global Installation
+#### Installation Options
 
-DQX is installed by default in the user home directory (under `/Users/<user>/.dqx`). You can also install DQX globally
-by setting the 'DQX_FORCE_INSTALL' environment variable. The following options are available:
+DQX offers flexible installation options. By default, DQX is installed in the user home directory under `/Users/<user>/.dqx`.
+You can also install DQX in a global folder or any custom workspace folder.
 
-* `DQX_FORCE_INSTALL=global databricks labs install dqx`: will force the installation to be for root only (`/Applications/dqx`)
-* `DQX_FORCE_INSTALL=user databricks labs install dqx`: will force the installation to be for user only (`/Users/<user>/.dqx`)
+**Environment Variable Override:**
+
+You can force global or user installation using the 'DQX_FORCE_INSTALL' environment variable:
+  * `DQX_FORCE_INSTALL=global databricks labs install dqx`: forces installation to `/Applications/dqx`
+  * `DQX_FORCE_INSTALL=user databricks labs install dqx`: forces installation to `/Users/<user>/.dqx` (default behaviour)
+
+**Custom Workspace Folder:**
+
+If you provide a custom path during installation (e.g., `/Shared/dqx-team` or `/Users/shared-user/dqx-project`), DQX will be installed there.
+You will be prompted to optionally enter a workspace path when installing DQX.
+
+The custom folder installation is required when using [Group Assigned cluster](https://docs.databricks.com/aws/en/compute/group-access),
+as the concept of a user home directory does not exist in this setup.
+
+<Admonition type="info" title="Complete Profiling Guide">
+If a custom folder is provided during installation, the installation folder will take precedence over any environment variables (e.g. `DQX_FORCE_INSTALL`).
+</Admonition> 
 
 #### Configuration file
 
 DQX configuration file can contain multiple run configurations for different pipelines or projects, each defining specific input, output and quarantine locations, etc.
-By default, the config is created in the installation directory under `/Users/<user>/.dqx/config.yml` or `/Applications/dqx/config.yml` if installed globally.
-The "default" run configuration is created during the installation. When DQX is upgraded, the configuration is preserved.
+The configuration file is created in the installation directory depending on installation options (see above):
+- User home (default): `/Users/<user>/.dqx/config.yml`
+- Global: `/Applications/dqx/config.yml` (when DQX_FORCE_INSTALL=global)
+- Custom folder: `<your-installation-folder>/config.yml` (when provided during installation)
+
+A "default" run configuration is created during the installation. When DQX is upgraded, the configuration is preserved.
 The configuration can be updated / extended manually by the user after the installation. Each run config defines configuration for one specific input and output location.
 
 Open the configuration file:

docs/dqx/docs/reference/cli.mdx

@@ -24,8 +24,9 @@ databricks labs uninstall dqx
 ```
 
 <Admonition type="info" title="Where is DQX installed?">
-By default, DQX is installed under the user's home (for example `/Users/<user>/.dqx`).
-Use the `DQX_FORCE_INSTALL` env var to force a global or user install. See the installation guide for details.
+By default, DQX is installed under the user's home folder (for example `/Users/<user>/.dqx`).
+Use the `DQX_FORCE_INSTALL` env var to force a global or user install. Provide a custom installation
+folder during installation to override the default location. See the installation guide for details.
 </Admonition>
 
 ## Configuration helpers