Skip to content
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/dqx/docs/guide/quality_checks_apply.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ You can apply quality checks to your data using the following approaches:
* [No-code approach (Workflows)](#no-code-approach-workflows): you can apply checks for data at-rest using workflows if DQX is installed in the workspace as a tool. This does not require code-level integration and is suitable for data already stored in Delta tables or files.

DQX offers a set of predefined built-in quality rules (checks) as described [here](/docs/guide/quality_checks_definition).
All DQX rules are evaluated independently and in parallel. If one rule fails, the others are still processed.

## Programmatic approach

Expand All @@ -30,7 +31,8 @@ Checks can be applied to the input data by one of the following methods of the `

You can see the full list of `DQEngine` methods [here](/docs/reference/engine/#dqx-engine-methods).

The engine will raise an error if you try to apply checks with invalid definition.
The engine ensures that the specified `column`, `columns`, or `filter` fields can be resolved in the input DataFrame. If any of these fields are invalid, the check evaluation is skipped, and the results include the check failure with a message identifying the invalid fields.
The engine will raise an error if you try to apply checks with invalid definition (e.g. wrong syntax).
In addition, you can also perform a standalone syntax validation of the checks as described [here](/docs/guide/quality_checks_definition#validating-syntax-of-quality-checks).

You can apply quality checks to streaming pipelines using the same methods as for batch processing.
Expand Down
14 changes: 7 additions & 7 deletions docs/dqx/docs/guide/quality_checks_storage.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -72,18 +72,18 @@ If you create checks as a list of `DQRule` objects, you can convert them to meta
# save checks as a YAML file in arbitrary workspace location (overwrite the file) using absolute path
dq_engine.save_checks(checks, config=WorkspaceFileChecksStorageConfig(location="/Shared/App1/checks.yml"))

# save checks in a Delta table to "default" run config (append checks)
# save checks in a Delta table to "default" run config, append checks
dq_engine.save_checks(checks, config=TableChecksStorageConfig(location="catalog.schema.checks_table", mode="append"))

# save checks in a Delta table with specific run config for filtering (overwrite checks)
dq_engine.save_checks(checks, config=TableChecksStorageConfig(location="catalog.schema.checks_table", run_config_name="workflow_001", mode="overwrite"))
# save checks in a Delta table with specific run config for filtering (e.g. input table name), overwrite checks
dq_engine.save_checks(checks, config=TableChecksStorageConfig(location="catalog.schema.checks_table", run_config_name="main.default.input_table", mode="overwrite"))
Comment thread
mwojtyczka marked this conversation as resolved.
Outdated

# save checks as a YAML in a Unity Catalog Volume location (overwrite the file)
dq_engine.save_checks(checks, config=VolumeFileChecksStorageConfig(location="/Volumes/dq/config/checks_volume/App1/checks.yml"))

# save checks as a YAML file or table defined in 'checks_location' of the run config
# only works if DQX is installed in the workspace
dq_engine.save_checks(checks, config=InstallationChecksStorageConfig(assume_user=True, run_config_name="default"))
dq_engine.save_checks(checks, config=InstallationChecksStorageConfig(assume_user=True, run_config_name="main.default.input_table"))
```
</TabItem>
<TabItem value="Workflows" label="Workflows">
Expand Down Expand Up @@ -210,15 +210,15 @@ If you create checks as a list of DQRule objects, you can convert them using the
# load checks from a Delta table and default run config name
checks: list[dict] = dq_engine.load_checks(config=TableChecksStorageConfig(location="catalog.schema.checks_table"))

# load checks from a Delta table with specific run config for filtering
checks: list[dict] = dq_engine.load_checks(config=TableChecksStorageConfig(location="catalog.schema.checks_table", run_config_name="workflow_001"))
# load checks from a Delta table with specific run config for filtering (e.g. table name)
checks: list[dict] = dq_engine.load_checks(config=TableChecksStorageConfig(location="catalog.schema.checks_table", run_config_name="main.default.input_table"))

# load checks from a Unity Catalog Volume
checks: list[dict] = dq_engine.load_checks(config=VolumeFileChecksStorageConfig(location="/Volumes/dq/config/checks_volume/App1/checks.yml"))

# load checks from a file or table defined in the run config ('checks_location' field)
# only works if DQX is installed in the workspace
checks: list[dict] = dq_engine.load_checks(config=InstallationChecksStorageConfig(run_config_name="default"))
checks: list[dict] = dq_engine.load_checks(config=InstallationChecksStorageConfig(run_config_name="main.default.input_table"))

# validate loaded checks
assert not dq_engine.validate_checks(checks).has_errors
Expand Down
Loading
Loading