Update handling of metadata columns during schema validation

[ghanse]

Changes

This PR adds capabilities in DQEngine to ignore metadata columns and internal columns during dataset-level checks (e.g. has_valid_schema) which depend on a stable input schema.

I also adds optional exclude_columns which is list of columns in the checked DataFrame schema to ignore for validation.

Check functions can be decorated to register them for schema preselection:

@register_for_original_columns_preselection()
def has_valid_schema(
    expected_schema: str | types.StructType | None = None,
    ref_df_name: str | None = None,
    ref_table: str | None = None,
    columns: list[str | Column] | None = None,
    strict: bool = False,
    exclude_columns: list[str] | None = None,
) -> tuple[Column, Callable]:
...

Rules decorated in this way will have original dataframe columns preselected.

Linked issues

Resolves #989

Tests

• manually tested
• added unit tests
• added integration tests
• added end-to-end tests
• added performance tests

[github-actions]

✅ 508/508 passed, 1 flaky, 41 skipped, 3h34m6s total

Flaky tests:

• 🤪 test_e2e_workflow_serverless (10m21.703s)

_{Running from acceptance #3685}

[Copilot]

Pull request overview

This PR adds support for handling metadata columns (result columns added by DQEngine) during schema validation by introducing an ignore_columns parameter to the has_valid_schema check function. The engine automatically adds result column names to the ignore list when performing schema validation, preventing false positives when result columns are included in the DataFrame schema.

Changes:

• Added ignore_columns parameter to has_valid_schema function to allow excluding specific columns from schema validation
• Added _get_checks_with_ignored_result_columns method in DQEngine to automatically ignore result columns during schema validation
• Added integration test for the new ignore_columns parameter functionality
• Updated documentation to reflect the new parameter

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
src/databricks/labs/dqx/engine.py	Added method to automatically append result column names to ignore_columns for schema validation checks, and imported check_funcs module
src/databricks/labs/dqx/check_funcs.py	Added ignore_columns parameter to has_valid_schema function to support excluding columns from schema comparison
tests/integration/test_dataset_checks.py	Added integration test for ignore_columns parameter in has_valid_schema function
docs/dqx/docs/reference/quality_checks.mdx	Updated documentation with examples and parameter description for ignore_columns

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mwojtyczka]

you can remove downstreams workflow, we don't need it

[mwojtyczka]

LGTM