Update has_valid_schema check to accept a reference dataframe or table

[ghanse]

Changes

This PR introduces reference datasets (either tables or dataframes) for the has_valid_schema check function.

The behavior is as follows:

• When ref_dfs is created in-code and ref_df_name is specified, the valid schema will be determined from the reference dataframe
• When ref_table is specified, the valid schema will be determined by loading the reference table as a Spark dataframe

Specifying multiple valid schema sources (e.g. expected_schema and ref_df_name or ref_table) will raise an InvalidParameterError.

Linked issues

Resolves #959

Tests

• manually tested
• added unit tests
• added integration tests
• added end-to-end tests
• added performance tests

[github-actions]

✅ 482/482 passed, 1 flaky, 41 skipped, 3h37m9s total

Flaky tests:

• 🤪 test_e2e_workflow_serverless (9m44.999s)

_{Running from acceptance #3373}

[Copilot]

Pull request overview

This PR adds support for specifying a reference table as the source of the expected schema in the has_valid_schema check function. Instead of providing an explicit schema string or StructType, users can now pass a ref_table parameter pointing to a table in the catalog, and the check will load that table's schema as the expected schema.

Key changes:

• Added ref_table parameter to has_valid_schema function as an alternative to expected_schema
• Added validation to ensure exactly one of expected_schema or ref_table is specified
• Updated the apply method signature to accept a spark parameter for loading reference tables

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
src/databricks/labs/dqx/check_funcs.py	Added ref_table parameter, validation logic, and schema loading from reference table
tests/unit/test_dataset_checks.py	Added unit tests for parameter validation
tests/integration/test_dataset_checks.py	Updated all test calls to pass spark parameter and added integration test for ref_table functionality
docs/dqx/docs/reference/quality_checks.mdx	Updated documentation with examples of using ref_table parameter

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mwojtyczka]

LGTM