619 feature generate dqx rules from odcs data contracts by vb-dbrks · Pull Request #932 · databrickslabs/dqx

vb-dbrks · 2025-11-14T22:56:15Z

Changes

Added support for generating DQ rules from ODCS (Open Data Contract Standard) v3.0.x data contracts. This enables customers implementing data contracts to automatically derive quality checks from their standardised contract definitions.

Key features:

Implicit rule generation: Automatically infers checks from schema properties (required, unique, pattern, minValue/maxValue, validValues, format)
Explicit rule generation: Supports custom DQX-native rules embedded in ODCS contracts
Text-based rules: Converts natural language quality expectations to DQX rules using LLM (optional)
Dataset-level rules: Supports both property-level and dataset-level quality checks (e.g., freshness, row counts)
Full validation: Validates contracts against official ODCS JSON schemas (v3.0.0-v3.0.2)
Lineage tracking: Preserves contract metadata (name, version, property, dimension) in generated rules

New components:

src/databricks/labs/dqx/datacontract/ - New module for contract parsing and rule generation
src/databricks/labs/dqx/datacontract/schemas/ - Bundled ODCS JSON schemas for validation
demos/dqx_demo_odcs.py - Interactive demo notebook
Added jsonschema dependency for contract validation

Usage:

generator = DQGenerator(workspace_client=ws)
rules = generator.generate_rules_from_contract(
    contract=odcs_contract_dict,
    generate_implicit_rules=True,
    process_text_rules=True,
    default_criticality="error"
)

Linked issues

Resolves #...

Tests

manually tested
added unit tests (30 test cases for parsing and rule generation)
added integration tests (8 end-to-end tests with Spark DataFrames)
added end-to-end tests
added performance tests

Adding support for automated rule generation from ODCS data contracts. ODCS (Open Data Contract Standard) v3.0.x data contracts are supported. Rules can be generated implicitly from schema properties (required fields, value ranges, patterns, formats) or explicitly from quality definitions and text-based expectations! Main entry point is DQGenerator.generate_rules_from_contract() which supports: - Implicit rules from schema constraints (required, unique, min/max, pattern, etc) - Explicit rules in DQX native format - Text-based expectations processed via LLM with full schema context - Custom criticality mapping by quality dimension - Complete contract metadata tracking in generated rules Includes unit tests, integration tests, demo notebook, and sample contracts for testing and documentation.

…ity in explicit custom rules, so adding an external mapping parameter was redundant and complex.

…validation - Enforce full JSON Schema validation against bundled ODCS schemas (v3.0.0, v3.0.1, v3.0.2) - Implement proper ODCS quality check format with required fields: * type: "text" or "custom" (required) * engine: "dqx" for custom checks (required) * implementation: contains DQX rule details (required) - Use ODCS-compliant logical types: * 'date' for both dates and timestamps (distinguished by format) * 'number' instead of 'numeric' - Detect timestamps vs dates based on format string (HH, mm, ss) - Extract DQX checks from 'implementation' field per ODCS custom format spec

…om validation logic. improved the demo example.

…ple of the ODCS contract. Improved the demo notebook.

Copilot

Pull Request Overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 9 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Marcin Wojtyczka <marcin.wojtyczka@databricks.com>

Copilot

Pull Request Overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mwojtyczka

LGTM

…, TypeError instead of a generic exception

…rning

Copilot

Pull Request Overview

Copilot reviewed 17 out of 19 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

vb-dbrks added 9 commits November 14, 2025 20:16

Removed criticality_mapping Parameter. ODCS already supports critical…

ae2ef62

…ity in explicit custom rules, so adding an external mapping parameter was redundant and complex.

improved example and test data

60023d3

minor fixes to odcs parser and validator

79f7cf5

updated and fixed the demo. formatting and linting fixes.

8e4ef1f

code quality improvements

d6a509d

code quality improvements for testing scripts

c89b619

bug fix: changes to contract format not updated in tests

128fa6c

vb-dbrks linked an issue Nov 14, 2025 that may be closed by this pull request

[FEATURE]: Generate DQX rules from ODCS data contracts #619

Closed

1 task

Code quality and readability improvements.

c8cf62e

vb-dbrks assigned mwojtyczka Nov 14, 2025

vb-dbrks added 6 commits November 17, 2025 11:08

simplify: added datacontract-cli as optional dependency. reduced cust…

76d5324

…om validation logic. improved the demo example.

tests: improving unit and integration test and code coverage and exam…

80eb759

…ple of the ODCS contract. Improved the demo notebook.

code quality: linting, formatting and refactoring.

5d5e249

formatting: pylint for imports added and updated the monketpatch

1964c7e

test: fixture redacted the actual workspace name for mocked llm output

0b2777c

docs: added new docs for data contract based DQ generation.

5dc719e

vb-dbrks marked this pull request as ready for review November 17, 2025 16:27

vb-dbrks requested a review from a team as a code owner November 17, 2025 16:27

vb-dbrks requested review from tombonfert and removed request for a team November 17, 2025 16:27

vb-dbrks had a problem deploying to tool November 17, 2025 16:27 — with GitHub Actions Failure

vb-dbrks had a problem deploying to tool November 17, 2025 16:27 — with GitHub Actions Error

vb-dbrks added documentation Improvements or additions to documentation enhancement New feature or request dependencies Pull requests that update a dependency file labels Nov 17, 2025

vb-dbrks added 7 commits November 18, 2025 23:33

improved integration tests

1cefb02

Refactor datacontract import handling and improve error messaging.

5abd915

fix: future import type checking for DataContract

7bdc240

minor: updated docs

c248a57

fix: ODCS 3.x specification

d6c0bc2

code comment improvements

fb772c3

docs: anchor link fix

93abcaf

mwojtyczka requested changes Nov 19, 2025

View reviewed changes

Copilot AI reviewed Nov 19, 2025

View reviewed changes

vb-dbrks and others added 7 commits November 19, 2025 19:01

PR: Code review comments addressed.

a7a848d

Update docs/dqx/docs/guide/data_contract_quality_rules_generation.mdx

b5242e1

Co-authored-by: Marcin Wojtyczka <marcin.wojtyczka@databricks.com>

fix test: test asserts for unit

ee62a0e

improved coverage of tests

3dcfef0

improve tests: increase test coverage

ad3cbdd

fix: ODCS demo sample data

b13a3bc

fix: demo dqx check save

b92869c

Copilot AI reviewed Nov 20, 2025

View reviewed changes

mwojtyczka reviewed Nov 20, 2025

View reviewed changes

Comment thread tests/resources/sample_datacontract_old.yaml Outdated

refactor

0f5ec94

mwojtyczka approved these changes Nov 20, 2025

View reviewed changes

Comment thread tests/integration/test_datacontract_integration.py Outdated

mwojtyczka reviewed Nov 21, 2025

View reviewed changes

Comment thread src/databricks/labs/dqx/datacontract/contract_rules_generator.py Outdated

mwojtyczka reviewed Nov 21, 2025

View reviewed changes

Comment thread src/databricks/labs/dqx/datacontract/contract_rules_generator.py Outdated

vb-dbrks and others added 6 commits November 21, 2025 09:53

improvement: added specific ValidationError, AttributeError, KeyError…

b4628fd

…, TypeError instead of a generic exception

refactor

6e9e63a

simplify: removed ODCS object checks

042b2ba

simplify: data-types and additional tests

6197476

bugs: fixed slient errors and filter dq rules with errors and show wa…

e027e14

…rning

refactor, added test

d5d4683

Copilot AI reviewed Nov 21, 2025

View reviewed changes

Comment thread src/databricks/labs/dqx/utils.py

mwojtyczka mentioned this pull request Dec 1, 2025

Release v0.11.0 #955

Merged

Uh oh!

Conversation

vb-dbrks commented Nov 14, 2025 • edited by mwojtyczka Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Linked issues

Tests

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mwojtyczka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vb-dbrks commented Nov 14, 2025 •

edited by mwojtyczka

Loading