[FEATURE]: Generate DQX rules from ODCS data contracts

[mwojtyczka]

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

Open Data Contract Standard (ODCS) is becoming quite popular. This is a contract, so does not run rules as such. DQX rules can be generated for the contracts.

Proposed Solution

Converter to generated DQX rules from ODCS contract.

Example:
https://github.com/bitol-io/open-data-contract-standard/blob/main/docs/examples/quality/column-accuracy.odcs.yaml

Docs:
https://bitol-io.github.io/open-data-contract-standard/v3.0.2/

Additional Context

Requested by some customers and Ontos

[vb-dbrks]

Would love to pick this up :)

[vb-dbrks]

@mwojtyczka Based on discussions today with you and the Astellas team. Please let me know if anything here is unclear or not aligned.

What’s the need/requirement?

Customers implementing federated governance strategies need a standardised approach for defining data contracts for their Data Products. The Open Data Contract Standard (ODCS), part of the Linux Foundation, provides a comprehensive specification format that captures data product descriptors and properties.

E.g. Astellas requires the ability to generate DQ checks from ODCS contracts:

• Implicitly - Auto-derived from schema properties and logical types
• Explicitly - Defined within the contract's quality section

Proposed Capability

1. Implicitly Derived DQ Rules
DQ rules should be automatically inferred from ODCS schema properties, including:

• Logical Types: Parse and derive checks based on ODCS logical type definitions
• Schema Constraints: Extract validation rules from property definitions
• Format Specifications: Generate format validation checks

Example Mappings:
ODCS Property -> Description -> DQX Check
unique: true -> Primary key/unique constraint -> is_unique
required: true -> Non-null constraint -> is_not_null
minValue/maxValue -> Range constraints -> is_in_range
minItems/maxItems -> Array size constraints -> Custom array validation
pattern: -> Regex pattern -> regex_match
format: -> Date/timestamp format -> is_valid_date/is_valid_timestamp

2. Explicitly Defined DQ Rules
ODCS supports multiple DQ rule definition approaches (see ODCS Data Quality section):

• Text-based expectations - Natural language rules (can leverage LLM-to-DQX converter)
• ODCS DQ Library - Standard library format rules
• SQL-based checks - Direct SQL expressions (maps to sql_expression in DQX)
• Custom properties - External tool formats (including DQX native format)

Design Principles

All generated DQ rules must:

1. Be traceable to the source contract - Track ODCS contract version, name, and metadata
2. Inherit scheduling from ODCS - Use scheduling defined in the contract
3. Include contract versioning - Capture ODCS contract version in generated rules metadata
4. Persist results with lineage - Save DQ check results alongside associated ODCS contract metadata (version, name, descriptors).

Side note: Versioning DQRules is Out of scope for this feature, and will be implemented as a part another; but it will have to be included later in the lineage ODCS Version -> DQ Check Version -> Results/Metrics Summary

[mwojtyczka]

Thank you for the summary. Happy to see the customer contributing to this as well. For versioning added another ticket.

[Palkers76]

Hey, will it also save rules to contract i.e if created using LLM ? it would be nice

[vb-dbrks]

Hey, will it also save rules to contract i.e if created using LLM ? it would be nice

Hey @Palkers76 , this feature will Generate DQ rules from

1. Predefined rules in ODCS contracts
2. Text rules defined in the ODCS contract using the previously release DQX LLM feature
3. Custom DQX rules that you've defined which you were anyways able to do today.
   https://bitol-io.github.io/open-data-contract-standard/v3.0.1/#data-quality

The generated rules will also have metadata to allow you to maintain a lineage back to the contract versions!