Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
396ea41
feature -
vb-dbrks Nov 14, 2025
ae2ef62
Removed criticality_mapping Parameter. ODCS already supports critical…
vb-dbrks Nov 14, 2025
60023d3
improved example and test data
vb-dbrks Nov 14, 2025
c85d5af
Fully conform to official ODCS v3.0.x specification with JSON Schema …
vb-dbrks Nov 14, 2025
79f7cf5
minor fixes to odcs parser and validator
vb-dbrks Nov 14, 2025
8e4ef1f
updated and fixed the demo. formatting and linting fixes.
vb-dbrks Nov 14, 2025
d6a509d
code quality improvements
vb-dbrks Nov 14, 2025
c89b619
code quality improvements for testing scripts
vb-dbrks Nov 14, 2025
128fa6c
bug fix: changes to contract format not updated in tests
vb-dbrks Nov 14, 2025
c8cf62e
Code quality and readability improvements.
vb-dbrks Nov 14, 2025
76d5324
simplify: added datacontract-cli as optional dependency. reduced cust…
vb-dbrks Nov 17, 2025
80eb759
tests: improving unit and integration test and code coverage and exam…
vb-dbrks Nov 17, 2025
5d5e249
code quality: linting, formatting and refactoring.
vb-dbrks Nov 17, 2025
1964c7e
formatting: pylint for imports added and updated the monketpatch
vb-dbrks Nov 17, 2025
0b2777c
test: fixture redacted the actual workspace name for mocked llm output
vb-dbrks Nov 17, 2025
5dc719e
docs: added new docs for data contract based DQ generation.
vb-dbrks Nov 17, 2025
08fb100
Update tests/unit/test_datacontract_utils.py
vb-dbrks Nov 17, 2025
33e8351
copilot suggestions: changes.
vb-dbrks Nov 17, 2025
7185ae9
feat: updated the implementation for string length tests using sql_ex…
vb-dbrks Nov 17, 2025
6b5c3d5
bug fix: issues with docs.
vb-dbrks Nov 17, 2025
58535e8
bugfix: sql_expression and DataContractSpecification
vb-dbrks Nov 17, 2025
244f9f9
fix: imports for integration and e2e tests
vb-dbrks Nov 18, 2025
60d81ad
fix: Quality class import not available as datacontract library isn't…
vb-dbrks Nov 18, 2025
3982189
minor:
vb-dbrks Nov 18, 2025
265e28e
fix: unquoted type annotations
vb-dbrks Nov 18, 2025
961a5cb
Merge branch 'main' into 619-feature-generate-dqx-rules-from-odcs-dat…
mwojtyczka Nov 18, 2025
3fce468
Update demos/dqx_demo_datacontract_odcs.py
vb-dbrks Nov 18, 2025
a9805e4
PR Comments: addressed
vb-dbrks Nov 18, 2025
598f959
PR Code Review Comments:
vb-dbrks Nov 18, 2025
1cefb02
improved integration tests
vb-dbrks Nov 18, 2025
5abd915
Refactor datacontract import handling and improve error messaging.
vb-dbrks Nov 19, 2025
7bdc240
fix: future import type checking for DataContract
vb-dbrks Nov 19, 2025
c248a57
minor: updated docs
vb-dbrks Nov 19, 2025
d6c0bc2
fix: ODCS 3.x specification
vb-dbrks Nov 19, 2025
fb772c3
code comment improvements
vb-dbrks Nov 19, 2025
93abcaf
docs: anchor link fix
vb-dbrks Nov 19, 2025
a7a848d
PR: Code review comments addressed.
vb-dbrks Nov 19, 2025
b5242e1
Update docs/dqx/docs/guide/data_contract_quality_rules_generation.mdx
vb-dbrks Nov 19, 2025
ee62a0e
fix test: test asserts for unit
vb-dbrks Nov 19, 2025
3dcfef0
improved coverage of tests
vb-dbrks Nov 19, 2025
ad3cbdd
improve tests: increase test coverage
vb-dbrks Nov 19, 2025
b13a3bc
fix: ODCS demo sample data
vb-dbrks Nov 20, 2025
b92869c
fix: demo dqx check save
vb-dbrks Nov 20, 2025
0f5ec94
refactor
mwojtyczka Nov 20, 2025
b4628fd
improvement: added specific ValidationError, AttributeError, KeyError…
vb-dbrks Nov 21, 2025
6e9e63a
refactor
mwojtyczka Nov 21, 2025
042b2ba
simplify: removed ODCS object checks
vb-dbrks Nov 21, 2025
6197476
simplify: data-types and additional tests
vb-dbrks Nov 21, 2025
e027e14
bugs: fixed slient errors and filter dq rules with errors and show wa…
vb-dbrks Nov 21, 2025
d5d4683
refactor, added test
mwojtyczka Nov 21, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
466 changes: 466 additions & 0 deletions demos/dqx_demo_datacontract_odcs.py

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions docs/dqx/docs/demos.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ Import the following notebooks in the Databricks workspace to try DQX out:
* [DQX Demo Notebook for Data Quality Summary Metrics](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_demo_summary_metrics.py) - demonstrates how to generate summary-level data quality metrics when validating data with DQX.
* [DQX Demo Notebook for Profiling and Applying Checks at Scale on Multiple Tables](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_multi_table_demo.py) - demonstrates how to use DQX as a library at scale to apply checks on multiple tables.
* [DQX Demo Notebook for AI-assisted checks generation](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_demo_ai_assisted_checks_generation.py) - demonstrates how to generate DQX rules/checks with LLM using natural language.
* [DQX Demo Notebook for Data Contract Integration (ODCS)](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_demo_datacontract_odcs.py) - demonstrates how to generate DQX quality rules from ODCS (Open Data Contract Standard) data contracts, including predefined rules from schema constraints, explicit custom rules, and contract metadata tracking.
* [DQX Demo Notebook for Spark Structured Streaming (Native End-to-End Approach)](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_streaming_demo_native.py) - demonstrates how to use DQX as a library with Spark Structured Streaming, using the built-in end-to-end method to handle both reading and writing.
* [DQX Demo Notebook for Spark Structured Streaming (DIY Approach)](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_streaming_demo_diy.py) - demonstrates how to use DQX as a library with Spark Structured Streaming, while handling reading and writing on your own outside DQX using Spark API.
* [DQX Demo Notebook for Lakeflow Pipelines (formerly DLT)](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_dlt_demo.py) - demonstrates how to use DQX as a library with Lakeflow Pipelines.
Expand Down
Loading
Loading