Skip to content

Commit 7f3abb6

Browse files
EscotomwojtyczkaCopilot
authored
Created a quick start demo (#367)
This PR adds a demo notebook that illustrates how to do Data Quality checking with DQX. **WHY** I created this demo to provide an easy-to-follow, hands-on example for users who want to quickly test Databricks DQX features using the **Databricks Power Tools** and **Databricks Extension** in VS Code. The goal is to help new users understand and compare both configuration approaches in a practical context. **How** * The notebook contains both configuration styles side by side, using the same three rules for direct comparison. * A small, hardcoded sample dataset is included for quick experimentation. * The demo is designed to be executed cell-by-cell in VS Code. **Testing** * Tested in VS Code using: -- Databricks Extension -- Databricks Power Tools Extension * All notebook cells execute successfully and produce expected results with library version 0.5.0. I have hardcoded the library version to 0.5.0 because the project is under active development. While I am committed to using and maintaining this demo as the library evolves, I recognize that ongoing maintenance may be needed. If this contribution is accepted, I am happy to submit regular updates to keep the demo aligned with future releases as I plan to extensively use this library. --------- Co-authored-by: Marcin Wojtyczka <marcin.wojtyczka@databricks.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 parent 98c3ef1 commit 7f3abb6

2 files changed

Lines changed: 212 additions & 0 deletions

File tree

‎demos/notebooks/dqx_quick_start_demo_library.ipynb‎

Lines changed: 211 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,211 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# DQX - Use as library demo\n",
8+
"\n",
9+
"In this demo we demonstrate how to create and apply a set of rules from YAML configuration. \n",
10+
"\n",
11+
"**Note:**\n",
12+
"This notebook can be executed without any modifications when using the `VS Code Databricks Extension`"
13+
]
14+
},
15+
{
16+
"cell_type": "markdown",
17+
"metadata": {},
18+
"source": [
19+
"### Install DQX"
20+
]
21+
},
22+
{
23+
"cell_type": "code",
24+
"execution_count": null,
25+
"metadata": {},
26+
"outputs": [],
27+
"source": [
28+
"%pip install databricks-labs-dqx\n",
29+
"%restart_python"
30+
]
31+
},
32+
{
33+
"cell_type": "markdown",
34+
"metadata": {},
35+
"source": [
36+
"### Import Required Modules"
37+
]
38+
},
39+
{
40+
"cell_type": "code",
41+
"execution_count": null,
42+
"metadata": {},
43+
"outputs": [],
44+
"source": [
45+
"import yaml\n",
46+
"from databricks.labs.dqx.engine import DQEngine\n",
47+
"from databricks.sdk import WorkspaceClient\n",
48+
"from pyspark.sql import SparkSession, Row"
49+
]
50+
},
51+
{
52+
"cell_type": "markdown",
53+
"metadata": {},
54+
"source": [
55+
"### Configure Test Data\n",
56+
"\n",
57+
"The result of this next step is `new_users_df`, which represents a dataframe of new users which requires quality validation."
58+
]
59+
},
60+
{
61+
"cell_type": "code",
62+
"execution_count": null,
63+
"metadata": {},
64+
"outputs": [],
65+
"source": [
66+
"spark = SparkSession.builder.appName(\"DQX_demo_library\").getOrCreate()\n",
67+
"\n",
68+
"# Create a sample DataFrame\n",
69+
"new_users_sample_data = [\n",
70+
" Row(id=1, age=23, country='Germany'),\n",
71+
" Row(id=2, age=30, country='France'),\n",
72+
" Row(id=3, age=16, country='Germany'), # Invalid -> age - LT 18\n",
73+
" Row(id=None, age=29, country='France'), # Invalid -> id - NULL\n",
74+
" Row(id=4, age=29, country=''), # Invalid -> country - Empty\n",
75+
" Row(id=5, age=23, country='Italy'), # Invalid -> country - not in\n",
76+
" Row(id=6, age=123, country='France') # Invalid -> age - GT 120\n",
77+
"]\n",
78+
"\n",
79+
"new_users_df = spark.createDataFrame(new_users_sample_data)"
80+
]
81+
},
82+
{
83+
"cell_type": "markdown",
84+
"metadata": {},
85+
"source": [
86+
"### Demonstrating Functions\n",
87+
"- `is_not_null_and_not_empty`\n",
88+
"- `is_in_range`\n",
89+
"- `is_in_list`\n",
90+
"\n",
91+
"You can find a list of all available built-in checks in the documentation [here](https://databrickslabs.github.io/dqx/docs/reference/quality_rules/) and in the code [here](https://github.com/databrickslabs/dqx/blob/main/src/databricks/labs/dqx/check_funcs.py).\n",
92+
"\n",
93+
"We are demonstrating creating and validating a set of `Quality Checks` defined declaratively using YAML.\n",
94+
"\n",
95+
"We can use `validate_checks` to verify that the checks are defined correctly."
96+
]
97+
},
98+
{
99+
"cell_type": "code",
100+
"execution_count": null,
101+
"metadata": {},
102+
"outputs": [],
103+
"source": [
104+
"checks_from_yaml = yaml.safe_load(\"\"\"\n",
105+
"- check:\n",
106+
" function: is_not_null_and_not_empty\n",
107+
" for_each_column: # define check for multiple columns at once\n",
108+
" - id\n",
109+
" - age\n",
110+
" - country\n",
111+
" criticality: error\n",
112+
"- check:\n",
113+
" function: is_in_range\n",
114+
" filter: country in ['Germany', 'France']\n",
115+
" arguments:\n",
116+
" column: age # define check for a single column\n",
117+
" min_limit: 18\n",
118+
" max_limit: 120\n",
119+
" criticality: warn\n",
120+
" name: age_not_in_range # optional check name, auto-generated if not provided\n",
121+
"- check:\n",
122+
" function: is_in_list\n",
123+
" for_each_column:\n",
124+
" - country\n",
125+
" arguments:\n",
126+
" allowed:\n",
127+
" - Germany\n",
128+
" - France\n",
129+
" criticality: warn\n",
130+
"\"\"\")\n",
131+
"\n",
132+
"# Validate YAML checks\n",
133+
"status = DQEngine.validate_checks(checks_from_yaml)\n",
134+
"print(f\"Checks from YAML: {status}\")"
135+
]
136+
},
137+
{
138+
"cell_type": "markdown",
139+
"metadata": {
140+
"vscode": {
141+
"languageId": "polyglot-notebook"
142+
}
143+
},
144+
"source": [
145+
"### Setup `DQEngine`"
146+
]
147+
},
148+
{
149+
"cell_type": "code",
150+
"execution_count": null,
151+
"metadata": {},
152+
"outputs": [],
153+
"source": [
154+
"ws = WorkspaceClient() # auto-authenticated inside Databricks\n",
155+
"dq_engine = DQEngine(ws)"
156+
]
157+
},
158+
{
159+
"cell_type": "markdown",
160+
"metadata": {},
161+
"source": [
162+
"### Apply Rules\n",
163+
"`apply_checks_by_metadata` results in one `DataFrame` with `_errors` and `_warnings` metadata columns added."
164+
]
165+
},
166+
{
167+
"cell_type": "code",
168+
"execution_count": null,
169+
"metadata": {},
170+
"outputs": [],
171+
"source": [
172+
"validated_df = dq_engine.apply_checks_by_metadata(new_users_df, checks_from_yaml)\n",
173+
"display(validated_df)"
174+
]
175+
},
176+
{
177+
"cell_type": "markdown",
178+
"metadata": {},
179+
"source": [
180+
"### Apply Rules And Split\n",
181+
"`apply_checks_by_metadata_and_split` results in a `tuple[DataFrame, DataFrame]` with `_errors` and `_warnings` metadata columns added. The first DF contains valid records, and the second invalid/quarantined records."
182+
]
183+
},
184+
{
185+
"cell_type": "code",
186+
"execution_count": null,
187+
"metadata": {},
188+
"outputs": [],
189+
"source": [
190+
"valid_records_df, invalid_records_df = dq_engine.apply_checks_by_metadata_and_split(new_users_df, checks_from_yaml)\n",
191+
"display(valid_records_df)"
192+
]
193+
},
194+
{
195+
"cell_type": "code",
196+
"execution_count": null,
197+
"metadata": {},
198+
"outputs": [],
199+
"source": [
200+
"display(invalid_records_df)"
201+
]
202+
}
203+
],
204+
"metadata": {
205+
"language_info": {
206+
"name": "python"
207+
}
208+
},
209+
"nbformat": 4,
210+
"nbformat_minor": 2
211+
}

‎docs/dqx/docs/demos.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import Admonition from '@theme/Admonition';
88

99
Import the following notebooks in the Databricks workspace to try DQX out:
1010
* [DQX Demo Notebook (library)](https://github.com/databrickslabs/dqx/blob/main/demos/dqx_demo_library.py) - demonstrates how to use DQX as a library.
11+
* [DQX Quick Start Demo Notebook (library)](https://github.com/databrickslabs/dqx/blob/main/demos/notebooks/dqx_quick_start_demo_library.ipynb) - quickstart on how to use DQX as a library.
1112
* [DQX Demo Notebook (tool)](https://github.com/databrickslabs/dqx/blob/main/demos/dqx_demo_tool.py) - demonstrates how to use DQX as a tool when installed in the workspace.
1213
* [DQX DLT Demo Notebook](https://github.com/databrickslabs/dqx/blob/main/demos/dqx_dlt_demo.py) - demonstrates how to use DQX with Delta Live Tables (DLT).
1314
* [DQX for PII Detection Notebook](https://github.com/databrickslabs/dqx/blob/main/demos/dqx_demo_pii_detection.py) - demonstrates how to use DQX to check data for Personally Identifiable Information (PII).

0 commit comments

Comments
 (0)