Demo for Spark Structured Streaming#518
Merged
Merged
Conversation
ghanse
requested changes
Aug 7, 2025
mwojtyczka
reviewed
Aug 8, 2025
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds a new demo for Spark Structured Streaming that demonstrates how to use the DQX library with streaming data. The demo is based on the existing DLT demo and processes NYC Taxi data with data quality checks, splitting the stream into silver and quarantine tables.
Key changes:
- Added a new streaming demo notebook that applies DQX data quality checks to streaming data
- Added integration test to validate the streaming demo execution
- Updated documentation to reference the new streaming demo
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| demos/dqx_streaming_demo.py | New notebook demonstrating DQX with Spark Structured Streaming using NYC taxi data |
| tests/e2e/test_run_demos.py | Added integration test for the streaming demo with proper parameterization |
| docs/dqx/docs/demos.mdx | Updated documentation to include link to the new streaming demo |
mwojtyczka
reviewed
Aug 8, 2025
Contributor
|
✅ 8/8 passed, 57m55s total Running from acceptance #1725 |
mwojtyczka
approved these changes
Aug 10, 2025
mwojtyczka
left a comment
mwojtyczka
left a comment
Contributor
There was a problem hiding this comment.
LGTM - will merge after testing
AdityaMandiwal
pushed a commit
that referenced
this pull request
Aug 21, 2025
## Changes * Added a demo for Spark Structured Streaming. This is based on the existing dlt demo: dqx_dlt_demo.py ### Linked issues This PR addresses the [FEATURE]: Create demo for Spark Structured Streaming #489 ### Tests - [X] manually tested - [ ] added unit tests - [X] added integration tests --------- Co-authored-by: Marcin Wojtyczka <marcin.wojtyczka@databricks.com>
mwojtyczka
added a commit
that referenced
this pull request
Aug 22, 2025
* Added quality checker and end to end workflows ([#519](#519)). This release introduces no-code solution for applying checks. The following workflows were added: quality-checker (apply checks and save results to tables) and end-to-end (e2e) workflows (profile input data, generate quality checks, apply the checks, save results to tables). The workflows enable quality checking for data at-rest without the need for code-level integration. It supports reference data for checks using tables (e.g., required by foreign key or compare datasets checks) as well as custom python check functions (mapping of custom check funciton to the module path in the workspace or Unity Catalog volume containing the function definition). The workflows handle one run config for each job run. Future release will introduce functionality to execute this across multiple tables. In addition, CLI commands have been added to execute the workflows. Additionaly, DQX workflows are configured now to execute using serverless clusters, with an option to use standards clusters as well. InstallationChecksStorageHandler now support absolute workspace path locations. * Added built-in row-level check for PII detection ([#486](#486)). Introduced a new built-in check for Personally Identifiable Information (PII) detection, which utilizes the Presidio framework and can be configured using various parameters, such as NLP entity recognition configuration. This check can be defined using the `does_not_contain_pii` check function and can be customized to suit specific use cases. The check requires `pii` extras to be installed: `pip install databricks-labs-dqx[pii]`. Furthermore, a new enum class `NLPEngineConfig` has been introduced to define various NLP engine configurations for PII detection. Overall, these updates aim to provide more robust and customizable quality checking capabilities for detecting PII data. * Added equality row-level checks ([#535](#535)). Two new row-level checks, `is_equal_to` and `is_not_equal_to`, have been introduced to enable equality checks on column values, allowing users to verify whether the values in a specified column are equal to or not equal to a given value, which can be a numeric literal, column expression, string literal, date literal, or timestamp literal. * Added demo for Spark Structured Streaming ([#518](#518)). Added demo to showcase usage of DQX with Spark Structured Streaming for in-transit data quality checking. The demo is available as Databricks notebook, and can be run on any Databricks workspace. * Added clarification to profiler summary statistics ([#523](#523)). Added new section on understanding summary statistics, which explains how these statistics are computed on a sampled subset of the data and provides a reference for the various summary statistics fields. * Fixed rounding datetimes in the checks generator ([#517](#517)). The generator has been enhanced to correctly handle midnight values when rounding "up", ensuring that datetime values already at midnight remain unchanged, whereas previously they were rounded to the next day. * Added API Docs ([#520](#520)). The DQX API documentation is generated automatically using docstrings. As part of this change the library's documentation has been updated to follow Google style. * Improved test automation by adding end-to-end test for the asset bundles demo ([#533](#533)). BREAKING CHANGES! * `ExtraParams` was moved from `databricks.labs.dqx.rule` module to `databricks.labs.dqx.config`
Merged
mwojtyczka
added a commit
that referenced
this pull request
Aug 23, 2025
* Added quality checker and end to end workflows ([#519](#519)). This release introduces no-code solution for applying checks. The following workflows were added: quality-checker (apply checks and save results to tables) and end-to-end (e2e) workflows (profile input data, generate quality checks, apply the checks, save results to tables). The workflows enable quality checking for data at-rest without the need for code-level integration. It supports reference data for checks using tables (e.g., required by foreign key or compare datasets checks) as well as custom python check functions (mapping of custom check funciton to the module path in the workspace or Unity Catalog volume containing the function definition). The workflows handle one run config for each job run. Future release will introduce functionality to execute this across multiple tables. In addition, CLI commands have been added to execute the workflows. Additionaly, DQX workflows are configured now to execute using serverless clusters, with an option to use standards clusters as well. InstallationChecksStorageHandler now support absolute workspace path locations. * Added built-in row-level check for PII detection ([#486](#486)). Introduced a new built-in check for Personally Identifiable Information (PII) detection, which utilizes the Presidio framework and can be configured using various parameters, such as NLP entity recognition configuration. This check can be defined using the `does_not_contain_pii` check function and can be customized to suit specific use cases. The check requires `pii` extras to be installed: `pip install databricks-labs-dqx[pii]`. Furthermore, a new enum class `NLPEngineConfig` has been introduced to define various NLP engine configurations for PII detection. Overall, these updates aim to provide more robust and customizable quality checking capabilities for detecting PII data. * Added equality row-level checks ([#535](#535)). Two new row-level checks, `is_equal_to` and `is_not_equal_to`, have been introduced to enable equality checks on column values, allowing users to verify whether the values in a specified column are equal to or not equal to a given value, which can be a numeric literal, column expression, string literal, date literal, or timestamp literal. * Added demo for Spark Structured Streaming ([#518](#518)). Added demo to showcase usage of DQX with Spark Structured Streaming for in-transit data quality checking. The demo is available as Databricks notebook, and can be run on any Databricks workspace. * Added clarification to profiler summary statistics ([#523](#523)). Added new section on understanding summary statistics, which explains how these statistics are computed on a sampled subset of the data and provides a reference for the various summary statistics fields. * Fixed rounding datetimes in the checks generator ([#517](#517)). The generator has been enhanced to correctly handle midnight values when rounding "up", ensuring that datetime values already at midnight remain unchanged, whereas previously they were rounded to the next day. * Added API Docs ([#520](#520)). The DQX API documentation is generated automatically using docstrings. As part of this change the library's documentation has been updated to follow Google style. * Improved test automation by adding end-to-end test for the asset bundles demo ([#533](#533)). BREAKING CHANGES! * `ExtraParams` was moved from `databricks.labs.dqx.rule` module to `databricks.labs.dqx.config`
mwojtyczka
added a commit
that referenced
this pull request
Aug 25, 2025
## 0.9.1 * Added quality checker and end to end workflows ([#519](#519)). This release introduces no-code solution for applying checks. The following workflows were added: quality-checker (apply checks and save results to tables) and end-to-end (e2e) workflows (profile input data, generate quality checks, apply the checks, save results to tables). The workflows enable quality checking for data at-rest without the need for code-level integration. It supports reference data for checks using tables (e.g., required by foreign key or compare datasets checks) as well as custom python check functions (mapping of custom check funciton to the module path in the workspace or Unity Catalog volume containing the function definition). The workflows handle one run config for each job run. Future release will introduce functionality to execute this across multiple tables. In addition, CLI commands have been added to execute the workflows. Additionaly, DQX workflows are configured now to execute using serverless clusters, with an option to use standards clusters as well. InstallationChecksStorageHandler now support absolute workspace path locations. * Added built-in row-level check for PII detection ([#486](#486)). Introduced a new built-in check for Personally Identifiable Information (PII) detection, which utilizes the Presidio framework and can be configured using various parameters, such as NLP entity recognition configuration. This check can be defined using the `does_not_contain_pii` check function and can be customized to suit specific use cases. The check requires `pii` extras to be installed: `pip install databricks-labs-dqx[pii]`. Furthermore, a new enum class `NLPEngineConfig` has been introduced to define various NLP engine configurations for PII detection. Overall, these updates aim to provide more robust and customizable quality checking capabilities for detecting PII data. * Added equality row-level checks ([#535](#535)). Two new row-level checks, `is_equal_to` and `is_not_equal_to`, have been introduced to enable equality checks on column values, allowing users to verify whether the values in a specified column are equal to or not equal to a given value, which can be a numeric literal, column expression, string literal, date literal, or timestamp literal. * Added demo for Spark Structured Streaming ([#518](#518)). Added demo to showcase usage of DQX with Spark Structured Streaming for in-transit data quality checking. The demo is available as Databricks notebook, and can be run on any Databricks workspace. * Added clarification to profiler summary statistics ([#523](#523)). Added new section on understanding summary statistics, which explains how these statistics are computed on a sampled subset of the data and provides a reference for the various summary statistics fields. * Fixed rounding datetimes in the checks generator ([#517](#517)). The generator has been enhanced to correctly handle midnight values when rounding "up", ensuring that datetime values already at midnight remain unchanged, whereas previously they were rounded to the next day. * Added API Docs ([#520](#520)). The DQX API documentation is generated automatically using docstrings. As part of this change the library's documentation has been updated to follow Google style. * Improved test automation by adding end-to-end test for the asset bundles demo ([#533](#533)). BREAKING CHANGES! * `ExtraParams` was moved from `databricks.labs.dqx.rule` module to `databricks.labs.dqx.config`
Merged
mwojtyczka
added a commit
that referenced
this pull request
Aug 25, 2025
The release replaces v0.9.0, which were missing PyPI package and will be removed. ## 0.9.1 * Added quality checker and end to end workflows ([#519](#519)). This release introduces no-code solution for applying checks. The following workflows were added: quality-checker (apply checks and save results to tables) and end-to-end (e2e) workflows (profile input data, generate quality checks, apply the checks, save results to tables). The workflows enable quality checking for data at-rest without the need for code-level integration. It supports reference data for checks using tables (e.g., required by foreign key or compare datasets checks) as well as custom python check functions (mapping of custom check funciton to the module path in the workspace or Unity Catalog volume containing the function definition). The workflows handle one run config for each job run. Future release will introduce functionality to execute this across multiple tables. In addition, CLI commands have been added to execute the workflows. Additionaly, DQX workflows are configured now to execute using serverless clusters, with an option to use standards clusters as well. InstallationChecksStorageHandler now support absolute workspace path locations. * Added built-in row-level check for PII detection ([#486](#486)). Introduced a new built-in check for Personally Identifiable Information (PII) detection, which utilizes the Presidio framework and can be configured using various parameters, such as NLP entity recognition configuration. This check can be defined using the `does_not_contain_pii` check function and can be customized to suit specific use cases. The check requires `pii` extras to be installed: `pip install databricks-labs-dqx[pii]`. Furthermore, a new enum class `NLPEngineConfig` has been introduced to define various NLP engine configurations for PII detection. Overall, these updates aim to provide more robust and customizable quality checking capabilities for detecting PII data. * Added equality row-level checks ([#535](#535)). Two new row-level checks, `is_equal_to` and `is_not_equal_to`, have been introduced to enable equality checks on column values, allowing users to verify whether the values in a specified column are equal to or not equal to a given value, which can be a numeric literal, column expression, string literal, date literal, or timestamp literal. * Added demo for Spark Structured Streaming ([#518](#518)). Added demo to showcase usage of DQX with Spark Structured Streaming for in-transit data quality checking. The demo is available as Databricks notebook, and can be run on any Databricks workspace. * Added clarification to profiler summary statistics ([#523](#523)). Added new section on understanding summary statistics, which explains how these statistics are computed on a sampled subset of the data and provides a reference for the various summary statistics fields. * Fixed rounding datetimes in the checks generator ([#517](#517)). The generator has been enhanced to correctly handle midnight values when rounding "up", ensuring that datetime values already at midnight remain unchanged, whereas previously they were rounded to the next day. * Added API Docs ([#520](#520)). The DQX API documentation is generated automatically using docstrings. As part of this change the library's documentation has been updated to follow Google style. * Improved test automation by adding end-to-end test for the asset bundles demo ([#533](#533)). BREAKING CHANGES! * `ExtraParams` was moved from `databricks.labs.dqx.rule` module to `databricks.labs.dqx.config`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Linked issues
This PR addresses the [FEATURE]: Create demo for Spark Structured Streaming #489
Tests