Skip to content

Adding Lakebase checks storage backend#550

Merged
mwojtyczka merged 145 commits into
mainfrom
feature_503_store_retrieve_checks
Oct 16, 2025
Merged

Adding Lakebase checks storage backend#550
mwojtyczka merged 145 commits into
mainfrom
feature_503_store_retrieve_checks

Conversation

@tlgnr

@tlgnr tlgnr commented Aug 28, 2025

Copy link
Copy Markdown
Contributor

Changes

Adding Lakebase checks storage backend.

Linked issues

Resolves #444

Tests

  • manually tested
  • added unit tests
  • added integration tests
  • added end-to-end tests

@tlgnr tlgnr linked an issue Aug 28, 2025 that may be closed by this pull request
1 task
@github-actions

Copy link
Copy Markdown
Contributor

All commits in PR should be signed ('git commit -S ...'). See https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits

@mwojtyczka mwojtyczka changed the title Feature 503 store retrieve checks Adding Lakebase checks storage backend Aug 29, 2025
@mwojtyczka mwojtyczka requested a review from Copilot August 29, 2025 09:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for Lakebase as a new storage backend for quality checks, allowing users to store and retrieve checks from Lakebase tables instead of only Delta tables or files.

  • Introduces LakebaseChecksStorageConfig configuration class with instance name and database location parameters
  • Implements LakebaseChecksStorageHandler with PostgreSQL-based operations for loading and saving checks
  • Updates documentation and factory methods to include the new Lakebase backend option

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.

File Description
src/databricks/labs/dqx/engine.py Updates documentation to include Lakebase storage option
src/databricks/labs/dqx/config.py Adds LakebaseChecksStorageConfig class with validation
src/databricks/labs/dqx/checks_storage.py Implements LakebaseChecksStorageHandler with PostgreSQL connection pooling

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py Outdated

@mwojtyczka mwojtyczka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

Please also updated docs and add tests.

Please sign commits: https://databrickslabs.github.io/dqx/docs/dev/contributing/

Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py
Comment thread src/databricks/labs/dqx/config.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
@tlgnr tlgnr force-pushed the feature_503_store_retrieve_checks branch from e7b8ac3 to 9bb5e48 Compare September 10, 2025 11:21
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@tlgnr tlgnr force-pushed the feature_503_store_retrieve_checks branch from 9bb5e48 to b333413 Compare September 10, 2025 11:23
@mwojtyczka mwojtyczka requested a review from Copilot September 22, 2025 16:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.


Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread tests/unit/test_load_checks.py Outdated
@mwojtyczka mwojtyczka requested a review from Copilot September 25, 2025 09:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.


Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread tests/unit/test_save_checks.py Outdated
Comment thread tests/unit/test_load_checks.py Outdated
Comment thread tests/unit/test_load_checks.py Outdated
Comment thread src/databricks/labs/dqx/config.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.


Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread tests/integration/test_save_and_load_checks_from_lakebase_table.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread src/databricks/labs/dqx/config.py Outdated
Comment thread src/databricks/labs/dqx/checks_storage.py Outdated
Comment thread src/databricks/labs/dqx/config.py Outdated

@mwojtyczka mwojtyczka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mwojtyczka mwojtyczka mentioned this pull request Nov 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE]: Storing/Retrieving checks (API) [FEATURE]: Option to save checks in Lakebase

3 participants