Fork test: PR #1319 - Persist check metadata across table storage backends (message_expr + typed user_metadata)#1322
Open
mwojtyczka wants to merge 6 commits into
Open
Fork test: PR #1319 - Persist check metadata across table storage backends (message_expr + typed user_metadata)#1322mwojtyczka wants to merge 6 commits into
mwojtyczka wants to merge 6 commits into
Conversation
Add a message_expr column to the Delta and Lakebase checks-table schemas and JSON-encode user_metadata values on the Delta path so custom messages and non-string metadata survive a save->load round-trip through a table backend (previously they only round-tripped via File/Volume). Existing tables are migrated in place: Delta generalizes the versioning-column ALTER to add message_expr, and Lakebase adds it idempotently on save and detects its presence on load. safe_json_load tolerates None so legacy NULL map values decode safely. Resolves #1308
mwojtyczka
added
do-not-merge
mwojtyczka
added
do-not-merge
Contributor
|
❌ 796/800 passed, 28 flaky, 4 failed, 42 skipped, 4h25m23s total ❌ test_complete_installation: databricks.sdk.errors.platform.BadRequest: The quota for the number of jobs has been reached. The current quota is 12000. This quota is only applied to jobs created through the UI or through the /jobs/create endpoint, which are displayed in the Jobs UI. If this limit is hit, it is very likely some user programmatically created many jobs that only need to run once. In this case, we recommend to delete those jobs and to submit one-time job runs instead using the /jobs/runs/submit endpoint. There is no quota on the number of one-time job runs you can create. For more questions, please contact support. (23.279s)❌ test_complete_installation_with_custom_folder: databricks.sdk.errors.platform.BadRequest: The quota for the number of jobs has been reached. The current quota is 12000. This quota is only applied to jobs created through the UI or through the /jobs/create endpoint, which are displayed in the Jobs UI. If this limit is hit, it is very likely some user programmatically created many jobs that only need to run once. In this case, we recommend to delete those jobs and to submit one-time job runs instead using the /jobs/runs/submit endpoint. There is no quota on the number of one-time job runs you can create. For more questions, please contact support. (13.654s)❌ test_profiler_workflow_with_ai_rules_generation_and_model_api_keys_as_secrets: databricks.sdk.errors.platform.BadRequest: The quota for the number of jobs has been reached. The current quota is 12000. This quota is only applied to jobs created through the UI or through the /jobs/create endpoint, which are displayed in the Jobs UI. If this limit is hit, it is very likely some user programmatically created many jobs that only need to run once. In this case, we recommend to delete those jobs and to submit one-time job runs instead using the /jobs/runs/submit endpoint. There is no quota on the number of one-time job runs you can create. For more questions, please contact support. (10.968s)❌ test_profiler_workflow_class_serverless: databricks.sdk.errors.platform.BadRequest: The quota for the number of jobs has been reached. The current quota is 12000. This quota is only applied to jobs created through the UI or through the /jobs/create endpoint, which are displayed in the Jobs UI. If this limit is hit, it is very likely some user programmatically created many jobs that only need to run once. In this case, we recommend to delete those jobs and to submit one-time job runs instead using the /jobs/runs/submit endpoint. There is no quota on the number of one-time job runs you can create. For more questions, please contact support. (8.193s)Flaky tests:
Running from acceptance #5146 |
Contributor
|
✅ 194/194 passed, 4 flaky, 2 skipped, 7h0m13s total Flaky tests:
Running from anomaly #1260 |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1322 +/- ##
===========================================
- Coverage 92.60% 74.66% -17.95%
===========================================
Files 103 103
Lines 10378 10410 +32
===========================================
- Hits 9611 7773 -1838
- Misses 767 2637 +1870
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The Pydantic migration (#1259) narrowed DQRule/DQForEachColRule.user_metadata to dict[str, str], regressing checks that carry non-string metadata values (e.g. {"confidence": 0.95, "enabled": true}). Pre-migration dataclasses never enforced the value type, and the table storage backends persist these values faithfully, so validation rejected them on load with InvalidCheckError. Relax both fields to dict[str, Any] | None to restore released behaviour. Fixes the failing unit test test_lakebase_normalize_checks_preserves_non_string_user_metadata and integration test test_save_and_load_checks_from_table_preserves_user_metadata_value_types.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated sync from fork PR for CI testing.
Original PR: #1319
All tests, including unit and integration tests run on this PR (they are skipped for fork PRs).