Skip check evaluation if columns or filter cannot be resolved in the input DataFrame#609
Merged
Conversation
mwojtyczka
changed the title
Contributor
|
✅ 400/400 passed, 6 flaky, 1 skipped, 4h31m51s total Flaky tests:
Running from acceptance #2794 |
alexott
reviewed
Oct 14, 2025
Comment on lines
+75
to
+79
| if self.check.column is not None and self._is_invalid_column(self.check.column): | ||
| invalid_cols.append(get_column_name_or_alias(self.check.column)) | ||
|
|
||
| # Validate multiple columns | ||
| if self.check.columns is not None: |
Contributor
There was a problem hiding this comment.
Should we have
if
...
elif
....
?
We shouldn't have both check.column and check.columns at the same time?
| """ | ||
| Returns a boolean indicating whether any of the specified check columns are invalid in the input DataFrame. | ||
| """ | ||
| return len(self.invalid_columns) > 0 |
Contributor
There was a problem hiding this comment.
could be expressed as return bool(self.invalid_columns)...
Contributor
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
ghanse
approved these changes
Oct 14, 2025
ghanse
left a comment
ghanse
left a comment
Collaborator
There was a problem hiding this comment.
LGTM! Left a comment on the docs.
alexott
approved these changes
Oct 15, 2025
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Currently, applying a check to a column or list of columns that are invalid (not present in the input DataFrame or containing an unresolvable expression) would cause the check to fail. With this change, the
column,columns, andfilterfields are validated against the input DataFrame. If any of these fields are invalid, the check is skipped, and the results include a failure message identifying the invalid fields. This ensures that the data quality process can continue gracefully, without failing due to missing or invalid fields.Linked issues
Resolves #608
Tests