Skip to content

Added case-insensitive comparison support to is_in_list and is_not_null_and_is_in_list #673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mwojtyczka merged 12 commits into from
Nov 11, 2025
Merged

Added case-insensitive comparison support to is_in_list and is_not_null_and_is_in_list #673

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
b2d19b0
Added case-insensitive comparison support to is_in_list and is_not_nu…
AdityaMandiwal Oct 31, 2025
27e3e3a
Merge branch 'main' into feature_638
mwojtyczka Oct 31, 2025
b24e26a
fixed documentation and improved integration tests
AdityaMandiwal Nov 3, 2025
f70eb14
Fix tests
ghanse Nov 3, 2025
8215377
Merge remote-tracking branch 'origin/feature_638' into feature_638
ghanse Nov 3, 2025
851f6fd
Fix tests
ghanse Nov 3, 2025
cee3812
Added lowercase checks for array type columns and fixed documentation
AdityaMandiwal Nov 5, 2025
1dc87a3
Added additional tests
AdityaMandiwal Nov 6, 2025
4f6a43d
refactor, fmt
mwojtyczka Nov 11, 2025
e09bb56
Merge branch 'main' into feature_638
mwojtyczka Nov 11, 2025
dca2c82
added integration tests
mwojtyczka Nov 11, 2025
8c446a8
Merge remote-tracking branch 'origin/feature_638' into feature_638
mwojtyczka Nov 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/dqx/docs/reference/quality_checks.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ You can also define your own custom checks (see [Creating custom checks](#creati
| `is_not_null` | Checks whether the values in the input column are not null. | `column`: column to check (can be a string column name or a column expression) |
| `is_not_empty` | Checks whether the values in the input column are not empty (but may be null). | `column`: column to check (can be a string column name or a column expression) |
| `is_not_null_and_not_empty` | Checks whether the values in the input column are not null and not empty. | `column`: column to check (can be a string column name or a column expression); `trim_strings`: optional boolean flag to trim spaces from strings |
| `is_in_list` | Checks whether the values in the input column are present in the list of allowed values (null values are allowed). This check is not suited for large lists of allowed values. In such cases, it’s recommended to use the `foreign_key` dataset-level check instead. | `column`: column to check (can be a string column name or a column expression); `allowed`: list of allowed values |
| `is_not_null_and_is_in_list` | Checks whether the values in the input column are not null and present in the list of allowed values. This check is not suited for large lists of allowed values. In such cases, it’s recommended to use the `foreign_key` dataset-level check instead. | `column`: column to check (can be a string column name or a column expression); `allowed`: list of allowed values |
| `is_in_list` | Checks whether the values in the input column are present in the list of allowed values (null values are allowed). We can pass an additional case_sensitive parameter as False for a case insensitive check. This check is not suited for large lists of allowed values. In such cases, it’s recommended to use the `foreign_key` dataset-level check instead. | `column`: column to check (can be a string column name or a column expression); `allowed`: list of allowed values |
Comment thread
mwojtyczka marked this conversation as resolved.
Outdated
| `is_not_null_and_is_in_list` | Checks whether the values in the input column are not null and present in the list of allowed values. We can pass an additional case_sensitive parameter as False for a case insensitive check. This check is not suited for large lists of allowed values. In such cases, it’s recommended to use the `foreign_key` dataset-level check instead. | `column`: column to check (can be a string column name or a column expression); `allowed`: list of allowed values |
Comment thread
mwojtyczka marked this conversation as resolved.
Outdated
Comment thread
mwojtyczka marked this conversation as resolved.
Outdated
| `is_not_null_and_not_empty_array` | Checks whether the values in the array input column are not null and not empty. | `column`: column to check (can be a string column name or a column expression) |
| `is_in_range` | Checks whether the values in the input column are in the provided range (inclusive of both boundaries). | `column`: column to check (can be a string column name or a column expression); `min_limit`: min limit as number, date, timestamp, column name or sql expression; `max_limit`: max limit as number, date, timestamp, column name or sql expression |
| `is_not_in_range` | Checks whether the values in the input column are outside the provided range (inclusive of both boundaries). | `column`: column to check (can be a string column name or a column expression); `min_limit`: min limit as number, date, timestamp, column name or sql expression; `max_limit`: max limit as number, date, timestamp, column name or sql expression |
Expand Down
46 changes: 36 additions & 10 deletions src/databricks/labs/dqx/check_funcs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,12 +129,13 @@ def is_not_null(column: str | Column) -> Column:


@register_rule("row")
def is_not_null_and_is_in_list(column: str | Column, allowed: list) -> Column:
def is_not_null_and_is_in_list(column: str | Column, allowed: list, case_sensitive: bool = True) -> Column:
"""Checks whether the values in the input column are not null and present in the list of allowed values.

Can optionally perform a case-insensitive comparison.
Args:
column: column to check; can be a string column name or a column expression
allowed: list of allowed values (actual values or Column objects)
case_sensitive: whether to perform a case-sensitive comparison (default: True)

Returns:
Column object for condition
Expand All @@ -152,31 +153,44 @@ def is_not_null_and_is_in_list(column: str | Column, allowed: list) -> Column:
if not allowed:
raise InvalidParameterError("allowed list must not be empty.")

allowed_cols = [item if isinstance(item, Column) else F.lit(item) for item in allowed]
# Keep original values for display in error message
allowed_cols_display = [item if isinstance(item, Column) else F.lit(item) for item in allowed]

col_str_norm, col_expr_str, col_expr = _get_normalized_column_and_expr(column)
condition = col_expr.isNull() | ~col_expr.isin(*allowed_cols)
Comment thread
mwojtyczka marked this conversation as resolved.

# Create columns for comparison
allowed_cols_compare = allowed_cols_display[:]
col_expr_compare = col_expr

# Case-insensitive normalization
if not case_sensitive:
col_expr_compare = F.lower(col_expr)
allowed_cols_compare = [F.lower(c) for c in allowed_cols_display]

condition = col_expr.isNull() | ~col_expr_compare.isin(*allowed_cols_compare)
return make_condition(
condition,
F.concat_ws(
"",
F.lit("Value '"),
F.when(col_expr.isNull(), F.lit("null")).otherwise(col_expr.cast("string")),
F.lit(f"' in Column '{col_expr_str}' is null or not in the allowed list: ["),
F.concat_ws(", ", *allowed_cols),
F.concat_ws(", ", *allowed_cols_display),
F.lit("]"),
),
f"{col_str_norm}_is_null_or_is_not_in_the_list",
)


@register_rule("row")
def is_in_list(column: str | Column, allowed: list) -> Column:
def is_in_list(column: str | Column, allowed: list, case_sensitive: bool = True) -> Column:
"""Checks whether the values in the input column are present in the list of allowed values
(null values are allowed).
(null values are allowed). Can optionally perform a case-insensitive comparison.
Comment thread
ghanse marked this conversation as resolved.

Args:
column: column to check; can be a string column name or a column expression
allowed: list of allowed values (actual values or Column objects)
case_sensitive: whether to perform a case-sensitive comparison (default: True)

Returns:
Column object for condition
Expand All @@ -194,17 +208,29 @@ def is_in_list(column: str | Column, allowed: list) -> Column:
if not allowed:
raise InvalidParameterError("allowed list must not be empty.")

allowed_cols = [item if isinstance(item, Column) else F.lit(item) for item in allowed]
# Keep original values for display in error message
allowed_cols_display = [item if isinstance(item, Column) else F.lit(item) for item in allowed]
Comment thread
mwojtyczka marked this conversation as resolved.
Outdated

col_str_norm, col_expr_str, col_expr = _get_normalized_column_and_expr(column)
condition = ~col_expr.isin(*allowed_cols)

# Create columns for comparison
allowed_cols_compare = allowed_cols_display[:]
col_expr_compare = col_expr

# Case-insensitive normalization
if not case_sensitive:
col_expr_compare = F.lower(col_expr)
allowed_cols_compare = [F.lower(c) for c in allowed_cols_display]

condition = ~col_expr_compare.isin(*allowed_cols_compare)
return make_condition(
condition,
F.concat_ws(
"",
F.lit("Value '"),
F.when(col_expr.isNull(), F.lit("null")).otherwise(col_expr.cast("string")),
F.lit(f"' in Column '{col_expr_str}' is not in the allowed list: ["),
F.concat_ws(", ", *allowed_cols),
F.concat_ws(", ", *allowed_cols_display),
F.lit("]"),
),
f"{col_str_norm}_is_not_in_the_list",
Expand Down
30 changes: 15 additions & 15 deletions tests/integration/test_row_checks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,10 +166,10 @@ def test_col_is_not_null_and_is_in_list(spark):
)

actual = test_df.select(
is_not_null_and_is_in_list("a", ["str1"]),
Comment thread
mwojtyczka marked this conversation as resolved.
is_not_null_and_is_in_list("b", [F.lit(3)]),
is_not_null_and_is_in_list(F.col("c").getItem("val"), [F.lit("a")]),
is_not_null_and_is_in_list(F.try_element_at("d", F.lit(2)), ["b"]),
is_not_null_and_is_in_list("a", ["STR1"], case_sensitive=False),
is_not_null_and_is_in_list("b", [F.lit(3)], case_sensitive=True),
is_not_null_and_is_in_list(F.col("c").getItem("val"), [F.lit("A")], case_sensitive=False),
is_not_null_and_is_in_list(F.try_element_at("d", F.lit(2)), ["b"], case_sensitive=True),
)

checked_schema = (
Expand All @@ -182,15 +182,15 @@ def test_col_is_not_null_and_is_in_list(spark):
[
[None, "Value '1' in Column 'b' is null or not in the allowed list: [3]", None, None],
[
"Value 'str2' in Column 'a' is null or not in the allowed list: [str1]",
Comment thread
mwojtyczka marked this conversation as resolved.
"Value 'str2' in Column 'a' is null or not in the allowed list: [STR1]",
"Value 'null' in Column 'b' is null or not in the allowed list: [3]",
"Value 'str2' in Column 'UnresolvedExtractValue(c, val)' is null or not in the allowed list: [a]",
"Value 'str2' in Column 'UnresolvedExtractValue(c, val)' is null or not in the allowed list: [A]",
"Value 'a' in Column 'try_element_at(d, 2)' is null or not in the allowed list: [b]",
],
[
"Value ' ' in Column 'a' is null or not in the allowed list: [str1]",
"Value ' ' in Column 'a' is null or not in the allowed list: [STR1]",
None,
"Value ' ' in Column 'UnresolvedExtractValue(c, val)' is null or not in the allowed list: [a]",
"Value ' ' in Column 'UnresolvedExtractValue(c, val)' is null or not in the allowed list: [A]",
"Value ' ' in Column 'try_element_at(d, 2)' is null or not in the allowed list: [b]",
],
],
Expand All @@ -212,10 +212,10 @@ def test_col_is_not_in_list(spark):
)

actual = test_df.select(
is_in_list("a", ["str1"]),
Comment thread
mwojtyczka marked this conversation as resolved.
is_in_list("b", [F.lit(3)]),
is_in_list(F.col("c").getItem("val"), [F.lit("a")]),
is_in_list(F.try_element_at("d", F.lit(2)), ["b"]),
is_in_list("a", ["STR1"], case_sensitive=False),
is_in_list("b", [F.lit(3)], case_sensitive=True),
is_in_list(F.col("c").getItem("val"), [F.lit("A")], case_sensitive=False),
is_in_list(F.try_element_at("d", F.lit(2)), ["b"], case_sensitive=True),
)

checked_schema = (
Expand All @@ -228,13 +228,13 @@ def test_col_is_not_in_list(spark):
[
[None, "Value '1' in Column 'b' is not in the allowed list: [3]", None, None],
[
"Value 'str2' in Column 'a' is not in the allowed list: [str1]",
Comment thread
mwojtyczka marked this conversation as resolved.
"Value 'str2' in Column 'a' is not in the allowed list: [STR1]",
None,
"Value 'str2' in Column 'UnresolvedExtractValue(c, val)' is not in the allowed list: [a]",
"Value 'str2' in Column 'UnresolvedExtractValue(c, val)' is not in the allowed list: [A]",
"Value 'a' in Column 'try_element_at(d, 2)' is not in the allowed list: [b]",
],
[
"Value ' ' in Column 'a' is not in the allowed list: [str1]",
"Value ' ' in Column 'a' is not in the allowed list: [STR1]",
None,
None,
"Value 'a' in Column 'try_element_at(d, 2)' is not in the allowed list: [b]",
Expand Down
Loading