Skip to content

Added case-insensitive comparison support to is_in_list and is_not_null_and_is_in_list #673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mwojtyczka merged 12 commits into from
Nov 11, 2025
Merged

Added case-insensitive comparison support to is_in_list and is_not_null_and_is_in_list #673

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
b2d19b0
Added case-insensitive comparison support to is_in_list and is_not_nu…
AdityaMandiwal Oct 31, 2025
27e3e3a
Merge branch 'main' into feature_638
mwojtyczka Oct 31, 2025
b24e26a
fixed documentation and improved integration tests
AdityaMandiwal Nov 3, 2025
f70eb14
Fix tests
ghanse Nov 3, 2025
8215377
Merge remote-tracking branch 'origin/feature_638' into feature_638
ghanse Nov 3, 2025
851f6fd
Fix tests
ghanse Nov 3, 2025
cee3812
Added lowercase checks for array type columns and fixed documentation
AdityaMandiwal Nov 5, 2025
1dc87a3
Added additional tests
AdityaMandiwal Nov 6, 2025
4f6a43d
refactor, fmt
mwojtyczka Nov 11, 2025
e09bb56
Merge branch 'main' into feature_638
mwojtyczka Nov 11, 2025
dca2c82
added integration tests
mwojtyczka Nov 11, 2025
8c446a8
Merge remote-tracking branch 'origin/feature_638' into feature_638
mwojtyczka Nov 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/dqx/docs/reference/quality_checks.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@ You can also define your own custom checks in Python (see [Creating custom check
| `is_not_null` | Checks whether the values in the input column are not null. | `column`: column to check (can be a string column name or a column expression) |
| `is_not_empty` | Checks whether the values in the input column are not empty (but may be null). | `column`: column to check (can be a string column name or a column expression) |
| `is_not_null_and_not_empty` | Checks whether the values in the input column are not null and not empty. | `column`: column to check (can be a string column name or a column expression); `trim_strings`: optional boolean flag to trim spaces from strings |
| `is_in_list` | Checks whether the values in the input column are present in the list of allowed values (null values are allowed). This check is not suited for large lists of allowed values. In such cases, it’s recommended to use the `foreign_key` dataset-level check instead. | `column`: column to check (can be a string column name or a column expression); `allowed`: list of allowed values |
| `is_not_null_and_is_in_list` | Checks whether the values in the input column are not null and present in the list of allowed values. This check is not suited for large lists of allowed values. In such cases, it’s recommended to use the `foreign_key` dataset-level check instead. | `column`: column to check (can be a string column name or a column expression); `allowed`: list of allowed values |
| `is_in_list` | Checks whether the values in the input column are present in the list of allowed values (null values are allowed). This check is not suited for large lists of allowed values. In such cases, it’s recommended to use the `foreign_key` dataset-level check instead. This check is not suited for `MapType` or `StructType` columns. | `column`: column to check (can be a string column name or a column expression); `allowed`: list of allowed values; `case_sensitive`: optional boolean flag for case-sensitive comparison (default: True) |
| `is_not_null_and_is_in_list` | Checks whether the values in the input column are not null and present in the list of allowed values. This check is not suited for large lists of allowed values. In such cases, it’s recommended to use the `foreign_key` dataset-level check instead. This check is not suited for `MapType` or `StructType` columns. | `column`: column to check (can be a string column name or a column expression); `allowed`: list of allowed values; `case_sensitive`: optional boolean flag for case-sensitive comparison (default: True) |
| `is_not_null_and_not_empty_array` | Checks whether the values in the array input column are not null and not empty. | `column`: column to check (can be a string column name or a column expression) |
| `is_in_range` | Checks whether the values in the input column are in the provided range (inclusive of both boundaries). | `column`: column to check (can be a string column name or a column expression); `min_limit`: min limit as number, date, timestamp, column name or sql expression; `max_limit`: max limit as number, date, timestamp, column name or sql expression |
| `is_not_in_range` | Checks whether the values in the input column are outside the provided range (inclusive of both boundaries). | `column`: column to check (can be a string column name or a column expression); `min_limit`: min limit as number, date, timestamp, column name or sql expression; `max_limit`: max limit as number, date, timestamp, column name or sql expression |
Expand Down
48 changes: 37 additions & 11 deletions src/databricks/labs/dqx/check_funcs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
is_sql_query_safe,
normalize_col_str,
get_columns_as_strings,
to_lowercase,
)
from databricks.labs.dqx.errors import MissingParameterError, InvalidParameterError, UnsafeSqlQueryError

Expand Down Expand Up @@ -129,12 +130,15 @@ def is_not_null(column: str | Column) -> Column:


@register_rule("row")
def is_not_null_and_is_in_list(column: str | Column, allowed: list) -> Column:
def is_not_null_and_is_in_list(column: str | Column, allowed: list, case_sensitive: bool = True) -> Column:
"""Checks whether the values in the input column are not null and present in the list of allowed values.
Can optionally perform a case-insensitive comparison.
Comment thread
ghanse marked this conversation as resolved.
This check is not suited for `MapType` or `StructType` columns.

Args:
column: column to check; can be a string column name or a column expression
allowed: list of allowed values (actual values or Column objects)
case_sensitive: whether to perform a case-sensitive comparison (default: True)

Returns:
Column object for condition
Expand All @@ -145,38 +149,50 @@ def is_not_null_and_is_in_list(column: str | Column, allowed: list) -> Column:
"""
if allowed is None:
raise MissingParameterError("allowed list is not provided.")

if not isinstance(allowed, list):
raise InvalidParameterError(f"allowed parameter must be a list, got {str(type(allowed))} instead.")

if not allowed:
raise InvalidParameterError("allowed list must not be empty.")

allowed_cols = [item if isinstance(item, Column) else F.lit(item) for item in allowed]
col_str_norm, col_expr_str, col_expr = _get_normalized_column_and_expr(column)
condition = col_expr.isNull() | ~col_expr.isin(*allowed_cols)
Comment thread
mwojtyczka marked this conversation as resolved.

# Apply case-insensitive transformation if needed
if not case_sensitive:
has_arrays = any(isinstance(item, (list, tuple)) for item in allowed if not isinstance(item, Column))
Comment thread
mwojtyczka marked this conversation as resolved.
Comment thread
mwojtyczka marked this conversation as resolved.
col_expr_compare = to_lowercase(col_expr, is_array=has_arrays)
allowed_cols_compare = [
to_lowercase(c, is_array=isinstance(allowed[i], (list, tuple))) for i, c in enumerate(allowed_cols)
]
else:
col_expr_compare, allowed_cols_compare = col_expr, allowed_cols

condition = col_expr.isNull() | ~col_expr_compare.isin(*allowed_cols_compare)

return make_condition(
condition,
F.concat_ws(
"",
F.lit("Value '"),
F.when(col_expr.isNull(), F.lit("null")).otherwise(col_expr.cast("string")),
F.lit(f"' in Column '{col_expr_str}' is null or not in the allowed list: ["),
F.concat_ws(", ", *allowed_cols),
F.concat_ws(", ", *[c.cast("string") for c in allowed_cols]),
F.lit("]"),
),
f"{col_str_norm}_is_null_or_is_not_in_the_list",
)


@register_rule("row")
def is_in_list(column: str | Column, allowed: list) -> Column:
def is_in_list(column: str | Column, allowed: list, case_sensitive: bool = True) -> Column:
"""Checks whether the values in the input column are present in the list of allowed values
(null values are allowed).
(null values are allowed). Can optionally perform a case-insensitive comparison.
Comment thread
ghanse marked this conversation as resolved.
This check is not suited for `MapType` or `StructType` columns.

Args:
column: column to check; can be a string column name or a column expression
allowed: list of allowed values (actual values or Column objects)
case_sensitive: whether to perform a case-sensitive comparison (default: True)

Returns:
Column object for condition
Expand All @@ -187,24 +203,34 @@ def is_in_list(column: str | Column, allowed: list) -> Column:
"""
if allowed is None:
raise MissingParameterError("allowed list is not provided.")

if not isinstance(allowed, list):
raise InvalidParameterError(f"allowed parameter must be a list, got {str(type(allowed))} instead.")

if not allowed:
raise InvalidParameterError("allowed list must not be empty.")

allowed_cols = [item if isinstance(item, Column) else F.lit(item) for item in allowed]
col_str_norm, col_expr_str, col_expr = _get_normalized_column_and_expr(column)
condition = ~col_expr.isin(*allowed_cols)

# Apply case-insensitive transformation if needed
if not case_sensitive:
has_arrays = any(isinstance(item, (list, tuple)) for item in allowed if not isinstance(item, Column))
col_expr_compare = to_lowercase(col_expr, is_array=has_arrays)
allowed_cols_compare = [
to_lowercase(c, is_array=isinstance(allowed[i], (list, tuple))) for i, c in enumerate(allowed_cols)
]
else:
col_expr_compare, allowed_cols_compare = col_expr, allowed_cols

condition = ~col_expr_compare.isin(*allowed_cols_compare)

return make_condition(
condition,
F.concat_ws(
"",
F.lit("Value '"),
F.when(col_expr.isNull(), F.lit("null")).otherwise(col_expr.cast("string")),
F.lit(f"' in Column '{col_expr_str}' is not in the allowed list: ["),
F.concat_ws(", ", *allowed_cols),
F.concat_ws(", ", *[c.cast("string") for c in allowed_cols]),
F.lit("]"),
),
f"{col_str_norm}_is_not_in_the_list",
Expand Down
17 changes: 16 additions & 1 deletion src/databricks/labs/dqx/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,12 @@
except ImportError:
ConnectColumn = None # type: ignore

import pyspark.sql.functions as F
from databricks.sdk import WorkspaceClient
from databricks.labs.blueprint.limiter import rate_limited
from databricks.labs.dqx.errors import InvalidParameterError
from databricks.sdk.errors import NotFound


logger = logging.getLogger(__name__)


Expand Down Expand Up @@ -450,3 +450,18 @@ def _match_table_patterns(table: str, patterns: list[str]) -> bool:
bool: True if the table name matches any of the patterns, False otherwise.
"""
return any(fnmatch(table, pattern) for pattern in patterns)


def to_lowercase(col_expr: Column, is_array: bool = False) -> Column:
"""Converts a column expression to lowercase, handling both scalar and array types.

Args:
col_expr: Column expression to convert
is_array: Whether the column contains array values

Returns:
Column expression with lowercase transformation applied
"""
if is_array:
return F.transform(col_expr, F.lower)
return F.lower(col_expr)
Loading
Loading