Added tolerance to equality and non-equality check functions

[dwanneruchi]

Changes

Linked issues

Resolves #1004

Tests

• manually tested
• added unit tests
• added integration tests
• added end-to-end tests
• added performance tests

Manul Test Examples:

Note: Below is the proper testing version:

%pip install git+https://github.com/dwanneruchi/dqx.git@aggr-rounding-issue-1004

dbutils.library.restartPython()

import pyspark.sql.functions as F

from databricks.labs.dqx.engine import DQEngine
from databricks.labs.dqx.check_funcs import get_normalized_column_and_expr, get_limit_expr, _match_values_with_tolerance, make_condition, is_equal_to, is_not_equal_to
from databricks.labs.dqx.rule import DQRowRule
from databricks.sdk import WorkspaceClient

dq_engine = DQEngine(WorkspaceClient())

is_equal_to:

Presently we will see a violation from C & D since they don't equal 200

data = [
    ("A", 200),
    ("B",200),
    ("C", 199),  # VIOLATION!
    ("D", 198),  # VIOLATION!
]

df = spark.createDataFrame(
    data,
    schema=[
        "id",
        "sales",
    ],
)

# List of checks
dqx_check = [
  DQRowRule(
      name="equal_200",
      criticality="warn",
      check_func=is_equal_to,
      column="sales",
      check_func_kwargs = {"value": 200}
  ),
]

valid_df, quarantine_df = dq_engine.apply_checks_and_split(df, dqx_check)

quarantine_df.display()

But we can use rel_tolerance and abs_tolerance to resolve:

# List of checks
dqx_check = [
  DQRowRule(
      name="equal_within_1_to_200",
      criticality="error",
      check_func=is_equal_to,
      column="sales",
      check_func_kwargs = {"value": 200, "abs_tolerance": 1.0, "rel_tolerance": 0.01 }
  ),
]

valid_df, quarantine_df = dq_engine.apply_checks_and_split(df, dqx_check)

quarantine_df.display()

This yields 0 violations, as expected:

• C -> is within 1 (abs_tolerance) and within 1% (rel_tolerance)
• D -> is NOT within 1 (abs_tolerance) but is exactly within 1% (rel_tolerance) since max(a,b) == 200

is_not_equal_to:

# List of checks - C will raise warning without tolerance
data = [
    ("A", 200.0),
    ("B",200.0),
    ("C", 199.0), 
]

df = spark.createDataFrame(
    data,
    schema=[
        "id",
        "sales",
    ],
)

# List of checks
dqx_check = [
  DQRowRule(
      name="nequal_200",
      criticality="warn",
      check_func=is_not_equal_to,
      column="sales",
      check_func_kwargs = {"value": 199}
  ),
]

valid_df, quarantine_df = dq_engine.apply_checks_and_split(df, dqx_check)

And adding abs_tolerance will cause all to raise warnings:

# List of checks
dqx_check = [
  DQRowRule(
      name="nequal_200",
      criticality="warn",
      check_func=is_not_equal_to,
      column="sales",
      check_func_kwargs = {"value": 199, "abs_tolerance": 1.0}
  ),
]

valid_df, quarantine_df = dq_engine.apply_checks_and_split(df, dqx_check)

is_aggr_equal

We see id2 == "B" will violate due to avg == 125.5

from databricks.labs.dqx.check_funcs import is_aggr_equal
from databricks.labs.dqx.rule import DQDatasetRule

data = [
    ("A", "A", 100),
    ("B", "A", 150),
    ("E", "A", None),
    ("C", "B", 100), # VIOLATION
    ("D", "B", 151), # VIOLATION
]

df = spark.createDataFrame(
    data,
    schema=[
        "id",
        "id2",
        "sales",
    ],
)

# List of checks
dqx_check = [
  DQDatasetRule(  
    name="agg_avg",
    criticality="warn",
    check_func=is_aggr_equal,
    column = "sales",
    check_func_kwargs={
      "group_by": ['id2'],
      "aggr_type": "avg",
      "limit": 125}
  ),
]

valid_df, quarantine_df = dq_engine.apply_checks_and_split(df, dqx_check)
quarantine_df.display()

Adding an abs_tolerance removes the violation:

• Note that null is handled correctly

# List of checks
dqx_check = [
  DQDatasetRule(  
    name="agg_avg",
    criticality="warn",
    check_func=is_aggr_equal,
    column = "sales",
    check_func_kwargs={
      "group_by": ['id2'],
      "aggr_type": "avg",
      "limit": 125,
      "abs_tolerance": 0.5
    }
  ),
]

valid_df, quarantine_df = dq_engine.apply_checks_and_split(df, dqx_check)
quarantine_df.display()

is_aggr_not_equal

Same example, except the negative - now id2 == A` will fail:

# List of checks
dqx_check = [
  DQDatasetRule(  
    name="agg_avg_neq",
    criticality="warn",
    check_func=is_aggr_not_equal,
    column = "sales",
    check_func_kwargs={
      "group_by": ['id2'],
      "aggr_type": "avg",
      "limit": 125}
  ),
]

valid_df, quarantine_df = dq_engine.apply_checks_and_split(df, dqx_check)

Adding abs_tolerance ==1.0 will cause all to be violations as expected:

# List of checks
dqx_check = [
  DQDatasetRule(  
    name="agg_avg_neq",
    criticality="warn",
    check_func=is_aggr_not_equal,
    column = "sales",
    check_func_kwargs={
      "group_by": ['id2'],
      "aggr_type": "avg",
      "limit": 125,
      "abs_tolerance": 1.0}
  ),
]

valid_df, quarantine_df = dq_engine.apply_checks_and_split(df, dqx_check)

quarantine_df.display()

[Copilot]

Pull request overview

This PR adds tolerance parameters (abs_tolerance and rel_tolerance) to equality comparison functions, enabling flexible numeric comparisons that account for floating-point precision issues and scale variations. This addresses issue #1004 related to aggregation rounding.

Changes:

• Added optional abs_tolerance and rel_tolerance parameters to is_equal_to, is_not_equal_to, is_aggr_equal, and is_aggr_not_equal functions
• Implemented _match_values_with_tolerance helper function for tolerance-based numeric comparisons
• Created _build_aggregate_check_metadata helper to extract metadata building logic from _is_aggr_compare

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

All commits in PR should be signed ('git commit -S ...'). See https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits

[dwanneruchi]

Still going to add tests in tests/integration/test_dataset_checks.py for is_equal_to and is_not_equal_to with new tolerance args, then this should be ready for a review.

[dwanneruchi]

A few notes for reviewers:

• Per @mwojtyczka (comment: [FEATURE]: Add optional rounding for _aggr check_funcs #1004 (comment)), I only added tests to the integration
• I got my local set-up going but some of the make integration failed for reasons that might be connected to the cluster I have set-up, however I did the following and both passed:

hatch run pytest tests/integration/test_row_checks.py -v -s
hatch run pytest tests/integration/test_dataset_checks.py -v -s

Otherwise ready for review, open to any feedback as this is my first contribution to dqx!

[davidwanner-8451]

This should be ready for review, anything pausing that @mwojtyczka and / or @ghanse ?

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mwojtyczka]

Implementation LGTM. I still need to check tests.

[mwojtyczka]

LGTM, added missing docs