Add built-in methods for PII detection

[ghanse]

Changes

This PR introduces the pii module which users can install to access built-in checks for Personally Identifiable Information (PII). These built-in checks use the Presidio framework.

Installation

Users install PII detection capabilities as an extras module with additional dependencies:

pip install databricks-labs-dqx[pii]

Usage

Users can define checks using a new built-in function, contains_pii.

from databricks.labs.dqx.rule import DQRowRule
from databricks.labs.dqx.pii.pii_detection_funcs import contains_pii

checks = [
    # PII detection check using stanford-deidentifier-base as a named entity recognizer
    DQRowRule(
        name="col_description_contains_pii",
        criticality="error",
        column="description",
        check_func=contains_pii
    ),    
]

Configuration

Users can configure the underlying model by passing parameters to contains_pii. This includes NLP entity recognition configuration which can be passed as a Python dictionary or as an NLPEngineConfig. In the future, we can add NLPEngineConfig values to support different named entity recognizers.

TO-DO:

• Introduce methods to configure the underlying named entity recognizer
• Add tests
• Add documentation
• Update demos to use the built-in check

Linked issues

Resolves #419

Tests

• manually tested
• added unit tests
• added integration tests

[github-actions]

✅ 9/9 passed, 54m20s total

_{Running from acceptance #1730}

[mwojtyczka]

would be useful to have info why we cannot use databricks connect

[mwojtyczka]

LGTM - left minor comment. Will merge after testing.

[mwojtyczka]

func name needs to be updated

[mwojtyczka]

LGTM - will merge after testing