[BUG]: does_not_contain_pii function is not working

[nilaksh92]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

have added custom_check_functions, still it is failing.

function

Error:

Could you please help here.

Expected Behavior

No response

Steps To Reproduce

No response

Cloud

AWS

Operating System

macOS

Relevant log output

[mwojtyczka]

For general info. The need for specifying a custom function has already been fixed and will be available in the next release as per the issue: #998
PR: #1003

[mwojtyczka]

It seems dqx cannot download the spacy model. Is this the complete stacktrace?

DQX automatically installs the built-in entity recognition models at runtime if they are not already available. However, if you see issues with downloading the models, you can try to pre-install them as seen in this notebook: https://github.com/databrickslabs/dqx/blob/v0.12.0/demos/dqx_demo_pii_detection.py

Ensure that pii extras are installed: pip install 'databricks-labs-dqx[pii]', then try to install the model explicitely as in the demo: https://github.com/databrickslabs/dqx/blob/v0.12.0/demos/dqx_demo_pii_detection.py

By default we use spacy small model.

# Installing spaCy's small English model (default for DQX PII checks):
%pip install "en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl"
# Installing spaCy's medium English model:
# %pip install "en_core_web_md @ https://github.com/explosion/spacy-models/releases/download/en_core_web_md-3.8.0/en_core_web_md-3.8.0-py3-none-any.whl"
# Installing spaCy's large English model (not used in this demo):
#%pip install "en_core_web_lg @ https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.8.0/en_core_web_lg-3.8.0-py3-none-any.whl"

[nilaksh92]

i have already installed 'databricks-labs-dqx[pii]

[mwojtyczka]

sounds good, pls also install the default spacy model: %pip install "en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl"
It shouldn't be required, as we download it automatically if it's not present, but maybe something specific in your environment is blocking it.

[nilaksh92]

hi @mwojtyczka we have databricks hosted on private network, not sure if we would be able to install something from git.

[mwojtyczka]

Do you have private PyPi repo? you could host the library there. Alternatively, you could save the spacy wheel model in in Unity Catalog volume or workspace file.

[nilaksh92]

yes, we have private pypi repo hosted on nexus. can you please guide how to host both dqx and dqx[pii] and other required libraries over there?

[mwojtyczka]

Please follow:

• library: https://databrickslabs.github.io/dqx/docs/installation/#installing-dqx-with-company-hosted-pypi-mirror
• if using workflows: https://databrickslabs.github.io/dqx/docs/installation/#installing-dqx-with-company-hosted-pypi-mirror-1

Alternatively, you could just host the spacy model (wheel) in Nexus, and pre-install it. Unfortunately, spacy models are only published in git.

[mwojtyczka]

I'm closing this ticket. If you need further support, feel free to ask here.