Investigate validation against JSON schema directly from (Py)Spark

[tskir]

Currently, we validate every file generated or ingested by the Data team against the JSON schema. We use the internally developed opentargets_validator package, which in turn depends on the popular jsonschema library.

The issue with this approach is that it limits the output of our pipelines to JSON, where it would potentially be easier for downstream pipelines (ETL) if we used for example Parquet or another similar format.

The goal of this issue is to find a way of validating the data directly inside PySpark, while still using the JSON schema as the single source of truth (at least, for now)

[tskir]

Approaches which don't work

• A promising library for validation would be Cerberus, for which there is a nearly complete tutorial provided for example here. However, the tutorial doesn't work with modern versions of Python/PySpark because the library was last updated 1.5 years ago and is no longer maintained.
• jsonschema, the same library we use inside opentargets_validator, does not work with PySpark UDF parallelisation, because it is being done using pickling. The author of jsonschema said that he will not support pickle, and pickle/cloudpickle does not support decorated functions/classes, which is a known and yet unfixed limitation.

[tskir]

The recommended approach

The fastjsonschema is actively maintained, and is one of the implementations mentioned on the JSON Schema website.

With fastjsonschema, validation is about three times quicker than the existing approach we use, and because it can be pickled, this allows us to use it inside a UDF (or other parallel approach) in PySpark.

The primer on how this library can be used to validate our data is available in this Jupyter notebook: https://github.com/opentargets/validator/blob/master/notebooks/spark-fastjsonschema.ipynb

[tskir]

Decided to close this for now in discussion with @DSuveges and @ireneisdoomed. This will be addressed again/reused shortly.