jsonschema 4.18 incompatible with asdf

[braingram]

Changes to ref resolution introduced here:
python-jsonschema/jsonschema#1049

are incompatible with asdf uses of jsonschema because of deprecations and apparently further changes.

Ignoring the test failures due to warnings from deprecation of the following:

• jsonschema.exceptions.RefResolutionError
• jsonschema.validators.RefResolver
• Validator.resolver
  (by filtering the deprecation warnings) still results in numerous test failures.

FAILED asdf/_tests/test_resource.py::test_jsonschema_resource_mapping - FileNotFoundError: [Errno 2] No such file or directory: '/Users/bgraham/projects/230314_jsonschema_ref_resolver/jsonschema/jsonschema/schemas/draft4.json'
FAILED asdf/_tests/test_schema.py::test_load_schema - jsonschema.exceptions._RefResolutionError: Unresolvable JSON pointer: 'definitions/stringArray'
FAILED asdf/_tests/test_schema.py::test_load_schema_with_full_tag - jsonschema.exceptions._RefResolutionError: Unresolvable JSON pointer: 'definitions/stringArray'
FAILED asdf/_tests/test_api.py::test_overwrite - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/transform/affine-1.3.0': 'http://stsci.edu/schemas/asdf/transform/affine-1.3.0'
FAILED asdf/_tests/test_schema.py::test_load_schema_with_tag_address - jsonschema.exceptions._RefResolutionError: Unresolvable JSON pointer: 'definitions/stringArray'
FAILED asdf/_tests/test_schema.py::test_load_schema_with_file_url - jsonschema.exceptions._RefResolutionError: Unresolvable JSON pointer: 'definitions/stringArray'
FAILED asdf/_tests/test_schema.py::test_load_schema_with_asdf_uri_scheme - jsonschema.exceptions._RefResolutionError: Unresolvable JSON pointer: 'definitions/local_bar'
FAILED asdf/_tests/test_schema.py::test_nested_array_yaml - jsonschema.exceptions._RefResolutionError: Unresolvable JSON pointer: 'definitions/simpleTypes'
FAILED asdf/_tests/test_types.py::test_custom_reference_cycle - RecursionError: maximum recursion depth exceeded while calling a Python object
FAILED asdf/_tests/test_schema.py::test_invalid_schema - jsonschema.exceptions._RefResolutionError: Unresolvable JSON pointer: 'definitions/simpleTypes'
FAILED asdf/_tests/test_schema.py::test_default_check_in_schema - jsonschema.exceptions._RefResolutionError: Unresolvable JSON pointer: 'definitions/simpleTypes'
FAILED asdf/_tests/test_schema.py::test_check_complex_default - jsonschema.exceptions._RefResolutionError: Unresolvable JSON pointer: 'definitions/simpleTypes'
FAILED asdf/_tests/test_schema.py::test_tag_reference_validation - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:nowhere.org:custom/tag_reference-1.0.0': 'http://nowhere.org/schemas/custom/tag_reference-1.0.0'
FAILED asdf/_tests/test_schema.py::test_foreign_tag_reference_validation - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:nowhere.org:custom/foreign_tag_reference-1.0.0': 'http://nowhere.org/schemas/custom/foreign_tag_reference-1.0.0'
FAILED asdf/_tests/commands/tests/test_diff.py::test_diff_command - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/wcs/celestial_frame-1.1.0': 'http://stsci.edu/schemas/asdf/wcs/celestial_frame-1.1.0'
FAILED asdf/_tests/commands/tests/test_diff.py::test_diff - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/wcs/celestial_frame-1.1.0': 'http://stsci.edu/schemas/asdf/wcs/celestial_frame-1.1.0'
FAILED asdf/_tests/commands/tests/test_diff.py::test_diff_minimal - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/wcs/celestial_frame-1.1.0': 'http://stsci.edu/schemas/asdf/wcs/celestial_frame-1.1.0'
FAILED asdf/_tests/commands/tests/test_diff.py::test_diff_ignore[frames_ignore_asdf_library.diff-ignore0] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/wcs/celestial_frame-1.1.0': 'http://stsci.edu/schemas/asdf/wcs/celestial_frame-1.1.0'
FAILED asdf/_tests/commands/tests/test_diff.py::test_diff_ignore[frames_ignore_reference_frame.diff-ignore1] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/wcs/celestial_frame-1.1.0': 'http://stsci.edu/schemas/asdf/wcs/celestial_frame-1.1.0'
FAILED asdf/_tests/commands/tests/test_diff.py::test_diff_ignore[frames_ignore_both.diff-ignore2] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/wcs/celestial_frame-1.1.0': 'http://stsci.edu/schemas/asdf/wcs/celestial_frame-1.1.0'
FAILED asdf/_tests/core/tests/test_integration.py::test_get_resource_mappings[http://json-schema.org/draft-04/schema] - FileNotFoundError: [Errno 2] No such file or directory: '/Users/bgraham/projects/230314_jsonschema_ref_resolver/jsonschema/jsonschema/schemas/draft4.json'
FAILED asdf/_tests/commands/tests/test_info.py::test_info_command - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/wcs/celestial_frame-1.1.0': 'http://stsci.edu/schemas/asdf/wcs/celestial_frame-1.1.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[266712839015571394777949972451543073253-+] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[710317309215200623862088544052-+] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[162177539112100312349055553100577848672-+] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[162177539112100312349055553100577848672--] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[7106521602475165645-+] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[16422101724900707500-+] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[452825560189989708422940210859154079599013498570713706563199-+] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[710317309215200623862088544052--] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[452825560189989708422940210859154079599013498570713706563199--] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_storage[False] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[7106521602475165645--] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_storage_duplication - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[266712839015571394777949972451543073253--] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_storage[True] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[16422101724900707500--] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_ndarray.py::test_array_inline_threshold_recursive - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/transform/affine-1.3.0': 'http://stsci.edu/schemas/asdf/transform/affine-1.3.0'
FAILED asdf/tags/core/integer.py::asdf.tags.core.integer.IntegerType

[braingram]

Some of the errors above are due to asdf expecting jsonschema to provide draft schemas.

asdf/asdf/resource.py

Line 184 in d33ca09

return pkgutil.get_data("jsonschema", f"schemas/{filename}")

The above mentioned PR removed these schemas. jsonschema instead pulls the draft schemas in from the following package:
https://github.com/python-jsonschema/jsonschema-specifications/

Even after fixing that path issue, errors are still seen:

FAILED asdf/_tests/test_schema.py::test_load_schema - jsonschema.exceptions._RefResolutionError: Unresolvable JSON pointer: 'definitions/stringArray'
FAILED asdf/_tests/test_schema.py::test_load_schema_with_full_tag - jsonschema.exceptions._RefResolutionError: Unresolvable JSON pointer: 'definitions/stringArray'
FAILED asdf/_tests/test_api.py::test_overwrite - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/transform/affine-1.3.0': 'http://stsci.edu/schemas/asdf/transform/affine-1.3.0'
FAILED asdf/_tests/test_schema.py::test_load_schema_with_tag_address - jsonschema.exceptions._RefResolutionError: Unresolvable JSON pointer: 'definitions/stringArray'
FAILED asdf/_tests/test_schema.py::test_load_schema_with_file_url - jsonschema.exceptions._RefResolutionError: Unresolvable JSON pointer: 'definitions/stringArray'
FAILED asdf/_tests/test_schema.py::test_load_schema_with_asdf_uri_scheme - jsonschema.exceptions._RefResolutionError: Unresolvable JSON pointer: 'definitions/local_bar'
FAILED asdf/_tests/test_types.py::test_custom_reference_cycle - RecursionError: maximum recursion depth exceeded while calling a Python object
FAILED asdf/_tests/test_schema.py::test_invalid_schema - jsonschema.exceptions._RefResolutionError: Unresolvable JSON pointer: 'definitions/simpleTypes'
FAILED asdf/_tests/test_schema.py::test_default_check_in_schema - jsonschema.exceptions._RefResolutionError: Unresolvable JSON pointer: 'definitions/simpleTypes'
FAILED asdf/_tests/test_schema.py::test_check_complex_default - jsonschema.exceptions._RefResolutionError: Unresolvable JSON pointer: 'definitions/simpleTypes'
FAILED asdf/_tests/test_schema.py::test_nested_array_yaml - jsonschema.exceptions._RefResolutionError: Unresolvable JSON pointer: 'definitions/simpleTypes'
FAILED asdf/_tests/test_schema.py::test_tag_reference_validation - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:nowhere.org:custom/tag_reference-1.0.0': 'http://nowhere.org/schemas/custom/tag_reference-1.0.0'
FAILED asdf/_tests/test_schema.py::test_foreign_tag_reference_validation - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:nowhere.org:custom/foreign_tag_reference-1.0.0': 'http://nowhere.org/schemas/custom/foreign_tag_reference-1.0.0'
FAILED asdf/_tests/commands/tests/test_diff.py::test_diff_ignore[frames_ignore_asdf_library.diff-ignore0] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/wcs/celestial_frame-1.1.0': 'http://stsci.edu/schemas/asdf/wcs/celestial_frame-1.1.0'
FAILED asdf/_tests/commands/tests/test_diff.py::test_diff_ignore[frames_ignore_reference_frame.diff-ignore1] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/wcs/celestial_frame-1.1.0': 'http://stsci.edu/schemas/asdf/wcs/celestial_frame-1.1.0'
FAILED asdf/_tests/commands/tests/test_diff.py::test_diff_ignore[frames_ignore_both.diff-ignore2] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/wcs/celestial_frame-1.1.0': 'http://stsci.edu/schemas/asdf/wcs/celestial_frame-1.1.0'
FAILED asdf/_tests/commands/tests/test_diff.py::test_diff - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/wcs/celestial_frame-1.1.0': 'http://stsci.edu/schemas/asdf/wcs/celestial_frame-1.1.0'
FAILED asdf/_tests/commands/tests/test_diff.py::test_diff_minimal - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/wcs/celestial_frame-1.1.0': 'http://stsci.edu/schemas/asdf/wcs/celestial_frame-1.1.0'
FAILED asdf/_tests/commands/tests/test_diff.py::test_diff_command - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/wcs/celestial_frame-1.1.0': 'http://stsci.edu/schemas/asdf/wcs/celestial_frame-1.1.0'
FAILED asdf/_tests/commands/tests/test_info.py::test_info_command - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/wcs/celestial_frame-1.1.0': 'http://stsci.edu/schemas/asdf/wcs/celestial_frame-1.1.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[16422101724900707500-+] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[266712839015571394777949972451543073253-+] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[7106521602475165645-+] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_storage[False] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[162177539112100312349055553100577848672-+] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[162177539112100312349055553100577848672--] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[710317309215200623862088544052-+] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[452825560189989708422940210859154079599013498570713706563199--] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[452825560189989708422940210859154079599013498570713706563199-+] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[16422101724900707500--] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[7106521602475165645--] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[266712839015571394777949972451543073253--] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_storage[True] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_storage_duplication - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_integer.py::test_integer_value[710317309215200623862088544052--] - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/core/integer-1.0.0': 'http://stsci.edu/schemas/asdf/core/integer-1.0.0'
FAILED asdf/_tests/tags/core/tests/test_ndarray.py::test_array_inline_threshold_recursive - asdf.exceptions.AsdfWarning: Unable to locate schema file for 'tag:stsci.edu:asdf/transform/affine-1.3.0': 'http://stsci.edu/schemas/asdf/transform/affine-1.3.0'
FAILED asdf/tags/core/integer.py::asdf.tags.core.integer.IntegerType

[jdavies-st]

To add urgency to resolving this issue, the latest jupyter install via pip install jupyter is now incompatible with asdf as of yesterday.

asdf 2.15.0 requires jsonschema<4.18,>=4.0.1, but you have jsonschema 4.18.4 which is incompatible.

jupyter-events 0.7.0 requires jsonschema[format-nongpl]>=4.18.0, but you have jsonschema 4.17.3 which is incompatible.

The only way to get around this now is to install jupyter-events==0.6.3 first, then install jupyter and asdf. The pip resolver is not smart enough to do this on its own.

[pllim]

Re: #1485 (comment)

Oh no... this might affect Jdaviz (in addition to James Davies). cc @kecnry

[jemorrison]

I do those steps and get a strange version of a Jupyter notebook that does not allow me to add or delete cells. It just allows me to run an existing notebook

[braingram]

@jdavies-st thanks for reporting the work around (it was helpful in getting the issue mostly resolved with @jemorrison).

This should be addressed by #1591
which was backported and released in 2.15.1.
I just tested and jupyterlab installs without issue with asdf 2.15.1.
Please feel free to comment we can reopen this if there are more issues.

[pllim]

#1591 has +99,200 −35 😱

Also, what does this mean? asdf will just bundle an old version of jsonschema forever?

[braingram]

For asdf 2.15.1, jsonschema 4.17.3 is bundled internally (not part of the public api). The docs have some more details:
https://asdf.readthedocs.io/en/latest/asdf/whats_new.html#jsonschema
Hopefully this is temporary and we can go back to using jsonschema or another library once one supports all the features necessary for asdf.