Implement referencing.retrieval.to_cached_resource

This is a fairly simple caching decorator (which just delegates to
lru_cache), but it's useful because:

    * Most dynamic retrievers will probably want this
    * It saves a small bit of referencing-specific boilerplate, letting
      users know to return some JSON and the rest is handled (as long as
      their schemas contain $schema as usual)

It also allows for continuing to support use cases that *don't* want
caching (by of course not using this decorator) such as when you *do*
want to dynamically re-retrieve a URI because it may have changed
contents.

Some tweaks may still be necessary here, but it does work for the
initial example.

Refs: python-jsonschema/jsonschema#1065

Author: Julian

docs/api.rst

@@ -18,7 +18,7 @@ API Reference
    :undoc-members:
 
 
-.. autoclass:: referencing._core.T
+.. autoclass:: referencing._core.AnchorOrResource
 
 
 .. autoclass:: referencing._core.Resolver
@@ -51,6 +51,17 @@ referencing.exceptions
    :undoc-members:
 
 
+referencing.retrieval
+^^^^^^^^^^^^^^^^^^^^^
+
+.. automodule:: referencing.retrieval
+   :members:
+   :undoc-members:
+
+
+.. autoclass:: referencing.retrieval._T
+
+
 referencing.typing
 ^^^^^^^^^^^^^^^^^^
 

docs/changes.rst

@@ -2,6 +2,11 @@
 Changelog
 =========
 
+v0.29.0
+-------
+
+* Add ``referencing.retrieval.to_cached_resource``, a simple caching decorator useful when writing a retrieval function turning JSON text into resources without repeatedly hitting the network, filesystem, etc.
+
 v0.28.6
 -------
 

docs/intro.rst

@@ -236,3 +236,31 @@ Here's an example of automatically retrieving external references by downloading
     See :kw:`schema-references` in particular.
 
     `referencing` will of course therefore not do any such thing automatically, and this section generally assumes that you have personally considered the security implications for your own use case.
+
+A common concern in these situations is also to *cache* the resulting resource such that repeated lookups of the same URI do not repeatedly make network calls, or hit the filesystem, etc.
+
+You are of course free to use whatever caching mechanism is convenient (e.g. one specific to ``httpx`` in the above example).
+
+Because of how common it is to retrieve a JSON string and construct a resource from it however, a decorator which specifically does so is also provided called `referencing.retrieval.to_cached_resource`.
+If you use it, note that your retrieval callable should return `str`, not a `Resource`, as the decorator will handle deserializing your response (this is mostly because otherwise, deserialized JSON is generally not hashable).
+
+The above example would be written:
+
+
+.. code:: python
+
+    from referencing import Registry, Resource
+    import httpx
+    import referencing.retrieval
+
+
+    @referencing.retrieval.to_cached_resource()
+    def cached_retrieve_via_httpx(uri):
+        return httpx.get(uri).text
+
+
+    registry = Registry(retrieve=cached_retrieve_via_httpx)
+    resolver = registry.resolver()
+    print(resolver.lookup("https://json-schema.org/draft/2020-12/schema"))
+
+and besides than that it will cache responses and not repeatedly call the retrieval function, it is otherwise functionally equivalent.

docs/spelling-wordlist.txt

@@ -5,13 +5,16 @@ changelog
 deduplication
 dereferenced
 deserialized
+deserializing
 discoverability
 docstrings
 filesystem
+hashable
 implementers
 instantiable
 instantiation
 iterable
+lookups
 metaschemas
 referenceable
 resolvers

referencing/_core.py

@@ -505,16 +505,16 @@ def resolver_with_root(self, resource: Resource[D]) -> Resolver[D]:
 
 
 #: An anchor or resource.
-T = TypeVar("T", AnchorType[Any], Resource[Any])
+AnchorOrResource = TypeVar("AnchorOrResource", AnchorType[Any], Resource[Any])
 
 
 @frozen
-class Retrieved(Generic[D, T]):
+class Retrieved(Generic[D, AnchorOrResource]):
     """
     A value retrieved from a `Registry`.
     """
 
-    value: T
+    value: AnchorOrResource
     registry: Registry[D]
 
 

referencing/retrieval.py

@@ -0,0 +1,83 @@
+"""
+Helpers related to (dynamic) resource retrieval.
+"""
+from __future__ import annotations
+
+from functools import lru_cache
+from typing import Callable, TypeVar
+import json
+
+from referencing import Resource
+from referencing.typing import URI, D, Retrieve
+
+#: A serialized document (e.g. a JSON string)
+_T = TypeVar("_T")
+
+
+def to_cached_resource(
+    cache: Callable[[Retrieve[D]], Retrieve[D]] | None = None,
+    loads: Callable[[_T], D] = json.loads,
+    from_contents: Callable[[D], Resource[D]] = Resource.from_contents,
+) -> Callable[[Callable[[URI], _T]], Retrieve[D]]:
+    """
+    Create a retriever which caches its return values from a simpler callable.
+
+    Takes a function which returns things like serialized JSON (strings) and
+    returns something suitable for passing to `Registry` as a retrieve
+    function.
+
+    This decorator both reduces a small bit of boilerplate for a common case
+    (deserializing JSON from strings and creating `Resource` objects from the
+    result) as well as makes the probable need for caching a bit easier.
+    Retrievers which otherwise do expensive operations (like hitting the
+    network) might otherwise be called repeatedly.
+
+    Examples
+    --------
+
+    .. testcode::
+
+        from referencing import Registry
+        import referencing.retrieval
+
+
+        @referencing.retrieval.to_cached_resource()
+        def retrieve(uri: str):
+            print(f"Retrieved {uri}")
+
+            # Normally, go get some expensive JSON from the network, a file ...
+            return '''
+                {
+                    "$schema": "https://json-schema.org/draft/2020-12/schema",
+                    "foo": "bar"
+                }
+            '''
+
+        one = Registry(retrieve=retrieve).get_or_retrieve("urn:example:foo")
+        print(one.value.contents["foo"])
+
+        # Retrieving the same URI again reuses the same value (and thus doesn't
+        # print another retrieval message here)
+        two = Registry(retrieve=retrieve).get_or_retrieve("urn:example:foo")
+        print(two.value.contents["foo"])
+
+    .. testoutput::
+
+        Retrieved urn:example:foo
+        bar
+        bar
+
+    """
+    if cache is None:
+        cache = lru_cache(maxsize=None)
+
+    def decorator(retrieve: Callable[[URI], _T]):
+        @cache
+        def cached_retrieve(uri: URI):
+            response = retrieve(uri)
+            contents = loads(response)
+            return from_contents(contents)
+
+        return cached_retrieve
+
+    return decorator

referencing/tests/test_core.py

@@ -357,7 +357,7 @@ def test_combine_with_common_retrieve(self):
         two = ID_AND_CHILDREN.create_resource({"foo": "bar"})
         three = ID_AND_CHILDREN.create_resource({"baz": "quux"})
 
-        def retrieve(uri):  #  pragma: no cover
+        def retrieve(uri):  # pragma: no cover
             pass
 
         first = Registry(retrieve=retrieve).with_resource(

referencing/tests/test_retrieval.py

@@ -0,0 +1,106 @@
+from functools import lru_cache
+import json
+
+import pytest
+
+from referencing import Registry, Resource, exceptions
+from referencing.jsonschema import DRAFT202012
+from referencing.retrieval import to_cached_resource
+
+
+class TestToCachedResource:
+    def test_it_caches_retrieved_resources(self):
+        contents = {"$schema": "https://json-schema.org/draft/2020-12/schema"}
+        stack = [json.dumps(contents)]
+
+        @to_cached_resource()
+        def retrieve(uri):
+            return stack.pop()
+
+        registry = Registry(retrieve=retrieve)
+
+        expected = Resource.from_contents(contents)
+
+        got = registry.get_or_retrieve("urn:example:schema")
+        assert got.value == expected
+
+        # And a second time we get the same value.
+        again = registry.get_or_retrieve("urn:example:schema")
+        assert again.value is got.value
+
+    def test_custom_loader(self):
+        contents = {"$schema": "https://json-schema.org/draft/2020-12/schema"}
+        stack = [json.dumps(contents)[::-1]]
+
+        @to_cached_resource(loads=lambda s: json.loads(s[::-1]))
+        def retrieve(uri):
+            return stack.pop()
+
+        registry = Registry(retrieve=retrieve)
+
+        expected = Resource.from_contents(contents)
+
+        got = registry.get_or_retrieve("urn:example:schema")
+        assert got.value == expected
+
+        # And a second time we get the same value.
+        again = registry.get_or_retrieve("urn:example:schema")
+        assert again.value is got.value
+
+    def test_custom_from_contents(self):
+        contents = {}
+        stack = [json.dumps(contents)]
+
+        @to_cached_resource(from_contents=DRAFT202012.create_resource)
+        def retrieve(uri):
+            return stack.pop()
+
+        registry = Registry(retrieve=retrieve)
+
+        expected = DRAFT202012.create_resource(contents)
+
+        got = registry.get_or_retrieve("urn:example:schema")
+        assert got.value == expected
+
+        # And a second time we get the same value.
+        again = registry.get_or_retrieve("urn:example:schema")
+        assert again.value is got.value
+
+    def test_custom_cache(self):
+        schema = {"$schema": "https://json-schema.org/draft/2020-12/schema"}
+        mapping = {
+            "urn:example:1": dict(schema, foo=1),
+            "urn:example:2": dict(schema, foo=2),
+            "urn:example:3": dict(schema, foo=3),
+        }
+
+        resources = {
+            uri: Resource.from_contents(contents)
+            for uri, contents in mapping.items()
+        }
+
+        @to_cached_resource(cache=lru_cache(maxsize=2))
+        def retrieve(uri):
+            return json.dumps(mapping.pop(uri))
+
+        registry = Registry(retrieve=retrieve)
+
+        got = registry.get_or_retrieve("urn:example:1")
+        assert got.value == resources["urn:example:1"]
+        assert registry.get_or_retrieve("urn:example:1").value is got.value
+        assert registry.get_or_retrieve("urn:example:1").value is got.value
+
+        got = registry.get_or_retrieve("urn:example:2")
+        assert got.value == resources["urn:example:2"]
+        assert registry.get_or_retrieve("urn:example:2").value is got.value
+        assert registry.get_or_retrieve("urn:example:2").value is got.value
+
+        # This still succeeds, but evicts the first URI
+        got = registry.get_or_retrieve("urn:example:3")
+        assert got.value == resources["urn:example:3"]
+        assert registry.get_or_retrieve("urn:example:3").value is got.value
+        assert registry.get_or_retrieve("urn:example:3").value is got.value
+
+        # And now this fails (as we popped the value out of `mapping`)
+        with pytest.raises(exceptions.Unretrievable):
+            registry.get_or_retrieve("urn:example:1")