Support ETag for cache validation when Last-Modified is unavailable

[altendky]

Summary

When a remote schema endpoint provides an ETag header but no Last-Modified header, check-jsonschema currently treats the cache as always-fresh, leading to stale schemas being served indefinitely.

Current Behavior

The cache hit logic in cachedownloader.py relies solely on Last-Modified:

def _cache_hit(cachefile: str, response: requests.Response) -> bool:
    if not os.path.exists(cachefile):
        return False
    local_mtime = os.path.getmtime(cachefile)
    remote_mtime = _lastmod_from_response(response)
    return local_mtime >= remote_mtime

When Last-Modified is missing, _lastmod_from_response() returns 0.0:

def _lastmod_from_response(response: requests.Response) -> float:
    try:
        return calendar.timegm(
            time.strptime(response.headers["last-modified"], _LASTMOD_FMT)
        )
    except (OverflowError, ValueError, LookupError):
        return 0.0

Since any cached file's mtime is >= 0.0, the cache never invalidates.

Real-World Impact

Mergify's schema endpoint (https://docs.mergify.com/mergify-configuration-schema.json) provides ETag but not Last-Modified:

$ curl -sI https://docs.mergify.com/mergify-configuration-schema.json | grep -E '^(cache-control|etag|last-modified):'
cache-control: public, max-age=600, no-transform
etag: "afd19c79c195c2e76f1d37bd12421d88"

When Mergify adds new config options, users get false validation failures until they manually clear ~/.cache/check_jsonschema/.

I've filed Mergifyio/mergify#5161 requesting they add Last-Modified, but this is likely a common pattern for CDN-served content (Cloudflare in their case).

Suggested Enhancement

Support ETag as a fallback when Last-Modified is unavailable:

1. Store the ETag value alongside cached files (e.g., in a .etag sidecar file or a metadata store)
2. On subsequent requests, send If-None-Match: <stored-etag> header
3. If server returns 304 Not Modified, treat as cache hit
4. If server returns 200 with new ETag, update cache and stored ETag

This follows standard HTTP caching semantics where clients should support both Last-Modified/If-Modified-Since and ETag/If-None-Match.

Alternatives Considered

• Require servers to set Last-Modified: Not always under user control, especially for third-party schemas
• Use Cache-Control: max-age: Already present in some responses (e.g., max-age=600), could be used as a TTL, though this would mean re-downloading more frequently than necessary when content hasn't changed

References

• HTTP Conditional Requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Conditional_requests
• Related Mergify issue: Schema endpoint missing Last-Modified header, causing stale cache issues with check-jsonschema Mergifyio/mergify#5161

[sirosen]

Yep! I don't need much convincing on this one. Respecting ETag seems like much better behavior than ignoring it.

I'm a little surprised to learn that CDNs aren't consistently providing Last-Modified, but this is an extremely reasonable feature request.

I prefer checking Last-Modified first because checking it will generally be slightly faster, but if it's missing an ETag is present, that should be checked before giving up.

[altendky]

any interest in switching to https://pypi.org/project/CacheControl/? it's used by pip (https://github.com/pypa/pip/tree/fc9550be97a09d3752b7ee77791418f17c27bb6e/src/pip/_vendor/cachecontrol) and integrates with requests. would allow just offloading all the handling on them. i was planning out a PR for this.

[sirosen]

👍 introducing cachecontrol looks to me like a good change!

I'm not sure what version bounds we should use though. It needs to be pointed at the current cache dir, but I assume it has APIs for that.

Thanks very much for working on this!

[altendky]

it does have some options around backing store control, but i don't think it is likely to be able to be directly compatible with existing caches. certainly that's problematic both from the perspective of every 'user' needing to rebuild their cache and also in terms of wanting to handle removal of the existing cache that would no longer be used. is this a blocker or an annoyance?

[altendky]

sorry, i should clarify. we should be able to tell it what dir. i don't think it will be able to reuse the existing cached content.

[sirosen]

I think it's a mild annoyance that the cache will be effectively discarded, but not a blocker.

check-jsonschema is still 0-ver. It would be a bigger deal if there were a stronger implicit compatibility promise, but "we changed how the cache is arranged, sorry" is, IMO, acceptable right now.

---

Related aside: there was a security issue a couple of years back because I made a silly error in how I was handling the cache. cachecontrol docs are pretty clear about the security concerns that are still present, but it does eliminate some of the potential sources of error.

[altendky]

alrighty, thanks. so that means the main concern around this cache incompatibility aspect is how we can smooth the transition.

• keeping the same directory avoids people needing to change things like any ci caching actions and the paths the process
• are there places (such as a helper github action for check-jsonschema) where we could actively bump the cache version to express this change
• do we want to provide any cache cleaning helpers, automatic or manual

that's just a quick list of what popped into my head