HTTP 302 Found responses with Cache Headers not being cached

[mklein0]

The particular use case for this problem is the JFrog Artifactory uses 302 Found responses for signed URLs to cloud storage in HTTP requests to their PYPI proxy cache. Python PIP which leverages the CacheControl library in turn fails to cache those responses and downloads the packages everytime. This is a pain for CI processes which download large packages like PyTorch.

Additionally RFC2616 section 10.3.3 claims the 302 Found HTTP response can be cached if the Cache-Control or Expires response headers are present.

[woodruffw]

Hi @mklein0, thanks for opening an issue.

I think the reason we don't cache this is because the various RFCs are ambiguous about temporary redirects vs. permanent ones: RFC 2616 doesn't proscribe caching for these ("cacheable" isn't qualified by SHOULD or similar), while RFC 7231 doesn't mention caching for 302 at all and RFC 7234 mentions only permanent redirects explicitly.

I'm not opposed to changing this per se, but I'd like to better understand why JFrog is using HTTP 302 instead of HTTP 301 for these redirects -- ideally this is something they'd fix on their own end, since other mirror indices don't appear to have this problem.

[woodruffw]

(More broadly, I agree with the appraisal in pypa/pip#13396 (comment) -- it would be good to better understand the intended behavior here, since a signed URL strongly suggests that caching shouldn't be done.)

[mklein0]

As you pointed out the standards are vague on required behavior, so I interpretted it as a suggestion as well. I do not know how common the usage of a private CDN redirect is, so I thought to bring it up and see if there is a best practice.

I will take these two tickets to JFROG and have them advocate for a behavior or have them alter theirs. Thank you.

[nethsix]

@woodruffw I believe JFrog is not using HTTP 301 for redirects likely the 'Location:' of the redirect has same url but different query component, i.e., the signature, e.g., X-Goog-Signature is different each time?

Not caching the initial request that results in the HTTP 302 redirect may not be that critical since the relevant content returned is tiny with just the Location: being important.

Would the concern be more of NOT caching the redirected Location: url which is the larger piece of content being transferred. I did some tests, and is it correct to say that pip performs a cache hit/miss based on the entire url including the query components? This results in the caching NOT happening because each time the X-Goog-Signature changes?

[woodruffw]

I believe JFrog is not using HTTP 301 for redirects likely the 'Location:' of the redirect has same url but different query component, i.e., the signature, e.g., X-Goog-Signature is different each time?

Sorry, could you clarify what you mean here? "A same URL but different query component" is not the same URL, since the query is part of the URL. I'm guessing you mean "same path component," but I want to make sure I understand correctly.

I did some tests, and is it correct to say that pip performs a cache hit/miss based on the entire url including the query components? This results in the caching NOT happening because each time the X-Goog-Signature changes?

I think the pip maintainers would better be able to answer what pip specifically does -- all I know is what the caching middleware does, which pip may or may not tweak, do a superset of, etc.

For CacheControl specifically, we (currently) key off of the query string exactly as it appears in the request via _urlnorm here:

cachecontrol/cachecontrol/controller.py

Lines 65 to 83 in 9af76f7

	@classmethod
	def _urlnorm(cls, uri: str) -> str:
	"""Normalize the URL to create a safe key for the cache"""
	(scheme, authority, path, query, fragment) = parse_uri(uri)
	if not scheme or not authority:
	raise Exception("Only absolute URIs are allowed. uri = %s" % uri)
	scheme = scheme.lower()
	authority = authority.lower()
	if not path:
	path = "/"
	# Could do syntax based normalization of the URI before
	# computing the digest. See Section 6.2.2 of Std 66.
	request_uri = query and "?".join([path, query]) or path
	defrag_uri = scheme + "://" + authority + request_uri
	return defrag_uri

(Based on my read of the RFCs, this is correct enough, and ignoring the query URL would cause overly permissive cache hits.)

[nethsix]

@woodruffw Sorry that I was not clear with what I meant by "A same URL but different query component". I have an AWS cloudfront/s3 pre-signed url on hand so I will use that as an example.

https://d3njxchdk64paq.cloudfront.net/development/2/7/pypi/packages/8.whl?Expires=1752474000&Signature=xMOpa8TJo3OgXQRdI3ASkvvwQ5OVNBhSX1kUQjkJ964CvYsDxLXr8Dg8w0PraCRchLOP1fw-bO1UWEC7xiHGRYHYvMo27u0mCfnZDhj~q-sxEO9uYzK-Q7QNdRUBGoiveYB87g9HgzKh3G0Im8TuRmdyhpOFXOgdWlZK1JgDZG9vPOvynJjGzazkuIb9S~Wj8YfcTy3ul7hDzIJpfi24VeIP9fG3O9rmQM8cZ1SRmAgPZeC5Y3-zXSn9Hpu2bT346CW7uhmoSdGyqSAcUewZ1vCpo-UpxiJvg~TQ9KkAuYJsVFxFuJY9~bsvkqgLQ82Kad31w-i~bOb6Ym7nnSAJnw__&Key-Pair-Id=K1126YPJP5H8P

The 'main' url is https://d3njxchdk64paq.cloudfront.net/development/2/7/pypi/packages/8.whl and this does not change for a pre-signed url. Then there is the query components/url parameters, i.e., everything after ?, e.g., Expires, Signature, etc., which usually changes each time the 'main' url is pre-signed.

Thank you for clarifying with:
(a) This quote "... the query is part of the URL"
(b) The code snippet showing (a)

I think this is the crux of the discussion (I hope I didn't misinterpret anyone. At the very least our users (Buildkite/Packagecloud) are seeing the same phenomenon), i.e., each pre-signed url (whether google storage, aws cloudfront/s3, etc.) if considered with query component/string, is always different even though the pre-signed url is pointing to the same object in the bucket, thus cache hit never happens. This is because the pre-signed url contains query component/string with Expires and Signature (for aws) and similar counterparts for other cloud, that is always changing each time a pre-signed url is generated.

uv, which is a pip alternative on the other hand behaves as expected, i.e., performing cache hit/miss based on just the url without considering the query components/strings.

If you can help point to any RFCs that indicates cache hit/miss should take the full url including query components/string, it will be super helpful to determine where we can fix this issue.

[woodruffw]

If you can help point to any RFCs that indicates cache hit/miss should take the full url including query components/string, it will be super helpful to determine where we can fix this issue.

See this part of RFC 7234: https://datatracker.ietf.org/doc/html/rfc7234#section-4

Specifically, a cache MUST NOT reuse a response if the "effective request URI" does not match. That is then defined here: https://datatracker.ietf.org/doc/html/rfc7230#section-5.5

And to my read, the requirement here is to include the query component in the "effective request URI" (since it's a semantically relevant part of the URI).

I don't know exactly how uv behaves in this regard, but if you're seeing cache hits on signed URLs that vary by query component then I suspect that something, somewhere, is violating cache-control semantics. That, or there's some special casing around these kinds of signed URLs, since they're uncacheable by design.

However, to take a step back: can you confirm that what you're seeing is related to the original issue here? Specifically, is your first URL performing a 301 redirect or a 302 redirect?

[woodruffw]

(Another possibility is that uv is doing application level catching instead of HTTP response caching. That would sidestep the problem entirely, but I don't know if that's actually what's happening.)

[woodruffw]

I looked into this, and I don't believe uv is doing application-level caching -- it has HTTP caching middleware that (like CacheControl) doesn't cache the HTTP 302 redirect. As far as I can tell uv's caching should also invalidate on URL query changes (there are some places in uv where the query component is removed, but FWICT these don't affect the cache key).

[nethsix]

However, to take a step back: can you confirm that what you're seeing is related to the original issue here? Specifically, is your first URL performing a 301 redirect or a 302 redirect?

This is what pip is doing when I did pip install verysimplemodule===0.0.1 the second time (I deleted the package after first install but left the cache untouched)

[A] pip request for simple index
[B] pip gets 302 redirect
[C] pip says 302 is NOT in (200, 203, 300, 301, 308)
[D] pip looks up the redirect Location: header which points to AWS cloudfront pre-signed url (which has Expires query parameter that is constantly changing each this the same AWS cloudfront url is pre-signed
[E] pip found not cache hit for the AWS cloudfront pre-signed url as it considers all query parameters when searching for cache hit
[F] pip downloads the python package from AWS cloudfront pre-signed url

2025-07-14T06:07:35,373 Found index url http://packagecloud.localhost/khor/another-repo/pypi/simple/ <====== [A]
2025-07-14T06:07:35,373 Looking up "http://packagecloud.localhost/khor/another-repo/pypi/packages/verysimplemodule-0.0.1-py3-none-any.whl" in the cache
2025-07-14T06:07:35,373 No cache entry available
2025-07-14T06:07:35,373 No cache entry available
2025-07-14T06:07:35,403 http://packagecloud.localhost:80 "GET /khor/another-repo/pypi/packages/verysimplemodule-0.0.1-py3-none-any.whl HTTP/1.1" 302 0  <======== [B]
2025-07-14T06:07:35,403 Status code 302 not in (200, 203, 300, 301, 308) <======== [C]
2025-07-14T06:07:35,404 Looking up "https://d3njxchdk64paq.cloudfront.net/development/2/7/pypi/packages/8.whl?Expires=1752474000&Signature=qORpa8TJo3OgXQRdI3ASkvvwQ5OVNBhSX1kUQjkJ964CvYsDxLXr8Dg8w0PraCRchLOP1fw-bO1UWEC7xiHGRYHYvMo27u0mCfnZDhf~q-sxEO9uZdK-Q7QNdRUBGoiveYB87g9HgzKh3G0Im8TuRmdyhpOFXOgdWlZK1JgDZG9vPOvynJjGzazkuIb9S~Wj8YfcTy3ul7hDzIJpfi24VeIP9fG3O9rmQM8cZ1SRmAgPZeC5Y3-zXSn9Hpu2bT346CW7uhmoSdGyqSAcUewZ1vCpo-UpxiJvg~TQ9KkAuYJsVFxFuJY9~bsvkqgLQ82Kad31w-i~bOb6Ym7nnSAJnw__&Key-Pair-Id=K1126YPJP5H8P0" in the cache <======== [D]
2025-07-14T06:07:35,404 No cache entry available <======== [E]
2025-07-14T06:07:35,404 No cache entry available
2025-07-14T06:07:35,876 https://d3njxchdk64paq.cloudfront.net:443 "GET /development/2/7/pypi/packages/8.whl?Expires=1752474000&Signature=pMRpa8TJo3OgXQRdI3ASkvvwQ5OVNBhSX1kUQjkJ964CvYsDxLXr8Dg8w0PraCRchLOP1fw-bO1UWEC7xiHGRYHYvMo27u0mCfnZDhf~q-sxEO9uYzK-Q7QNdRUBGoiveYB87g9HgzKh3G0Im8TuRmdyhpOFXOgdWlZK1JgDZG9vPOvynJjGzazkuIb9S~Wj8YfcTy3ul7hDzIJpfi24VeIP9fG3O9rmQM8cZ1SRmAgPZeC5Y3-zXSn9Hpu2bT346CW7uhmoSdGyqSAcUewZ1vCpo-UpxiJvg~TQ9KkAuYJsVFxFuJY9~bsvkqgLQ82Kad31w-i~bOb6Ym7nnSAJnw__&Key-Pair-Id=K1126YPJP5H8P0 HTTP/1.1" 200 1242
2025-07-14T06:07:35,877 Downloading http://packagecloud.localhost/khor/another-repo/pypi/packages/verysimplemodule-0.0.1-py3-none-any.whl (1.2 kB) <======== [F]

Likewise when using uv to download verysimplemodule===0.0.1 the second time (I deleted the package after first install but left the cache untouched):

[A] uv request for simple index
[B] uv seems to have found the cached url for simple index, and requested revalidation
[C] uv got modified response for simple index
[D] uv retrieved the newer simple index
[E] uv conclude that there is a cache hit for the python package so no need for redownload

DEBUG Found stale response for: http://packagecloud.localhost/khor/another-repo/pypi/simple/verysimplemodule/ <===== [A]
DEBUG Sending revalidation request for: http://packagecloud.localhost/khor/another-repo/pypi/simple/verysimplemodule/ <===== [B]
DEBUG Found modified response for: http://packagecloud.localhost/khor/another-repo/pypi/simple/verysimplemodule/ <===== [C]
DEBUG Searching for a compatible version of verysimplemodule (>=0.0.1, <0.0.1+)
DEBUG Selecting: verysimplemodule==0.0.1 [compatible] (verysimplemodule-0.0.1-py3-none-any.whl)
DEBUG Found fresh response for: http://packagecloud.localhost/khor/another-repo/pypi/packages/verysimplemodule-0.0.1-py3-none-any.whl.metadata <===== [D]
DEBUG Tried 2 versions: test 1, verysimplemodule 1
DEBUG all marker environments resolution took 0.408s
Resolved 2 packages in 409ms
DEBUG Using request timeout of 30s
DEBUG Registry requirement already cached: verysimplemodule==0.0.1 <===== [E]
Installed 1 package in 2ms
 + verysimplemodule==0.0.1

[notatallshaw]

uv, which is a pip alternative on the other hand behaves as expected, i.e., performing cache hit/miss based on just the url without considering the query components/strings.

I wasn't aware uv cached HTTP responses at all. I know it caches built packages and other application specific artifacts.

Can you point to what you mean in the uv codebase, prior art of other applications implementing your intended behavior is useful to make informed choices.

[woodruffw]

@nethsix Thanks for the detailed logs, that's very helpful. Could you re-run the uv one at TRACE? That might reveal more of the redirects.

@notatallshaw Yep, uv uses a subset of the cache-control semantics -- you can see those here (in part): https://github.com/astral-sh/uv/blob/00efde06b61756f0f305fcf67b12db71a29063d3/crates/uv-client/src/httpcache/mod.rs

[notatallshaw]

Thanks for the info.

This is probably obvious to everyone in the discussion but cache control is far more of a general middleware than pip's specific use case, and using http cache as the primary mechanism for catching is probably not perfect for pip, but cache control shouldn't be changed for that purpose.

I'm going to privately ping the uv team in case they have any input on the question of query components.

[woodruffw]

This is probably obvious to everyone in the discussion but cache control is far more of a general middleware than pip's specific use case, and using http cache as the primary mechanism for catching is probably not perfect for pip, but cache control shouldn't be changed for that purpose.

Yep, agreed -- I also think (barring more evidence) that what CacheControl and pip are doing here is probably more correct, since the observed states (302 redirects and changing queries) are straightforwardly not cacheable at the HTTP layer. But to your point, this is a discrepancy between what pip semantically wants to cache (application-level contents) and what HTTP caching actually enables (responses).

I'm going to privately ping the uv team in case they have any input on the question of query components.

I've already raised it internally 😉