Respect Cache-Control header for fetch data-cache

[rattrayalex]

Goals

1. Allow library authors to disable caching in vercel's fetch data cache without leaking Next-specific types or runtime behavior.
2. Respect standards.

Non-Goals

No response

Background

A user making requests with the openai TS SDK had to do some backflips to force Next to not use the cache for requests:

openai/openai-node#182 (comment)

We have workarounds available, but at least for certain endpoints, I'd expect ~nobody would want to cache these endpoints; it'd be nice if the library could either disable this cache by default, or allow the user to do so easily (passing headers is often easier through another library than nonstandard requestinit options).

Proposal

I don't have specific proposals, but following https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching more closely or at least respecting Cache-Control: no-cache would be great.

[rattrayalex]

cc @ijjk as #49638 looks relevant

[davidbarratt]

I recently read the blog post for Next.js 15 RC and while the new caching heuristics are a step in the right direction, It still leaves me puzzled how Next.js could come to the conclusion that the cache instruction ought to be reversed.

The Cache-Control header was introduced in 1999 and has received many updates and revisions since its introduction.

What has remained constant since then is that the server has always instructed the client on how the response can be cached.

Next.js reverses this basic assumption about HTTP cacheability and instead completely ignores how a server provides instructions on how a resource can be cached by the client and instead imposes its own heuristics. This of course leads to severe under or over caching.

While the browser does have a mechanism for storing HTTP Responses manually, this is completely redundant if the server responds with a proper Cache-Control header. Of course, this mechanism could be useful for legacy or 3rd-party systems that are difficult or impossible to update. However, modern APIs should respond with the proper response headers, and if they do not, they ought to be updated to do so.

It seems sensible to me that Next.js could respect the Cache-Control header and act as a shared cache in the same way that Varnish, Squid, Cloudflare, Fastly, Akamai, and many others do.

Instead, Next.js simply ignores decades of existing software and charts its own path, one that makes little sense to this engineer who has been making websites since before the Cache-Control header was invented. I hope that Next.js changes this behavior and brings a consistent and coherent HTTP caching strategy to its users.