Caching with Content Loader API

[elbotho]

As far as I found the docs only mention that there is caching when using the Content Loader API right now:

The Content Layer API allows data to be cached between builds and is suitable for tens of thousands of content entries.

It would be really helpful to explain how it works in more detail. Some question I was hoping to finds answers for:

• How and where is the data cached?
• Can I use a cache when building in e.g. a GitHub Action?
• Revalidation: how do build-in and custom loaders check if an entry has to be revalidated?
• Can I access the collections from outside .astro files? (e.g. a helper script that exports a sitemap.xml?)

[sarah11918]

I think there's definitely a place for some additional content around caching.

I'll note that we have a couple of open roadmap proposals that won't change what currently happens for collections using static builds, but will add to the complete picture of content collections caching:

Live content loaders with an optional integration with route caching, allowing loaders to define cache tags, TTL etc associated with the data, which the user could decide to use

I think both of these accepted proposals (Stage 2 right now) will dictate how we eventually describe caching for content collections, and for anyone reading, you might find some helpful answers about how content loader caching currently works in the discussions about how to add to existing functionality.

In the mean time, I'll ask @ascorbic if he can answer these questions, and what of this makes sense to add to docs directly especially given the upcoming planned features. (Maybe we'll develop new docs in conjunction with the new features.)

[ascorbic]

• The data is cached in .astro or node_modules/.astro depending on whether it's dev or production
• If it caches node modules it should cache the data store
• The built-in loaders check the hash of the contents of the source files. Custom loaders can use their own checks. There is a generateDigest helper provided to loaders to help with this.
• You can only access the data inside the Astro build, where you can import from astro:content. You shouldn't try to load the data store directly, as it's a private API and likely to move, change format etc without warning.

[Fryuni]

How and where is the data cached?

The data is cached in .astro or node_modules/.astro depending on whether it's dev or production

I think this portion is worth being on the docs (it might be, haven't looked too deeply into it) because the caching strategy of a CI might not be to cache node_modules. Both npm, yarn and bun have a cache folder which can be used as a common cache across multiple projects and their install command will copy the relevant parts from the cache. pnpm doesn't even write data into node_modules, only link to its global store, which again can be cached and reused across projects. This is common on large companies with hundreds of projects.

For such cases, the specific node_modules/.astro path have to be configured for caching or it would be missed.

Can I access the collections from outside .astro files? (e.g. a helper script that exports a sitemap.xml?)

You can only access the data inside the Astro build, where you can import from astro:content.

Just to clarify that this doesn't have to be an .astro file, as mentioned in the question, it can be any .ts file as well. So if OP wants to make a helper function to generate the information for a sitemap.xml.ts route it is totally ok and intended use.

If "a helper script that exports a sitemap.xml" refers to a separate thing to execute outside of the Astro project (either prior or after the build) then it is not an intended and supported use case (possible, sure, but not supported)

[ascorbic]

It is documented here, though it doesn't explicitly say that this includes the data store.