HNT-1890: shared curated recommendation cache

[mmiermans]

References

JIRA: HNT-1890

Description

Adds a shared Redis cache between the per-pod in-memory cache and the Corpus GraphQL API. ~300 pods currently hit the API independently every ~60s. With this, one pod fetches and the rest read from Redis.

How it works

Requests are always served from the in-memory cache (L1) — either fresh or stale. Stale data is served whenever possible so that the Redis and API calls never block the request. Revalidation happens in a background task: it checks Redis (L2) first, and only the pod that acquires the distributed lock fetches from the API.

flowchart TB
    req["Firefox NewTab Request"]

    subgraph L1 ["L1 — Per-Pod In-Memory Cache"]
        check_l1{{"Check in-memory cache"}}
        l1_lock{{"asyncio.Lock (per entry)"}}
    end

    respond["Respond with data"]
    return_error["BackendError"]

    subgraph bg ["Background Revalidation Task"]

        subgraph L2 ["L2 — Shared Redis"]
            check_l2{{"Check Redis cache"}}
            acquire_lock{{"Try distributed lock"}}
        end

        api["Fetch from Corpus GraphQL API"]
        write["Write to Redis + release lock"]
        retry_redis["Wait 500ms, retry Redis"]
        serve_stale["Serve stale + update L1"]
        return_data["Return data + update L1"]
        return_503["503 Service Unavailable"]
    end

    req --> check_l1

    check_l1 -- "FRESH or STALE" --> respond
    check_l1 -. "MISS (cold start)" .-> l1_lock

    respond -. "if STALE, spawns task" .-> l1_lock
    l1_lock -- "acquired" --> check_l2
    l1_lock -. "waited, value populated" .-> respond
    l1_lock -. "waited, fetch failed" .-> return_error

    check_l2 -- "FRESH HIT" --> return_data
    check_l2 -. "STALE or MISS" .-> acquire_lock

    acquire_lock -- "LOCK ACQUIRED" --> api
    acquire_lock -. "LOCK HELD + stale exists" .-> serve_stale
    acquire_lock -. "LOCK HELD + no data" .-> retry_redis

    retry_redis -- "HIT" --> return_data
    retry_redis -. "MISS" .-> return_503

    api --> write --> return_data

    style req fill:#2c3e50,stroke:#1a252f,color:#ecf0f1,stroke-width:2px
    style check_l1 fill:#2980b9,stroke:#1f6da0,color:#fff,stroke-width:2px
    style l1_lock fill:#2980b9,stroke:#1f6da0,color:#fff,stroke-width:2px
    style check_l2 fill:#d35400,stroke:#a04000,color:#fff,stroke-width:2px
    style acquire_lock fill:#e67e22,stroke:#bf6516,color:#fff,stroke-width:2px
    style api fill:#1e8449,stroke:#145a32,color:#fff,stroke-width:2px
    style write fill:#1e8449,stroke:#145a32,color:#fff,stroke-width:2px
    style respond fill:#27ae60,stroke:#1e8449,color:#fff,stroke-width:2px
    style serve_stale fill:#f4d03f,stroke:#d4ac0f,color:#333
    style retry_redis fill:#e67e22,stroke:#bf6516,color:#fff,stroke-width:2px
    style return_data fill:#27ae60,stroke:#1e8449,color:#fff
    style return_503 fill:#e74c3c,stroke:#c0392b,color:#fff
    style return_error fill:#e74c3c,stroke:#c0392b,color:#fff
    style L1 fill:#eaf2f8,stroke:#2980b9,stroke-width:2px,color:#2c3e50
    style L2 fill:#fef5e7,stroke:#d35400,stroke-width:2px,color:#2c3e50
    style bg fill:#f4f6f7,stroke:#95a5a6,stroke-width:2px,stroke-dasharray: 8 4,color:#2c3e50

Loading

Design decisions

Decision	Choice	Why
Cache layer	Redis L2 behind existing in-memory L1	Keeps per-pod latency low, Redis only consulted on L1 miss or stale
Write pattern	Distributed stale-while-revalidate	One pod revalidates, others serve stale. Avoids thundering herd
L1 lock	asyncio.Lock per cache entry	Coordinates concurrent revalidation within a single pod
L2 lock	SET NX EX with TTL	Distributed, self-expiring. Worst case on timeout: one extra API call
Soft/hard TTL	60s soft, 1 day hard	Soft triggers revalidation. Hard is a safety net for extended API outages
Cache format	Pydantic model dicts via orjson	Processed models cached, not raw GraphQL. Saves CPU across pods
Redis cluster	Same shared cluster as other providers	~36 keys, ~4MB total. Not worth a separate instance
Cold miss (lock held)	503 Service Unavailable	Prevents connection pile-up; Firefox shows cached NewTab content
No circuit breaker	L1 asyncio.Lock limits Redis traffic	Only one coroutine per cache entry per pod reaches Redis. In steady state, L1 serves stale data and only the background task hits Redis
Rollout	Global config flag, off by default	Enable in staging first, then production

Rollout

0. Checked with Nan; 5 MB storage and 40 IOPS are fine.
1. Enabled in staging via stage.toml. Monitor metrics, validate API call reduction.
2. Enable in production: add cache = "redis" to production.toml or set MERINO_CURATED_RECOMMENDATIONS__CORPUS_CACHE__CACHE=redis.
3. After production rollout: add a request volume alert in Apollo based on the new (lower) baseline, to detect when the Redis cache fails and pods fall back to direct API calls.

PR Review Checklist

Put an x in the boxes that apply

• This PR conforms to the Contribution Guidelines
• The PR title starts with the JIRA issue reference, format example [DISCO-####], and has the same title (if applicable)
• [load test: (abort|skip|warn)] keywords are applied to the last commit message (if applicable)
• Documentation has been updated (if applicable)
• Functional and performance test coverage has been expanded and maintained (if applicable)

[mmiermans]

Returning a 201 code in the scenario where we do not have any (stale) data would improve reliability, by avoiding connections piling up.

TODO: Check how Firefox Desktop & Mobile would handle 201 status code? Does it retry?

[mmiermans]

Redis L2 behind existing in-memory L1

Q: Are we consistent, or does weather and other providers hit Redis on every request? Are the use-cases different?

Q: What's the existing pattern for testing with Redis?

Q: What are the existing Redis patterns?

[mmiermans]

We probably want this to be high: more like 1-24 hours? Days? To match our response time.

[mmiermans]

Again, check if this is a new pattern, and whether it should be one.

[mmiermans]

Redis locks are a new pattern because the curated recommendations use-case is unique, in that a lot of the same requests arrive at the same time.

[mmiermans]

TODO: Check if there's a client network timeout of 30s?

[mmiermans]

Checked: the HTTP client for Corpus API requests uses httpx.AsyncClient with a 5-second request timeout (default from merino/utils/http_client.py:12). The lock_ttl_sec = 30 here is for the Redis distributed lock auto-expiry, not for network timeouts.

So the worst case for a single get_or_fetch call with Redis timing out is bounded by the Redis socket_timeout_sec (configured in [default.redis]), not 30 seconds per API call.

[mmiermans]

Let's verify whether we still need this? Do we need to fetch yesterday's data for the schedule?

[mmiermans]

Let's check if there's a pattern to create an envelope or have a separate expiration key?

[mmiermans]

Discussed with Herraj: no existing pattern. Envelope is probably easiest.

[mmiermans]

Return 201 if this happens?

[mmiermans]

Why is this the first time we need to do a shutdown in the curated-recommendations provider?

[mmiermans]

Because previous resources did not use external connections.

[mmiermans]

Is there any benefit to severing GCS connections here? Other providers may do this. What happens if we don't?

[mmiermans]

Key files is probably not useful; I'd remove it.

[mmiermans]

Initially roll out to the staging environment for testing.

[mmiermans]

Re: days_offset comment

Verified: days_offset is never called with a non-zero value. The only call site is provider.py:195 which uses the default (days_offset=0).

The parameter is part of the pre-existing ScheduledSurfaceProtocol interface, so the cache wrapper implements the same signature. Since it's always 0, the cache key doesn't need to include it.

[mmiermans]

I'm curious whether we'd like to see more documentation around the Curated Recommendations endpoint. I think yes, but out-of-scope for this PR.

[mmiermans]

This could be an enum, although I don't know if it matters. The scheduled backend is going away in 2026Q2.

[mmiermans]

I didn't differentiate between the two backend endpoints, because scheduled is going away in Q2, and the main goal is to reduce total Apollo calls.

[mmiermans]

This reverts #1305, and keeps the total TTL including Redis the same at about 2 minutes.

[jpetto]

requesting splitting up this PR - i think focusing on the merino/cache files (and relevant docs) would be a good first step to break these changes into smaller, easier to review chunks. set the groundwork for the actual implementation in one PR, then implement in a service in a follow-up.

[jpetto]

i think it's worth mentioning that a value of none will increase corpus load by the number of k8s pods (300).

[jpetto]

can you add a comment about the practical use of this value?

[jpetto]

same here - add a comment on the practical use.