Repository navigation
🛡️ fix: Invalidate Regional Role Caches - #18099
dustinhealy wants to merge 5 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Generation-write failures can silently leave stale permissions cached after reported invalidation, and forced in-memory role caching with Redis enabled lacks an expiry sweep, allowing expired entries to accumulate.
2 open issues
What changed in this PR
Adds opt-in role-cache invalidation across application instances using independent caches. Database change streams rotate cache generations, while bounded expiry provides recovery when events are unavailable; legacy behavior remains the default.
Changes:
- Add generation-scoped caching, primary-database reads on cache misses, and configurable TTL-only fallback.
- Wire change-stream watchers into startup and graceful shutdown.
- Add unit and MongoDB/Redis integration coverage, CI execution, and configuration examples.
| File | Description |
|---|---|
packages/data-schemas/src/types/cache.ts |
Add snapshot and primary-read cache capabilities. |
packages/data-schemas/src/roles/changes.ts |
Define the role change-stream adapter. |
packages/data-schemas/src/methods/role.ts |
Use cache snapshots and primary reads when enabled. |
packages/data-schemas/src/index.ts |
Export role change-stream interfaces and adapter. |
packages/data-provider/src/config.ts |
Define optional regional role-cache settings. |
packages/api/src/cache/roles.ts |
Implement generation caching and watcher lifecycle. |
packages/api/src/cache/roles.spec.ts |
Test expiry, invalidation, failures, and recovery. |
packages/api/src/cache/roles.redis.spec.ts |
Test propagation across three independent Redis caches. |
packages/api/src/cache/index.ts |
Export regional role-cache functionality. |
packages/api/package.json |
Add the role-cache integration test command. |
librechat.example.yaml |
Document opt-in settings and deployment requirements. |
api/server/index.js |
Register watcher startup and shutdown. |
api/server/experimental.js |
Wire watcher lifecycle into the experimental server. |
api/cache/getLogStores.js |
Wrap the role cache and include its base in expiry cleanup. |
.github/workflows/cache-integration-tests.yml |
Provision services and run regional cache integration tests. |
Review effort: Balanced
|
/quality check-all |
|
@codex review |
Codex Review Summary: 🔄 RunningThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
ac237e1 to
2ccb680
Compare
|
/quality check-all |
Quality reportOutdated: this request is for an earlier PR commit. Gate: open. The selected checks completed. Scores and risk levels are advisory. Status: completed (success). Actions run, attempt 1.
Change riskTakeaway: 49 changed lines have no test. Highest CRAP among changed functions:
Changed lines with no test
|
| Function | Complexity | Coverage | CRAP | Location |
|---|---|---|---|---|
startServer |
36 | 0% | 1332.0 | api/server/experimental.js:474 |
startServer |
43 | 91% | 44.2 | api/server/index.js:208 |
getRoleByName |
15 | 91% | 15.1 | packages/data-schemas/src/methods/role.ts:256 |
run |
15 | 92% | 15.1 | packages/api/src/cache/roles.ts:191 |
(anonymous) |
3 | 0% | 12.0 | packages/api/src/cache/roles.ts:147 |
(anonymous) |
3 | 0% | 12.0 | packages/api/src/cache/roles.ts:148 |
(anonymous) |
3 | 0% | 12.0 | packages/data-schemas/src/roles/changes.ts:16 |
configure |
6 | 78% | 6.4 | packages/api/src/cache/roles.ts:125 |
tryNext |
2 | 0% | 6.0 | packages/data-schemas/src/roles/changes.ts:30 |
startRoleCacheInvalidation |
5 | 92% | 5.0 | packages/api/src/cache/roles.ts:158 |
23 more functions below the threshold (or cut for size) are in report.md.
CRAP = complexity² × (1 − coverage)³ + complexity
Mutation testing
Takeaway: 23 survivors; the most are in packages/data-schemas/src/roles/changes.ts (18).
Score: killed by assertion, Killed/(Killed+Survived+NoCoverage); the Stryker score also counts timeouts.
| Workspace | Killed | Survived | Timeout | No coverage | Invalid | Ignored | Score | Run |
|---|---|---|---|---|---|---|---|---|
| data-schemas | 0 | 23 | 0 | 2 | 0 | 0 | 0.0% (Stryker 0.0%) | complete |
Survivors (tests still passed after this change to the code; not confirmed bugs), most important first
packages/data-schemas/src/methods/role.ts: 5 survivors on 3 lines
Line 271 · authorization · 3 survivors
- if (cache?.primaryReads) {
@@ ConditionalExpression @@
+ if (true) {
@@ ConditionalExpression @@
+ if (false) {
@@ OptionalChaining @@
+ if (cache.primaryReads) {Line 260 · authorization · 1 survivor
- const store = deps.getCache?.(CacheKeys.ROLES);
+ const store = deps.getCache(CacheKeys.ROLES);Line 261 · authorization · 1 survivor
- const cache = store?.snapshot ? await store.snapshot() : store;
+ const cache = store.snapshot ? await store.snapshot() : store;packages/data-schemas/src/roles/changes.ts: 18 survivors on 11 lines
Lines 15-37 · authorization, tenant, credentials, data-loss · 1 survivor
-export function createRoleChangeSource(mongoose: typeof import('mongoose')): RoleChangeSource {
- return ({ resumeAfter, maxAwaitTimeMs }) => {
- const db = mongoose.connection.db;
- if (!db) {
- throw new Error('Role change source requires a connected database');
- }
- // This invalidates a deployment-wide namespace, including deletions and renames.
- // It intentionally carries neither tenant context nor full role documents.
- // eslint-disable-next-line no-restricted-syntax -- System-wide coherence consumer, never a tenant API: projection retains only resume token/type. A tenant …
- const stream = db.collection('roles').watch([{ $project: { _id: 1, operationType: 1 } }], {
- readPreference: 'primary',
- maxAwaitTimeMS: maxAwaitTimeMs,
-… 11 more lines
+export function createRoleChangeSource(mongoose: typeof import('mongoose')): RoleChangeSource {}Lines 16-36 · authorization, tenant, credentials, data-loss · 1 survivor
- return ({ resumeAfter, maxAwaitTimeMs }) => {
- const db = mongoose.connection.db;
- if (!db) {
- throw new Error('Role change source requires a connected database');
- }
- // This invalidates a deployment-wide namespace, including deletions and renames.
- // It intentionally carries neither tenant context nor full role documents.
- // eslint-disable-next-line no-restricted-syntax -- System-wide coherence consumer, never a tenant API: projection retains only resume token/type. A tenant …
- const stream = db.collection('roles').watch([{ $project: { _id: 1, operationType: 1 } }], {
- readPreference: 'primary',
- maxAwaitTimeMS: maxAwaitTimeMs,
- ...(resumeAfter ? { resumeAfter } : {}),
-… 9 more lines
+ return ({ resumeAfter, maxAwaitTimeMs }) => {};Lines 18-20 · authorization, data-loss · 4 survivors
- if (!db) {
- throw new Error('Role change source requires a connected database');
- }
@@ BlockStatement @@
+ if (!db) {}
@@ BooleanLiteral @@
- if (!db) {
+ if (db) {
@@ ConditionalExpression @@
- if (!db) {
+ if (true) {1 more survivor on this line in report.md.
Line 19 · authorization · 1 survivor
- throw new Error('Role change source requires a connected database');
+ throw new Error("");Line 24 · authorization, data-loss · 5 survivors
- const stream = db.collection('roles').watch([{ $project: { _id: 1, operationType: 1 } }], {
@@ StringLiteral @@
+ const stream = db.collection("").watch([{ $project: { _id: 1, operationType: 1 } }], {
@@ ObjectLiteral @@
+ const stream = db.collection('roles').watch([{}], {
@@ ObjectLiteral @@
+ const stream = db.collection('roles').watch([{ $project: {} }], {2 more survivors on this line in report.md.
Line 27 · authorization, data-loss · 1 survivor
- ...(resumeAfter ? { resumeAfter } : {}),
+ ...(resumeAfter ? {} : {}),Lines 29-35 · authorization, credentials, data-loss · 1 survivor
- return {
- tryNext: async () => {
- const event = await stream.tryNext();
- return event ? { token: event._id } : null;
- },
- close: () => stream.close(),
- };
+ return {};Line 32 · authorization, credentials, data-loss · 1 survivor
- return event ? { token: event._id } : null;
+ return event ? {} : null;Line 25 · authorization · 1 survivor
- readPreference: 'primary',
+ readPreference: "",Lines 30-33 · authorization, credentials, data-loss · 1 survivor
- tryNext: async () => {
- const event = await stream.tryNext();
- return event ? { token: event._id } : null;
- },
+ tryNext: async () => {},Line 34 · authorization · 1 survivor
- close: () => stream.close(),
+ close: () => undefined,Changed lines that no test executed during mutation testing (1 range)
packages/data-schemas/src/methods/role.ts: 271-273
Jobs (8)
| Check | Result |
|---|---|
| Change risk (data-schemas) | success |
| Change risk (api) | success |
| Change risk (server) | success |
| Mutation testing 1/4 (data-schemas) | success |
| Mutation testing 2/4 (data-schemas) | success |
| Mutation testing 3/4 (data-schemas) | success |
| Mutation testing 4/4 (data-schemas) | success |
| Mutation report (data-schemas) | success |
All data (quality-report artifact) · Run page
The gate opens only when the workflow, source selection, tooling verification, and every report job succeed. A report job fails on a failed test in its workspace's own Jest run, a tool error, a job timeout, or an incomplete mutation report, including a mutation shard that reaches its execution budget. A cancelled workflow keeps the gate closed. Scores are advisory. A change-risk file marked incomplete or unavailable, for example by a failed api/ source-alias suite, does not close the gate.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2ccb68076c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
2ccb680 to
5513373
Compare
|
/quality check-all |
Quality reportGate: pending. Checks are running. Status: running. Actions run, attempt 1.
The gate opens only when the workflow, source selection, tooling verification, and every report job succeed. A report job fails on a failed test in its workspace's own Jest run, a tool error, a job timeout, or an incomplete mutation report, including a mutation shard that reaches its execution budget. A cancelled workflow keeps the gate closed. Scores are advisory. A change-risk file marked incomplete or unavailable, for example by a failed |
|
@codex review |
|
Codex Review: Something went wrong. Try again later by commenting “@codex review”. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |


Summary
Role changes can leave stale permissions in application instances that use independent role caches. Updating one cache does not invalidate the others, and persistent cached roles can outlive the database change.
Add opt-in role-cache invalidation driven by the authoritative database's role change stream, with bounded expiry as recovery when events are unavailable. Permission reads retain Redis cache hits; only cold misses read the primary database. Existing behavior remains the default.
How it works
Each application process watches the roles collection and rotates a generation token in its cache namespace when a change arrives. Reads capture that generation before fetching a role, so an in-flight read cannot repopulate the active namespace with an obsolete value. Cached values expire from the start of the database read rather than its completion.
The watcher invalidates after establishing or reconnecting its cursor, retries failures, and closes during graceful shutdown. Foreground mutations rotate the generation instead of publishing potentially reordered update results. Enabling regional caching switches both Keyv and its adapter to propagate write failures, while read fills remain best effort. The TypeScript cache layer selects the stores for the existing expiry sweep. When Redis is enabled and role caching is forced into memory, the sweep reclaims expired generations from the underlying role store. A TTL-only mode supports deployments without change streams. The optional
roleCacheblock is set inlibrechat.yamland takes effect on restart. The admin config API reports it as startup-only and skips profile overrides.Type of change
Testing
Tested environments/configuration:
ROLES, preserved an adjacent namespace, and returned the committed setting after restarting both app instances with legacy caching.Automated tests:
roleCacheis startup-only and that stored overrides are reported as ignored.Risk / compatibility
The feature is disabled by default. Upgrade every replica before enabling it, then restart all replicas together with the same setting. Mixed settings use different cache keys even when instances share Redis. Change-stream mode requires a database deployment that supports collection change streams and a database identity allowed to watch the roles collection. Each process owns one cursor, and changes invalidate the whole role-cache namespace, including tenant-scoped entries.
Change events normally invalidate caches promptly. During a watcher outage, cached values expire within the configured TTL while cache operations remain available. A warm hit adds one serial generation lookup in Redis and makes zero database queries. In a loopback benchmark of warm reads through the production cache wrapper, the median batch mean increased from 0.051 ms to 0.082 ms per sequential read. Production network latency was outside that measurement. To roll back, stop all replicas, clear the
ROLESnamespace in each backing Redis store, and restart with the feature disabled. With Redis enabled, startup waits for the initial generation write through the shared client, subject to its reconnect and offline-queue settings. Already-open browser panels reflect changes on their next qualifying request.Checklist