Skip to content

🛡️ fix: Invalidate Regional Role Caches - #18099

Open
dustinhealy wants to merge 5 commits into
devfrom
fix/regional-role-cache-invalidation
Open

dustinhealy wants to merge 5 commits into
devfrom
fix/regional-role-cache-invalidation

Conversation

@dustinhealy

@dustinhealy dustinhealy commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Role changes can leave stale permissions in application instances that use independent role caches. Updating one cache does not invalidate the others, and persistent cached roles can outlive the database change.

Add opt-in role-cache invalidation driven by the authoritative database's role change stream, with bounded expiry as recovery when events are unavailable. Permission reads retain Redis cache hits; only cold misses read the primary database. Existing behavior remains the default.

How it works

Each application process watches the roles collection and rotates a generation token in its cache namespace when a change arrives. Reads capture that generation before fetching a role, so an in-flight read cannot repopulate the active namespace with an obsolete value. Cached values expire from the start of the database read rather than its completion.

role database change
  each process's change-stream cursor
    rotate its role-cache generation
      next miss reads the primary database
        fill only the captured generation, with bounded TTL

The watcher invalidates after establishing or reconnecting its cursor, retries failures, and closes during graceful shutdown. Foreground mutations rotate the generation instead of publishing potentially reordered update results. Enabling regional caching switches both Keyv and its adapter to propagate write failures, while read fills remain best effort. The TypeScript cache layer selects the stores for the existing expiry sweep. When Redis is enabled and role caching is forced into memory, the sweep reclaims expired generations from the underlying role store. A TTL-only mode supports deployments without change streams. The optional roleCache block is set in librechat.yaml and takes effect on restart. The admin config API reports it as startup-only and skips profile overrides.

Type of change

  • Bug fix
  • Tests / tooling / CI

Testing

Tested environments/configuration:

  • Local MongoDB replica set and three independent disposable Redis instances. Two full application processes backed by separate Redis instances both returned the updated role setting within 29 ms of starting an authenticated update in the integration fixture. The ordinary authenticated MCP tools route returned success on both instances.
  • Feature disabled, change streams enabled, TTL-only mode, and interrupted watcher recovery. Standalone MongoDB served the updated setting within the two-second TTL used by the fixture. A scoped rollback rehearsal cleared ROLES, preserved an adjacent namespace, and returned the committed setting after restarting both app instances with legacy caching.

Automated tests:

  • 22 cache tests, including six real MongoDB/Redis integration tests covering grants, revocations, cache hits, tenant isolation, delayed fills, missed events, and mutation publication failures, and actual Redis server write rejections.
  • Role methods, server startup, role and adjacent MCP routes, native Keyv expiry selection, the forced-memory role sweep, and CI startup contracts. Admin metadata and merge tests verify that roleCache is startup-only and that stored overrides are reported as ignored.
  • Production package and client builds, workspace TypeScript checks, and static checks. Lighthouse passed at 250 ms injected per MongoDB query. Median LCP was 1.65 s with the default settings. An earlier TTL-only run also passed at 1.54 s.

Risk / compatibility

The feature is disabled by default. Upgrade every replica before enabling it, then restart all replicas together with the same setting. Mixed settings use different cache keys even when instances share Redis. Change-stream mode requires a database deployment that supports collection change streams and a database identity allowed to watch the roles collection. Each process owns one cursor, and changes invalidate the whole role-cache namespace, including tenant-scoped entries.

Change events normally invalidate caches promptly. During a watcher outage, cached values expire within the configured TTL while cache operations remain available. A warm hit adds one serial generation lookup in Redis and makes zero database queries. In a loopback benchmark of warm reads through the production cache wrapper, the median batch mean increased from 0.051 ms to 0.082 ms per sequential read. Production network latency was outside that measurement. To roll back, stop all replicas, clear the ROLES namespace in each backing Redis store, and restart with the feature disabled. With Redis enabled, startup waits for the initial generation write through the shared client, subject to its reconnect and offline-queue settings. Already-open browser panels reflect changes on their next qualifying request.

Checklist

  • I reviewed my own changes
  • Relevant tests have been added or updated
  • Existing relevant tests pass
  • The change does not introduce new warnings or errors
  • User-facing or complex behavior is documented where necessary
  • Required dependency changes have been merged/published — no new dependencies
  • Required documentation PR: N/A; configuration example included

Copilot AI balanced review requested due to automatic review settings November 12, 2026 17:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Generation-write failures can silently leave stale permissions cached after reported invalidation, and forced in-memory role caching with Redis enabled lacks an expiry sweep, allowing expired entries to accumulate.

2 open issues
What changed in this PR

Adds opt-in role-cache invalidation across application instances using independent caches. Database change streams rotate cache generations, while bounded expiry provides recovery when events are unavailable; legacy behavior remains the default.

Changes:

  • Add generation-scoped caching, primary-database reads on cache misses, and configurable TTL-only fallback.
  • Wire change-stream watchers into startup and graceful shutdown.
  • Add unit and MongoDB/Redis integration coverage, CI execution, and configuration examples.
File Description
packages/​data-schemas/​src/​types/​cache.ts Add snapshot and primary-read cache capabilities.
packages/​data-schemas/​src/​roles/​changes.ts Define the role change-stream adapter.
packages/​data-schemas/​src/​methods/​role.ts Use cache snapshots and primary reads when enabled.
packages/​data-schemas/​src/​index.ts Export role change-stream interfaces and adapter.
packages/​data-provider/​src/​config.ts Define optional regional role-cache settings.
packages/​api/​src/​cache/​roles.ts Implement generation caching and watcher lifecycle.
packages/​api/​src/​cache/​roles.spec.ts Test expiry, invalidation, failures, and recovery.
packages/​api/​src/​cache/​roles.redis.spec.ts Test propagation across three independent Redis caches.
packages/​api/​src/​cache/​index.ts Export regional role-cache functionality.
packages/​api/​package.json Add the role-cache integration test command.
librechat.example.yaml Document opt-in settings and deployment requirements.
api/​server/​index.js Register watcher startup and shutdown.
api/​server/​experimental.js Wire watcher lifecycle into the experimental server.
api/​cache/​getLogStores.js Wrap the role cache and include its base in expiry cleanup.
.github/​workflows/​cache-integration-tests.yml Provision services and run regional cache integration tests.

Review effort: Balanced

Comment thread api/cache/getLogStores.js Resolved
Comment thread api/cache/getLogStores.js Resolved
@dustinhealy

Copy link
Copy Markdown
Collaborator Author

/quality check-all

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Quality request

Outdated: this request is for an earlier PR commit.

Commit: 437c59c6029c08a9c6a29e399318df500724c99f. Requested by @dustinhealy.
Checks: check-all. Scopes: data-schemas, api, server. Static: off.
Risk: data-schemas, api, server. Mutation: data-schemas (four shards).
Mutation testing is experimental. Only data-schemas is supported today. Support for more scopes is planned.

Status: completed (success). Actions run (attempt 1).

Check Result
Mutation testing 1/4 (data-schemas) success
Mutation testing 3/4 (data-schemas) success
Mutation testing 2/4 (data-schemas) success
Mutation testing 4/4 (data-schemas) success
Change risk (server) success
Change risk (data-schemas) success
Change risk (api) success
Mutation report (data-schemas) success

Reports and scores:

  1. change-risk-api-437c59c6029c08a9c6a29e399318df500724c99f
  2. change-mutation-data-schemas-437c59c6029c08a9c6a29e399318df500724c99f-shard-3
  3. change-mutation-data-schemas-437c59c6029c08a9c6a29e399318df500724c99f
  4. change-mutation-data-schemas-437c59c6029c08a9c6a29e399318df500724c99f-shard-2
  5. change-mutation-data-schemas-437c59c6029c08a9c6a29e399318df500724c99f-shard-1
  6. change-risk-server-437c59c6029c08a9c6a29e399318df500724c99f
  7. change-risk-data-schemas-437c59c6029c08a9c6a29e399318df500724c99f
  8. change-mutation-data-schemas-437c59c6029c08a9c6a29e399318df500724c99f-shard-4

Scores are advisory. Failed tests, tool errors and incomplete reports do not clear the gate.

@dustinhealy

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: 🔄 Running

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-11-19T07:27:40.444921Z 5513373 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 437c59c602

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@dustinhealy
dustinhealy force-pushed the fix/regional-role-cache-invalidation branch from ac237e1 to 2ccb680 Compare November 19, 2026 04:22

Copy link
Copy Markdown
Collaborator Author

/quality check-all

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Quality report

Outdated: this request is for an earlier PR commit.

Gate: open. The selected checks completed. Scores and risk levels are advisory.

Status: completed (success). Actions run, attempt 1.

  • Commit: 2ccb68076c2c3c4c03ebf6893a0d68119639ee94
  • Requested by: @dustinhealy
  • Check: check-all
  • Scopes: data-schemas, api, server
  • Static mutants: off
  • Change risk: data-schemas, api, server
  • Mutation testing (experimental, data-schemas only): data-schemas (four shards)

Change risk

Takeaway: 49 changed lines have no test. Highest CRAP among changed functions: startServer at api/server/experimental.js:474, CRAP 1332.0 with 0% coverage.

Workspace Files measured Incomplete Unavailable Changed lines with no test Highest CRAP
data-schemas 4 of 4 0 0 19 15.1
api 2 of 2 0 0 16 15.1
server 3 of 3 0 0 14 1332.0

Changed lines with no test

packages/data-schemas/src/roles/changes.ts: 18 lines (data-schemas)

Lines 16-20

  return ({ resumeAfter, maxAwaitTimeMs }) => {
    const db = mongoose.connection.db;
    if (!db) {
      throw new Error('Role change source requires a connected database');
    }

Lines 24-36

    const stream = db.collection('roles').watch([{ $project: { _id: 1, operationType: 1 } }], {
      readPreference: 'primary',
      maxAwaitTimeMS: maxAwaitTimeMs,
      ...(resumeAfter ? { resumeAfter } : {}),
    });
    return {
      tryNext: async () => {
        const event = await stream.tryNext();
… 5 more lines
packages/api/src/cache/roles.ts: 16 lines (api)

Lines 16-19

  logger.warn(
    '[roleCache] Invalidation or caching failed; bounded expiry remains active:',
    getSafeErrorMetadata(error),
  );

Line 23

  logger.info('[roleCache] Regional generation invalidated', { reason });

Line 81

          throw new Error('Role cache generation initialization failed');

Lines 91-92

            onError(error);
            return undefined;

Lines 104-105

            onError(error);
            return false;

Line 127

        throw new RangeError('Role cache TTL must be a positive integer');

Line 134

        adapter.throwOnErrors = true;

Line 180

    onError(error);

Lines 220-222

          if (!closed) {
            onError(error);
          }
api/server/experimental.js: 14 lines (server)

Lines 22-24

const { logger, runAsSystem, createRoleChangeSource } = require('@librechat/data-schemas');
const { CacheKeys } = require('librechat-data-provider');
const getLogStores = require('~/cache/getLogStores');

Lines 537-547

    registerShutdownTask(
      'role cache invalidation',
      await startRoleCacheInvalidation({
        config: appConfig?.config?.roleCache,
        cache: getLogStores(CacheKeys.ROLES),
        open: createRoleChangeSource(mongoose),
        onError: reportRoleCacheError,
        onInvalidate: reportRoleCacheInvalidation,
… 3 more lines
packages/data-schemas/src/methods/role.ts: 1 line (data-schemas)

Line 272

        query = query.read('primary');

Refactor hints (highest CRAP first)

Function Complexity Coverage CRAP Location
startServer 36 0% 1332.0 api/server/experimental.js:474
startServer 43 91% 44.2 api/server/index.js:208
getRoleByName 15 91% 15.1 packages/data-schemas/src/methods/role.ts:256
run 15 92% 15.1 packages/api/src/cache/roles.ts:191
(anonymous) 3 0% 12.0 packages/api/src/cache/roles.ts:147
(anonymous) 3 0% 12.0 packages/api/src/cache/roles.ts:148
(anonymous) 3 0% 12.0 packages/data-schemas/src/roles/changes.ts:16
configure 6 78% 6.4 packages/api/src/cache/roles.ts:125
tryNext 2 0% 6.0 packages/data-schemas/src/roles/changes.ts:30
startRoleCacheInvalidation 5 92% 5.0 packages/api/src/cache/roles.ts:158

23 more functions below the threshold (or cut for size) are in report.md.

CRAP = complexity² × (1 − coverage)³ + complexity

Mutation testing

Takeaway: 23 survivors; the most are in packages/data-schemas/src/roles/changes.ts (18).

Score: killed by assertion, Killed/(Killed+Survived+NoCoverage); the Stryker score also counts timeouts.

Workspace Killed Survived Timeout No coverage Invalid Ignored Score Run
data-schemas 0 23 0 2 0 0 0.0% (Stryker 0.0%) complete

Survivors (tests still passed after this change to the code; not confirmed bugs), most important first

packages/data-schemas/src/methods/role.ts: 5 survivors on 3 lines

Line 271 · authorization · 3 survivors

-      if (cache?.primaryReads) {
@@ ConditionalExpression @@
+      if (true) {
@@ ConditionalExpression @@
+      if (false) {
@@ OptionalChaining @@
+      if (cache.primaryReads) {

Line 260 · authorization · 1 survivor

-    const store = deps.getCache?.(CacheKeys.ROLES);
+    const store = deps.getCache(CacheKeys.ROLES);

Line 261 · authorization · 1 survivor

-    const cache = store?.snapshot ? await store.snapshot() : store;
+    const cache = store.snapshot ? await store.snapshot() : store;
packages/data-schemas/src/roles/changes.ts: 18 survivors on 11 lines

Lines 15-37 · authorization, tenant, credentials, data-loss · 1 survivor

-export function createRoleChangeSource(mongoose: typeof import('mongoose')): RoleChangeSource {
-  return ({ resumeAfter, maxAwaitTimeMs }) => {
-    const db = mongoose.connection.db;
-    if (!db) {
-      throw new Error('Role change source requires a connected database');
-    }
-    // This invalidates a deployment-wide namespace, including deletions and renames.
-    // It intentionally carries neither tenant context nor full role documents.
-    // eslint-disable-next-line no-restricted-syntax -- System-wide coherence consumer, never a tenant API: projection retains only resume token/type. A tenant …
-    const stream = db.collection('roles').watch([{ $project: { _id: 1, operationType: 1 } }], {
-      readPreference: 'primary',
-      maxAwaitTimeMS: maxAwaitTimeMs,
-… 11 more lines
+export function createRoleChangeSource(mongoose: typeof import('mongoose')): RoleChangeSource {}

Lines 16-36 · authorization, tenant, credentials, data-loss · 1 survivor

-  return ({ resumeAfter, maxAwaitTimeMs }) => {
-    const db = mongoose.connection.db;
-    if (!db) {
-      throw new Error('Role change source requires a connected database');
-    }
-    // This invalidates a deployment-wide namespace, including deletions and renames.
-    // It intentionally carries neither tenant context nor full role documents.
-    // eslint-disable-next-line no-restricted-syntax -- System-wide coherence consumer, never a tenant API: projection retains only resume token/type. A tenant …
-    const stream = db.collection('roles').watch([{ $project: { _id: 1, operationType: 1 } }], {
-      readPreference: 'primary',
-      maxAwaitTimeMS: maxAwaitTimeMs,
-      ...(resumeAfter ? { resumeAfter } : {}),
-… 9 more lines
+  return ({ resumeAfter, maxAwaitTimeMs }) => {};

Lines 18-20 · authorization, data-loss · 4 survivors

-    if (!db) {
-      throw new Error('Role change source requires a connected database');
-    }
@@ BlockStatement @@
+    if (!db) {}
@@ BooleanLiteral @@
-    if (!db) {
+    if (db) {
@@ ConditionalExpression @@
-    if (!db) {
+    if (true) {

1 more survivor on this line in report.md.

Line 19 · authorization · 1 survivor

-      throw new Error('Role change source requires a connected database');
+      throw new Error("");

Line 24 · authorization, data-loss · 5 survivors

-    const stream = db.collection('roles').watch([{ $project: { _id: 1, operationType: 1 } }], {
@@ StringLiteral @@
+    const stream = db.collection("").watch([{ $project: { _id: 1, operationType: 1 } }], {
@@ ObjectLiteral @@
+    const stream = db.collection('roles').watch([{}], {
@@ ObjectLiteral @@
+    const stream = db.collection('roles').watch([{ $project: {} }], {

2 more survivors on this line in report.md.

Line 27 · authorization, data-loss · 1 survivor

-      ...(resumeAfter ? { resumeAfter } : {}),
+      ...(resumeAfter ? {} : {}),

Lines 29-35 · authorization, credentials, data-loss · 1 survivor

-    return {
-      tryNext: async () => {
-        const event = await stream.tryNext();
-        return event ? { token: event._id } : null;
-      },
-      close: () => stream.close(),
-    };
+    return {};

Line 32 · authorization, credentials, data-loss · 1 survivor

-        return event ? { token: event._id } : null;
+        return event ? {} : null;

Line 25 · authorization · 1 survivor

-      readPreference: 'primary',
+      readPreference: "",

Lines 30-33 · authorization, credentials, data-loss · 1 survivor

-      tryNext: async () => {
-        const event = await stream.tryNext();
-        return event ? { token: event._id } : null;
-      },
+      tryNext: async () => {},

Line 34 · authorization · 1 survivor

-      close: () => stream.close(),
+      close: () => undefined,
Changed lines that no test executed during mutation testing (1 range)
  • packages/data-schemas/src/methods/role.ts: 271-273
Jobs (8)
Check Result
Change risk (data-schemas) success
Change risk (api) success
Change risk (server) success
Mutation testing 1/4 (data-schemas) success
Mutation testing 2/4 (data-schemas) success
Mutation testing 3/4 (data-schemas) success
Mutation testing 4/4 (data-schemas) success
Mutation report (data-schemas) success

All data (quality-report artifact) · Run page

The gate opens only when the workflow, source selection, tooling verification, and every report job succeed. A report job fails on a failed test in its workspace's own Jest run, a tool error, a job timeout, or an incomplete mutation report, including a mutation shard that reaches its execution budget. A cancelled workflow keeps the gate closed. Scores are advisory. A change-risk file marked incomplete or unavailable, for example by a failed api/ source-alias suite, does not close the gate.

@dustinhealy
dustinhealy marked this pull request as ready for review November 19, 2026 04:35

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2ccb68076c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread api/cache/getLogStores.js Resolved
@dustinhealy
dustinhealy force-pushed the fix/regional-role-cache-invalidation branch from 2ccb680 to 5513373 Compare November 19, 2026 07:07

Copy link
Copy Markdown
Collaborator Author

/quality check-all

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Quality report

Gate: pending. Checks are running.

Status: running. Actions run, attempt 1.

  • Commit: 5513373273ae36b0c547f606dd48759aa47b5b82
  • Requested by: @dustinhealy
  • Check: check-all
  • Scopes: data-schemas, api, server
  • Static mutants: off
  • Change risk: data-schemas, api, server
  • Mutation testing (experimental, data-schemas only): data-schemas (four shards)
Check Result
Change risk (data-schemas) success
Change risk (api) success
Change risk (server) success
Mutation testing 1/4 (data-schemas) success
Mutation testing 2/4 (data-schemas) success
Mutation testing 3/4 (data-schemas) success
Mutation testing 4/4 (data-schemas) success
Mutation report (data-schemas) in_progress

Run page

The gate opens only when the workflow, source selection, tooling verification, and every report job succeed. A report job fails on a failed test in its workspace's own Jest run, a tool error, a job timeout, or an incomplete mutation report, including a mutation shard that reaches its execution budget. A cancelled workflow keeps the gate closed. Scores are advisory. A change-risk file marked incomplete or unavailable, for example by a failed api/ source-alias suite, does not close the gate.

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Unknown error
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copy link
Copy Markdown
Collaborator Author

@codex review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants