Skip to content

Endpoint CRUD does not invalidate CreateMessageApp cache, causing missed deliveries to new endpoints #2191

@orhanrauf

Description

@orhanrauf

Summary

When an endpoint is created (or updated/deleted), the CreateMessageApp cache is not invalidated. This causes the delivery worker to use a stale endpoint list for up to 30 seconds (the TTL), meaning newly created endpoints miss all messages dispatched during that window.

I understand this is not a usual case. We only discovered this in our E2E test suite that creates multiple endpoints in a short period and expects immediate delivery. In normal production usage the 30s window is unlikely to matter, but for any workflow where an endpoint is created and a message follows shortly after, the delivery is silently lost.

Reproduction

  1. Set cache_type = "memory" (the default) or cache_type = "redis"
  2. Send a message to an application (this populates the SVIX_CACHE_APP_v3_{org}_{app} cache)
  3. Within 30 seconds, create a new endpoint on that application
  4. Send another message

Expected: The new endpoint receives the second message.
Actual: The delivery worker uses the cached CreateMessageApp (which doesn't include the new endpoint) and never attempts delivery. The message exists in the DB but is never dispatched to the new endpoint.

Root cause

CreateMessageApp::layered_fetch in message_app.rs caches the application and all its endpoints with a 30s TTL:

// worker.rs - called for every message dispatch
let Some(create_message_app) = CreateMessageApp::layered_fetch(
    cache, db, None, msg.org_id.clone(), msg.app_id.clone(),
    Duration::from_secs(30),
).await?

let endpoints = create_message_app
    .filtered_endpoints(trigger_type, &msg.event_type, msg.channels.as_ref())

But the endpoint CRUD handlers in endpoint/crud.rs (create_endp_from_data, update_endp_from_data, and delete_endpoint) write to Postgres but never invalidate or update the cache.

Impact

This is especially problematic when:

  • Messages are dispatched frequently (keeping the cache permanently warm)
  • Endpoints are created and expected to receive events immediately (e.g., a user subscribes and triggers a job in quick succession)
  • Multiple instances share a Redis-backed cache

Workaround

Setting SVIX_CACHE_TYPE=none avoids the issue but loses the performance benefit of caching.

Suggested fix

Invalidate (delete) the AppEndpointKey cache entry when an endpoint is created, updated, or deleted:

// In create_endp_from_data / update_endp_from_data / delete_endpoint:
let cache_key = AppEndpointKey::new(&app.org_id, &app.id);
let _ = cache.delete(&cache_key).await;

This is a one-line addition in each handler and ensures the next layered_fetch hits Postgres with fresh data.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions