-
Notifications
You must be signed in to change notification settings - Fork 233
Description
Summary
When an endpoint is created (or updated/deleted), the CreateMessageApp cache is not invalidated. This causes the delivery worker to use a stale endpoint list for up to 30 seconds (the TTL), meaning newly created endpoints miss all messages dispatched during that window.
I understand this is not a usual case. We only discovered this in our E2E test suite that creates multiple endpoints in a short period and expects immediate delivery. In normal production usage the 30s window is unlikely to matter, but for any workflow where an endpoint is created and a message follows shortly after, the delivery is silently lost.
Reproduction
- Set
cache_type = "memory"(the default) orcache_type = "redis" - Send a message to an application (this populates the
SVIX_CACHE_APP_v3_{org}_{app}cache) - Within 30 seconds, create a new endpoint on that application
- Send another message
Expected: The new endpoint receives the second message.
Actual: The delivery worker uses the cached CreateMessageApp (which doesn't include the new endpoint) and never attempts delivery. The message exists in the DB but is never dispatched to the new endpoint.
Root cause
CreateMessageApp::layered_fetch in message_app.rs caches the application and all its endpoints with a 30s TTL:
// worker.rs - called for every message dispatch
let Some(create_message_app) = CreateMessageApp::layered_fetch(
cache, db, None, msg.org_id.clone(), msg.app_id.clone(),
Duration::from_secs(30),
).await?
let endpoints = create_message_app
.filtered_endpoints(trigger_type, &msg.event_type, msg.channels.as_ref())But the endpoint CRUD handlers in endpoint/crud.rs (create_endp_from_data, update_endp_from_data, and delete_endpoint) write to Postgres but never invalidate or update the cache.
Impact
This is especially problematic when:
- Messages are dispatched frequently (keeping the cache permanently warm)
- Endpoints are created and expected to receive events immediately (e.g., a user subscribes and triggers a job in quick succession)
- Multiple instances share a Redis-backed cache
Workaround
Setting SVIX_CACHE_TYPE=none avoids the issue but loses the performance benefit of caching.
Suggested fix
Invalidate (delete) the AppEndpointKey cache entry when an endpoint is created, updated, or deleted:
// In create_endp_from_data / update_endp_from_data / delete_endpoint:
let cache_key = AppEndpointKey::new(&app.org_id, &app.id);
let _ = cache.delete(&cache_key).await;This is a one-line addition in each handler and ensures the next layered_fetch hits Postgres with fresh data.