Since upgrading to version 1.10.0, the Dify platform has experienced incompatibility issues with Langfuse.

[yue1239067739]

Describe your question

Background: The monitoring module on Deify is correctly connected to Langfuse. These issues did not occur before the Deify update (version 1.9.0).

Previously, when running single-test or batch data on Deify, Langfuse would record the corresponding data and costs with a delay of only about 10 seconds.

However, after the Deify update, Langfuse cannot record data and costs promptly, sometimes with a delay of 8-12 hours. Sometimes, data is even lost, meaning no historical batch data is recorded.

I have upgraded from version v3.120.0 to the latest Langfuse version v3.134.0, but the problem persists.

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

v3.134.0 OSS

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Based on the available Langfuse documentation, the delay you're experiencing is consistent with Langfuse v3's asynchronous architecture(1).

In Langfuse v3, the /api/public/ingestion endpoint is now asynchronous(1). It accepts all events as they come in and queues them for processing before returning a 207 status code(1). This means that events will not be available immediately after acceptance by the backend and instead will appear eventually in subsequent read requests(1).

The 8-12 hour delays you're experiencing suggest potential issues with your background processing infrastructure. Langfuse v3 introduced several new components that handle asynchronous processing(1):

1. Worker Container - processes events asynchronously from a Redis queue(1)
2. Redis/Valkey - used for queue and cache operations(1)
3. ClickHouse - stores traces, observations, and scores(1)

To troubleshoot your delays:

1. Verify all v3 infrastructure is running properly:

   • Ensure the Langfuse Worker container is running and processing events(1)
   • Check Redis connection and queue health(1)
   • Verify ClickHouse connectivity(1)

2. Check background migrations status - Langfuse v3 introduced background migrations that run after deployment(1). If these are still running or stuck, they could impact ingestion performance(1).

3. Review worker container logs - The worker container handles the heavy processing of events asynchronously(1). Check for errors or performance issues there.

The data loss you mentioned could occur if events are being dropped from the Redis queue or if the worker container isn't processing them successfully(1).

Since this issue appeared after Dify's upgrade (not Langfuse's), it's also worth checking if Dify 1.10.0 changed how it sends data to Langfuse's ingestion endpoint.

📚 Sources:

1. Migrate Langfuse v2 to v3

---

Have another question? Just tag @inkeep.

[Lotte-Verheyden]

Hi @yue1239067739, was the Inkeep bot's answer helpful for you?