Observation exist, but trace not exist

[mark-qin-derbysoft]

1. Self host;
   2.Version: 3.132.0;
   3.langfuse-web pod log:

langfuse-web-error.log

4.check clickhouse:
SELECT *
FROM default.observations
WHERE trace_id = '162894ae-08e9-4154-8855-ac2b2924f836';
return result with trace_id = 162894ae-08e9-4154-8855-ac2b2924f836

SELECT *
FROM default.traces
WHERE id = '162894ae-08e9-4154-8855-ac2b2924f836'
AND project_id = 'cmi2u0gm10006zo071v1908ag';

can't not find any trace;

observation exist, but trace not exist

5. On UI Page right top corn: rase error: Trace not found

[inkeep-ai-assistant[bot]]

This issue indicates that observations were successfully ingested into ClickHouse, but the corresponding trace record is missing. This can occur in Langfuse v3's asynchronous architecture where events are queued and processed separately(1).

Potential Causes:

1. Asynchronous Processing Delay: In Langfuse v3, the /api/public/ingestion endpoint is asynchronous and queues events for processing(1). Observations may appear before their parent trace is fully processed.

2. Event Processing Order: If observations arrive before the trace event, they'll be stored but the trace lookup will fail until the trace event is processed by the worker container(1).

3. Worker Container Issues: Check if your langfuse-worker container is running and processing the ingestion queue properly(1). The worker is responsible for upserting events into ClickHouse(1).

Troubleshooting Steps:

1. Check Worker Logs: Examine the langfuse-worker container logs for any processing errors or queue backlog indicators(2).

2. Verify Queue Length: Monitor the langfuse.queue.ingestion.length metric to see if events are backing up(2). High queue length suggests the worker is saturated(2).

3. Check S3/Blob Storage: Verify that the trace event was successfully written to your S3/Blob storage bucket (LANGFUSE_S3_EVENT_UPLOAD_BUCKET)(1). All incoming events are stored there before processing(1).

4. Wait for Processing: Since processing is asynchronous, the trace may appear after a short delay. Events should appear within a few seconds of being ingested under normal conditions(1).

5. Review Background Migrations: If you recently upgraded from v2 to v3, check that background migrations completed successfully(1). The traces migration should have finished before observations migration(1).

If the issue persists and the trace never appears despite observations being present, this may indicate a worker processing failure or event ordering issue that requires further investigation of your worker container logs and queue status.

📚 Sources:

1. Migrate Langfuse v2 to v3
2. Scaling

---

Have another question? Just tag @inkeep.

[jannikmaierhoefer]

Hi @mark-qin-derbysoft, was inkeep able to help you here?

[mark-qin-derbysoft]

I found something very strange, all lost trace with observation are related to dify integration with langfuse, and dify's Generate Name Trace, check:


this issue still exist, can anyone help me.

[mark-qin-derbysoft]

@jannikmaierhoefer BTW, only web-pod log have error info, for worker, clickhouse, zookeeper self deploy containers, are all work normaly

[jannikmaierhoefer]

Hi @mark-qin-derbysoft, could you share some more context what you are expecting to see? I am not sure if I understand your issue.
If you click on the observations, you are seeing in the table, you should also see the parent trace of these observations.

[mark-qin-derbysoft]

Sure, i can replay the issue step by step:
Step 1: Self-Host Langfuse version

Step 2: In traces tab, filter by name: generate_conversation_name

Step 3: Switch to Observations tab:

Step 4: Click input/ouput empty row:

Step 5: Chrome F12 dev tool check error:

Step 6: connect to clickhouse, query observation by id: 6478b899-bc96-44df-a4cd-c8432340641c

find observation and parent trace:
SQL: select * from observations where project_id='cmi2u0gm10006zo071v1908ag' and id='6478b899-bc96-44df-a4cd-c8432340641c';
Result: {
name: 'generate_conversation_name',
id: '6478b899-bc96-44df-a4cd-c8432340641c',
trace_id: '399591c7-e685-4b86-b6f1-a10e10343884',
project_id: 'cmi2u0gm10006zo071v1908ag',
type: 'span',
parent_observation: null,
metadata: { "conversation_id": "399591c7-e685-4b86-b6f1-a10e10343884", "tenant_id": "6b18ec24-f52b-49c4-9d79-2d1b2ddca5bd" },
input: '[{"role":"user","content":"You are asked to generate a concise chat title by decomposing the user’s input into two parts: “Intention” and “Subject”.\n\n1. Detect Input Language\nAutomatically identify the language of the user’s input (e.g. English, Chinese, Italian, Español, Arabic, Japanese, French, and etc.).\n\n2. Generate Title\n- Combine Intention + Subject into a single, as-short-as-possible phrase.\n- The title must be natural, friendly, and in the same language as the input.\n- If the input is a direct question to the model, you may add an emoji at the end.\n\n3. Output Format\nReturn only a valid JSON object with these exact keys and no additional text:\n{\n "Language Type": "",\n "Your Reasoning": "",\n "Your Output": "<Intention + Subject>"\n}\n\nUser Input:\ndear team please help cancle this booking *************** due to pax insist wrong date, thankyou\n"}]'

output: [{"role":"assistant","content":"Cancel Booking Request"}]
is_deleted: 0,
................
}

Query trace info by trace id in the same project:
SQL :select * from traces where project_id='cmi2u0gm10006zo071v1908ag' and id='399591c7-e685-4b86-b6f1-a10e10343884'

Result: Nothing found