High Inode Usage on MinIO Backend (~8.6M files/day) Leading to Rapid Filesystem Exhaustion

[weicheng59]

GitHub Issue Title: High Inode Usage on MinIO Backend (~8.6M files/day) Leading to Rapid Filesystem Exhaustion

Body:

Hello Langfuse Team,

I am experiencing an issue where my Langfuse instance is generating an extremely high number of objects in its MinIO backend, leading to rapid inode exhaustion on the filesystem. I'm hoping you can help me understand the source of this behavior.

Background & Context

1. Initial Problem: I was initially running the standard Docker Compose setup on a server with an ext4 root filesystem. After a few weeks of use, my server's root partition ran out of inodes (IUse% was over 43%), caused by a massive number of small files in the langfuse_minio_data Docker volume.

2. Migration to XFS: To solve the inode exhaustion problem, I migrated the MinIO backend to a dedicated 2TB XFS filesystem. I created a 2TB file on a separate drive, formatted it with XFS, and mounted it via a loopback device at /data/minio. I then updated my docker-compose.yaml to use this as a bind mount for MinIO's data directory.

3. Clean Start: The data migration was taking too long, so I opted for a "clean start". I wiped the new XFS volume and started the Langfuse stack fresh, pointing to the empty MinIO volume.

The Current Problem: Unsustainable Inode Growth

After starting fresh, the new XFS volume is filling up with inodes at an alarming and unsustainable rate.

Here is the inode usage from df -i over the last 12 days:

On August 13, 2025 (2 days after clean start):

Filesystem      Inodes    IUsed     IFree IUse% Mounted on
/dev/loop7     214748352 15643535 199104817    8% /data/minio

On August 25, 2025 (Today):

Filesystem      Inodes     IUsed      IFree IUse% Mounted on
/dev/loop7     214748352 119163214   95585138   56% /data/minio

Analysis:

• Inodes added in 12 days: ~103.5 million
• Average daily rate: ~8.6 million new inodes (files) per day.
• Projected runway: At this rate, the remaining ~95 million inodes will be exhausted in approximately 11 days.

My Expectation vs. Reality

My understanding is that MinIO is primarily used for storing multimedia uploads (images, etc.) associated with traces. These are typically larger files, so I was very surprised to see the inode count (the number of files) become the bottleneck again, especially at this extreme rate. This behavior seems inconsistent with storing only multimedia content.

What I've Tried

• I have checked the langfuse-worker and langfuse-web logs using docker-compose logs, and there are no obvious repeating errors, stack traces, or indications of a runaway process loop.
• I have confirmed using sudo du --inodes that the vast majority of the files are being created within the langfuse bucket directory on the filesystem (/data/minio/langfuse).
• I am aware that a paid data retention policy would be a long-term solution, but my immediate concern is understanding the source of this massive object creation, as it seems anomalous.

My Questions for the Team

1. Could you please help me understand what process within Langfuse would generate such a massive number of small objects in MinIO?
2. Is this volume of object creation (~8.6 million/day) expected for a certain type of workload? I am writing about 0.25m traces each day. each trace may come with about 10 spans and scores.
3. Are there any specific configurations, trace types, or features (aside from paid data retention) that are known to cause this behavior?

Thank you for your time and for this fantastic tool. Any insight you can provide would be greatly appreciated.

System Details

• Langfuse Version: 3.97.2
• Deployment: Docker Compose on Ubuntu 22.04 LTS
• Filesystem: MinIO backend on XFS, hosted on an ext4 partition via a loopback device.
• docker-compose.yaml: Standard file from the documentation, with the MinIO volume modified to be a bind mount (- /data/minio:/data).

[maxdeichmann]

Hi @weicheng59, thanks for raising this. our current ingestion pipeline is heavily using S3 under the hood. I would recommend to:

• use langfuse cloud to make operations easier
• use AWS S3 APIs instead of minio
• move to the new JS Python SDKs. S3 usage rapidly decreases with these new SDKs.

[weicheng59]

Thank you for the response. I have migrated my application to the new Python SDK, but my S3 object creation is still very high.

Questions

• Is there a specific configuration or feature flag required to enable the new, more efficient storage mechanism in the latest SDK?
• Could you provide an example of how to correctly instrument a trace with the new SDK to ensure minimal S3 object creation?

Current Implementation

Currently, I am using the following way to create a trace:

langfuse = Langfuse(
    public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
    secret_key=os.environ["LANGFUSE_SECRET_KEY"],
    host="http://192.168.20.236:3000",
    environment="production",
    tracing_enabled=True
)

# This first part seems to be separate from the main logic below
trace_id = langfuse.create_trace_id(seed=request_id)
langfuse.start_span(name="final", trace_context={"trace_id": trace_id})
span.update_trace(name=request_id, input=langfuse_input, output=response_text)
span.end()


# Main tracing logic
trace_id = langfuse.create_trace_id(seed=request_id)

with langfuse.start_as_current_span(name="before_stage1", input=langfuse_input, trace_context={"trace_id": trace_id}) as before_stage1_span:
    before_stage1_span.update_trace(
        name=str(request_id),
        user_id=json_data["tdxid"],
        session_id=str(request_id),
        metadata={
            "input": langfuse_input,
            "knowledge_type": json_data["knowledge_type"],
            "screenType": json_data["screenType"],
            "ip": json_data["ip"]
        }
    )
    before_stage1_span.update(output=stage1_prompt["prompt"])

    with langfuse.start_as_current_generation(name="stage1", model=stage1_prompt.get('model'), input=[{"role": "user", "content": stage1_prompt['prompt']}], trace_context={"trace_id": trace_id}) as stage1_span:
        res_str = await stage1_llm_call()
        stage1_span.update(
            output=json.loads(res_str)["choices"][0]['text'],
            usage=json.loads(res_str).get("usage", {})
        )

    with langfuse.start_as_current_span(name="after_stage1", input=json.loads(res_str)["choices"][0]['text'], trace_context={"trace_id": trace_id}) as after_stage1_span:
        outputs1 = await after_stage1_wrapper()
        after_stage1_span.update(output=outputs1)
        
        trace_id = langfuse.create_trace_id(seed=request_id)

[weicheng59]

In our case, we didn't observe S3 API usage drop by observing the inodes usage in minio. We end up applying a policy in minio to delete objects that are longer than 5 days. It works for us and no traces was lost from the first look.
Thanks for the reply and building such a great project.

[maxdeichmann]

@weicheng59, thanks so much for letting us know! In this case you could end up with inconsistent updates if you try to update a trace that is > 5 days old. As this is not the default behavior, i expect no issues here for you.

[iRyanRib]

@weicheng59 How you apply this policy ? Im facing the same problem in minio ..... i try some tricks but without success, i really appreciate if you share some details, thank you !

[weicheng59]

This is what worked for us.

Create a file lifecycle.json:

{
  "Rules": [
    {
      "ID": "expire-objects-after-5-days",
      "Status": "Enabled",
      "Filter": {
        "Prefix": ""
      },
      "Expiration": {
        "Days": 5
      }
    }
  ]
}

Apply the policy:

Bash

~/minio-binaries/mc ilm import local/langfuse < lifecycle.json