Support Ollama API in langfuse playground (and LLM API / Gateway)

[LiveOverflow]

Describe the feature or potential improvement

I would like to see the Ollama API added to the supported LLM API / Gateway. This would allow users to use the langfuse playground and LLM-as-a-judge with local hosted models.

Business case:
I am currently experimenting with sensitive user data and confidential documents, thus I don't want to use cloud services at the moment. Once going to production, GDPR compliant DPAs can be setup. But during local experimentation and local evaluation for a use-cases, it would be great to support locally hosted models in langfuse.

Why Ollama?
From what I can tell, Ollama is one of the most common ways to run local models and is largely also compatible with the OpenAI API. The other one is LM Studio which would also be a good choice and has OpenAI API compatibility.

Due to the OpenAPI compatibilities, I would expect integration can be based on the existing OpenAI code.

Additional information

No response

[marcklingen]

This works, you can just select the openai adapter when adding a model and point it to your ollama endpoint. Let me know in case you run into issues while setting this up.

[MathiasBoisgardDev]

Glad to hear that you solved it too :) Happy testing !

[ianferreira]

Hi there, I am unable to add ollama using above. I didn't try the extra-hosts. I have ollama in one session and docker-compose in the other. The ollama is not in the same docker compose so tried localhost:11434. Keep getting cannot connect.

[ianferreira]

figured it out

[MathiasBoisgardDev]

Sorry, just saw you previous comment with this answer, nice that you figured it out! :) what was the problem?

[4l1fe]

I'd like to improve the solution as am dealing with the same issue. One can struggle with "connection refused error" while requesting Ollama's API from a docker compose container(that's what LangFuse does when you press the button Create Connection) after adding extra_hosts as @fitlemon mentioned.

You have to listen requests by ollama serve on host broadcast address 0.0.0.0

Generally, the issue is connection between the docker container service and host service.

[firasarfa]

@ianferreira Could you please share what you did in order to solve . I am facing the same problem . I want to locally evaluate MLLMs using Langfuse , Ollama and Docker. I succesfully can trace the generations but nothing more.