LLM-as-a-Judge: Categorical and Boolean Scores

[jannikmaierhoefer]

Describe the feature or potential improvement

Currently, LLM-as-a-Judge evaluations in Langfuse can only create numerical scores.

It would be helpful if this feature could also create categorical or boolean scores (e.g. for intent classification).

Additional information

No response

[tangochris]

Yes. As an example use case, in our legal information chatbot, we would like to use LLM-as-judge to categorize (or "score") a trace into one of these topics:

[
"Business & non-profits",
"Cars & getting around",
"Consumer",
"Crime",
"Families & children",
"Health",
"Home & neighbours",
"Money & debt",
"Plan for your future care",
"Resolving disputes",
"Rights & citizenship",
"Wills & estates",
"Work"
]

[marcklingen]

thanks for sharing!

[bmeetzejanuary]

+1 this - This would allow us to categorize contact types more easily, and I believe allow us to run different evaluations on different categorizations automatically. Given this has been on the roadmap since January, is there a possible timing update?

[Qanpi]

Agreed. This doesn't seem like a huge burden since score configs for categorical/boolean already exist, so it would be a question of supporting that as structured output from the LLMJ model.