JsonPath in LLM as a Judge returns ["value"] instead of value when using a dataset

[misterbridge]

Describe your question

Hi team,

I’m seeing unexpected behavior when using LLM as a Judge with a dataset as input.

Goal

I want to inject a field from my dataset input JSON directly into the evaluator prompt.
The example below is truncated only in its content, not in its structure. The actual JSON has the same top-level structure and the same path to locale:

{
  "locale": "fr_FR",
  "project": ""
}

The variable {{locale}} should be replaced by:

fr_FR

Mapping configuration

In the evaluator’s Map Variables section:

• Object: Dataset item
• Object variable: Input
• JsonPath: locale

I also tried (from discussions with Langfuse AI):

• $.locale
• $['locale']

Actual result

Instead of inserting the raw value, Langfuse injects it with unexpected brackets and quotes:

based on the language from locale: "["fr_FR"]"

The expected output is:

based on the language from locale: "fr_FR"

So the variable becomes ["fr_FR"] instead of fr_FR.

Question

Is this a known issue with JsonPath resolution for datasets in LLM as a Judge?
And is there a way to force Langfuse to return the raw string value (fr_FR) instead of the wrapped form (["fr_FR"])?

Happy to provide a minimal dataset sample if needed.

Thanks!

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

v3.134.0

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Based on the available documentation, the specific issue you're describing—where JsonPath returns ["fr_FR"] instead of fr_FR when extracting values from dataset items in LLM-as-a-Judge evaluators—is not explicitly documented(1)(2).

However, there was a known issue with JsonPath extraction in LLM-as-a-Judge evaluators that could cause unexpected behavior, including empty values or incorrect extraction. This was fixed in versions after 3.96.2 OSS(3). Since you're running v3.134.0, you should already have this fix.

For your specific case, here are some troubleshooting steps based on the documentation:

1. Verify your JsonPath expression in the live prompt preview(1). The preview should show exactly how your variables are being populated. If you see ["fr_FR"] in the preview, this confirms the extraction behavior.

2. Try these JsonPath variations:

   • $.locale (standard JsonPath for top-level property)
   • $['locale'] (bracket notation)
   • locale (simple property name)

3. Check your dataset item structure(1). The mapping relies on the data being structured exactly as expected. Ensure your dataset item's input field contains the JSON structure you showed.

If the live prompt preview continues to show the wrapped array format ["fr_FR"], this may indicate either:

• An edge case in JsonPath extraction not covered in the current documentation
• A potential issue with how the evaluator processes dataset item inputs specifically

I'd recommend reaching out to Langfuse support with your specific dataset structure and evaluator configuration, as this behavior isn't documented in the official troubleshooting guides(2).

📚 Sources:

1. LLM-as-a-Judge
2. Troubleshooting and FAQ
3. LLM as a judge gets empty values

---

Have another question? Just tag @inkeep.

[misterbridge]

Previsualisation does not work on dataset, only on traces (or at least I couldn't make it work). (I'm working on datasets)
I already tried the variations without success.
I gave you the actual structure, which should map my mapping.