bug: LangGraph Playground Disabled With tool calls

[takefy-dev]

Describe the bug

Langfuse expect ML format for the playground but the problem is that langchain provides ML Format but not langgraph for tool calls.
LangChain Exemple:
[ { "content": "Can you multiply 8 by 9 and then tell me the weather in London?", "role": "user" }, { "content": "", "role": "assistant" }, { "content": "Interesting! Given both the calculation result and London weather, what would you recommend for outdoor activities?", "role": "user" } ]
{ "content": "", "role": "assistant", "additional_kwargs": { "tool_calls": [ { "index": 0, "id": "tool_0_calculator", "type": "function", "function": { "name": "calculator", "arguments": { "input": { "operation": "*", "a": 8, "b": 9 } } } }, { "index": 1, "id": "tool_1_get_weather", "type": "function", "function": { "name": "get_weather", "arguments": { "input": { "city": "London" } } } } ] } }
LangChain Ouput:
Just normal ML Format

LangGraph Trace:
[ { "content": "Can you multiply 8 by 9 and then tell me the weather in London?", "role": "user" }, { "content": "", "role": "assistant", "additional_kwargs": { "tool_calls": [ { "index": 0, "id": "tool_0_calculator", "type": "function", "function": { "name": "calculator", "arguments": { "input": { "operation": "*", "a": 8, "b": 9 } } } }, { "index": 1, "id": "tool_1_get_weather", "type": "function", "function": { "name": "get_weather", "arguments": { "input": { "city": "London" } } } } ] } }, { "content": "Tools output: Unknown operation, this message has already been sent to the user, never send it again.", "additional_kwargs": {}, "role": "calculator" }, { "content": "Tools output: Weather in London: Cloudy, 15°C, this message has already been sent to the user, never send it again.", "additional_kwargs": {}, "role": "get_weather" }, { "content": "8 multiplied by 9 is 72. The weather in London is Cloudy, 15°C.\n", "role": "assistant" }, { "content": "Interesting! Given both the calculation result and London weather, what would you recommend for outdoor activities?", "role": "user" } ]
You can clearly see that the role takes the tool call name, and this cause problem because we can't open the playground

To reproduce

import { ChatOpenAI } from '@langchain/openai';
import { DynamicTool } from '@langchain/core/tools';
import { CallbackHandler } from 'langfuse-langchain';
import { StateGraph, START, END } from '@langchain/langgraph';
import { ToolNode } from '@langchain/langgraph/prebuilt';
import { AIMessage, BaseMessage, HumanMessage } from '@langchain/core/messages';
import { Annotation } from '@langchain/langgraph';

// Define a simple calculator tool
const calculatorTool = new DynamicTool({
  name: 'calculator',
  description:
    'Performs basic arithmetic operations. Input should be a JSON string with operation, a, and b fields.',
  func: async (input: string) => {
    try {
      const { operation, a, b } = JSON.parse(input);
      switch (operation) {
        case 'add':
          return `${a} + ${b} = ${a + b}`;
        case 'subtract':
          return `${a} - ${b} = ${a - b}`;
        case 'multiply':
          return `${a} * ${b} = ${a * b}`;
        case 'divide':
          if (b === 0) return 'Error: Division by zero';
          return `${a} / ${b} = ${a / b}`;
        default:
          return 'Unknown operation';
      }
    } catch {
      return 'Error: Invalid input format. Expected JSON with operation, a, and b fields.';
    }
  },
});

// Define a mock weather tool
const weatherTool = new DynamicTool({
  name: 'get_weather',
  description:
    'Gets the current weather for a given city. Input should be a JSON string with a city field.',
  func: async (input: string) => {
    try {
      const { city } = JSON.parse(input);
      // Mock weather data
      const mockWeather: Record<string, string> = {
        'new york': 'Sunny, 72°F',
        london: 'Cloudy, 15°C',
        tokyo: 'Rainy, 18°C',
        paris: 'Partly cloudy, 20°C',
      };

      const weather = mockWeather[city.toLowerCase()] || 'Weather data not available';
      return `Weather in ${city}: ${weather}`;
    } catch {
      return 'Error: Invalid input format. Expected JSON with a city field.';
    }
  },
});

// Define the graph state
const GraphState = Annotation.Root({
  messages: Annotation<BaseMessage[]>({
    reducer: (x, y) => x.concat(y),
  }),
});

// Agent node that calls the model
async function agentNode(state: typeof GraphState.State) {
  const model = new ChatOpenAI({
    model: 'google/gemini-2.0-flash-001',
    temperature: 0.2,
    configuration: {
      baseURL: 'https://openrouter.ai/api/v1',
      apiKey: process.env.OPENROUTER_API_KEY,
    },
  }).bindTools([calculatorTool, weatherTool]);

  const response = await model.invoke(state.messages);
  return { messages: [response] };
}

// Tool node that executes tools
async function toolNode(state: typeof GraphState.State) {
  const toolNodeInstance = new ToolNode([calculatorTool, weatherTool]);
  const result = await toolNodeInstance.invoke(state);
  
  // Log tool execution details (similar to your production code)
  console.log('\n🔧 Tool Execution Results:');
  for (const message of result.messages as BaseMessage[]) {
    if (message.getType() === "tool") {
      console.log(`  - Tool: ${message.name}`);
      console.log(`  - Content: ${message.content}`);
      console.log(`  - Role: ${message.getType()}`);
      console.log(`  - Message Name: ${(message as { name?: string }).name || 'unknown'}`);
      
      // Simulate the content modification like in your production code
    //   message.content = `Tools output: ${message.content}, this message has already been sent to the user, never send it again.`;
    }
  }
  
  return result;
}

// Router function to decide next step
function shouldContinue(state: typeof GraphState.State) {
  const lastMessage = state.messages[state.messages.length - 1] as AIMessage;
  if (lastMessage.tool_calls?.length) {
    return "tools";
  }
  return END;
}

// Create the LangGraph
function createTestGraph() {
  const workflow = new StateGraph(GraphState)
    .addNode("agent", agentNode)
    .addNode("tools", toolNode)
    .addEdge(START, "agent")
    .addConditionalEdges("agent", shouldContinue, {
      tools: "tools",
      [END]: END,
    })
    .addEdge("tools", "agent");

  return workflow.compile();
}

async function testLangChainTools() {
  console.log('🚀 Testing LangGraph Tool Calls with Langfuse Tracing\n');

  // Validate required environment variables
  const requiredEnvVars = {
    OPENROUTER_API_KEY: process.env.OPENROUTER_API_KEY,
    LANGFUSE_PUBLIC_KEY: process.env.LANGFUSE_PUBLIC_KEY,
    LANGFUSE_SECRET_KEY: process.env.LANGFUSE_SECRET_KEY,
  };

  const missingVars = Object.entries(requiredEnvVars)
    .filter(([_, value]) => !value)
    .map(([key]) => key);

  if (missingVars.length > 0) {
    console.error('❌ Missing required environment variables:');
    missingVars.forEach(varName => console.error(`   - ${varName}`));
    console.error('\nPlease set these variables in your .env.local file or environment.');
    console.error('\nExample:');
    console.error('OPENROUTER_API_KEY=your_openrouter_api_key_here');
    console.error('LANGFUSE_PUBLIC_KEY=your_langfuse_public_key_here');
    console.error('LANGFUSE_SECRET_KEY=your_langfuse_secret_key_here');
    console.error('LANGFUSE_HOST=https://cloud.langfuse.com');
    return;
  }

  // Initialize Langfuse callback handler
  const langfuseHandler = new CallbackHandler({
    publicKey: process.env.LANGFUSE_PUBLIC_KEY,
    secretKey: process.env.LANGFUSE_SECRET_KEY,
    baseUrl: process.env.LANGFUSE_HOST || 'https://cloud.langfuse.com',
    sessionId: `langgraph-tools-test-${Date.now()}`,
    userId: 'test-user',
  });

  // Create the LangGraph
  const graph = createTestGraph();

  // Test cases with follow-up discussions
  const testCases = [
    {
      initial: 'What is 15 + 27?',
      followUp: 'Great! Now can you multiply that result by 3?'
    },
    {
      initial: 'Calculate 144 divided by 12',
      followUp: 'Perfect! Can you tell me what that result plus 5 equals?'
    },
    {
      initial: "What's the weather like in New York?",
      followUp: 'Thanks! Based on that weather, should I bring an umbrella?'
    },
    {
      initial: 'Can you multiply 8 by 9 and then tell me the weather in London?',
      followUp: 'Interesting! Given both the calculation result and London weather, what would you recommend for outdoor activities?'
    },
  ];

  for (const [index, testCase] of testCases.entries()) {
    console.log(`\n📝 Test ${index + 1}: ${testCase.initial}`);
    console.log('─'.repeat(70));

    try {
      // Create a new trace for each test case
      const traceHandler = new CallbackHandler({
        publicKey: process.env.LANGFUSE_PUBLIC_KEY,
        secretKey: process.env.LANGFUSE_SECRET_KEY,
        baseUrl: process.env.LANGFUSE_HOST || 'https://cloud.langfuse.com',
        sessionId: `langgraph-tools-test-${Date.now()}-${index}`,
        userId: 'test-user',
        metadata: {
          testCase: index + 1,
          initialQuestion: testCase.initial,
          followUpQuestion: testCase.followUp,
          timestamp: new Date().toISOString(),
        },
        tags: ['langgraph-test', 'tool-calling', 'gemini-2.0', 'discussion-flow'],
      });

      // PART 1: Initial tool call using LangGraph
      console.log('🔵 Initial Request:', testCase.initial);
      
      const initialState = {
        messages: [new HumanMessage(testCase.initial)]
      };

      const result = await graph.invoke(initialState, {
        callbacks: [traceHandler],
      });

      console.log('\n📊 Graph Execution Complete');
      console.log('📝 Final Messages:');
      
             result.messages.forEach((msg: BaseMessage, i: number) => {
         console.log(`  ${i + 1}. ${msg.getType()}: ${msg.content}`);
         if (msg.getType() === "tool") {
           console.log(`     - Tool Name: ${(msg as { name?: string }).name || 'unknown'}`);
           console.log(`     - Tool Role: ${msg.getType()}`);
         }
       });

      // PART 2: Follow-up discussion
      console.log('\n🟡 Follow-up Question:', testCase.followUp);
      
      const followUpState = {
        messages: [...result.messages, new HumanMessage(testCase.followUp)]
      };

      const followUpResult = await graph.invoke(followUpState, {
        callbacks: [traceHandler],
      });

      console.log('\n📊 Follow-up Graph Execution Complete');
      console.log('📝 New Messages:');
      
      const newMessages = followUpResult.messages.slice(result.messages.length);
      newMessages.forEach((msg: BaseMessage, i: number) => {
        console.log(`  ${i + 1}. ${msg.getType()}: ${msg.content}`);
        if (msg.getType() === "tool") {
          console.log(`     - Tool Name: ${(msg as { name?: string }).name || 'unknown'}`);
          console.log(`     - Tool Role: ${msg.getType()}`);
        }
      });

      // Flush the trace to Langfuse
      await traceHandler.shutdownAsync();
      
    } catch (error) {
      console.error('❌ Error:', error.message);
      console.error('Stack:', error.stack);
    }
  }

  // Final flush to ensure all traces are sent
  await langfuseHandler.shutdownAsync();
  console.log('\n✅ All traces sent to Langfuse!');
}

// Run the test if this file is executed directly
testLangChainTools().catch(console.error);



### SDK and container versions

 "@langchain/core": "^0.3.39",
    "@langchain/langgraph": "^0.2.45",
    "@langchain/langgraph-checkpoint-postgres": "^0.0.4",
    "@langchain/openai": "^0.3.17",
  "langchain": "^0.3.19",
    "langfuse-langchain": "^3.37.2",
Langfuse V 3.63.0

### Additional information

This is really frustrating because we can't use the playground to debug tool calls

Are you interested to contribute a fix for this bug?

No

[dosubot]

You've identified a real incompatibility between LangGraph's tool call message format and what Langfuse Playground expects.

What's happening:
Langfuse expects messages to follow a ChatML-like schema, where each message has a role (such as "assistant", "user", or "tool") and content. For tool calls, the assistant's message should have role: "assistant" and include tool calls in a structured way (typically in an additional_kwargs or tool_calls field). Tool results should have role: "tool" and reference the tool call ID. This is the format LangChain outputs and what Langfuse Playground is designed to parse and display correctly [1] [2] [3].

LangGraph, however, emits tool result messages where the role is set to the tool's name (e.g., "calculator" or "get_weather"), not "tool". This breaks the expected schema and prevents the Playground from parsing and displaying the trace properly.

Why this matters:
Langfuse's Playground and trace viewer rely on the role field to distinguish between user, assistant, and tool messages. When the role is set to an arbitrary tool name, the UI cannot recognize or render these messages as tool results, causing the playground to fail to open or display tool calls correctly [2].

What you can do:

• The root cause is that LangGraph (or your integration code) is setting the role to the tool name instead of "tool" for tool result messages.
• To work around this, you would need to post-process your LangGraph trace before sending it to Langfuse, rewriting any message with a tool name as the role to instead use "tool" and include the tool name in a separate field (such as name or tool_name).
• If you have control over the LangGraph tool node output, you could modify it to emit messages with role: "tool" and the correct structure expected by Langfuse.

Reference formats:

• Assistant tool call message:

  {
    "role": "assistant",
    "content": "",
    "tool_calls": [ ... ]
  }

• Tool result message:

  {
    "role": "tool",
    "content": "...",
    "tool_call_id": "tool_0_calculator"
  }

[2]

Why this happens:
Langfuse is permissive and allows any string as a role via the API/SDK, but the Playground and trace viewer expect specific roles to function correctly [2] [4].

No existing fix:
There are currently no existing tickets or discussions about this specific incompatibility in the Langfuse repository.

Summary:
To use the Langfuse Playground with LangGraph traces, ensure that tool result messages use role: "tool" (not the tool name) and follow the expected schema. This will allow the Playground to parse and display tool calls as intended.

If you need a code snippet for post-processing or have further questions about the schema, let me know!

To reply, just mention my name @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

[takefy-dev]

The problem is in the Langfuse integration itself - it's not properly handling the ChatML format for tool messages. When LangGraph serializes a ToolMessage, it's using the tool name as the role instead of "tool".
Here's what's happening:
LangGraph creates a ToolMessage with the correct structure
When sending to Langfuse, the message gets serialized
During serialization, the role gets set to the tool name instead of "tool"

[hassiebp]

Thanks for raising this! We we will take a look

[takefy-dev]

Thanks @hassiebp

[takefy-dev]

Hey any news on this @hassiebp

[Kevinello]

any news? facing the same issue here

[christiannwamba]

This is also an issue with the Vercel AI SDK.

1. When you use a tool call, the Playground button is disabled
2. To walk around, append a message that does not use a tool call and the playground button will be enabled. Even if you used tool call in a previous message, as long as the last message does not have a tool call, the Playground button is enabled

[takefy-dev]

Yeah but this issue should be fixed in langfuse sdk to support everything

[christiannwamba]

100% agree. Just reporting that it ALSO happens with Vercel AI SDK

[LouisHaoSZER]

Any news here？facing the same problem in version 3.75.2

[takefy-dev]

@LouisHaoSZER I've ping them in discord directly witht the github link will see but been a month i've reported it already no one classified it has a bug.