How to customize the callback handler when multiple llm calls are being made in a langgraph state?

[dahnny012]

Describe your question

Hello

I have a use case where I have a streaming LLM call. On some X amount of chunks I run it through a LLM Moderation call such that it looks like the the following

def call_model()
   streaming_response = call_main_model() # Uses langchain's `ChatBedrockConverse` 
   call_moderation_model(streaming_response)  # Also uses langchain's `ChatBedrockConverse` 


def call_moderation_model(streaming_response):
    with span. start_as_current_span("critic.orchestrator.evaluate"):
           ChatBedrockConverse(streaming_response)

This is a state in langgraph and I use the callback handler to automatically observe it. The issue is that the traces now look like this.

• I have 3 moderation models running creating 3 bedrock spans. Is there a way I can customize it in such a way I can customize the name of the bedrock span?

• The bedrock calls appear outside of the critic.orchestrator.evaluate even though they were executed within the span. How do i make it so its a sub span of the critic.orchestrator.evaluate call.

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

3.131

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Based on the available sources, there are known context propagation issues with the CallbackHandler in LangGraph, particularly when trying to nest observations properly(1)(2)(3).

Issue with Nesting

The LangChain CallbackHandler uses start_span instead of start_as_current_span, which means it doesn't register observations in the current context(2)(3). This is why your Bedrock calls appear outside of your critic.orchestrator.evaluate span even though they're executed within it(3).

Recommended Approach

The recommended way to ensure correct span context and nesting is to use the @observe decorator or the langfuse.start_as_current_span() context manager, which leverage OpenTelemetry's automatic context propagation(3).

Here's how you could restructure your code:

from langfuse import observe

@observe()
def call_moderation_model(streaming_response):
    # This will automatically create a span and set it as current
    ChatBedrockConverse(streaming_response)

(3)

Alternative: Explicit Context Passing

If you need to work within the current structure, you can explicitly pass trace_id and parent_observation_id to ensure correct nesting(3):

from langfuse import get_client

langfuse = get_client()

def call_moderation_model(streaming_response):
    with langfuse.start_as_current_span(name="critic.orchestrator.evaluate") as span:
        # Initialize handler within the span context
        langfuse_handler = CallbackHandler()
        
        # Your LangChain call will now be nested under this span
        ChatBedrockConverse(
            streaming_response,
            callbacks=[langfuse_handler]
        )

(4)

Known Limitations

The current CallbackHandler implementation has limitations with context propagation in concurrent or complex scenarios, which is what you're experiencing(2)(3). The handler's use of start_span is a deliberate choice to avoid breaking context in multi-threaded or async environments, but it means you can't reliably access the current observation context everywhere(2)(3).

For customizing span names, the sources don't provide a direct method when using the CallbackHandler with LangChain models. The span names are typically derived from the LangChain component names automatically(4).

📚 Sources:

1. bug: LangGraph CallbackHandler does not link observations to managed prompts
2. Let LangchainCallbackHandler to register each observation as current
3. Let LangchainCallbackHandler to register each observation as current
4. LangChain & LangGraph Integration

---

Have another question? Just tag @inkeep.

[dahnny012]

@inkeep

i added @observe to the call_moderation_model call and it did not work as expected. For reference im running langfuse 3.9 of the sdk