Repository navigation
Conversation
# Conflicts: # scenarios/gen-ai/python/agno/openinference/pyproject.toml # scenarios/gen-ai/python/agno/openllmetry/pyproject.toml # scenarios/gen-ai/python/agno/opentelemetry-agno/pyproject.toml
Pull request dashboard statusWaiting on the author · refreshed 2026-10-23 05:45 UTC Move out of draft to request review. Status above doesn't look right?
|
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Cross-framework execution and report reproducibility need integration confirmation, and both DSPy environments need the LiteLLM configuration correction.
1 open issue
What changed in this PR
Adds DSPy to the GenAI conformance suite with OpenTelemetry and OpenInference implementations, building on the retrieval coverage from #372.
Changes:
- Adds DSPy agent, automatic tool-calling, and in-memory retrieval scenarios with recorded results.
- Extends the mock server to return DSPy ChatAdapter fields, with four focused tests.
- Includes prerequisite retrieval, workflow, and conversation-ID coverage for existing frameworks.
| File | Description |
|---|---|
| tools/gen-ai/mock-server/tests/test_mock_server.py | Tests DSPy response formatting and tool selection. |
| tools/gen-ai/mock-server/src/genai_mock_server/openai.py | Adds ChatAdapter-compatible mock responses. |
| scenarios/gen-ai/README.md | Documents retrieval and DSPy coverage. |
| scenarios/gen-ai/python/llama-index/scenarios/retrieval.py | Adds in-memory vector retrieval. |
| scenarios/gen-ai/python/llama-index/openinference/uv.lock | Locks the embedding dependency. |
| scenarios/gen-ai/python/llama-index/openinference/pyproject.toml | Adds OpenAI embeddings support. |
| scenarios/gen-ai/python/llama-index/openinference/data.json | Records embedding and retrieval results. |
| scenarios/gen-ai/python/llama-index/openinference/conformance.yaml | Registers retrieval execution. |
| scenarios/gen-ai/python/llama-index/native/uv.lock | Locks the embedding dependency. |
| scenarios/gen-ai/python/llama-index/native/retrieval.py | Initializes telemetry for shared retrieval. |
| scenarios/gen-ai/python/llama-index/native/pyproject.toml | Adds OpenAI embeddings support. |
| scenarios/gen-ai/python/llama-index/native/data.json | Records retrieval-run findings. |
| scenarios/gen-ai/python/llama-index/native/conformance.yaml | Registers retrieval execution. |
| scenarios/gen-ai/python/langchain/scenarios/retrieval.py | Adds in-memory vector retrieval. |
| scenarios/gen-ai/python/langchain/scenarios/invoke_agent.py | Adds a checkpointed conversation thread. |
| scenarios/gen-ai/python/langchain/opentelemetry-langchain/uv.lock | Locks checkpoint and scoring dependencies. |
| scenarios/gen-ai/python/langchain/opentelemetry-langchain/pyproject.toml | Adds checkpoint and scoring dependencies. |
| scenarios/gen-ai/python/langchain/opentelemetry-langchain/data.json | Records conversation and retrieval coverage. |
| scenarios/gen-ai/python/langchain/opentelemetry-langchain/conformance.yaml | Registers retrieval execution. |
| scenarios/gen-ai/python/langchain/openllmetry/uv.lock | Locks checkpoint and scoring dependencies. |
| scenarios/gen-ai/python/langchain/openllmetry/pyproject.toml | Adds checkpoint and scoring dependencies. |
| scenarios/gen-ai/python/langchain/openllmetry/data.json | Records conversation and retrieval results. |
| scenarios/gen-ai/python/langchain/openllmetry/conformance.yaml | Registers retrieval execution. |
| scenarios/gen-ai/python/langchain/openinference/uv.lock | Locks checkpoint and scoring dependencies. |
| scenarios/gen-ai/python/langchain/openinference/pyproject.toml | Adds checkpoint and scoring dependencies. |
| scenarios/gen-ai/python/langchain/openinference/data.json | Records conversation and retrieval results. |
| scenarios/gen-ai/python/langchain/openinference/conformance.yaml | Registers retrieval execution. |
| scenarios/gen-ai/python/dspy/scenarios/retrieval.py | Adds deterministic in-memory retrieval. |
| scenarios/gen-ai/python/dspy/scenarios/invoke_agent.py | Runs ReAct without caller-provided tools. |
| scenarios/gen-ai/python/dspy/scenarios/automatic_tool_calling.py | Runs ReAct with a weather tool. |
| scenarios/gen-ai/python/dspy/opentelemetry-dspy/pyproject.toml | Defines the OpenTelemetry environment. |
| scenarios/gen-ai/python/dspy/opentelemetry-dspy/data.json | Records OpenTelemetry conformance results. |
| scenarios/gen-ai/python/dspy/opentelemetry-dspy/conformance.yaml | Configures three DSPy scenarios. |
| scenarios/gen-ai/python/dspy/openinference/pyproject.toml | Defines the OpenInference environment. |
| scenarios/gen-ai/python/dspy/openinference/data.json | Records OpenInference conformance results. |
| scenarios/gen-ai/python/dspy/openinference/conformance.yaml | Configures three DSPy scenarios. |
| scenarios/gen-ai/python/agno/scenarios/workflow.py | Adds a model-calling workflow step. |
| scenarios/gen-ai/python/agno/scenarios/retrieval.py | Adds local Qdrant knowledge retrieval. |
| scenarios/gen-ai/python/agno/opentelemetry-agno/pyproject.toml | Adds the Qdrant dependency. |
| scenarios/gen-ai/python/agno/opentelemetry-agno/data.json | Records workflow and retrieval coverage. |
| scenarios/gen-ai/python/agno/opentelemetry-agno/conformance.yaml | Registers workflow and retrieval scenarios. |
| scenarios/gen-ai/python/agno/openllmetry/pyproject.toml | Adds the Qdrant dependency. |
| scenarios/gen-ai/python/agno/openllmetry/conformance.yaml | Registers workflow and retrieval scenarios. |
| scenarios/gen-ai/python/agno/openinference/uv.lock | Locks Qdrant and supporting dependencies. |
| scenarios/gen-ai/python/agno/openinference/pyproject.toml | Adds the Qdrant dependency. |
| scenarios/gen-ai/python/agno/openinference/data.json | Records workflow findings. |
| scenarios/gen-ai/python/agno/openinference/conformance.yaml | Registers workflow and retrieval scenarios. |
Review effort: Balanced
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #372 and is based on it. Only the last commit belongs to this PR.
Adds
scenarios/gen-ai/python/dspywithopentelemetry-dspy(opentelemetry-instrumentation-genai-dspy==1.2b0) andopeninference(openinference-instrumentation-dspy==0.1.48), covering invoke_agent, automatic_tool_calling and retrieval. There is no OpenLLMetry package for dspy.invoke_agent is
dspy.ReActwith no tools, which still runs ReAct's built-infinishstep. retrieval isdspy.Retrieve(k=2)over an in-memory RM with no LLM call.dspy's ChatAdapter expects each output field back between
[[ ## field ## ]]markers, and the mock replied with plain text, so both agent runs failed to parse. The mock now answers in that format when the request carries ChatAdapter's closing instruction. ReAct picks the first tool, thenfinishonce an observation is in the trajectory. Four tests cover it.opentelemetry-dspyemits agent, tool and retrieval spans and leaves model calls to other instrumentations. The agent name is hardcoded todspy.ReAct. The retrieval span has data source, query andtop_k, but documents carry no id or score becauseRetrievereturns text only. OpenInference retrieval spans have no query,top_kor data source, and its chat spans have no usage or response model.