Repository navigation
Description
Describe the bug
responses_api_models/openai_model turns a successful upstream OpenAI Responses API response into HTTP 500 when the response contains a hosted-tool output item such as web_search_call.
The upstream response is valid according to the repository's pinned OpenAI SDK (openai==2.7.2). Gym rejects the same payload during NeMoGymResponse validation.
Tested against upstream/main at de2dbda81068a6d43d188b46cc2f4528d57d5f37.
Steps to reproduce
Set OPENAI_API_KEY in the environment, then run this from the Gym repository:
import os
from openai import OpenAI
from openai.types.responses.response_output_item import ResponseOutputItem
from pydantic import TypeAdapter, ValidationError
from nemo_gym.openai_utils import NeMoGymResponse
response = OpenAI(api_key=os.environ["OPENAI_API_KEY"]).responses.create(
model="gpt-5.4-mini",
input=(
"Use web search to find the official OpenAI homepage domain. "
"Return only the domain."
),
tools=[{"type": "web_search"}],
tool_choice="required",
reasoning={"effort": "low"},
max_output_tokens=1024,
)
payload = response.model_dump(mode="json")
print([item["type"] for item in payload["output"]])
for item in payload["output"]:
TypeAdapter(ResponseOutputItem).validate_python(item)
print("OpenAI SDK validation: passed")
try:
NeMoGymResponse.model_validate(payload)
except ValidationError as exc:
print(f"Gym validation errors: {len(exc.errors())}")
raiseObserved output:
['web_search_call', 'message']
OpenAI SDK validation: passed
Gym validation errors: 53
A live end-to-end request through SimpleModelServer with a forced web search returns:
POST /v1/responses -> 500 Internal Server Error
Live compatibility matrix
Every response below completed successfully upstream and passed the pinned SDK's ResponseOutputItem validation.
| Model | Live output item types | Gym validation | Exact-payload endpoint replay |
|---|---|---|---|
gpt-4.1-mini |
web_search_call, message |
53 errors | HTTP 500 |
gpt-5.4-mini |
web_search_call, message |
53 errors | HTTP 500 |
gpt-5.6-luna |
web_search_call, reasoning, web_search_call, message |
106 errors | HTTP 500 |
gpt-5.6-terra |
web_search_call, message |
53 errors | HTTP 500 |
For gpt-4.1-mini, the HTTP 500 was additionally reproduced fully end-to-end through Gym's real upstream client, rather than only replaying the exact live payload.
Expected behavior
Gym's OpenAI model proxy should accept and return valid Responses API output item variants emitted by its pinned SDK. A successful hosted-tool response should not become an internal server error.
Configuration
openai_base_url: https://api.openai.com/v1
openai_model: gpt-5.4-miniThe same failure occurs with the other models listed above.
Environment
- macOS 26.2
- Python 3.13.14
- NeMo Gym
de2dbda81068a6d43d188b46cc2f4528d57d5f37 openai==2.7.2pydantic==2.13.4fastapi==0.135.2
Root cause
The output type is currently aliased to the narrower input union:
NeMoGymResponseOutputItem = NeMoGymResponseInputItemThe OpenAI model adapter then validates the upstream payload.
The pinned SDK's output union includes output-only variants absent from NeMoGymResponseInputItem, including web/file search, computer, image generation, code interpreter, local shell, and custom tool calls.
Only web_search_call was exercised live in this report; the other missing variants are based on schema comparison.
Activity
fix(openai_utils): accept Responses API output-call items
fix(openai_utils): accept Responses API output-call items
fix(openai_utils): accept hosted-tool responses api output items (#2438)
test(responses): gate the item union and boundary sets against the in…
test(responses): check the item union and boundary sets against the i…
test(responses): check the item union and boundary sets against the i…
test(responses): check the item union and boundary sets against the i…
test(responses): check the item union and boundary sets against the i…
test(responses): check the item union and boundary sets against the i…
test(responses): check the item union and boundary sets against the i…
test(responses): check the item union and boundary sets against the i…
test(responses): check the item union and boundary sets against the i…
test(responses): check the item union and boundary sets against the i…
test(responses): check the item union and boundary sets against the i…