Skip to content

openai_model returns HTTP 500 for Responses API hosted-tool outputs #2436

Closed

Description

@Atharva-Kanherkar

Describe the bug

responses_api_models/openai_model turns a successful upstream OpenAI Responses API response into HTTP 500 when the response contains a hosted-tool output item such as web_search_call.

The upstream response is valid according to the repository's pinned OpenAI SDK (openai==2.7.2). Gym rejects the same payload during NeMoGymResponse validation.

Tested against upstream/main at de2dbda81068a6d43d188b46cc2f4528d57d5f37.

Steps to reproduce

Set OPENAI_API_KEY in the environment, then run this from the Gym repository:

import os

from openai import OpenAI
from openai.types.responses.response_output_item import ResponseOutputItem
from pydantic import TypeAdapter, ValidationError

from nemo_gym.openai_utils import NeMoGymResponse

response = OpenAI(api_key=os.environ["OPENAI_API_KEY"]).responses.create(
    model="gpt-5.4-mini",
    input=(
        "Use web search to find the official OpenAI homepage domain. "
        "Return only the domain."
    ),
    tools=[{"type": "web_search"}],
    tool_choice="required",
    reasoning={"effort": "low"},
    max_output_tokens=1024,
)
payload = response.model_dump(mode="json")

print([item["type"] for item in payload["output"]])
for item in payload["output"]:
    TypeAdapter(ResponseOutputItem).validate_python(item)
print("OpenAI SDK validation: passed")

try:
    NeMoGymResponse.model_validate(payload)
except ValidationError as exc:
    print(f"Gym validation errors: {len(exc.errors())}")
    raise

Observed output:

['web_search_call', 'message']
OpenAI SDK validation: passed
Gym validation errors: 53

A live end-to-end request through SimpleModelServer with a forced web search returns:

POST /v1/responses -> 500 Internal Server Error

Live compatibility matrix

Every response below completed successfully upstream and passed the pinned SDK's ResponseOutputItem validation.

Model Live output item types Gym validation Exact-payload endpoint replay
gpt-4.1-mini web_search_call, message 53 errors HTTP 500
gpt-5.4-mini web_search_call, message 53 errors HTTP 500
gpt-5.6-luna web_search_call, reasoning, web_search_call, message 106 errors HTTP 500
gpt-5.6-terra web_search_call, message 53 errors HTTP 500

For gpt-4.1-mini, the HTTP 500 was additionally reproduced fully end-to-end through Gym's real upstream client, rather than only replaying the exact live payload.

Expected behavior

Gym's OpenAI model proxy should accept and return valid Responses API output item variants emitted by its pinned SDK. A successful hosted-tool response should not become an internal server error.

Configuration

openai_base_url: https://api.openai.com/v1
openai_model: gpt-5.4-mini

The same failure occurs with the other models listed above.

Environment

  • macOS 26.2
  • Python 3.13.14
  • NeMo Gym de2dbda81068a6d43d188b46cc2f4528d57d5f37
  • openai==2.7.2
  • pydantic==2.13.4
  • fastapi==0.135.2

Root cause

The output type is currently aliased to the narrower input union:

NeMoGymResponseOutputItem = NeMoGymResponseInputItem

The OpenAI model adapter then validates the upstream payload.

The pinned SDK's output union includes output-only variants absent from NeMoGymResponseInputItem, including web/file search, computer, image generation, code interpreter, local shell, and custom tool calls.

Only web_search_call was exercised live in this report; the other missing variants are based on schema comparison.

Activity

  1. added a commit that references this issue on Aug 8, 2026
    98e597f
  2. added a commit that references this issue on Aug 10, 2026
    16ac038
  3. added a commit that references this issue on Aug 10, 2026
    88d1692
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    community-requestIssue reported or requested by someone from the community

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions