Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions nemo_gym/anthropic_converter.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,11 +41,12 @@

import base64
import binascii
import json
from time import time
from typing import Any, Dict, Iterator, List, Optional
from uuid import uuid4

import orjson

# Types only — never the `anthropic` client. The client uses httpx (O(n^2) connection
# pooling at high concurrency); all transport in Gym stays on aiohttp via server_utils.
# MessageCreateParams (request) is a TypedDict used purely as a hint; NeMoGymAnthropicMessage
Expand Down Expand Up @@ -264,7 +265,7 @@ def _anthropic_content_to_output_items(self, content: List[Dict[str, Any]]) -> L
self._flush_text_output(pending_text, output)
output.append(
NeMoGymResponseFunctionToolCall(
arguments=json.dumps(block.get("input", {})),
arguments=orjson.dumps(block.get("input", {})).decode(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the new serializer rejects tool inputs that stdlib serialized successfully

orjson.dumps() raises JSONEncodeError: Integer exceeds 64-bit range for arbitrary-precision Python integers, while stdlib emits the exact JSON number. For example:

value = {"account_id": 18446744073709551617}
json.dumps(value)
# '{"account_id": 18446744073709551617}'

orjson.dumps(value)
# JSONEncodeError: Integer exceeds 64-bit range

I think this should be addressed through one shared Gym serialization helper. The implementation can try orjson.dumps() and fall back on any orjson.JSONEncodeError to compact stdlib JSON.

def json_dumps_bytes(value: JsonValue, *, sort_keys: bool = False) -> bytes:
    option = orjson.OPT_SORT_KEYS if sort_keys else 0
    try:
        return orjson.dumps(value, option=option)
    except orjson.JSONEncodeError:
        return json.dumps(
            value,
            sort_keys=sort_keys,
            ensure_ascii=False,
            separators=(",", ":"),
            allow_nan=False,
        ).encode("utf-8")

def json_dumps(value: JsonValue, *, sort_keys: bool = False) -> str:
    return json_dumps_bytes(value, sort_keys=sort_keys).decode("utf-8")

there are still some differences:

  • orjson silently serializes NaN and infinities as null vs stdlib normally emits the non-standard NaN/Infinity tokens, while allow_nan=False raises.
  • orjson also natively serializes datetime/dataclass/UUID values that stdlib rejects

call_id=block["id"],
name=block["name"],
id=block["id"],
Expand Down Expand Up @@ -369,7 +370,7 @@ def flush_message() -> None:
flush_message()
items.append(
NeMoGymResponseFunctionToolCall(
arguments=json.dumps(block.get("input", {})),
arguments=orjson.dumps(block.get("input", {})).decode(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — Do not reject previously supported tool inputs during encoding

Failure / blast radius: An Anthropic request with messages=[{"role":"assistant","content":[{"type":"tool_use","id":"t1","name":"lookup","input":{"account_id":18446744073709551617}}]}] now raises orjson.JSONEncodeError: Integer exceeds 64-bit range here. The default Messages endpoint invokes this conversion before model dispatch, so valid history that previously converted now fails. The same limitation affects Anthropic response conversion, _block_deltas, and SSE envelopes containing unconstrained tool schemas.

Correction: The smallest compatibility fix is to keep stdlib encoding at these unconstrained sites, reintroducing import json. In both tool-use → function-call branches use:

arguments=json.dumps(block.get("input", {})),

In _block_deltas restore:

return [{"type": "input_json_delta", "partial_json": json.dumps(block.get("input", {}))}]

The two changed SSE emitters also need lossless encoding for arbitrary JSON values: restore json.dumps(data) in the Anthropic emitter and json.dumps(payload) in the Responses emitter, or implement a tested compatibility fallback. Do not convert integers to floats/strings or stringify the entire input object. The invariant is that supported JSON inputs remain serializable without changing their values.

Preservation / acceptance (proposed, not run): Convert the request above and an Anthropic response carrying the same block; stdlib decoding of the resulting arguments must recover the exact integer. Generate Anthropic SSE and assert decoded partial_json recovers it and the stream reaches message_stop. Generate Responses SSE with a function schema containing enum: [18446744073709551617]; it must reach response.completed with the integer intact. Cover the signed/unsigned 64-bit endpoints as neighboring valid cases. Retain call IDs, names, status, text/thinking ordering, and the existing SSE assertions: tool delta decodes to {"a":1}, stop reason remains tool_use, and output usage remains 4.

call_id=block["id"],
name=block["name"],
id=block["id"],
Expand Down Expand Up @@ -599,11 +600,11 @@ def _block_deltas(self, block: Dict[str, Any]) -> List[Dict[str, Any]]:
if block_type == "thinking":
return [{"type": "thinking_delta", "thinking": block.get("thinking", "")}]
if block_type == "tool_use":
return [{"type": "input_json_delta", "partial_json": json.dumps(block.get("input", {}))}]
return [{"type": "input_json_delta", "partial_json": orjson.dumps(block.get("input", {})).decode()}]
raise NotImplementedError(f"Unsupported Anthropic block for SSE synthesis: {block_type}")

def _sse_event(self, event_type: str, data: Dict[str, Any]) -> str:
return f"event: {event_type}\ndata: {json.dumps(data)}\n\n"
return f"event: {event_type}\ndata: {orjson.dumps(data).decode()}\n\n"

############################################################################
# Shared structural helpers
Expand Down Expand Up @@ -738,7 +739,7 @@ def _function_call_to_tool_use(self, item: Dict[str, Any]) -> Dict[str, Any]:
}

def _json_object_from_arguments(self, arguments: str) -> Dict[str, Any]:
parsed = json.loads(arguments or "{}")
parsed = orjson.loads(arguments or "{}")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

orjson.loads() can silently corrupt valid tool arguments outside its 64-bit integer range

With the installed orjson==3.11.3:

json.loads('{"account_id":18446744073709551617}')
# {'account_id': 18446744073709551617}

orjson.loads('{"account_id":18446744073709551617}')
# {'account_id': 1.8446744073709552e+19}

The second result is a float, is not equal to the original integer. No exception is raised. At the runtime tool boundary it changes behavior; in capture paths it can silently make the recorded request/response differ from the actual wire payload, which is especially concerning if captures are later used as training or evaluation data.

I think we should keep stdlib json.loads() for externally supplied or otherwise unconstrained JSON

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKER — Preserve exact integers when parsing tool arguments

Failure / blast radius: _json_object_from_arguments('{"account_id":18446744073709551617}') now returns a float (1.8446744073709552e+19) rather than the exact integer. No exception triggers a fallback. The caller inserts this altered value into Anthropic tool_use.input for request and response conversion. The former parser preserved it. The same replacement also affects structured tool-call observations and unconstrained capture payloads.

Correction: Reintroduce import json and replace this statement with:

parsed = json.loads(arguments or "{}")

Likewise retain stdlib parsing in _as_arguments, _parse_sse_events, _reconstruct_anthropic_sse, _record, and the non-streaming body parse in _CaptureMiddleware._parse_and_record. Falling back only when orjson.loads raises cannot detect this rounding. Restore the invariant that parsed numbers are exact, or capture is explicitly incomplete—not silently rounded. Update the changed parser-failure test to patch obs.json.loads and save that function as real_loads.

Preservation / acceptance (proposed, not run): Assert both this helper and _as_arguments return an int equal to 18446744073709551617; repeat with -9223372036854775809. Reconstruct an Anthropic tool block whose partial_json contains that integer and assert its input remains exact. Retain empty arguments → {}, list arguments → ValueError, ordinary nested objects/Unicode, and malformed _as_arguments input → {"_raw": input}. Retain the request-parser fallback assertions. Because the existing CaptureStore writer rejects oversized integers, restoring lossless parsing must at least mark such captures incomplete rather than save a clean rounded exchange; supporting exact persistence additionally requires a lossless writer and reader.

if not isinstance(parsed, dict):
raise ValueError(f"Anthropic tool_use input must be a JSON object, got {type(parsed).__name__}")
return parsed
Expand Down
17 changes: 9 additions & 8 deletions nemo_gym/base_responses_api_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@
import asyncio
import fcntl
import inspect
import json
import logging
import os
import re
Expand Down Expand Up @@ -142,7 +141,9 @@ def _request_envelope(getter: Any) -> list[dict]:
if not instructions and not tools:
return []
envelope = {"instructions": _plain(instructions), "tools": _plain(tools)}
return [{"role": _ENVELOPE_ROLE, "content": json.dumps(envelope, sort_keys=True, default=str)}]
return [
{"role": _ENVELOPE_ROLE, "content": orjson.dumps(envelope, option=orjson.OPT_SORT_KEYS, default=str).decode()}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — Preserve the versioned request-envelope hash representation

Failure / blast radius: For instructions="be brief", the envelope content changes from '{"instructions": "be brief", "tools": null}' to '{"instructions":"be brief","tools":null}'. This is not merely wire whitespace: conversation hashing treats string content literally, and parent verification compares that context digest. Existing persisted records still have fingerprint version 2, as does this head. A continuation of a pre-change record with unchanged instructions/tools therefore resolves UNRESOLVED/no_match, losing its valid parent link and potentially masking the rollout. Sorting keys alone does not preserve the digest.

Correction: Preserve the previous canonical representation at this hash boundary. Reintroduce import json and restore:

return [{"role": _ENVELOPE_ROLE, "content": json.dumps(envelope, sort_keys=True, default=str)}]

Do not route this envelope through a compact serializer intended for SSE/tool arguments. The invariant is hash-for-hash agreement with existing same-version records, including spacing and Unicode escaping.

Preservation / acceptance (proposed, not run): Build a legacy request from the literal spaced envelope above plus a user turn; record it with RolloutLineage.record, an assistant reply, and context_len=len(legacy_request). Resolve _request_messages for the same instructions and echoed turns plus a new user turn: expect RESOLVED and the recorded call ID, not no_match. Repeat with non-ASCII instructions and a nested tool schema. Retain the changed-system-prompt rejection and unchanged-prompt link assertions, and changed-tool rejection, envelope exclusion from the assistant lookup key, and dict/model equivalence.

]


def _plain(value: Any) -> Any:
Expand Down Expand Up @@ -632,7 +633,7 @@ def _as_arguments(arguments: Any) -> dict[str, Any]:
return arguments
if isinstance(arguments, str):
try:
return json.loads(arguments)
return orjson.loads(arguments)
except Exception:
return {"_raw": arguments}
return {}
Expand Down Expand Up @@ -910,7 +911,7 @@ def _consume_terminal_sse_event(buffer: bytearray, dialect: str) -> Optional[str
if not line.startswith(b"data:"):
continue
try:
payload = json.loads(line[5:].lstrip())
payload = orjson.loads(line[5:].lstrip())
except Exception:
continue
if isinstance(payload, dict) and payload.get("error") is not None:
Expand Down Expand Up @@ -1007,7 +1008,7 @@ def _parse_sse_events(raw: bytes) -> list[dict[str, Any]]:
if payload == "[DONE]":
continue
try:
parsed = json.loads(payload)
parsed = orjson.loads(payload)
except Exception:
continue
if isinstance(parsed, dict):
Expand Down Expand Up @@ -1054,7 +1055,7 @@ def _reconstruct_anthropic_sse(events: list[dict[str, Any]]) -> Optional[dict[st
block = blocks[idx]
if block.get("type") == "tool_use" and idx in tool_json and not block.get("input"):
try:
block["input"] = json.loads(tool_json[idx]) if tool_json[idx] else {}
block["input"] = orjson.loads(tool_json[idx]) if tool_json[idx] else {}
except Exception:
block["input"] = {"_raw": tool_json[idx]}
content.append(block)
Expand Down Expand Up @@ -1177,7 +1178,7 @@ def _record(
request_raw = None
if request_bytes:
try:
parsed_request = json.loads(request_bytes)
parsed_request = orjson.loads(request_bytes)
if isinstance(parsed_request, dict):
request_body = parsed_request
else:
Expand Down Expand Up @@ -1494,7 +1495,7 @@ def _parse_and_record() -> None:
if body_bytes:
try:
response_body = (
_reconstruct_streamed_response(body_bytes, dialect) if streaming else json.loads(body_bytes)
_reconstruct_streamed_response(body_bytes, dialect) if streaming else orjson.loads(body_bytes)
)
if not isinstance(response_body, dict):
response_body = None
Expand Down
4 changes: 2 additions & 2 deletions nemo_gym/responses_streaming.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,12 @@
function-call names back into ``namespace`` + ``name`` on the way out.
"""

import json
import logging
from copy import deepcopy
from typing import Any, Iterator, Optional
from uuid import uuid4

import orjson
from openai.types.responses.response_create_params import ToolParam
from pydantic import TypeAdapter, ValidationError

Expand Down Expand Up @@ -238,7 +238,7 @@ def validate_streaming_responses_params(body: dict[str, Any]) -> NeMoGymResponse


def _sse_event(payload: dict[str, Any]) -> str:
return f"event: {payload['type']}\ndata: {json.dumps(payload)}\n\n"
return f"event: {payload['type']}\ndata: {orjson.dumps(payload).decode()}\n\n"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKER — Preserve Unicode separators in captured SSE payloads

Failure / blast radius: The new encoder emits U+2028, U+2029, and U+0085 literally rather than as ASCII escapes. However, capture's SSE parser uses str.splitlines(), which treats these characters as line boundaries inside JSON strings. For example, synthesize a response with status="completed" and one output-text item containing "a\u2028b", then pass its encoded SSE to _reconstruct_streamed_response(raw, "responses"). The output-item and completed payloads are discarded as malformed; reconstruction falls back to response.created, returning status="in_progress", output=[]. Terminal detection still sees completion, so capture can record success with missing content. Anthropic text deltas suffer the same loss through its changed emitter.

Correction: If retaining literal Unicode emission, change the physical-line splitter in _parse_sse_events at line 1004—not the model text—to recognize only SSE CR/LF boundaries:

data_lines = [
    line[5:].lstrip()
    for line in re.split(r"\r\n|\r|\n", block)
    if line.startswith("data:")
]

The invariant is that Unicode characters inside a JSON string never become SSE framing. This replacement uses the existing re import and leaves JSON parsing and multi-data: joining unchanged.

Preservation / acceptance (proposed, not run): Parameterize synthesized Responses text over U+0085/U+2028/U+2029; reconstruction must retain the exact text, completed status, and output list. Do the equivalent Anthropic synthesis/reconstruction check and assert the reconstructed text block is unchanged. Retain LF and CRLF streams, escaped \n inside JSON strings, multiple data: lines, and malformed/non-object event skipping. In the existing CRLF chat reconstruction test, retain content == "Hello", reasoning_content == "hmm", and assembled arguments '{"a":1}'.



def restore_namespace_tool_calls(items: list[dict], ns_map: NamespaceMap) -> list[dict]:
Expand Down
13 changes: 12 additions & 1 deletion tests/unit_tests/test_base_responses_api_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -768,7 +768,18 @@ def test_record_swallows_store_failure_and_marks_capture_incomplete(tmp_path, mo
def test_record_falls_back_to_raw_when_request_parser_raises(tmp_path, monkeypatch):
import nemo_gym.base_responses_api_model as obs

monkeypatch.setattr(obs.json, "loads", MagicMock(side_effect=RecursionError))
# `_record`'s request-body parse and `CaptureStore.read`'s exchange-line parse both go
# through `orjson.loads` now, so the failure must be scoped to the exact malformed
# payload under test rather than blanket-patching `orjson.loads` (which would also break
# this test's own `store.read()` call below).
real_loads = obs.orjson.loads

def _raise_for_malformed_request(data):
if data == b"deeply-nested-request":
raise RecursionError
return real_loads(data)

monkeypatch.setattr(obs.orjson, "loads", MagicMock(side_effect=_raise_for_malformed_request))
store = CaptureStore(tmp_path)
obs._record(
store,
Expand Down
Loading