Repository navigation
Conversation
orjson is already a hard dependency and already used elsewhere in these same files; these are the remaining stdlib json.dumps/json.loads call sites that were never converted: - responses_streaming.py: _sse_event, one per streamed Responses-API SSE event. - anthropic_converter.py: tool-call argument serialization (both directions), the Anthropic SSE _sse_event/_block_deltas emitters, and _json_object_from_arguments. - base_responses_api_model.py: the instructions/tools envelope hash, tool-call argument parsing, the capture middleware's per-chunk SSE terminal-event probe, full-stream SSE reconstruction (chat + anthropic), and the request-body parse in _record. All are drop-in swaps (bytes/str input either module accepts identically, and every call site is already wrapped in a broad try/except), except _sse_event-style call sites which need `.decode()` since callers expect `str`, not `bytes`. One test needed updating: test_record_falls_back_to_raw_when_request_parser_raises patched `json.loads` directly, which no longer exists on the module now that `_record`'s request-body parse and CaptureStore.read's exchange-line parse both go through orjson.loads. Scoped the patch to the specific malformed payload under test instead of blanket-patching orjson.loads, since that would also break the test's own store.read() call. Signed-off-by: Kajal Jain <kajalj@nvidia.com>
|
/ok to test 1f6ccf1 |
|
/ok to test b0f9e14 |
|
/claude review |
|
SHIP — LGTM, no reliability concerns. Reviewed the
The only residual |
|
|
||
| def _json_object_from_arguments(self, arguments: str) -> Dict[str, Any]: | ||
| parsed = json.loads(arguments or "{}") | ||
| parsed = orjson.loads(arguments or "{}") |
There was a problem hiding this comment.
orjson.loads() can silently corrupt valid tool arguments outside its 64-bit integer range
With the installed orjson==3.11.3:
json.loads('{"account_id":18446744073709551617}')
# {'account_id': 18446744073709551617}
orjson.loads('{"account_id":18446744073709551617}')
# {'account_id': 1.8446744073709552e+19}The second result is a float, is not equal to the original integer. No exception is raised. At the runtime tool boundary it changes behavior; in capture paths it can silently make the recorded request/response differ from the actual wire payload, which is especially concerning if captures are later used as training or evaluation data.
I think we should keep stdlib json.loads() for externally supplied or otherwise unconstrained JSON
| output.append( | ||
| NeMoGymResponseFunctionToolCall( | ||
| arguments=json.dumps(block.get("input", {})), | ||
| arguments=orjson.dumps(block.get("input", {})).decode(), |
There was a problem hiding this comment.
the new serializer rejects tool inputs that stdlib serialized successfully
orjson.dumps() raises JSONEncodeError: Integer exceeds 64-bit range for arbitrary-precision Python integers, while stdlib emits the exact JSON number. For example:
value = {"account_id": 18446744073709551617}
json.dumps(value)
# '{"account_id": 18446744073709551617}'
orjson.dumps(value)
# JSONEncodeError: Integer exceeds 64-bit rangeI think this should be addressed through one shared Gym serialization helper. The implementation can try orjson.dumps() and fall back on any orjson.JSONEncodeError to compact stdlib JSON.
def json_dumps_bytes(value: JsonValue, *, sort_keys: bool = False) -> bytes:
option = orjson.OPT_SORT_KEYS if sort_keys else 0
try:
return orjson.dumps(value, option=option)
except orjson.JSONEncodeError:
return json.dumps(
value,
sort_keys=sort_keys,
ensure_ascii=False,
separators=(",", ":"),
allow_nan=False,
).encode("utf-8")
def json_dumps(value: JsonValue, *, sort_keys: bool = False) -> str:
return json_dumps_bytes(value, sort_keys=sort_keys).decode("utf-8")there are still some differences:
orjsonsilently serializesNaNand infinities asnullvs stdlib normally emits the non-standardNaN/Infinitytokens, whileallow_nan=Falseraises.orjsonalso natively serializes datetime/dataclass/UUID values that stdlib rejects
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
HOLD — the swaps are not behaviorally equivalent: they introduce integer compatibility failures, lose valid Unicode SSE content during capture, and change persisted lineage hashes.
Reviewed all four changed files and the directly affected conversion, capture, and lineage paths. The updated parser-failure test still exercises raw fallback, but does not cover these serialization differences. The existing integer concerns remain supported by the current source.
Static review only; tests and suggested corrections were not executed.
Review generated by codex
|
|
||
| def _json_object_from_arguments(self, arguments: str) -> Dict[str, Any]: | ||
| parsed = json.loads(arguments or "{}") | ||
| parsed = orjson.loads(arguments or "{}") |
There was a problem hiding this comment.
BLOCKER — Preserve exact integers when parsing tool arguments
Failure / blast radius: _json_object_from_arguments('{"account_id":18446744073709551617}') now returns a float (1.8446744073709552e+19) rather than the exact integer. No exception triggers a fallback. The caller inserts this altered value into Anthropic tool_use.input for request and response conversion. The former parser preserved it. The same replacement also affects structured tool-call observations and unconstrained capture payloads.
Correction: Reintroduce import json and replace this statement with:
parsed = json.loads(arguments or "{}")Likewise retain stdlib parsing in _as_arguments, _parse_sse_events, _reconstruct_anthropic_sse, _record, and the non-streaming body parse in _CaptureMiddleware._parse_and_record. Falling back only when orjson.loads raises cannot detect this rounding. Restore the invariant that parsed numbers are exact, or capture is explicitly incomplete—not silently rounded. Update the changed parser-failure test to patch obs.json.loads and save that function as real_loads.
Preservation / acceptance (proposed, not run): Assert both this helper and _as_arguments return an int equal to 18446744073709551617; repeat with -9223372036854775809. Reconstruct an Anthropic tool block whose partial_json contains that integer and assert its input remains exact. Retain empty arguments → {}, list arguments → ValueError, ordinary nested objects/Unicode, and malformed _as_arguments input → {"_raw": input}. Retain the request-parser fallback assertions. Because the existing CaptureStore writer rejects oversized integers, restoring lossless parsing must at least mark such captures incomplete rather than save a clean rounded exchange; supporting exact persistence additionally requires a lossless writer and reader.
| items.append( | ||
| NeMoGymResponseFunctionToolCall( | ||
| arguments=json.dumps(block.get("input", {})), | ||
| arguments=orjson.dumps(block.get("input", {})).decode(), |
There was a problem hiding this comment.
RISK — Do not reject previously supported tool inputs during encoding
Failure / blast radius: An Anthropic request with messages=[{"role":"assistant","content":[{"type":"tool_use","id":"t1","name":"lookup","input":{"account_id":18446744073709551617}}]}] now raises orjson.JSONEncodeError: Integer exceeds 64-bit range here. The default Messages endpoint invokes this conversion before model dispatch, so valid history that previously converted now fails. The same limitation affects Anthropic response conversion, _block_deltas, and SSE envelopes containing unconstrained tool schemas.
Correction: The smallest compatibility fix is to keep stdlib encoding at these unconstrained sites, reintroducing import json. In both tool-use → function-call branches use:
arguments=json.dumps(block.get("input", {})),In _block_deltas restore:
return [{"type": "input_json_delta", "partial_json": json.dumps(block.get("input", {}))}]The two changed SSE emitters also need lossless encoding for arbitrary JSON values: restore json.dumps(data) in the Anthropic emitter and json.dumps(payload) in the Responses emitter, or implement a tested compatibility fallback. Do not convert integers to floats/strings or stringify the entire input object. The invariant is that supported JSON inputs remain serializable without changing their values.
Preservation / acceptance (proposed, not run): Convert the request above and an Anthropic response carrying the same block; stdlib decoding of the resulting arguments must recover the exact integer. Generate Anthropic SSE and assert decoded partial_json recovers it and the stream reaches message_stop. Generate Responses SSE with a function schema containing enum: [18446744073709551617]; it must reach response.completed with the integer intact. Cover the signed/unsigned 64-bit endpoints as neighboring valid cases. Retain call IDs, names, status, text/thinking ordering, and the existing SSE assertions: tool delta decodes to {"a":1}, stop reason remains tool_use, and output usage remains 4.
|
|
||
| def _sse_event(payload: dict[str, Any]) -> str: | ||
| return f"event: {payload['type']}\ndata: {json.dumps(payload)}\n\n" | ||
| return f"event: {payload['type']}\ndata: {orjson.dumps(payload).decode()}\n\n" |
There was a problem hiding this comment.
BLOCKER — Preserve Unicode separators in captured SSE payloads
Failure / blast radius: The new encoder emits U+2028, U+2029, and U+0085 literally rather than as ASCII escapes. However, capture's SSE parser uses str.splitlines(), which treats these characters as line boundaries inside JSON strings. For example, synthesize a response with status="completed" and one output-text item containing "a\u2028b", then pass its encoded SSE to _reconstruct_streamed_response(raw, "responses"). The output-item and completed payloads are discarded as malformed; reconstruction falls back to response.created, returning status="in_progress", output=[]. Terminal detection still sees completion, so capture can record success with missing content. Anthropic text deltas suffer the same loss through its changed emitter.
Correction: If retaining literal Unicode emission, change the physical-line splitter in _parse_sse_events at line 1004—not the model text—to recognize only SSE CR/LF boundaries:
data_lines = [
line[5:].lstrip()
for line in re.split(r"\r\n|\r|\n", block)
if line.startswith("data:")
]The invariant is that Unicode characters inside a JSON string never become SSE framing. This replacement uses the existing re import and leaves JSON parsing and multi-data: joining unchanged.
Preservation / acceptance (proposed, not run): Parameterize synthesized Responses text over U+0085/U+2028/U+2029; reconstruction must retain the exact text, completed status, and output list. Do the equivalent Anthropic synthesis/reconstruction check and assert the reconstructed text block is unchanged. Retain LF and CRLF streams, escaped \n inside JSON strings, multiple data: lines, and malformed/non-object event skipping. In the existing CRLF chat reconstruction test, retain content == "Hello", reasoning_content == "hmm", and assembled arguments '{"a":1}'.
| envelope = {"instructions": _plain(instructions), "tools": _plain(tools)} | ||
| return [{"role": _ENVELOPE_ROLE, "content": json.dumps(envelope, sort_keys=True, default=str)}] | ||
| return [ | ||
| {"role": _ENVELOPE_ROLE, "content": orjson.dumps(envelope, option=orjson.OPT_SORT_KEYS, default=str).decode()} |
There was a problem hiding this comment.
RISK — Preserve the versioned request-envelope hash representation
Failure / blast radius: For instructions="be brief", the envelope content changes from '{"instructions": "be brief", "tools": null}' to '{"instructions":"be brief","tools":null}'. This is not merely wire whitespace: conversation hashing treats string content literally, and parent verification compares that context digest. Existing persisted records still have fingerprint version 2, as does this head. A continuation of a pre-change record with unchanged instructions/tools therefore resolves UNRESOLVED/no_match, losing its valid parent link and potentially masking the rollout. Sorting keys alone does not preserve the digest.
Correction: Preserve the previous canonical representation at this hash boundary. Reintroduce import json and restore:
return [{"role": _ENVELOPE_ROLE, "content": json.dumps(envelope, sort_keys=True, default=str)}]Do not route this envelope through a compact serializer intended for SSE/tool arguments. The invariant is hash-for-hash agreement with existing same-version records, including spacing and Unicode escaping.
Preservation / acceptance (proposed, not run): Build a legacy request from the literal spaced envelope above plus a user turn; record it with RolloutLineage.record, an assistant reply, and context_len=len(legacy_request). Resolve _request_messages for the same instructions and echoed turns plus a new user turn: expect RESOLVED and the recorded call ID, not no_match. Repeat with non-ASCII instructions and a nested tool schema. Retain the changed-system-prompt rejection and unchanged-prompt link assertions, and changed-tool rejection, envelope exclusion from the assistant lookup key, and dict/model equivalence.
Summary
orjsonis already a hard dependency and already used elsewhere in these same files; this converts the remaining stdlibjson.dumps/json.loadscall sites in the model-server subsystem's hot paths:responses_streaming.py:_sse_event, fired once per streamed Responses-API SSE event.anthropic_converter.py: tool-call argument serialization (both directions), the Anthropic SSE_sse_event/_block_deltasemitters,_json_object_from_arguments.base_responses_api_model.py: the instructions/tools envelope hash, tool-call argument parsing, the model-call capture middleware's per-chunk SSE terminal-event probe, full-stream SSE reconstruction (chat + anthropic dialects), and the request-body parse in_record.try/except._sse_event-style sites add.decode()since callers expectstr.orjsonswap already applied tochat_streaming.py; this closes out the rest.Test plan
pytest tests/unit_tests/— 4776 passed, 15 pre-existing failures (sandbox provider tests requiringray/piponPATH; reproduced identically on unmodifiedmainwith this diff stashed out, unrelated to this change)ruff check/ruff format --checkon all changed files — cleantest_record_falls_back_to_raw_when_request_parser_raises, which patchedjson.loadsdirectly — no longer exists on the module now that_record's request-body parse andCaptureStore.read's exchange-line parse both go throughorjson.loads. Scoped the patch to the specific malformed payload under test instead of blanket-patchingorjson.loads, since that would also break the test's ownstore.read()call.