Skip to content

perf(model-server): swap stdlib json for orjson on remaining hot paths - #3509

Draft
kajalj22 wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
kajalj22:gym-tier1-orjson
Draft

kajalj22 wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
kajalj22:gym-tier1-orjson

Conversation

@kajalj22

Copy link
Copy Markdown
Contributor

Summary

  • orjson is already a hard dependency and already used elsewhere in these same files; this converts the remaining stdlib json.dumps/json.loads call sites in the model-server subsystem's hot paths:
    • responses_streaming.py: _sse_event, fired once per streamed Responses-API SSE event.
    • anthropic_converter.py: tool-call argument serialization (both directions), the Anthropic SSE _sse_event/_block_deltas emitters, _json_object_from_arguments.
    • base_responses_api_model.py: the instructions/tools envelope hash, tool-call argument parsing, the model-call capture middleware's per-chunk SSE terminal-event probe, full-stream SSE reconstruction (chat + anthropic dialects), and the request-body parse in _record.
  • All are drop-in swaps — orjson accepts the same bytes/str input, and every call site was already wrapped in a broad try/except. _sse_event-style sites add .decode() since callers expect str.
  • Companion to the SSE orjson swap already applied to chat_streaming.py; this closes out the rest.

Test plan

  • pytest tests/unit_tests/ — 4776 passed, 15 pre-existing failures (sandbox provider tests requiring ray/pip on PATH; reproduced identically on unmodified main with this diff stashed out, unrelated to this change)
  • ruff check / ruff format --check on all changed files — clean
  • Updated test_record_falls_back_to_raw_when_request_parser_raises, which patched json.loads directly — no longer exists on the module now that _record's request-body parse and CaptureStore.read's exchange-line parse both go through orjson.loads. Scoped the patch to the specific malformed payload under test instead of blanket-patching orjson.loads, since that would also break the test's own store.read() call.

orjson is already a hard dependency and already used elsewhere in these
same files; these are the remaining stdlib json.dumps/json.loads call
sites that were never converted:

- responses_streaming.py: _sse_event, one per streamed Responses-API SSE
  event.
- anthropic_converter.py: tool-call argument serialization (both
  directions), the Anthropic SSE _sse_event/_block_deltas emitters, and
  _json_object_from_arguments.
- base_responses_api_model.py: the instructions/tools envelope hash,
  tool-call argument parsing, the capture middleware's per-chunk SSE
  terminal-event probe, full-stream SSE reconstruction (chat + anthropic),
  and the request-body parse in _record.

All are drop-in swaps (bytes/str input either module accepts
identically, and every call site is already wrapped in a broad
try/except), except _sse_event-style call sites which need `.decode()`
since callers expect `str`, not `bytes`.

One test needed updating: test_record_falls_back_to_raw_when_request_parser_raises
patched `json.loads` directly, which no longer exists on the module now
that `_record`'s request-body parse and CaptureStore.read's exchange-line
parse both go through orjson.loads. Scoped the patch to the specific
malformed payload under test instead of blanket-patching orjson.loads,
since that would also break the test's own store.read() call.

Signed-off-by: Kajal Jain <kajalj@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 18, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@kajalj22

Copy link
Copy Markdown
Contributor Author

/ok to test 1f6ccf1

@kajalj22

Copy link
Copy Markdown
Contributor Author

/ok to test b0f9e14

@kajalj22
kajalj22 marked this pull request as ready for review September 18, 2026 00:54
@yaoyu-33 yaoyu-33 added area:core Shared APIs, servers, telemetry, health, and registries complexity:low Localized change in one scope with a small, straightforward review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer labels Sep 18, 2026
@kajalj22

Copy link
Copy Markdown
Contributor Author

/claude review

@claude

claude Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

SHIP — LGTM, no reliability concerns.

Reviewed the json→orjson migration across anthropic_converter.py, base_responses_api_model.py, and responses_streaming.py. It's complete and behaviorally sound:

  • Dependency declared. orjson is already a first-class dep (pyproject.toml:216, pinned via uv.lock) and used widely elsewhere in nemo_gym/ — no new import hygiene issue.
  • Bytes/str boundary handled. Every orjson.dumps(...) that feeds a str context (arguments=, SSE data: frames, envelope content, partial_json) correctly appends .decode(). orjson.loads accepts both bytes and str, so the parse sites are fine.
  • Semantics preserved. The fingerprint envelope keeps deterministic output via option=orjson.OPT_SORT_KEYS + default=str (equivalent to the prior sort_keys=True, default=str). No downstream code catches json.JSONDecodeError on these paths, and the SSE whitespace difference (orjson emits compact separators) isn't asserted by any test or contract.
  • Test rescoping is correct and necessary. test_record_falls_back_to_raw_when_request_parser_raises now scopes the RecursionError to the specific b"deeply-nested-request" payload instead of blanket-patching orjson.loads; a blanket patch would have broken the test's own store.read() call. Assertions still verify real fallback behavior (request is None, request_raw populated).

The only residual json. reference is in a docstring comment (base_responses_api_model.py:168), which is accurate. No async, verifier-correctness, or public-API concerns.

@ananthsub
ananthsub self-requested a review September 18, 2026 16:19

def _json_object_from_arguments(self, arguments: str) -> Dict[str, Any]:
parsed = json.loads(arguments or "{}")
parsed = orjson.loads(arguments or "{}")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

orjson.loads() can silently corrupt valid tool arguments outside its 64-bit integer range

With the installed orjson==3.11.3:

json.loads('{"account_id":18446744073709551617}')
# {'account_id': 18446744073709551617}

orjson.loads('{"account_id":18446744073709551617}')
# {'account_id': 1.8446744073709552e+19}

The second result is a float, is not equal to the original integer. No exception is raised. At the runtime tool boundary it changes behavior; in capture paths it can silently make the recorded request/response differ from the actual wire payload, which is especially concerning if captures are later used as training or evaluation data.

I think we should keep stdlib json.loads() for externally supplied or otherwise unconstrained JSON

output.append(
NeMoGymResponseFunctionToolCall(
arguments=json.dumps(block.get("input", {})),
arguments=orjson.dumps(block.get("input", {})).decode(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the new serializer rejects tool inputs that stdlib serialized successfully

orjson.dumps() raises JSONEncodeError: Integer exceeds 64-bit range for arbitrary-precision Python integers, while stdlib emits the exact JSON number. For example:

value = {"account_id": 18446744073709551617}
json.dumps(value)
# '{"account_id": 18446744073709551617}'

orjson.dumps(value)
# JSONEncodeError: Integer exceeds 64-bit range

I think this should be addressed through one shared Gym serialization helper. The implementation can try orjson.dumps() and fall back on any orjson.JSONEncodeError to compact stdlib JSON.

def json_dumps_bytes(value: JsonValue, *, sort_keys: bool = False) -> bytes:
    option = orjson.OPT_SORT_KEYS if sort_keys else 0
    try:
        return orjson.dumps(value, option=option)
    except orjson.JSONEncodeError:
        return json.dumps(
            value,
            sort_keys=sort_keys,
            ensure_ascii=False,
            separators=(",", ":"),
            allow_nan=False,
        ).encode("utf-8")

def json_dumps(value: JsonValue, *, sort_keys: bool = False) -> str:
    return json_dumps_bytes(value, sort_keys=sort_keys).decode("utf-8")

there are still some differences:

  • orjson silently serializes NaN and infinities as null vs stdlib normally emits the non-standard NaN/Infinity tokens, while allow_nan=False raises.
  • orjson also natively serializes datetime/dataclass/UUID values that stdlib rejects

@kajalj22
kajalj22 marked this pull request as draft September 28, 2026 20:17
@kajalj22

kajalj22 commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD — the swaps are not behaviorally equivalent: they introduce integer compatibility failures, lose valid Unicode SSE content during capture, and change persisted lineage hashes.

Reviewed all four changed files and the directly affected conversion, capture, and lineage paths. The updated parser-failure test still exercises raw fallback, but does not cover these serialization differences. The existing integer concerns remain supported by the current source.

Static review only; tests and suggested corrections were not executed.

Review generated by codex


def _json_object_from_arguments(self, arguments: str) -> Dict[str, Any]:
parsed = json.loads(arguments or "{}")
parsed = orjson.loads(arguments or "{}")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKER — Preserve exact integers when parsing tool arguments

Failure / blast radius: _json_object_from_arguments('{"account_id":18446744073709551617}') now returns a float (1.8446744073709552e+19) rather than the exact integer. No exception triggers a fallback. The caller inserts this altered value into Anthropic tool_use.input for request and response conversion. The former parser preserved it. The same replacement also affects structured tool-call observations and unconstrained capture payloads.

Correction: Reintroduce import json and replace this statement with:

parsed = json.loads(arguments or "{}")

Likewise retain stdlib parsing in _as_arguments, _parse_sse_events, _reconstruct_anthropic_sse, _record, and the non-streaming body parse in _CaptureMiddleware._parse_and_record. Falling back only when orjson.loads raises cannot detect this rounding. Restore the invariant that parsed numbers are exact, or capture is explicitly incomplete—not silently rounded. Update the changed parser-failure test to patch obs.json.loads and save that function as real_loads.

Preservation / acceptance (proposed, not run): Assert both this helper and _as_arguments return an int equal to 18446744073709551617; repeat with -9223372036854775809. Reconstruct an Anthropic tool block whose partial_json contains that integer and assert its input remains exact. Retain empty arguments → {}, list arguments → ValueError, ordinary nested objects/Unicode, and malformed _as_arguments input → {"_raw": input}. Retain the request-parser fallback assertions. Because the existing CaptureStore writer rejects oversized integers, restoring lossless parsing must at least mark such captures incomplete rather than save a clean rounded exchange; supporting exact persistence additionally requires a lossless writer and reader.

items.append(
NeMoGymResponseFunctionToolCall(
arguments=json.dumps(block.get("input", {})),
arguments=orjson.dumps(block.get("input", {})).decode(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — Do not reject previously supported tool inputs during encoding

Failure / blast radius: An Anthropic request with messages=[{"role":"assistant","content":[{"type":"tool_use","id":"t1","name":"lookup","input":{"account_id":18446744073709551617}}]}] now raises orjson.JSONEncodeError: Integer exceeds 64-bit range here. The default Messages endpoint invokes this conversion before model dispatch, so valid history that previously converted now fails. The same limitation affects Anthropic response conversion, _block_deltas, and SSE envelopes containing unconstrained tool schemas.

Correction: The smallest compatibility fix is to keep stdlib encoding at these unconstrained sites, reintroducing import json. In both tool-use → function-call branches use:

arguments=json.dumps(block.get("input", {})),

In _block_deltas restore:

return [{"type": "input_json_delta", "partial_json": json.dumps(block.get("input", {}))}]

The two changed SSE emitters also need lossless encoding for arbitrary JSON values: restore json.dumps(data) in the Anthropic emitter and json.dumps(payload) in the Responses emitter, or implement a tested compatibility fallback. Do not convert integers to floats/strings or stringify the entire input object. The invariant is that supported JSON inputs remain serializable without changing their values.

Preservation / acceptance (proposed, not run): Convert the request above and an Anthropic response carrying the same block; stdlib decoding of the resulting arguments must recover the exact integer. Generate Anthropic SSE and assert decoded partial_json recovers it and the stream reaches message_stop. Generate Responses SSE with a function schema containing enum: [18446744073709551617]; it must reach response.completed with the integer intact. Cover the signed/unsigned 64-bit endpoints as neighboring valid cases. Retain call IDs, names, status, text/thinking ordering, and the existing SSE assertions: tool delta decodes to {"a":1}, stop reason remains tool_use, and output usage remains 4.


def _sse_event(payload: dict[str, Any]) -> str:
return f"event: {payload['type']}\ndata: {json.dumps(payload)}\n\n"
return f"event: {payload['type']}\ndata: {orjson.dumps(payload).decode()}\n\n"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKER — Preserve Unicode separators in captured SSE payloads

Failure / blast radius: The new encoder emits U+2028, U+2029, and U+0085 literally rather than as ASCII escapes. However, capture's SSE parser uses str.splitlines(), which treats these characters as line boundaries inside JSON strings. For example, synthesize a response with status="completed" and one output-text item containing "a\u2028b", then pass its encoded SSE to _reconstruct_streamed_response(raw, "responses"). The output-item and completed payloads are discarded as malformed; reconstruction falls back to response.created, returning status="in_progress", output=[]. Terminal detection still sees completion, so capture can record success with missing content. Anthropic text deltas suffer the same loss through its changed emitter.

Correction: If retaining literal Unicode emission, change the physical-line splitter in _parse_sse_events at line 1004—not the model text—to recognize only SSE CR/LF boundaries:

data_lines = [
    line[5:].lstrip()
    for line in re.split(r"\r\n|\r|\n", block)
    if line.startswith("data:")
]

The invariant is that Unicode characters inside a JSON string never become SSE framing. This replacement uses the existing re import and leaves JSON parsing and multi-data: joining unchanged.

Preservation / acceptance (proposed, not run): Parameterize synthesized Responses text over U+0085/U+2028/U+2029; reconstruction must retain the exact text, completed status, and output list. Do the equivalent Anthropic synthesis/reconstruction check and assert the reconstructed text block is unchanged. Retain LF and CRLF streams, escaped \n inside JSON strings, multiple data: lines, and malformed/non-object event skipping. In the existing CRLF chat reconstruction test, retain content == "Hello", reasoning_content == "hmm", and assembled arguments '{"a":1}'.

envelope = {"instructions": _plain(instructions), "tools": _plain(tools)}
return [{"role": _ENVELOPE_ROLE, "content": json.dumps(envelope, sort_keys=True, default=str)}]
return [
{"role": _ENVELOPE_ROLE, "content": orjson.dumps(envelope, option=orjson.OPT_SORT_KEYS, default=str).decode()}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — Preserve the versioned request-envelope hash representation

Failure / blast radius: For instructions="be brief", the envelope content changes from '{"instructions": "be brief", "tools": null}' to '{"instructions":"be brief","tools":null}'. This is not merely wire whitespace: conversation hashing treats string content literally, and parent verification compares that context digest. Existing persisted records still have fingerprint version 2, as does this head. A continuation of a pre-change record with unchanged instructions/tools therefore resolves UNRESOLVED/no_match, losing its valid parent link and potentially masking the rollout. Sorting keys alone does not preserve the digest.

Correction: Preserve the previous canonical representation at this hash boundary. Reintroduce import json and restore:

return [{"role": _ENVELOPE_ROLE, "content": json.dumps(envelope, sort_keys=True, default=str)}]

Do not route this envelope through a compact serializer intended for SSE/tool arguments. The invariant is hash-for-hash agreement with existing same-version records, including spacing and Unicode escaping.

Preservation / acceptance (proposed, not run): Build a legacy request from the literal spaced envelope above plus a user turn; record it with RolloutLineage.record, an assistant reply, and context_len=len(legacy_request). Resolve _request_messages for the same instructions and echoed turns plus a new user turn: expect RESOLVED and the recorded call ID, not no_match. Repeat with non-ASCII instructions and a nested tool schema. Retain the changed-system-prompt rejection and unchanged-prompt link assertions, and changed-tool rejection, envelope exclusion from the assistant lookup key, and dict/model equivalence.

This branch was previously deployed

1 inactive deployment
public — b0f9e14d Deployed Sep 18, 2026 by copy-pr-bot[bot] via release / finalize / notify #3139
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Shared APIs, servers, telemetry, health, and registries complexity:low Localized change in one scope with a small, straightforward review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants