Repository navigation
perf(model-server): swap stdlib json for orjson on remaining hot paths #3509
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -41,11 +41,12 @@ | |
|
|
||
| import base64 | ||
| import binascii | ||
| import json | ||
| from time import time | ||
| from typing import Any, Dict, Iterator, List, Optional | ||
| from uuid import uuid4 | ||
|
|
||
| import orjson | ||
|
|
||
| # Types only — never the `anthropic` client. The client uses httpx (O(n^2) connection | ||
| # pooling at high concurrency); all transport in Gym stays on aiohttp via server_utils. | ||
| # MessageCreateParams (request) is a TypedDict used purely as a hint; NeMoGymAnthropicMessage | ||
|
|
@@ -264,7 +265,7 @@ def _anthropic_content_to_output_items(self, content: List[Dict[str, Any]]) -> L | |
| self._flush_text_output(pending_text, output) | ||
| output.append( | ||
| NeMoGymResponseFunctionToolCall( | ||
| arguments=json.dumps(block.get("input", {})), | ||
| arguments=orjson.dumps(block.get("input", {})).decode(), | ||
| call_id=block["id"], | ||
| name=block["name"], | ||
| id=block["id"], | ||
|
|
@@ -369,7 +370,7 @@ def flush_message() -> None: | |
| flush_message() | ||
| items.append( | ||
| NeMoGymResponseFunctionToolCall( | ||
| arguments=json.dumps(block.get("input", {})), | ||
| arguments=orjson.dumps(block.get("input", {})).decode(), | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. RISK — Do not reject previously supported tool inputs during encoding Failure / blast radius: An Anthropic request with Correction: The smallest compatibility fix is to keep stdlib encoding at these unconstrained sites, reintroducing arguments=json.dumps(block.get("input", {})),In return [{"type": "input_json_delta", "partial_json": json.dumps(block.get("input", {}))}]The two changed SSE emitters also need lossless encoding for arbitrary JSON values: restore Preservation / acceptance (proposed, not run): Convert the request above and an Anthropic response carrying the same block; stdlib decoding of the resulting |
||
| call_id=block["id"], | ||
| name=block["name"], | ||
| id=block["id"], | ||
|
|
@@ -599,11 +600,11 @@ def _block_deltas(self, block: Dict[str, Any]) -> List[Dict[str, Any]]: | |
| if block_type == "thinking": | ||
| return [{"type": "thinking_delta", "thinking": block.get("thinking", "")}] | ||
| if block_type == "tool_use": | ||
| return [{"type": "input_json_delta", "partial_json": json.dumps(block.get("input", {}))}] | ||
| return [{"type": "input_json_delta", "partial_json": orjson.dumps(block.get("input", {})).decode()}] | ||
| raise NotImplementedError(f"Unsupported Anthropic block for SSE synthesis: {block_type}") | ||
|
|
||
| def _sse_event(self, event_type: str, data: Dict[str, Any]) -> str: | ||
| return f"event: {event_type}\ndata: {json.dumps(data)}\n\n" | ||
| return f"event: {event_type}\ndata: {orjson.dumps(data).decode()}\n\n" | ||
|
|
||
| ############################################################################ | ||
| # Shared structural helpers | ||
|
|
@@ -738,7 +739,7 @@ def _function_call_to_tool_use(self, item: Dict[str, Any]) -> Dict[str, Any]: | |
| } | ||
|
|
||
| def _json_object_from_arguments(self, arguments: str) -> Dict[str, Any]: | ||
| parsed = json.loads(arguments or "{}") | ||
| parsed = orjson.loads(arguments or "{}") | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
With the installed json.loads('{"account_id":18446744073709551617}')
# {'account_id': 18446744073709551617}
orjson.loads('{"account_id":18446744073709551617}')
# {'account_id': 1.8446744073709552e+19}The second result is a I think we should keep stdlib
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. BLOCKER — Preserve exact integers when parsing tool arguments Failure / blast radius: Correction: Reintroduce parsed = json.loads(arguments or "{}")Likewise retain stdlib parsing in Preservation / acceptance (proposed, not run): Assert both this helper and |
||
| if not isinstance(parsed, dict): | ||
| raise ValueError(f"Anthropic tool_use input must be a JSON object, got {type(parsed).__name__}") | ||
| return parsed | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -29,7 +29,6 @@ | |
| import asyncio | ||
| import fcntl | ||
| import inspect | ||
| import json | ||
| import logging | ||
| import os | ||
| import re | ||
|
|
@@ -142,7 +141,9 @@ def _request_envelope(getter: Any) -> list[dict]: | |
| if not instructions and not tools: | ||
| return [] | ||
| envelope = {"instructions": _plain(instructions), "tools": _plain(tools)} | ||
| return [{"role": _ENVELOPE_ROLE, "content": json.dumps(envelope, sort_keys=True, default=str)}] | ||
| return [ | ||
| {"role": _ENVELOPE_ROLE, "content": orjson.dumps(envelope, option=orjson.OPT_SORT_KEYS, default=str).decode()} | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. RISK — Preserve the versioned request-envelope hash representation Failure / blast radius: For Correction: Preserve the previous canonical representation at this hash boundary. Reintroduce return [{"role": _ENVELOPE_ROLE, "content": json.dumps(envelope, sort_keys=True, default=str)}]Do not route this envelope through a compact serializer intended for SSE/tool arguments. The invariant is hash-for-hash agreement with existing same-version records, including spacing and Unicode escaping. Preservation / acceptance (proposed, not run): Build a legacy request from the literal spaced envelope above plus a user turn; record it with |
||
| ] | ||
|
|
||
|
|
||
| def _plain(value: Any) -> Any: | ||
|
|
@@ -632,7 +633,7 @@ def _as_arguments(arguments: Any) -> dict[str, Any]: | |
| return arguments | ||
| if isinstance(arguments, str): | ||
| try: | ||
| return json.loads(arguments) | ||
| return orjson.loads(arguments) | ||
| except Exception: | ||
| return {"_raw": arguments} | ||
| return {} | ||
|
|
@@ -910,7 +911,7 @@ def _consume_terminal_sse_event(buffer: bytearray, dialect: str) -> Optional[str | |
| if not line.startswith(b"data:"): | ||
| continue | ||
| try: | ||
| payload = json.loads(line[5:].lstrip()) | ||
| payload = orjson.loads(line[5:].lstrip()) | ||
| except Exception: | ||
| continue | ||
| if isinstance(payload, dict) and payload.get("error") is not None: | ||
|
|
@@ -1007,7 +1008,7 @@ def _parse_sse_events(raw: bytes) -> list[dict[str, Any]]: | |
| if payload == "[DONE]": | ||
| continue | ||
| try: | ||
| parsed = json.loads(payload) | ||
| parsed = orjson.loads(payload) | ||
| except Exception: | ||
| continue | ||
| if isinstance(parsed, dict): | ||
|
|
@@ -1054,7 +1055,7 @@ def _reconstruct_anthropic_sse(events: list[dict[str, Any]]) -> Optional[dict[st | |
| block = blocks[idx] | ||
| if block.get("type") == "tool_use" and idx in tool_json and not block.get("input"): | ||
| try: | ||
| block["input"] = json.loads(tool_json[idx]) if tool_json[idx] else {} | ||
| block["input"] = orjson.loads(tool_json[idx]) if tool_json[idx] else {} | ||
| except Exception: | ||
| block["input"] = {"_raw": tool_json[idx]} | ||
| content.append(block) | ||
|
|
@@ -1177,7 +1178,7 @@ def _record( | |
| request_raw = None | ||
| if request_bytes: | ||
| try: | ||
| parsed_request = json.loads(request_bytes) | ||
| parsed_request = orjson.loads(request_bytes) | ||
| if isinstance(parsed_request, dict): | ||
| request_body = parsed_request | ||
| else: | ||
|
|
@@ -1494,7 +1495,7 @@ def _parse_and_record() -> None: | |
| if body_bytes: | ||
| try: | ||
| response_body = ( | ||
| _reconstruct_streamed_response(body_bytes, dialect) if streaming else json.loads(body_bytes) | ||
| _reconstruct_streamed_response(body_bytes, dialect) if streaming else orjson.loads(body_bytes) | ||
| ) | ||
| if not isinstance(response_body, dict): | ||
| response_body = None | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -31,12 +31,12 @@ | |
| function-call names back into ``namespace`` + ``name`` on the way out. | ||
| """ | ||
|
|
||
| import json | ||
| import logging | ||
| from copy import deepcopy | ||
| from typing import Any, Iterator, Optional | ||
| from uuid import uuid4 | ||
|
|
||
| import orjson | ||
| from openai.types.responses.response_create_params import ToolParam | ||
| from pydantic import TypeAdapter, ValidationError | ||
|
|
||
|
|
@@ -238,7 +238,7 @@ def validate_streaming_responses_params(body: dict[str, Any]) -> NeMoGymResponse | |
|
|
||
|
|
||
| def _sse_event(payload: dict[str, Any]) -> str: | ||
| return f"event: {payload['type']}\ndata: {json.dumps(payload)}\n\n" | ||
| return f"event: {payload['type']}\ndata: {orjson.dumps(payload).decode()}\n\n" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. BLOCKER — Preserve Unicode separators in captured SSE payloads Failure / blast radius: The new encoder emits U+2028, U+2029, and U+0085 literally rather than as ASCII escapes. However, capture's SSE parser uses Correction: If retaining literal Unicode emission, change the physical-line splitter in data_lines = [
line[5:].lstrip()
for line in re.split(r"\r\n|\r|\n", block)
if line.startswith("data:")
]The invariant is that Unicode characters inside a JSON string never become SSE framing. This replacement uses the existing Preservation / acceptance (proposed, not run): Parameterize synthesized Responses text over U+0085/U+2028/U+2029; reconstruction must retain the exact text, completed status, and output list. Do the equivalent Anthropic synthesis/reconstruction check and assert the reconstructed text block is unchanged. Retain LF and CRLF streams, escaped |
||
|
|
||
|
|
||
| def restore_namespace_tool_calls(items: list[dict], ns_map: NamespaceMap) -> list[dict]: | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the new serializer rejects tool inputs that stdlib serialized successfully
orjson.dumps()raisesJSONEncodeError: Integer exceeds 64-bit rangefor arbitrary-precision Python integers, while stdlib emits the exact JSON number. For example:I think this should be addressed through one shared Gym serialization helper. The implementation can try
orjson.dumps()and fall back on anyorjson.JSONEncodeErrorto compact stdlib JSON.there are still some differences:
orjsonsilently serializesNaNand infinities asnullvs stdlib normally emits the non-standardNaN/Infinitytokens, whileallow_nan=Falseraises.orjsonalso natively serializes datetime/dataclass/UUID values that stdlib rejects