Skip to content

fix(client): keep JSON values exact across Gym boundaries - #5784

Draft
ananthsub wants to merge 10 commits into
mainfrom
ananthsub/exact-json-boundaries
Draft

ananthsub wants to merge 10 commits into
mainfrom
ananthsub/exact-json-boundaries

Conversation

@ananthsub

@ananthsub ananthsub commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

What and why

#5049 taught request() to send a body that orjson cannot encode. This PR finishes that fix. Any value Gym sends or stores now crosses each JSON boundary exactly, or the write fails with an error.

orjson has two gaps that matter here:

  • It cannot write an integer wider than 64 bits or a string holding a lone surrogate.
  • It reads an integer outside [-2**63, 2**64) back as a float, and raises no error. For example, 1180591620717411303424 comes back as 1.1805916207174113e+21.

A model can put either kind of value into tool-call arguments. From there it reaches request bodies, model server replies, and the rollout and capture records Gym writes and reads back.

The one exception to "exactly": NaN and infinity become null, as orjson already writes them. Gym now states this in the codec's module docstring. Note that server_runtime/_checkpoint/workers.py deliberately uses the standard library so a checkpoint keeps NaN, which makes this a second, documented NaN policy rather than a change to that one.

Shared encoder and decoder

The new module nemo_gym/utils/json_codec.py has two helpers. It imports nothing from Gym, so any module can use it.

dumps_json(value, *, default=None, option=None) mirrors orjson.dumps, so each call site is a one-word swap:

  • Same bytes as before. Any value orjson accepts gets exactly the bytes orjson writes, so no existing record, hash or fingerprint changes.
  • Fallback. For a value orjson rejects, it uses the standard library with compact separators and allow_nan=False. It honors OPT_SORT_KEYS, OPT_INDENT_2 and OPT_APPEND_NEWLINE. If the call passes any other orjson option, it raises orjson's error instead of silently dropping the option.
  • NaN and infinity. The fallback writes them as null, as orjson does. Without this, the same NaN would mean null in one body and either fail or produce invalid JSON in another, only because the second body also held a wide integer.
  • Non-ASCII text stays as UTF-8, as orjson writes it. It is escaped only when the value holds a lone surrogate, so the output is always valid UTF-8.
  • Object keys. It refuses keys that are not strings rather than coerce them, so 1 and "1" never merge into one key. This is the guard from feat(capture): support framework-owned context reconstruction #4128's fingerprint fallback, moved into the shared helper.
  • Same types as orjson. The fallback accepts only what orjson accepts. A datetime, UUID, enum or dataclass is written the way orjson writes it. Dataclasses and enums are unpacked first, so a wide integer inside one is written rather than handed to default with the whole object. A namedtuple, a float or tuple subclass, or a NumPy float goes to default if the caller passed one, and otherwise raises, as orjson does. The fallback reuses what default returned or raised during the orjson pass, so a default that consumes its argument, such as list on a generator, writes the same output whatever the key order.

loads_json(data) parses with orjson, except in two cases where it uses the standard library:

  • The bytes may hold an integer that orjson would turn into a float.
  • orjson rejects the body, but the standard library reads it exactly: an escaped lone surrogate, or nesting deeper than orjson's limit of 1,024.

A whole number of 19 or more digits with no fraction or exponent now reads as an int, as the standard library reads it. JavaScript and Go write floats from 1e19 to 1e21 in that form, so such a value changes from float to int. That follows the JSON grammar and json.loads.

The fallback logs at debug level. It still refuses NaN, Infinity and numbers that overflow a float, as orjson does. A body both parsers reject raises orjson's own JSONDecodeError. Input of the wrong type, or a non-contiguous memoryview, raises orjson's error before anything is copied. Existing except orjson.JSONDecodeError handlers therefore keep working.

How the decoder spots a wide integer

orjson loses an integer only when it has at least 19 digits. The check runs in two stages, each a byte translation into character classes followed by one reverse substring search:

  1. Cheap stage, on every body. Classes are digit, ., and other. It looks for a non-digit followed by 19 digits. A run after . is a fraction, which orjson reads exactly. Python writers never put 17 or more digits before the point, so logprobs and token IDs don't match.
  2. Tighter stage, only when the first matches. Classes are digit, number-start, and other. A JSON number can only start at the beginning of the input or after whitespace, [, ,, : or -. This drops matches inside strings, such as OpenAI msg_ and resp_ IDs, git hashes and nanosecond timestamps in paths. It runs second because spaces make its search slower on text replies.

Across the resources_servers/*/data/*.jsonl files, 6 of 2,440 valid lines still take the standard library path, down from 39 with the first stage alone. Nothing that still matches is lossy. A 19-digit integer that fits in 64 bits, or a digit string after a space inside a string, costs only a standard-library parse of that body, which gives the same values. Fuzzing during review compared the decoder with json.loads on 620,000 random documents and found no mismatches.

Call sites

Routed through the helpers:

  • HTTP: request() and get_response_json.
  • Model server: the non-streaming reply _orjson_dispatch_response. A tool call whose arguments hold 2**70 returned 500 there.
  • Rollout records: the writers, readers and hashes in rollout_store, rollout_records, rollout_collection and batch_status.
  • Capture records: model_call_capture, token_capture/store and token_capture/lineage.
  • Fingerprints: fingerprint._canonical_json now calls the shared encoder instead of keeping its own fallback.
  • Other modules that read or write the same rows and payloads: reverification, rollout health, the reward profile, the comparison loaders and reports, the W&B and MLflow exporters, environment validation, and checkpoint coordination.

Every call site that orjson already handles writes the same bytes as before.

Agents that parse tool arguments now treat a ValueError or RecursionError from json.loads like a decode error. Those come from an integer over 4,300 digits, which is Python's limit, or nesting deeper than the interpreter's recursion limit. Before, simple_agent aborted the whole /run. Now the model sees an invalid-arguments tool output.

The same change applies to:

  • simple_agent_with_compaction
  • remote_agent, whose request and reply now also go through the helpers, as does the vcqa_agent judge request
  • vcqa_agent
  • browsecomp_agent
  • the LangChain and Haystack bridges
  • osworld_agent
  • rna_design_agent, in both its structured and text tool-call parsers
  • verifiers_agent
  • tau2, which now raises its malformed-tool-call error for this case too

Left as is, and why

  • The streaming Anthropic reply. It was already exact, because the reply goes through a Pydantic JSON dump before it is streamed. That dump also turns NaN into null. A test now guards it, and that test passes on the old code too.
  • _check_json and _plain_json in server_runtime/_checkpoint/control.py. They are an acceptance check for a checkpoint store that writes with the standard library, not an encoder, and they already accept wide integers. Merging them into the encoder would change which payloads the checkpoint store accepts.
  • A resources server whose reply contains a lone surrogate. FastAPI fails that reply with a 500. That is a loud failure, which is the stated bar, and the agent passes the error text back to the model. A lone surrogate does reach the tool intact: a test checks the value the endpoint sees.
  • Resources-server verifiers that parse tool arguments with their own except json.JSONDecodeError. An integer over 4,300 digits there fails verification with an error rather than a wrong reward. Changing each server's verifier is out of scope here.
  • orjson calls inside individual agents and resources servers that never see model-written values. Those are left alone.
  • Draft PR perf(model-server): swap stdlib json for orjson on remaining hot paths #3509 proposes swapping json.loads for orjson.loads in anthropic_converter._json_object_from_arguments and in the model_call_capture body parsers. That would bring the silent float back on those paths. If it lands, it should use loads_json.

Behavior changes to review

  • Non-ASCII text in fix(server_utils): send request bodies that hold integers wider than 64 bits #5049's fallback. That fallback escaped every non-ASCII character in a body holding a wide integer. The shared encoder writes it as UTF-8, as orjson does, and escapes only when a lone surrogate is present. fix(server_utils): send request bodies that hold integers wider than 64 bits #5049's request test is updated to match.
  • Fingerprints for NaN beside a wide integer. canonicalize_tool_arguments used to raise for this value. It now writes null, matching its output for NaN alone. A golden-vector test that expected the error now uses a set, which is still refused.
  • A TypeError from orjson.dumps now triggers the fallback. orjson's JSONEncodeError is TypeError. One merge test injected a TypeError through orjson.dumps to simulate an interrupted serialization. It now patches dumps_json directly.

Hot-path cost

The encoder costs nothing extra: orjson runs first and the standard library runs only when orjson raises. The decoder cannot work that way. orjson raises no error on a wide integer; it returns a float. So loads_json checks the bytes on every call, and that check is the cost below. orjson upstream has discussed and declined a strict decode mode (ijl/orjson#116, ijl/orjson#473). With one, the decoder could become a pure fallback too.

Measured with timeit, taking the best of seven repeats, against plain orjson on the same bytes:

Body Size Encode, orjson Encode, helper Decode, orjson Decode, helper
Tool call arguments 55 B 0.1 µs 0.1 µs 0.2 µs 0.3 µs
Responses create params, 20 turns 40 KB 1.5 µs 1.5 µs 37.1 µs 51.4 µs
Responses reply, text only 10 KB 0.5 µs 0.6 µs 8.6 µs 12.4 µs
Responses reply, 8k prompt and 2k generated token IDs with float32 logprobs 117 KB 92.8 µs 93.0 µs 192.7 µs 283.0 µs

The text-only reply is the realistic per-turn case. With NeMo RL's external capture, token IDs and logprobs no longer travel through Gym. For that reply, decoding plus NeMoGymResponse.model_validate takes 18 µs instead of 14 µs: about 4 µs per 10 KB reply. The token-heavy row still applies when token IDs do travel through Gym. That includes the 249 KB token-mode vLLM chat reply the model server decodes under vllm_model_for_training.yaml and vllm_model_for_compaction.yaml, measured during review at about +165 µs per turn. There, decoding plus validation takes about 457 µs instead of 367 µs.

The check is a byte translation plus one reverse substring search, with an inline path for bytes, which is what replies and record lines arrive as. Alternatives I measured:

  • A plain digit-run regex: about ten times orjson's parse time.
  • A boundary-free digit search: about as cheap as the translation alone. But float32 logprob fractions run past 19 digits, so most token replies would fall back to the standard library, which is about 5x slower.
  • pydantic_core.from_json: reads wide integers exactly, but costs about 50% more than orjson on every body.
  • Decoding straight into the model with model_validate_json: also exact. During review it measured about the same as orjson plus validation on a text reply, and slightly cheaper on a token reply. It only applies where a call site validates into a model, so it is a possible follow-up for typed readers, not a replacement for loads_json.

Why one exact reader everywhere, not a fast reader for model replies. A split would be correct only if a fast reader never saw a wide integer as a JSON number. Nothing enforces that, and it is already false:

  • Anthropic Messages replies carry tool input as a JSON object, not a string.
  • Responses metadata and provider-specific fields are open-ended.

A wrong choice of reader would fail silently. get_response_json has hundreds of call sites, and the same bytes could decode differently depending on which reader a path used. One exact path costs about 4 µs per text reply and leaves nothing to get wrong.

Tests

Each test asserts the exact value that comes back, not just that no exception was raised.

Codec unit tests

  • Values orjson accepts keep orjson's bytes, with each supported option.
  • A wide integer, a lone surrogate, and NaN mixed with a wide integer each round-trip exactly.
  • Keys that are not strings, sets, and options the fallback cannot honor all raise.
  • Integers at and just past the 64-bit limits decode as int, from bytes, str, bytearray and memoryview.
  • Long fractions, long digit strings inside JSON strings, and floats written with exponents decode exactly as orjson decodes them.
  • Bodies orjson rejects still raise its error, as do input of the wrong type and a non-contiguous memoryview.
  • Types orjson refuses are refused by the fallback too, alone and beside a wide integer, and are passed to default when it is given.
  • Long digit runs inside strings, and long fractions, decode without touching the standard library.
  • Token-ID and logprob bodies decode without touching the standard library, as bytes and as str.
  • An invalid body holding a wide integer, and nesting too deep for the standard library, raise orjson's error.
  • A str holding a lone surrogate decodes.
  • NaN inside a tuple beside a wide integer is written as null.
  • Object keys that are str subclasses, such as StrEnum members, are refused as orjson refuses them.
  • A default that drains a generator writes the same output in either key order.
  • A default that raises on its first call still fails the write.
  • Round-trip assertions compare repr or check for int, because 2**70 as a float compares equal to the integer.

Tool call to a resources server and back

  • A real uvicorn FastAPI tool endpoint receives a body sent through request(), and its reply is read through get_response_json.
  • A wide integer and an out-of-range negative come back as exact ints, and NaN comes back as null.
  • A lone surrogate reaches the tool intact.

Model server replies

  • Covers the non-streaming and streaming /v1/messages replies.
  • In both, tool-call arguments holding a wide integer, a lone surrogate and NaN decode to the exact input.

Rollout records

  • A result and a failure holding the same values are written through RolloutStore and loaded back with RolloutRecords. They come back exact.
  • The re-judgment predecessor digest is the same whether taken before the write or after the read.

Agents

  • When a tool argument holds an integer over 4,300 digits, or nesting past the recursion limit, simple_agent, simple_agent_with_compaction and remote_agent return an invalid-arguments tool output naming the error instead of failing the run.
  • tau2 raises its malformed-tool-call error for the same two cases.
  • The other agents call json.loads on a string and catch the same exceptions, so these four stand in for them.

Proof that the new tests fail on the old code

Validation

PYTHONPATH=$PWD ~/dev/Gym/.venv/bin/python -m pytest tests/unit_tests -n 16 --ignore=tests/unit_tests/artifacts/test_exporters.py

After rebasing onto main, this fails exactly the same tests as main itself, run in the same venv. Those failures come from optional dependencies missing in the local venv, such as mlflow, wandb and the sandbox tooling. CI runs those.

PYTHONPATH=$PWD ~/dev/Gym/.venv/bin/python -m pytest responses_api_agents/{simple_agent,simple_agent_with_compaction,remote_agent,vcqa_agent,browsecomp_agent,osworld_agent,rna_design_agent}/tests -n 8

These run with the root venv's editable finder disabled, so the worktree's agent code is the code under test. They pass. The tau2 tests pass in tau2's own venv. The Haystack, LangChain and verifiers agent tests need dependencies the local venv lacks, and are left to CI.

pre-commit run --files <changed files>

All hooks pass. The commit is signed off for DCO and signed.

Rollout evidence

N/A. The encode path writes byte-identical output for everything orjson already handled, which the codec tests check directly. The only agent change turns a crash into an invalid-arguments tool output, and that only happens for model output that can't be produced on demand: an integer over 4,300 digits. A smoke rollout would exercise only the unchanged orjson path.

@copy-pr-bot

copy-pr-bot Bot commented Nov 7, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added bug Something isn't working area:core Shared APIs, servers, telemetry, health, and registries needs-review PR is ready for code review and waiting on a reviewer labels Nov 7, 2026
Base automatically changed from cmunley1/fingerprint-bigint-args to main November 8, 2026 00:47
orjson cannot write an integer wider than 64 bits or a lone surrogate, and
it reads such an integer back as a float without an error. A model can put
either into tool call arguments, so they reach request bodies, model server
replies, and the rollout and capture records Gym writes and reads back.

Add dumps_json_bytes and loads_json in nemo_gym.utils.json_codec. The
encoder keeps orjson's bytes for every value orjson accepts and falls back
to the standard library otherwise, writing NaN as null like orjson and
refusing non-string keys. The decoder parses with the standard library only
when the body may hold an integer orjson would change, or when orjson
rejects it. Route request(), get_response_json, the model server's
non-streaming reply, and the record writers, readers and hashes through
them. Agents now treat a ValueError from json.loads on tool arguments,
such as an integer over 4,300 digits, as invalid arguments.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Replies and record lines arrive as bytes, so loads_json checks them for wide
integers without the type dispatch the other input types need.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
The standard library writes namedtuples, float and tuple subclasses and
NumPy floats, which orjson refuses, so they started encoding whenever a
body took the fallback. Convert the value to plain JSON types first,
accepting only what orjson accepts and calling default for the rest.

loads_json no longer hands input of the wrong type to bytes(), which
decoded a list of byte values or allocated an integer's worth of zeros.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
- Rename dumps_json_bytes to dumps_json, matching loads_json.
- Check for a wide integer in two stages. The cheap digit-run check runs
  first; when it matches, a second check keeps only runs that start where
  a JSON number can start, so IDs, hashes and timestamps inside strings no
  longer send rollout and dataset lines to the standard library.
- Refuse a non-contiguous memoryview before copying it, as orjson does.
- Log decoder fallbacks at debug level, as the encoder does.
- State in the module docstring that NaN and infinity become null, and
  document the fallback's second call to default and its error cases.
- Route remote_agent's request and reply and the vcqa judge request
  through the helpers.
- Catch RecursionError beside ValueError where agents parse tool
  arguments, and widen the rna_design_agent and verifiers_agent parsers
  this change missed.
- Patch the merge test's serialization at dumps_json directly.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
orjson rejects a dataclass or enum whose value holds an integer wider than
64 bits, so the fallback handed the whole object to default, and
default=str stringified it. Unpack them as orjson would before encoding.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
orjson does not write dataclass fields whose names start with an
underscore, so the fallback must not either.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
orjson writes an instance only of a class decorated itself, and refuses an
undecorated subclass of a dataclass. Apply the same rule, so the fallback
neither accepts such an instance nor bypasses default for it.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
- Compare decoded round trips by repr or type: 2**70 is a power of two,
  so a float compares equal to it and == alone passed a lossy decoder.
- Cover an invalid body holding a wide integer, a str holding a lone
  surrogate, nesting deeper than the standard library allows, NaN in a
  tuple beside a wide integer, and token-ID and logprob bodies staying
  on orjson.
- Test the over-limit integer and deep nesting cases in the compaction,
  remote and tau2 agents too.

Also refuse str subclasses as object keys in the encoder fallback, as
orjson does.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
The fallback called default a second time for each value orjson could
not write. A default that consumes its argument, such as list on a
generator, then wrote an empty list when orjson had drained the
generator before reaching the wide integer, so the output depended on
key order. Record what default returns during the orjson pass and
reuse it in the fallback.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
A default that raised during the orjson pass was called again by the
fallback, so one that failed only on its first call turned a failed
write into a success. Record the raised error and raise it again.

Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
@ananthsub
ananthsub force-pushed the ananthsub/exact-json-boundaries branch from ca2924a to 12063a6 Compare November 9, 2026 03:43

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Shared APIs, servers, telemetry, health, and registries bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants