Repository navigation
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
orjson cannot write an integer wider than 64 bits or a lone surrogate, and it reads such an integer back as a float without an error. A model can put either into tool call arguments, so they reach request bodies, model server replies, and the rollout and capture records Gym writes and reads back. Add dumps_json_bytes and loads_json in nemo_gym.utils.json_codec. The encoder keeps orjson's bytes for every value orjson accepts and falls back to the standard library otherwise, writing NaN as null like orjson and refusing non-string keys. The decoder parses with the standard library only when the body may hold an integer orjson would change, or when orjson rejects it. Route request(), get_response_json, the model server's non-streaming reply, and the record writers, readers and hashes through them. Agents now treat a ValueError from json.loads on tool arguments, such as an integer over 4,300 digits, as invalid arguments. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
Replies and record lines arrive as bytes, so loads_json checks them for wide integers without the type dispatch the other input types need. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
The standard library writes namedtuples, float and tuple subclasses and NumPy floats, which orjson refuses, so they started encoding whenever a body took the fallback. Convert the value to plain JSON types first, accepting only what orjson accepts and calling default for the rest. loads_json no longer hands input of the wrong type to bytes(), which decoded a list of byte values or allocated an integer's worth of zeros. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
- Rename dumps_json_bytes to dumps_json, matching loads_json. - Check for a wide integer in two stages. The cheap digit-run check runs first; when it matches, a second check keeps only runs that start where a JSON number can start, so IDs, hashes and timestamps inside strings no longer send rollout and dataset lines to the standard library. - Refuse a non-contiguous memoryview before copying it, as orjson does. - Log decoder fallbacks at debug level, as the encoder does. - State in the module docstring that NaN and infinity become null, and document the fallback's second call to default and its error cases. - Route remote_agent's request and reply and the vcqa judge request through the helpers. - Catch RecursionError beside ValueError where agents parse tool arguments, and widen the rna_design_agent and verifiers_agent parsers this change missed. - Patch the merge test's serialization at dumps_json directly. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
orjson rejects a dataclass or enum whose value holds an integer wider than 64 bits, so the fallback handed the whole object to default, and default=str stringified it. Unpack them as orjson would before encoding. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
orjson does not write dataclass fields whose names start with an underscore, so the fallback must not either. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
orjson writes an instance only of a class decorated itself, and refuses an undecorated subclass of a dataclass. Apply the same rule, so the fallback neither accepts such an instance nor bypasses default for it. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
- Compare decoded round trips by repr or type: 2**70 is a power of two, so a float compares equal to it and == alone passed a lossy decoder. - Cover an invalid body holding a wide integer, a str holding a lone surrogate, nesting deeper than the standard library allows, NaN in a tuple beside a wide integer, and token-ID and logprob bodies staying on orjson. - Test the over-limit integer and deep nesting cases in the compaction, remote and tau2 agents too. Also refuse str subclasses as object keys in the encoder fallback, as orjson does. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
The fallback called default a second time for each value orjson could not write. A default that consumes its argument, such as list on a generator, then wrote an empty list when orjson had drained the generator before reaching the wide integer, so the output depended on key order. Record what default returns during the orjson pass and reuse it in the fallback. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
A default that raised during the orjson pass was called again by the fallback, so one that failed only on its first call turned a failed write into a success. Record the raised error and raise it again. Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com>
ananthsub
force-pushed
the
ananthsub/exact-json-boundaries
branch
from
November 9, 2026 03:43
ca2924a to
12063a6
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
#5049 taught
request()to send a body that orjson cannot encode. This PR finishes that fix. Any value Gym sends or stores now crosses each JSON boundary exactly, or the write fails with an error.orjson has two gaps that matter here:
[-2**63, 2**64)back as a float, and raises no error. For example,1180591620717411303424comes back as1.1805916207174113e+21.A model can put either kind of value into tool-call arguments. From there it reaches request bodies, model server replies, and the rollout and capture records Gym writes and reads back.
The one exception to "exactly": NaN and infinity become
null, as orjson already writes them. Gym now states this in the codec's module docstring. Note thatserver_runtime/_checkpoint/workers.pydeliberately uses the standard library so a checkpoint keeps NaN, which makes this a second, documented NaN policy rather than a change to that one.Shared encoder and decoder
The new module
nemo_gym/utils/json_codec.pyhas two helpers. It imports nothing from Gym, so any module can use it.dumps_json(value, *, default=None, option=None)mirrorsorjson.dumps, so each call site is a one-word swap:allow_nan=False. It honorsOPT_SORT_KEYS,OPT_INDENT_2andOPT_APPEND_NEWLINE. If the call passes any other orjson option, it raises orjson's error instead of silently dropping the option.null, as orjson does. Without this, the same NaN would meannullin one body and either fail or produce invalid JSON in another, only because the second body also held a wide integer.1and"1"never merge into one key. This is the guard from feat(capture): support framework-owned context reconstruction #4128's fingerprint fallback, moved into the shared helper.datetime, UUID, enum or dataclass is written the way orjson writes it. Dataclasses and enums are unpacked first, so a wide integer inside one is written rather than handed todefaultwith the whole object. A namedtuple, a float or tuple subclass, or a NumPy float goes todefaultif the caller passed one, and otherwise raises, as orjson does. The fallback reuses whatdefaultreturned or raised during the orjson pass, so adefaultthat consumes its argument, such asliston a generator, writes the same output whatever the key order.loads_json(data)parses with orjson, except in two cases where it uses the standard library:A whole number of 19 or more digits with no fraction or exponent now reads as an
int, as the standard library reads it. JavaScript and Go write floats from 1e19 to 1e21 in that form, so such a value changes from float to int. That follows the JSON grammar andjson.loads.The fallback logs at debug level. It still refuses
NaN,Infinityand numbers that overflow a float, as orjson does. A body both parsers reject raises orjson's ownJSONDecodeError. Input of the wrong type, or a non-contiguous memoryview, raises orjson's error before anything is copied. Existingexcept orjson.JSONDecodeErrorhandlers therefore keep working.How the decoder spots a wide integer
orjson loses an integer only when it has at least 19 digits. The check runs in two stages, each a byte translation into character classes followed by one reverse substring search:
., and other. It looks for a non-digit followed by 19 digits. A run after.is a fraction, which orjson reads exactly. Python writers never put 17 or more digits before the point, so logprobs and token IDs don't match.[,,,:or-. This drops matches inside strings, such as OpenAImsg_andresp_IDs, git hashes and nanosecond timestamps in paths. It runs second because spaces make its search slower on text replies.Across the
resources_servers/*/data/*.jsonlfiles, 6 of 2,440 valid lines still take the standard library path, down from 39 with the first stage alone. Nothing that still matches is lossy. A 19-digit integer that fits in 64 bits, or a digit string after a space inside a string, costs only a standard-library parse of that body, which gives the same values. Fuzzing during review compared the decoder withjson.loadson 620,000 random documents and found no mismatches.Call sites
Routed through the helpers:
request()andget_response_json._orjson_dispatch_response. A tool call whose arguments hold2**70returned 500 there.rollout_store,rollout_records,rollout_collectionandbatch_status.model_call_capture,token_capture/storeandtoken_capture/lineage.fingerprint._canonical_jsonnow calls the shared encoder instead of keeping its own fallback.Every call site that orjson already handles writes the same bytes as before.
Agents that parse tool arguments now treat a
ValueErrororRecursionErrorfromjson.loadslike a decode error. Those come from an integer over 4,300 digits, which is Python's limit, or nesting deeper than the interpreter's recursion limit. Before,simple_agentaborted the whole/run. Now the model sees an invalid-arguments tool output.The same change applies to:
simple_agent_with_compactionremote_agent, whose request and reply now also go through the helpers, as does thevcqa_agentjudge requestvcqa_agentbrowsecomp_agentosworld_agentrna_design_agent, in both its structured and text tool-call parsersverifiers_agenttau2, which now raises its malformed-tool-call error for this case tooLeft as is, and why
null. A test now guards it, and that test passes on the old code too._check_jsonand_plain_jsoninserver_runtime/_checkpoint/control.py. They are an acceptance check for a checkpoint store that writes with the standard library, not an encoder, and they already accept wide integers. Merging them into the encoder would change which payloads the checkpoint store accepts.except json.JSONDecodeError. An integer over 4,300 digits there fails verification with an error rather than a wrong reward. Changing each server's verifier is out of scope here.json.loadsfororjson.loadsinanthropic_converter._json_object_from_argumentsand in themodel_call_capturebody parsers. That would bring the silent float back on those paths. If it lands, it should useloads_json.Behavior changes to review
canonicalize_tool_argumentsused to raise for this value. It now writesnull, matching its output for NaN alone. A golden-vector test that expected the error now uses a set, which is still refused.TypeErrorfromorjson.dumpsnow triggers the fallback. orjson'sJSONEncodeErrorisTypeError. One merge test injected aTypeErrorthroughorjson.dumpsto simulate an interrupted serialization. It now patchesdumps_jsondirectly.Hot-path cost
The encoder costs nothing extra: orjson runs first and the standard library runs only when orjson raises. The decoder cannot work that way. orjson raises no error on a wide integer; it returns a float. So
loads_jsonchecks the bytes on every call, and that check is the cost below. orjson upstream has discussed and declined a strict decode mode (ijl/orjson#116, ijl/orjson#473). With one, the decoder could become a pure fallback too.Measured with
timeit, taking the best of seven repeats, against plain orjson on the same bytes:The text-only reply is the realistic per-turn case. With NeMo RL's external capture, token IDs and logprobs no longer travel through Gym. For that reply, decoding plus
NeMoGymResponse.model_validatetakes 18 µs instead of 14 µs: about 4 µs per 10 KB reply. The token-heavy row still applies when token IDs do travel through Gym. That includes the 249 KB token-mode vLLM chat reply the model server decodes undervllm_model_for_training.yamlandvllm_model_for_compaction.yaml, measured during review at about +165 µs per turn. There, decoding plus validation takes about 457 µs instead of 367 µs.The check is a byte translation plus one reverse substring search, with an inline path for
bytes, which is what replies and record lines arrive as. Alternatives I measured:pydantic_core.from_json: reads wide integers exactly, but costs about 50% more than orjson on every body.model_validate_json: also exact. During review it measured about the same as orjson plus validation on a text reply, and slightly cheaper on a token reply. It only applies where a call site validates into a model, so it is a possible follow-up for typed readers, not a replacement forloads_json.Why one exact reader everywhere, not a fast reader for model replies. A split would be correct only if a fast reader never saw a wide integer as a JSON number. Nothing enforces that, and it is already false:
metadataand provider-specific fields are open-ended.A wrong choice of reader would fail silently.
get_response_jsonhas hundreds of call sites, and the same bytes could decode differently depending on which reader a path used. One exact path costs about 4 µs per text reply and leaves nothing to get wrong.Tests
Each test asserts the exact value that comes back, not just that no exception was raised.
Codec unit tests
int, from bytes, str, bytearray and memoryview.defaultwhen it is given.strholding a lone surrogate decodes.null.strsubclasses, such asStrEnummembers, are refused as orjson refuses them.defaultthat drains a generator writes the same output in either key order.defaultthat raises on its first call still fails the write.repror check forint, because2**70as a float compares equal to the integer.Tool call to a resources server and back
request(), and its reply is read throughget_response_json.ints, and NaN comes back asnull.Model server replies
/v1/messagesreplies.Rollout records
RolloutStoreand loaded back withRolloutRecords. They come back exact.Agents
simple_agent,simple_agent_with_compactionandremote_agentreturn an invalid-arguments tool output naming the error instead of failing the run.tau2raises its malformed-tool-call error for the same two cases.json.loadson a string and catch the same exceptions, so these four stand in for them.Proof that the new tests fail on the old code
simple_agent,simple_agent_with_compaction,remote_agentandtau2, the fingerprint NaN case, and the updated fix(server_utils): send request bodies that hold integers wider than 64 bits #5049 request case.Validation
After rebasing onto
main, this fails exactly the same tests asmainitself, run in the same venv. Those failures come from optional dependencies missing in the local venv, such as mlflow, wandb and the sandbox tooling. CI runs those.These run with the root venv's editable finder disabled, so the worktree's agent code is the code under test. They pass. The tau2 tests pass in tau2's own venv. The Haystack, LangChain and verifiers agent tests need dependencies the local venv lacks, and are left to CI.
All hooks pass. The commit is signed off for DCO and signed.
Rollout evidence
N/A. The encode path writes byte-identical output for everything orjson already handled, which the codec tests check directly. The only agent change turns a crash into an invalid-arguments tool output, and that only happens for model output that can't be produced on demand: an integer over 4,300 digits. A smoke rollout would exercise only the unchanged orjson path.