Skip to content

json5: parse hex numbers wider than 64 bits as the nearest double instead of throwing - #42136

Open
robobun wants to merge 1 commit into
mainfrom
robobun/8d349f54/json5-hex-wider-than-u64
Open

robobun wants to merge 1 commit into
mainfrom
robobun/8d349f54/json5-hex-wider-than-u64

Conversation

@robobun

@robobun robobun commented Sep 9, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • Bun.JSON5.parse("0x1ffffffffffffffff") throws SyntaxError: JSON5 Parse error: Invalid hex number. The json5 package and the JS literal give 36893488147419103000 (2^65). .json5 imports fail the same way.
  • scan_hex_number() (src/parsers/json5.rs:1039) parses the digits with parse_int::<u64> and maps the overflow to InvalidHexNumber.

Fix

  • scan_hex_number() collects the digits in bun_core::fmt::RadixInteger (Round long hexadecimal, octal and binary literals to the nearest double #45290), as the JavaScript lexers and the JSONC parser do. The result is the nearest double, or Infinity past Number.MAX_VALUE.
  • Values below 2^64 do not change: the old u64 as f64 cast also rounded once.
  • Two tests replace error messages > hex number too large, which pinned the throw.
  • Verified: test/js/bun/json5/json5.test.ts (both new tests fail on 1.4.3). Also json5-test-suite.test.ts, jsonc.test.ts and the JSON5 tests of test/cli/lint/js-parsers.test.ts.

Background

  • JSON5 numbers are ES5 NumericLiterals, so a hex number has no digit limit. The json5 package returns Number("0x..."): the nearest double, ties to even.
  • RadixInteger keeps the leading digits in a u64 and counts the bits of the rest. to_f64() rounds once.
  • Considered a u64 fast path, with RadixInteger only on overflow: two paths, and slower from three digits up.

Downsides

  • A hex number of 2^64 or more no longer throws. Code that relied on the SyntaxError now gets a rounded Number or Infinity.
  • Instructions per hex number: 320 to 341 for 1 digit, 667 to 614 for 8 digits (method and table in Notes).
  • Size: 48 bytes less text and 8 bytes more data in an optimized build of bun_parsers.
Notes
  • History: this PR was stacked on Bump WebKit (oven-sh/WebKit#597 preview): parseInt, Number() and eval round binary, octal and hex digits to the nearest double #42112 and called parse_power_of_two_radix_digits, a helper backed by JSC that Bump WebKit (oven-sh/WebKit#597 preview): parseInt, Number() and eval round binary, octal and hex digits to the nearest double #42112 added. Round long hexadecimal, octal and binary literals to the nearest double #45290 then put RadixInteger on main and Bump WebKit (oven-sh/WebKit#597 preview): parseInt, Number() and eval round binary, octal and hex digits to the nearest double #42112 dropped its helper. This PR is now based on main and needs no WebKit change.
  • bun lint already states this behavior. json/no-unsafe-values reports "The number '+0xf..f' will evaluate to Infinity" for a 300-digit hex number in a .json5 file (test/cli/lint/js-parsers.test.ts). On main, Bun.JSON5.parse throws for that number.
  • Sweep test: it takes the hex cases of integersThatNeedRounding() (test/harness.ts, from Round long hexadecimal, octal and binary literals to the nearest double #45290), adds each with - and with +, and compares with Number(BigInt(text)). That is 989 numbers, 203 of the 331 unsigned ones are 2^64 or more.
  • Reference: json5@2.2.3 on node v26.3.0 returns the expected value for each explicit assertion and for all 989 numbers of the sweep. Number(BigInt) gives the same bits in JSC and in V8 for them.
  • Fuzz: Bun.JSON5.parse("0x" + digits) equals Number(BigInt("0x" + digits)) for 52,773 digit strings, each also negated: 0 mismatches. The strings are random runs of 1 to 300 digits, a tie-biased band of 53 to 80 bits, 2^k ± small with zero and non-zero tails, the Number.MAX_VALUE / Infinity boundary, and one number of 100,001 digits.
  • Measurement: a standalone optimized build of bun_parsers (the setup of scripts/bench-json-rust.sh) parses an array of 1,000 copies of one hex number 2,000 times. perf_event_open counts the user-space instructions. Repeated runs agree within 700 instructions of 1.3 billion. Per number, before and after: 0x0 320.23 and 341.23, 0xF 331.23 and 341.23, 0xFF 379.23 and 380.23, 0xFFF 427.23 and 419.23, 0xFFFF 475.23 and 458.23, 0xFFFFFF 571.23 and 536.23, 0xDEADBEEF 667.23 and 614.23, 12 digits 859.23 and 770.23, 16 digits 1051.23 and 926.23. Each count is the whole cost of one array element. perf, valgrind and bloaty are not installed in my environment, so the sizes come from size: text 605,095 and 605,047 bytes, data 25,008 and 25,016 bytes. These are not sizes of the release bun binary.
  • The extra cost for a number of one digit is the powi call in RadixInteger::to_f64(). An early return for dropped_bits == 0 there would remove it for all five users. This PR does not change the shared helper.
  • Lint tests: I ran these tests of test/cli/lint/js-parsers.test.ts one at a time: "lints JSON, JSONC and JSON5 without the eslint package", "a rejected file gets exactly one fatal problem", "accepts and rejects the right inputs", "follows the spec where the plugin's parser doesn't", "parses JSON5-only syntax in JSON5" and "all of the plugin's rules". All pass. Under my debug build the tests of that file exceed the 5 second limit when they run at once, and "many keys out of order, with comments" takes 7.4 seconds alone. So I ran them one at a time with a longer limit. I also ran the json5 case of test/bundler/bundler_loader.test.ts.
  • No allocation is added. SourceTokens (the as_written mode that bun lint reads tokens with) returns before the conversion, as before.
  • Not changed: Bun.YAML.parse returns a string for a 0x scalar of 2^64 or more (src/parsers/yaml.rs). That is a schema question for the YAML parser.
  • An earlier self-review raised one concern: a second rounding helper beside the shared one. RadixInteger resolves it. The new tests build long digit runs with Buffer.alloc(n, "0").toString(), as the review of the first version asked.

[human-review] gate passed · iteration 1 · 2 files touched

fails on main (without fix)
ASAN without fix: 2 FAILED
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/pr_gate.xml" "test/js/bun/json5/json5.test.ts"
bun test v1.4.4 (367d939d9)

test/js/bun/json5/json5.test.ts:
(pass) escape sequences > \v vertical tab [2.66ms]
(pass) escape sequences > \0 null character [1.85ms]
(pass) escape sequences > \0 followed by non-digit [1.35ms]
(pass) escape sequences > \0 followed by digit throws [8.22ms]
(pass) escape sequences > \1 through \9 throw [8.22ms]
(pass) escape sequences > \xHH hex escape [1.97ms]
(pass) escape sequences > \xHH hex escape lowercase [1.41ms]
(pass) escape sequences > \xHH hex escape high byte [1.91ms]
(pass) escape sequences > \xHH hex escape null [8.39ms]
(pass) escape sequences > \x with insufficient hex digits throws [7.17ms]
(pass) escape sequences > \uHHHH unicode escape A [1.56ms]
(pass) escape sequences > \uHHHH unicode escape e-acute [1.59ms]
(pass) escape sequences > \uHHHH unicode escape CJK [1.18ms]
(pass) escape sequences > \u with insufficient hex digits throws [5.09ms]
(pass) escape sequences > hex and unicode escape errors point at the first byte that is not a hex digit [433.76m
... (truncated)

release without fix: 4 FAILED
bun test v1.4.3-canary.1 (367d939d9)

test/js/bun/json5/json5.test.ts:
(pass) escape sequences > \v vertical tab [24.61ms]
(pass) escape sequences > \0 null character [0.23ms]
(pass) escape sequences > \0 followed by non-digit [0.04ms]
(pass) escape sequences > \0 followed by digit throws [21.25ms]
(pass) escape sequences > \1 through \9 throw [0.28ms]
(pass) escape sequences > \xHH hex escape [0.06ms]
(pass) escape sequences > \xHH hex escape lowercase [0.02ms]
(pass) escape sequences > \xHH hex escape high byte [0.04ms]
(pass) escape sequences > \xHH hex escape null [0.01ms]
(pass) escape sequences > \x with insufficient hex digits throws [0.08ms]
(pass) escape sequences > \uHHHH unicode escape A [0.01ms]
(pass) escape sequences > \uHHHH unicode escape e-acute [0.02ms]
(pass) escape sequences > \uHHHH unicode escape CJK [0.01ms]
(pass) escape sequences > \u with insufficient hex digits throws [0.05ms]
(pass) escape sequences > hex and unicode escape errors point at the first byte that is not a hex digit [84.53ms]
(pass) escape sequences > surrogate pairs [0.08ms]
(pass) escape sequences > identity escapes [0.02ms]
(pass) escape sequences > identity escape single c
... (truncated)
passes on PR (with fix)
ASAN with fix: all passed
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test "--reporter=junit" "--reporter-outfile=/tmp/pr_gate.xml" "test/js/bun/json5/json5.test.ts"
rustup spent 7s installing the pinned toolchain (nightly-2026-09-15); it was missing or incomplete on this machine
bun test v1.4.4 (367d939d9)

test/js/bun/json5/json5.test.ts:
(pass) escape sequences > \v vertical tab [2.80ms]
(pass) escape sequences > \0 null character [1.31ms]
(pass) escape sequences > \0 followed by non-digit [5.63ms]
(pass) escape sequences > \0 followed by digit throws [27.65ms]
(pass) escape sequences > \1 through \9 throw [8.04ms]
(pass) escape sequences > \xHH hex escape [1.39ms]
(pass) escape sequences > \xHH hex escape lowercase [1.57ms]
(pass) escape sequences > \xHH hex escape high byte [1.22ms]
(pass) escape sequences > \xHH hex escape null [1.47ms]
(pass) escape sequences > \x with insufficient hex digits throws [6.62ms]
(pass) escape sequences > \uHHHH unicode escape A [1.28ms]
(pass) escape sequences > \uHHHH unicode escape e-acute [1.65ms]
(pass) escape sequences > \uHHHH unicode escape CJK [1.75ms]
(pass) escape sequences > \u with insufficient hex digits throws [6.98
... (truncated)

release with fix: all passed
$ bun scripts/build.ts --profile=release
[configured] bun-profile → bun (stripped)
  target       linux-x64-gnu
  build type   Release
  build dir    ./build/release
  revision     35e913cc6e
  features     lto, baseline

23 deps, 228 codegen, 1218 objects in 2445ms

ninja: Entering directory `/workspace/bun/build/release'
[1/4] fetch lolhtml
[lolhtml] up to date
[2/4] fetch rust-argon2
[rust-argon2] up to date
[2/4] cargo plan → /workspace/bun/build/release/rust-target/plan.json
270 units: 194 lib, 16 proc-macro (host), 21 custom-build (host), 17 run custom-build, 17 lib (host), 4 run custom-build (host), 1 rlib
[3/4] reconfigure
[1/1577] mkdir stamps
[2/1577] mkdir codegen
[3/1577] gen bake.{client,server,error}.js
-> bake.client.js, bake.server.js, bake.error.js
[4/1577] install /workspace/bun
bun install v1.4.3-canary.1 (367d939d9)

Checked 22 installs across 44 packages (no changes) [5.59s]
[5/1577] install /workspace/bun/packages/bun-error
bun install v1.4.3-canary.1 (367d939d9)

Checked 1 install across 2 packages (no changes) [227.00ms]
[6/1577] rustc unicode_ident 
[7/1577] rustc unicode_xid 
[8/1577] gen string-map defines_table
[9/1577] rustc heck 
[1
... (truncated)
diff hotspot
src/parsers/json5.rs            | 13 +++++++++----
 test/js/bun/json5/json5.test.ts | 38 ++++++++++++++++++++++++++++++++------
 2 files changed, 41 insertions(+), 10 deletions(-)

gate history · 1 passed · 0 rejected · iteration 1

evidence per changed file
file                             reads  edits  tests
src/parsers/json5.rs                 5      5     31
test/js/bun/json5/json5.test.ts      7      6     28

@github-actions github-actions Bot added the claude label Sep 9, 2026
@robobun

robobun commented Sep 9, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 5:10 AM PT - Oct 22nd, 2026

✅ @robobun, your commit 35e913cc6e12a2c2c04deb64ca5168726cebb305 passed in Build #130727! 🎉


🧪   To try this PR locally:

bunx bun-pr 42136

That installs a local version of the PR into your bun-42136 executable, so you can run:

bun-42136 --bun

@robobun

robobun commented Sep 9, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status

Reproduced on Bun 1.4.3: Bun.JSON5.parse("0x1ffffffffffffffff") and Bun.JSON5.parse("0xFFFFFFFFFFFFFFFFFF") throw SyntaxError: JSON5 Parse error: Invalid hex number. json5@2.2.3 on node returns 36893488147419103000 and 4.722366482869645e+21.

The two new tests in test/js/bun/json5/json5.test.ts fail on 1.4.3 (USE_SYSTEM_BUN=1 bun test test/js/bun/json5/json5.test.ts -t "hex number") and pass with this change (bun bd test test/js/bun/json5/json5.test.ts).

This PR is rebased onto main and no longer depends on #42112. It uses bun_core::fmt::RadixInteger from #45290.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

LGTM — swaps the u64-capped parse for the shared radix→double helper already used by the lexer and JSONC parser, with thorough boundary and tie-to-even coverage. The inline .repeat note is a convention nit only; string lengths here are ≤256 so it's not load-bearing.

Extended reasoning...

Overview

The PR changes scan_hex_number() in src/parsers/json5.rs to route hex digit runs through bun_core::fmt::parse_power_of_two_radix_digits(digits, 16) instead of parse_int::<u64>. This lifts the artificial 64-bit ceiling and matches ES HexIntegerLiteral / Number("0x…") semantics, which is what the reference json5 npm package does. The scanner loop above already guarantees the slice contains only ASCII hex digits and is non-empty, so the helper's precondition is satisfied and the removed error branch was genuinely unreachable for well-formed digit runs. The same helper is already the established pattern in src/js_parser/lexer.rs and src/parsers/json_stage2.rs, so this is a consistency fix as much as a correctness one.

Test additions in test/js/bun/json5/json5.test.ts cover the variant matrix well: values just past 2^64, signed forms, uppercase 0X with leading zeros, embedded-in-object/array contexts, the Number.MAX_VALUE / Infinity boundary at both sides of the half-ulp tie, and explicit round-half-to-even cases at 2^53 and 2^64 with a Number(BigInt(...)) oracle loop. The deleted "hex number too large" negative test's exact input (0xFFFFFFFFFFFFFFFFFF) reappears as a positive .toBe(2 ** 72) assertion, satisfying the "every deleted test needs a stated replacement" rule.

Security risks

None. This is numeric-literal parsing with no allocation proportional to input beyond what the scanner already advanced; the helper takes a bounded slice of pre-validated hex digits. No untrusted-length arithmetic, no I/O, no auth/crypto surface.

Level of scrutiny

Low-to-moderate. Five-line Rust change swapping one bun_core::fmt helper for another that's already exercised on identical input shapes elsewhere in the tree; the interesting correctness work (correct rounding) lives in the stacked #42112 helper, not here. No CODEOWNERS entry covers src/parsers/ or test/js/bun/json5/.

Other factors

The one confirmed finding is a test-harness convention nit (.repeat(n) vs Buffer.alloc(n, fill).toString()) at n ≤ 256, which the finding itself notes has negligible cost. It doesn't affect correctness or CI runtime meaningfully and doesn't warrant holding the PR. No prior reviews or outstanding objections in the timeline; exit reason was dry_streak.

Comment thread test/js/bun/json5/json5.test.ts Resolved
@robobun

robobun commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator Author

Looked at the review. The one note was the optional .repeat convention nit in the new tests. I kept .repeat: each call repeats a single character at most 256 times, which JSC handles on its native single-character path, and the test runs in about 7 ms under the debug ASAN build. The thread is resolved with that explanation. No code change.

@robobun
robobun changed the base branch from robobun/77c9783c/webkit-parseint-pow2-radix to main October 20, 2026 23:14
@robobun
robobun force-pushed the robobun/8d349f54/json5-hex-wider-than-u64 branch from 329973e to 0104327 Compare October 20, 2026 23:14
@coderabbitai

coderabbitai Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: oven-sh/bun/.coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: df044311-c6ba-48f7-a1cb-162e563d0ac2
📥 Commits

Reviewing files that changed from the base of the PR and between 9b4284f and 0104327.

📒 Files selected for processing: 2
  • src/parsers/json5.rs
  • test/js/bun/json5/json5.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 2 remain after this review.

Walkthrough

JSON5 hexadecimal parsing now supports literals beyond the u64 range by converting them through RadixInteger to the nearest f64. Tests cover large and signed values, rounding, overflow to Infinity, and generated values compared with JavaScript Number.

Changes

JSON5 hexadecimal parsing

Layer / File(s) Summary
Hexadecimal conversion and tests
src/parsers/json5.rs, test/js/bun/json5/json5.test.ts
scan_hex_number converts hexadecimal literals through RadixInteger. Tests cover large and signed values, rounding, finite and infinite results, and remove the expectation that an oversized literal is invalid.

Suggested reviewers: jarred-sumner

Priority: ⬇️ Low

Merge Risk: ⚪ Minimal · up to 01043

The JSON5 hex conversion is ready to merge after normal checks; no actionable issue remains.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title clearly and concisely describes the primary change: parsing JSON5 hexadecimal numbers wider than 64 bits as the nearest double instead of throwing.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description clearly explains the problem, implementation, behavior change, tradeoffs, and verification. It does not use the template headings exactly, but it provides the required information in e…
Full details: Description check

Explanation

The description clearly explains the problem, implementation, behavior change, tradeoffs, and verification. It does not use the template headings exactly, but it provides the required information in equivalent sections.

✨ Finishing Touches: 💡 1
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — small, self-contained change that routes JSON5 hex literals through the shared RadixInteger helper, and the earlier .repeat nit is addressed with Buffer.alloc.

What was reviewed:

  • scan_hex_number (src/parsers/json5.rs:1038-1046): the scanner's is_ascii_hexdigit pre-filter and to_digit(16) accept exactly the same bytes, so filter_map drops nothing; push_digit(4, ..) only starts dropping once leading holds 61+ bits, so the sticky-bit rounding in to_f64 is correct and all-zero digit runs stay exact 0.
  • Sign and -Infinity are applied by the caller scan_signed_value, so -0x1 followed by 256 zeros gives -Infinity as the test asserts; the as_written early return used by bun lint is untouched and lint already reports such literals as evaluating to Infinity.
  • The deleted "hex number too large" test is replaced by a positive assertion on the same literal plus a deterministic (seeded) sweep against Number(BigInt) that guards against an empty case list and skips 0n so -0 is not compared with +0.
Extended reasoning...

The diff touches one function in src/parsers/json5.rs (8 lines) and test/js/bun/json5/json5.test.ts, replacing a u64 parse-with-overflow-error by the existing bun_core::fmt::RadixInteger used by four sibling radix-number sites, so hex literals of any width become the nearest double or Infinity as json5 and ECMAScript specify. It touches no security-sensitive surface; the input is already bounded to hex digits by the scanner and no allocation is added. The new commit addresses the only prior inline comment, no other reviewer objection is recorded, the changed files have no CODEOWNER, the hunt exited by dry_streak with no findings, and the tests cover 2^64 boundaries, both signs, the finite/Infinity limit, and a seeded rounding sweep.

…d of rejecting them

Bun.JSON5.parse("0x1ffffffffffffffff") threw "Invalid hex number" because
scan_hex_number() parsed the digits into a u64. A JSON5 hex number is an
ECMAScript HexIntegerLiteral, a Number of any magnitude, and the json5
package returns Number("0x...") for it. Collect the digits in
bun_core::fmt::RadixInteger, which the JavaScript lexers and the JSONC
parser already use. It rounds once to the nearest double, ties to even, and
gives Infinity past Number.MAX_VALUE.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review found no issues

No high-confidence issues detected in this change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant