Repository navigation
Conversation
The fetch Body mixin defines text() as 'UTF-8 decode' and json() as
'parse JSON from bytes', both of which (per the Encoding Standard)
remove a leading U+FEFF byte-order mark.
When a Request/Response is constructed with a JS string body, Bun stores
the WTFStringImpl directly and hands it back to .text()/.json() without
the UTF-8 encode/decode round trip, so a leading BOM was preserved.
This meant new Response('\ufeff{...}').json() rejected with a syntax
error while the same payload via a Blob or Uint8Array body parsed fine,
and diverged from Node and browsers.
Strip a leading U+FEFF in the WTFStringImpl arms of Any::to_string and
Any::to_json, matching the sibling Blob/InternalBlob arms which already
strip the EF BB BF prefix from byte-backed bodies.
|
Updated 3:05 PM PT - Jul 28th, 2026
❌ @robobun, your commit b367d5e has some failures in 🧪 To try this PR locally: bunx bun-pr 33656That installs a local version of the PR into your bun-33656 --bun |
|
Warning Review limit reached
Next review available in: 3 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (2)
WalkthroughModifies Blob.rs's Any::to_json and Any::to_string to detect and skip a leading UTF-8 BOM character in WTFStringImpl strings before JSON parsing or string conversion. Adds tests validating this behavior for Response and Request objects constructed with string bodies. ChangesUTF-8 BOM Stripping
Possibly related PRs
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
test/js/web/fetch/utf8-bom.test.ts (1)
1-114: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick winMove these tests into the existing
fetch.test.tsinstead of a new file.This creates a new dedicated test file for a fetch/
Response/Requestbehavior change, but the project convention requires new tests to be added to the existing test file for the module being changed unless it qualifies for the reserved regression path (which this doesn't, since BOM-stripping wasn't previously working and then broke).As per coding guidelines: "Default: add your test to the existing test file for the code you're changing. Do not create a new file. A fetch bug goes in
test/js/web/fetch/fetch.test.ts, aBun.servebug goes intest/js/bun/http/serve.test.ts, and so on."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/js/web/fetch/utf8-bom.test.ts` around lines 1 - 114, The BOM-stripping coverage was added in a new dedicated test file, but this behavior belongs with the existing fetch module tests. Move the UTF-8 BOM cases into the current fetch test suite in fetch.test.ts, keeping the same coverage for Blob, Response, and Request methods like text(), json(), and formData(), and remove the standalone file so the module’s tests stay consolidated.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/js/web/fetch/utf8-bom.test.ts`:
- Around line 95-113: The Request BOM test coverage is incomplete compared with
Response, so extend the existing Request (string body) cases in the utf8-bom
test to include the same edge scenarios: BOM-only bodies, double-BOM stripping
only one leading BOM, interior BOM preservation, and emoji handling for both
text() and json(). Reuse the Request constructor and its text()/json()
assertions so this mirrors the BodyMixin/Any::to_string/Any::to_json behavior
already exercised by Response.
---
Outside diff comments:
In `@test/js/web/fetch/utf8-bom.test.ts`:
- Around line 1-114: The BOM-stripping coverage was added in a new dedicated
test file, but this behavior belongs with the existing fetch module tests. Move
the UTF-8 BOM cases into the current fetch test suite in fetch.test.ts, keeping
the same coverage for Blob, Response, and Request methods like text(), json(),
and formData(), and remove the standalone file so the module’s tests stay
consolidated.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 9f26afd2-8811-46f9-86a4-bcdc41f5d6b2
📒 Files selected for processing (2)
src/runtime/webcore/Blob.rstest/js/web/fetch/utf8-bom.test.ts
|
Addressed review feedback in 627fc19:
|
There was a problem hiding this comment.
LGTM — brings the string-body text()/json() path in line with the spec and the sibling byte-body arms.
Extended reasoning...
Overview
This PR adds a leading U+FEFF BOM strip to the Any::WTFStringImpl arms of Any::to_json and Any::to_string in src/runtime/webcore/Blob.rs (10 lines), and appends a describe.each([Response, Request]) block to the existing test/js/web/fetch/utf8-bom.test.ts covering ASCII/emoji payloads, BOM-only, double-BOM (only one stripped), and interior BOM (preserved).
Security risks
None. This is pure output normalization on the fetch Body mixin — no auth, crypto, permissions, or untrusted-input parsing surface is touched. The added char_at(0) is guarded by a length check in both arms (length() == 0 early-return in to_json, length() > 0 && short-circuit in to_string), so there's no OOB read.
Level of scrutiny
Low-to-medium. The change is small, mechanical, and spec-driven: the fetch Body mixin defines text() as "UTF-8 decode" which removes a leading EF BB BF, and the sibling Any::Blob/Any::InternalBlob arms already do this via strings::BOM::detect_and_split. char_at returns u16 so the == 0xFEFF comparison is correct for 16-bit storage and a no-op for 8-bit Latin-1 (which can't hold U+FEFF anyway). substring(1) returns a borrowed ZigString view over the OwnedString's storage, which stays alive through the to_js/to_js_by_parse_json call before Drop releases the WTF ref — the ownership comments already in the surrounding code cover this. The no-BOM common path is byte-for-byte unchanged.
Other factors
- The PR description shows 35 pass on the debug build and 12/14 new cases failing on system Bun (the two interior-BOM cases were always correct), satisfying the fails-for-the-right-reason requirement.
- Author ran
body.test.ts,response.test.ts,body-mixin-errors.test.ts(373 pass, including string-body leak tests) andBUN_JSC_validateExceptionChecks=1. - CodeRabbit's only feedback (parametrize over Request) was addressed in 627fc19 and the thread is resolved.
- No prior claude[bot] reviews on this PR; no CODEOWNERS entries for the touched paths.
- The bug hunting system found no issues.
There was a problem hiding this comment.
LGTM — small spec-aligned fix bringing the string-body arm to parity with the byte-body arms, with thorough edge-case coverage.
Extended reasoning...
Overview
Two match arms in src/runtime/webcore/Blob.rs (Any::to_json and Any::to_string, the WTFStringImpl variant) now check for a leading U+FEFF code unit and, if present, operate on str.substring(1) instead. Tests in test/js/web/fetch/utf8-bom.test.ts add a describe.each block over Response/Request string bodies covering BOM-only, double-BOM (strip one), interior BOM (preserved), and emoji payloads for both text() and json().
Security risks
None. This is a pure string-prefix check on data the caller supplied to their own Request/Response constructor; no auth, crypto, filesystem, or network parsing is involved.
Level of scrutiny
Low-to-moderate. The change is ~10 lines of Rust and mirrors what the sibling Any::Blob / Any::InternalBlob arms already do via strings::BOM::detect_and_split / without_utf8_bom. It aligns Bun with the fetch spec ("UTF-8 decode" strips a leading BOM) and with Node/browser behavior. I verified:
char_at(0)is guarded bylength() > 0in both call sites (satisfying itsdebug_assert).char_atreturnsu16; on 8-bit Latin1 storage the max is 0xFF, so the== 0xFEFFcheck can only fire on 16-bit storage — which is exactly where a JS-string BOM would land.substring(1)returns aZigString-backed borrowed view into the WTFStringImpl's UTF-16 buffer; theOwnedStringstrholds the +1 WTF ref and drops it at scope exit, so the storage is valid throughoutto_js()/to_js_by_parse_json().
Other factors
The bug-hunting system found nothing. CodeRabbit's only feedback (parametrize Request coverage to match Response) was addressed in 627fc19. The PR description reports the new tests fail on system Bun and pass on the debug build, plus clean runs of the neighboring body/response test suites and BUN_JSC_validateExceptionChecks=1. No CODEOWNERS entry covers this path.
|
CI status: the diff is green. Remaining red is unrelated infrastructure:
Ready for a maintainer to retry the red lanes or merge. |
Move the fix from consumption time (Any::to_string/to_json) to construction time (body::Value::from_js). A string BodyInit's bytes are its UTF-8 encoding (fetch 'extract a body', USVString), and text()/json() are 'UTF-8 decode' of those bytes, which strips a leading BOM and cannot produce lone surrogates. The previous fast path stored the raw JS string and handed it straight back from text(), so both a leading U+FEFF and any lone surrogate survived, while arrayBuffer()/bytes() of the same body would encode them away. Now, when the incoming string is 16-bit and either starts with U+FEFF or fails simdutf::validate::utf16le, encode it to UTF-8 and store as InternalBlob so every consumer (text/json/arrayBuffer/bytes/blob) agrees. 8-bit Latin-1 strings cannot hold U+FEFF or surrogates, so they (and all well-formed 16-bit strings) keep the zero-copy WTFStringImpl path. Tests: add lone-surrogate cases and a parametrized text() === new TextDecoder().decode(arrayBuffer()) invariant over a dozen string shapes, run for both Request and Response.
|
The diff is ready. Verified locally at b367d5e on a clean tree: The internal fail-before gate is bouncing on its own build environment: both ASAN lanes report |
There was a problem hiding this comment.
I didn't find any bugs, but this changes body::Value::from_js — the construction path every string Response/Request body flows through — and adds a simdutf::validate::utf16le scan to every 16-bit string body, so it's worth a human look at the perf/behavior tradeoff before merge.
Checked: utf16[0] is guarded by the length() == 0 early return above; bun_core::String is Copy with no Drop, so the explicit str.deref() correctly releases the +1 from to_bun_string without a double-deref; the utf16le validator on host-native &[u16] matches the existing pattern in shell_body.rs (Bun is LE-only); the 8-bit Latin-1 arm is correctly skipped since it can't hold U+FEFF or surrogates.
Extended reasoning...
Overview
Two files: a ~12-line addition to body::Value::from_js in src/runtime/webcore/Body.rs that detects a leading BOM or lone surrogates in 16-bit string bodies and eagerly transcodes them to UTF-8 bytes (InternalBlob) instead of storing the raw WTFStringImpl; and ~70 lines of new test cases in test/js/web/fetch/utf8-bom.test.ts covering Response/Request string bodies via describe.each.
Security risks
None identified. No user-controlled paths, no auth/crypto, no untrusted-size arithmetic. The only new indexing (utf16[0]) is guarded by the pre-existing str.length() == 0 → Value::Empty return.
Level of scrutiny
Medium-high. from_js runs for every new Response(str) / new Request(url, {body: str}), so this is a hot, production-critical path. The change is a behavior change (spec-aligning, matches Node/browsers) and adds a full-string SIMD validation pass to every 16-bit string body. That's likely fine — simdutf is fast and 16-bit strings are already the slow-ish path — but a maintainer should sign off on adding work to this constructor.
Other factors
- Memory management verified:
bun_core::Stringis#[derive(Copy)]with noDropimpl (perstring/mod.rs:1251-1255), so the explicitstr.deref()on the new early-return path is the sole release of the +1 ref fromto_bun_string()— no double-deref on scope exit. Theutf16borrow ends atto_utf8_alloc(utf16)before the deref. simdutf::validate::utf16leon host-endian&[u16]mirrorssrc/runtime/shell/shell_body.rs:986; Bun targets only LE hosts perbun_core/lib.rs:1760.- The InternalBlob →
.text()path already strips a leading BOM (proven by the pre-existingBuffer.from("\uFEFF...")tests in the same file), so routing through it produces the spec-correct result. - Test coverage is thorough (BOM-only, double-BOM, interior BOM preserved, lone high/low surrogates, and a
text() === TextDecoder().decode(arrayBuffer())invariant over a dozen shapes across bothResponseandRequest). - The last CI status comment reports the ASAN lanes failed to build (infra, not the diff) and the fail-before gate ran a stale binary — so CI hasn't produced a clean full-matrix pass on the final commit. That's another reason for a human to gate the merge.
|
Closing as part of a cleanup of stale pull requests. This PR has had no new commits since 2026-07-28, it conflicts with main, and its last CI run failed. This is not a judgment on the fix itself. If the problem still reproduces on a current build, reopen this PR after a rebase or open a new one against main. |
What
new Response(str).text()/.json()now matchnew TextDecoder().decode(await new Response(str).arrayBuffer())for every string body: a leading U+FEFF BOM is stripped and lone surrogates become U+FFFD, matching Node, browsers, and Bun's own byte-backed body paths.Repro
Node v26 and browsers return
{a: 1}/"a\uFFFDb"on every row.Cause
A string
BodyInitis its UTF-8 encoding (fetch "extract a body", USVString), and the Body mixintext()/json()are "UTF-8 decode" of those bytes. UTF-8 decode strips a leadingEF BB BFand cannot produce a lone surrogate.Bun's string-body fast path stores the raw
WTFStringImpl(body::Value::from_js,src/runtime/webcore/Body.rs) and.text()hands it straight back (Any::to_string,Any::WTFStringImplarm) with no encode/decode round trip, so a leading BOM and any lone surrogate survived..arrayBuffer()/.bytes()of the same body transcode viato_utf8_allocand do normalize, so the two disagreed.Fix
In
body::Value::from_js, when the incoming JS string is 16-bit and either starts with0xFEFFor failssimdutf::validate::utf16le(i.e. contains a lone surrogate), encode it to UTF-8 and store asInternalBlobso every consumer agrees. 8-bit Latin-1 strings cannot hold U+FEFF or surrogates, and well-formed 16-bit strings pass the SIMD validator, so both keep the zero-copyWTFStringImplpath.Verification
test/js/web/fetch/utf8-bom.test.tsgains adescribe.eachoverResponse/Requeststring bodies covering BOM-only, double-BOM, interior BOM preserved, emoji, lone high/low surrogates, and atext() === new TextDecoder().decode(arrayBuffer())invariant over a dozen string shapes.Also ran
body.test.ts,response.test.ts,body-mixin-errors.test.ts,body-clone.test.ts(all green) andBUN_JSC_validateExceptionChecks=1.