Skip to content

fetch: normalize string bodies so text() matches the UTF-8 decode of arrayBuffer() - #33656

Closed
robobun wants to merge 7 commits into
mainfrom
claude/farm/1f3e685b/body-string-bom-strip
Closed

robobun wants to merge 7 commits into
mainfrom
claude/farm/1f3e685b/body-string-bom-strip

Conversation

@robobun

@robobun robobun commented Jul 7, 2026 •

Copy link
Copy Markdown
Collaborator

What

new Response(str).text() / .json() now match new TextDecoder().decode(await new Response(str).arrayBuffer()) for every string body: a leading U+FEFF BOM is stripped and lone surrogates become U+FFFD, matching Node, browsers, and Bun's own byte-backed body paths.

Repro

const doc = "\ufeff" + '{"a":1}';
await new Response(doc).json();                            // before: SyntaxError: Failed to parse JSON
await new Response(new TextEncoder().encode(doc)).json();  // always: {a: 1}

await new Response("a\uD800b").text();                     // before: "a\uD800b" (lone surrogate kept)
new TextDecoder().decode(await new Response("a\uD800b").arrayBuffer());  // always: "a\uFFFDb"

Node v26 and browsers return {a: 1} / "a\uFFFDb" on every row.

Cause

A string BodyInit is its UTF-8 encoding (fetch "extract a body", USVString), and the Body mixin text() / json() are "UTF-8 decode" of those bytes. UTF-8 decode strips a leading EF BB BF and cannot produce a lone surrogate.

Bun's string-body fast path stores the raw WTFStringImpl (body::Value::from_js, src/runtime/webcore/Body.rs) and .text() hands it straight back (Any::to_string, Any::WTFStringImpl arm) with no encode/decode round trip, so a leading BOM and any lone surrogate survived. .arrayBuffer()/.bytes() of the same body transcode via to_utf8_alloc and do normalize, so the two disagreed.

Fix

In body::Value::from_js, when the incoming JS string is 16-bit and either starts with 0xFEFF or fails simdutf::validate::utf16le (i.e. contains a lone surrogate), encode it to UTF-8 and store as InternalBlob so every consumer agrees. 8-bit Latin-1 strings cannot hold U+FEFF or surrogates, and well-formed 16-bit strings pass the SIMD validator, so both keep the zero-copy WTFStringImpl path.

Verification

test/js/web/fetch/utf8-bom.test.ts gains a describe.each over Response / Request string bodies covering BOM-only, double-BOM, interior BOM preserved, emoji, lone high/low surrogates, and a text() === new TextDecoder().decode(arrayBuffer()) invariant over a dozen string shapes.

bun bd test test/js/web/fetch/utf8-bom.test.ts   # 63 pass / 0 fail
USE_SYSTEM_BUN=1 bun test ...                    # 33 pass / 30 fail

Also ran body.test.ts, response.test.ts, body-mixin-errors.test.ts, body-clone.test.ts (all green) and BUN_JSC_validateExceptionChecks=1.

The fetch Body mixin defines text() as 'UTF-8 decode' and json() as
'parse JSON from bytes', both of which (per the Encoding Standard)
remove a leading U+FEFF byte-order mark.

When a Request/Response is constructed with a JS string body, Bun stores
the WTFStringImpl directly and hands it back to .text()/.json() without
the UTF-8 encode/decode round trip, so a leading BOM was preserved.
This meant new Response('\ufeff{...}').json() rejected with a syntax
error while the same payload via a Blob or Uint8Array body parsed fine,
and diverged from Node and browsers.

Strip a leading U+FEFF in the WTFStringImpl arms of Any::to_string and
Any::to_json, matching the sibling Blob/InternalBlob arms which already
strip the EF BB BF prefix from byte-backed bodies.
@github-actions github-actions Bot added the claude label Jul 7, 2026
@robobun

robobun commented Jul 7, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 3:05 PM PT - Jul 28th, 2026

❌ @robobun, your commit b367d5e has some failures in Build #84140 (All Failures)


🧪   To try this PR locally:

bunx bun-pr 33656

That installs a local version of the PR into your bun-33656 executable, so you can run:

bun-33656 --bun

@coderabbitai

coderabbitai Bot commented Jul 7, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

@robobun, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 3 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 10b0cf0b-b86b-45fe-9ca7-eceed62b5ad9

📥 Commits

Reviewing files that changed from the base of the PR and between 1537724 and b367d5e.

📒 Files selected for processing (2)
  • src/runtime/webcore/Body.rs
  • test/js/web/fetch/utf8-bom.test.ts

Walkthrough

Modifies Blob.rs's Any::to_json and Any::to_string to detect and skip a leading UTF-8 BOM character in WTFStringImpl strings before JSON parsing or string conversion. Adds tests validating this behavior for Response and Request objects constructed with string bodies.

Changes

UTF-8 BOM Stripping

Layer / File(s) Summary
BOM detection in Blob conversion paths
src/runtime/webcore/Blob.rs
Any::to_json and Any::to_string now check for a leading UTF-8 BOM (0xFEFF) in WTFStringImpl strings and, when present, operate on str.substring(1) instead of the full string.
Response/Request BOM tests
test/js/web/fetch/utf8-bom.test.ts
New test groups verify that text() and json() on Response and Request with string bodies strip a leading BOM, handle BOM-only input, strip only one leading BOM, and preserve interior BOM characters.

Possibly related PRs

  • oven-sh/bun#30139: Fixes BOM splitting/freeing logic in Blob.toJSONWithBytes, part of the same BOM→JSON flow in Blob.rs.
  • oven-sh/bun#30991: Also modifies Any::to_json/Any::to_string WTFStringImpl handling in Blob.rs, addressing WTF string ref release behavior.
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the main change: normalizing string bodies for text() and JSON behavior.
Description check ✅ Passed It includes the purpose and verification details, though the headings differ from the template.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
test/js/web/fetch/utf8-bom.test.ts (1)

1-114: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Move these tests into the existing fetch.test.ts instead of a new file.

This creates a new dedicated test file for a fetch/Response/Request behavior change, but the project convention requires new tests to be added to the existing test file for the module being changed unless it qualifies for the reserved regression path (which this doesn't, since BOM-stripping wasn't previously working and then broke).

As per coding guidelines: "Default: add your test to the existing test file for the code you're changing. Do not create a new file. A fetch bug goes in test/js/web/fetch/fetch.test.ts, a Bun.serve bug goes in test/js/bun/http/serve.test.ts, and so on."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/js/web/fetch/utf8-bom.test.ts` around lines 1 - 114, The BOM-stripping
coverage was added in a new dedicated test file, but this behavior belongs with
the existing fetch module tests. Move the UTF-8 BOM cases into the current fetch
test suite in fetch.test.ts, keeping the same coverage for Blob, Response, and
Request methods like text(), json(), and formData(), and remove the standalone
file so the module’s tests stay consolidated.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/js/web/fetch/utf8-bom.test.ts`:
- Around line 95-113: The Request BOM test coverage is incomplete compared with
Response, so extend the existing Request (string body) cases in the utf8-bom
test to include the same edge scenarios: BOM-only bodies, double-BOM stripping
only one leading BOM, interior BOM preservation, and emoji handling for both
text() and json(). Reuse the Request constructor and its text()/json()
assertions so this mirrors the BodyMixin/Any::to_string/Any::to_json behavior
already exercised by Response.

---

Outside diff comments:
In `@test/js/web/fetch/utf8-bom.test.ts`:
- Around line 1-114: The BOM-stripping coverage was added in a new dedicated
test file, but this behavior belongs with the existing fetch module tests. Move
the UTF-8 BOM cases into the current fetch test suite in fetch.test.ts, keeping
the same coverage for Blob, Response, and Request methods like text(), json(),
and formData(), and remove the standalone file so the module’s tests stay
consolidated.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 9f26afd2-8811-46f9-86a4-bcdc41f5d6b2

📥 Commits

Reviewing files that changed from the base of the PR and between 3f5d816 and 1537724.

📒 Files selected for processing (2)
  • src/runtime/webcore/Blob.rs
  • test/js/web/fetch/utf8-bom.test.ts

Comment thread test/js/web/fetch/utf8-bom.test.ts Resolved
@robobun

robobun commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed review feedback in 627fc19:

  • Request coverage: the string-body block is now a describe.each over Response and Request, so every edge case (BOM-only, double-BOM, interior BOM preserved, emoji in text()/json()) runs against both. 35 pass on the debug build, 12 of the 14 new cases fail on the released bun (the two interior-BOM cases were always correct).
  • "Move into fetch.test.ts": skipping this one. test/js/web/fetch/utf8-bom.test.ts is an existing file (added in Fixes #4876 #7391, extended in Fixes #7503 #7511) that already groups the BOM coverage for Blob/Response/Request/ReadableStream. This PR appends to it rather than creating a new file, so the guideline is already satisfied.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — brings the string-body text()/json() path in line with the spec and the sibling byte-body arms.

Extended reasoning...

Overview

This PR adds a leading U+FEFF BOM strip to the Any::WTFStringImpl arms of Any::to_json and Any::to_string in src/runtime/webcore/Blob.rs (10 lines), and appends a describe.each([Response, Request]) block to the existing test/js/web/fetch/utf8-bom.test.ts covering ASCII/emoji payloads, BOM-only, double-BOM (only one stripped), and interior BOM (preserved).

Security risks

None. This is pure output normalization on the fetch Body mixin — no auth, crypto, permissions, or untrusted-input parsing surface is touched. The added char_at(0) is guarded by a length check in both arms (length() == 0 early-return in to_json, length() > 0 && short-circuit in to_string), so there's no OOB read.

Level of scrutiny

Low-to-medium. The change is small, mechanical, and spec-driven: the fetch Body mixin defines text() as "UTF-8 decode" which removes a leading EF BB BF, and the sibling Any::Blob/Any::InternalBlob arms already do this via strings::BOM::detect_and_split. char_at returns u16 so the == 0xFEFF comparison is correct for 16-bit storage and a no-op for 8-bit Latin-1 (which can't hold U+FEFF anyway). substring(1) returns a borrowed ZigString view over the OwnedString's storage, which stays alive through the to_js/to_js_by_parse_json call before Drop releases the WTF ref — the ownership comments already in the surrounding code cover this. The no-BOM common path is byte-for-byte unchanged.

Other factors

  • The PR description shows 35 pass on the debug build and 12/14 new cases failing on system Bun (the two interior-BOM cases were always correct), satisfying the fails-for-the-right-reason requirement.
  • Author ran body.test.ts, response.test.ts, body-mixin-errors.test.ts (373 pass, including string-body leak tests) and BUN_JSC_validateExceptionChecks=1.
  • CodeRabbit's only feedback (parametrize over Request) was addressed in 627fc19 and the thread is resolved.
  • No prior claude[bot] reviews on this PR; no CODEOWNERS entries for the touched paths.
  • The bug hunting system found no issues.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — small spec-aligned fix bringing the string-body arm to parity with the byte-body arms, with thorough edge-case coverage.

Extended reasoning...

Overview

Two match arms in src/runtime/webcore/Blob.rs (Any::to_json and Any::to_string, the WTFStringImpl variant) now check for a leading U+FEFF code unit and, if present, operate on str.substring(1) instead. Tests in test/js/web/fetch/utf8-bom.test.ts add a describe.each block over Response/Request string bodies covering BOM-only, double-BOM (strip one), interior BOM (preserved), and emoji payloads for both text() and json().

Security risks

None. This is a pure string-prefix check on data the caller supplied to their own Request/Response constructor; no auth, crypto, filesystem, or network parsing is involved.

Level of scrutiny

Low-to-moderate. The change is ~10 lines of Rust and mirrors what the sibling Any::Blob / Any::InternalBlob arms already do via strings::BOM::detect_and_split / without_utf8_bom. It aligns Bun with the fetch spec ("UTF-8 decode" strips a leading BOM) and with Node/browser behavior. I verified:

  • char_at(0) is guarded by length() > 0 in both call sites (satisfying its debug_assert).
  • char_at returns u16; on 8-bit Latin1 storage the max is 0xFF, so the == 0xFEFF check can only fire on 16-bit storage — which is exactly where a JS-string BOM would land.
  • substring(1) returns a ZigString-backed borrowed view into the WTFStringImpl's UTF-16 buffer; the OwnedString str holds the +1 WTF ref and drops it at scope exit, so the storage is valid throughout to_js() / to_js_by_parse_json().

Other factors

The bug-hunting system found nothing. CodeRabbit's only feedback (parametrize Request coverage to match Response) was addressed in 627fc19. The PR description reports the new tests fail on system Bun and pass on the debug build, plus clean runs of the neighboring body/response test suites and BUN_JSC_validateExceptionChecks=1. No CODEOWNERS entry covers this path.

@robobun

robobun commented Jul 7, 2026 •

Copy link
Copy Markdown
Collaborator Author

CI status: the diff is green. test/js/web/fetch/utf8-bom.test.ts passes on every lane that ran it.

Remaining red is unrelated infrastructure:

  • build 69876: darwin-aarch64 buildkite artifact download timed out after 120s (never reached tests).
  • build 69882: alpine x64 / x64-baseline test-docker-build-debian.ts failed with gpg: keyserver receive failed: Connection timed out; plus Windows flakes (prisma tarball fetch, EBUSY/EBADF/EPIPE) that all passed on retry.
  • build 84140: no tests ran at all. 33 jobs expired in queue, 252 waiting_failed on those, 0 annotations.

Ready for a maintainer to retry the red lanes or merge.

Move the fix from consumption time (Any::to_string/to_json) to
construction time (body::Value::from_js). A string BodyInit's bytes are
its UTF-8 encoding (fetch 'extract a body', USVString), and text()/json()
are 'UTF-8 decode' of those bytes, which strips a leading BOM and cannot
produce lone surrogates. The previous fast path stored the raw JS string
and handed it straight back from text(), so both a leading U+FEFF and any
lone surrogate survived, while arrayBuffer()/bytes() of the same body
would encode them away.

Now, when the incoming string is 16-bit and either starts with U+FEFF or
fails simdutf::validate::utf16le, encode it to UTF-8 and store as
InternalBlob so every consumer (text/json/arrayBuffer/bytes/blob) agrees.
8-bit Latin-1 strings cannot hold U+FEFF or surrogates, so they (and all
well-formed 16-bit strings) keep the zero-copy WTFStringImpl path.

Tests: add lone-surrogate cases and a parametrized
text() === new TextDecoder().decode(arrayBuffer()) invariant over a dozen
string shapes, run for both Request and Response.
Comment thread src/runtime/webcore/Body.rs Resolved
@robobun robobun changed the title fetch: strip leading UTF-8 BOM from string-body text()/json() fetch: normalize string bodies so text() matches the UTF-8 decode of arrayBuffer() Jul 28, 2026
Comment thread src/runtime/webcore/Body.rs Resolved
@robobun

robobun commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator Author

The diff is ready. Verified locally at b367d5e on a clean tree:

bun bd test test/js/web/fetch/utf8-bom.test.ts       # 63 pass / 0 fail (debug+ASAN)
./build/release/bun test ...                         # 63 pass / 0 fail (release)
USE_SYSTEM_BUN=1 bun test ...                        # 33 pass / 30 fail (fail-before)

The internal fail-before gate is bouncing on its own build environment: both ASAN lanes report BUILD FAILED (no junit output) with and without the src/ diff applied, and its "without fix" release lane is running a stale cached binary (reports revision 476ff7629, which already contains the fix). Re-ran once with no change. This needs a maintainer to merge on the strength of the local verification above.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't find any bugs, but this changes body::Value::from_js — the construction path every string Response/Request body flows through — and adds a simdutf::validate::utf16le scan to every 16-bit string body, so it's worth a human look at the perf/behavior tradeoff before merge.

Checked: utf16[0] is guarded by the length() == 0 early return above; bun_core::String is Copy with no Drop, so the explicit str.deref() correctly releases the +1 from to_bun_string without a double-deref; the utf16le validator on host-native &[u16] matches the existing pattern in shell_body.rs (Bun is LE-only); the 8-bit Latin-1 arm is correctly skipped since it can't hold U+FEFF or surrogates.

Extended reasoning...

Overview

Two files: a ~12-line addition to body::Value::from_js in src/runtime/webcore/Body.rs that detects a leading BOM or lone surrogates in 16-bit string bodies and eagerly transcodes them to UTF-8 bytes (InternalBlob) instead of storing the raw WTFStringImpl; and ~70 lines of new test cases in test/js/web/fetch/utf8-bom.test.ts covering Response/Request string bodies via describe.each.

Security risks

None identified. No user-controlled paths, no auth/crypto, no untrusted-size arithmetic. The only new indexing (utf16[0]) is guarded by the pre-existing str.length() == 0 → Value::Empty return.

Level of scrutiny

Medium-high. from_js runs for every new Response(str) / new Request(url, {body: str}), so this is a hot, production-critical path. The change is a behavior change (spec-aligning, matches Node/browsers) and adds a full-string SIMD validation pass to every 16-bit string body. That's likely fine — simdutf is fast and 16-bit strings are already the slow-ish path — but a maintainer should sign off on adding work to this constructor.

Other factors

  • Memory management verified: bun_core::String is #[derive(Copy)] with no Drop impl (per string/mod.rs:1251-1255), so the explicit str.deref() on the new early-return path is the sole release of the +1 ref from to_bun_string() — no double-deref on scope exit. The utf16 borrow ends at to_utf8_alloc(utf16) before the deref.
  • simdutf::validate::utf16le on host-endian &[u16] mirrors src/runtime/shell/shell_body.rs:986; Bun targets only LE hosts per bun_core/lib.rs:1760.
  • The InternalBlob → .text() path already strips a leading BOM (proven by the pre-existing Buffer.from("\uFEFF...") tests in the same file), so routing through it produces the spec-correct result.
  • Test coverage is thorough (BOM-only, double-BOM, interior BOM preserved, lone high/low surrogates, and a text() === TextDecoder().decode(arrayBuffer()) invariant over a dozen shapes across both Response and Request).
  • The last CI status comment reports the ASAN lanes failed to build (infra, not the diff) and the fail-before gate ran a stale binary — so CI hasn't produced a clean full-matrix pass on the final commit. That's another reason for a human to gate the merge.

@robobun

robobun commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator Author

Closing as part of a cleanup of stale pull requests. This PR has had no new commits since 2026-07-28, it conflicts with main, and its last CI run failed. This is not a judgment on the fix itself. If the problem still reproduces on a current build, reopen this PR after a rebase or open a new one against main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant