Skip to content

Honor Blob slice offsets on non-seekable sources - #33510

Closed
robobun wants to merge 3 commits into
mainfrom
farm/c731f65c/stdin-slice-offset
Closed

robobun wants to merge 3 commits into
mainfrom
farm/c731f65c/stdin-slice-offset

Conversation

@robobun

@robobun robobun commented Jul 6, 2026 •

Copy link
Copy Markdown
Collaborator

What

Bun.stdin.slice(start, end) ignores start when stdin is a pipe. end survives, but as a length measured from 0:

// child.js
console.log(JSON.stringify({
  "slice(2,7)": await Bun.stdin.slice(2, 7).text(),
  "slice(3)": await Bun.stdin.slice(3).text(),
}));
$ printf 'abcdefghij' | bun child.js
{"slice(2,7)":"abcde","slice(3)":"abcdefghij"}    # actual
{"slice(2,7)":"cdefg","slice(3)":"defghij"}       # Bun.file(path).slice() / new Blob([...]).slice()

Every read method is affected: .text(), .bytes(), .arrayBuffer(), .json(), .stream(), and new Response(blob). Consumers silently get the wrong byte range with no error. Redirecting a regular file into stdin (bun child.js < file) is correct, so the bug only shows up for pipes, sockets and ttys.

Why

A blob's start offset is applied by seeking. lseek fails with ESPIPE on anything that is not seekable, and the failure was discarded (src/runtime/webcore/blob/read_file.rs):

if self.offset > 0 {
    // We DO support offset in Bun.file()
    // we ignore errors because it should continue to work even if its a pipe
    let _ = bun_sys::set_file_offset(fd, self.offset);
}

So the read started at byte 0 while max_length (end - start) still capped it, which is exactly the observed [0, end - start) window. The streaming path has the same hole one layer down: BufferedReader::start_file_offset only arms pread, and pread is never used for a pipe, socket or tty.

How

When the source cannot seek, the offset is read out of the stream and discarded before any byte is kept. Seekable sources keep using lseek/pread, so concurrent slices of the same file are untouched.

  • ReadFile (POSIX): when set_file_offset fails, skip_remaining is armed and the read loop drains it into its stack buffer. The existing poll/EAGAIN re-arm already covers an offset that straddles several reads.
  • ReadFileUV (Windows): uv_fs_read only honors an offset on seekable handles, so for a non-regular file the skipped head is dropped from the buffer after each read and read_off advances only by the kept bytes.
  • BufferedReader (both platforms) for .stream(): start_file_offset arms _skip_remaining when the source is not a file. POSIX drains it before the normal read loop runs; Windows shifts the kept tail down inside the uncommitted spare capacity, so the skipped bytes are never committed and FileReader never sees them.

Readiness is the subtle part of the POSIX drain. sys::read_nonblocking is a plain blocking read on macOS (and on Linux once RWF_NOWAIT is disabled), and FileType::Pipe fds are blocking, so the drain polls before every read — the same hazard read_blocking_pipe documents a few lines above. It then re-confirms readiness once more before handing control to the normal read loop, because read_blocking_pipe is only entered after its caller has proven the fd readable, and the drain has just spent that proof. Without it, a writer that emits exactly the offset and then pauses leaves the next read blocking on an empty pipe (thanks @claude for catching this one).

One existing assertion in test/js/bun/util/bun-stdin-slice.test.ts encoded the bug: Bun.stdin.slice(1) on "hello world" expected "hello world", and now expects "ello world".

Verification

$ printf 'abcdefghij' | ./build/debug/bun-debug child.js
{"slice(2,7)":"cdefg","slice(3)":"defghij"}

bun bd test test/js/bun/util/bun-stdin-slice.test.ts — 12 pass; 10 of the 12 fail on main. They cover a piped stdin through .text(), .bytes(), .stream() and Response, unbounded slice(start), an offset spanning many reads, an offset past EOF, a writer that trickles bytes so the offset straddles poll re-arms, and a writer that closes with the offset exactly consumed. The regular-file case guards against the offset being consumed twice.

The two boundary tests cannot distinguish a blocking read on Linux, where RWF_NOWAIT turns it into EAGAIN — they hang rather than fail on a macOS regression. Proving that one deterministically needs instrumentation in src/.

Also checked by hand
  • A named FIFO (FileType::Pipe, the blocking-read path) for both .text() and .stream().
  • A FIFO writer that sleeps mid-offset: the result is correct and a setInterval in the reader keeps ticking, i.e. the event loop is not blocked.
  • A writer that pauses, and one that closes, with the offset exactly consumed.
  • EOF reached before the offset, on both paths: empty result, clean exit, no hang.
  • spawn, streams, bun-write, blob, bun-serve-file, filesink: 459 pass, 0 new failures. (streams-leak.test.ts fails on an ASAN RSS ceiling both with and without this diff — 755 MB on main, 700 MB here.)
  • bun run rust:check-all clean across all 10 targets, which is what type-checks the Windows branches.

Touches src/io/PipeReader.rs alongside #31680, which fixes an unrelated hang in the same file; the two changes are in different functions.

@github-actions github-actions Bot added the claude label Jul 6, 2026
@coderabbitai

coderabbitai Bot commented Jul 6, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@robobun, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 22 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: d0d9e5ee-a558-4c00-95bb-73c576d69180

📥 Commits

Reviewing files that changed from the base of the PR and between 48ff9eb and 8380e2c.

📒 Files selected for processing (3)
  • src/io/PipeReader.rs
  • src/runtime/webcore/blob/read_file.rs
  • test/js/bun/util/bun-stdin-slice.test.ts

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Jul 6, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 6:08 PM PT - Jul 6th, 2026

❌ @robobun, your commit 8380e2c has some failures in Build #69270 (All Failures)


🧪   To try this PR locally:

bunx bun-pr 33510

That installs a local version of the PR into your bun-33510 executable, so you can run:

bun-33510 --bun

@robobun

robobun commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 6, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Comment thread src/io/PipeReader.rs Resolved

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — c2d053e addresses the readiness re-check I flagged in read_blocking_pipe, and the new trickle/close-at-offset tests cover it. I'm deferring to a human because this threads new state through four separate read loops (POSIX/Windows × buffered/one-shot) with event-loop-blocking implications, and the Windows paths are only type-checked.

Extended reasoning...

Overview

Fixes Bun.stdin.slice(start, end) ignoring start on non-seekable sources (pipes, sockets, ttys). Adds a skip_remaining counter to four read paths — PosixBufferedReader, WindowsBufferedReader, ReadFile (POSIX), ReadFileUV (Windows) — that drains and discards the leading bytes when lseek/pread cannot apply the offset. Adds a drain_skipped helper for the POSIX buffered reader with per-iteration readiness polling, and post-read head-dropping for the Windows paths. ~200 lines of native code changed plus 10 tests (8 new, 1 assertion corrected).

Since my last review, commit c2d053e added the is_readable re-check after drain_skipped returns true in read_blocking_pipe (exactly the fix suggested), plus two new tests exercising the offset-boundary and close-at-offset cases.

Security risks

None identified. No untrusted-input parsing, no auth/crypto/permissions surface. The change reads and discards bytes from an fd the caller already owns.

Level of scrutiny

High. PipeReader.rs is core I/O infrastructure shared by subprocess pipes, shell, stdin, and file streaming. The new drain_skipped path interacts with blocking-vs-nonblocking reads, poll re-arming, and event-loop liveness — the exact area where my previous review found a bug (now fixed). The Windows on_read change reslices the libuv buffer inside uncommitted spare capacity and early-returns before the debug-assertion pointer-range check and commit_spare; that looks correct but is subtle. Both Windows paths (WindowsBufferedReader::on_read, ReadFileUV::on_read) were verified only by rust:check-all, not by running tests.

Other factors

  • Test coverage is thorough for POSIX: covers .text(), .bytes(), .stream(), Response, unbounded slices, multi-read offsets, offset-past-EOF, trickle writer, close-at-offset, and a regular-file guard against double-consumption.
  • The ReadFileUV change to remaining_buffer() uses saturating_add(skip_remaining) on a value derived from max_length - read_off; when max_length == MAX_SIZE and skip_remaining > 0 this saturates rather than overflowing, which is the intent.
  • The corrected assertion in the existing overflow test ("hello world" → "ello world") is a genuine behavior change — the old test encoded the bug.
  • No new bugs found by the current bug-hunting pass.

@robobun

robobun commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the re-review. Two notes on the reservations, for whoever picks this up.

The Windows paths are executed by this build, not only type-checked. The two pre-existing tests in bun-stdin-slice.test.ts are skipIf(isWindows); the ten added ones are not. So windows-2019-x64-test-bun, windows-2019-x64-baseline-test-bun and windows-11-aarch64-test-bun run the piped-stdin slice cases through WindowsBufferedReader::on_read and ReadFileUV::on_read for real, which is the coverage I could not produce locally. All three Windows build lanes are green; the test lanes are still running, and if one of them goes red that is where to look.

One case no lane can turn red on. The readiness re-check only changes observable behavior where sys::read_nonblocking is a blocking read(2), and on Linux it is not, since RWF_NOWAIT gives EAGAIN instead. The trickle and close-at-offset tests run the path everywhere, but a regression there can only surface on macOS, and as a hang rather than a failed assertion. Pinning it deterministically would mean instrumenting src/, so it rests on the analysis plus the darwin-*-test-bun lanes.

And yes, the saturating_add in remaining_buffer is load-bearing rather than incidental: max_length is MAX_SIZE for an unbounded slice(start), so the add has to clamp rather than wrap.

robobun added 2 commits July 6, 2026 18:39
Bun.stdin.slice(2, 7).text() returned bytes [0, 5) when stdin was a pipe:
the blob's start offset was applied with lseek, which fails with ESPIPE on a
pipe, socket or tty, and the error was swallowed. The end bound survived as a
read length, so start was silently dropped.

The readers now consume and discard the offset from the stream when the
underlying fd cannot seek:

- ReadFile (POSIX) skips into a scratch buffer when lseek fails.
- ReadFileUV (Windows) drops the skipped head after each libuv read, since
  uv_fs_read only honors an offset on seekable handles.
- BufferedReader (both platforms) does the same for the streaming path, where
  start_file_offset() previously only armed pread, which pipes ignore.

Seekable sources keep using lseek/pread, so concurrent slices of the same file
are unaffected.
read_blocking_pipe is only entered once its caller has confirmed the fd is
readable, which is what makes its first read safe to block in. Draining the
start offset spends that readiness, so the read that follows could block the
event loop on an empty pipe: sys::read_nonblocking is a plain read(2) on macOS
and on Linux once RWF_NOWAIT is disabled, and FileType::Pipe fds are blocking.

A writer that emits exactly the offset and then pauses, without closing, was
enough to trigger it.
@robobun
robobun force-pushed the farm/c731f65c/stdin-slice-offset branch from c2d053e to 461e11e Compare July 6, 2026 18:44
@robobun

robobun commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

CI status

The diff is green on every lane that ran it. The remaining red on build #69270 is infrastructure, not this change.

The one failure is an artifact-download timeout on a macOS shard — the test-bun binary never downloaded, so no test executed there:

Error: buildkite-agent artifact download timed out after 120s for step 'darwin-aarch64-build-bun'.
Refusing to continue with a partial download (would silently fall back to the wrong binary).
The command exited with status 1

A re-run of that single job should clear it.

What did run, passed — including the paths that earlier review noted were only type-checked:

  • windows 11 aarch64, windows 2019 x64, windows 2019 x64-baseline — all green. These exercise the piped-stdin slice cases through WindowsBufferedReader::on_read and ReadFileUV::on_read for real (10 of the 12 tests are not skipIf(isWindows)).
  • debian 13 x64-asan — green.
  • all alpine / ubuntu / debian / linux lanes — green.

The two prior reds on this PR were also unrelated infra: a stale-base cookie-map assertion (fixed by rebasing onto #33425) and network timeouts in bun-install-proxy / the debian docker build. None of the three touched pipe or blob reading.

I've used my automated re-roll for this round, so I'm leaving the macOS job for a maintainer to re-run rather than pushing another no-op commit.

Jarred-Sumner pushed a commit that referenced this pull request Aug 16, 2026
…after it (#39296)

Follow-up to #39201, where moving the slice window into the reader was
agreed as a separate PR.

### Problem
- Streaming a slice of a pipe, socket or stdin consumes bytes past the
slice. With `0123456789` written to a child's stdin, `for await (const c
of Bun.stdin.slice(0, 3).stream())` delivers `012` and whoever reads
stdin next gets nothing: the other 7 bytes were read and dropped. Same
through a native sink (`HTMLRewriter.transform(new
Response(Bun.stdin.slice(0, 3)))`). With 60000 bytes written, the stream
leaves 0 of them in the pipe. Same on Windows.
- `Bun.stdin.slice(0, 0).stream()` (any empty slice of a pollable
source) never ends while the pipe stays open and silent; it ends only
when a byte arrives, which it drops, or at EOF. Same through a native
sink.
- `Bun.serve` responding with `Bun.file(p).slice(0, 2048)` of a 4 KiB
file issues two `read()`s per response (a 256 KiB request that returns
the whole file, then an EOF probe) and ends the response from a queued
task. Harmless for regular files, but it is the same after-the-fact
window as above, implemented a second time.
- Cause: the window is applied by the reader's parents after the bytes
have been read. `FileReader::on_read_chunk` cuts the delivered chunk
down (`src/runtime/webcore/FileReader.rs:636`); the only place that cut
the read itself was `on_pull`'s `read_into` branch
(`FileReader.rs:849`), which is skipped whenever a poll is registered,
i.e. for every pollable fd, and `read_into` is a stub on Windows. The
read loop in `src/io/PipeReader.rs` reads up to its 256 KiB scratch
buffer (`fill_scratch`, line 653) and the Windows reader hands libuv its
whole spare capacity (`get_read_buffer_with_stable_memory_address`, line
1166); neither knows about the window. The empty-window hang is the same
gap: the check lived on the `read_into` branch, and a pollable source
parks on its poll before reaching it. `FileResponseStream` keeps a third
copy of the window (`src/runtime/server/FileResponseStream.rs:277`) and
needs the `FileResponseStreamEof` task to end itself.

### Fix
- `PosixBufferedReader` and `WindowsBufferedReader` get a `ReadLimit`
next to the pread offset, set with `set_limit` before starting.
`read_once` (every POSIX read: loop, buffer and `read_into`) and the
libuv buffer on Windows cut each read to what is left of it, and
charging it down to zero is reported the way EOF already is (`Stop::Eof`
/ `ReadState::Eof`, then `close` and `on_reader_done`). A limit that
starts at zero makes `start` register no poll and issue no read; the
parent's first read request (`read`, `read_into`, Windows `unpause`)
reports EOF without touching the fd.
- `FileReader` passes the blob's size to the reader at start and
`FileResponseStream` passes the body length; both now treat the end of
the window exactly like EOF, because that is what the reader reports.
Deleted: `FileReader`'s `total_readed` / `window_remaining` /
`consume_window` / `end_at_window`, `FileResponseStream`'s `max_size`
and the `FileResponseStreamEof` task (tag, dispatch arm, `Taskable`
impl), and `Chunk::truncate`, which only existed for this.
- `FileResponseStream` implements the reader's `ref_` / `deref` (as
`FileReader` does). The chunk that fills the window is now followed by
`on_reader_done`, which releases the stream's owning ref, while the
reader's read loop (POSIX) or libuv completion (Windows, which re-checks
the reader's flags after delivering a data read) is still on the stack
with the reader embedded in the stream; the pin keeps the stream alive
until those frames return. This is the job the queued task used to do.
- Why this is the right place: the window is a bound on how many bytes
may be taken out of the source, and on anything that is not seekable the
bytes a read returns are gone, so the bound has to be applied to the
read, and only the reader knows whether a read is needed at all.
Reporting it as EOF is correct because for this reader the source does
end there, and every consumer already handles EOF (real EOF and
`maxBuffer` exhaustion end streams through the same path), which is what
lets the two parent implementations (three code paths, two of which
#39201 had to bring back into agreement) go away. For regular files the
delivered bytes are unchanged; only the reads get shorter.
- Verified with `bun bd test test/js/bun/util/bun-stdin-slice.test.ts
test/js/bun/http/serve-file-slice-read-error.test.ts`. The five new
`bun-stdin-slice` cases run the slice in a grandchild sharing the
child's stdin and then read what is left: a write larger than the window
and an empty window with nothing written, each via `stream()` and via
HTMLRewriter, plus a window filled by a write larger than its remainder.
With `src/` stashed, the three over-read cases get `""` instead of
`3456789` / `456` back and the two empty-window cases time out; with the
fix all pass on Linux and on a Windows x64 debug build (the block is not
Windows-skipped; the same protocol reproduces the over-read with the
current Windows canary). `serve-file-slice-read-error.test.ts` kept its
ptrace read counter: 3 responses now take 3 `read()`s and the EIO
injected into the second one fails that response alone (`DONE 2`);
without the fix it is 6 reads and `DONE 3`.
- Also run with the fix: `blob.test.ts` (the #39201 window cases:
`read_into`, sink, Windows `on_read_chunk`, empty window on a regular
file), `bun-serve-file`, `bun-serve-static`, `serve-directory-routes`
(empty files give `FileResponseStream` a zero length), `bun-file*`,
`spawn.test.ts`, `spawn-maxbuf` (the budget shares `read_once` /
`on_read`), `spawn-streaming-stdout`, `spawn-stream-serve`,
`spawn-stdout-filereader-gc-uaf`, `streams.test.js`, `html-rewriter*`,
`body*.test.ts`, `process-stdin`, `bun-write`; on Windows additionally
`blob`, `bun-serve-file`, `bun-serve-static`, `spawn-maxbuf`,
`bun-file*`. `cargo fmt` clean.

### Background
- `BufferedReader` (`src/io/PipeReader.rs`, one POSIX and one libuv
implementation behind a cfg alias) reads an fd on behalf of a parent
embedded around it and reports through a vtable: `on_read_chunk` per
read, then `on_reader_done` at EOF or `on_reader_error`. `FileReader` is
the parent behind a file-backed `ReadableStream` (`Bun.file().stream()`,
`new Response(file).body`, the stream a native sink such as HTMLRewriter
or a fetch body pulls from); `FileResponseStream` is the parent that
writes a `Bun.serve` file body to the socket.
- A file blob carries an offset and a size; for a slice these describe
the window. `ReadableStream::from_blob_copy_ref` copies them into
`FileReader` as `start_offset` / `max_size`; `Bun.serve` computes the
body's offset and length from the blob (or a Range header) and passes
them to `FileResponseStream`. The offset was already the reader's job
(`start_file_offset`); this PR gives it the length as well.
- On POSIX a pollable fd (pipe, socket, tty) is read when its poll
fires, from the reader's own loop; a regular file is read synchronously
when the parent asks (`read`, or `read_into` straight into the JS pull
buffer). `FileReader::on_pull` only uses `read_into` when no poll is
registered, which is why a window applied there never affected pollable
sources. On Windows every read completes through libuv into a buffer the
reader allocates up front, so that allocation is where a read's size is
decided.
- `ref_` / `deref` on the parent vtable are taken by the reader around a
read dispatch (`ParentKeepAlive`) because the reader touches its own
fields after calling into the parent; a parent whose `on_reader_done`
can free it needs them so that free happens after the dispatch returns.
`maxBuffer` (`MaxBuf`) is the reader's other byte bound; the limit is
charged and reported alongside it.

<details>
<summary>Related open PRs</summary>

- #39257 (alii) fixes the empty-window hang inside `FileReader` with the
helpers this PR deletes; if it lands first this PR will be rebased to
drop those two guards (the reader covers them) and keep its tests, which
pass here as well. The over-read is not covered there.
- #31680 and #33510 are about the buffered (`.text()` /
`.arrayBuffer()`) path in `blob/read_file.rs`, which does not use
`BufferedReader`; #31680's `FileReader.rs` hunk was made unnecessary by
#39201.
- #34185 wanted `FileResponseStream`'s remaining count to tell a short
file from a complete one; with this PR that is the bytes it has written
versus the length it was given.
</details>
@robobun

robobun commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator Author

Closing as part of a cleanup of stale pull requests. This PR has had no new commits since 2026-07-06, it conflicts with main, and its last CI run failed. This is not a judgment on the fix itself. If the problem still reproduces on a current build, reopen this PR after a rebase or open a new one against main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant