Repository navigation
Conversation
|
Status: draft, stacked on #37082. This revision (e1bb808) tests a regular file whose How to reproduce on main or bun 1.4.2, Linux: await using server = Bun.serve({ port: 0, fetch: () => new Response(Bun.file("/proc/version")) });
const res = await fetch(server.url);
console.log(res.headers.get("content-length"), (await res.text()).length);
// main: "0" 0
// this PR: "219" 219The same happens for a static route. #44064 tracks the parts that this PR does not change. |
5561b9f to
86b62a0
Compare
|
Updated 5:18 PM PT - Sep 29th, 2026
✅ @robobun, your commit e1bb80848d035899fdee2fe4be79c60496219382 passed in 🧪 To try this PR locally: bunx bun-pr 43974That installs a local version of the PR into your bun-43974 --bun |
86b62a0 to
64489d0
Compare
64489d0 to
1924ab8
Compare
|
@alii this PR needs one decision from you, and it is the decision that is open on #41593 too. Edited on 2026-09-29: the branch now tests a file for content before it answers, and it covers HEAD. The table and the questions describe this revision (e1bb808). A regular file on procfs or cgroupfs reports
Shape A is on this branch. Shape B is on Questions:
|
1924ab8 to
7cb6d8c
Compare
A regular file on procfs or cgroupfs reports st_size 0 and has content. do_sendfile and FileRoute::serve framed the response from st_size, so they sent Content-Length: 0 and no body. For a regular file whose st_size is 0, one pread of one byte now tells a file with content from an empty file. A file with content takes the arm of a pipe: FileResponseStream reads it to EOF, and the response has no Content-Length, Range or validator from the stat. A response to HEAD has none of them either. A slice passes its window as the limit of the reader (BodyLength::UpTo). An empty file, a descriptor of the caller, a directory route, and a static route with a Content-Length header of its own keep the answers of the stat. StartOptions.length is now a BodyLength. Exact is a Content-Length that the producer wrote. ToEof and UpTo are not.
7cb6d8c to
e1bb808
Compare
|
Status on 2026-10-23. The bug is still on main. On the canary build (1.4.4, 5c6d4e2), Two things changed since the last push:
The questions at the top of the description are still open. The rebase waits for that answer and for #46528, because a rebase onto main today conflicts again when #46528 merges. |
Part of #44064
Stacked on #37082. Questions for alii or Jarred-Sumner:
preadof one byte the right test, orfstatfs?Problem
Bun.servesendsContent-Length: 0and no body fornew Response(Bun.file("/proc/version"))(219 bytes).do_sendfile(RequestContext.rs:1857) andFileRoute::serve(FileRoute.rs:335) frame fromst_size. procfs and cgroupfs report 0.Fix
preadof one byte finds content in a regular file withst_size0.FileResponseStreamreads it to EOF. GET and HEAD get noContent-Length, Range or validator from thestat.stat.bun-serve-file.test.ts,serve-file-slice-read-error.test.ts: 29 new tests, 22 fail on the base. Self-reviewed: 27 concerns raised on the revision before, 24 addressed.Background
FileResponseStreamowns each file body. A body with no length ends at EOF: aContent-Lengthwhen the first read ends it, else chunked (last chunk: Bun.serve: terminate FIFO/pipe file responses at EOF #37082).Downsides
BunFileafter.sizeorexists()(Bun.file: do not use a cached stat size as the byte budget of a read or a copy #43910),Bun.file(fd),{ dir }routes.pread(HEAD in a handler:open,pread,close). A sized file pays 3 to 20 instructions, no syscall..textgrows by 3,072 bytes./now sends/proc/self/environ. The read holds the JS thread: 350 ms for/proc/pagetypeinfo(100 ms for thepread).Notes
What the server sends
Raw HTTP/1.1,
Connection: close, Linux x64, release builds. Base: #37082 at 26376b9./proc/versionhas 219 bytes,/proc/cpuinfo177,934 and/proc/kallsyms12.5 MB here. "fetch handler" isfetch: () => new Response(Bun.file(path)). "static route" isroutes: { "/x": new Response(Bun.file(path)) }./proc/version, fetch handler and static routeContent-Length: 0Content-Length: 219Bun.file("/proc/version").slice(10, 60)Content-Length: 0Content-Length: 50/proc/cpuinfo(65 reads)Content-Length: 0/proc/kallsymsContent-Length: 0Range: bytes=0-9416,Content-Range: bytes */0200, whole bodyIf-Modified-Sincein the future, static route304200, whole bodyIf-None-Match: *, static route304,Last-Modified304/proc/version, fetch handler and static routeContent-Length: 0Content-LengthContent-Length: 0Last-Modified,304forIf-Modified-Since,416for a Range/proc/self/mem(preadfails with EIO)Content-Length: 0/proc/self/pagemap(preadof one byte fails with EINVAL)Content-Length: 0BunFileafter.sizeorawait exists()Bun.file(fd), two timesResponsehas aContent-LengthheaderContent-Length: 0Content-Length: 0The test for content
zero_size_file_has_contentruns only for a regular file whosest_sizeis 0. It reads one byte withpreadat the offset of theBlob. One byte back means content. End of file or an error means an empty file, and thestatanswers as before. So a file that cannot give one byte keeps the answer of the base:/proc/self/mem(EIO) and/proc/self/pagemap(EINVAL, its records have 8 bytes).The cost is in the kernel. A procfs file makes its content at
read, so thepreadmakes the first record and the reader makes it again. Median of 11 runs on the test machine:preadof one byte and thenreadto EOF, againstreadto EOF with nopread.preadreadto EOF after itreadto EOF alone/sys/fs/cgroup/memory.current(10 B)/proc/meminfo(1.6 KB)/proc/stat(11.6 KB)/proc/self/smaps_rollup(698 B)/proc/kallsyms(12.5 MB)/proc/slabinfo(45 KB)/proc/pagetypeinfo(3.9 KB)/proc/vmallocinfo(3.9 MB)fstatfsis the other test: the filesystem type tells procfs from a disk. It costs one syscall and no second record. It needs a list of filesystem types (procfs, cgroupfs, debugfs, tracefs, securityfs, FUSE with direct I/O), and Bun has no such list today. With it,/proc/self/pagemapis a body with no end and/proc/self/memcloses the connection. Question 2 asks a maintainer to choose.The first revision of this shape (1924ab8) had no test: each file with
st_size0 went through the reader. On a static route, an empty file then lostLast-Modified, the304forIf-Modified-Sinceand the416for a Range. It paid onereadand 1 to 2 allocations.BodyLengthStartOptions.lengthwas anOption<u64>.Some(n)said two things: the producer wroteContent-Length: n, and the reader stops afternbytes. A slice of a file with no size needs the second and must not claim the first.BodyLengthhas one variant for each case:Exact(n): the producer wroteContent-Length: n. Only this variant can usesendfile.ToEof: noContent-Length, read to EOF. Pipes, sockets, and a whole file with no size.UpTo(n): noContent-Length, read to EOF and not more thannbytes. A slice of a file with no size.Code that decides the framing calls
exact(). Open PRs that readStartOptions.lengthas anOptionneed that change at a rebase: #41622, #41474, #38107, #40875.Measurements
Release builds of the base (26376b9) and of this PR, same machine and toolchain.
strace,perf,valgrindandbloatyare not installed on the test machine. The counts come from two smallptracetools. One logs each syscall of each thread. The other single-steps the JS thread. It counts the instructions and the calls of the mimalloc entry points (mi_malloc,mi_calloc,mi_zalloc,mi_realloc, the aligned and heap variants). One keep-alive connection, one request at a time.User instructions of the JS thread for one request, from the return of
recvfromto the last write to the connection. The static and directory routes give the same count for each request. The fetch handler runs JavaScript, so its count changes from request to request. For a sized file its row is the least count of 60 requests after 300 warm-up requests. That count is also the one that most requests have. The other fetch handler rows give the median of 30 requests.Range: bytes=0-9, static route/proc/versionGET, static route/proc/versionHEAD, static route/proc/versionGET, fetch handler (median of 30)/proc/versionHEAD, fetch handler (median of 30)Syscalls per request,
futexleft out:Range: bytes=0-9, static routesendfile): fetch handler, static routesetImmediatesendto)sendto)pread)pread)open,pread,close)/proc/versionGET: fetch handler, static routepread, tworead)/proc/versionGET, handler that resolves aftersetImmediatesendto)pread, tworead, 9sendto)/proc/versionHEAD, static routepread)/proc/versionHEAD, fetch handleropen,pread,close)/proc/cpuinfoGET: fetch handler, static routepread, 65read, 8sendto)/proc/self/pagemapGET, static routepreadthat fails)Heap allocations per request:
/proc/versionGET, static route/proc/versionGET, fetch handler/proc/versionHEAD: static route / fetch handlerTime that one GET from a fetch handler holds the event loop: the greatest gap of a 1 ms timer, median of 15 requests. Two rounds for each build, one build after the other, uid 0, load average 850 to 1300. The 1 ms of the timer is the floor.
/proc/version(219 B)/proc/cpuinfo(178 KB)/proc/self/smaps_rollup(698 B) at 0.03, 0.5 and 2 GiB resident/proc/slabinfo(45 KB, mode 0400)/proc/kallsyms(12.5 MB)/proc/vmallocinfo(2.7 MB to 3.2 MB, mode 0400)/proc/pagetypeinfo(3.9 KB, mode 0400)The time is the time the kernel takes to make the content. The reader of
FileResponseStreamrunsreadon the JS thread for each file that does not go throughsendfile, on the base too. The four Dockerfiles indockerhub/set noUSER, so the images run as uid 0, which can read the files of mode 0400.Binary size:
.textsection (size -A)textcolumn ofsize,bun-profilebun, stripped, file bytesFrom
nm -S, the functions grow by 3,192 bytes:do_sendfile+1,408 (176 in each of its 8 copies),do_render_head_response+620 (its 8 copies together),zero_size_path_has_content584 (new),FileRoute::on+440,zero_size_file_has_content111 (new),FileResponseStream::start+29.StartOptionskeeps its size:BodyLengthhas the 16 bytes thatOption<u64>had.Source: +159 -33 lines in 4 files of
src/.Tests
test/js/bun/http/bun-serve-file.test.ts: 28 tests, Linux only but one. On a release build of the base, 21 fail and 7 pass. Of those 7, 6 hold an answer that must not change (an empty file, a file that cannot be read,Bun.file(fd)). One is atest.failingrow for aBunFileafter.sizeorexists()(Bun.file: do not use a cached stat size as the byte budget of a read or a copy #43910).test/js/bun/http/serve-file-slice-read-error.test.ts: one test. A supervisor process makes the secondreadof/proc/loadavgfail. The response must not be complete. It fails on the base./proc/versionand two files of a child process that stopped itself with SIGSTOP. Itsenvironhas about 480 KB. Itsmapshas 64 added mappings and comes in reads of at most one page.Content-Length. It fails for a chunk size that is not hexadecimal, and for chunks with no last chunk. The FIFO tests of Bun.serve: terminate FIFO/pipe file responses at EOF #37082 use the same parser.BunFilethat all the responses of a handler share.serve-directory-routes.test.ts,bun-serve-static.test.ts,serve-if-none-match.test.ts,bun-serve-headers.test.ts,tls-bunfile-leak.test.ts,serve-http2.test.ts,serve-http3.test.ts.cargo clippy -p bun_runtime,bun run rust:check-all(12 targets) andtest/internal/source-lints/pass.bun-serve-file.test.tsthat this PR does not change ("pollable Bun.file ...", from Bun.serve: pause the file reader while the response is backpressured #41244) start a debug build of bun. There they take 6 to 9 s and end at the 5 s limit. They pass with a longer limit, and with release builds of the base and of this PR.Limits
BunFilewhose size something asked for holds a window of 0 bytes:.size,await exists(), a read ofresponse.body,Bun.inspect(response). The server cannot tell it fromslice(0, 0).stat(),lastModifiedandclone()do not store it. Bun.file: do not use a cached stat size as the byte budget of a read or a copy #43910 and Leave a body stream unsized when the file's st_size is 0 #41593 own that.Bun.file(fd)keeps the answer of itsstat. A read moves the position of a descriptor that the caller owns.{ dir }route keeps the answer of itsstat. There the client names the file. A maintainer decides if such a route reads these files.statwhen itsResponsehas aContent-Lengthheader. So does a route whose status has no body. Bun.serve: drop Transfer-Encoding and Content-Length from a Bun.file() route's Response headers #43108 owns the header./proc/kmsgandtrace_pipegive bytes only when the kernel has some, and a read takes them away. Neither can be opened on the test machine (EPERM, not mounted), so nothing here is from a run. If thepreadgets a byte there, the response does not have that byte. The reader does not wait for a regular file (src/io/PipeReader.rs, "Received EAGAIN while reading from a file"). So the response then stays open until the idle timeout. The base answers an empty200at once.slice()counts from the largest size of aBlob, as inBun.file(path).slice(0, -10).bytes(). Bun.file().slice(): count a negative index back from the file's real end #41257 owns that.st_size4096 and holds fewer bytes. It is not in this class. Close the connection when a Bun.file response ends short of its Content-Length #41622 covers a body that ends short of itsContent-Length.respondWithFileofnode:http2frames fromstat.sizetoo. node:http2: handle an empty byte range in respondWithFile and respondWithFD #41516 covers it.HAS_PROCFSis false there. It is true on Linux and Android.Self-review
The review of the revision before (1924ab8) raised 27 concerns. 24 are addressed. The main ones:
test.failingrow and the strict parserContent-LengthheaderBodyLengthcontract aboveNot taken:
bun_systhat the sibling PRs can call.zero_size_file_has_contentstays in the server. The reads of the siblings are in other crates and other PRs.fstatfsas the test for content. It is question 2.Other PRs
FileResponseStream::finishends withresp.end(b""), which writes the last chunk.do_sendfile, for each file. It changes the same lines. After it,do_sendfiletests the file for HEAD too, andzero_size_path_has_contentcan go.Blob::resolve_size. #43910 changesRequestContext.rsand this test file.Bun.filereads,fetchbodies, the module loader, dotenv, FormData. No file in common.StartOptions.lengthas anOption<u64>. SeeBodyLengthabove.FileResponseStream.rs.src/io/PipeReader.rs), which this PR does not change.Found on the way, not in this PR:
uws_res_end_without_body(src/uws_sys/libuwsockets.cpp) namesFileResponseStream::finishas a caller. With Bun.serve: terminate FIFO/pipe file responses at EOF #37082 it is not one.sendto: 7 calls for a sized file on the base. Cork every HttpResponse write so post-await responses leave in one send() #41341 covers it.The branches
robobun/00c06367/serve-file-whole-body(64489d0) androbobun/00c06367/serve-file-chunked-body(86b62a0) hold the two shapes that read the file before the status line.