Repository navigation
Conversation
|
Status: draft. The head is aae9f44 (the same commit as a8b74c2, rebased on the narrowed #47524 and #47647 with no change of content), with the rule that the last status announced. Before the import check of #47666, Bun decides whether it fills the addon's imports of the 13 zlib functions: all of them or none. It fills only when every relocation that names one of them is unbound and storable and Closed on this head: two of the three holes of the version before. An address-taken zlib import that the loader bound to a zlib that no lookup finds, and a weak import that the addon tests before the call: both addons are now refused with the base's text, or filled as a whole. Limits that stay (the PR description has both under "Limits"): a library that the addon links and that imports zlib from the host ends the process at its first zlib call, and an addon that keeps the address of a zlib function outside the 13 under a library that has its own zlib gets one stream in two zlibs. uWebSockets.js has neither shape. One decision for a maintainer: how an addon gets zlib from Bun. The answer differs by object format. On ELF it is this private binding, or an export of all 88 zlib names with no version. On macOS and Windows only an export can work (the 13 names in How I reproduced it: with the base's The program of #9785: a local Linux x64 debug build of this stack (at 7ca23d0) with #47619, #47607, #47876 and #46585 merged in prints CI (build 137600, head aae9f44, finished): both test files have 0 failures on every lane. |
4246a31 to
ac1eeb3
Compare
d8e329f to
3011653
Compare
ac1eeb3 to
f4b61ac
Compare
3011653 to
1d0c07d
Compare
e7f121a to
46fbfb8
Compare
65cfd3e to
9c1a9ae
Compare
46fbfb8 to
e1c022b
Compare
e1c022b to
6f9f156
Compare
83ff774 to
6af437e
Compare
7ca23d0 to
a8b74c2
Compare
…ets Bun's zlib
Node's executable exports its zlib to addons. uWebSockets.js imports 13
zlib functions from it and links no libz. Bun links zlib-ng and does not
export it, so the import check refuses the addon on its first zlib
import.
With glibc, Bun now fills an addon's imports of these 13 functions with
its own functions: all of them or none. The filling is a step of its
own before the import check. When Bun fills nothing, the check runs
exactly as without this change.
1. Before the check, Bun reads every relocation of the addon that
stores the address of one of the 13 functions: a jump slot, a slot of
the global offset table or a word of data, in the table of DT_JMPREL
and in that of DT_RELA, for a strong and for a weak import. A slot
that holds Bun's own function already is left out: an earlier load
has filled it, or a load on another thread is filling it now. Bun
fills the other slots, unless one of these holds:
- one of them is bound to something else. The loader has then found
a zlib in the addon's scope, also where no lookup finds one;
- a word of data has an addend;
- Bun cannot read a table, a symbol, a name or a slot;
- dlsym finds one of the 13 names, in the global scope or in the
addon's own tree;
- a slot is not storable: it is outside the writable PT_LOAD ranges
or inside PT_GNU_RELRO.
In each of these cases Bun fills nothing. None of them is an error.
2. The import check runs as before. An import whose slot Bun fills
counts as bound. With nothing to fill, an import of zlib is treated
like every other import: a weak one is skipped, a strong one is asked
about, and the first one that nothing defines refuses the addon by
its name.
3. If the check has found nothing, Bun stores its functions in the
slots.
So nothing is stored in an addon that is refused, and Bun never fills a
part of these imports. An addon keeps the zlib of its scope where a
bound slot or dlsym shows that zlib. An addon without a relocation that
names one of the 13 makes the same dlsym calls as before.
Two limits stay.
Bun reads only the object that has the initializer. A library that the
addon links and that imports zlib from the host is not filled: the
addon loads, and the library's first zlib call ends the process. The
import check has the same limit for every import of such a library.
No lookup searches the scope of another library that has loaded the
addon as its dependency. A zlib that is only there shows only in a
bound slot of one of the 13 names. If the addon's imports of the 13 are
all calls that the loader has not bound yet, Bun fills them. A zlib
function outside the 13 that the loader has bound there in dlopen(),
such as the address of deflateParams, then gets a stream of Bun's zlib.
The reader has one entry point for every target. musl and FreeBSD call
it with an empty table and get the import check alone: musl binds a
pending import again at each later dlopen(), which would undo a stored
slot, and nothing has run this on FreeBSD.
The test addon with the shape of uWebSockets.js has a second build that
deflates its response with the host's zlib.
Part of #9785
1f6914b to
41b7658
Compare
a8b74c2 to
aae9f44
Compare
Part of #9785
Problem
it imports 'inflateReset', and Bun found no definition of it.undefined symbol: deflateInit2_, exit 127).Fix
node_register_module_v<ABI>on Linux with glibc, Bun fills the imports of these 13 with its own zlib: all or none.dlsymfinds none of them. It stores after the import check.test/js/node/process/dlopen-duplicate-load.test.ts,test/v8/v8.test.ts), 11 fail on the base. 15 unit tests, also with Miri.Background
RTLD_LAZYthe loader fills an import's slot at the first call and never looks up a filled slot.z_streammust not pass through two zlib implementations, so Bun never fills a part.Downsides
.text+2,944 bytes.Bun.buildwith a pendingsetup()) can end the process until Stop running the event loop inside JavaScript calls #46545 (Notes).Notes
One decision for a maintainer: how does an addon get zlib
The answer differs by object format. uWebSockets.js ships binaries for five targets.
src/symbols.txtsrc/symbols.defsrc/libc-functions.lds, which exports 12 libc functions that way since Compile TypeScript to machine code ahead of time:bun build --compile --aot#45631. A model of the other forms (a small host linked with Bun's link flags and Bun's zlib objects): an export of only the 13 names, or under theBUN_1.2version, crashes the distribution's libpng16 in a PNG read. With all 88 names and no version, libpng16 and every library that linkslibz.so.1bind to the host's zlib, and a caller that openslibz.so.1by handle (asbun:ffidoes) aborts in a streaming call. The nodejs.org build of Node has that last effect too.--shared-zlib: Homebrew's Linux build of Node exports 0 zlib names, and the 13 imports of the addon bind tolibz.so.1.LD_PRELOAD=libz.so.1loads the addon on the system's zlib. This PR removes that step and the need for a system zlib.src/changes of this PR.Limits of this head
In each case the base refuses the addon with a catchable error, and Node v26.3.0 works.
deflateParams) in its data, and a zlib is only in the scope of another library that loaded the addon as its dependency. The loader binds that address there indlopen(). No lookup searches that scope, and the 13 imports are calls that the loader has not bound yet, so Bun fills them. A stream of Bun's zlib then goes into the other zlib:zlib failed at deflateParams. It needs both things at once. uWebSockets.js imports only the 13.Two ways to close the second one, neither built: take a bound relocation of any zlib API name as the proof of a zlib (that needs a list made from zlib's own export list), or ask
dlsymalso on the handle of every loaded object that needs the addon.The version before this head had two more holes, which are closed. An address-taken import of one of the 13 that the loader bound to a zlib that no lookup finds, beside 12 jump slots: one stream went through two zlibs. A weak import that the addon tests before the call (
if (deflate)): the addon got 12 functions and a nulldeflate. Both addons are now refused with the base's text, or filled as a whole.Order
v8::Global) is an open draft on main, v8: add the 23 members that uWebSockets.js imports #47619 (V8 members) is a draft on v8 tests: run on musl #47863 (a test change on main), and v8::Value::Uint32Value: convert with ToUint32 instead of testing for a uint32 #46585 (Uint32Value) is open on main.awaitends the process.app.listen(host, "3000", cb)aborts with exit 134.Bun.buildfor a pendingsetup(), on the build of "End to end": four exit 0, one loses anonAbortedcall, one ends with the addon's ownCork buffer must not be held across event loop iterations!(exit 134), and one with a segmentation fault in the sweep timer callback of uSockets.ArrayBuffer::Detach()in the Notes of v8: add the 23 members that uWebSockets.js imports #47619. Until the cure in JavaScriptCore ([JSC] A pinned ArrayBuffer keeps its bytes allocated when it is detached or its wasm memory moves WebKit#833) is in, a handler that gives a body chunk to an asynchronous Bun API keeps a view of memory that the addon reuses.process.dlopen: process.dlopen: load an addon and save its registrations under one lock #47732, process.dlopen: honorflags, split the.nodehandler from it, and use Node's codes for load failures #47711, process.dlopen: give every init function ToObject(module.exports), read once before the load #47700, process.dlopen: refuse the V8 build of better-sqlite3 by what it registers, not by its file name #47648. process.dlopen: load an addon and save its registrations under one lock #47732 puts a load under one lock. The store of this PR needs no lock and takes none.The rule
The table is the 13 functions that uWebSockets.js imports:
deflate,deflateBound,deflateEnd,deflateInit2_,deflateInit_,deflateReset,deflateSetDictionary,inflate,inflateEnd,inflateInit2_,inflateInit_,inflateReset,inflateSetDictionary.The filling is a step of its own before the import check of #47666. When Bun fills nothing, the check runs exactly as without this PR.
Bun reads every relocation of the addon that stores the address of one of the 13 functions: a jump slot, a slot of the global offset table or a word of data, in the table of
DT_JMPRELand in that ofDT_RELA, for a strong and for a weak import. A slot that holds Bun's own function already is left out: an earlier load has filled it, or a load on another thread is filling it now. Bun fills the other slots, unless one of these holds:dlsymfinds one of the 13 names, in the global scope or in the addon's own libraries.In each of these cases Bun fills nothing. None of them is an error.
The import check runs as on the base. An import whose slot Bun fills counts as bound. With nothing to fill, an import of zlib is treated like every other import: a weak one is skipped, a strong one is asked about, and the first one that nothing defines refuses the addon by its name.
If the check has found nothing, Bun stores its functions in the slots.
So nothing is stored in an addon that is refused, and Bun never fills a part of these imports. A zlib import outside the 13 is an ordinary import. With a zlib in scope the loader binds it. Without one the addon is refused by its name.
The store
PT_LOADrange that is readable and writable, and outsidePT_GNU_RELRO.bun_exe_format::elf_imports::resolve_lazy_imports, decides, checks and stores. Every ELF target calls it. Only glibc passes a table. With an empty table it is the check of process.dlopen: load an addon through node_register_module_v<ABI> where its imports can be checked #47666.Who gets Bun's zlib, and who does not
node_register_module_v<ABI>, on Linux with glibc, opened lazily, with nothing in its scope that defines one of the 13 names and with no import of the 13 that the loader has bound.LD_PRELOAD,db.loadExtension(), a library opened withRTLD_GLOBAL) or in the addon's own libraries, the addon runs on that zlib, as on the base. On 18 fixtures with the system's zlib in the global scope, this build gives the base's result on all 18.dlopen(), which would undo a stored slot.-z now,LD_BIND_NOW=1) or built with-fno-plt, with nothing that defines zlib:dlopen()fails first, with the loader's own error.dlopen()(a static constructor) runs before Bun stores anything.What each runtime does (Linux x64, release builds.
[534,385]are the sizes of a raw deflate of the test text at levels 1 and 6 from zlib 1.3.1, which is Node's and this system's.[817,425]are Bun's. "Slots" is the count of the addon's zlib slots that hold a function of another object, before and after the load.)[534,385]deflateInit_[817,425], slots 0 then 13[534,385]twice[817,425]twice, slots 0, 13, 13inflateCopy, with two weak ones[534,385][534,385], slots 0 and 0deflate[534,385][534,385], slots 1 and 1libz.so.1[534,385]deflateInit_deflateInit_, slots 1 and 1[534,385]deflateInit_[817,425], slots 0 then 13deflatethat the code tests, with RELRO[534,385]deflateInit_deflateInit_, slots 0 and 0-z norelro[534,385]deflateInit_[817,425], slots 0 then 13[534,385]deflateInit_[817,425], slots 0 then 13[534,385]deflateInit_[817,425], slots 0 then 13inflateCopy, nothing defines zlib[534,385]deflateInit_inflateCopy, slots 0 and 0[534,385]inflateSetDictionaryinflateSetDictionary, slots 0 and 0[534,385][817,425], slots 0 then 13-z now, the system's zlib in the global scope[534,385][534,385], slots 13 and 13-z now, nothing defines zlib[534,385]undefined symbol: deflateInit_(loader)[534,385][534,385], slots 0 and 0200,deflate, the bodydeflateInit2_deflateParamsbeside the 13, under a library that needslibz.so.1[534,385]deflateInit_zlib failed at deflateParamsEnd to end (Linux x64, glibc 2.41, Node v26.3.0. A local debug build of the heads of the series on Nov 4: this stack at aae9f44 with #47619 at 02c6c91, #47607 at dbfbb42, which has #47876 below it, and #46585 merged in. The real prebuilt addon, with no shim in front of it.)
200 Hello uWS, for the head of the addon'sbinariesbranch and for its tag v20.71.0.fetchsends.await, a read of the request after it, a throw under anuncaughtExceptionlistener): the same output as Node.onData,onDataV2andcollectBody,res.endwith an ArrayBuffer and a Uint8Array,tryEnd,getRemoteAddress, six WebSocket cases with open, echo, send, options, upgrade data and publish,listenwith a string port): 19 print what Node prints. One of them times out on both.examples/WorkerThreads.mjs(an acceptor app gives a socket to an app on a Worker): the same two lines as Node. Node then aborts atprocess.exit()(uv loop ... has open handles), Bun exits 0.Tests
test/js/node/process/dlopen-duplicate-load.test.ts, describe "an addon with only this entry that imports zlib": 23 tests, one of them for musl. A helper holds the addon withbun:ffiand counts its zlib slots that hold a function of another object, before and afterprocess.dlopen.test/v8/v8.test.ts: "serves a response that the addon deflates with the host's zlib, like in Node".src/of the base (process.dlopen: load an addon through node_register_module_v<ABI> where its imports can be checked #47666), 10 of the 22 that run on glibc fail, and the deflate response fails. The other 12 pin that Bun changes nothing where it fills nothing.src/of the version before this head, 11 fail: the two holes that are closed (both loaded and then failed), the weak address in writable data (a crash), all 13 weak beside a partial library (refused), and the tests that expect the new refusal text.test/v8/v8.test.ts.cargo test -p bun_exe_format: 316 pass, 61 of them in the reader.scripts/rust-miri.ts): 59 pass, 2 ignored. Its two-thread test fails under Miri when the store is a plain write or the read of a slot a plain read.BUN_JSC_validateExceptionChecks=1on the first file: no report.libz.so.1are skipped without it.Measurements (Linux x64, release builds of this head and of the head of #47666)
.text+2,944 bytes, all sections +2,944. Dynamic names: 704 defined and 394 undefined before and after, none of them a zlib name.dlsymcalls of a first and of a secondprocess.dlopenin one process, counted in gdb:LD_PRELOAD=libz.so.1LD_PRELOAD=libz.so.1-z nowprocess.dlopen(a hardware counter, first and second load):-z nowLD_PRELOAD=libz.so.1)Self-review
35 concerns raised on the first version of this PR. 34 are addressed:
deflateandLD_PRELOAD=libz.so.1gotZ_STREAM_ERROR. It also crashed on weak imports, and its list of the 88 zlib names was a hand copy that was 7 names short. The list is gone.-z nowaddon is no longer skipped by its flags. The check reads the slots.--shared-zlibbuilds, theLD_PRELOADbaseline and the upstream way. The memory cost and the merge order are stated.Rejected as a change of this PR (1): a library that the addon links and that imports zlib. It needs a walk of every library of the addon, and the base has the same limit. Limit 2 above was found while this head was built, not by the review.
Other platforms
test/js/node/process/dlopen-duplicate-load.test.ts: 63 pass and 2 skip on Linux x64 and aarch64 with glibc, also with ASAN. So the filling ran on aarch64. 36 pass and 29 skip on both Alpine lanes (musl), where Bun fills nothing. 17 pass and 48 skip on Windows x64 and aarch64, 20 pass and 45 skip on macOS x64 and aarch64.test/v8/v8.test.ts, which has the uWebSockets.js-shaped tests: 108 pass on Linux x64 and aarch64 with glibc, 107 pass and 1 skip with ASAN, 88 pass and 20 skip on Windows, 104 pass and 4 skip on macOS. With musl the whole file istodo, as on main.clang-format, the source lints and the type checks for musl, FreeBSD and aarch64 passed locally.test/aot/**family andtest/bundler/compile-aot-size.test.ts, which are red on main, andtest/js/web/request/request-clone-leak.test.tson the ASAN lane, which this diff does not touch.