Skip to content

arena: allocate the abandoned-page bitmaps on first abandon, and not at all for a heap that is being released - #32

Merged
Jarred-Sumner merged 7 commits into
oven-sh:bun-dev3-v2from
robobun:robobun/9dfcc910/lazy-abandoned-bitmaps
Sep 3, 2026
Merged

Jarred-Sumner merged 7 commits into
oven-sh:bun-dev3-v2from
robobun:robobun/9dfcc910/lazy-abandoned-bitmaps

Conversation

@robobun

@robobun robobun commented Sep 2, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • Every (heap, arena) pair lays out MI_ARENA_BIN_COUNT (49) abandoned-page bitmaps up front, each sized by the arena: ~110 KiB per heap for the default 1 GiB arena, ~410 KiB for a heap in a 4 GiB arena. The header write of each bitmap touches its own OS page. So a heap pays ~50 page faults, or a memset of that size, before its first allocation. Most heaps never abandon a page.
  • In bun this is one heap per transpile (created and destroyed on a thread pool) plus JSC's structure heap, a 4 GiB exclusive arena: about 250 of the ~1000 page faults of a bun file.js start.

Fix

  • pages_abandoned[bin] starts NULL and is allocated the first time a page of that bin is abandoned (mi_arena_pages_abandoned_ensure), from the meta-data heap, published with a CAS. Readers treat NULL as all-clear. They were already guarded by heap->abandoned_count[bin]. If the allocation fails, the page is abandoned unmapped, like a full page.
  • A heap that is being deleted or destroyed (heap->releasing) abandons its pages unmapped. Its teardown claims every page through arena_pages->pages, and _mi_arenas_page_unabandon already handles an unmapped page. Without this, each teardown allocated a map per size class under the meta lock, only to free it a moment later.
  • Verified: test/test-abandoned-lazy.c (new). It abandons from 8 threads, reclaims, re-abandons by a foreign free, deletes while 4 threads free into the heap, and counts the maps that released heaps allocate (0, and 600 without the second commit). ctest Debug (MI_DEBUG_FULL): 25/25. ctest Release with MI_USE_CXX: 24/24. The new test also passes under ASAN.

Background

  • An abandoned page is a page whose owning thread exited with blocks still live. The per-bin bitmaps let a thread that allocates that size class find and reclaim such a page.
  • The meta-data heap (_mi_meta_zalloc_aligned) is usable on the abandon paths (thread teardown, a foreign free), where a regular heap is not.

Measured in bun (release, x64 Linux, interleaved A/B, median), main against this PR plus bun's other startup changes:

before after
bun empty.js 3.99 ms, 1039 faults 3.29 ms, 789 faults
entry with 300 TS imports, CPU time 27.4 ms, 2578 faults 25.0 ms, 2054 faults
Notes
  • The first version of this PR had no teardown commit. With 300 imports it used about 4 ms more CPU than the base: each transpile heap allocated and freed its maps under the meta lock on the thread pool. The second commit removes that.
  • Two more commits were in the first version: the NUMA node count from /sys/devices/system/node/online, and option init without an environment scan when no MIMALLOC_* variable is set. Measured alone they save 3 syscalls and 2 page faults per process, and the time difference is below noise. They are not in this PR.
  • I could not run the TSAN configuration. Its runtime needs lower ASLR entropy, and the container blocks personality, so every test stops before main, with or without this change.
  • Wall times came from a loaded 16-core VM. Page faults and CPU time are the numbers to compare.

@robobun
robobun force-pushed the robobun/9dfcc910/lazy-abandoned-bitmaps branch from 8c1a4f3 to 9567ee9 Compare September 3, 2026 02:44
@robobun robobun changed the title arena: allocate the abandoned-page bitmaps on first abandon; cheaper NUMA and option init arena: allocate the abandoned-page bitmaps on first abandon, and not at all for a heap that is being released Sep 3, 2026
Jarred-Sumner pushed a commit to oven-sh/bun that referenced this pull request Sep 3, 2026
### Problem
- `Bun__StackCheck__initialize`
(`src/jsc/bindings/wtf-bindings.cpp:298`) computes the thread's stack
bounds each time it is called. The main thread calls it twice: from
`main`, and again from `VirtualMachine::init`.
- For the main thread on Linux, `pthread_getattr_np` reads and parses
all of `/proc/self/maps`. So each `bun file.js` and `bun -e` start
parses it once for nothing.

### Fix
- The function returns when this thread's bounds are already set. A
thread's bounds are fixed. On Windows they are the reserved range of the
stack, and the stack grows inside that range.
- The call in `VirtualMachine::init` stays. It sets the bound for a
thread that skipped it at start (`configure_thread_no_js`). Its old
comment said JSC may change the stack size, which it does not.
- Verified on release builds (numbers below). CI runs the stack tests:
`test/js/bun/test/expect-stack-overflow-crash.test.ts`,
`test/js/bun/toml/toml.test.ts`, `test/js/bun/yaml/yaml.test.ts`, and
`test/regression/issue/26142.test.ts`.

Release x64 Linux, against `main`, interleaved, 600 rounds, median:

| | main | this PR |
|---|---|---|
| `/proc/self/maps` opens, `bun empty.js` | 3 | 2 |
| syscalls, `bun empty.js` | 312 | 300 |
| `bun empty.js` | 4.41 ms | 4.36 ms |

### Background
- `StackCheck` is bun's recursion guard for its own parsers (JS, JSON,
TOML, YAML). JSC's stack limit is separate.
- WTF computes the main thread's bounds once more for itself, with the
same `/proc/self/maps` parse. A change to `StackBounds.cpp` in
oven-sh/WebKit can remove both remaining parses. That is a separate
change.

<details><summary>Notes</summary>

- Windows: `WTF::StackBounds` calls `GetCurrentThreadStackLimits`. A
probe linked with bun's `/STACK:0x1200000,0x200000` read the same range
at start and 10 MB deep in a recursion. Only the committed part grew,
from 2 MB to 10 MB. The output is in the conversation below.
- WTF makes the same assumption: `Thread::initializeInThread` computes a
thread's bounds once, and JSC's stack limit uses that value.
- I found no thread that creates a VM and skips the bound at start. Each
`ThreadPool` sets it (`needs_stack_bounds` is `true` in all pools), and
the watcher thread runs no JS. So the call in `VirtualMachine::init` is
one thread-local check today.
- `bun --version` is unchanged: it never creates a VM, so it computes
the bounds once.
- A forked child inherits the thread-local bounds of the thread that
forked, and that thread's stack is the child's stack. So the early
return holds there too.
- The syscall and `/proc/self/maps` counts are from `strace -f` on the
two release builds.
- My local debug build and test run of this change were cut off twice by
container resets. An earlier version of this change, which moved the
same bound computation, passed the same stack tests.
- Other startup work, measured separately: #41233 (cpuid without the
libgcc constructor), oven-sh/mimalloc#32, oven-sh/WebKit#553 (merged)
and oven-sh/WebKit#554.
</details>
Jarred-Sumner pushed a commit to oven-sh/bun that referenced this pull request Sep 3, 2026
#41233)

### Problem
- `bun_cpu_features()` (`src/jsc/bindings/CPUFeatures.cpp`) uses
`__builtin_cpu_supports` on Linux and macOS. That builtin links libgcc's
constructor `__cpu_indicator_init`, which runs a dozen cpuid
instructions before `main`. Under a hypervisor each one is a VM exit.
- On Windows it uses `IsProcessorFeaturePresent`. On Windows Server 2019
that returns false for SSE4.2 and AVX on CPUs that have them. The crash
report then has no `CPU:` line and says "CPU lacks AVX support".

### Fix
- The function runs cpuid itself on every x86_64 OS, with the XCR0
checks of compiler-rt: bits 1 and 2 for AVX, bits 5 to 7 for AVX-512
(not on macOS).
- Nothing else references `__cpu_model`. So the binary has one
constructor less, and the vendored compiler-rt `cpu_model` sources for
the macOS x64 cross link are removed.
- Verified: `test/cli/run/run-crash-handler.test.ts` checks the `CPU:`
line, now on Windows too. On Windows Server 2019 it fails on main and
passes here. `test/js/bun/perf/static-initializers.test.ts` expects two
constructors on macOS x64.

Release x64 Linux, against main, interleaved, 1500 rounds, median:

| | main | this PR |
|---|---|---|
| `.init_array` entries | 11 | 10 |
| `bun --version` | 1.033 ms, 212 faults | 1.014 ms, 209 faults |

### Background
- The crash report is the only reader: its `CPU:` line, its AVX message,
and the `no_avx` counters.
- XCR0 tells which register state the OS saves. A CPU can have AVX while
the OS does not save the YMM registers. Darwin enables the AVX-512 state
at a process's first AVX-512 instruction.

<details><summary>Notes</summary>

- Windows x64: on a Xeon Platinum 8573C with Windows Server 2019 (build
17763), `IsProcessorFeaturePresent` returned 0 for the SSE4.2, AVX,
AVX2, and AVX-512 constants. cpuid reported all four, and XCR0 was 0xe7.
The report now says `CPU: sse42 popcnt avx avx2 avx512`.
- Windows arm64 keeps `IsProcessorFeaturePresent`. On Windows 11 arm64
(build 26100) the canary reports `CPU: neon fp aes crc32 atomics sve`,
so the test passes there.
- `<cpuid.h>` and the `xgetbv` inline asm compile with clang-cl 21 next
to `<windows.h>` and `<intrin.h>`.
- `CachedTypes.cpp` in WebKit already calls `__get_cpuid` to avoid the
same constructor.
- Searched for other users of `__cpu_model`: `src/`, `packages/`,
`vendor/`, the fetched deps, the cargo registry, and the undefined
symbols of the prebuilt JSC. None. On Linux, libgcc's `cpuinfo.o` leaves
the link with this change. CI's macOS x64 cross link checks the rest.
- `-e 1` shows no difference above the noise of this VM.
- The other startup work: oven-sh/mimalloc#32 (per-bin abandoned
bitmaps, about 250 faults per start), oven-sh/WebKit#553 (merged: the
warm-up thread starts after 64 blocks), oven-sh/WebKit#554 (atom table
reservation, cgroup files, structure arena), oven-sh/WebKit#556 (main
thread stack bounds without `/proc/self/maps`), #41238.
</details>

<!-- robobun:evidence:begin -->

---

**no test proof** · iteration 0 · platform-specific test(s) that do not
run on this machine, deferring to CI, which covers all platforms:
test/js/bun/perf/static-initializers.test.ts,
test/cli/run/run-crash-handler.test.ts

<!-- robobun:evidence:end -->
robobun and others added 7 commits September 2, 2026 22:48
Every (heap, arena) pair carried MI_ARENA_BIN_COUNT abandoned-page
bitmaps next to its `pages` bitmap, laid out and initialized up front:
once in the arena's info slices for the main heap, and once more in a
zeroed allocation for every other heap that touched the arena
(`mi_heap_ensure_arena_pages`). Each bitmap is sized by the arena
(2 KiB per GiB of arena), so with the default 1 GiB reservation that is
~110 KiB per heap, and ~410 KiB for a heap in a 4 GiB arena. Writing the
header of each bitmap touched one OS page per bitmap, so a heap paid
about 50 page faults (or the memset, for external memory that is not
known to be zero) before its first allocation, and most heaps never
abandon a page at all: short-lived heaps are destroyed, not abandoned.

Bun creates one heap per transpile and one for JSC's 4 GiB structure
heap, so this was about 250 of the ~1000 page faults of `bun file.js`.

Now `pages_abandoned[bin]` starts NULL and is allocated the first time a
page of that bin is abandoned, from the subproc meta-data heap (safe on
the abandon paths, where a regular heap allocation is not), published
with a CAS. Readers treat NULL as all-clear; they were already guarded
by `heap->abandoned_count[bin]`. `_mi_arena_pages_free` frees them with
the heap's arena pages. If the allocation fails the page is abandoned
unmapped, like a full page, and mapped once a free brings it back.

The new test-abandoned-lazy exercises the abandon, reclaim,
re-abandon, visit and delete paths from several threads at once.
`mi_heap_delete` and `mi_heap_destroy` abandon every page of the heap
(step 2 of the teardown) and then claim each one back through the
`arena_pages->pages` bitmap (step 3). With the per-bin abandoned maps
allocated on first abandon, step 2 allocated a map for every size class
the heap had used, from the meta-data heap and under its lock, only to
free them with the heap a moment later. A process that creates and
destroys a heap per job on several threads (bun creates one per
transpile, on a thread pool) paid that on every job, and the threads
contended on the meta lock: with 300 imports, bun used about 4 ms more
CPU than with the maps allocated up front.

The heap now records that its release has started (`heap->releasing`).
`_mi_arenas_page_abandon` leaves the pages of such a heap out of the
per-bin maps, as it already does for a full page, and a concurrent free
does not re-map them. Step 3 finds them through `pages` as before, and
`_mi_arenas_page_unabandon` already handles an unmapped page. A heap
that lives only to be released now never allocates a map, and its
`arena_pages` block holds just the `pages` bitmap.

test-abandoned-lazy counts the maps published while heaps are created,
used, destroyed and deleted on one thread (a debug-build counter, as
test-heap-teardown uses a debug hook): 0 now, 600 without this change.
It also deletes heaps on four threads while four other threads free
their blocks during the delete.
…elease on many threads

`mi_heap_delete` walks `arena_pages->pages` and then frees the heap and the bitmap, while a
foreign `mi_free` that frees a page of that heap reads both and clears the page's bit as its
last access. The walk's loads were relaxed, so nothing formally ordered those reads before
the free of the memory they read. They are acquire now.

The walk also located the page struct (which reads `block_size` with separated page meta)
before pinning the slice, when the slice may already be freed or be a fresh page of another
heap. It is located after the pin now.

On macOS the theap TLS slot is a raw slot in the thread's control block, which the OS reuses
for a later thread: the exiting thread's last store and the next thread's first load are the
same address from two threads. Use relaxed atomics there (same single load/store).

`test/test-heap-release-mt.c` (new): heaps created and deleted/destroyed on 8 threads while 4
threads free their blocks and batches of short-lived threads allocate from them and exit.
Clean under TSAN, ASAN and MI_DEBUG_FULL; before this it produced TSAN reports on the base
branch as well.
…ger sweeps them

The second half of the test runs the same churn with each churn thread parked
(`mi_on_thread_idle_start`) before its heap goes: the scavenger then sweeps that thread's
pages and the abandoned-page maps of its heaps. Half of those heaps are deleted by the owner
right after `mi_on_thread_idle_end`, racing the end of the sweep; the other half by a freer
thread while the owner is still parked, so the delete's detach has to get past the scavenger
holding the owner's theaps. A `-no-scavenger` ctest variant covers the inline sweep.
…ther heap tests (MSVC has no <stdatomic.h> in C mode)
…efore-init a pre-main hook on MSVC

MSVC builds the library as C++ and mangles global variables, so a debug hook the C tests link
against has to be declared in the `extern "C"` block in `internal.h` like the other hooks;
`mi_debug_abandoned_maps_allocated` was not, and `test-abandoned-lazy` failed to link.

`test-free-before-init` only had a pre-main hook for ELF (`.preinit_array`) and GCC/Clang
(constructor), so on MSVC it ran zero times and the test failed. Use a `.CRT$XCU` initializer
there.
@Jarred-Sumner
Jarred-Sumner force-pushed the robobun/9dfcc910/lazy-abandoned-bitmaps branch from 44e1ba4 to 1515c3c Compare September 3, 2026 05:50
@Jarred-Sumner
Jarred-Sumner merged commit b20b60d into oven-sh:bun-dev3-v2 Sep 3, 2026
14 of 15 checks passed
Jarred-Sumner added a commit to oven-sh/bun that referenced this pull request Sep 3, 2026
Bumps oven-sh/mimalloc from 942b834 to 6a64e1b (12 commits).

- Allocate the per-bin abandoned-page bitmaps on first abandon
(oven-sh/mimalloc#32)
- A heap being released abandons its pages unmapped
- Order the heap delete walk against concurrent frees
- Take the meta-data lock before the arena reserve lock in the fork
handlers (oven-sh/mimalloc#33)
- Tests for heap release on many threads
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants