Repository navigation
Port Bun.stringWidth to C++ with explicit SIMD - #31351
Conversation
Move the visible-width engine (ANSI escape handling, grapheme clustering, East Asian Width) from Rust (src/bun_core/string/immutable/visible.rs) to C++ (src/jsc/bindings/stringWidth.cpp) and register Bun.stringWidth directly in the BunObject lookup table. The ASCII fast paths now use explicit highway SIMD kernels (highway_visible_latin1_width, highway_count_printable_ascii16, highway_first_non_ascii8/16) instead of scalar loops that relied on LLVM's SLP autovectorizer, so throughput no longer depends on the linker's LTO pass pipeline. Escape-sequence scanning reuses the WTF::SIMD helpers in ANSIHelpers.h. The grapheme break algorithm and its uucode-derived tables move to C++ as well (stringWidthTables.h); sliceAnsi.cpp/wrapAnsi.cpp keep calling the same Bun__* symbols, now defined in C++. Remaining Rust callers (console.table column sizing, the markdown ANSI renderer) go through a thin FFI shim that preserves the bun.strings.visible module paths. The readline/util.inspect/ console.table builtins switch from $newZigFunction to $newCppFunction. Deleted: the Rust implementation and everything that only supported it (grapheme.rs, grapheme_tables.rs, the .zig porting references, the icu_hasBinaryProperty shim, the visible_fallback module).
|
Warning Review limit reached
Your plan includes 5 reviews of capacity. Refill in 46 minutes and 34 seconds. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more review capacity refills, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than trial, open-source, and free plans. In all cases, review capacity refills continuously over time. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (2)
WalkthroughReplaces Rust/Zig string-width and grapheme logic with a C++ implementation (Bun.stringWidth), adds Highway SIMD kernels and C wrappers, updates JS/JSC bindings and FFI, removes legacy Rust/Zig exports, updates baseline allowlists and table-generation script, and adds extensive tests. ChangesString Width Implementation Migration
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Comment |
|
Updated 12:40 AM PT - May 25th, 2026
✅ @robobun, your commit 03dfc8e6bb783b95dfce2bad6c1900cc850f9768 passed in 🧪 To try this PR locally: bunx bun-pr 31351That installs a local version of the PR into your bun-31351 --bun |
|
Found 4 issues this PR may fix:
🤖 Generated with Claude Code |
|
This PR may be a duplicate of:
🤖 Generated with Claude Code |
|
Re the duplicate-PR flag: #29543 is a different change — a rope-string fast path (walking 8-bit rope fibers without resolving them) written against the pre-Rust-port tree ( |
The new VisibleLatin1Width/CountPrintableAscii16/FirstNonAscii8/16 highway kernels are runtime-dispatched via hwy::SupportedTargets (same gate as the rest of highway_strings.cpp), so their per-target variants need entries in the baseline ISA allowlists. Symbols and feature ceilings taken from the verify-baseline-static reports for linux-x64(-musl), windows-x64 and linux-aarch64. Also reword the Latin-1 comment in jsFunctionBunStringWidth: some Latin-1 codepoints are East-Asian-Ambiguous; the path ignores ambiguousIsNarrow for parity with the previous implementation, not because of a spec property.
77609f9 to
bb18854
Compare
The aarch64 verify-baseline scan only flagged the N_SVE2_128 variants (the other SVE targets fold to identical code in the current toolchains), but list all four SVE namespaces like the rest of the highway_strings kernels so a different folding outcome doesn't trip the check.
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/js/builtins/ConsoleObject.ts (1)
168-175:⚠️ Potential issue | 🟠 Major | ⚡ Quick win
removeControlCharsis now ignored due to argument-shape mismatch.
internalGetStringWidthnow points tojsFunctionBunStringWidth, whose second parameter is an options object. This call still passes a boolean, soremoveControlChars=falseno longer changes behavior.Suggested fix
var getStringWidth = function getStringWidth(str, removeControlChars = true) { - return internalGetStringWidth(str, removeControlChars); + return internalGetStringWidth(str, { + // removeControlChars=true => do NOT count ANSI escapes + countAnsiEscapeCodes: !removeControlChars, + }); };🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/js/builtins/ConsoleObject.ts` around lines 168 - 175, getStringWidth currently passes a boolean as the second argument to internalGetStringWidth, but internalGetStringWidth is bound to jsFunctionBunStringWidth which expects an options object; this makes removeControlChars ignored. Fix by changing getStringWidth to accept either a boolean or an options object and call internalGetStringWidth(str, optionsObj) — e.g. if removeControlChars is a boolean, convert it to {removeControlChars: removeControlChars}, else pass the object through; update the parameter default and any callers accordingly to ensure the second argument is an options object matching jsFunctionBunStringWidth.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/jsc/bindings/stringWidth.cpp`:
- Around line 1172-1175: Validate inputs at the C ABI boundary in
Bun__graphemeBreak: check that cp1 and cp2 are within the Unicode scalar range
(<= 0x10FFFF and not out-of-range values) before calling into
StringWidth::graphemeBreak/graphemeBreakClass and check that state is non-null
before dereferencing; if any check fails, return a safe default (e.g., false or
the non-break behavior) rather than proceeding to table indexing. Update
Bun__graphemeBreak to perform these validations and only call
StringWidth::graphemeBreak when inputs are safe, or dispatch to a safe fallback
path to avoid out-of-bounds access.
---
Outside diff comments:
In `@src/js/builtins/ConsoleObject.ts`:
- Around line 168-175: getStringWidth currently passes a boolean as the second
argument to internalGetStringWidth, but internalGetStringWidth is bound to
jsFunctionBunStringWidth which expects an options object; this makes
removeControlChars ignored. Fix by changing getStringWidth to accept either a
boolean or an options object and call internalGetStringWidth(str, optionsObj) —
e.g. if removeControlChars is a boolean, convert it to {removeControlChars:
removeControlChars}, else pass the object through; update the parameter default
and any callers accordingly to ensure the second argument is an options object
matching jsFunctionBunStringWidth.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: ffa9fcb9-4a8e-456e-89c9-6f8c6af8e5e2
📒 Files selected for processing (24)
scripts/verify-baseline-static/allowlist-aarch64.txtscripts/verify-baseline-static/allowlist-x64-windows.txtscripts/verify-baseline-static/allowlist-x64.txtsrc/bun_core/string/immutable.rssrc/bun_core/string/immutable/grapheme.rssrc/bun_core/string/immutable/grapheme.zigsrc/bun_core/string/immutable/grapheme_tables.zigsrc/bun_core/string/immutable/visible.rssrc/bun_core/string/immutable/visible.zigsrc/bun_core/string/mod.rssrc/js/builtins/ConsoleObject.tssrc/js/internal/util/inspect.jssrc/js/node/readline.tssrc/jsc/bindings/BunObject+exports.hsrc/jsc/bindings/BunObject.cppsrc/jsc/bindings/highway_strings.cppsrc/jsc/bindings/stringWidth.cppsrc/jsc/bindings/stringWidth.hsrc/jsc/bindings/stringWidthTables.hsrc/jsc/bindings/workaround-missing-symbols.cppsrc/jsc/bun_string_jsc.rssrc/runtime/api/BunObject.rssrc/runtime/dispatch_js2native.rstest/js/bun/util/stringWidth.test.ts
💤 Files with no reviewable changes (9)
- src/bun_core/string/immutable/visible.zig
- src/bun_core/string/immutable/grapheme.zig
- src/bun_core/string/immutable/grapheme.rs
- src/jsc/bun_string_jsc.rs
- src/jsc/bindings/workaround-missing-symbols.cpp
- src/jsc/bindings/BunObject+exports.h
- src/runtime/dispatch_js2native.rs
- src/runtime/api/BunObject.rs
- src/bun_core/string/mod.rs
8f7366d to
d02e539
Compare
The grapheme class lookup indexes fixed tables, so validate the exported entry point instead of relying on callers to pass valid Unicode scalars; invalid input is treated as a cluster boundary.
Same reasoning as the SVE namespaces: the current toolchains fold these into the AVX3 variants so the scanner does not flag them, but list them like the other highway_strings kernels so a different folding outcome doesn't trip the check.
Read countAnsiEscapeCodes / ambiguousIsNarrow with getIfPropertyExistsPrototypePollutionMitigation, matching the lookup the previous implementation and the other Bun option parsers use (stops before Object.prototype), and add a regression test that polluting Object.prototype does not change results.
|
CI note: the red on build 57626 (sha 5ac4e7e) was a single test lane — |
There was a problem hiding this comment.
All my earlier inline findings are addressed and I didn't spot anything new, but this is a ~1.3k-line reimplementation of the grapheme/width engine plus new Highway SIMD kernels — worth a human pass on the port fidelity and the generated tables before merging.
Extended reasoning...
Overview
This PR replaces the Rust/Zig Bun.stringWidth implementation with a new ~1250-line C++ engine (stringWidth.cpp/.h + generated stringWidthTables.h), adds four Highway SIMD kernels to highway_strings.cpp, rewires the BunObject LUT and three JS builtins to the new host function, reduces visible.rs to a thin FFI shim, deletes the Rust/Zig grapheme modules, and updates the three verify-baseline allowlists. 24 files touched, net several thousand lines moved across languages.
Security risks
Low. stringWidth is not a security boundary. Inputs are JSC string views (length-bounded spans), the C ABI exports now guard against out-of-range codepoints and null state, and option-bag reads use the prototype-pollution-mitigated lookup (fixed in 5ac4e7e after my inline comment, with a regression test). No auth/crypto/permissions surface.
Level of scrutiny
High. This is a behavior-preserving port of a non-trivial Unicode algorithm (UAX #29 grapheme breaking incl. GB9c, East Asian Width, ZWJ/flag/keycap/VS handling, CSI/OSC parsing) into hand-written C++, plus brand-new SIMD code paths whose correctness depends on Highway lane semantics across six x64 and four aarch64 targets. The 123-test suite plus the sliceAnsi/wrapAnsi/console.table/readline suites give reasonable confidence, but the generated stringWidthTables.h and the constexpr 8 KiB grapheme decision table are not directly reviewable from the diff. This is well beyond the "simple/mechanical" bar for auto-approval.
Other factors
All four of my earlier inline comments (Latin-1 ambiguous-width comment wording, missing SVE/AVX10_2 allowlist entries, orphaned grapheme*.rs files, and the prototype-pollution option lookup) plus the CodeRabbit Bun__graphemeBreak boundary check have been addressed and the threads resolved. CI was green on 63/64 lanes with one unrelated S3 flake being retriggered. No human reviewer has weighed in yet; given the scope, one should.
|
CI status: the only red on build 57630 is Everything else is green: 63/65 GitHub checks, 273 Buildkite jobs passed — every build lane and every test lane (including the ASAN, Windows, musl and baseline ones); only the darwin-x64 steps downstream of that fetch are blocked. This PR touches no vendor/CMake/CI files, and the same tree built fine on that lane in 57626 (whose single red was the Cloudflare R2 outage in I've already used a retrigger for the 57626 flake and don't have Buildkite permissions to retry a job, so leaving this to a maintainer: a "Retry" on the Final state of build 57630 (finished 20:21 UTC): 279/287 jobs passed. The only job that executed and failed is So: nothing here implicates the PR. A maintainer retrying the failed |
|
@robobun Benchmarked this branch ( Results (largest size of each input)
There is a 1000× spread between the ASCII path (137 GB/s) and the CJK path (0.13 GB/s, ~65 cycles per character). 1. Regression: dense SGR is 25% slower than the current implementation (must fix)
The cause is in Fix: single-pass block classification (simdjson stage-1 style). Instead of restarting a scan at every escape, process each 32/64-byte block once:
One pass over each block, zero per-escape kernel calls. This should take dense-SGR well past the current implementation rather than behind it. At minimum, the regression on 2. CJK is ~65 cycles per character (15.9 ns/char)Every non-ASCII codepoint currently pays, in sequence:
Fix (a) — fuse everything into the existing stage-3 table. Fix (b) — add // Consumes units until it hits one that needs the scalar grapheme path.
// Returns the number of units consumed; *width accumulates the prefix's width.
size_t highway_visible_utf16_width(const uint16_t* input, size_t len, size_t* width);Per 16-unit vector, range-compare against a conservative allowlist of "always its own grapheme, fixed width" BMP blocks:
3. Drop the ICU dependency in
|
… text visibleLatin1WidthExcludeANSI previously restarted a dynamically-dispatched scan at every escape sequence (index-of-ESC, width of the visible run, scan for the CSI final byte), so dense SGR input like shell prompts paid the dispatch overhead every few bytes and ended up slower than the previous implementation. Replace it with highway_visible_latin1_width_exclude_ansi: each 16-64 byte chunk is classified once into printable / ESC / CSI-final / OSC-terminator bitmasks, escape regions are carved out of the printable mask with a few scalar bit operations per escape, and in-CSI / in-OSC state carries across chunk boundaries so sequences may straddle chunks. Chunks with no escapes reduce to one popcount; short inputs and partial tail chunks use a masked load. Escape semantics are unchanged. Covers the new kernel with chunk-boundary, straddling-terminator, malformed and unterminated escape tests plus a deterministic fuzz against a scalar reference, and allowlists the new per-target symbols for the baseline verifiers.
…emoji lookup - Carry the previous codepoint's grapheme break class across the UTF-16 width loop instead of re-deriving it for every pair, halving the trie walks on non-ASCII text. - Fast-path the common always-wide blocks (kana letters, CJK Unified Ideographs, Hangul syllables, fullwidth forms) in visibleCodepointWidth before the zero-width chain and the East Asian Width binary search. - Replace u_hasBinaryProperty(UCHAR_EMOJI) with a generated kEmojiPresentationRanges table in stringWidthTables.h (same binary search as the East Asian Width tables, regeneration snippet documented in the header), removing the out-of-line ICU call from isEmojiPresentation.
|
Benchmarks for the regression fix are in. Summary of what changed on the branch since the review:
|
| input | chars | canary (Rust) | this PR | speedup |
|---|---|---|---|---|
| ascii | 25,000 | 12.54 µs | 713.2 ns | 17.58× |
| hyperlink-bel | 430,000 | 329.20 µs | 56.03 µs | 5.88× |
| hyperlink-st | 440,000 | 333.99 µs | 58.99 µs | 5.66× |
| hyperlink-c1st | 430,000 | 332.33 µs | 57.13 µs | 5.82× |
| emoji | 60,000 | 331.00 µs | 193.78 µs | 1.71× |
| cjk | 35,000 | 1.22 ms | 328.23 µs | 3.72× |
| ansi+emoji | 150,000 | 727.42 µs | 630.15 µs | 1.15× |
| truecolor | 140,000 | 133.25 µs | 42.67 µs | 3.12× |
| short-sgr | 70,000 | 107.10 µs | 41.93 µs | 2.55× |
| dense-sgr | 205,000 | 322.50 µs | 110.46 µs | 2.92× |
No input is slower than canary at any size — the previous 0.72–0.92× on dense-sgr/short-sgr is now 1.9–2.9×, and the worst remaining ratio in the whole sweep is ansi+emoji at 1.12× (that input is dominated by the UTF-16 emoji path that 2a-full/2b will address).
Full sweep — every input at every size
| input | chars | canary (Rust) | this PR | speedup |
|---|---|---|---|---|
| ascii | 5 | 27.6 ns | 16.6 ns | 1.67× |
| ascii | 50 | 53.5 ns | 13.2 ns | 4.06× |
| ascii | 500 | 282.8 ns | 31.2 ns | 9.07× |
| ascii | 5,000 | 2.54 µs | 156.2 ns | 16.26× |
| ascii | 25,000 | 12.54 µs | 713.2 ns | 17.58× |
| short-sgr | 14 | 45.6 ns | 22.0 ns | 2.07× |
| short-sgr | 140 | 264.9 ns | 96.4 ns | 2.75× |
| short-sgr | 1,400 | 2.15 µs | 790.6 ns | 2.72× |
| short-sgr | 14,000 | 21.37 µs | 7.61 µs | 2.81× |
| short-sgr | 70,000 | 107.10 µs | 41.93 µs | 2.55× |
| truecolor | 28 | 51.5 ns | 36.1 ns | 1.43× |
| truecolor | 280 | 308.6 ns | 105.8 ns | 2.92× |
| truecolor | 2,800 | 2.71 µs | 785.2 ns | 3.45× |
| truecolor | 28,000 | 26.62 µs | 7.40 µs | 3.60× |
| truecolor | 140,000 | 133.25 µs | 42.67 µs | 3.12× |
| hyperlink-bel | 86 | 92.0 ns | 44.6 ns | 2.06× |
| hyperlink-bel | 860 | 701.9 ns | 144.0 ns | 4.88× |
| hyperlink-bel | 8,600 | 6.63 µs | 1.13 µs | 5.87× |
| hyperlink-bel | 86,000 | 65.97 µs | 11.30 µs | 5.84× |
| hyperlink-bel | 430,000 | 329.20 µs | 56.03 µs | 5.88× |
| hyperlink-st | 88 | 94.0 ns | 45.0 ns | 2.09× |
| hyperlink-st | 880 | 714.7 ns | 172.4 ns | 4.15× |
| hyperlink-st | 8,800 | 6.74 µs | 1.21 µs | 5.57× |
| hyperlink-st | 88,000 | 66.85 µs | 11.82 µs | 5.66× |
| hyperlink-st | 440,000 | 333.99 µs | 58.99 µs | 5.66× |
| hyperlink-c1st | 86 | 92.9 ns | 44.9 ns | 2.07× |
| hyperlink-c1st | 860 | 818.5 ns | 164.0 ns | 4.99× |
| hyperlink-c1st | 8,600 | 6.69 µs | 1.24 µs | 5.40× |
| hyperlink-c1st | 86,000 | 66.54 µs | 11.89 µs | 5.60× |
| hyperlink-c1st | 430,000 | 332.33 µs | 57.13 µs | 5.82× |
| dense-sgr | 41 | 90.6 ns | 48.2 ns | 1.88× |
| dense-sgr | 410 | 692.3 ns | 265.8 ns | 2.61× |
| dense-sgr | 4,100 | 6.48 µs | 2.28 µs | 2.84× |
| dense-sgr | 41,000 | 64.37 µs | 22.16 µs | 2.90× |
| dense-sgr | 205,000 | 322.50 µs | 110.46 µs | 2.92× |
| emoji | 12 | 97.1 ns | 69.8 ns | 1.39× |
| emoji | 120 | 716.3 ns | 440.3 ns | 1.63× |
| emoji | 1,200 | 6.69 µs | 3.94 µs | 1.70× |
| emoji | 12,000 | 66.53 µs | 41.34 µs | 1.61× |
| emoji | 60,000 | 331.00 µs | 193.78 µs | 1.71× |
| ansi+emoji | 30 | 175.1 ns | 155.8 ns | 1.12× |
| ansi+emoji | 300 | 1.53 µs | 1.30 µs | 1.18× |
| ansi+emoji | 3,000 | 14.50 µs | 12.67 µs | 1.14× |
| ansi+emoji | 30,000 | 144.77 µs | 126.76 µs | 1.14× |
| ansi+emoji | 150,000 | 727.42 µs | 630.15 µs | 1.15× |
| cjk | 7 | 262.7 ns | 78.1 ns | 3.36× |
| cjk | 70 | 2.47 µs | 674.4 ns | 3.66× |
| cjk | 700 | 24.54 µs | 6.53 µs | 3.76× |
| cjk | 7,000 | 244.06 µs | 65.47 µs | 3.73× |
| cjk | 35,000 | 1.22 ms | 328.23 µs | 3.72× |
Correctness: the benchmark's built-in bunStringWidth(str) !== npmStringWidth(str) check passes on every input it applies to, test/js/bun/util/stringWidth.test.ts passes (136 tests, including the new chunk-boundary suite), and the sliceAnsi/wrapAnsi/stripANSI/console.table/inspect.table/readline/markdown-renderer suites pass against the debug build.
|
@robobun Do the full fused stage-3 table (width+class+emoji in one trie walk) and 2b (highway_visible_utf16_width BMP kernel) are not in this push. |
|
Benchmarked this PR's This PR is the fastest binary on 32 of the 33 rows measured. Plain text
ANSI escapes
Single linesThe megabyte inputs above stress the scan loops, but the common caller is a terminal renderer measuring output one line at a time — syntax-highlighted code, diff lines, table rows, status lines, all 12–80 columns. Best of 3 runs:
The Latin-1 lines (plain/highlighted code, diffs, dim paths) are 1.4–3.6× faster than main even at this size — the fixed cost of entering the highway kernel doesn't eat the win. The rows containing a non-ASCII glyph ( Notes
|
Each codepoint now maps to a single packed byte (grapheme break class, width class incl. East Asian Ambiguous, and the Emoji property) through the existing 3-stage table, so classifying a codepoint is one trie walk instead of a branchy zero-width check, a binary search over the East Asian Width ranges, a separate grapheme-class walk and an emoji range lookup. GraphemeState and the UTF-16 width loop consume the packed value directly. The table is produced by the new scripts/generate-stringwidth-tables.mjs, which derives the width and emoji bits from the range tables kept in stringWidthTables.h and carries the grapheme classes over unchanged (verified by a full-range round trip at generation time plus compile-time spot checks). The exported C ABI now guards codepoints outside the Unicode scalar range before indexing the tables.
highway_visible_utf16_width consumes runs of code units that are always their own grapheme cluster with a fixed width — printable ASCII, most Latin-1/Latin-Extended/IPA, Greek and Cyrillic letters (narrow), and the main kana / CJK Unified + Extension A / Hangul-syllable / fullwidth blocks (wide) — in one vector pass per chunk, bailing to the scalar grapheme loop for anything else (surrogates, combining marks, ZWJ/VS, jamo, ESC). The last consumed unit seeds the grapheme state so combining marks and jamo after a run still join their cluster, and the path is skipped when ambiguous-width characters count as wide since Greek/Cyrillic are East-Asian-Ambiguous. Also annotates the vector-taking lambdas in the Latin-1 kernel with HWY_ATTR so they always compile for the enclosing kernel's target. The allowlisted ranges are pinned by tests that sweep every codepoint in them against the scalar classifier, plus cluster-joining checks at the bail points cross-checked with the previous implementation.
|
Both remaining items are now on the branch:
|
| input | chars | canary (Rust) | this PR | speedup |
|---|---|---|---|---|
| ascii | 25,000 | 12.53 µs | 534.0 ns | 23.46× |
| hyperlink-bel | 430,000 | 329.61 µs | 56.28 µs | 5.86× |
| hyperlink-st | 440,000 | 333.60 µs | 58.78 µs | 5.68× |
| hyperlink-c1st | 430,000 | 331.93 µs | 56.23 µs | 5.90× |
| emoji | 60,000 | 331.42 µs | 145.87 µs | 2.27× |
| cjk | 35,000 | 1.24 ms | 5.54 µs | 223.83× |
| ansi+emoji | 150,000 | 730.68 µs | 461.34 µs | 1.58× |
| truecolor | 140,000 | 133.48 µs | 42.45 µs | 3.14× |
| short-sgr | 70,000 | 107.06 µs | 41.79 µs | 2.56× |
| dense-sgr | 205,000 | 322.96 µs | 110.69 µs | 2.92× |
No input is slower than canary at any size. The headline change from the two new commits: cjk went from 3.7× to ~220× vs canary (1.24 ms → 5.5 µs for 35k chars, ~0.16 ns/char on this box — the whole string stays in the UTF-16 bulk kernel), emoji 1.7× → 2.3×, ansi+emoji 1.15× → 1.6× (those are dominated by non-BMP emoji, which always take the scalar grapheme path; the fused table is what speeds that up). The Latin-1/escape numbers are unchanged from the previous comment, as expected.
Full sweep — every input at every size
| input | chars | canary (Rust) | this PR | speedup |
|---|---|---|---|---|
| ascii | 5 | 28.2 ns | 20.1 ns | 1.40× |
| ascii | 50 | 54.5 ns | 12.9 ns | 4.22× |
| ascii | 500 | 285.1 ns | 22.1 ns | 12.91× |
| ascii | 5,000 | 2.54 µs | 115.7 ns | 21.95× |
| ascii | 25,000 | 12.53 µs | 534.0 ns | 23.46× |
| short-sgr | 14 | 45.8 ns | 42.7 ns | 1.07× |
| short-sgr | 140 | 265.2 ns | 111.3 ns | 2.38× |
| short-sgr | 1,400 | 2.15 µs | 791.7 ns | 2.72× |
| short-sgr | 14,000 | 21.40 µs | 7.61 µs | 2.81× |
| short-sgr | 70,000 | 107.06 µs | 41.79 µs | 2.56× |
| truecolor | 28 | 63.0 ns | 36.6 ns | 1.72× |
| truecolor | 280 | 342.5 ns | 105.9 ns | 3.23× |
| truecolor | 2,800 | 2.73 µs | 788.0 ns | 3.46× |
| truecolor | 28,000 | 26.59 µs | 7.39 µs | 3.60× |
| truecolor | 140,000 | 133.48 µs | 42.45 µs | 3.14× |
| hyperlink-bel | 86 | 92.9 ns | 38.8 ns | 2.39× |
| hyperlink-bel | 860 | 702.7 ns | 138.8 ns | 5.06× |
| hyperlink-bel | 8,600 | 6.62 µs | 1.13 µs | 5.86× |
| hyperlink-bel | 86,000 | 65.92 µs | 11.30 µs | 5.83× |
| hyperlink-bel | 430,000 | 329.61 µs | 56.28 µs | 5.86× |
| hyperlink-st | 88 | 97.1 ns | 40.1 ns | 2.42× |
| hyperlink-st | 880 | 714.3 ns | 170.5 ns | 4.19× |
| hyperlink-st | 8,800 | 6.72 µs | 1.19 µs | 5.65× |
| hyperlink-st | 88,000 | 66.79 µs | 11.79 µs | 5.66× |
| hyperlink-st | 440,000 | 333.60 µs | 58.78 µs | 5.68× |
| hyperlink-c1st | 86 | 93.4 ns | 39.2 ns | 2.39× |
| hyperlink-c1st | 860 | 814.1 ns | 140.1 ns | 5.81× |
| hyperlink-c1st | 8,600 | 6.71 µs | 1.12 µs | 5.99× |
| hyperlink-c1st | 86,000 | 66.52 µs | 11.30 µs | 5.89× |
| hyperlink-c1st | 430,000 | 331.93 µs | 56.23 µs | 5.90× |
| dense-sgr | 41 | 93.9 ns | 48.9 ns | 1.92× |
| dense-sgr | 410 | 727.4 ns | 255.1 ns | 2.85× |
| dense-sgr | 4,100 | 6.49 µs | 2.23 µs | 2.91× |
| dense-sgr | 41,000 | 64.51 µs | 22.20 µs | 2.91× |
| dense-sgr | 205,000 | 322.96 µs | 110.69 µs | 2.92× |
| emoji | 12 | 98.0 ns | 48.7 ns | 2.01× |
| emoji | 120 | 716.5 ns | 319.0 ns | 2.25× |
| emoji | 1,200 | 6.72 µs | 2.95 µs | 2.28× |
| emoji | 12,000 | 66.27 µs | 29.16 µs | 2.27× |
| emoji | 60,000 | 331.42 µs | 145.87 µs | 2.27× |
| ansi+emoji | 30 | 175.9 ns | 114.5 ns | 1.54× |
| ansi+emoji | 300 | 1.51 µs | 955.8 ns | 1.58× |
| ansi+emoji | 3,000 | 14.62 µs | 9.34 µs | 1.57× |
| ansi+emoji | 30,000 | 145.87 µs | 92.09 µs | 1.58× |
| ansi+emoji | 150,000 | 730.68 µs | 461.34 µs | 1.58× |
| cjk | 7 | 262.1 ns | 36.4 ns | 7.21× |
| cjk | 70 | 2.46 µs | 48.4 ns | 50.86× |
| cjk | 700 | 24.39 µs | 190.3 ns | 128.19× |
| cjk | 7,000 | 243.55 µs | 1.16 µs | 209.96× |
| cjk | 35,000 | 1.24 ms | 5.54 µs | 223.83× |
Correctness: the benchmark's built-in npm string-width mismatch check passes, stringWidth.test.ts is at 143 tests (the new range-sweep and cluster-boundary tests included), and the sliceAnsi / wrapAnsi / stripANSI / console.table / Bun.inspect.table / readline / markdown-renderer suites pass against the debug (ASAN) build.
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/jsc/bindings/stringWidth.cpp (1)
915-933:⚠️ Potential issue | 🟠 Major | ⚡ Quick winKeep grapheme state in sync after bare ESC fallback.
When
excludeAnsiColorsis enabled andESCis followed by a non-sequence byte, this path adds width directly and skipsgraphemeState/prevClassupdates. That can mis-cluster the next non-ASCII codepoint (e.g. emoji modifier/ZWJ) against an earlier visible codepoint.💡 Suggested fix
if (saw1b) { if (cp == '[') { sawCsi = true; continue; } if (cp == ']') { sawOsc = true; continue; } if (cp == 0x1b) { // Another ESC — this one starts a new potential // sequence (ESC itself is zero-width anyway). continue; } - // ESC followed by an ordinary char: the ESC is dropped, - // the char is counted directly. - len += visibleCodepointWidth(cp, ambiguousAsWide); + // ESC followed by an ordinary char: the ESC is dropped. + // Process the char through the normal grapheme path so + // `prevClass`/cluster state stay consistent. saw1b = false; - continue; } if (!excludeAnsiColors || cp != 0x1b) { const uint8_t packed = fusedClassify(cp); const GraphemeBreakClass cpClass = graphemeBreakClassFromFused(packed);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/jsc/bindings/stringWidth.cpp` around lines 915 - 933, When handling a bare ESC fallback in the saw1b branch you add the visible width but forget to update grapheme clustering state; after the len += visibleCodepointWidth(cp, ambiguousAsWide); line you must update graphemeState and prevClass the same way the normal visible-codepoint path does (use the existing helper(s) used elsewhere in this file to compute the grapheme break class for cp and to advance graphemeState), so add calls to compute the grapheme class for cp (e.g., classify/graphemeBreak function used in this file) and assign prevClass and then update graphemeState accordingly to keep clusters in sync with symbols saw1b, visibleCodepointWidth, graphemeState, and prevClass.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@src/jsc/bindings/stringWidth.cpp`:
- Around line 915-933: When handling a bare ESC fallback in the saw1b branch you
add the visible width but forget to update grapheme clustering state; after the
len += visibleCodepointWidth(cp, ambiguousAsWide); line you must update
graphemeState and prevClass the same way the normal visible-codepoint path does
(use the existing helper(s) used elsewhere in this file to compute the grapheme
break class for cp and to advance graphemeState), so add calls to compute the
grapheme class for cp (e.g., classify/graphemeBreak function used in this file)
and assign prevClass and then update graphemeState accordingly to keep clusters
in sync with symbols saw1b, visibleCodepointWidth, graphemeState, and prevClass.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 18dcab07-b892-4501-9c32-08c17d907cf6
📒 Files selected for processing (8)
scripts/generate-stringwidth-tables.mjsscripts/verify-baseline-static/allowlist-aarch64.txtscripts/verify-baseline-static/allowlist-x64-windows.txtscripts/verify-baseline-static/allowlist-x64.txtsrc/jsc/bindings/highway_strings.cppsrc/jsc/bindings/stringWidth.cppsrc/jsc/bindings/stringWidthTables.htest/js/bun/util/stringWidth.test.ts
…y field The decision-table key packs the state into bits 0-2; guard the state count the same way the class count already guards its 5-bit field.
|
Addressed the two bot findings on the latest push:
|
The top-of-file paragraph still described stage3 as bare GraphemeBreakClass ordinals; it now packs class, width and the Emoji property, as the banner above the tables and stringWidth.cpp document.
Adds support for compiling and linking Bun for macOS (arm64 **and** x64)
on a Linux host, and switches CI over to it: the darwin build lanes now
run on the Linux fleet and **the macOS binaries that ship from the
release step are cross-compiled** — the mac fleet only runs tests.
Requested in Slack: get cross compilation for macOS working locally
first, then in CI.
> [!NOTE]
> `WEBKIT_VERSION` now points at a regular `autobuild-<sha>` release
from oven-sh/WebKit main (the macOS-cross + Windows-cross WebKit work
has merged there). The macOS `-lto` prebuilts are ThinLTO bitcode; the
Linux `-lto` prebuilts are full-LTO bitcode — matching how each
platform's final link runs (see the LTO section below).
### How it works
Same model as the existing Android/FreeBSD cross targets — host clang +
`--target`/sysroot — extended to Mach-O:
- **SDK**: `resolveMacosSdkPath()` (new `scripts/build/macos-sdk.ts`)
finds a macOS SDK via `--macos-sdk` / `$MACOS_SDK_PATH`, a well-known
install (`/opt/MacOSX*.sdk`, osxcross tree), or fetches the pinned
`MacOSX26.5.sdk` into the build cache at configure time via `xmac`
(vendored `scripts/build/xmac.mjs`), which downloads the Command Line
Tools package straight from Apple's public software-update CDN and
extracts the SDK (~59 MB, same model as xwin uses for the Windows SDK —
nothing is redistributed, Apple's license notice is printed before
`--accept-license`). Deployment target defaults to the existing 13.0
floor; the SDK version only affects `.tbd` stub coverage.
- **Compile**: `--target=arm64|x86_64-apple-macosx -isysroot <SDK>
-mmacosx-version-min=13.0`, and C++ uses the SDK's libc++ headers
(`-nostdinc++ -isystem <SDK>/usr/include/c++/v1`) so out-of-line libc++
symbols match the `libc++.tbd` actually linked (host libc++ 21 headers
reference `std::__hash_memory`, which Apple's libc++ doesn't export).
- **Link**: lld's Mach-O port (`ld64.lld`, ships with the LLVM 21
toolchain we already require) driven by clang with
`-mlinker-version=705` so the driver emits modern `-platform_version`
args. `-Wl,-ld_new` is native-only (ld64.lld doesn't know it).
- **Post-link**: `llvm-strip` + LLVM `dsymutil` (both handle Mach-O on
Linux) instead of GNU strip / native-only dsymutil.
- **Rust**: `aarch64/x86_64-apple-darwin` are Tier 2 targets — the
staticlib cross-compiles on Linux with prebuilt std; `SDKROOT` /
`CFLAGS_<triple>` / `MACOSX_DEPLOYMENT_TARGET` are forwarded for any
`cc`-crate build script.
- **Gaps filled**:
- x64 only: vendored compiler-rt `cpu_model/x86.c` (Apache-2.0 WITH
LLVM-exception) provides `___cpu_model` / `___cpu_indicator_init` for
`__builtin_cpu_supports` — on native builds these come from Apple's
static `libclang_rt.osx.a`, which the Linux toolchain doesn't ship.
Registered in `workarounds.ts` so it self-obsoletes if the SDK ever
exports them.
- x64 only: `-Wl,-sectalign,__BUN,__bun,0x1000` — ld64.lld otherwise
16K-aligns the placeholder section's file offset inside a 4K-aligned
segment and produces `filesize > vmsize` (invalid Mach-O that
llvm-strip/dsymutil reject). The placeholder only needs 8-byte alignment
and `bun build --compile` rewrites the segment at 16K anyway.
- `workaround-missing-symbols.cpp`: `std::__libcpp_verbose_abort` now
matches the header's exception spec for libc++ ≤ 19 (Apple SDK headers
declare it without `noexcept`).
- ASAN/UBSan are forced off for darwin cross builds (no darwin sanitizer
runtimes in the Linux toolchain), mirroring the Android/FreeBSD
behavior. Release CI cross builds get **ThinLTO with cross-language
importing**: C++/JSC link against the ThinLTO macOS `-lto` WebKit
prebuilts, Rust participates via `-Clinker-plugin-lto`, the Mach-O link
runs through rustc's `ld64.lld` (rustc's LLVM is newer than clang's, and
bitcode is only forward-compatible), and `-Wl,-object_path_lto` keeps
the dSYM usable under LTO. See the LTO section below for why darwin is
thin and linux stays on full LTO.
### Usage (local)
```sh
bun scripts/build.ts --profile=release --os=darwin --arch=aarch64 --build-dir=build/darwin-arm64
bun scripts/build.ts --profile=release --os=darwin --arch=x64 --build-dir=build/darwin-x64
# optional: --macos-sdk=/path/to/MacOSX26.5.sdk or MACOS_SDK_PATH=...
```
Debug profiles work too (`--profile=debug --os=darwin ...`); ASAN is
automatically disabled for the cross target. Native Linux/macOS builds
are unaffected — the native Linux debug graph reconfigures byte-for-byte
identical with this change.
### CI
The darwin build lanes now run on the Linux fleet and produce the
artifacts that ship — the native mac build lanes are deleted and the mac
fleet only runs tests:
- `darwin-{aarch64,x64}` `build-{cpp,rust,bun}` run on Linux agents
(cpp/link on the amazonlinux Docker fleet, rust on the shared Linux rust
box), driven by the `crossCompile` flag + explicit `--os`/`--arch` build
args.
- Step keys, labels, and artifact names are the canonical ones
(`bun-darwin-{aarch64,x64}[-profile].zip`) — no `-cross` suffix — so the
release step, the darwin test steps, and `upload-release.sh` consume
them with no further changes. The release ships the Linux-built darwin
artifacts and binary-size tracking tracks them.
- The darwin test steps download these artifacts and run the full macOS
test suite on real macOS agents (macOS 14 + 26, Intel + Apple Silicon).
- No image changes required: the SDK is fetched per build from Apple's
CDN (~59 MB, cached like the WebKit prebuilt). `bootstrap.sh` now
preinstalls the apple-darwin rust targets on Linux so the rust step
stops needing `rustup target add` at build time once images are next
refreshed.
### Verification
Both targets were built end-to-end on a Linux x64 host (clang 21.1.8 +
ld64.lld + the MacOSX15.5 SDK those early builds used; numbers below are
from the pre-LTO local builds — the CI darwin lanes now build with full
+ cross-language LTO against the `-lto` WebKit prebuilts, and the SDK
pin has since moved to 26.5 fetched from Apple's CDN):
| | darwin-arm64 | darwin-x64 |
|---|---|---|
| stripped `bun` | 57.5 MB Mach-O arm64 | 61.8 MB Mach-O x86_64 |
| `bun-profile` + dSYM + linker map | ✅ | ✅ |
| `LC_BUILD_VERSION` | minos 13.0, sdk 15.5 | minos 13.0, sdk 15.5 |
| ad-hoc code signature + entitlements.plist (XML + DER blobs) | ✅ | ✅
(the shipping config has since dropped the x64 signature — see below) |
| `LC_MAIN.stacksize` | 0x1200000 (18 MB) | 0x1200000 (18 MB) |
| linked dylibs | libicucore, libresolv, libc++, libSystem | same |
| dyld export trie | all 554 symbols from `src/symbols.txt` incl. 145
`napi_*` | same |
| segment layout | valid | valid (after `-sectalign`) |
`ninja check` (smoke test) is skipped for cross targets, same as
Android/FreeBSD — the cross-built binaries now run the **full test suite
on real macOS hardware** (Intel + Apple Silicon test lanes) and have
been benchmarked on an M-series MacBook — see the Benchmarks section
below.
### Benchmarks (M-series MacBook, arm64)
Two questions, two comparisons. **(1) Is the artifact this PR ships
faster than the last released Zig build?** — `bun-darwin-aarch64` from
this branch (ThinLTO + cross-language importing + `--lto-O3` +
`--icf=safe` + the ThinLTO WebKit prebuilt) vs bun v1.3.14. **(2) What
does LTO itself buy?** — the same artifact vs a native non-LTO release
build at a nearby commit. Microbenchmarks via mitata / oha (3
interleaved rounds for servers); real-app suite via
`oven-sh/bun-perf-tester` (hyperfine + bombardier + GNU time).
#### vs bun v1.3.14 (the binary this replaces): wins 24, ties 7, loses 5
| | LTO cross build | v1.3.14 (zig) | Δ |
|---|---|---|---|
| HTTP plaintext, oha -c64 (req/s) | **150,757** | 149,358 | +0.9%,
ahead in 3/3 interleaved rounds |
| HTTP React SSR (req/s) | **96,780** | 94,854 | +2.0% |
| `next start` (req/s / max RSS) | **13,594 / 391 MB** | 12,887 / 433 MB
| +5.5% / −10% |
| `next build` | **4.99 s** | 5.39 s | +7.5% |
| vite build / `tsc -b` / eslint | **794 / 434 / 326 ms** | 818 / 448 /
371 ms | +3.0% / +3.1% / +12% |
| elysia build / test suite | **1.50 s / 1.61 s** | 1.57 s / 1.66 s |
+4.2% / +3.0% |
| `bun build` three.js ×10 --minify --sourcemap | **77.5 ms** | 83.1 ms
| +6.7% |
| JS↔native binding micros — `Headers.set`, `Response.json`,
`structuredClone`, `URLSearchParams`, `FormData.append`, `Bun.semver`,
`createHash`, promise resolution, small base64 | **wins all** | | +6% to
+37% |
| stripped binary size | **56.0 MB** | 60.2 MB | −7.0% |
| WebSocket pub/sub, startup, `bun install`, `Bun.serve` req/s under
bombardier, gzip, sqlite | | | ties |
| `TextEncoder.encode` ASCII / `escapeHTML` no-escape / `stringWidth`
ASCII / `formData()` round-trip | | **2× / 7.7× / 4.4× / 1.5× faster** |
pre-existing Zig→Rust port gaps — see notes |
#### vs the native non-LTO release build (what LTO itself buys)
| workload | LTO vs native non-LTO | notes |
|---|---|---|
| JS ↔ native binding micros | **+5–17%** (`Headers.set` +21%) | exactly
where cross-language inlining should help |
| `next build` / vite / tsc / eslint / prettier / elysia tests |
**+1–4.5%** | sweeps every "run a real JS app/tool" workload |
| `Bun.serve` max RSS | **lower** | single sample; directionally
consistent |
| stripped binary size | **−2.2 MB** | on top of the −2.8 MB from ICF |
| HTTP plaintext throughput | −6% on the first LTO build → recovered |
see the HTTP note below |
| `bun install` (real registry) | −6% | |
| `Bun.stringWidth` ASCII SIMD path | −2× | SLP vectorizer — see below;
superseded by #31351 |
**The `stringWidth` regression is root-caused** (by replaying the exact
bitcode through the exact linker — the repro is bit-identical to the
shipped binary): `ld64.lld`'s LTO pipeline never runs the **SLP
vectorizer** at any `--lto-O` level. `lld/ELF/LTO.cpp` sets
`PTO.SLPVectorization = OptLevel > 1`; `lld/MachO/LTO.cpp` is missing
that line, so it inherits the default of *off* (#31343 tracks
upstreaming the one-liner). `printable_mask_latin1` (the `stringWidth`
ASCII kernel) is a Zig port whose `u8x16` "vectors" are scalar `for i in
0..16` stand-ins that depend 100% on SLP autovectorization — under the
Mach-O LTO it comes out fully scalar (131 instructions / 16 bytes, 0
NEON ops) instead of partially vectorized (75 instructions, 28 NEON
ops). Loop-vectorized and explicit-intrinsic code (simdutf, highway,
zlib, WebKit) is unaffected, which is why everything else got faster.
**#31351 supersedes this**: it rewrites `stringWidth` in C++ with
explicit highway SIMD, which doesn't depend on any autovectorizer — that
build measures 1.5–4.7× *faster* than the v1.3.14 Zig build on the same
inputs.
**The HTTP plaintext deficit is recovered.** The first LTO build trailed
both the native build (−6% under oha) and v1.3.14 on plaintext
throughput; `--lto-O3` alone did not move it (±0.1%). Marking the
`uws_*` C-ABI wrappers and the 1-line usockets accessors `always_inline`
did: the current build is ahead of v1.3.14 in all 3 interleaved rounds
(it previously trailed in 3/3) and ahead on React SSR, `next start`, and
the bundler.
The remaining v1.3.14 wins (`escapeHTML` no-escape 7.7×, `TextEncoder`
ASCII 1.4–2×, `formData()` round-trip 1.5×) predate this PR, are
unaffected by it, and share the same root cause as the `stringWidth`
row: the original `@Vector` SIMD code was ported as scalar stand-ins
behind `ENABLE_SIMD = false` / `ScalarVec` in
`bun_core/string/immutable.rs`. They reproduce identically on the native
non-LTO build.
### LTO: ThinLTO on darwin, full LTO on linux
The darwin release links use **ThinLTO** (`-flto=thin
-fno-split-lto-unit` + `-mllvm -whole-program-visibility` +
`--lto-O3/--lto-CGO3` on arm64): every module carries a summary, the LTO
backends run in parallel, and the link drops from **860 s to 81 s** on a
64-core box. ThinLTO also makes **Rust↔C++ cross-language inlining real
for the first time** — under full LTO the C++ lands in one regular-LTO
partition and the Rust CGUs in ThinLTO partitions, and the two never
exchange function bodies, so every one of the ~15,700 calls across the
`extern "C"` boundary (e.g. 940 calls to `JSC__JSGlobalObject__vm`,
which is a single load) stayed an out-of-line call. With ThinLTO +
`__attribute__((always_inline))` on the hottest tiny boundary functions
(the JSC binding getters, the `uws_*` C-ABI wrappers, the 1-line
usockets accessors), **4,663 cross-language calls are eliminated** for
+131 KB.
**Linux stays on full LTO.** The LLVM 22 ThinLTO backend pipeline
(rust-lld) miscompiles JavaScriptCore on linux — a JIT-tier correctness
bug (`require("axobject-query")` fails once the surrounding functions
tier up to the DFG) plus several bundler-test hangs, on x64 and aarch64,
glibc and musl, absent on darwin. Bisected with a relink-only harness:
fixed at `--lto-O0`, broken at O1/O2/O3, broken with cross-module
importing disabled, with WPD disabled, with split LTO units disabled,
and with TBAA disabled; ICF was separately ruled out. The miscompiling
pass is in the core O1 simplification pipeline; isolating it needs a
per-module bisection and is a follow-up. Linux therefore keeps the exact
full-LTO configuration every previous green build used (the `-flto=full`
entry in `flags.ts` documents the repro).
### Differences vs the previously mac-built binaries (these now ship —
validated by the full test suite on real macOS hardware and the
benchmarks above)
- `-Wl,-no_compact_unwind` is accepted but not implemented by ld64.lld
21 → the shipped binary now carries a `__unwind_info` section the
mac-built one didn't. (`-Wl,-stack_size` is also unimplemented, but the
post-link fixup below patches `LC_MAIN.stacksize` to the same 18 MB the
native link got.)
- The darwin link now enables `-faddrsig` + `-Wl,--icf=safe`
(identical-code folding, same as the Linux release link) and, for
release CI builds, full + cross-language LTO. Apple's ld has no ICF and
the previous native release builds didn't do LTO, so the shipped binary
comes out smaller than before (~5% from ICF alone in the pre-LTO local
builds, another −2.2 MB from LTO — see Benchmarks).
- arm64: the ad-hoc signature is generated by `macho-postlink` rather
than Apple's `codesign`: same CodeDirectory layout, entitlements, and
DER encoding (byte-identical to the previously shipped binary's), but no
Developer ID / hardened runtime — those need Apple tooling and are
applied by the release pipeline, not the build. x64 ships unsigned,
exactly like the previously mac-built x64 binary (Apple's toolchain only
auto-signs arm64).
- The x64 `__BUN` placeholder section is 4 KB-aligned instead of 16 KB
(see above; rewritten on `bun build --compile` anyway).
- Binaries are linked against SDK 26.5 stubs fetched from Apple's CDN
(previously whatever Xcode shipped on the mac builders); deployment
target is unchanged at 13.0.
### Post-link fixups (`scripts/build/shims/macho-postlink.c`)
A standalone host tool (single C11 file, libc only — `cc -O2
macho-postlink.c` works on any POSIX host including a Mac) appended to
the link **and** strip rule commands for darwin cross targets (`… -o
$out && macho-postlink $out --stack-size=0x1200000
--entitlements=entitlements.plist`, the `--entitlements` part on arm64
only):
- **Stack size** — ld64.lld parses `-stack_size` but doesn't implement
it, so the tool sets `LC_MAIN.stacksize` in place. Registered in
`workarounds.ts` with a probe that fails configure once ld64.lld starts
honoring the flag we already pass.
- **Code signature + entitlements (arm64 only)** — the header edit
invalidates the linker's ad-hoc signature (page 0 is hashed into the
CodeDirectory), so the tool regenerates it, embedding
`entitlements.plist` as both the XML blob and the DER blob (AMFI
requires the DER form for minos ≥ 12). `-Wl,-adhoc_codesign` is forced
on the arm64 link so `LC_CODE_SIGNATURE` always exists for the tool to
replace, and the tool errors if asked to embed entitlements into an
unsigned binary — arm64 can't silently ship unsigned. x64 ships unsigned
like the native x64 build always did (x64 macOS runs unsigned binaries
fine; the CodeDirectory would cost ~0.8% of the binary) and only gets
the stack-size patch. Debug builds sign with `entitlements.debug.plist`
(adds `get-task-allow`/`cs.debugger`), matching `scripts/trace.sh`.
llvm-strip regenerates a bare signature on its output, so the stripped
arm64 `bun` gets its own fixup pass.
- Verified against the official `bun-v1.3.14` darwin release binaries
(arm64 + x64) and the actual cross-built artifacts with an independent
parser that recomputes every code-page hash and special-slot hash; the
DER entitlements blob comes out byte-for-byte identical to the one
Apple's `codesign` embedded in the shipped binary for the same plist.
The cross-built binaries pass the full test suite on real macOS hardware
(Intel + Apple Silicon test lanes).
### features.json
The cross lanes previously uploaded a `{ crossTarget }` stub because the
binary can't run on the build host. Every field is a build-time
constant, so it's now generated host-side: the feature list is parsed
out of the `define_features!` invocation in `src/analytics/lib.rs`
(`scripts/build/features-json.ts`), and version/is_canary/revision come
from the build config. A test runs the debug build's
`crash_handler.getFeatureData()` and requires the parsed list to match
exactly, so the parser can't silently drift from the binary (the list
maps crash-report bit indices to names — order matters).
### Tests
`test/internal/macos-cross-config.test.ts` pins the cross-compile
configure logic: `--os=darwin` on a non-darwin host resolves to
`arm64|x86_64-apple-macosx` with ld64.lld/llvm-strip, sanitizers forced
off (LTO stays on for release CI cross builds), SDK path resolution
(`--macos-sdk` / `MACOS_SDK_PATH` / cache fallback, and the cache never
serving a stale SDK version), the cross compile/link flag set
(`-isysroot`, SDK libc++, no `-Wl,-ld_new`, x64 `-sectalign`),
apple-darwin rust triples, macOS WebKit prebuilt naming, and that native
Linux configs resolve exactly as before. Also: the post-link command on
cross links only (stack size on both arches, per-profile entitlements
plist on arm64), `-adhoc_codesign` on arm64 but not x64, and the
host-side features.json parser pinned against the real binary's
`getFeatureData()`.
### Follow-ups (tracked, none blocking this PR)
- [x] Repoint `WEBKIT_VERSION` at a regular `autobuild-<sha>` tag —
done; oven-sh/WebKit#238 merged and the pin now tracks main.
- [ ] Migrate the Linux release links to ThinLTO once the LLVM 22
ThinLTO-backend miscompile of JSC is isolated (repro + bisection harness
documented in the `-flto=full` comment in `flags.ts`); that also unlocks
Rust↔C++ cross-language inlining on Linux and drops the 14-minute serial
link.
- [x] #31342 — the darwin arm64 LTO link now passes
`--lto-O3`/`--lto-CGO3` (#31348 applied onto this branch; the bare `-O2`
it replaces was silently dropped by the driver). x64 keeps lld's default
`--lto-O2/CGO2` — O3 codegen costs +3.1 MB there vs +0.3 MB on arm64.
The HTTP plaintext A/B has since been re-run: `--lto-O3` alone did not
move it; `always_inline` on the uws_* boundary wrappers recovered it
(see Benchmarks).
- [ ] #31343 — `ld64.lld`'s LTO pipeline never runs the SLP vectorizer
(`lld/MachO/LTO.cpp` is missing the `PTO.SLPVectorization` line the ELF
port has); upstream the one-liner. Root cause of the `Bun.stringWidth`
−2×.
- [ ] #31351 — `Bun.stringWidth` → C++ with explicit highway SIMD,
deleting the Rust scalar-stand-in implementation. Measures 1.5–4.7×
faster than the 1.3.14 Zig build (and 1.4–3.8× faster than main).
- [ ] #31344 — Windows ASAN bun-webkit prebuilt (the cross-compile
plumbing is fully wired; needs a matrix entry and a consumer).
- [ ] #31345 — re-add `bun-webkit-windows-arm64-lto` once LLVM's
CodeView emitter maps ARM64 NEON register tuples.
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Jarred Sumner <jarred@jarredsumner.com>
* oven/main (20 new commits): webcore: free Blob's owned content type on drop (oven-sh#31358) Support cross-compiling macOS binaries from Linux (oven-sh#31303) test: forward keep-alive requests in proxy.test.ts's mock proxy (oven-sh#31352) Port Bun.stringWidth to C++ with explicit SIMD (oven-sh#31351) Fix quadratic hang reporting duplicate-binding parse errors in the transpiler (oven-sh#31341) shell: don't abort when a glob's directory prefix doesn't exist (oven-sh#31367) Error instead of crashing on deeply nested statements in the transpiler (oven-sh#31333) Fix JSX transform panic when a bare `key` prop precedes `key` with a value (oven-sh#31350) Cap ANSI markdown indentation so deeply nested lists render in linear time (oven-sh#31366) css: bound selector-list expansion when compiling nesting for older targets (oven-sh#31277) node:http2: reassemble HEADERS+CONTINUATION before HPACK decoding (oven-sh#31323) Fix `await using` expression printing `using` as `await` (oven-sh#31324) Parenthesize `async` when it starts a for-of loop initializer (oven-sh#31326) Print Infinity and negative numeric property keys as computed properties (oven-sh#31328) css: keep required grouping parens in @container conditions when minifying (oven-sh#31330) Fix panic on anonymous export default class with an auto-accessor field (oven-sh#31331) node:http2: send GOAWAY frames on stream 0 (oven-sh#31353) parser: fix Scope mismatch while visiting panic from decorators on dropped class members (oven-sh#31340) webcrypto: reject oversized BufferSource inputs instead of aborting (oven-sh#31356) Error instead of crashing on deeply nested TypeScript types in the transpiler (oven-sh#31361) Resolved conflicts: - scripts/build.ts: kept both OHOS and macOS-cross argv entries - scripts/build/config.ts: kept both OHOS and macOS-cross config fields - scripts/build/deps/webkit.ts: kept OHOS fno-pic exclusion, adopted upstream -flto=thin
| alignas(8) uint8_t maskBytes[8]; | ||
| const auto maskToBits = [&](auto mask) HWY_ATTR -> uint64_t { | ||
| std::memset(maskBytes, 0, sizeof(maskBytes)); | ||
| hn::StoreMaskBits(d, mask, maskBytes); |
There was a problem hiding this comment.
Nice work and speedups :) Here, we might get better codegen from BitsFromMask?
| } | ||
|
|
||
| for (; i < len; ++i) { | ||
| if (input[i] > 0x7F) { |
There was a problem hiding this comment.
For Ice lake / Zen4 (if lengths are often not multiples of 64), it might help to have one final iteration that uses LoadN to load remainders, and otherwise uses the same vector logic.
## What
Moves the `Bun.stringWidth` implementation from Rust to C++ with
explicit SIMD, as requested.
- **New `src/jsc/bindings/stringWidth.cpp` / `stringWidth.h`** — the
full visible-width engine: codepoint classification (zero-width, East
Asian `W`/`F`, ambiguous), grapheme cluster break (UAX #29 incl. GB9c,
the uucode-derived 3-stage tables now live in the generated
`stringWidthTables.h`), the grapheme-cluster width accumulator (ZWJ
emoji, regional-indicator flags, keycaps, skin tones, VS15/VS16),
Latin-1 / UTF-8 / UTF-16 width functions with CSI/OSC handling,
`utf8IndexAtWidth`, and the `Bun.stringWidth` host function (same
`countAnsiEscapeCodes` / `ambiguousIsNarrow` options, same defaults,
same coercion semantics).
- **Explicit SIMD via highway** — five kernels in `highway_strings.cpp`
(`highway_visible_latin1_width`,
`highway_visible_latin1_width_exclude_ansi`,
`highway_count_printable_ascii16`, `highway_first_non_ascii8/16`) using
the existing dynamic-dispatch pattern. The default (ANSI-excluding)
Latin-1 path is a single-pass kernel: each chunk is classified once into
printable/ESC/CSI-final/OSC-terminator bitmasks and escape regions are
carved out with a few bit ops per escape, with in-CSI/in-OSC state
carried across chunks. The UTF-16 path's escape scanning still reuses
the `WTF::SIMD` helpers in `ANSIHelpers.h` shared with
stripANSI/wrapAnsi/sliceAnsi.
- **Registration** — `stringWidth` now points at
`jsFunctionBunStringWidth` in the BunObject LUT; the
`BunObject_callback_stringWidth` Rust callback and `macro(stringWidth)`
are gone. The readline / `util.inspect` / console.table builtins switch
from `$newZigFunction("string.zig", "String.jsGetStringWidth")` to
`$newCppFunction("stringWidth.cpp", "jsFunctionBunStringWidth")`.
- **Rust deletions** — `visible.rs` (1.6k lines) is now a ~70-line FFI
shim that preserves the
`bun.strings.visible::width::exclude_ansi_colors::{latin1,utf8,utf16,utf8_index_at_width}`
paths for the remaining Rust callers (console.table column sizing in
`ConsoleObject.rs`, the markdown ANSI renderer). `grapheme.rs`,
`grapheme_tables.rs`, the `.zig` porting references, the dead
`visible_fallback` module, `index_of_char16_usize`, and the
`icu_hasBinaryProperty` shim (all only existed to support the Rust width
code) are deleted. `String::visible_width` (include-ANSI variant) had no
remaining callers and is removed;
`String::visible_width_exclude_ansi_colors` keeps working through the
shim.
- **`sliceAnsi.cpp` / `wrapAnsi.cpp` are untouched** — they keep calling
`Bun__codepointWidth`, `Bun__graphemeBreak`, `Bun__isEmojiPresentation`,
`Bun__visibleWidthExcludeANSI_{latin1,utf16}`, which are now defined in
`stringWidth.cpp` instead of Rust.
## Why
The Rust implementation's `u8x16`/`u16x8` "vectors" are scalar `for`
loops that rely on LLVM's SLP autovectorizer to become SIMD — ~2.3×
slower than real vector code on the ASCII path, and much worse under
macOS LTO where `ld64.lld`'s pipeline doesn't run the SLP pass (see
#31348 / #31349 for the build-side workarounds). Explicit highway
kernels remove the dependency on the optimizer entirely and let x64 use
256/512-bit vectors via dynamic dispatch.
## SIMD choice: highway vs WTF::SIMD
Microbenchmark of the "count visible Latin-1 bytes" kernel (x64,
AVX-512-capable host, clang 21 `-O3 -march=haswell`, same flags as the
build):
| input | scalar (autovec, ≈ current Rust) | scalar no-vec (≈ macOS LTO
case) | 128-bit SSE2 (≈ WTF::SIMD / old Zig `@Vector`) | highway static
(AVX2) | highway dynamic (selects AVX3_DL) |
|---|---|---|---|---|---|
| 16 B | 8.1 ns | 45.6 ns | 3.2 ns | 2.0 ns | 6.0 ns |
| 64 B | 28.4 ns | 175 ns | 5.9 ns | 5.1 ns | **3.9 ns** |
| 256 B | 110 ns | 702 ns | 14.8 ns | 10.4 ns | **7.8 ns** |
| 1 KiB | 437 ns | 2843 ns | 61 ns | 37 ns | **14.2 ns** |
| 16 KiB | 7165 ns | 44987 ns | 959 ns | 588 ns | **128 ns** |
Highway dynamic dispatch (the existing `highway_strings.cpp` pattern)
wins from 64 B up — 2–7× over the 128-bit WTF::SIMD-style kernel and
14–56× over the scalar path — because it can use AVX2/AVX-512 vectors at
runtime. Only at ≤16 B does the dispatch indirection cost ~3 ns, which
is noise next to the JS→native call. On arm64 NEON both highway and
WTF::SIMD compile to 128-bit vectors, so expect parity with the old Zig
`@Vector(16, u8)` code there (not measured here — x64-only container).
## End-to-end numbers (mitata, `bench/snippets/string-width.mjs`)
> **Note:** the table below is from the initial version of this PR.
After the review, the dense-SGR regression was fixed and the
classification path tightened — see [the post-review benchmark
comment](oven-sh/bun#31351 (comment))
for current numbers (no input slower than the current canary at any
size).
Release build of this branch vs release build of main (`f161e0311`),
same machine:
| input (release builds, x64 AVX-512 host) | main (Rust) | this PR (C++
+ highway) | speedup |
|---|---|---|---|
| ascii, 11 chars | 33.4 ns | 24.9 ns | 1.3× |
| ascii, 46 chars | 56.3 ns | 27.8 ns | 2.0× |
| ascii, 460 chars | 262 ns | 28.8 ns | 9.1× |
| ascii, 4.6k chars | 2.33 µs | 106 ns | 22× |
| ansi-heavy (SGR + truecolor), 1× | 115 ns | 83.5 ns | 1.4× |
| ansi-heavy, 10× | 818 ns | 774 ns | 1.06× |
| ansi-heavy, 100× | 7.74 µs | 7.61 µs | 1.02× |
| OSC-8 hyperlinks, 10× | 1.75 µs | 1.20 µs | 1.5× |
| emoji (ZWJ/flags/keycaps), 1× | 506 ns | 316 ns | 1.6× |
| emoji, 20× | 9.36 µs | 6.28 µs | 1.5× |
| CJK, 1× | 606 ns | 347 ns | 1.7× |
| CJK, 20× | 11.8 µs | 6.93 µs | 1.7× |
| mixed unicode, 10× | 11.2 µs | 7.05 µs | 1.6× |
| ascii 460 chars, countAnsiEscapeCodes | 406 ns | 139 ns | 2.9× |
| ansi-heavy 10×, countAnsiEscapeCodes | 487 ns | 157 ns | 3.1× |
| emoji 20×, countAnsiEscapeCodes | 9.47 µs | 6.37 µs | 1.5× |
(mitata, `avg` column; "main" = release build of f161e03, "this PR" =
release build of this branch on the same machine. The short-string cases
are dominated by the JS→native call itself; the per-escape-sequence
structure of the ansi-heavy case limits it to parity-or-better — the win
there comes from the `countAnsiEscapeCodes: true` path and the OSC
scanning.)
## Behavior / tests
- The existing `Bun.stringWidth` suite passes unchanged (112 tests),
plus new cases for the ASCII fast path (SIMD chunk-boundary lengths,
Latin-1 high bytes, ASCII stored in 16-bit strings), ANSI sequences
mid-string (SGR + OSC-8 in both Latin-1 and UTF-16 strings,
`countAnsiEscapeCodes` both ways), emoji/ZWJ sequences, and East Asian
wide/ambiguous characters — 123 total.
- `sliceAnsi` (incl. fuzz), `wrapAnsi`, `stripANSI`, `console.table`
snapshot, `Bun.inspect.table`, markdown-renderer, node `readline`
`getStringWidth` suites all pass against the debug (ASAN) build.
- Edge-case coercion parity spot-checked against the previous
implementation (`undefined`/`null`/numbers/objects as input,
falsy-string vs falsy-number option values, Symbol throwing): identical
results.
- One pre-existing exception: `sliceAnsi adversarial > many SGR codes
don't overflow or quadratic-slow` exceeds its 5 s debug-build time bound
on this (heavily throttled) container — 20.6 ms/iter inside
`sliceAnsi.cpp`'s own SGR handling under ASAN, vs 0.07 ms/iter in
release. The code it times is untouched by this PR (`Bun.stringWidth` on
the same adversarial input takes 0.17 ms/iter on the same debug build).
## Related issues
- #8782 (switch from `@Vector`-style SIMD to Google Highway with runtime
dispatch) — this PR does that for the stringWidth / visible-width paths,
but not codebase-wide, so it is not closed by this PR.
- The find-issues bot also suggested #8329, #6776 and #30693; those are
behavior bugs/requests that a behavior-preserving port does not address,
so they are intentionally not auto-closed here.
---------
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Bun does not link uucode. #31254 removed the vendored Zig library. #31351 added src/jsc/bindings/stringWidth.cpp, which ports the uucode grapheme break algorithm and carries over the grapheme break class bits of its tables. Word the row as that derived code in both tables, like the libbase64 row. The link and the MIT license stay.
What
Moves the
Bun.stringWidthimplementation from Rust to C++ with explicit SIMD, as requested.src/jsc/bindings/stringWidth.cpp/stringWidth.h— the full visible-width engine: codepoint classification (zero-width, East AsianW/F, ambiguous), grapheme cluster break (UAX Check requirements for the build #29 incl. GB9c, the uucode-derived 3-stage tables now live in the generatedstringWidthTables.h), the grapheme-cluster width accumulator (ZWJ emoji, regional-indicator flags, keycaps, skin tones, VS15/VS16), Latin-1 / UTF-8 / UTF-16 width functions with CSI/OSC handling,utf8IndexAtWidth, and theBun.stringWidthhost function (samecountAnsiEscapeCodes/ambiguousIsNarrowoptions, same defaults, same coercion semantics).highway_strings.cpp(highway_visible_latin1_width,highway_visible_latin1_width_exclude_ansi,highway_count_printable_ascii16,highway_first_non_ascii8/16) using the existing dynamic-dispatch pattern. The default (ANSI-excluding) Latin-1 path is a single-pass kernel: each chunk is classified once into printable/ESC/CSI-final/OSC-terminator bitmasks and escape regions are carved out with a few bit ops per escape, with in-CSI/in-OSC state carried across chunks. The UTF-16 path's escape scanning still reuses theWTF::SIMDhelpers inANSIHelpers.hshared with stripANSI/wrapAnsi/sliceAnsi.stringWidthnow points atjsFunctionBunStringWidthin the BunObject LUT; theBunObject_callback_stringWidthRust callback andmacro(stringWidth)are gone. The readline /util.inspect/ console.table builtins switch from$newZigFunction("string.zig", "String.jsGetStringWidth")to$newCppFunction("stringWidth.cpp", "jsFunctionBunStringWidth").visible.rs(1.6k lines) is now a ~70-line FFI shim that preserves thebun.strings.visible::width::exclude_ansi_colors::{latin1,utf8,utf16,utf8_index_at_width}paths for the remaining Rust callers (console.table column sizing inConsoleObject.rs, the markdown ANSI renderer).grapheme.rs,grapheme_tables.rs, the.zigporting references, the deadvisible_fallbackmodule,index_of_char16_usize, and theicu_hasBinaryPropertyshim (all only existed to support the Rust width code) are deleted.String::visible_width(include-ANSI variant) had no remaining callers and is removed;String::visible_width_exclude_ansi_colorskeeps working through the shim.sliceAnsi.cpp/wrapAnsi.cppare untouched — they keep callingBun__codepointWidth,Bun__graphemeBreak,Bun__isEmojiPresentation,Bun__visibleWidthExcludeANSI_{latin1,utf16}, which are now defined instringWidth.cppinstead of Rust.Why
The Rust implementation's
u8x16/u16x8"vectors" are scalarforloops that rely on LLVM's SLP autovectorizer to become SIMD — ~2.3× slower than real vector code on the ASCII path, and much worse under macOS LTO whereld64.lld's pipeline doesn't run the SLP pass (see #31348 / #31349 for the build-side workarounds). Explicit highway kernels remove the dependency on the optimizer entirely and let x64 use 256/512-bit vectors via dynamic dispatch.SIMD choice: highway vs WTF::SIMD
Microbenchmark of the "count visible Latin-1 bytes" kernel (x64, AVX-512-capable host, clang 21
-O3 -march=haswell, same flags as the build):@Vector)Highway dynamic dispatch (the existing
highway_strings.cpppattern) wins from 64 B up — 2–7× over the 128-bit WTF::SIMD-style kernel and 14–56× over the scalar path — because it can use AVX2/AVX-512 vectors at runtime. Only at ≤16 B does the dispatch indirection cost ~3 ns, which is noise next to the JS→native call. On arm64 NEON both highway and WTF::SIMD compile to 128-bit vectors, so expect parity with the old Zig@Vector(16, u8)code there (not measured here — x64-only container).End-to-end numbers (mitata,
bench/snippets/string-width.mjs)Release build of this branch vs release build of main (
f161e0311), same machine:(mitata,
avgcolumn; "main" = release build of f161e03, "this PR" = release build of this branch on the same machine. The short-string cases are dominated by the JS→native call itself; the per-escape-sequence structure of the ansi-heavy case limits it to parity-or-better — the win there comes from thecountAnsiEscapeCodes: truepath and the OSC scanning.)Behavior / tests
Bun.stringWidthsuite passes unchanged (112 tests), plus new cases for the ASCII fast path (SIMD chunk-boundary lengths, Latin-1 high bytes, ASCII stored in 16-bit strings), ANSI sequences mid-string (SGR + OSC-8 in both Latin-1 and UTF-16 strings,countAnsiEscapeCodesboth ways), emoji/ZWJ sequences, and East Asian wide/ambiguous characters — 123 total.sliceAnsi(incl. fuzz),wrapAnsi,stripANSI,console.tablesnapshot,Bun.inspect.table, markdown-renderer, nodereadlinegetStringWidthsuites all pass against the debug (ASAN) build.undefined/null/numbers/objects as input, falsy-string vs falsy-number option values, Symbol throwing): identical results.sliceAnsi adversarial > many SGR codes don't overflow or quadratic-slowexceeds its 5 s debug-build time bound on this (heavily throttled) container — 20.6 ms/iter insidesliceAnsi.cpp's own SGR handling under ASAN, vs 0.07 ms/iter in release. The code it times is untouched by this PR (Bun.stringWidthon the same adversarial input takes 0.17 ms/iter on the same debug build).Related issues
@Vector-style SIMD to Google Highway with runtime dispatch) — this PR does that for the stringWidth / visible-width paths, but not codebase-wide, so it is not closed by this PR.