Wasm build doesn't work on Windows

[arcanis]

Running a build on Windows doesn't work when using the esbuild-wasm package. It works fine when using esbuild (but we'd prefer to avoid that since it requires a postinstall script).

Repro:

git clone git@github.com:yarnpkg/berry && cd berry
git checkout mael/esbuild-2
yarn build:cli

Error:

error: Invalid path: C:\berry\packages\yarnpkg-cli\bundles\yarn.js
    at failureErrorWithLog (C:\berry\.yarn\cache\esbuild-wasm-npm-0.8.33-0589ac914f-8d0542a70c.zip\node_modules\esbuild-wasm\lib\main.js:969:15)
    at buildResponseToResult (C:\berry\.yarn\cache\esbuild-wasm-npm-0.8.33-0589ac914f-8d0542a70c.zip\node_modules\esbuild-wasm\lib\main.js:767:32)
    at C:\berry\.yarn\cache\esbuild-wasm-npm-0.8.33-0589ac914f-8d0542a70c.zip\node_modules\esbuild-wasm\lib\main.js:819:20
    at handleIncomingPacket (C:\berry\.yarn\cache\esbuild-wasm-npm-0.8.33-0589ac914f-8d0542a70c.zip\node_modules\esbuild-wasm\lib\main.js:566:9)

[evanw]

Ah, I think I understand why. It looks like perhaps Go's path internals are locked to Unix-style paths when using WebAssembly. Specifically, building with WebAssembly sets GOOS=js and I think Go's path/filepath module only uses Windows-style paths when GOOS=windows. I clearly haven't ever tried this before. This appears to be a shortcoming of Go. I'll ask around to see what I should do here. I'd certainly like for this to work.

FWIW the WebAssembly version of esbuild is mainly intended to be used in the browser where running a native executable is impossible. The native version is intended to be used outside of the browser. The native version is around 10x faster than the WebAssembly version for various reasons. I'm not saying you have to switch away from the WebAssembly version, especially if I can figure out how to fix this problem. I'm just telling you that it'll go a lot faster if you do.

[arcanis]

Yep - our Webpack build was 28s, with esbuild-wasm it's 5s, and with esbuild it's 750ms. Still, not having to run yarn install when cloning the repo or switching between branches is a significant advantage, so for now we'll likely stay on the wasm version.

Fwiw I'm also thinking about adding proper builtin support for multi-platform packages into Yarn - similar to what you're mimicking with the "download the right package on-demand" postinstall script, but more integrated with the package manager infrastructure. If that works out native builds could be a viable option.

[kzc]

Off topic... comment from the cross linked yarn PR above:

Switches the build to ESBuild

It used to take 28s to build the CLI. With this diff it becomes 6s (on the other hand, went 1.62MB -> 1.92MB).

The development iteration speed improvement is fantastic and certainly justifies the change, but I wonder what accounts for the 300KB bundle size increase with esbuild compared to webpack+terser. An optimization opportunity for esbuild perhaps?

[evanw]

The development iteration speed improvement is fantastic and certainly justifies the change, but I wonder what accounts for the 300KB bundle size increase with esbuild compared to webpack+terser. An optimization opportunity for esbuild perhaps?

When I read the thread it looked like this might be more about which code was selected for bundling instead of the result of minifier optimizations: yarnpkg/berry#2362 (comment). Not sure if that's still current or not though.

[arcanis]

When I read the thread it looked like this might be more about which code was selected for bundling instead of the result of minifier optimizations: yarnpkg/berry#2362 (comment). Not sure if that's still current or not though.

I've managed to bring the size down to 1.83MB (don't remember how), but I couldn't go lower. The size seems the same whether I have platform: "node" set or not (in both cases esprima is missing from the bundle, as wanted).

[evanw]

I took a look at fixing this and it looks like this is an inherently buggy scenario. I've been trying all day and I still haven't gotten it working yet, but I've already hit these issues:

1. path/filepath: WebAssembly uses *nix conventions on Windows golang/go#43768: Go's path/filepath module is broken on Windows in WebAssembly.

   To fix this, I can fork Go's path/filepath module and include both the Unix and Windows implementations and then switch between them at run-time.

2. fs.readSync throws EOF Error instead of returning 0 (Windows, if STDIN is a pipe) nodejs/node#35997: Node's fs.read throws an error at the end of a pipe on Windows, meaning passing stdin to esbuild with WebAssembly is broken on Windows.

   I should be able to monkey-patch fs.read to swallow the incorrect error on Windows.

3. tiered WebAssembly compilation broken nodejs/node#36616: Node's unfortunate handling of WebAssembly compilation can make node temporarily hang on exit, which means the WebAssembly-powered esbuild command often takes 10x longer than necessary.

   You can almost do process.kill(process.pid) except the exit code will be non-zero, which isn't right. So I was thinking of trying a native node extension that calls _exit(0) instead.

I'm hoping that these are the main issues with the esbuild-wasm package on Windows.

[arcanis]

I seem to remember that the error is triggered by a Abs validation in esbuild - would it be possible to disable those checks in wasm mode, and just assume that all paths are always absolute?

[kzc]

So I was thinking of trying a native node extension that calls _exit(0) instead.

I would advise against this. Native compilation defeats the purpose of using wasm on NodeJS. It doesn't work on all platforms, like the older version of Mac OS I use which cannot build fsevents natively.

[kzc]

After reviewing 9ea2076 I can report that npm/esbuild-wasm/exit0/darwin-x64-LE.node works correctly on mac OS 10.9. The prebuilt zero dependency wasm-napi-exit0 dynamic libraries are a good workaround.

[evanw]

I would advise against this. Native compilation defeats the purpose of using wasm on NodeJS. It doesn't work on all platforms, like the older version of Mac OS I use which cannot build fsevents natively.

I figured it's harmless if it doesn't work for whatever reason, because node is exiting anyway and any errors running the thing are ignored. Worst case node will just exit many seconds later as usual.

Edit: Here are the times on my end for running the WebAssembly esbuild --version command:

	Before	After	Improvement
macOS 10.15	2.900s	0.153s	19x faster
Windows 10 VM	16.164s	0.397s	41x faster

[kzc]

Worst case node will just exit many seconds later as usual

On my Mac I see a roughly 0.4s improvement with esbuild-wasm with 9ea2076...

before 9ea2076:

$ /usr/bin/time npm/esbuild-wasm/bin/esbuild --version
0.8.34
        0.62 real         1.88 user         0.19 sys

$ echo 'console.log(1 + 2)' | /usr/bin/time npm/esbuild-wasm/bin/esbuild --minify
console.log(1+2);
        0.60 real         1.86 user         0.20 sys

after 9ea2076:

$ /usr/bin/time npm/esbuild-wasm/bin/esbuild --version
0.8.34
        0.22 real         0.68 user         0.06 sys

$ echo 'console.log(1 + 2)' | /usr/bin/time npm/esbuild-wasm/bin/esbuild --minify
console.log(1+2);
        0.23 real         0.69 user         0.07 sys

[kzc]

macOS 10.15 2.900s 0.153s 19x faster
Windows 10 VM 16.164s 0.397s 41x faster

It's hard to believe this NodeJS bug has gone unnoticed given its performance on recent OS versions. Wasm modules must not be widely used.