EMSDK support for wasm backend

[jgravelle-google]

As part of using the wasm backend by default, we should make it easy for users to download and install it. This means having emsdk install upstream LLVM instead of fastcomp.

Steps:

1. Add wasm backend to emsdk behind a flag
2. Switch to wasm backend by default (fastcomp behind a flag)
3. Deprecate fastcomp backend

There's the question of how to organize installing the two backends. The options I know of are

1. Download only one backend
   • need to rerun emsdk install to download the other backend later (which is bound to be a frustrating UX)
   • need to emsdk activate to change backends
2. Download both backends
   • no need to download/compile the other backend later, but instead frontloads that cost
   • may need to edit ~/.emscripten to change the LLVM_ROOT variable
3. Download one LLVM that has both backends enabled
   • integrate upstream changes into fastcomp branch to get latest wasm backend, build fastcomp with js and wasm32 targets enabled
   • need to merge regularly going forward
   • changing backends is a matter of changing ~/.emscripten to set... something. Currently EMCC_WASM_BACKEND is an environment flag, can .emscripten set that?

[kripken]

Current status is that the emsdk can install latest-upstream which gets the latest lkgr. Users can install both that and fastcomp, and switch between them.

Not yet done:

• non-linux
• emcc version integration, i.e., something other than "last lkgr" (still thinking about what to do with this in our testbots and builders)

[DragonOsman]

(I'm posting this question here, but also wondering if it's okay for me to do so)

So this doesn't support Windows yet (at least at the time of writing)? And am I correct in assuming that when it's complete, developers would be able to use the WASM Backend through EMSDK? What about the global Node installation? I have one for the latest version of Node, and I want to use that with EMSDK if possible, preferably without having to keep changing the .emscripten file every time I update the EMSDK (both for the path to the Node installation and for the LLVM root setting).

And if it still doesn't support Windows yet, does anyone have any idea when it might be able to? Just wondering. Thanks.

[kripken]

Yes, eventually the wasm backend will replace the older backend, and builds will be available on all platforms. Meanwhile, you can build it from source, or if some other place has recent LLVM builds you can use those (since it's just a normal LLVM build - for example, maybe wasi has builds for windows).

[DragonOsman]

Right now the wasm backend seems to only work on Linux distributions. Lately, when I try to do it on Windows, I get an error from gen_struct_info.py. Invoking Node gives a non-zero value, with Node itself just giving an empty output instead of some sort of message. It used to work before, so I don't know what happened. So right now it seems like I'll have to use the WSL if I want to do it on Windows.

[DragonOsman]

This is the error message I get when trying to use the wasm backend with Emscripten to compile C or C++ code to WebAssembly:

cache:INFO: generating system asset: generated_struct_info.json... (this will be cached in "C:\Users\Osman\.emscripten_cache\wasm-obj-pic\generated_struct_info.json" for subsequent builds)
Traceback (most recent call last):
  File "C:\emsdk\emscripten\incoming\emcc.py", line 3334, in <module>
    sys.exit(run(sys.argv))
  File "C:\emsdk\emscripten\incoming\emcc.py", line 1947, in run
    final = shared.Building.emscripten(final, target + '.mem', js_libraries)
  File "C:\emsdk\emscripten\incoming\tools\shared.py", line 2204, in emscripten
    emscripten.run(infile, outfile, memfile, js_libraries)
  File "C:\emsdk\emscripten\incoming\emscripten.py", line 2560, in run
    shared.Settings.STRUCT_INFO = shared.Cache.get(generated_struct_info_name, ensure_struct_info)
  File "C:\emsdk\emscripten\incoming\tools\cache.py", line 133, in get
    temp = creator()
  File "C:\emsdk\emscripten\incoming\emscripten.py", line 2557, in ensure_struct_info
    gen_struct_info.main(['-qo', out, path_from_root('src', 'struct_info.json')])
  File "C:\emsdk\emscripten\incoming\tools\gen_struct_info.py", line 529, in main
    struct_info = inspect_code(header_files, cpp_opts, structs, defines)
  File "C:\emsdk\emscripten\incoming\tools\gen_struct_info.py", line 407, in inspect_code
    info = shared.run_js(js_file[1]).splitlines()
  File "C:\emsdk\emscripten\incoming\tools\shared.py", line 1090, in run_js
    return jsrun.run_js(filename, engine, *args, **kw)
  File "C:\emsdk\emscripten\incoming\tools\jsrun.py", line 155, in run_js
    raise Exception('Expected the command ' + str(command) + ' to finish with return code ' + str(assert_returncode) + ', but it returned with code ' + str(proc.returncode) + ' instead! Output: ' + str(ret)[:error_limit])
Exception: Expected the command ['C:/Program Files/nodejs/node.exe', 'c:\\users\\osman\\appdata\\local\\temp\\tmpzu370u.js'] to finish with return code 0, but it returned with code -1073741819 instead! Output:

Note the empty output from Node, as well as the other stuff mentioned in the exception.

Do I need to debug the Python script?

[kripken]

I think we can close this issue. At this point we have wasm backend support in emsdk for windows, mac and linux, and tested on our new CI, all installable with ./emsdk install latest-upstream.