Compile ASM without Atomics to support older browsers

[Jaifroid]

TLDR: As Kiwix Offline Wikipedia reader supports legacy browsers, we need to find a way to compile a complex library to ASM without Atomics / pthreads. The problem seems to arise in the zlib library, but --disable-pthreads is not supported by the zlib configure command. We also use WORKERFS. Is there a way to prevent Emscripten compiling with Atomics?

DETAILS: The Kiwix offline Wikipedia browser needs to support a wide range of legacy browsers in its JavaScript version. Until recently, we used specific Emscripten ports of XZ and ZStandard to provide decompression of strings from very large (up to 93GB) archives, and this works well even in Internet Explorer. We recently managed to compile our entire libzim application to WASM (and ASM) with Emscripten, which provides significant advantages over the earlier ports, e.g. use of Xapian full-text search. However, it only works in the latest browsers.

The issue is that the Emscripten port of libzim uses Atomics. Unfortunately, a number of our target browsers don't support the JS Atomics object. We understand that we need to compile without pthread support in order to disable Atomics, but it's proving difficult to do so. We believe we've narrowed the problem down to the linked zlib library, used by a number of the libraries in the project, though the issue may also be that we are forced to use the WORKERFS file system because it is the only one that supports passing a reference to 93GB File objects. Unlike with the ZStandard port, the libzim application needs to work on files rather than on strings or file slices.

This is how we compile zlib in our makefile:

        wget -N https://zlib.net/zlib-1.2.13.tar.gz
	tar xf zlib-1.2.13.tar.gz
	cd zlib-1.2.13 ; emconfigure ./configure --prefix=`pwd`/../build
	cd zlib-1.2.13 ; emmake make
	cd zlib-1.2.13 ; emmake make install

Unfortunately, the zlib configure command does not support --disable-pthreads or --disable-threaded-resolver. It lists very few, basic options (configure [--const] [--zprefix] [--prefix=PREFIX] [--eprefix=EXPREFIX] [--static] [--64] [--libdir=LIBDIR] [--sharedlibdir=LIBDIR] [--includedir=INCLUDEDIR] [--archs="-arch i386 -arch x86_64"]).

When we link the required libraries together for the ASM version, we don't link with -lpthread, but we are still getting Atomics emitted in the JavaScript:

em++ -o libzim-asm.js --bind libzim_bindings.cpp -I/src/build/include -L/src/build/lib -lzim -llzma -lzstd -lxapian -lz -licui18n -licuuc -licudata -O3 --pre-js prejs_file_api.js --post-js postjs_file_api.js -s WASM=0 --memory-init-file 0 -s MIN_EDGE_VERSION=40 -s "EXPORTED_RUNTIME_METHODS=['ALLOC_NORMAL','printErr','ALLOC_STACK','print']" -s INITIAL_MEMORY=83886080 -s ALLOW_MEMORY_GROWTH=1 -lworkerfs.js

Is there a way to disable Atomics? The -s MIN_IE_VERSION=11 or -s MIN_EDGE_VERSION=40 don't make any difference in this regard. Oddly, ZStandard also requires zlib, but we have been able to compile that in a version that worked even with IE11. Is WORKERFS the real problem here?

[sbc100]

Firstly, I don't think workerfs has any bearing on the use of pthreads or atomics.

Secondly, unless you build with -sUSE_PTHREADS (or equivalently -pthread), then all of your object files and libraries should be free from atomics, since without those flags llvm will lower away any atomics. So I imagine what is happening is that one or more of your libraries is being build with the -pthread or the -sUSE_PTHREADS flag.

I would recommend disassembly your binary (I normally use wasm-objdump -d -r for this) to find out which function contains the atomic instruction, then figuring out which object file or library it is coming from and looking at the command line that was use to build that object

Finally, emscripten provides a bundled version of zlib that doesn't require building it yourself. You can just use -sUSE_ZLIB instead if you like.

[sbc100]

BTW, what makes you think that zlib is the cause the of the atomic operations? I don't imagine zlib would contains such things, but I could be wrong.

[Jaifroid]

BTW, what makes you think that zlib is the cause the of the atomic operations? I don't imagine zlib would contains such things, but I could be wrong.

It may be a red herring. I was able to add --disable-pthreads to every other library compilation command except to zlib. But it may arise from another library using zlib in a multi-threaded way.

Thank you for the wasm-objdump suggestion. I'll try using that to isolate the problem. I did try -sUSE_ZLIB with the other libraries that use zlib, but then I get emcc complaining that several of the _deflate functions can't be found and would need to be exported explicitly.

[sbc100]

zlib, but then I get emcc complaining that several of the _deflate functions can't be found and would need to be exported explicitly.

That is odd, if you get a chance maybe you could file a separate bug about that.

[Jaifroid]

That is odd, if you get a chance maybe you could file a separate bug about that.

OK, I'll check and add details.

[Jaifroid]

Just to say that the cause of the pthreading turned out to be multithreading in our libzim binary, and not zlib. It's a shame there doesn't seem to be a way in Emscripten to mimic pthreads in JS without using JS Atomic operations (as the latter limits browser support quite drastically). Thank you for the help in locating the issue.