Error: "exports count of 143958 exceeds internal limit of 10000"

[anutosh491]

Hi,

I am facing this issue

RuntimeError: Aborted(CompileError: WebAssembly.instantiate(): exports count of 143958 exceeds internal limit of 100000 @+214486)

I am not planning on missing out any function/symbol (hence providing EXPORT_ALL=1 to em++ and --export-all to wasm-ld)

When building with EMCC_DEBUG=1 emmake make -j16 install VERBOSE=1, it generates a file (/var/folders/m1/cdn74f917994jd99d_2cpf440000gn/T/emscripten_temp/tmp6qr4um78.json) which is giving me more info on the exported functions. I see a lot of them hence wanted to confirm if this is due to the huge list of functions I am interested in and is there a way to get past this error ?

I've pasted the contents of the file here for anyone interested (https://raw.githubusercontent.com/anutosh491/xeus-cpp-lite-debug/refs/heads/main/tmp6qr4um78.json)

[anutosh491]

When i get rid of the --export-all flags I end up with this file (https://github.com/anutosh491/xeus-cpp-lite-debug/blob/main/tmptqjqh2js.json) but then I lose out of symbols which are all present in the above and I end up with these errors when I host target.js and target.wasm

wasm-ld: error: incr_module_2.wasm: undefined symbol: __clang_Interpreter_SetValueNoAlloc

[anutosh491]

I guess adding -s EXPORTED_FUNCTIONS=['_clang_Interpreter_SetValueNoAlloc'] would add all exported functions and shown in the above + _clang_Interpreter_SetValueNoAlloc ?

I could try that but I was interested in all of them cause I am not sure what sort of symbols I need and the ones I can skip !

[sbc100]

Yes, you almost certainly don't want to be passing --export-all to the linker. By the way, are you running wasm-ld directly? Is there some reason you are not using em++ as your linker?

In terms of exporting symbols, you should only export symbol that you intent to use directly from JS, i.e. that you want to call from the embedder. What are you trying to build here? Are you trying to just run clang? If so, than you don't need any exports other than the default main symbol.

[anutosh491]

What are you trying to build here?

The issues I have been opening lately (1, 2 which I think you self assigned and this one) are all towards the same goal that is to get xeus-cpp-lite (xeus-cpp + jupyter-lite) working

Xeus-cpp is a Jupyter kernel that allows us to run C++ code on a jupyter notebook. Jupyter-lite basically aids running jupyter kernels so that you can have something completely in browser.
So basically we can say that the bigger goal is to run clang-repl in the browser.

Basically there are 4 layers here and all of them should be compiled against emscripten

1. xeus-cpp (the kernel)
2. CppInterOp (a thin wrapper/helper) over clang-repl/cling
3. llvm (for clangInterpreter, lldWasm etc)

i) Now llvm has been compiled cleanly to I have access to libclangInterpreter.a
ii) I am trying to either get libclangCppInterOp.a or libclangCppInterOp.so
iii) For libclangCppInterOp.a (I did this as you suggested to fix the undefined symbols and I can successfully come up with libclangCppInterOp.a but then when I'm trying to generate xcpp.js and xcpp.wasm (simply having target_link_libraries (xeus-cpp PUBLIC clangCppInterOp foo bar) I get an lld crash as noted on 2)
iv) Generating libclangCppInterOp.so wasn't tough (inspired by this issue) and the linking step and generation etc went smoothly (no undefined symbols or anything atleast while building)

But I end up with this error xcpp.js:2931 wasm-ld: error: incr_module_2.wasm: undefined symbol: __clang_Interpreter_SetValueNoAlloc

The stack trace tells me that

1. We have clang-repl instantiated.
2. We can Parse stuff (Parse from ParseAndExecute works)
3. Maybe execution is an issue. (Execute function fails)

[anutosh491]

So the error is basically coming from here

Pasting the stack trace too for more info

So I am kinda trying to make sure where the error is

1. The llvm build looks good

 anutosh491@Anutoshs-MacBook-Air lib % nm -C libclangInterpreter.a --defined-only | grep __clang_Interpreter_SetValueNoAlloc
00005cf4 T __clang_Interpreter_SetValueNoAlloc

2. Having a shared build or a static build for CppInterOp, looking at the error I am guessing __clang_Interpreter_SetValueNoAlloc should be present in the WASM_EXPORTS when finally build xeus-cpp.

So i am not sure what might be going wrong. Hence I was just passing (-s EXPORT-ALL =1 and target_link_options(xcpp PUBLIC "-Wl,--export-all") while building xeus-cpp.

Also now that I don't see any undefined symbol during build time but I see the above undefined symbol during runtime. I am just curious if this has something to do with mangling. I can see that almost all stuff coming out of clangInterpreter is like _ZNXXX... but then the above symbol defined here and being used here is using extern "C"

[anutosh491]

By the way, are you running wasm-ld directly? Is there some reason you are not using em++ as your linker?

No, I am not running wasm-ld directly. I am running em++ itself. My final link.txt looks like this when I use both the export flags
/Users/anutosh491/micromamba/envs/xeus-cpp-wasm-build/lib/python3.13/site-packages/emsdk/upstream/emscripten/em++ -Wl,--export-all -O0 -g --bind -Wno-unused-command-line-argument -fexceptions -s MODULARIZE=1 -s EXPORT_NAME=createXeusModule -s EXPORT_ES6=0 -s USE_ES6_IMPORT_META=0 -s DEMANGLE_SUPPORT=0 -s ASSERTIONS=2 -s ALLOW_MEMORY_GROWTH=1 -s EXIT_RUNTIME=1 -s WASM=1 -s -s STACK_SIZE=64mb -s INITIAL_MEMORY=256mb -s WASM_BIGINT -s FORCE_FILESYSTEM -s MAIN_MODULE=1 -s EXPORT_ALL=1 @CMakeFiles/xcpp.dir/objects1.rsp -o xcpp.js @CMakeFiles/xcpp.dir/linkLibs.rsp

That being said I am seeing this.

1. If I don't have the -Wl,--export-all and only have the EXPORT_ALL=1 here... it make no difference. I end up with the undefined symbols error
2. But the other way round (including --export-all) I see that __clang_Interpreter_SetValueNoAlloc is present in the EXPORTED_FUNCTIONS and the WASM_EXPORTS section (in the tmp6qr4um78.json produced when i run with emcc_debug=1)

But when I try option 2) I end up getting the above error

[anutosh491]

That being said, I think it would be appropriate to close this issue.

After exporting the required symbol something like --export=__clang_Interpreter_SetValueNoAlloc I still see the same wasm-ld error .... so I guess it might not even an issue related to exporting symbols. The error still shows that it is coming out of scanRelocations

[anutosh491]

Although I have closed the issue, I am curious to know what might be giving us the error.

For example now i am running this on the code cell

extern "C" int printf(const char*,...);
auto r1 = printf("r=%d\n", 1+1);

so I get this error xcpp.js:2972 wasm-ld: error: incr_module_2.wasm: undefined symbol: printf

1. So the error comes out of here (place which reports the undefined symbol)
2. The place where the symbol is extracted is this
3. I see that there is something called reloc.type on which the operations are based so we need to figure out what this is for our case.
   Also something like file->getSymbols() which gives us the symbols, so we kinda need to understand how the file incr_module_2.wasm is created and what symbols does it carry.
   Actually even to start with something like this ... I haven't understood as to what creates incr_module_0.wasm OR incr_module_1.wasm if they are even created. Obviously the file creation is happening here

This doesn't look like a symbol not present issue.

[sbc100]

Looking at the code in https://github.com/llvm/llvm-project/blob/9f796159f28775b3f93d77e173c1fd3413c2e60e/clang/lib/Interpreter/Wasm.cpp#L74-L86 it doesn't link in any standard libraries so it not suprising that things like printf are not defined.

Do you what that clang/lib/Interpreter/Wasm.cpp is designed to do? It certainly doesn't looks like its designed to link general purpose programs because there is not libc or compiler-rt mentioned there.

[anutosh491]

Hey @sbc100

Thanks for the reply

it doesn't link in any standard libraries

Ahh yes that's something I missed totally.

Do you what that clang/lib/Interpreter/Wasm.cpp is designed to do?

Yes, so this is where it at all starts. So Wasm.cpp was introduced as an attempt to help running clang-repl in the browser. The approach can be found here (https://compiler-research.org/assets/presentations/Anubhab_Ghosh_wasm_clangrepl.pdf)

Wasm.cpp basically is a code based representation of the approach stated above. So Wasm.cpp basically does this (contents on the 1st slide are basically something we can do, 2nd slide is basically how wasm.cpp helps us)

[anutosh491]

Well so as the above approach goes we would be interested in how we do the linking. Your reply tells me that we might be interested in more flags .... not sure maybe some more flags for standard libraries as follows !

-whole-archive
-lstubs-debug
-lc-debug
-ldlmalloc-debug
-lcompiler_rt
-lc++
-lc++abi-debug
--no-whole-archive

Along with what we already have. Does that look like the way to go here ?

[sbc100]

If you want to make the code in https://github.com/llvm/llvm-project/blob/9f796159f28775b3f93d77e173c1fd3413c2e60e/clang/lib/Interpreter/Wasm.cpp#L74-L86 somehow compatible with emscripten linker, I think that would represent a lot of work.

You would basically have to duplicate all/most of the logic in the emscripten linker in that Wasm.cpp file. I would not recommend attempted that unless are ready to dive deep into all the internal details of emscripten and add a lot of code to Wasm.cpp.

BTW, if you were going to try to make this work I would suggest getting it all working first with the normal native/desktop version of clang (i.e. solve the problem of making /clang/lib/Interpreter/Wasm.cpp much smarter before trying to make it all run inside of wasm).

[sbc100]

To give you an idea of the complexity here is just a small subset of the logic in the emscripten linker for generating the link command:

emscripten/tools/building.py

Lines 133 to 293 in 4b3bace

	def lld_flags_for_executable(external_symbols):
	cmd = []
	if external_symbols:
	if settings.INCLUDE_FULL_LIBRARY:
	# When INCLUDE_FULL_LIBRARY is set try to export every possible
	# native dependency of a JS function.
	all_deps = set()
	for deps in external_symbols.values():
	for dep in deps:
	if dep not in all_deps:
	cmd.append('--export-if-defined=' + dep)
	all_deps.add(dep)
	stub = create_stub_object(external_symbols)
	cmd.append(stub)
	if not settings.ERROR_ON_UNDEFINED_SYMBOLS:
	cmd.append('--import-undefined')
	if settings.IMPORTED_MEMORY:
	cmd.append('--import-memory')
	if settings.SHARED_MEMORY:
	cmd.append('--shared-memory')
	# wasm-ld can strip debug info for us. this strips both the Names
	# section and DWARF, so we can only use it when we don't need any of
	# those things.
	if settings.DEBUG_LEVEL < 2 and (not settings.EMIT_SYMBOL_MAP and
	not settings.EMIT_NAME_SECTION and
	not settings.ASYNCIFY):
	cmd.append('--strip-debug')
	if settings.LINKABLE:
	cmd.append('--export-dynamic')
	if settings.LTO and not settings.EXIT_RUNTIME:
	# The WebAssembly backend can generate new references to `__cxa_atexit` at
	# LTO time. This `-u` flag forces the `__cxa_atexit` symbol to be
	# included at LTO time. For other such symbols we exclude them from LTO
	# and always build them as normal object files, but that would inhibit the
	# LowerGlobalDtors optimization which allows destructors to be completely
	# removed when __cxa_atexit is a no-op.
	cmd.append('-u__cxa_atexit')
	c_exports = [e for e in settings.EXPORTED_FUNCTIONS if is_c_symbol(e)]
	# Strip the leading underscores
	c_exports = [demangle_c_symbol_name(e) for e in c_exports]
	# Filter out symbols external/JS symbols
	c_exports = [e for e in c_exports if e not in external_symbols]
	c_exports += settings.REQUIRED_EXPORTS
	if settings.MAIN_MODULE:
	c_exports += side_module_external_deps(external_symbols)
	for export in c_exports:
	if settings.ERROR_ON_UNDEFINED_SYMBOLS:
	cmd.append('--export=' + export)
	else:
	cmd.append('--export-if-defined=' + export)
	for e in settings.EXPORT_IF_DEFINED:
	cmd.append('--export-if-defined=' + e)
	if settings.RELOCATABLE:
	cmd.append('--experimental-pic')
	cmd.append('--unresolved-symbols=import-dynamic')
	if not settings.WASM_BIGINT:
	# When we don't have WASM_BIGINT available, JS signature legalization
	# in binaryen will mutate the signatures of the imports/exports of our
	# shared libraries. Because of this we need to disabled signature
	# checking of shared library functions in this case.
	cmd.append('--no-shlib-sigcheck')
	if settings.SIDE_MODULE:
	cmd.append('-shared')
	else:
	cmd.append('-pie')
	if not settings.LINKABLE:
	cmd.append('--no-export-dynamic')
	else:
	cmd.append('--export-table')
	if settings.ALLOW_TABLE_GROWTH:
	cmd.append('--growable-table')
	if not settings.SIDE_MODULE:
	cmd += ['-z', 'stack-size=%s' % settings.STACK_SIZE]
	if settings.ALLOW_MEMORY_GROWTH:
	cmd += ['--max-memory=%d' % settings.MAXIMUM_MEMORY]
	else:
	cmd += ['--no-growable-memory']
	if settings.INITIAL_HEAP != -1:
	cmd += ['--initial-heap=%d' % settings.INITIAL_HEAP]
	if settings.INITIAL_MEMORY != -1:
	cmd += ['--initial-memory=%d' % settings.INITIAL_MEMORY]
	if settings.STANDALONE_WASM:
	# when settings.EXPECT_MAIN is set we fall back to wasm-ld default of _start
	if not settings.EXPECT_MAIN:
	cmd += ['--entry=_initialize']
	else:
	if settings.PROXY_TO_PTHREAD:
	cmd += ['--entry=_emscripten_proxy_main']
	else:
	# TODO(sbc): Avoid passing --no-entry when we know we have an entry point.
	# For now we need to do this since the entry point can be either `main` or
	# `__main_argv_argc`, but we should address that by using a single `_start`
	# function like we do in STANDALONE_WASM mode.
	cmd += ['--no-entry']
	if settings.STACK_FIRST:
	cmd.append('--stack-first')
	if not settings.RELOCATABLE:
	cmd.append('--table-base=%s' % settings.TABLE_BASE)
	if not settings.STACK_FIRST:
	cmd.append('--global-base=%s' % settings.GLOBAL_BASE)
	return cmd
	def link_lld(args, target, external_symbols=None):
	if not os.path.exists(WASM_LD):
	exit_with_error('linker binary not found in LLVM directory: %s', WASM_LD)
	# runs lld to link things.
	# lld doesn't currently support --start-group/--end-group since the
	# semantics are more like the windows linker where there is no need for
	# grouping.
	args = [a for a in args if a not in ('--start-group', '--end-group')]
	# Emscripten currently expects linkable output (SIDE_MODULE/MAIN_MODULE) to
	# include all archive contents.
	if settings.LINKABLE:
	args.insert(0, '--whole-archive')
	args.append('--no-whole-archive')
	if settings.STRICT and '--no-fatal-warnings' not in args:
	args.append('--fatal-warnings')
	if any(a in args for a in ('--strip-all', '-s')):
	# Tell wasm-ld to always generate a target_features section even if --strip-all/-s
	# is passed.
	args.append('--keep-section=target_features')
	cmd = [WASM_LD, '-o', target] + args
	for a in llvm_backend_args():
	cmd += ['-mllvm', a]
	if settings.WASM_EXCEPTIONS:
	cmd += ['-mllvm', '-wasm-enable-eh']
	if settings.WASM_EXCEPTIONS or settings.SUPPORT_LONGJMP == 'wasm':
	cmd += ['-mllvm', '-exception-model=wasm']
	if settings.MEMORY64:
	cmd.append('-mwasm64')
	# For relocatable output (generating an object file) we don't pass any of the
	# normal linker flags that are used when building and executable
	if '--relocatable' not in args and '-r' not in args:
	cmd += lld_flags_for_executable(external_symbols)
	cmd = get_command_with_possible_response_file(cmd)
	check_call(cmd)

[anutosh491]

BTW, if you were going to try to make this work I would suggest getting it all working first with the normal native/desktop version of clang (i.e. solve the problem of making /clang/lib/Interpreter/Wasm.cpp much smarter before trying to make it all run inside of wasm).

Few comments here

1. I see that Wasm.cpp only comes into play when building clang (or rather clangInterpreter) against emscripten. (https://github.com/llvm/llvm-project/blob/ae509a085836079585228aede8a5017ad80e1aa9/clang/lib/Interpreter/Interpreter.cpp#L18-L20)
2. Also the discussion here([clang-repl] Support wasm execution llvm/llvm-project#86402) tells me that there might not be straightforward ways inside llvm just as of now to test this out through a bot. So I am guessing it is only through downstream projects like xeus-cpp/CppInterOp that we can test this out.

That being said, would this approach make sense

1. We run the script you showed above at the xeus-cpp layer, select the options and put them through an env variable and trickle it down to the wasm.cpp linker command

[anutosh491]

I have been discussing this with @vgvassilev and @argentite who were the authors of Wasm.cpp. As far as my understanding goes they could get a very similar demo working not neccessarily passing the flags to the linker (and hence I think the LinkerArgs in wasm.cpp doesn't possess flags like standard libraries as of now)

The demo can be seen here (https://wasmdemo.argentite.me/)

[anutosh491]

Hey @sbc100

I would like to try something out. The following is my understanding. Maybe you could let me know if you think I am wrong somewhere

1. So while dynamically linking Wasm binaries ..... The main module holds all the symbols required from the common libraries, along with any other project symbols you choose to compile.
2. The side modules hold any other code you desire, and importantly, cannot operate without being loaded in conjunction with a main module.

Now while building with Emscripten (we link again -lc or -lc-debug by default) and hence xcpp.wasm (our main module) has have access to printf (as can bee seen in this json file that gets produced while building with emcc_debug=1)

This tells me that if we add --allow-undefined (nothing but --unresolve-symbols=ignore + --import-undefined) for our linker flags

1. The linker will not fail due to printf being undefined in the incr_module_2.wasm (It should simply defer the resolution of printf to runtime)
2. Since printf is exported from xcpp.wasm, when we later load the side module (incr_module_2.wasm) at runtime (using dlopen or similar), the system will attempt to resolve printf.
3. My understanding tells me that printf must be available in the global symbol table when the side module is loaded. If the side module (incr_module_2.wasm) doesn't directly include printf, it will attempt to resolve it from the global context—which is where xcpp.wasm (the main module) comes into play

I can see here (https://github.com/llvm/llvm-project/blob/0c5bf565ba7059ca8542c522fcab019f2e2c82bb/clang/lib/Interpreter/Wasm.cpp#L91C1-L92C62) we use this

  void *LoadedLibModule =
      dlopen(OutputFileName.c_str(), RTLD_NOW | RTLD_GLOBAL);

And the docs tell me

1. RTLD_NOW
   If this value is specified all undefined symbols in the library are resolved before dlopen() returns. If this cannot be done, an error is returned.
2. RTLD_GLOBAL
   The symbols defined by this library will be made available for symbol resolution of subsequently loaded libraries.

So my understanding here tells me that maybe we can not resolve symbols at link time rather do it when we dynamically load libraries using dlopen ?

The change involved is just the following (which would be made here https://github.com/llvm/llvm-project/blob/9f796159f28775b3f93d77e173c1fd3413c2e60e/clang/lib/Interpreter/Wasm.cpp#L74-L86)

  std::vector<const char *> LinkerArgs = {"wasm-ld",
                                          "-pie",
                                          "--import-memory",
                                          "--no-entry",
                                          "--export-all",
                                          "--experimental-pic",
                                          "--no-export-dynamic",
                                          "--stack-first",
+                                         "--allow-undefined"
                                          OutputFileName.c_str(),
                                          "-o",
                                          OutputFileName.c_str()};
                                          

[sbc100]

• So while dynamically linking Wasm binaries ..... The main module holds all the symbols required from the common libraries, along with any other project symbols you choose to compile.
• The side modules hold any other code you desire, and importantly, cannot operate without being loaded in conjunction with a main module.

Yes, both of those assertions are correct.

[sbc100]

So my understanding here tells me that maybe we can not resolve symbols at link time rather do it when we dynamically load libraries using dlopen ?

Yes, you can either have the static linker (wasm-ld) resolve the symbols, or you can have the dyanmic linker (the code in library_dylink.js) resolve the symbols when dlopen is called.

The change involved is just the following (which would be made here https://github.com/llvm/llvm-project/blob/9f796159f28775b3f93d77e173c1fd3413c2e60e/clang/lib/Interpreter/Wasm.cpp#L74-L86)

  std::vector<const char *> LinkerArgs = {"wasm-ld",
                                          "-pie",
                                          "--import-memory",
                                          "--no-entry",
                                          "--export-all",
                                          "--experimental-pic",
                                          "--no-export-dynamic",
                                          "--stack-first",
+                                         "--allow-undefined"
                                          OutputFileName.c_str(),
                                          "-o",
                                          OutputFileName.c_str()};

If you want to load the resulting module with dlopen you should be using -shared rather that -pie.

[anutosh491]

Thanks for the quick reply.

So as the approach I pasted above also speaks about generating a "shared library"

As can be seen this also mentions about resolving the symbols at the dynamic loading step.

So what I'll do is, maybe try with this building llvm against emscripten with this patch

  std::vector<const char *> LinkerArgs = {"wasm-ld",
 -                                        "-pie",
 +                                        "-shared",
                                          "--import-memory",
                                          "--no-entry",
                                          "--export-all",
                                          "--experimental-pic",
                                          "--no-export-dynamic",
                                          "--stack-first",
+                                         "--allow-undefined"
                                          OutputFileName.c_str(),
                                          "-o",
                                          OutputFileName.c_str()};
                                          

I guess the above confirms that we are looking for a shared library (that avoids the symbol resolution at link time) that can be dynamically loaded using dlopen.

Do you see any other necessary change ?

[sbc100]

Do you see any other necessary change ?

Building a shared libraries correctly will likely involve mimicking the current logic in emscripten for doing so:

emscripten/tools/building.py

Lines 133 to 249 in 5c81135

	def lld_flags_for_executable(external_symbols):
	cmd = []
	if external_symbols:
	if settings.INCLUDE_FULL_LIBRARY:
	# When INCLUDE_FULL_LIBRARY is set try to export every possible
	# native dependency of a JS function.
	all_deps = set()
	for deps in external_symbols.values():
	for dep in deps:
	if dep not in all_deps:
	cmd.append('--export-if-defined=' + dep)
	all_deps.add(dep)
	stub = create_stub_object(external_symbols)
	cmd.append(stub)
	if not settings.ERROR_ON_UNDEFINED_SYMBOLS:
	cmd.append('--import-undefined')
	if settings.IMPORTED_MEMORY:
	cmd.append('--import-memory')
	if settings.SHARED_MEMORY:
	cmd.append('--shared-memory')
	# wasm-ld can strip debug info for us. this strips both the Names
	# section and DWARF, so we can only use it when we don't need any of
	# those things.
	if settings.DEBUG_LEVEL < 2 and (not settings.EMIT_SYMBOL_MAP and
	not settings.EMIT_NAME_SECTION and
	not settings.ASYNCIFY):
	cmd.append('--strip-debug')
	if settings.LINKABLE:
	cmd.append('--export-dynamic')
	if settings.LTO and not settings.EXIT_RUNTIME:
	# The WebAssembly backend can generate new references to `__cxa_atexit` at
	# LTO time. This `-u` flag forces the `__cxa_atexit` symbol to be
	# included at LTO time. For other such symbols we exclude them from LTO
	# and always build them as normal object files, but that would inhibit the
	# LowerGlobalDtors optimization which allows destructors to be completely
	# removed when __cxa_atexit is a no-op.
	cmd.append('-u__cxa_atexit')
	c_exports = [e for e in settings.EXPORTED_FUNCTIONS if is_c_symbol(e)]
	# Strip the leading underscores
	c_exports = [demangle_c_symbol_name(e) for e in c_exports]
	# Filter out symbols external/JS symbols
	c_exports = [e for e in c_exports if e not in external_symbols]
	c_exports += settings.REQUIRED_EXPORTS
	if settings.MAIN_MODULE:
	c_exports += side_module_external_deps(external_symbols)
	for export in c_exports:
	if settings.ERROR_ON_UNDEFINED_SYMBOLS:
	cmd.append('--export=' + export)
	else:
	cmd.append('--export-if-defined=' + export)
	for e in settings.EXPORT_IF_DEFINED:
	cmd.append('--export-if-defined=' + e)
	if settings.RELOCATABLE:
	cmd.append('--experimental-pic')
	cmd.append('--unresolved-symbols=import-dynamic')
	if not settings.WASM_BIGINT:
	# When we don't have WASM_BIGINT available, JS signature legalization
	# in binaryen will mutate the signatures of the imports/exports of our
	# shared libraries. Because of this we need to disabled signature
	# checking of shared library functions in this case.
	cmd.append('--no-shlib-sigcheck')
	if settings.SIDE_MODULE:
	cmd.append('-shared')
	else:
	cmd.append('-pie')
	if not settings.LINKABLE:
	cmd.append('--no-export-dynamic')
	else:
	cmd.append('--export-table')
	if settings.ALLOW_TABLE_GROWTH:
	cmd.append('--growable-table')
	if not settings.SIDE_MODULE:
	cmd += ['-z', 'stack-size=%s' % settings.STACK_SIZE]
	if settings.ALLOW_MEMORY_GROWTH:
	cmd += ['--max-memory=%d' % settings.MAXIMUM_MEMORY]
	else:
	cmd += ['--no-growable-memory']
	if settings.INITIAL_HEAP != -1:
	cmd += ['--initial-heap=%d' % settings.INITIAL_HEAP]
	if settings.INITIAL_MEMORY != -1:
	cmd += ['--initial-memory=%d' % settings.INITIAL_MEMORY]
	if settings.STANDALONE_WASM:
	# when settings.EXPECT_MAIN is set we fall back to wasm-ld default of _start
	if not settings.EXPECT_MAIN:
	cmd += ['--entry=_initialize']
	else:
	if settings.PROXY_TO_PTHREAD:
	cmd += ['--entry=_emscripten_proxy_main']
	else:
	# TODO(sbc): Avoid passing --no-entry when we know we have an entry point.
	# For now we need to do this since the entry point can be either `main` or
	# `__main_argv_argc`, but we should address that by using a single `_start`
	# function like we do in STANDALONE_WASM mode.
	cmd += ['--no-entry']
	if settings.STACK_FIRST:
	cmd.append('--stack-first')
	if not settings.RELOCATABLE:
	cmd.append('--table-base=%s' % settings.TABLE_BASE)
	if not settings.STACK_FIRST:
	cmd.append('--global-base=%s' % settings.GLOBAL_BASE)
	return cmd

.

As you can see there is a fair amount of complexity there so trying to reimplement that in Wasm.cpp will likely involve a fair amount of work, and back and forth. I'm not saying its impossible, but be aware that you could be signing up for a fair bit more work/debugging here.

[anutosh491]

Hmm, well this is something I really need to confirm cause the above approach (or the people who executed it) don't neccessarily mention this and were successfully able to get a REPL running in the browser.

You can try it out here too (https://wasmdemo.argentite.me/)

Now obviously each iteration or code input in the REPL ends up being a shared/side module and needs to be loaded on top of the main module and these modules obviously need to have access to symbols (I am guessing atleast those that are available in the main module or from standard libraries) and even these side modules should keep propagating the symbols to the other side modules that would keep on being generated as we provide code through the REPL.

That being said llvm (or wasm.cpp) currently shows these linker flags (https://github.com/llvm/llvm-project/blob/9f796159f28775b3f93d77e173c1fd3413c2e60e/clang/lib/Interpreter/Wasm.cpp#L74-L86)

Now for sure I can say that the approach above says that the undefined symbols should be resolved at the dlopen step so surely we need to use "--allow-undefined" or something at the link step. Apart from that atleast the above running repl didn't involve anything else is what I can see.

[anutosh491]

Also in my comment above(#22863 (comment)) , I wrote about an approach

We run the script you showed above at the xeus-cpp layer, select the options and put them through an env variable and trickle it down to the wasm.cpp linker command

Maybe you could validate if this is a valid approach. So I say that as you mention the script in emscripten/tools/building.py... we could run it at the highest level and make a note of what all is required concering the main module I guess and pass them down to wasm.cpp (not having any linkers flag already present by default)

[anutosh491]

Going through the lld_flags_for_executable function (these flags look essential to me)

What we know as of now

1. We want to generate shared libraries
2. The symbol resolution or accessing symbols should be done through the main module at the dynamic loading step (dlopen) rather the linking step (wasm-ld)

Talking about the flags that should end up here ( https://github.com/llvm/llvm-project/blob/9f796159f28775b3f93d77e173c1fd3413c2e60e/clang/lib/Interpreter/Wasm.cpp#L74-L86)

1. --import-undefined : We can skip undefined symbol or defer it to runtime
2. --import-memory: To share the memory defined by the main module.
3. --export-dynamic: I am guessing this is crucial if dynamic linking with dlopen and shared symbols across modules.
   (if you see what's implemented in wasm.cpp is --no-export-dynamic which is contradictory )

Okay this is confusing

According to the approach Each code-block in the repl maps to a side module

So cell1 defines and implements sqrt
cell 2 tries printf(sqrt(25)) .. we want sqrt symbol from incr_module_2.wasm and printf symbol from xcpp.wasm.

So I guess we need to propagate symbols through the shared libraries being built too ?

4. -shared
5. --experimental-pic and --unresolved-symbols=import-dynamic to enable runtime symbol resolution and ensure position-independent code. (I have used --allow-undefined for now as I think that covers --unresolved-symbols while linking)

NOTE

1. I didn't see any condition talking about export-all so not sure if it's relevant for shared libraries.
2. I see --no-entry being added in WASM.cpp but I see the following

  if not settings.SIDE_MODULE:
        .......
        cmd += ['--no-entry']   

--no-entry is only added if not a side module, but in our case it is a side module, so not sure if this is necessary in wasm.cpp too !

[anutosh491]

Hmm, so I am basically trying this out (@sbc100)

  std::vector<const char *> LinkerArgs = {"wasm-ld",
                                          "-shared",
                                          "--import-memory",
                                          "--no-entry",
                                          "--export-all",
                                          "--experimental-pic",
                                          "--export-dynamic",
                                          "--stack-first",
                                          "--allow-undefined",
                                          OutputFileName.c_str(),
                                          "-o",
                                          OutputFileName.c_str()};                                  

Maybe you could confirm the last 2 points I raised above (about requirement of export-all and no-entry)

[anutosh491]

I am also kinda confused as to whether pie and shared can be passed together while creating a compiler instance (https://github.com/llvm/llvm-project/blob/c7df10643bda4acdc9a02406a2eee8aa4ced747f/clang/lib/Interpreter/Interpreter.cpp#L199-L200)

1. Are these flags meant to passed to the compiler builder (I mean they need to be propagated to the link step if they are)
2. I see the following in the console when running xeus-cpp-lite (I think comes out of https://github.com/llvm/llvm-project/blob/c7df10643bda4acdc9a02406a2eee8aa4ced747f/clang/lib/Interpreter/Interpreter.cpp#L182)

-cc1 -triple wasm32-unknown-emscripten -emit-obj -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name "<<< inputs >>>" -mrelocation-model static -mframe-pointer=none -ffp-contract=on -fno-rounding-math -mconstructor-aliases -target-cpu generic -fvisibility=hidden -debugger-tuning=gdb -fdebug-compilation-dir=/ -v -fcoverage-compilation-dir=/ -resource-dir  -internal-isystem include -internal-isystem /include/wasm32-emscripten -internal-isystem /include -std=c++14 -fdeprecated-macro -ferror-limit 19 -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -fcxx-exceptions -fexceptions -fincremental-extensions -o "<<< inputs >>>.o" -x c++ "<<< inputs >>>"

I don't see any pie or shared flag here. So I am guessing it is omitted somehow which makes me kinda confused if we even need it there.

[anutosh491]

As an update I could atleast get something running when I used the above patch (#22863 (comment))

Obviously lot of stuff doesn't work just yet

So my doubts related to the flags that are neccesaary still prevail (#22863 (comment)) especially the ones with export-dynamic and --no-export-dynamic

Also can't thank you enough for the help. This atleast gets me started !!!

[anutosh491]

Gentle ping @sbc100.

Would be really helpful if I could know your thoughts/comments on the approach/ideas I have written above !

[sbc100]

I would download the compiled module (incr_module_5.wasm) and then disassemble it. Does it validate? What is the global its trying to assign to? Can you attach it here?

[anutosh491]

I would download the compiled module (incr_module_5.wasm) and then disassemble it. Does it validate? What is the global its trying to assign to? Can you attach it here?

Ahh yess this is obviously one doubt I have but more importantly I kinda need to have your views on some fundamental doubts. Once we have more info here, we can proceed to reviewing the wasm binaries

1. Error: "exports count of 143958 exceeds internal limit of 10000" #22863 (comment)
2. Error: "exports count of 143958 exceeds internal limit of 10000" #22863 (comment)
3. Also here is an approach I am curious about rather than dumping everything in wasm.cpp (Error: "exports count of 143958 exceeds internal limit of 10000" #22863 (comment))

Once I have info on these things, we can look into other stuff.

Also one thing I am curious about is if you had a chance to look at llvm/llvm-project#114651

[anutosh491]

I have been experimenting around and have quite some stuff to discuss regarding the wasm binaries being generated (the first couple might be fine after which the subsequent incr_module_XX.wasm generated are error prone)

I think I shall be in a much better position to discuss this once I get answers for the above questions !
But I shall leave you with a couple wasm binaries just in case you would be interested.

1. So as soon as I load jupyter-lite and run xeus-cpp I see an incr_module_0.wasm so we have something already produced. I think this base wasm binary comes out of here when the interpreter is initialized.
   I have pasted the contents here incr_module_0.wasm

2. After which in the first code block when I run something like

extern "C" int abs(int x);
extern "C" int printf(const char*,...);
auto result = abs(-42);
printf("r=%d\n", result);

I don't get a incr_module_1.wasm but a incr_module_2.wasm
The contents are present here incr_module_2.wasm

I shall share more soon!

[anutosh491]

Cc @sbc100

[anutosh491]

Another error I am curious about is this. So even though I can run the REPL, I see this

Now xcpp.js tells me the following

        function unexportedRuntimeSymbol(sym) {
            if (!Object.getOwnPropertyDescriptor(Module, sym)) {
                Object.defineProperty(Module, sym, {
                    configurable: true,
                    get() {
                        var msg = "'" + sym + "' was not exported. add it to EXPORTED_RUNTIME_METHODS (see the Emscripten FAQ)";
                        if (isExportedByForceFilesystem(sym)) {
                            msg += '. Alternatively, forcing filesystem support (-sFORCE_FILESYSTEM) can export this for you';
                        }
                        abort(msg);
                    }
                });
            }
        }

Now I have always been using FORCE_FILESYSTEM (hence I think I shouldn't be seeing this error right ?)

My link.txt shows me this

/Users/anutosh491/micromamba/envs/xeus-cpp-wasm-build/lib/python3.13/site-packages/emsdk/upstream/emscripten/em++ 
-00 -g --bind -Wno-unused-command-line-argument -fexceptions -s MODULARIZE=1 -s EXPORT_NAME=createXeusModule -s EXPORT_ES6=0 -s USE_ES6_IMPORT_META=0 
-s DEMANGLE_SUPPORT=0 -s ASSERTIONS=2 -s ALLOW_MEMORY_GROWTH=1 
-s EXIT_RUNTIME=1 -s WASM=1 -s USE_PTHREADS=0 -s STACK_SIZE=64mb 
-s INITIAL_MEMORY=256mb -s WASM_BIGINT -s FORCE_FILESYSTEM 
-s MAIN_MODULE=1 @CMakeFiles/xcpp.dir/objects1.rsp 
-o xcpp.js @CMakeFiles/xcpp.dir/linkLibs.rsp

Not sure if I am missing something

[sbc100]

If you are building with -O0 and -g then I would expect to see full symbol names in your backtrace not minified names. Are you running some kind of minifier on the emscripten output?

Have symbol names would allow you to see where the call to FS.mount is coming from.

[anutosh491]

Hey Sam,

Thanks a lot for the reply, shall look into it. I am also looking forward to your reply on a couple of my comments above especially (#22863 (comment))

[anutosh491]

Hey @sbc100,

We had the above changes merged related to the linker flags merged (llvm/llvm-project#116735)

But that being said I have a doubt now that I can get stuff to run (obviously if you could answer #22863 (comment) that would be great as well)

So if you see the output, you can see duplication of output from cell 1 to cell 2

The wasm modules being generates show straightaway why we have the duplication.

1. incr_module_2.wasm (basically 1st code block)

(module $incr_module_2.wasm
  (memory $env.memory (;0;) (import "env" "memory") 1)
  (table $env.__indirect_function_table (;0;) (import "env" "__indirect_function_table") 0 funcref)
  (global $__stack_pointer (;0;) (import "env" "__stack_pointer") (mut i32))
  (global $__memory_base (;1;) (import "env" "__memory_base") i32)
  (global $__table_base (;2;) (import "env" "__table_base") i32)
  (func $printf (;0;) (import "env" "printf") (param i32 i32) (result i32))
  (global $result (;3;) (export "result") i32 (i32.const 8))
  (global $__dso_handle (;4;) (export "__dso_handle") i32 (i32.const 0))
  (func $__wasm_call_ctors (;1;)
  )
  (func $__wasm_apply_data_relocs (;2;)
  )
  (func $__wasm_call_ctors (;3;) (export "__wasm_call_ctors")
    call $_GLOBAL__sub_I_incr_module_2
  )
  (func $__wasm_apply_data_relocs (;4;) (export "__wasm_apply_data_relocs")
  )
  (func $__cxx_global_var_init (;5;)
    global.get $__memory_base
    i32.const 8
    i32.add
    i32.const 42
    i32.store
  )
  (func $__stmts__1 (;6;)
    (local $var0 i32)
    (local $var1 i32)
    global.get $__stack_pointer
    i32.const 16
    i32.sub
    local.tee $var0
    global.set $__stack_pointer
    local.get $var0
    global.get $__memory_base
    local.tee $var1
    i32.const 8
    i32.add
    i32.load
    i32.store
    local.get $var1
    i32.const 0
    i32.add
    local.get $var0
    call $printf
    drop
    local.get $var0
    i32.const 16
    i32.add
    global.set $__stack_pointer
  )
  (func $_GLOBAL__sub_I_incr_module_2 (;7;)
    call $__cxx_global_var_init
    call $__stmts__1
  )
  (data (global.get $__memory_base) "r=%d\0a\00\00\00\00\00\00\00")
)

2. incr_module_3.wasm (2nd code block)

(module $incr_module_3.wasm
  (memory $env.memory (;0;) (import "env" "memory") 1)
  (table $env.__indirect_function_table (;0;) (import "env" "__indirect_function_table") 0 funcref)
  (global $__stack_pointer (;0;) (import "env" "__stack_pointer") (mut i32))
  (global $__memory_base (;1;) (import "env" "__memory_base") i32)
  (global $__table_base (;2;) (import "env" "__table_base") i32)
  (func $printf (;0;) (import "env" "printf") (param i32 i32) (result i32))
  (global $result (;3;) (import "GOT.mem" "result") (mut i32))
  (global $__dso_handle (;4;) (export "__dso_handle") i32 (i32.const 0))
  (func $__wasm_call_ctors (;1;)
  )
  (func $__wasm_apply_data_relocs (;2;)
  )
  (func $__wasm_call_ctors (;3;)
    call $__cxx_global_var_init
  )
  (func $__wasm_apply_data_relocs (;4;)
  )
  (func $__wasm_call_ctors (;5;) (export "__wasm_call_ctors")
    call $_GLOBAL__sub_I_incr_module_2
    call $_GLOBAL__sub_I_incr_module_3
  )
  (func $__wasm_apply_data_relocs (;6;) (export "__wasm_apply_data_relocs")
  )
  (func $__cxx_global_var_init (;7;)
    global.get $__memory_base
    i32.const 8
    i32.add
    i32.const 42
    i32.store
  )
  (func $__stmts__1 (;8;)
    (local $var0 i32)
    (local $var1 i32)
    global.get $__stack_pointer
    i32.const 16
    i32.sub
    local.tee $var0
    global.set $__stack_pointer
    local.get $var0
    global.get $__memory_base
    local.tee $var1
    i32.const 8
    i32.add
    i32.load
    i32.store
    local.get $var1
    i32.const 0
    i32.add
    local.get $var0
    call $printf
    drop
    local.get $var0
    i32.const 16
    i32.add
    global.set $__stack_pointer
  )
  (func $_GLOBAL__sub_I_incr_module_2 (;9;)
    call $__cxx_global_var_init
    call $__stmts__1
  )
  (func $__stmts__0 (;10;)
    (local $var0 i32)
    global.get $__stack_pointer
    i32.const 16
    i32.sub
    local.tee $var0
    global.set $__stack_pointer
    local.get $var0
    global.get $result
    i32.load
    i32.store
    global.get $__memory_base
    i32.const 0
    i32.add
    local.get $var0
    call $printf
    drop
    local.get $var0
    i32.const 16
    i32.add
    global.set $__stack_pointer
  )
  (func $_GLOBAL__sub_I_incr_module_3 (;11;)
    call $__stmts__0
  )
  (data (global.get $__memory_base) "r=%d\0a\00\00\00\00\00\00\00")
)

So incr_module_3.wasm is basically also calling _GLOBAL__sub_I_incr_module_2 which results into execution of the previous code blocks

So there are two things that go on here , Parse followed by Execute. Parse works same for both and generates a PTU for a wasm or non wasm case. But in Execute we have a separate AddModule for wasm (as this doesn't use the llvm JIT)

So the error is out of this function

So what I can see is that the 2nd module loads the 1st one again or maybe the linker combines the two modules. So not sure of a fix here because not sure if a flag might be promoting this behaviour but I think the error is out of either

1. The linker args (https://github.com/llvm/llvm-project/blob/fb4ecada815ceee37536a26b4ff5ce231226b23e/clang/lib/Interpreter/Wasm.cpp#L74-L84)
2. The link call (https://github.com/llvm/llvm-project/blob/fb4ecada815ceee37536a26b4ff5ce231226b23e/clang/lib/Interpreter/Wasm.cpp#L85-L86)

If It is the link step at fault here. What I think needs to be done is after every dlopen call, we need to remove the llvm IR (or rather any sort of static initialization) I guess, so that we are good before the next linkage occurs ?

[anutosh491]

To solve the above I thought that we could make _GLOBAL__sub_I_incr_module_XX a no-op after the dlopen call (I think linker relies on the state of the LLVM Module to decide what to include in the next .wasm binary and hence we just remove this from the llvm IR)

So I framed this

void removeGlobalInit(llvm::Module *M) {
  std::string targetName = "_GLOBAL__sub_I_" + M->getName().str();
  for (auto &Func : *M) {
    if (Func.hasName() && Func.getName().str() == targetName) {
      Func.deleteBody();
    }
  }
}

And placed a call for after the dlopen step (https://github.com/llvm/llvm-project/blob/32da1fd8c7d45d5209c6c781910c51940779ec52/clang/lib/Interpreter/Wasm.cpp#L91C1-L97C4)

  void *LoadedLibModule =
      dlopen(OutputFileName.c_str(), RTLD_NOW | RTLD_GLOBAL);
  if (LoadedLibModule == nullptr) {
    llvm::errs() << dlerror() << '\n';
    return llvm::make_error<llvm::StringError>(
        "Failed to load incremental module", llvm::inconvertibleErrorCode());
  }
  
+  removeGlobalInit(PTU.TheModule.get());

This results into the following

It says

Could not load dynamic lib: incr_module_6.wasm
CompileError: WebAssembly.Module(): Compiling function #19:"__wasm_call_ctors" failed: not enough arguments on the stack for call (need 1, got 0) @+4255

So I am not sure if my thinking is correct here but I thought what if once we've made use of a module, we can just get rid of this _GLOBAL__sub_I_incr_module_XX (or make it a no-op) so that it is not being used by the subsequent Modules.
Does that make sense ?

[sbc100]

Is the notebook style execution supposed to be a program that you are incrementally building?

Each time you add new code are you building a completely new program or building on top of the old one? If you are building on top of the new one then it makes sense that the new program (which includes both of the old and new code) would execute the static constructors from both the old and the new program (i.e. it represents the sum of all snippets).

How does this notebook work when you execute on non-wasm platforms? Or doesn't it?

[anutosh491]

Hey @sbc100 thanks for your reply

Giving you some more context (I think I gave some in the past but reiterating)

Each time you add new code are you building a completely new program or building on top of the old one?

So basically as the approach I pasted above says

1. Hence we have a main module (xeus-cpp.wasm)
2. Answering your question xeus-cpp is an established project that I maintain. Yes it works much better locally. We can do all of this (https://github.com/compiler-research/xeus-cpp/blob/main/notebooks/xeus-cpp.ipynb)
3. The readme shows both .... how to run stuff locally and how to try an emscripten build and run it with jupyter-lite (notebook but completely in browser. Now our goal is to support xeus-cpp in jupyterlite (or fundamentally getting clang-repl to run in the browser)
4. Obviously for executing the above we use llvm and xeus-cpp compiled against emscripten for wasm.
5. Continuing with the approach
   i) We have a main module (xeus-cpp.wasm)
   ii) Each cell gives us a incr_module_XX.wasm (where xx is the cell number) through this
   iii) Then we load the side module or the shared wasm binary created on top of the main module through this

Now although we fixed most of the linker args for the linking step and also the loading step through dlopen looks correct the error I face as also pointed out in the above comment (#22863 (comment)) is that I see the content being duplicated from the previous modules into the latest modules.

So basically if we have two repl's/ code block

// block 1
int x = 10;
cout << x << endl;

// block 2
cout << x + 5 << endl;

The desired out for the shared wasm binary generated from block 2 is to only have _GLOBAL__sub_I_incr_module_2 being framed and called from __wasm_call_ctors I guess but I see duplication here !!!

Now my thought here was what if we remove the static initializers from the module once it has been loaded using dlopen, hence I came up with

void removeGlobalInit(llvm::Module *M) {
  std::string targetName = "_GLOBAL__sub_I_" + M->getName().str();
  for (auto &Func : *M) {
    if (Func.hasName() && Func.getName().str() == targetName) {
      Func.deleteBody();
    }
  }
}

So that I could call removeGlobalInit(PTU.TheModule.get()); after the dlopen step .

[anutosh491]

If you are building on top of the new one then it makes sense that the new program (which includes both of the old and new code) would execute the static constructors from both the old and the new program (i.e. it represents the sum of all snippets).

Yeah exactly, we build/load on top of the old module. So the point is obviously we want to use global variables and function declarations and other stuff from the symbol table or memory but we don't want to execute the function body that defined in the previous cell blocks !!!

Is there a way to achieve this (through a linker flag or a custom logic like what I did) . I thought my logic would work as expected but it didn't !

[anutosh491]

I pasted the code above for the module being generated out of the second code block
It goes like this

(module $incr_module_3.wasm
  (memory $env.memory (;0;) (import "env" "memory") 1)
  (table $env.__indirect_function_table (;0;) (import "env" "__indirect_function_table") 0 funcref)
  (global $__stack_pointer (;0;) (import "env" "__stack_pointer") (mut i32))
  (global $__memory_base (;1;) (import "env" "__memory_base") i32)
  (global $__table_base (;2;) (import "env" "__table_base") i32)
  (func $printf (;0;) (import "env" "printf") (param i32 i32) (result i32))
  (global $result (;3;) (import "GOT.mem" "result") (mut i32))
  (global $__dso_handle (;4;) (export "__dso_handle") i32 (i32.const 0))
  (func $__wasm_call_ctors (;1;)
  )
  (func $__wasm_apply_data_relocs (;2;)
  )
  (func $__wasm_call_ctors (;3;)
    call $__cxx_global_var_init
  )
  (func $__wasm_apply_data_relocs (;4;)
  )
  (func $__wasm_call_ctors (;5;) (export "__wasm_call_ctors")
    call $_GLOBAL__sub_I_incr_module_2
    call $_GLOBAL__sub_I_incr_module_3
  )
  (func $__wasm_apply_data_relocs (;6;) (export "__wasm_apply_data_relocs")
  )
  (func $__cxx_global_var_init (;7;)
    global.get $__memory_base
    i32.const 8
    i32.add
    i32.const 42
    i32.store
  )
  (func $__stmts__1 (;8;)
    (local $var0 i32)
    (local $var1 i32)
    global.get $__stack_pointer
    i32.const 16
    i32.sub
    local.tee $var0
    global.set $__stack_pointer
    local.get $var0
    global.get $__memory_base
    local.tee $var1
    i32.const 8
    i32.add
    i32.load
    i32.store
    local.get $var1
    i32.const 0
    i32.add
    local.get $var0
    call $printf
    drop
    local.get $var0
    i32.const 16
    i32.add
    global.set $__stack_pointer
  )
  (func $_GLOBAL__sub_I_incr_module_2 (;9;)
    call $__cxx_global_var_init
    call $__stmts__1
  )
  (func $__stmts__0 (;10;)
    (local $var0 i32)
    global.get $__stack_pointer
    i32.const 16
    i32.sub
    local.tee $var0
    global.set $__stack_pointer
    local.get $var0
    global.get $result
    i32.load
    i32.store
    global.get $__memory_base
    i32.const 0
    i32.add
    local.get $var0
    call $printf
    drop
    local.get $var0
    i32.const 16
    i32.add
    global.set $__stack_pointer
  )
  (func $_GLOBAL__sub_I_incr_module_3 (;11;)
    call $__stmts__0
  )
  (data (global.get $__memory_base) "r=%d\0a\00\00\00\00\00\00\00")
)

Can we get to this

(module $incr_module_3.wasm
  (memory $env.memory (;0;) (import "env" "memory") 1)
  (table $env.__indirect_function_table (;0;) (import "env" "__indirect_function_table") 0 funcref)
  (global $__stack_pointer (;0;) (import "env" "__stack_pointer") (mut i32))
  (global $__memory_base (;1;) (import "env" "__memory_base") i32)
  (global $__table_base (;2;) (import "env" "__table_base") i32)
  (func $printf (;0;) (import "env" "printf") (param i32 i32) (result i32))
  (global $result (;3;) (import "GOT.mem" "result") (mut i32))
  (global $__dso_handle (;4;) (export "__dso_handle") i32 (i32.const 0))
  (func $__wasm_call_ctors (;1;)
  )
  (func $__wasm_apply_data_relocs (;2;)
  )
  (func $__wasm_call_ctors (;3;)
    call $__cxx_global_var_init
  )
  (func $__wasm_apply_data_relocs (;4;)
  )
  (func $__wasm_call_ctors (;5;) (export "__wasm_call_ctors")
    call $_GLOBAL__sub_I_incr_module_3
  )
  (func $__wasm_apply_data_relocs (;6;) (export "__wasm_apply_data_relocs")
  )
  (func $__stmts__0 (;10;)
    (local $var0 i32)
    global.get $__stack_pointer
    i32.const 16
    i32.sub
    local.tee $var0
    global.set $__stack_pointer
    local.get $var0
    global.get $result
    i32.load
    i32.store
    global.get $__memory_base
    i32.const 0
    i32.add
    local.get $var0
    call $printf
    drop
    local.get $var0
    i32.const 16
    i32.add
    global.set $__stack_pointer
  )
  (func $_GLOBAL__sub_I_incr_module_3 (;11;)
    call $__stmts__0
  )
  (data (global.get $__memory_base) "r=%d\0a\00\00\00\00\00\00\00")
)

Basically using stuff like printf or result that has now been declared globally by point but not make use of the function body !

[sbc100]

2. Answering your question xeus-cpp is an established project that I maintain. Yes it works much better locally. We can do all of this (https://github.com/compiler-research/xeus-cpp/blob/main/notebooks/xeus-cpp.ipynb)

If you are putting the top level code from each cell into a static constructor, then how to you avoid this issue on native platforms? i.e. how do you avoid running that static constructors from each cell each time you run the program?

I guess my point is that Wasm should be no different to native platforms in how it runs or doesn't run static constructors in the LLVM IR. I doubt you want to me trying to manually prune static constructors from the IR, unless that is the solution you also using on native platforms?

[sbc100]

call $_GLOBAL__sub_I_incr_module_2
call $_GLOBAL__sub_I_incr_module_3

The fact that both of these constructors are included in the IR means you much be including both blocks of code right? Are you compiling each block to its own object file? Or just creating one big object file?

[anutosh491]

If you are putting the top level code from each cell into a static constructor, then how to you avoid this issue on native platforms? i.e. how do you avoid running that static constructors from each cell each time you run the program?

Hmmm well for native/non-wasm cases we don't need to do anything. So we use clang-repl in the background and for native cases I think clang-repl uses the LLVM JIT but for the wasm case, it uses the above approach through the linker and dynamic loading through dlopen . So every thing boils down to the addModule function

1. For native cases - https://github.com/llvm/llvm-project/blob/32da1fd8c7d45d5209c6c781910c51940779ec52/clang/lib/Interpreter/IncrementalExecutor.cpp#L73
2. For wasm case - https://github.com/llvm/llvm-project/blob/32da1fd8c7d45d5209c6c781910c51940779ec52/clang/lib/Interpreter/Wasm.cpp#L40

So there are 2 different incremental executor responsible for addition of modules. In the first case the lllvm JIT takes care of everything (through addMoudle and runctors I suppose) but I don't think the same is being done through the Linker approach for wasm cases.

So the important point here is xeus-cpp is only providing the frontend (a notebook and a kernel to run stuff) . All heavy lifiting is being done by clang-repl .... so we don't do anything special from our side to cater to any build platform.

[anutosh491]

The fact that both of these constructors are included in the IR means you much be including both blocks of code right? Are you compiling each block to its own object file? Or just creating one big object file?

I mean yeah both blocks of code are being included. We just want 1 main module and each code block should give us a side module that we keep loading on top of the main one as we run code in the repl. So just each code block maps a separate wasm shared binary !

Now if you see the linker and also the flags present in dlopen

  std::vector<const char *> LinkerArgs = {"wasm-ld",
                                          "-shared",
                                          "--import-memory",
                                          "--no-entry",
                                          "--export-all",
                                          "--experimental-pic",
                                          "--stack-first",
                                          "--allow-undefined",
                                          OutputFileName.c_str(),
                                          "-o",
                                          OutputFileName.c_str()};
  int Result =
      lld::wasm::link(LinkerArgs, llvm::outs(), llvm::errs(), false, false);

  void *LoadedLibModule =
      dlopen(OutputFileName.c_str(), RTLD_NOW | RTLD_GLOBAL);

you can see we use

1. --import-memory hoping that these modules share the same memory
2. RTLD_GLOBAL and export-all (not sure if both are required) but the whole point is to convey the symbols from one module across all future modules.

call $_GLOBAL__sub_I_incr_module_2
call $_GLOBAL__sub_I_incr_module_3

Exactly what I am trying to have your views on ;)

1. So dlopen loads symbols out of module1 into the runtime.
2. But the linker relies on the state of the LLVM Module to decide what to include in the next .wasm binary.

So can we do something (maybe through a linkerarg passed to wasm-ld or changing the way lld::wasm::link works or maybe through the method I suggested to get rid of the the global initializers after the dlopen) to avoid the duplication ?!

[sbc100]

I mean yeah both blocks of code are being included.

It doesn't sound to me like that approach your are taking will work since you will end up with side module N include all the code from the N-1 other side modules. This includes global data and static constructors. You could try to had the top level code for the previous N-1 modules when compiling module N, but its does at least needs to see all the declarations and types from the N-1 previous modules.

For example imagine if the fist module declare some global data and mutates in the static constructor (top level code).

int data = [1, 2, 3];
data[2] = 99;

If the next module duplicates this code it would get is own copy of data and you would want to rerun the constructors to ensure the value of data[2] is correct. The alternative would be to try to have the second module see only the declaration of the extern data, rather then a copy of it .e.g

extern int* data;

The approach sounds really tricky though.

Perhaps it might be worth attempting to use the same IncrementalExecutor.cpp method using LLVM JIT? That way the solution will be closer to one that you know already works.

[anutosh491]

Screen.Recording.2024-12-04.at.11.03.06.AM.mov

Thanks a lot for the discussion here. I finally got clang-repl running completely in browser. Sharing a screen recording .. not sure how clear it turns out to be though (I think making it full screen makes it much better)