Should WASI APIs all be compatible with "shared nothing linking"?

[alexcrichton]

The concept of "shared nothing linking" has become quite interesting recently with the advent of WebAssembly Interface Types. The general idea is that if you link wasm modules together, they don't actually need to share anything, not even memories, which means that as a consumer or a producer you can guarantee that all your memory/globals/etc are purely internal state.

Currently WASI, however, largely requires that its APIs have access to an exported memory from a wasm module. This is mostly evident through APIs like

• APIs that fill in a caller-provided buffer, args_get, environ_get, fd_prestat_dir_name, fd_readdir, path_readlink, random_get, and anything using iovec (e.g. fd_read). These APIs all work with mutable pointers right now in the *.witx file, unlike other syscalls like path_link which can purely work with a string type which doesn't (in the long run) require sharing memories.

• Additionally poll_oneoff has trickery about filling in a caller-provided buffer.

Do we want to move to a world where these raw mutable pointers are exposed as part of the API in *.witx files? Do we want to move to a world which is more supportive of "shared nothing linking" where you can link in a WASI implementation that doesn't require the consuming module to export its memory?

I personally think that we may want to move in this direction, but it's not entirely clear to me how we'd be handling some of these APIs. Also to be clear I do think that the performance is pretty critical here that extraneous memory copies are introduced, but interface types' laziness property we have a lot of flexibility to expose apis which might look like they're copying when actually under the hood very little copying is happening.

For example many of the above APIs can probably move towards a paradigm where they return a string or an array of strings. This means that the API is specified in terms of returning buffers, but interface adapters could adapt the current ABI to this API in a module-specific fashion. (and today we'd sort of bless "one implicit adapter" as the one true ABI).

Other APIs like fd_readdir could perhaps return a handle for each directory entry, and then the directory entry's handle could then be used to fetch more information via more syscalls.

Cases like random_get are sort of interesting because you're not really requesting an array of random bytes, but rather an array of random bytes with a particular length, but I think we could still massage this all together with interface types.

In any case I'm curious to see what others think? Is this something that we should just trend towards in the limit? Or perhaps try to move more closely to in terms of polyfills today?

[sbc100]

My understanding is that "shared nothing linking" mostly relates to sharing between wasm modules, rather than between a wasm module and the system/embedder. Whereas WASI is specifically all about interfacing with the system (and specifically not related to interfacing with other modules). See #68 for more on this.

I guess the real question is: can we simultaneously do "shared memory linking" with the embedder while doing "shared nothing linking" with other modules. If not, then that kind forces shared nothing linking into WASI I guess?

From what I understand of what you have written, if we do decide to go in this direction we can continue to maintain source compat using wrappers to emulate the current behaviour. i.e. it would be possible experiment with shared nothing WASI syscalls without needing to modify large amounts of wasi-libc? it could be done initially with an extra shim layer?

[sunfishcode]

@alexcrichton Yes, I agree with this overall direction. witx is aimed at being aligned with interface types in the long run. See the string and handle types, for example. Mutable output buffers in particular are an ongoing design discussion on the interface types side. Witx's dedicated pointer type is a temporary mechanism in order to mark the places where we'll want mutable output buffers, once we have a more clear picture of how those will look.

@sbc100 Eventually, instead of sharing the whole memory with the embedder at all, with interface types, we want to have APIs which effectively pass views of memory to each other, rather than exporting their entire memories and passing offsets.

[devsnek]

this is also important for better interactions with js+esm, where memory can't be shared prior to evaluation.

[alexcrichton]

Another way to sort of look at this issue is that if WASI wants to be entirely defined in terms of interface types, I think that one specific hurdle we need to overcome is handling the mutable output buffer problem in interface types land. Currently the *.witx format is a hybrid between "just get stuff done" and interface types, which is completely expected at this stage, but I wanted to make sure an issue was on file for changing this calculus over time.

[linclark]

Since the consensus now is to define WASI entirely in terms of interface types, I believe this issue is resolved. Do free free to reopen if I'm missing something, though.