WASI support

[connor4312]

I've made a fork of coreutils that implements provisional support for running most coreutils under WASI.

Currently this involves quite a bit of conditional flagging, in large part because Rust hasn't adopted newer proposals that implement file permissions to WASI. However, the number of changes is not terrible, though there are still quite a few warnings I'd need to fix up.

I was wondering what the appetite is of coreutils maintainers to merge something like this into the repo and start officially supporting WASI, at least for a subset of tools.

[tertsdiepraam]

That's very cool!

Not sure how the other maintainers feel, but I'm in favor of merging something like that! It would be another step towards making uutils truly "universal" :)

The number of differences with other platforms indeed feels manageable. In order to make the support "official" we will have to add it to the CI though, so we don't break it by accident.

at least for a subset of tools.

Do you know what subset of utils is useful on WASI? Do unix-specific utilities like chmod make sense? I think the first version we could merge would require support for at least the "common core" set of utils (the ones enabled by default).

Edit: Why are you requiring path dependencies for libc and nix? Do you need to patch them to work with WASI?

[connor4312]

Great! 🙂 This is still a rough sketch and there's polishing to do before we'd be ready to PR. I don't know if that'll happen this month or next.

Also cc @dbaeumer who is leading the wasi work for VS Code

Do you know what subset of utils is useful on WASI? Do unix-specific utilities like chmod make sense? I think the first version we could merge would require support for at least the "common core" set of utils (the ones enabled by default).

chmod would be relevant when file permissions are available in Rust's wasi libc. I think this is even possible today if one uses the wasi-sdk, though I have not tried it, and imo we should not write code that doesn't work with Rust/clang out of the box.

However, not all utils reference functionality present in WASI proposals. For example, there are no processes for kill to kill.

I agree that targeting a common core set of utils makes sense.

Edit: Why are you requiring path dependencies for libc and nix? Do you need to patch them to work with WASI?

Yea, I have a couple of PR's into both nix and libc at the moment (rust-lang/libc#3142 rust-lang/libc#3143 nix-rust/nix#2011)

[sylvestre]

Not sure how the other maintainers feel, but I'm in favor of merging something like that! It would be another step
towards making uutils truly "universal" :)

+1 :)

[dbaeumer]

Our current goal is to implement a WASM/WASI host for VS Code running in the browser so that web assemblies have transparent access to the VS Code file system to the console and the terminal. Here is a small demo running CPython in the browser with access to the files in the workspace which are hosted in a GitHub repository:

What we would like to provide as well is some sort of basic terminal support so that users can execute python directly or run command like ls, cat, ...

This being said commands like kill or chmod are not high priority for that support.

[tertsdiepraam]

That sounds awesome! I'd love for uutils to be a part of that effort!

[RReverser]

FWIW I had a WASI-enabled fork of coreutils a while back as well, with my custom JS runtime to have coreutils working in a browser: https://wasi.rreverser.com/ (https://github.com/GoogleChromeLabs/wasi-fs-access, fork here https://github.com/RReverser/coreutils)

It allows to mount even with an actual filesystem in Chromium-based browsers! (video demo here https://www.youtube.com/watch?v=qRmO-8b4WmE)

Unfortunately, I never got around to upstreaming my changes and they're probably more outdated than @connor4312's, just thought I'd post for posterity :)

[chadbrewbaker]

@connor4312 I have some time to write a GitHub worker YAML. Sorry, I didn't see in your code. Were you using Chromium or some other WASM/WASI runtime? Also what is the list of WASM/WASM runtimes that we are expecting to support?

[connor4312]

I was using wasmer to develop locally. But out of the box rust targets a very minimal set of WASI APIs so I would expect it to be quite portable. Dirk was able to build upon this work to run the subset of coreutils in Chrome.

[RReverser]

I see there was an attempt to land this in #7596, but it never got a response. Is this still planned to be merged?

[sylvestre]

@RReverser if someone finishes the work, sure :)

[RReverser]

I guess I can try to rebase my patches from earlier if they are not horribly outdated. I just figured those branches by others are more up-to-date :)

[RReverser]

Just noticed @GPlaczek forked my wasi-enabled fork of coreutils into https://github.com/antmicro/coreutils/commits/wasi/, which should presumably be more up-to-date. @GPlaczek do you want to make a PR upstreaming those changes or should I?

[GPlaczek]

Just noticed @GPlaczek forked my wasi-enabled fork of coreutils into https://github.com/antmicro/coreutils/commits/wasi/, which should presumably be more up-to-date. @GPlaczek do you want to make a PR upstreaming those changes or should I?

Hi @RReverser!

Our fork is heavily modified and uses a lot of non-standard hacks specific to our use case in jswasi. We've had some successful attempts to backport some of the applets. Rebasing all of your and our changes over updated upstream was difficult and we didn't have enough time to do that.

I can reorganize our changes to remove our hacks to make everything mergeable. I will also try to rebase over updated upstream and resolve conflicts.

I'll try to do that over the weekend, I'll update you with my progress in a few days.

[GPlaczek]

@RReverser I rebased our fork over updated main from the upstream and removed our hacks. I pushed it to my fork: https://github.com/GPlaczek/coreutils/tree/rebased-nohacks.

It does not build yet, I will keep working on it but if you want to take a look and try to fix it, feel free.

Many of your patches were outdated and couldn't be easily applied to the upstream, I think that this code needs a lot of cleaning up before it is mergeable.