Port std::process

[skade]

Design document with active discussion:

https://paper.dropbox.com/doc/async-process--Ae7VXYrJ4sSucoYlMC7XYBQvAg-Fbg2Jq7UbhqihtnWpc1EY

External process execution is clearly asynchronous. This API should make it possible to both asynchronously handle IO streams and also just wait for them to finish their execution.

[OddCoincidence]

I'd like to work on this if nobody else is already

[skade]

@OddCoincidence I know of no one currently working. Please keep in mind many of us are on RustConf this week, so mentoring might be a little delayed :).

Feel free to hop on our discord or open a draft PR here!

[yaahc]

just pinging the issue to let ppl know that I'm interested in working on this as well.

[ghost]

@OddCoincidence Hi, sorry about the slow response! Did you perhaps make any progress?

@yaahc Let's see what's the current status first. If the issue is not taken yet, I can assign it to you.

As for the implementation plan, my suggestion is we kick off by building a minimum working module with:

1. Add Command struct with methods new and spawn.
2. Add Child struct with no fields and no methods.
3. Implement Future for Child.

Then, we can do the remaining work in follow-up PRs (in no particular order):

• Add the remaining methods on Command.
• Add the remaining methods on Child.
• Add ChildStdin, ChildStdout, and ChildStderr.

[yaahc]

On it boss salutes @stjepang

[ghost]

@yaahc Hey, just wanted to check in and ask how it's going :) If you need help with this task or have any questions, let me know!

[yaahc]

Hey @stjepang, I haven't been able to start yet because I've been swamped with procrastinating / slowly working on my talk for cogoldrust, ADHD problems >_>, I'm still interested in working on this but its an intimidatingly large task so I'm having trouble starting it. My hope is to start working on it after Thursday, which is when I'm giving the trial version of my talk @ the bay area rust meetup. After that I should be essentially done with my talk and back in "ready for other things" mode.

I'm sorry for claiming this task and taking so long to do it :S, if you're feeling pressure to start this soon lmk and that might help me get going easier or feel free to assign to someone else or w/e.

[vhakulinen]

Has there been any progress with this?

(I'm trying to evaluate current async support for std::process::Command for implementing neovim-lib in async fashion with subprocessing)

[yaahc]

I haven't done anything yet. Last I heard @stjepang was planning to lay the groundwork and split it up into pieces of work that need to be done, so I've been watching for that to happen before engaging.

[frehberg]

Proposal: AFAICS,, dealing with async termination of subprocesses is quiet challenging. For example the Posix framework does not permit to poll for term/read-events on process-ids (pid), but provides an async signalling mechanism to tell the parent process about termination of child-processes. The problem is, that this signal may be delivered to any thread within the parent-process (assuming NPTL): no thread-locality and the SIGCHLD-signal might occur even before we got the chance to store the pid of the child-process in any form of shared database with within the parent process.
On the other side Win32 etc. provides a nice HANDLE-API to deal with child-processes, permitting async polling on process-handles and socket-handles combined (WaitForMultipleObjects)

For the Rust API "async_std::process::Child" I would like to propose the concept "forkfd" internally, as used by the implementation of QProcess. Here, a pipe is established between the parent process and the child-process, and within the parent-process the file-descriptor of the pipe-end may be used to poll for READ-events . In case the child-process terminates, the pipe-file-descriptor is receiving the EOF read-event, waking up the the parent-task in async fashion.

AFAICS, this forkfd-concept would fit nicely into the async_std library. Forkfd can be implemented on top of Posix API (usage of fork and pipe calls) and even more easily on top of Win32 process handles.

What do you think?

[ghost]

@frehberg Interesting, that sounds very reasonable to me, although I'm not super familiar with this area. :)

I wonder - how does your strategy differ from what was implemented in tokio-process?
https://github.com/alexcrichton/tokio-process

If your strategy is different, what are the pros/cons?

Thank you for chiming in and offering help, this is very helpful and much appreciated! <3

[frehberg]

@stjepang The tokio-process implementation (POSIX) is using the painfull signal-approach (as documented at the top of file). This solution requires a global child-queue and does not scale well. This solution might even interfere with signal handling in legacy C/C++ or for example Erlang apps.

In contrast, the forkfd-concept allows locality, it does not require a global child-process queue (optional), nor does it require signal handling,. And it would scale well.
The Pro is, that the monitoring of a child-process can be realized by a simple pipe and the corresponding file-descriptor async-event-handling, achieving locality, and even the ownership of a Child-items might be moved between threads (moving the FD between threads) The forkfd concept is portable (Qt-lib is the proof), it can be implemented on top of Posix-layer, or using platform specific solutions on BSD (pdfork-function) or on Windows (process handle).

A cons might be, that the forkfd-solution allocates an additional pipe-resource per sub-process (just on Posix-platform), which costs additional time when spawning, and the number of open-files of the parent process will grow faster. On BSD or Windows, no additional costs will arise. Another cons would be, that the forkfd-concept is introducing a clean owner-ship-model for child-processes, and this might conflict with the shared-ownership concept and application-wide signal-handling in present applications, as performed by tokio-process or other libs (for example a application-wide signal handler might grab/consume the exit status of the child-process)

The question is: for backward-compatibility to tokio-process etal., should a forkfd-solution deal with a shared child-process-queue, for example to store the exit-status of child-process in an application-wide fashion?

[frehberg]

I have some time to work on a demonstrator, or did you port the QProcess to Rust already?

[frehberg]

please see a first sketch of the proposed forkfd process-handle (under construction)
https://github.com/frehberg/rust/tree/process_handle

It is important that the feature will be available for the Posix platforms as well as VxWorks.
AFAICS, As the internal Windows-process API did already propose the function 'handle()', I reused this name instead of 'fd()'

[frehberg]

I filed a feature request rust-lang/rfcs#2817

[mmstick]

Linux -- as of 5.3 -- supports a "pidfd" concept, which would eliminate the need for creating pipes between processes. Unlike a traditional PID, a PID fd is guaranteed to always point to the original process which was spawned, and can be passed into poll() for tracking the exit status. So you can achieve the same as forkfd, but without the need for pipes. The forkfd approach would still be useful for Linux kernels before 5.3, however, and other UNIX-like platforms.

[mmstick]

Related: mio-pidfd

[yaahc]

I'm still following this but I think it should be unassigned from me because when i tried digging into it, it seemed a bit more complicated than I originally anticipated, and now that there's seemingly a lot of progress it makes sense for someone else to take over the issue, I'm still here and willing to help though.

[mmstick]

I can work on this. It's critical for the kind of work that I do on the Linux desktop. I've been stuck with using tokio for all of our software because of needing the process support more than anything else.

[mmstick]

I've created a pidfd crate, which provides a PidFd type, which implements std::future::Future using libc::poll. Any std::process::Child can be converted into a PidFd for awaiting process termination concurrently. You may also get the RawFd from the type, as it implements AsRawFd.

• Repository: https://github.com/pop-os/pidfd
• Crates: https://crates.io/crates/pidfd

Given that this only works for Linux, we would need solutions to other platforms, too.

[frehberg]

Please comment PR RFC process-handle-for-async
rust-lang/rfcs#2823

[KillTheMule]

Has there been any progress with this?

(I'm trying to evaluate current async support for std::process::Command for implementing neovim-lib in async fashion with subprocessing)

I've taken that up in nvim-rs. One component that's very much missing is connecting to a child process via async stdio.

[frehberg]

I am no longer searching for an abstraction working for all platforms, I am willing to implement invidual child-process handling for each platform. I did some recherché for Linux, and found an acceptable solution for Linux-2.6 upwards using "signalfd()", without the nead for so called death-pipes.
Using the signalfd() solution, we would get rid of the async SYS-V signal handlers.
Right now I try to figure out an architecture, how a generalized API might look, covering signalfd() (>Linux.2.6), pidfd(>Linux-5.3), HANDLE (Windows-NT), or pdfork(FreeBSD). I will ignore other platforms so far,

[mmstick]

For Linux >= 5.3, I've been successfully using pidfd in all Pop!_OS projects. It uses fd-reactor currently, which is a simple libc::poll-based reactor for registering and unregistering file descriptor interests in a single static global background thread. If async-std ever makes its internal reactor publicly accessible, I could have it use that, too.

[joshtriplett]

@frehberg signalfd helps, but you can only use signalfd to watch for SIGCHLD if you block SIGCHLD in the process, which means no possibility of cooperation with anything else in the process that wants SIGCHLD.

[dignifiedquire]

First simple implementation is up here: #723

[ghost]

This might be helpful for implementing async_std::process: https://github.com/stjepang/async-process

[dignifiedquire]

very cool @stjepang

[detly]

Please note that Linux 5.x is a long way away for some embedded systems (eg. any OpenWrt system), so if you don't want to lock them out you might want to consider something that can be used on Linux 4.x.

[detly]

(I suppose "do it in a thread" is something that can be used on Linux 4.x.)