Shared subprocess impl#1235
Conversation
Move the subprocess lifecycle logic (Process, spawning and reaping) into a single implementation in anyio._core._subprocesses, built on subprocess.Popen() plus a small set of backend primitives: * create_subprocess_stdin_pipe / create_subprocess_output_pipe * wait_for_child_exit The asyncio backend implements the pipes with loop.connect_read_pipe / connect_write_pipe (IOCP on Windows via the ProactorEventLoop) and waits on the process handle via the proactor on Windows; on POSIX it reaps via pidfd/waitid. The Trio backend uses raw fd pipe streams on POSIX and keeps its native implementation on Windows. This removes the reliance on undocumented backend internals (e.g. StreamReader.set_exception()). This also resolves the asyncio/Trio inconsistencies from discussion agronholm#828: * Process.wait() no longer waits for the standard streams to close, so it returns promptly once the process exits (even with a pipe inherited by a grandchild) * Process.returncode now polls on both backends, so it no longer returns a stale None after exit * signalling an already-exited process is consistently a no-op Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LZMpdx7hNR89fwdmiMFmVJ
- macOS: os.waitid is unavailable before Python 3.13, so reap via a portable helper that uses waitid(WNOWAIT) where available and falls back to Popen.wait() otherwise (pidfd is still used on Linux) - pyright: annotate wait_for_child_exit's parameter as subprocess.Popen[bytes] so AsyncBackend (and everything referencing it, e.g. EventLoopToken) stays type-complete - Windows: use a duplex pipe for the child's stdin, since asyncio's write-pipe transport reads our end to detect closure and therefore needs read access Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LZMpdx7hNR89fwdmiMFmVJ
- Wait for child exit on Windows via RegisterWaitForSingleObject (a Windows thread-pool wait delivered through call_soon_threadsafe), which works on the stdlib ProactorEventLoop, winloop and SelectorEventLoop alike and doesn't tie up a Python thread. Replaces the ProactorEventLoop-only _proactor.wait_for_handle. - Create the subprocess pipes appropriately for the running loop: the stdlib ProactorEventLoop takes a PipeHandle, while winloop (libuv) wants a file object backed by a real overlapped file descriptor, like uvloop on POSIX. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LZMpdx7hNR89fwdmiMFmVJ
PyPy (and other interpreters/older kernels) may not provide os.pidfd_open; accessing it raised AttributeError rather than OSError, so it wasn't caught. Fall back to the worker-thread reaping path when it's unavailable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LZMpdx7hNR89fwdmiMFmVJ
Use the native IOCP handle wait on the stdlib ProactorEventLoop, falling back to the RegisterWaitForSingleObject helper only on loops without a proactor (winloop, SelectorEventLoop). wait_for_pid only needs call_soon_threadsafe, so it works on any of them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LZMpdx7hNR89fwdmiMFmVJ
UnregisterWait() may return while a RegisterWaitForSingleObject callback is still in flight, so on cancellation the callback could fire after the process handle was closed and the ctypes trampoline collected. UnregisterWaitEx() with INVALID_HANDLE_VALUE blocks until in-flight callbacks finish, making cleanup safe. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LZMpdx7hNR89fwdmiMFmVJ
The SelectorEventLoop can't do overlapped pipe I/O, so run the subprocess pipes on a background ProactorEventLoop (or winloop) thread, mirroring the existing selector thread helper (started lazily, shut down via threading._register_atexit). Pipe operations are marshalled onto that loop with run_coroutine_threadsafe and awaited on the caller's loop via wrap_future. Waiting for the child still uses the loop-agnostic RegisterWaitForSingleObject helper. A Windows SelectorEventLoop entry is added to the test matrix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LZMpdx7hNR89fwdmiMFmVJ
- Move the Python 3.10 asyncio.Runner backport out of _backends/_asyncio.py into a standalone _core/_asyncio_runner.py (re-exporting the stdlib Runner on 3.11+). - Drive the proactor thread's loop with asyncio.Runner (loop_factory) and a stop-event serve coroutine, so leftover tasks, async generators and the default executor are cleaned up and the loop is closed properly on shutdown. - Give the subprocess pipe streams (and their SelectorEventLoop proxies) a synchronous _abort() so the process pool's forced shutdown can tear down worker transports at loop completion; the proxy schedules the close on the proactor loop. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LZMpdx7hNR89fwdmiMFmVJ
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LZMpdx7hNR89fwdmiMFmVJ
Covers the worker-thread reaping fallback (no os.pidfd_open, and no os.waitid), the spawn-failure cleanup path, and a single PathLike command, which were previously uncovered by the test suite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LZMpdx7hNR89fwdmiMFmVJ
There was a problem hiding this comment.
Pull request overview
This PR refactors AnyIO’s subprocess support by moving process lifecycle management into a shared, backend-agnostic implementation in anyio._core._subprocesses, with backends providing only a small set of primitives for pipe creation and child-exit waiting. It also aims to eliminate asyncio/Trio behavioral inconsistencies discussed in #828 and expands Windows coverage to include asyncio’s SelectorEventLoop.
Changes:
- Introduces a shared
subprocess.Popen()-basedProcessimplementation and POSIX child-exit waiting helpers (pidfd on Linux, worker-thread fallback elsewhere). - Updates asyncio and Trio backends to use the shared implementation where applicable, adding Windows-specific helpers (proactor thread + threadpool wait) for asyncio.
- Adds/updates tests and changelog entries, including a Windows
SelectorEventLooptest-matrix entry.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_subprocesses.py | Adds regression tests for returncode polling, signalling exited processes, and POSIX reaping fallbacks. |
| tests/conftest.py | Adds Windows asyncio SelectorEventLoop coverage to the backend test matrix. |
| src/anyio/abc/_eventloop.py | Extends backend interface with subprocess pipe creation and child-exit waiting primitives. |
| src/anyio/_core/_subprocesses.py | Implements shared Process + spawn/reap logic using subprocess.Popen() and backend primitives. |
| src/anyio/_core/_asyncio_windows_process.py | Adds loop-agnostic Windows child-exit waiting via RegisterWaitForSingleObject. |
| src/anyio/_core/_asyncio_runner.py | Extracts/backports asyncio.Runner for Python 3.10 to support background-loop usage. |
| src/anyio/_core/_asyncio_proactor_thread.py | Adds a long-lived background proactor loop for overlapped pipe I/O when main loop is a Windows selector loop. |
| src/anyio/_backends/_trio.py | Switches Trio POSIX subprocesses to the shared implementation and adds pipe primitives. |
| src/anyio/_backends/_asyncio.py | Switches asyncio subprocesses to the shared implementation; adds pipe primitives and new pipe stream implementations. |
| docs/versionhistory.rst | Adds UNRELEASED changelog entries describing the subprocess refactor and behavior fixes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
| - Added support for running subprocesses on the asyncio ``SelectorEventLoop`` on Windows, | ||
| which does not support subprocesses natively; the pipe I/O is handled on a background | ||
| ``ProactorEventLoop`` (or winloop) thread | ||
| (`#1235 <https://github.com/agronholm/anyio/pull/1235>`_; PR by @graingert) | ||
| - Changed subprocess handling to use a shared, backend-agnostic implementation built on | ||
| ``subprocess.Popen()`` plus a small set of backend primitives (asynchronous pipes and | ||
| child-process reaping). This is used by the asyncio backend on all platforms and by the | ||
| Trio backend on POSIX (Trio keeps its native implementation on Windows), and removes the | ||
| reliance on undocumented backend internals | ||
| (`#783 <https://github.com/agronholm/anyio/issues/783>`_; PR by @graingert) | ||
| - Fixed several inconsistencies between the asyncio and Trio backends when working with | ||
| subprocesses (`#828 <https://github.com/agronholm/anyio/discussions/828>`_; PR by | ||
| @graingert): | ||
|
|
||
| * ``Process.wait()`` no longer waits for the standard streams to close, so it returns | ||
| promptly once the process exits, even when a pipe is inherited by a grandchild process | ||
| * ``Process.returncode`` now polls the process on both backends, so it no longer returns | ||
| a stale ``None`` after the process has exited | ||
| * Signalling an already-exited process via ``terminate()``, ``kill()`` or | ||
| ``send_signal()`` is now consistently a no-op instead of raising on asyncio |
There was a problem hiding this comment.
This needs to be severely trimmed down before we merge.
|
There's so much low-level code added here that it would be hard to justify merging all of it. |
tapetersen
left a comment
There was a problem hiding this comment.
Looked through and couldn't see any obvious bugs but it's a lot of fiddly code.
| raise BrokenResourceError from exc | ||
| else: | ||
| if data: | ||
| return data |
There was a problem hiding this comment.
This doesn't hit a checkpoint if the read is successful.
Is that intentional for speed? (also not sure how committed to following trio's guarantees regarding that)
There was a problem hiding this comment.
This is certainly missing the checkpoint
| Exercise the worker-thread reaping fallback used when ``os.pidfd_open`` is unavailable | ||
| (e.g. PyPy or kernels older than 5.3) and, in turn, when ``os.waitid`` is unavailable |
There was a problem hiding this comment.
This is actually a lot more common than that as the standalone python builds from uv don't include it
$ uv run --isolated --python 3.14 python -c "import os; print(hasattr(os, 'pidfd_open'))"
False
See astral-sh/python-build-standalone#193
(Just a note to as it surprised me)
| await open_process( | ||
| [os.path.join(os.getcwd(), "nonexistent-anyio-test-executable")] | ||
| ) |
There was a problem hiding this comment.
It doesn't check for any pipe-cleanup as mentioned but that may be implicit (if so just resolve)
|
For what it's worth I at least got the (We should fix it work-around it anyway but it al least confirms it as a bug and not intended behavior). |
NOTE Erasing or replacing the contents of this template will result in your pull
request being summarily closed without consideration!
Changes
Fixes #783. Also resolves the asyncio/Trio inconsistencies reported in #828.
Moves the subprocess lifecycle logic (the
Processobject, spawning and reaping) into asingle, backend-agnostic implementation in
anyio._core._subprocesses, built onsubprocess.Popen()plus a small set of backend primitives:create_subprocess_stdin_pipe/create_subprocess_output_pipewait_for_child_exitThis removes the reliance on undocumented backend internals such as
StreamReader.set_exception().Backends / platforms
loop.connect_read_pipe/loop.connect_write_pipe:ProactorEventLoop(Windows default): overlapped IOCP pipes viaPipeHandle;the child is waited on with
proactor.wait_for_handle.SelectorEventLoop(which can't do overlapped pipe I/O): the pipes run ona background
ProactorEventLoop/winloop thread (mirroring the existing selector-threadhelper), and operations are proxied to it.
os.pipe()+wait_readable/wait_writable._proactor(winloop / SelectorEventLoop)uses a loop-agnostic
RegisterWaitForSingleObjecthelper (thread-pool wait deliveredvia
call_soon_threadsafe, no Python thread tied up).Windows.
waitid(WNOWAIT)/Popen.wait()in a workerthread elsewhere (
os.waitidis unavailable on macOS before Python 3.13).The Python 3.10
asyncio.Runnerbackport was moved out of the asyncio backend intoanyio._core._asyncio_runnerso the proactor thread can reuse it.Fixed inconsistencies (#828)
Process.wait()no longer waits for the standard streams to close, so it returnspromptly once the process exits, even when a pipe is inherited by a grandchild process.
Process.returncodenow polls the process on both backends, so it no longer returns astale
Noneafter the process has exited.terminate()/kill()/send_signal()isnow consistently a no-op instead of raising on asyncio.
A Windows
SelectorEventLoopentry was added to the test matrix.Checklist
If this is a user-facing code change, like a bugfix or a new feature, please ensure that
you've fulfilled the following conditions (where applicable):
tests/) which would fail without your patchdocs/), in case of behavior changes or newfeatures
docs/versionhistory.rst).If this is a trivial change, like a typo fix or a code reformatting, then you can ignore
these instructions.
Updating the changelog
If there are no entries after the last release, use
**UNRELEASED**as the version.If, say, your patch fixes issue #123, the entry should look like this:
If there's no issue linked, just link to your pull request instead by updating the
changelog after you've created the PR.
🤖 Generated with Claude Code