Skip to content

Shared subprocess impl#1235

Draft
graingert wants to merge 17 commits into
agronholm:masterfrom
graingert:shared-subprocess-impl
Draft

Shared subprocess impl#1235
graingert wants to merge 17 commits into
agronholm:masterfrom
graingert:shared-subprocess-impl

Conversation

@graingert

@graingert graingert commented Jul 19, 2026

Copy link
Copy Markdown
Collaborator

NOTE Erasing or replacing the contents of this template will result in your pull
request being summarily closed without consideration!

Changes

Fixes #783. Also resolves the asyncio/Trio inconsistencies reported in #828.

Moves the subprocess lifecycle logic (the Process object, spawning and reaping) into a
single, backend-agnostic implementation in anyio._core._subprocesses, built on
subprocess.Popen() plus a small set of backend primitives:

  • create_subprocess_stdin_pipe / create_subprocess_output_pipe
  • wait_for_child_exit

This removes the reliance on undocumented backend internals such as
StreamReader.set_exception().

Backends / platforms

  • asyncio uses the shared implementation on every platform. Pipes are created with
    loop.connect_read_pipe / loop.connect_write_pipe:
    • stdlib ProactorEventLoop (Windows default): overlapped IOCP pipes via PipeHandle;
      the child is waited on with proactor.wait_for_handle.
    • winloop (libuv): pipes backed by real overlapped file descriptors.
    • Windows SelectorEventLoop (which can't do overlapped pipe I/O): the pipes run on
      a background ProactorEventLoop/winloop thread (mirroring the existing selector-thread
      helper), and operations are proxied to it.
    • POSIX: os.pipe() + wait_readable/wait_writable.
    • On Windows, waiting for a child that has no _proactor (winloop / SelectorEventLoop)
      uses a loop-agnostic RegisterWaitForSingleObject helper (thread-pool wait delivered
      via call_soon_threadsafe, no Python thread tied up).
  • Trio uses the shared implementation on POSIX and keeps its native implementation on
    Windows.
  • Reaping on POSIX uses a pidfd on Linux, or waitid(WNOWAIT) / Popen.wait() in a worker
    thread elsewhere (os.waitid is unavailable on macOS before Python 3.13).

The Python 3.10 asyncio.Runner backport was moved out of the asyncio backend into
anyio._core._asyncio_runner so the proactor thread can reuse it.

Fixed inconsistencies (#828)

  • Process.wait() no longer waits for the standard streams to close, so it returns
    promptly once the process exits, even when a pipe is inherited by a grandchild process.
  • Process.returncode now polls the process on both backends, so it no longer returns a
    stale None after the process has exited.
  • Signalling an already-exited process via terminate() / kill() / send_signal() is
    now consistently a no-op instead of raising on asyncio.

A Windows SelectorEventLoop entry was added to the test matrix.

Checklist

If this is a user-facing code change, like a bugfix or a new feature, please ensure that
you've fulfilled the following conditions (where applicable):

  • You've added tests (in tests/) which would fail without your patch
  • You've updated the documentation (in docs/), in case of behavior changes or new
    features
  • You've added a new changelog entry (in docs/versionhistory.rst).

If this is a trivial change, like a typo fix or a code reformatting, then you can ignore
these instructions.

Updating the changelog

If there are no entries after the last release, use **UNRELEASED** as the version.
If, say, your patch fixes issue #123, the entry should look like this:

- Fix big bad boo-boo in task groups
  (`#123 <https://github.com/agronholm/anyio/issues/123>`_; PR by @yourgithubaccount)

If there's no issue linked, just link to your pull request instead by updating the
changelog after you've created the PR.

🤖 Generated with Claude Code

graingert and others added 12 commits July 19, 2026 13:44
Move the subprocess lifecycle logic (Process, spawning and reaping) into a
single implementation in anyio._core._subprocesses, built on subprocess.Popen()
plus a small set of backend primitives:

  * create_subprocess_stdin_pipe / create_subprocess_output_pipe
  * wait_for_child_exit

The asyncio backend implements the pipes with loop.connect_read_pipe /
connect_write_pipe (IOCP on Windows via the ProactorEventLoop) and waits on the
process handle via the proactor on Windows; on POSIX it reaps via pidfd/waitid.
The Trio backend uses raw fd pipe streams on POSIX and keeps its native
implementation on Windows. This removes the reliance on undocumented backend
internals (e.g. StreamReader.set_exception()).

This also resolves the asyncio/Trio inconsistencies from discussion agronholm#828:

  * Process.wait() no longer waits for the standard streams to close, so it
    returns promptly once the process exits (even with a pipe inherited by a
    grandchild)
  * Process.returncode now polls on both backends, so it no longer returns a
    stale None after exit
  * signalling an already-exited process is consistently a no-op

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LZMpdx7hNR89fwdmiMFmVJ
- macOS: os.waitid is unavailable before Python 3.13, so reap via a portable
  helper that uses waitid(WNOWAIT) where available and falls back to
  Popen.wait() otherwise (pidfd is still used on Linux)
- pyright: annotate wait_for_child_exit's parameter as subprocess.Popen[bytes]
  so AsyncBackend (and everything referencing it, e.g. EventLoopToken) stays
  type-complete
- Windows: use a duplex pipe for the child's stdin, since asyncio's write-pipe
  transport reads our end to detect closure and therefore needs read access

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LZMpdx7hNR89fwdmiMFmVJ
- Wait for child exit on Windows via RegisterWaitForSingleObject (a Windows
  thread-pool wait delivered through call_soon_threadsafe), which works on the
  stdlib ProactorEventLoop, winloop and SelectorEventLoop alike and doesn't tie
  up a Python thread. Replaces the ProactorEventLoop-only _proactor.wait_for_handle.
- Create the subprocess pipes appropriately for the running loop: the stdlib
  ProactorEventLoop takes a PipeHandle, while winloop (libuv) wants a file object
  backed by a real overlapped file descriptor, like uvloop on POSIX.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LZMpdx7hNR89fwdmiMFmVJ
PyPy (and other interpreters/older kernels) may not provide os.pidfd_open;
accessing it raised AttributeError rather than OSError, so it wasn't caught.
Fall back to the worker-thread reaping path when it's unavailable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LZMpdx7hNR89fwdmiMFmVJ
Use the native IOCP handle wait on the stdlib ProactorEventLoop, falling back to
the RegisterWaitForSingleObject helper only on loops without a proactor (winloop,
SelectorEventLoop). wait_for_pid only needs call_soon_threadsafe, so it works on
any of them.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LZMpdx7hNR89fwdmiMFmVJ
UnregisterWait() may return while a RegisterWaitForSingleObject callback is still
in flight, so on cancellation the callback could fire after the process handle
was closed and the ctypes trampoline collected. UnregisterWaitEx() with
INVALID_HANDLE_VALUE blocks until in-flight callbacks finish, making cleanup safe.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LZMpdx7hNR89fwdmiMFmVJ
The SelectorEventLoop can't do overlapped pipe I/O, so run the subprocess pipes on
a background ProactorEventLoop (or winloop) thread, mirroring the existing selector
thread helper (started lazily, shut down via threading._register_atexit). Pipe
operations are marshalled onto that loop with run_coroutine_threadsafe and awaited on
the caller's loop via wrap_future. Waiting for the child still uses the loop-agnostic
RegisterWaitForSingleObject helper. A Windows SelectorEventLoop entry is added to the
test matrix.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LZMpdx7hNR89fwdmiMFmVJ
- Move the Python 3.10 asyncio.Runner backport out of _backends/_asyncio.py into a
  standalone _core/_asyncio_runner.py (re-exporting the stdlib Runner on 3.11+).
- Drive the proactor thread's loop with asyncio.Runner (loop_factory) and a stop-event
  serve coroutine, so leftover tasks, async generators and the default executor are
  cleaned up and the loop is closed properly on shutdown.
- Give the subprocess pipe streams (and their SelectorEventLoop proxies) a synchronous
  _abort() so the process pool's forced shutdown can tear down worker transports at loop
  completion; the proxy schedules the close on the proactor loop.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LZMpdx7hNR89fwdmiMFmVJ
Covers the worker-thread reaping fallback (no os.pidfd_open, and no os.waitid),
the spawn-failure cleanup path, and a single PathLike command, which were
previously uncovered by the test suite.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LZMpdx7hNR89fwdmiMFmVJ
Comment thread docs/versionhistory.rst Outdated
Comment thread docs/versionhistory.rst Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors AnyIO’s subprocess support by moving process lifecycle management into a shared, backend-agnostic implementation in anyio._core._subprocesses, with backends providing only a small set of primitives for pipe creation and child-exit waiting. It also aims to eliminate asyncio/Trio behavioral inconsistencies discussed in #828 and expands Windows coverage to include asyncio’s SelectorEventLoop.

Changes:

  • Introduces a shared subprocess.Popen()-based Process implementation and POSIX child-exit waiting helpers (pidfd on Linux, worker-thread fallback elsewhere).
  • Updates asyncio and Trio backends to use the shared implementation where applicable, adding Windows-specific helpers (proactor thread + threadpool wait) for asyncio.
  • Adds/updates tests and changelog entries, including a Windows SelectorEventLoop test-matrix entry.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
tests/test_subprocesses.py Adds regression tests for returncode polling, signalling exited processes, and POSIX reaping fallbacks.
tests/conftest.py Adds Windows asyncio SelectorEventLoop coverage to the backend test matrix.
src/anyio/abc/_eventloop.py Extends backend interface with subprocess pipe creation and child-exit waiting primitives.
src/anyio/_core/_subprocesses.py Implements shared Process + spawn/reap logic using subprocess.Popen() and backend primitives.
src/anyio/_core/_asyncio_windows_process.py Adds loop-agnostic Windows child-exit waiting via RegisterWaitForSingleObject.
src/anyio/_core/_asyncio_runner.py Extracts/backports asyncio.Runner for Python 3.10 to support background-loop usage.
src/anyio/_core/_asyncio_proactor_thread.py Adds a long-lived background proactor loop for overlapped pipe I/O when main loop is a Windows selector loop.
src/anyio/_backends/_trio.py Switches Trio POSIX subprocesses to the shared implementation and adds pipe primitives.
src/anyio/_backends/_asyncio.py Switches asyncio subprocesses to the shared implementation; adds pipe primitives and new pipe stream implementations.
docs/versionhistory.rst Adds UNRELEASED changelog entries describing the subprocess refactor and behavior fixes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/anyio/_core/_subprocesses.py Outdated
Comment thread src/anyio/_backends/_asyncio.py Outdated
Comment thread src/anyio/_backends/_asyncio.py
Comment thread src/anyio/_backends/_asyncio.py
graingert and others added 3 commits July 19, 2026 15:44
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Comment thread docs/versionhistory.rst
Comment on lines +8 to +27
- Added support for running subprocesses on the asyncio ``SelectorEventLoop`` on Windows,
which does not support subprocesses natively; the pipe I/O is handled on a background
``ProactorEventLoop`` (or winloop) thread
(`#1235 <https://github.com/agronholm/anyio/pull/1235>`_; PR by @graingert)
- Changed subprocess handling to use a shared, backend-agnostic implementation built on
``subprocess.Popen()`` plus a small set of backend primitives (asynchronous pipes and
child-process reaping). This is used by the asyncio backend on all platforms and by the
Trio backend on POSIX (Trio keeps its native implementation on Windows), and removes the
reliance on undocumented backend internals
(`#783 <https://github.com/agronholm/anyio/issues/783>`_; PR by @graingert)
- Fixed several inconsistencies between the asyncio and Trio backends when working with
subprocesses (`#828 <https://github.com/agronholm/anyio/discussions/828>`_; PR by
@graingert):

* ``Process.wait()`` no longer waits for the standard streams to close, so it returns
promptly once the process exits, even when a pipe is inherited by a grandchild process
* ``Process.returncode`` now polls the process on both backends, so it no longer returns
a stale ``None`` after the process has exited
* Signalling an already-exited process via ``terminate()``, ``kill()`` or
``send_signal()`` is now consistently a no-op instead of raising on asyncio

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be severely trimmed down before we merge.

@agronholm

Copy link
Copy Markdown
Owner

There's so much low-level code added here that it would be hard to justify merging all of it.

@tapetersen tapetersen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looked through and couldn't see any obvious bugs but it's a lot of fiddly code.

raise BrokenResourceError from exc
else:
if data:
return data

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't hit a checkpoint if the read is successful.
Is that intentional for speed? (also not sure how committed to following trio's guarantees regarding that)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is certainly missing the checkpoint

Comment on lines +532 to +533
Exercise the worker-thread reaping fallback used when ``os.pidfd_open`` is unavailable
(e.g. PyPy or kernels older than 5.3) and, in turn, when ``os.waitid`` is unavailable

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually a lot more common than that as the standalone python builds from uv don't include it

$ uv run --isolated --python 3.14 python -c "import os; print(hasattr(os, 'pidfd_open'))"
False

See astral-sh/python-build-standalone#193

(Just a note to as it surprised me)

Comment on lines +555 to +557
await open_process(
[os.path.join(os.getcwd(), "nonexistent-anyio-test-executable")]
)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't check for any pipe-cleanup as mentioned but that may be implicit (if so just resolve)

@tapetersen

Copy link
Copy Markdown
Collaborator

For what it's worth I at least got the Process.wait() waiting on streams and not just the process fixed upstream python/cpython#119710 . Looks like it may be backported to 3.13-

(We should fix it work-around it anyway but it al least confirms it as a bug and not intended behavior).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Look into creating a shared subprocess implementation between backends

4 participants