process: do not poll join() in MultiprocessingProcesss

Based on the initial analysis of Esteve Varela Colominas, the polling
on join() with the fixed 100ms delay incurs a significant performance
penalty, especially for short-lived processes. And since portage is
prone to spawning many of those, the penalty adds up easily.

Instead of pooling proc.join() with a fixed 100ms delay, we now use
the blocking variant of join() started in an executor which we await
in the coroutine.

Before this change:

( cd lib; python3 -m timeit 'import portage.process; portage.process.spawn("true")' )
2 loops, best of 5: 104 msec per loop

After this change:

( cd lib; python3 -m timeit 'import portage.process; portage.process.spawn("true")' )
50 loops, best of 5: 4.45 msec per loop

Bug: https://bugs.gentoo.org/958635
Signed-off-by: Florian Schmaus <[email protected]>
Part-of: #1452
Closes: #1452
Signed-off-by: Sam James <[email protected]>

Author: Flowdalic

lib/portage/process.py

@@ -426,10 +426,6 @@ class MultiprocessingProcess(AbstractProcess):
     An object that wraps OS processes created by multiprocessing.Process.
     """
 
-    # Number of seconds between poll attempts for process exit status
-    # (after the sentinel has become ready).
-    _proc_join_interval = 0.1
-
     def __init__(self, proc: multiprocessing.Process):
         self._proc = proc
         self.pid = proc.pid
@@ -480,13 +476,7 @@ async def _proc_join(self):
             except ValueError:
                 pass
 
-        # Now that proc.sentinel is ready, poll until process exit
-        # status has become available.
-        while True:
-            proc.join(0)
-            if proc.exitcode is not None:
-                break
-            await asyncio.sleep(self._proc_join_interval, loop=loop)
+        await loop.run_in_executor(None, proc.join)
 
     def _proc_join_done(self, future):
         # The join task should never be cancelled, so let it raise