gh-96471: Add asyncio queue shutdown #104228

EpicWink · 2023-05-06T04:54:02Z

asyncio-only changes from #102499 (which supercedes #96474), updated to match the API introduced by #104750

Issue: Queue shutdown #96471

📚 Documentation preview 📚: https://cpython-previews--104228.org.readthedocs.build/

* Include docs

* Queue state enum members are capitalised * Termination state in str/repr * Include raised exception in docstrings * Factor out queue-state checks and updates to methods * Logic fixes in get_nowait and shutdown * Handle queue shutdown in task_done and join * Updated tests * Document feature added in 3.13

EpicWink · 2023-05-06T04:54:17Z

@willingc ready for review

gvanrossum · 2023-05-09T17:56:46Z

asyncio-only changes from #102499 (on top of #96474)

I assume the "on top of ..." part is obsolete, since that PR was closed without merging. I'll add @willingc as a reviewer per your request.

gvanrossum · 2024-02-10T05:00:20Z

@EpicWink Could you merge main?

EpicWink · 2024-02-20T11:29:59Z

Could you merge main?

Done, but like #104750, we may want to modify the implementation of immediate=True to simply consume the queue rather than having a separate state.

gvanrossum · 2024-02-20T15:21:50Z

Eh, @EpicWink, did you see the hubbub about the hanging test in your previous PR? The test was disabled because it kept hanging in CI on various platforms. Could you look into what's wrong with it? (At least the feature wasn't reverted.)

EpicWink · 2024-02-20T23:27:52Z

Eh, @EpicWink, did you see the hubbub about the hanging test in your previous PR? The test was disabled because it kept hanging in CI on various platforms. Could you look into what's wrong with it?

I did see the follow-on issue and pull-requests. Yves and I are working on fixing it (you can follow the changes here: main...EpicWink:cpython:fix-thread-queue-shutdown-test)

gvanrossum · 2024-02-21T00:09:49Z

Awesome! (An acknowledgement that you were working on it would have lessened my stress. :-)

gvanrossum · 2024-02-22T00:59:55Z

Looking at this with an eye towards review, I realized that the threaded-queue implementation doesn't appear to match its documentation: It appears that q.join() does not raise ShutDown when the queue is shutdown immediately. The tests don't seem to be testing for this either. (Come to think of it, the test never seems to take the except self.queue.ShutDown path -- if I put a breakpoint there it never gets hit during any of the tests. The test logic is pretty convoluted, which is my only excuse for not having caught this in the review of gh-104750.)

EpicWink · 2024-02-22T01:04:29Z

Now that the immediate-consume implementation of the threading queue shutdown has been accepted, I'm going to do the same here and for multiprocessing. I'll personally rewrite the tests to be more readable and obvious. Laurie

…

________________________________ From: Guido van Rossum ***@***.***> Sent: Thursday, February 22, 2024 11:00:09 AM To: python/cpython ***@***.***> Cc: Laurie O ***@***.***>; Mention ***@***.***> Subject: Re: [python/cpython] gh-96471: Add asyncio queue shutdown (PR #104228) Looking at this with an eye towards review, I realized that the threaded-queue implementation doesn't appear to match its documentation: It appears that q.join() does not raise ShutDown when the queue is shutdown immediately. The tests don't seem to be testing for this either. (Come to think of it, the test never seems to take the except self.queue.ShutDown path -- if I put a breakpoint there it never gets hit during any of the tests. The test logic is pretty convoluted, which is my only excuse for not having caught this in the review of gh-104750<#104750>.) — Reply to this email directly, view it on GitHub<#104228 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AF72GRNKMPJRCRIJHOB4NUDYU2KBTAVCNFSM6AAAAAAXX3PWIGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJYGQ3TENZYGQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

gvanrossum · 2024-02-22T01:08:52Z

Now that the immediate-consume implementation of the threading queue shutdown has been accepted, I'm going to do the same here and for multiprocessing. I'll personally rewrite the tests to be more readable and obvious.

And what about queue.Queue.join()? Should it raise after shutdown(immediate=True)? Either way some changes are needed -- either in the implementation or in the docs. And the tests there too (what you wrote above appears to apply to the asyncio and multiprocessing versions only.)

gvanrossum

I'll wait reviewing until you've rewritten shutdown, but here's one doc markup nit.

Also, please mark PRs that aren't ready for review as Draft.

Doc/library/asyncio-queue.rst

EpicWink · 2024-02-22T03:41:38Z

And what about queue.Queue.join()? Should it raise after shutdown(immediate=True)? Either way some changes are needed -- either in the implementation or in the docs. And the tests there too (what you wrote above appears to apply to the asyncio and multiprocessing versions only.)

The behaviour should (and I intend to implement it to) be the same between the threading, multiprocessing and asyncio queues

YvesDup · 2024-02-22T09:32:12Z

By the way, should task_done() raise an exception after shutdown or not ?
update; see https://docs.python.org/3.13/library/queue.html#queue.Queue.task_done

EpicWink · 2024-02-25T12:14:17Z

By the way, should task_done() raise an exception after shutdown or not ?

@YvesDup No (except the usual ValueError); see #115838

EpicWink · 2024-04-04T01:42:35Z

I want to trust that you two have done the right thing there

@gvanrossum please don't trust, have you seen xz-utils's backdoor?

but not push until the big test run completes? (Buildbot tests can take many hours, sorry.)

Oops, sorry. I forgot to hold off my most recent push

ALso, why is the DO-NOT-MERGE label set? Is it because of #104228?

No, I think it was because I hadn't updated the implementation to match the new agreed-upon behaviour (this PR was in draft at that time anyway, so I don't think the label was needed).

wondering why you always wake up all getters -- and came to the conclusion that it is needed

I should probably add a comment

EpicWink · 2024-04-04T02:09:20Z

there is the same bug on threading queue shutdown. I will open an issue.

@YvesDup I've created issue #117531, and fix in PR #117532

YvesDup · 2024-04-04T06:36:27Z

Lib/test/test_asyncio/test_queues.py

+        # Setup queue with 2 items (1 retrieved), and a join() task
+        q = self.q_class()
+        loop = asyncio.get_running_loop()
+        q.put_nowait("data")


I suggest:

q.put_nowait("data1") q.put_nowait("data2")

YvesDup · 2024-04-04T06:36:29Z

Lib/test/test_asyncio/test_queues.py

+        q.put_nowait("data")
+        q.put_nowait("data")
+        join_task = loop.create_task(q.join())
+        self.assertEqual(await q.get(), "data")


and testing on "data1".

Except when q is an instance of LifoQueue, when the value should be "data2". Is it necessary to test queue ordering in the shutdown tests?

I forgot the LifoQueue class. Is this worth commenting on?

I have implemented your suggestion, but I have not committed. I am considering it. I don't think it's necessary.

The test case (thus including LifoQueue) is implicit. A comment would be enough.

YvesDup · 2024-04-04T06:46:40Z

there is the same bug on threading queue shutdown. I will open an issue.

@YvesDup I've created issue #117531, and fix in PR #117532

I have seen and commented

Lib/asyncio/queues.py

gvanrossum

Yup! This version looks great. Let me know if you agree -- I've removed the DO-NOT-MERGE label, but I'll wait until you and @YvesDup are happy too.

EpicWink · 2024-04-06T06:39:51Z

I think it's good to go

wondering why you always wake up all getters -- and came to the conclusion that it is needed

I should probably add a comment

I could do this, but I don't think it's worth revoking the PR's approval

YvesDup · 2024-04-06T17:46:17Z

Doc/library/asyncio-queue.rst

+      All blocked callers of :meth:`~Queue.put` will be unblocked. If
+      *immediate* is true, also unblock callers of :meth:`~Queue.get`
+      and :meth:`~Queue.join`.
+


Sorry but I have a doubt, shouldn't this documentation block be rather:

All blocked callers of :meth:`~Queue.put` and :meth:`~Queue.get` will be unblocked. If *immediate* is true, also unblock callers of :meth:`~Queue.join`.

In event of change, the docstring of the shutdown method must be updated.

join callers aren't necessarily even unblocked anyway, if consumers are processing any items. I should probably say that a task is marked as done for each item in the queue if immediate shutdown.

Also, I think the threading queue docs are the same.

a task is marked as done for each item in the queue if immediate shutdown.

It's very precise, better.

Also, I think the threading queue docs are the same.

Yes, I commented here so as not to forget (see #117532 (comment)).

English is your native language, I think it'is best if you update documentations and docstrings.

Update: but I can create the follow-up PR.

I've made a PR: #117621

YvesDup · 2024-04-06T17:47:04Z

Lib/asyncio/queues.py

+
+        All blocked callers of put() will be unblocked, and also get()
+        and join() if 'immediate'.
+        """


Docstring to modify depending of agree/disagree on my first remark about `blocked callers'.
Changes could be:

All blocked callers of put() and get() will be unblocked, and also join() if 'immediate

YvesDup · 2024-04-06T17:47:39Z

Doc/whatsnew/3.13.rst

+* Add :meth:`asyncio.Queue.shutdown` (along with
+  :exc:`asyncio.QueueShutDown`) for queue termination.
+  (Contributed by Laurie Opperman in :gh:`104228`.)
+


We've worked well together, I'd be nice if you'd mention me.
:-)

gvanrossum · 2024-04-06T18:37:00Z

Sorry for merging prematurely. @EpicWink Can you prepare a follow-up PR? I will merge it as soon as @YvesDup approves it. (Or, alternatively, @YvesDup can make a PR and I will merge when @EpicWink approves.)

EpicWink · 2024-04-08T05:30:31Z

@gvanrossum see #117621

Co-authored-by: Duprat <[email protected]>

john-parton · 2024-08-27T04:09:38Z

I'd like to add that the shutdown() method and task groups makes certain patterns really easy to express. I was struggling with how to properly make a generic async map function with a fixed number of workers (mostly as an exercise), but when I saw the new shutdown method, it came together really easily:

async def async_map[T, R](
    func: Callable[[T], Awaitable[R]],
    iterable: AsyncIterable[T],
    *,
    limit: int,
    maxsize: int = -1,
) -> AsyncIterator[R]:
    if maxsize < 0:
        maxsize = limit

    arguments_queue = Queue[T](maxsize=maxsize)
    results_queue = Queue[R](maxsize=maxsize)

    async def drain():
        async for argument in iterable:
            await arguments_queue.put(argument)
        arguments_queue.shutdown(immediate=False)

    async def worker():
        while True:
            try:
                argument = await arguments_queue.get()
            except QueueShutDown:
                break
            await results_queue.put(await func(argument))

    async def background():
        async with asyncio.TaskGroup() as background_task_group:
            background_task_group.create_task(drain())
            for _ in range(limit):
                background_task_group.create_task(worker())
        results_queue.shutdown(immediate=False)
 
    async with asyncio.TaskGroup() as task_group:
        task_group.create_task(background())

        while True:
            try:
                yield await results_queue.get()
            except QueueShutDown:
                break

I believe this has a common issue that a lot of async generators do, in that if you don't consume the entire generator, it will still continue processing and end up with a lot of futures never awaited.

I know there's some backports. Is there a backports package for asyncio stuff already?

EpicWink and others added 2 commits May 6, 2023 14:50

Add asyncio queue shutdown

440a702

* Include docs

EpicWink requested a review from rhettinger as a code owner May 6, 2023 04:54

bedevere-bot mentioned this pull request May 6, 2023

Queue shutdown #96471

Open

bedevere-bot added the awaiting review label May 6, 2023

📜🤖 Added by blurb_it.

e5951ac

This was referenced May 6, 2023

gh-96471: Add queue shutdown #96474

Closed

gh-96471: Add queue shutdown, next step. #102499

Closed

gvanrossum added the topic-asyncio label May 6, 2023

rhettinger removed their request for review May 6, 2023 19:16

gvanrossum requested a review from willingc May 9, 2023 17:56

EpicWink added 2 commits February 20, 2024 20:58

Merge remote-tracking branch 'upstream/main' into asyncio-queue-shutdown

a72aedd

Add references in docs and news entry

d5e925d

gvanrossum mentioned this pull request Feb 22, 2024

gh-96471: Add threading queue shutdown #104750

Merged

gvanrossum reviewed Feb 22, 2024

View reviewed changes

Doc/library/asyncio-queue.rst Outdated Show resolved Hide resolved

EpicWink marked this pull request as draft February 22, 2024 01:14

bedevere-app bot removed the awaiting review label Feb 22, 2024

EpicWink added 7 commits April 4, 2024 11:20

Simplify shutdown-check in put and get

aef4063

Format shutdown docstring

d49c6dd

Check for 0 unfinised tasks in shutdown

5a435a6

Use asyncio.sleep to run other tasks

c8db40e

Use public method to shut down queue in format test

ca01ee1

Only start queue join after shutdown in test

b02c4dd

Test join before failing task-done

8deca77

EpicWink mentioned this pull request Apr 4, 2024

Queue non-immediate shutdown doesn't unblock getters #117531

Closed

YvesDup reviewed Apr 4, 2024

View reviewed changes

Lib/asyncio/queues.py Show resolved Hide resolved

gvanrossum removed the DO-NOT-MERGE label Apr 6, 2024

gvanrossum approved these changes Apr 6, 2024

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting review labels Apr 6, 2024

gvanrossum merged commit df4d84c into python:main Apr 6, 2024

bedevere-app bot removed the awaiting merge label Apr 6, 2024

YvesDup reviewed Apr 6, 2024

View reviewed changes

EpicWink deleted the asyncio-queue-shutdown branch April 8, 2024 00:49

EpicWink mentioned this pull request Apr 8, 2024

gh-96471: Correct documentation for asyncio queue shutdown #117621

Merged

diegorusso pushed a commit to diegorusso/cpython that referenced this pull request Apr 17, 2024

pythongh-96471: Add asyncio queue shutdown (python#104228)

21be21c

Co-authored-by: Duprat <[email protected]>

Uh oh!

gh-96471: Add asyncio queue shutdown #104228

gh-96471: Add asyncio queue shutdown #104228

Uh oh!

Conversation

EpicWink commented May 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EpicWink commented May 6, 2023

Uh oh!

gvanrossum commented May 9, 2023

Uh oh!

gvanrossum commented Feb 10, 2024

Uh oh!

EpicWink commented Feb 20, 2024

Uh oh!

gvanrossum commented Feb 20, 2024

Uh oh!

EpicWink commented Feb 20, 2024

Uh oh!

gvanrossum commented Feb 21, 2024

Uh oh!

gvanrossum commented Feb 22, 2024

Uh oh!

EpicWink commented Feb 22, 2024 via email

Uh oh!

gvanrossum commented Feb 22, 2024

Uh oh!

gvanrossum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

EpicWink commented Feb 22, 2024

Uh oh!

YvesDup commented Feb 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EpicWink commented Feb 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EpicWink commented Apr 4, 2024

Uh oh!

EpicWink commented Apr 4, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

EpicWink Apr 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

YvesDup commented Apr 4, 2024

Uh oh!

Uh oh!

gvanrossum left a comment

Choose a reason for hiding this comment

Uh oh!

EpicWink commented Apr 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

YvesDup Apr 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

EpicWink commented May 6, 2023 •

edited

Loading

YvesDup commented Feb 22, 2024 •

edited

Loading

EpicWink commented Feb 25, 2024 •

edited

Loading

EpicWink Apr 4, 2024 •

edited

Loading

EpicWink commented Apr 6, 2024 •

edited

Loading

YvesDup Apr 7, 2024 •

edited

Loading

john-parton commented Aug 27, 2024 •

edited

Loading