gh-143055: Implementation of PEP 798 (#143056)

Co-authored-by: Jelle Zijlstra <[email protected]>

Author: adqm

Doc/reference/expressions.rst

@@ -266,17 +266,19 @@ called "displays", each of them in two flavors:
 Common syntax elements for comprehensions are:
 
 .. productionlist:: python-grammar
-   comprehension: `assignment_expression` `comp_for`
+   comprehension: `flexible_expression` `comp_for`
    comp_for: ["async"] "for" `target_list` "in" `or_test` [`comp_iter`]
    comp_iter: `comp_for` | `comp_if`
    comp_if: "if" `or_test` [`comp_iter`]
 
 The comprehension consists of a single expression followed by at least one
-:keyword:`!for` clause and zero or more :keyword:`!for` or :keyword:`!if` clauses.
-In this case, the elements of the new container are those that would be produced
-by considering each of the :keyword:`!for` or :keyword:`!if` clauses a block,
-nesting from left to right, and evaluating the expression to produce an element
-each time the innermost block is reached.
+:keyword:`!for` clause and zero or more :keyword:`!for` or :keyword:`!if`
+clauses.  In this case, the elements of the new container are those that would
+be produced by considering each of the :keyword:`!for` or :keyword:`!if`
+clauses a block, nesting from left to right, and evaluating the expression to
+produce an element each time the innermost block is reached.  If the expression
+is starred, the result will instead be unpacked to produce zero or more
+elements.
 
 However, aside from the iterable expression in the leftmost :keyword:`!for` clause,
 the comprehension is executed in a separate implicitly nested scope. This ensures
@@ -321,6 +323,9 @@ See also :pep:`530`.
    asynchronous functions. Outer comprehensions implicitly become
    asynchronous.
 
+.. versionchanged:: next
+   Unpacking with the ``*`` operator is now allowed in the expression.
+
 
 .. _lists:
 
@@ -396,8 +401,8 @@ enclosed in curly braces:
 .. productionlist:: python-grammar
    dict_display: "{" [`dict_item_list` | `dict_comprehension`] "}"
    dict_item_list: `dict_item` ("," `dict_item`)* [","]
+   dict_comprehension: `dict_item` `comp_for`
    dict_item: `expression` ":" `expression` | "**" `or_expr`
-   dict_comprehension: `expression` ":" `expression` `comp_for`
 
 A dictionary display yields a new dictionary object.
 
@@ -419,10 +424,21 @@ earlier dict items and earlier dictionary unpackings.
 .. versionadded:: 3.5
    Unpacking into dictionary displays, originally proposed by :pep:`448`.
 
-A dict comprehension, in contrast to list and set comprehensions, needs two
-expressions separated with a colon followed by the usual "for" and "if" clauses.
-When the comprehension is run, the resulting key and value elements are inserted
-in the new dictionary in the order they are produced.
+A dict comprehension may take one of two forms:
+
+- The first form  uses two expressions separated with a colon followed by the
+  usual "for" and "if" clauses.  When the comprehension is run, the resulting
+  key and value elements are inserted in the new dictionary in the order they
+  are produced.
+
+- The second form uses a single expression prefixed by the ``**`` dictionary
+  unpacking operator followed by the usual "for" and "if" clauses.  When the
+  comprehension is evaluated, the expression is evaluated and then unpacked,
+  inserting zero or more key/value pairs into the new dictionary.
+
+Both forms of dictionary comprehension retain the property that if the same key
+is specified multiple times, the associated value in the resulting dictionary
+will be the last one specified.
 
 .. index:: pair: immutable; object
            hashable
@@ -439,6 +455,8 @@ prevails.
    the key.  Starting with 3.8, the key is evaluated before the value, as
    proposed by :pep:`572`.
 
+.. versionchanged:: next
+   Unpacking with the ``**`` operator is now allowed in dictionary comprehensions.
 
 .. _genexpr:
 
@@ -453,7 +471,7 @@ Generator expressions
 A generator expression is a compact generator notation in parentheses:
 
 .. productionlist:: python-grammar
-   generator_expression: "(" `expression` `comp_for` ")"
+   generator_expression: "(" `flexible_expression` `comp_for` ")"
 
 A generator expression yields a new generator object.  Its syntax is the same as
 for comprehensions, except that it is enclosed in parentheses instead of

Doc/tutorial/classes.rst

@@ -929,6 +929,25 @@ Examples::
    >>> list(data[i] for i in range(len(data)-1, -1, -1))
    ['f', 'l', 'o', 'g']
 
+   >>> x = [[1,2,3], [], [4, 5]]
+   >>> g = (*i for i in x)
+   >>> list(g)
+   [1, 2, 3, 4, 5]
+
+In most cases, generator expressions must be wrapped in parentheses.  As a
+special case, however, when provided as the sole argument to a function (as in
+the examples involving ``sum``, ``set``, ``max``, and ``list`` above), the
+generator expression does not need to be wrapped in an additional set of
+parentheses.  That is to say, the following two pieces of code are semantically
+equivalent::
+
+   >>> f(x for x in y)
+   >>> f((x for x in y))
+
+as are the following::
+
+   >>> f(*x for x in y)
+   >>> f((*x for x in y))
 
 
 .. rubric:: Footnotes

Doc/tutorial/datastructures.rst

@@ -333,6 +333,47 @@ The :func:`zip` function would do a great job for this use case::
 
 See :ref:`tut-unpacking-arguments` for details on the asterisk in this line.
 
+Unpacking in Lists and List Comprehensions
+------------------------------------------
+
+The section on :ref:`tut-unpacking-arguments` describes the use of ``*`` to
+"unpack" the elements of an iterable object, providing each one seperately as
+an argument to a function.  Unpacking can also be used in other contexts, for
+example, when creating lists.  When specifying elements of a list, prefixing an
+expression by a ``*`` will unpack the result of that expression, adding each of
+its elements to the list we're creating::
+
+   >>> x = [1, 2, 3]
+   >>> [0, *x, 4, 5, 6]
+   [0, 1, 2, 3, 4, 5, 6]
+
+This only works if the expression following the ``*`` evaluates to an iterable
+object; trying to unpack a non-iterable object will raise an exception::
+
+   >>> x = 1
+   >>> [0, *x, 2, 3, 4]
+   Traceback (most recent call last):
+     File "<python-input-1>", line 1, in <module>
+       [0, *x, 2, 3, 4]
+   TypeError: Value after * must be an iterable, not int
+
+Unpacking can also be used in list comprehensions, as a way to build a new list
+representing the concatenation of an arbitrary number of iterables::
+
+   >>> x = [[1, 2, 3], [4, 5, 6], [], [7], [8, 9]]
+   >>> [*element for element in x]
+   [1, 2, 3, 4, 5, 6, 7, 8, 9]
+
+Note that the effect is that each element from ``x`` is unpacked.  This works
+for arbitrary iterable objects, not just lists::
+
+   >>> x = [[1, 2, 3], 'cat', {'spam': 'eggs'}]
+   >>> [*element for element in x]
+   [1, 2, 3, 'c', 'a', 't', 'spam']
+
+But if the objects in ``x`` are not iterable, this expression would again raise
+an exception.
+
 .. _tut-del:
 
 The :keyword:`!del` statement
@@ -394,7 +435,10 @@ A tuple consists of a number of values separated by commas, for instance::
    >>> v = ([1, 2, 3], [3, 2, 1])
    >>> v
    ([1, 2, 3], [3, 2, 1])
-
+   >>> # they support unpacking just like lists:
+   >>> x = [1, 2, 3]
+   >>> 0, *x, 4
+   (0, 1, 2, 3, 4)
 
 As you see, on output tuples are always enclosed in parentheses, so that nested
 tuples are interpreted correctly; they may be input with or without surrounding
@@ -480,12 +524,16 @@ Here is a brief demonstration::
    {'r', 'd', 'b', 'm', 'z', 'l'}
 
 Similarly to :ref:`list comprehensions <tut-listcomps>`, set comprehensions
-are also supported::
+are also supported, including comprehensions with unpacking::
 
    >>> a = {x for x in 'abracadabra' if x not in 'abc'}
    >>> a
    {'r', 'd'}
 
+   >>> fruits = [{'apple', 'avocado', 'apricot'}, {'banana', 'blueberry'}]
+   >>> {*fruit for fruit in fruits}
+   {'blueberry', 'banana', 'avocado', 'apple', 'apricot'}
+
 
 .. _tut-dictionaries:
 
@@ -563,6 +611,18 @@ arbitrary key and value expressions::
    >>> {x: x**2 for x in (2, 4, 6)}
    {2: 4, 4: 16, 6: 36}
 
+And dictionary unpacking (via ``**``) can be used to merge multiple
+dictionaries::
+
+   >>> odds = {i: i**2 for i in (1, 3, 5)}
+   >>> evens = {i: i**2 for i in (2, 4, 6)}
+   >>> {**odds, **evens}
+   {1: 1, 3: 9, 5: 25, 2: 4, 4: 16, 6: 36}
+
+   >>> all_values = [odds, evens, {0: 0}]
+   >>> {**i for i in all_values}
+   {1: 1, 3: 9, 5: 25, 2: 4, 4: 16, 6: 36, 0: 0}
+
 When the keys are simple strings, it is sometimes easier to specify pairs using
 keyword arguments::
 

Doc/whatsnew/3.15.rst

@@ -69,6 +69,8 @@ Summary -- Release highlights
   profiling tools <whatsnew315-profiling-package>`
 * :pep:`799`: :ref:`Tachyon: High frequency statistical sampling profiler
   profiling tools <whatsnew315-sampling-profiler>`
+* :pep:`798`: :ref:`Unpacking in Comprehensions
+  <whatsnew315-unpacking-in-comprehensions>`
 * :pep:`686`: :ref:`Python now uses UTF-8 as the default encoding
   <whatsnew315-utf8-default>`
 * :pep:`782`: :ref:`A new PyBytesWriter C API to create a Python bytes object
@@ -187,6 +189,45 @@ available output formats, profiling modes, and configuration options.
 
 (Contributed by Pablo Galindo and László Kiss Kollár in :gh:`135953` and :gh:`138122`.)
 
+.. _whatsnew315-unpacking-in-comprehensions:
+
+:pep:`798`: Unpacking in Comprehensions
+---------------------------------------
+
+List, set, and dictionary comprehensions, as well as generator expressions, now
+support unpacking with ``*`` and ``**``.  This extends the unpacking syntax
+from :pep:`448` to comprehensions, providing a new syntax for combining an
+arbitrary number of iterables or dictionaries into a single flat structure.
+This new syntax is a direct alternative to nested comprehensions,
+:func:`itertools.chain`, and :meth:`itertools.chain.from_iterable`.  For
+example::
+
+    >>> lists = [[1, 2], [3, 4], [5]]
+    >>> [*L for L in lists]  # equivalent to [x for L in lists for x in L]
+    [1, 2, 3, 4, 5]
+
+    >>> sets = [{1, 2}, {2, 3}, {3, 4}]
+    >>> {*s for s in sets}  # equivalent to {x for s in sets for x in s}
+    {1, 2, 3, 4}
+
+    >>> dicts = [{'a': 1}, {'b': 2}, {'a': 3}]
+    >>> {**d for d in dicts}  # equivalent to {k: v for d in dicts for k,v in d.items()}
+    {'a': 3, 'b': 2}
+
+Generator expressions can similarly use unpacking to yield values from multiple
+iterables::
+
+    >>> gen = (*L for L in lists)  # equivalent to (x for L in lists for x in L)
+    >>> list(gen)
+    [1, 2, 3, 4, 5]
+
+This change also extends to asynchronous generator expressions, such that, for
+example, ``(*a async for a in agen())`` is equivalent to ``(x async for a in
+agen() for x in a)``.
+
+.. seealso:: :pep:`798` for further details.
+
+(Contributed by Adam Hartz in :gh:`143055`.)
 
 .. _whatsnew315-improved-error-messages: