PEP 646: Explain the results of the new grammar/compiler change

pep-0646.rst

@@ -873,7 +873,7 @@ Grammar Changes
 
 This PEP requires two grammar changes. Full diffs of ``python.gram``
 and simple tests to confirm correct behaviour are available at
-https://github.com/mrahtz/cpython/commits/pep646-grammar.
+https://github.com/mrahtz/cpython/commits/pep646-grammar-attempt2-rebased.
 
 Change 1: Star Expressions in Indexes
 -------------------------------------
@@ -1015,21 +1015,57 @@ Where:
 
     star_annotation: ':' star_expression
 
-This accomplishes the desired outcome (making ``*args: *Ts`` not be a syntax
-error) while matching the behaviour of star-unpacking in other contexts:
-at runtime, ``__iter__`` is called on the starred object, and a tuple
-containing the items of the resulting iterator is set as the type annotion
-for ``args``. In other words, at runtime ``*args: *foo`` is equivalent to
-``*args: tuple(foo)``.
+We also need to deal with the ``star_expression`` that results from this
+construction. Normally, a ``star_expression`` occurs within the context
+of e.g. a list, so a ``star_expression`` is handled by essentially
+calling ``iter()`` on the starred object, and inserting the results
+of the resulting iterator into the list at the appropriate place. For
+``*args: *Ts``, however, we must process the ``star_expression`` in a
+different way.
+
+We do this by instead making a special case for the ``star_expression``
+resulting from ``*args: *Ts``, modifying the compiler to emit bytecode
+corresponding to ``next(iter(Ts))``. While slightly inconsistent with
+other uses of the star operator, this results in the unpacked
+``TypeVarTuple`` being set directly as the runtime annotation for ``*args``:
 
 ::
 
     >>> Ts = TypeVarTuple('Ts')
-    >>> def foo(*args: *Ts): pass  # Equivalent to `*args: tuple(Ts)`
+    >>> def foo(*args: *Ts): pass
     >>> foo.__annotations__
-    {'args': (*Ts,)}
+    {'args': *Ts}
     # *Ts is the repr() of Ts._unpacked, an instance of UnpackedTypeVarTuple
 
+This is important, because it allows the runtime annotation to be
+consistent with an AST representation that uses a ``Starred`` node
+for the annotations of ``args`` - in turn important for tools that
+rely on the AST such as mypy to correctly recognise the construction:
+
+::
+
+    >>> print(ast.dump(ast.parse('def foo(*args: *Ts): pass'), indent=2))
+    Module(
+      body=[
+        FunctionDef(
+          name='foo',
+          args=arguments(
+            posonlyargs=[],
+            args=[],
+            vararg=arg(
+              arg='args',
+              annotation=Starred(
+                value=Name(id='Ts', ctx=Load()),
+                ctx=Load())),
+            kwonlyargs=[],
+            kw_defaults=[],
+            defaults=[]),
+          body=[
+            Pass()],
+          decorator_list=[])],
+      type_ignores=[])
+
+
 Note that the only scenario in which this grammar change allows ``*Ts`` to be
 used as a direct annotation (rather than being wrapped in e.g. ``Tuple[*Ts]``)
 is ``*args``. Other uses are still invalid:
@@ -1045,14 +1081,14 @@ Implications
 As with the first grammar change, this change also has a number of side effects.
 In particular, the annotation of ``*args`` could be set to a starred object
 other than a ``TypeVarTuple`` - for example, the following nonsensical
-annotation is possible:
+annotation is possible, resulting in the following behaviour:
 
 ::
 
     >>> foo = [1, 2, 3]
-    >>> def bar(*args: *foo): pass  # Equivalent to `*args: tuple(foo)`
+    >>> def bar(*args: *foo): pass  # Roughly equivalent to `*args: next(iter(foo))`
     >>> bar.__annotations__
-    {'args': (1, 2, 3)}
+    {'args': 1}
 
 Again, prevention of such annotations will need to be done by, say, static
 checkers, rather than at the level of syntax.