Merge branch 'main' into test/executor-map-ref-cycles

Author: graingert

Doc/library/annotationlib.rst

@@ -204,12 +204,6 @@ Classes
       means may not have any information about their scope, so passing
       arguments to this method may be necessary to evaluate them successfully.
 
-      .. important::
-
-         Once a :class:`~ForwardRef` instance has been evaluated, it caches
-         the evaluated value, and future calls to :meth:`evaluate` will return
-         the cached value, regardless of the parameters passed in.
-
    .. versionadded:: 3.14
 
 

Doc/library/typing.rst

@@ -3449,7 +3449,9 @@ Introspection helpers
    .. versionadded:: 3.7.4
 
    .. versionchanged:: 3.14
-      This is now an alias for :class:`annotationlib.ForwardRef`.
+      This is now an alias for :class:`annotationlib.ForwardRef`. Several undocumented
+      behaviors of this class have been changed; for example, after a ``ForwardRef`` has
+      been evaluated, the evaluated value is no longer cached.
 
 .. function:: evaluate_forward_ref(forward_ref, *, owner=None, globals=None, locals=None, type_params=None, format=annotationlib.Format.VALUE)
 

Doc/whatsnew/3.14.rst

@@ -209,7 +209,7 @@ This example shows how these formats behave:
      ...
    NameError: name 'Undefined' is not defined
    >>> get_annotations(func, format=Format.FORWARDREF)
-   {'arg': ForwardRef('Undefined')}
+   {'arg': ForwardRef('Undefined', owner=<function func at 0x...>)}
    >>> get_annotations(func, format=Format.STRING)
    {'arg': 'Undefined'}
 
@@ -455,6 +455,13 @@ Other language changes
   The testbed can also be used to run the test suite of projects other than
   CPython itself. (Contributed by Russell Keith-Magee in :gh:`127592`.)
 
+* Add a built-in implementation for HMAC (:rfc:`2104`) using formally verified
+  code from the `HACL* <https://github.com/hacl-star/hacl-star/>`__ project.
+  This implementation is used as a fallback when the OpenSSL implementation
+  of HMAC is not available.
+  (Contributed by Bénédikt Tran in :gh:`99108`.)
+
+
 .. _whatsnew314-pep765:
 
 PEP 765: Disallow return/break/continue that exit a finally block
@@ -464,6 +471,7 @@ The compiler emits a :exc:`SyntaxWarning` when a :keyword:`return`, :keyword:`br
 :keyword:`continue` statements appears where it exits a :keyword:`finally` block.
 This change is specified in :pep:`765`.
 
+
 New modules
 ===========
 
@@ -705,6 +713,14 @@ graphlib
   (Contributed by Daniel Pope in :gh:`130914`)
 
 
+hmac
+----
+
+* Add a built-in implementation for HMAC (:rfc:`2104`) using formally verified
+  code from the `HACL* <https://github.com/hacl-star/hacl-star/>`__ project.
+  (Contributed by Bénédikt Tran in :gh:`99108`.)
+
+
 http
 ----
 
@@ -1059,6 +1075,54 @@ turtle
   (Contributed by Marie Roald and Yngve Mardal Moe in :gh:`126350`.)
 
 
+types
+-----
+
+* :class:`types.UnionType` is now an alias for :class:`typing.Union`.
+  See :ref:`below <whatsnew314-typing-union>` for more details.
+  (Contributed by Jelle Zijlstra in :gh:`105499`.)
+
+
+typing
+------
+
+.. _whatsnew314-typing-union:
+
+* :class:`types.UnionType` and :class:`typing.Union` are now aliases for each other,
+  meaning that both old-style unions (created with ``Union[int, str]``) and new-style
+  unions (``int | str``) now create instances of the same runtime type. This unifies
+  the behavior between the two syntaxes, but leads to some differences in behavior that
+  may affect users who introspect types at runtime:
+
+  - Both syntaxes for creating a union now produce the same string representation in
+    ``repr()``. For example, ``repr(Union[int, str])``
+    is now ``"int | str"`` instead of ``"typing.Union[int, str]"``.
+  - Unions created using the old syntax are no longer cached. Previously, running
+    ``Union[int, str]`` multiple times would return the same object
+    (``Union[int, str] is Union[int, str]`` would be ``True``), but now it will
+    return two different objects. Users should use ``==`` to compare unions for equality, not
+    ``is``. New-style unions have never been cached this way.
+    This change could increase memory usage for some programs that use a large number of
+    unions created by subscripting ``typing.Union``. However, several factors offset this cost:
+    unions used in annotations are no longer evaluated by default in Python 3.14
+    because of :pep:`649`; an instance of :class:`types.UnionType` is
+    itself much smaller than the object returned by ``Union[]`` was on prior Python
+    versions; and removing the cache also saves some space. It is therefore
+    unlikely that this change will cause a significant increase in memory usage for most
+    users.
+  - Previously, old-style unions were implemented using the private class
+    ``typing._UnionGenericAlias``. This class is no longer needed for the implementation,
+    but it has been retained for backward compatibility, with removal scheduled for Python
+    3.17. Users should use documented introspection helpers like :func:`typing.get_origin`
+    and :func:`typing.get_args` instead of relying on private implementation details.
+  - It is now possible to use :class:`typing.Union` itself in :func:`isinstance` checks.
+    For example, ``isinstance(int | str, typing.Union)`` will return ``True``; previously
+    this raised :exc:`TypeError`.
+  - The ``__args__`` attribute of :class:`typing.Union` objects is no longer writable.
+
+  (Contributed by Jelle Zijlstra in :gh:`105499`.)
+
+
 unicodedata
 -----------
 
@@ -1592,6 +1656,11 @@ Changes in the Python API
   This temporary change affects other threads.
   (Contributed by Serhiy Storchaka in :gh:`69998`.)
 
+* :class:`types.UnionType` is now an alias for :class:`typing.Union`,
+  causing changes in some behaviors.
+  See :ref:`above <whatsnew314-typing-union>` for more details.
+  (Contributed by Jelle Zijlstra in :gh:`105499`.)
+
 
 Build changes
 =============

Include/cpython/pystats.h

@@ -31,7 +31,7 @@
 
 #define PYSTATS_MAX_UOP_ID 512
 
-#define SPECIALIZATION_FAILURE_KINDS 37
+#define SPECIALIZATION_FAILURE_KINDS 44
 
 /* Stats for determining who is calling PyEval_EvalFrame */
 #define EVAL_CALL_TOTAL 0

Include/internal/pycore_ceval.h

@@ -196,7 +196,7 @@ extern void _PyEval_DeactivateOpCache(void);
 
 /* --- _Py_EnterRecursiveCall() ----------------------------------------- */
 
-#if !_Py__has_builtin(__builtin_frame_address)
+#if !_Py__has_builtin(__builtin_frame_address) && !defined(_MSC_VER)
 static uintptr_t return_pointer_as_int(char* p) {
     return (uintptr_t)p;
 }
@@ -206,6 +206,8 @@ static inline uintptr_t
 _Py_get_machine_stack_pointer(void) {
 #if _Py__has_builtin(__builtin_frame_address)
     return (uintptr_t)__builtin_frame_address(0);
+#elif defined(_MSC_VER)
+    return (uintptr_t)_AddressOfReturnAddress();
 #else
     char here;
     /* Avoid compiler warning about returning stack address */

Include/internal/pycore_opcode_metadata.h

@@ -4,11 +4,6 @@
 extern "C" {
 #endif
 
-// Define this to get precise tracking of closed stackrefs.
-// This will use unbounded memory, as it can only grow.
-// Use this to track double closes in short-lived programs
-// #define Py_STACKREF_CLOSE_DEBUG 1
-
 #ifndef Py_BUILD_CORE
 #  error "this header requires Py_BUILD_CORE define"
 #endif

Include/internal/pycore_stackref.h

@@ -57,6 +57,12 @@ typedef struct {
 // Define this to get precise tracking of stackrefs.
 // #define Py_STACKREF_DEBUG 1
 
+// Define this to get precise tracking of closed stackrefs.
+// This will use unbounded memory, as it can only grow.
+// Use this to track double closes in short-lived programs
+// #define Py_STACKREF_CLOSE_DEBUG 1
+
+
 typedef union _PyStackRef {
 #if !defined(Py_GIL_DISABLED) && defined(Py_STACKREF_DEBUG)
     uint64_t index;

Include/internal/pycore_structs.h

@@ -3,7 +3,6 @@
 import ast
 import builtins
 import enum
-import functools
 import keyword
 import sys
 import types
@@ -33,18 +32,16 @@ class Format(enum.IntEnum):
 # preserved for compatibility with the old typing.ForwardRef class. The remaining
 # names are private.
 _SLOTS = (
-    "__forward_evaluated__",
-    "__forward_value__",
     "__forward_is_argument__",
     "__forward_is_class__",
     "__forward_module__",
     "__weakref__",
     "__arg__",
-    "__ast_node__",
-    "__code__",
     "__globals__",
-    "__owner__",
+    "__code__",
+    "__ast_node__",
     "__cell__",
+    "__owner__",
     "__stringifier_dict__",
 )
 
@@ -77,14 +74,12 @@ def __init__(
             raise TypeError(f"Forward reference must be a string -- got {arg!r}")
 
         self.__arg__ = arg
-        self.__forward_evaluated__ = False
-        self.__forward_value__ = None
         self.__forward_is_argument__ = is_argument
         self.__forward_is_class__ = is_class
         self.__forward_module__ = module
+        self.__globals__ = None
         self.__code__ = None
         self.__ast_node__ = None
-        self.__globals__ = None
         self.__cell__ = None
         self.__owner__ = owner
 
@@ -96,17 +91,11 @@ def evaluate(self, *, globals=None, locals=None, type_params=None, owner=None):
 
         If the forward reference cannot be evaluated, raise an exception.
         """
-        if self.__forward_evaluated__:
-            return self.__forward_value__
         if self.__cell__ is not None:
             try:
-                value = self.__cell__.cell_contents
+                return self.__cell__.cell_contents
             except ValueError:
                 pass
-            else:
-                self.__forward_evaluated__ = True
-                self.__forward_value__ = value
-                return value
         if owner is None:
             owner = self.__owner__
 
@@ -172,8 +161,6 @@ def evaluate(self, *, globals=None, locals=None, type_params=None, owner=None):
         else:
             code = self.__forward_code__
             value = eval(code, globals=globals, locals=locals)
-        self.__forward_evaluated__ = True
-        self.__forward_value__ = value
         return value
 
     def _evaluate(self, globalns, localns, type_params=_sentinel, *, recursive_guard):
@@ -231,18 +218,30 @@ def __forward_code__(self):
     def __eq__(self, other):
         if not isinstance(other, ForwardRef):
             return NotImplemented
-        if self.__forward_evaluated__ and other.__forward_evaluated__:
-            return (
-                self.__forward_arg__ == other.__forward_arg__
-                and self.__forward_value__ == other.__forward_value__
-            )
         return (
             self.__forward_arg__ == other.__forward_arg__
             and self.__forward_module__ == other.__forward_module__
+            # Use "is" here because we use id() for this in __hash__
+            # because dictionaries are not hashable.
+            and self.__globals__ is other.__globals__
+            and self.__forward_is_class__ == other.__forward_is_class__
+            and self.__code__ == other.__code__
+            and self.__ast_node__ == other.__ast_node__
+            and self.__cell__ == other.__cell__
+            and self.__owner__ == other.__owner__
         )
 
     def __hash__(self):
-        return hash((self.__forward_arg__, self.__forward_module__))
+        return hash((
+            self.__forward_arg__,
+            self.__forward_module__,
+            id(self.__globals__),  # dictionaries are not hashable, so hash by identity
+            self.__forward_is_class__,
+            self.__code__,
+            self.__ast_node__,
+            self.__cell__,
+            self.__owner__,
+        ))
 
     def __or__(self, other):
         return types.UnionType[self, other]
@@ -251,11 +250,14 @@ def __ror__(self, other):
         return types.UnionType[other, self]
 
     def __repr__(self):
-        if self.__forward_module__ is None:
-            module_repr = ""
-        else:
-            module_repr = f", module={self.__forward_module__!r}"
-        return f"ForwardRef({self.__forward_arg__!r}{module_repr})"
+        extra = []
+        if self.__forward_module__ is not None:
+            extra.append(f", module={self.__forward_module__!r}")
+        if self.__forward_is_class__:
+            extra.append(", is_class=True")
+        if self.__owner__ is not None:
+            extra.append(f", owner={self.__owner__!r}")
+        return f"ForwardRef({self.__forward_arg__!r}{''.join(extra)})"
 
 
 class _Stringifier:
@@ -277,8 +279,6 @@ def __init__(
         # represent a single name).
         assert isinstance(node, (ast.AST, str))
         self.__arg__ = None
-        self.__forward_evaluated__ = False
-        self.__forward_value__ = None
         self.__forward_is_argument__ = False
         self.__forward_is_class__ = is_class
         self.__forward_module__ = None
@@ -765,9 +765,10 @@ def get_annotations(
             if hasattr(unwrap, "__wrapped__"):
                 unwrap = unwrap.__wrapped__
                 continue
-            if isinstance(unwrap, functools.partial):
-                unwrap = unwrap.func
-                continue
+            if functools := sys.modules.get("functools"):
+                if isinstance(unwrap, functools.partial):
+                    unwrap = unwrap.func
+                    continue
             break
         if hasattr(unwrap, "__globals__"):
             obj_globals = unwrap.__globals__