Skip to content

bpo-36842: Implement PEP 578 #12613

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 33 commits into from
May 23, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
5085b3f
Implement PEP 578
zooba Mar 14, 2019
2e246b7
Move audit hooks to interpreter state
zooba May 6, 2019
1357172
Only preserve exception on Python thread
zooba May 6, 2019
445224c
Add test for subinterpreters
zooba May 6, 2019
af3dd4d
Add DTrace UDST audit hook
tiran May 7, 2019
f53004b
Add NEWS and update socket events
zooba May 7, 2019
a96e225
Fix socket events and add input event
zooba May 8, 2019
79dea7a
Add ..audit-event directive for docs
zooba May 8, 2019
f357525
Undo mismerged changes
zooba May 8, 2019
7f83da1
Fix docs and test
zooba May 8, 2019
475909e
Fix whitespace
zooba May 10, 2019
0fb5b3a
Merge remote-tracking branch 'cpython/master' into pep-578
zooba May 10, 2019
a208832
Merge remote-tracking branch 'cpython/master' into pep-578
zooba May 13, 2019
1d028ca
Fix merge issues
zooba May 13, 2019
68746dd
Ensure auditing hooks are testing
zooba May 13, 2019
00687c1
Code review feedback and fixes
zooba May 13, 2019
8a6cb1e
Format tests and add new header to various file lists
zooba May 13, 2019
3c106f7
Fix doc bug
zooba May 13, 2019
2a18358
Improve dangling thread output and fix reference leak
zooba May 13, 2019
9fa5939
Disable tracing of audit hooks
zooba May 13, 2019
89b24eb
Fix new parser interface
zooba May 13, 2019
5a5b51e
Apply review feedback
zooba May 14, 2019
ec98f69
Merge remote-tracking branch 'cpython/master' into pep-578
zooba May 14, 2019
47a911c
Avoid creating PyObject without an interpreter.
zooba May 14, 2019
b573d3a
Fix expected result for test
zooba May 14, 2019
a30f018
Merge remote-tracking branch 'cpython/master' into pep-578
zooba May 15, 2019
0d25ac7
Merge remote-tracking branch 'cpython/master' into pep-578
zooba May 15, 2019
b3c1792
Merge remote-tracking branch 'cpython/master' into pep-578
zooba May 16, 2019
b78e468
Revert save_env change and filed bpo-36949
zooba May 17, 2019
bfb52fb
Add comment about importing _io
zooba May 17, 2019
5bb335c
Merge remote-tracking branch 'cpython/master' into pep-578
zooba May 20, 2019
b36c25a
Merge remote-tracking branch 'cpython/master' into pep-578
zooba May 22, 2019
969f1cb
Merge remote-tracking branch 'cpython/master' into pep-578
zooba May 23, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Doc/c-api/code.rst
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ bound into a function.
:c:func:`PyCode_New` directly can bind you to a precise Python
version since the definition of the bytecode changes often.

.. audit-event:: code.__new__ "code filename name argcount kwonlyargcount nlocals stacksize flags"

.. c:function:: PyCodeObject* PyCode_NewEmpty(const char *filename, const char *funcname, int firstlineno)

Expand Down
26 changes: 26 additions & 0 deletions Doc/c-api/file.rst
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,32 @@ the :mod:`io` APIs instead.
raised if the end of the file is reached immediately.


.. c:function:: int PyFile_SetOpenCodeHook(Py_OpenCodeHookFunction handler)

Overrides the normal behavior of :func:`io.open_code` to pass its parameter
through the provided handler.

The handler is a function of type :c:type:`PyObject *(\*)(PyObject *path,
void *userData)`, where *path* is guaranteed to be :c:type:`PyUnicodeObject`.

The *userData* pointer is passed into the hook function. Since hook
functions may be called from different runtimes, this pointer should not
refer directly to Python state.

As this hook is intentionally used during import, avoid importing new modules
during its execution unless they are known to be frozen or available in
``sys.modules``.

Once a hook has been set, it cannot be removed or replaced, and later calls to
:c:func:`PyFile_SetOpenCodeHook` will fail. On failure, the function returns
-1 and sets an exception if the interpreter has been initialized.

This function is safe to call before :c:func:`Py_Initialize`.

.. versionadded:: 3.8



.. c:function:: int PyFile_WriteObject(PyObject *obj, PyObject *p, int flags)

.. index:: single: Py_PRINT_RAW
Expand Down
50 changes: 50 additions & 0 deletions Doc/c-api/sys.rst
Original file line number Diff line number Diff line change
Expand Up @@ -289,6 +289,56 @@ accessible to C code. They all work with the current interpreter thread's
.. versionadded:: 3.2


.. c:function:: int PySys_Audit(const char *event, const char *format, ...)

.. index:: single: audit events

Raises an auditing event with any active hooks. Returns zero for success
and non-zero with an exception set on failure.

If any hooks have been added, *format* and other arguments will be used
to construct a tuple to pass. Apart from ``N``, the same format characters
as used in :c:func:`Py_BuildValue` are available. If the built value is not
a tuple, it will be added into a single-element tuple. (The ``N`` format
option consumes a reference, but since there is no way to know whether
arguments to this function will be consumed, using it may cause reference
leaks.)

:func:`sys.audit` performs the same function from Python code.

.. versionadded:: 3.8


.. c:function:: int PySys_AddAuditHook(Py_AuditHookFunction hook, void *userData)

.. index:: single: audit events

Adds to the collection of active auditing hooks. Returns zero for success
and non-zero on failure. If the runtime has been initialized, also sets an
error on failure. Hooks added through this API are called for all
interpreters created by the runtime.

This function is safe to call before :c:func:`Py_Initialize`. When called
after runtime initialization, existing audit hooks are notified and may
silently abort the operation by raising an error subclassed from
:class:`Exception` (other errors will not be silenced).

The hook function is of type :c:type:`int (*)(const char *event, PyObject
*args, void *userData)`, where *args* is guaranteed to be a
:c:type:`PyTupleObject`. The hook function is always called with the GIL
held by the Python interpreter that raised the event.

The *userData* pointer is passed into the hook function. Since hook
functions may be called from different runtimes, this pointer should not
refer directly to Python state.

See :pep:`578` for a detailed decription of auditing. Functions in the
runtime and standard library that raise events include the details in each
function's documentation.

.. versionadded:: 3.8


.. _processcontrol:

Process Control
Expand Down
9 changes: 9 additions & 0 deletions Doc/howto/instrumentation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -332,6 +332,15 @@ Available static markers
.. versionadded:: 3.7


.. c:function:: audit(str event, void *tuple)

Fires when :func:`sys.audit` or :c:func:`PySys_Audit` is called.
``arg0`` is the event name as C string, ``arg1`` is a :c:type:`PyObject`
pointer to a tuple object.

.. versionadded:: 3.8


SystemTap Tapsets
-----------------

Expand Down
1 change: 1 addition & 0 deletions Doc/library/array.rst
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ The module defines the following type:
to add initial items to the array. Otherwise, the iterable initializer is
passed to the :meth:`extend` method.

.. audit-event:: array.__new__ "typecode initializer"

.. data:: typecodes

Expand Down
17 changes: 17 additions & 0 deletions Doc/library/ctypes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1509,6 +1509,17 @@ object is available:
:c:type:`int`, which is of course not always the truth, so you have to assign
the correct :attr:`restype` attribute to use these functions.

.. audit-event:: ctypes.dlopen name

Loading a library through any of these objects raises an
:ref:`auditing event <auditing>` ``ctypes.dlopen`` with string argument
``name``, the name used to load the library.

.. audit-event:: ctypes.dlsym "library name"

Accessing a function on a loaded library raises an auditing event
``ctypes.dlsym`` with arguments ``library`` (the library object) and ``name``
(the symbol's name as a string or integer).

.. _ctypes-foreign-functions:

Expand Down Expand Up @@ -2032,6 +2043,12 @@ Data types
This method returns a ctypes type instance using the memory specified by
*address* which must be an integer.

.. audit-event:: ctypes.cdata address

This method, and others that indirectly call this method, raises an
:func:`auditing event <sys.audit>` ``ctypes.cdata`` with argument
``address``.

.. method:: from_param(obj)

This method adapts *obj* to a ctypes type. It is called with the actual
Expand Down
31 changes: 31 additions & 0 deletions Doc/library/functions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,12 @@ are always available. They are listed here in alphabetical order.
If you want to parse Python code into its AST representation, see
:func:`ast.parse`.

.. audit-event:: compile "source filename"

Raises an :func:`auditing event <sys.audit>` ``compile`` with arguments
``source`` and ``filename``. This event may also be raised by implicit
compilation.

.. note::

When compiling a string with multi-line code in ``'single'`` or
Expand Down Expand Up @@ -473,6 +479,11 @@ are always available. They are listed here in alphabetical order.
See :func:`ast.literal_eval` for a function that can safely evaluate strings
with expressions containing only literals.

.. audit-event:: exec code_object

Raises an :func:`auditing event <sys.audit>` ``exec`` with the code object as
the argument. Code compilation events may also be raised.

.. index:: builtin: exec

.. function:: exec(object[, globals[, locals]])
Expand Down Expand Up @@ -502,6 +513,11 @@ are always available. They are listed here in alphabetical order.
builtins are available to the executed code by inserting your own
``__builtins__`` dictionary into *globals* before passing it to :func:`exec`.

.. audit-event:: exec code_object

Raises an :func:`auditing event <sys.audit>` ``exec`` with the code object as
the argument. Code compilation events may also be raised.

.. note::

The built-in functions :func:`globals` and :func:`locals` return the current
Expand Down Expand Up @@ -747,6 +763,16 @@ are always available. They are listed here in alphabetical order.
If the :mod:`readline` module was loaded, then :func:`input` will use it
to provide elaborate line editing and history features.

.. audit-event:: builtins.input prompt

Raises an :func:`auditing event <sys.audit>` ``builtins.input`` with
argument ``prompt`` before reading input

.. audit-event:: builtins.input/result result

Raises an auditing event ``builtins.input/result`` with the result after
successfully reading input.


.. class:: int([x])
int(x, base=10)
Expand Down Expand Up @@ -1176,6 +1202,11 @@ are always available. They are listed here in alphabetical order.
(where :func:`open` is declared), :mod:`os`, :mod:`os.path`, :mod:`tempfile`,
and :mod:`shutil`.

.. audit-event:: open "file mode flags"

The ``mode`` and ``flags`` arguments may have been modified or inferred from
the original call.

.. versionchanged::
3.3

Expand Down
21 changes: 21 additions & 0 deletions Doc/library/io.rst
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,27 @@ High-level Module Interface

This is an alias for the builtin :func:`open` function.

.. audit-event:: open "path mode flags"

This function raises an :func:`auditing event <sys.audit>` ``open`` with
arguments ``path``, ``mode`` and ``flags``. The ``mode`` and ``flags``
arguments may have been modified or inferred from the original call.


.. function:: open_code(path)

Opens the provided file with mode ``'rb'``. This function should be used
when the intent is to treat the contents as executable code.

``path`` should be an absolute path.

The behavior of this function may be overridden by an earlier call to the
:c:func:`PyFile_SetOpenCodeHook`, however, it should always be considered
interchangeable with ``open(path, 'rb')``. Overriding the behavior is
intended for additional validation or preprocessing of the file.

.. versionadded:: 3.8


.. exception:: BlockingIOError

Expand Down
2 changes: 2 additions & 0 deletions Doc/library/mmap.rst
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ To map anonymous memory, -1 should be passed as the fileno along with the length
will be relative to the offset from the beginning of the file. *offset*
defaults to 0. *offset* must be a multiple of the :const:`ALLOCATIONGRANULARITY`.

.. audit-event:: mmap.__new__ "fileno length access offset"

.. class:: mmap(fileno, length, flags=MAP_SHARED, prot=PROT_WRITE|PROT_READ, access=ACCESS_DEFAULT[, offset])
:noindex:
Expand Down Expand Up @@ -155,6 +156,7 @@ To map anonymous memory, -1 should be passed as the fileno along with the length

mm.close()

.. audit-event:: mmap.__new__ "fileno length access offset"

Memory-mapped file objects support the following methods:

Expand Down
12 changes: 11 additions & 1 deletion Doc/library/os.rst
Original file line number Diff line number Diff line change
Expand Up @@ -651,7 +651,7 @@ process and user.
File Object Creation
--------------------

This function creates new :term:`file objects <file object>`. (See also
These functions create new :term:`file objects <file object>`. (See also
:func:`~os.open` for opening file descriptors.)


Expand Down Expand Up @@ -829,11 +829,14 @@ as internal buffering of data.
most *length* bytes in size. As of Python 3.3, this is equivalent to
``os.truncate(fd, length)``.

.. audit-event:: os.truncate "fd length"

.. availability:: Unix, Windows.

.. versionchanged:: 3.5
Added support for Windows


.. function:: get_blocking(fd)

Get the blocking mode of the file descriptor: ``False`` if the
Expand All @@ -845,6 +848,7 @@ as internal buffering of data.

.. versionadded:: 3.5


.. function:: isatty(fd)

Return ``True`` if the file descriptor *fd* is open and connected to a
Expand Down Expand Up @@ -912,6 +916,8 @@ as internal buffering of data.
This function can support :ref:`paths relative to directory descriptors
<dir_fd>` with the *dir_fd* parameter.

.. audit-event:: open "path mode flags"

.. versionchanged:: 3.4
The new file descriptor is now non-inheritable.

Expand Down Expand Up @@ -2756,6 +2762,8 @@ features:

This function can support :ref:`specifying a file descriptor <path_fd>`.

.. audit-event:: os.truncate "path length"

.. availability:: Unix, Windows.

.. versionadded:: 3.3
Expand Down Expand Up @@ -3715,6 +3723,8 @@ written in Python, such as a mail server's external command delivery program.
to using this function. See the :ref:`subprocess-replacements` section in
the :mod:`subprocess` documentation for some helpful recipes.

.. audit-event:: os.system command

.. availability:: Unix, Windows.


Expand Down
1 change: 1 addition & 0 deletions Doc/library/pickle.rst
Original file line number Diff line number Diff line change
Expand Up @@ -427,6 +427,7 @@ The :mod:`pickle` module exports two classes, :class:`Pickler` and
how they can be loaded, potentially reducing security risks. Refer to
:ref:`pickle-restrict` for details.

.. audit-event:: pickle.find_class "module name"

.. _pickle-picklable:

Expand Down
Loading