Skip to content

Reformat "Important Files" section in compiler.rst #1004

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Dec 15, 2022
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
180 changes: 76 additions & 104 deletions internals/compiler.rst
Original file line number Diff line number Diff line change
Expand Up @@ -524,140 +524,112 @@ statement in ``_PyEval_EvalFrameDefault()``.
Important Files
===============

+ Parser/
* :cpy-file:`Parser/`
* :cpy-file:`Parser/Python.asdl`: ASDL syntax file.

Python.asdl
ASDL syntax file
* :cpy-file:`Parser/asdl.py`: Parser for ASDL definition files.
Reads in an ASDL description and parses it into an AST that describes it.

asdl.py
Parser for ASDL definition files. Reads in an ASDL description
and parses it into an AST that describes it.
* :cpy-file:`Parser/asdl_c.py`: Generate C code from an ASDL description.
Generates :cpy-file:`Python/Python-ast.c` and
:cpy-file:`Include/internal/pycore_ast.h`.

asdl_c.py
"Generate C code from an ASDL description." Generates
:cpy-file:`Python/Python-ast.c` and :cpy-file:`Include/internal/pycore_ast.h`.
* :cpy-file:`Parser/parser.c`: The new PEG parser introduced in Python 3.9.
Generated by :cpy-file:`Tools/peg_generator/pegen/c_generator.py`
from the grammar :cpy-file:`Grammar/python.gram`. Creates the AST from
source code. Rule functions for their corresponding production rules
are found here.

parser.c
The new PEG parser introduced in Python 3.9.
Generated by :cpy-file:`Tools/peg_generator/pegen/c_generator.py`
from the grammar :cpy-file:`Grammar/python.gram`. Creates the AST from
source code. Rule functions for their corresponding production rules
are found here.
* :cpy-file:`Parser/peg_api.c`: Contains high-level functions which are
used by the interpreter to create an AST from source code.

peg_api.c
Contains high-level functions which are used by the interpreter to
create an AST from source code .
* :cpy-file:`Parser/pegen.c`: Contains helper functions which are used
by functions in :cpy-file:`Parser/parser.c` to construct the AST.
Also contains helper functions which help raise better error messages
when parsing source code.

pegen.c
Contains helper functions which are used by functions in
:cpy-file:`Parser/parser.c` to construct the AST. Also contains helper
functions which help raise better error messages when parsing source
code.
* :cpy-file:`Parser/pegen.h`: Header file for the corresponding
:cpy-file:`Parser/pegen.c`. Also contains definitions of the ``Parser``
and ``Token`` structs.

pegen.h
Header file for the corresponding :cpy-file:`Parser/pegen.c`. Also contains
definitions of the ``Parser`` and ``Token`` structs.
* :cpy-file:`Python/`
* :cpy-file:`Python/Python-ast.c`: Creates C structs corresponding to
the ASDL types. Also contains code for marshalling AST nodes (core
ASDL types have marshalling code in :cpy-file:`Python/asdl.c`).
"File automatically generated by :cpy-file:`Parser/asdl_c.py`".
This file must be committed separately after every grammar change
is committed since the ``__version__`` value is set to the latest
grammar change revision number.

+ Python/
* :cpy-file:`Python/asdl.c`: Contains code to handle the ASDL sequence type.
Also has code to handle marshalling the core ASDL types, such as number
and identifier. Used by :cpy-file:`Python/Python-ast.c` for marshalling
AST nodes.

Python-ast.c
Creates C structs corresponding to the ASDL types. Also
contains code for marshalling AST nodes (core ASDL types have
marshalling code in :cpy-file:`Python/asdl.c`). "File automatically
generated by :cpy-file:`Parser/asdl_c.py`". This file must be
committed separately after every grammar change is committed since
the ``__version__`` value is set to the latest grammar change
revision number.
* :cpy-file:`Python/ast.c`: Used for validating the AST.

asdl.c
Contains code to handle the ASDL sequence type. Also has code
to handle marshalling the core ASDL types, such as number and
identifier. Used by :cpy-file:`Python/Python-ast.c` for marshalling
AST nodes.
* :cpy-file:`Python/ast_opt.c`: Optimizes the AST.

ast.c
Used for validating the AST.
* :cpy-file:`Python/ast_unparse.c`: Converts the AST expression node
back into a string (for string annotations).

ast_opt.c
Optimizes the AST.
* :cpy-file:`Python/ceval.c`: Executes byte code (aka, eval loop).

ast_unparse.c
Converts the AST expression node back into a string
(for string annotations).
* :cpy-file:`Python/compile.c`: Emits bytecode based on the AST.

ceval.c
Executes byte code (aka, eval loop).
* :cpy-file:`Python/symtable.c`: Generates a symbol table from AST.

compile.c
Emits bytecode based on the AST.
* :cpy-file:`Python/peephole.c`: Optimizes the bytecode.

symtable.c
Generates a symbol table from AST.
* :cpy-file:`Python/pyarena.c`: Implementation of the arena memory manager.

peephole.c
Optimizes the bytecode.
* :cpy-file:`Python/wordcode_helpers.h`: Helpers for generating bytecode.

pyarena.c
Implementation of the arena memory manager.
* :cpy-file:`Python/opcode_targets.h`: One of the files that must be
modified if :cpy-file:`Lib/opcode.py` is.

wordcode_helpers.h
Helpers for generating bytecode.
* :cpy-file:`Include/`
* :cpy-file:`Include/code.h`: Header file for :cpy-file:`Objects/codeobject.c`;
contains definition of ``PyCodeObject``.

opcode_targets.h
One of the files that must be modified if :cpy-file:`Lib/opcode.py` is.
* :cpy-file:`Include/opcode.h`: One of the files that must be modified if
:cpy-file:`Lib/opcode.py` is.

+ Include/
* :cpy-file:`Include/internal/pycore_ast.h`: Contains the actual definitions
of the C structs as generated by :cpy-file:`Python/Python-ast.c`.
"Automatically generated by :cpy-file:`Parser/asdl_c.py`".

code.h
Header file for :cpy-file:`Objects/codeobject.c`; contains definition of
``PyCodeObject``.
* :cpy-file:`Include/internal/pycore_asdl.h`: Header for the corresponding
:cpy-file:`Python/ast.c`.

opcode.h
One of the files that must be modified if :cpy-file:`Lib/opcode.py` is.
* :cpy-file:`Include/internal/pycore_ast.h`: Declares ``_PyAST_Validate()``
external (from :cpy-file:`Python/ast.c`).

+ internal/
* :cpy-file:`Include/internal/pycore_symtable.h`: Header for
:cpy-file:`Python/symtable.c`. ``struct symtable`` and ``PySTEntryObject``
are defined here.

pycore_ast.h
Contains the actual definitions of the C structs as generated by
:cpy-file:`Python/Python-ast.c`.
"Automatically generated by :cpy-file:`Parser/asdl_c.py`".
* :cpy-file:`Include/internal/pycore_parser.h`: Header for the
corresponding :cpy-file:`Parser/peg_api.c`.

pycore_asdl.h
Header for the corresponding :cpy-file:`Python/ast.c`
* :cpy-file:`Include/internal/pycore_pyarena.h`: Header file for the
corresponding :cpy-file:`Python/pyarena.c`.

pycore_ast.h
Declares ``_PyAST_Validate()`` external (from :cpy-file:`Python/ast.c`).
* :cpy-file:`Objects/`
* :cpy-file:`Objects/codeobject.c`: Contains PyCodeObject-related code
(originally in :cpy-file:`Python/compile.c`).

pycore_symtable.h
Header for :cpy-file:`Python/symtable.c`. ``struct symtable`` and
``PySTEntryObject`` are defined here.
* :cpy-file:`Objects/frameobject.c`: Contains the ``frame_setlineno()``
function which should determine whether it is allowed to make a jump
between two points in a bytecode.

pycore_parser.h
Header for the corresponding :cpy-file:`Parser/peg_api.c`.
* :cpy-file:`Lib/`
* :cpy-file:`Lib/opcode.py`: Master list of bytecode; if this file is
modified you must modify several other files accordingly
(see "`Introducing New Bytecode`_")

pycore_pyarena.h
Header file for the corresponding :cpy-file:`Python/pyarena.c`.


+ Objects/

codeobject.c
Contains PyCodeObject-related code (originally in
:cpy-file:`Python/compile.c`).

frameobject.c
Contains the ``frame_setlineno()`` function which should determine
whether it is allowed to make a jump between two points in a bytecode.

+ Lib/

opcode.py
Master list of bytecode; if this file is modified you must modify
several other files accordingly (see "`Introducing New Bytecode`_")

importlib/_bootstrap_external.py
Home of the magic number (named ``MAGIC_NUMBER``) for bytecode
versioning.
* :cpy-file:`Lib/importlib/_bootstrap_external.py`: Home of the magic number
(named ``MAGIC_NUMBER``) for bytecode versioning.


Known Compiler-related Experiments
Expand Down