From d9bfe4d2d26974ece504b8a00c148d41f3c262dc Mon Sep 17 00:00:00 2001 From: Ezio Melotti Date: Fri, 9 Dec 2022 03:14:59 -0700 Subject: [PATCH 1/2] Reformat the "Important file" section. --- internals/compiler.rst | 257 ++++++++++++++++++++--------------------- 1 file changed, 125 insertions(+), 132 deletions(-) diff --git a/internals/compiler.rst b/internals/compiler.rst index 7b5df7b87e..9e8f45fc72 100644 --- a/internals/compiler.rst +++ b/internals/compiler.rst @@ -524,140 +524,133 @@ statement in ``_PyEval_EvalFrameDefault()``. Important Files =============== -+ Parser/ +* :cpy-file:`Parser/` + * :cpy-file:`Parser/Python.asdl` + ASDL syntax file + + * :cpy-file:`Parser/asdl.py` + Parser for ASDL definition files. Reads in an ASDL description + and parses it into an AST that describes it. + + * :cpy-file:`Parser/asdl_c.py` + "Generate C code from an ASDL description." Generates + :cpy-file:`Python/Python-ast.c` and :cpy-file:`Include/internal/pycore_ast.h`. + + * :cpy-file:`Parser/parser.c` + The new PEG parser introduced in Python 3.9. + Generated by :cpy-file:`Tools/peg_generator/pegen/c_generator.py` + from the grammar :cpy-file:`Grammar/python.gram`. Creates the AST from + source code. Rule functions for their corresponding production rules + are found here. - Python.asdl - ASDL syntax file + * :cpy-file:`Parser/peg_api.c` + Contains high-level functions which are used by the interpreter to + create an AST from source code . + + * :cpy-file:`Parser/pegen.c` + Contains helper functions which are used by functions in + :cpy-file:`Parser/parser.c` to construct the AST. Also contains helper + functions which help raise better error messages when parsing source + code. + + * :cpy-file:`Parser/pegen.h` + Header file for the corresponding :cpy-file:`Parser/pegen.c`. Also contains + definitions of the ``Parser`` and ``Token`` structs. + +* :cpy-file:`Python/` + * :cpy-file:`Python/Python-ast.c` + Creates C structs corresponding to the ASDL types. Also + contains code for marshalling AST nodes (core ASDL types have + marshalling code in :cpy-file:`Python/asdl.c`). "File automatically + generated by :cpy-file:`Parser/asdl_c.py`". This file must be + committed separately after every grammar change is committed since + the ``__version__`` value is set to the latest grammar change + revision number. + + * :cpy-file:`Python/asdl.c` + Contains code to handle the ASDL sequence type. Also has code + to handle marshalling the core ASDL types, such as number and + identifier. Used by :cpy-file:`Python/Python-ast.c` for marshalling + AST nodes. + + * :cpy-file:`Python/ast.c` + Used for validating the AST. + + * :cpy-file:`Python/ast_opt.c` + Optimizes the AST. + + * :cpy-file:`Python/ast_unparse.c` + Converts the AST expression node back into a string + (for string annotations). + + * :cpy-file:`Python/ceval.c` + Executes byte code (aka, eval loop). + + * :cpy-file:`Python/compile.c` + Emits bytecode based on the AST. + + * :cpy-file:`Python/symtable.c` + Generates a symbol table from AST. + + * :cpy-file:`Python/peephole.c` + Optimizes the bytecode. + + * :cpy-file:`Python/pyarena.c` + Implementation of the arena memory manager. + + * :cpy-file:`Python/wordcode_helpers.h` + Helpers for generating bytecode. + + * :cpy-file:`Python/opcode_targets.h` + One of the files that must be modified if :cpy-file:`Lib/opcode.py` is. + +* :cpy-file:`Include/` + * :cpy-file:`code.h` + Header file for :cpy-file:`Objects/codeobject.c`; contains definition of + ``PyCodeObject``. + + * :cpy-file:`opcode.h` + One of the files that must be modified if :cpy-file:`Lib/opcode.py` is. + + * :cpy-file:`internal/` + * :cpy-file:`pycore_ast.h` + Contains the actual definitions of the C structs as generated by + :cpy-file:`Python/Python-ast.c`. + "Automatically generated by :cpy-file:`Parser/asdl_c.py`". + + * :cpy-file:`pycore_asdl.h` + Header for the corresponding :cpy-file:`Python/ast.c` + + * :cpy-file:`pycore_ast.h` + Declares ``_PyAST_Validate()`` external (from :cpy-file:`Python/ast.c`). + + * :cpy-file:`pycore_symtable.h` + Header for :cpy-file:`Python/symtable.c`. ``struct symtable`` and + ``PySTEntryObject`` are defined here. + + * :cpy-file:`pycore_parser.h` + Header for the corresponding :cpy-file:`Parser/peg_api.c`. + + * :cpy-file:`pycore_pyarena.h` + Header file for the corresponding :cpy-file:`Python/pyarena.c`. + +* :cpy-file:`Objects/` + * :cpy-file:`codeobject.c` + Contains PyCodeObject-related code (originally in + :cpy-file:`Python/compile.c`). + + * :cpy-file:`frameobject.c` + Contains the ``frame_setlineno()`` function which should determine + whether it is allowed to make a jump between two points in a bytecode. - asdl.py - Parser for ASDL definition files. Reads in an ASDL description - and parses it into an AST that describes it. - - asdl_c.py - "Generate C code from an ASDL description." Generates - :cpy-file:`Python/Python-ast.c` and :cpy-file:`Include/internal/pycore_ast.h`. - - parser.c - The new PEG parser introduced in Python 3.9. - Generated by :cpy-file:`Tools/peg_generator/pegen/c_generator.py` - from the grammar :cpy-file:`Grammar/python.gram`. Creates the AST from - source code. Rule functions for their corresponding production rules - are found here. - - peg_api.c - Contains high-level functions which are used by the interpreter to - create an AST from source code . - - pegen.c - Contains helper functions which are used by functions in - :cpy-file:`Parser/parser.c` to construct the AST. Also contains helper - functions which help raise better error messages when parsing source - code. - - pegen.h - Header file for the corresponding :cpy-file:`Parser/pegen.c`. Also contains - definitions of the ``Parser`` and ``Token`` structs. - -+ Python/ - - Python-ast.c - Creates C structs corresponding to the ASDL types. Also - contains code for marshalling AST nodes (core ASDL types have - marshalling code in :cpy-file:`Python/asdl.c`). "File automatically - generated by :cpy-file:`Parser/asdl_c.py`". This file must be - committed separately after every grammar change is committed since - the ``__version__`` value is set to the latest grammar change - revision number. - - asdl.c - Contains code to handle the ASDL sequence type. Also has code - to handle marshalling the core ASDL types, such as number and - identifier. Used by :cpy-file:`Python/Python-ast.c` for marshalling - AST nodes. - - ast.c - Used for validating the AST. - - ast_opt.c - Optimizes the AST. - - ast_unparse.c - Converts the AST expression node back into a string - (for string annotations). - - ceval.c - Executes byte code (aka, eval loop). - - compile.c - Emits bytecode based on the AST. - - symtable.c - Generates a symbol table from AST. - - peephole.c - Optimizes the bytecode. - - pyarena.c - Implementation of the arena memory manager. - - wordcode_helpers.h - Helpers for generating bytecode. - - opcode_targets.h - One of the files that must be modified if :cpy-file:`Lib/opcode.py` is. - -+ Include/ - - code.h - Header file for :cpy-file:`Objects/codeobject.c`; contains definition of - ``PyCodeObject``. - - opcode.h - One of the files that must be modified if :cpy-file:`Lib/opcode.py` is. - - + internal/ - - pycore_ast.h - Contains the actual definitions of the C structs as generated by - :cpy-file:`Python/Python-ast.c`. - "Automatically generated by :cpy-file:`Parser/asdl_c.py`". - - pycore_asdl.h - Header for the corresponding :cpy-file:`Python/ast.c` - - pycore_ast.h - Declares ``_PyAST_Validate()`` external (from :cpy-file:`Python/ast.c`). - - pycore_symtable.h - Header for :cpy-file:`Python/symtable.c`. ``struct symtable`` and - ``PySTEntryObject`` are defined here. - - pycore_parser.h - Header for the corresponding :cpy-file:`Parser/peg_api.c`. - - pycore_pyarena.h - Header file for the corresponding :cpy-file:`Python/pyarena.c`. - - -+ Objects/ - - codeobject.c - Contains PyCodeObject-related code (originally in - :cpy-file:`Python/compile.c`). - - frameobject.c - Contains the ``frame_setlineno()`` function which should determine - whether it is allowed to make a jump between two points in a bytecode. - -+ Lib/ - - opcode.py - Master list of bytecode; if this file is modified you must modify - several other files accordingly (see "`Introducing New Bytecode`_") - - importlib/_bootstrap_external.py - Home of the magic number (named ``MAGIC_NUMBER``) for bytecode - versioning. +* :cpy-file:`Lib/` + * :cpy-file:`opcode.py` + Master list of bytecode; if this file is modified you must modify + several other files accordingly (see "`Introducing New Bytecode`_") + + * :cpy-file:`importlib/_bootstrap_external.py` + Home of the magic number (named ``MAGIC_NUMBER``) for bytecode + versioning. Known Compiler-related Experiments From aad34d282a45f3d20be8a89e58cc03a1b1ba085d Mon Sep 17 00:00:00 2001 From: Ezio Melotti Date: Fri, 9 Dec 2022 04:00:03 -0700 Subject: [PATCH 2/2] More reformatting. --- internals/compiler.rst | 175 ++++++++++++++++++----------------------- 1 file changed, 77 insertions(+), 98 deletions(-) diff --git a/internals/compiler.rst b/internals/compiler.rst index 9e8f45fc72..2785aab51d 100644 --- a/internals/compiler.rst +++ b/internals/compiler.rst @@ -525,132 +525,111 @@ Important Files =============== * :cpy-file:`Parser/` - * :cpy-file:`Parser/Python.asdl` - ASDL syntax file - - * :cpy-file:`Parser/asdl.py` - Parser for ASDL definition files. Reads in an ASDL description - and parses it into an AST that describes it. - - * :cpy-file:`Parser/asdl_c.py` - "Generate C code from an ASDL description." Generates - :cpy-file:`Python/Python-ast.c` and :cpy-file:`Include/internal/pycore_ast.h`. - - * :cpy-file:`Parser/parser.c` - The new PEG parser introduced in Python 3.9. - Generated by :cpy-file:`Tools/peg_generator/pegen/c_generator.py` - from the grammar :cpy-file:`Grammar/python.gram`. Creates the AST from - source code. Rule functions for their corresponding production rules - are found here. - - * :cpy-file:`Parser/peg_api.c` - Contains high-level functions which are used by the interpreter to - create an AST from source code . - - * :cpy-file:`Parser/pegen.c` - Contains helper functions which are used by functions in - :cpy-file:`Parser/parser.c` to construct the AST. Also contains helper - functions which help raise better error messages when parsing source - code. - - * :cpy-file:`Parser/pegen.h` - Header file for the corresponding :cpy-file:`Parser/pegen.c`. Also contains - definitions of the ``Parser`` and ``Token`` structs. + * :cpy-file:`Parser/Python.asdl`: ASDL syntax file. + + * :cpy-file:`Parser/asdl.py`: Parser for ASDL definition files. + Reads in an ASDL description and parses it into an AST that describes it. + + * :cpy-file:`Parser/asdl_c.py`: Generate C code from an ASDL description. + Generates :cpy-file:`Python/Python-ast.c` and + :cpy-file:`Include/internal/pycore_ast.h`. + + * :cpy-file:`Parser/parser.c`: The new PEG parser introduced in Python 3.9. + Generated by :cpy-file:`Tools/peg_generator/pegen/c_generator.py` + from the grammar :cpy-file:`Grammar/python.gram`. Creates the AST from + source code. Rule functions for their corresponding production rules + are found here. + + * :cpy-file:`Parser/peg_api.c`: Contains high-level functions which are + used by the interpreter to create an AST from source code. + + * :cpy-file:`Parser/pegen.c`: Contains helper functions which are used + by functions in :cpy-file:`Parser/parser.c` to construct the AST. + Also contains helper functions which help raise better error messages + when parsing source code. + + * :cpy-file:`Parser/pegen.h`: Header file for the corresponding + :cpy-file:`Parser/pegen.c`. Also contains definitions of the ``Parser`` + and ``Token`` structs. * :cpy-file:`Python/` - * :cpy-file:`Python/Python-ast.c` - Creates C structs corresponding to the ASDL types. Also - contains code for marshalling AST nodes (core ASDL types have - marshalling code in :cpy-file:`Python/asdl.c`). "File automatically - generated by :cpy-file:`Parser/asdl_c.py`". This file must be - committed separately after every grammar change is committed since - the ``__version__`` value is set to the latest grammar change - revision number. + * :cpy-file:`Python/Python-ast.c`: Creates C structs corresponding to + the ASDL types. Also contains code for marshalling AST nodes (core + ASDL types have marshalling code in :cpy-file:`Python/asdl.c`). + "File automatically generated by :cpy-file:`Parser/asdl_c.py`". + This file must be committed separately after every grammar change + is committed since the ``__version__`` value is set to the latest + grammar change revision number. - * :cpy-file:`Python/asdl.c` - Contains code to handle the ASDL sequence type. Also has code - to handle marshalling the core ASDL types, such as number and - identifier. Used by :cpy-file:`Python/Python-ast.c` for marshalling - AST nodes. + * :cpy-file:`Python/asdl.c`: Contains code to handle the ASDL sequence type. + Also has code to handle marshalling the core ASDL types, such as number + and identifier. Used by :cpy-file:`Python/Python-ast.c` for marshalling + AST nodes. - * :cpy-file:`Python/ast.c` - Used for validating the AST. + * :cpy-file:`Python/ast.c`: Used for validating the AST. - * :cpy-file:`Python/ast_opt.c` - Optimizes the AST. + * :cpy-file:`Python/ast_opt.c`: Optimizes the AST. - * :cpy-file:`Python/ast_unparse.c` - Converts the AST expression node back into a string - (for string annotations). + * :cpy-file:`Python/ast_unparse.c`: Converts the AST expression node + back into a string (for string annotations). - * :cpy-file:`Python/ceval.c` - Executes byte code (aka, eval loop). + * :cpy-file:`Python/ceval.c`: Executes byte code (aka, eval loop). - * :cpy-file:`Python/compile.c` - Emits bytecode based on the AST. + * :cpy-file:`Python/compile.c`: Emits bytecode based on the AST. - * :cpy-file:`Python/symtable.c` - Generates a symbol table from AST. + * :cpy-file:`Python/symtable.c`: Generates a symbol table from AST. - * :cpy-file:`Python/peephole.c` - Optimizes the bytecode. + * :cpy-file:`Python/peephole.c`: Optimizes the bytecode. - * :cpy-file:`Python/pyarena.c` - Implementation of the arena memory manager. + * :cpy-file:`Python/pyarena.c`: Implementation of the arena memory manager. - * :cpy-file:`Python/wordcode_helpers.h` - Helpers for generating bytecode. + * :cpy-file:`Python/wordcode_helpers.h`: Helpers for generating bytecode. - * :cpy-file:`Python/opcode_targets.h` - One of the files that must be modified if :cpy-file:`Lib/opcode.py` is. + * :cpy-file:`Python/opcode_targets.h`: One of the files that must be + modified if :cpy-file:`Lib/opcode.py` is. * :cpy-file:`Include/` - * :cpy-file:`code.h` - Header file for :cpy-file:`Objects/codeobject.c`; contains definition of - ``PyCodeObject``. + * :cpy-file:`Include/code.h`: Header file for :cpy-file:`Objects/codeobject.c`; + contains definition of ``PyCodeObject``. - * :cpy-file:`opcode.h` - One of the files that must be modified if :cpy-file:`Lib/opcode.py` is. + * :cpy-file:`Include/opcode.h`: One of the files that must be modified if + :cpy-file:`Lib/opcode.py` is. - * :cpy-file:`internal/` - * :cpy-file:`pycore_ast.h` - Contains the actual definitions of the C structs as generated by - :cpy-file:`Python/Python-ast.c`. - "Automatically generated by :cpy-file:`Parser/asdl_c.py`". + * :cpy-file:`Include/internal/pycore_ast.h`: Contains the actual definitions + of the C structs as generated by :cpy-file:`Python/Python-ast.c`. + "Automatically generated by :cpy-file:`Parser/asdl_c.py`". - * :cpy-file:`pycore_asdl.h` - Header for the corresponding :cpy-file:`Python/ast.c` + * :cpy-file:`Include/internal/pycore_asdl.h`: Header for the corresponding + :cpy-file:`Python/ast.c`. - * :cpy-file:`pycore_ast.h` - Declares ``_PyAST_Validate()`` external (from :cpy-file:`Python/ast.c`). + * :cpy-file:`Include/internal/pycore_ast.h`: Declares ``_PyAST_Validate()`` + external (from :cpy-file:`Python/ast.c`). - * :cpy-file:`pycore_symtable.h` - Header for :cpy-file:`Python/symtable.c`. ``struct symtable`` and - ``PySTEntryObject`` are defined here. + * :cpy-file:`Include/internal/pycore_symtable.h`: Header for + :cpy-file:`Python/symtable.c`. ``struct symtable`` and ``PySTEntryObject`` + are defined here. - * :cpy-file:`pycore_parser.h` - Header for the corresponding :cpy-file:`Parser/peg_api.c`. + * :cpy-file:`Include/internal/pycore_parser.h`: Header for the + corresponding :cpy-file:`Parser/peg_api.c`. - * :cpy-file:`pycore_pyarena.h` - Header file for the corresponding :cpy-file:`Python/pyarena.c`. + * :cpy-file:`Include/internal/pycore_pyarena.h`: Header file for the + corresponding :cpy-file:`Python/pyarena.c`. * :cpy-file:`Objects/` - * :cpy-file:`codeobject.c` - Contains PyCodeObject-related code (originally in - :cpy-file:`Python/compile.c`). + * :cpy-file:`Objects/codeobject.c`: Contains PyCodeObject-related code + (originally in :cpy-file:`Python/compile.c`). - * :cpy-file:`frameobject.c` - Contains the ``frame_setlineno()`` function which should determine - whether it is allowed to make a jump between two points in a bytecode. + * :cpy-file:`Objects/frameobject.c`: Contains the ``frame_setlineno()`` + function which should determine whether it is allowed to make a jump + between two points in a bytecode. * :cpy-file:`Lib/` - * :cpy-file:`opcode.py` - Master list of bytecode; if this file is modified you must modify - several other files accordingly (see "`Introducing New Bytecode`_") + * :cpy-file:`Lib/opcode.py`: Master list of bytecode; if this file is + modified you must modify several other files accordingly + (see "`Introducing New Bytecode`_") - * :cpy-file:`importlib/_bootstrap_external.py` - Home of the magic number (named ``MAGIC_NUMBER``) for bytecode - versioning. + * :cpy-file:`Lib/importlib/_bootstrap_external.py`: Home of the magic number + (named ``MAGIC_NUMBER``) for bytecode versioning. Known Compiler-related Experiments