Generating code for the ceval loop/switch

[gvanrossum]

I'm collecting ideas for discussion, and this is the first one.

@markshannon Can you clear up any misunderstandings in my summary of your idea here?

IIUC Mark's idea is that we'd replace the big switch (and perhaps the loop around it as well) with code generated by a script from "opcode definitions", a bit similar to the way Argument Clinic works. Not sure if the opcode definition would be inline as Argument Clinic does or if they would be C snippets contained in a separate file.

Somehow this would allow us to create a version of the main loop that doesn't have to check for tracing on every opcode.

Not sure of other benefits yet.

[markshannon]

The main benefit is maintainability.
E.g, we can:

• generate different interpreters for Windows and Linux without drowning in macros.
• add things like super-instructions automatically, without code duplication.
• easily experiment with different interpreter layouts and dispatch mechanisms.

[markshannon]

Another reason, is debugging and instrumentation.
Adding timers, counters or logging in the template is so much nicer than adding macros to every instruction.

[ericsnowcurrently]

One thing Guido and I were discussing the other day is generating one copy of the eval loop with all the Python-level tracing/profiling machinery, and one without. We'd default to the latter and switch to the former when tracing is turned on. That could be facilitated by PyInterpreterState.eval_frame.

[markshannon]

Currently, we seem to going down the manual route for adding new instructions. Which fine.
We may well want to factor out the code for automatic layout of the interpreter, and generation of IR for the compiler, but that can be done as needed.

[gvanrossum]

Reopening, see #454 (comment)

[jneb]

Of course this is a great idea, making it easier to experiment with opcodes; especially when multiple people are tweaking.
I guess it would also make it easier to change one or more of the operations, for example for the register based interpreter.

[gvanrossum]

We now have a working generator (see Tools/cases_generator). While there's a whole lot more to do, let's close this and discuss details in separate issues under the label epic-generator.