[C API] Add explicit support for Cython to the C API

[markshannon]

BPO	45247
Nosy	@scoder, @vstinner, @markshannon, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-09-20.10:59:47.125>
labels = ['expert-C-API', 'type-feature']
title = '[C API] Add explicit support for Cython to the C API'
updated_at = <Date 2022-02-23.15:21:36.827>
user = 'https://github.com/markshannon'

bugs.python.org fields:

activity = <Date 2022-02-23.15:21:36.827>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['C API']
creation = <Date 2021-09-20.10:59:47.125>
creator = 'Mark.Shannon'
dependencies = []
files = []
hgrepos = []
issue_num = 45247
keywords = []
message_count = 7.0
messages = ['402224', '403812', '403819', '403821', '403823', '403894', '413804']
nosy_count = 4.0
nosy_names = ['scoder', 'vstinner', 'Mark.Shannon', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45247'
versions = []

[markshannon]

As the C API has evolved it has grown features in an ad-hoc way, driven by the needs to whoever has bothered to add the code.

Maybe we should be a bit more principled about this. Specifically we should make sure that there is a well defined interface between CPython and the other major components of the Python ecosystem.
The obvious places to start are with Cython and Numpy.

This issue deals specifically with Cython. I will leave it to someone who know more about Numpy to open an issue for Numpy.

Matching Cython issue: cython/cython#4382

This issue is an attempt to stop the annual occurrence of bugs like https://bugs.python.org/issue43760#msg393401

[vstinner]

We need to add C API to abstract access to structures used by Cython:

• bpo-39947: PyThreadState
• bpo-40421: PyFrameObject

[markshannon]

I disagree. All that is doing is locking in the current poor interface.

We do need to extend the C API for these uses, yes. But we need an API that addresses Cython's needs directly and at a higher level.

For example, Cython often wants to insert a frame into the call stack for debugging and introspection.

Rather than have Cython laboriously create a Python frame with fake code object, etc., it would make much more sense for us to offer an API to insert a "native" frame into the stack and update the line number.

E.g. something like:

int PushNativeFrame(const char *filename);
void PopNativeFrame(void);
int CurrentNativeFrame_SetLineNumber(int lineno);

We might even want to use such an API ourselves in some modules.

[vstinner]

Rather than have Cython laboriously create a Python frame with fake code object, etc., it would make much more sense for us to offer an API to insert a "native" frame into the stack and update the line number.

I agree with that.

[serhiy-storchaka]

I agree with Mark. Instead of exposing internal details in low-level API we should add more high-level API to satisfy concrete needs. It will give us more freedom of changing internals in future.

[vstinner]

My first attempt: Add PyThreadState_EnterTracing() and PyThreadState_LeaveTracing()
#28542

It adds an abstraction on accesses to PyThreadState.tracing and PyThreadState.cframe.use_tracing members.

[vstinner]

I created bpo-46836: "[C API] Move PyFrameObject to the internal C API".

[vstinner]

I closed #107076 as a duplicate of this issue.