Deprecate the private _PyUnicodeWriter C API

[vstinner]

Hi,

I would like to deprecate private C APIs which now have a replacement in the public C API, especially deprecate the private _PyUnicodeWriter API.

Since Python 3.14 offers a new clean public PyUnicodeWriter C API (tested and documented), I propose to deprecate the private _PyUnicodeWriter C API which is not tested and not documented. I wanted to only deprecate for 2 Python versions (3.14 and 3.15), but Inada-san asked me to deprecate it for 4 releases (Python 3.14 to 3.17) and so only remove the API in Python 3.18.

@encukou is in disagreement with me, and asks to either do nothing, or only deprecate and not schedule the removal yet. See the discussion on Discourse.

A code search on PyPI top 8,000 projects on _PyUnicodeWriter_Init only finds 5 projects using it:

• asyncpg (0.30.0)
• catboost (1.2.7)
• cython (3.0.11)
• immutables (0.21)
• orjson (3.10.15)

IMO 4 years is more than enough to update these 5 projects to the public C API, knowing that the pythoncapi-compat project already provides the new public API on Python 3.6 and newer.

My motivation to deprecate (and then remove) private functions is to:

• Clarify the scope of the C API: define better what can be used outside Python, and which API is really “internal”.
• Provide better APIs, less error-prone, well tested, documented. For example, public PyDict_Pop() has a better API than private _PyDict_Pop(), the public API is easier to use and less error-prone.
• Make the C API smaller, and so easier to maintain in CPython, and easier to implement in other Python implementations.

The private _PyUnicodeWriter C API has multiple issues:

• It has low-level functions such as _PyUnicodeWriter_Prepare(writer, length, maxchar) which are specific to PEP 393 internals. I would like to get rid of these to get more freedom on how the public PyUnicodeWriter API is implemented (since it's currently implemented on top of the private API).
• The API allocates a structure on the stack, structure members are public.
• The private API has no documentation and is not tested. Removing it reduces the size of the C API and the maintenance burden.
• It's also a way to promote the public C API which doesn't expose PEP 393 internals.

Discussions:

• gh-128863: Deprecate the private _PyUnicodeWriter API python/cpython#129245
• https://discuss.python.org/t/remove-again-private-apis-which-now-have-new-public-replacement/68081

---

I propose 3 choices:

• do nothing (keep the private API forever)
• only deprecate
• deprecate and remove in Python 3.18

Vote to do nothing:

• @encukou
• @erlend-aasland
• @mdboom
• @serhiy-storchaka
• @vstinner
• @zooba

Vote to only deprecate:

• @encukou
• @erlend-aasland
• @mdboom
• @serhiy-storchaka
• @vstinner
• @zooba

Vote to deprecate and remove in Python 3.18:

• @encukou
• @erlend-aasland
• @mdboom
• @serhiy-storchaka
• @vstinner
• @zooba

[serhiy-storchaka]

I am fine with removing without deprecation or with very short deprecation period. This is an internal API, we have no obligation to support it. It was created for our own needs. Others used it on they own risk because there was no adequate public alternative, but now it is.

To be super nice, we should only deprecate the public C API when the alternative is available in all maintained Python versions (and preferably in all versions that get security fixes). I am more conservative here. But the private C API is a different thing. These that use it do it on their own risk, they should test their software for every bugfix release and they should have a backup plan for a case when it will be broken.

[encukou]

As in #7 (comment), I don't think we should remove private API pre-emptively, in bulk. That hurts users who ended up using the API.

For the guals:

Clarify the scope of the C API: define better what can be used outside Python, and which API is really “internal”.

A deprecation should suffice here.

Provide better APIs, less error-prone, well tested, documented. For example, public PyDict_Pop() has a better API than private _PyDict_Pop(), the public API is easier to use and less error-prone.

You already did. Thank you!

Make the C API smaller, and so easier to maintain in CPython, and easier to implement in other Python implementations.

Other implementations are not under any obligation to implement private/deprecated API. If they do that, it most likely means that the API is/was important to some user.

As for CPython, responding to users who are surprised that their code broke is itself a maintenance burden.

[vstinner]

Ping @zooba.

[zooba]

I think there's enough difference between the old API and the new API that we should clean up the old.

If it were basically just private vs. public aliases of the same API, I'd vote to keep the old around.

[encukou]

Count me as -0 so I don't block the work.

[vstinner]

The working group decided to deprecate the API and remove it in Python 3.18. I close the issue. Thanks for the vote.