Provide clear documentation on not supporting free threaded Python

[rogerbinns]

Description

The latest cibuildwheel automatically enables free threaded Python 3.14 compilation. It isn't clear from the documentation how to disable it.

Doc search

The first result shows how to enable but not disable.

The second result shows enable option but there is no corresponding disable option.

It seems like enable uses cpython-freethreading while disabling requires skip using a glob pattern with t somewhere in the name, hoping that you have the exact right pattern and don't accidentally match an unintended t. Hence why clear documentation is needed.

My C extension uses the GIL so it compiles fine, but loading the extension re-enables the GIL. It also isn't clear if I should disable the building, or let it be published to pypi anyway.

Build log

No response

CI config

No response

[henryiii]

Before it was enabled by default, so not surprising that we missed showing how to disable it. :)

If you don't push to PyPI, then people using the (non-experimental) version of Python with free-threading won't be able to use your library (or it will build from source, if that works). Even if it requires the GIL, it's still better than not supporting that at all. For example, I wasn't able to try a package that required h5py on no-GIL Python, even though I don't care about loading files in threads (and it probably doesn't need to lock Python, just needs it's own global lock, but I'd be happy with just loading the file with the GIL).

Also, even worse, this means packages can't have any dependencies that don't have no-GIL builds, since you can't specify nogil as a selector yet.

We should add a disable example, and a little warning with some of the info above.

As for the skip string, it's "cp3*t-*", I've got it in a previous PR/Issue description. "cp31?t-*" also works until 3.20, though by that time I think free-threading will not only be default, I expect there won't be a GIL version.

[rogerbinns]

Thanks for the clarification - my extension builds and runs just fine on free threaded, but you get the message about GIL being enabled on importing it. Based on your advice, I'm happy to provide free threaded compatible builds to pypi that enable the GIL. Please also include in the doc if Python 3.14+ free threaded should be provided, or 3.13 too.

Just to confirm, cp314-* will do regular Python and cp314t-* will do free threaded. The reason behind this is my github action includes this:

      matrix:
       os: [ubuntu-latest, ubuntu-24.04-arm, windows-latest, macos-13, macos-14]
       pyver: [cp39, cp310, cp311, cp312, cp313]
...
        env:
          CIBW_BUILD: ${{ matrix.pyver }}-*

The reason is a build and test for one architecture, one Python version, and one C library takes 5 minutes. If I followed the standard instructions then on ubuntu-latest with AMD64 and x86, 7 versions of Python, glibc and musl, we are looking at 28 combinations which will take ~3 hours to complete, with the ARM runner being even slower. Using a matrix like the above means each individual matrix combination is only 20 minutes so the whole thing takes not too much longer than that due to parallel running.

I expect there won't be a GIL version.

One frustration so far is that there is no automated error detection. By that I mean I want compile time errors, or at least runtime errors on "wrong" code like sanitizers provide. The free threaded interpreter won't even build for me if I configure with the thread sanitizer. Hopefully this will be addressed.

[henryiii]

I think I remember free-threading and sanitizers as something mentioned in the latest Core.py podcast, so I think it will work eventually. Certainly the core devs have been prioritizing

Why is the ARM runner slower? Emulation is slow, but I've found the runners are very fast (possibly because they are newer machines). Is it building a dependency?

Yes, here's a expanded version:

      matrix:
       os: [ubuntu-latest, ubuntu-24.04-arm, windows-latest, windows-11-arm, macos-13, macos-15]
       pyver: [cp39, cp310, cp311, cp312, cp313, cp313t, cp314, cp314t]
       exclude:
         - os: windows-11-arm
           pyver: cp39
...
        env:
          CIBW_BUILD: ${{ matrix.pyver }}-*

(I don't remember what the minimum version for Windows ARM is, it might be 3.10 or 3.11, but I added an example just to fill it out more)

Dropping 3.9 would help a little in a couple of months. But it's more builds, something we are stuck with. Hopefully (a new?) Stable ABI will be supported for free threading eventually.

3.13t was experimental; I don't have a good feel for how important supporting it will be once 3.14 final is released. I'd rather say it's not important, and the focus should be on 3.14t+. But I don't have any feel yet for what the community will do with 3.13t once 3.14t is out. https://py-free-threading.github.io would be a good place for that sort of recommendation.

Also, Intel macOS (macos-13) is going away in September. :(

[henryiii]

Since you are using CPython's API directly, I think I'd focus on 3.14t+ and only backport if someone really wants it and you feel like supporting it, at least once you actually start support running freely. C API like PyCriticalSection work better in 3.14+.

[rogerbinns]

I'll stick to 3.14+ for free threaded builds.

ARM takes 23 minutes while AMD64 takes 19 for my build. Not a huge difference, and likely a side effect of a single very large translation unit. I used to use the qemu emulation - that was slow. It is also why I haven't enabled ppc, riscv, s390 etc. My project will build correctly from the source pypi distribution, but does require a local compiler.

In your new matrix above I see macos-15 while the docs still have macos-14. I hadn't spotted Windows ARM being added - nice! The docs don't have the cp39 exclude.

My version policy is that I do one more release after a Python version goes EOL, and then remove support for it, so cp39 will go away late in the year.

[rogerbinns]

I've done a release with your recommendations and updates above. It results in 41 jobs in the matrix of platform and Python version and 81 binary files at PyPI !!!

Using the matrix meant wall clock build time was half an hour, while using the approach in the cibuildwheel docs would have been 3 hours.

I'm looking forward to updated cibuildwheel docs about free threading.

[mxmlnkn]

I was also looking for how to disable the threaded version.

• cp14t failed to build on the macos-CI with unknown type name '__pyx_vectorcallfunc'. It seems to have disappeared after relaxing my cython < 3.1 requirement in pyprojects.toml.
• I had this requirement because of error: use of undeclared identifier 'PyDescr_NewMember', but I am not building for Python 3.8 anymore, so it seems to be fine. The linked fix commit seems to be more recent than the most recent Cython release, which is unfortunate.
• I am starting to have all these problems since relaxing my cibuildwheel~=2.0, i.e., after updating to cibuildwheel 3.0, which I did not do because of Python 3.8 support. But Python 3.8 seems to be broken now, even with cibuildwheel-2.23.3, because I get a 404 error when it tries to set up virtualenv https://github.com/pypa/get-virtualenv/blob/20.30.0/public/virtualenv.pyz?raw=true
• After fixing the above errors, the Windows build is failing and blocking me because of: LINK : fatal error LNK1104: cannot open file 'python314.lib' when building cp14t. I get the feeling that it should have linked python314**t**.lib instead, but I have no idea where I would even begin to change that.

Unfortunately, I am hopping from one bug to the next one without an end in sight.

[tfpf]

@mxmlnkn Were you able to fix the link error? My CI pipeline fails with a similar message.

LINK : fatal error LNK1104: cannot open file 'python314.lib'
error: command 'C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\bin\HostX86\x64\link.exe' failed with exit code 1104

If you didn't, perhaps I should create a new issue?

[mxmlnkn]

@tfpf I can still reproduce the problem on the Github Actions windows-latest runner. Yes, it would be nice if you could open a separate issue.

[mxmlnkn]

I am also getting these errors on another project, for which I already tried to enable ABI3:

   /opt/_internal/cpython-3.14.0-nogil/include/python3.14t/Python.h:51:4: error: #error "The limited API is not currently supported in the free-threaded build"
     51 | #  error "The limited API is not currently supported in the free-threaded build"
        |    ^~~~~

I am unsure how to handle this. I'll have to either not build 3.14t wheels or somehow detect building for that and disable the ABI3 build options in that case. Currently, I am setting them for Cython in the setup.py. It seems I need to check sysconfig.