Can't include Python.h from C++ code on macOS Sonoma / Clang 15

[lunixbochs]

Bug report

Bug description:

Including Python.h on <3.13 from a .cpp file results in the following:

In file included from include/python3.11/Python.h:50:
include/python3.11/bytesobject.h:10:1: error: import of C++ module 'Darwin.C.stdarg' appears within extern "C" language linkage specification [-Wmodule-import-in-extern-c]

You can work around this on a per project basis with -Wno-module-import-in-extern-c, but that's still pretty annoying.

This is due to an include of a system header from inside an extern "C" block, which LLVM doesn't seem to like anymore with default flags on macOS.

I believe the changes related to #108769 will fix this. Can we get them backported?

CPython versions tested on:

3.11, 3.12

Operating systems tested on:

macOS

[vstinner]

What is your clang version?

[ronaldoussoren]

Also, please add a minimal example on how to reproduce the issue.

FWIW, I tried to reproduce using the following trivial code as a c++ file (.cc suffix):

#include <cctype>
#include "Python.h"

This worked fine when building with "cc -c repro.cc" using the clang included with the current Xcode 15.1 beta.

Version Details

Xcode:

Xcode 15.1
Build version 15C5042i

Clang:

Apple clang version 15.0.0 (clang-1500.1.0.2.4)
Target: arm64-apple-darwin23.1.0
Thread model: posix
InstalledDir: /Users/ronald/Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

[lunixbochs]

This triggers it for me:

cc `python3.11-config --includes` -fmodules -fcxx-modules test.cpp

[vstinner]

Can you provide the minimum test.cpp file reproducing the issue?

[ronaldoussoren]

Can you provide the minimum test.cpp file reproducing the issue?

The two-line file in my previous comment is enough to reproduce the issue (with 3.9 or later)

The error goes away if I add -Wno-module-import-in-extern-c to the compiler arguments, but that's almost certainly not the right fix (and I haven't tried building an extension with modules enabled and this flag to suppress the error).

Actually avoiding the error requires non-trivial reorganisation of our header files to ensure that all includes of system headers are done outside of extern "C" blocks.

[lunixbochs]

Actually avoiding the error requires non-trivial reorganisation of our header files to ensure that all includes of system headers are done outside of extern "C" blocks.

It looks like these changes for 3.13 already have the correct reorganization, they'd just need to be backported: #108769

[ronaldoussoren]

@vstinner : Are the changes in #108769 something that can be back ported to 3.11 and 3.12? I'm worried that moving includes around can affect 3th-party extensions.

Alternatively we can document that Python 3.12 and earlier don't support building with C++ modules enabled, and add a test based on my earlier message to 3.13 to ensure that building with modules continues working there.

[vstinner]

I reproduced a similar error using a manually created C++20 module.

Work in the Lib/test/test_cppext/ directory.

Create Lib/test/test_cppext/mymodule.cppm:

export module mymodule;

Create Lib/test/test_cppext/Makefile:

CXX=clang++
CXXFLAGS=-std=c++20 -fmodules -fprebuilt-module-path=. -fno-strict-overflow -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -O0 -fPIC

mymodule.pcm: mymodule.cppm
	$(CXX) $(CXXFLAGS) --precompile mymodule.cppm -o $@
	$(CXX) $(CXXFLAGS) -c mymodule.pcm -o mymodule.o

clean:
	rm -rf mymodule.pcm mymodule.o

Pre-compile mymodule.cppm using the Makefile in Lib/test/test_cppext/:

make

Modify Lib/test/test_cppext/extension.cpp, before #include "Python.h", add:

extern "C" {
    import mymodule;
}

Build the C++ extension with:

../../../python -m venv env
env/bin/python -m pip install setuptools
rm -rf build
CC=clang++ CXX=clang++ CFLAGS="-fmodules -fprebuilt-module-path=." CPYTHON_TEST_CPP_STD=c++20 CPYTHON_TEST_EXT_NAME=extname env/bin/python setup.py build_ext

Building the C++ extension fails with:

extension.cpp:10:1: error: unknown type name 'import'
    2 | import mymodule;        // import declaration
      | ^
1 error generated.

What I understood:

• On macOS, some system header files use import ... if built with C++.
• <Python.h> includes some system header files inside extern "C" { ... }. In this case, building <Python.h> with C++20 modules fails with an error.

In short:

• #include ... must not be done inside extern "C" { ... }.
• #include ... can be indirect, header A can include header B which includes a system header.
• Moving most system headers to Python.h makes it easier since Python.h doesn't use extern "C" { ... }.

[vstinner]

Moving most system headers to Python.h makes it easier since Python.h doesn't use extern "C" { ... }.

The problem also exists for the internal C API: see #115105.

[vstinner]

Since only Python 3.12 is impacted (Python 3.11 no longer gets bugfixes) and the lack of activity for one year, I suggest closing this old issue as "WONT FIX".

[vstinner]

This issue has been fixed in Python 3.13.