Undefined references to HACL symbols when statically linking Python 3.12

[luis605]

Bug report

Bug description:

I am developing an application that uses Python 3.12's static library (libpython3.12.a). Due to some restrictions, I am required to use a package manager to install Python and cannot compile it from source. Additionally, I can't use dynamic libraries, so I must link python statically.

When linking libpython3.12.a to my project, I encounter undefined references to HACL (HACL*) symbols, such as:

/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu/libpython3.12.a(sha2module.o): in function `update_256':
(.text.unlikely+0x9ea): undefined reference to `python_hashlib_Hacl_Hash_SHA2_update_256'
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/13/../../../x86_64-linux-gnu/libpython3.12.a(sha2module.o): in function `SHA512Type_copy':
(.text.unlikely+0xcd3): undefined reference to `python_hashlib_Hacl_Hash_SHA2_copy_512'
[etc.]

These symbols appear to be related to the HACL library, which is included in Python 3.12's development. However, I cannot locate the corresponding object files or static libraries for HACL in the Python installation provided by the package manager.

I was previously using python3.11 - which worked flawlessly -, however the dev package has been removed from apt and other similar package managers. For this specific reason, I had to upgrade.

A quick note, the command nm -gA /usr/lib/x86_64-linux-gnu/*.a 2>/dev/null | grep python_hashlib_Hacl_Hash_SHA2_update_512 shows that these symbols are shipped with the static package: /usr/lib/x86_64-linux-gnu/libpython3.12.a:sha2module.o: U python_hashlib_Hacl_Hash_SHA2_update_512.

Steps to Reproduce:

1. Install Python 3.12 using your system's package manager (e.g., apt on Ubuntu).
2. Attempt to statically link the Python static library libpython3.12.a into a C/C++ project.
3. Observe linker errors related to undefined references to HACL symbols.

Expected Behavior:
The static library libpython3.12.a should include all necessary dependencies, including HACL, or provide a way to link against HACL statically.

Actual Behavior:
The linker fails with undefined references to HACL symbols, as the required HACL objects or static libraries are not provided.

Questions:

1. Are the HACL object files or static libraries supposed to be included in the Python 3.12 package from the package manager?
2. If not, is there a recommended way to obtain the necessary HACL static library for linking with libpython3.12.a? And if so, is it possible to do so without having to compile Python or any other large project?
3. Could this be an issue with the packaging of Python 3.12 in the package manager, or is it a broader issue with the static linking setup?
4. My primary goal is to link python statically. If there isn't any solution available that follows the requirements of using a package manager, are there any work arounds to bypass the linker issue?

Thank you for your assistance.

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

• gh-131298: eliminate HACL* static libraries for cryptographic modules #132438
• gh-131298: update Modules/Setup after HACL static linking changes #133012

[picnixz]

I think this issue is known but I can't remember where it is. I've statically built HACL* sources for HMAC in #130157. What should we do for 3.12? cc @gpshead @msprotz

[msprotz]

#116043 is the original report

I proposed a PR at the time #119320 but I don't think it was deemed critical enough for the PR to be merged

[luis605]

If the PR wasn't deemed necessary then it won't probably be implemented. Is there any workaround that doesn't include compiling Python from source?

[luis605]

A possible workaround, if you don't need HACL is to create and compile the following .cpp file in your project:

#include <cstddef>
#include <cstdlib>

extern "C" {
    void* python_hashlib_Hacl_Hash_SHA2_malloc_512() {}
    void python_hashlib_Hacl_Hash_SHA2_free_512(void* ptr) {}
    void python_hashlib_Hacl_Hash_SHA2_update_512(void*, const void*, size_t) {}
    void python_hashlib_Hacl_Hash_SHA2_digest_512(void*, void*) {}
    void python_hashlib_Hacl_Hash_SHA2_copy_512(void*, void*) {}
    void* python_hashlib_Hacl_Hash_SHA2_malloc_256() {}
    void python_hashlib_Hacl_Hash_SHA2_free_256(void* ptr) {}
    void python_hashlib_Hacl_Hash_SHA2_update_256(void*, const void*, size_t) {}
    void python_hashlib_Hacl_Hash_SHA2_digest_256(void*, void*) {}
    void python_hashlib_Hacl_Hash_SHA2_copy_256(void*, void*) {}
    void* python_hashlib_Hacl_Hash_SHA2_malloc_224() {}
    void* python_hashlib_Hacl_Hash_SHA2_malloc_384() {}
}

[picnixz]

Ah. I think I should have complicated the builds even more :D in my latest HMAC PR, I actually made all objects as being static because it was easier. I'll eliminate those for the upcoming 3.14 beta and make them fully dynamic (I think we can mostly do this now). My very very bad.

[picnixz]

There is #133012 that is pending but now it should work for Python 3.14 and later. I don't want to backport the fix though because 3.12 is security-only. Maybe a backport to 3.13 is possible though but the build process has been changed too much and I fear too many conflicts.