Fails to build in Alpine 3.9

[certik]

Here is the error:

      [ 70%] No install step for 'download_ninja_source'
      [ 80%] Completed 'download_ninja_source'
      [ 80%] Built target download_ninja_source
      Scanning dependencies of target build_ninja
      [ 90%] Generating ../../../src/ninja
      ./src/disk_interface.cc: In member function 'virtual TimeStamp RealDiskInterface::Stat(const string&, std::__cxx11::string*) const':
      ./src/disk_interface.cc:219:51: error: 'struct stat' has no member named 'st_mtimensec'; did you mean 'st_mtim'?
         return (int64_t)st.st_mtime * 1000000000LL + st.st_mtimensec;
                                                         ^~~~~~~~~~~~
                                                         st_mtim

I can provide the exact steps to reproduce. This looks like a bug, as things should compile. I think this is related to Alpine using musl instead of glibc.

[jcfr]

After #23 is integrated and a new set of wheels is generated, it may be possible to install the available binary wheels.

This looks like a bug

Most likely, the following should be updated into the upstream project:

https://github.com/ninja-build/ninja/blob/b25c08bda4949192c69cea4cee057887341a2ffc/src/disk_interface.cc#L205-L220

[certik]

@jcfr thanks! I am interested in getting this working, and I am happy to help. The motivation is that LFortran's Python package depends on Ninja and so I want things to just pip install on all platforms. It already seems to work on Linux, Mac, Windows, but currently fails on Alpine linux (which I use as Docker images for fast CI builds) due to this bug.

#23 is merged.

I am new to Python wheels --- when reading the documentation I was left with the impression that binary wheels for linux can only be made for the manylinux1 compatible linux environments, and they currently require glibc, so Alpine is not manylinux1 compatible. You can hack it to be, as I did here:

scikit-build/cmake-python-distributions#69 (comment)

But then it fails, as described in the comment, because glic is not really compatible with musl.

When you said

it may be possible to install the available binary wheels.

Can you be more specific? Are those manylinux1 compatible? If so, pip will not install them on Alpine.

My understanding was that on Alpine, pip install will install from source. Which is fine, because I can prebuilt the Docker images. As long as things actually build from source.

How can I help to resolve this?

[certik]

Here is how to reproduce this error:

$ docker run -it alpine:3.9
# apk add python3 python3-dev cmake make g++
# ln /usr/bin/python3 /usr/bin/python
# ln /usr/bin/pip3 /usr/bin/pip
# pip install scikit-build
# pip install ninja
[...]
    [ 90%] Generating ../../../src/ninja
    ./src/disk_interface.cc: In member function 'virtual TimeStamp RealDiskInterface::Stat(const string&, std::__cxx11::string*) const':
    ./src/disk_interface.cc:219:51: error: 'struct stat' has no member named 'st_mtimensec'; did you mean 'st_mtim'?
       return (int64_t)st.st_mtime * 1000000000LL + st.st_mtimensec;
                                                       ^~~~~~~~~~~~
                                                       st_mtim

Note: I get exactly the same error if I install the latest version by pip install ninja==1.9.0.post1.

[certik]

I debugged the issue. The problem is that this line should be compiled on Alpine:

https://github.com/ninja-build/ninja/blob/28a7d1491367de7b39c854d166114f76d272f04f/src/disk_interface.cc#L215

But instead this line gets compiled:

https://github.com/ninja-build/ninja/blob/28a7d1491367de7b39c854d166114f76d272f04f/src/disk_interface.cc#L219

and fails. To fix it, we have to define some preprocessor directives that the code expects, e.g., we can define _BSD_SOURCE. Here is one way to get it installed in Alpine:

$ docker run -it alpine:3.9
# apk add python3 python3-dev cmake make g++
# ln /usr/bin/python3 /usr/bin/python
# ln /usr/bin/pip3 /usr/bin/pip
# pip install scikit-build
# export CXXFLAGS="-D_BSD_SOURCE"
# pip install ninja

Now it works.

How should this be fixed so that one does not have to execute export CXXFLAGS="-D_BSD_SOURCE" by hand?

[jcfr]

support for building on Alpine

How should this be fixed so that one does not have to execute export CXXFLAGS="-D_BSD_SOURCE" by hand?

We could change this line with the following:

${CMAKE_COMMAND} -E  env CXXFLAGS=-D_BSD_SOURCE ${PYTHON_EXECUTABLE} configure.py --bootstrap

Or more complete solution would be:

set(bootstrap_command ${PYTHON_EXECUTABLE} configure.py --bootstrap)
# Explicitly defining _BSD_SOURCE is required to support building the wheel on Alpine
# See issue  #22
if(UNIX AND NOT APPLE)
  set(bootstrap_command -E  env ${CMAKE_COMMAND} CXXFLAGS=-D_BSD_SOURCE ${bootstrap_command})
endif()
add_custom_command(
    COMMAND ${bootstrap_command}
    OUTPUT ${ninja_executable}
    WORKING_DIRECTORY ${Ninja_SOURCE_DIR}
)

... and I then realized submitting a PR would be as easy ... see #25

Let me know if you prefer not to be added as co-author.

manylinux and musl

Can you be more specific? Are those manylinux1 compatible?

I was incorrect. I don't think they are.

To answer that question, could you run the following command in your python environment:

from pip._internal.pep425tags import get_supported
print('\n'.join('-'.join(x) for x in get_supported()))

This should list all supported tag on your platform.

As you reported, manylinux1 and manylinux2010 require glibc, and since musl and glibc does not seem to be ABI compatible, package have to explicitly compile for Alpine.

What happen if you running pip install numpy, I am assuming it downloading the sdist and trying to build from source ?

[jcfr]

You could test doing:

pip install git+https://github.com/jcfr/ninja-python-distributions@support-building-on-alpine#egg=ninja

See https://pip.pypa.io/en/stable/reference/pip_install/#vcs-support

[certik]

Yes, pip install numpy starts installing from source. Here are the tags on Alpine linux:

~ $ python
Python 3.6.8 (default, Apr  8 2019, 18:17:52) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pip._internal.pep425tags import get_supported
>>> print('\n'.join('-'.join(x) for x in get_supported()))
cp36-cp36m-linux_x86_64
cp36-abi3-linux_x86_64
cp36-none-linux_x86_64
cp35-abi3-linux_x86_64
cp34-abi3-linux_x86_64
cp33-abi3-linux_x86_64
cp32-abi3-linux_x86_64
py3-none-linux_x86_64
cp36-none-any
cp3-none-any
py36-none-any
py3-none-any
py35-none-any
py34-none-any
py33-none-any
py32-none-any
py31-none-any
py30-none-any

[certik]

You could test doing:

pip install git+https://github.com/jcfr/ninja-python-distributions@support-building-on-alpine#egg=ninja

See https://pip.pypa.io/en/stable/reference/pip_install/#vcs-support

That worked!!

[certik]

Note, I had to do this:

pip install scikit-build
apk add python3 python3-dev cmake make g++
pip install git+https://github.com/jcfr/ninja-python-distributions@support-building-on-alpine#egg=ninja

You have to install scikit-build by hand, otherwise ninja fails. That's another bug, one should depend on scikit-build in pyproject.toml.

The other issue is that you have to install cmake in order to install ninja, and you need ninja to install cmake, so it's a chicken and an egg problem. So I install cmake using apk (Alpine's package manager), then pip install ninja, and finally after that you can pip install cmake. That's not optimal -- I thought the point of ninja and cmake packages was precisely that I do not need cmake to have installed on my system and can just pip install it.

[jcfr]

automatic install of scikit-build

one should depend on scikit-build in pyproject.toml

This PR should take care of this: #26

Could you test ?

pip install git+https://github.com/jcfr/support-pep-518@support-building-on-alpine#egg=ninja

a simple approach for distributing Alpine wheel

Here are the tags on Alpine linux:

Since it only list *-linux_x86_64, it will not grab wheels from pypi.

Waiting pypi officially host Alpine wheel, here is an idea:

Release assets of a particular github project could be used to host generated binary wheels.

Such wheels could be installed doing:

pip install -f https://github.com/awesome-org/alpine-python-wheels/releases/tag/3.9

This means that after uploading wheels built for Alpine 3.9 and upload them as GitHub release assets into a release named 3.9, you wouldn't have to build the package from source ...

https://github.com/scikit-build/alpine-python-wheels

[jcfr]

Ps: Headed on a 🏃‍♂️ , will be back shortly.

Then, we will:

• do an other release of ninja
• look at building cmake with manylinux1 so that we can re-upload the latest CMake sdist on pypi. We add to remove them, see this for details: Revisit use of manylinux2010 to build wheel cmake-python-distributions#78

[grische]

@jcfr as far as I understand, the CXXFLAGS override is the currently only workaround, isn't it?

So my Dockerfile would have to look like this (just for ninja to build), including the fix for #27

FROM alpine:3.10

# install python3 (incl. pip3)
RUN apk add --no-cache --update python3

# install ninja dependency manually, see bug #27 
RUN pip3 install 'scikit-build==0.10.0'

# add build time dependencies
RUN apk add --no-cache python3-dev cmake make g++

# help ninja build itself
ENV CXXFLAGS="-D_BSD_SOURCE"

# build ninja
RUN pip3 install ninja

[jcfr]

Thanks for the follow up.

I understand, the CXXFLAGS override is the currently only workaround, isn't it?

For reference, find below the error reported if _BSD_SOURCE is not specified. It looks like it is related to what was discussed in the ninja issue tracker at ninja-build/ninja#1510

Ideally, the ninja sources should be fixed. If for some reason this is not possible, an other option could be to update ninja-python-distributions/CMakeLists.txt

Error:

./src/disk_interface.cc: In member function 'virtual TimeStamp RealDiskInterface::Stat(const string&, std::__cxx11::string*) const':
  ./src/disk_interface.cc:219:51: error: 'struct stat' has no member named 'st_mtimensec'; did you mean 'st_mtim'?
     return (int64_t)st.st_mtime * 1000000000LL + st.st_mtimensec;
                                                     ^~~~~~~~~~~~
                                                     st_mtim

[bsolomon1124]

@jcfr is there a reason that all of

• https://github.com/jcfr/support-pep-518
• https://github.com/jcfr/ninja-python-distributions@support-building-on-alpine,
• https://github.com/scikit-build/alpine-python-wheels

have been deleted? Is there a working solution to this issue?

[jcfr]

I think that due to the lack of manpower to maintain the build of Alpine wheels, I removed the branch and repo I created. Honestly I don't recall the details, and I should have add a note here.

All of that said, is it something you would like to help move forward ?

[mayeut]

Fixed in 1.10.0.post3

Wheel for musllinux will be available once all tooling around PEP 656 is available.

For now, the following works on alpine:3.9:

apk update
apk add python3 py3-pip cmake make g++
python3 -m pip install -v ninja
ninja --version
  1.10.0.git.kitware.jobserver-1