Name clash with multiple python impls

[SoapGentoo]

On Gentoo, we usually build Boost with multiple python impls. Since 1.72 we are getting name clashes when building with MPI and multiple python impls:

error: Name clash for '<p/var/tmp/portage/dev-libs/boost-1.72.0/work/boost_1_72_0-abi_x86_64.amd64/stage/lib>mpi.so'
error: 
error: Tried to build the target twice, with property sets having 
error: these incompatible properties:
error: 
error:     -  <include>/usr/include/python2.7 <python>2.7 <xdll-path>/var/tmp/portage/dev-libs/boost-1.72.0/work/boost_1_72_0-abi_x86_64.amd64/bin.v2/libs/mpi/build/gcc-9.2/gentoorelease/local-visibility-global/pch-off/python-2.7/threading-multi/visibility-hidden <xdll-path>/var/tmp/portage/dev-libs/boost-1.72.0/work/boost_1_72_0-abi_x86_64.amd64/bin.v2/libs/python/build/gcc-9.2/gentoorelease/pch-off/python-2.7/threading-multi/visibility-hidden
error:     -  <include>/usr/include/python3.6m <python>3.6 <xdll-path>/var/tmp/portage/dev-libs/boost-1.72.0/work/boost_1_72_0-abi_x86_64.amd64/bin.v2/libs/mpi/build/gcc-9.2/gentoorelease/local-visibility-global/pch-off/python-3.6/threading-multi/visibility-hidden <xdll-path>/var/tmp/portage/dev-libs/boost-1.72.0/work/boost_1_72_0-abi_x86_64.amd64/bin.v2/libs/python/build/gcc-9.2/gentoorelease/pch-off/python-3.6/threading-multi/visibility-hidden

this is very likely due to the change in 3ecbf83. @Lastique @pdimov any ideas on how we could fix this? I tried injecting all sorts of tags into the Jamfile, but to no avail, it always crashes with the name clash error above. Our bug report: https://bugs.gentoo.org/703036 building it against a single python impl solves the issue, but isn't a tractable solution for gentoo users.

[Lastique]

As a workaround, you could invoke b2 separately for every Python version you require.

As to why b2 fails in a single run, I'm not sure. Python version already seems to be part of the target path, so I don't understand the problem.

[SoapGentoo]

While that's a possibility, it's prohibitively expensive if you have 5 python impls enabled. @stefanseefeld seems to suggest that Boost.MPI should do the same thing in boostorg/python#203 (comment). I've tried a lot, but have failed to get any workable solution that tags all Boost.MPI libs in a similar fashion to Boost.Python.

[Lastique]

While that's a possibility, it's prohibitively expensive if you have 5 python impls enabled.

Why? You don't have to build the whole Boost for each Python, just Boost.Python and Boost.MPI, and you have to do this in a single run anyway.

[pdimov]

Building with both 2 and 3 in a single run was certainly intended to be supported, and breaking it is no good.

[pdimov]

As far as I can see, fixing this should be as easy as copying these two lines:

https://github.com/boostorg/mpi/blob/develop/build/Jamfile.v2#L136-L137

to the corresponding place at

https://github.com/boostorg/mpi/blob/develop/build/Jamfile.v2#L171

and then adding PYTHON_EXTENSION to the list STATIC_LIB SHARED_LIB IMPORT_LIB at

https://github.com/boostorg/mpi/blob/develop/build/Jamfile.v2#L61

and

https://github.com/boostorg/mpi/blob/develop/build/Jamfile.v2#L67

But I can't test this at the moment because I've no idea how to set up MPI under Windows and have no time to figure it out.

Let me know if this doesn't work and I'll try to look into it more deeply.

[stefanseefeld]

Please consider cloning the Boost.Python re-implementation of the python-tag function (https://github.com/boostorg/python/blob/develop/Jamfile#L52-L68) to add a Python-version-specific suffix to the respective libraries.
Once that works correctly for Boost.MPI, we can fix the upstream version of that function (https://github.com/boostorg/boost/blob/master/boostcpp.jam#L222-L250) and get rid of the overrides in both Boost.Python as well as Boost.MPI.

[pdimov]

It's already cloned. https://github.com/boostorg/mpi/blob/develop/build/Jamfile.v2#L58-L74

[pdimov]

In addition, the upstream version seems to do the same thing as these local overrides. What specifically do you want fixed?

[SoapGentoo]

This is my current patch:

--- a/libs/mpi/build/Jamfile.v2
+++ b/libs/mpi/build/Jamfile.v2
@@ -58,13 +58,13 @@
 rule python-tag ( name : type ? : property-set )
 {
     local result = $(name) ;
-    if $(type) in STATIC_LIB SHARED_LIB IMPORT_LIB
+    if $(type) in STATIC_LIB SHARED_LIB IMPORT_LIB python-extension
     {
         local version = [ $(property-set).get <python> ] ;
         local lib-suffix = [ version-suffix $(version) ] ;
         result = $(result)$(lib-suffix) ;
     }
-    if $(type) in STATIC_LIB SHARED_LIB IMPORT_LIB && $(PYTHON_ID)
+    if $(type) in STATIC_LIB SHARED_LIB IMPORT_LIB python-extension && $(PYTHON_ID)
     {
         result = $(result)-$(PYTHON_ID) ;
     }
@@ -168,6 +168,8 @@
                 <link>shared:<define>BOOST_PYTHON_DYN_LINK=1
                 <link>shared <runtime-link>shared
                 <python-debugging>on:<define>BOOST_DEBUG_PYTHON
+                -<tag>@$(BOOST_JAMROOT_MODULE)%$(BOOST_JAMROOT_MODULE).tag
+                <tag>@$(__name__).python-tag
               ;
 
             mpi_python_libs = boost_mpi_python mpi ;

and it doesn't solve the issue. My gut feeling is that it's related to the python-extension not getting the tag.

[SoapGentoo]

I have removed the if $(type) in STATIC_LIB SHARED_LIB IMPORT_LIB conditional, thereby applying the python-tag modifications unconditionally, and still get the unmodified mpi.so target.

[pdimov]

Should be PYTHON_EXTENSION, not python-extension. Although if with the if removed it still doesn't apply, that's not the problem.

[SoapGentoo]

@pdimov before trying python-extension, I tried PYTHON_EXTENSION too, still fails.

[pdimov]

@SoapGentoo You're right, the superproject definition of tag needs PYTHON_EXTENSION added too for this to work.

[pdimov]

Unrelated to this build problem, the addition of mpi.so to boost-install is questionable for other reasons; Boost should in all probability not be installing a file named mpi.so to /usr/lib (which is the default for b2 install.) I suspect that we just don't want mpi in the boost-install line. (And if we insist on installing it, it should be prefixed with boost_ like everything else, and named something like boost_mpi_python_extension.so.)

[pdimov]

I can fix this build issue by adding PYTHON_EXTENSION here:

https://github.com/boostorg/boost/blob/master/boostcpp.jam#L188

and perhaps here for consistency:

https://github.com/boostorg/boost/blob/master/boostcpp.jam#L226

However, this will likely break Boost.Python's examples such as this one:

https://github.com/boostorg/python/blob/develop/example/quickstart/Jamfile#L15

which at the moment assume an unadorned name for the extension. If I make the change in boostcpp.jam, this example will start building something like extending-mt-gd-x64-1_73.pyd (or whatever the exact name will turn out to be) on Windows, and will in all probability fail to load it from Python.

So I can't go ahead with the change before all "Python stakeholders" (so to speak) weigh in and tell me what the behavior when building a Python extension .dll/.so ought to be when --layout is tagged or versioned.

(The example as-is will already acquire a version suffix (extending27.so) if built with python=2.7, and I'm not sure how it tolerates that.)