Address external libevent once again

[rhc54]

Assume that a user has both a lib64 and lib directory under the location of $with_libevent. However, libevent itself was installed in the lib directory, not lib64. Assume also that $with_libevent/lib is in the default LD_LIBRARY_PATH.

We need the configury to be smart enough to see that lib64 exists, but that libevent isn't in it so that it moves along and finds libevent in the correct place. I'm not wild about this solution, but I found that just running AC_CHECK_LIB always succeeded with -L$with_libevent/lib64. I'm open to alternative solutions.

Signed-off-by: Ralph Castain [email protected]

[jsquyres]

Can we just ask the user to install libevent into lib64?

I'm guessing that this scenario only applies when libevent was built/installed from a source tarball (because if this were a distro-installed libevent and the distro was a) using lib64 and b) building a 64-bit version of libevent, then it would have taken care to install libevent into lib64), and therefore it's not unreasonable to ask the user to install it a specific way.

I.e., can the user just ./configure --prefix=blah --libdir=blah/lib64 when installing libevent?

I ask because having 64 bit libraries in prefix/lib and prefix/lib64 does really flout the convention of having lib (which, when there is also a lib64 dir, is usually only for 32 bit libs) and lib64 (where is typically only for 64 bit libs).

[jsquyres]

I guess what I'm trying to say is: Open MPI does try to have a very "full-service" configure script. We take a lot of care to generally make sure that configure "just works."

But this case is not just an uncommon user scenario -- it actually seems to go against common conventions. I'm therefore not sure we should apply our typical "full service" attitude in this scenario.

[rhc54]

It may come as a surprise to some, but I actually installed all the software in that location using simply a "prefix" in configure. The individual software packages chose, completely on their own, whether to use lib or lib64. So I suspect this is going to be more the rule rather than the exception case.

The problem with the code as it currently sits in master is that it solely checks for the existence of the $with_libevent/lib64 directory, and then decides that this is where libevent should be located. This isn't the default libevent behavior - by default, it installs under $with_libevent/lib.

That said, I'm comfortable revising this to simply allow the user to point directly to the lib directory where libevent is located in these cases.

[rhc54]

Just to clarify my comment. I do not have a mixed 32/64-bit system. Everything is 64-bits. What apparently happened is that one package I installed (looks like it was gcc) decided to install by default into <prefix>/lib64, while all other packages (libevent, hwloc, autotools, etc) installed into <prefix>/lib. This is what has created the confusion.

I think the bottom line is that this supposed convention isn't really being followed by the individual package developers - including OMPI, by the way. We just blindly install into <prefix>/lib without checking for the existence of a lib64.

So I'll reiterate that the scenario I am encountering is probably the more common situation and not the exception case. I'm trying to add some smarts to at least tell the user they have to do something more than the usual to work around it.

[rhc54]

Okay, I have found the problem, though I'm not sure how to solve it. The root cause of the problem is that OPAL_CHECK_PACKAGE isn't doing what we thought it was doing. Looking at that macro, it thinks it is checking if the library is present and usable in the provided libdir. However, what it actually is doing is seeing if the library is present and usable anywhere in your LD_LIBRARY_PATH plus your libdir.

In this case, libevent is in my LD_LIBRARY_PATH. As a result, passing any directory to OPAL_CHECK_PACKAGE results in a positive response, leaving the LDFLAGS incorrectly set. If I try passing --with-libevent=external, the it still finds the library, but doesn't tell me where it is (which is the info I'm trying to get so I can link a library against libevent).

[jsquyres]

@rhc54 You're right: Autoconf installs into $prefix/lib by default. So unless a package author does something specific (which libevent and OMPI don't), it'll always go to $prefix/lib unless the user overrides with --libdir=something_else. We don't override this to install into lib64 because it's a bit of an art to figure out what the installer wants... and it would end up being a[nother] heuristic.

My only point was that distros tend to do the --libdir=blah/lib64, so in a distro situation, all such packages (including libevent and OMPI) will be in lib64. It's typically going to only be in users-builds-from-source situations (where they don't override this default) that packages (such as libevent and OMPI) end up in prefix/lib when others -- perhaps installed via the distro -- end up in prefix/lib64. And since we can tell users that they should re-install into lib64, that might be a good enough situation...? (or, as Ralph points out, it's not a sure thing -- so perhaps our help message can suggest that users can try re-installing into lib64...?).

So, back to @rhc's last comment...

Yes, it was always expected that OPAL_CHECK_PACKAGE was a less-than-optimal heuristic. 😦 The problem (as you probably know) is that we can't set LD_LIBRARY_PATH to just the directory under test because then we might not find dependencies in the actual library that we're searching for (i.e., if those dependencies live elsewhere in LD_LIBRARY_PATH).

We for for the presence of the actual library file -- which is a good first check, but it's definitely not sufficient (because, for example, there could be 32/64 versions). So we also do the AC_CHECK_LIB thing. But it's admittedly not a perfect check for the reasons cited. 😦

So I think I'm more agreeing with @rhc54 more than anything else, but I still don't know what the correct answers are. 🤷‍♂️

I'm open to suggestions here...

[rhc54]

As I said in the commit message, I think we are concluding there is no good answer here. Accordingly, I've changed the opal/pmix configury so it no longer tries to link the standalone libpmix back to libevent. It means users who want to work with it will have to use mpicc to build, and will have to explicitly link -lpmix into their application, but I have no good alternative.

This commit just adds a little more verbose output to the external event configury and modifies opal/mca/pmix/pmix3x/configure.m4 as per above.

[ggouaillardet]

i think lib64 vs lib is a now a de facto standards when packaging both 64 and 32 bits libraries on linux.
as far as i am concerned, an ISV should package its software in /opt whereas distros should package theirs in /usr. Having said, that, i think Open MPI should try both directories (e.g. "full service").

One oddity is that when Open MPI is built as a 32 bits library, and libFOO is available in both 64 and 32 bits in a non standard location, it is likely the enduser will have to configure --with-FOO=/opt/FOO --with-FOO-libdir=/opt/FOO/lib.

My main issue is the library detection is not consistent accross the various configure files

• OPAL_CHECK_PACKAGE tries /opt/FOO/lib and if it fails, tries /opt/FOO/lib64.
• OPAL_CHECK_PMI_LIB tries /opt/FOO/lib64 if /opt/FOO/lib64/libFOO.* matches something, and if not, it tries /opt/FOO/lib if /opt/FOO/lib/libFOO.* matches something
• MCA_opal_event_external/Makefile tries /opt/FOO/lib64 if that directory exists, and if not, tries /opt/FOO/lib if this directory exists.

at first glance, OPAL_CHECK_PACKAGE looks like the best out of these three methods.
We can improve it by checking the existence of the library (libFOO.* ? libFOO.{a,so,dylib} ?), so if there
is no library in this directory, but the required library is somewhere in the LD_LIBRARY_PATH (or LIBRARY_PATH ?) we would abort.
For the sake of completion, if /opt/FOO/libFOO.so is present but with not the expected endianness, and libFOO.so is available with the expected endianness somewhere in the path, we would fail to abort, unless we explicitly test for the library endianness.

[rhc54]

I agree that OPAL_CHECK_PACKAGE needs a little updating to check for existence of the library in the directories. It should also update the provided lib directory variable to point to the directory where the library was eventually found.

Just to again be clear: this issue was talking about a pure 64-bit system, with software built/installed from tarballs and not rpms. The problem is that there is no standard, de facto or otherwise, for installing in to lib vs lib64 in this scenario. Some packages build into one, some into the other. Thus, you wind up with both a lib and lib64 in the target area, which is what causes the problem.