-s USE_XXX should accept -lXXX

[Beuc]

Hi,

I get a lot of errors like in my builds with 1.38.32 such as:

emcc -s USE_ZLIB=1 -o python Modules/python.o libpython2.7.a -ldl -lz -lm  
shared:ERROR: emcc: cannot find library "z" (`-s ERROR_ON_MISSING_LIBRARIES=0` to disable this error)

This is quite puzzling.
It would make sense for emcc to accept -lXXX when using a built-in port -s USE_XXX.

[sbc100]

The -s USE_FOO flags for libraries implicitly add the include paths and link flags that you need in order to use the library. You don't need to add additional flags. They would be definition be redundant. I don't love the USE_FOO system for including ports, but that is how it currently works.

I recently re-worked the ports so at least they have the correct names (libfoo.a).. but the name of the actual library produced by the port is still currently and implementation default and passing -lz implies that you know that filename is libz.a under the hood somewhere.

I kind of agree with your though, it would be nice some day to be able to consume ports simply by passing -lz. I imagine this being separate to the -s USE_FOO flags though. We could have embuilder be the way to build the libraries and have -lz be the way to consume them. Currently we have USE_ZLIB which means "build and consume".

[sbc100]

Are you saying that it used to work to pass -lz? Perhaps it was just failing silently prior to #8461?

[Beuc]

From a porter PoV, I ./configure the target project with appropriate CFLAGS/LDFLAGS, which include -L and -s XXX options. The target project's build system will then expect my configuration to satisfy its link dependencies, which it unconditionally specifies as -lXXX.

That's what I think it makes sense for -s USE_XXX to provide a high-precedence equivalent to -L/hidden-emscripten-path -lXXX. Otherwise I have to patch the target build system, which would be emscripten-specific, time consuming and error-prone.

The issue is indeed all the more present with #8461 - as emcc used to warn in the past, and now fails. I can pass -s ERROR_ON_MISSING_LIBRARIES=0 everywhere but that sounds like a work-around and not a proper fix.

[sbc100]

Yes, its seems that this has always been an issue just hidden by the fact that ERROR_ON_MISSING_LIBRARIES used default to false. I suggest you add ERROR_ON_MISSING_LIBRARIES=0 to get the old behaviour until we can come up with a more elegant solution.

[Beuc]

ERROR_ON_MISSING_LIBRARIES=0 is not not available anymore - any progress on the more elegant solution?
I'm starting to implement silly work-arounds like creating dummy libXXX.a so the build system can work, which is probably a bad sign ;)

[kripken]

Looks like ERROR_ON_MISSING_LIBRARIES was removed in #9223.

@sbc100 maybe we removed it too early? The workarounds @Beuc describes do sound bad :(

[sbc100]

No I still don't think ERROR_ON_MISSING_LIBRARIES is the right solution here.

@Beuc, why can't you remove -lz to you link flags? That seems like the correct solution for now.

[sbc100]

In the long run it would be nice to make -l work with emscripten ports. Today -s USE_ZLIB is the way to use the port. I'll leave this bug open so we get it fixed, but simply removing the offending argument for the link line seems like a good enough workaround until this is fixed.

[Beuc]

@Beuc, why can't you remove -lz to you link flags?

That's because, as I explained above, this is not a trivial project, but I'm interacting with moderately complex build systems, this concerns -lz but also -lSDL2* and -lfreetype, and it's more of a work-around to patch-up the various build systems than it is to make dummy .a libs.

[Beuc]

By the way, my idea of an elegant solution would be a simple include/ & lib/ hierarchy where ports would be installed (or symlinked/copied on demand). Something like https://mxe.cc/ does for cross-compiling to windows. It would silently add -I and -Lflags and so would work as the rest of my dependencies, could be easily overridden with a patched/in-progress port, etc.
For the record :)

[sbc100]

I think we are already moving in exactly that direction: #9353. Hopefully we will get there sooner rather than later.

[sbc100]

There are two problems emscripten faces that not all crosstools face:

1. How to handle the case of a read-only emscripten install (i think in this case we have to fall back to some kind $HOME-based solution for usr/local).
2. How do the auto-built ports interact with user-built ports (we never want to clobber things the user has installed by hand). We can hopefully handle this by having the ports system target an non-obvious location (e.g. usr/ports/lib rather than just usr/lib).

[Beuc]

Currently the ports do require write access to build and install the port, I wouldn't change that. Similarly MXC builds in $(checkout-path)/usr/bin. It doesn't matter to me if the ports are installed in a specific location, as a porter/cross-compiler I'm used to that. If we want to enable/disable them on demand the ports would probably reside not in Emscripten's global sysroot, but in a throw-away root (though it seems you may not want to keep that).

Your second point makes rethink my workflow the other way around: currently my root issue is I'm stuck with the SDL2 port because the official SDL2 version sadly lags behind (I used to install the official SDL2 in my build root, until it became incompatible with the current emscripten).
So if I can build an Emscripten port, with the same (ad-hoc) build-system from tools/ports/xxx.py, from a separate command line, but installing in my build root, that would suit my needs. (I want to keep the same build instructions so I can patch and contribute to the port. If the ports switch back to ./configure&&make that works too.)

[hikari-no-yume]

This seems to be causing me trouble with Rust. I want to use a Rust SDL2 wrapper crate (see https://puddleofcode.com/story/definitive-guide-to-rust-sdl2-and-emscriptem, I'm roughly copying that guide), but it causes -lSDL2 to be passed to emcc and so building always fails with = note: shared:ERROR: emcc: cannot find library "SDL2", regardless of whether I have EMMAKEN_CFLAGS="-s USE_SDL=2" set 😢

Of course, that crate has not been designed with Emscripten in mind, so the solution may just be to modify that crate (I may well do so at some point!). But it is a shame that Emscripten can't “just work” here.

[bvibber]

As a workaround, you should be able to create an empty (or mostly-empty) libSDL2.a of your own and stick it in the library path. This'll get linked in harmlessly while the -s USE_SDL=2 links in the real one... I think. :)

[hikari-no-yume]

@Brion Hey, I think that worked (— at least, it made it compile!), thanks so much for the suggestion!

[tlively]

@hikari-no-yume I'd like to play around with this and see if I can improve this experience. Can you share your project or describe how to reproduce the issue?

[hikari-no-yume]

@tlively Unfortunately I am not able to share the code right now (much as I'd like to), but I was roughly following this guide: https://puddleofcode.com/story/definitive-guide-to-rust-sdl2-and-emscriptem

Basically, use the Rust sdl2 crate and try building with Cargo for wasm32-unknown-emscripten (with EMMAKEN_CFLAGS='-s USE_SDL=2) and it should fail because that crate currently makes Cargo pass -lSDL2 to emcc. This is a problem with the crate of course (will probably eventually fix it myself if I can), but there's an opportunity here for emscripten to be helpful and just ignore the flag.

[Beuc]

It looks like #9983 fixes this issue!

[sbc100]

Yup! Thanks.

[hikari-no-yume]

Oh, cool, thank you! But why does it fix it?

[sbc100]

Because it puts the directory where the ports get installed to on the library path and include path.