Discussion: Polly in Gentoo LLVM Stack

[elsandosgrande]

Pardon using an issue ticket like a discussion post, but, as Discussions doesn't seem to be enabled for this repository…

As of yesterday, specifically gentoo/gentoo@75d67b1, AMD64 23.0 profiles have been stabilised. Do you have any plans to stabilise your 23.0 Plasma LLVM systemd profile, or does it not matter in practice?

Also, is there anything that I should be aware of coming from the LLVM systemd merged-usr profile present in the main Gentoo repository? Since I'm on a time crunch and your profile seems to just pull in the relevant building blocks from the main Gentoo repository, I'll switch to your profile right away, but I get the feeling that it would still be good to double-check with you, the author, especially since I'm not well-versed in exactly how Gentoo profiles are structured — I've only figured out enough to copy the Plasma desktop bits into /etc/portage/profile last summer.

[xarblu]

or does it not matter in practice

As far as I know the labels dev/exp/stable (e.g. shown by eselect profile) are just that - labels and don't have any deeper meaning. I just have them set to exp because the 17.1/clang profiles were labelled as that.
I could mark them stable now that 23.0/llvm is marked stable but that shouldn't change anything functionality wise. I'll think about it but I do like to keep things marked "experimental" as a sort of "get-out-of-jail-free-card" just in case they ever break :P

your profile seems to just pull in the relevant building blocks from the main Gentoo repository

Yup, that's pretty much all this profile does. The only difference is I have a "slimmer" package.use.force compared too ::gentoo 23.0/llvm profiles.
This one only sets the bare minimum of abi_x86_32 flags for LLVM runtimes so clang can build 32bit things (which is just sys-devel/clang-runtime[abi_x86_32] + deps). These are force enabled because changing them later if your clang can't build 32bit is painful.
The ::gentoo version of that file has a couple extra packages that aren't needed for this (notably sys-devel/clang and sys-devel/llvm which can build 32bit just fine without abi_x86_32 but enabling that flag basically doubles their compile times because you build them twice).

[elsandosgrande]

All right then, I should be good to go with it as-is.

Thanks for the info! Also, many thanks for continuing to package Polly for Gentoo!!!

Edit

[…] enabling that flag basically doubles their compile times because you build them twice).

That explains things.

[xarblu]

many thanks for continuing to package Polly for Gentoo!!!

Oh another user of my hacky Polly solution :P

But jokes aside it's nice to see see people using it.
And just as a heads-up if you use sys-devel/llvm[polly] - I'll probably change the way Polly is loaded soon-ish.
The current method (patching libLLVM.so), requires a needless rebuild of LLVM, causes circular dependencies and is just a big hack all around (loading it for everything using LLVM even though just clang and opt can use it).

I'm currently testing loading polly via -fplugin=LLVMPolly.so in /etc/clang (provided by sys-devel/clang-common) which mostly works. The only issue is that the package isn't slotted (meaning it applies to all clang versions, causing errors when no matching polly version is installed).

I feel like the cleanest, "most intended" solution would be to just have users add -fplugin=LLVMPolly.so to their C(XX)FLAGS that way they're responsible for having matching clang and polly versions installed.

What do you as a user think?

[elsandosgrande]

@xarblu I can't speak for others or for what is the best/ideal solution, but I personally prefer the current method compared to having to handle that in the compiler flags, since the double emerge, however annoying, only occurs once when emerging a new version of the toolchain, so it's a personally acceptable compromise.

That said, the following bug ticket might be of more worth as reference on outside opinion and approach to this problem: https://bugs.gentoo.org/715612. Interestingly enough, it also ends up having to use compiler flags to pull Polly in, but using an arguably less elegant flag.

[xarblu]

Yea the patched LLVM will probably stay for now (even if it's not the prettiest solution). Any other way I tried causes seemingly random issues (e.g. with -fplugin=LLVMPolly.so clang breaks a step when building LLVM itself).

The -Xclang -load -Xclang libPolly.so flags from https://bugs.gentoo.org/715612 afaik is just "the old way" to load plugins. Not sure if they are different in any meaningful way compared to -fplugin=LLVMPolly.so.

[elsandosgrande]

[…] (loading it for everything using LLVM even though just clang and opt can use it).

For the record, rustc can use it too via -C llvm-args=--polly. I'm not sure if that's an extraneous hyphen though 😅

[elsandosgrande]

@xarblu Looks like your work has reached at least one more person: gentoo/gentoo#36158

[…]

Took this idea from xarblu-overlay but having a separate polly package causes llvm-polly-llvm rebuild cycles which is not desireable. This PR makes llvm ebuild handle everything.

[…]

[StormBytePP]

@xarblu Looks like your work has reached at least one more person: gentoo/gentoo#36158

[…]
Took this idea from xarblu-overlay but having a separate polly package causes llvm-polly-llvm rebuild cycles which is not desireable. This PR makes llvm ebuild handle everything.
[…]

And let's hope they finally accept this and we all have polly!

[elsandosgrande]

@StormBytePP Since I don't yet know the etiquette for commenting on Gentoo pull requests, I imagine that this is a safer place to talk.

I have a question about the new non-automatically-linked approach: How will this work with rustc? As can be seen a few comments above, I've been able to use Polly with rustc (or at least rustc hasn't complained about it in any way that I could ascertain), but that has been without any -Xclang […] flags being needed.

[StormBytePP]

Current aproach modifies clang-common to include -Xclang... into /etc/clang/gentoo-runtimes.cfg so every call to clang will contain this command so it should work with rust and with everything.

Note that this is my third try to get this accepted:

• First I tried to build polly with llvm and link static -> got rejected because lots of archive files were needed to install
• Second I took @xarblu patchelf method but compiling it along with llvm and got rejected also because of that force linking
• Current aproach, builds polly separatelly and gets the needed commands being present transparently, they will not be visible in command line but they will be used internally for clang.

[elsandosgrande]

@StormBytePP Oh, so they outright don't want Polly to just be unconditionally linked. Well, that slightly sucks.

I haven't looked into it much before because it was never a concern before, but I'm worried about there possibly being no flags for rustc specifically to pull in the Polly library. I'm not talking about calls to Clang during the building of a primarily Rust project with some C/C++ in it, or when linking — Clang is sort of treated like a linker for some reason — I'm talking about when invoking Polly straight from rustc with no Clang in sight.

[StormBytePP]

I don't know much about rust, sorry so I can't help.

But thanks to this thread I have just force-pushed a change to load it with -fplugin=LLVMPolly.so as @xarblu said above.

[StormBytePP]

many thanks for continuing to package Polly for Gentoo!!!

Oh another user of my hacky Polly solution :P

But jokes aside it's nice to see see people using it. And just as a heads-up if you use sys-devel/llvm[polly] - I'll probably change the way Polly is loaded soon-ish. The current method (patching libLLVM.so), requires a needless rebuild of LLVM, causes circular dependencies and is just a big hack all around (loading it for everything using LLVM even though just clang and opt can use it).

I'm currently testing loading polly via -fplugin=LLVMPolly.so in /etc/clang (provided by sys-devel/clang-common) which mostly works. The only issue is that the package isn't slotted (meaning it applies to all clang versions, causing errors when no matching polly version is installed).

I feel like the cleanest, "most intended" solution would be to just have users add -fplugin=LLVMPolly.so to their C(XX)FLAGS that way they're responsible for having matching clang and polly versions installed.

What do you as a user think?

Me too, but I've been silent a lot of time until I decided to improve it and try to get accepted by gentoo ;)

With my PR this polly/llvm version mismatches are gone as it is a direct dependency of clang-common

See more info at gentoo/gentoo#36158

[elsandosgrande]

Oh, looks like that's supported: rust-lang/rust#86267

It requires a nightly build of Rust (unless something has changed in the meantime and I haven't been able to find that change in my short time looking into this), but, since I already build the Rust package as nightly so as to have DWARF 5 debugging information as opposed to DWARF 4, looks like I'm all set.