enhance CUDA easyblock to create version independent pkgconfig files #2656

damianam · 2022-01-18T08:17:26Z

as some distros do and as expected by some packages (eg: xpra)

easybuild/easyblocks/c/cuda.py

SebastianAchilles · 2022-01-18T11:49:10Z

Overview of tested easyconfigs (in order)

SUCCESS CUDA-11.3.1.eb
SUCCESS CUDA-11.4.1.eb
SUCCESS CUDA-11.4.2.eb
SUCCESS CUDA-11.5.0.eb

Build succeeded for 4 out of 4 (4 easyconfigs in total)
jsfc01.int.jusuf.sebastian.cluster - Linux rocky linux 8.4, x86_64, AMD EPYC 7742 64-Core Processor (zen2), 1 x NVIDIA GRID V100-4C, 460.73.01, Python 3.6.8
See https://gist.github.com/673610e120b78349fde1f55d0c7f2190 for a full test report.

easybuild/easyblocks/c/cuda.py

boegel · 2022-01-19T07:54:55Z

easybuild/easyblocks/c/cuda.py

+        # See e.g. https://github.com/Xpra-org/xpra/blob/master/setup.py#L206
+        # Distros provide these files, so let's do it here too
+        pkgconfig_dir = os.path.join(self.installdir, 'pkgconfig')
+        pc_files = expand_glob_paths([os.path.join(pkgconfig_dir, '*.pc')])


This doesn't always work apparently, fails with CUDA 10.1.105 because there are no pkgconfig files included at all:

No files found using glob pattern '/user/gent/400/vsc40023/eb_arcaninescratch/CO7/skylake-ib/software/CUDA/10.1.105-GCC-8.2.0-2.31.1/pkgconfig/*.pc`

So this should be guarded with:

if os.path.exists(pkgconfig_dir):

@damianam Does that make sense to you?

None of the 10.x versions we have installed have a pkgconfig dir. So it's clearly needed

Also the module generation needs to have this protected. Don't add nonexisting paths to PKG_CONFIG_PATH

Makes sense. Update coming

I pushed an update to address that. Note that I readded some of the pc files in the sanity check list. That's because those files are explicitly checked by other tools (and actually nvjpeg.pc is the whole reason for this PR), so I guess it makes sense to add them. Or are they not present in older CUDA releases?

SebastianAchilles · 2022-01-19T08:32:19Z

Test report by @SebastianAchilles

Overview of tested easyconfigs (in order)

FAIL (build issue) CUDA-10.1.105.eb (partial log available at https://gist.github.com/4e6bae26d6021bb91b3d99e894c6be08)
FAIL (build issue) CUDA-10.1.168.eb (partial log available at https://gist.github.com/c1f9808372c4971e0aa5e13bb33168c4)
FAIL (build issue) CUDA-10.1.243.eb (partial log available at https://gist.github.com/2cb5916da879d8b84c6104b6ee9512a3)
SUCCESS CUDA-11.3.1.eb
SUCCESS CUDA-11.4.1.eb
FAIL (build issue) CUDA-11.4.2.eb (partial log available at https://gist.github.com/346f6edb6658441e8e0671327e2ba10e)
SUCCESS CUDA-11.5.0.eb
FAIL (build issue) CUDA-5.5.22.eb (partial log available at https://gist.github.com/29024be2d236e3f4997dbd1ddea280bf)
FAIL (build issue) CUDA-6.0.37.eb (partial log available at https://gist.github.com/2d7af8643de199a7e6ca4553a348d440)
FAIL (build issue) CUDA-6.5.14.eb (partial log available at https://gist.github.com/1bf6dd01b47274b6d3f5a9b8daec803c)
FAIL (build issue) CUDA-7.0.28.eb (partial log available at https://gist.github.com/5ad8248c63dea68b659af1f78175521a)
FAIL (build issue) CUDA-7.5.18.eb (partial log available at https://gist.github.com/4ba727799b6f333c6f6c4b720c704933)
FAIL (build issue) CUDA-8.0.44.eb (partial log available at https://gist.github.com/234e37c032e8e4825f49655a5a0db8ef)
FAIL (build issue) CUDA-8.0.61.eb (partial log available at https://gist.github.com/f276f6b0203321d989348fcc648f55f2)
FAIL (build issue) CUDA-9.0.176.eb (partial log available at https://gist.github.com/3d4f2a036412ca16930fd3a3eb889dea)
FAIL (build issue) CUDA-9.1.85.eb (partial log available at https://gist.github.com/3e9ec1aa36892204254b632deb772985)
FAIL (build issue) CUDA-9.2.148.1.eb (partial log available at https://gist.github.com/09ef346ae845e894374532ed70b47621)
FAIL (build issue) CUDA-9.2.88.eb (partial log available at https://gist.github.com/94949e73716abc8152f607f73c6e651f)

Build succeeded for 3 out of 18 (18 easyconfigs in total)
jsfc01.int.jusuf.sebastian.cluster - Linux rocky linux 8.4, x86_64, AMD EPYC 7742 64-Core Processor (zen2), 1 x NVIDIA GRID V100-4C, 460.73.01, Python 3.6.8
See https://gist.github.com/c865046effdab2ea5f7f719fd7bd057f for a full test report.

boegel · 2022-01-19T09:03:25Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS CUDA-7.0.28.eb
SUCCESS CUDA-8.0.61.eb
SUCCESS CUDA-9.0.176.eb
SUCCESS CUDA-9.2.88-GCC-7.3.0-2.30.eb
FAIL (build issue) CUDA-10.1.105-GCC-8.2.0-2.31.1.eb (partial log available at https://gist.github.com/9e854eadbe91330e388a67c874c9a948)
FAIL (build issue) CUDA-10.1.243.eb (partial log available at https://gist.github.com/001fe9888f9230bbd9a311459186e437)
SUCCESS CUDA-11.5.0.eb

Build succeeded for 5 out of 7 (7 easyconfigs in total)
node3104.skitty.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz (skylake_avx512), Python 3.6.8
See https://gist.github.com/9d56c24b62359722f741c2df111fa13a for a full test report.

SebastianAchilles · 2022-01-19T11:13:20Z

Test report by @SebastianAchilles

Overview of tested easyconfigs (in order)

FAIL (build issue) CUDA-10.1.105.eb (partial log available at https://gist.github.com/62914ffa0caca0afc8cf73b301cacfd6)
FAIL (build issue) CUDA-10.1.168.eb (partial log available at https://gist.github.com/e402e0b6bc85ac9b328c3cec37d99611)
FAIL (build issue) CUDA-10.1.243.eb (partial log available at https://gist.github.com/7c362d8069326ddfbc6787669e09fc79)
SUCCESS CUDA-11.3.1.eb
SUCCESS CUDA-11.4.1.eb
SUCCESS CUDA-11.4.2.eb
SUCCESS CUDA-11.5.0.eb
FAIL (build issue) CUDA-5.5.22.eb (partial log available at https://gist.github.com/8040ef4a4b13cacc7dcbd1bd43ff2d17)
FAIL (build issue) CUDA-6.0.37.eb (partial log available at https://gist.github.com/1147b92122f85dd330411e0d0eaf5f48)
FAIL (build issue) CUDA-6.5.14.eb (partial log available at https://gist.github.com/36669cbf92c0b3fa856d9866890d382a)
FAIL (build issue) CUDA-7.0.28.eb (partial log available at https://gist.github.com/f1f693f4e69fa9a9a9edac81f08b2646)
FAIL (build issue) CUDA-7.5.18.eb (partial log available at https://gist.github.com/959616af1fa2734247c99e6a1396a74f)
FAIL (build issue) CUDA-8.0.44.eb (partial log available at https://gist.github.com/73370123d3de0e07587b06d24c164c91)
FAIL (build issue) CUDA-8.0.61.eb (partial log available at https://gist.github.com/069463847b066fda2a4b2711fd0b1c73)
FAIL (build issue) CUDA-9.0.176.eb (partial log available at https://gist.github.com/273f66562d58598b344171495c292830)
FAIL (build issue) CUDA-9.1.85.eb (partial log available at https://gist.github.com/ebe6e7390991e2710c09128c4d6ff834)
FAIL (build issue) CUDA-9.2.148.1.eb (partial log available at https://gist.github.com/838be1ea5e89960c5fe94338604b47d8)
FAIL (build issue) CUDA-9.2.88.eb (partial log available at https://gist.github.com/6d643e27500f4d32f8ace048362cc9cb)

Build succeeded for 4 out of 18 (18 easyconfigs in total)
cnx2 - Linux rocky linux 8.4, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/c24d7374c144bf3a37f1b99ad035b050 for a full test report.

…d-easyblocks into cuda_pkgconfig

easybuild/easyblocks/c/cuda.py

…o dict returns by make_module_req_guess in CUDA easyblock

…ld CUDA versions

requested changes made

boegel · 2022-01-19T18:41:10Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS CUDA-7.0.28.eb
SUCCESS CUDA-8.0.61.eb
SUCCESS CUDA-9.0.176.eb
SUCCESS CUDA-9.2.88-GCC-7.3.0-2.30.eb
SUCCESS CUDA-10.1.105-GCC-8.2.0-2.31.1.eb
SUCCESS CUDA-10.1.243.eb
SUCCESS CUDA-11.5.0.eb

Build succeeded for 7 out of 7 (7 easyconfigs in total)
node3104.skitty.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz (skylake_avx512), Python 3.6.8
See https://gist.github.com/ce56c51dc9534e593b188a4c482b859d for a full test report.

boegel · 2022-01-19T18:41:14Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS CUDAcore-11.2.1.eb
SUCCESS CUDA-11.1.1-GCC-10.2.0.eb
SUCCESS CUDA-11.4.1.eb

Build succeeded for 3 out of 3 (3 easyconfigs in total)
node3589.doduo.os - Linux RHEL 8.2, x86_64, AMD EPYC 7552 48-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/2be3434eb292214c9e14e2275c5874f1 for a full test report.

damianam added 2 commits January 18, 2022 09:16

To make the CUDA easyblock create version independent pkgconfig files,

6250169

as some distros do and as expected by some packages (eg: xpra)

To appease the hound

50793b8

akesandgren previously requested changes Jan 18, 2022

View reviewed changes

easybuild/easyblocks/c/cuda.py Outdated Show resolved Hide resolved

SebastianAchilles added the enhancement label Jan 18, 2022

SebastianAchilles added this to the 4.5.2 milestone Jan 18, 2022

SebastianAchilles reviewed Jan 18, 2022

View reviewed changes

easybuild/easyblocks/c/cuda.py Outdated Show resolved Hide resolved

cuda.py use append in files sanity check

bcd34ce

boegel requested changes Jan 18, 2022

View reviewed changes

easybuild/easyblocks/c/cuda.py Outdated Show resolved Hide resolved

easybuild/easyblocks/c/cuda.py Outdated Show resolved Hide resolved

fix check for pkgconfig files in CUDA sanity check

d81d069

boegel requested changes Jan 19, 2022

View reviewed changes

boegel changed the title ~~To make the CUDA easyblock create version independent pkgconfig files,~~ enhance CUDA easyblock to create version independent pkgconfig files Jan 19, 2022

damianam added 2 commits January 19, 2022 14:37

To add safeguards for old CUDA versions that do not have pkgconfig files

84f9e30

Merge branch 'cuda_pkgconfig' of https://github.com/damianam/easybuil…

4ba76e8

…d-easyblocks into cuda_pkgconfig

boegel requested changes Jan 19, 2022

View reviewed changes

easybuild/easyblocks/c/cuda.py Outdated Show resolved Hide resolved

easybuild/easyblocks/c/cuda.py Outdated Show resolved Hide resolved

boegel added 2 commits January 19, 2022 19:26

no need to check for existence of pkgconfig subdir before adding it t…

0202a4f

…o dict returns by make_module_req_guess in CUDA easyblock

don't check for nvidia-ml.pc and nvjpeg.pc, which are not there for o…

c8f31cb

…ld CUDA versions

boegel approved these changes Jan 19, 2022

View reviewed changes

boegel merged commit 2984a22 into easybuilders:develop Jan 19, 2022

enhance CUDA easyblock to create version independent pkgconfig files #2656

enhance CUDA easyblock to create version independent pkgconfig files #2656

Uh oh!

Conversation

damianam commented Jan 18, 2022

Uh oh!

Uh oh!

SebastianAchilles commented Jan 18, 2022

Overview of tested easyconfigs (in order)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

boegel Jan 19, 2022

Choose a reason for hiding this comment

Uh oh!

akesandgren Jan 19, 2022

Choose a reason for hiding this comment

Uh oh!

akesandgren Jan 19, 2022

Choose a reason for hiding this comment

Uh oh!

damianam Jan 19, 2022

Choose a reason for hiding this comment

Uh oh!

damianam Jan 19, 2022

Choose a reason for hiding this comment

Uh oh!

SebastianAchilles commented Jan 19, 2022

Overview of tested easyconfigs (in order)

Uh oh!

boegel commented Jan 19, 2022

Overview of tested easyconfigs (in order)

Uh oh!

SebastianAchilles commented Jan 19, 2022

Overview of tested easyconfigs (in order)

Uh oh!

Uh oh!

Uh oh!

boegel commented Jan 19, 2022

Overview of tested easyconfigs (in order)

Uh oh!

boegel commented Jan 19, 2022

Overview of tested easyconfigs (in order)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants