{lib} [foss/2023a] Bazel v6.1.0, grpcio v1.57.0, TensorFlow v2.15.1

[yqshao]

(created using eb --new-pr)

depends on:

• update custom easyblock for Tensorflow for versions 2.14 + 2.15 easybuild-easyblocks#3303
• {tools}[gfbf/2023a] jax v0.4.25 w/ CUDA 12.1.1 #20119

[yqshao]

I attempted to get TF-2.15.1 building in EB. For imcompatibilities I was trying to go back to TF-pinned versions when it happens; not sure whether I'm doing it the right way, so I attached a bit detailed rationale/background info. :>

Bazel 6.1.0 dependence (with patch from 6.3.1)

There seems to be issue with bazel linking cc_proto_library (bazel#17091), in connection to the introduction of riegeli in approx. TF-2.14 (tf@8e84d71), sympton with the available 6.3.1 is a missing riegeli::RecordsMetadata::Clear() symbol in libtensorflow_framework.so ;

Notebly I also switched compiler to Clang following TensorFlow's new default.

Protobuf 23.4 & Protobuf-python 4.23.4

I was missing a bunch of bazel rules using systemlib protobuf, on top of that I am not sure if the issue with EB+TF-2.13 would persist (tf#61593); and protobuf-24.0 is explicitly excluded since TF-2.15 due to known bug (protobuf#13485). So I went back to 23.4.

But since there's a patch for the bug (easyconfig#19671), maybe we can do without this version?

TensorFlow 2.15.1 patches

Those give me an initial working build:

• fix-flatbuffer-license: @flatbuffers//:LICENSE.txt in tensorflow/tools/pip_package/BUILD;
• fix-pybind11-build: osx config required by pybind11 in third_party/systemlibs/pybind11.BUILD;
• add-default-shell-env: some api generators are missing from PYTHONPATH (at least when libraries need to come from lmod, I seem to manage without the previous TF default-shell-env patches);

Havn't probed everything with the exts_list versions and also havn't read and adapted all the TF<=2.13.1 patches; I'll do that if needed.

Misc notes

Some are perhaps better suited for easyblock changes, but for now I am monkey patching in the config:

• opt-einsum removed from VALID_LIBS in third_party/systemlibs/syslibs_configure.bzl as of TF-2.15 (tf@fce6afd), from my understanding I could only do patching TF_SYSTEM_LIBS and exclude opt-einsum for now?
• TF adapted "hermetic Python" as of TF-2.14 (tf@e85860e), need one more env: TF_PYTHON_VERSION;
• switched to wheel for ml_dytpes due reported to failure installing from source (slack);
• I also didn't get the GRPC_PYTHON_BUILD_SYSTEM_* flags to work;

[yqshao]

Test report by @yqshao
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
alvis1-01 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz, 2 x NVIDIA Tesla V100-SXM2-32GB, 550.54.14, Python 3.6.8
See https://gist.github.com/yqshao/e9a90314997675f1b8917e523eb1c7b8 for a full test report.

[casparvl]

Likewise, good to mention that these are deps of grpcio

[casparvl]

@Flamefire I'm really missing some expertise here... Any chance you can have a look at this PR and give your opinion?

[casparvl]

I'm not sure about using Clang here, even if that is what upstream TensorFlow is now doing. In principle the toolchain defines the compiler. I.e. using foss means we are using GCC as compiler. We do have Clang toolchains (https://docs.easybuild.io/version-specific/toolchains/) in EasyBuild, but they are not widely used and you'd miss a lot of the dependencies needed for TensorFlow...

Was the only reason to switch to Clang that it was a tested build configuration, as per https://www.tensorflow.org/install/source#tested_build_configurations ? I'd say for us that's not per se a reason to use that compiler. But maybe you had other reasons?

[yqshao]

Yes that was the only reason; thanks for the explanation, I'll work on a GCC build.

[Flamefire]

Wow! Hats up for the courage to battle the new TF version. Haven't dared to look at it yet. ;-)

I was missing a bunch of bazel rules using systemlib protobuf, on top of that I am not sure if the issue with EB+TF-2.13 would persist (tf#61593); and protobuf-24.0 is explicitly excluded since TF-2.15 due to known bug (protobuf#13485). So I went back to 23.4.

But since there's a patch for the bug (easyconfig#19671), maybe we can do without this version?

Possibly. Worth a try at least as another protobuf in the toolchain might cause headaches later on.

opt-einsum removed from VALID_LIBS in third_party/systemlibs/syslibs_configure.bzl as of TF-2.15 (tf@fce6afd), from my understanding I could only do patching TF_SYSTEM_LIBS and exclude opt-einsum for now?

Awesome -.- Not only that, they also seem to have killed more of the "System" python packages we were able to specify, but not bothered to remove them from the "allow list". Might be related to the hermetic Python thing below. Not sure what it really does.
In general changes like this need updates to the tensorflow easyblock such that it knows the allowed values for this version. The code should be pretty clear in how to do that.

TF adapted "hermetic Python" as of TF-2.14 (tf@e85860e), need one more env: TF_PYTHON_VERSION;

That sounds like more fun issues for us. Not sure if this means that they now install their own Python and python environment with packages from PyPI but sounds like it. -.-
But yeah, that looks like a change to the easyblock to set TF_PYTHON_VERSION

switched to wheel for ml_dytpes due reported to failure installing from source (slack);

The last message to this was "I'd want to install ml_dtypes from source, instead of wheel, before I'd make a PR." from Simon.
Using the whl here is not a good option as it is x86 only. Better than nothing but restricts us to x86 which would be a bummer.

* I also didn't get the `GRPC_PYTHON_BUILD_SYSTEM_*` flags to work;

Why? At least for reference it would be good to know. They also seem to support(?) a few more: https://github.com/grpc/grpc/blob/c2758e87fcc8146ea9b7ed5999f863f2f2106df6/setup.py#L142-L168

[branfosj]

switched to wheel for ml_dytpes due reported to failure installing from source (slack);

The last message to this was "I'd want to install ml_dtypes from source, instead of wheel, before I'd make a PR." from Simon. Using the whl here is not a good option as it is x86 only. Better than nothing but restricts us to x86 which would be a bummer.

See #20119 for ml_dtypes.

[yqshao]

Thanks for comments! I got the following to work on then:

• Get the current config to work with gcc;
• Plug in protobuf-24.0-GCCcore-12.3.0 protobuf-python-4.24.0-GCCcore-12.3.0;
• Get grpc to build with system libs (or at least get more information on what's wrong);
• Plug in EB/ml-dtypes;
• Bring back relevant comments along the way :-)

Also:

Not only that, they also seem to have killed more of the "System" python packages we were able to specify, but not bothered to remove them from the "allow list". Might be related to the hermetic Python thing below. Not sure what it really does.
In general changes like this need updates to the tensorflow easyblock such that it knows the allowed values for this version. The code should be pretty clear in how to do that.

• check what VALID_LIBS and TF_PYTHON_VERSION actually do now and attempt updating easyblock

[Flamefire]

* [ ]  Get grpc to build with system libs (or at least get more information on what's wrong);

Check reusing the tensorboard EC which IIRC includes grpc, although it doesn't build with system libs (yet, we should improve there)

* [ ]  Check what `VALID_LIBS` and `TF_PYTHON_VERSION` actually do now and attempt updating easyblock

The former doesn't do much/anything IIRC. It is just a list of possible "system" dependencies in TF which is read by the easyblock and compared against the result of its get_system_libs_for_version which in turn is based on a "table" with dependency to TF-version mapping.

[yqshao]

Some small progress...

• GCC works without problem, just export TF_NEED_CLANG=no;
• grpcio seems to work with system_lib after patching some hard coded lib paths, but I'm not sure what's the best way forward (after some offline discussion with @Micket I still have some assorted questions);

---

1. Should grpcio get its own easyconfig? only TF&tensorboard dependents on it but maybe there will be some benefit?
2. I didn't manage to get TF to work with sytem Abseil (missing @com_google_absl//absl/log at least in the system lib bazel rules)... so I excluded that as well; TF is now pulling its own com_google_absl (to which tensorflow_framwork.so is statically linked to if I understood right); I reckon this would not be a huge problem for absl at least (because it has versioned symbols)?
3. if (1), then does it make sense to also statically link grpcio(cygrpc.cpython-*.so) so that abseil can become a build dependency only? This way we avoid having to load abseil when building tensorflow.
4. I think the next is to look into patching system_lib rules for TF now... I guess it would be the best if one can have both from EasyBuild. Is there any hint or known good ways to generate/fix those?

[Flamefire]

1. Should grpcio get its own easyconfig? only TF&tensorboard dependents on it but maybe there will be some benefit?

Might make sense especially given how picky/difficult it is... Generally I think if we use something non-trivial in 2 ECs it is worth adding it as an extra EC

2. I didn't manage to get TF to work with sytem Abseil (missing `@com_google_absl//absl/log` at least in the system lib bazel rules)... so I excluded that as well; TF is now pulling its own `com_google_absl` (to which `tensorflow_framwork.so` is statically linked to if I understood right); I reckon this would not be a huge problem for absl at least (because it has versioned symbols)?

IIRC I also never managed to get TF to work with our Abseil. Always issues over issues... I might have some actual issues open in the TF repo about that.

3. if (1), then does it make sense to also statically link grpcio(`cygrpc.cpython-*.so`) so that abseil can become a build dependency only? This way we avoid having to load abseil when building tensorflow.

Maybe. But check the install folder of TF for any references to e.g #include <absl/ as it might use its Abseil for JIT compilation.
But if this only about Abseil and Abseil is statically linked into grpcio it would work already, doesn't it? Or is that what you meant?

4. I think the next is to look into patching system_lib rules for TF now... I guess it would be the best if one can have both from EasyBuild. Is there any hint or known good ways to generate/fix those?

I'm not sure what you exactly want to patch here. What is the intended change? I.e. what is "one can have both from EasyBuild"?
General hint: Search the TF repo for related changes & issues and try working from there or other similar changes.

[Micket]

Maybe. But check the install folder of TF for any references to e.g #include <absl/ as it might use its Abseil for JIT compilation.

The main problem is that including our Abseil easyconfig (even if just intended for the grpcio extension) as a dependency for TensorFlow easyconfig, it now conflicts with the (different?) absl that TF's bazel pulls in.

By breaking out grpcio into it's own easyconfig which only uses Abseil as a builddep, one can avoid having our Abseil visible in CPATH/LIBRARY_PATH when building TF.
Though, one can of course also try and convince TF to use external Abseil as well, as a bonus.

[yqshao]

Sorry, was distracted by holidy etc.... back now

But check the install folder of TF for any references to e.g #include <absl/ as it might use its Abseil for JIT compilation.

No hit with a naive grep, hopefully it's OK...

But if this only about Abseil and Abseil is statically linked into grpcio it would work already, doesn't it? Or is that what you meant?

Yes, that is what I was thinking

I'm not sure what you exactly want to patch here. What is the intended change? I.e. what is "one can have both from EasyBuild"? General hint: Search the TF repo for related changes & issues and try working from there or other similar changes.

I was looking at things like this...

ERROR:
/dev/shm/TensorFlow/2.15.1/foss-2023a/TensorFlow/bazel-root/8d83.../
external/pybind11_protobuf/pybind11_protobuf/BUILD:46:15:
no such target '@com_google_protobuf//:proto_api': target 'proto_api'
not declared in package '' defined by
/dev/shm/TensorFlow/2.15.1/foss-2023a/TensorFlow/bazel-root/8d83.../
external/com_google_protobuf/BUILD.bazel
(did you mean 'protobuf'? Tip: use `query "@com_google_protobuf//:*"`
to see all the targets in that package) and referenced by
'@pybind11_protobuf//pybind11_protobuf:proto_cast_util'

I reckon that this has to do with change of bazel rules updated in the third party dep., which do not get a corresponding systemlibs/xxx.BUILD update. Going to the third party repos does give some hint, e.g. pybind11@c8cc30 and protobuf@b3cbea1;

Above case one have just a rename so rather easy (edit: wrong statement, the actual cause is pybind@10fc4b, so also addition dep and not quite easy for me...), but sometimes one have a new set of shared library to include (tf@f8ac04c7), in that case my plan would be to go to the Absl build rules and try figuring out the dependencies so that I can compose a new systemlibs rule for it (maybe following its siblings in TF like system.absl.random.BUILD).

Maybe that's how it is? But it would be a bit tedius and ad-hoc. I wonder if there is some good magic to generate the "would be needed" systemlib rules from a bazel query or something...

[yqshao]

Not much progress with bazel rules, maybe I can proceed without abseil and protobuf in systemlibs? I'll still try to make it work with #20119 and work on the TF easyblock and an easyconfig for grpcio...

[casparvl]

Ugh, disk quota exceeded. I'll clean up and try again :)

[boegelbot]

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3303
SUCCESS
Build succeeded for 4 out of 4 (1 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/b4846a011780bba6614b6fd651fd8816 for a full test report.

[casparvl]

Test report by @casparvl
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3303
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/079ac0be19ed395862b5d59187bdcebe for a full test report.

[casparvl]

@boegelbot please test @ generoso
EB_ARGS="--include-easyblocks-from-pr 3303 TensorFlow-2.15.1-foss-2023a.eb"

[boegelbot]

@casparvl: Request for testing this PR well received on login1

PR test command 'EB_PR=20191 EB_ARGS="--include-easyblocks-from-pr 3303 TensorFlow-2.15.1-foss-2023a.eb" EB_CONTAINER= EB_REPO=easybuild-easyconfigs /opt/software/slurm/bin/sbatch --job-name test_PR_20191 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

• exit code: 0
• output:

Submitted batch job 13616

Test results coming soon (I hope)...

Details

- notification for comment with ID 2146216179 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

[boegelbot]

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3303
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
cns2 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/19e10acea1819e27310ab86e73679230 for a full test report.

Edit by @casparvl : this build failed because of an existing lock file. I restarted a build since I figured the first never came back - but it just seems to have taken very long (see next test report).

[boegelbot]

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3303
SUCCESS
Build succeeded for 4 out of 4 (1 easyconfigs in total)
cns1 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/ff0d9c05e45dc892732fc9d85f537467 for a full test report.

[yqshao]

Am I correct that this TensorFlow doesn't really need the jax EasyConfig? If so, I'd propose to take it out of this PR and leave contributing jax with CUDA up to this PR. They had plenty of problems there, so I think it'd be wise if we can focus (only) on TF and its direct deps in the current PR (20191). Note that #20119 also adds ml_dtypes. For that, the reverse argument applies: I'd like to keep that in this PR (20191) and proposed to take it out of 20119 (but let's see if the contributor of that PR agrees :)). If needed, we could even get that split off and merged quickly, so that both PRs can easily build on it. EDIT: it's split off in #20707 now

Thanks, I took the merged version and removed the jax files.

For the rest, I think you made some great progress on this PR. It is quite difficult to keep track of everything, but as I understand it we now rely on bazel pulling in & statically linking abseil and protobuf, since we don't immediately see a way to convince Bazel to use the 'system' libraries (i.e. EasyBuild-provided deps), for this, correct? If so, I think that's a fair approach for now: it's better to have this, than no EasyConfig at all. If we ever figure out a way to make it link to it's EasyBuild-provided deps for these two: great, but we can worry about that later.

Right, we are relying on Abseil being statically linked and its ABI being versioned, but since Abseil is still a dependency this is not optimal. To be more specific about the difficulty, I was stopped by how the Bazel requries access to build rules of the dependencies (that changes between versions) without providing updated "system" counterparts. I didn't find a good strategy to track this problem.

I'll check the changes in a bit more detail now, but I think you've tackled all of my previous requests :) If not, I'll make inline remarks :)

[yqshao]

Test report by @yqshao
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3303
FAILED
Build succeeded for 1 out of 3 (3 easyconfigs in total)
alvis2-02 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz, 8 x NVIDIA Tesla T4, 550.54.14, Python 3.6.8
See https://gist.github.com/yqshao/eab786e0cad977554c708a7d11b2e7b6 for a full test report.

[casparvl]

@boegelbot please test @ generoso
EB_ARGS="--include-easyblocks-from-pr 3303"

[casparvl]

I'm not sure why your grpcio build fails (there seems to be no gist/error log for that one...), let me test if it still works on my system with your latest changes :)

[boegelbot]

@casparvl: Request for testing this PR well received on login1

PR test command 'EB_PR=20191 EB_ARGS="--include-easyblocks-from-pr 3303" EB_CONTAINER= EB_REPO=easybuild-easyconfigs /opt/software/slurm/bin/sbatch --job-name test_PR_20191 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

• exit code: 0
• output:

Submitted batch job 13620

Test results coming soon (I hope)...

Details

- notification for comment with ID 2147136409 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

[yqshao]

Probably a silly mistake with indentation, testing now...

[casparvl]

Test report by @casparvl
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3303
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/f0742c53b4d72724acf6506f645386eb for a full test report.

[yqshao]

Test report by @yqshao
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3303
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in total)
alvis2-02 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz, 8 x NVIDIA Tesla T4, 550.54.14, Python 3.6.8
See https://gist.github.com/yqshao/1e7c978cd1dc522b531a974de65a3bab for a full test report.

[casparvl]

@boegelbot please test @ jsc-zen3
EB_ARGS="--include-easyblocks-from-pr 3303"

[boegelbot]

@casparvl: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=20191 EB_ARGS="--include-easyblocks-from-pr 3303" EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_20191 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

• exit code: 0
• output:

Submitted batch job 4274

Test results coming soon (I hope)...

Details

- notification for comment with ID 2147526608 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

[casparvl]

Test report by @casparvl
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3303
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in total)
tcn1.local.snellius.surf.nl - Linux RHEL 8.6, x86_64, AMD EPYC 7H12 64-Core Processor, Python 3.6.8
See https://gist.github.com/casparvl/b3b660381bc60092283a03c9ed3215c7 for a full test report.

[boegelbot]

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3303
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.4, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.18
See https://gist.github.com/boegelbot/f9ed1e380bfc9b23a2c1fec25f4a4519 for a full test report.

[boegelbot]

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3303
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in total)
cns1 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/32bf12c9c4349eaf81168c5ce1842a52 for a full test report.

[casparvl]

Changes look good to me, time to merge :)

[casparvl]

Going in, thanks @yqshao!