RFdiffusion-1.1.0-foss-2022a-CUDA-11.7.0.eb with models and missing dependencies #18359

ThomasHoffmann77 · 2023-07-20T11:46:13Z

(created using eb --new-pr)

requires {bio}[system/system] RFdiffusion-models v1.1.0, RFdiffusion-schedules v1.1.0 #20019

…ependencies

…n/1.5.0-foss-2022a

lexming · 2023-07-23T21:52:50Z

Test report by @lexming
FAILED
Build succeeded for 6 out of 10 (9 easyconfigs in total)
node200.hydra.os - Linux CentOS Linux 7.9.2009, x86_64, Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz (broadwell), Python 3.6.8
See https://gist.github.com/lexming/92ef10c7ef1ab4a78e49ae586dd4cada for a full test report.

ThomasHoffmann77 · 2023-07-24T09:11:42Z

Test report by @lexming FAILED Build succeeded for 6 out of 10 (9 easyconfigs in total) node200.hydra.os - Linux CentOS Linux 7.9.2009, x86_64, Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz (broadwell), Python 3.6.8 See https://gist.github.com/lexming/92ef10c7ef1ab4a78e49ae586dd4cada for a full test report.
@lexming I updated the checksums for the DGL easyconfigs.

easybuild/easyconfigs/r/RFdiffusion/RFdiffusion-1.1.0-foss-2022a.eb

se3-transformer/model/basis.py imports torch.cuda.nvtx

model checksum exit

comment models urls and checksums

smoors · 2023-11-17T13:21:10Z

easybuild/easyconfigs/d/DGL/DGL-1.1.1-foss-2022a-CUDA-11.7.0.eb

+    {'dgl-1.1.1.tar.gz': '076026a7818f2396056252269d7960c561840bb0fe89494e11f8d6c524891c49'},
+    {'metis-5.1.1-DistDGL-v0.5.tar.gz': 'cedf0b32d32a8496bac7eb078b2b8260fb00ddb8d50c27e4082968a01bc33331'},
+    {'GKlib-METIS-v5.1.1-DistDGL-0.5.tar.gz': '52aa0d383d42360f4faa0ae9537ba2ca348eeab4db5f2dfd6343192d0ff4b833'},
+    {'tensorpipe-20230206.tar.gz': '97dcc6ab648d38f62ddc9d2a33de2a789c967e43e819481f817341881912a524'},


checksums for git checkouts are not stable, so should be removed

@smoors I was not aware of the unstable checksums. Is this the case for any commit or only for recursive checkouts?

it's the case whenever you are not downloading a tarball directly, but creating the tarball yourself, so any git clone

remove unstable checksums for tensorpipe, pcg, and libxsmm

update cub comment

easybuild/easyconfigs/r/RFdiffusion/RFdiffusion-1.1.0-foss-2022a-CUDA-11.7.0.eb

smoors · 2023-11-20T15:54:20Z

easybuild/easyconfigs/r/RFdiffusion/RFdiffusion-1.1.0-foss-2022a-CUDA-11.7.0.eb

+# uncomment for installing models and schedules as components.
+# Then comment out _get_schedules and _get_models in postinstallcmds


what's the difference between the components and _get_schedules/_get_models?
i would prefer to use the best method and remove the other one to keep things clean

@smoors Using components would archive the models and would allow to reinstall RFdiffusion even if the models were not available for download any more. I'd prefer this option since RFdiffusion is not usable without the models.
However, I understand @lexming's concerns about to store the models data (3.9GB) at least twice, in the module directory and the sources directory.
Therefore I kept the components approach commented out.

Which solution do you prefer and which should I delete?

hm, i agree with you that having to re-download on reinstall is suboptimal, but i also agree with @lexming that storing the models twice is suboptimal.

a possible solution is to create a custom easyblock that checks the installdir and does not delete/re-download the models if they are already present and if their checksums are correct.
to make this configurable, we could define an extra parameter in the easyblock, e.g. keep_models_if_installed.
this is similar to easyblock Tarball, which allows merging an installation into an existing installation instead of wiping it first.

what do you think?

For long term it is probably better to place the data into a separate module RFdiffusion_data, which can be reused for minor updates of RFdiffusion, if the data does not change.
It might be good to have a generic easyblock for such data modules, because we can expect more of them with all the upcoming AI packages. (see also #19157 Model-Angelo_data, #19240 relion-classranker_data + relion-blush_data).
There was also a brief discussion at https://github.com/easybuilders/easybuild/wiki/Conference-call-notes-20230719

that's indeed a better solution long-term.
i'll try to give a stab at a generic data easyblock, but feel free to do this yourself if you're up for it.

in the meantime, i suggest to already go ahead with creating a separate easyconfig for RFdiffusion_data. you should be able to use the Bundle easyblock (which does not install anything), and download the models in a postinstallcmds, as you have done here.

I'll have a look on RFdiffusion_data in the next few days. RFdiffusion-1.1.0_data_paths.patch requires an update allowing to read the models and schedules from elsewhere defined in a environment variable.

pavelToman · 2024-02-15T13:31:05Z

Hi! I juts start to work on vscentrum/vsc-software-stack#283 pipeline of RFdiffusion - ProteinMPNN -AlphaFold. I'd like to use RFd with data models either. Do you working on that or you stop?

ThomasHoffmann77 · 2024-02-15T14:06:10Z

Hi! I juts start to work on vscentrum/vsc-software-stack#283 pipeline of RFdiffusion - ProteinMPNN -AlphaFold. I'd like to use RFd with data models either. Do you working on that or you stop?

@pavelToman I am currently working on AlphaFold/2.3.2-foss-2023a-CUDA-12.1.1. see also #19841

….0.eb

…ing_test.patch

…usion110

pavelToman · 2024-06-05T10:13:35Z

{lib}[GCC/11.3.0,GCCcore/11.3.0] DGL v1.1.1, DLPack v0.8, DMLC-Core v0.5, ... w/ Python 3.10.4 #19095

WilleBell · 2024-09-04T12:39:15Z

easybuild/easyconfigs/r/RFdiffusion/RFdiffusion-1.1.0-foss-2022a-CUDA-11.7.0.eb

+
+sanity_check_commands = [
+    "run_inference.py --help",
+    "run_inference.py 'contigmap.contigs=[150-150]' " +


This command will create two output directories in %(installdir)s, is there a way to avoid this?

WilleBell · 2024-09-04T12:53:01Z

easybuild/easyconfigs/r/RFdiffusion/RFdiffusion-1.1.0-foss-2022a-CUDA-11.7.0.eb

        ],
    }),
 ]

-sanity_pip_check = True
+postinstallcmds = [
+    'cp -rpP config examples %s/%%(namelower)s' % _pysp,


Suggested change

'cp -rpP config examples %s/%%(namelower)s' % _pysp,

'cp -rpP config examples helper_scripts %s/%%(namelower)s' % _pysp,

'cp -pP %%(namelower)s/inference/sym_rots.npz %s/%%(namelower)s/inference/' % _pysp,

RFdiffusion needs sym_rots.npz to perform symmetry operations which is needed to use different symmetry settings.

added in #23110

garadar · 2024-09-19T15:23:59Z

From https://github.com/easybuilders/easybuild-easyconfigs/pull/18359/files#diff-03d4d74a13067c23b1773097b070efb41c27ac715ad10f07b0fd98775ba1a144

    {
        'source_urls': ['https://github.com/KarypisLab/GKlib/archive'],
        'download_filename': 'METIS-v6.1.1-DistDGL-0.5.tar.gz',
        'filename': 'GKlib-METIS-v6.1.1-DistDGL-0.5.tar.gz',
        'extract_cmd': "tar -C %(namelower)s-%(version)s/third_party/METIS/GKlib --strip-components=2 -xf %s",
    },

It seems the source v6.1.1-DistDGL-0.5 is not available : https://github.com/KarypisLab/GKlib/tags

sassy-crick · 2025-06-16T10:01:01Z

@ThomasHoffmann77 You might want to sync your PR with development, something like this:

eb --sync-pr-with-develop NUMBER_OF_PR

Also, there are some changes regarding GitHub and stable checksums in EasyBuild-5. Finally, RFdiffusion-1.1.0-foss-2022a-CUDA-11.7.0.eb is merged in EasyBuild-5.1.0 so do we still need this PR?

jfgrimm · 2025-06-16T13:17:30Z

I'll close this, since we have already merged RFdiffusion easyconfigs for this version and toolchain
There are some differences -- if you feel that changes to the existing easyconfigs are appropriate, please open a new PR with those changes.

ThomasHoffmann77 · 2025-06-16T13:38:20Z

I'll close this, since we have already merged RFdiffusion easyconfigs for this version and toolchain There are some differences -- if you feel that changes to the existing easyconfigs are appropriate, please open a new PR with those changes.

I thinks the one already merged still does not take care of the data downloads. That's why we have RFdiffusion-models and -schedules.
See also https://github.com/easybuilders/easybuild/wiki/Sync%E2%80%90meeting%E2%80%90EMBL%E2%80%9020240522#rfdiffusion

It's probably still also missing some required dependencies, e.g., as DGL and Hydra.

jfgrimm · 2025-06-16T13:47:56Z

@ThomasHoffmann77 the data stuff can probably use the Dataset easyblock, which is now available (easybuilders/easybuild-easyblocks#3246)
I'd suggest still tackling this in a new PR, and maybe targetting a newer toolchain

ThomasHoffmann77 · 2025-06-16T14:39:11Z

@ThomasHoffmann77 the data stuff can probably use the Dataset easyblock, which is now available (easybuilders/easybuild-easyblocks#3246) I'd suggest still tackling this in a new PR, and maybe targetting a newer toolchain

this was already implemented with commit b7ec8ea

I'll open a new pr with foss/2022a again and then one with an update to 2023a

ThomasHoffmann77 · 2025-06-16T15:39:23Z

reopened as #23110

ThomasHoffmann77 and others added 6 commits July 20, 2023 13:46

RFdiffusion-1.1.0-foss-2022a-CUDA-11.7.0.eb with models and missing d…

1082a62

…ependencies

checksums

8b47f1b

checksums moved

bcf7c89

remove download_dep_fail

1851f8f

add missing DLPack/0.8-foss-2022a, DMLC-Core/0.5-foss-2022a, nanoflan…

27b635a

…n/1.5.0-foss-2022a

Remove end of file blank lines for DLPack

d4f32d7

fix checksums

d8b60fd

lexming requested changes Aug 1, 2023

View reviewed changes

easybuild/easyconfigs/r/RFdiffusion/RFdiffusion-1.1.0-foss-2022a.eb Outdated Show resolved Hide resolved

ThomasHoffmann77 and others added 6 commits August 9, 2023 10:07

postinstall data

7348a5e

typos

0c7ba6a

typos

a45f57e

Delete RFdiffusion-1.1.0-foss-2022a.eb

2f142e8

se3-transformer/model/basis.py imports torch.cuda.nvtx

Update RFdiffusion-1.1.0-foss-2022a-CUDA-11.7.0.eb

97b130a

model checksum exit

Update RFdiffusion-1.1.0-foss-2022a-CUDA-11.7.0.eb

28c433f

comment models urls and checksums

ThomasHoffmann77 requested a review from lexming November 13, 2023 17:40

ThomasHoffmann77 mentioned this pull request Nov 17, 2023

{bio}[foss/2022a] RFdiffusion v1.1.0, e3nn v0.3.3 w/ CUDA 11.7.0 #19183

Merged

smoors reviewed Nov 17, 2023

View reviewed changes

ThomasHoffmann77 added 4 commits November 17, 2023 15:48

Update DGL-1.1.1-foss-2022a-CUDA-11.7.0.eb

8248e55

remove unstable checksums for tensorpipe, pcg, and libxsmm

Update DGL-1.1.1-foss-2022a.eb

f3458b0

remove unstable checksums for tensorpipe, pcg, and libxsmm

Update DGL-1.1.1-foss-2022a-CUDA-11.7.0.eb

a0c1945

update cub comment

Merge branch 'develop' into 20230720134603_new_pr_RFdiffusion110

38cd29f

ThomasHoffmann77 requested a review from smoors November 20, 2023 12:41

smoors reviewed Nov 20, 2023

View reviewed changes

Update RFdiffusion-1.1.0-foss-2022a-CUDA-11.7.0.eb

7a2f9f7

easybuilders deleted a comment from boegelbot Nov 20, 2023

ThomasHoffmann77 added 3 commits February 15, 2024 15:14

Delete easybuild/easyconfigs/n/nanoflann/nanoflann-1.5.0-GCCcore-11.3…

08fa70b

….0.eb

Delete easybuild/easyconfigs/n/nanoflann/nanoflann-1.5.0_disable_fail…

976eead

…ing_test.patch

Merge branch 'easybuilders:develop' into 20230720134603_new_pr_RFdiff…

029bf36

…usion110

pavelToman mentioned this pull request Feb 15, 2024

RFdiffusion/ProteinMPNN/AlphaFold pipeline + ColabDesign vscentrum/vsc-software-stack#283

Closed

smoors mentioned this pull request Mar 3, 2024

{bio}[system/system] RFdiffusion-models v1.1.0, RFdiffusion-schedules v1.1.0 #20019

Merged

ThomasHoffmann77 and others added 3 commits March 11, 2024 17:53

use data from PR easybuilders#20019

b7ec8ea

fix style

386881a

Merge branch 'easybuilders:develop' into 20230720134603_new_pr_RFdiff…

2953c1e

…usion110

boegel mentioned this pull request May 22, 2024

{ai}[foss/2022a] DGL v1.1.3 w/ CUDA 11.7.0 #20092

Merged

3 tasks

WilleBell mentioned this pull request Sep 2, 2024

{lib}[GCC/11.3.0,GCCcore/11.3.0] DGL v1.1.1, DLPack v0.8, DMLC-Core v0.5, ... w/ Python 3.10.4 #19095

Closed

WilleBell reviewed Sep 4, 2024

View reviewed changes

sassy-crick added the 2022a label Jun 16, 2025

jfgrimm closed this Jun 16, 2025

ThomasHoffmann77 added a commit to ThomasHoffmann77/easybuild-easyconfigs that referenced this pull request Jun 16, 2025

reopen easybuilders#18359

4551067

boegel mentioned this pull request Jun 16, 2025

{bio}[foss/2022a] RFdiffusion v1.1.0 w/ CUDA 11.7.0 #23110

Open

1 task

		# uncomment for installing models and schedules as components.
		# Then comment out _get_schedules and _get_models in postinstallcmds

	'cp -rpP config examples %s/%%(namelower)s' % _pysp,
	'cp -rpP config examples helper_scripts %s/%%(namelower)s' % _pysp,
	'cp -pP %%(namelower)s/inference/sym_rots.npz %s/%%(namelower)s/inference/' % _pysp,

RFdiffusion-1.1.0-foss-2022a-CUDA-11.7.0.eb with models and missing dependencies #18359

RFdiffusion-1.1.0-foss-2022a-CUDA-11.7.0.eb with models and missing dependencies #18359

Uh oh!

Conversation

ThomasHoffmann77 commented Jul 20, 2023 • edited by lexming Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lexming commented Jul 23, 2023

Uh oh!

ThomasHoffmann77 commented Jul 24, 2023

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smoors Nov 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pavelToman commented Feb 15, 2024

Uh oh!

ThomasHoffmann77 commented Feb 15, 2024

Uh oh!

pavelToman commented Jun 5, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WilleBell Sep 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

garadar commented Sep 19, 2024

Uh oh!

sassy-crick commented Jun 16, 2025

Uh oh!

jfgrimm commented Jun 16, 2025

Uh oh!

ThomasHoffmann77 commented Jun 16, 2025

Uh oh!

jfgrimm commented Jun 16, 2025

Uh oh!

ThomasHoffmann77 commented Jun 16, 2025

Uh oh!

ThomasHoffmann77 commented Jun 16, 2025

Uh oh!

Uh oh!

ThomasHoffmann77 commented Jul 20, 2023 •

edited by lexming

Loading

smoors Nov 21, 2023 •

edited

Loading

WilleBell Sep 4, 2024 •

edited

Loading