implement exponential backoff in download_file + allow max. 6 attempts to download PR diff in fetch_files_from_pr and fetch_files_from_commit

[boegel]

workaround/fix for #4869

[boegel]

I tested this with an "extreme" case, and it seems to hold up well:

$ eb --debug --from-pr 22698,22852,22827 --disable-cleanup-tmpdir -M --include-easyblocks-from-pr 3707
== Temporary log file in case of crash /tmp/eb-o9bahsfx/easybuild-zumur0uf.log
== Waiting for 20 seconds before trying download of https://github.com/easybuilders/easybuild-easyblocks/pull/3707.diff again...
== easyblock mathematica.py included from PR #3707
== Waiting for 20 seconds before trying download of https://github.com/easybuilders/easybuild-easyconfigs/pull/22698.diff again...
== Waiting for 40 seconds before trying download of https://github.com/easybuilders/easybuild-easyconfigs/pull/22698.diff again...
== Waiting for 20 seconds before trying download of https://github.com/easybuilders/easybuild-easyconfigs/pull/22852.diff again...
== Waiting for 40 seconds before trying download of https://github.com/easybuilders/easybuild-easyconfigs/pull/22852.diff again...
== Waiting for 20 seconds before trying download of https://github.com/easybuilders/easybuild-easyconfigs/pull/22827.diff again...
== Waiting for 40 seconds before trying download of https://github.com/easybuilders/easybuild-easyconfigs/pull/22827.diff again...

143 out of 146 required modules missing:

The printed messages I added for debugging purposes, but it's probably wise to actually have those, so I'll update the PR accordingly.

The first wait time is also not the intended 10 seconds, it straight goes to 2*10, need to fix that too.

[sassy-crick]

Just to add: I tried eb --copy-ec --from-pr 22824 which fails for the reasons mentioned above but with the fixes it works well. Also, I tried two different IP addresses, home and work, and I had the same issue. Thus, the IP address where you are coming from does not seem to make a difference. At home I got the full GitHub integration setup, and that did not make a difference too.

[smoors]

works well for me too

[lexming]

LGTM

[lexming]

LGTM 2