use of `--from-pr` or `--include-easyblocks-from-pr` sometimes fails because PR patch file was not downloaded

[boegel]

Symptom (with --include-easyblocks-from-pr 3705):

ERROR: Failed to read /tmp/eb-lot97q4h/ebs_pr3705/3705.diff: [Errno 2] No such file or directory: '/tmp/eb-lot97q4h/ebs_pr3705/3705.diff'

[verdurin]

Further detail from another local occurrence:

== 2025-05-06 14:28:43,286 filetools.py:785 DEBUG Trying to download 3705.diff from https://github.com/easybuilders/easybuild-easyblocks/pull/3705.diff to /tmp/eb-c8_n7_ph/ebs_pr3705/3705.diff
== 2025-05-06 14:28:43,287 filetools.py:788 DEBUG Using timeout of 10 seconds for initiating download
== 2025-05-06 14:28:43,287 filetools.py:2026 DEBUG Not creating existing path /tmp/eb-c8_n7_ph/ebs_pr3705
== 2025-05-06 14:28:43,470 filetools.py:865 WARNING URL https://github.com/easybuilders/easybuild-easyblocks/pull/3705.diff was not found (HTTP response code 429), not trying again

[Thyre]

This also occurs without patch files, on different systems, with different IP addresses.
Based on the status code, some rate limit might be hit (error code TOO_MANY_REQUESTS).

I ran into this with pure EasyConfig tests, e.g. easybuilders/easybuild-easyconfigs#22845 and with ones having requiring an changed EasyBlock as well, e.g. easybuilders/easybuild-easyconfigs#22842. Having both an EasyBlock and at least one EasyConfig to download triggers this more often.

The LLVM test runs on jsc-zen3 by @Crivella also trigger this issue A LOT, e.g easybuilders/easybuild-easyconfigs#22517

For me, it started to happen this week on my systems. I haven't checked all of them yet, but all are running pretty much the latest develop.

[Thyre]

This might help is in finding out if we hit a rate limit:

https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28#checking-the-status-of-your-rate-limit

The error described when one hits it matches what we're seeing:

https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28#exceeding-the-rate-limit

[Thyre]

I just added a print statement to check the error message.:

      <h1>Whoa there!</h1>
      <p>You have exceeded a secondary rate limit.<br><br>
        Please wait a few minutes before you try again;<br>
        in some cases this may take up to an hour.
      </p>
      <div id="suggestions">
        <a href="https://support.github.com/contact">Contact Support</a> &mdash;
        <a href="https://githubstatus.com">GitHub Status</a> &mdash;
        <a href="https://twitter.com/githubstatus">@githubstatus</a>
      </div>

Let's see what else I can extract...

---

The response object does not contain retry-after, x-ratelimit-remaining or x-ratelimit-reset.
Looking at the doc., we cannot find out what the current state of the secondary rate limit is: There is not a way to check the status of your secondary rate limit.

Maybe adding the GitHub token, if available, helps in working around the issue?

[ocaisa]

A valid token would almost certainly solve the problem. The rate limit issue is something we hit quite a bit in EESSI, and hence the move towards using --*from-commit instead, that's a bit more effort in terms of user experience though. Catching the error and suggesting the installation of a token if they plan on using these options a lot seems like the way forward (also informing about the alternative --from-commit in the docs perhaps).

[Thyre]

I tried passing the token when doing the requests (by more or less hacking it in, as it is not done by default as far as I can see), but some requests still failed. This may be due to us using https://raw.githubusercontent.com, but that's just a guess from my side. Maybe I missed something.

I also found a somewhat related discussion: https://github.com/orgs/community/discussions/157887
Looks like others run into the same issue (even without the AcceptLanguage flag).

[boegel]

This is not related to the GitHub token at all, I think.

We're doing a "raw" download here from a URL, we're not getting the *.diff file for a PR via the GitHub API (we're getting the URL via the API, but the actual download of that file is essentially done via urllib.request.urlopen.

Easy to trigger with:

python -c "import urllib.request; [urllib.request.urlopen('https://github.com/easybuilders/easybuild-easyblocks/pull/3705.diff').read() for _ in range(10)]"

which quickly leads to:

urllib.error.HTTPError: HTTP Error 429: Too Many Requests

[boegel]

fix/workaround implemented in:

• implement exponential backoff in download_file + allow max. 6 attempts to download PR diff in fetch_files_from_pr and fetch_files_from_commit #4870

[Thyre]

https://github.blog/changelog/2025-05-08-updated-rate-limits-for-unauthenticated-requests/

Finally an official update regarding changed rate limits, though no specific numbers are given. The documentation still shows the old numbers.
raw.githubusercontent.com is explicitly mentioned in this short changelog message.

[smoors]

using the github API for downloading the diff apparently has a much higher rate limit, even without a token. this runs without error for me:

repo="easybuilders/easybuild-easyblocks"
pr=3705
for x in {1..100}; do
    echo $x
    curl \ 
    --header "Accept: application/vnd.github.diff" \ 
    -Lo /tmp/diff \ 
    "https://api.github.com/repos/$repo/pulls/$pr"
done