Openaddresses failing to download planet file #527
Description
Describe the bug
Running the openaddresses downloader for the planet (utils/download_all.js), the OA planet file fails to download with HTTP 429 (too many requests). I got this error initially, and on various machines, so I don't think the issue is that I've made a ton of requests.
info: [openaddresses-download] Attempting to download all data
error: [openaddresses-download] Failed to download data message=cURL request failed, HTTP status: 429, exit code: 22, stack=Error: cURL request failed, HTTP status: 429, exit code: 22
at ChildProcess.exithandler (node:child_process:422:12)
at ChildProcess.emit (node:events:517:28)
at maybeClose (node:internal/child_process:1098:16)
at Socket.<anonymous> (node:internal/child_process:450:11)
at Socket.emit (node:events:517:28)
at Pipe.<anonymous> (node:net:350:12), code=22, killed=false, signal=null, cmd=curl --request GET --silent --location --fail --write-out "%{http_code}" --referer https://pelias-results.openaddresses.io --output /tmp/202543-1-8rtbub.hd7x4.zip --retry 5 --retry-connrefused --retry-delay 5 https://data.openaddresses.io/openaddr-collected-global.zip
When I run the curl command directly, I see the same error.
Steps to Reproduce
Run:
curl --request GET --silent --location --fail --write-out "%{http_code}" --referer https://pelias-results.openaddresses.io --output /tmp/202543-1-8rtbub.hd7x4.zip --retry 5 --retry-connrefused --retry-delay 5 https://data.openaddresses.io/openaddr-collected-global.zip
Expected behavior
The container should download the OA planet file.
Environment (please complete the following information):
Linux and MacOS
Additional context
I'm running Pelias in the context of docker via Earthly, but I don't think it's relevant to the issue.
I know some time ago, OA introduced API tokens, and that pelias has a community token baked into the build process. #520 documented how you can add your own. I tried this, but it didn't change the behavior.
Looking at the code a bit, I don't actually see anywhere that this token is used.
On a lark I tried to pass in my own token into the above curl command as a Bearer token, but that also didn't change any behavior:
curl -H "Authorization: Bearer XXX...