Skip to content

Retry failed partial downloads with HTTP Range header when supported - #19075

Closed
apaz-cli wants to merge 14 commits into
astral-sh:mainfrom
apaz-cli:resumable-wheel-downloads
Closed

apaz-cli wants to merge 14 commits into
astral-sh:mainfrom
apaz-cli:resumable-wheel-downloads

Conversation

@apaz-cli

@apaz-cli apaz-cli commented Apr 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

For whatever reason, pypi resets the connection a lot, and we run out of retries. pip does not have this problem because it retries with http Range requests, and does not have to redownload the whole file.

I have been plagued with this issue for about a year, and it has made uv near-unusable for me, unless I am using an index other than official pypi. When I use uv I usually have to use the Tsinghua mirror, because pypi is flaky and uv cannot handle it.

I've gotten so frustrated with uv recently that I decided to take matters into my own hands.

It looks like someone else tried to address this in #17656, but that PR was rejected for being complicated, and the author never followed up. There have been other attempts too, if I recall.

Fixes #16934
Fixes #13359
Fixes #18642
Fixes #18538

Testing

Passed all tests in:

cargo insta test --accept --test-runner nextest

Except for python_upgrade::python_upgrade_implementation, which appears to be broken on main.

@apaz-cli
apaz-cli force-pushed the resumable-wheel-downloads branch from 43d4aaf to 313b371 Compare April 20, 2026 05:45
@konstin

konstin commented Apr 21, 2026

Copy link
Copy Markdown
Member

Can you add a test for this in network.rs? There is already logic for retry testing in there, hopefully we can reuse that logic for this check too

@apaz-cli

Copy link
Copy Markdown
Contributor Author

Should be ready now @konstin, the release binary tests appear to be failing, but unrelated.

@codspeed

codspeed Bot commented Apr 23, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 6 untouched benchmarks


Comparing apaz-cli:resumable-wheel-downloads (52df791) with main (496dbae)

Open in CodSpeed

@apaz-cli

Copy link
Copy Markdown
Contributor Author

More interesting failures

@woodruffw

woodruffw commented Apr 23, 2026 •

Copy link
Copy Markdown
Member

Those are from a GitHub outage earlier. I'll requeue those.

@apaz-cli

Copy link
Copy Markdown
Contributor Author

Looks like one error persists, the sdist build is attempting to build with an outdated version of Rust.

@woodruffw

Copy link
Copy Markdown
Member

Looks like one error persists, the sdist build is attempting to build with an outdated version of Rust.

Yeah, you'll need to rebase or merge main for that one: #19131

@apaz-cli
apaz-cli requested a review from konstin April 24, 2026 13:06
@apaz-cli

This comment was marked as off-topic.

@apaz-cli

This comment was marked as off-topic.

@woodruffw

Copy link
Copy Markdown
Member

@apaz-cli Please don't send us repeated pings to review; we review all PRs that come in as we're able.

@apaz-cli

Copy link
Copy Markdown
Contributor Author

fair enough, I will fork for now

@konstin konstin changed the title Retry failed partial downloads with HTTP Range header when supported. Retry failed partial downloads with HTTP Range header when supported May 16, 2026
@apaz-cli

Copy link
Copy Markdown
Contributor Author

It's been over a month, please review or close :)

@woodruffw
woodruffw self-requested a review May 24, 2026 19:34
@woodruffw

Copy link
Copy Markdown
Member

Time permitting, I will review this week.

@apaz-cli

Copy link
Copy Markdown
Contributor Author

ty :)

@notatallshaw

notatallshaw commented May 24, 2026 •

Copy link
Copy Markdown
Collaborator

I'll just note here to set expectations for when contributing to an open source project in general that the equivalent PR for pip had to wait 5 months for reviews to begin in earnest and 7 months to merge: pypa/pip#12991

Pinging maintainers can be helpful to make sure something isn't forgotten, but part of contributing to projects usually involves some patience on the timescale of months, not days.

@woodruffw woodruffw left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll take over the remainder of this PR. Some initial review below.

woodruffw added 2 commits May 26, 2026 10:21
Signed-off-by: William Woodruff <william@yossarian.net>
Signed-off-by: William Woodruff <william@yossarian.net>
@astral-sh-bot

astral-sh-bot Bot commented May 26, 2026

Copy link
Copy Markdown

uv test inventory changes

This PR changes the tests when compared with the latest main baseline.

  • Added tests: 2
  • Removed tests: 0
  • Changed suites: 1
uv::it: +2 / -0

Added:

  • uv::it::network::direct_url_no_range_resume
  • uv::it::network::direct_url_range_resume

Removed: none

Comment on lines +1111 to +1112
/// A valid wheel whose data descriptors require the on-disk extraction fallback.
async fn wheel_requiring_download() -> Vec<u8> {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noting: I'm not in love this this test fixture, since it relies on a piece of unrelated weirdness/implicit state (the fact that the default writer from async-zip uses data descriptors).

woodruffw added 2 commits May 26, 2026 11:41
Signed-off-by: William Woodruff <william@yossarian.net>
Signed-off-by: William Woodruff <william@yossarian.net>
woodruffw added 2 commits May 26, 2026 11:58
Signed-off-by: William Woodruff <william@yossarian.net>
Signed-off-by: William Woodruff <william@yossarian.net>
@woodruffw

Copy link
Copy Markdown
Member

The state machine is a little hard to follow here, so here's my attempt to map it out:

  1. Wheel downloads happen with our normal top-level middleware/retry stack.
  2. Happy path: normal non-ranged wheel download; any failures punt to the retry stack
  3. Sad path: non-ranged download partially fails
    • If the original request doesn't advertise support for ranges, we punt back to the retry stack
    • Otherwise, we try to resume at the last byte saved
    • If the range request fails, we punt back to the retry stack
    • If the range request succeeds but isn't a valid range response (206) or isn't for the right range, we punt back to the retry stack
    • Otherwise, we continue with our range loop, and break after a full conclusion of the download.

In effect this means that we'll use the retry machinery on pretty much every failure, except the subset of failures that we know correspond to a partial copy. When that occurs we'll entry our retry-independent loop, which should generally only execute once (either resuming the download to completion, or booting back to retry).

@konstin
konstin removed their request for review July 21, 2026 08:20
@apaz-cli

apaz-cli commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

Status on this? uv is still not usable for me.

@woodruffw

Copy link
Copy Markdown
Member

I'm continuing this in #21570, since the current branch is pretty far behind. My hope is we can land some version of this soon, but I need to be more confident that I understand the retry/resume state machine (especially around misbehaving origins, partial resumptions, etc.).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

4 participants