Remove restriction that a crate must be in the database to get a download URL

[carols10cents]

So I've been thinking about/working on making it easier to run a crates.io mirror. The mirror in particular that I'm working on would basically just mirror the crates.io API:

• A cargo user would change their registry index URL in their .cargo/config to point to an alternate index
• That index would be a clone of the official index with one extra commit changing the config.json to point to an alternate crates.io instance
• The crates.io instance would have its S3_BUCKET set to crates-io and its S3_REGION set to us-west-1, so it would NOT be mirroring the actual crate files

The problem I hit was that in order to get a redirect from /crates/:crate_id/:version/download to S3, the crate and version needed to be in the database, because the current code would try to update the download counts.

I thought about creating a script that would load the contents of the index back into a database, but then it would have to keep that in sync as changes to the index happened, so I stepped back from that tactic and decided to try this.

This PR tries to increment the download count, but if that isn't possible for any reason, return the redirect to S3 anyway. Cargo checks the index it has before requesting the URL from the API anyway, so as long as the index is a (possibly lagging behind) mirror, cargo should not be requesting files that don't exist on S3. If it does, cargo will just say "failed to download package whatever from http://...s3". This might cause other problems to go unnoticed, but seeing as this is just the download count and any problems with the database would likely manifest in other ways, I think this is ok.

Is there a more idiomatic way to use the Result but ignore errors? I know that's not recommended in general, but that's really what I'm trying to do here :)

With this patch deployed to https://cratesio-mirror.herokuapp.com/, I'm able to set my .cargo/config to use https://gitlab.com/integer32llc/crates.io-index and install crates!

I'm totally open to other ideas if there's a better way to solve this problem!

[alexcrichton]

Thanks for the PR @carols10cents! This sounds pretty reasonable to me, and I'm game! Exciting to see work on mirrors :)

I personally prefer to ignore results through either let _ = foo(); or just drop(foo()); with a comment indicating why the error is ignored.

I wonder though, perhaps just the failed-to-lookup error could be ignored, but things like db errors could still get plumbed through?

Also, as a side note, with source replacement now on nightly I believe it should be as simple as:

# in .cargo/config
[source.crates-io]
registry = 'https://github.com/rust-lang/crates.io-index'
replace-with = 'my-mirror'

[source.my-mirror]
registry = 'https://...' 

[alexcrichton]

And with that as well Cargo will generate lock files that are interoperable with the default registry as well as mirror registries!

[carols10cents]

I wonder though, perhaps just the failed-to-lookup error could be ignored, but things like db errors could still get plumbed through?

If we're a mirror, though, we'd still want to ignore all errors, including db errors, right? And send the download URL regardless?

I have this thought that I haven't researched the implications of yet, that some mirrors might not need to have a database at all.

And if we weren't acting as a mirror (in this case, that we do want to be counting downloads), then we actually don't want to ignore ANY errors from updating the download counts, right? :-/

It almost feels like we'd need to have a setting that said whether we were a mirror or not, but then we'd also want to be able to say what kind of mirror we were: are we trying to be an exact match of crates.io? A superset that allows additional (private) uploads? A subset that has crates that have been audited and approved?

This is where I start getting overwhelmed as to how much to start with, and how to implement a little bit at a time but be extensible :) I'd love ideas!

[carols10cents]

I added a new commit to change to let _ = and have more of an explanation!

And with that as well Cargo will generate lock files that are interoperable with the default registry as well as mirror registries!

This is so awesome!!! My .cargo/config is now:

[source.mirror]
registry = "https://gitlab.com/integer32llc/crates.io-index"

[source.crates-io]
replace-with = "mirror"

and I can remove that, do a clean rebuild, and the Cargo.lock stays the same!!! 😻

[alexcrichton]

Thanks!

So thinking about this a bit more, this means that if you're trying to download "crate-that-does-not-exist" at "version-that-does-not-exist", you'd get a 302 to go talk to S3? We may want to head that off quickly with a 404 (or at least a 200 with a JSON error like we do today), though?

Or at least crates.io itself may wish to do that even if mirrors do not

[carols10cents]

So thinking about this a bit more, this means that if you're trying to download "crate-that-does-not-exist" at "version-that-does-not-exist", you'd get a 302 to go talk to S3? We may want to head that off quickly with a 404 (or at least a 200 with a JSON error like we do today), though?

Before we even get to that point, though, cargo looks in its locally checked out index and won't even make a request to this route if the crate+version isn't in its index. I'm not sure how/if/how likely it is to get into a situation where a crate+version IS in the index but ISN'T on s3... do you have any ideas?

Or at least crates.io itself may wish to do that even if mirrors do not

The problem I'm having is how to tell if we're crates.io or if we're a mirror. What mechanism would you suggest? Should I add another environment variable that's like... MIRROR?

[alexcrichton]

Yeah it's true that Cargo itself may not send these requests to S3, but I'm also somewhat worried about other clients who download directly from crates.io for whatever reason (or future Cargo implementations perhaps). Maybe it's not worth worrying about them though?

I guess thinking a bit more about what this should be (in terms of mechanisms), what kind of mirror are you going for here? I believe that this is going to a mirror of the index, but not necessarily the crates themselves, right? Maybe a caching mirror would look different where it'd lazily fill in packages from S3 but it still would require everything to exist locally?

[carols10cents]

For this step, I'm going for an API-only mirror. I thought this would be a good midpoint between what exists now and a mirror that lazily caches the packages to its own storage, but if you'd rather I proceed directly to trying to get a mirror like that working, I'd be into that :)

A lazy cache mirror still would have the same problem, in that when you hit the download route, it needs to have different behavior than crates.io does now, so how do we configure a mirror vs not a mirror?

[alexcrichton]

Hm yeah so if we want to end up at just an API mirror then this makes sense to me (perhaps with a setting for crates.io to differ), but if that's not the end goal we'd end up changing this anyway, right?

I guess I'd expect a download route to behave with logic like:

• Check if it's in the database
• If so, we've probably got it cached somewhere or we've cached that it's not on crates.io
• If not, we fetch it from crates.io inline and cache it

In any case though I'd be fine merging this for now with something like MIRROR=1 to enable it if it helps move forward!

[carols10cents]

Yeah i have no idea if an API-only mirror is anything that's useful by itself! I mean, it would work, I just don't know if that's anything useful enough for anyone to run. (shrug)

[carols10cents]

Added the MIRROR=1 setting! :)

[alexcrichton]

Looks good to me, thanks!