When should we next squash the index?

[Eh2406]

Last (only) time: https://internals.rust-lang.org/t/cargos-crate-index-upcoming-squash-into-one-commit/8440 we had 100k+ commits and we thought we weighted a little too long (given how smoothly it went), now we have 51k + ~1.5k/week.

The Cargo team discussed this today and we think we should do this soon. Not interrupt whatever you are working on, but when you have a chance. Who has the permissions to run that script? Is it just @alexcrichton?

As the index grows we should have a policy for when we plan to do the squash. When we have a policy we should plan to make a bot to ensure we follow it. It is reasonable to say that it is too soon. Or we could make a simple policy for now and grow it as we need. The Cargo team discussed a policy like "when we remember approximately every 3-6 months" or "... approximately at 50k commits" or "... approximately when the squash is half the size of the history"

[alexcrichton]

The actual script:

the script

set -ex

now=`date '+%Y-%m-%d'`
git fetch origin
git reset --hard origin/master
head=`git rev-parse HEAD`
git push -f git@github.com:rust-lang/crates.io-index $head:refs/heads/snapshot-$now

msg=$(cat <<-END
Collapse index into one commit

Previous HEAD was $head, now on the \`snapshot-$now\` branch

More information about this change can be found [online] and on [this issue]

[online]: https://internals.rust-lang.org/t/cargos-crate-index-upcoming-squash-into-one-commit/8440
[this issue]: https://github.com/rust-lang/crates-io-cargo-teams/issues/47
END
)

new_rev=$(git commit-tree HEAD^{tree} -m "$msg")

git push \
  git@github.com:rust-lang/crates.io-index \
  $new_rev:refs/heads/master \
  --force-with-lease=refs/heads/master:$head

Edit: to include the critical --force-with-lease that was in https://internals.rust-lang.org/t/cargos-crate-index-upcoming-squash-into-one-commit/8440/31?u=eh2406

only requires push access to the crates.io-index, which any admin of the rust-lang GitHub organization has (and probably more).

I think it'd be best to do some measurements here directly correlated with the metrics we care about. The original rationale for squashing was that initial clones took quite a long time downloading so much history. As a result I would suspect that we should establish thresholds along the lines of "how big is the download and how much would we save with a squash"?

[sgrif]

which any admin of the rust-lang GitHub organization has (and probably more).

I'm also able to do it, and bors of course can (dunno if bors is an admin). I think that's it though.

This was discussed at the crates.io meeting. Here were the key points.

• It makes sense for some amount of this to be automated. crates.io has the infra for this, and is happy to have this on our servers.
• The last time this happened there was some public communication around it. Is this something that we want to do for each squash? If not, we can just automate the whole process.
• @sgrif points out that at least one crawler looks at commit dates to avoid having to ask crates.io for update times, so we should ensure that the history branch remains consistently named. The script already does this, but we should make sure that piece doesn't change
• We can automate this either by time or by commit count.
• If we don't want to completely automate the whole process, we can still automate checking whatever threshold we want and just send an email to relevant folks.
• The crates.io team is happy to take on the work around this (it's relatively minor for us)
• @joshtriplett is interested in seeing if it's possible to set this up so that git is able to do a smaller update for folks who would have done a fast forward had we not squashed.
• Ultimately if we get the "when to squash" threshold wrong, there was consensus that the cost is relatively minor

The main unresolved questions, which we'd like to get answers from the Cargo team on, are:

• Are we ok with completely automating the whole process, and therefore losing the ability to communicate beforehand?
• Should the threshold be time based or commit based?
• What should the threshold be?

My personal answer to those questions, which does not represent consensus among any team(s) are:

• Yes, we should automate it and not communicate when it happens. Nobody noticed last time. The only parties affected are crawlers, who have to handle this either way, and are better served by this being automated (and therefore ensured consistent).
• Commit based. While time based is probably better for crawlers (they can just assume there's a history branch every 6 months, etc), our primary focus should be on human users of Cargo. Commit count is ultimately the main factor behind all the problems we intend to solve by squashing.
• 75k commits. This is very uninformed, and entirely based on the issue description saying we thought 100k was too long to wait last time. This is an easy number to make configurable, and we should probably just experiment with what feels like the best balance.

[Eh2406]

A follow up to @joshtriplett suggestion.

To clone the index as is
git clone -b master --single-branch https://github.com/rust-lang/crates.io-index.git
downloads 61.9MiB
Then to fetch the squash that I made is
git fetch https://github.com/Eh2406/crates.io-index.git master
does not redownload the data!
If I dell that checkout, and clone the index from my squash
git clone -b master --single-branch https://github.com/Eh2406/crates.io-index.git
downloads 17.26MiB

So apparently we can get git to do this correctly! (Others should check if they are getting the same results.) The thing I tried https://github.com/Eh2406/crates.io-index/commit/65419fd5f5b9758b95fa08f207276639b1426e43 is to add a new squash commit on top of the existing one from last time. I did not make a script just did it manually. It may be sufficient to just share the same root commit, if someone wants to give that a try.

[Eh2406]

Looks like it works with the root in common, using git fetch https://github.com/Eh2406/crates.io-index.git test.

The root can be found with root = git rev-list --max-parents=0 HEAD
Then the penultimate line can be new_rev=$(git commit-tree HEAD^{tree} -m "$msg" -p $root)
And everything should work.

[alexcrichton]

For my own personal takes on some of the unresolved questions:

Are we ok with completely automating the whole process, and therefore losing the ability to communicate beforehand?

I don't have any problem with losing communication about this, I don't think it's really all that important especially now that it went so smoothly the first time. I do have a slightly different concern though. I think it would be a failure mode of Cargo if the index were automatically rolled up every day (defeating the purpose of delta updates), and having a fully automated process may cause us to not realize we're getting close to that situation.

I am, however, very much in favor of automation. So to allay my concern I would request that a notification of some form be sent out to interested team members when a squash happens. (aka I just want an email of some form)

Should the threshold be time based or commit based?

I would personally measure this in megabytes of data to download rather then either metric you mentioned, but commits are likely a good proxy for the megabytes being downloaded. My ideal metric would be something like "we shave 100MB off a clean download of the index", and the 100 number there is pulled out of thin air and could be more like 50 or something like that.

What should the threshold be?

I think the first index squash went from roughly 90MB to 10MB (ish) for a clean initial download. Along those lines I'd say that a squash should save at least 70MB before squashing.

[Eh2406]

I think it would be a failure mode of Cargo if the index were automatically rolled up every day (defeating the purpose of delta updates)

@alexcrichton One question, if git can download a roll up in O(delta) work would you still think this is a failure mode?

[alexcrichton]

AFAIK git just downloads objects and doesn't do any diffing at the fetch layer. Delta updates work because most indexes have a huge shared history. If we roll into one commit frequently there's no shared history so git will keep downloading the entire new history, which would be fresh each time.

So to answer your question, I don't believe git can have any sort of delta update when the history is changed and so I would still consider it a failure mode.

[smarnach]

For users who already have the latest version of the index, Git will generally see that the tree object for the single squashed commit is identical to the tree object it already has (since it has the same hash), so it will only donwload the single new commit object.

So another solution may be to always keep, say, the last month's worth of commits in the history, and only squash the bits that are older than one month. All users who have updated in the month before squashing will be able to download deltas, and only users with an even older version of the index will have to redownload it in full.

When squashing the old commits, all commits on top of them will have to be rewritten, so users will have to redownload the commit objects. However, commit objects hardly contain any data, and the associated tree objects are identical, so they won't be retransmitted.

I did some experiments for this approach, and got somewhat mixed results with what Git is able to detect, but I believe it is possible to make it work. It would require some work to figure out the details, though.

[smarnach]

We had some discussion in the crates.io Discord channel (can't figure out how to permalink it), and things aren't quite as easy as indicated in my previous comment. I may have time to do some experiments later this week, but I don't make any promises.

[Eh2406]

link to the discussion: https://discordapp.com/channels/442252698964721669/448525639469891595/597888610376613901

[Eh2406]

We did not have time to discuss this at the Cargo meeting today. So we don't have any new answers for @sgrif.

I would request that a notification of some form be sent out

I was thinking maybe we open and issue on the index repo and have the script add a comment there, then anyone interested (in teams or not) can subscribe to that issue to get notifications. I would want to look into @Nemo157 suggestions for how to get git not to download the history at all well before we start doing a squash every week.

I think the first index squash went from roughly 90MB to 10MB (ish)

>git clone -b master --single-branch https://github.com/rust-lang/crates.io-index.git
...
Receiving objects: 100% (297740/297740), 67.54 MiB | 5.79 MiB/s, done.

>git clone -b master --single-branch https://github.com/smarnach/crates.io-index
Cloning into 'crates.io-index'...
...
Receiving objects: 100% (36539/36539), 14.01 MiB | 5.75 MiB/s, done.

So it looks like we save ~54 MiB today. Assuming a linear size per commit then we would hit 70 MiB saved at ~ 72K Commits. So it looks like people's instincts are approximately in the same ballpark.

[joshtriplett]

It sounds like we don't need to keep a window of commits on the main branch, and we just need to archive the squashed-away commits on an archive branch? And since the server has those available it can do deltas from those objects? That sounds perfect.

[Eh2406]

We discussed this at the Cargo meeting today.

The main unresolved questions, which we'd like to get answers from the Cargo team on, are:

• Are we ok with completely automating the whole process, and therefore losing the ability to communicate beforehand?

Yes! Several of us would like some form of notification when it happens, but it does not need to be in advance and we do not need to publicize the event.

• Should the threshold be time based or commit based?

We realized that it was hard to make a decision do to a bikeshed effect, we all had different opinions but not strong enough to convince anyone. So we decided whatever is easiest for you to set up. If you need someone to make a decision, A daly check if we are over the commit limit.

• What should the threshold be?

After some discussion @ehuss pointed out that it is already noticeable, and @nrc pointed out that we want to have the script do something the first time it runs. We don't want it to break things on some random day in 3 month when we have non of this paged in. So if it is time based then every 6 months, if it is commit based then 50k. Most importantly We can monitor it and adjust the threshold later if needed.

We had some discussion of whether this will cause existing users to download the full index on each squash day. My understanding from our discussion with @Nemo157 and @smarnach on discord is that the current plan will not trigger a full download. The Github repo will always have a commit referencing all tree objects that the client will have, so Github will have what it needs to do a delta even when master has just been squashed. No git-gc can remove the tree objects as there used by a backup branch. @ehuss wanted to recheck to make sure that this works as hoped.