missing rustc commits

[Mark-Simulacrum]

We are not infrequently losing rustc commits from the database log. We should try to prevent this by auto-healing more reliably.

A query like select parent_sha from rustc_commits where parent_sha not in (select sha from rustc_commits) and time > '2021-01-01'; against the database can give us the list of SHAs we're missing (potentially incomplete, but these should all definitely be in the DB). From there, we should be able to query GitHub for their information (PR, parent SHA, time) and fill the database in.

It is likely that doing this instead of / in place of the current logic which attempts to "scan back" on a git push to rust-lang/rust is preferable, and should be fairly reliable. We will need to be careful to avoid falling into any infinite loops, but that should not be terribly hard.

Implementing this should also hopefully be not too hard, and I'd be happy for someone to invest time into doing so -- please feel free to ping me on Zulip (or this issue). I'll likely get to it eventually because it's becoming a hassle to re-add commits manually, but that may take some time.

[Kobzol]

I'll try to write up my understanding of the whole process, it could also serve as some basic
documentation for future reference.

So my understanding is that currently the triagebot stores Rust master commits into database
by listening for comments on PRs. If it finds a comment that is from bors and deals with a merge
into master, is gets the SHA from it, then it downloads data from GH based on this SHA and stores
the data into the database.

Why does the bot scan comments instead of listening for push events to master? Would it
be more difficult to find the corresponding PR from the push?

When a merge commit happens, I'm not exactly sure what are the corresponding SHAs. Is it like this?
sha: commit of the new (merged) merge commit
parent sha: parent of the merged commit, i.e. the previous merge commit

Then when the commit is recorded, its parent shas are fetched recursively. I'm not sure how the
stop condition works though - it seems that the search stops when a SHA can't be resolved or
when it's message doesn't contain a PR number (or when it's already in the database?).

Why is the recursive search performed in the first place? Do the events sometimes fail and then
the commits are missed? How can it happen that some commit is missing?

When it happens, what problems does it cause? /bors-commit-list shows some commit that is then
used by rustc-perf, but the commit does not actually exist in master?

[Mark-Simulacrum]

So my understanding is that currently the triagebot stores Rust master commits into database
by listening for comments on PRs. If it finds a comment that is from bors and deals with a merge
into master, is gets the SHA from it, then it downloads data from GH based on this SHA and stores
the data into the database.

Yep, this is correct.

Why does the bot scan comments instead of listening for push events to master? Would it
be more difficult to find the corresponding PR from the push?

The comment gives us direct access to the PR number related to the merge, which works slightly better than push events IMO. In general it doesn't actually matter though since we do want to support "backscanning" to recover from missed webhooks, so we could switch to looking for pushes to master as well.

• sha is the bors 'auto merge of ...' commit for a given PR
• parent sha is the previous bors auto merge commit

I think this matches your impression.

Then when the commit is recorded, its parent shas are fetched recursively. I'm not sure how the
stop condition works though - it seems that the search stops when a SHA can't be resolved or
when it's message doesn't contain a PR number (or when it's already in the database?).

The goal is to keep scanning back until we have recovered fully -- ideally, that means it's in the DB, but it's possible this should use a different approach (e.g., using the DB with the select parent_sha from rustc_commits where parent_sha not in (select sha from rustc_commits) and time > '2021-01-01'; query until that doesn't return any rows), and doing that on a ~30 minute sleep background thread or something.

---

Why is the recursive search performed in the first place? Do the events sometimes fail and then
the commits are missed? How can it happen that some commit is missing?

Yes, GitHub can forget to deliver webhooks or we might be restarting or otherwise miss the webhook event (the first of these maybe surprisingly happens much more often, I think).

When it happens, what problems does it cause? /bors-commit-list shows some commit that is then
used by rustc-perf, but the commit does not actually exist in master?

When we're missing a commit, tools that use bors-commit-list to scan back (e.g., rustc-perf's queue) or bisect tooling may get confused, as they expect and somewhat rely on an unbroken chain of commits (i.e., each parent_sha points to a known commit) -- this is not the case when we're missing a commit or more in the bors-commit-list. Those tools code be adjusted to be more resilient, but in any case we want those commits in the list as we do eventually want to run perf on them :)

[Kobzol]

I see, thanks for the explanation. It seems to me that the easiest way to go forward is to replace the current recursive search with the query that you have mentioned (or more like combine these two approaches). It should probably be done with some timeout or maximum number of tries to avoid "infinite" loops.

[Mark-Simulacrum]

That seems broadly correct to me; a background task or thread would likely work well, though some logging/upper bound is still a good idea I think.