Some GH PR references were not ported correctly

[JustAnotherArchivist]

python/cpython#90908 (comment) has two incorrect references. The first one goes to GH-75453, which is bpo-31270. It should be GH-31270 (https://bugs.python.org/issue46752#msg413260). Note that the reference is correct in the following 'New changeset' comment.

This is the only case I've seen so far, but I haven't systematically looked for more.

[JustAnotherArchivist]

Found two more:

python/cpython#91294 (comment) references GH-76305 instead of GH-32124.
python/cpython#91302 (comment) references GH-76242 instead of GH-32061.

[ezio-melotti]

It seems like GH-* were interpreted as issue number and converted automatically to the new migrated issue during the transfer. This automatic conversion was supposed to be disabled during the transfer, but apparently it wasn't.

[vstinner]

Another example. In https://bugs.python.org/issue47075 I explicitly wrote I proposed GH-32112 for that. to get a link to my PR python/cpython#32112 but the converted issue python/cpython#91231 points to python/cpython#76293 i don't understand how GH-32112 was transformed to #76293 which is an unrelated issue (not a PR)

[ezio-melotti]

i don't understand how GH-32112 was transformed to #76293 which is an unrelated issue (not a PR)

During the transfer, https://bugs.python.org/issue32112 was transferred to python/cpython#76293. The transfer tool thought that your GH-32112 was a reference to bpo-32112 (rather than referring to a GitHub PR), and since bpo-32112 is now GH-76293, it converted GH-32112 into GH-76293.

[gvanrossum]

The transfer tool thought that your GH-32112 was a reference to bpo-32112 (rather than referring to a GitHub PR)

How could that happen? That seems terrible. :-(

[JustAnotherArchivist]

Any update on whether/how/when this could hopefully get fixed?

[vstinner]

The transfer tool thought that your GH-32112 was a reference to bpo-32112

Would it be possible to process all data once more time to detect the issue and fix it? This issue is quite annoying.

The current workaround is to go to the original issue, find the message to look for the correct GH issue/PR number.

[serhiy-storchaka]

I agree that it is very serious issue. For now we have incorrect references and the reader can not even know that they are incorrect.

Can we repeat translation which correctly detects GitHub references locally, compare it with the old incorrect translation, and apply the diff? The diff should be orders smaller that the whole data (but it still can be thousands messages).

[ezio-melotti]

Since this happened at transfer time, one of the test repos that I kept around might still have the correct references and it might be used to fix the links. Doing it from bpo is a bit trickier because there is quite a bit of processing that happened at import time (even though it should be possible to extract only the references).

The main issue is that this requires a mass-rewrite which will have to go through thousands of issues and comments and edit them, and this is not trivial to write and test. At this point we will also have to use the public API, which is rate limited. If it is done, the same infrastructure could also be used to replace bpo links with GH links in the first message of the PRs, and to @mention again nosy list members in the first message of issues. I think this will also edit the "last updated" field of all the affected issues/PRs, making it more difficult to find old/stale issues/PRs.

[serhiy-storchaka]

How many issues are affected? 100, 1000 or 10000? If it is only few 100s, we can survive editing the "last updated" field. But in long term it would be better to find a way to edit messages without changing the "last updated" field. We may need it to fix references to old Subversion and Mercurial revisions.

[ezio-melotti]

I don't have a number, and when we discussed different options to edit the messages after the fact and they all changed the "last updated" field (technically GitHub could rewrite the messages directly in the DB, but it's not something they would like to do and we would need their help).

[JustAnotherArchivist]

I can get an estimate of this shortly.

[JustAnotherArchivist]

There are about 19270 lines in messages on bpo (as of late March, shortly before the migration) that contain a link with the text GH-*. This number includes the 'New changeset' messages, which were migrated correctly, and I'm not sure how to filter them out (without a complete export of the GitHub issues to compare it directly). I don't see any differences between the two in bpo's HTML. For example, the links mentioned in the original issue are:

• <a href="https://github.com/python/cpython/pull/31270" class="closed" title="GitHub PR 31270: [merged] Merged">GH-31270</a> – Incorrect in GH issue
• <a href="https://github.com/python/cpython/pull/31313" title="GitHub PR 31313">GH-31313</a> – Incorrect in GH issue
• <a href="https://github.com/python/cpython/pull/31270" class="closed" title="GitHub PR 31270: [merged] Merged">GH-31270</a> – Correct in GH issue

In addition, there are about 5993 lines in bpo messages that contain other links to GH PRs. Roughly half of these are PR # links (example), the others are mostly just plain URLs (example). A brief sampling seems to indicate that those were transferred correctly.

[JustAnotherArchivist]

One thing I suspected regarding those correctly migrated links is that perhaps messages that first contain a bpo link and then a GH PR link worked. This does not appear to be the case: exhibit A, exhibit B.

I also found one issue with a weird link on GitHub: this comment was rewritten as python/issues-test-cpython#30631. It should clearly point to python/cpython#30631.

[ezio-melotti]

A few things that might help you:

• the migration happened in two stages: the import from bpo to a new GitHub repo, and the transfer from that repo to python/cpython.
• before the import, I did a lot of processing to fix/convert some links, add MarkDown markup, add the first message with the table, etc. In particular, I converted GH-* and PR-* references (but not plain URLs) using this regex:

    CPYTHON_PR = 'https://github.com/python/cpython/pull/'
    # PR-123, PR 123, PR123, pull request 123, GH 123, GH123
    # use a markdown link to preserve the original text -- the popup works
    (re.compile(r'(?P<text>(\b(?<![-/_])(PR-?|GH|pull\s*request))\s*'
                r'(?P<pr_no>\d+))\b', re.I),
     r'[\g<text>](%s\g<pr_no>)' % CPYTHON_PR),

  and bpo issues with

    # issue1234, issue 1234, bpo 1234, #1234
    (re.compile(r'(?:(?<!\w)\#|\b(?<![-/_])(?:issue|bpo))'
                r' ?(1?\d{4,6})\b', re.I),
     r'bpo-\1'),

• during the transfer, the tool used by GitHub apparently converted some of the links. I know that one limitation of the tool is that it could only convert issues that had already been migrated. For example, if there was an issue with id 1234 that was migrated to e.g. https://github.com/python/cpython/issue/4321, issues that had references like #1234 could have been converted to #4321, but only in messages migrated after issue 1234. This was the reason why we converted all the issue references to bpo-* instead of relying on their link conversion (that was supposed to be disabled, since it wasn't relyable).

This number includes the 'New changeset' messages, which were migrated correctly, and I'm not sure how to filter them out (without a complete export of the GitHub issues to compare it directly).

Maybe they were migrated correctly because the regex used by GitHub to find links/references didn't match when the reference was surrounded by (...), like in the "New changesets" messages. All those message have the same format, so it shouldn't be difficult to filter them out (a msg.startswith('New changeset') is probably enough).

If you want to see the pre-transfer messages, I can give you access to one of the test repos. If you have other questions let me know.