Update override for .co and be more selective about following related links

[elidickinson]

Hello!

The .co registry moved their RDAP server recently. I updated it in the overrides and it seems to work now.

I also included I guess a somewhat related feature: the .co RDAP data always includes a value like:

    {
      "title": ".co Registry",
      "rel": "related",
      "type": "text/html",
      "value": "https://rdap.registry.co/co/domain/foo.co",
      "href": "https://registry.co/"
    }

Which obviously goes to their homepage and not to a JSON with more info.

I added a check in _domain to ignore related links with a type like text/html. I tested domains at a bunch of different registries and they all seem to use the correct content type of application/rdap+json so I think this should be pretty safe (but I admit I didn't test all of them). Seems like you probably intended to make that the client's problem, which fair enough, but it's annoying that it fails every time. An alternative approach would be adding something like ignore_related=True to overrides for .co

[meeb]

Thanks! Yeah the override endpoints are annoying to keep up to date.

[case]

Thanks! Yeah the override endpoints are annoying to keep up to date.

I’m going to set up a bit of monitoring and try to do a better job of keeping them up to date. 😅

[elidickinson]

No worries at all. And yeah I guess the ccTLDs are kind of a mess huh? Thanks for maintaining the library. Perfect fit for a thing I needed to alert on domain status changes for a big list of domains.

You might have something fancier in mind for monitoring, but if it were me, I'd stuff a list of domains in a test case and set up a github action that runs the test on a schedule.

[case]

but if it were me, I'd stuff a list of domains in a test case and set up a github action that runs the test on a schedule.

LOL you read my mind.

[meeb]

That's actually what I was also originally planning, I'm just pretty busy these days!

[case]

Fwiw I got a bit carried away and made this - https://github.com/case/iana-data ;)

There's a dashboard here - https://cctld-rdap.checkly-dashboards.com/ - which keeps an eye on the various "unlisted" ccTLD RDAP servers.

…which I'm keeping track of here - https://github.com/case/iana-data/blob/main/data/manual/supplemental-cctld-rdap.json

[meeb]

Thanks! Looks useful. I'll add a tool that can reference your supplemental ccTLD list which can be periodically used to update the overrides.

[case]

The key takeaway is that when a gTLD registry backend provider (like Identity Digital or CentralNic) also operates ccTLDs, for whatever reason their RDAP endpoints tend to respond to ccTLD queries as well. So we can poke around their RDAP servers to see if they have ccTLD responses.

[meeb]

Yeah, happy to send queries to a backend provider by default if that works! Hopefully at some point more ccTLDs get on board the RDAP train properly.

edit: Sync'd off your data: https://github.com/meeb/whoisit/blob/main/whoisit/overrides.py

There's a hacky script in tools/ to sync the overrides.py file. I'll bundle the updates into the next release.