Self-hosted PDS not being picked up by relay

[kaylathar]

Hiya folks. I migrated to a self hosted PDS recently and am running into an issue where it seems like relays are not picking up events from the PDS - so while making posts and whatnot will successfully write data that's readable in clearsky or pdsls it's not appearing in any clients, and I have traced it to not propagating to the relay.

Running relay hound against my PDS and bsky.network and performing an action leads to it being picked up on the websocket from the pds, but not the relay. I've already requested crawl many hours ago as well. Screenshot of relay output below. My DID is did:plc:obkeyqheqcqn65w4zwr6kzli in case that helps.

I've confirmed my previous PDS (bsky itself) has my account deactivated as well. I tried some various solutions as well like bumping seq in case that was the issue, to no avail.

I'm only hosting one account, but could this be hitting some weird rate limit or caching issue?

[kaylathar]

Am now seeing actions propagate to the relay, going to leave this open though as I have heard others have seen similar issues, and would be good to understand why the relay seems to have such a long delay before picking things up.

[gitlimes]

in case it's useful for debugging this, i am also having the exact same issue, and i have also tried bumping seq.

did: did:plc:4hvfudiuyuwyhbtwcakljvit
pds: https://pds.limes.pink

[gitlimes]

i can confirm it's now working. see this post

[gitlimes]

ok i think i just need to shut up because it's broken again (https://pdsls.dev/at://did:plc:4hvfudiuyuwyhbtwcakljvit/app.bsky.feed.post/3m2nbwhec7s2v)

[bnewbold]

I'll need to confirm, but I think your PDS hit a rare race-condition bug with our old relay implementation, which breaks all events.

[pfych]

I think I've also run into this issue - weirdly had my first post index correctly but no subsequent actions seem to be indexed.

did: did:plc:lpxtxymhlfbs5foxba7mfewt
pds: https://atmosphere.pfy.ch

[DavidBuchanan314]

As Bryan mentions, there is an unfortunate race-condition bug in the relay implementation that means some PDSes don't get relayed properly. The bug gets triggered (occasionally) when a new PDS connects for the first time, and from that point onwards relaying will be broken or inconsistent for that PDS (hence the occasional "sometimes it works" observations).

In the near/medium-term this will be fixed by switching to the new relay implementation, but in the interim there is a possible workaround:

(These steps assume you are both the admin of the PDS, and owner of any accounts hosted on it)

• Set up a new PDS at a different hostname (e.g. if your original was pds.example.com, spin up another at pds2.example.com )
• Create a test account on the new PDS, or migrate an existing user to the new PDS (following the standard migration procedures)
• Check if the account on the new PDS works. Are their records getting relayed? (you probably won't lose the race condition this time around)
• If it works, you can migrate any remaining users from the old PDS to the new PDS (again, following standard migration procedures)
• If there are no users left on the original PDS, you can safely shut it down.

This is a fairly hands-on workaround, so apologies for the inconvenience. If you try this and it works, please let me know!

And just to confirm, the following PDSes are currently affected by the race-condition bug: pds.limes.pink, atmosphere.pfy.ch, pds.kayla.is

[gitlimes]

Yup, spun up a new pds at https://pds.atproto.limes.pink, and that one seems to be working fine! Migrated everything and yeeted the old one with no issues. Thanks!

[jegtnes]

I had the same issue as the OP but my PDS did not get picked up after a couple of days except for a small period. It was probably exacerbated by me not realising that the initial Bluesky PDS -> self-hosted PDS migration process had a partial failure and didn't deactivate the account on the Bluesky PDS, which I fixed after about a day, but it never recovered from there. This was only an issue with the bsky.network relay; relay1.us-{west,east}.bsky.network as well as third-party relays such as relay3.fr.hose.cam picked up my PDS without any issues.

Eventually I just tried what you suggested and migrated to a brand new PDS on a new VPS with a new domain and everything worked pretty much instantly, including bsky.network. Thanks for the suggestion and thanks to everybody else in this thread for chiming in.