bug: synchronization sometimes hangs while still reporting connected peers

[zvolin]

[zvolin]

I've identified 3 different issues causing this.

panic in libp2p-kad

The kbucket uses std::time::Instant sometimes triggering runtime panic. After that the node is in kinda undefined state, I saw it either stopping logging completely or remaining active only on libp2p-gossipsub but not syncing, not updating peers etc.
We managed to get the fix for this in scope of libp2p/rust-libp2p#5347. We'll either have to wait for the 0.54 release of libp2p or try to get a backport.

IndexedDb hanging on committing in append_single_unchecked

Happened to me on firefox. Switching from .commit() to .done() didn't solve this, it hung infinitely. This causes syncer to hang too, and also our UI updates as we await node.syncer_stats() which never resolves. The only thing that helped was clearing the whole Idb store. I didn't see it happening any time later, not sure what caused that.

header-ex request never resolved on libp2p level

With Sessions we make a few requests in parallel (8 currently) and retry the ones which errored out / provided incomplete ranges of headers. It sometimes happen that for a single request we never get any event from the request-response behavior. It should time-out, finish or error, but in this case we just don't get any event. The Session.run() then hangs on

        while self.ongoing > 0 {
            let (height, requested_amount, res) = self.recv_response().await;

with self.ongoing == 1 and syncer waits for it to finish. We could solve this by re-introducing timeouts by hand in our header-ex behavior, however it'd be good to know what's going on in libp2p. I spotted one place which could lead to this bug, but there should be log indicating it which wasn't present in my reproduction. Additional debugging is needed here.

[zvolin]

I have a branch for debugging this here. It uses my fork of libp2p where I already backported the fix for the kbucket, but it's better to clone it locally, update patches to path based to be able to add logs manually in libp2p. It also has the bulk inserts into indexeddb implemented.

When debugging, run the node in chromium. There will be a lot of logs because it's on trace level. Wait for a 2-3 syncer batches to see if it reproduced. If it reproduced then wait half a minute or more, toggle the preserve logs button in dev tools and refresh page to stop the new logs from appearing to not flood you. You can then check the logs, they should persist the refresh. If it didn't reproduce, then just refresh and repeat, I found it hard to debug if I have more logs than from like 2-3 batches because there is a lot of them. Firefox has cool feature of saving all logs to the file too.

[oblique]

Blockers:

• fix(request-response): Report dial IO errors to the user libp2p/rust-libp2p#5429
• fix(request-response): Report failure when streams are at capacity libp2p/rust-libp2p#5417
• deps: use web-time instead of instant libp2p/rust-libp2p#5347
  • feat: Use web-time instead of instant libp2p/rust-yamux#191
  • Release of new yamux version with the above patch
  • chore: Use libp2p-tls from workspace libp2p/rust-libp2p#5452
• libp2p-ping sometimes panics in WASM libp2p/rust-libp2p#5442
  • (Proper fix) Fix panic when Delay is used after Poll::Ready async-rs/futures-timer#74
  • (Workaround for the above) fix(ping): Fix panic in WASM caused by retrying on dial upgrade errors libp2p/rust-libp2p#5447
• (Not really a blocker) Chromium throttles WebTransport connections causing connection instability #287
• Relase of libp2p v0.54