Allow robust reconnects when client do not perform graceful disconnects

[pranavkm]

The renderer goes in a “buffering” mode when the client disconnects and spits it back out once it reconnects. In the most common case where the client has a transient network failure, this is what it would look like:

Connection
   | Send diff 1
   | Send diff 2
   | Send diff 3
Disconnected (client network error)
   | Send diff 4
   | Send diff 5
SignalR OnDisconnected is executed
  | Buffer diff 6 
Client reconnected
  | Send diff 6
  | Send diff 7

SignalR \ Kestrel do not complain if a websocket send failed due to a client error. Consequently, there will be a gap where the server thinks it successfully sent the diffs to the client, but it ended up in the ether.

The fix for this will be that we're going to queue the render batches on the server, and hold onto them until the client can ACK each one.

[pranavkm]

@SteveSandersonMS:

This is definitely a difficult and important problem to solve. Unfortunately, even when we know that some diffs have been lost, recovery is not as simple as “render the affected components again” (e.g., by keeping them in the queue until we get an ACK), because by that stage we have lost the “before” state that the client last knew about, so subsequent diffs will be based on the wrong starting point.

It’s also not enough to say “well just re-render everything from scratch, and have the client rebuild the UI completely”. This would lead to broken scenarios, e.g., when people are using JS interop to interact with 3rd-party JS libraries. We can’t replay the sequence of calls they have made to the JS libraries, and hence can’t ever reliably recreate the state of the client from scratch. We can only safely do diffs from the known previous state.

We already do have a system for knowing when the client has acknowledged receipt of a given render batch. We put this in so that we’d know when it was safe to dispose event handlers (we can only do that after the client acknowledges that it’s removed all its own references to a given event handler ID, so we know it’s definitely not going to try to raise more events for that handler ID).

We might end up needing to solve this by actually buffering the binary renderbatch data in a queue that we only remove from when we receive the client’s acknowledgements that it applied each batch. Then on reconnect, we replay everything the client hasn’t yet acknowledged. We’d probably need to introduce some notion of a batch sequence number, so the client can skip repeats of batches it already did apply when it’s the ACK that was lost, not the batch itself.

Even if we do this, it’s still valuable to have the server switch into a “client is offline, so I don’t do more renders until it comes back” mode, so that we can save memory and CPU time by not actually doing renders and expanding the renderbatch buffer once we know we’re not able to send. So Pranav’s current work is definitely not wasted effort.

Summary: I think we can solve this, it just needs some more machinery. And I’d be fine with Pranav’s current PR only handling the case where the disconnection was “clean” (i.e., there were no batches in flight, and the server found out about the disconnection before its next render attempt). We’ll have to not promise to customers that the current reconnection logic is robust.

Maybe with Pranav’s PR we should add a sequence number to render batches so that we can at least detect the “lost diffs” case on the client, and not just have corrupted behaviour in that case.

@javiercn:

I actually think this is the most straightforward solution. I would draw a parallel with how TCP works:
• Batches are stored and sent to the client with a sequence number and the client is responsible for applying them in order and sending acks (which as you mention already does).
• The server retries sending a batch X times before giving up.
• We keep a window size Y for each renderer and we put batch entries in that window until we fill it. (At which point we prevent new renders from happening).
• When the client acks the next batch in the sequence we slide the window and new space is opened for pending renders to continue.

For prerendering you simply start disconnected and capture the initial batches. When the client reconnects, from the prerendered state (which we'll do by imprinting the circuit id into the selector tag, retrieving it on the JS side and resuming the connection) you start sending the batches.

The window ensures limits how much memory we use (framework, individual batch size is always up to the app) and also makes it easy to establish a total memory limit on the set of rendered batches.

[mkArtakMSFT]

Let's re-evaluate what we want to do here in the future.