`STREAM_DATA_BLOCKED` and `DATA_BLOCKED` sent prematurely?

[larseggert]

I may have always misunderstood the RFC, or maybe we have an actual bug?

I see STREAM_DATA_BLOCKED and DATA_BLOCKED frames sent already when the application's write buffer is larger than remaining flow control credit, not when credit is actually exhausted. I think this violates RFC 9000 Section 4.1:

"If a sender has sent data up to the limit, it will be unable to send new data and is considered blocked. A sender SHOULD send a STREAM_DATA_BLOCKED or DATA_BLOCKED frame to indicate to the receiver that it has data to write but is blocked by flow control limits."

The frame should be sent when available() == 0, not when available() < app_write_size?

fn send_blocked_if_space_needed(&mut self, needed_space: usize) {
    if fc.available() <= needed_space {    // stream-level
        fc.blocked();
    }
    if conn_fc.borrow().available() <= needed_space {  // connection-level
        conn_fc.borrow_mut().blocked();
    }
}

When an application writes 100MB to a stream with a 1MB FC limit, blocked() fires immediately (1MB < 100MB) even though the sender has nearly 1MB of credit remaining.

The other call site, send_blocked_if_space_needed(0) after mark_as_sent, is correct — it checks for actual credit exhaustion.

[martinthomson]

I don't know why you closed this, because there's a real bug on our end here.

The way we manage STREAMS_BLOCKED is fine. We send it when the application wants a stream and can't have one. With data buffering, our current strategy isn't great. Because, as you say, we trigger the sending of the various blocked frames at the point of the write, not the point at which our buffer runs dry.

At the same time, we probably do benefit from sending this notice a little earlier. If we only sent it when we reach the end, there's a risk that we send it too late. Consider the interaction between atomic sends and the send_blocked_if_space_needed(0) in mark_as_sent. If we are using atomic sends, we might not completely fill the buffer.

(I'm also concerned that this arrangement ends up sending many blocked frames, because when the buffer is full we will send a blocked frame after every send.)

What I think we should do is something else: when there is a send and there isn't enough space for that data in the buffer, we remember that we wanted to send more than could be buffered. Clear that marker if we get more credit in the meantime. But if we get below a certain threshold (which might depend on our understanding of the BDP), sending the relevant blocked frame at that point will ensure that we are sending the signal early enough. My though is to set more than one threshold (e.g., 1/4th, 1/8th, empty) to ensure that the signal is stronger as the space drains more, rather than try to retransmit the frame, because retransmissions don't really make sense for blocked frames.

Of course, the goal is to only have a BDP worth of data buffered, so having a blocked message sent when you have 1/8th of a BDP left in the buffer isn't that much different from sending it when you are completely out, at least in terms of being a useful trigger for getting more credit. The difference is 7/8th of an RTT of dead air instead of a whole RTT. That just demonstrates that blocked frames aren't really that useful as a way to guarantee good performance, they are more useful to inform developers that they might need to tune their credit algorithms.

[larseggert]

(I think I misclicked.)

I wish the spec was a bit clearer on this. It reads like a sender should signal blockage only when no more packets can be sent (implying that it is the receiver who is tasked with making the sender avoid running into that.) But then there is also the sender who has an incentive to signal blockage "early" to "optimize" but then the receiver cannot trust the signal...

[martinthomson]

I've been pushing @mxinden to publish something about tuning flow control. Producing some more concrete guidance would seem to fit within that remit.