ACK frames are unnecessarily limited to 32 ranges

[larseggert]

Problem

write_frame in tracking.rs caps ACK frames at MAX_ACKS_PER_FRAME (32) ranges to "keep plenty of space for other stuff." This cap is overly conservative in two situations:

1. ACK-only packets (cwnd < 256): These are limited to 255 bytes by ACK_ONLY_SIZE_LIMIT, which fits only ~13 ranges. The cap of 32 never fires — the space constraint is always binding. These packets could use the full MTU instead, since ACK-only packets are exempt from congestion control per RFC 9002 Section 6.1.

2. Non-ACK-only packets with nothing else to send: When cwnd >= 256 but no data or control frames are queued, the packet is effectively ACK-only, yet the 32-range cap still applies. At a 1200-byte MTU, up to ~72 ranges could fit, but 40 of those slots go unused.

Impact

After #3639 increased MAX_TRACKED_RANGES to 128, we can now track many more ranges — but we can only communicate 32 per frame. Under reordering, this means it takes more round trips to acknowledge all received packets, prolonging the window where the sender might falsely declare packets lost.

Possible approaches

• Remove MAX_ACKS_PER_FRAME entirely. The space-based formula 1 + (avail / 16) already uses a pessimistic 16-bytes-per-range estimate. In practice, most ranges encode much smaller, so even at the calculated maximum, the ACK frame leaves substantial room for other frames. The cap is belt-and-suspenders on top of an already conservative estimate.

• Lift ACK_ONLY_SIZE_LIMIT to the full MTU. Since ACK-only packets don't count against the congestion window (RFC 9002 Section 6.1), there's no reason to artificially constrain them to 255 bytes. This was an implementation choice from Pacing #618 (2020), not an RFC requirement.

• Write other frames first, ACK last. This would let the ACK frame fill whatever space remains, but is a more invasive change.

Context

ACK_ONLY_SIZE_LIMIT (256) was introduced in #618 and MAX_ACKS_PER_FRAME (32) in #998, both without recorded discussion about the specific values.

[martinthomson]

I don't think that we can write ACK frames last. Not without reserving space for at least some ranges.

Nor can we lift the ACK-only size limit to the full MTU, at least not without tweaking some of the code that depends on that to determine that the packet is ACK-only.

We could switch to a size-based limit on what we send in ACKs. I didn't do that originally because that degrades as the connection starts getting used more. But you are right to point out that 32 ranges often encode to not much more than the 70-odd byte minimum size that implies.

There's still a good case for having the number of ACK ranges in each packet. I think. Consider that we send N ACKs per RTT. In that case, when we are tracking M ranges, it might be reasonable to have each ACK contain M/N ranges. Of course, that only works if the ranges we send are different each time. We'd need to be smarter about which ranges we communicate, which we don't do (see below for one idea). If things are moving along, we might reasonably expect newer ranges (the latest ones; the ones we currently pick) to be the ones that are sent each time, so maybe that doesn't need a whole lot of extra smarts. However, if we are seeing reordering in any significant measure, there could be packets arriving that attach to ranges outside of the 32 we send.

Another idea here is that we could include counters on each ACK range. Fresh ranges have been extended since the last ACK frame. Or the ACK frame they were originally sent in was marked lost. Those both reset to 0. A counter increments each time the range is sent in a frame. We could then skip non-fresh ranges when assembling the ACK frame. Or ranges that have been sent in frames fewer than some number of times.

The way to decide this sort of thing is to test this out. And to set some performance targets. The simulator can probably help there.

[mvanhorn]

Implemented in PR #3674 - range count now sized against actual packet space, including the 2-byte varint boundary at 64+ ranges.