Description
StreamingPull currently has a "protocol" level bug affecting all languages.
TLDR: client libs should send modify ack deadline for all messages upon receiving them.
Problem description
When streaming, StreamingPullResponse can be buffered, either by gRPC itself or by proxy/firewall.
There might be significant delay between the server sending a response and the client library reading it.
For this reason, by the time client library decides it should send a modify ack deadline ("modack"), it might be too late. The message might have already expired and resent.
Experimenting with Java client, we have seen many messages delivered 3-5 times because of this problem.
What server-side is doing
To fix this, pubsub team is implementing the "receipt" feature.
The server will not start the expiration timer until it receives a receipt from the client [1]. A receipt here is similar to an ack. A receipt notifies the server that the client has seen the message, while an ack notifies that the client has finished with it.
This should make the client and server start expiration timer closer to each other.
[1] There's actually a limit on how long the server is willing to wait, so that a message isn't lost forever if the client doesn't send a receipt.
What the client needs to do
The client should be modified to send these receipts.
For compatibility, these "receipts" are just modack requests. The client should modack messages after seeing it, in addition to what it is currently doing. These modacks can be batched for performance.
Few more details for the curious
These modacks/receipts aren't strictly necessary for avoiding duplication, but it will help pubsub server expire messages on time. Without them, messages might expire too late.
Each StreamingPullResponse can contain multiple messages. In an implementation about to roll out, a modack for any message in the response will serve as receipts for all messages in that response. Still, client libs are encouraged to actually modack all messages since this behavior might change in the future.