[QUESTION] EventBatch doesn't clear after being sent

[ArmandoLacerda]

Describe the bug
When sending multiple messages through the eventBatch helper, it doesn't clear out after the batch is submitted. I have a 5MB payload to send to the hub. I break the payload in many smaller messages and loop through them, adding them up to the eventBatch until it gets close to 1MB. Then I invoke producerClient.SendAsync() to submit the batch and resume the loop for the next collection of messages. eventBatch doesn't zero out and the loop tries to resubmit the same batch. The eventBatch.SizeInBytes still shows the already submitted payload.

Expected behavior
After invoking producerClient.SendAsync(eventBatch) I expect to get a clean batch to start filling up with new messages.

Actual behavior (include Exception or Stack Trace)
The eventBatch does not clear out. It is of no use in a loop to send a payload larger than 1MB.

To Reproduce
A slightly modified version of the sample official sample:

`
public static void trySendingMultipleBatches()
{
{
// Create a producer client that you can use to send events to an event hub
await using (var producerClient = new EventHubProducerClient(connectionString, eventHubName))
{
// Create a batch of events
using EventDataBatch eventBatch = await producerClient.CreateBatchAsync();

        // Try 3 batches 
        for (int i = 0; i < 3; i++)
        {

            while (eventBatch.SizeInBytes < 800000)
            {
                // Add events to the batch. An event is a represented by a collection of bytes and metadata. 
                eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("First event")));
                eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Second event")));
                eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Third event")));
            }
            // Use the producer client to send the batch of events to the event hub
            await producerClient.SendAsync(eventBatch);
            Console.WriteLine("A batch of about 1MB has been published.");
        }
    }
}

}

`

Environment:

• Azure.Messaging.EventHubs 5.2.0

• Hosting platform or OS and .NET runtime version (dotnet --info)
  .NET Core SDK (reflecting any global.json):
  Version: 3.1.300
  Commit: b2475c1295

Runtime Environment:
OS Name: Windows
OS Version: 10.0.18363
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\3.1.300\

Host (useful for support):
Version: 3.1.4
Commit: 0c2e69caa6

.NET Core SDKs installed:
3.1.300 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

• Visual Studio Code
  Version: 1.50.1 (user setup)
  Commit: d2e414d9e4239a252d1ab117bd7067f125afd80a
  Date: 2020-10-13T15:06:15.712Z
  Electron: 9.2.1
  Chrome: 83.0.4103.122
  Node.js: 12.14.1
  V8: 8.3.110.13-electron.0
  OS: Windows_NT x64 10.0.18363

[jsquire]

Hi @ArmandoLacerda, thank you for reaching out and your feedback. The behavior that you're describing is by-design; EventDataBatch instances are intended to be single use for most scenarios, with the notable exception being when you want to resend a set of events.

For sending multiple batches of distinct events, a new batch should be created after each publish operation. This sample illustrates one approach for doing so.

In your snippet above, the pattern that you're using isn't ideal for taking advantage of the batch. Instead of:

while (eventBatch.SizeInBytes < 800000)
{
    // ...
}

I'd advise something similar to:

var batchOptions = new CreateBatchOptions
{
    MaximumSizeInBytes = 800000
};

for (var count = 0; count < 3; ++count)
{
    using var batch = await producerClient.CreateBatchAsync(batchOptions);

    // --
    // Add events until batch.TryAdd  returns false.
    //-- 
      
    await producerClient.SendAsync(eventBatch);
}

That said, I'm not sure of your intent to constraining the batch to a specific size. By default CreateBatchAsync will query the Event Hubs service to understand the size limit for the associated Event Hub. If you're using a n Event Hubs instance with a 256k limit, the resulting batch will be limited to 256k. Likewise, if you are using an instance that allows 1mb or 5mb, the resulting batch will allow that size. By not explicitly specifying the size, your code will automatically adapt to the limit without you needing to track the specifics.

[ArmandoLacerda]

Thanks for your reply, @jsquire . Please help me to understand this in regards to eventBatch dynamically adjusting the batch size.

If I create a 5MB batch in send it, will SendAsync + eventBatch break the full payload in multiple submissions and get the full payload pushed to the hub? Or will it discard the tail end of the payload that didn't fit in the maxSize?

I didn't get from the documentation whether it would go one way or another so I coded the size control myself.

[jsquire]

If I create a 5MB batch in send it, will SendAsync + eventBatch break the full payload in multiple submissions and get the full payload pushed to the hub? Or will it discard the tail end of the payload that didn't fit in the maxSize?

The EventDataBatch is scoped to a single operation. It won't allow you to add more events to it than can be sent in one SendAsync call. The EventDataBatch knows the maximum size for a single send operation allowed by the service. It will not allow that size to be exceeded by returning false when you call its TryAdd method. So long as TryAdd returns true, you know that your event was accepted into the batch and will fit into a single send call.

The caller has responsibility for honoring the TryAdd return value. If that return is false, your event was not accepted into the batch because the size would be exceeded. That's the signal to send the current batch and create a new one if you have events left to publish. If you continue to TryAdd when the batch is full, the batch will not accept it and will not track it.