How to calculate buffer size?

[szerwi]

I have a problem with understanding buffer size.
I set batch size to 1 and buffer size to 65535, which is the maximum number for uint16_t. I also set flush interval to 65535.
My code is sending data to database every 10 seconds. Each point contains 3 fields.
Once every 10 minutes the point is bigger, it contains 27 fields.

I understand it in this way: if buffer size is set to 2, it can hold 2 points (regardless the fields number).
In this situation I should be able to store 182 hours of data. Unfortunately, the buffer gets full after around 40 minutes.

How to properly calculate buffer size? Is there any way to store a few hours of data in case there is no Internet connection?

[vlastahajek]

@szerwi, thanks for posting this issue.

Both batch size and buffer size mean a number of points.

To calculate total buffer size, you must consider available RAM (~250k for ESP32), and an average point length.
Point is internally stored in the InfluxDB line protocol format, which means string.
If a single point is small, with some reasonable tags, it can be 50 chars long.
In such a case there can be only 5_000 such points.

The write buffer is actually limited to 255 batches and each batch can hold a maximum of 255 lines. So, if your batch size is 1, written 6x per minute, the time to fill the buffer is ~42min, so it confirms your observation.

Getting back to the calculation, 5_000 points, with a maximum of 255 batches, means ~19 for batch size.
For these sizes, you have to set batch size to 19 and buffer size to 5_000.

Data type uint16_t for the batch size doesn't make sense, I will fix it.

[szerwi]

@vlastahajek, thank you for explanation.
Unfortunately, when I set batch size to 19 and buffer size to 5000, it still gets full after 42 minutes.
May it be because I call client.flushBuffer() every 10 seconds, because I need readings in real time in database?

Anyway, I think I need to save the reading in SPIFFS instead of RAM. There will be more space & data will be saved even if I restart the ESP.

[szerwi]

I've tried to set batch size to 75 and buffer size to 255, but it still gets full after 42 minutes. Is there any way to make it bigger?
Or maybe is there any way to add only specific points to buffer? I am also thinking about skipping 10sec readings and add only 10min reading to buffer, but I cannot find any function in the library that will send data to database, but it won't add data to buffer in case sending failed.

[vlastahajek]

@szerwi, the problem is in conversion from uint16_t to uint8_t, when calculating writeBuffer=bufferSize/batchSize If the result is higher than 255, writeBuffer get the size of lower 8bits. To achieve maximum writeBuffer, you have to set buffer size and batch size to produce a division result maximum of 255. I previously wrongly advised you batchSize=19 and buffer size=5000. This gets a write buffer of 7.
To get maximum, either set batchsize 19 and buffer size 4845, or batchsize 20 and buffer size 5100.

[szerwi]

Thank you for the explanation. Anyway, I have made my own buffer in which I store only 10min readings, skipping 10sec readings.