DeepCopyArrray vs. ShallowCopyArrayByReference

[PaulChristopher]

Bonjour Benoît and many thanks for this well-done and mature library! I am also very impressed by the web page, the assistant and trouble shooter, wow! That is really very user-friendly, thank you!

I am writing to you, because I have trouble understanding the need for the copyArray function: It seems really to create a clone / a deep-copy of an existing C-array and creates a new JsonArray in a JsonDocument. But this also means: The array's data needs to reside in memory twice. No problem for small arrays but a problem for big arrays e.g. with values of sensor measurements.

Why is it not possible to just pass in a reference/pointer to the JsonDocument and have somekind of new type JsonRefArray as a shallow copy with its own custom string_writer?

I use the ArduinoJson library in a small project on an ESP32. The ESP32 ADC samples values of a pressure sensor at 20kHz. If pressure is applied, I need to record those values and if pressure is gone, I send those values via websockets to a webbased GUI, which draws a time-based chart of the pressure applied using µPlot.js. This should help the user to analyze and calibrate my device with the pressure sensor.

At the moment, I directly save the gathered values into a JsonDocument with JsonArray.add. But you pointed out yourself: We are not supposed to do that: overhead and execution speed, see https://arduinojson.org/v7/faq/why-should-i-create-a-separate-config-object/ . And execution speed is in my case top priority so as to be able to keep up with the sampling rate.

I would not mind, if serialization is a little bit slower with a custom string_writer of the JsonRefArray. But I cannot afford 1) slow performance of an array (JsonArray.add vs. native array) 2) duplication of an array just for serialization due to memory limits.

IMHO, you have tackled the problems of limited memory of Arduino-like devices on the deserialization-side very well with the input filters. I could no see offered solutions on the serialization side which avoids internal duplication of large arrays.

Or to put it short: I am looking for a way to hook in a custom writer which is memory efficient and does not create a second JsonArray. I.e. If I call serializeJson(mydoc, myjsonstring) and mydoc["pressure_values"] is of type JsonRefArray, I would like ArduinoJson to call my regsitered custom string_writer function, which returns the string representation to ArduinoJson of my 1D C-array of floats, maybe even incrementally to avoid large temporary strings during transformation.

Can I do that with the current library? What would be the steps to do that?

Thanks for the well done library again and its intuitive syntax. It is really fun working with it!

Paul

[bblanchon]

Hi Paul,

many thanks for this well-done and mature library!

"Mature", yes, but definitely not stale!
I haven't published any release recently, but I'm still actively working on #2198.
Development is very slow because I'm overwhelmed with all my other jobs, but it should only be temporary.

I have trouble understanding the need for the copyArray function

copyArray() is a convenience function to copy data from/to a C-style array.

The array's data needs to reside in memory twice

Indeed, "copy" is literally in the function's name.

I directly save the gathered values into a JsonDocument with JsonArray.add. But you pointed out yourself: We are not supposed to do that

The page says "should", not "must".
Sometimes it's okay.

Why is it not possible to just pass in a reference/pointer

ArduinoJson 6 had shallowCopy() for a short period. It also allowed referencing strings rather than copying the characters.

I decided to remove all these footguns in ArduinoJson 7 after realizing they were the cause of most issues people were having with the library.
Still, I don't remember anyone complaining about the removal of these optimizations.

In ArduinoJson 7, I'm trying to reduce memory usage without pointing to external variables:

• Long strings are deduplicated
• Short strings are stored directly in the node's payload
• Arrays consume less RAM since 7.2

I'll continue to reduce the overhead, but this is a long-term effort.

Can I do that with the current library?

No, and I have no plan to add such a feature, at least not in the foreseeable future.

What would be the steps to do that?

If I were you, I would reconsider using JSON to transmit the pressure data.
JSON is great for sending objects, but not so great for large arrays, images, or streams.
Instead, I would simply send the live values (possibly as strings) to the WebSocket.

Best regards,
Benoit