Support for numpy in Javascript through another serilization like MessagePack

[VigneshVSV]

Numpy arrays are serialized as fast as python's pickle by MessagePack format with msgspec.msgpack . However, to unpack it on the javascript side, a suitable deserialization is necessary. MessagePack is also safer than pickle.

The expected server side code is as follows:

class MyThing(Thing)
    
    numpy_event = Event(doc="event with a large numpy array as memory view") 
    
    @action()
    def numpy_gen(self) -> memoryview:
         "action returning a binary payload of numpy array"
         return memoryview(self._large_numpy_array)
       
    def event_pub(self) -> None:
         while True:
             self.numpy_event.push(memoryview(self._large_numpy_array)
             time.sleep(1)

if __name__ == '___main__':
     MyThing(id='my-thing', serializer='msgpack').run_with_http_server()

However, a javascript library which can read this memoryview data is not known. The other option compared to a memoryview is numpy's builtin save() method which can dump the binary data onto a io.BytesIO() object. See here

Some speed test results (not sure if its upto date):

Serializer	Spectrum 1e3	Spectrum 1e4 Double Entries	Spectrum 1e5 Entries	Spectrum 1e6 Entries
JSONSerializer	0.05121	0.61983	6.29486	65.279
PickleSerializer	0.01586	0.01927	0.26054	8.31536
MsgpackSerializer	0.01237	0.09472	1.13373	15.2071

Rough steps:

• research existing serializers in Javascript
• Implement a C/C++ based numpy to javascript parser
• Show code examples to be used in node-wot. In node-wot, you can always access the binary buffer and pass it to a custom deserializer. Better may be to register a content-type with their existing content type management.

This is a javascript related issue, not python. This can also be released as a separate package or add-on in npm.