Serializing Class Hierarchy

[ovz]

What is the recommended way to serialize the state of a derived class along with all the states of it's parents? Looks like MSGPACK_DEFINE doesn't directly support invoking the base class. Shall a macro with name like MSGPACK_DEFINE_BASE allowing to specify a list of base classes that has to be serialized too?

[redboltz]

Unfortunately, msgpack-c doesn't support packing and unpacking base classes directly.
So far, we need to write functions as follows:

#include <msgpack.hpp>
#include <iostream>
#include <string>

struct base {
    std::string m_base1;
    std::string m_base2;
    // for itself
    MSGPACK_DEFINE(m_base1, m_base2);
};

struct mid1:virtual base {
    int m_mid1;
    // for itself
    MSGPACK_DEFINE(m_mid1);
};

struct mid2:virtual base {
    int m_mid2;
    // for itself
    MSGPACK_DEFINE(m_mid2);
};

struct derived:mid1, mid2 {
    int m_derived;
    // for itself
    // It is never used in this example
    MSGPACK_DEFINE(m_derived);
};

// for derived class member and base classes
// You need to define the following functions to
// pack and unpack 'derived'
// ----------------------------------------------------------------------------------
inline derived& operator>> (msgpack::object o, derived& v)
{
    if(o.type != msgpack::type::ARRAY) { throw msgpack::type_error(); }
    // 4 means base, mid1, mid2, and m_derived
    if(o.via.array.size != 4) { throw msgpack::type_error(); }
    o.via.array.ptr[0].convert(&static_cast<base&>(v));
    o.via.array.ptr[1].convert(&static_cast<mid1&>(v));
    o.via.array.ptr[2].convert(&static_cast<mid2&>(v));
    o.via.array.ptr[3].convert(&v.m_derived);
    return v;
}

template <typename Stream>
inline msgpack::packer<Stream>& operator<< (msgpack::packer<Stream>& o, derived const& v)
{
    // 4 means base, mid1, mid2, and m_derived
    o.pack_array(4);
    o.pack(static_cast<base const&>(v));
    o.pack(static_cast<mid1 const&>(v));
    o.pack(static_cast<mid2 const&>(v));
    o.pack(v.m_derived);
    return o;
}
// ----------------------------------------------------------------------------------


int main() {
    msgpack::sbuffer sbuf;
    derived d;
    d.m_base1 = "m_base1";
    d.m_base2 = "m_base2";
    d.m_mid1 = 1;
    d.m_mid2 = 2;
    d.m_derived = 3;

    msgpack::pack(sbuf, d);

    msgpack::unpacker unp;
    unp.reserve_buffer(sbuf.size());
    memcpy(unp.buffer(), sbuf.data(), sbuf.size());
    unp.buffer_consumed(sbuf.size());

    msgpack::unpacked result;
    while(unp.next(&result)) {
        msgpack::object obj = result.get();
        derived converted;
        obj.convert(&converted);

        std::cout << converted.m_base1 << std::endl;
        std::cout << converted.m_base2 << std::endl;
        std::cout << converted.m_mid1 << std::endl;
        std::cout << converted.m_mid2 << std::endl;
        std::cout << converted.m_derived << std::endl;
    }

}

MSGPACK_DEFINE_BASE is an interesting idea. I think that combine member variables and base classes to one array is required. How about the following syntax?

MSGPACK_DEFINE( \
    MSGPACK_BASE(base), \
    MSGPACK_BASE(mid1), \
    MSGPACK_BASE(mid2), \
    m_derived\
);

[ovz]

Would be great if one doesn't have to maintain the whole hierarchy in either operator overloads or MSGPACK_DEFINE. I agree that for virtual base class it is necessary. If we didn't have a virtual base, mentioning just immediate bases in MSGPACK_DEFINE should be enough, correct?

MSGPACK_DEFINE(
MSGPACK_BASE(mid1),
MSGPACK_BASE(mid2),
m_derived
);

Of course mid1 and mid2 do their own MSGPACK_BASE as necessary.

[redboltz]

Yes, if we didn't have a virtual base class.
If we have a virtual base class, the situation is complecated.

struct base {
    std::string m_base1;
    std::string m_base2;
    // for itself
    MSGPACK_DEFINE(m_base1, m_base2);
};

struct mid1:virtual base {
    int m_mid1;
    // for itself
    MSGPACK_DEFINE(MSGPACK_BASE(base), m_mid1);
};

struct mid2:virtual base {
    int m_mid2;
    // for itself
    MSGPACK_DEFINE(MSGPACK_BASE(base), m_mid2);
};

struct derived:mid1, mid2 {
    int m_derived;
    // for itself
    // It is never used in this example
    MSGPACK_DEFINE(MSGPACK_BASE(mid1), MSGPACK_BASE(mid2), m_derived);
};

When we pack/unpack an instance of mid1 or mid2, 'base' should be packed/unpacked automatically. In this case, there is no difference between virtual inheritance or not.
When we pack/unpack an instance of derived, 'base' should be packed/unpacked exactly once.

BTW, I'm taling about the syntax. How to implement them is a different problem.

[lzuwei]

I have an issue Serializing Class Hierarchies

I tried running the codes that were written here but I was unable to deserialize the base classes and mid classes properly...

I am running on
Mac OSX yosemite
clang Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
Target: x86_64-apple-darwin14.1.0
Thread model: posix

My output for the above example:

213315584
197156352
3

I have written another class hierarchy structure to test

typedef uint32_t RequestId;

struct Request {
    int type;
    RequestId id;
    MSGPACK_DEFINE(type, id);
};

/*
 * Search Request is a client request to search an image
 */
struct SearchRequest : public Request {
public:
    std::string image_data;
    MSGPACK_DEFINE(image_data);
};

inline SearchRequest& operator >> (msgpack::object o, SearchRequest& r) {
    if(o.type != msgpack::type::ARRAY) {
        throw msgpack::type_error();
    }
    if(o.via.array.size != 2) {
        throw msgpack::type_error();
    }
    o.via.array.ptr[0].convert(&static_cast<Request&>(r));
    o.via.array.ptr[1].convert(&r);
    return r;
}

template<typename Stream>
inline msgpack::packer<Stream>& operator<< (msgpack::packer<Stream>& o, const SearchRequest& r) {
    o.pack_array(2);
    o.pack(static_cast<const Request&>(r));
    o.pack(r);
    return o;
}

std::ostream& operator<< (std::ostream& o, const SearchRequest& r) {
    o << "Type: " << r.type << ", Id: " << r.id << ", Data: " << r.image_data;
    return o;
}

int main() {
    msgpack::sbuffer sbuf;
    SearchRequest r;
    r.type = 100;
    r.id = 15;
    r.image_data = "example";
    msgpack::pack(sbuf, r);

    msgpack::unpacker unp;
    unp.reserve_buffer(sbuf.size());
    memcpy(unp.buffer(), sbuf.data(), sbuf.size());
    unp.buffer_consumed(sbuf.size());

    msgpack::unpacked result;
    SearchRequest search_request;
    while(unp.next(&result)) {
        msgpack::object obj = result.get();
        obj.convert(&search_request);
    }

    std::cout << search_request << std::endl;
    return 0;
}

the output for the my code:
Type: 142389248, Id: 1, Data: example

Any help is greatly appreciated! The value of Type keeps changing in every run. the Derived class member is correctly deserialized.

[redboltz]

Hi @lzuwei, I updated your code. Could you try it using msgpack 1.1.0?

#include <msgpack.hpp>
#include <iostream>

typedef uint32_t RequestId;

struct Request {
    int type;
    RequestId id;
    MSGPACK_DEFINE(type, id);
};

/*
 * Search Request is a client request to search an image
 */
struct SearchRequest : public Request {
public:
    // In the case using non-intrusive approach,
    // you don't need to write MSGPACK_DEFINE(image_data)
    std::string image_data;
};

// This is the new adaptation implementation
// See:
//   https://github.com/msgpack/msgpack-c/wiki
//   https://github.com/msgpack/msgpack-c/wiki/v1_1_cpp_adaptor#non-intrusive-approach
namespace msgpack {
MSGPACK_API_VERSION_NAMESPACE(MSGPACK_DEFAULT_API_NS) {
namespace adaptor {

template<>
struct convert<SearchRequest> {
    // Return msgpack::object const&, instead of r.
    msgpack::object const& operator()(msgpack::object const& o, SearchRequest& r) const {
        if(o.type != msgpack::type::ARRAY) {
            throw msgpack::type_error();
        }
        if(o.via.array.size != 2) {
            throw msgpack::type_error();
        }
        // The reference version is recommended
        o.via.array.ptr[0].convert(static_cast<Request&>(r));
        // convert image_data here.
        o.via.array.ptr[1].convert(r.image_data);
        return o;
    }
};

template<>
struct pack<SearchRequest> {
    template <typename Stream>
    msgpack::packer<Stream>& operator()(msgpack::packer<Stream>& o, SearchRequest const& r) const {
        o.pack_array(2);
        o.pack(static_cast<const Request&>(r));
        // pack image_data here.
        o.pack(r.image_data);
        return o;
    }
};

} // namespace adaptor
} // MSGPACK_API_VERSION_NAMESPACE(MSGPACK_DEFAULT_API_NS)
} // namespace msgpack

std::ostream& operator<< (std::ostream& o, const SearchRequest& r) {
    o << "Type: " << r.type << ", Id: " << r.id << ", Data: " << r.image_data;
    return o;
}

int main() {
    msgpack::sbuffer sbuf;
    SearchRequest r;
    r.type = 100;
    r.id = 15;
    r.image_data = "example";
    msgpack::pack(sbuf, r);

    msgpack::unpacker unp;
    unp.reserve_buffer(sbuf.size());
    memcpy(unp.buffer(), sbuf.data(), sbuf.size());
    unp.buffer_consumed(sbuf.size());

    msgpack::unpacked result;
    SearchRequest search_request;
    while(unp.next(&result)) {
        msgpack::object obj = result.get();
        obj.convert(&search_request);
    }

    std::cout << search_request << std::endl;
    return 0;
}

I got the following output:

Type: 100, Id: 15, Data: example

[lzuwei]

Hi @redboltz,

I will update my msgpack versions and try out the following that you have suggested.
Thank you for the prompt reply!

Best Regards,
Zuwei

[lzuwei]

Hi @redboltz,

I have updated the codes and tested the modified codes you have provided, they work perfectly.
Thank you for all the comments as well!

Regards,
Zuwei

[redboltz]

Hi @lzuwei. Sounds good :)

[redboltz]

@ovz, I sent the PR to support base class serialization, just before. It is #265.

Here is an example:

struct top {
    int t;
    MSGPACK_DEFINE(t);
};

struct mid1 : top {
    int m1;
    MSGPACK_DEFINE(MSGPACK_BASE(top), m1);
};

struct mid2 : top {
    int m2;
    MSGPACK_DEFINE(m2, MSGPACK_BASE(top));
};

struct bottom : mid1, mid2 {
    int b;
    MSGPACK_DEFINE(MSGPACK_BASE(mid1), MSGPACK_BASE(mid2), b);
};

TEST(object_with_zone, user_defined_non_virtual)
{
    bottom b;
    b.b = 1;
    b.m1 = 2;
    b.m2 = 3;
    b.mid1::t = 4;
    b.mid2::t = 5;

    msgpack::zone z;
    msgpack::object obj(b, z);
    bottom br = obj.as<bottom>();
    EXPECT_EQ(b.b, br.b);
    EXPECT_EQ(b.m1, br.m1);
    EXPECT_EQ(b.m2, br.m2);
    EXPECT_EQ(b.mid1::t, br.mid1::t);
    EXPECT_EQ(b.mid2::t, br.mid2::t);
}

[redboltz]

This issue has been fixed by #265.