Skip to content

Unpacker.read_bytes behaves unpredictably with a size of -1 / support for reading all outstanding bytes? #259

Closed
@RR2DO2

Description

@RR2DO2

Some python read() methods take -1 as a parameter to return all remaining bytes in a stream/buffer. While msgpack's Unpacker read_bytes isn't explicitly documented as supporting this, it used to accept -1.

This lead to various corruptions though; the c implementation would not return all bytes. At least with 0.4.8 it now raises an error stating -1 is invalid (side-effect of calling the CPython related methods to obtain a size_t).

The fallback method however still accepts -1, but returns bad data in this case.

Our current workaround is to use read_bytes(sys.maxint) to obtain any other bytes from the reader passed into Unpacker, however this only works in the c implementation. The fallback returns OutOfData since there aren't that many bytes available.

There are a few things that likely should be done. read_bytes should behave consistently between the c and fallback implementations. Passing -1 into any implementation should not corrupt any internal state or return invalid data. Finally, having a dedicated Unpacker.read() that would return all data in the reader still outstanding would be invaluable.

Not sure if help would appreciated in writing a patch for this? Also not sure why there is no .read() method already - instead of the less standard read_bytes?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions