Is flate / gzip really as fast as it should be?

[splitice]

Using this library in docker/moby I noticed (on ARMv7 at-least) gzip decompression seems to occupy the largest share (33%) of work in a docker pull. I found this a bit surprising, especially what with all that filesystem work to install the extracted files in their layers.

This got me looking through the algorithm and I noticed that its reading byte by byte from the bufio source. While it is bufio surely this is not the fastest way to decode a hufman table?

[klauspost]

First of all, without having the flamegraph, I am pretty sure you are misreading it. The read part is only a tiny fraction of the 33%:

and

The huffmanBufioReader function is doing the actual decoding.

The standard library guarantees that if you provide an io.ByteReader it will not over-read beyond the input. That means you can't just read, say 1K ahead. To remain compatible we "respect" that.

But even if we didn't, it wouldn't make sense to when we are given a bufio.Reader, since you would just introduce a memcopy to an internal buffer, which would serve pretty much the same purpose as the bufio.Reader.

Compared to the stdlib we generate the readers for each stdlib implementation of io.ByteReader. This allows the compiler to much more efficiently read the bufio.Reader - since it is hitting the implementation and not the interface.

The reason that people are moving to zstd is mostly because of decompression speed. There isn't too much more that can be squeezed from inflate.