Repository navigation
compress/flate: compression degradation in streaming mode with intermediate Flush() calls #81542
Description
Go version
go1.27.1 linux/amd64
Output of go env in your module/workspace:
AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN='/home/dop/projects/test/bin'
GOCACHE='/home/dop/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/dop/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3155541213=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOOS='linux'
GOPACKAGESDRIVER=''
GOROOT='/usr/local/go1.27.1'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/dop/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go1.27.1/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.27.1'
GOWORK=''
PKG_CONFIG='pkg-config'What did you do?
I'm using flate to compress individial messages in a streaming protocol. After switching from 1.26 to 1.27 I've noticed that if two identical messages are sent with another short message between, the second copy of the message does not use a reference to the first copy and is sent in full instead.
In 1.26 this behaviour is only observed on compression level 1, in 1.27 -- on levels 1-6 inclusive.
Reproducing code: https://go.dev/play/p/9V1TyhjEzS6
Note that if there is no intermediate message or if there is no Flush() call after that intermediate message is sent, it works as expected.
This may be an acceptable behaviour in general, but in my case this is a very typical traffic pattern. With the new version I'll have to raise the compression level to 7 which incurs a significant CPU and memory penalty.
What did you see happen?
2009/11/10 23:00:00 Wrote 227 bytes
2009/11/10 23:00:00 Wrote 17 bytes
2009/11/10 23:00:00 Wrote 227 bytes
2009/11/10 23:00:00 too many bytes
What did you expect to see?
2009/11/10 23:00:00 Wrote 176 bytes
2009/11/10 23:00:00 Wrote 13 bytes
2009/11/10 23:00:00 Wrote 9 bytes
Activity
gabyhelp commentedon Sep 15, 2026
Related Issues
Related Discussions
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)
seankhliao commentedon Sep 15, 2026
cc @klauspost
klauspost commentedon Sep 16, 2026
Seems like you are hitting the "too small to bother compressing" limit (128B), so blocks are just huffman encoded without trying to generate matches.
Arguably it is a bit aggressive on default level - since it doesn't discriminate between flush and close. I will see if I can make it softer.
klauspost commentedon Sep 18, 2026
Got a solution that I am happy with. Will do some cleanup and send next week.
dop251 commentedon Oct 22, 2026
Any update (sorry for asking)? Even if it's fixed in https://github.com/klauspost/compress, I could just switch to it.
klauspost commentedon Oct 23, 2026
Apologies. It got lost in my mind. Sent in #82452
gopherbot commentedon Oct 23, 2026
Change https://go.dev/cl/853285 mentions this issue:
compress/flate: keep match history across small flushes