Capacity inconsistency for byte slices from strings

[dlsniper]

What version of Go are you using (go version)?

• the Go Playground
• go version go1.7.4 linux/amd64
• go version go1.8beta2 linux/amd64

What operating system and processor architecture are you using (go env)?

• the Go Playground

• go 1.7:

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/florin/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build121149425=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

• go 1.8 beta 2

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/florin/go"
GORACE=""
GOROOT="/usr/local/go1.8"
GOTOOLDIR="/usr/local/go1.8/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build149699806=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?

https://play.golang.org/p/mvU3emq7Yj

What did you expect to see?

// from struct
len:	4
cap:	4
// from var
len:	4
cap:	4

What did you see instead?

• In playground:

// from struct
len:	4
cap:	32
// from var
len:	4
cap:	32

• go 1.7.4:

// from struct
len:    4
cap:    8
// from var
len:    4
cap:    32

• go 1.8 beta 2:

// from struct
len:    4
cap:    8
// from var
len:    4
cap:    32

I would expect that no extra memory is allocated beyond the needs of the string but the capacity seems different in both cases. What's even more bizarre is that the values are different from between locations where this is allocated and it seems that the playground is also different from my linux machine.

[minux]

Why is this a problem? The runtime could preallocate a few bytes in case you want to grow the byte slice later.

[dlsniper]

For me it's unexpected. Worse, in a scenario where you have a high-traffic server or an app with hundreds of thousands of objects like this, those "few bytes" might make some differences (for example here, those few bytes are about 400 https://play.golang.org/p/oDTX87BviO ).

I understand how the runtime might preallocate for that scenario and I'm familiar with the append mechanics however, I'm expecting that what my code declares to be respected. The compiler could hint if there's an append later on or not if that's the concern we have.

[dominikh]

Related: #14232

[minux]

The waste of memory is not too bad when the string length is larger. https://play.golang.org/p/oq91-ho64B Compiling hinting won't help much because unless the compiler could do precise full program analysis, it can't see (at the allocation site) whether the program is going to append to the []byte or not (except for toy programs). The only bug I can see is that somehow Go 1.8 (tip) always starts with the 32-byte size class for string to []byte conversions. Otherwise there is no bug. See https://play.golang.org/p/oq91-ho64B. The runtime allocate memory using about 67 size classes, and the spacing in the lower end is 16 byte.

[minux]

OK, I see the reason. When the string is small, and it's not escaping, the []byte will be stack allocated and for simplicity, the stack buffer is always 32 byte. If the result escapes, then it will be heap allocated, and will use the correct size class. https://play.golang.org/p/I_OV63vNgP It's all working as intended. The take-away is: the compiler and runtime is getting smarter in each release. Do not infer anything from memory usage of toy examples. Measure real application code instead.

[randall77]

See #14235 and https://go-review.googlesource.com/#/c/22424/ for more context on why this behaves as it does.

[dlsniper]

Thank you all for the quick reply and pointers on this