encoding/gob: TestFuzz can use a lot of memory

[mikioh]

What steps will reproduce the problem?
1. run go test -v std
2.
3.

What is the expected output? What do you see instead?
all tests pass

Please use labels and text to provide additional information.
--- PASS: TestDebugStruct (0.00 seconds)
=== RUN TestFuzz
runtime: out of memory: cannot allocate 268500992-byte block (269549568 in use)
throw: out of memory

goroutine 23 [running]:
encoding/gob.(*Decoder).readMessage(0x98298140, 0x10000766, 0x98262200)
    /home/mikioh/go/src/pkg/encoding/gob/decoder.go:92 +0x5a
encoding/gob.(*Decoder).recvMessage(0x98298140, 0x0, 0x1)
    /home/mikioh/go/src/pkg/encoding/gob/decoder.go:84 +0xaa
encoding/gob.(*Decoder).decodeTypeSequence(0x98298140, 0x98267d00, 0x160, 0x0)
    /home/mikioh/go/src/pkg/encoding/gob/decoder.go:142 +0x57
encoding/gob.(*Decoder).DecodeValue(0x98298140, 0x80f98f0, 0x98267dc0, 0x160, 0x0, ...)
    /home/mikioh/go/src/pkg/encoding/gob/decoder.go:206 +0xe4
encoding/gob.(*Decoder).Decode(0x98298140, 0x80f98e8, 0x98267dc0, 0x98267dc0,
0x98296de0, ...)
    /home/mikioh/go/src/pkg/encoding/gob/decoder.go:185 +0x168
encoding/gob.encFuzzDec(0x98261cc8, 0x80f9640, 0x98278bf8, 0x982373e0, 0x982021a0, ...)
    /home/mikioh/go/src/pkg/encoding/gob/codec_test.go:1429 +0x287
encoding/gob.testFuzz(0x98263440, 0x383acc68, 0x1278f820, 0x64, 0xb8244f68, ...)
    /home/mikioh/go/src/pkg/encoding/gob/codec_test.go:1468 +0x1a4
encoding/gob.TestFuzz(0x98263440, 0xe)
    /home/mikioh/go/src/pkg/encoding/gob/codec_test.go:1452 +0x2fa
testing.tRunner(0x98263440, 0x82113d0, 0x0)
    /tmp/bindist445861990/go/src/pkg/testing/testing.go:271 +0x6e
created by testing.RunTests
    /tmp/bindist445861990/go/src/pkg/testing/testing.go:346 +0x687

goroutine 1 [chan receive]:
testing.RunTests(0x8048c00, 0x82112e0, 0x49, 0x49, 0x8202101, ...)
    /tmp/bindist445861990/go/src/pkg/testing/testing.go:347 +0x6a7
testing.Main(0x8048c00, 0x82112e0, 0x49, 0x49, 0x820dd58, ...)
    /tmp/bindist445861990/go/src/pkg/testing/testing.go:282 +0x46
main.main()
    /tmp/go-build055329031/encoding/gob/_test/_testmain.go:191 +0x4e

goroutine 2 [syscall]:
created by runtime.main
    /tmp/bindist445861990/go/src/pkg/runtime/proc.c:219
FAIL    encoding/gob    0.012s

[mikioh]

Comment 1:

% ulimit -a
socket buffer size       (bytes, -b) unlimited
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) 2097152
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 7207
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 65536
cpu time               (seconds, -t) unlimited
max user processes              (-u) 3603
virtual memory          (kbytes, -v) unlimited
swap size               (kbytes, -w) unlimited

[mikioh]

Comment 2:

% go version
go version weekly.2012-03-04 +f4470a54e6db

[rsc]

Comment 3:

There is a separate bug about not being able to use
more than 512 MB on 32-bit.  Do you believe that
the fuzz input should not be causing gob to try to
allocate that much memory?
If the allocation is appropriate for the input, then this
is not a bug in gob but a dup of the runtime bug, which
Remy has a CL pending for.
Russ

[mikioh]

Comment 4:

I agree with you, that supposed root cause would be runtime.
I just filed for the record of first binary distribution trial. ;)
By the way, unfortunately CL 5725045 doesn't fix the issue
for 32-bit memory allocator.
http://golang.org/cl/5725045/
% go test -v encoding/gob
(snip)
=== RUN TestFuzz
runtime: out of memory: cannot allocate 806223872-byte block (1048576 in use)
throw: out of memory
goroutine 23 [running]:
encoding/gob.(*Decoder).readMessage(0x9828a820, 0x300d0369, 0x98269b40)
    /home/mikioh/go/src/pkg/encoding/gob/decoder.go:92 +0x5a
encoding/gob.(*Decoder).recvMessage(0x9828a820, 0x0, 0x1)
    /home/mikioh/go/src/pkg/encoding/gob/decoder.go:84 +0xaa
encoding/gob.(*Decoder).decodeTypeSequence(0x9828a820, 0x98289400, 0x160, 0x0)
    /home/mikioh/go/src/pkg/encoding/gob/decoder.go:142 +0x57
encoding/gob.(*Decoder).DecodeValue(0x9828a820, 0x80f9c70, 0x98289410, 0x160, 0x0, ...)
    /home/mikioh/go/src/pkg/encoding/gob/decoder.go:206 +0xe4
encoding/gob.(*Decoder).Decode(0x9828a820, 0x80f9c68, 0x98289410, 0x98289410,
0x9828d4e0, ...)
    /home/mikioh/go/src/pkg/encoding/gob/decoder.go:185 +0x168
encoding/gob.encFuzzDec(0x9827bcc8, 0x80f9ba0, 0x9827bc70, 0x982373e0, 0x982381a0, ...)
    /home/mikioh/go/src/pkg/encoding/gob/codec_test.go:1429 +0x287
encoding/gob.testFuzz(0x98281d00, 0xf9d05b78, 0x1278fce3, 0x64, 0xb8245f68, ...)
    /home/mikioh/go/src/pkg/encoding/gob/codec_test.go:1468 +0x1a4
encoding/gob.TestFuzz(0x98281d00, 0xe)
    /home/mikioh/go/src/pkg/encoding/gob/codec_test.go:1452 +0x2fa
testing.tRunner(0x98281d00, 0x8212bf8, 0x0)
    /home/mikioh/go/src/pkg/testing/testing.go:271 +0x6e
created by testing.RunTests
    /home/mikioh/go/src/pkg/testing/testing.go:346 +0x687
goroutine 1 [chan receive]:
testing.RunTests(0x8048c00, 0x8212b08, 0x49, 0x49, 0x8203101, ...)
    /home/mikioh/go/src/pkg/testing/testing.go:347 +0x6a7
testing.Main(0x8048c00, 0x8212b08, 0x49, 0x49, 0x820f570, ...)
    /home/mikioh/go/src/pkg/testing/testing.go:282 +0x46
main.main()
    /tmp/go-build357021947/encoding/gob/_test/_testmain.go:191 +0x4e
goroutine 2 [syscall]:
created by runtime.main
    /home/mikioh/go/src/pkg/runtime/proc.c:221
exit status 2
FAIL    encoding/gob    0.034s
% hg id
99a66fff2af2+ tip
% hg p
5725045:
    runtime: try extending arena size in 32-bit allocator.

[rsc]

Comment 5:

Well, asking for 800 MB on a 32-bit system is probably
too much anyway.

[mikioh]

Comment 6:

What about making TestFuzz run w/ a flag that default is false,
at least on 386 runtime environment?

[mikioh]

Comment 7:

Also on linux/386.
=== RUN TestFuzz
runtime: memory allocated by OS not in usable range
runtime: out of memory: cannot allocate 268500992-byte block (272695296 in use)
throw: out of memory

[remyoudompheng]

Comment 8:

mikioh: can you retry with latest version of my CL? Here 
  GOARCH=386 go test encoding/gob
works. I tested it with a program that takes up to 1.8GB and it didn't die (but I have
3GB physical RAM)

[mikioh]

Comment 9:

Sure -- Mikio

[mikioh]

Comment 10:

vm5% dmesg | grep memory
real memory  = 2147483648 (2048 MB)
avail memory = 2091053056 (1994 MB)
// with CL 5725045
vm5% go test -v encoding/gob -run=TestFuzz
=== RUN TestFuzz
--- PASS: TestFuzz (0.01 seconds)
    codec_test.go:1465: seed=1331108516916737000 n=100 e=*int
    codec_test.go:1465: seed=1331108516916737000 n=100 e=*float32
    codec_test.go:1465: seed=1331108516916737000 n=100 e=*float64
    codec_test.go:1465: seed=1331108516916737000 n=100 e=*complex128
    codec_test.go:1465: seed=1331108516916737000 n=100 e=*gob.ByteStruct
    codec_test.go:1465: seed=1331108516916737000 n=100 e=*gob.ArrayStruct
    codec_test.go:1465: seed=1331108516916737000 n=100 e=*gob.StringStruct
    codec_test.go:1465: seed=1331108516916737000 n=100 e=*gob.GobTest1
=== RUN TestFuzzRegressions
runtime: out of memory: cannot allocate 268500992-byte block (271712256 in use)
throw: out of memory
goroutine 4 [running]:
encoding/gob.(*Decoder).readMessage(0xa8481be0, 0x1000cf66, 0xa847b620)
    /home/mikioh/go/src/pkg/encoding/gob/decoder.go:92 +0x5a
encoding/gob.(*Decoder).recvMessage(0xa8481be0, 0x0, 0x1)
    /home/mikioh/go/src/pkg/encoding/gob/decoder.go:84 +0xaa
encoding/gob.(*Decoder).decodeTypeSequence(0xa8481be0, 0xa8479c00, 0x160, 0x0)
    /home/mikioh/go/src/pkg/encoding/gob/decoder.go:142 +0x57
encoding/gob.(*Decoder).DecodeValue(0xa8481be0, 0x80fd260, 0xa8479c68, 0x160, 0x0, ...)
    /home/mikioh/go/src/pkg/encoding/gob/decoder.go:206 +0xe4
encoding/gob.(*Decoder).Decode(0xa8481be0, 0x80fd258, 0xa8479c68, 0xa8479c68,
0xa8484f00, ...)
    /home/mikioh/go/src/pkg/encoding/gob/decoder.go:185 +0x168
encoding/gob.encFuzzDec(0xa84654b0, 0x80fcf88, 0xa8465488, 0x982333e0, 0x9823e1b0, ...)
    /home/mikioh/go/src/pkg/encoding/gob/codec_test.go:1429 +0x287
encoding/gob.testFuzz(0xa8468380, 0x994c53f0, 0x126fc084, 0x64, 0xb824cfa4, ...)
    /home/mikioh/go/src/pkg/encoding/gob/codec_test.go:1468 +0x1a4
encoding/gob.TestFuzzRegressions(0xa8468380, 0xe)
    /home/mikioh/go/src/pkg/encoding/gob/codec_test.go:1457 +0x99
testing.tRunner(0xa8468380, 0x8219c44, 0x0)
    /home/mikioh/go/src/pkg/testing/testing.go:271 +0x6e
created by testing.RunTests
    /home/mikioh/go/src/pkg/testing/testing.go:346 +0x687
goroutine 1 [chan receive]:
testing.RunTests(0x8048c00, 0x8219b48, 0x49, 0x49, 0x820a101, ...)
    /home/mikioh/go/src/pkg/testing/testing.go:347 +0x6a7
testing.Main(0x8048c00, 0x8219b48, 0x49, 0x49, 0x82165b0, ...)
    /home/mikioh/go/src/pkg/testing/testing.go:282 +0x46
main.main()
    /tmp/go-build628245346/encoding/gob/_test/_testmain.go:191 +0x4e
goroutine 2 [syscall]:
created by runtime.main
    /home/mikioh/go/src/pkg/runtime/proc.c:221
exit status 2
FAIL    encoding/gob    0.040s

[rsc]

Comment 11:

How big is the input being passed to gob?

[rsc]

Comment 12:

Labels changed: added priority-go1, removed priority-triage.

[alberts]

Comment 13:

It shouldn't ever be more than a few bytes:
http://code.google.com/p/go/source/browse/src/pkg/encoding/gob/codec_test.go#1436
the fuzz test just corrupts the bytes for a single encoded object at this point.

[rsc]

Comment 14:

This test is already skipped for -short.
It is true that the fuzzer can create gobs that happen
to describe very large data structures.
Sometimes they don't fit completely in memory.
I think this is nearly unavoidable.

Labels changed: added priority-later, removed priority-go1.

Status changed to LongTerm.

[davecheney]

Comment 15:

Should this issue be marked as resolved ?
https://code.google.com/p/go/source/detail?r=547c731ab1299c7d8b4b7075195254ea18bbd488