cmd/compile: 1.3MB program takes 290MB of memory to compile

[davecheney]

When compiling pkg/exp/eval or test/64bit.go, RSS memory usage using 5g or 6g exceeds
100mb. 

What is the expected output?

RSS usage should be more inline with other packages, around 10-20 mb

What do you see instead?

When compiling pkg/exp/eval or test/64bit.go, RSS memory usage using 5g or 6g exceeds
100mb. 

Which compiler are you using (5g, 6g, 8g, gccgo)?

5g and 6g

Which operating system are you using?

Debian 6 amd64 and arm5

Which revision are you using?  (hg identify)

+tip

Please provide any additional information below.

[rsc]

Comment 1:

It is probably not test/64bit.go but its *output* that 6g uses a lot
of memory to compile.
That's a big program.

Status changed to LongTerm.

[davecheney]

Comment 2:

Ahh yes, tmp.go is 1,300kb of source, which consumes 289,145,039 bytes (according to
valgrind). Looking at the way subr.c:mal works this isn't an easy problem to fix (or a a
very high priority one)

[rsc]

Comment 3:

Labels changed: added priority-later.

[rsc]

Comment 4:

I would like to make constants use int64 until it is necessary to use a bignum.
That would probably fix this issue.

Labels changed: added performance.

[rsc]

Comment 5:

Labels changed: added go1.1maybe.

[remyoudompheng]

Comment 6:

I am starting having problems to compile go on a amd64 machine I don't use really often.
$ uname -a
Linux maison 3.7.1-2-ARCH #1 SMP PREEMPT Thu Dec 20 17:57:00 CET 2012 x86_64 GNU/Linux
$ sysctl vm.overcommit_memory
vm.overcommit_memory = 1
$ free -h
             total       used       free     shared    buffers     cached
Mem:          998M       675M       322M         0B        13M        88M
-/+ buffers/cache:       573M       424M
Swap:           0B         0B         0B
$ cat /proc/cpuinfo
[...]
model name  : Intel(R) Pentium(R) D CPU 2.80GHz
[...]
cpu MHz     : 2792.945
[...]
cpu cores   : 2
[...]
$ go install -a -v std
[...]
compress/lzw
hash/adler32
compress/zlib
container/list
container/ring
crypto/sha512
database/sql/driver
database/sql
debug/gosym
encoding/ascii85
cmd/go
expvar
net/http/pprof
cmd/godoc
encoding/base32
go build encoding/base32: fork/exec /opt/remy/go/pkg/tool/linux_amd64/6g: cannot
allocate memory
encoding/csv
go build encoding/csv: fork/exec /opt/remy/go/pkg/tool/linux_amd64/6g: cannot allocate
memory
exp/cookiejar
go build exp/cookiejar: fork/exec /opt/remy/go/pkg/tool/linux_amd64/6g: cannot allocate
memory
exp/ebnf
go build exp/ebnf: fork/exec /opt/remy/go/pkg/tool/linux_amd64/6g: cannot allocate memory
exp/types
go build exp/types: fork/exec /opt/remy/go/pkg/tool/linux_amd64/6g: cannot allocate
memory
exp/html/atom
go build exp/html/atom: fork/exec /opt/remy/go/pkg/tool/linux_amd64/6g: cannot allocate
memory
[...]
Restoring vm.overcommit_memory=1 solves the problem. The case described by this issue
happens for real in the standard library now:
$ ls -l src/pkg/exp/locale/collate
total 2892
drwxr-xr-x 2 remy remy    4096 Dec 23 21:28 build
-rw-r--r-- 1 remy remy    7284 Dec 23 21:28 colelem.go
-rw-r--r-- 1 remy remy    4368 Dec 23 21:28 colelem_test.go
-rw-r--r-- 1 remy remy   12571 Dec 23 21:28 collate.go
-rw-r--r-- 1 remy remy   13089 Dec 23 21:28 collate_test.go
-rw-r--r-- 1 remy remy    2392 Dec 23 21:28 contract.go
-rw-r--r-- 1 remy remy    2291 Dec 23 21:28 contract_test.go
-rw-r--r-- 1 remy remy    1112 Dec 23 21:28 export.go
-rw-r--r-- 1 remy remy    1957 Dec 23 21:28 export_test.go
-rw-r--r-- 1 remy remy     403 Dec 23 21:28 Makefile
-rw-r--r-- 1 remy remy   17770 Dec 23 21:28 maketables.go
-rw-r--r-- 1 remy remy    7009 Dec 23 21:28 regtest.go
-rw-r--r-- 1 remy remy    3914 Dec 23 21:28 table.go
-rw-r--r-- 1 remy remy 2828722 Dec 23 21:28 tables.go
-rw-r--r-- 1 remy remy    8641 Dec 23 21:28 table_test.go
drwxr-xr-x 3 remy remy    4096 Dec 23 21:28 tools
-rw-r--r-- 1 remy remy    2264 Dec 23 21:28 trie.go
-rw-r--r-- 1 remy remy    2596 Dec 23 21:28 trie_test.go

[robpike]

Comment 8:

Labels changed: removed go1.1maybe.

[rsc]

Comment 9:

Labels changed: added go1.3maybe.

[rsc]

Comment 10:

Labels changed: added release-none, removed go1.3maybe.

[rsc]

Comment 11:

Labels changed: added repo-main.

[bankole7782]

This doesn't seem to be a problem. Computers are not in the days of 512mb RAM but 8gb RAM common place.

[ianlancetaylor]

A large multiplier in resource use when compiling is always a problem.

[mdempsky]

On my system, cmd/compile appears to take about 106MB of memory to compile test/64bit.go's output now:

$ /usr/bin/time go tool compile <(go run $(go env GOROOT)/test/64bit.go)
1.08user 0.05system 0:00.94elapsed 120%CPU (0avgtext+0avgdata 106680maxresident)k
0inputs+0outputs (0major+29288minor)pagefaults 0swaps

Do we have a particular goal in mind for when we can consider this issue fixed? For comparison, cmd/compile is taking less memory than gccgo does even without optimizations:

$ /usr/bin/time gccgo -c -o x.o -x go <(go run $(go env GOROOT)/test/64bit.go)
2.67user 0.08system 0:02.92elapsed 94%CPU (0avgtext+0avgdata 191528maxresident)k
256inputs+0outputs (1major+50944minor)pagefaults 0swaps

$ /usr/bin/time gccgo -c -o x.o -O2 -x go <(go run $(go env GOROOT)/test/64bit.go)
7.65user 0.12system 0:08.04elapsed 96%CPU (0avgtext+0avgdata 249000maxresident)k
44184inputs+0outputs (190major+69875minor)pagefaults 0swaps