cmd/go, cmd/cgo: repeatable builds on Solaris

[neild]

Binaries built with "go build" that use cgo or include packages that use cgo contain references to a temporary directory. Multiple builds for the same binary will produce inconsistent results.

Simple reproduction:

// foo.go
package main

// #include <math.h>
// #cgo LDFLAGS: -lm
import "C"

import "fmt"

func main() {
        fmt.Println(C.sqrt(4))
}

$ go build foo.go && md5sum foo
d4cc4febe540953e8115417476adc4a4  foo
$ go build foo.go && md5sum foo
28f5f670a48e6f72a2f31405d9fbf2cc  foo
$ strings foo | grep go-build
/tmp/go-build847878379/command-line-arguments/_obj/_cgo_export.c
/tmp/go-build847878379/command-line-arguments/_obj/foo.cgo2.c
/tmp/go-build988195549/runtime/cgo/_obj/_cgo_export.c
/tmp/go-build988195549/runtime/cgo/_obj/cgo.cgo2.c

Some build systems require reproducible results: The same inputs should produce precisely the same outputs. The above behavior violates that requirement.

The problem appears to be that the gcc command invoked by "go build" includes the absolute path of the source file in $WORKDIR, which gcc then bakes into the resulting object file.

One fix might be to execute gcc from within $WORKDIR. There is, however, a comment in cmd/go/build.go indicating that the current behavior is intentional: "We always pass absolute paths of source files so that the error messages will include the full path to a file in need of attention."

Another possibility might be to use -fdebug-prefix-map to elide $WORKDIR from the debugging information written by gcc. I don't know if this can be generalized to other compilers.

[neild]

Change (*builder).ccompile to execute gcc from within $WORKDIR: Works, but something is still inserting $WORKDIR into the object file. I haven't yet figured out what.

$ strings bin | grep go-build
/tmp/go-build502746782/cgotest/_obj
/tmp/go-build159213004/runtime/cgo/_obj

Change (*builder).ccompilerCmd to include -fdebug-prefix-map: Works, but the compiler command line appears to be included in the object file somewhere:

$ strings bin | grep go-build
GNU C 4.8.4 -m64 -mtune=generic -march=x86-64 -g -O2 -fPIC -fno-working-directory -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build629323208=WORK -fstack-protector
GNU C 4.8.4 -m64 -mtune=generic -march=x86-64 -g -O2 -fPIC -fno-working-directory -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build738763988=WORK -fstack-protector

Also -fdebug-prefix-map doesn't appear to be supported by clang, so that's probably not a useful approach to continue pursuing.

[ALTree]

#9206

[mdempsky]

We could post-process the compiled .o file's DWARF debug strings to replace all occurrences of /tmp/go-build12345 with /tmp/go-buildXXXXX.

[mdempsky]

Hm, post-processing the DWARF debug strings doesn't seem to reliably work either, as the debug string's ordering is also influenced by the actual build directory name:

$ go build -o foo1 foo.go
$ go build -o foo2 foo.go
$ diff -u <(llvm-dwarfdump-3.5 foo1) <(llvm-dwarfdump-3.5 foo2)
[...]
@@ -134762,9 +134762,9 @@
 0x0000009e: "GNU C 4.8.4 -m64 -mtune=generic -march=x86-64 -g -O2 -fPIC -fmessage-length=0 -fstack-protector"
 0x000000fe: "short int"
 0x00000108: "complex double"
-0x00000117: "_cgo_e6732a3e38c3_Cfunc_sqrt"
-0x00000134: "_cgo_topofstack"
-0x00000144: "/tmp/go-buildXXXXXXXXX/command-line-arguments/_obj/foo.cgo2.c"
+0x00000117: "/tmp/go-buildXXXXXXXXX/command-line-arguments/_obj/foo.cgo2.c"
+0x00000155: "_cgo_e6732a3e38c3_Cfunc_sqrt"
+0x00000172: "_cgo_topofstack"
 0x00000182: "stktop"
 0x00000189: "/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/_cgo_export.c"
 0x000001bf: "/tmp/go-buildXXXXXXXXX/runtime/cgo/_obj/cgo.cgo2.c"

[mdempsky]

BTW, the temporary directory generated by cmd/link for external linking is also being embedded into the output files (as a FILE symbol in the ELF .symtab):

$ strings foo1 foo2 | grep go-link
/tmp/go-link-676379844/go.o
/tmp/go-link-961034572/go.o

[rsc]

I looked briefly into this but it's like gcc is actively working against us. I love the fact that the -fdebug-prefix-map command-line argument gets embedded into the binary when the whole point of that flag is to specify something to keep out.

https://go-review.googlesource.com/17943 has my work in progress. Until gcc or clang wants to cooperate I don't see much point in heroics. I do see that clang seems to have added support for -fdebug-prefix-map very recently, not that we can rely on it being available.

[neild]

The magic additional incantation appears to be -gno-record-gcc-switches to keep the flags out of the binary.

[rsc]

Yay!

[gopherbot]

CL https://golang.org/cl/19363 mentions this issue.

[neild]

Fixed with a caveat: The system compiler needs to support -fdebug-prefix-map for builds to be reproducible. That's gcc 4.3 and clang 3.8 (not released yet).

[neild]

Reopening, because Solaris builds remain inconsistent.

[binarycrusader]

Is this still a problem after my fix for #14957?

I can't reproduce the issue locally with my fix applied:
$ go build foo.go && md5sum foo; rm -f foo
2d826543fb363898a060dce93121cb46 foo
$ go build foo.go && md5sum foo; rm -f foo
2d826543fb363898a060dce93121cb46 foo

Historically, I would note that the Solaris linker would insert the working directory if no STT_FILE was found, but that's no longer true in current Solaris releases.

So I think what was causing the changes made for this bug to not work on Solaris was the problem fixed in #14957.

[bradfitz]

Sounds like this is fixed then.

[ALTree]

I believe #9206 too can be closed.

[gopherbot]

CL https://golang.org/cl/23741 mentions this issue.

[mikioh]

No, this issue is not fixed on Solaris. See https://build.golang.org/log/efb56d2bc3049a131016966cff929c44137968d8

[rasky]

I also confirm #9206 cannot be closed, I can still reproduce the issue.

[ianlancetaylor]

Postponing until 1.8.

[gopherbot]

CL https://golang.org/cl/24460 mentions this issue.

[rsc]

Confirmed this is still a problem on Solaris:

$ VM=$(gomote create solaris-amd64-smartosbuildlet)
$ gomote run $VM go/src/make.bash
...
$ gomote run $VM /bin/bash -c '
	/tmp/workdir/go/bin/go build -ldflags=-linkmode=external -o /tmp/gofmt1 cmd/gofmt; 
	/tmp/workdir/go/bin/go build -ldflags=-linkmode=external -o /tmp/gofmt2 cmd/gofmt; 
	od -t x1 /tmp/gofmt1 >/tmp/gofmt1.o
	od -t x1 /tmp/gofmt2 >/tmp/gofmt2.o
	diff /tmp/gofmt[12].o
'
4906c4906
< 0231220 2d 62 75 69 6c 64 30 36 37 31 39 35 33 33 35 2f
---
> 0231220 2d 62 75 69 6c 64 39 32 31 38 35 38 32 39 32 2f
140441c140441
< 10457460 f8 fd ff 6f 00 00 00 00 cb f3 00 00 00 00 00 00
---
> 10457460 f8 fd ff 6f 00 00 00 00 d2 f3 00 00 00 00 00 00
154056c154056
< 11336120 2d 62 75 69 6c 64 30 36 37 31 39 35 33 33 35 2f
---
> 11336120 2d 62 75 69 6c 64 39 32 31 38 35 38 32 39 32 2f
Error running run: exit status 1
$

The first and third diff are the work tempdir leaking into the binary. I'm not sure what the second diff is. I have no solution. If someone knows how to tell gcc on Solaris not to record the absolute object paths used during a link, please let us know.