dashboard: mail being sent too eagerly

[rsc]

The recent dashboard failure mail below was sent out without sufficient cause.
I have attached a screen shot of build.golang.org shortly after the mail was received.
You can see that while there is a "fail" in the freebsd-amd64 column for
change fc75f13840b8, there is not an 'ok' in the row below it, and without that 'ok',
there is no proof that this change is the one that broke the build, only that something
did. The dashboard should only send mail for the specific case where a change has caused
a 'fail' _and_ the row below it was 'ok'. This mail was sent too early, and in fact a
minute later we got the second mail below, which blamed cfa8a..., again without
sufficient evidence (although this time correctly, it turns out).

Please make the spam stop.


---------- Forwarded message ----------
From:  <[email protected]>
Date: Tue, May 29, 2012 at 1:48 PM
Subject: [golang-dev] freebsd-amd64 broken by runtime: make go work on netbsd/386
To: [email protected]


Change fc75f13840b8 broke the freebsd-amd64 build:
http://build.golang.org/log/7ce2247ca88a1cdb3e31fe6d5af41dc93f9ee5f6

runtime: make go work on netbsd/386

R=golang-dev, rsc
CC=golang-dev
http://golang.org/cl/6254055

http://code.google.com/p/go/source/detail?r=fc75f13840b8

$ tail -200 < log
ok      cmd/gofmt       0.133s
?       cmd/vet [no test files]
?       cmd/yacc        [no test files]
ok      archive/tar     0.005s
ok      archive/zip     0.038s
ok      bufio   0.050s
ok      bytes   0.030s
ok      compress/bzip2  0.052s
ok      compress/flate  0.537s
ok      compress/gzip   0.007s
ok      compress/lzw    0.096s
ok      compress/zlib   0.842s
ok      container/heap  0.005s
ok      container/list  0.004s
ok      container/ring  0.020s
?       crypto  [no test files]
ok      crypto/aes      0.025s
ok      crypto/cipher   0.057s
ok      crypto/des      0.021s
ok      crypto/dsa      0.019s
ok      crypto/ecdsa    0.024s
ok      crypto/elliptic 0.012s
ok      crypto/hmac     0.005s
ok      crypto/md5      0.006s
ok      crypto/rand     0.062s
ok      crypto/rc4      0.011s
ok      crypto/rsa      0.080s
ok      crypto/sha1     0.012s
ok      crypto/sha256   0.007s
ok      crypto/sha512   0.063s
ok      crypto/subtle   0.013s
ok      crypto/tls      0.090s
ok      crypto/x509     0.460s
?       crypto/x509/pkix        [no test files]
ok      database/sql    0.011s
ok      database/sql/driver     0.026s
ok      debug/dwarf     0.007s
ok      debug/elf       0.013s
ok      debug/gosym     0.005s
ok      debug/macho     0.010s
ok      debug/pe        0.006s
ok      encoding/ascii85        0.005s
ok      encoding/asn1   0.009s
ok      encoding/base32 0.006s
ok      encoding/base64 0.006s
ok      encoding/binary 0.517s
ok      encoding/csv    0.006s
ok      encoding/gob    0.029s
ok      encoding/hex    0.005s
ok      encoding/json   0.047s
ok      encoding/pem    0.007s
ok      encoding/xml    0.012s
ok      errors  0.009s
ok      exp/ebnf        0.005s
ok      exp/ebnflint    0.011s
ok      exp/gotype      0.181s
ok      exp/html        0.094s
ok      exp/locale/collate      0.017s
ok      exp/locale/collate/build        0.008s
ok      exp/norm        0.797s
ok      exp/proxy       0.006s
panic: go/types internal error: assertion failed [recovered]
       panic: interface conversion: interface is string, not types.importError
[recovered]
       panic: interface conversion: interface is string, not types.importError

goroutine 5 [running]:
testing._func_003(0x800802fa8, 0x800802100, 0x800802520, 0x8008023e0)
       /usr/local/go/src/pkg/testing/testing.go:268 +0xf6
----- stack segment boundary -----
exp/types._func_002(0x8008028e8, 0x800802100, 0x800802fb8, 0x800802568)
     
 /tmp/gobuilder/freebsd-amd64-fc75f13840b8/go/src/pkg/exp/types/gcimporter.go:99 +0x67
----- stack segment boundary -----
exp/types.assert(0x4ffe00, 0xf8401046e0)
       /tmp/gobuilder/freebsd-amd64-fc75f13840b8/go/src/pkg/exp/types/const.go:37
+0x4e
exp/types.(*gcParser).declare(0xf8400b2580, 0xf84013b090, 0xf800000002, 0xf8401071f8,
0x5, ...)
     
 /tmp/gobuilder/freebsd-amd64-fc75f13840b8/go/src/pkg/exp/types/gcimporter.go:200 +0x92
exp/types.(*gcParser).parseConstDecl(0xf8400b2580, 0x800000005)
     
 /tmp/gobuilder/freebsd-amd64-fc75f13840b8/go/src/pkg/exp/types/gcimporter.go:703 +0xbc
exp/types.(*gcParser).parseDecl(0xf8400b2580, 0xf840059820)
     
 /tmp/gobuilder/freebsd-amd64-fc75f13840b8/go/src/pkg/exp/types/gcimporter.go:837 +0xaf
exp/types.(*gcParser).parseExport(0xf8400b2580, 0xf84015c820, 0xf80000004b, 0xf840145850)
     
 /tmp/gobuilder/freebsd-amd64-fc75f13840b8/go/src/pkg/exp/types/gcimporter.go:876 +0x217
exp/types.GcImportData(0xf840059820, 0xf84015c820, 0x4b, 0xf840145850, 0xf80000000a, ...)
     
 /tmp/gobuilder/freebsd-amd64-fc75f13840b8/go/src/pkg/exp/types/gcimporter.go:105 +0x207
exp/types.GcImport(0xf840059820, 0xf840145850, 0xa0000000a, 0x0, 0x0, ...)
     
 /tmp/gobuilder/freebsd-amd64-fc75f13840b8/go/src/pkg/exp/types/gcimporter.go:153 +0x398
exp/types.testPath(0xf8400af1c0, 0xf840145850, 0xf80000000a, 0xf80000000a, 0x3, ...)
     
 /tmp/gobuilder/freebsd-amd64-fc75f13840b8/go/src/pkg/exp/types/gcimporter_test.go:55
+0x42
exp/types.testDir(0xf8400af1c0, 0xf840132d58, 0xe00000006, 0xec75702aa, 0x33f05748, ...)
     
 /tmp/gobuilder/freebsd-amd64-fc75f13840b8/go/src/pkg/exp/types/gcimporter_test.go:82
+0x6ce
exp/types.testDir(0xf8400af1c0, 0x5a7734, 0x0, 0xec75702aa, 0x33f05748, ...)
     
 /tmp/gobuilder/freebsd-amd64-fc75f13840b8/go/src/pkg/exp/types/gcimporter_test.go:88
+0x81c
exp/types.TestGcImport(0xf8400af1c0, 0x3177bdf8)
     
 /tmp/gobuilder/freebsd-amd64-fc75f13840b8/go/src/pkg/exp/types/gcimporter_test.go:108
+0x1ec
testing.tRunner(0xf8400af1c0, 0x69e3e8, 0x0, 0x0)
       /usr/local/go/src/pkg/testing/testing.go:273 +0x6f
created by testing.RunTests
       /usr/local/go/src/pkg/testing/testing.go:349 +0x784

goroutine 1 [chan receive]:
testing.RunTests(0x400c00, 0x69e3d0, 0x200000002, 0x430401, 0x8006e0f18, ...)
       /usr/local/go/src/pkg/testing/testing.go:350 +0x7a7
testing.Main(0x400c00, 0x69e3d0, 0x200000002, 0x6a21e8, 0x0, ...)
       /usr/local/go/src/pkg/testing/testing.go:285 +0x7a
main.main()
       /tmp/go-build159262258/exp/types/_test/_testmain.go:45 +0x91

goroutine 2 [syscall]:
created by runtime.main
       /usr/local/go/src/pkg/runtime/proc.c:221

goroutine 3 [syscall]:
created by addtimer
       /usr/local/go/src/pkg/runtime/ztime_freebsd_amd64.c:69

goroutine 7 [finalizer wait]:
created by runtime.gc
       /usr/local/go/src/pkg/runtime/mgc0.c:849
FAIL    exp/types       0.217s
ok      exp/utf8string  0.012s
ok      expvar  0.013s
ok      flag    0.008s
ok      fmt     0.061s
ok      go/ast  0.007s
ok      go/build        0.431s
ok      go/doc  0.147s
ok      go/parser       0.034s
ok      go/printer      0.273s
ok      go/scanner      0.006s
ok      go/token        0.047s
?       hash    [no test files]
ok      hash/adler32    0.010s
ok      hash/crc32      0.011s
ok      hash/crc64      0.004s
ok      hash/fnv        0.007s
ok      html    0.010s
ok      html/template   0.062s
ok      image   0.071s
ok      image/color     0.013s
ok      image/draw      0.025s
?       image/gif       [no test files]
ok      image/jpeg      0.056s
ok      image/png       0.044s
ok      index/suffixarray       0.012s
ok      io      0.231s
ok      io/ioutil       0.006s
ok      log     0.012s
ok      log/syslog      0.221s
ok      math    0.012s
ok      math/big        0.083s
ok      math/cmplx      0.005s
ok      math/rand       0.096s
ok      mime    0.006s
ok      mime/multipart  0.135s
ok      net     0.683s
ok      net/http        1.693s
ok      net/http/cgi    0.215s
ok      net/http/fcgi   0.010s
ok      net/http/httptest       0.010s
ok      net/http/httputil       0.073s
?       net/http/pprof  [no test files]
ok      net/mail        0.007s
ok      net/rpc 0.065s
ok      net/rpc/jsonrpc 0.015s
ok      net/smtp        0.008s
ok      net/textproto   0.008s
ok      net/url 0.007s
ok      old/netchan     0.019s
ok      os      0.676s
ok      os/exec 0.122s
ok      os/signal       0.005s
ok      os/user 0.007s
ok      path    0.005s
ok      path/filepath   0.052s
ok      reflect 0.007s
ok      regexp  0.065s
ok      regexp/syntax   0.498s
ok      runtime 0.328s
?       runtime/cgo     [no test files]
ok      runtime/debug   0.005s
ok      runtime/pprof   0.340s
ok      sort    0.059s
ok      strconv 0.374s
ok      strings 0.010s
ok      sync    0.031s
ok      sync/atomic     0.046s
?       syscall [no test files]
?       testing [no test files]
?       testing/iotest  [no test files]
ok      testing/quick   0.027s
ok      text/scanner    0.013s
ok      text/tabwriter  0.011s
ok      text/template   0.016s
ok      text/template/parse     0.017s
ok      time    3.597s
ok      unicode 0.006s
ok      unicode/utf16   0.020s
ok      unicode/utf8    0.052s
?       unsafe  [no test files]





---------- Forwarded message ----------
From:  <[email protected]>
Date: Tue, May 29, 2012 at 1:52 PM
Subject: [golang-dev] freebsd-amd64 broken by crypto: housekeeping
To: [email protected]


Change cfa8a356dcc8 broke the freebsd-amd64 build:
http://build.golang.org/log/29c22d4868f5fabe4b9e051359ea3906d28a84c9

crypto: housekeeping

Rename _Block to block, don't bother making it compute count.
Add benchmarks.

R=agl, agl
CC=golang-dev
http://golang.org/cl/6243053

http://code.google.com/p/go/source/detail?r=cfa8a356dcc8

$ tail -200 < log
ok      cmd/gofmt       0.376s
?       cmd/vet [no test files]
?       cmd/yacc        [no test files]
ok      archive/tar     0.007s
ok      archive/zip     0.039s
ok      bufio   0.052s
ok      bytes   0.025s
ok      compress/bzip2  0.049s
ok      compress/flate  0.460s
ok      compress/gzip   0.008s
ok      compress/lzw    0.089s
ok      compress/zlib   0.851s
ok      container/heap  0.005s
ok      container/list  0.005s
ok      container/ring  0.017s
?       crypto  [no test files]
ok      crypto/aes      0.026s
ok      crypto/cipher   0.005s
ok      crypto/des      0.067s
ok      crypto/dsa      0.009s
ok      crypto/ecdsa    0.039s
ok      crypto/elliptic 0.011s
ok      crypto/hmac     0.008s
ok      crypto/md5      0.006s
ok      crypto/rand     0.015s
ok      crypto/rc4      0.005s
ok      crypto/rsa      0.084s
ok      crypto/sha1     0.010s
ok      crypto/sha256   0.007s
ok      crypto/sha512   0.007s
ok      crypto/subtle   0.012s
ok      crypto/tls      0.083s
ok      crypto/x509     0.471s
?       crypto/x509/pkix        [no test files]
ok      database/sql    0.012s
ok      database/sql/driver     0.057s
ok      debug/dwarf     0.011s
ok      debug/elf       0.009s
ok      debug/gosym     0.009s
ok      debug/macho     0.007s
ok      debug/pe        0.007s
ok      encoding/ascii85        0.013s
ok      encoding/asn1   0.043s
ok      encoding/base32 0.009s
ok      encoding/base64 0.009s
ok      encoding/binary 0.043s
ok      encoding/csv    0.011s
ok      encoding/gob    0.030s
ok      encoding/hex    0.005s
ok      encoding/json   0.043s
ok      encoding/pem    0.008s
ok      encoding/xml    0.016s
ok      errors  0.006s
ok      exp/ebnf        0.005s
ok      exp/ebnflint    0.007s
ok      exp/gotype      0.182s
ok      exp/html        0.133s
ok      exp/locale/collate      0.020s
ok      exp/locale/collate/build        0.090s
ok      exp/norm        0.625s
ok      exp/proxy       0.343s
panic: go/types internal error: assertion failed [recovered]
       panic: interface conversion: interface is string, not types.importError
[recovered]
       panic: interface conversion: interface is string, not types.importError

goroutine 5 [running]:
testing._func_003(0x800962fa8, 0x800962100, 0x800962520, 0x8009623e0)
       /usr/local/go/src/pkg/testing/testing.go:268 +0xf6
----- stack segment boundary -----
exp/types._func_002(0x8009628e8, 0x800962100, 0x800962fb8, 0x800962568)
     
 /tmp/gobuilder/freebsd-amd64-cfa8a356dcc8/go/src/pkg/exp/types/gcimporter.go:99 +0x67
----- stack segment boundary -----
exp/types.assert(0x4ffe00, 0xf84015f520)
       /tmp/gobuilder/freebsd-amd64-cfa8a356dcc8/go/src/pkg/exp/types/const.go:37
+0x4e
exp/types.(*gcParser).declare(0xf84018c000, 0xf8401ce7b0, 0xf800000002, 0xf8401c6590,
0x5, ...)
     
 /tmp/gobuilder/freebsd-amd64-cfa8a356dcc8/go/src/pkg/exp/types/gcimporter.go:200 +0x92
exp/types.(*gcParser).parseConstDecl(0xf84018c000, 0x800000005)
     
 /tmp/gobuilder/freebsd-amd64-cfa8a356dcc8/go/src/pkg/exp/types/gcimporter.go:703 +0xbc
exp/types.(*gcParser).parseDecl(0xf84018c000, 0xf840059820)
     
 /tmp/gobuilder/freebsd-amd64-cfa8a356dcc8/go/src/pkg/exp/types/gcimporter.go:837 +0xaf
exp/types.(*gcParser).parseExport(0xf84018c000, 0xf840197c30, 0xf80000004b, 0xf8401ce170)
     
 /tmp/gobuilder/freebsd-amd64-cfa8a356dcc8/go/src/pkg/exp/types/gcimporter.go:876 +0x217
exp/types.GcImportData(0xf840059820, 0xf840197c30, 0x4b, 0xf8401ce170, 0xf80000000a, ...)
     
 /tmp/gobuilder/freebsd-amd64-cfa8a356dcc8/go/src/pkg/exp/types/gcimporter.go:105 +0x207
exp/types.GcImport(0xf840059820, 0xf8401ce170, 0xa0000000a, 0x0, 0x0, ...)
     
 /tmp/gobuilder/freebsd-amd64-cfa8a356dcc8/go/src/pkg/exp/types/gcimporter.go:153 +0x398
exp/types.testPath(0xf8400c01c0, 0xf8401ce170, 0xf80000000a, 0xf80000000a, 0x3, ...)
     
 /tmp/gobuilder/freebsd-amd64-cfa8a356dcc8/go/src/pkg/exp/types/gcimporter_test.go:55
+0x42
exp/types.testDir(0xf8400c01c0, 0xf840144880, 0xe00000006, 0xec7570399, 0x3846a1a8, ...)
     
 /tmp/gobuilder/freebsd-amd64-cfa8a356dcc8/go/src/pkg/exp/types/gcimporter_test.go:82
+0x6ce
exp/types.testDir(0xf8400c01c0, 0x5a7734, 0x0, 0xec7570399, 0x3846a1a8, ...)
     
 /tmp/gobuilder/freebsd-amd64-cfa8a356dcc8/go/src/pkg/exp/types/gcimporter_test.go:88
+0x81c
exp/types.TestGcImport(0xf8400c01c0, 0x368f3960)
     
 /tmp/gobuilder/freebsd-amd64-cfa8a356dcc8/go/src/pkg/exp/types/gcimporter_test.go:108
+0x1ec
testing.tRunner(0xf8400c01c0, 0x69e3e8, 0x0, 0x0)
       /usr/local/go/src/pkg/testing/testing.go:273 +0x6f
created by testing.RunTests
       /usr/local/go/src/pkg/testing/testing.go:349 +0x784

goroutine 1 [chan receive]:
testing.RunTests(0x400c00, 0x69e3d0, 0x200000002, 0x430401, 0x8006e0f18, ...)
       /usr/local/go/src/pkg/testing/testing.go:350 +0x7a7
testing.Main(0x400c00, 0x69e3d0, 0x200000002, 0x6a21e8, 0x0, ...)
       /usr/local/go/src/pkg/testing/testing.go:285 +0x7a
main.main()
       /tmp/go-build096326926/exp/types/_test/_testmain.go:45 +0x91

goroutine 2 [syscall]:
created by runtime.main
       /usr/local/go/src/pkg/runtime/proc.c:221

goroutine 3 [syscall]:
created by addtimer
       /usr/local/go/src/pkg/runtime/ztime_freebsd_amd64.c:69

goroutine 7 [finalizer wait]:
created by runtime.gc
       /usr/local/go/src/pkg/runtime/mgc0.c:849
FAIL    exp/types       0.060s
ok      exp/utf8string  0.005s
ok      expvar  0.008s
ok      flag    0.005s
ok      fmt     0.086s
ok      go/ast  0.081s
ok      go/build        0.160s
ok      go/doc  0.048s
ok      go/parser       0.036s
ok      go/printer      0.244s
ok      go/scanner      0.007s
ok      go/token        0.033s
?       hash    [no test files]
ok      hash/adler32    0.011s
ok      hash/crc32      0.008s
ok      hash/crc64      0.009s
ok      hash/fnv        0.005s
ok      html    0.006s
ok      html/template   0.041s
ok      image   0.082s
ok      image/color     0.014s
ok      image/draw      0.068s
?       image/gif       [no test files]
ok      image/jpeg      0.099s
ok      image/png       0.060s
ok      index/suffixarray       0.011s
ok      io      0.248s
ok      io/ioutil       0.064s
ok      log     0.006s
ok      log/syslog      0.224s
ok      math    0.009s
ok      math/big        0.239s
ok      math/cmplx      0.008s
ok      math/rand       0.129s
ok      mime    0.007s
ok      mime/multipart  0.143s
ok      net     0.661s
ok      net/http        1.570s
ok      net/http/cgi    0.274s
ok      net/http/fcgi   0.009s
ok      net/http/httptest       0.021s
ok      net/http/httputil       0.019s
?       net/http/pprof  [no test files]
ok      net/mail        0.006s
ok      net/rpc 0.063s
ok      net/rpc/jsonrpc 0.013s
ok      net/smtp        0.008s
ok      net/textproto   0.015s
ok      net/url 0.006s
ok      old/netchan     0.014s
ok      os      0.663s
ok      os/exec 0.108s
ok      os/signal       0.005s
ok      os/user 0.014s
ok      path    0.006s
ok      path/filepath   0.053s
ok      reflect 0.009s
ok      regexp  0.060s
ok      regexp/syntax   0.534s
ok      runtime 0.248s
?       runtime/cgo     [no test files]
ok      runtime/debug   0.011s
ok      runtime/pprof   0.858s
ok      sort    0.070s
ok      strconv 0.406s
ok      strings 0.010s
ok      sync    0.060s
ok      sync/atomic     0.059s
?       syscall [no test files]
?       testing [no test files]
?       testing/iotest  [no test files]
ok      testing/quick   0.063s
ok      text/scanner    0.007s
ok      text/tabwriter  0.016s
ok      text/template   0.051s
ok      text/template/parse     0.014s
ok      time    3.700s
ok      unicode 0.006s
ok      unicode/utf16   0.005s
ok      unicode/utf8    0.007s
?       unsafe  [no test files]

Attachments:

1. Screen Shot 2012-05-29 at 1.51.42 PM.png (74749 bytes)

[adg]

Comment 1:

Weird. The code is there to only notify on an ok->bad transition. Not
sure why it decided to stop working now.
http://code.google.com/p/go/source/browse/misc/dashboard/app/build/notify.go?spec=svnf82e8102025c9866f250ceb667a06fe3dbe50446&name=f82e8102025c&r=f82e8102025c9866f250ceb667a06fe3dbe50446#32

[rsc]

Comment 2:

The way it was behaving suggests that somehow it was interpreting a
missing result as 'ok'. Looking at the code I can't see why.

[rsc]

Comment 3:

This happened again overnight with the 3 freebsd/386 mails that should
have been 1.
Can you add some logging to the app, recording the builder result that
triggered the mail? For example instead of returning ok, present, the
function could just return the *Result  and then you could check r !=
nil && r.OK in the test instead of present && ok. Doing that would
mean you're holding the result and can log it as a reason to send the
mail. In fact it would be nice to log the entire result string.

[adg]

Comment 4:

I've done what you described. Let's see what happens.
http://code.google.com/p/go/source/detail?r=d137b2a8007ae64eaeeeec0156f8d2a9eb2b5903

[adg]

Comment 5:

Status changed to Fixed.

[adg]

Comment 6:

http://code.google.com/p/go/source/detail?r=ff2e12aef85e37e73894941c69e55c4796fb919e