[dev.fuzz] Crash data written to corpus is incorrect

[stevenjohnstone]

What version of Go are you using (go version)?

$ go version
go version devel go1.17-2a0825d01f Tue Jul 20 00:06:06 2021 +0000 linux/amd64

Does this issue reproduce with the latest release?

n/a

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/stevie/.cache/go-build"
GOENV="/home/stevie/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/stevie/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/stevie/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/stevie/sdk/gotip"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/stevie/sdk/gotip/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="devel go1.17-2a0825d01f Tue Jul 20 00:06:06 2021 +0000"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/stevie/unicode/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build426468616=/tmp/go-build -gno-record-gcc-switches"

What did you do?

// +build gofuzzbeta

package unicode

import (
	"fmt"
	"strings"
	"testing"
)

func FuzzBeta(f *testing.F) {
	f.Fuzz(func(t *testing.T, data []byte) {
		s := string(data)
		valid := strings.ToValidUTF8(s, "")
		if len(valid) == 0 {
			return
		}

		lower := strings.ToLower(valid)
		upper := strings.ToUpper(lower)

		roundtrip := strings.ToLower(upper)

		if roundtrip != lower {
			panic(fmt.Sprintf("%x: %v (%x): %v (%x) != %v (%x), upper = %v (%x)", data, valid, []byte(valid), roundtrip, []byte(roundtrip), lower, []byte(lower), upper, []byte(upper)))
		}
	})
}

Run the fuzzer and it finds (correctly) unicode examples which are modified in a strings.ToUpper strings.ToLower roundtrip

$ gotip test -fuzz=Fuzz
found a crash, minimizing...
fuzzing, elapsed: 0.1s, execs: 995 (7461/sec), workers: 8, interesting: 16
--- FAIL: FuzzBeta (0.13s)
        panic: c2b5: µ (c2b5): μ (cebc) != µ (c2b5), upper = Μ (ce9c)
        goroutine 264 [running]:
        runtime/debug.Stack()
        	/home/stevie/sdk/gotip/src/runtime/debug/stack.go:24 +0x90
        testing.tRunner.func1.2({0x5a8b60, 0xc006707ab0})
        	/home/stevie/sdk/gotip/src/testing/testing.go:1281 +0x267
        testing.tRunner.func1()
        	/home/stevie/sdk/gotip/src/testing/testing.go:1288 +0x218
        panic({0x5a8b60, 0xc006707ab0})
        	/home/stevie/sdk/gotip/src/runtime/panic.go:1038 +0x215
        github.com/stevenjohnstone/unicode.FuzzBeta.func1(0x0, {0xc00013e4c0, 0x2, 0x40})
        	/home/stevie/unicode/fuzz_test.go:25 +0x3e8
        reflect.Value.call({0x5aa780, 0x5e36e0, 0x13}, {0x5d5c43, 0x4}, {0xc0067144b0, 0x2, 0x2})
        	/home/stevie/sdk/gotip/src/reflect/value.go:543 +0xf0d
        reflect.Value.Call({0x5aa780, 0x5e36e0, 0xc006703930}, {0xc0067144b0, 0x2, 0x2})
        	/home/stevie/sdk/gotip/src/reflect/value.go:339 +0x13e
        testing.(*F).Fuzz.func1.1(0xc0002080c0)
        	/home/stevie/sdk/gotip/src/testing/fuzz.go:377 +0x1c6
        testing.tRunner(0xc00673b6c0, 0xc006726a80)
        	/home/stevie/sdk/gotip/src/testing/testing.go:1335 +0x102
        created by testing.(*F).Fuzz.func1
        	/home/stevie/sdk/gotip/src/testing/fuzz.go:366 +0x505
        
        --- FAIL: FuzzBeta (0.00s)
    
    Crash written to testdata/corpus/FuzzBeta/e2422ad459420d437cced43ea44570336d2ccae8d700406a987c1b59cab89e23
    To re-run:
    go test github.com/stevenjohnstone/unicode -run=FuzzBeta/e2422ad459420d437cced43ea44570336d2ccae8d700406a987c1b59cab89e23
FAIL
exit status 1
FAIL	github.com/stevenjohnstone/unicode	0.139s

When I run the suggested command to re-run the crasher, the test passes:

$ gotip test github.com/stevenjohnstone/unicode -run=FuzzBeta/e2422ad459420d437cced43ea44570336d2ccae8d700406a987c1b59cab89e23
ok  	github.com/stevenjohnstone/unicode	0.003s

Looking at the contents of the corpus file I see

$ cat testdata/corpus/FuzzBeta/e2422ad459420d437cced43ea44570336d2ccae8d700406a987c1b59cab89e23
go test fuzz v1
[]byte("\xb5\xb5")

The panic message shows that the actual input was "\xc2\xb5". It appears that incorrect data is written to the corpus?

What did you expect to see?

I expect the re-run command to result in a test failure. I expected the corpus file to contain the input which causes a panic.

What did you see instead?

Incorrect data in the corpus

[stevenjohnstone]

When I disable crasher minimization with this patch to the toolchain

diff --git a/src/internal/fuzz/worker.go b/src/internal/fuzz/worker.go
index e3029bcd66..fca250077d 100644
--- a/src/internal/fuzz/worker.go
+++ b/src/internal/fuzz/worker.go
@@ -206,14 +206,8 @@ func (w *worker) coordinate(ctx context.Context) error {
 
                case crasher := <-w.coordinator.minimizeC:
                        // Received input to minimize from coordinator.
-                       minRes, err := w.minimize(ctx, crasher)
-                       if err != nil {
-                               // Failed to minimize. Send back the original crash.
-                               fmt.Fprintln(w.coordinator.opts.Log, err)
-                               minRes = crasher
-                               minRes.minimized = true
-                       }
-                       w.coordinator.resultC <- minRes
+                       crasher.minimized = true
+                       w.coordinator.resultC <- crasher
                }
        }
 }

the problem goes away.

Interestingly, it's always the penultimate byte of the corpus entry which is incorrect and is always equal to the last byte.

[stevenjohnstone]

The bug seems to be caused by assigning a byte slice to the collection of minimized inputs which can be modified after assignment by minimizeBytes. This should be a copy of the data rather than a reference. This patch fixes the problem for me:

diff --git a/src/internal/fuzz/worker.go b/src/internal/fuzz/worker.go
index e3029bcd66..bab0f07036 100644
--- a/src/internal/fuzz/worker.go
+++ b/src/internal/fuzz/worker.go
@@ -787,7 +787,7 @@ func (ws *workerServer) minimizeInput(ctx context.Context, vals []interface{}, c
                case []byte:
                        switch prev.(type) {
                        case []byte:
-                               vals[valI] = c
+                               vals[valI] = append([]byte{}, c...)
                        case string:
                                vals[valI] = string(c)
                        default:

[gopherbot]

Change https://golang.org/cl/340630 mentions this issue: [dev.fuzz] internal/fuzz: avoid corruption of minimized crashers

[katiehockman]

/cc @golang/fuzzing

[katiehockman]

I can reproduce this, and confirm that your fix stops the issue from happening, but I'm not actually seeing exactly why this bug is occurring in the first place. In theory, I can see how assigning vals[valI] the way that it does would allow it to be modified later on, but I also don't see anywhere in the code where it is actually modifying it.

I'd like to figure that out before we merge the fix.

[katiehockman]

Actually I think I know what is going on. I think the fix is still a reasonable one, but here is my theory for how the memory is actually changing later on:

My suspicion is that https://github.com/golang/go/blob/dev.fuzz/src/internal/fuzz/minimize.go#L38 is the root of how this is happening. We use this same tmp slice every time, pointing to the same slots in memory. We set candidate to tmp every iteration of the loop. Let's say that one iteration finds a smaller input that works: vals[valI] will be set to candidate which is actually just pointing to tmp. Even though we create a new candidate in the next iteration of the loop, we are still setting candidate to the same tmp that we used before, thus the same memory that vals[valI] is currently pointing to. This next iteration, we change candidate, which changes tmp, which changes vals[valI], and there's our issue.