ioutil: deadlock when WriteFile called with a 5<<30 byte buffer on Windows

[prvst]

What version of Go are you using (go version)?

$ go version go1.12 linux/amd64

Does this issue reproduce with the latest release?

Yes but only on Windows, not on Linux.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN="/home/prvst/go/bin"
GOCACHE="/home/prvst/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/prvst/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build007601782=/tmp/go-build -gno-record-gcc-switches"

What did you do?

The program has a function that parses a text file, creates a data structure and serializes to disk using msgpack. The stack trace seems to indicate the error on the ioutil.WriteFile call to create the file (see * below). What seems to have caused the issue is the size of the text file, ~ 5GB. No errors with files smaller than 1 GB.

	b, er := msgpack.Marshal(&d)
	if er != nil {
		return &err.Error{Type: err.CannotOpenFile, Class: err.FATA, Argument: "database structure"}
	}

*	er = ioutil.WriteFile(sys.DBBin(), b, sys.FilePermission())
	if er != nil {
		return &err.Error{Type: err.CannotSerializeData, Class: err.FATA, Argument: er.Error()}
	}

The program was developed on Linux and compiled to different platforms, the error is not reproducible when running on Linux.

What did you expect to see?

no errors

What did you see instead?

fatal error: all goroutines are asleep - deadlock!

goroutine 1 [semacquire]:
sync.runtime_SemacquireMutex(0xc00009a954, 0xc0fad3c600)
        /usr/local/go/src/runtime/sema.go:71 +0x44
sync.(*Mutex).Lock(0xc00009a950)
        /usr/local/go/src/sync/mutex.go:134 +0x106
internal/poll.(*FD).Write(0xc00009a780, 0xc31f400000, 0xfc26204c, 0x11296f667, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_windows.go:673 +0x139
os.(*File).write(...)
        /usr/local/go/src/os/file_windows.go:224
os.(*File).Write(0xc000098078, 0xc31f400000, 0xfc26204c, 0x11296f667, 0xc000098078, 0x0, 0x0)
        /usr/local/go/src/os/file.go:145 +0x77
io/ioutil.WriteFile(0xc0004a6030, 0xc, 0xc31f400000, 0xfc26204c, 0x11296f667, 0xc0000001a4, 0xc, 0xc0000f4500)
        /usr/local/go/src/io/ioutil/ioutil.go:84 +0x9b
github.com/prvst/philosopher/lib/dat.(*Base).Serialize(0xc000153740, 0xc00006e0e0)
        /home/prvst/go/src/github.com/prvst/philosopher/lib/dat/dat.go:358 +0x238
github.com/prvst/philosopher/lib/dat.Run(0xc000084570, 0x24, 0xc00006e600, 0x15, 0xc0000c64b0, 0x4c, 0xc0000845a0, 0x24, 0xc00006e620, 0x1b, ...)
        /home/prvst/go/src/github.com/prvst/philosopher/lib/dat/dat.go:55 +0x949
github.com/prvst/philosopher/cmd.glob..func4(0x8422500, 0xc00006b5a0, 0x0, 0x2)
        /home/prvst/go/src/github.com/prvst/philosopher/cmd/database.go:21 +0x16d
github.com/spf13/cobra.(*Command).execute(0x8422500, 0xc00006b580, 0x2, 0x2, 0x8422500, 0xc00006b580)
        /home/prvst/go/src/github.com/spf13/cobra/command.go:766 +0x2b5
github.com/spf13/cobra.(*Command).ExecuteC(0x8423f20, 0x4057f6, 0xc000062058, 0x0)
        /home/prvst/go/src/github.com/spf13/cobra/command.go:852 +0x2c7
github.com/spf13/cobra.(*Command).Execute(...)
        /home/prvst/go/src/github.com/spf13/cobra/command.go:800
github.com/prvst/philosopher/cmd.Execute()
        /home/prvst/go/src/github.com/prvst/philosopher/cmd/root.go:32 +0x35
main.main()
        /home/prvst/go/src/github.com/prvst/philosopher/main.go:19 +0x75

[egonelbre]

Here's a short reproducer:

package main

import "io/ioutil"

func main() {
	data := make([]byte, 5<<30)
	ioutil.WriteFile("example.dat", data, 0755)
}

[egonelbre]

Seems to be working on tip and commit 40d8c3d probably fixed this.

[deanveloper]

I was able to reproduce on go version go1.12.1 windows/amd64, although I guess if it's already fixed I don't think that matters too much

[andybons]

@ianlancetaylor @alexbrainman this looks to be a regression from 1.11 (introduced in golang.org/cl//129137).

Should the fix above be backported?

[ianlancetaylor]

@andybons Sure. We can backport CL 165598 to 1.12.

[ianlancetaylor]

I guess this bug is already fixed on tip so retargeting for 1.12.

[gopherbot]

Change https://golang.org/cl/170680 mentions this issue: [release-branch.go1.12] internal/poll: fix deadlock in Write if len(buf) > maxRW

[gopherbot]

Closed by merging 499088f to release-branch.go1.12.

[alexbrainman]

We can backport CL 165598 to 1.12.

I agree with what Ian said.

Alex

[prvst]

Hey everyone, thanks for the fast reply. Keep up with the good work, Go is awesome !
Cheers