net: broken sendfile on SmartOS/Solaris? large files garbled when sent over network

[doublerebel]

What version of Go are you using (go version)? go version go1.5.1 solaris/amd64
What operating system and processor architecture are you using? SunOS 5.11 joyent_20141030T081701Z i86pc i386 i86pc Solaris, SunOS 5.11 joyent_20151002T091442Z i86pc i386 i86pc Solaris
What did you do? Retrieved a >250K file over http from a golang server
What did you expect to see? The correct contents of the file
What did you see instead? A file with the correct size, but the first ~25% of the file repeated to fill the file.

If I access the file through localhost, the file is served correctly. If I send it over any network interface, the bug appears. Different lengths of the file are repeated to fill the space in each case. All smaller files come through without error.

I attempted to step through using godebug but ran into mailgun/godebug#12, since the issue is clearly somewhere in the standard library. So I have setup a temporary fileserver to demonstrate the issue. text.text.2 contains the beginning of Sherlock Holmes from Project Gutenberg (thanks Norvig).

Control: python -m SimpleHTTPServer
http://165.225.174.156:8000/
http://165.225.174.156:8000/test.txt.2
Serves correct contents of file.

Bug: gohttp (itang/gohttp)
http://165.225.174.156:8080/
http://165.225.174.156:8080/test.txt.2
Serves garbled file, every time.

Localhost: gohttp with a barebones nginx proxy from localhost
http://165.225.174.156/
http://165.225.174.156/test.txt.2
Serves correct contents of file.

I discovered this bug because we just recently have hashicorp/consul building on SmartOS (hashicorp/consul#159), and the Consul UI contains a ~750K minified JS file. Consul uses gorilla/mux and http.FileServer, but I am able to reproduce with the much simpler case of gohttp.

The test suite for the http response methods doesn't have tests from a remote machine, or use large files, so this bug can slip through in most cases unnoticed. Please let me know if I can provide more details. We really need a way to step through and debug the standard libs!

[bradfitz]

I suspect the sendfile implementation for Solaris is broken, then.

Can you modify src/net/sendfile_solaris.go and add a return statement on the first line of func sendfile?

func sendFile(c *netFD, r io.Reader) (written int64, err error, handled bool) {
    return

And see if that fixes it? (it would revert to Read+Write)

/cc @4ad

[doublerebel]

@bradfitz You nailed it. I added a log line and the return statement, and I can now see the file being served correctly. Really appreciate the fast diagnosis.

[4ad]

Could you please try with the latest SmartOS image? Thanks.

[doublerebel]

@4ad sure thing, tested both versions on the latest base-64 image: SunOS 5.11 joyent_20151002T091442Z i86pc i386 i86pc Solaris

The current version fails as on the earlier SmartOS image, and the patched version suggested by @bradfitz works.

[4ad]

Just for confirmation, you tested on a recent SmartOS image, a.i. a recent kernel, not an older SmartOS image with a recent base-64 image?

[doublerebel]

@4ad the tests were all run on instances in the Joyent Public Cloud, base-64 is the name of their default basic 64-bit SmartOS image. joyent_20151002T091442Z is SmartOS Base Image v15.3.0. I can donate an instance for testing, if that helps.

EDIT: FWIW, I just rebuilt Consul with net/sendfile patched, and it now too works as expected.

[bradfitz]

Perhaps we should just disable sendfile on Solaris for Go 1.6 as a safe option. We can re-enable in Go 1.7 once we understand and fix the problem.

[davecheney]

I think that is a good idea

On Tue, 12 Jan 2016, 05:57 Brad Fitzpatrick notifications@github.com
wrote:

Perhaps we should just disable sendfile on Solaris for Go 1.6 as a safe
option. We can re-enable in Go 1.7 once we understand and fix the problem.

—
Reply to this email directly or view it on GitHub
#13892 (comment).