Skip to content

net: sendFile can't handle os.fileWithoutWriteTo #66988

Closed
@philwo

Description

@philwo

Go version

go version go1.22.2 darwin/arm64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/philwo/Library/Caches/go-build'
GOENV='/Users/philwo/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/philwo/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/philwo/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.22.2/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.22.2/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.22.2'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/Users/philwo/src/philtools/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/ft/3rpc4dkj09b3lw6pbm_j0j7w005v1x/T/go-build2443549281=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

  1. Open a regular file f for reading.
  2. Connect to a TCP socket s.
  3. Copy f into s by calling io.Copy(s, f)

Example: https://go.dev/play/p/xJbcSMVNLqy (deadlocks when run on Go Playground due to go.dev/issue/48394)

What did you see happen?

io.Copy does not use sendfile on macOS to copy f into s, but instead falls back to using a buffer due to a regression caused by a side-effect of go.dev/cl/472475.

Stepping through the code with a debugger, I think this is what happens:

io.Copy implements three different strategies to copy src to `dst:

  1. Call src.WriteTo(dst) if src implements the WriterTo interface.
  2. Call dst.ReadFrom(src) if dst implements the ReaderFrom interface.
  3. Use a buffer to handle the copying ourselves.

In the first iteration, this goes through os.File's WriteTo method, which calls into zero_copy_stub.go's writeTo, which is just a stub and thus can't do anything. It thus falls back to calling io.Copy again with itself wrapped into fileWithoutWriteTo, preventing it from taking the first branch.

In this second iteration, it thus calls ReadFrom on s and passes it f wrapped into fileWithoutWriteTo. tcpsock_posix.go attempts to use sendFile now, passing f into its r io.Reader parameter. sendFile needs to ensure that the passed Reader supports Stat, Seek and SyscallConn, and thus attempts a type assertion to *os.File.

However, this fails, because r isn't a File, it's a fileWithoutWriteTo.

Prior to go.dev/cl/472475 this all happened to work, because we never took the first branch in io.Copy, because os.File didn't implement io.WriterTo, so we passed the unmodified arg to the socket's ReadFrom.

What did you expect to see?

io.Copy should use sendfile on macOS to copy f into s, like it did prior to go.dev/cl/472475 (and still does on Linux, because os/zero_copy_linux.go handles it there and is not affected by the regression).

Metadata

Metadata

Assignees

No one assigned

    Labels

    FrozenDueToAgeNeedsFixThe path to resolution is known, but the work has not been done.

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions