io: Unable to write more than 65536 (2^16) bytes

[msc24x]

Go version

go version go1.20.11 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/msc24x/.cache/go-build"
GOENV="/home/msc24x/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/msc24x/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/msc24x/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.11"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build510038043=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Trying to write a byte array into a command's input stream (stdinpipe)

Following is the dummy snippet wrote for demonstration, which is running a command, reading a file with over 70,000 bytes.
It remains stuck on write_closer.Write(file_bytes) forever.

Maybe what I'm doing is not the correct way?

Sinppet

package main
import (
	"fmt"
	"os"
	"os/exec"
)

func main() {
	cmd := exec.Command("echo")
	write_closer, err := cmd.StdinPipe()
	if err != nil {
		panic(err.Error())
	}
	file_bytes, err := os.ReadFile("./70kchars.txt")
	if err != nil {
		panic(err.Error())
	}
	fmt.Println("size", len(file_bytes))
	write_closer.Write(file_bytes) # Remains stuck on this line forever
	fmt.Println("written", err)
	write_closer.Close()
}

Output

size 77324
^Csignal: interrupt

What did you see happen?

It was observed that the program remains stuck on line write_closer.Write(file_bytes)

What did you expect to see?

I expected it to write all the bytes.

[gabyhelp]

Related Issues and Documentation

• os.Pipe() issue: a function to read a line from STDIN and output the line to STDOUT not able to write a line of size 100MB back to STDOUT #52777 (closed)
• execution hangs when I try to open named pipe file #58854 (closed)
• io/ioutil hangs with too big output from os/exec stderr and stdout pipes #16787 (closed)
• runtime: infinite loop causing stack overflow in bufio #38481 (closed)
• ioutil: deadlock when WriteFile called with a 5<<30 byte buffer on Windows #31211 (closed)
• File descriptor slice bounds out of range error #30330 (closed)
• ssh: writing more than 655360 bytes to remote stdin intermittently hangs on Wait() #3479 (closed)
• net: goroutine stuck in waitWrite() #66543 (closed)
• os/exec: Stdout/Stderr not closed after data copied #22610 (closed)
• os: File.ReadFrom copy_file_range: bad file descriptor on append only file #60181 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

pipes are not unlimited buffers, and echo does not read from stdin.

Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.

For questions please refer to https://github.com/golang/go/wiki/Questions

[msc24x]

echo was only an example.

Yes, its probably due to the following:
From https://www.man7.org/linux/man-pages/man7/pipe.7.html

Pipe capacity
       A pipe has a limited capacity.  If the pipe is full, then a
       [write(2)](https://www.man7.org/linux/man-pages/man2/write.2.html) will block or fail, depending on whether the O_NONBLOCK
       flag is set (see below).  Different implementations have
       different limits for the pipe capacity.  Applications should not
       rely on a particular capacity: an application should be designed
       so that a reading process consumes data as soon as it is
       available, so that a writing process does not remain blocked.

       Before Linux 2.6.11, the capacity of a pipe was the same as the
       system page size (e.g., 4096 bytes on i386).  Since Linux 2.6.11,
       the pipe capacity is 16 pages (i.e., 65,536 bytes in a system
       with a page size of 4096 bytes).  Since Linux 2.6.35, the default
       pipe capacity is 16 pages, but the capacity can be queried and
       set using the [fcntl(2)](https://www.man7.org/linux/man-pages/man2/fcntl.2.html) F_GETPIPE_SZ and F_SETPIPE_SZ operations.
       See [fcntl(2)](https://www.man7.org/linux/man-pages/man2/fcntl.2.html) for more information.

       The following [ioctl(2)](https://www.man7.org/linux/man-pages/man2/ioctl.2.html) operation, which can be applied to a file
       descriptor that refers to either end of a pipe, places a count of
       the number of unread bytes in the pipe in the int buffer pointed
       to by the final argument of the call:

           ioctl(fd, FIONREAD, &nbytes);

       The FIONREAD operation is not specified in any standard, but is
       provided on many implementations.

What worked for me:

Should have directly set the input stream to a file (*os.File) to approach this situation.
cmd.Stdin = file