os.Pipe() issue: a function to read a line from STDIN and output the line to STDOUT not able to write a line of size 100MB back to STDOUT

[smartgolang]

What version of Go are you using (go version)?

$ go version
go version go1.18.1 darwin/arm64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="arm64"
GOHOSTARCH="arm64"
GOHOSTOS="darwin"

What did you do?

// a function to read a line from STDIN and output the line to STDOUT

func LargeDataLine(r io.Reader, w io.Writer) {
	s := ReadWithReadLine(r)
        w.Write([]byte(s))
}
// Read with Readline function
func read(r *bufio.Reader) ([]byte, error) {
	var (
		isPrefix = true
		err      error
		line, ln []byte
	)
	for isPrefix && err == nil {
		line, isPrefix, err = r.ReadLine()
		ln = append(ln, line...)
	}
	return ln, err
}

func ReadWithReadLine(r io.Reader) string {
	reader := bufio.NewReader(r)
	var line []byte
	var err error
	for {
		line, err = read(reader)
		if err != nil {
			if err == io.EOF {
				break
			}
			log.Fatalf("a real error happened here: %v\n", err)
		}

		// log.Println("readWithReadLine:")
		// log.Println(string(line))
		return string(line)
	}
	return ""
}

Testing:
Temporary redirecting Stdout

	tempStdout := os.Stdout
	r, w, _ := os.Pipe()
	os.Stdout = w

	dbfolder := "../test-data/"
	testdataFile := fmt.Sprintf("%stestdata/testdata_%d.csv", dbfolder, i)
        file, _ := os.Open(testdataFile)
        defer file.Close()

	r := bufio.NewReader(file)
	s := ReadWithReadLine(r)
	LargeDataLine(strings.NewReader(s), os.Stdout)
	w.Close()
		
        out, _ := ioutil.ReadAll(r)
        os.Stdout = tempStdout

	LargeDataLineStr := string(out)

What did you expect to see?

w.Write([]byte(s)) working with a line of hundreds of megabytes.

What did you see instead?

Waiting forever on the following: w.Write([]byte(s))

[ZekeLu]

Have you read from the pipe?

See https://www.man7.org/linux/man-pages/man7/pipe.7.html

Pipe capacity
A pipe has a limited capacity. If the pipe is full, then a write(2) will block or fail, depending on whether the O_NONBLOCK flag is set

[smartgolang]

My function
func LargeDataLine(r io.Reader, w io.Writer) {
needs to write to STDOUT
I need a way to use the Golang testing framework
here how I'm calling it with the pipe redirection:
LargeDataLine(strings.NewReader(s), os.Stdout)

What you'll recommend in this case?

[ZekeLu]

Sorry, I don't understand your question.

And unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.

For questions please refer to https://github.com/golang/go/wiki/Questions

[seankhliao]

I don't think you've connected both ends of the pipe properly.

Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.

For questions please refer to https://github.com/golang/go/wiki/Questions