Skip to content

os.Pipe() issue: a function to read a line from STDIN and output the line to STDOUT not able to write a line of size 100MB back to STDOUT  #52777

Closed
@smartgolang

Description

@smartgolang

What version of Go are you using (go version)?

$ go version
go version go1.18.1 darwin/arm64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="arm64"
GOHOSTARCH="arm64"
GOHOSTOS="darwin"

What did you do?

// a function to read a line from STDIN and output the line to STDOUT

func LargeDataLine(r io.Reader, w io.Writer) {
	s := ReadWithReadLine(r)
        w.Write([]byte(s))
}
// Read with Readline function
func read(r *bufio.Reader) ([]byte, error) {
	var (
		isPrefix = true
		err      error
		line, ln []byte
	)
	for isPrefix && err == nil {
		line, isPrefix, err = r.ReadLine()
		ln = append(ln, line...)
	}
	return ln, err
}

func ReadWithReadLine(r io.Reader) string {
	reader := bufio.NewReader(r)
	var line []byte
	var err error
	for {
		line, err = read(reader)
		if err != nil {
			if err == io.EOF {
				break
			}
			log.Fatalf("a real error happened here: %v\n", err)
		}

		// log.Println("readWithReadLine:")
		// log.Println(string(line))
		return string(line)
	}
	return ""
}

Testing:
Temporary redirecting Stdout

	tempStdout := os.Stdout
	r, w, _ := os.Pipe()
	os.Stdout = w
	dbfolder := "../test-data/"
	testdataFile := fmt.Sprintf("%stestdata/testdata_%d.csv", dbfolder, i)
        file, _ := os.Open(testdataFile)
        defer file.Close()

	r := bufio.NewReader(file)
	s := ReadWithReadLine(r)
	LargeDataLine(strings.NewReader(s), os.Stdout)
	w.Close()
		
        out, _ := ioutil.ReadAll(r)
        os.Stdout = tempStdout

	LargeDataLineStr := string(out)

What did you expect to see?

w.Write([]byte(s)) working with a line of hundreds of megabytes.

What did you see instead?

Waiting forever on the following: w.Write([]byte(s))

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions