optimalPartInfo() default partSize results in running out of memory

[sb10]

Like #404, I'm streaming data which could be over 1GB and trying to use PutObjectStreaming(), but this results in optimalPartInfo(-1) being called and a partSize of 603979776 being returned.

The problem I face is that this partSize is used as, essentially, the size of the read buffer, and I'm running out of memory (I end up with errors like fork/exec ...: cannot allocate memory after the stream completes).

I've tried setting partSize = minPartSize in optimalPartInfo() when the input size is -1, and this solves my memory issue.

Regardless of exactly how it's done, can we get (or choose) a reasonable sized read buffer during PutObjectStreaming() so that we can stream in constant small amount of memory?

[sb10]

(In my test doing minPartSize chunks also resulted in greater speed, so I'm not sure why larger sizes are considered more optimal?)

[harshavardhana]

You should always make sure that the size of the stream is known for AWS S3 APIs to work in optimal manner. If not we fallback to a buffering technique where maximum object chosen is 5TB (lack of knowing file size) - If you do know the file size wrap it around into a struct with Len() function then optimalPartInfo() will calculate the right value @sb10 .

So this is an expected behavior. Also the buffering technique only uses 576MB re-usable buffer are you saying you don't have 1GB > memory on the system where you are using this?

[sb10]

@harshavardhana I don't know the size of the file beforehand; the use case here is creating data that is too big to fit on the local hard drive and instead directly streaming it to S3.

From my brief reading of the code a new ~600MB buffer is created for every call to PutObjectStreaming(), so if someone wanted to stream multiple files simultaneously they'd run out of memory even on large memory systems.

But I'm running out of memory streaming just 1 file on a 4GB system. According to free -h I go from 68MB used to 2.7GB used (~900MB available) during the stream of a 1GB file. I run out of memory during a subsequent unrelated call. This may be due to being forced to buffer the writes while waiting for minio-go to read, combined with the data generator holding an input buffer of 1GB in my test case. But this change to minio-go fixes the problem:

$ git diff upstream/master
diff --git a/api-put-object-common.go b/api-put-object-common.go
index 213fc21..fb26417 100644
--- a/api-put-object-common.go
+++ b/api-put-object-common.go
@@ -56,19 +56,23 @@ func isReadAt(reader io.Reader) (ok bool) {
 //  maxMultipartPutObjectSize - 5TiB
 //
 func optimalPartInfo(objectSize int64) (totalPartsCount int, partSize int64, lastPartSize int64, err error) {
-       // object size is '-1' set it to 5TiB.
-       if objectSize == -1 {
-               objectSize = maxMultipartPutObjectSize
-       }
        // object size is larger than supported maximum.
        if objectSize > maxMultipartPutObjectSize {
                err = ErrEntityTooLarge(objectSize, maxMultipartPutObjectSize, "", "")
                return
        }
-       // Use floats for part size for all calculations to avoid
-       // overflows during float64 to int64 conversions.
-       partSizeFlt := math.Ceil(float64(objectSize / maxPartsCount))
-       partSizeFlt = math.Ceil(partSizeFlt/minPartSize) * minPartSize
+       var partSizeFlt float64
+       if objectSize == -1 {
+               // object size is '-1' set it to 5TiB and make our part size the minimum
+               // size, so that our read buffer isn't unnecessarily large
+               partSizeFlt = minPartSize
+               objectSize = maxMultipartPutObjectSize
+       } else {
+               // Use floats for part size for all calculations to avoid
+               // overflows during float64 to int64 conversions.
+               partSizeFlt = math.Ceil(float64(objectSize / maxPartsCount))
+               partSizeFlt = math.Ceil(partSizeFlt/minPartSize) * minPartSize
+       }
        // Total parts count.
        totalPartsCount = int(math.Ceil(float64(objectSize) / partSizeFlt))
        // Part size.

Now, during the stream of a 1GB file I go from 68MB used to 1.3GB used (2.3GB available) during the stream, and don't run out of memory.

(If it makes any difference, I'm using io.Pipe() to connect the io.Writer of my data generation to the io.Reader of PutObjectStreaming().)

[harshavardhana]

@harshavardhana I don't know the size of the file beforehand; the use case here is creating data that is too big to fit on the local hard drive and instead directly streaming it to S3.

From my brief reading of the code a new ~600MB buffer is created for every call to PutObjectStreaming(), so if someone wanted to stream multiple files simultaneously they'd run out of memory even on large memory systems.

But I'm running out of memory streaming just 1 file on a 4GB system. According to free -h I go from 68MB used to 2.7GB used (~900MB available) during the stream of a 1GB file. I run out of memory during a subsequent unrelated call. This may be due to being forced to buffer the writes while waiting for minio-go to read, combined with the data generator holding an input buffer of 1GB in my test case. But this change to minio-go fixes the problem:

It is incorrect to assume a lower size just to make sure that it fits into your memory. I agree that the crash is incorrect and at max we should only be using 600MB max. Unfortunately on AWS S3 it is not possible to upload a file for which content length is not known. So we optimize based on available information to provide a best effort mechanism.

From my brief reading of the code a new ~600MB buffer is created for every call to PutObjectStreaming(), so if someone wanted to stream multiple files simultaneously they'd run out of memory even on large memory systems.

I see now you are right and it should be avoided, there should be a fixed buffer which should be re-used. The problem is caused by usage of bytes.Buffer which does 2x allocation growth. The code can be modified to have a capped behavior. On the other hand reducing the optimal part size is not acceptable solution.

[harshavardhana]

Once this PR is merged #728 will look at the memory problems of choosing a 600MB part size. Avoiding bytes.Buffer should solve this problem will keep you posted @sb10