use a consistent part size so that ETags are predictable

[andrewrk]

S3 uses this algorithm for ETags for multipart uploads:

1. Compute the MD5 sum of each part
2. Compute the MD5 sum of the concatenation of the MD5 sum digests of each part
3. Digest of that + '-' + part count

Sadly this means that when you want to check a local file to see if it matches an S3 object that was uploaded via multipart, you must also know what size each of the parts are, and this information is not available in S3 metadata.

One way to mitigate this problem is to use consistent part sizes when uploading files. For example, if I set maxPartSize to 5MB, then each part uploaded to S3 should be exactly 5MB, except for the last one. Currently the code flushes a part when the part size is slightly above maxPartSize. This makes it impossible to do client side ETag calculation.

Note that s3cmd, which by default does 15MB multipart uploads, has behavior like I am describing where each part is exactly 15MB (except for the last one).

[nathanpeck]

Good catch. You are correct, right now I'm using maxPartSize as a minimum threshold for flushing instead of a strict limit. I will make this change so that it uploads in parts that are exactly maxPartSize in length, as it will also allow me to add some features that I want such as better parallel uploading of multiple parts simultaneously.

[nathanpeck]

This has been fixed in version 0.5.0

Let me know if you experience any issues.

[andrewrk]

Cheers!

BTW I'm not sure if this is useful to you but maybe it is: https://github.com/andrewrk/node-fd-slicer

The idea is that you can create multiple read streams from different slices in a file with the same fd. Looks like your code does something similar.

(edit) oops, never mind. I forgot the whole point of this module was using a stream instead of a file.

[nathanpeck]

That is very, very interesting. I'll keep it in mind if I ever need it. I want this particular module to be read stream agnostic so I don't think I would use such a technique in this module as it only applies to files.

But that definitely seems great for some use cases.