Reduce multipart form upload memory usage!

[n0v3xx]

Hi,
the problem with this multipart form implementation is that they need huge amount of RAM. You should include an option to switch to a diffrent method where you can use io.Pipe to stream the file.
Would be great if you can implement this.

[jeevatkm]

@n0v3xx Thank you for sharing your experience. I will look into it when I get a chance.

If you're interested, you can take a stab at it too.

[amarjeetanandsingh]

@jeevatkm
Could you please help me a bit to gather the requirement? Am interested in taking up this task.

As far as I can understand, we need to implement an alternate of SetMultiValueFormData(params url.Values) method so that we can stream the values of multipart form data.
Since SetMultiValueFormData() takes url.Values(which is a map of string, string) as input, it stores all the data in memory that we need to escape that.
Am i correct?

[jeevatkm]

@amarjeetanandsingh This issue created/reported to propose to use io.Pipe while processing multipart files to reduce memory usage during a request.

[bartzz]

Can confirm this is an issue

(pprof) top
Showing nodes accounting for 781.27MB, 100% of 781.27MB total
Showing top 10 nodes out of 19
flat flat% sum% cum cum%
781.27MB 100% 100% 781.27MB 100% bytes.makeSlice
0 0% 100% 195.24MB 24.99% bytes.(*Buffer).Grow
0 0% 100% 586.02MB 75.01% bytes.(*Buffer).Write
0 0% 100% 781.27MB 100% bytes.(*Buffer).grow
0 0% 100% 586.02MB 75.01% bytes.(*Reader).WriteTo
0 0% 100% 586.02MB 75.01% gopkg.in/resty%2ev1.(*Client).execute
0 0% 100% 586.02MB 75.01% gopkg.in/resty%2ev1.(*Request).Execute
0 0% 100% 586.02MB 75.01% gopkg.in/resty%2ev1.(*Request).Post
0 0% 100% 586.02MB 75.01% gopkg.in/resty%2ev1.addFileReader
0 0% 100% 586.02MB 75.01% gopkg.in/resty%2ev1.handleMultipart
(pprof)

[jeevatkm]

@bartzz Thanks for sharing profiling info. Yeah, its good to optimize memory usage for multipart file upload follow.
BTW, I noticed, you're using resty v1, I would recommend you to upgrade it to v2 😄

[segevda]

Hi @jeevatkm , this might be a good candidate for V3, wdyt?

[jeevatkm]

Agreed @segevda