Expose the length arg from shutil.copyfileobj for public use

[tgoodlet]

BPO	29659
Nosy	@rhettinger, @terryjreedy, @methane, @serhiy-storchaka, @tgoodlet
PRs	bpo-29659: Expose copyfileobj() length arg for public use #328

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-02-25.09:27:42.340>
created_at = <Date 2017-02-27.00:57:24.705>
labels = ['3.8', 'type-feature', 'library']
title = 'Expose the `length` arg from shutil.copyfileobj for public use'
updated_at = <Date 2019-02-25.09:27:42.339>
user = 'https://github.com/tgoodlet'

bugs.python.org fields:

activity = <Date 2019-02-25.09:27:42.339>
actor = 'methane'
assignee = 'none'
closed = True
closed_date = <Date 2019-02-25.09:27:42.340>
closer = 'methane'
components = ['Library (Lib)']
creation = <Date 2017-02-27.00:57:24.705>
creator = 'goodboy'
dependencies = []
files = []
hgrepos = []
issue_num = 29659
keywords = []
message_count = 5.0
messages = ['288616', '288962', '288974', '305625', '336504']
nosy_count = 5.0
nosy_names = ['rhettinger', 'terry.reedy', 'methane', 'serhiy.storchaka', 'goodboy']
pr_nums = ['328']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue29659'
versions = ['Python 3.8']

[tgoodlet]

It would be handy to be able to pass the length default argument defined by shutil.copyfileobj() to other public functions througout the module. When copying very large files (1GB +), increasing the memory buffer can divide the copy time in half and currently the only way to accomplish this is by overriding the needed function(s) which call shutil.copyfileobj().

I propose a simple non-invasive change where the following functions will also expose the length kwarg and pass it downwards to copyfileobj:

• copyfile
• copy
• copy2

[rhettinger]

This seems reasonable to me.

[methane]

How about increasing default value to 32KiB or 64KiB too?

binutils's cp works well for most cases, while it doesn't have option to specify blocksize.
I want Python's copy functions works nice enough for common cases too.

binutils cp uses 128KiB block for normal file. see more information in https://github.com/coreutils/coreutils/blob/master/src/ioblksize.h

But copyfileobj can be used other file-likes.
On Linux, typical socket buffer size is 128KiB, and typical pipe buffer size is 64KiB. So block size larger than 64KiB may cause unneeded blocking.
That's why I suggested 32KiB or 64KiB.

[serhiy-storchaka]

Could you please provide any benchmarks Tyler?

[methane]

bpo-33671 fixed this.