Skip to content

gh-129005: Fix buffer expansion in _pyio.FileIO.readall #129541

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

gh-129005: Fix buffer expansion in _pyio.FileIO.readall #129541

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion Lib/_pyio.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
import sys
# Import _thread instead of threading to reduce startup cost
from _thread import allocate_lock as Lock
from itertools import repeat
if sys.platform in {'win32', 'cygwin'}:
from msvcrt import setmode as _setmode
else:
Expand Down Expand Up @@ -1686,7 +1687,8 @@ def readall(self):
if addend < DEFAULT_BUFFER_SIZE:
addend = DEFAULT_BUFFER_SIZE
bufsize += addend
result[bytes_read:bufsize] = b'\0'
result.extend(repeat(0, addend))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just b'\0' * addend?

Copy link
Contributor Author

@cmaloney cmaloney Feb 1, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That uses a lot more memory (And overall goal is to get _io and _pyio to use the same). result is expanded in length by addend bytes, a new bytes is constructed which is of length addend containing just null, then there's a memcpy from the new temporary bytes object to result... That means get 2x addend memory usage, rather than just expanding the buffer by addend bytes.

In this case, bytearray.extend also doesn't prefer working in place / creates a PyByteArray_FromStringAndSize that it copies the data out of a iterator on the object passed to it as an argument...

My preference is definitely bytearray.resize which does what I'd need here, but didn't want that to be a blocker for getting bots back to green (#129560). Reading through the slice, extend, append, etc. code in bytearray haven't found any other efficient ways to just "expand capacity, in place if possible without multiplying existing space" and (ideally) without requiring a by-byte write to every byte (the read loop is about to do that anyways).

assert len(result) == bufsize, "Should have expanded in size"
assert bufsize - bytes_read > 0, "Should always try and read at least one byte"
try:
n = os.readinto(self._fd, memoryview(result)[bytes_read:])
Expand Down
1 change: 1 addition & 0 deletions Misc/NEWS.d/next/Library/2025-01-31-19-07-31.gh-issue-129005.II6go0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
:mod:`!pyio`: Fix expansion of buffer in _pyio.FileIO.readall
Loading