Unbounded growth in SpooledTemporaryFile.writelines()

[bertptrs]

Bug report

Bug description:

SpooledTemporaryFile provides a temporary file backed by a buffer in memory that spills over to disk when it gets too large. However, the writelines() method only checks whether it should roll over after the entire lines iterator is exhausted. This causes unexpectedly high memory use when feeding it large iterators.

from tempfile import SpooledTemporaryFile

with SpooledTemporaryFile(mode="w", max_size=1024) as temp:
    temp.writelines(map(str, range(1000)))

With the above code, one might expect that the buffer doesn't grow (much) past 1024 bytes, but it grows to almost three times that size before finally rolling over.

CPython versions tested on:

3.12, 3.13, CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-127371 Avoid unbounded growth SpooledTempfile.writelines #127372
• gh-127371: Fix high memory usage in SpooledTemporaryFile.writelines() when handling large iterators #127378
• [3.12] gh-127371 Avoid unbounded growth SpooledTempfile.writelines (GH-127372) #130885
• [3.13] gh-127371 Avoid unbounded growth SpooledTempfile.writelines (GH-127372) #130886

[serhiy-storchaka]

Thank you for your report and PR @bertptrs. I do not think that it is worth to backport this change, it looks more as a new feature (minor improvement of memory use).

[bertptrs]

@serhiy-storchaka I personally do think is better to backport, as it is a security issue. I discovered this bug because of a DoS issue in our code base. The developer who wrote the original code reasonably assumed that he had set a buffer size limit, while he effectively hadn't.

How much effort is it to backport this?

[serhiy-storchaka]

I'm not strictly against it, I just have doubts. I left this on the release manager, @Yhg1s.

[Yhg1s]

Backporting the fix makes sense to me.

[bertptrs]

Thanks for backporting! Looking forward to seeing these changes drop in a release.