Running Python in unbuffered mode may not write all contents to the console

[fabioz]

BPO	42044
Nosy	@fabioz, @eryksun
PRs	gh-86210: Write all bytes to the console in unbuffered mode on Windows #26678

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-10-15.13:22:04.781>
labels = ['type-bug', '3.8', 'expert-IO']
title = 'Running Python in unbuffered mode may not write all contents to the console'
updated_at = <Date 2021-06-11.18:26:14.867>
user = 'https://github.com/fabioz'

bugs.python.org fields:

activity = <Date 2021-06-11.18:26:14.867>
actor = 'fabioz'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['IO']
creation = <Date 2020-10-15.13:22:04.781>
creator = 'fabioz'
dependencies = []
files = []
hgrepos = []
issue_num = 42044
keywords = ['patch']
message_count = 3.0
messages = ['378684', '378693', '395654']
nosy_count = 2.0
nosy_names = ['fabioz', 'eryksun']
pr_nums = ['26678']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue42044'
versions = ['Python 3.8']

[fabioz]

When running Python in unbuffered mode it may fail to write all the contents to the actual console (on Windows).

The code below can reproduce the issue:

import sys
s = ''
for i in range(1,301):
    s += f"{str(i*100).zfill(10)}{'x' * 89}\n"

sys.stdout.write(s)

When calling it with python -u code.py it'll write only up to line 15000 and when calling it with python code.py it'll write up to line 30000.

This fails because in _textiowrapper_writeflush it doesn't verify if all the contents have been indeed written and thus fails in a partial write. In buffered mode it works because _io_BufferedWriter_write_impl does the job properly.

I'm a bit uncertain on why doesn't _io__WindowsConsoleIO_write_impl itself do the loop to write everything instead of leaving it up to callers to do that work (apparently due to bpo-11395 it says that it only writes partially, but maybe the fix could've been to loop inside of _io__WindowsConsoleIO_write_impl to write everything instead of expecting callers to handle partial writes...

[eryksun]

Text mode without a buffer isn't reliable. That said, Python 3.9 no longer supports Windows 7, so it can remove the 32 KiB limit on console I/O files.

The size limit in Windows 7 and earlier is due to the LPC-based pseudo-files that it uses for I/O. Under the hood, console pseudo-files use a 64 KiB heap that's shared between the console host process and client processes.

In Windows 8+, console files are real files that use the ConDrv device instead of LPC, in which case there is practically no limit on the size of buffers that can be read and written.

[fabioz]

Seems fair. I just did a pull request to remove those limits.

Please let me know if you think something else is needed there.