Skip to content

Use Literal to improve SpooledTemporaryFile #3526

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Dec 5, 2019

Conversation

aloisklink
Copy link
Contributor

Specifies that a SpooledTemporaryFile[bytes] is constructed if we open the file in bytes mode,
and a SpooledTemporaryFile[str] is constructed if we open the file in str mode.

I've also ran black on the stdlib/3/tempfile.pyi file I changed, since CONTRIBUTING.md says that style-fixes are accepted.

I'm using the newly added to mypy python/typing#680 to do this, as @srittau recommended.

Fixes #3360.

Test case:

import tempfile
with tempfile.SpooledTemporaryFile() as defaultmpfile:
    reveal_type(defaultmpfile)

with tempfile.SpooledTemporaryFile(mode="w+b") as bytetmpfile:
    reveal_type(bytetmpfile)

with tempfile.SpooledTemporaryFile(mode="w+") as strtmpfile:
    reveal_type(strtmpfile)

Prints the following with the latest version of mypy, so it looks like everything is working fine.

me@test:~/typeshed/test_mypy_dir(improve-spooled-temporary-file)$ PYTHONPATH="mypy/:${PYTHONPATH}" python3 -m mypy test.py
test.py:3: note: Revealed type is 'tempfile.SpooledTemporaryFile[builtins.bytes]'
test.py:6: note: Revealed type is 'tempfile.SpooledTemporaryFile[builtins.bytes]'
test.py:9: note: Revealed type is 'tempfile.SpooledTemporaryFile[builtins.str]'

Previously, SpooledTemporaryFile was always an AnyStr.
Now, we load a SpooledTemporaryFile[bytes] if we open in bytes mode,
and we load a SpooledTemporaryFile[str] if we open in str mode.
Copy link
Collaborator

@srittau srittau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Two small remarks below.

def __init__(
self: SpooledTemporaryFile[bytes],
max_size: int = ...,
mode: Literal["rb", "wb", "ab", "xb", "r+b", "w+b", "a+b", "x+b"] = ...,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also just "b".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mode "b" and "" both cause errors in Python 2.7 and 3.7. I haven't tested on other Python versions, but I don't see why they would be different.

Trying to instantiate a TemporaryFile (done when a SpooledTemporaryFile rolls over) with a mode string not containing exactly one of rwax and more than one + will raise ValueError: Must have exactly one of create/read/write/append mode and at most one plus on Python 3.7.4.

Python 2.7.16 has a similar error: ValueError: mode string must begin with one of 'r', 'w', 'a' or 'U', not 'b'.

Confusingly, the SpooledTemporaryFile constructor accepts any mode, but an exception is raised only when it creates a TemporaryFile internally. This can be forced by using the rollover() function.

Test case:

import tempfile
# this line works fine
with tempfile.SpooledTemporaryFile(mode="b") as tmpfile:
    # this line fails, since the invalid mode is checked here
    tmpfile.rollover()

def __init__(
self: SpooledTemporaryFile[str],
max_size: int = ...,
mode: Literal["r", "w", "a", "x", "r+", "w+", "a+", "x+", "rt", "wt", "at", "xt", "r+t", "w+t", "a+t", "x+t"] = ...,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also just "".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same error as above on Python 3.7.4, Python 2.7.16 has it's own ValueError: empty mode string error. It is only seen after the SpooledTemporaryFile creates a TemporaryFile internally, which can be forced using ~SpooledTemporaryFile.rollover().

import tempfile

with tempfile.SpooledTemporaryFile(mode="") as tmpfile:
    tmpfile.rollover() # empty mode fails on this line, not at constructor

@aloisklink
Copy link
Contributor Author

Thanks for the review @srittau. I think the two modes you asked me to add are not valid modes. Confusingly, SpooledTemporaryFile only raises an Error about an incorrect modes when it internally calls TemporaryFile, which does not happen during the constructor.

There is a mode U that I haven't added, but it does nothing in Python 3, and the official docs for Python 2 say "should not be used in new code.", since it is replaced by the newline parameter.

Copy link
Collaborator

@srittau srittau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanation. I only tried constructing the file and reading from it, which worked.

@srittau srittau merged commit 4766ca0 into python:master Dec 5, 2019
@aloisklink aloisklink deleted the improve-spooled-temporary-file branch December 9, 2019 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

read of SpooledTemporaryFile returns bytes, mypy assumes str
2 participants