Update `open`, `os.fspath`, `os.fsencode`, `os.fsdecode`, `pathlib.PurePath` and `pathlib.Path` stubs due to PEP-519 #991

sproshev · 2017-03-13T15:50:11Z

Update stubs for:

builtin open function
os.path, os.fsencode and os.fsdecode functions
pathlib.PurePath and pathlib.Path classes

due to PEP-519.

… due to PEP-519.

… PEP-519.

JelleZijlstra · 2017-03-14T02:05:00Z

This change looks good to me but there's a lot more to be done in relation to PEP 519.

First, many additional functions should be updated to accept os.PathLike in addition to str for paths. We should maybe add a type alias (typing.Path?) for Union[str, os.PathLike] to typing, because otherwise we'll have to put a lot of version checks for 3.6 in various places.

Also, os.PathLike should perhaps be generic over AnyStr to express the bytes/str situation more precisely. For example, the Path constructor only accepts os.PathLike[str], not os.PathLike[bytes]. On the other hand, most use cases for Paths probably don't care about bytes paths, so maybe we don't need this complication.

Anyway, those issues are outside the scope of this PR. I'll think about them more and maybe write up a proposal.

JelleZijlstra · 2017-03-14T02:05:57Z

stdlib/3/os/__init__.pyi

-def fsencode(filename: str) -> bytes: ...
-def fsdecode(filename: bytes) -> str: ...
+if sys.version_info >= (3, 6):
+    def fsencode(filename: Union[str, PathLike]) -> bytes: ...


It actually also accepts bytes, and this is documented at https://docs.python.org/3/library/os.html#os.fsencode.

JelleZijlstra · 2017-03-14T02:08:17Z

stdlib/3.4/pathlib.pyi

-                    **kwargs: Any) -> _P: ...
+        if sys.version_info < (3, 6):
+            def __new__(cls: Type[_P], *args: Union[str, PurePath],
+                        **kwargs: Any) -> _P: ...


Interesting, apparently Path takes arbitrary kwargs and then ignores them. This doesn't appear to be documented. Do you know why this is? I might open a bug against CPython about this.

I don't know. I think it is better to open a bug.

http://bugs.python.org/issue29847

Need python/typeshed#991 for proper checking

…de` (+str).

JelleZijlstra · 2017-03-21T05:15:30Z

stdlib/3/os/__init__.pyi

+    @overload
+    def fspath(path: bytes) -> bytes: ...
+    @overload
+    def fspath(path: PathLike) -> Union[str, bytes]: ...


I'd prefer to have this overload return Any since Union return types are troublesome. Sorry for not noticing this before.

…rn types are troublesome

JelleZijlstra · 2017-03-21T14:38:57Z

Thanks!

gvanrossum · 2017-03-21T17:44:28Z

Sadly I have to revert one part of this PR -- the changes to builtins.pyi introduced a new element in mypy's "bootstrap" import cycle (by adding a from os import PathLike to it). I will rewrite it as follows:

define _PathLike in builtins
use that in the definition of open()
import that as PathLike into builtins

All that only for Python 3.6.

This reverts commit 48b271c. See #991 (comment) for details; basically this disturbed mypy's bootstrap import cycle.

JelleZijlstra · 2017-03-21T18:32:53Z

Oh no. :( Why didn't Travis catch this?

gvanrossum · 2017-03-21T18:41:10Z

Good question. I think the travis mypy test script runs mypy with a whole bunch of filenames (of stub files) as arguments, and that may force a different processing order. Maybe we should add a plain mypy -c pass run?

ambv · 2017-03-21T20:59:55Z

Yeah, it would be totally worth it to run mypy's own testsuite against typeshed. There are two risks:

is mypy master stable enough for this exercise? I think it's decently stable, most of issues would come from chicken-and-egg problems when support needs to be added on both ends.
are mypy tests fast enough? I think they're decently fast, we've done some work on them.

This would make the life of Dropboxers easier by providing early warnings if something would fail when syncing the submodule.

gvanrossum · 2017-03-21T21:12:08Z

Can either of you create a new issue for this? I don't like new discussions under closed issues/PRs if it's not directly "this broke X".

Summary: python/typeshed#991 (comment) Reviewed By: sinancepel Differential Revision: D8335824 fbshipit-source-id: 7926319534b9e0bb8039d7786a1dc56a9da2eeb5

sproshev added 3 commits March 13, 2017 18:33

Update stub for builtin open function due to PEP-519.

48b271c

Update stubs for os.path, os.fsencode and os.fsdecode functions…

1086038

… due to PEP-519.

Update stubs for pathlib.PurePath and pathlib.Path classes due to…

e4fdd0e

… PEP-519.

JelleZijlstra reviewed Mar 14, 2017

View reviewed changes

yan12125 added a commit to yan12125/python3-android that referenced this pull request Mar 18, 2017

pybuild: Fix some mypy errors

6b2d11a

Need python/typeshed#991 for proper checking

Update input parameter type for os.fsencode (+bytes) and `os.fsdeco…

99a1afd

…de` (+str).

JelleZijlstra approved these changes Mar 18, 2017

View reviewed changes

sproshev changed the title ~~PEP-519~~ Update open, os.fspath, os.fsencode, os.fsdecode, pathlib.PurePath and pathlib.Path stubs due to PEP-519 Mar 18, 2017

JelleZijlstra mentioned this pull request Mar 19, 2017

os.open() can take Path objects in Python 3.6 #1030

Closed

JelleZijlstra reviewed Mar 21, 2017

View reviewed changes

Update os.fspath(path: PathLike) to return Any since Union retu…

89de36a

…rn types are troublesome

JelleZijlstra merged commit 25464cf into python:master Mar 21, 2017

sproshev deleted the sproshev/pep-519 branch March 21, 2017 15:58

gvanrossum pushed a commit that referenced this pull request Mar 21, 2017

Revert "Update stub for builtin open function due to PEP-519."

baa61a1

This reverts commit 48b271c. See #991 (comment) for details; basically this disturbed mypy's bootstrap import cycle.

gvanrossum mentioned this pull request Mar 21, 2017

Add PathLike to typing python/typing#402

Closed

meliache mentioned this pull request Jun 28, 2020

Allow hybrid_config type to be any pathlike, e.g. pathlib.Path b2-hive/eFFORT#11

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update `open`, `os.fspath`, `os.fsencode`, `os.fsdecode`, `pathlib.PurePath` and `pathlib.Path` stubs due to PEP-519 #991

Update `open`, `os.fspath`, `os.fsencode`, `os.fsdecode`, `pathlib.PurePath` and `pathlib.Path` stubs due to PEP-519 #991

sproshev commented Mar 13, 2017

JelleZijlstra commented Mar 14, 2017

JelleZijlstra Mar 14, 2017

sproshev Mar 18, 2017

JelleZijlstra Mar 14, 2017

sproshev Mar 18, 2017

JelleZijlstra Mar 18, 2017

JelleZijlstra Mar 21, 2017

sproshev Mar 21, 2017

JelleZijlstra commented Mar 21, 2017

gvanrossum commented Mar 21, 2017

JelleZijlstra commented Mar 21, 2017

gvanrossum commented Mar 21, 2017

ambv commented Mar 21, 2017

gvanrossum commented Mar 21, 2017

Update open, os.fspath, os.fsencode, os.fsdecode, pathlib.PurePath and pathlib.Path stubs due to PEP-519 #991

Update open, os.fspath, os.fsencode, os.fsdecode, pathlib.PurePath and pathlib.Path stubs due to PEP-519 #991

Conversation

sproshev commented Mar 13, 2017

JelleZijlstra commented Mar 14, 2017

JelleZijlstra Mar 14, 2017

Choose a reason for hiding this comment

sproshev Mar 18, 2017

Choose a reason for hiding this comment

JelleZijlstra Mar 14, 2017

Choose a reason for hiding this comment

sproshev Mar 18, 2017

Choose a reason for hiding this comment

JelleZijlstra Mar 18, 2017

Choose a reason for hiding this comment

JelleZijlstra Mar 21, 2017

Choose a reason for hiding this comment

sproshev Mar 21, 2017

Choose a reason for hiding this comment

JelleZijlstra commented Mar 21, 2017

gvanrossum commented Mar 21, 2017

JelleZijlstra commented Mar 21, 2017

gvanrossum commented Mar 21, 2017

ambv commented Mar 21, 2017

gvanrossum commented Mar 21, 2017

Update `open`, `os.fspath`, `os.fsencode`, `os.fsdecode`, `pathlib.PurePath` and `pathlib.Path` stubs due to PEP-519 #991

Update `open`, `os.fspath`, `os.fsencode`, `os.fsdecode`, `pathlib.PurePath` and `pathlib.Path` stubs due to PEP-519 #991