Skip to content

Commit a8741d8

Browse files
authored
Improve stubgen tests (#16760)
Improve test cases around #16486 This PR does not change any actual mypy behavior, only hardens the stubgen tests. The specific changes are: - **dedicated test cases**: The existing pybind11 test cases originate from a pybind11 demo. They cover a specific topic involving geometry types and semi-implemented logic related to it. This somehow distracts from the aspects we are trying to test here from the mypy stubgen perspective, because it hides the actual intent of the bindings. I've simply started adding new test cases that clearly express via their name what the test case is addressing. I've kept the original demo stuff for now, so that the new cases are just an addition (overhead is negligible). - **running mypy on the generated stubs**: In general it is crucial that the output produced by the stubgen can actually be type checked by mypy (this was the regression in #18486). This wasn't covered by the CI check so far. I've added check now, which would have avoided the regression. My goal for follow up PRs would be that we can use `mypy --disallow-untyped-defs` or even `mypy --strict` on the output. - **minor directory restructuring**: So far the expected stub outputs were stored in folder names `stubgen` and `stubgen-include-docs`. This was a bit confusing, because the folder `stubgen` suggested it contains _the_ stubgen (implementation). I went for `expected_stubs_no_docs` and `expected_stubs_with_docs` to make the role of the two folders more explicit. - **minor script bugfix**: Fix a bug in `test-stubgen.sh`: The pre-delete functionality was broken, because the `*` was quoted and therefore did not match.
1 parent 1fd29ac commit a8741d8

File tree

11 files changed

+227
-94
lines changed

11 files changed

+227
-94
lines changed

.github/workflows/test_stubgenc.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Test stubgenc on pybind11-mypy-demo
1+
name: Test stubgenc on pybind11_fixtures
22

33
on:
44
workflow_dispatch:

misc/test-stubgenc.sh

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ cd "$(dirname "$0")/.."
77

88
# Install dependencies, demo project and mypy
99
python -m pip install -r test-requirements.txt
10-
python -m pip install ./test-data/pybind11_mypy_demo
10+
python -m pip install ./test-data/pybind11_fixtures
1111
python -m pip install .
1212

1313
EXIT=0
@@ -17,19 +17,29 @@ EXIT=0
1717
# everything else is passed to stubgen as its arguments
1818
function stubgenc_test() {
1919
# Remove expected stubs and generate new inplace
20-
STUBGEN_OUTPUT_FOLDER=./test-data/pybind11_mypy_demo/$1
21-
rm -rf "${STUBGEN_OUTPUT_FOLDER:?}/*"
20+
STUBGEN_OUTPUT_FOLDER=./test-data/pybind11_fixtures/$1
21+
rm -rf "${STUBGEN_OUTPUT_FOLDER:?}"
22+
2223
stubgen -o "$STUBGEN_OUTPUT_FOLDER" "${@:2}"
2324

25+
# Check if generated stubs can actually be type checked by mypy
26+
if ! mypy "$STUBGEN_OUTPUT_FOLDER";
27+
then
28+
echo "Stubgen test failed, because generated stubs failed to type check."
29+
EXIT=1
30+
fi
31+
2432
# Compare generated stubs to expected ones
2533
if ! git diff --exit-code "$STUBGEN_OUTPUT_FOLDER";
2634
then
35+
echo "Stubgen test failed, because generated stubs differ from expected outputs."
2736
EXIT=1
2837
fi
2938
}
3039

3140
# create stubs without docstrings
32-
stubgenc_test stubgen -p pybind11_mypy_demo
41+
stubgenc_test expected_stubs_no_docs -p pybind11_fixtures
3342
# create stubs with docstrings
34-
stubgenc_test stubgen-include-docs -p pybind11_mypy_demo --include-docstrings
43+
stubgenc_test expected_stubs_with_docs -p pybind11_fixtures --include-docstrings
44+
3545
exit $EXIT
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
import os
2+
from . import demo as demo
3+
from typing import List, Optional, Tuple, overload
4+
5+
class StaticMethods:
6+
def __init__(self, *args, **kwargs) -> None: ...
7+
@overload
8+
@staticmethod
9+
def overloaded_static_method(value: int) -> int: ...
10+
@overload
11+
@staticmethod
12+
def overloaded_static_method(value: float) -> float: ...
13+
@staticmethod
14+
def some_static_method(a: int, b: int) -> int: ...
15+
16+
class TestStruct:
17+
field_readwrite: int
18+
field_readwrite_docstring: int
19+
def __init__(self, *args, **kwargs) -> None: ...
20+
@property
21+
def field_readonly(self) -> int: ...
22+
23+
def func_incomplete_signature(*args, **kwargs): ...
24+
def func_returning_optional() -> Optional[int]: ...
25+
def func_returning_pair() -> Tuple[int, float]: ...
26+
def func_returning_path() -> os.PathLike: ...
27+
def func_returning_vector() -> List[float]: ...

test-data/pybind11_mypy_demo/stubgen/pybind11_mypy_demo/basics.pyi renamed to test-data/pybind11_fixtures/expected_stubs_no_docs/pybind11_fixtures/demo.pyi

Lines changed: 0 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -3,17 +3,6 @@ from typing import ClassVar, List, overload
33
PI: float
44
__version__: str
55

6-
class Foo:
7-
def __init__(self, *args, **kwargs) -> None: ...
8-
@overload
9-
@staticmethod
10-
def overloaded_static_method(value: int) -> int: ...
11-
@overload
12-
@staticmethod
13-
def overloaded_static_method(value: float) -> float: ...
14-
@staticmethod
15-
def some_static_method(a: int, b: int) -> int: ...
16-
176
class Point:
187
class AngleUnit:
198
__members__: ClassVar[dict] = ... # read-only
Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
import os
2+
from . import demo as demo
3+
from typing import List, Optional, Tuple, overload
4+
5+
class StaticMethods:
6+
def __init__(self, *args, **kwargs) -> None:
7+
"""Initialize self. See help(type(self)) for accurate signature."""
8+
@overload
9+
@staticmethod
10+
def overloaded_static_method(value: int) -> int:
11+
"""overloaded_static_method(*args, **kwargs)
12+
Overloaded function.
13+
14+
1. overloaded_static_method(value: int) -> int
15+
16+
2. overloaded_static_method(value: float) -> float
17+
"""
18+
@overload
19+
@staticmethod
20+
def overloaded_static_method(value: float) -> float:
21+
"""overloaded_static_method(*args, **kwargs)
22+
Overloaded function.
23+
24+
1. overloaded_static_method(value: int) -> int
25+
26+
2. overloaded_static_method(value: float) -> float
27+
"""
28+
@staticmethod
29+
def some_static_method(a: int, b: int) -> int:
30+
"""some_static_method(a: int, b: int) -> int
31+
32+
None
33+
"""
34+
35+
class TestStruct:
36+
field_readwrite: int
37+
field_readwrite_docstring: int
38+
def __init__(self, *args, **kwargs) -> None:
39+
"""Initialize self. See help(type(self)) for accurate signature."""
40+
@property
41+
def field_readonly(self) -> int: ...
42+
43+
def func_incomplete_signature(*args, **kwargs):
44+
"""func_incomplete_signature() -> dummy_sub_namespace::HasNoBinding"""
45+
def func_returning_optional() -> Optional[int]:
46+
"""func_returning_optional() -> Optional[int]"""
47+
def func_returning_pair() -> Tuple[int, float]:
48+
"""func_returning_pair() -> Tuple[int, float]"""
49+
def func_returning_path() -> os.PathLike:
50+
"""func_returning_path() -> os.PathLike"""
51+
def func_returning_vector() -> List[float]:
52+
"""func_returning_vector() -> List[float]"""

test-data/pybind11_mypy_demo/stubgen-include-docs/pybind11_mypy_demo/basics.pyi renamed to test-data/pybind11_fixtures/expected_stubs_with_docs/pybind11_fixtures/demo.pyi

Lines changed: 15 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -3,52 +3,22 @@ from typing import ClassVar, List, overload
33
PI: float
44
__version__: str
55

6-
class Foo:
7-
def __init__(self, *args, **kwargs) -> None:
8-
"""Initialize self. See help(type(self)) for accurate signature."""
9-
@overload
10-
@staticmethod
11-
def overloaded_static_method(value: int) -> int:
12-
"""overloaded_static_method(*args, **kwargs)
13-
Overloaded function.
14-
15-
1. overloaded_static_method(value: int) -> int
16-
17-
2. overloaded_static_method(value: float) -> float
18-
"""
19-
@overload
20-
@staticmethod
21-
def overloaded_static_method(value: float) -> float:
22-
"""overloaded_static_method(*args, **kwargs)
23-
Overloaded function.
24-
25-
1. overloaded_static_method(value: int) -> int
26-
27-
2. overloaded_static_method(value: float) -> float
28-
"""
29-
@staticmethod
30-
def some_static_method(a: int, b: int) -> int:
31-
"""some_static_method(a: int, b: int) -> int
32-
33-
None
34-
"""
35-
366
class Point:
377
class AngleUnit:
388
__members__: ClassVar[dict] = ... # read-only
399
__entries: ClassVar[dict] = ...
4010
degree: ClassVar[Point.AngleUnit] = ...
4111
radian: ClassVar[Point.AngleUnit] = ...
4212
def __init__(self, value: int) -> None:
43-
"""__init__(self: pybind11_mypy_demo.basics.Point.AngleUnit, value: int) -> None"""
13+
"""__init__(self: pybind11_fixtures.demo.Point.AngleUnit, value: int) -> None"""
4414
def __eq__(self, other: object) -> bool:
4515
"""__eq__(self: object, other: object) -> bool"""
4616
def __hash__(self) -> int:
4717
"""__hash__(self: object) -> int"""
4818
def __index__(self) -> int:
49-
"""__index__(self: pybind11_mypy_demo.basics.Point.AngleUnit) -> int"""
19+
"""__index__(self: pybind11_fixtures.demo.Point.AngleUnit) -> int"""
5020
def __int__(self) -> int:
51-
"""__int__(self: pybind11_mypy_demo.basics.Point.AngleUnit) -> int"""
21+
"""__int__(self: pybind11_fixtures.demo.Point.AngleUnit) -> int"""
5222
def __ne__(self, other: object) -> bool:
5323
"""__ne__(self: object, other: object) -> bool"""
5424
@property
@@ -63,15 +33,15 @@ class Point:
6333
mm: ClassVar[Point.LengthUnit] = ...
6434
pixel: ClassVar[Point.LengthUnit] = ...
6535
def __init__(self, value: int) -> None:
66-
"""__init__(self: pybind11_mypy_demo.basics.Point.LengthUnit, value: int) -> None"""
36+
"""__init__(self: pybind11_fixtures.demo.Point.LengthUnit, value: int) -> None"""
6737
def __eq__(self, other: object) -> bool:
6838
"""__eq__(self: object, other: object) -> bool"""
6939
def __hash__(self) -> int:
7040
"""__hash__(self: object) -> int"""
7141
def __index__(self) -> int:
72-
"""__index__(self: pybind11_mypy_demo.basics.Point.LengthUnit) -> int"""
42+
"""__index__(self: pybind11_fixtures.demo.Point.LengthUnit) -> int"""
7343
def __int__(self) -> int:
74-
"""__int__(self: pybind11_mypy_demo.basics.Point.LengthUnit) -> int"""
44+
"""__int__(self: pybind11_fixtures.demo.Point.LengthUnit) -> int"""
7545
def __ne__(self, other: object) -> bool:
7646
"""__ne__(self: object, other: object) -> bool"""
7747
@property
@@ -90,38 +60,38 @@ class Point:
9060
"""__init__(*args, **kwargs)
9161
Overloaded function.
9262
93-
1. __init__(self: pybind11_mypy_demo.basics.Point) -> None
63+
1. __init__(self: pybind11_fixtures.demo.Point) -> None
9464
95-
2. __init__(self: pybind11_mypy_demo.basics.Point, x: float, y: float) -> None
65+
2. __init__(self: pybind11_fixtures.demo.Point, x: float, y: float) -> None
9666
"""
9767
@overload
9868
def __init__(self, x: float, y: float) -> None:
9969
"""__init__(*args, **kwargs)
10070
Overloaded function.
10171
102-
1. __init__(self: pybind11_mypy_demo.basics.Point) -> None
72+
1. __init__(self: pybind11_fixtures.demo.Point) -> None
10373
104-
2. __init__(self: pybind11_mypy_demo.basics.Point, x: float, y: float) -> None
74+
2. __init__(self: pybind11_fixtures.demo.Point, x: float, y: float) -> None
10575
"""
10676
def as_list(self) -> List[float]:
107-
"""as_list(self: pybind11_mypy_demo.basics.Point) -> List[float]"""
77+
"""as_list(self: pybind11_fixtures.demo.Point) -> List[float]"""
10878
@overload
10979
def distance_to(self, x: float, y: float) -> float:
11080
"""distance_to(*args, **kwargs)
11181
Overloaded function.
11282
113-
1. distance_to(self: pybind11_mypy_demo.basics.Point, x: float, y: float) -> float
83+
1. distance_to(self: pybind11_fixtures.demo.Point, x: float, y: float) -> float
11484
115-
2. distance_to(self: pybind11_mypy_demo.basics.Point, other: pybind11_mypy_demo.basics.Point) -> float
85+
2. distance_to(self: pybind11_fixtures.demo.Point, other: pybind11_fixtures.demo.Point) -> float
11686
"""
11787
@overload
11888
def distance_to(self, other: Point) -> float:
11989
"""distance_to(*args, **kwargs)
12090
Overloaded function.
12191
122-
1. distance_to(self: pybind11_mypy_demo.basics.Point, x: float, y: float) -> float
92+
1. distance_to(self: pybind11_fixtures.demo.Point, x: float, y: float) -> float
12393
124-
2. distance_to(self: pybind11_mypy_demo.basics.Point, other: pybind11_mypy_demo.basics.Point) -> float
94+
2. distance_to(self: pybind11_fixtures.demo.Point, other: pybind11_fixtures.demo.Point) -> float
12595
"""
12696
@property
12797
def length(self) -> float: ...

test-data/pybind11_mypy_demo/setup.py renamed to test-data/pybind11_fixtures/setup.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,14 +5,14 @@
55
# Documentation: https://pybind11.readthedocs.io/en/stable/compiling.html
66
ext_modules = [
77
Pybind11Extension(
8-
"pybind11_mypy_demo",
8+
"pybind11_fixtures",
99
["src/main.cpp"],
1010
cxx_std=17,
1111
),
1212
]
1313

1414
setup(
15-
name="pybind11-mypy-demo",
15+
name="pybind11_fixtures",
1616
version="0.0.1",
1717
ext_modules=ext_modules,
1818
)

0 commit comments

Comments
 (0)