Improve annotations for various set methods

[AlexWaygood]

• Fixes some regressions introduced by Improve in-place BinOp methods for sets #7161 (see Annotations for set.__sub__ and related methods are inconsistent #9004 (comment)).
• Fixes Annotations for set.__sub__ and related methods are inconsistent #9004
• Fixes set[str].discard(str | None) #7121
• Fixes mypy doesn't understand set operations on optional mypy#13192 (cc. @finite-state-machine).
  • I wasn't initially sure that set[str | None] - set[None] came up often enough to be worth special-casing, but the activity on that issue has persuaded me otherwise.

It's a bit unfortunate how many type: ignores we have to introduce here, but I can't see a good way round them. I've added some test cases to make sure everything's working correctly.

[AlexWaygood]

Primer analysis:

• ibis is just an error message changing slightly

• werkzeug would just need to change their annotations on their subclass to match our new annotation for discard()

• The three pydantic hits are all new false positives caused by making set.__sub__ stricter.

  (You could argue that the third one is, strictly speaking, a true positive, in that it's working the way we "want" -- mypy is warning that there's a possibility AbstractSet[int] might be being subtracted from AbstractSet[str], which would be a no-op. But it doesn't seem like a "helpful" true positive. You'd have to do some more work to narrow the type before doing the subtraction, but it wouldn't really make your code any more type-safe.)

  These seem pretty unfortunate to me, and make me think that the better way of fixing Annotations for set.__sub__ and related methods are inconsistent #9004 is to make it so that __sub__, __isub__, difference and difference_update all accept AbstractSet[object] or Iterable[Any] rather than AbstractSet[_T | None]. @Akuli, thoughts on that?

  • https://github.com/pydantic/pydantic/blob/832225b90672c68e2d4067bd0ffecf834d62b48b/pydantic/v1/main.py#L901
  • https://github.com/pydantic/pydantic/blob/832225b90672c68e2d4067bd0ffecf834d62b48b/pydantic/main.py#L1257
  • https://github.com/pydantic/pydantic/blob/832225b90672c68e2d4067bd0ffecf834d62b48b/pydantic/deprecated/copy_internals.py#L222

• The new jax hit looks like a false positive, probably due to the fact that set.__sub__ is now an overloaded method. https://github.com/google/jax/blob/9352a00dab3c465a4e2c651cc6a0fa91e50ae7a2/jax/_src/lax/parallel.py#L512

[srittau]

This has conflicts now.

[AlexWaygood]

This has conflicts now.

I fixed the merge conflicts, but it's still not clear to me whether we want to fix the inconsistency between the different methods by making some annotations stricter (as this PR does), which seems to lead to false positives when running type checkers on user code. I think a better solution might be to make the methods consistent by making some annotations looser instead -- what do you think?

[Bruntaz]

I believe the correct signature for this should be def discard(self, element: Hashable, /) -> None: ....

set.discard will only raise an exception if an unhashable type is passed into it. Any other input is valid (just if it doesn't match _T, it will do nothing instead of raising like set.remove():

>>> a = {1, 2, 3}
>>> a.discard(1)
>>> a
{2, 3}
>>> a.discard("Hello")
>>> a
{2, 3}
>>> a.discard([])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

pydantic (https://github.com/pydantic/pydantic)
+ pydantic/v1/main.py:901: error: Set comprehension has incompatible type Set[int | str]; expected Set[None]  [misc]
+ pydantic/main.py:1474: error: Argument 1 to "set" has incompatible type "AbstractSet[int] | AbstractSet[str] | Mapping[int, Any] | Mapping[str, Any]"; expected "Iterable[str | None]"  [arg-type]
+ pydantic/deprecated/copy_internals.py:222: error: Set comprehension has incompatible type Set[int | str]; expected Set[str | None]  [misc]

ibis (https://github.com/ibis-project/ibis)
- ibis/backends/sql/datatypes.py:1263: error: Unsupported operand types for - ("set[type[Never]]" and "set[type[SqlglotType]]")  [operator]
+ ibis/backends/sql/datatypes.py:1263: error: Argument 1 to <set> has incompatible type "type[SqlglotType]"; expected "None"  [arg-type]
+ ibis/backends/sql/datatypes.py:1263: error: Argument 2 to <set> has incompatible type "type[BigQueryUDFType]"; expected "None"  [arg-type]
- ibis/expr/datatypes/tests/test_core.py:548: error: Unsupported operand types for - ("set[type[Never]]" and "set[AnnotableMeta]")  [operator]
+ ibis/expr/datatypes/tests/test_core.py:550: error: Argument 1 to <set> has incompatible type "type[Array[T]]"; expected "None"  [arg-type]
+ ibis/expr/datatypes/tests/test_core.py:551: error: Argument 2 to <set> has incompatible type "type[String]"; expected "None"  [arg-type]
+ ibis/expr/datatypes/tests/test_core.py:552: error: Argument 3 to <set> has incompatible type "type[Floating]"; expected "None"  [arg-type]
+ ibis/expr/datatypes/tests/test_core.py:553: error: Argument 4 to <set> has incompatible type "type[GeoSpatial]"; expected "None"  [arg-type]
+ ibis/expr/datatypes/tests/test_core.py:554: error: Argument 5 to <set> has incompatible type "type[Integer]"; expected "None"  [arg-type]
+ ibis/expr/datatypes/tests/test_core.py:555: error: Argument 6 to <set> has incompatible type "type[Map[K, V]]"; expected "None"  [arg-type]
+ ibis/expr/datatypes/tests/test_core.py:556: error: Argument 7 to <set> has incompatible type "type[Numeric]"; expected "None"  [arg-type]
+ ibis/expr/datatypes/tests/test_core.py:557: error: Argument 8 to <set> has incompatible type "type[Primitive]"; expected "None"  [arg-type]
+ ibis/expr/datatypes/tests/test_core.py:558: error: Argument 9 to <set> has incompatible type "type[SignedInteger]"; expected "None"  [arg-type]
+ ibis/expr/datatypes/tests/test_core.py:559: error: Argument 10 to <set> has incompatible type "type[Struct]"; expected "None"  [arg-type]
+ ibis/expr/datatypes/tests/test_core.py:560: error: Argument 11 to <set> has incompatible type "type[Temporal]"; expected "None"  [arg-type]
+ ibis/expr/datatypes/tests/test_core.py:561: error: Argument 12 to <set> has incompatible type "type[UnsignedInteger]"; expected "None"  [arg-type]
+ ibis/expr/datatypes/tests/test_core.py:562: error: Argument 13 to <set> has incompatible type "type[Variadic]"; expected "None"  [arg-type]
+ ibis/expr/datatypes/tests/test_core.py:563: error: Argument 14 to <set> has incompatible type "type[Parametric]"; expected "None"  [arg-type]
+ ibis/expr/datatypes/tests/test_core.py:564: error: Argument 15 to <set> has incompatible type "type[Interval]"; expected "None"  [arg-type]

werkzeug (https://github.com/pallets/werkzeug)
+ src/werkzeug/datastructures/structures.py:1121: error: Argument 1 of "discard" is incompatible with supertype "MutableSet"; supertype defines the argument type as "Optional[str]"  [override]
+ src/werkzeug/datastructures/structures.py:1121: note: This violates the Liskov substitution principle
+ src/werkzeug/datastructures/structures.py:1121: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides

xarray (https://github.com/pydata/xarray)
+ xarray/core/utils.py: note: In member "discard" of class "OrderedSet":
+ xarray/core/utils.py:574: error: Argument 1 of "discard" is incompatible with supertype "MutableSet"; supertype defines the argument type as "T | None"  [override]
+ xarray/core/utils.py:574: note: This violates the Liskov substitution principle
+ xarray/core/utils.py:574: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides

jax (https://github.com/google/jax)
+ jax/_src/sharding_impls.py:107: error: Need type annotation for "remaining_axes" (hint: "remaining_axes: set[<type>] = ...")  [var-annotated]

graphql-core (https://github.com/graphql-python/graphql-core)
+ src/graphql/pyutils/ref_set.py:58: error: Argument 1 of "discard" is incompatible with supertype "MutableSet"; supertype defines the argument type as "T | None"  [override]
+ src/graphql/pyutils/ref_set.py:58: note: This violates the Liskov substitution principle
+ src/graphql/pyutils/ref_set.py:58: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides

schemathesis (https://github.com/schemathesis/schemathesis)
- src/schemathesis/cli/commands/run/hypothesis.py:35: error: Argument 1 to "set" has incompatible type "list[schemathesis.cli.commands.run.hypothesis.Phase]"; expected "Iterable[Optional[hypothesis._settings.Phase]]"  [arg-type]
+ src/schemathesis/cli/commands/run/hypothesis.py:35: error: Argument 1 to "set" has incompatible type "list[Phase]"; expected "Iterable[None]"  [arg-type]

django-stubs (https://github.com/typeddjango/django-stubs)
+ django-stubs/utils/datastructures.pyi:56: error: Argument 1 of "discard" is incompatible with supertype "MutableSet"; supertype defines the argument type as "Optional[_K]"  [override]
+ django-stubs/utils/datastructures.pyi:56: note: This violates the Liskov substitution principle
+ django-stubs/utils/datastructures.pyi:56: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides

core (https://github.com/home-assistant/core)
+ homeassistant/core_config.py:530: error: Argument 1 of "discard" is incompatible with supertype "set"; supertype defines the argument type as "str | None"  [override]
+ homeassistant/core_config.py:530: note: This violates the Liskov substitution principle
+ homeassistant/core_config.py:530: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
+ homeassistant/core_config.py:530: error: Argument 1 of "discard" is incompatible with supertype "MutableSet"; supertype defines the argument type as "str | None"  [override]

[srittau]

While I'm not terribly worried about the new LSP warnings, this PR doesn't seem to fix any actual problems, only introduce new ones – at least not according to primer. I'm -0 on this solution (at least for now).

[ferdnyc]

The number of primer changes/additions where the expected type has demonstrably worsened is concerning.

schemathesis

Take the 'schemathesis' diff:

Before

error: Argument 1 to "set" has incompatible type "list[schemathesis.cli.commands.run.hypothesis.Phase]"; expected "Iterable[Optional[hypothesis._settings.Phase]]" [arg-type]

After

error: Argument 1 to "set" has incompatible type "list[Phase]"; expected "Iterable[None]" [arg-type]

Something's just plain wrong, if the expected type is Iterable[None].

pydantic

Same thing in the first new 'pydantic' message:

error: Set comprehension has incompatible type Set[int | str]; expected Set[None] [misc]

Not sure what kind of set comprehension is expected to contain only None values, but that feels less-than-useful in a type-checking context, and presumably has little-to-no relevance to the runtime type situation.