-
Notifications
You must be signed in to change notification settings - Fork 256
Can we have a generic Type[C]? #107
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Unless this is trivial to add to mypy I propose to punt this to a future version of type hints. FWIW type(X) is a shorthand for |
Yeah, but I'm pretty sure you have to reassign Anyway, I recognize that this is probably a lot more complex than it looks, and that the first 3.5 beta is rather alarmingly close, but I do feel this is essential for anyone using first-class classes in any significant way. Otherwise, you just have a |
|
Here's an example use case, straight from the code of my new framework: @typechecked
def register_extension_type(ext_type: str, extension_class: type, replace: bool=False):
"""
Adds a new extension type that can be used with a dictionary based configuration.
:param ext_type: the extension type identifier
:param extension_class: a class that implements IExtension
:param replace: ``True`` to replace an existing type
"""
assert_subclass('extension_class', extension_class, IExtension)
if ext_type in extension_types and not replace:
raise ValueError('Extension type "{}" already exists'.format(ext_type))
extension_types[ext_type] = extension_class I would like to declare the second argument as |
I understand the time pressure but without |
Could you submit a patch to mypy? That would make a world of difference. Until then, incompleteness of the typesystem doesn't bother me that much -- unlike statically-typed languages, Python programs don't have to be fully specified, you can always use |
I totally agree, one of the good points of gradual typing is the possibility to introduce it gradually :-) IMO it is perfectly pythonic - one could take exactly as much typing as one wants, practicality beats purity here. And yes I will try to make a patch. |
@ilevkivskyi If you're serious about making a patch, make sure type inference of this line doesn't crash or break things too badly:
You could infer |
Another issue related to For example, consider |
@JukkaL It gets worse:
I agree with Jukka that this code is wrong. We can easily fix it by splitting the "do something" off into a separate (underscore-prefixed) method and calling |
The pattern (helper class methods that create and return instances) is indeed very popular. I think it's usually done for classes that won't be subclassed, or where the subclasses don't override I wonder if a good type checker should be complaining only about an |
Frankly I don't understand what the fuss is about. Type[Foo] would not guarantee anything beyond the type being a subclass of Foo. It would not guarantee compatibility of method signatures. Just like when annotating an argument as Foo, it would still accept an instance of any Foo's subclass, which may have arbitrary method overrides, yes? |
@agronholm That's why we have the Liskov substitution principle. Usually, constructors and initializers are exempt from that rule since they're (effectively) static, but when you start to allow covariant first-class classes, that decision becomes a bit more questionable. Should static/class methods be exempt from LSP at all? |
My point was that Python does not adhere to the LSP since you can arbitrarily override any methods, even if the overridden signatures are incompatible. In Java, this would simply cause the superclass method to be called in such cases. Are you proposing that the LSP should now be enforced somehow? |
@NYKevin, One of the ideas could be to make first-class classes invariant and add a keyword, something like |
I don't think that is valid Python syntax. |
Good point :-) One can of course define a constant @JukkaL I think |
@agronholm The only reason Java calls the superclass implementation in that case is because it supports overloading. Python doesn't, so I would expect LSP adherence in practice. It may be difficult to enforce statically, however (cf. the Circle-Ellipse problem). But that doesn't mean you can get away with violating it for non-static methods. Things will break, whether the linter catches it or not. @ilevkivskyi That seems a rather odd syntax to me. I'm not sure invariant |
@NYKevin, I agree, it is not very useful, but one can use it for something like |
This seems to run into some of the same objections as handling the classmethod first argument does in #292, where we also have the problem of how to type-check calling the class, since we don't know what the signature of a subclass's constructor is. Maybe we can cut through these objections using a similar approach: Never mind the non-Liskov properties of constructor signatures, at least in the first round. If someone defines a class C with a certain constructor signature, and they define an API that takes an argument c annotated with Type[C], then suitable argument values for c are the class C and any subclass thereof. In the API, one can call class methods of C on the argument c, and one can also call c(), with the same signature as C(). In the future we can come up with some way to handle subclasses of C with a different constructor signature. Perhaps we could give Type an optional second parameter that constrains the constructor signature, so that e.g. Type[C, [int, int]] would mean subclasses of C whose constructor takes two integers. Hm... maybe it should be Type[[int, int], C] so it matches Callable? In any case, Type[C] should probably then be limited to subclasses whose constructor signature matches that of C. (To unconstrain the constructor signature, Type[C, ...] or Type[..., C] could be used, similar to Callable[..., C].) |
If I needed something like that, I would just write I'd much rather assume something like this: "Well-behaved constructors support cooperative multiple inheritance; they should consume the arguments they recognize and pass unrecognized arguments up the inheritance chain via Another option would be to ban calling types entirely. If you want a callable, you should take a callable. |
That's great if all you need is a factory, and if that's all we needed we wouldn't need Type[C] at all. But what if you're also using it as a class? E.g. call a class method, use it with isinstance() or issubclass(), or possibly even use it as a base class for a dynamically created new class.
IMO that's a dangerous myth -- it is essentially claiming a very specific protocol for constructors as the one true way. That's okay for a library or framework, but I wouldn't want Python itself to enforce it, and I don't think it's appropriate for a type checker either. (Different frameworks may have different religions about constructors and a type checker should be able to support them all equally well.) (Also, why shouldn't a recognized argument also be passed via super()? That's how non-constructor methods do it after all.)
I originally started out thinking that's a reasonable alternative, but looking at actual code where developers have asked for Type[C] I found that they are often constructing instances (but not just that, so Callable isn't a substitute). However: I am totally fine with punting on all this and just supporting Type[C] for now. (I am also considering making it so that at runtime you can call Type[] with any number of parameters -- this would make it easier to evolve the rules in mypy without having to distribute a new version of typing.py.) |
Because (Of course, in practice, a lot of code out there takes the third approach of "swallow everything immediately and don't even call
Fair enough, though I believe the system I just described is implicitly endorsed by the design of super(). But some people might reasonably disagree, and even if I am right, most developers won't want to go back and change their constructors just to make the type checker happy.
I would refactor that code into a classmethod, if possible. But again, we don't want to demand that developers refactor their code in arbitrary ways just to use the type checker, so we'd need to support this use case even if we were sure it was always a bad idea (which we're not).
I like this, as a compromise. Maybe throw in a warning in strict mode (is strict mode still a thing?). |
Why would such a root class be a bad idea? To me it sounds more sensible that assuming you can just multiply-inherit from anything that inherits from object. That's patently false anyways (try MI from list and dict :-). MI is a great idea but it needs to be tamed. We used to have a meme that mix-in classes were the only way to tame it. That was probably true in the Python 2 days of classic classes. But nowadays a framework-specific root class sounds way better. You're never going to be successful using MI with two base classes that weren't designed to cooperate. Sharing a framework-specific root class sounds like a perfect way to signal that you're expecting to cooperate, and that you're using the framework's protocol for constructor super-calls. The MRO mostly d
The design of super() is primarily meant to support "Liskov" for ordinary (non-constructor) methods. It is additionally constrained by the need for backwards compatibility, e.g. you can choose not to use it. If you don't call your super method you're implicitly constraining yourself to single inheritance. If you do call your super method you may support MI, if you do it right. For ordinary methods that's simple. For constructors, Python doesn't provide any additional support, but the needs and conventions are nevertheless different. Again, for SI you can do it any way you like (Liskov is not needed), and for MI you have to agree on a protocol. But I don't think that that protocol has to be the one implemented by object.
The idea is that at runtime it's completely silent, but a type checker can require a specific format. So if mypy doesn't support Type[X, Y] it should certainly flag that as an error. And we should all agree on what Type[X, Y] means before supporting it -- but my hope is that we won't be constrained by what the runtime library in 3.5.2 supports, since that library couldn't care less. (It's much easier to roll out a new version of mypy than to roll out a new version of typing.py, since the latter is baked into the stdlib of 3.5.x, and a PyPI package can't override a stdlib module.) |
Are types and classes different ? Would we need Type to accept stuff like str, dict, int and custom classes, while Class would accept only classes ? |
Hmm... The distinction between "type" and "class" as we're trying to distinguish in PEP 484 is that a class is a runtime concept while a type is a concept in the type checker. So str, dict etc. are classes. They are also types, but only because basically every class is considered a type -- and then there are some things that are types but not classes, such as Any, Union[int, str], Tuple[int, int], and type variables. (However, List[int] is both, though for fairly obscure reasons.) I don't recall whether we considered Class[C]; I kind of like Type[C] because we can make Type equal to Type[object] and which is exactly what Still I'm reluctant to give up the pun or rhyme with |
Right, the ability to substitute |
Everyone of python-ideas seems to prefer Type[C] over Class[C], so let's not fret over the naming. Next we need to come up with some text for PEP 484, and a PR for typing.py (both the Python 3 version and the Python 2 backport). All preferably before PyCon, i.e. before May 28 or so (because right after PyCon I'll be depeleted and the 3.5.2 RC will be on June 12, i.e. really soon afterwards). |
This addresses #107 (but it doen't close it because the issue also calls for an implementation in typing.py).
This addresses #107 (but it doen't close it because the issue also calls for an implementation in typing.py).
I think my text about covariance "Type[T] should be considered covariant, since for a concrete class C , Type[C] matches C and any of its subclasses" is bogus. Looking at the actual class definition in the PR, it seems the type variable used (CT) represents a metaclass. The "variance" allowed here is subclassing the metaclass. I think it's still correct to state that it's covariant, but the text needs to clarify that this refers to the metaclass. HELP!! |
With generic collections, the type parameter denotes the type of the contained elements. But for What to do here may depend on the definition of a "generic class". If FWIW, I assumed all along that |
I think the problem here is about classes vs types. It is not safe to talk about variance with respect to subclassing, one should talk about variance of subtyping. Therefore I would propose wording like this: " |
@bintoro Although |
@ilevkivskyi I did not refer to
I wouldn't because generic classes are a thing. A class that inherits from |
@gvanrossum In addition to my first explanation, I think |
Hm, I agree CT does not represent a metaclass. The example Type[User] makes this pretty clear. It's Type itself that's like a metaclass (it inherits from 'type'). This is also pretty clear from the new_user() example: def new_user(user_class: XXX):
...
new_user(User) Compare this to def foo(arg: int):
...
foo(42) Here, 42 is an instance of int, just like in the previous example User is an instance of type. So where the annotation for arg is a class, the annotation for user_class is a metaclass. Now back to variance. For this we need a different "regular" example. Let's use def bar(args: Sequence[int]):
... If we had a subclass of int, MyInt, then Sequence[MyInt] would be a valid type for a call to bar(), and that's what covariance means. So in our Type example, we have Type[User] as the argument's annotation. For Type to be covariant would mean that if we have another function with an argument declared as Type[ProUser], we could call new_user() with that argument: def new_pro_user(pro_user_class: Type[ProUser]):
user = new_user(pro_user_class)
... Indeed that sounds fine to me, so I agree intuitively (without having proven so rigorously) that Type is indeed covariant. (That's a big deal because previously I was just repeating "Type is covariant" because that's what the experts said; I hadn't actually visualized what that statement meant. :-) Having gotten this far, I like Ivan's first sentence:
I don't think the second sentence adds much (even though I agree it's true). |
I agree that To test whether
(Why do these rules not start simply "If Now suppose |
One problem with the typing rules above is that they don't actually correspond to python's runtime semantics. For example, every class is a subclass of The language in the PEP about compatibility of method signatures is the assumption that is needed for the typing rule to match the runtime reality. mypy currently checks compatibility for ordinary methods, but not for the constructor. The true covariance rule for runtime behavior is something like " Operationally I think mypy should implement these rules by just picking |
Also, I quite like @gvanrossum's class A: ...
class AImpl1(A): ...
class AImpl2(A): ...
if some_condition:
aImpl = AImpl1 # type: Type[A]
else:
aImpl = AImpl2
# use class methods on aImpl Technically this does not require variance since the rule for typing classes could be " |
OK, thanks for the (semi-?)formal proof, it helps to know that I didn't miss a case (reasoning about variance just doesn't come naturally to me, I'm probably a closet Eiffel programmer :-). Of course the weakness of the whole scheme is that the constructor signature of a subclass doesn't have to match that of the base class -- but that's not specific to the argument for covariance. There's already text in the PEP that promises to get back to this issue in the future. The other thought that this triggered for me is that a class method might itself be a factory that uses Type[T] in its return type, for some type variable T whose upper bound is C. Then c.m() needn't have the same type as C.m() -- wherever the return type of C.m() uses C, the return type of c.m() uses c. But honestly I don't want to weigh the PEP down with this formalism anyways, so we can hand-wave this away. |
I'm not sure which "it" you're referring to here: mypy or Trying this out, mypy already gives an error
but with a type alias mypy accepts the Running under python gave a baffling error, looks like an unrelated issue?:
For context, that from typing import TypeVar, Generic
T = TypeVar('T')
class Type(type, Generic[T], extra=type): pass
Type_int = Type[int]
isinstance(1, Type_int) |
Something like this crossed my mind too; I think it might fit better with the SelfType stuff discussed elsewhere and we should not worry about it right now. |
(Our remarks about constructor signatures and LSP crossed, but I think we're in agreement. Note that object is even more special than most other classes, because it also has a wacko rule about compatibility between |
Me neither, but most likely I was thinking about the type checker. Or perhaps the PEP. Since at runtime isinstance() involving special stuff is almost always forbidden, and issubclass() is forbidden by the PEP (though the runtime still allows it -- ripping it out is still a task (#136) but it's stalled by some unforeseen problems). |
@gvanrossum I agree that my second sentence does not add much, but I do not see a shorter way of explaining the covariance than @rwbarton did. I think probably it would be better to simply add a short example, illustrating that a value of type ``Type`` is covariant in its parameter, because ``Type[Derived]`` is a subtype of ``Type[Base]``::
def new_pro_user(pro_user_class: Type[ProUser]):
user = new_user(pro_user_class) # OK
... |
Done! I've also added a similar call to |
Closing -- both the PEP and typing.py have been updated. We're waiting for mypy but that's also very close (and not essential to this issue): python/mypy#1569 |
The
type
object occupies a rather strange place in the type hierarchy:(I'm pretty sure that's a flat lie, since you can't instantiate something from itself, but regardless...)
In Java, the (very rough) equivalent is a class, specifically
Class<T>
. It's also generic; the type variable refers to the instance. Java has it easy because they don't support metaclasses. Classes are not first class in Java, so their type hierarchy doesn't have to deal with the strange loops shown above.I realize metaclasses in their full generality are out of scope (at least for now), but a generic
Type[T]
like Java's would be nice to have. So far as I can tell from the Mypy documentation, it doesn't currently exist.Here's some example code which might like to have this feature:
The text was updated successfully, but these errors were encountered: