Problem with @overloaded decorators

[o11c]

Currently, the best possible signature for a decorator is

T = TypeVar('T')
def foo(fun: T) -> T: ...

because there is no other way to express that the callable will be preserved exactly.

However, in the wild, and this fails to work:

T = TypeVar('T')
@overload
def foo(fun: T) -> T: ...
@overload
def foo(mes: str) -> Callable[[T], T]: ...

Possible solutions:

• Define a strict order in which overloads are attempted (likely to happen if runtime @overload proposal, but a bad idea for producing meaningful error messages from a static type checker).
• Add a @fallback_overload decorator that only is considered if none of the others applies.
• Support Intersection since it's needed for a coherent type system anyway.
• Support typeof(expr) and use the argument names in the return value.
• Add syntax for packs, named arguments, and named argument packs in Callable, possibly with the following syntax:

R = TypeVar('R')
A = TypeVarPack('A')
K = TypeVarKwPack('K')

def foo(fun: Callable[[A, K], R]) -> Callable[[A, K], R]: ...

[gvanrossum]

Part of this problem is just decorators that are trying to be too clever. In particular, they want to let you write

@blah
def fun(...): ...

but also

@blah(option)
def fun(...): ...

Of course if we had better type checking in the first place we could have simplified things and required that the first example be written

@blah()
def fun(...): ...

Barring that, the problem seems to be that a decorator's type, even that for one which doesn't alter the signature of the decorated function, needs a more powerful way of expressing its signature than what you say.

I think it should be possible to write this in a stub file:

Func = TypeVar('Func', bound=Callable[..., Any])
@overload
def logger(func: Func) -> Func: ...
@overload
def logger(msg: str) -> Callable[[Func], Func]: ...

and then use it like this:

@logger
def foo(...): ...

@logger('Hello')
def bar(...): ...

I played with this and mypy doesn't yet support TypeVar(..., bound=...). There's an issue for that: python/mypy#689 but I think it's not a high priority -- the type of a decorator is hard (e.g. what if it changes the sugnature by inserting or removing a parameter) and we're punting on this until we're farther along with the "easy" stuff.

[JukkaL]

I think that using a type variable with a bound is the most reasonable way of implementing this. It's not a top priority thing for mypy right now as there are more serious issues that I want to address first. Falling back to dynamic typing (signature Any -> Any) is always an option, and there will always be examples where this is required.

I've also been thinking about mypy specific, ad hoc type system plugins for commonly used library things that can't be represented using the current type system. So instead of adding a general (and complex) new type system feature that should be supported by all tools, we could have a dynamically or partially typed fallback that is 'good enough' and supported by all tools + individual tools can optionally special case some things to get better type checking coverage.

[ilevkivskyi]

PEP 484 now specifies type variable bounds, and the pattern in question is fully supported by mypy (I just tried on master).