Callable has no attribute __kwdefaults__

[ahmedsoe]

Here's an example (pycharm is also syntax highlighting it):

def print_first_number_decorator(f: Callable[Any]) -> Callable:
    """Example decorator"""
    
    @functools.wraps(f)
    def wrapper(*args: List[Any], **kwargs: Dict[Any, Any]) -> int:
        print(f.__kwdefaults__["num2"])  # Callable[Any]" has no attribute "__kwdefaults__"
        return f(*args)
    
    return wrapper
    
    
@print_first_number_decorator
def add_(num1=3, *, num2=4) -> int:
    return num1 + num2
    
    
if __name__ == "__main__":
    print(add_(5))

Mypy error: error: "Callable[Any]" has no attribute "__kwdefaults__"

[ilevkivskyi]

I think this can be fixed quickly just by a typeshed change, for historical reasons mypy uses builtins.function instead of types.FunctionType, see #3171. Fixing that issue is not very hard but tedious, so I would propose in the meantime just copy all the content from types.FunctionType to builtins.function (except for __get__ maybe?). @JelleZijlstra what do you think?

[JelleZijlstra]

Sure. Could we just do from types import FunctionType as function in builtins.pyi instead? (Or the other way around.)

[ilevkivskyi]

Could we just do from types import FunctionType as function in builtins.pyi instead

This may work, mypy uses various builtin_type() functions for this, and there is no guarantee they all will work. Could you please try this? If it works, this will be great.

[JukkaL]

Accessing __kwdefaults__ is not safe, since a callable object can be an instance of an arbitrary class that implements __call__:

from typing import Callable

def f(fn: Callable[[], None]) -> None:
    fn()
    print(fn.__kwdefaults__)  # Unsafe

class A:
    def __call__(self) -> None: pass

f(A())

If we had intersection types, we could use Intersection[Callable[[], None], FunctionType] to represent a callable that must be a function object.

[JukkaL]

Perhaps more importantly, type objects and C functions don't have __kwdefaults__, and they are commonly used as callable objects.

The current fallback is already unsafe, but we perhaps shouldn't make it any more unsafe. An alternative approach could be something like this:

• Remove at least __code__ and __annotations__ from function since they aren't very generally available (removing __name__ might cause too much trouble).
• If we see an access to some of these missing attributes, suggest casting the callable to FunctionType instead. Maybe also explain that some callables such as type objects don't have these defined.

[ilevkivskyi]

Accessing __kwdefaults__ is not safe

Everything except __module__ if already unsafe, but I didn't hear a single complain about this. Avoiding a false negative in an edge case (callable object) at a cost of false positive in a common case (normal function) is not wise.

Also it is a false positive, the same error appears for clearly valid code:

def f() -> None: ...
f.__kwdefaults__

[JukkaL]

Also it is a false positive, the same error appears for clearly valid code:

This is a fair point. However, the use __kwdefaults__ seems very rare, and requiring a cast is not a big burden. In an internal codebase of several million lines of Python I only found a few uses of __kwdefaults__ (or the Python 2 equivalent). If this would be a typical result, this seems a very low-priority thing.

Everything except module if already unsafe, but I didn't hear a single complain about this.

My argument is that at the moment there is only a weak case for making things even unsafer, since the enabled use case is probably rare. FunctionType as the fallback would be a lie. function is already a lie, but at least it's not shy about it -- as the type doesn't exist at runtime, it's more clear that something special is going on. We could make this clearer in the stubs by adding some comments.

Some attributes like __name__ are defined for multiple kinds of callables and used all over the place, so removing them is probably not a realistic option. But the fact that things are incorrect right now feels like a questionable argument for making things even less correct. We should generally try to make things more correct, unless this would have other drawbacks such as generating many false positives, which doesn't seem to be the case here (but I could be wrong here -- I'm happy to accept evidence that shows otherwise).

Also, I'd rather talk about how to fix this in a more principled fashion, even if we won't be doing this in the immediate future. For example, maybe Python function objects should have a different fallback from plain Callable types? Or should we use an intersection type, if/when they are supported?

[JukkaL]

Here are some additional related examples that generate false positives at the moment:

ord.__text_signature__  # Works at runtime, but mypy complains

class A:
    def f(self): pass
A().f.__self__  # Works at runtime, but mypy complains

The move to FunctionType wouldn't help with these, and I don't immediately see why we should ignore these over supporting __kwdefaults__. Using __self__, at least, seems somewhat common in the wild, perhaps significantly more common than __kwdefaults__.

The problem is much wider than the original reported issue, and just fixing the reported issue without considering the wider problem risks moving sideways at best, I think.

[JukkaL]

To get the ball rolling, here's a random idea about solving the problem more generally:

1. Change function to a protocol type that contains attributes that most(?) callable objects should have. At least __name__ should be here, even if it's unsafe, to avoid breaking existing code. A protocol has the benefit of not being usable in isinstance checks. It's also more correct, since there is no single concrete class that is a base class of all callables (beyond object).
2. For references to user-defined functions infer FunctionType as the callable fallback. This allows access to __kwdefaults__, etc.
3. For bound methods infer MethodType as the fallback (only when we are certain about this).
4. When inferring types for variables, normalize callable fallback to the protocol mentioned in (1). This can be important to avoid false positives and general confusion, since there's no way to specify a custom fallback for a callable using the annotation syntax.
5. User-defined classes with __call__ would still be compatible with callable types, even though they may not define all the attributes defined by the function protocol. This would be the primary remaining unsafety.
6. Give an informative note when accessing __kwdefaults__ (and other similar attributes) of callables with the default fallback.
7. Optionally, figure out a way to infer the correct fallback for C/Python functions defined in stubs. This is probably not super important, though, since generally whether a module is implemented in C or Python should be an implementation detail that programs shouldn't depend on.

This wouldn't directly solve the original issue, but it would have the benefit of more closely modelling what happens at runtime.

Another idea would be to use intersection types, but that would be problematic since types like FunctionType define a permissible __call__ which seems problematic -- an intersection with FunctionType would apparently allow arbitrary calls. Using fallback types doesn't have the same problem.

[ilevkivskyi]

Another example of this appeared in python/typing#598

[kasium]

Hey,

I currently working with decorators and signatures. Each time I assign a signature to __signature__ I need to add a myyp ignore.
Is there any plan to solve this?

[AlexWaygood]

The original example in this issue is no longer reproducible due to changes made in typeshed.