Soundness bugs with Callable[..., T] and partial

[drvink]

See XXX comments for highlights.

Callable[..., T]:

#!/usr/bin/env python
from typing import Any, Callable, Optional, TypeVar
import socket, sys

class CodedException(Exception):
    code = None # type: int
class CommandLineError(CodedException): code = 3

T = TypeVar('T')

def of_string_werr(fc # type: Callable[..., T]
):
    # type: (...) -> Callable[[Callable[[], Any]], Callable[[str], Callable[..., T]]]
    def curry_handler(fh):
        # type: (Callable[[], Any]) -> Callable[[str], Callable[..., T]]
        def curry_arg(x): # type: (str) -> Callable[..., T]
            def mkf(*args, **kw):
                # type: (*Any, **Any) -> T
                try: return fc(x, *args, **kw)
                except ValueError: fh()
            return mkf
        return curry_arg
    return curry_handler

def int_of_string_werr(fh):
    # type: (Callable[[], Any]) -> Callable[[str], Callable[..., int]]
    return of_string_werr(int)(fh)

def fail(ex=None): # type: (Optional[CodedException]) -> None
    if ex:
        print('%s: error: %s' % (sys.argv[0], ex), file=sys.stderr)
        sys.exit(ex.code)
    else:
        print('doop', file=sys.stderr)
        sys.exit(CommandLineError.code)

def ff(): # type: () -> None
    fail(CommandLineError('port must be an integer'))

# XXX the buggy line; should be: port = int_of_string_werr(ff)('1234')()
port = int_of_string_werr(ff)('1234')
# reveal_type(port): def (*Any, **Any) -> builtins.int

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# XXX typechecks but fails at runtime with
# TypeError: an integer is required (got type function)
sock.connect(('blah', port))

partial:

#!/usr/bin/env python
from typing import Any, Callable, Optional, TypeVar
from functools import partial
import socket, sys

class CodedException(Exception):
    code = None # type: int
class CommandLineError(CodedException): code = 3

T = TypeVar('T')

def of_string_werr(fc # type: Callable[..., T]
):
    # type: (...) -> Callable[[Callable[[], Any]], Callable[[str], partial[T]]]
    def curry_handler(fh):
        # type: (Callable[[], Any]) -> Callable[[str], partial[T]]
        def curry_arg(x): # type: (str) -> partial[T]
            def mkf(*args, **kw):
                # type: (*Any, **Any) -> T
                try: return fc(x, *args, **kw)
                except ValueError: fh()
            return partial(mkf)
        return curry_arg
    return curry_handler

def int_of_string_werr(fh):
    # type: (Callable[[], Any]) -> Callable[[str], partial[int]]

    # XXX
    # Incompatible return value type (got Callable[[str], partial[object]],
    # expected Callable[[str], partial[int]])
    return of_string_werr(int)(fh)

def fail(ex=None): # type: (Optional[CodedException]) -> None
    if ex:
        print('%s: error: %s' % (sys.argv[0], ex), file=sys.stderr)
        sys.exit(ex.code)
    else:
        print('doop', file=sys.stderr)
        sys.exit(CommandLineError.code)

def ff(): # type: () -> None
    fail(CommandLineError('port must be an integer'))

# XXX the buggy line; should be: port = int_of_string_werr(ff)('999')()
port = int_of_string_werr(ff)('999')
# reveal_type(port): def (*Any, **Any) -> builtins.int

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# XXX typechecks but fails at runtime with
# TypeError: an integer is required (got type function)
sock.connect(('blah', port))

As an aside, Callable and especially partial are very clumsy to use with typing; the use of higher-order functions is greatly impaired. The fact that def is not Callable is not lambda is not partial is not any number of other things vs. having a single simple ML-style arrow type is unfortunate.

[elazarg]

I agree that there should be a single SignatureType. Note that arrow type is not enough, since signatures in Python are more complex. SignatureType should have a binding operation, which will easily handle most uses of partial, and may also make hellp SelfType (#1212) "fall out of the implementation".

(I also do not understand why an overloaded function is not simply a Union of SignatureTypes)

[JukkaL]

An overloaded function could perhaps be an intersection of signatures, but even then they are currently not quite intersection types, since the order of signatures is significant, whereas intersection types are commutative.

It can't be a union since then a call would be valid only if it was compatible with every component signature.

[elazarg]

@JukkaL thanks I got it backwards indeed...

Why don't we have an intersection type yet? And thinking about it, if fallback means intersection, it happens to be ordered too. And so does multiple inheritance.

[JukkaL]

Yes, an ordered intersection type could perhaps replace overloaded callables and fallbacks -- not sure about multiple inheritance. I think about it every once in a while but I've never spent much time thinking about the implications. It would likely be a big change, and until we can demonstrate a concrete net benefit it's unlikely to happen. Also, there could be some edge cases that it doesn't deal well with -- it's possible that it would make some things cleaner and other things messier.

[elazarg]

Introducing it incrementally may help in assessing its value.

I think that a concrete benefit will be the ability to understand the code, especially fallbacks.

[alunduil]

I'm not sure if this smaller example is part of this issue or something different, but I'm having trouble getting Callable[..., T] to unify with a concrete type.

T = TypeVar("T", covariant=True)

f: Optional[Callable[[str], T]] = None
f  = lambda x: str(x)

This fails with:

 Incompatible types in assignment (expression has type "Callable[[str], str]", variable has type "Optional[Callable[[str], T]]")

Is there a way to workaround this issue? If there is a better place for this discussion, let me know.

[JukkaL]

@alunduil Your issue seems different from the original issue. It would be great if you could open a new GitHub issue about it.

Here's my analysis anyway:

f basically seems to have an existential type in your example (Optional[Callable[...]] for some T). Unfortunately, mypy doesn't support existential types, and mypy should actually complain about the type annotation of f instead of the assignment.

The closest supported type might be Optional[Callable[[str], object]]. You can always fall back to Optional[Callable[[str], Any]], but it's less precise.

[JelleZijlstra]

(Also, I think the example lambda is a Callable[[T], str], not Callable[[str], T].)

[alunduil]

@JelleZijlstra, It's Callable[[T₂], str] going into a Callable[[str], T₁] which should be unifiable (and might be the problem), but I'll document all of this in another issue so we don't pollute this one.