ParamSpec: Allow only P.args

[cdce8p]

I would like to use ParamSpec to type a function that accepts a Callable[P, Any] and *args which will be passed to the callable. The function itself does not accept keyword arguments. As an example, AbstractEventLoop.call_soon behaves similar. This is the current Typeshed definition

    def call_soon(self, callback: Callable[..., Any], *args: Any) -> Handle: ...

typeshed -> stdlib/asyncio/events.pyi -> AbstractEventLoop

P = ParamSpec("P")

    def call_soon(self, callback: Callable[P, Any], *args: P.args) -> Handle: ...

Intuitively adding a ParamSpec variable like this would make sense: "Only except *args. If any arguments are passed, they need to match those of the callback." I would also expect that if callback has required arguments, they need to be passed with *args to call_soon.

The issue here is that PEP 612 explicitly forbids specifying P.args or P.kwargs alone. They always need to be together.
https://www.python.org/dev/peps/pep-0612/#id2

Furthermore, because the default kind of parameter in Python ((x: int)) may be addressed
both positionally and through its name, two valid invocations of a (*args: P.args, **kwargs: P.kwargs)
function may give different partitions of the same set of parameters. Therefore, we need to make
sure that these special types are only brought into the world together, and are used together,
so that our usage is valid for all possible partitions.

I do wonder if this strict limitation makes sense or if there are other ways to work around it so the case described above could be supported. After all it's similar to adding **kwargs to the function signature without ever passing keyword arguments.

    def call_soon(self, callback: Callable[P, Any], *args: P.args, **kwargs: P.kwargs) -> Handle: ...

[TeamSpen210]

Perhaps this could require having ParamSpec("Name", positional_only=True)? Then it would be clear you don't care about keyword arguments, and any callable with required keyword args would be invalid.

[JelleZijlstra]

I believe PEP 646 would allow this: https://www.python.org/dev/peps/pep-0646/#type-variable-tuples-with-callable.

[erictraut]

@JelleZijlstra is correct. This is supported by PEP 646.

Ts = TypeVarTuple("Ts")

def call_soon(self, callback: Callable[[*Ts], Any], *args: *Ts) -> Handle:
    ...

[cdce8p]

This is supported by PEP 646.

Thanks for the responses! This certainly looks promising. I do wonder though why it wouldn't work with ParamSpec, too. The PEP says it's forbidden, but is there an actual limitation? I would have assumed that only specifying one would just set an additional constraint for the other to be empty. E.g. if only P.args is given, no kwargs should be accepted.

There is also the question about kwargs only calls. Something like this here seems reasonable to require keyword arguments for external functions.

P = ParamSpec("P")
R = TypeVar("R")

def decorator(
    func: Callable[P, R]
) -> Callable[P, R]:
    def wrapper(**kwargs: P.kwargs) -> R:
        return func(**kwargs)
    return wrapper

[srittau]

I have a similar use case with Ariadne middleware functions, but with kwargs. In Ariadne, a "resolver" function takes two positional arguments and a varying number of keyword arguments that depend on the API the resolver implements. As the name implies, a middleware function sits between Ariadne and the resolver function and forwards arguments.

Ideally, I would like to type a middleware function as follows:

_P = ParamSpec("_P")
_R = TypeVar("_R")
_T = TypeVar("_T")

def foo_middleware(
    resolver: Callable[Concatenate[_T, GraphQLResolveInfo, _P], _R],
    obj: _T,
    info: GraphQLResolveInfo,
    /,
    **kwargs: _P.kwargs,
) -> _R:
    ...
    return resolver(obj, info, **kwargs)

But currently this isn't possible, because _P.args is "missing".

[JelleZijlstra]

Closing as PEP 646 provides support for @cdce8p's use case.

[cdce8p]

Closing as PEP 646 provides support for @cdce8p's use case.

Unfortunately, it only fully works for Mypy. Pyright doesn't support callables with defaults.

from typing import Callable, TypeVarTuple

Ts = TypeVarTuple("Ts")

def f(a: int) -> None: ...
def g(a: int, b: str = "") -> None: ...

def call_soon(func: Callable[[*Ts], None], *args: *Ts) -> None:
    return func(*args)


call_soon(f, 1)
call_soon(g, 1)  # pyright error
call_soon(g, 1, "Hello")

Whereas mypy can decide if Ts should be expanded to (int,) or (int, str), pyright always assumes the latter which results in an error for call_soon(g, 1).

  test.py:13:1 - error: Argument of type "*tuple[int]" cannot be assigned to parameter "args" of type "*Ts@call_soon" in function "call_soon"
    Type "*tuple[int]" cannot be assigned to type "*tuple[int, str]"
      "*tuple[int]" is incompatible with "*tuple[int, str]"
        Tuple size mismatch; expected 2 but received 1 (reportGeneralTypeIssues)

@erictraut mentioned in microsoft/pyright#3775 that this was be design. If so, TypeVarTuple can't be used to annotated functions like these.

@AlexWaygood mentioned a few other examples for TypeVarTuple in the typeshed tracking issue which wouldn't work then: python/typeshed#8708 (comment)

--
mypy playground
pyright playground
pyre doesn't seem to work at all? pyre playground

[AlexWaygood]

@erictraut mentioned in microsoft/pyright#3775 that this was be design. If so, TypeVarTuple can't be used to annotated functions like these.

@AlexWaygood mentioned a few other examples for TypeVarTuple in the typeshed tracking issue which wouldn't work then: python/typeshed#8708 (comment)

I think it would be unfortunate if this interpretation of the spec meant we had to stick with the (significantly more unsafe) signatures indefinitely for the many functions in asyncio that follow this pattern. I had been hoping we were close to being able to significantly increase the type safety of asyncio's functions: see python/typeshed#11015.

If it is decided that pyright's current interpretation of the spec is correct, I think the outcome will be that there will be essentially nowhere in typeshed that we are able to make use of TypeVarTuple.

[erictraut]

If mypy figured out a way to implement this, I'll take another look at adding support in pyright. I'm not sure how mypy is capturing the value of Ts in the example above when g is passed as an argument. The captured type could be either tuple[int] or tuple[int, str]. Pyright's implementation chooses tuple[int, str]. It looks like mypy makes the same choice by default.

def g(a: int, b: str = "") -> None: ...
def func(func: Callable[[*Ts], None]) -> tuple[*Ts]: ...
reveal_type(func(g))  # mypy and pyright both reveal "tuple[int, str]"

@JelleZijlstra, do you happen to know who implemented this in mypy? Was it Ivan? I could use some advice on how this works.

[JelleZijlstra]

Yes, Ivan did most of the PEP 646 work, though Jared Hance worked on it earlier.

[erictraut]

Good news — I figured out how to make this work in pyright without too much hackery. This support will be included in the next release of pyright.

It's not clear from a reading of PEP 646 that this case is meant to be supported — and if so, what the intended behavior is. If typeshed is going to depend on this behavior, we should work to mandate this behavior in the typing spec and include it in the typing conformance test suite. I've added this to a long list that I've been compiling.

[AlexWaygood]

Thanks Eric! Yes, agreed, let's formalise this in the spec.

[cdce8p]

Now that both mypy and pyright support using TypeVarTuples for it, I guess the only thing left would be to eventually formalize this in the typing spec. Would be fine with closing this issue if nobody has objections.

https://github.com/microsoft/pyright/releases/tag/1.1.339

--
There is still the **kwargs case @srittau mentioned earlier but that might need a new kind of TypeVar if we don't want to touch ParamSpec. Might be best to track that separately though.

[srittau]

I split the kwargs case into #1524.