CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 612 – Parameter Specification Variables | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0612/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 612 – Parameter Specification Variables | peps.python.org'> <meta property="og:description" content="There currently are two ways to specify the type of a callable, the Callable[[int, str], bool] syntax defined in PEP 484, and callback protocols from PEP 544. Neither of these support forwarding the parameter types of one callable over to another calla..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0612/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="There currently are two ways to specify the type of a callable, the Callable[[int, str], bool] syntax defined in PEP 484, and callback protocols from PEP 544. Neither of these support forwarding the parameter types of one callable over to another calla..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 612</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 612 – Parameter Specification Variables</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Mark Mendoza <mendoza.mark.a at gmail.com></dd> <dt class="field-even">Sponsor:</dt> <dd class="field-even">Guido van Rossum <guido at python.org></dd> <dt class="field-odd">BDFL-Delegate:</dt> <dd class="field-odd">Guido van Rossum <guido at python.org></dd> <dt class="field-even">Discussions-To:</dt> <dd class="field-even"><a class="reference external" href="https://mail.python.org/archives/list/typing-sig@python.org/">Typing-SIG list</a></dd> <dt class="field-odd">Status:</dt> <dd class="field-odd"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-even">Type:</dt> <dd class="field-even"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-odd">Topic:</dt> <dd class="field-odd"><a class="reference external" href="../topic/typing/">Typing</a></dd> <dt class="field-even">Created:</dt> <dd class="field-even">18-Dec-2019</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">3.10</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even">18-Dec-2019, 13-Jul-2020</dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#paramspec-variables"><code class="docutils literal notranslate">ParamSpec</code> Variables</a><ul> <li><a class="reference internal" href="#declaration">Declaration</a></li> <li><a class="reference internal" href="#valid-use-locations">Valid use locations</a></li> <li><a class="reference internal" href="#user-defined-generic-classes">User-Defined Generic Classes</a></li> <li><a class="reference internal" href="#semantics">Semantics</a></li> </ul> </li> <li><a class="reference internal" href="#the-components-of-a-paramspec">The components of a <code class="docutils literal notranslate">ParamSpec</code></a><ul> <li><a class="reference internal" href="#id1">Valid use locations</a></li> <li><a class="reference internal" href="#id2">Semantics</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#rejected-alternatives">Rejected Alternatives</a><ul> <li><a class="reference internal" href="#using-list-variadics-and-map-variadics">Using List Variadics and Map Variadics</a></li> <li><a class="reference internal" href="#defining-parametersof">Defining ParametersOf</a></li> <li><a class="reference internal" href="#concatenating-keyword-parameters">Concatenating Keyword Parameters</a></li> <li><a class="reference internal" href="#naming-this-a-parameterspecification">Naming this a <code class="docutils literal notranslate">ParameterSpecification</code></a></li> <li><a class="reference internal" href="#naming-this-an-argspec">Naming this an <code class="docutils literal notranslate">ArgSpec</code></a></li> </ul> </li> <li><a class="reference internal" href="#acknowledgements">Acknowledgements</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <div class="pep-banner canonical-typing-spec sticky-banner admonition attention"> Attention This PEP is a historical document: see <a class="reference external" href="https://typing.readthedocs.io/en/latest/spec/generics.html#paramspec" title="(in typing)">ParamSpec</a> and <a class="reference external" href="https://docs.python.org/3/library/typing.html#typing.ParamSpec" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">typing.ParamSpec</code></a> for up-to-date specs and documentation. Canonical typing specs are maintained at the <a class="reference external" href="https://typing.readthedocs.io/en/latest/spec/">typing specs site</a>; runtime typing behaviour is described in the CPython documentation. × See the <a class="reference external" href="https://typing.readthedocs.io/en/latest/spec/meta.html">typing specification update process</a> for how to propose changes to the typing spec. </div> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> There currently are two ways to specify the type of a callable, the <code class="docutils literal notranslate">Callable[[int, str], bool]</code> syntax defined in <a class="pep reference internal" href="../pep-0484/" title="PEP 484 – Type Hints">PEP 484</a>, and callback protocols from <a class="pep reference internal" href="../pep-0544/#callback-protocols" title="PEP 544 – Protocols: Structural subtyping (static duck typing) § Callback protocols">PEP 544</a>. Neither of these support forwarding the parameter types of one callable over to another callable, making it difficult to annotate function decorators. This PEP proposes <code class="docutils literal notranslate">typing.ParamSpec</code> and <code class="docutils literal notranslate">typing.Concatenate</code> to support expressing these kinds of relationships. </section> <section id="motivation"> <h2><a class="toc-backref" href="#motivation" role="doc-backlink">Motivation</a></h2> The existing standards for annotating higher order functions don’t give us the tools to annotate the following common decorator pattern satisfactorily: <div class="highlight-default notranslate"><div class="highlight"><pre>from typing import Awaitable, Callable, TypeVar R = TypeVar("R") def add_logging(f: Callable[..., R]) -> Callable[..., Awaitable[R]]: async def inner(*args: object, **kwargs: object) -> R: await log_to_database() return f(*args, **kwargs) return inner @add_logging def takes_int_str(x: int, y: str) -> int: return x + 7 await takes_int_str(1, "A") await takes_int_str("B", 2) # fails at runtime </pre></div> </div> <code class="docutils literal notranslate">add_logging</code>, a decorator which logs before each entry into the decorated function, is an instance of the Python idiom of one function passing all arguments given to it over to another function. This is done through the combination of the <code class="docutils literal notranslate">*args</code> and <code class="docutils literal notranslate">**kwargs</code> features in both parameters and in arguments. When one defines a function (like <code class="docutils literal notranslate">inner</code>) that takes <code class="docutils literal notranslate">(*args, **kwargs)</code> and goes on to call another function with <code class="docutils literal notranslate">(*args, **kwargs)</code>, the wrapping function can only be safely called in all of the ways that the wrapped function could be safely called. To type this decorator, we’d like to be able to place a dependency between the parameters of the callable <code class="docutils literal notranslate">f</code> and the parameters of the returned function. <a class="pep reference internal" href="../pep-0484/" title="PEP 484 – Type Hints">PEP 484</a> supports dependencies between single types, as in <code class="docutils literal notranslate">def append(l: typing.List[T], e: T) -> typing.List[T]: ...</code>, but there is no existing way to do so with a complicated entity like the parameters of a function. Due to the limitations of the status quo, the <code class="docutils literal notranslate">add_logging</code> example will type check but will fail at runtime. <code class="docutils literal notranslate">inner</code> will pass the string “B” into <code class="docutils literal notranslate">takes_int_str</code>, which will try to add 7 to it, triggering a type error. This was not caught by the type checker because the decorated <code class="docutils literal notranslate">takes_int_str</code> was given the type <code class="docutils literal notranslate">Callable[..., Awaitable[int]]</code> (an ellipsis in place of parameter types is specified to mean that we do no validation on arguments). Without the ability to define dependencies between the parameters of different callable types, there is no way, at present, to make <code class="docutils literal notranslate">add_logging</code> compatible with all functions, while still preserving the enforcement of the parameters of the decorated function. With the addition of the <code class="docutils literal notranslate">ParamSpec</code> variables proposed by this PEP, we can rewrite the previous example in a way that keeps the flexibility of the decorator and the parameter enforcement of the decorated function. <div class="highlight-default notranslate"><div class="highlight"><pre>from typing import Awaitable, Callable, ParamSpec, TypeVar P = ParamSpec("P") R = TypeVar("R") def add_logging(f: Callable[P, R]) -> Callable[P, Awaitable[R]]: async def inner(*args: P.args, **kwargs: P.kwargs) -> R: await log_to_database() return f(*args, **kwargs) return inner @add_logging def takes_int_str(x: int, y: str) -> int: return x + 7 await takes_int_str(1, "A") # Accepted await takes_int_str("B", 2) # Correctly rejected by the type checker </pre></div> </div> Another common decorator pattern that has previously been impossible to type is the practice of adding or removing arguments from the decorated function. For example: <div class="highlight-default notranslate"><div class="highlight"><pre>class Request: ... def with_request(f: Callable[..., R]) -> Callable[..., R]: def inner(*args: object, **kwargs: object) -> R: return f(Request(), *args, **kwargs) return inner @with_request def takes_int_str(request: Request, x: int, y: str) -> int: # use request return x + 7 takes_int_str(1, "A") takes_int_str("B", 2) # fails at runtime </pre></div> </div> With the addition of the <code class="docutils literal notranslate">Concatenate</code> operator from this PEP, we can even type this more complex decorator. <div class="highlight-default notranslate"><div class="highlight"><pre>from typing import Concatenate def with_request(f: Callable[Concatenate[Request, P], R]) -> Callable[P, R]: def inner(*args: P.args, **kwargs: P.kwargs) -> R: return f(Request(), *args, **kwargs) return inner @with_request def takes_int_str(request: Request, x: int, y: str) -> int: # use request return x + 7 takes_int_str(1, "A") # Accepted takes_int_str("B", 2) # Correctly rejected by the type checker </pre></div> </div> </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> <section id="paramspec-variables"> <h3><a class="toc-backref" href="#paramspec-variables" role="doc-backlink"><code class="docutils literal notranslate">ParamSpec</code> Variables</a></h3> <section id="declaration"> <h4><a class="toc-backref" href="#declaration" role="doc-backlink">Declaration</a></h4> A parameter specification variable is defined in a similar manner to how a normal type variable is defined with <code class="docutils literal notranslate">typing.TypeVar</code>. <div class="highlight-default notranslate"><div class="highlight"><pre>from typing import ParamSpec P = ParamSpec("P") # Accepted P = ParamSpec("WrongName") # Rejected because P =/= WrongName </pre></div> </div> The runtime should accept <code class="docutils literal notranslate">bound</code>s and <code class="docutils literal notranslate">covariant</code> and <code class="docutils literal notranslate">contravariant</code> arguments in the declaration just as <code class="docutils literal notranslate">typing.TypeVar</code> does, but for now we will defer the standardization of the semantics of those options to a later PEP. </section> <section id="valid-use-locations"> <h4><a class="toc-backref" href="#valid-use-locations" role="doc-backlink">Valid use locations</a></h4> Previously only a list of parameter arguments (<code class="docutils literal notranslate">[A, B, C]</code>) or an ellipsis (signifying “undefined parameters”) were acceptable as the first “argument” to <code class="docutils literal notranslate">typing.Callable</code> . We now augment that with two new options: a parameter specification variable (<code class="docutils literal notranslate">Callable[P, int]</code>) or a concatenation on a parameter specification variable (<code class="docutils literal notranslate">Callable[Concatenate[int, P], int]</code>). <div class="highlight-default notranslate"><div class="highlight"><pre>callable ::= Callable "[" parameters_expression, type_expression "]" parameters_expression ::= | "..." | "[" [ type_expression ("," type_expression)* ] "]" | parameter_specification_variable | concatenate "[" type_expression ("," type_expression)* "," parameter_specification_variable "]" </pre></div> </div> where <code class="docutils literal notranslate">parameter_specification_variable</code> is a <code class="docutils literal notranslate">typing.ParamSpec</code> variable, declared in the manner as defined above, and <code class="docutils literal notranslate">concatenate</code> is <code class="docutils literal notranslate">typing.Concatenate</code>. As before, <code class="docutils literal notranslate">parameters_expression</code>s by themselves are not acceptable in places where a type is expected <div class="highlight-default notranslate"><div class="highlight"><pre>def foo(x: P) -> P: ... # Rejected def foo(x: Concatenate[int, P]) -> int: ... # Rejected def foo(x: typing.List[P]) -> None: ... # Rejected def foo(x: Callable[[int, str], P]) -> None: ... # Rejected </pre></div> </div> </section> <section id="user-defined-generic-classes"> <h4><a class="toc-backref" href="#user-defined-generic-classes" role="doc-backlink">User-Defined Generic Classes</a></h4> Just as defining a class as inheriting from <code class="docutils literal notranslate">Generic[T]</code> makes a class generic for a single parameter (when <code class="docutils literal notranslate">T</code> is a <code class="docutils literal notranslate">TypeVar</code>), defining a class as inheriting from <code class="docutils literal notranslate">Generic[P]</code> makes a class generic on <code class="docutils literal notranslate">parameters_expression</code>s (when <code class="docutils literal notranslate">P</code> is a <code class="docutils literal notranslate">ParamSpec</code>). <div class="highlight-default notranslate"><div class="highlight"><pre>T = TypeVar("T") P_2 = ParamSpec("P_2") class X(Generic[T, P]): f: Callable[P, int] x: T def f(x: X[int, P_2]) -> str: ... # Accepted def f(x: X[int, Concatenate[int, P_2]]) -> str: ... # Accepted def f(x: X[int, [int, bool]]) -> str: ... # Accepted def f(x: X[int, ...]) -> str: ... # Accepted def f(x: X[int, int]) -> str: ... # Rejected </pre></div> </div> By the rules defined above, spelling a concrete instance of a class generic with respect to only a single <code class="docutils literal notranslate">ParamSpec</code> would require unsightly double brackets. For aesthetic purposes we allow these to be omitted. <div class="highlight-default notranslate"><div class="highlight"><pre>class Z(Generic[P]): f: Callable[P, int] def f(x: Z[[int, str, bool]]) -> str: ... # Accepted def f(x: Z[int, str, bool]) -> str: ... # Equivalent # Both Z[[int, str, bool]] and Z[int, str, bool] express this: class Z_instantiated: f: Callable[[int, str, bool], int] </pre></div> </div> </section> <section id="semantics"> <h4><a class="toc-backref" href="#semantics" role="doc-backlink">Semantics</a></h4> The inference rules for the return type of a function invocation whose signature contains a <code class="docutils literal notranslate">ParamSpec</code> variable are analogous to those around evaluating ones with <code class="docutils literal notranslate">TypeVar</code>s. <div class="highlight-default notranslate"><div class="highlight"><pre>def changes_return_type_to_str(x: Callable[P, int]) -> Callable[P, str]: ... def returns_int(a: str, b: bool) -> int: ... f = changes_return_type_to_str(returns_int) # f should have the type: # (a: str, b: bool) -> str f("A", True) # Accepted f(a="A", b=True) # Accepted f("A", "A") # Rejected expects_str(f("A", True)) # Accepted expects_int(f("A", True)) # Rejected </pre></div> </div> Just as with traditional <code class="docutils literal notranslate">TypeVars</code>, a user may include the same <code class="docutils literal notranslate">ParamSpec</code> multiple times in the arguments of the same function, to indicate a dependency between multiple arguments. In these cases a type checker may choose to solve to a common behavioral supertype (i.e. a set of parameters for which all of the valid calls are valid in both of the subtypes), but is not obligated to do so. <div class="highlight-default notranslate"><div class="highlight"><pre>P = ParamSpec("P") def foo(x: Callable[P, int], y: Callable[P, int]) -> Callable[P, bool]: ... def x_y(x: int, y: str) -> int: ... def y_x(y: int, x: str) -> int: ... foo(x_y, x_y) # Should return (x: int, y: str) -> bool foo(x_y, y_x) # Could return (__a: int, __b: str) -> bool # This works because both callables have types that are # behavioral subtypes of Callable[[int, str], int] def keyword_only_x(*, x: int) -> int: ... def keyword_only_y(*, y: int) -> int: ... foo(keyword_only_x, keyword_only_y) # Rejected </pre></div> </div> The constructors of user-defined classes generic on <code class="docutils literal notranslate">ParamSpec</code>s should be evaluated in the same way. <div class="highlight-default notranslate"><div class="highlight"><pre>U = TypeVar("U") class Y(Generic[U, P]): f: Callable[P, str] prop: U def __init__(self, f: Callable[P, str], prop: U) -> None: self.f = f self.prop = prop def a(q: int) -> str: ... Y(a, 1) # Should resolve to Y[(q: int), int] Y(a, 1).f # Should resolve to (q: int) -> str </pre></div> </div> The semantics of <code class="docutils literal notranslate">Concatenate[X, Y, P]</code> are that it represents the parameters represented by <code class="docutils literal notranslate">P</code> with two positional-only parameters prepended. This means that we can use it to represent higher order functions that add, remove or transform a finite number of parameters of a callable. <div class="highlight-default notranslate"><div class="highlight"><pre>def bar(x: int, *args: bool) -> int: ... def add(x: Callable[P, int]) -> Callable[Concatenate[str, P], bool]: ... add(bar) # Should return (__a: str, x: int, *args: bool) -> bool def remove(x: Callable[Concatenate[int, P], int]) -> Callable[P, bool]: ... remove(bar) # Should return (*args: bool) -> bool def transform( x: Callable[Concatenate[int, P], int] ) -> Callable[Concatenate[str, P], bool]: ... transform(bar) # Should return (__a: str, *args: bool) -> bool </pre></div> </div> This also means that while any function that returns an <code class="docutils literal notranslate">R</code> can satisfy <code class="docutils literal notranslate">typing.Callable[P, R]</code>, only functions that can be called positionally in their first position with a <code class="docutils literal notranslate">X</code> can satisfy <code class="docutils literal notranslate">typing.Callable[Concatenate[X, P], R]</code>. <div class="highlight-default notranslate"><div class="highlight"><pre>def expects_int_first(x: Callable[Concatenate[int, P], int]) -> None: ... @expects_int_first # Rejected def one(x: str) -> int: ... @expects_int_first # Rejected def two(*, x: int) -> int: ... @expects_int_first # Rejected def three(**kwargs: int) -> int: ... @expects_int_first # Accepted def four(*args: int) -> int: ... </pre></div> </div> There are still some classes of decorators still not supported with these features: <ul class="simple"> <li>those that add/remove/change a variable number of parameters (for example, <code class="docutils literal notranslate">functools.partial</code> will remain untypable even after this PEP)</li> <li>those that add/remove/change keyword-only parameters (See <a class="reference internal" href="#concatenating-keyword-parameters">Concatenating Keyword Parameters</a> for more details).</li> </ul> </section> </section> <section id="the-components-of-a-paramspec"> <h3><a class="toc-backref" href="#the-components-of-a-paramspec" role="doc-backlink">The components of a <code class="docutils literal notranslate">ParamSpec</code></a></h3> A <code class="docutils literal notranslate">ParamSpec</code> captures both positional and keyword accessible parameters, but there unfortunately is no object in the runtime that captures both of these together. Instead, we are forced to separate them into <code class="docutils literal notranslate">*args</code> and <code class="docutils literal notranslate">**kwargs</code>, respectively. This means we need to be able to split apart a single <code class="docutils literal notranslate">ParamSpec</code> into these two components, and then bring them back together into a call. To do this, we introduce <code class="docutils literal notranslate">P.args</code> to represent the tuple of positional arguments in a given call and <code class="docutils literal notranslate">P.kwargs</code> to represent the corresponding <code class="docutils literal notranslate">Mapping</code> of keywords to values. <section id="id1"> <h4><a class="toc-backref" href="#id1" role="doc-backlink">Valid use locations</a></h4> These “properties” can only be used as the annotated types for <code class="docutils literal notranslate">*args</code> and <code class="docutils literal notranslate">**kwargs</code>, accessed from a ParamSpec already in scope. <div class="highlight-default notranslate"><div class="highlight"><pre>def puts_p_into_scope(f: Callable[P, int]) -> None: def inner(*args: P.args, **kwargs: P.kwargs) -> None: # Accepted pass def mixed_up(*args: P.kwargs, **kwargs: P.args) -> None: # Rejected pass def misplaced(x: P.args) -> None: # Rejected pass def out_of_scope(*args: P.args, **kwargs: P.kwargs) -> None: # Rejected pass </pre></div> </div> Furthermore, because the default kind of parameter in Python (<code class="docutils literal notranslate">(x: int)</code>) may be addressed both positionally and through its name, two valid invocations of a <code class="docutils literal notranslate">(*args: P.args, **kwargs: P.kwargs)</code> function may give different partitions of the same set of parameters. Therefore, we need to make sure that these special types are only brought into the world together, and are used together, so that our usage is valid for all possible partitions. <div class="highlight-default notranslate"><div class="highlight"><pre>def puts_p_into_scope(f: Callable[P, int]) -> None: stored_args: P.args # Rejected stored_kwargs: P.kwargs # Rejected def just_args(*args: P.args) -> None: # Rejected pass def just_kwargs(**kwargs: P.kwargs) -> None: # Rejected pass </pre></div> </div> </section> <section id="id2"> <h4><a class="toc-backref" href="#id2" role="doc-backlink">Semantics</a></h4> With those requirements met, we can now take advantage of the unique properties afforded to us by this set up: <ul class="simple"> <li>Inside the function, <code class="docutils literal notranslate">args</code> has the type <code class="docutils literal notranslate">P.args</code>, not <code class="docutils literal notranslate">Tuple[P.args, ...]</code> as would be with a normal annotation (and likewise with the <code class="docutils literal notranslate">**kwargs</code>)<ul> <li>This special case is necessary to encapsulate the heterogeneous contents of the <code class="docutils literal notranslate">args</code>/<code class="docutils literal notranslate">kwargs</code> of a given call, which cannot be expressed by an indefinite tuple/dictionary type.</li> </ul> </li> <li>A function of type <code class="docutils literal notranslate">Callable[P, R]</code> can be called with <code class="docutils literal notranslate">(*args, **kwargs)</code> if and only if <code class="docutils literal notranslate">args</code> has the type <code class="docutils literal notranslate">P.args</code> and <code class="docutils literal notranslate">kwargs</code> has the type <code class="docutils literal notranslate">P.kwargs</code>, and that those types both originated from the same function declaration.</li> <li>A function declared as <code class="docutils literal notranslate">def inner(*args: P.args, **kwargs: P.kwargs) -> X</code> has type <code class="docutils literal notranslate">Callable[P, X]</code>.</li> </ul> With these three properties, we now have the ability to fully type check parameter preserving decorators. <div class="highlight-default notranslate"><div class="highlight"><pre>def decorator(f: Callable[P, int]) -> Callable[P, None]: def foo(*args: P.args, **kwargs: P.kwargs) -> None: f(*args, **kwargs) # Accepted, should resolve to int f(*kwargs, **args) # Rejected f(1, *args, **kwargs) # Rejected return foo # Accepted </pre></div> </div> To extend this to include <code class="docutils literal notranslate">Concatenate</code>, we declare the following properties: <ul class="simple"> <li>A function of type <code class="docutils literal notranslate">Callable[Concatenate[A, B, P], R]</code> can only be called with <code class="docutils literal notranslate">(a, b, *args, **kwargs)</code> when <code class="docutils literal notranslate">args</code> and <code class="docutils literal notranslate">kwargs</code> are the respective components of <code class="docutils literal notranslate">P</code>, <code class="docutils literal notranslate">a</code> is of type <code class="docutils literal notranslate">A</code> and <code class="docutils literal notranslate">b</code> is of type <code class="docutils literal notranslate">B</code>.</li> <li>A function declared as <code class="docutils literal notranslate">def inner(a: A, b: B, *args: P.args, **kwargs: P.kwargs) -> R</code> has type <code class="docutils literal notranslate">Callable[Concatenate[A, B, P], R]</code>. Placing keyword-only parameters between the <code class="docutils literal notranslate">*args</code> and <code class="docutils literal notranslate">**kwargs</code> is forbidden.</li> </ul> <div class="highlight-default notranslate"><div class="highlight"><pre>def add(f: Callable[P, int]) -> Callable[Concatenate[str, P], None]: def foo(s: str, *args: P.args, **kwargs: P.kwargs) -> None: # Accepted pass def bar(*args: P.args, s: str, **kwargs: P.kwargs) -> None: # Rejected pass return foo # Accepted def remove(f: Callable[Concatenate[int, P], int]) -> Callable[P, None]: def foo(*args: P.args, **kwargs: P.kwargs) -> None: f(1, *args, **kwargs) # Accepted f(*args, 1, **kwargs) # Rejected f(*args, **kwargs) # Rejected return foo </pre></div> </div> Note that the names of the parameters preceding the <code class="docutils literal notranslate">ParamSpec</code> components are not mentioned in the resulting <code class="docutils literal notranslate">Concatenate</code>. This means that these parameters can not be addressed via a named argument: <div class="highlight-default notranslate"><div class="highlight"><pre>def outer(f: Callable[P, None]) -> Callable[P, None]: def foo(x: int, *args: P.args, **kwargs: P.kwargs) -> None: f(*args, **kwargs) def bar(*args: P.args, **kwargs: P.kwargs) -> None: foo(1, *args, **kwargs) # Accepted foo(x=1, *args, **kwargs) # Rejected return bar </pre></div> </div> This is not an implementation convenience, but a soundness requirement. If we were to allow that second calling style, then the following snippet would be problematic. <div class="highlight-default notranslate"><div class="highlight"><pre>@outer def problem(*, x: object) -> None: pass problem(x="uh-oh") </pre></div> </div> Inside of <code class="docutils literal notranslate">bar</code>, we would get <code class="docutils literal notranslate">TypeError: foo() got multiple values for argument 'x'</code>. Requiring these concatenated arguments to be addressed positionally avoids this kind of problem, and simplifies the syntax for spelling these types. Note that this also why we have to reject signatures of the form <code class="docutils literal notranslate">(*args: P.args, s: str, **kwargs: P.kwargs)</code> (See <a class="reference internal" href="#concatenating-keyword-parameters">Concatenating Keyword Parameters</a> for more details). If one of these prepended positional parameters contains a free <code class="docutils literal notranslate">ParamSpec</code>, we consider that variable in scope for the purposes of extracting the components of that <code class="docutils literal notranslate">ParamSpec</code>. That allows us to spell things like this: <div class="highlight-default notranslate"><div class="highlight"><pre>def twice(f: Callable[P, int], *args: P.args, **kwargs: P.kwargs) -> int: return f(*args, **kwargs) + f(*args, **kwargs) </pre></div> </div> The type of <code class="docutils literal notranslate">twice</code> in the above example is <code class="docutils literal notranslate">Callable[Concatenate[Callable[P, int], P], int]</code>, where <code class="docutils literal notranslate">P</code> is bound by the outer <code class="docutils literal notranslate">Callable</code>. This has the following semantics: <div class="highlight-default notranslate"><div class="highlight"><pre>def a_int_b_str(a: int, b: str) -> int: pass twice(a_int_b_str, 1, "A") # Accepted twice(a_int_b_str, b="A", a=1) # Accepted twice(a_int_b_str, "A", 1) # Rejected </pre></div> </div> </section> </section> </section> <section id="backwards-compatibility"> <h2><a class="toc-backref" href="#backwards-compatibility" role="doc-backlink">Backwards Compatibility</a></h2> The only changes necessary to existing features in <code class="docutils literal notranslate">typing</code> is allowing these <code class="docutils literal notranslate">ParamSpec</code> and <code class="docutils literal notranslate">Concatenate</code> objects to be the first parameter to <code class="docutils literal notranslate">Callable</code> and to be a parameter to <code class="docutils literal notranslate">Generic</code>. Currently <code class="docutils literal notranslate">Callable</code> expects a list of types there and <code class="docutils literal notranslate">Generic</code> expects single types, so they are currently mutually exclusive. Otherwise, existing code that doesn’t reference the new interfaces will be unaffected. </section> <section id="reference-implementation"> <h2><a class="toc-backref" href="#reference-implementation" role="doc-backlink">Reference Implementation</a></h2> The <a class="reference external" href="https://pyre-check.org/">Pyre</a> type checker supports all of the behavior described above. A reference implementation of the runtime components needed for those uses is provided in the <code class="docutils literal notranslate">pyre_extensions</code> module. A reference implementation for CPython can be found <a class="reference external" href="https://github.com/python/cpython/pull/23702">here</a>. </section> <section id="rejected-alternatives"> <h2><a class="toc-backref" href="#rejected-alternatives" role="doc-backlink">Rejected Alternatives</a></h2> <section id="using-list-variadics-and-map-variadics"> <h3><a class="toc-backref" href="#using-list-variadics-and-map-variadics" role="doc-backlink">Using List Variadics and Map Variadics</a></h3> We considered just trying to make something like this with a callback protocol which was parameterized on a list-type variadic, and a map-type variadic like so: <div class="highlight-default notranslate"><div class="highlight"><pre>R = typing.TypeVar(“R”) Tpositionals = ... Tkeywords = ... class BetterCallable(typing.Protocol[Tpositionals, Tkeywords, R]): def __call__(*args: Tpositionals, **kwargs: Tkeywords) -> R: ... </pre></div> </div> However, there are some problems with trying to come up with a consistent solution for those type variables for a given callable. This problem comes up with even the simplest of callables: <div class="highlight-default notranslate"><div class="highlight"><pre>def simple(x: int) -> None: ... simple <: BetterCallable[[int], [], None] simple <: BetterCallable[[], {“x”: int}, None] BetterCallable[[int], [], None] </: BetterCallable[[], {“x”: int}, None] </pre></div> </div> Any time where a type can implement a protocol in more than one way that aren’t mutually compatible, we can run into situations where we lose information. If we were to make a decorator using this protocol, we would have to pick one calling convention to prefer. <div class="highlight-default notranslate"><div class="highlight"><pre>def decorator( f: BetterCallable[[Ts], [Tmap], int], ) -> BetterCallable[[Ts], [Tmap], str]: def decorated(*args: Ts, **kwargs: Tmap) -> str: x = f(*args, **kwargs) return int_to_str(x) return decorated @decorator def foo(x: int) -> int: return x reveal_type(foo) # Option A: BetterCallable[[int], {}, str] # Option B: BetterCallable[[], {x: int}, str] foo(7) # fails under option B foo(x=7) # fails under option A </pre></div> </div> The core problem here is that, by default, parameters in Python can either be called positionally or as a keyword argument. This means we really have three categories (positional-only, positional-or-keyword, keyword-only) we’re trying to jam into two categories. This is the same problem that we briefly mentioned when discussing <code class="docutils literal notranslate">.args</code> and <code class="docutils literal notranslate">.kwargs</code>. Fundamentally, in order to capture two categories when there are some things that can be in either category, we need a higher level primitive (<code class="docutils literal notranslate">ParamSpec</code>) to capture all three, and then split them out afterward. </section> <section id="defining-parametersof"> <h3><a class="toc-backref" href="#defining-parametersof" role="doc-backlink">Defining ParametersOf</a></h3> Another proposal we considered was defining <code class="docutils literal notranslate">ParametersOf</code> and <code class="docutils literal notranslate">ReturnType</code> operators which would operate on a domain of a newly defined <code class="docutils literal notranslate">Function</code> type. <code class="docutils literal notranslate">Function</code> would be callable with, and only with <code class="docutils literal notranslate">ParametersOf[F]</code>. <code class="docutils literal notranslate">ParametersOf</code> and <code class="docutils literal notranslate">ReturnType</code> would only operate on type variables with precisely this bound. The combination of these three features could express everything that we can express with <code class="docutils literal notranslate">ParamSpecs</code>. <div class="highlight-default notranslate"><div class="highlight"><pre>F = TypeVar("F", bound=Function) def no_change(f: F) -> F: def inner( *args: ParametersOf[F].args, **kwargs: ParametersOf[F].kwargs ) -> ReturnType[F]: return f(*args, **kwargs) return inner def wrapping(f: F) -> Callable[ParametersOf[F], List[ReturnType[F]]]: def inner( *args: ParametersOf[F].args, **kwargs: ParametersOf[F].kwargs ) -> List[ReturnType[F]]: return [f(*args, **kwargs)] return inner def unwrapping( f: Callable[ParametersOf[F], List[R]] ) -> Callable[ParametersOf[F], R]: def inner( *args: ParametersOf[F].args, **kwargs: ParametersOf[F].kwargs ) -> R: return f(*args, **kwargs)[0] return inner </pre></div> </div> We decided to go with <code class="docutils literal notranslate">ParamSpec</code>s over this approach for several reasons: <ul class="simple"> <li>The footprint of this change would be larger, as we would need two new operators, and a new type, while <code class="docutils literal notranslate">ParamSpec</code> just introduces a new variable.</li> <li>Python typing has so far has avoided supporting operators, whether user-defined or built-in, in favor of destructuring. Accordingly, <code class="docutils literal notranslate">ParamSpec</code> based signatures look much more like existing Python.</li> <li>The lack of user-defined operators makes common patterns hard to spell. <code class="docutils literal notranslate">unwrapping</code> is odd to read because <code class="docutils literal notranslate">F</code> is not actually referring to any callable. It’s just being used as a container for the parameters we wish to propagate. It would read better if we could define an operator <code class="docutils literal notranslate">RemoveList[List[X]] = X</code> and then <code class="docutils literal notranslate">unwrapping</code> could take <code class="docutils literal notranslate">F</code> and return <code class="docutils literal notranslate">Callable[ParametersOf[F], RemoveList[ReturnType[F]]]</code>. Without that, we unfortunately get into a situation where we have to use a <code class="docutils literal notranslate">Function</code>-variable as an improvised <code class="docutils literal notranslate">ParamSpec</code>, in that we never actually bind the return type.</li> </ul> In summary, between these two equivalently powerful syntaxes, <code class="docutils literal notranslate">ParamSpec</code> fits much more naturally into the status quo. </section> <section id="concatenating-keyword-parameters"> <h3><a class="toc-backref" href="#concatenating-keyword-parameters" role="doc-backlink">Concatenating Keyword Parameters</a></h3> In principle the idea of concatenation as a means to modify a finite number of positional parameters could be expanded to include keyword parameters. <div class="highlight-default notranslate"><div class="highlight"><pre>def add_n(f: Callable[P, R]) -> Callable[Concatenate[("n", int), P], R]: def inner(*args: P.args, n: int, **kwargs: P.kwargs) -> R: # use n return f(*args, **kwargs) return inner </pre></div> </div> However, the key distinction is that while prepending positional-only parameters to a valid callable type always yields another valid callable type, the same cannot be said for adding keyword-only parameters. As alluded to <a class="reference internal" href="#above">above</a> , the issue is name collisions. The parameters <code class="docutils literal notranslate">Concatenate[("n", int), P]</code> are only valid when <code class="docutils literal notranslate">P</code> itself does not already have a parameter named <code class="docutils literal notranslate">n</code>. <div class="highlight-default notranslate"><div class="highlight"><pre>def innocent_wrapper(f: Callable[P, R]) -> Callable[P, R]: def inner(*args: P.args, **kwargs: P.kwargs) -> R: added = add_n(f) return added(*args, n=1, **kwargs) return inner @innocent_wrapper def problem(n: int) -> None: pass </pre></div> </div> Calling <code class="docutils literal notranslate">problem(2)</code> works fine, but calling <code class="docutils literal notranslate">problem(n=2)</code> leads to a <code class="docutils literal notranslate">TypeError: problem() got multiple values for argument 'n'</code> from the call to <code class="docutils literal notranslate">added</code> inside of <code class="docutils literal notranslate">innocent_wrapper</code>. This kind of situation could be avoided, and this kind of decorator could be typed if we could reify the constraint that a set of parameters not contain a certain name, with something like: <div class="highlight-default notranslate"><div class="highlight"><pre>P_without_n = ParamSpec("P_without_n", banned_names=["n"]) def add_n( f: Callable[P_without_n, R] ) -> Callable[Concatenate[("n", int), P_without_n], R]: ... </pre></div> </div> The call to <code class="docutils literal notranslate">add_n</code> inside of <code class="docutils literal notranslate">innocent_wrapper</code> could then be rejected since the callable was not guaranteed not to already have a parameter named <code class="docutils literal notranslate">n</code>. However, enforcing these constraints would require enough additional implementation work that we judged this extension to be out of scope of this PEP. Fortunately the design of <code class="docutils literal notranslate">ParamSpec</code>s are such that we can return to this idea later if there is sufficient demand. </section> <section id="naming-this-a-parameterspecification"> <h3><a class="toc-backref" href="#naming-this-a-parameterspecification" role="doc-backlink">Naming this a <code class="docutils literal notranslate">ParameterSpecification</code></a></h3> We decided that ParameterSpecification was a little too long-winded for use here, and that this style of abbreviated name made it look more like TypeVar. </section> <section id="naming-this-an-argspec"> <h3><a class="toc-backref" href="#naming-this-an-argspec" role="doc-backlink">Naming this an <code class="docutils literal notranslate">ArgSpec</code></a></h3> We think that calling this a ParamSpec is more correct than referring to it as an ArgSpec, since callables have parameters, which are distinct from the arguments which are passed to them in a given call site. A given binding for a ParamSpec is a set of function parameters, not a call-site’s arguments. </section> </section> <section id="acknowledgements"> <h2><a class="toc-backref" href="#acknowledgements" role="doc-backlink">Acknowledgements</a></h2> Thanks to all of the members of the Pyre team for their comments on early drafts of this PEP, and for their help with the reference implementation. Thanks are also due to the whole Python typing community for their early feedback on this idea at a Python typing meetup, leading directly to the much more compact <code class="docutils literal notranslate">.args</code>/<code class="docutils literal notranslate">.kwargs</code> syntax. </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0612.rst">https://github.com/python/peps/blob/main/peps/pep-0612.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0612.rst">2024-06-11 22:12:09 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#paramspec-variables"><code class="docutils literal notranslate">ParamSpec</code> Variables</a><ul> <li><a class="reference internal" href="#declaration">Declaration</a></li> <li><a class="reference internal" href="#valid-use-locations">Valid use locations</a></li> <li><a class="reference internal" href="#user-defined-generic-classes">User-Defined Generic Classes</a></li> <li><a class="reference internal" href="#semantics">Semantics</a></li> </ul> </li> <li><a class="reference internal" href="#the-components-of-a-paramspec">The components of a <code class="docutils literal notranslate">ParamSpec</code></a><ul> <li><a class="reference internal" href="#id1">Valid use locations</a></li> <li><a class="reference internal" href="#id2">Semantics</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#rejected-alternatives">Rejected Alternatives</a><ul> <li><a class="reference internal" href="#using-list-variadics-and-map-variadics">Using List Variadics and Map Variadics</a></li> <li><a class="reference internal" href="#defining-parametersof">Defining ParametersOf</a></li> <li><a class="reference internal" href="#concatenating-keyword-parameters">Concatenating Keyword Parameters</a></li> <li><a class="reference internal" href="#naming-this-a-parameterspecification">Naming this a <code class="docutils literal notranslate">ParameterSpecification</code></a></li> <li><a class="reference internal" href="#naming-this-an-argspec">Naming this an <code class="docutils literal notranslate">ArgSpec</code></a></li> </ul> </li> <li><a class="reference internal" href="#acknowledgements">Acknowledgements</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0612.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>