CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 637 – Support for indexing with keyword arguments | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0637/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 637 – Support for indexing with keyword arguments | peps.python.org'> <meta property="og:description" content="At present keyword arguments are allowed in function calls, but not in item access. This PEP proposes that Python be extended to allow keyword arguments in item access."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0637/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="At present keyword arguments are allowed in function calls, but not in item access. This PEP proposes that Python be extended to allow keyword arguments in item access."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 637</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 637 – Support for indexing with keyword arguments</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Stefano Borini</dd> <dt class="field-even">Sponsor:</dt> <dd class="field-even">Steven D’Aprano</dd> <dt class="field-odd">Discussions-To:</dt> <dd class="field-odd"><a class="reference external" href="https://mail.python.org/archives/list/python-ideas@python.org/">Python-Ideas list</a></dd> <dt class="field-even">Status:</dt> <dd class="field-even"><abbr title="Formally declined and will not be accepted">Rejected</abbr></dd> <dt class="field-odd">Type:</dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created:</dt> <dd class="field-even">24-Aug-2020</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">3.10</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even">23-Sep-2020</dd> <dt class="field-odd">Resolution:</dt> <dd class="field-odd"><a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/6TAQ2BEVSJNV4JM2RJYSSYFJUT3INGZD/">Python-Dev thread</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#overview">Overview</a><ul> <li><a class="reference internal" href="#background">Background</a></li> <li><a class="reference internal" href="#use-cases">Use cases</a></li> <li><a class="reference internal" href="#current-status-of-indexing-operation">Current status of indexing operation</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#indexing-behavior-in-standard-classes-dict-list-etc">Indexing behavior in standard classes (dict, list, etc.)</a></li> <li><a class="reference internal" href="#corner-case-and-gotchas">Corner case and Gotchas</a></li> </ul> </li> <li><a class="reference internal" href="#c-interface">C Interface</a></li> <li><a class="reference internal" href="#how-to-teach-recommendations">“How to teach” recommendations</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#workarounds">Workarounds</a></li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a><ul> <li><a class="reference internal" href="#previous-pep-472-solutions">Previous PEP 472 solutions</a></li> <li><a class="reference internal" href="#adding-new-dunders">Adding new dunders</a></li> <li><a class="reference internal" href="#adding-an-adapter-function">Adding an adapter function</a></li> <li><a class="reference internal" href="#create-a-new-kwslice-object">create a new “kwslice” object</a></li> <li><a class="reference internal" href="#using-a-single-bit-to-change-the-behavior">Using a single bit to change the behavior</a></li> <li><a class="reference internal" href="#allowing-for-empty-index-notation-obj">Allowing for empty index notation obj[]</a></li> <li><a class="reference internal" href="#sentinel-value-for-no-given-positional-index">Sentinel value for no given positional index</a></li> </ul> </li> <li><a class="reference internal" href="#common-objections">Common objections</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <div class="admonition note"> Note This PEP has been rejected. In general, the cost of introducing new syntax was not outweighed by the perceived benefits. See the link in the Resolution header field for details. </div> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> At present keyword arguments are allowed in function calls, but not in item access. This PEP proposes that Python be extended to allow keyword arguments in item access. The following example shows keyword arguments for ordinary function calls: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> val = f(1, 2, a=3, b=4) </pre></div> </div> The proposal would extend the syntax to allow a similar construct to indexing operations: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> val = x[1, 2, a=3, b=4] # getitem >>> x[1, 2, a=3, b=4] = val # setitem >>> del x[1, 2, a=3, b=4] # delitem </pre></div> </div> and would also provide appropriate semantics. Single- and double-star unpacking of arguments is also provided: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> val = x[*(1, 2), **{a=3, b=4}] # Equivalent to above. </pre></div> </div> This PEP is a successor to <a class="pep reference internal" href="../pep-0472/" title="PEP 472 – Support for indexing with keyword arguments">PEP 472</a>, which was rejected due to lack of interest in 2019. Since then there’s been renewed interest in the feature. </section> <section id="overview"> <h2><a class="toc-backref" href="#overview" role="doc-backlink">Overview</a></h2> <section id="background"> <h3><a class="toc-backref" href="#background" role="doc-backlink">Background</a></h3> <a class="pep reference internal" href="../pep-0472/" title="PEP 472 – Support for indexing with keyword arguments">PEP 472</a> was opened in 2014. The PEP detailed various use cases and was created by extracting implementation strategies from a broad discussion on the python-ideas mailing list, although no clear consensus was reached on which strategy should be used. Many corner cases have been examined more closely and felt awkward, backward incompatible or both. The PEP was eventually rejected in 2019 <a class="footnote-reference brackets" href="#rejection" id="id1">[1]</a> mostly due to lack of interest for the feature despite its 5 years of existence. However, with the introduction of type hints in <a class="pep reference internal" href="../pep-0484/" title="PEP 484 – Type Hints">PEP 484</a> the square bracket notation has been used consistently to enrich the typing annotations, e.g. to specify a list of integers as Sequence[int]. Additionally, there has been an expanded growth of packages for data analysis such as pandas and xarray, which use names to describe columns in a table (pandas) or axis in an nd-array (xarray). These packages allow users to access specific data by names, but cannot currently use index notation ([]) for this functionality. As a result, a renewed interest in a more flexible syntax that would allow for named information has been expressed occasionally in many different threads on python-ideas, recently by Caleb Donovick <a class="footnote-reference brackets" href="#request-1" id="id2">[2]</a> in 2019 and Andras Tantos <a class="footnote-reference brackets" href="#request-2" id="id3">[3]</a> in 2020. These requests prompted a strong activity on the python-ideas mailing list, where the various options have been re-discussed and a general consensus on an implementation strategy has now been reached. </section> <section id="use-cases"> <h3><a class="toc-backref" href="#use-cases" role="doc-backlink">Use cases</a></h3> The following practical use cases present different cases where a keyword specification would improve notation and provide additional value: <ol class="arabic"> <li>To provide a more communicative meaning to the index, preventing e.g. accidental inversion of indexes:<div class="highlight-default notranslate"><div class="highlight"><pre>>>> grid_position[x=3, y=5, z=8] >>> rain_amount[time=0:12, location=location] >>> matrix[row=20, col=40] </pre></div> </div> </li> <li>To enrich the typing notation with keywords, especially during the use of generics:<div class="highlight-default notranslate"><div class="highlight"><pre>def function(value: MyType[T=int]): </pre></div> </div> </li> <li>In some domain, such as computational physics and chemistry, the use of a notation such as <code class="docutils literal notranslate">Basis[Z=5]</code> is a Domain Specific Language notation to represent a level of accuracy:<div class="highlight-default notranslate"><div class="highlight"><pre>>>> low_accuracy_energy = computeEnergy(molecule, BasisSet[Z=3]) </pre></div> </div> </li> <li>Pandas currently uses a notation such as:<div class="highlight-default notranslate"><div class="highlight"><pre>>>> df[df['x'] == 1] </pre></div> </div> which could be replaced with <code class="docutils literal notranslate">df[x=1]</code>. </li> <li>xarray has named dimensions. Currently these are handled with functions .isel:<div class="highlight-default notranslate"><div class="highlight"><pre>>>> data.isel(row=10) # Returns the tenth row </pre></div> </div> which could also be replaced with <code class="docutils literal notranslate">data[row=10]</code>. A more complex example: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> # old syntax >>> da.isel(space=0, time=slice(None, 2))[...] = spam >>> # new syntax >>> da[space=0, time=:2] = spam </pre></div> </div> Another example: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> # old syntax >>> ds["empty"].loc[dict(lon=5, lat=6)] = 10 >>> # new syntax >>> ds["empty"][lon=5, lat=6] = 10 >>> # old syntax >>> ds["empty"].loc[dict(lon=slice(1, 5), lat=slice(3, None))] = 10 >>> # new syntax >>> ds["empty"][lon=1:5, lat=6:] = 10 </pre></div> </div> </li> <li>Functions/methods whose argument is another function (plus its arguments) need some way to determine which arguments are destined for the target function, and which are used to configure how they run the target. This is simple (if non-extensible) for positional parameters, but we need some way to distinguish these for keywords. <a class="footnote-reference brackets" href="#trio-run" id="id4">[4]</a>An indexed notation would afford a Pythonic way to pass keyword arguments to these functions without cluttering the caller’s code. <div class="highlight-default notranslate"><div class="highlight"><pre>>>> # Let's start this example with basic syntax without keywords. >>> # the positional values are arguments to `func` while >>> # `name=` is processed by `trio.run`. >>> trio.run(func, value1, value2, name="func") >>> # `trio.run` ends up calling `func(value1, value2)`. >>> # If we want/need to pass value2 by keyword (keyword-only argument, >>> # additional arguments that won't break backwards compatibility ...), >>> # currently we need to resort to functools.partial: >>> trio.run(functools.partial(func, param2=value2), value1, name="func") >>> trio.run(functools.partial(func, value1, param2=value2), name="func") >>> # One possible workaround is to convert `trio.run` to an object >>> # with a `__call__` method, and use an "option" helper, >>> trio.run.option(name="func")(func, value1, param2=value2) >>> # However, foo(bar)(baz) is uncommon and thus disruptive to the reader. >>> # Also, you need to remember the name of the `option` method. >>> # This PEP allows us to replace `option` with `__getitem__`. >>> # The call is now shorter, more mnemonic, and looks+works like typing >>> trio.run[name="func"](func, value1, param2=value2) </pre></div> </div> </li> <li>Availability of star arguments would benefit <a class="pep reference internal" href="../pep-0646/" title="PEP 646 – Variadic Generics">PEP 646</a> Variadic Generics, especially in the forms <code class="docutils literal notranslate">a[*x]</code> and <code class="docutils literal notranslate">a[*x, *y, p, q, *z]</code>. The PEP details exactly this notation in its “Unpacking: Star Operator” section.</li> </ol> It is important to note that how the notation is interpreted is up to the implementation. This PEP only defines and dictates the behavior of Python regarding passed keyword arguments, not how these arguments should be interpreted and used by the implementing class. </section> <section id="current-status-of-indexing-operation"> <h3><a class="toc-backref" href="#current-status-of-indexing-operation" role="doc-backlink">Current status of indexing operation</a></h3> Before detailing the new syntax and semantics to the indexing notation, it is relevant to analyse how the indexing notation works today, in which contexts, and how it is different from a function call. Subscripting <code class="docutils literal notranslate">obj[x]</code> is, effectively, an alternate and specialised form of function call syntax with a number of differences and restrictions compared to <code class="docutils literal notranslate">obj(x)</code>. The current Python syntax focuses exclusively on position to express the index, and also contains syntactic sugar to refer to non-punctiform selection (slices). Some common examples: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> a[3] # returns the fourth element of 'a' >>> a[1:10:2] # slice notation (extract a non-trivial data subset) >>> a[3, 2] # multiple indexes (for multidimensional arrays) </pre></div> </div> This translates into a <code class="docutils literal notranslate">__(get|set|del)item__</code> dunder call which is passed a single parameter containing the index (for <code class="docutils literal notranslate">__getitem__</code> and <code class="docutils literal notranslate">__delitem__</code>) or two parameters containing index and value (for <code class="docutils literal notranslate">__setitem__</code>). The behavior of the indexing call is fundamentally different from a function call in various aspects: The first difference is in meaning to the reader. A function call says “arbitrary function call potentially with side-effects”. An indexing operation says “lookup”, typically to point at a subset or specific sub-aspect of an entity (as in the case of typing notation). This fundamental difference means that, while we cannot prevent abuse, implementors should be aware that the introduction of keyword arguments to alter the behavior of the lookup may violate this intrinsic meaning. The second difference of the indexing notation compared to a function is that indexing can be used for both getting and setting operations. In Python, a function cannot be on the left hand side of an assignment. In other words, both of these are valid: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> x = a[1, 2] >>> a[1, 2] = 5 </pre></div> </div> but only the first one of these is valid: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> x = f(1, 2) >>> f(1, 2) = 5 # invalid </pre></div> </div> This asymmetry is important, and makes one understand that there is a natural imbalance between the two forms. It is therefore not a given that the two should behave transparently and symmetrically. The third difference is that functions have names assigned to their arguments, unless the passed parameters are captured with <code class="docutils literal notranslate">*args</code>, in which case they end up as entries in the args tuple. In other words, functions already have anonymous argument semantic, exactly like the indexing operation. However, <code class="docutils literal notranslate">__(get|set|del)item__</code> is not always receiving a tuple as the <code class="docutils literal notranslate">index</code> argument (to be uniform in behavior with <code class="docutils literal notranslate">*args</code>). In fact, given a trivial class: <div class="highlight-default notranslate"><div class="highlight"><pre>class X: def __getitem__(self, index): print(index) </pre></div> </div> The index operation basically forwards the content of the square brackets “as is” in the <code class="docutils literal notranslate">index</code> argument: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> x=X() >>> x[0] 0 >>> x[0, 1] (0, 1) >>> x[(0, 1)] (0, 1) >>> >>> x[()] () >>> x[{1, 2, 3}] {1, 2, 3} >>> x["hello"] hello >>> x["hello", "hi"] ('hello', 'hi') </pre></div> </div> The fourth difference is that the indexing operation knows how to convert colon notations to slices, thanks to support from the parser. This is valid: <div class="highlight-default notranslate"><div class="highlight"><pre>a[1:3] </pre></div> </div> this one isn’t: <div class="highlight-default notranslate"><div class="highlight"><pre>f(1:3) </pre></div> </div> The fifth difference is that there’s no zero-argument form. This is valid: <div class="highlight-default notranslate"><div class="highlight"><pre>f() </pre></div> </div> this one isn’t: <div class="highlight-default notranslate"><div class="highlight"><pre>a[] </pre></div> </div> </section> </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> Before describing the specification, it is important to stress the difference in nomenclature between positional index, final index and keyword argument, as it is important to understand the fundamental asymmetries at play. The <code class="docutils literal notranslate">__(get|set|del)item__</code> is fundamentally an indexing operation, and the way the element is retrieved, set, or deleted is through an index, the final index. The current status quo is to directly build the final index from what is passed between square brackets, the positional index. In other words, what is passed in the square brackets is trivially used to generate what the code in <code class="docutils literal notranslate">__getitem__</code> then uses for the indicisation operation. As we already saw for the dict, <code class="docutils literal notranslate">d[1]</code> has a positional index of <code class="docutils literal notranslate">1</code> and also a final index of <code class="docutils literal notranslate">1</code> (because it’s the element that is then added to the dictionary) and <code class="docutils literal notranslate">d[1, 2]</code> has positional index of <code class="docutils literal notranslate">(1, 2)</code> and final index also of <code class="docutils literal notranslate">(1, 2)</code> (because yet again it’s the element that is added to the dictionary). However, the positional index <code class="docutils literal notranslate">d[1,2:3]</code> is not accepted by the dictionary, because there’s no way to transform the positional index into a final index, as the slice object is unhashable. The positional index is what is currently known as the <code class="docutils literal notranslate">index</code> parameter in <code class="docutils literal notranslate">__getitem__</code>. Nevertheless, nothing prevents to construct a dictionary-like class that creates the final index by e.g. converting the positional index to a string. This PEP extends the current status quo, and grants more flexibility to create the final index via an enhanced syntax that combines the positional index and keyword arguments, if passed. The above brings an important point across. Keyword arguments, in the context of the index operation, may be used to take indexing decisions to obtain the final index, and therefore will have to accept values that are unconventional for functions. See for example use case 1, where a slice is accepted. The successful implementation of this PEP will result in the following behavior: <ol class="arabic"> <li>An empty subscript is still illegal, regardless of context (see Rejected Ideas):<div class="highlight-default notranslate"><div class="highlight"><pre>obj[] # SyntaxError </pre></div> </div> </li> <li>A single index value remains a single index value when passed:<div class="highlight-default notranslate"><div class="highlight"><pre>obj[index] # calls type(obj).__getitem__(obj, index) obj[index] = value # calls type(obj).__setitem__(obj, index, value) del obj[index] # calls type(obj).__delitem__(obj, index) </pre></div> </div> This remains the case even if the index is followed by keywords; see point 5 below. </li> <li>Comma-separated arguments are still parsed as a tuple and passed as a single positional argument:<div class="highlight-default notranslate"><div class="highlight"><pre>obj[spam, eggs] # calls type(obj).__getitem__(obj, (spam, eggs)) obj[spam, eggs] = value # calls type(obj).__setitem__(obj, (spam, eggs), value) del obj[spam, eggs] # calls type(obj).__delitem__(obj, (spam, eggs)) </pre></div> </div> The points above mean that classes which do not want to support keyword arguments in subscripts need do nothing at all, and the feature is therefore completely backwards compatible. </li> <li>Keyword arguments, if any, must follow positional arguments:<div class="highlight-default notranslate"><div class="highlight"><pre>obj[1, 2, spam=None, 3] # SyntaxError </pre></div> </div> This is like function calls, where intermixing positional and keyword arguments give a SyntaxError. </li> <li>Keyword subscripts, if any, will be handled like they are in function calls. Examples:<div class="highlight-default notranslate"><div class="highlight"><pre># Single index with keywords: obj[index, spam=1, eggs=2] # calls type(obj).__getitem__(obj, index, spam=1, eggs=2) obj[index, spam=1, eggs=2] = value # calls type(obj).__setitem__(obj, index, value, spam=1, eggs=2) del obj[index, spam=1, eggs=2] # calls type(obj).__delitem__(obj, index, spam=1, eggs=2) # Comma-separated indices with keywords: obj[foo, bar, spam=1, eggs=2] # calls type(obj).__getitem__(obj, (foo, bar), spam=1, eggs=2) obj[foo, bar, spam=1, eggs=2] = value # calls type(obj).__setitem__(obj, (foo, bar), value, spam=1, eggs=2) del obj[foo, bar, spam=1, eggs=2] # calls type(obj).__detitem__(obj, (foo, bar), spam=1, eggs=2) </pre></div> </div> Note that: <ul class="simple"> <li>a single positional index will not turn into a tuple just because one adds a keyword value.</li> <li>for <code class="docutils literal notranslate">__setitem__</code>, the same order is retained for index and value. The keyword arguments go at the end, as is normal for a function definition.</li> </ul> </li> <li>The same rules apply with respect to keyword subscripts as for keywords in function calls:<ul class="simple"> <li>the interpreter matches up each keyword subscript to a named parameter in the appropriate method;</li> <li>if a named parameter is used twice, that is an error;</li> <li>if there are any named parameters left over (without a value) when the keywords are all used, they are assigned their default value (if any);</li> <li>if any such parameter doesn’t have a default, that is an error;</li> <li>if there are any keyword subscripts remaining after all the named parameters are filled, and the method has a <code class="docutils literal notranslate">**kwargs</code> parameter, they are bound to the <code class="docutils literal notranslate">**kwargs</code> parameter as a dict;</li> <li>but if no <code class="docutils literal notranslate">**kwargs</code> parameter is defined, it is an error.</li> </ul> </li> <li>Sequence unpacking is allowed inside subscripts:<div class="highlight-default notranslate"><div class="highlight"><pre>obj[*items] </pre></div> </div> This allows notations such as <code class="docutils literal notranslate">[:, *args, :]</code>, which could be treated as <code class="docutils literal notranslate">[(slice(None), *args, slice(None))]</code>. Multiple star unpacking are allowed: <div class="highlight-default notranslate"><div class="highlight"><pre>obj[1, *(2, 3), *(4, 5), 6, foo=5] # Equivalent to obj[(1, 2, 3, 4, 5, 6), foo=3) </pre></div> </div> The following notation equivalence must be honored: <div class="highlight-default notranslate"><div class="highlight"><pre>obj[*()] # Equivalent to obj[()] obj[*(), foo=3] # Equivalent to obj[(), foo=3] obj[*(x,)] # Equivalent to obj[(x,)] obj[*(x,),] # Equivalent to obj[(x,)] </pre></div> </div> Note in particular case 3: sequence unpacking of a single element will not behave as if only one single argument was passed. A related case is the following example: <div class="highlight-default notranslate"><div class="highlight"><pre>obj[1, *(), foo=5] # Equivalent to obj[(1,), foo=5] # calls type(obj).__getitem__(obj, (1,), foo=5) </pre></div> </div> However, as we saw earlier, for backward compatibility a single index will be passed as is: <div class="highlight-default notranslate"><div class="highlight"><pre>obj[1, foo=5] # calls type(obj).__getitem__(obj, 1, foo=5) </pre></div> </div> In other words, a single positional index will be passed “as is” only if no sequence unpacking is present. If a sequence unpacking is present, then the index will become a tuple, regardless of the resulting number of elements in the index after the unpacking has taken place. </li> <li>Dict unpacking is permitted:<div class="highlight-default notranslate"><div class="highlight"><pre>items = {'spam': 1, 'eggs': 2} obj[index, **items] # equivalent to obj[index, spam=1, eggs=2] </pre></div> </div> The following notation equivalent should be honored: <div class="highlight-default notranslate"><div class="highlight"><pre>obj[**{}] # Equivalent to obj[()] obj[3, **{}] # Equivalent to obj[3] </pre></div> </div> </li> <li>Keyword-only subscripts are permitted. The positional index will be the empty tuple:<div class="highlight-default notranslate"><div class="highlight"><pre>obj[spam=1, eggs=2] # calls type(obj).__getitem__(obj, (), spam=1, eggs=2) obj[spam=1, eggs=2] = 5 # calls type(obj).__setitem__(obj, (), 5, spam=1, eggs=2) del obj[spam=1, eggs=2] # calls type(obj).__delitem__(obj, (), spam=1, eggs=2) </pre></div> </div> The choice of the empty tuple as a sentinel has been debated. Details are provided in the Rejected Ideas section. </li> <li>Keyword arguments must allow slice syntax:<div class="highlight-default notranslate"><div class="highlight"><pre>obj[3:4, spam=1:4, eggs=2] # calls type(obj).__getitem__(obj, slice(3, 4, None), spam=slice(1, 4, None), eggs=2) </pre></div> </div> This may open up the possibility to accept the same syntax for general function calls, but this is not part of this recommendation. </li> <li>Keyword arguments allow for default values:<div class="highlight-default notranslate"><div class="highlight"><pre># Given type(obj).__getitem__(obj, index, spam=True, eggs=2) obj[3] # Valid. index = 3, spam = True, eggs = 2 obj[3, spam=False] # Valid. index = 3, spam = False, eggs = 2 obj[spam=False] # Valid. index = (), spam = False, eggs = 2 obj[] # Invalid. </pre></div> </div> </li> <li>The same semantics given above must be extended to <code class="docutils literal notranslate">__class__getitem__</code>: Since <a class="pep reference internal" href="../pep-0560/" title="PEP 560 – Core support for typing module and generic types">PEP 560</a>, type hints are dispatched so that for <code class="docutils literal notranslate">x[y]</code>, if no <code class="docutils literal notranslate">__getitem__</code> method is found, and <code class="docutils literal notranslate">x</code> is a type (class) object, and <code class="docutils literal notranslate">x</code> has a class method <code class="docutils literal notranslate">__class_getitem__</code>, that method is called. The same changes should be applied to this method as well, so that a writing like <code class="docutils literal notranslate">list[T=int]</code> can be accepted.</li> </ol> <section id="indexing-behavior-in-standard-classes-dict-list-etc"> <h3><a class="toc-backref" href="#indexing-behavior-in-standard-classes-dict-list-etc" role="doc-backlink">Indexing behavior in standard classes (dict, list, etc.)</a></h3> None of what is proposed in this PEP will change the behavior of the current core classes that use indexing. Adding keywords to the index operation for custom classes is not the same as modifying e.g. the standard dict type to handle keyword arguments. In fact, dict (as well as list and other stdlib classes with indexing semantics) will remain the same and will continue not to accept keyword arguments. In other words, if <code class="docutils literal notranslate">d</code> is a <code class="docutils literal notranslate">dict</code>, the statement <code class="docutils literal notranslate">d[1, a=2]</code> will raise <code class="docutils literal notranslate">TypeError</code>, as their implementation will not support the use of keyword arguments. The same holds for all other classes (list, dict, etc.) </section> <section id="corner-case-and-gotchas"> <h3><a class="toc-backref" href="#corner-case-and-gotchas" role="doc-backlink">Corner case and Gotchas</a></h3> With the introduction of the new notation, a few corner cases need to be analysed. <ol class="arabic"> <li>Technically, if a class defines their getter like this:<div class="highlight-default notranslate"><div class="highlight"><pre>def __getitem__(self, index): </pre></div> </div> then the caller could call that using keyword syntax, like these two cases: <div class="highlight-default notranslate"><div class="highlight"><pre>obj[3, index=4] obj[index=1] </pre></div> </div> The resulting behavior would be an error automatically, since it would be like attempting to call the method with two values for the <code class="docutils literal notranslate">index</code> argument, and a <code class="docutils literal notranslate">TypeError</code> will be raised. In the first case, the <code class="docutils literal notranslate">index</code> would be <code class="docutils literal notranslate">3</code>, in the second case, it would be the empty tuple <code class="docutils literal notranslate">()</code>. Note that this behavior applies for all currently existing classes that rely on indexing, meaning that there is no way for the new behavior to introduce backward compatibility issues on this respect. Classes that wish to stress this behavior explicitly can define their parameters as positional-only: <div class="highlight-default notranslate"><div class="highlight"><pre>def __getitem__(self, index, /): </pre></div> </div> </li> <li>a similar case occurs with setter notation:<div class="highlight-default notranslate"><div class="highlight"><pre># Given type(obj).__setitem__(obj, index, value): obj[1, value=3] = 5 </pre></div> </div> This poses no issue because the value is passed automatically, and the Python interpreter will raise <code class="docutils literal notranslate">TypeError: got multiple values for keyword argument 'value'</code> </li> <li>If the subscript dunders are declared to use positional-or-keyword parameters, there may be some surprising cases when arguments are passed to the method. Given the signature:<div class="highlight-default notranslate"><div class="highlight"><pre>def __getitem__(self, index, direction='north') </pre></div> </div> if the caller uses this: <div class="highlight-default notranslate"><div class="highlight"><pre>obj[0, 'south'] </pre></div> </div> they will probably be surprised by the method call: <div class="highlight-default notranslate"><div class="highlight"><pre># expected type(obj).__getitem__(obj, 0, direction='south') # but actually get: type(obj).__getitem__(obj, (0, 'south'), direction='north') </pre></div> </div> Solution: best practice suggests that keyword subscripts should be flagged as keyword-only when possible: <div class="highlight-default notranslate"><div class="highlight"><pre>def __getitem__(self, index, *, direction='north') </pre></div> </div> The interpreter need not enforce this rule, as there could be scenarios where this is the desired behaviour. But linters may choose to warn about subscript methods which don’t use the keyword-only flag. </li> <li>As we saw, a single value followed by a keyword argument will not be changed into a tuple, i.e.: <code class="docutils literal notranslate">d[1, a=3]</code> is treated as <code class="docutils literal notranslate">__getitem__(d, 1, a=3)</code>, NOT <code class="docutils literal notranslate">__getitem__(d, (1,), a=3)</code>. It would be extremely confusing if adding keyword arguments were to change the type of the passed index. In other words, adding a keyword to a single-valued subscript will not change it into a tuple. For those cases where an actual tuple needs to be passed, a proper syntax will have to be used:<div class="highlight-default notranslate"><div class="highlight"><pre>obj[(1,), a=3] # calls type(obj).__getitem__(obj, (1,), a=3) </pre></div> </div> In this case, the call is passing a single element (which is passed as is, as from rule above), only that the single element happens to be a tuple. Note that this behavior just reveals the truth that the <code class="docutils literal notranslate">obj[1,]</code> notation is shorthand for <code class="docutils literal notranslate">obj[(1,)]</code> (and also <code class="docutils literal notranslate">obj[1]</code> is shorthand for <code class="docutils literal notranslate">obj[(1)]</code>, with the expected behavior). When keywords are present, the rule that you can omit this outermost pair of parentheses is no longer true: <div class="highlight-default notranslate"><div class="highlight"><pre>obj[1] # calls type(obj).__getitem__(obj, 1) obj[1, a=3] # calls type(obj).__getitem__(obj, 1, a=3) obj[1,] # calls type(obj).__getitem__(obj, (1,)) obj[(1,), a=3] # calls type(obj).__getitem__(obj, (1,), a=3) </pre></div> </div> This is particularly relevant in the case where two entries are passed: <div class="highlight-default notranslate"><div class="highlight"><pre>obj[1, 2] # calls type(obj).__getitem__(obj, (1, 2)) obj[(1, 2)] # same as above obj[1, 2, a=3] # calls type(obj).__getitem__(obj, (1, 2), a=3) obj[(1, 2), a=3] # calls type(obj).__getitem__(obj, (1, 2), a=3) </pre></div> </div> And particularly when the tuple is extracted as a variable: <div class="highlight-default notranslate"><div class="highlight"><pre>t = (1, 2) obj[t] # calls type(obj).__getitem__(obj, (1, 2)) obj[t, a=3] # calls type(obj).__getitem__(obj, (1, 2), a=3) </pre></div> </div> Why? because in the case <code class="docutils literal notranslate">obj[1, 2, a=3]</code> we are passing two elements (which are then packed as a tuple and passed as the index). In the case <code class="docutils literal notranslate">obj[(1, 2), a=3]</code> we are passing a single element (which is passed as is) which happens to be a tuple. The final result is that they are the same. </li> </ol> </section> </section> <section id="c-interface"> <h2><a class="toc-backref" href="#c-interface" role="doc-backlink">C Interface</a></h2> Resolution of the indexing operation is performed through a call to the following functions <ul class="simple"> <li><code class="docutils literal notranslate">PyObject_GetItem(PyObject *o, PyObject *key)</code> for the get operation</li> <li><code class="docutils literal notranslate">PyObject_SetItem(PyObject *o, PyObject *key, PyObject *value)</code> for the set operation</li> <li><code class="docutils literal notranslate">PyObject_DelItem(PyObject *o, PyObject *key)</code> for the del operation</li> </ul> These functions are used extensively within the Python executable, and are also part of the public C API, as exported by <code class="docutils literal notranslate">Include/abstract.h</code>. It is clear that the signature of this function cannot be changed, and different C level functions need to be implemented to support the extended call. We propose <ul class="simple"> <li><code class="docutils literal notranslate">PyObject_GetItemWithKeywords(PyObject *o, PyObject *key, PyObject *kwargs)</code></li> <li><code class="docutils literal notranslate">PyObject_SetItemWithKeywords(PyObject *o, PyObject *key, PyObject *value, PyObject *kwargs)</code></li> <li><code class="docutils literal notranslate">PyObject_GetItemWithKeywords(PyObject *o, PyObject *key, PyObject *kwargs)</code></li> </ul> New opcodes will be needed for the enhanced call. Currently, the implementation uses <code class="docutils literal notranslate">BINARY_SUBSCR</code>, <code class="docutils literal notranslate">STORE_SUBSCR</code> and <code class="docutils literal notranslate">DELETE_SUBSCR</code> to invoke the old functions. We propose <code class="docutils literal notranslate">BINARY_SUBSCR_KW</code>, <code class="docutils literal notranslate">STORE_SUBSCR_KW</code> and <code class="docutils literal notranslate">DELETE_SUBSCR_KW</code> for the new operations. The compiler will have to generate these new opcodes. The old C implementations will call the extended methods passing <code class="docutils literal notranslate">NULL</code> as kwargs. Finally, the following new slots must be added to the <code class="docutils literal notranslate">PyMappingMethods</code> struct: <ul class="simple"> <li><code class="docutils literal notranslate">mp_subscript_kw</code></li> <li><code class="docutils literal notranslate">mp_ass_subscript_kw</code></li> </ul> These slots will have the appropriate signature to handle the dictionary object containing the keywords. </section> <section id="how-to-teach-recommendations"> <h2><a class="toc-backref" href="#how-to-teach-recommendations" role="doc-backlink">“How to teach” recommendations</a></h2> One request that occurred during feedback sessions was to detail a possible narrative for teaching the feature, e.g. to students, data scientists, and similar audience. This section addresses that need. We will only describe the indexing from the perspective of use, not of implementation, because it is the aspect that the above mentioned audience will likely encounter. Only a subset of the users will have to implement their own dunder functions, and can be considered advanced usage. A proper explanation could be: <blockquote> <div>The indexing operation is generally used to refer to a subset of a larger dataset by means of an index. In the commonly seen cases, the index is made by one or more numbers, strings, slices, etc.Some types may allow indexing to occur not only with the index, but also with named values. These named values are given between square brackets using the same syntax used for function call keyword arguments. The meaning of the names and their use is found in the documentation of the type, as it varies from one type to another. </div></blockquote> The teacher will now show some practical real world examples, explaining the semantics of the feature in the shown library. At the time of writing these examples do not exist, obviously, but the libraries most likely to implement the feature are pandas and numpy, possibly as a method to refer to columns by name. </section> <section id="reference-implementation"> <h2><a class="toc-backref" href="#reference-implementation" role="doc-backlink">Reference Implementation</a></h2> A reference implementation is currently being developed here <a class="footnote-reference brackets" href="#reference-impl" id="id5">[6]</a>. </section> <section id="workarounds"> <h2><a class="toc-backref" href="#workarounds" role="doc-backlink">Workarounds</a></h2> Every PEP that changes the Python language should <a class="pep reference internal" href="../pep-0001/#what-belongs-in-a-successful-pep" title="PEP 1 – PEP Purpose and Guidelines § What belongs in a successful PEP?">“clearly explain why the existing language specification is inadequate to address the problem that the PEP solves”</a>. Some rough equivalents to the proposed extension, which we call work-arounds, are already possible. The work-arounds provide an alternative to enabling the new syntax, while leaving the semantics to be defined elsewhere. These work-arounds follow. In them the helpers <code class="docutils literal notranslate">H</code> and <code class="docutils literal notranslate">P</code> are not intended to be universal. For example, a module or package might require the use of its own helpers. <ol class="arabic"> <li>User defined classes can be given <code class="docutils literal notranslate">getitem</code> and <code class="docutils literal notranslate">delitem</code> methods, that respectively get and delete values stored in a container:<div class="highlight-default notranslate"><div class="highlight"><pre>>>> val = x.getitem(1, 2, a=3, b=4) >>> x.delitem(1, 2, a=3, b=4) </pre></div> </div> The same can’t be done for <code class="docutils literal notranslate">setitem</code>. It’s not valid syntax: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> x.setitem(1, 2, a=3, b=4) = val SyntaxError: can't assign to function call </pre></div> </div> </li> <li>A helper class, here called <code class="docutils literal notranslate">H</code>, can be used to swap the container and parameter roles. In other words, we use:<div class="highlight-default notranslate"><div class="highlight"><pre>H(1, 2, a=3, b=4)[x] </pre></div> </div> as a substitute for: <div class="highlight-default notranslate"><div class="highlight"><pre>x[1, 2, a=3, b=4] </pre></div> </div> This method will work for <code class="docutils literal notranslate">getitem</code>, <code class="docutils literal notranslate">delitem</code> and also for <code class="docutils literal notranslate">setitem</code>. This is because: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> H(1, 2, a=3, b=4)[x] = val </pre></div> </div> is valid syntax, which can be given the appropriate semantics. </li> <li>A helper function, here called <code class="docutils literal notranslate">P</code>, can be used to store the arguments in a single object. For example:<div class="highlight-default notranslate"><div class="highlight"><pre>>>> x[P(1, 2, a=3, b=4)] = val </pre></div> </div> is valid syntax, and can be given the appropriate semantics. </li> <li>The <code class="docutils literal notranslate">lo:hi:step</code> syntax for slices is sometimes very useful. This syntax is not directly available in the work-arounds. However:<div class="highlight-default notranslate"><div class="highlight"><pre>s[lo:hi:step] </pre></div> </div> provides a work-around that is available everything, where: <div class="highlight-default notranslate"><div class="highlight"><pre>class S: def __getitem__(self, key): return key s = S() </pre></div> </div> defines the helper object <code class="docutils literal notranslate">s</code>. </li> </ol> </section> <section id="rejected-ideas"> <h2><a class="toc-backref" href="#rejected-ideas" role="doc-backlink">Rejected Ideas</a></h2> <section id="previous-pep-472-solutions"> <h3><a class="toc-backref" href="#previous-pep-472-solutions" role="doc-backlink">Previous PEP 472 solutions</a></h3> <a class="pep reference internal" href="../pep-0472/" title="PEP 472 – Support for indexing with keyword arguments">PEP 472</a> presents a good amount of ideas that are now all to be considered Rejected. A personal email from D’Aprano to the author specifically said: <blockquote> <div>I have now carefully read through PEP 472 in full, and I am afraid I cannot support any of the strategies currently in the PEP.</div></blockquote> We agree that those options are inferior to the currently presented, for one reason or another. To keep this document compact, we will not present here the objections for all options presented in <a class="pep reference internal" href="../pep-0472/" title="PEP 472 – Support for indexing with keyword arguments">PEP 472</a>. Suffice to say that they were discussed, and each proposed alternative had one or few dealbreakers. </section> <section id="adding-new-dunders"> <h3><a class="toc-backref" href="#adding-new-dunders" role="doc-backlink">Adding new dunders</a></h3> It was proposed to introduce new dunders <code class="docutils literal notranslate">__(get|set|del)item_ex__</code> that are invoked over the <code class="docutils literal notranslate">__(get|set|del)item__</code> triad, if they are present. The rationale around this choice is to make the intuition around how to add kwd arg support to square brackets more obvious and in line with the function behavior. Given: <div class="highlight-default notranslate"><div class="highlight"><pre>def __getitem_ex__(self, x, y): ... </pre></div> </div> These all just work and produce the same result effortlessly: <div class="highlight-default notranslate"><div class="highlight"><pre>obj[1, 2] obj[1, y=2] obj[y=2, x=1] </pre></div> </div> In other words, this solution would unify the behavior of <code class="docutils literal notranslate">__getitem__</code> to the traditional function signature, but since we can’t change <code class="docutils literal notranslate">__getitem__</code> and break backward compatibility, we would have an extended version that is used preferentially. The problems with this approach were found to be: <ul> <li>It will slow down subscripting. For every subscript access, this new dunder attribute gets investigated on the class, and if it is not present then the default key translation function is executed. Different ideas were proposed to handle this, from wrapping the method only at class instantiation time, to add a bit flag to signal the availability of these methods. Regardess of the solution, the new dunder would be effective only if added at class creation time, not if it’s added later. This would be unusual and would disallow (and behave unexpectedly) monkeypatching of the methods for whatever reason it might be needed.</li> <li>It adds complexity to the mechanism.</li> <li>Will require a long and painful transition period during which time libraries will have to somehow support both calling conventions, because most likely, the extended methods will delegate to the traditional ones when the right conditions are matched in the arguments, or some classes will support the traditional dunder and others the extended dunder. While this will not affect calling code, it will affect development.</li> <li>it would potentially lead to mixed situations where the extended version is defined for the getter, but not for the setter.</li> <li>In the <code class="docutils literal notranslate">__setitem_ex__</code> signature, value would have to be made the first element, because the index is of arbitrary length depending on the specified indexes. This would look awkward because the visual notation does not match the signature:<div class="highlight-default notranslate"><div class="highlight"><pre>obj[1, 2] = 3 # calls type(obj).__setitem_ex__(obj, 3, 1, 2) </pre></div> </div> </li> <li>the solution relies on the assumption that all keyword indices necessarily map into positional indices, or that they must have a name. This assumption may be false: xarray, which is the primary Python package for numpy arrays with labelled dimensions, supports indexing by additional dimensions (so called “non-dimension coordinates”) that don’t correspond directly to the dimensions of the underlying numpy array, and those have no position to match up to. In other words, anonymous indexes are a plausible use case that this solution would remove, although it could be argued that using <code class="docutils literal notranslate">*args</code> would solve that issue.</li> </ul> </section> <section id="adding-an-adapter-function"> <h3><a class="toc-backref" href="#adding-an-adapter-function" role="doc-backlink">Adding an adapter function</a></h3> Similar to the above, in the sense that a pre-function would be called to convert the “new style” indexing into “old style indexing” that is then passed. Has problems similar to the above. </section> <section id="create-a-new-kwslice-object"> <h3><a class="toc-backref" href="#create-a-new-kwslice-object" role="doc-backlink">create a new “kwslice” object</a></h3> This proposal has already been explored in “New arguments contents” P4 in <a class="pep reference internal" href="../pep-0472/" title="PEP 472 – Support for indexing with keyword arguments">PEP 472</a>: <div class="highlight-default notranslate"><div class="highlight"><pre>obj[a, b:c, x=1] # calls type(obj).__getitem__(obj, a, slice(b, c), key(x=1)) </pre></div> </div> This solution requires everyone who needs keyword arguments to parse the tuple and/or key object by hand to extract them. This is painful and opens up to the get/set/del function to always accept arbitrary keyword arguments, whether they make sense or not. We want the developer to be able to specify which arguments make sense and which ones do not. </section> <section id="using-a-single-bit-to-change-the-behavior"> <h3><a class="toc-backref" href="#using-a-single-bit-to-change-the-behavior" role="doc-backlink">Using a single bit to change the behavior</a></h3> A special class dunder flag: <div class="highlight-default notranslate"><div class="highlight"><pre>__keyfn__ = True </pre></div> </div> would change the signature of the <code class="docutils literal notranslate">__get|set|delitem__</code> to a “function like” dispatch, meaning that this: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> d[1, 2, z=3] </pre></div> </div> would result in a call to: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> type(obj).__getitem__(obj, 1, 2, z=3) # instead of type(obj).__getitem__(obj, (1, 2), z=3) </pre></div> </div> This option has been rejected because it feels odd that a signature of a method depends on a specific value of another dunder. It would be confusing for both static type checkers and for humans: a static type checker would have to hard-code a special case for this, because there really is nothing else in Python where the signature of a dunder depends on the value of another dunder. A human that has to implement a <code class="docutils literal notranslate">__getitem__</code> dunder would have to look if in the class (or in any of its subclasses) for a <code class="docutils literal notranslate">__keyfn__</code> before the dunder can be written. Moreover, adding a base classes that have the <code class="docutils literal notranslate">__keyfn__</code> flag set would break the signature of the current methods. This would be even more problematic if the flag is changed at runtime, or if the flag is generated by calling a function that returns randomly True or something else. </section> <section id="allowing-for-empty-index-notation-obj"> <h3><a class="toc-backref" href="#allowing-for-empty-index-notation-obj" role="doc-backlink">Allowing for empty index notation obj[]</a></h3> The current proposal prevents <code class="docutils literal notranslate">obj[]</code> from being valid notation. However a commenter stated <blockquote> <div>We have <code class="docutils literal notranslate">Tuple[int, int]</code> as a tuple of two integers. And we have <code class="docutils literal notranslate">Tuple[int]</code> as a tuple of one integer. And occasionally we need to spell a tuple of no values, since that’s the type of <code class="docutils literal notranslate">()</code>. But we currently are forced to write that as <code class="docutils literal notranslate">Tuple[()]</code>. If we allowed <code class="docutils literal notranslate">Tuple[]</code> that odd edge case would be removed.So I probably would be okay with allowing <code class="docutils literal notranslate">obj[]</code> syntactically, as long as the dict type could be made to reject it. </div></blockquote> This proposal already established that, in case no positional index is given, the passed value must be the empty tuple. Allowing for the empty index notation would make the dictionary type accept it automatically, to insert or refer to the value with the empty tuple as key. Moreover, a typing notation such as <code class="docutils literal notranslate">Tuple[]</code> can easily be written as <code class="docutils literal notranslate">Tuple</code> without the indexing notation. However, subsequent discussion with Brandt Bucher during implementation has revealed that the case <code class="docutils literal notranslate">obj[]</code> would fit a natural evolution for variadic generics, giving more strength to the above comment. In the end, after a discussion between D’Aprano, Bucher and the author, we decided to leave the <code class="docutils literal notranslate">obj[]</code> notation as a syntax error for now, and possibly extend the notation with an additional PEP to hold the equivalence <code class="docutils literal notranslate">obj[]</code> as <code class="docutils literal notranslate">obj[()]</code>. </section> <section id="sentinel-value-for-no-given-positional-index"> <h3><a class="toc-backref" href="#sentinel-value-for-no-given-positional-index" role="doc-backlink">Sentinel value for no given positional index</a></h3> The topic of which value to pass as the index in the case of: <div class="highlight-default notranslate"><div class="highlight"><pre>obj[k=3] </pre></div> </div> has been considerably debated. One apparently rational choice would be to pass no value at all, by making use of the keyword only argument feature, but unfortunately will not work well with the <code class="docutils literal notranslate">__setitem__</code> dunder, as a positional element for the value is always passed, and we can’t “skip over” the index one unless we introduce a very weird behavior where the first argument refers to the index when specified, and to the value when index is not specified. This is extremely deceiving and error prone. The above consideration makes it impossible to have a keyword only dunder, and opens up the question of what entity to pass for the index position when no index is passed: <div class="highlight-default notranslate"><div class="highlight"><pre>obj[k=3] = 5 # would call type(obj).__setitem__(obj, ???, 5, k=3) </pre></div> </div> A proposed hack would be to let the user specify which entity to use when an index is not specified, by specifying a default for the <code class="docutils literal notranslate">index</code>, but this forces necessarily to also specify a (never going to be used, as a value is always passed by design) default for the <code class="docutils literal notranslate">value</code>, as we can’t have non-default arguments after defaulted one: <div class="highlight-default notranslate"><div class="highlight"><pre>def __setitem__(self, index=SENTINEL, value=NEVERUSED, *, k) </pre></div> </div> which seems ugly, redundant and confusing. We must therefore accept that some form of sentinel index must be passed by the Python implementation when the <code class="docutils literal notranslate">obj[k=3]</code> notation is used. This also means that default arguments to those parameters are simply never going to be used (but it’s already the case with the current implementation, so no change there). Additionally, some classes may want to use <code class="docutils literal notranslate">**kwargs</code>, instead of a keyword-only argument, meaning that having a definition like: <div class="highlight-default notranslate"><div class="highlight"><pre>def __setitem__(self, index, value, **kwargs): </pre></div> </div> and a user that wants to pass a keyword <code class="docutils literal notranslate">value</code>: <div class="highlight-default notranslate"><div class="highlight"><pre>x[value=1] = 0 </pre></div> </div> expecting a call like: <div class="highlight-default notranslate"><div class="highlight"><pre>type(obj).__setitem__(obj, SENTINEL, 0, **{"value": 1}) </pre></div> </div> will instead accidentally be caught by the named <code class="docutils literal notranslate">value</code>, producing a <code class="docutils literal notranslate">duplicate value error</code>. The user should not be worried about the actual local names of those two arguments if they are, for all practical purposes, positional only. Unfortunately, using positional-only values will ensure this does not happen but it will still not solve the need to pass both <code class="docutils literal notranslate">index</code> and <code class="docutils literal notranslate">value</code> even when the index is not provided. The point is that the user should not be prevented to use keyword arguments to refer to a column <code class="docutils literal notranslate">index</code>, <code class="docutils literal notranslate">value</code> (or <code class="docutils literal notranslate">self</code>) just because the class implementor happens to use those names in the parameter list. Moreover, we also require the three dunders to behave in the same way: it would be extremely inconvenient if only <code class="docutils literal notranslate">__setitem__</code> were to receive this sentinel, and <code class="docutils literal notranslate">__get|delitem__</code> would not because they can get away with a signature that allows for no index specification, thus allowing for a user-specified default index. Whatever the choice of the sentinel, it will make the following cases degenerate and thus impossible to differentiate in the dunder: <div class="highlight-default notranslate"><div class="highlight"><pre>obj[k=3] obj[SENTINEL, k=3] </pre></div> </div> The question now shifts to which entity should represent the sentinel: the options were: <ol class="arabic simple"> <li>Empty tuple</li> <li>None</li> <li>NotImplemented</li> <li>a new sentinel object (e.g. NoIndex)</li> </ol> For option 1, the call will become: <div class="highlight-default notranslate"><div class="highlight"><pre>type(obj).__getitem__(obj, (), k=3) </pre></div> </div> therefore making <code class="docutils literal notranslate">obj[k=3]</code> and <code class="docutils literal notranslate">obj[(), k=3]</code> degenerate and indistinguishable. This option sounds appealing because: <ol class="arabic"> <li>The numpy community was inquired <a class="footnote-reference brackets" href="#numpy-ml" id="id6">[5]</a>, and the general consensus of the responses was that the empty tuple felt appropriate.</li> <li>It shows a parallel with the behavior of <code class="docutils literal notranslate">*args</code> in a function, when no positional arguments are given:<div class="highlight-default notranslate"><div class="highlight"><pre>>>> def foo(*args, **kwargs): ... print(args, kwargs) ... >>> foo(k=3) () {'k': 3} </pre></div> </div> Although we do accept the following asymmetry in behavior compared to functions when a single value is passed, but that ship has sailed: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> foo(5, k=3) (5,) {'k': 3} # for indexing, a plain 5, not a 1-tuple is passed </pre></div> </div> </li> </ol> For option 2, using <code class="docutils literal notranslate">None</code>, it was objected that NumPy uses it to indicate inserting a new axis/dimensions (there’s a <code class="docutils literal notranslate">np.newaxis</code> alias as well): <div class="highlight-default notranslate"><div class="highlight"><pre>arr = np.array(5) arr.ndim == 0 arr[None].ndim == arr[None,].ndim == 1 </pre></div> </div> While this is not an insurmountable issue, it certainly will ripple onto numpy. The only issues with both the above is that both the empty tuple and None are potential legitimate indexes, and there might be value in being able to differentiate the two degenerate cases. So, an alternative strategy (option 3) would be to use an existing entity that is unlikely to be used as a valid index. One option could be the current built-in constant <code class="docutils literal notranslate">NotImplemented</code>, which is currently returned by operators methods to report that they do not implement a particular operation, and a different strategy should be attempted (e.g. to ask the other object). Unfortunately, its name and traditional use calls back to a feature that is not available, rather than the fact that something was not passed by the user. This leaves us with option 4: a new built-in constant. This constant must be unhashable (so it’s never going to be a valid key) and have a clear name that makes it obvious its context: <code class="docutils literal notranslate">NoIndex</code>. This would solve all the above issues, but the question is: is it worth it? From a quick inquire, it seems that most people on python-ideas seem to believe it’s not crucial, and the empty tuple is an acceptable option. Hence the resulting series will be: <div class="highlight-default notranslate"><div class="highlight"><pre>obj[k=3] # type(obj).__getitem__(obj, (), k=3). Empty tuple obj[1, k=3] # type(obj).__getitem__(obj, 1, k=3). Integer obj[1, 2, k=3] # type(obj).__getitem__(obj, (1, 2), k=3). Tuple </pre></div> </div> and the following two notation will be degenerate: <div class="highlight-default notranslate"><div class="highlight"><pre>obj[(), k=3] # type(obj).__getitem__(obj, (), k=3) obj[k=3] # type(obj).__getitem__(obj, (), k=3) </pre></div> </div> </section> </section> <section id="common-objections"> <h2><a class="toc-backref" href="#common-objections" role="doc-backlink">Common objections</a></h2> <ol class="arabic"> <li>Just use a method call.One of the use cases is typing, where the indexing is used exclusively, and function calls are out of the question. Moreover, function calls do not handle slice notation, which is commonly used in some cases for arrays. One problem is type hint creation has been extended to built-ins in Python 3.9, so that you do not have to import Dict, List, et al anymore. Without kwdargs inside <code class="docutils literal notranslate">[]</code>, you would not be able to do this: <div class="highlight-default notranslate"><div class="highlight"><pre>Vector = dict[i=float, j=float] </pre></div> </div> but for obvious reasons, call syntax using builtins to create custom type hints isn’t an option: <div class="highlight-default notranslate"><div class="highlight"><pre>dict(i=float, j=float) # would create a dictionary, not a type </pre></div> </div> Finally, function calls do not allow for a setitem-like notation, as shown in the Overview: operations such as <code class="docutils literal notranslate">f(1, x=3) = 5</code> are not allowed, and are instead allowed for indexing operations. </li> </ol> </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <aside class="footnote-list brackets"> <aside class="footnote brackets" id="rejection" role="doc-footnote"> <dt class="label" id="rejection">[<a href="#id1">1</a>]</dt> <dd>“Rejection of PEP 472” (<a class="reference external" href="https://mail.python.org/pipermail/python-dev/2019-March/156693.html">https://mail.python.org/pipermail/python-dev/2019-March/156693.html</a>)</aside> <aside class="footnote brackets" id="request-1" role="doc-footnote"> <dt class="label" id="request-1">[<a href="#id2">2</a>]</dt> <dd>“Allow kwargs in __{get|set|del}item__” (<a class="reference external" href="https://mail.python.org/archives/list/python-ideas@python.org/thread/EUGDRTRFIY36K4RM3QRR52CKCI7MIR2M/">https://mail.python.org/archives/list/python-ideas@python.org/thread/EUGDRTRFIY36K4RM3QRR52CKCI7MIR2M/</a>)</aside> <aside class="footnote brackets" id="request-2" role="doc-footnote"> <dt class="label" id="request-2">[<a href="#id3">3</a>]</dt> <dd>“PEP 472 – Support for indexing with keyword arguments” (<a class="reference external" href="https://mail.python.org/archives/list/python-ideas@python.org/thread/6OGAFDWCXT5QVV23OZWKBY4TXGZBVYZS/">https://mail.python.org/archives/list/python-ideas@python.org/thread/6OGAFDWCXT5QVV23OZWKBY4TXGZBVYZS/</a>)</aside> <aside class="footnote brackets" id="trio-run" role="doc-footnote"> <dt class="label" id="trio-run">[<a href="#id4">4</a>]</dt> <dd>“trio.run() should take **kwargs in addition to *args” (<a class="reference external" href="https://github.com/python-trio/trio/issues/470">https://github.com/python-trio/trio/issues/470</a>)</aside> <aside class="footnote brackets" id="numpy-ml" role="doc-footnote"> <dt class="label" id="numpy-ml">[<a href="#id6">5</a>]</dt> <dd>“[Numpy-discussion] Request for comments on PEP 637 - Support for indexing with keyword arguments” (<a class="reference external" href="http://numpy-discussion.10968.n7.nabble.com/Request-for-comments-on-PEP-637-Support-for-indexing-with-keyword-arguments-td48489.html">http://numpy-discussion.10968.n7.nabble.com/Request-for-comments-on-PEP-637-Support-for-indexing-with-keyword-arguments-td48489.html</a>)</aside> <aside class="footnote brackets" id="reference-impl" role="doc-footnote"> <dt class="label" id="reference-impl">[<a href="#id5">6</a>]</dt> <dd>“Reference implementation” (<a class="reference external" href="https://github.com/python/cpython/compare/master...stefanoborini:PEP-637-implementation-attempt-2">https://github.com/python/cpython/compare/master…stefanoborini:PEP-637-implementation-attempt-2</a>)</aside> </aside> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0637.rst">https://github.com/python/peps/blob/main/peps/pep-0637.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0637.rst">2025-02-01 08:59:27 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#overview">Overview</a><ul> <li><a class="reference internal" href="#background">Background</a></li> <li><a class="reference internal" href="#use-cases">Use cases</a></li> <li><a class="reference internal" href="#current-status-of-indexing-operation">Current status of indexing operation</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#indexing-behavior-in-standard-classes-dict-list-etc">Indexing behavior in standard classes (dict, list, etc.)</a></li> <li><a class="reference internal" href="#corner-case-and-gotchas">Corner case and Gotchas</a></li> </ul> </li> <li><a class="reference internal" href="#c-interface">C Interface</a></li> <li><a class="reference internal" href="#how-to-teach-recommendations">“How to teach” recommendations</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#workarounds">Workarounds</a></li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a><ul> <li><a class="reference internal" href="#previous-pep-472-solutions">Previous PEP 472 solutions</a></li> <li><a class="reference internal" href="#adding-new-dunders">Adding new dunders</a></li> <li><a class="reference internal" href="#adding-an-adapter-function">Adding an adapter function</a></li> <li><a class="reference internal" href="#create-a-new-kwslice-object">create a new “kwslice” object</a></li> <li><a class="reference internal" href="#using-a-single-bit-to-change-the-behavior">Using a single bit to change the behavior</a></li> <li><a class="reference internal" href="#allowing-for-empty-index-notation-obj">Allowing for empty index notation obj[]</a></li> <li><a class="reference internal" href="#sentinel-value-for-no-given-positional-index">Sentinel value for no given positional index</a></li> </ul> </li> <li><a class="reference internal" href="#common-objections">Common objections</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0637.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>