CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 576 – Rationalize Built-in function classes | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0576/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 576 – Rationalize Built-in function classes | peps.python.org'> <meta property="og:description" content="Expose the “FastcallKeywords” convention used internally by CPython to third-party code, and make the inspect module use duck-typing. In combination this will allow third-party C extensions and tools like Cython to create objects that use the same calli..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0576/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="Expose the “FastcallKeywords” convention used internally by CPython to third-party code, and make the inspect module use duck-typing. In combination this will allow third-party C extensions and tools like Cython to create objects that use the same calli..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 576</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> <span class="visually-hidden">Toggle light / dark / auto colour theme</span> </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 576 – Rationalize Built-in function classes</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author<span class="colon">:</span></dt> <dd class="field-odd">Mark Shannon <mark at hotpy.org></dd> <dt class="field-even">BDFL-Delegate<span class="colon">:</span></dt> <dd class="field-even">Petr Viktorin</dd> <dt class="field-odd">Status<span class="colon">:</span></dt> <dd class="field-odd"><abbr title="Removed from consideration by sponsor or authors">Withdrawn</abbr></dd> <dt class="field-even">Type<span class="colon">:</span></dt> <dd class="field-even"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-odd">Created<span class="colon">:</span></dt> <dd class="field-odd">10-May-2018</dd> <dt class="field-even">Python-Version<span class="colon">:</span></dt> <dd class="field-even">3.8</dd> <dt class="field-odd">Post-History<span class="colon">:</span></dt> <dd class="field-odd">17-May-2018, 23-Jun-2018, 08-Jul-2018, 29-Mar-2019</dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#motivation">Motivation</a><ul> <li><a class="reference internal" href="#introspection">Introspection</a></li> <li><a class="reference internal" href="#efficient-calls-to-third-party-callables">Efficient calls to third-party callables</a><ul> <li><a class="reference internal" href="#continued-prohibition-of-callable-classes-as-base-classes">Continued prohibition of callable classes as base classes</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#new-classes-and-changes-to-existing-classes">New classes and changes to existing classes</a><ul> <li><a class="reference internal" href="#python-visible-changes">Python visible changes</a></li> <li><a class="reference internal" href="#c-api-changes">C API changes</a></li> </ul> </li> <li><a class="reference internal" href="#retaining-backwards-compatibility-in-the-c-api-and-abi">Retaining backwards compatibility in the C API and ABI</a><ul> <li><a class="reference internal" href="#internal-c-changes">Internal C changes</a><ul> <li><a class="reference internal" href="#py-tpflags-extended-call">Py_TPFLAGS_EXTENDED_CALL</a></li> <li><a class="reference internal" href="#py-tpflags-function-descriptor">Py_TPFLAGS_FUNCTION_DESCRIPTOR</a></li> <li><a class="reference internal" href="#changes-to-existing-c-structs">Changes to existing C structs</a></li> </ul> </li> <li><a class="reference internal" href="#third-party-built-in-classes-using-the-new-extended-call-interface">Third-party built-in classes using the new extended call interface</a><ul> <li><a class="reference internal" href="#performance-implications-of-these-changes">Performance implications of these changes</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#alternative-suggestions">Alternative Suggestions</a></li> <li><a class="reference internal" href="#reference-implementation">Reference implementation</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> <p>Expose the “FastcallKeywords” convention used internally by CPython to third-party code, and make the <code class="docutils literal notranslate"><span class="pre">inspect</span></code> module use duck-typing. In combination this will allow third-party C extensions and tools like Cython to create objects that use the same calling conventions as built-in and Python functions, thus gaining performance parity with built-in functions like <code class="docutils literal notranslate"><span class="pre">len</span></code> or <code class="docutils literal notranslate"><span class="pre">print</span></code>.</p> <p>A small improvement in the performance of existing code is expected.</p> </section> <section id="motivation"> <h2><a class="toc-backref" href="#motivation" role="doc-backlink">Motivation</a></h2> <p>Currently third-party module authors face a dilemma when implementing functions in C. Either they can use one of the pre-existing built-in function or method classes or implement their own custom class in C. The first choice causes them to lose the ability to access the internals of the callable object. The second choice is an additional maintenance burden and, more importantly, has a significant negative impact on performance.</p> <p>This PEP aims to allow authors of third-party C modules, and tools like to Cython, to utilize the faster calling convention used internally by CPython for built-in functions and methods, and to do so without a loss of capabilities relative to a function implemented in Python.</p> <section id="introspection"> <h3><a class="toc-backref" href="#introspection" role="doc-backlink">Introspection</a></h3> <p>The inspect module will fully support duck-typing when introspecting callables.</p> <p>The <code class="docutils literal notranslate"><span class="pre">inspect.Signature.from_callable()</span></code> function computes the signature of a callable. If an object has a <code class="docutils literal notranslate"><span class="pre">__signature__</span></code> property, then <code class="docutils literal notranslate"><span class="pre">inspect.Signature.from_callable()</span></code> simply returns that. To further support duck-typing, if a callable has a <code class="docutils literal notranslate"><span class="pre">__text_signature__</span></code> then the <code class="docutils literal notranslate"><span class="pre">__signature__</span></code> will be created from that.</p> <p>This means that 3rd party builtin-functions can implement <code class="docutils literal notranslate"><span class="pre">__text_signature__</span></code> if sufficient, and the more expensive <code class="docutils literal notranslate"><span class="pre">__signature__</span></code> if necessary.</p> </section> <section id="efficient-calls-to-third-party-callables"> <h3><a class="toc-backref" href="#efficient-calls-to-third-party-callables" role="doc-backlink">Efficient calls to third-party callables</a></h3> <p>Currently the majority of calls are dispatched to <code class="docutils literal notranslate"><span class="pre">function</span></code>s and <code class="docutils literal notranslate"><span class="pre">method_descriptor</span></code>s in custom code, using the “FastcallKeywords” internal calling convention. This PEP proposes that this calling convention is implemented via a C function pointer. Third-party callables which implement this binary interface will have the potential to be called as fast as a built-in function.</p> <section id="continued-prohibition-of-callable-classes-as-base-classes"> <h4><a class="toc-backref" href="#continued-prohibition-of-callable-classes-as-base-classes" role="doc-backlink">Continued prohibition of callable classes as base classes</a></h4> <p>Currently any attempt to use <code class="docutils literal notranslate"><span class="pre">function</span></code>, <code class="docutils literal notranslate"><span class="pre">method</span></code> or <code class="docutils literal notranslate"><span class="pre">method_descriptor</span></code> as a base class for a new class will fail with a <code class="docutils literal notranslate"><span class="pre">TypeError</span></code>. This behaviour is desirable as it prevents errors when a subclass overrides the <code class="docutils literal notranslate"><span class="pre">__call__</span></code> method. If callables could be sub-classed then any call to a <code class="docutils literal notranslate"><span class="pre">function</span></code> or a <code class="docutils literal notranslate"><span class="pre">method_descriptor</span></code> would need an additional check that the <code class="docutils literal notranslate"><span class="pre">__call__</span></code> method had not been overridden. By exposing an additional call mechanism, the potential for errors becomes greater. As a consequence, any third-party class implementing the addition call interface will not be usable as a base class.</p> </section> </section> </section> <section id="new-classes-and-changes-to-existing-classes"> <h2><a class="toc-backref" href="#new-classes-and-changes-to-existing-classes" role="doc-backlink">New classes and changes to existing classes</a></h2> <section id="python-visible-changes"> <h3><a class="toc-backref" href="#python-visible-changes" role="doc-backlink">Python visible changes</a></h3> <ol class="arabic simple"> <li>A new built-in class, <code class="docutils literal notranslate"><span class="pre">builtin_function</span></code>, will be added.</li> <li><code class="docutils literal notranslate"><span class="pre">types.BuiltinFunctionType</span></code> will refer to <code class="docutils literal notranslate"><span class="pre">builtin_function</span></code> not <code class="docutils literal notranslate"><span class="pre">builtin_function_or_method</span></code>.</li> <li>Instances of the <code class="docutils literal notranslate"><span class="pre">builtin_function</span></code> class will retain the <code class="docutils literal notranslate"><span class="pre">__module__</span></code> property of <code class="docutils literal notranslate"><span class="pre">builtin_function_or_method</span></code> and gain the <code class="docutils literal notranslate"><span class="pre">func_module</span></code> and <code class="docutils literal notranslate"><span class="pre">func_globals</span></code> properties. The <code class="docutils literal notranslate"><span class="pre">func_module</span></code> allows access to the module to which the function belongs. Note that this is different from the <code class="docutils literal notranslate"><span class="pre">__module__</span></code> property which merely returns the name of the module. The <code class="docutils literal notranslate"><span class="pre">func_globals</span></code> property is equivalent to <code class="docutils literal notranslate"><span class="pre">func_module.__dict__</span></code> and is provided to mimic the Python function property of the same name.</li> <li>When binding a <code class="docutils literal notranslate"><span class="pre">method_descriptor</span></code> instance to an instance of its owning class, a <code class="docutils literal notranslate"><span class="pre">bound_method</span></code> will be created instead of a <code class="docutils literal notranslate"><span class="pre">builtin_function_or_method</span></code>. This means that the <code class="docutils literal notranslate"><span class="pre">method_descriptors</span></code> now mimic the behaviour of Python functions more closely. In other words, <code class="docutils literal notranslate"><span class="pre">[].append</span></code> becomes a <code class="docutils literal notranslate"><span class="pre">bound_method</span></code> instead of a <code class="docutils literal notranslate"><span class="pre">builtin_function_or_method</span></code>.</li> </ol> </section> <section id="c-api-changes"> <h3><a class="toc-backref" href="#c-api-changes" role="doc-backlink">C API changes</a></h3> <ol class="arabic simple"> <li>A new function <code class="docutils literal notranslate"><span class="pre">PyBuiltinFunction_New(PyMethodDef</span> <span class="pre">*ml,</span> <span class="pre">PyObject</span> <span class="pre">*module)</span></code> is added to create built-in functions.</li> <li><code class="docutils literal notranslate"><span class="pre">PyCFunction_NewEx()</span></code> and <code class="docutils literal notranslate"><span class="pre">PyCFunction_New()</span></code> are deprecated and will return a <code class="docutils literal notranslate"><span class="pre">PyBuiltinFunction</span></code> if able, otherwise a <code class="docutils literal notranslate"><span class="pre">builtin_function_or_method</span></code>.</li> </ol> </section> </section> <section id="retaining-backwards-compatibility-in-the-c-api-and-abi"> <h2><a class="toc-backref" href="#retaining-backwards-compatibility-in-the-c-api-and-abi" role="doc-backlink">Retaining backwards compatibility in the C API and ABI</a></h2> <p>The proposed changes are fully backwards and forwards compatible at both the API and ABI level.</p> <section id="internal-c-changes"> <h3><a class="toc-backref" href="#internal-c-changes" role="doc-backlink">Internal C changes</a></h3> <p>Two new flags will be allowed for the <code class="docutils literal notranslate"><span class="pre">typeobject.tp_flags</span></code> field. These are <code class="docutils literal notranslate"><span class="pre">Py_TPFLAGS_EXTENDED_CALL</span></code> and <code class="docutils literal notranslate"><span class="pre">Py_TPFLAGS_FUNCTION_DESCRIPTOR</span></code></p> <section id="py-tpflags-extended-call"> <h4><a class="toc-backref" href="#py-tpflags-extended-call" role="doc-backlink">Py_TPFLAGS_EXTENDED_CALL</a></h4> <p>For any built-in class that sets <code class="docutils literal notranslate"><span class="pre">Py_TPFLAGS_EXTENDED_CALL</span></code> The C struct corresponding to this built-in class must begin with the struct <code class="docutils literal notranslate"><span class="pre">PyExtendedCallable</span></code> which is defined as follows:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">typedef</span> <span class="n">PyObject</span> <span class="o">*</span><span class="p">(</span><span class="o">*</span><span class="n">extended_call_ptr</span><span class="p">)(</span><span class="n">PyObject</span> <span class="o">*</span><span class="nb">callable</span><span class="p">,</span> <span class="n">PyObject</span><span class="o">**</span> <span class="n">args</span><span class="p">,</span> <span class="nb">int</span> <span class="n">positional_argcount</span><span class="p">,</span> <span class="n">PyTupleObject</span><span class="o">*</span> <span class="n">kwnames</span><span class="p">);</span> <span class="n">typedef</span> <span class="n">struct</span> <span class="p">{</span> <span class="n">PyObject_HEAD</span> <span class="n">extended_call_ptr</span> <span class="n">ext_call</span><span class="p">;</span> <span class="p">}</span> <span class="n">PyExtendedCallable</span><span class="p">;</span> </pre></div> </div> <p>Any class that sets the <code class="docutils literal notranslate"><span class="pre">Py_TPFLAGS_EXTENDED_CALL</span></code> cannot be used as a base class and a TypeError will be raised if any Python code tries to use it a base class.</p> </section> <section id="py-tpflags-function-descriptor"> <h4><a class="toc-backref" href="#py-tpflags-function-descriptor" role="doc-backlink">Py_TPFLAGS_FUNCTION_DESCRIPTOR</a></h4> <p>If this flag is set for a built-in class <code class="docutils literal notranslate"><span class="pre">F</span></code>, then instances of that class are expected to behave the same as a Python function when used as a class attribute. Specifically, this mean that the value of <code class="docutils literal notranslate"><span class="pre">c.m</span></code> where <code class="docutils literal notranslate"><span class="pre">C.m</span></code> is an instanceof the built-in class <code class="docutils literal notranslate"><span class="pre">F</span></code> (and <code class="docutils literal notranslate"><span class="pre">c</span></code> is an instance of <code class="docutils literal notranslate"><span class="pre">C</span></code>) must be a bound-method binding <code class="docutils literal notranslate"><span class="pre">C.m</span></code> and <code class="docutils literal notranslate"><span class="pre">c</span></code>. Without this flag, it would be impossible for custom callables to behave like Python functions <em>and</em> be efficient as Python or built-in functions.</p> </section> <section id="changes-to-existing-c-structs"> <h4><a class="toc-backref" href="#changes-to-existing-c-structs" role="doc-backlink">Changes to existing C structs</a></h4> <p>The <code class="docutils literal notranslate"><span class="pre">function</span></code>, <code class="docutils literal notranslate"><span class="pre">method_descriptor</span></code> and <code class="docutils literal notranslate"><span class="pre">method</span></code> classes will have their corresponding structs changed to start with the <code class="docutils literal notranslate"><span class="pre">PyExtendedCallable</span></code> struct.</p> </section> </section> <section id="third-party-built-in-classes-using-the-new-extended-call-interface"> <h3><a class="toc-backref" href="#third-party-built-in-classes-using-the-new-extended-call-interface" role="doc-backlink">Third-party built-in classes using the new extended call interface</a></h3> <p>To enable call performance on a par with Python functions and built-in functions, third-party callables should set the <code class="docutils literal notranslate"><span class="pre">Py_TPFLAGS_EXTENDED_CALL</span></code> bit of <code class="docutils literal notranslate"><span class="pre">tp_flags</span></code> and ensure that the corresponding C struct starts with the <code class="docutils literal notranslate"><span class="pre">PyExtendedCallable</span></code>. Any built-in class that has the <code class="docutils literal notranslate"><span class="pre">Py_TPFLAGS_EXTENDED_CALL</span></code> bit set must also implement the <code class="docutils literal notranslate"><span class="pre">tp_call</span></code> function and make sure its behaviour is consistent with the <code class="docutils literal notranslate"><span class="pre">ext_call</span></code> function.</p> <section id="performance-implications-of-these-changes"> <h4><a class="toc-backref" href="#performance-implications-of-these-changes" role="doc-backlink">Performance implications of these changes</a></h4> <p>Adding a function pointer to each callable, rather than each class of callable, enables the choice of dispatching function (the code to shuffle arguments about and do error checking) to be made when the callable object is created rather than when it is called. This should reduce the number of instructions executed between the call-site in the interpreter and the execution of the callee.</p> </section> </section> </section> <section id="alternative-suggestions"> <h2><a class="toc-backref" href="#alternative-suggestions" role="doc-backlink">Alternative Suggestions</a></h2> <p><a class="pep reference internal" href="../pep-0580/" title="PEP 580 – The C call protocol">PEP 580</a> is an alternative approach to solving the same problem as this PEP.</p> </section> <section id="reference-implementation"> <h2><a class="toc-backref" href="#reference-implementation" role="doc-backlink">Reference implementation</a></h2> <p>A draft implementation can be found at <a class="reference external" href="https://github.com/markshannon/cpython/tree/pep-576-minimal">https://github.com/markshannon/cpython/tree/pep-576-minimal</a></p> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> <p>This document has been placed in the public domain.</p> </section> </section> <hr class="docutils" /> <p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0576.rst">https://github.com/python/peps/blob/main/peps/pep-0576.rst</a></p> <p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0576.rst">2023-09-09 17:39:29 GMT</a></p> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#motivation">Motivation</a><ul> <li><a class="reference internal" href="#introspection">Introspection</a></li> <li><a class="reference internal" href="#efficient-calls-to-third-party-callables">Efficient calls to third-party callables</a><ul> <li><a class="reference internal" href="#continued-prohibition-of-callable-classes-as-base-classes">Continued prohibition of callable classes as base classes</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#new-classes-and-changes-to-existing-classes">New classes and changes to existing classes</a><ul> <li><a class="reference internal" href="#python-visible-changes">Python visible changes</a></li> <li><a class="reference internal" href="#c-api-changes">C API changes</a></li> </ul> </li> <li><a class="reference internal" href="#retaining-backwards-compatibility-in-the-c-api-and-abi">Retaining backwards compatibility in the C API and ABI</a><ul> <li><a class="reference internal" href="#internal-c-changes">Internal C changes</a><ul> <li><a class="reference internal" href="#py-tpflags-extended-call">Py_TPFLAGS_EXTENDED_CALL</a></li> <li><a class="reference internal" href="#py-tpflags-function-descriptor">Py_TPFLAGS_FUNCTION_DESCRIPTOR</a></li> <li><a class="reference internal" href="#changes-to-existing-c-structs">Changes to existing C structs</a></li> </ul> </li> <li><a class="reference internal" href="#third-party-built-in-classes-using-the-new-extended-call-interface">Third-party built-in classes using the new extended call interface</a><ul> <li><a class="reference internal" href="#performance-implications-of-these-changes">Performance implications of these changes</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#alternative-suggestions">Alternative Suggestions</a></li> <li><a class="reference internal" href="#reference-implementation">Reference implementation</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <br> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0576.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>