CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 579 – Refactoring C functions and methods | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0579/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 579 – Refactoring C functions and methods | peps.python.org'> <meta property="og:description" content="This meta-PEP collects various issues with CPython’s existing implementation of built-in functions (functions implemented in C) and methods."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0579/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This meta-PEP collects various issues with CPython’s existing implementation of built-in functions (functions implemented in C) and methods."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 579</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> <span class="visually-hidden">Toggle light / dark / auto colour theme</span> </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 579 – Refactoring C functions and methods</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author<span class="colon">:</span></dt> <dd class="field-odd">Jeroen Demeyer <J.Demeyer at UGent.be></dd> <dt class="field-even">BDFL-Delegate<span class="colon">:</span></dt> <dd class="field-even">Petr Viktorin</dd> <dt class="field-odd">Status<span class="colon">:</span></dt> <dd class="field-odd"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-even">Type<span class="colon">:</span></dt> <dd class="field-even"><abbr title="Non-normative PEP containing background, guidelines or other information relevant to the Python ecosystem">Informational</abbr></dd> <dt class="field-odd">Created<span class="colon">:</span></dt> <dd class="field-odd">04-Jun-2018</dd> <dt class="field-even">Post-History<span class="colon">:</span></dt> <dd class="field-even">20-Jun-2018</dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#approval-notice">Approval Notice</a></li> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#issues">Issues</a><ul> <li><a class="reference internal" href="#naming">1. Naming</a></li> <li><a class="reference internal" href="#not-extendable">2. Not extendable</a></li> <li><a class="reference internal" href="#cfunctions-do-not-become-methods">3. cfunctions do not become methods</a></li> <li><a class="reference internal" href="#semantics-of-inspect-isfunction">4. Semantics of inspect.isfunction</a></li> <li><a class="reference internal" href="#c-functions-should-have-access-to-the-function-object">5. C functions should have access to the function object</a></li> <li><a class="reference internal" href="#meth-fastcall-is-private-and-undocumented">6. METH_FASTCALL is private and undocumented</a></li> <li><a class="reference internal" href="#allowing-native-c-arguments">7. Allowing native C arguments</a></li> <li><a class="reference internal" href="#complexity">8. Complexity</a></li> <li><a class="reference internal" href="#pymethoddef-is-too-limited">9. PyMethodDef is too limited</a></li> <li><a class="reference internal" href="#slot-wrappers-have-no-custom-documentation">10. Slot wrappers have no custom documentation</a></li> <li><a class="reference internal" href="#static-methods-and-class-methods-should-be-callable">11. Static methods and class methods should be callable</a></li> </ul> </li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="approval-notice"> <h2><a class="toc-backref" href="#approval-notice" role="doc-backlink">Approval Notice</a></h2> <p>This PEP describes design issues addressed in <a class="pep reference internal" href="../pep-0575/" title="PEP 575 – Unifying function/method classes">PEP 575</a>, <a class="pep reference internal" href="../pep-0580/" title="PEP 580 – The C call protocol">PEP 580</a>, <a class="pep reference internal" href="../pep-0590/" title="PEP 590 – Vectorcall: a fast calling protocol for CPython">PEP 590</a> (and possibly later proposals).</p> <p>As noted in <a class="pep reference internal" href="../pep-0001/#pep-types" title="PEP 1 – PEP Purpose and Guidelines § PEP Types">PEP 1</a>:</p> <blockquote> <div>Informational PEPs do not necessarily represent a Python community consensus or recommendation, so users and implementers are free to ignore Informational PEPs or follow their advice.</div></blockquote> <p>While there is no consensus on whether the issues or the solutions in this PEP are valid, the list is still useful to guide further design.</p> </section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> <p>This meta-PEP collects various issues with CPython’s existing implementation of built-in functions (functions implemented in C) and methods.</p> <p>Fixing all these issues is too much for one PEP, so that will be delegated to other standards track PEPs. However, this PEP does give some brief ideas of possible fixes. This is mainly meant to coordinate an overall strategy. For example, a proposed solution may sound too complicated for fixing any one single issue, but it may be the best overall solution for multiple issues.</p> <p>This PEP is purely informational: it does not imply that all issues will eventually be fixed, nor that they will be fixed using the solution proposed here.</p> <p>It also serves as a check-list of possible requested features to verify that a given fix does not make those other features harder to implement.</p> <p>The major proposed change is replacing <code class="docutils literal notranslate"><span class="pre">PyMethodDef</span></code> by a new structure <code class="docutils literal notranslate"><span class="pre">PyCCallDef</span></code> which collects everything needed for calling the function/method. In the <code class="docutils literal notranslate"><span class="pre">PyTypeObject</span></code> structure, a new field <code class="docutils literal notranslate"><span class="pre">tp_ccalloffset</span></code> is added giving an offset to a <code class="docutils literal notranslate"><span class="pre">PyCCallDef</span> <span class="pre">*</span></code> in the object structure.</p> <p><strong>NOTE</strong>: This PEP deals only with CPython implementation details, it does not affect the Python language or standard library.</p> </section> <section id="issues"> <h2><a class="toc-backref" href="#issues" role="doc-backlink">Issues</a></h2> <p>This lists various issues with built-in functions and methods, together with a plan for a solution and (if applicable) pointers to standards track PEPs discussing the details.</p> <section id="naming"> <h3><a class="toc-backref" href="#naming" role="doc-backlink">1. Naming</a></h3> <p>The word “built-in” is overused in Python. From a quick skim of the Python documentation, it mostly refers to things from the <code class="docutils literal notranslate"><span class="pre">builtins</span></code> module. In other words: things which are available in the global namespace without a need for importing them. This conflicts with the use of the word “built-in” to mean “implemented in C”.</p> <p><strong>Solution</strong>: since the C structure for built-in functions and methods is already called <code class="docutils literal notranslate"><span class="pre">PyCFunctionObject</span></code>, let’s use the name “cfunction” and “cmethod” instead of “built-in function” and “built-in method”.</p> </section> <section id="not-extendable"> <h3><a class="toc-backref" href="#not-extendable" role="doc-backlink">2. Not extendable</a></h3> <p>The various classes involved (such as <code class="docutils literal notranslate"><span class="pre">builtin_function_or_method</span></code>) cannot be subclassed:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">types</span> <span class="kn">import</span> <span class="n">BuiltinFunctionType</span> <span class="gp">>>> </span><span class="k">class</span> <span class="nc">X</span><span class="p">(</span><span class="n">BuiltinFunctionType</span><span class="p">):</span> <span class="gp">... </span> <span class="k">pass</span> <span class="gt">Traceback (most recent call last):</span> File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>, in <span class="n"><module></span> <span class="gr">TypeError</span>: <span class="n">type 'builtin_function_or_method' is not an acceptable base type</span> </pre></div> </div> <p>This is a problem because it makes it impossible to add features such as introspection support to these classes.</p> <p>If one wants to implement a function in C with additional functionality, an entirely new class must be implemented from scratch. The problem with this is that the existing classes like <code class="docutils literal notranslate"><span class="pre">builtin_function_or_method</span></code> are special-cased in the Python interpreter to allow faster calling (for example, by using <code class="docutils literal notranslate"><span class="pre">METH_FASTCALL</span></code>). It is currently impossible to have a custom class with the same optimizations.</p> <p><strong>Solution</strong>: make the existing optimizations available to arbitrary classes. This is done by adding a new <code class="docutils literal notranslate"><span class="pre">PyTypeObject</span></code> field <code class="docutils literal notranslate"><span class="pre">tp_ccalloffset</span></code> (or can we re-use <code class="docutils literal notranslate"><span class="pre">tp_print</span></code> for that?) specifying the offset of a <code class="docutils literal notranslate"><span class="pre">PyCCallDef</span></code> pointer. This is a new structure holding all information needed to call a cfunction and it would be used instead of <code class="docutils literal notranslate"><span class="pre">PyMethodDef</span></code>. This implements the new “C call” protocol.</p> <p>For constructing cfunctions and cmethods, <code class="docutils literal notranslate"><span class="pre">PyMethodDef</span></code> arrays will still be used (for example, in <code class="docutils literal notranslate"><span class="pre">tp_methods</span></code>) but that will be the <em>only</em> remaining purpose of the <code class="docutils literal notranslate"><span class="pre">PyMethodDef</span></code> structure.</p> <p>Additionally, we can also make some function classes subclassable. However, this seems less important once we have <code class="docutils literal notranslate"><span class="pre">tp_ccalloffset</span></code>.</p> <p><strong>Reference</strong>: <a class="pep reference internal" href="../pep-0580/" title="PEP 580 – The C call protocol">PEP 580</a></p> </section> <section id="cfunctions-do-not-become-methods"> <h3><a class="toc-backref" href="#cfunctions-do-not-become-methods" role="doc-backlink">3. cfunctions do not become methods</a></h3> <p>A cfunction like <code class="docutils literal notranslate"><span class="pre">repr</span></code> does not implement <code class="docutils literal notranslate"><span class="pre">__get__</span></code> to bind as a method:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">class</span> <span class="nc">X</span><span class="p">:</span> <span class="gp">... </span> <span class="n">meth</span> <span class="o">=</span> <span class="nb">repr</span> <span class="gp">>>> </span><span class="n">x</span> <span class="o">=</span> <span class="n">X</span><span class="p">()</span> <span class="gp">>>> </span><span class="n">x</span><span class="o">.</span><span class="n">meth</span><span class="p">()</span> <span class="gt">Traceback (most recent call last):</span> File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>, in <span class="n"><module></span> <span class="gr">TypeError</span>: <span class="n">repr() takes exactly one argument (0 given)</span> </pre></div> </div> <p>In this example, one would have expected that <code class="docutils literal notranslate"><span class="pre">x.meth()</span></code> returns <code class="docutils literal notranslate"><span class="pre">repr(x)</span></code> by applying the normal rules of methods.</p> <p>This is surprising and a needless difference between cfunctions and Python functions. For the standard built-in functions, this is not really a problem since those are not meant to used as methods. But it does become a problem when one wants to implement a new cfunction with the goal of being usable as method.</p> <p>Again, a solution could be to create a new class behaving just like cfunctions but which bind as methods. However, that would lose some existing optimizations for methods, such as the <code class="docutils literal notranslate"><span class="pre">LOAD_METHOD</span></code>/<code class="docutils literal notranslate"><span class="pre">CALL_METHOD</span></code> opcodes.</p> <p><strong>Solution</strong>: the same as the previous issue. It just shows that handling <code class="docutils literal notranslate"><span class="pre">self</span></code> and <code class="docutils literal notranslate"><span class="pre">__get__</span></code> should be part of the new C call protocol.</p> <p>For backwards compatibility, we would keep the existing non-binding behavior of cfunctions. We would just allow it in custom classes.</p> <p><strong>Reference</strong>: <a class="pep reference internal" href="../pep-0580/" title="PEP 580 – The C call protocol">PEP 580</a></p> </section> <section id="semantics-of-inspect-isfunction"> <h3><a class="toc-backref" href="#semantics-of-inspect-isfunction" role="doc-backlink">4. Semantics of inspect.isfunction</a></h3> <p>Currently, <code class="docutils literal notranslate"><span class="pre">inspect.isfunction</span></code> returns <code class="docutils literal notranslate"><span class="pre">True</span></code> only for instances of <code class="docutils literal notranslate"><span class="pre">types.FunctionType</span></code>. That is, true Python functions.</p> <p>A common use case for <code class="docutils literal notranslate"><span class="pre">inspect.isfunction</span></code> is checking for introspection: it guarantees for example that <code class="docutils literal notranslate"><span class="pre">inspect.getfile()</span></code> will work. Ideally, it should be possible for other classes to be treated as functions too.</p> <p><strong>Solution</strong>: introduce a new <code class="docutils literal notranslate"><span class="pre">InspectFunction</span></code> abstract base class and use that to implement <code class="docutils literal notranslate"><span class="pre">inspect.isfunction</span></code>. Alternatively, use duck typing for <code class="docutils literal notranslate"><span class="pre">inspect.isfunction</span></code> (as proposed in <a class="footnote-reference brackets" href="#bpo30071" id="id1">[2]</a>):</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">isfunction</span><span class="p">(</span><span class="n">obj</span><span class="p">):</span> <span class="k">return</span> <span class="nb">hasattr</span><span class="p">(</span><span class="nb">type</span><span class="p">(</span><span class="n">obj</span><span class="p">),</span> <span class="s2">"__code__"</span><span class="p">)</span> </pre></div> </div> </section> <section id="c-functions-should-have-access-to-the-function-object"> <h3><a class="toc-backref" href="#c-functions-should-have-access-to-the-function-object" role="doc-backlink">5. C functions should have access to the function object</a></h3> <p>The underlying C function of a cfunction currently takes a <code class="docutils literal notranslate"><span class="pre">self</span></code> argument (for bound methods) and then possibly a number of arguments. There is no way for the C function to actually access the Python cfunction object (the <code class="docutils literal notranslate"><span class="pre">self</span></code> in <code class="docutils literal notranslate"><span class="pre">__call__</span></code> or <code class="docutils literal notranslate"><span class="pre">tp_call</span></code>). This would for example allow implementing the C call protocol for Python functions (<code class="docutils literal notranslate"><span class="pre">types.FunctionType</span></code>): the C function which implements calling Python functions needs access to the <code class="docutils literal notranslate"><span class="pre">__code__</span></code> attribute of the function.</p> <p>This is also needed for <a class="pep reference internal" href="../pep-0573/" title="PEP 573 – Module State Access from C Extension Methods">PEP 573</a> where all cfunctions require access to their “parent” (the module for functions of a module or the defining class for methods).</p> <p><strong>Solution</strong>: add a new <code class="docutils literal notranslate"><span class="pre">PyMethodDef</span></code> flag to specify that the C function takes an additional argument (as first argument), namely the function object.</p> <p><strong>References</strong>: <a class="pep reference internal" href="../pep-0580/" title="PEP 580 – The C call protocol">PEP 580</a>, <a class="pep reference internal" href="../pep-0573/" title="PEP 573 – Module State Access from C Extension Methods">PEP 573</a></p> </section> <section id="meth-fastcall-is-private-and-undocumented"> <h3><a class="toc-backref" href="#meth-fastcall-is-private-and-undocumented" role="doc-backlink">6. METH_FASTCALL is private and undocumented</a></h3> <p>The <code class="docutils literal notranslate"><span class="pre">METH_FASTCALL</span></code> mechanism allows calling cfunctions and cmethods using a C array of Python objects instead of a <code class="docutils literal notranslate"><span class="pre">tuple</span></code>. This was introduced in Python 3.6 for positional arguments only and extended in Python 3.7 with support for keyword arguments.</p> <p>However, given that it is undocumented, it is presumably only supposed to be used by CPython itself.</p> <p><strong>Solution</strong>: since this is an important optimization, everybody should be encouraged to use it. Now that the implementation of <code class="docutils literal notranslate"><span class="pre">METH_FASTCALL</span></code> is stable, document it!</p> <p>As part of the C call protocol, we should also add a C API function</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">PyObject</span> <span class="o">*</span><span class="n">PyCCall_FastCall</span><span class="p">(</span><span class="n">PyObject</span> <span class="o">*</span><span class="n">func</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">const</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="n">Py_ssize_t</span> <span class="n">nargs</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">keywords</span><span class="p">)</span> </pre></div> </div> <p><strong>Reference</strong>: <a class="pep reference internal" href="../pep-0580/" title="PEP 580 – The C call protocol">PEP 580</a></p> </section> <section id="allowing-native-c-arguments"> <h3><a class="toc-backref" href="#allowing-native-c-arguments" role="doc-backlink">7. Allowing native C arguments</a></h3> <p>A cfunction always takes its arguments as Python objects (say, an array of <code class="docutils literal notranslate"><span class="pre">PyObject</span></code> pointers). In cases where the cfunction is really wrapping a native C function (for example, coming from <code class="docutils literal notranslate"><span class="pre">ctypes</span></code> or some compiler like Cython), this is inefficient: calls from C code to C code are forced to use Python objects to pass arguments.</p> <p>Analogous to the buffer protocol which allows access to C data, we should also allow access to the underlying C callable.</p> <p><strong>Solution</strong>: when wrapping a C function with native arguments (for example, a C <code class="docutils literal notranslate"><span class="pre">long</span></code>) inside a cfunction, we should also store a function pointer to the underlying C function, together with its C signature.</p> <p>Argument Clinic could automatically do this by storing a pointer to the “impl” function.</p> </section> <section id="complexity"> <h3><a class="toc-backref" href="#complexity" role="doc-backlink">8. Complexity</a></h3> <p>There are a huge number of classes involved to implement all variations of methods. This is not a problem by itself, but a compounding issue.</p> <p>For ordinary Python classes, the table below gives the classes for various kinds of methods. The columns refer to the class in the class <code class="docutils literal notranslate"><span class="pre">__dict__</span></code>, the class for unbound methods (bound to the class) and the class for bound methods (bound to the instance):</p> <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">kind</th> <th class="head">__dict__</th> <th class="head">unbound</th> <th class="head">bound</th> </tr> </thead> <tbody> <tr class="row-even"><td>Normal method</td> <td><code class="docutils literal notranslate"><span class="pre">function</span></code></td> <td><code class="docutils literal notranslate"><span class="pre">function</span></code></td> <td><code class="docutils literal notranslate"><span class="pre">method</span></code></td> </tr> <tr class="row-odd"><td>Static method</td> <td><code class="docutils literal notranslate"><span class="pre">staticmethod</span></code></td> <td><code class="docutils literal notranslate"><span class="pre">function</span></code></td> <td><code class="docutils literal notranslate"><span class="pre">function</span></code></td> </tr> <tr class="row-even"><td>Class method</td> <td><code class="docutils literal notranslate"><span class="pre">classmethod</span></code></td> <td><code class="docutils literal notranslate"><span class="pre">method</span></code></td> <td><code class="docutils literal notranslate"><span class="pre">method</span></code></td> </tr> <tr class="row-odd"><td>Slot method</td> <td><code class="docutils literal notranslate"><span class="pre">function</span></code></td> <td><code class="docutils literal notranslate"><span class="pre">function</span></code></td> <td><code class="docutils literal notranslate"><span class="pre">method</span></code></td> </tr> </tbody> </table> <p>This is the analogous table for extension types (C classes):</p> <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">kind</th> <th class="head">__dict__</th> <th class="head">unbound</th> <th class="head">bound</th> </tr> </thead> <tbody> <tr class="row-even"><td>Normal method</td> <td><code class="docutils literal notranslate"><span class="pre">method_descriptor</span></code></td> <td><code class="docutils literal notranslate"><span class="pre">method_descriptor</span></code></td> <td><code class="docutils literal notranslate"><span class="pre">builtin_function_or_method</span></code></td> </tr> <tr class="row-odd"><td>Static method</td> <td><code class="docutils literal notranslate"><span class="pre">staticmethod</span></code></td> <td><code class="docutils literal notranslate"><span class="pre">builtin_function_or_method</span></code></td> <td><code class="docutils literal notranslate"><span class="pre">builtin_function_or_method</span></code></td> </tr> <tr class="row-even"><td>Class method</td> <td><code class="docutils literal notranslate"><span class="pre">classmethod_descriptor</span></code></td> <td><code class="docutils literal notranslate"><span class="pre">builtin_function_or_method</span></code></td> <td><code class="docutils literal notranslate"><span class="pre">builtin_function_or_method</span></code></td> </tr> <tr class="row-odd"><td>Slot method</td> <td><code class="docutils literal notranslate"><span class="pre">wrapper_descriptor</span></code></td> <td><code class="docutils literal notranslate"><span class="pre">wrapper_descriptor</span></code></td> <td><code class="docutils literal notranslate"><span class="pre">method-wrapper</span></code></td> </tr> </tbody> </table> <p>There are a lot of classes involved and these two tables look very different. There is no good reason why Python methods should be treated fundamentally different from C methods. Also the features are slightly different: for example, <code class="docutils literal notranslate"><span class="pre">method</span></code> supports <code class="docutils literal notranslate"><span class="pre">__func__</span></code> but <code class="docutils literal notranslate"><span class="pre">builtin_function_or_method</span></code> does not.</p> <p>Since CPython has optimizations for calls to most of these objects, the code for dealing with them can also become complex. A good example of this is the <code class="docutils literal notranslate"><span class="pre">call_function</span></code> function in <code class="docutils literal notranslate"><span class="pre">Python/ceval.c</span></code>.</p> <p><strong>Solution</strong>: all these classes should implement the C call protocol. Then the complexity in the code can mostly be fixed by checking for the C call protocol (<code class="docutils literal notranslate"><span class="pre">tp_ccalloffset</span> <span class="pre">!=</span> <span class="pre">0</span></code>) instead of doing type checks.</p> <p>Furthermore, it should be investigated whether some of these classes can be merged and whether <code class="docutils literal notranslate"><span class="pre">method</span></code> can be re-used also for bound methods of extension types (see <a class="pep reference internal" href="../pep-0576/" title="PEP 576 – Rationalize Built-in function classes">PEP 576</a> for the latter, keeping in mind that this may have some minor backwards compatibility issues). This is not a goal by itself but just something to keep in mind when working on these classes.</p> </section> <section id="pymethoddef-is-too-limited"> <h3><a class="toc-backref" href="#pymethoddef-is-too-limited" role="doc-backlink">9. PyMethodDef is too limited</a></h3> <p>The typical way to create a cfunction or cmethod in an extension module is by using a <code class="docutils literal notranslate"><span class="pre">PyMethodDef</span></code> to define it. These are then stored in an array <code class="docutils literal notranslate"><span class="pre">PyModuleDef.m_methods</span></code> (for cfunctions) or <code class="docutils literal notranslate"><span class="pre">PyTypeObject.tp_methods</span></code> (for cmethods). However, because of the stable ABI (<a class="pep reference internal" href="../pep-0384/" title="PEP 384 – Defining a Stable ABI">PEP 384</a>), we cannot change the <code class="docutils literal notranslate"><span class="pre">PyMethodDef</span></code> structure.</p> <p>So, this means that we cannot add new fields for creating cfunctions/cmethods this way. This is probably the reason for the hack that <code class="docutils literal notranslate"><span class="pre">__doc__</span></code> and <code class="docutils literal notranslate"><span class="pre">__text_signature__</span></code> are stored in the same C string (with the <code class="docutils literal notranslate"><span class="pre">__doc__</span></code> and <code class="docutils literal notranslate"><span class="pre">__text_signature__</span></code> descriptors extracting the relevant part).</p> <p><strong>Solution</strong>: stop assuming that a single <code class="docutils literal notranslate"><span class="pre">PyMethodDef</span></code> entry is sufficient to describe a cfunction/cmethod. Instead, we could add some flag which means that one of the <code class="docutils literal notranslate"><span class="pre">PyMethodDef</span></code> fields is instead a pointer to an additional structure. Or, we could add a flag to use two or more consecutive <code class="docutils literal notranslate"><span class="pre">PyMethodDef</span></code> entries in the array to store more data. Then the <code class="docutils literal notranslate"><span class="pre">PyMethodDef</span></code> array would be used only to construct cfunctions/cmethods but it would no longer be used after that.</p> </section> <section id="slot-wrappers-have-no-custom-documentation"> <h3><a class="toc-backref" href="#slot-wrappers-have-no-custom-documentation" role="doc-backlink">10. Slot wrappers have no custom documentation</a></h3> <p>Right now, slot wrappers like <code class="docutils literal notranslate"><span class="pre">__init__</span></code> or <code class="docutils literal notranslate"><span class="pre">__lt__</span></code> only have very generic documentation, not at all specific to the class:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="nb">list</span><span class="o">.</span><span class="fm">__init__</span><span class="o">.</span><span class="vm">__doc__</span> <span class="go">'Initialize self. See help(type(self)) for accurate signature.'</span> <span class="gp">>>> </span><span class="nb">list</span><span class="o">.</span><span class="fm">__lt__</span><span class="o">.</span><span class="vm">__doc__</span> <span class="go">'Return self<value.'</span> </pre></div> </div> <p>The same happens for the signature:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="nb">list</span><span class="o">.</span><span class="fm">__init__</span><span class="o">.</span><span class="n">__text_signature__</span> <span class="go">'($self, /, *args, **kwargs)'</span> </pre></div> </div> <p>As you can see, slot wrappers do support <code class="docutils literal notranslate"><span class="pre">__doc__</span></code> and <code class="docutils literal notranslate"><span class="pre">__text_signature__</span></code>. The problem is that these are stored in <code class="docutils literal notranslate"><span class="pre">struct</span> <span class="pre">wrapperbase</span></code>, which is common for all wrappers of a specific slot (for example, the same <code class="docutils literal notranslate"><span class="pre">wrapperbase</span></code> is used for <code class="docutils literal notranslate"><span class="pre">str.__eq__</span></code> and <code class="docutils literal notranslate"><span class="pre">int.__eq__</span></code>).</p> <p><strong>Solution</strong>: rethink the slot wrapper class to allow docstrings (and text signatures) for each instance separately.</p> <p>This still leaves the question of how extension modules should specify the documentation. The <code class="docutils literal notranslate"><span class="pre">PyTypeObject</span></code> entries like <code class="docutils literal notranslate"><span class="pre">tp_init</span></code> are just function pointers, we cannot do anything with those. One solution would be to add entries to the <code class="docutils literal notranslate"><span class="pre">tp_methods</span></code> array just for adding docstrings. Such an entry could look like</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">{</span><span class="s2">"__init__"</span><span class="p">,</span> <span class="n">NULL</span><span class="p">,</span> <span class="n">METH_SLOTDOC</span><span class="p">,</span> <span class="s2">"pointer to __init__ doc goes here"</span><span class="p">}</span> </pre></div> </div> </section> <section id="static-methods-and-class-methods-should-be-callable"> <h3><a class="toc-backref" href="#static-methods-and-class-methods-should-be-callable" role="doc-backlink">11. Static methods and class methods should be callable</a></h3> <p>Instances of <code class="docutils literal notranslate"><span class="pre">staticmethod</span></code> and <code class="docutils literal notranslate"><span class="pre">classmethod</span></code> should be callable. Admittedly, there is no strong use case for this, but it has occasionally been requested (see for example <a class="footnote-reference brackets" href="#bpo20309" id="id2">[1]</a>).</p> <p>Making static/class methods callable would increase consistency. First of all, function decorators typically add functionality or modify a function, but the result remains callable. This is not true for <code class="docutils literal notranslate"><span class="pre">@staticmethod</span></code> and <code class="docutils literal notranslate"><span class="pre">@classmethod</span></code>.</p> <p>Second, class methods of extension types are already callable:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">fromhex</span> <span class="o">=</span> <span class="nb">float</span><span class="o">.</span><span class="vm">__dict__</span><span class="p">[</span><span class="s2">"fromhex"</span><span class="p">]</span> <span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="n">fromhex</span><span class="p">)</span> <span class="go"><class 'classmethod_descriptor'></span> <span class="gp">>>> </span><span class="n">fromhex</span><span class="p">(</span><span class="nb">float</span><span class="p">,</span> <span class="s2">"0xff"</span><span class="p">)</span> <span class="go">255.0</span> </pre></div> </div> <p>Third, one can see <code class="docutils literal notranslate"><span class="pre">function</span></code>, <code class="docutils literal notranslate"><span class="pre">staticmethod</span></code> and <code class="docutils literal notranslate"><span class="pre">classmethod</span></code> as different kinds of unbound methods: they all become <code class="docutils literal notranslate"><span class="pre">method</span></code> when bound, but the implementation of <code class="docutils literal notranslate"><span class="pre">__get__</span></code> is slightly different. From this point of view, it looks strange that <code class="docutils literal notranslate"><span class="pre">function</span></code> is callable but the others are not.</p> <p><strong>Solution</strong>: when changing the implementation of <code class="docutils literal notranslate"><span class="pre">staticmethod</span></code>, <code class="docutils literal notranslate"><span class="pre">classmethod</span></code>, we should consider making instances callable. Even if this is not a goal by itself, it may happen naturally because of the implementation.</p> </section> </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <aside class="footnote-list brackets"> <aside class="footnote brackets" id="bpo20309" role="doc-footnote"> <dt class="label" id="bpo20309">[<a href="#id2">1</a>]</dt> <dd>Not all method descriptors are callable (<a class="reference external" href="https://bugs.python.org/issue20309">https://bugs.python.org/issue20309</a>)</aside> <aside class="footnote brackets" id="bpo30071" role="doc-footnote"> <dt class="label" id="bpo30071">[<a href="#id1">2</a>]</dt> <dd>Duck-typing inspect.isfunction() (<a class="reference external" href="https://bugs.python.org/issue30071">https://bugs.python.org/issue30071</a>)</aside> </aside> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> <p>This document has been placed in the public domain.</p> </section> </section> <hr class="docutils" /> <p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0579.rst">https://github.com/python/peps/blob/main/peps/pep-0579.rst</a></p> <p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0579.rst">2023-09-09 17:39:29 GMT</a></p> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#approval-notice">Approval Notice</a></li> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#issues">Issues</a><ul> <li><a class="reference internal" href="#naming">1. Naming</a></li> <li><a class="reference internal" href="#not-extendable">2. Not extendable</a></li> <li><a class="reference internal" href="#cfunctions-do-not-become-methods">3. cfunctions do not become methods</a></li> <li><a class="reference internal" href="#semantics-of-inspect-isfunction">4. Semantics of inspect.isfunction</a></li> <li><a class="reference internal" href="#c-functions-should-have-access-to-the-function-object">5. C functions should have access to the function object</a></li> <li><a class="reference internal" href="#meth-fastcall-is-private-and-undocumented">6. METH_FASTCALL is private and undocumented</a></li> <li><a class="reference internal" href="#allowing-native-c-arguments">7. Allowing native C arguments</a></li> <li><a class="reference internal" href="#complexity">8. Complexity</a></li> <li><a class="reference internal" href="#pymethoddef-is-too-limited">9. PyMethodDef is too limited</a></li> <li><a class="reference internal" href="#slot-wrappers-have-no-custom-documentation">10. Slot wrappers have no custom documentation</a></li> <li><a class="reference internal" href="#static-methods-and-class-methods-should-be-callable">11. Static methods and class methods should be callable</a></li> </ul> </li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <br> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0579.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>