CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 620 – Hide implementation details from the C API | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0620/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 620 – Hide implementation details from the C API | peps.python.org'> <meta property="og:description" content="Introduce C API incompatible changes to hide implementation details."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0620/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="Introduce C API incompatible changes to hide implementation details."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 620</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 620 – Hide implementation details from the C API</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Victor Stinner <vstinner at python.org></dd> <dt class="field-even">Status:</dt> <dd class="field-even"><abbr title="Removed from consideration by sponsor or authors">Withdrawn</abbr></dd> <dt class="field-odd">Type:</dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created:</dt> <dd class="field-even">19-Jun-2020</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">3.12</dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#pep-withdrawn">PEP withdrawn</a></li> <li><a class="reference internal" href="#motivation">Motivation</a><ul> <li><a class="reference internal" href="#the-c-api-blocks-cpython-evolutions">The C API blocks CPython evolutions</a></li> <li><a class="reference internal" href="#same-cpython-design-since-1990-structures-and-reference-counting">Same CPython design since 1990: structures and reference counting</a></li> <li><a class="reference internal" href="#why-is-pypy-more-efficient-than-cpython">Why is PyPy more efficient than CPython?</a></li> <li><a class="reference internal" href="#pypy-bottleneck-the-python-c-api">PyPy bottleneck: the Python C API</a></li> </ul> </li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#hide-implementation-details">Hide implementation details</a></li> <li><a class="reference internal" href="#relationship-with-the-limited-c-api">Relationship with the limited C API</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#summary">Summary</a></li> <li><a class="reference internal" href="#reorganize-the-c-api-header-files">Reorganize the C API header files</a></li> <li><a class="reference internal" href="#move-private-functions-to-the-internal-c-api">Move private functions to the internal C API</a></li> <li><a class="reference internal" href="#convert-macros-to-static-inline-functions">Convert macros to static inline functions</a></li> <li><a class="reference internal" href="#make-structures-opaque">Make structures opaque</a></li> <li><a class="reference internal" href="#disallow-using-py-type-as-l-value">Disallow using Py_TYPE() as l-value</a></li> <li><a class="reference internal" href="#new-c-api-functions-must-not-return-borrowed-references">New C API functions must not return borrowed references</a></li> <li><a class="reference internal" href="#avoid-functions-returning-pyobject">Avoid functions returning PyObject**</a></li> <li><a class="reference internal" href="#new-pythoncapi-compat-h-header-file">New pythoncapi_compat.h header file</a></li> </ul> </li> <li><a class="reference internal" href="#process-to-reduce-the-number-of-broken-c-extensions">Process to reduce the number of broken C extensions</a></li> <li><a class="reference internal" href="#version-history">Version History</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> Introduce C API incompatible changes to hide implementation details. Once most implementation details will be hidden, evolution of CPython internals would be less limited by C API backward compatibility issues. It will be way easier to add new features. It becomes possible to experiment with more advanced optimizations in CPython than just micro-optimizations, like tagged pointers. Define a process to reduce the number of broken C extensions. The implementation of this PEP is expected to be done carefully over multiple Python versions. It already started in Python 3.7 and most changes are already completed. The <a class="reference internal" href="#process-to-reduce-the-number-of-broken-c-extensions">Process to reduce the number of broken C extensions</a> dictates the rhythm. </section> <section id="pep-withdrawn"> <h2><a class="toc-backref" href="#pep-withdrawn" role="doc-backlink">PEP withdrawn</a></h2> This PEP was withdrawn by its author since the scope is too broad and the work is distributed over multiple Python versions, which makes it difficult to make a decision on the overall PEP. It was split into new PEPs with narrower and better defined scopes, like <a class="pep reference internal" href="../pep-0670/" title="PEP 670 – Convert macros to functions in the Python C API">PEP 670</a>. </section> <section id="motivation"> <h2><a class="toc-backref" href="#motivation" role="doc-backlink">Motivation</a></h2> <section id="the-c-api-blocks-cpython-evolutions"> <h3><a class="toc-backref" href="#the-c-api-blocks-cpython-evolutions" role="doc-backlink">The C API blocks CPython evolutions</a></h3> Adding or removing members of C structures is causing multiple backward compatibility issues. Adding a new member breaks the stable ABI (<a class="pep reference internal" href="../pep-0384/" title="PEP 384 – Defining a Stable ABI">PEP 384</a>), especially for types declared statically (e.g. <code class="docutils literal notranslate">static PyTypeObject MyType = {...};</code>). In Python 3.4, the <a class="pep reference internal" href="../pep-0442/" title="PEP 442 – Safe object finalization">PEP 442</a> “Safe object finalization” added the <code class="docutils literal notranslate">tp_finalize</code> member at the end of the <code class="docutils literal notranslate">PyTypeObject</code> structure. For ABI backward compatibility, a new <code class="docutils literal notranslate">Py_TPFLAGS_HAVE_FINALIZE</code> type flag was required to announce if the type structure contains the <code class="docutils literal notranslate">tp_finalize</code> member. The flag was removed in Python 3.8 (<a class="reference external" href="https://bugs.python.org/issue32388">bpo-32388</a>). The <code class="docutils literal notranslate">PyTypeObject.tp_print</code> member, deprecated since Python 3.0 released in 2009, has been removed in the Python 3.8 development cycle. But the change broke too many C extensions and had to be reverted before 3.8 final release. Finally, the member was removed again in Python 3.9. C extensions rely on the ability to access structure members, indirectly through the C API, or even directly. Modifying structures like <code class="docutils literal notranslate">PyListObject</code> cannot be even considered. The <code class="docutils literal notranslate">PyTypeObject</code> structure is the one which evolved the most, simply because there was no other way to evolve CPython than modifying it. A C extension can technically dereference a <code class="docutils literal notranslate">PyObject*</code> pointer and access <code class="docutils literal notranslate">PyObject</code> members. This prevents experiments like tagged pointers (storing small values as <code class="docutils literal notranslate">PyObject*</code> which does not point to a valid <code class="docutils literal notranslate">PyObject</code> structure). Replacing Python garbage collector with a tracing garbage collector would also need to remove <code class="docutils literal notranslate">PyObject.ob_refcnt</code> reference counter, whereas currently <code class="docutils literal notranslate">Py_INCREF()</code> and <code class="docutils literal notranslate">Py_DECREF()</code> macros access directly to <code class="docutils literal notranslate">PyObject.ob_refcnt</code>. </section> <section id="same-cpython-design-since-1990-structures-and-reference-counting"> <h3><a class="toc-backref" href="#same-cpython-design-since-1990-structures-and-reference-counting" role="doc-backlink">Same CPython design since 1990: structures and reference counting</a></h3> When the CPython project was created, it was written with one principle: keep the implementation simple enough so it can be maintained by a single developer. CPython complexity grew a lot and many micro-optimizations have been implemented, but CPython core design has not changed. Members of <code class="docutils literal notranslate">PyObject</code> and <code class="docutils literal notranslate">PyTupleObject</code> structures have not changed since the “Initial revision” commit (1990): <div class="highlight-default notranslate"><div class="highlight"><pre>#define OB_HEAD \ unsigned int ob_refcnt; \ struct _typeobject *ob_type; typedef struct _object { OB_HEAD } object; typedef struct { OB_VARHEAD object *ob_item[1]; } tupleobject; </pre></div> </div> Only names changed: <code class="docutils literal notranslate">object</code> was renamed to <code class="docutils literal notranslate">PyObject</code> and <code class="docutils literal notranslate">tupleobject</code> was renamed to <code class="docutils literal notranslate">PyTupleObject</code>. CPython still tracks Python objects lifetime using reference counting internally and for third party C extensions (through the Python C API). All Python objects must be allocated on the heap and cannot be moved. </section> <section id="why-is-pypy-more-efficient-than-cpython"> <h3><a class="toc-backref" href="#why-is-pypy-more-efficient-than-cpython" role="doc-backlink">Why is PyPy more efficient than CPython?</a></h3> The PyPy project is a Python implementation which is 4.2x faster than CPython on average. PyPy developers chose to not fork CPython, but start from scratch to have more freedom in terms of optimization choices. PyPy does not use reference counting, but a tracing garbage collector which moves objects. Objects can be allocated on the stack (or even not at all), rather than always having to be allocated on the heap. Objects layouts are designed with performance in mind. For example, a list strategy stores integers directly as integers, rather than objects. Moreover, PyPy also has a JIT compiler which emits fast code thanks to the efficient PyPy design. </section> <section id="pypy-bottleneck-the-python-c-api"> <h3><a class="toc-backref" href="#pypy-bottleneck-the-python-c-api" role="doc-backlink">PyPy bottleneck: the Python C API</a></h3> While PyPy is way more efficient than CPython to run pure Python code, it is as efficient or slower than CPython to run C extensions. Since the C API requires <code class="docutils literal notranslate">PyObject*</code> and allows to access directly structure members, PyPy has to associate a CPython object to PyPy objects and maintain both consistent. Converting a PyPy object to a CPython object is inefficient. Moreover, reference counting also has to be implemented on top of PyPy tracing garbage collector. These conversions are required because the Python C API is too close to the CPython implementation: there is no high-level abstraction. For example, structures members are part of the public C API and nothing prevents a C extension to get or set directly <code class="docutils literal notranslate">PyTupleObject.ob_item[0]</code> (the first item of a tuple). See <a class="reference external" href="https://morepypy.blogspot.com/2018/09/inside-cpyext-why-emulating-cpython-c.html">Inside cpyext: Why emulating CPython C API is so Hard</a> (Sept 2018) by Antonio Cuni for more details. </section> </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> <section id="hide-implementation-details"> <h3><a class="toc-backref" href="#hide-implementation-details" role="doc-backlink">Hide implementation details</a></h3> Hiding implementation details from the C API has multiple advantages: <ul class="simple"> <li>It becomes possible to experiment with more advanced optimizations in CPython than just micro-optimizations. For example, tagged pointers, and replace the garbage collector with a tracing garbage collector which can move objects.</li> <li>Adding new features in CPython becomes easier.</li> <li>PyPy should be able to avoid conversions to CPython objects in more cases: keep efficient PyPy objects.</li> <li>It becomes easier to implement the C API for a new Python implementation.</li> <li>More C extensions will be compatible with Python implementations other than CPython.</li> </ul> </section> <section id="relationship-with-the-limited-c-api"> <h3><a class="toc-backref" href="#relationship-with-the-limited-c-api" role="doc-backlink">Relationship with the limited C API</a></h3> The <a class="pep reference internal" href="../pep-0384/" title="PEP 384 – Defining a Stable ABI">PEP 384</a> “Defining a Stable ABI” is implemented in Python 3.4. It introduces the “limited C API”: a subset of the C API. When the limited C API is used, it becomes possible to build a C extension only once and use it on multiple Python versions: that’s the stable ABI. The main limitation of the <a class="pep reference internal" href="../pep-0384/" title="PEP 384 – Defining a Stable ABI">PEP 384</a> is that C extensions have to opt-in for the limited C API. Only very few projects made this choice, usually to ease distribution of binaries, especially on Windows. This PEP moves the C API towards the limited C API. Ideally, the C API will become the limited C API and all C extensions will use the stable ABI, but this is out of this PEP scope. </section> </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> <section id="summary"> <h3><a class="toc-backref" href="#summary" role="doc-backlink">Summary</a></h3> <ul class="simple"> <li>(Completed) Reorganize the C API header files: create <code class="docutils literal notranslate">Include/cpython/</code> and <code class="docutils literal notranslate">Include/internal/</code> subdirectories.</li> <li>(Completed) Move private functions exposing implementation details to the internal C API.</li> <li>(Completed) Convert macros to static inline functions.</li> <li>(Completed) Add new functions <code class="docutils literal notranslate">Py_SET_TYPE()</code>, <code class="docutils literal notranslate">Py_SET_REFCNT()</code> and <code class="docutils literal notranslate">Py_SET_SIZE()</code>. The <code class="docutils literal notranslate">Py_TYPE()</code>, <code class="docutils literal notranslate">Py_REFCNT()</code> and <code class="docutils literal notranslate">Py_SIZE()</code> macros become functions which cannot be used as l-value.</li> <li>(Completed) New C API functions must not return borrowed references.</li> <li>(In Progress) Provide <code class="docutils literal notranslate">pythoncapi_compat.h</code> header file.</li> <li>(In Progress) Make structures opaque, add getter and setter functions.</li> <li>(Not Started) Deprecate <code class="docutils literal notranslate">PySequence_Fast_ITEMS()</code>.</li> <li>(Not Started) Convert <code class="docutils literal notranslate">PyTuple_GET_ITEM()</code> and <code class="docutils literal notranslate">PyList_GET_ITEM()</code> macros to static inline functions.</li> </ul> </section> <section id="reorganize-the-c-api-header-files"> <h3><a class="toc-backref" href="#reorganize-the-c-api-header-files" role="doc-backlink">Reorganize the C API header files</a></h3> The first consumer of the C API was Python itself. There is no clear separation between APIs which must not be used outside Python, and API which are public on purpose. Header files must be reorganized in 3 API: <ul class="simple"> <li><code class="docutils literal notranslate">Include/</code> directory is the limited C API: no implementation details, structures are opaque. C extensions using it get a stable ABI.</li> <li><code class="docutils literal notranslate">Include/cpython/</code> directory is the CPython C API: less “portable” API, depends more on the Python version, expose some implementation details, few incompatible changes can happen.</li> <li><code class="docutils literal notranslate">Include/internal/</code> directory is the internal C API: implementation details, incompatible changes are likely at each Python release.</li> </ul> The creation of the <code class="docutils literal notranslate">Include/cpython/</code> directory is fully backward compatible. <code class="docutils literal notranslate">Include/cpython/</code> header files cannot be included directly and are included automatically by <code class="docutils literal notranslate">Include/</code> header files when the <code class="docutils literal notranslate">Py_LIMITED_API</code> macro is not defined. The internal C API is installed and can be used for specific usage like debuggers and profilers which must access structures members without executing code. C extensions using the internal C API are tightly coupled to a Python version and must be recompiled at each Python version. STATUS: Completed (in Python 3.8) The reorganization of header files started in Python 3.7 and was completed in Python 3.8: <ul class="simple"> <li><a class="reference external" href="https://bugs.python.org/issue35134">bpo-35134</a>: Add a new Include/cpython/ subdirectory for the “CPython API” with implementation details.</li> <li><a class="reference external" href="https://bugs.python.org/issue35081">bpo-35081</a>: Move internal headers to <code class="docutils literal notranslate">Include/internal/</code></li> </ul> </section> <section id="move-private-functions-to-the-internal-c-api"> <h3><a class="toc-backref" href="#move-private-functions-to-the-internal-c-api" role="doc-backlink">Move private functions to the internal C API</a></h3> Private functions which expose implementation details must be moved to the internal C API. If a C extension relies on a CPython private function which exposes CPython implementation details, other Python implementations have to re-implement this private function to support this C extension. STATUS: Completed (in Python 3.9) Private functions moved to the internal C API in Python 3.8: <ul class="simple"> <li><code class="docutils literal notranslate">_PyObject_GC_TRACK()</code>, <code class="docutils literal notranslate">_PyObject_GC_UNTRACK()</code></li> </ul> Macros and functions excluded from the limited C API in Python 3.9: <ul class="simple"> <li><code class="docutils literal notranslate">_PyObject_SIZE()</code>, <code class="docutils literal notranslate">_PyObject_VAR_SIZE()</code></li> <li><code class="docutils literal notranslate">PyThreadState_DeleteCurrent()</code></li> <li><code class="docutils literal notranslate">PyFPE_START_PROTECT()</code>, <code class="docutils literal notranslate">PyFPE_END_PROTECT()</code></li> <li><code class="docutils literal notranslate">_Py_NewReference()</code>, <code class="docutils literal notranslate">_Py_ForgetReference()</code></li> <li><code class="docutils literal notranslate">_PyTraceMalloc_NewReference()</code></li> <li><code class="docutils literal notranslate">_Py_GetRefTotal()</code></li> </ul> Private functions moved to the internal C API in Python 3.9: <ul class="simple"> <li>GC functions like <code class="docutils literal notranslate">_Py_AS_GC()</code>, <code class="docutils literal notranslate">_PyObject_GC_IS_TRACKED()</code> and <code class="docutils literal notranslate">_PyGCHead_NEXT()</code></li> <li><code class="docutils literal notranslate">_Py_AddToAllObjects()</code> (not exported)</li> <li><code class="docutils literal notranslate">_PyDebug_PrintTotalRefs()</code>, <code class="docutils literal notranslate">_Py_PrintReferences()</code>, <code class="docutils literal notranslate">_Py_PrintReferenceAddresses()</code> (not exported)</li> </ul> Public “clear free list” functions moved to the internal C API and renamed to private functions in Python 3.9: <ul class="simple"> <li><code class="docutils literal notranslate">PyAsyncGen_ClearFreeLists()</code></li> <li><code class="docutils literal notranslate">PyContext_ClearFreeList()</code></li> <li><code class="docutils literal notranslate">PyDict_ClearFreeList()</code></li> <li><code class="docutils literal notranslate">PyFloat_ClearFreeList()</code></li> <li><code class="docutils literal notranslate">PyFrame_ClearFreeList()</code></li> <li><code class="docutils literal notranslate">PyList_ClearFreeList()</code></li> <li><code class="docutils literal notranslate">PyTuple_ClearFreeList()</code></li> <li>Functions simply removed:<ul> <li><code class="docutils literal notranslate">PyMethod_ClearFreeList()</code> and <code class="docutils literal notranslate">PyCFunction_ClearFreeList()</code>: bound method free list removed in Python 3.9.</li> <li><code class="docutils literal notranslate">PySet_ClearFreeList()</code>: set free list removed in Python 3.4.</li> <li><code class="docutils literal notranslate">PyUnicode_ClearFreeList()</code>: Unicode free list removed in Python 3.3.</li> </ul> </li> </ul> </section> <section id="convert-macros-to-static-inline-functions"> <h3><a class="toc-backref" href="#convert-macros-to-static-inline-functions" role="doc-backlink">Convert macros to static inline functions</a></h3> Converting macros to static inline functions has multiple advantages: <ul class="simple"> <li>Functions have well defined parameter types and return type.</li> <li>Functions can use variables with a well defined scope (the function).</li> <li>Debugger can be put breakpoints on functions and profilers can display the function name in the call stacks. In most cases, it works even when a static inline function is inlined.</li> <li>Functions don’t have <a class="reference external" href="https://gcc.gnu.org/onlinedocs/cpp/Macro-Pitfalls.html">macros pitfalls</a>.</li> </ul> Converting macros to static inline functions should only impact very few C extensions that use macros in unusual ways. For backward compatibility, functions must continue to accept any type, not only <code class="docutils literal notranslate">PyObject*</code>, to avoid compiler warnings, since most macros cast their parameters to <code class="docutils literal notranslate">PyObject*</code>. Python 3.6 requires C compilers to support static inline functions: the <a class="pep reference internal" href="../pep-0007/" title="PEP 7 – Style Guide for C Code">PEP 7</a> requires a subset of C99. STATUS: Completed (in Python 3.9) Macros converted to static inline functions in Python 3.8: <ul class="simple"> <li><code class="docutils literal notranslate">Py_INCREF()</code>, <code class="docutils literal notranslate">Py_DECREF()</code></li> <li><code class="docutils literal notranslate">Py_XINCREF()</code>, <code class="docutils literal notranslate">Py_XDECREF()</code></li> <li><code class="docutils literal notranslate">PyObject_INIT()</code>, <code class="docutils literal notranslate">PyObject_INIT_VAR()</code></li> <li><code class="docutils literal notranslate">_PyObject_GC_TRACK()</code>, <code class="docutils literal notranslate">_PyObject_GC_UNTRACK()</code>, <code class="docutils literal notranslate">_Py_Dealloc()</code></li> </ul> Macros converted to regular functions in Python 3.9: <ul class="simple"> <li><code class="docutils literal notranslate">Py_EnterRecursiveCall()</code>, <code class="docutils literal notranslate">Py_LeaveRecursiveCall()</code> (added to the limited C API)</li> <li><code class="docutils literal notranslate">PyObject_INIT()</code>, <code class="docutils literal notranslate">PyObject_INIT_VAR()</code></li> <li><code class="docutils literal notranslate">PyObject_GET_WEAKREFS_LISTPTR()</code></li> <li><code class="docutils literal notranslate">PyObject_CheckBuffer()</code></li> <li><code class="docutils literal notranslate">PyIndex_Check()</code></li> <li><code class="docutils literal notranslate">PyObject_IS_GC()</code></li> <li><code class="docutils literal notranslate">PyObject_NEW()</code> (alias to <code class="docutils literal notranslate">PyObject_New()</code>), <code class="docutils literal notranslate">PyObject_NEW_VAR()</code> (alias to <code class="docutils literal notranslate">PyObject_NewVar()</code>)</li> <li><code class="docutils literal notranslate">PyType_HasFeature()</code> (always call <code class="docutils literal notranslate">PyType_GetFlags()</code>)</li> <li><code class="docutils literal notranslate">Py_TRASHCAN_BEGIN_CONDITION()</code> and <code class="docutils literal notranslate">Py_TRASHCAN_END()</code> macros now call functions which hide implementation details, rather than accessing directly members of the <code class="docutils literal notranslate">PyThreadState</code> structure.</li> </ul> </section> <section id="make-structures-opaque"> <h3><a class="toc-backref" href="#make-structures-opaque" role="doc-backlink">Make structures opaque</a></h3> The following structures of the C API become opaque: <ul class="simple"> <li><code class="docutils literal notranslate">PyInterpreterState</code></li> <li><code class="docutils literal notranslate">PyThreadState</code></li> <li><code class="docutils literal notranslate">PyGC_Head</code></li> <li><code class="docutils literal notranslate">PyTypeObject</code></li> <li><code class="docutils literal notranslate">PyObject</code> and <code class="docutils literal notranslate">PyVarObject</code></li> <li><code class="docutils literal notranslate">PyTypeObject</code></li> <li>All types which inherit from <code class="docutils literal notranslate">PyObject</code> or <code class="docutils literal notranslate">PyVarObject</code></li> </ul> C extensions must use getter or setter functions to get or set structure members. For example, <code class="docutils literal notranslate">tuple->ob_item[0]</code> must be replaced with <code class="docutils literal notranslate">PyTuple_GET_ITEM(tuple, 0)</code>. To be able to move away from reference counting, <code class="docutils literal notranslate">PyObject</code> must become opaque. Currently, the reference counter <code class="docutils literal notranslate">PyObject.ob_refcnt</code> is exposed in the C API. All structures must become opaque, since they “inherit” from PyObject. For, <code class="docutils literal notranslate">PyFloatObject</code> inherits from <code class="docutils literal notranslate">PyObject</code>: <div class="highlight-default notranslate"><div class="highlight"><pre>typedef struct { PyObject ob_base; double ob_fval; } PyFloatObject; </pre></div> </div> Making <code class="docutils literal notranslate">PyObject</code> fully opaque requires converting <code class="docutils literal notranslate">Py_INCREF()</code> and <code class="docutils literal notranslate">Py_DECREF()</code> macros to function calls. This change has an impact on performance. It is likely to be one of the very last changes when making structures opaque. Making <code class="docutils literal notranslate">PyTypeObject</code> structure opaque breaks C extensions declaring types statically (e.g. <code class="docutils literal notranslate">static PyTypeObject MyType = {...};</code>). C extensions must use <code class="docutils literal notranslate">PyType_FromSpec()</code> to allocate types on the heap instead. Using heap types has other advantages like being compatible with subinterpreters. Combined with <a class="pep reference internal" href="../pep-0489/" title="PEP 489 – Multi-phase extension module initialization">PEP 489</a> “Multi-phase extension module initialization”, it makes a C extension behavior closer to a Python module, like allowing to create more than one module instance. Making <code class="docutils literal notranslate">PyThreadState</code> structure opaque requires adding getter and setter functions for members used by C extensions. STATUS: In Progress (started in Python 3.8) The <code class="docutils literal notranslate">PyInterpreterState</code> structure was made opaque in Python 3.8 (<a class="reference external" href="https://bugs.python.org/issue35886">bpo-35886</a>) and the <code class="docutils literal notranslate">PyGC_Head</code> structure (<a class="reference external" href="https://bugs.python.org/issue40241">bpo-40241</a>) was made opaque in Python 3.9. Issues tracking the work to prepare the C API to make following structures opaque: <ul class="simple"> <li><code class="docutils literal notranslate">PyObject</code>: <a class="reference external" href="https://bugs.python.org/issue39573">bpo-39573</a></li> <li><code class="docutils literal notranslate">PyTypeObject</code>: <a class="reference external" href="https://bugs.python.org/issue40170">bpo-40170</a></li> <li><code class="docutils literal notranslate">PyFrameObject</code>: <a class="reference external" href="https://bugs.python.org/issue40421">bpo-40421</a><ul> <li>Python 3.9 adds <code class="docutils literal notranslate">PyFrame_GetCode()</code> and <code class="docutils literal notranslate">PyFrame_GetBack()</code> getter functions, and moves <code class="docutils literal notranslate">PyFrame_GetLineNumber</code> to the limited C API.</li> </ul> </li> <li><code class="docutils literal notranslate">PyThreadState</code>: <a class="reference external" href="https://bugs.python.org/issue39947">bpo-39947</a><ul> <li>Python 3.9 adds 3 getter functions: <code class="docutils literal notranslate">PyThreadState_GetFrame()</code>, <code class="docutils literal notranslate">PyThreadState_GetID()</code>, <code class="docutils literal notranslate">PyThreadState_GetInterpreter()</code>.</li> </ul> </li> </ul> </section> <section id="disallow-using-py-type-as-l-value"> <h3><a class="toc-backref" href="#disallow-using-py-type-as-l-value" role="doc-backlink">Disallow using Py_TYPE() as l-value</a></h3> The <code class="docutils literal notranslate">Py_TYPE()</code> function gets an object type, its <code class="docutils literal notranslate">PyObject.ob_type</code> member. It is implemented as a macro which can be used as an l-value to set the type: <code class="docutils literal notranslate">Py_TYPE(obj) = new_type</code>. This code relies on the assumption that <code class="docutils literal notranslate">PyObject.ob_type</code> can be modified directly. It prevents making the <code class="docutils literal notranslate">PyObject</code> structure opaque. New setter functions <code class="docutils literal notranslate">Py_SET_TYPE()</code>, <code class="docutils literal notranslate">Py_SET_REFCNT()</code> and <code class="docutils literal notranslate">Py_SET_SIZE()</code> are added and must be used instead. The <code class="docutils literal notranslate">Py_TYPE()</code>, <code class="docutils literal notranslate">Py_REFCNT()</code> and <code class="docutils literal notranslate">Py_SIZE()</code> macros must be converted to static inline functions which can not be used as l-value. For example, the <code class="docutils literal notranslate">Py_TYPE()</code> macro: <div class="highlight-default notranslate"><div class="highlight"><pre>#define Py_TYPE(ob) (((PyObject*)(ob))->ob_type) </pre></div> </div> becomes: <div class="highlight-default notranslate"><div class="highlight"><pre>#define _PyObject_CAST_CONST(op) ((const PyObject*)(op)) static inline PyTypeObject* _Py_TYPE(const PyObject *ob) { return ob->ob_type; } #define Py_TYPE(ob) _Py_TYPE(_PyObject_CAST_CONST(ob)) </pre></div> </div> STATUS: Completed (in Python 3.10) New functions <code class="docutils literal notranslate">Py_SET_TYPE()</code>, <code class="docutils literal notranslate">Py_SET_REFCNT()</code> and <code class="docutils literal notranslate">Py_SET_SIZE()</code> were added to Python 3.9. In Python 3.10, <code class="docutils literal notranslate">Py_TYPE()</code>, <code class="docutils literal notranslate">Py_REFCNT()</code> and <code class="docutils literal notranslate">Py_SIZE()</code> can no longer be used as l-value and the new setter functions must be used instead. </section> <section id="new-c-api-functions-must-not-return-borrowed-references"> <h3><a class="toc-backref" href="#new-c-api-functions-must-not-return-borrowed-references" role="doc-backlink">New C API functions must not return borrowed references</a></h3> When a function returns a borrowed reference, Python cannot track when the caller stops using this reference. For example, if the Python <code class="docutils literal notranslate">list</code> type is specialized for small integers, store directly “raw” numbers rather than Python objects, <code class="docutils literal notranslate">PyList_GetItem()</code> has to create a temporary Python object. The problem is to decide when it is safe to delete the temporary object. The general guidelines is to avoid returning borrowed references for new C API functions. No function returning borrowed references is scheduled for removal by this PEP. STATUS: Completed (in Python 3.9) In Python 3.9, new C API functions returning Python objects only return strong references: <ul class="simple"> <li><code class="docutils literal notranslate">PyFrame_GetBack()</code></li> <li><code class="docutils literal notranslate">PyFrame_GetCode()</code></li> <li><code class="docutils literal notranslate">PyObject_CallNoArgs()</code></li> <li><code class="docutils literal notranslate">PyObject_CallOneArg()</code></li> <li><code class="docutils literal notranslate">PyThreadState_GetFrame()</code></li> </ul> </section> <section id="avoid-functions-returning-pyobject"> <h3><a class="toc-backref" href="#avoid-functions-returning-pyobject" role="doc-backlink">Avoid functions returning PyObject**</a></h3> The <code class="docutils literal notranslate">PySequence_Fast_ITEMS()</code> function gives a direct access to an array of <code class="docutils literal notranslate">PyObject*</code> objects. The function is deprecated in favor of <code class="docutils literal notranslate">PyTuple_GetItem()</code> and <code class="docutils literal notranslate">PyList_GetItem()</code>. <code class="docutils literal notranslate">PyTuple_GET_ITEM()</code> can be abused to access directly the <code class="docutils literal notranslate">PyTupleObject.ob_item</code> member: <div class="highlight-default notranslate"><div class="highlight"><pre>PyObject **items = &PyTuple_GET_ITEM(0); </pre></div> </div> The <code class="docutils literal notranslate">PyTuple_GET_ITEM()</code> and <code class="docutils literal notranslate">PyList_GET_ITEM()</code> macros are converted to static inline functions to disallow that. STATUS: Not Started </section> <section id="new-pythoncapi-compat-h-header-file"> <h3><a class="toc-backref" href="#new-pythoncapi-compat-h-header-file" role="doc-backlink">New pythoncapi_compat.h header file</a></h3> Making structures opaque requires modifying C extensions to use getter and setter functions. The practical issue is how to keep support for old Python versions which don’t have these functions. For example, in Python 3.10, it is no longer possible to use <code class="docutils literal notranslate">Py_TYPE()</code> as an l-value. The new <code class="docutils literal notranslate">Py_SET_TYPE()</code> function must be used instead: <div class="highlight-default notranslate"><div class="highlight"><pre>#if PY_VERSION_HEX >= 0x030900A4 Py_SET_TYPE(&MyType, &PyType_Type); #else Py_TYPE(&MyType) = &PyType_Type; #endif </pre></div> </div> This code may ring a bell to developers who ported their Python code base from Python 2 to Python 3. Python will distribute a new <code class="docutils literal notranslate">pythoncapi_compat.h</code> header file which provides new C API functions to old Python versions. Example: <div class="highlight-default notranslate"><div class="highlight"><pre>#if PY_VERSION_HEX < 0x030900A4 static inline void _Py_SET_TYPE(PyObject *ob, PyTypeObject *type) { ob->ob_type = type; } #define Py_SET_TYPE(ob, type) _Py_SET_TYPE((PyObject*)(ob), type) #endif // PY_VERSION_HEX < 0x030900A4 </pre></div> </div> Using this header file, <code class="docutils literal notranslate">Py_SET_TYPE()</code> can be used on old Python versions as well. Developers can copy this file in their project, or even to only copy/paste the few functions needed by their C extension. STATUS: In Progress (implemented but not distributed by CPython yet) The <code class="docutils literal notranslate">pythoncapi_compat.h</code> header file is currently developed at: <a class="reference external" href="https://github.com/pythoncapi/pythoncapi_compat">https://github.com/pythoncapi/pythoncapi_compat</a> </section> </section> <section id="process-to-reduce-the-number-of-broken-c-extensions"> <h2><a class="toc-backref" href="#process-to-reduce-the-number-of-broken-c-extensions" role="doc-backlink">Process to reduce the number of broken C extensions</a></h2> Process to reduce the number of broken C extensions when introducing C API incompatible changes listed in this PEP: <ul class="simple"> <li>Estimate how many popular C extensions are affected by the incompatible change.</li> <li>Coordinate with maintainers of broken C extensions to prepare their code for the future incompatible change.</li> <li>Introduce the incompatible changes in Python. The documentation must explain how to port existing code. It is recommended to merge such changes at the beginning of a development cycle to have more time for tests.</li> <li>Changes which are the most likely to break a large number of C extensions should be announced on the capi-sig mailing list to notify C extensions maintainers to prepare their project for the next Python.</li> <li>If the change breaks too many projects, reverting the change should be discussed, taking in account the number of broken packages, their importance in the Python community, and the importance of the change.</li> </ul> The coordination usually means reporting issues to the projects, or even proposing changes. It does not require waiting for a new release including fixes for every broken project. Since more and more C extensions are written using Cython, rather directly using the C API, it is important to ensure that Cython is prepared in advance for incompatible changes. It gives more time for C extension maintainers to release a new version with code generated with the updated Cython (for C extensions distributing the code generated by Cython). Future incompatible changes can be announced by deprecating a function in the documentation and by annotating the function with <code class="docutils literal notranslate">Py_DEPRECATED()</code>. But making a structure opaque and preventing the usage of a macro as l-value cannot be deprecated with <code class="docutils literal notranslate">Py_DEPRECATED()</code>. The important part is coordination and finding a balance between CPython evolutions and backward compatibility. For example, breaking a random, old, obscure and unmaintained C extension on PyPI is less severe than breaking numpy. If a change is reverted, we move back to the coordination step to better prepare the change. Once more C extensions are ready, the incompatible change can be reconsidered. </section> <section id="version-history"> <h2><a class="toc-backref" href="#version-history" role="doc-backlink">Version History</a></h2> <ul class="simple"> <li>Version 3, June 2020: PEP rewritten from scratch. Python now distributes a new <code class="docutils literal notranslate">pythoncapi_compat.h</code> header and a process is defined to reduce the number of broken C extensions when introducing C API incompatible changes listed in this PEP.</li> <li>Version 2, April 2020: <a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/HKM774XKU7DPJNLUTYHUB5U6VR6EQMJF/#TKHNENOXP6H34E73XGFOL2KKXSM4Z6T2">PEP: Modify the C API to hide implementation details</a>.</li> <li>Version 1, July 2017: <a class="reference external" href="https://mail.python.org/archives/list/python-ideas@python.org/thread/6XATDGWK4VBUQPRHCRLKQECTJIPBVNJQ/#HFBGCWVLSM47JEP6SO67MRFT7Y3EOC44">PEP: Hide implementation details in the C API</a> sent to python-ideas</li> </ul> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0620.rst">https://github.com/python/peps/blob/main/peps/pep-0620.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0620.rst">2025-02-01 08:55:40 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#pep-withdrawn">PEP withdrawn</a></li> <li><a class="reference internal" href="#motivation">Motivation</a><ul> <li><a class="reference internal" href="#the-c-api-blocks-cpython-evolutions">The C API blocks CPython evolutions</a></li> <li><a class="reference internal" href="#same-cpython-design-since-1990-structures-and-reference-counting">Same CPython design since 1990: structures and reference counting</a></li> <li><a class="reference internal" href="#why-is-pypy-more-efficient-than-cpython">Why is PyPy more efficient than CPython?</a></li> <li><a class="reference internal" href="#pypy-bottleneck-the-python-c-api">PyPy bottleneck: the Python C API</a></li> </ul> </li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#hide-implementation-details">Hide implementation details</a></li> <li><a class="reference internal" href="#relationship-with-the-limited-c-api">Relationship with the limited C API</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#summary">Summary</a></li> <li><a class="reference internal" href="#reorganize-the-c-api-header-files">Reorganize the C API header files</a></li> <li><a class="reference internal" href="#move-private-functions-to-the-internal-c-api">Move private functions to the internal C API</a></li> <li><a class="reference internal" href="#convert-macros-to-static-inline-functions">Convert macros to static inline functions</a></li> <li><a class="reference internal" href="#make-structures-opaque">Make structures opaque</a></li> <li><a class="reference internal" href="#disallow-using-py-type-as-l-value">Disallow using Py_TYPE() as l-value</a></li> <li><a class="reference internal" href="#new-c-api-functions-must-not-return-borrowed-references">New C API functions must not return borrowed references</a></li> <li><a class="reference internal" href="#avoid-functions-returning-pyobject">Avoid functions returning PyObject**</a></li> <li><a class="reference internal" href="#new-pythoncapi-compat-h-header-file">New pythoncapi_compat.h header file</a></li> </ul> </li> <li><a class="reference internal" href="#process-to-reduce-the-number-of-broken-c-extensions">Process to reduce the number of broken C extensions</a></li> <li><a class="reference internal" href="#version-history">Version History</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0620.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>