CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 733 – An Evaluation of Python’s Public C API | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0733/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 733 – An Evaluation of Python’s Public C API | peps.python.org'> <meta property="og:description" content="This informational PEP describes our shared view of the public C API. The document defines:"> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0733/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This informational PEP describes our shared view of the public C API. The document defines:"> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 733</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> <span class="visually-hidden">Toggle light / dark / auto colour theme</span> </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 733 – An Evaluation of Python’s Public C API</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author<span class="colon">:</span></dt> <dd class="field-odd">Erlend Egeberg Aasland <erlend at python.org>, Domenico Andreoli <domenico.andreoli at linux.com>, Stefan Behnel <stefan_ml at behnel.de>, Carl Friedrich Bolz-Tereick <cfbolz at gmx.de>, Simon Cross <hodgestar at gmail.com>, Steve Dower <steve.dower at python.org>, Tim Felgentreff <tim.felgentreff at oracle.com>, David Hewitt <1939362+davidhewitt at users.noreply.github.com>, Shantanu Jain <hauntsaninja at gmail.com>, Wenzel Jakob <wenzel.jakob at epfl.ch>, Irit Katriel <irit at python.org>, Marc-Andre Lemburg <mal at lemburg.com>, Donghee Na <donghee.na at python.org>, Karl Nelson <nelson85 at llnl.gov>, Ronald Oussoren <ronaldoussoren at mac.com>, Antoine Pitrou <solipsis at pitrou.net>, Neil Schemenauer <nas at arctrix.com>, Mark Shannon <mark at hotpy.org>, Stepan Sindelar <stepan.sindelar at oracle.com>, Gregory P. Smith <greg at krypto.org>, Eric Snow <ericsnowcurrently at gmail.com>, Victor Stinner <vstinner at python.org>, Guido van Rossum <guido at python.org>, Petr Viktorin <encukou at gmail.com>, Carol Willing <willingc at gmail.com>, William Woodruff <william at yossarian.net>, David Woods <dw-git at d-woods.co.uk>, Jelle Zijlstra <jelle.zijlstra at gmail.com></dd> <dt class="field-even">Status<span class="colon">:</span></dt> <dd class="field-even"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-odd">Type<span class="colon">:</span></dt> <dd class="field-odd"><abbr title="Non-normative PEP containing background, guidelines or other information relevant to the Python ecosystem">Informational</abbr></dd> <dt class="field-even">Created<span class="colon">:</span></dt> <dd class="field-even">16-Oct-2023</dd> <dt class="field-odd">Post-History<span class="colon">:</span></dt> <dd class="field-odd"><a class="reference external" href="https://discuss.python.org/t/pep-733-an-evaluation-of-python-s-public-c-api/37618" title="Discourse thread">01-Nov-2023</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#introduction">Introduction</a></li> <li><a class="reference internal" href="#c-api-stakeholders">C API Stakeholders</a><ul> <li><a class="reference internal" href="#common-actions-for-all-stakeholders">Common Actions for All Stakeholders</a></li> <li><a class="reference internal" href="#extension-writers">Extension Writers</a></li> <li><a class="reference internal" href="#authors-of-embedded-python-applications">Authors of Embedded Python Applications</a></li> <li><a class="reference internal" href="#python-implementations">Python Implementations</a></li> <li><a class="reference internal" href="#alternative-apis-and-binding-generators">Alternative APIs and Binding Generators</a></li> </ul> </li> <li><a class="reference internal" href="#strengths-of-the-c-api">Strengths of the C API</a></li> <li><a class="reference internal" href="#c-api-problems">C API problems</a><ul> <li><a class="reference internal" href="#api-evolution-and-maintenance">API Evolution and Maintenance</a></li> <li><a class="reference internal" href="#api-specification-and-abstraction">API Specification and Abstraction</a></li> <li><a class="reference internal" href="#object-reference-management">Object Reference Management</a></li> <li><a class="reference internal" href="#type-definition-and-object-creation">Type Definition and Object Creation</a></li> <li><a class="reference internal" href="#error-handling">Error Handling</a></li> <li><a class="reference internal" href="#api-tiers-and-stability-guarantees">API Tiers and Stability Guarantees</a></li> <li><a class="reference internal" href="#use-of-the-c-language">Use of the C Language</a></li> <li><a class="reference internal" href="#implementation-flaws">Implementation Flaws</a></li> <li><a class="reference internal" href="#missing-functionality">Missing Functionality</a><ul> <li><a class="reference internal" href="#debug-mode">Debug Mode</a></li> <li><a class="reference internal" href="#introspection">Introspection</a></li> <li><a class="reference internal" href="#improved-interaction-with-other-languages">Improved Interaction with Other Languages</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> <p>This <strong>informational</strong> PEP describes our shared view of the public C API. The document defines:</p> <ul class="simple"> <li>purposes of the C API</li> <li>stakeholders and their particular use cases and requirements</li> <li>strengths of the C API</li> <li>problems of the C API categorized into nine areas of weakness</li> </ul> <p>This document does not propose solutions to any of the identified problems. By creating a shared list of C API issues, this document will help to guide continuing discussion about change proposals and to identify evaluation criteria.</p> </section> <section id="introduction"> <h2><a class="toc-backref" href="#introduction" role="doc-backlink">Introduction</a></h2> <p>Python’s C API was not designed for the different purposes it currently fulfills. It evolved from what was initially the internal API between the C code of the interpreter and the Python language and libraries. In its first incarnation, it was exposed to make it possible to embed Python into C/C++ applications and to write extension modules in C/C++. These capabilities were instrumental to the growth of Python’s ecosystem. Over the decades, the C API grew to provide different tiers of stability, conventions changed, and new usage patterns have emerged, such as bindings to languages other than C/C++. In the next few years, new developments are expected to further test the C API, such as the removal of the GIL and the development of a JIT compiler. However, this growth was not supported by clearly documented guidelines, resulting in inconsistent approaches to API design in different subsystems of CPython. In addition, CPython is no longer the only implementation of Python, and some of the design decisions made when it was, are difficult for alternative implementations to work with [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/64">Issue 64</a>]. In the meantime, lessons were learned and mistakes in both the design and the implementation of the C API were identified.</p> <p>Evolving the C API is hard due to the combination of backwards compatibility constraints and its inherent complexity, both technical and social. Different types of users bring different, sometimes conflicting, requirements. The tradeoff between stability and progress is an ongoing, highly contentious topic of discussion when suggestions are made for incremental improvements. Several proposals have been put forward for improvement, redesign or replacement of the C API, each representing a deep analysis of the problems. At the 2023 Language Summit, three back-to-back sessions were devoted to different aspects of the C API. There is general agreement that a new design can remedy the problems that the C API has accumulated over the last 30 years, while at the same time updating it for use cases that it was not originally designed for.</p> <p>However, there was also a sense at the Language Summit that we are trying to discuss solutions without a clear common understanding of the problems that we are trying to solve. We decided that we need to agree on the current problems with the C API, before we are able to evaluate any of the proposed solutions. We therefore created the <a class="reference external" href="https://github.com/capi-workgroup/problems/issues/">capi-workgroup</a> repository on GitHub in order to collect everyone’s ideas on that question.</p> <p>Over 60 different issues were created on that repository, each describing a problem with the C API. We categorized them and identified a number of recurring themes. The sections below mostly correspond to these themes, and each contains a combined description of the issues raised in that category, along with links to the individual issues. In addition, we included a section that aims to identify the different stakeholders of the C API, and the particular requirements that each of them has.</p> </section> <section id="c-api-stakeholders"> <h2><a class="toc-backref" href="#c-api-stakeholders" role="doc-backlink">C API Stakeholders</a></h2> <p>As mentioned in the introduction, the C API was originally created as the internal interface between CPython’s interpreter and the Python layer. It was later exposed as a way for third-party developers to extend and embed Python programs. Over the years, new types of stakeholders emerged, with different requirements and areas of focus. This section describes this complex state of affairs in terms of the actions that different stakeholders need to perform through the C API.</p> <section id="common-actions-for-all-stakeholders"> <h3><a class="toc-backref" href="#common-actions-for-all-stakeholders" role="doc-backlink">Common Actions for All Stakeholders</a></h3> <p>There are actions which are generic, and required by all types of API users:</p> <ul class="simple"> <li>Define functions and call them</li> <li>Define new types</li> <li>Create instances of builtin and user-defined types</li> <li>Perform operations on object instances</li> <li>Introspect objects, including types, instances, and functions</li> <li>Raise and handle exceptions</li> <li>Import modules</li> <li>Access to Python’s OS interface</li> </ul> <p>The following sections look at the unique requirements of various stakeholders.</p> </section> <section id="extension-writers"> <h3><a class="toc-backref" href="#extension-writers" role="doc-backlink">Extension Writers</a></h3> <p>Extension writers are the traditional users of the C API. Their requirements are the common actions listed above. They also commonly need to:</p> <ul class="simple"> <li>Create new modules</li> <li>Efficiently interface between modules at the C level</li> </ul> </section> <section id="authors-of-embedded-python-applications"> <h3><a class="toc-backref" href="#authors-of-embedded-python-applications" role="doc-backlink">Authors of Embedded Python Applications</a></h3> <p>Applications with an embedded Python interpreter. Examples are <a class="reference external" href="https://docs.blender.org/api/current/info_overview.html">Blender</a> and <a class="reference external" href="https://obsproject.com/wiki/Getting-Started-With-OBS-Scripting">OBS</a>.</p> <p>They need to be able to:</p> <ul class="simple"> <li>Configure the interpreter (import paths, inittab, <code class="docutils literal notranslate"><span class="pre">sys.argv</span></code>, memory allocator, etc.).</li> <li>Interact with the execution model and program lifetime, including clean interpreter shutdown and restart.</li> <li>Represent complex data models in a way Python can use without having to create deep copies.</li> <li>Provide and import frozen modules.</li> <li>Run and manage multiple independent interpreters (in particular, when embedded in a library that wants to avoid global effects).</li> </ul> </section> <section id="python-implementations"> <h3><a class="toc-backref" href="#python-implementations" role="doc-backlink">Python Implementations</a></h3> <p>Python implementations such as <a class="reference external" href="https://www.python.org">CPython</a>, <a class="reference external" href="https://www.pypy.org">PyPy</a>, <a class="reference external" href="https://www.graalvm.org/python/">GraalPy</a>, <a class="reference external" href="https://ironpython.net">IronPython</a>, <a class="reference external" href="https://github.com/RustPython/RustPython">RustPython</a>, <a class="reference external" href="https://micropython.org">MicroPython</a>, and <a class="reference external" href="https://www.jython.org">Jython</a>), may take very different approaches for the implementation of different subsystems. They need:</p> <ul class="simple"> <li>The API to be abstract and hide implementation details.</li> <li>A specification of the API, ideally with a test suite that ensures compatibility.</li> <li>It would be nice to have an ABI that can be shared across Python implementations.</li> </ul> </section> <section id="alternative-apis-and-binding-generators"> <h3><a class="toc-backref" href="#alternative-apis-and-binding-generators" role="doc-backlink">Alternative APIs and Binding Generators</a></h3> <p>There are several projects that implement alternatives to the C API, which offer extension users advantanges over programming directly with the C API. These APIs are implemented with the C API, and in some cases by using CPython internals.</p> <p>There are also libraries that create bindings between Python and other object models, paradigms or languages.</p> <p>There is overlap between these categories: binding generators usually provide alternative APIs, and vice versa.</p> <p>Examples are <a class="reference external" href="https://cython.org">Cython</a>, <a class="reference external" href="https://cffi.readthedocs.io/">cffi</a>, <a class="reference external" href="https://pybind11.readthedocs.io/en/stable/">pybind11</a> and <a class="reference external" href="https://github.com/wjakob/nanobind">nanobind</a> for C++, <a class="reference external" href="https://github.com/PyO3/pyo3">PyO3</a> for Rust, <a class="reference external" href="https://doc.qt.io/qtforpython-6/shiboken6/index.html">Shiboken</a> used by PySide for Qt, <a class="reference external" href="https://pygobject.readthedocs.io/en/latest/">PyGObject</a> for GTK, <a class="reference external" href="https://gitlab.com/pygolo/py">Pygolo</a> for Go, <a class="reference external" href="https://github.com/jpype-project/jpype/">JPype</a> for Java, <a class="reference external" href="https://github.com/kivy/pyjnius/">PyJNIus</a> for Android, <a class="reference external" href="https://pyobjc.readthedocs.io">PyObjC</a> for Objective-C, <a class="reference external" href="https://swig.org/">SWIG</a> for C/C++, <a class="reference external" href="https://github.com/pythonnet/pythonnet">Python.NET</a> for .NET (C#), <a class="reference external" href="https://hpyproject.org">HPy</a>, <a class="reference external" href="https://mypyc.readthedocs.io/en/latest/introduction.html">Mypyc</a>, <a class="reference external" href="https://pythran.readthedocs.io">Pythran</a> and <a class="reference external" href="https://pythoncapi-compat.readthedocs.io/en/latest/">pythoncapi-compat</a>. CPython’s DSL for parsing function arguments, the <a class="reference external" href="https://devguide.python.org/development-tools/clinic/">Argument Clinic</a>, can also be seen as belonging to this category of stakeholders.</p> <p>Alternative APIs need minimal building blocks for accessing CPython efficiently. They don’t necessarily need an ergonomic API, because they typically generate code that is not intended to be read by humans. But they do need it to be comprehensive enough so that they can avoid accessing internals, without sacrificing performance.</p> <p>Binding generators often need to:</p> <ul class="simple"> <li>Create custom objects (e.g. function/module objects and traceback entries) that match the behavior of equivalent Python code as closely as possible.</li> <li>Dynamically create objects which are static in traditional C extensions (e.g. classes/modules), and need CPython to manage their state and lifetime.</li> <li>Dynamically adapt foreign objects (strings, GC’d containers), with low overhead.</li> <li>Adapt external mechanisms, execution models and guarantees to the Python way (stackful coroutines, continuations, one-writer-or-multiple-readers semantics, virtual multiple inheritance, 1-based indexing, super-long inheritance chains, goroutines, channels, etc.).</li> </ul> <p>These tools might also benefit from a choice between a more stable and a faster (possibly lower-level) API. Their users could then decide whether they can afford to regenerate the code often or trade some performance for more stability and less maintenance work.</p> </section> </section> <section id="strengths-of-the-c-api"> <h2><a class="toc-backref" href="#strengths-of-the-c-api" role="doc-backlink">Strengths of the C API</a></h2> <p>While the bulk of this document is devoted to problems with the C API that we would like to see fixed in any new design, it is also important to point out the strengths of the C API, and to make sure that they are preserved.</p> <p>As mentioned in the introduction, the C API enabled the development and growth of the Python ecosystem over the last three decades, while evolving to support use cases that it was not originally designed for. This track record in itself is indication of how effective and valuable it has been.</p> <p>A number of specific strengths were mentioned in the capi-workgroup discussions. Heap types were identified as much safer and easier to use than static types [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/4#issuecomment-1542324451">Issue 4</a>].</p> <p>API functions that take a C string literal for lookups based on a Python string are very convenient [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/30#issuecomment-1550098113">Issue 30</a>].</p> <p>The limited API demonstrates that an API which hides implementation details makes it easier to evolve Python [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/30#issuecomment-1560083258">Issue 30</a>].</p> </section> <section id="c-api-problems"> <h2><a class="toc-backref" href="#c-api-problems" role="doc-backlink">C API problems</a></h2> <p>The remainder of this document summarizes and categorizes the problems that were reported on the <a class="reference external" href="https://github.com/capi-workgroup/problems/issues/">capi-workgroup</a> repository. The issues are grouped into several categories.</p> <section id="api-evolution-and-maintenance"> <h3><a class="toc-backref" href="#api-evolution-and-maintenance" role="doc-backlink">API Evolution and Maintenance</a></h3> <p>The difficulty of making changes in the C API is central to this report. It is implicit in many of the issues we discuss here, particularly when we need to decide whether an incremental bugfix can resolve the issue, or whether it can only be addressed as part of an API redesign [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/44">Issue 44</a>]. The benefit of each incremental change is often viewed as too small to justify the disruption. Over time, this implies that every mistake we make in an API’s design or implementation remains with us indefinitely.</p> <p>We can take two views on this issue. One is that this is a problem and the solution needs to be baked into any new C API we design, in the form of a process for incremental API evolution, which includes deprecation and removal of API elements. The other possible approach is that this is not a problem to be solved, but rather a feature of any API. In this view, API evolution should not be incremental, but rather through large redesigns, each of which learns from the mistakes of the past and is not shackled by backwards compatibility requirements (in the meantime, new API elements may be added, but nothing can ever be removed). A compromise approach is somewhere between these two extremes, fixing issues which are easy or important enough to tackle incrementally, and leaving others alone.</p> <p>The problem we have in CPython is that we don’t have an agreed, official approach to API evolution. Different members of the core team are pulling in different directions and this is an ongoing source of disagreements. Any new C API needs to come with a clear decision about the model that its maintenance will follow, as well as the technical and organizational processes by which this will work.</p> <p>If the model does include provisions for incremental evolution of the API, it will include processes for managing the impact of the change on users [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/60">Issue 60</a>], perhaps through introducing an external backwards compatibility module [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/62">Issue 62</a>], or a new API tier of “blessed” functions [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/55">Issue 55</a>].</p> </section> <section id="api-specification-and-abstraction"> <h3><a class="toc-backref" href="#api-specification-and-abstraction" role="doc-backlink">API Specification and Abstraction</a></h3> <p>The C API does not have a formal specification, it is currently defined as whatever the reference implementation (CPython) contains in a particular version. The documentation acts as an incomplete description, which is not sufficient for verifying the correctness of either the full API, the limited API, or the stable ABI. As a result, the C API may change significantly between releases without needing a more visible specification update, and this leads to a number of problems.</p> <p>Bindings for languages other than C/C++ must parse C code [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/7">Issue 7</a>]. Some C language features are hard to handle in this way, because they produce compiler-dependent output (such as enums) or require a C preprocessor/compiler rather than just a parser (such as macros) [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/35">Issue 35</a>].</p> <p>Furthermore, C header files tend to expose more than what is intended to be part of the public API [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/34">Issue 34</a>]. In particular, implementation details such as the precise memory layouts of internal data structures can be exposed [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/22">Issue 22</a> and <a class="pep reference internal" href="../pep-0620/" title="PEP 620 – Hide implementation details from the C API">PEP 620</a>]. This can make API evolution very difficult, in particular when it occurs in the stable ABI as in the case of <code class="docutils literal notranslate"><span class="pre">ob_refcnt</span></code> and <code class="docutils literal notranslate"><span class="pre">ob_type</span></code>, which are accessed via the reference counting macros [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/45">Issue 45</a>].</p> <p>We identified a deeper issue in relation to the way that reference counting is exposed. The way that C extensions are required to manage references with calls to <code class="docutils literal notranslate"><span class="pre">Py_INCREF</span></code> and <code class="docutils literal notranslate"><span class="pre">Py_DECREF</span></code> is specific to CPython’s memory model, and is hard for alternative Python implementations to emulate. [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/12">Issue 12</a>].</p> <p>Another set of problems arises from the fact that a <code class="docutils literal notranslate"><span class="pre">PyObject*</span></code> is exposed in the C API as an actual pointer rather than a handle. The address of an object serves as its ID and is used for comparison, and this complicates matters for alternative Python implementations that move objects during GC [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/37">Issue 37</a>].</p> <p>A separate issue is that object references are opaque to the runtime, discoverable only through calls to <code class="docutils literal notranslate"><span class="pre">tp_traverse</span></code>/<code class="docutils literal notranslate"><span class="pre">tp_clear</span></code>, which have their own purposes. If there was a way for the runtime to know the structure of the object graph, and keep up with changes in it, this would make it possible for alternative implementations to implement different memory management schemes [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/33">Issue 33</a>].</p> </section> <section id="object-reference-management"> <h3><a class="toc-backref" href="#object-reference-management" role="doc-backlink">Object Reference Management</a></h3> <p>There does not exist a consistent naming convention for functions which makes their reference semantics obvious, and this leads to error prone C API functions, where they do not follow the typical behaviour. When a C API function returns a <code class="docutils literal notranslate"><span class="pre">PyObject*</span></code>, the caller typically gains ownership of a reference to the object. However, there are exceptions where a function returns a “borrowed” reference, which the caller can access but does not own a reference to. Similarly, functions typically do not change the ownership of references to their arguments, but there are exceptions where a function “steals” a reference, i.e., the ownership of the reference is permanently transferred from the caller to the callee by the call [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/8">Issue 8</a> and <a class="reference external" href="https://github.com/capi-workgroup/problems/issues/52">Issue 52</a>]. The terminology used to describe these situations in the documentation can also be improved [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/11">Issue 11</a>].</p> <p>A more radical change is necessary in the case of functions that return “borrowed” references (such as <code class="docutils literal notranslate"><span class="pre">PyList_GetItem</span></code>) [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/5">Issue 5</a> and <a class="reference external" href="https://github.com/capi-workgroup/problems/issues/21">Issue 21</a>] or pointers to parts of the internal structure of an object (such as <code class="docutils literal notranslate"><span class="pre">PyBytes_AsString</span></code>) [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/57">Issue 57</a>]. In both cases, the reference/pointer is valid for as long as the owning object holds the reference, but this time is hard to reason about. Such functions should not exist in the API without a mechanism that can make them safe.</p> <p>For containers, the API is currently missing bulk operations on the references of contained objects. This is particularly important for a stable ABI where <code class="docutils literal notranslate"><span class="pre">INCREF</span></code> and <code class="docutils literal notranslate"><span class="pre">DECREF</span></code> cannot be macros, making bulk operations expensive when implemented as a sequence of function calls [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/15">Issue 15</a>].</p> </section> <section id="type-definition-and-object-creation"> <h3><a class="toc-backref" href="#type-definition-and-object-creation" role="doc-backlink">Type Definition and Object Creation</a></h3> <p>The C API has functions that make it possible to create incomplete or inconsistent Python objects, such as <code class="docutils literal notranslate"><span class="pre">PyTuple_New</span></code> and <code class="docutils literal notranslate"><span class="pre">PyUnicode_New</span></code>. This causes problems when the object is tracked by GC or its <code class="docutils literal notranslate"><span class="pre">tp_traverse</span></code>/<code class="docutils literal notranslate"><span class="pre">tp_clear</span></code> functions are called. A related issue is with functions such as <code class="docutils literal notranslate"><span class="pre">PyTuple_SetItem</span></code> which is used to modify a partially initialized tuple (tuples are immutable once fully initialized) [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/56">Issue 56</a>].</p> <p>We identified a few issues with type definition APIs. For legacy reasons, there is often a significant amount of code duplication between <code class="docutils literal notranslate"><span class="pre">tp_new</span></code> and <code class="docutils literal notranslate"><span class="pre">tp_vectorcall</span></code> [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/24">Issue 24</a>]. The type slot function should be called indirectly, so that their signatures can change to include context information [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/13">Issue 13</a>]. Several aspects of the type definition and creation process are not well defined, such as which stage of the process is responsible for initializing and clearing different fields of the type object [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/49">Issue 49</a>].</p> </section> <section id="error-handling"> <h3><a class="toc-backref" href="#error-handling" role="doc-backlink">Error Handling</a></h3> <p>Error handling in the C API is based on the error indicator which is stored on the thread state (in global scope). The design intention was that each API function returns a value indicating whether an error has occurred (by convention, <code class="docutils literal notranslate"><span class="pre">-1</span></code> or <code class="docutils literal notranslate"><span class="pre">NULL</span></code>). When the program knows that an error occurred, it can fetch the exception object which is stored in the error indicator. We identified a number of problems which are related to error handling, pointing at APIs which are too easy to use incorrectly.</p> <p>There are functions that do not report all errors that occur while they execute. For example, <code class="docutils literal notranslate"><span class="pre">PyDict_GetItem</span></code> clears any errors that occur when it calls the key’s hash function, or while performing a lookup in the dictionary [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/51">Issue 51</a>].</p> <p>Python code never executes with an in-flight exception (by definition), and typically code using native functions should also be interrupted by an error being raised. This is not checked in most C API functions, and there are places in the interpreter where error handling code calls a C API function while an exception is set. For example, see the call to <code class="docutils literal notranslate"><span class="pre">PyUnicode_FromString</span></code> in the error handler of <code class="docutils literal notranslate"><span class="pre">_PyErr_WriteUnraisableMsg</span></code> [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/2">Issue 2</a>].</p> <p>There are functions that do not return a value, so a caller is forced to query the error indicator in order to identify whether an error has occurred. An example is <code class="docutils literal notranslate"><span class="pre">PyBuffer_Release</span></code> [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/20">Issue 20</a>]. There are other functions which do have a return value, but this return value does not unambiguously indicate whether an error has occurred. For example, <code class="docutils literal notranslate"><span class="pre">PyLong_AsLong</span></code> returns <code class="docutils literal notranslate"><span class="pre">-1</span></code> in case of error, or when the value of the argument is indeed <code class="docutils literal notranslate"><span class="pre">-1</span></code> [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/1">Issue 1</a>]. In both cases, the API is error prone because it is possible that the error indicator was already set before the function was called, and the error is incorrectly attributed. The fact that the error was not detected before the call is a bug in the calling code, but the behaviour of the program in this case doesn’t make it easy to identify and debug the problem.</p> <p>There are functions that take a <code class="docutils literal notranslate"><span class="pre">PyObject*</span></code> argument, with special meaning when it is <code class="docutils literal notranslate"><span class="pre">NULL</span></code>. For example, if <code class="docutils literal notranslate"><span class="pre">PyObject_SetAttr</span></code> receives <code class="docutils literal notranslate"><span class="pre">NULL</span></code> as the value to set, this means that the attribute should be cleared. This is error prone because it could be that <code class="docutils literal notranslate"><span class="pre">NULL</span></code> indicates an error in the construction of the value, and the program failed to check for this error. The program will misinterpret the <code class="docutils literal notranslate"><span class="pre">NULL</span></code> to mean something different than error [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/47">Issue 47</a>].</p> </section> <section id="api-tiers-and-stability-guarantees"> <h3><a class="toc-backref" href="#api-tiers-and-stability-guarantees" role="doc-backlink">API Tiers and Stability Guarantees</a></h3> <p>The different API tiers provide different tradeoffs of stability vs API evolution, and sometimes performance.</p> <p>The stable ABI was identified as an area that needs to be looked into. At the moment it is incomplete and not widely adopted. At the same time, its existence is making it hard to make changes to some implementation details, because it exposes struct fields such as <code class="docutils literal notranslate"><span class="pre">ob_refcnt</span></code>, <code class="docutils literal notranslate"><span class="pre">ob_type</span></code> and <code class="docutils literal notranslate"><span class="pre">ob_size</span></code>. There was some discussion about whether the stable ABI is worth keeping. Arguments on both sides can be found in [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/4">Issue 4</a>] and [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/9">Issue 9</a>].</p> <p>Alternatively, it was suggested that in order to be able to evolve the stable ABI, we need a mechanism to support multiple versions of it in the same Python binary. It was pointed out that versioning individual functions within a single ABI version is not enough because it may be necessary to evolve, together, a group of functions that interoperate with each other [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/39">Issue 39</a>].</p> <p>The limited API was introduced in 3.2 as a blessed subset of the C API which is recommended for users who would like to restrict themselves to high quality APIs which are not likely to change often. The <code class="docutils literal notranslate"><span class="pre">Py_LIMITED_API</span></code> flag allows users to restrict their program to older versions of the limited API, but we now need the opposite option, to exclude older versions. This would make it possible to evolve the limited API by replacing flawed elements in it [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/54">Issue 54</a>]. More generally, in a redesign we should revisit the way that API tiers are specified and consider designing a method that will unify the way we currently select between the different tiers [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/59">Issue 59</a>].</p> <p>API elements whose names begin with an underscore are considered private, essentially an API tier with no stability guarantees. However, this was only clarified recently, in <a class="pep reference internal" href="../pep-0689/" title="PEP 689 – Unstable C API tier">PEP 689</a>. It is not clear what the change policy should be with respect to such API elements that predate PEP 689 [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/58">Issue 58</a>].</p> <p>There are API functions which have an unsafe (but fast) version as well as a safe version which performs error checking (for example, <code class="docutils literal notranslate"><span class="pre">PyTuple_GET_ITEM</span></code> vs <code class="docutils literal notranslate"><span class="pre">PyTuple_GetItem</span></code>). It may help to be able to group them into their own tiers - the “unsafe API” tier and the “safe API” tier [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/61">Issue 61</a>].</p> </section> <section id="use-of-the-c-language"> <h3><a class="toc-backref" href="#use-of-the-c-language" role="doc-backlink">Use of the C Language</a></h3> <p>A number of issues were raised with respect to the way that CPython uses the C language. First there is the issue of which C dialect we use, and how we test our compatibility with it, as well as API header compatibility with C++ dialects [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/42">Issue 42</a>].</p> <p>Usage of <code class="docutils literal notranslate"><span class="pre">const</span></code> in the API is currently sparse, but it is not clear whether this is something that we should consider changing [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/38">Issue 38</a>].</p> <p>We currently use the C types <code class="docutils literal notranslate"><span class="pre">long</span></code> and <code class="docutils literal notranslate"><span class="pre">int</span></code>, where fixed-width integers such as <code class="docutils literal notranslate"><span class="pre">int32_t</span></code> and <code class="docutils literal notranslate"><span class="pre">int64_t</span></code> may now be better choices [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/27">Issue 27</a>].</p> <p>We are using C language features which are hard for other languages to interact with, such as macros, variadic arguments, enums, bitfields, and non-function symbols [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/35">Issue 35</a>].</p> <p>There are API functions that take a <code class="docutils literal notranslate"><span class="pre">PyObject*</span></code> arg which must be of a more specific type (such as <code class="docutils literal notranslate"><span class="pre">PyTuple_Size</span></code>, which fails if its arg is not a <code class="docutils literal notranslate"><span class="pre">PyTupleObject*</span></code>). It is an open question whether this is a good pattern to have, or whether the API should expect the more specific type [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/31">Issue 31</a>].</p> <p>There are functions in the API that take concrete types, such as <code class="docutils literal notranslate"><span class="pre">PyDict_GetItemString</span></code> which performs a dictionary lookup for a key specified as a C string rather than <code class="docutils literal notranslate"><span class="pre">PyObject*</span></code>. At the same time, for <code class="docutils literal notranslate"><span class="pre">PyDict_ContainsString</span></code> it is not considered appropriate to add a concrete type alternative. The principle around this should be documented in the guidelines [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/23">Issue 23</a>].</p> </section> <section id="implementation-flaws"> <h3><a class="toc-backref" href="#implementation-flaws" role="doc-backlink">Implementation Flaws</a></h3> <p>Below is a list of localized implementation flaws. Most of these can probably be fixed incrementally, if we choose to do so. They should, in any case, be avoided in any new API design.</p> <p>There are functions that don’t follow the convention of returning <code class="docutils literal notranslate"><span class="pre">0</span></code> for success and <code class="docutils literal notranslate"><span class="pre">-1</span></code> for failure. For example, <code class="docutils literal notranslate"><span class="pre">PyArg_ParseTuple</span></code> returns 0 for success and non-zero for failure [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/25">Issue 25</a>].</p> <p>The macros <code class="docutils literal notranslate"><span class="pre">Py_CLEAR</span></code> and <code class="docutils literal notranslate"><span class="pre">Py_SETREF</span></code> access their arg more than once, so if the arg is an expression with side effects, they are duplicated [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/3">Issue 3</a>].</p> <p>The meaning of <code class="docutils literal notranslate"><span class="pre">Py_SIZE</span></code> depends on the type and is not always reliable [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/10">Issue 10</a>].</p> <p>Some API function do not have the same behaviour as their Python equivalents. The behaviour of <code class="docutils literal notranslate"><span class="pre">PyIter_Next</span></code> is different from <code class="docutils literal notranslate"><span class="pre">tp_iternext</span></code>. [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/29">Issue 29</a>]. The behaviour of <code class="docutils literal notranslate"><span class="pre">PySet_Contains</span></code> is different from <code class="docutils literal notranslate"><span class="pre">set.__contains__</span></code> [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/6">Issue 6</a>].</p> <p>The fact that <code class="docutils literal notranslate"><span class="pre">PyArg_ParseTupleAndKeywords</span></code> takes a non-const <code class="docutils literal notranslate"><span class="pre">char*</span></code> array as argument makes it more difficult to use [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/28">Issue 28</a>].</p> <p><code class="docutils literal notranslate"><span class="pre">Python.h</span></code> does not expose the whole API. Some headers (like <code class="docutils literal notranslate"><span class="pre">marshal.h</span></code>) are not included from <code class="docutils literal notranslate"><span class="pre">Python.h</span></code>. [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/43">Issue 43</a>].</p> <p><strong>Naming</strong></p> <p><code class="docutils literal notranslate"><span class="pre">PyLong</span></code> and <code class="docutils literal notranslate"><span class="pre">PyUnicode</span></code> use names which no longer match the Python types they represent (<code class="docutils literal notranslate"><span class="pre">int</span></code>/<code class="docutils literal notranslate"><span class="pre">str</span></code>). This could be fixed in a new API [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/14">Issue 14</a>].</p> <p>There are identifiers in the API which are lacking a <code class="docutils literal notranslate"><span class="pre">Py</span></code>/<code class="docutils literal notranslate"><span class="pre">_Py</span></code> prefix [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/46">Issue 46</a>].</p> </section> <section id="missing-functionality"> <h3><a class="toc-backref" href="#missing-functionality" role="doc-backlink">Missing Functionality</a></h3> <p>This section consists of a list of feature requests, i.e., functionality that was identified as missing in the current C API.</p> <section id="debug-mode"> <h4><a class="toc-backref" href="#debug-mode" role="doc-backlink">Debug Mode</a></h4> <p>A debug mode that can be activated without recompilation and which activates various checks that can help detect various types of errors [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/36">Issue 36</a>].</p> </section> <section id="introspection"> <h4><a class="toc-backref" href="#introspection" role="doc-backlink">Introspection</a></h4> <p>There aren’t currently reliable introspection capabilities for objects defined in C in the same way as there are for Python objects [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/32">Issue 32</a>].</p> <p>Efficient type checking for heap types [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/17">Issue 17</a>].</p> </section> <section id="improved-interaction-with-other-languages"> <h4><a class="toc-backref" href="#improved-interaction-with-other-languages" role="doc-backlink">Improved Interaction with Other Languages</a></h4> <p>Interfacing with other GC based languages, and integrating their GC with Python’s GC [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/19">Issue 19</a>].</p> <p>Inject foreign stack frames to the traceback [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/18">Issue 18</a>].</p> <p>Concrete strings that can be used in other languages [<a class="reference external" href="https://github.com/capi-workgroup/problems/issues/16">Issue 16</a>].</p> </section> </section> </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <ol class="arabic simple"> <li><a class="reference external" href="https://docs.python.org/3/c-api/index.html">Python/C API Reference Manual</a></li> <li><a class="reference external" href="https://pyfound.blogspot.com/2023/05/the-python-language-summit-2023-three.html">2023 Language Summit Blog Post: Three Talks on the C API</a></li> <li><a class="reference external" href="https://github.com/capi-workgroup">capi-workgroup on GitHub</a></li> <li><a class="reference external" href="https://github.com/iritkatriel/talks/blob/main/2023_Sprint_Brno_C_API.pdf">Irit’s Core Sprint 2023 slides about C API workgroup</a></li> <li><a class="reference external" href="https://drive.google.com/file/d/148NLRPXGZGI1SXfKLMFvQc_iv67hPJQS/view?usp=sharing">Petr’s Core Sprint 2023 slides</a></li> <li><a class="reference external" href="https://hpyproject.org/talks/2023/10/things_to_learn_from_hpy.pdf">HPy team’s Core Sprint 2023 slides for Things to Learn from HPy</a></li> <li><a class="reference external" href="https://github.com/vstinner/talks/blob/main/2023-CoreDevSprint-Brno/c-api.pdf">Victor’s slides of Core Sprint 2023 Python C API talk</a></li> <li><a class="reference external" href="https://www.youtube.com/watch?v=iiBJF0kM-P8">The Python’s stability promise — Cristián Maureira-Fredes, PySide maintainer</a></li> <li><a class="reference external" href="https://github.com/pyside/pyside2-setup/blob/5.11/sources/shiboken2/libshiboken/pep384impl_doc.rst">Report on the issues PySide had 5 years ago when switching to the stable ABI</a></li> </ol> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> <p>This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.</p> </section> </section> <hr class="docutils" /> <p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0733.rst">https://github.com/python/peps/blob/main/peps/pep-0733.rst</a></p> <p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0733.rst">2024-10-28 18:52:58 GMT</a></p> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#introduction">Introduction</a></li> <li><a class="reference internal" href="#c-api-stakeholders">C API Stakeholders</a><ul> <li><a class="reference internal" href="#common-actions-for-all-stakeholders">Common Actions for All Stakeholders</a></li> <li><a class="reference internal" href="#extension-writers">Extension Writers</a></li> <li><a class="reference internal" href="#authors-of-embedded-python-applications">Authors of Embedded Python Applications</a></li> <li><a class="reference internal" href="#python-implementations">Python Implementations</a></li> <li><a class="reference internal" href="#alternative-apis-and-binding-generators">Alternative APIs and Binding Generators</a></li> </ul> </li> <li><a class="reference internal" href="#strengths-of-the-c-api">Strengths of the C API</a></li> <li><a class="reference internal" href="#c-api-problems">C API problems</a><ul> <li><a class="reference internal" href="#api-evolution-and-maintenance">API Evolution and Maintenance</a></li> <li><a class="reference internal" href="#api-specification-and-abstraction">API Specification and Abstraction</a></li> <li><a class="reference internal" href="#object-reference-management">Object Reference Management</a></li> <li><a class="reference internal" href="#type-definition-and-object-creation">Type Definition and Object Creation</a></li> <li><a class="reference internal" href="#error-handling">Error Handling</a></li> <li><a class="reference internal" href="#api-tiers-and-stability-guarantees">API Tiers and Stability Guarantees</a></li> <li><a class="reference internal" href="#use-of-the-c-language">Use of the C Language</a></li> <li><a class="reference internal" href="#implementation-flaws">Implementation Flaws</a></li> <li><a class="reference internal" href="#missing-functionality">Missing Functionality</a><ul> <li><a class="reference internal" href="#debug-mode">Debug Mode</a></li> <li><a class="reference internal" href="#introspection">Introspection</a></li> <li><a class="reference internal" href="#improved-interaction-with-other-languages">Improved Interaction with Other Languages</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <br> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0733.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>