CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 697 – Limited C API for Extending Opaque Types | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0697/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 697 – Limited C API for Extending Opaque Types | peps.python.org'> <meta property="og:description" content="Add Limited C API support for extending some types with opaque data by allowing code to only deal with data specific to a particular (sub)class."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0697/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="Add Limited C API support for extending some types with opaque data by allowing code to only deal with data specific to a particular (sub)class."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 697</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 697 – Limited C API for Extending Opaque Types</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Petr Viktorin <encukou at gmail.com></dd> <dt class="field-even">Discussions-To:</dt> <dd class="field-even"><a class="reference external" href="https://discuss.python.org/t/19743">Discourse thread</a></dd> <dt class="field-odd">Status:</dt> <dd class="field-odd"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-even">Type:</dt> <dd class="field-even"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-odd">Created:</dt> <dd class="field-odd">23-Aug-2022</dd> <dt class="field-even">Python-Version:</dt> <dd class="field-even">3.12</dd> <dt class="field-odd">Post-History:</dt> <dd class="field-odd"><a class="reference external" href="https://mail.python.org/archives/list/capi-sig@python.org/thread/SIP3VP7JU4OBWP62KBOYGOYCVIOTXEFH/" title="Capi-SIG thread">24-May-2022</a>, <a class="reference external" href="https://discuss.python.org/t/19743" title="Discourse thread">06-Oct-2022</a></dd> <dt class="field-even">Resolution:</dt> <dd class="field-even"><a class="reference external" href="https://discuss.python.org/t/19743/30">Discourse message</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#extending-opaque-types">Extending opaque types</a></li> <li><a class="reference internal" href="#extending-variable-size-objects">Extending variable-size objects</a><ul> <li><a class="reference internal" href="#variable-size-layouts">Variable-size layouts</a></li> <li><a class="reference internal" href="#extending-classes-with-the-pyheaptypeobject-like-layout">Extending classes with the <code class="docutils literal notranslate">PyHeapTypeObject</code>-like layout</a></li> <li><a class="reference internal" href="#big-picture">Big picture</a></li> </ul> </li> <li><a class="reference internal" href="#relative-member-offsets">Relative member offsets</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#relative-basicsize">Relative <code class="docutils literal notranslate">basicsize</code></a></li> <li><a class="reference internal" href="#inheriting-itemsize">Inheriting <code class="docutils literal notranslate">itemsize</code></a></li> <li><a class="reference internal" href="#id2">Relative member offsets</a></li> </ul> </li> <li><a class="reference internal" href="#list-of-new-api">List of new API</a></li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a></li> <li><a class="reference internal" href="#assumptions">Assumptions</a></li> <li><a class="reference internal" href="#security-implications">Security Implications</a></li> <li><a class="reference internal" href="#endorsements">Endorsements</a></li> <li><a class="reference internal" href="#how-to-teach-this">How to Teach This</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#possible-future-enhancements">Possible Future Enhancements</a><ul> <li><a class="reference internal" href="#alignment-performance">Alignment & Performance</a></li> <li><a class="reference internal" href="#other-layouts-for-variable-size-types">Other layouts for variable-size types</a></li> </ul> </li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a></li> <li><a class="reference internal" href="#footnotes">Footnotes</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <div class="pep-banner canonical-doc sticky-banner admonition important"> Important This PEP is a historical document. The up-to-date, canonical documentation can now be found at <a class="reference external" href="https://docs.python.org/3.12/c-api/type.html#c.PyType_Spec.basicsize" title="(in Python v3.12)"><code class="xref c c-member docutils literal notranslate">PyType_Spec.basicsize</code></a>, <a class="reference external" href="https://docs.python.org/3.12/c-api/object.html#c.PyObject_GetTypeData" title="(in Python v3.12)"><code class="xref c c-func docutils literal notranslate">PyObject_GetTypeData()</code></a>, <a class="reference external" href="https://docs.python.org/3.12/c-api/typeobj.html#c.Py_TPFLAGS_ITEMS_AT_END" title="(in Python v3.12)"><code class="xref c c-macro docutils literal notranslate">Py_TPFLAGS_ITEMS_AT_END</code></a>, <a class="reference external" href="https://docs.python.org/3.12/c-api/structures.html#c.Py_RELATIVE_OFFSET" title="(in Python v3.12)"><code class="xref c c-macro docutils literal notranslate">Py_RELATIVE_OFFSET</code></a>, <a class="reference external" href="https://docs.python.org/3.12/c-api/object.html#c.PyObject_GetItemData" title="(in Python v3.12)"><code class="xref c c-func docutils literal notranslate">PyObject_GetItemData()</code></a>. × See <a class="pep reference internal" href="../pep-0001/" title="PEP 1 – PEP Purpose and Guidelines">PEP 1</a> for how to propose changes. </div> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> Add <a class="reference external" href="https://docs.python.org/3.11/c-api/stable.html#stable-application-binary-interface">Limited C API</a> support for extending some types with opaque data by allowing code to only deal with data specific to a particular (sub)class. This mechanism is required to be usable with <code class="docutils literal notranslate">PyHeapTypeObject</code>. This PEP does not propose allowing to extend non-dynamically sized variable sized objects such as <code class="docutils literal notranslate">tuple</code> or <code class="docutils literal notranslate">int</code> due to their different memory layout and perceived lack of demand for doing so. This PEP leaves room to do so in the future via the same mechanism if ever desired. </section> <section id="motivation"> <h2><a class="toc-backref" href="#motivation" role="doc-backlink">Motivation</a></h2> The motivating problem this PEP solves is attaching C-level state to custom types — i.e. metaclasses (subclasses of <a class="reference external" href="https://docs.python.org/3/library/functions.html#type" title="(in Python v3.13)"><code class="docutils literal notranslate">type</code></a>). This is often needed in “wrappers” that expose another type system (e.g. C++, Java, Rust) as Python classes. These typically need to attach information about the “wrapped” non-Python class to the Python type object. This should be possible to do in the Limited API, so that the language wrappers or code generators can be used to create Stable ABI extensions. (See <a class="pep reference internal" href="../pep-0652/" title="PEP 652 – Maintaining the Stable ABI">PEP 652</a> for the benefits of providing a stable ABI.) Extending <code class="docutils literal notranslate">type</code> is an instance of a more general problem: extending a class while maintaining loose coupling – that is, not depending on the memory layout used by the superclass. (That’s a lot of jargon; see Rationale for a concrete example of extending <code class="docutils literal notranslate">list</code>.) </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> <section id="extending-opaque-types"> <h3><a class="toc-backref" href="#extending-opaque-types" role="doc-backlink">Extending opaque types</a></h3> In the Limited API, most <code class="docutils literal notranslate">struct</code>s are opaque: their size and memory layout are not exposed, so they can be changed in new versions of CPython (or alternate implementations of the C API). This means that the usual subclassing pattern – making the <code class="docutils literal notranslate">struct</code> used for instances of the base type be the first element of the <code class="docutils literal notranslate">struct</code> used for instances of the derived type – does not work. To illustrate with code, the <a class="reference external" href="https://docs.python.org/3.11/extending/newtypes_tutorial.html#subclassing-other-types">example from the tutorial</a> extends <a class="reference external" href="https://docs.python.org/3/c-api/list.html#c.PyListObject" title="(in Python v3.13)"><code class="xref c c-type docutils literal notranslate">PyListObject</code></a> (<a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#list" title="(in Python v3.13)"><code class="docutils literal notranslate">list</code></a>) using the following <code class="docutils literal notranslate">struct</code>: <div class="highlight-c notranslate"><div class="highlight"><pre>typedef struct { PyListObject list; int state; } SubListObject; </pre></div> </div> This won’t compile in the Limited API, since <code class="docutils literal notranslate">PyListObject</code> is opaque (to allow changes as features and optimizations are implemented). Instead, this PEP proposes using a <code class="docutils literal notranslate">struct</code> with only the state needed in the subclass, that is: <div class="highlight-c notranslate"><div class="highlight"><pre>typedef struct { int state; } SubListState; // (or just `typedef int SubListState;` in this case) </pre></div> </div> The subclass can now be completely decoupled from the memory layout (and size) of the superclass. This is possible today. To use such a struct: <ul class="simple"> <li>when creating the class, use <code class="docutils literal notranslate">PyListObject->tp_basicsize + sizeof(SubListState)</code> as <code class="docutils literal notranslate">PyType_Spec.basicsize</code>;</li> <li>when accessing the data, use <code class="docutils literal notranslate">PyListObject->tp_basicsize</code> as the offset into the instance (<code class="docutils literal notranslate">PyObject*</code>).</li> </ul> However, this has disadvantages: <ul class="simple"> <li>The base’s <code class="docutils literal notranslate">basicsize</code> may not be properly aligned, causing issues on some architectures if not mitigated. (These issues can be particularly nasty if alignment changes in a new release.)</li> <li><code class="docutils literal notranslate">PyTypeObject.tp_basicsize</code> is not exposed in the Limited API, so extensions that support Limited API need to use <code class="docutils literal notranslate">PyObject_GetAttrString(obj, "__basicsize__")</code>. This is cumbersome, and unsafe in edge cases (the Python attribute can be overridden).</li> <li>Variable-size objects are not handled (see <a class="reference internal" href="#extending-variable-size-objects">Extending variable-size objects</a> below).</li> </ul> To make this easy (and even best practice for projects that choose loose coupling over maximum performance), this PEP proposes an API to: <ol class="arabic"> <li>During class creation, specify that <code class="docutils literal notranslate">SubListState</code> should be “appended” to <code class="docutils literal notranslate">PyListObject</code>, without passing any additional details about <code class="docutils literal notranslate">list</code>. (The interpreter itself gets all necessary info, like <code class="docutils literal notranslate">tp_basicsize</code>, from the base).This will be specified by a negative <code class="docutils literal notranslate">PyType_Spec.basicsize</code>: <code class="docutils literal notranslate">-sizeof(SubListState)</code>. </li> <li>Given an instance, and the subclass <code class="docutils literal notranslate">PyTypeObject*</code>, get a pointer to the <code class="docutils literal notranslate">SubListState</code>. A new function, <code class="docutils literal notranslate">PyObject_GetTypeData</code>, will be added for this.</li> </ol> The base class is not limited to <code class="docutils literal notranslate">PyListObject</code>, of course: it can be used to extend any base class whose instance <code class="docutils literal notranslate">struct</code> is opaque, unstable across releases, or not exposed at all – including <a class="reference external" href="https://docs.python.org/3/library/functions.html#type" title="(in Python v3.13)"><code class="docutils literal notranslate">type</code></a> (<code class="docutils literal notranslate">PyHeapTypeObject</code>) or third-party extensions (for example, NumPy arrays <a class="footnote-reference brackets" href="#f1" id="id1">[1]</a>). For cases where no additional state is needed, a zero <code class="docutils literal notranslate">basicsize</code> will be allowed: in that case, the base’s <code class="docutils literal notranslate">tp_basicsize</code> will be inherited. (This currently works, but lacks explicit documentation and tests.) The <code class="docutils literal notranslate">tp_basicsize</code> of the new class will be set to the computed total size, so code that inspects classes will continue working as before. </section> <section id="extending-variable-size-objects"> <h3><a class="toc-backref" href="#extending-variable-size-objects" role="doc-backlink">Extending variable-size objects</a></h3> Additional considerations are needed to subclass <a class="reference external" href="https://docs.python.org/3/c-api/structures.html#c.PyVarObject" title="(in Python v3.13)"><code class="xref c c-type docutils literal notranslate">variable-sized objects</code></a> while maintaining loose coupling: the variable-sized data can collide with subclass data (<code class="docutils literal notranslate">SubListState</code> in the example above). Currently, CPython doesn’t provide a way to prevent such collisions. So, the proposed mechanism of extending opaque classes (negative <code class="docutils literal notranslate">base->tp_itemsize</code>) will fail by default. We could stop there, but since the motivating type — <code class="docutils literal notranslate">PyHeapTypeObject</code> — is variable sized, we need a safe way to allow subclassing it. A bit of background first: <section id="variable-size-layouts"> <h4><a class="toc-backref" href="#variable-size-layouts" role="doc-backlink">Variable-size layouts</a></h4> There are two main memory layouts for variable-sized objects. In types such as <code class="docutils literal notranslate">int</code> or <code class="docutils literal notranslate">tuple</code>, the variable data is stored at a fixed offset. If subclasses need additional space, it must be added after any variable-sized data: <div class="highlight-default notranslate"><div class="highlight"><pre>PyTupleObject: ┌───────────────────┬───┬───┬╌╌╌╌┐ │ PyObject_VAR_HEAD │var. data │ └───────────────────┴───┴───┴╌╌╌╌┘ tuple subclass: ┌───────────────────┬───┬───┬╌╌╌╌┬─────────────┐ │ PyObject_VAR_HEAD │var. data │subclass data│ └───────────────────┴───┴───┴╌╌╌╌┴─────────────┘ </pre></div> </div> In other types, like <code class="docutils literal notranslate">PyHeapTypeObject</code>, variable-sized data always lives at the end of the instance’s memory area: <div class="highlight-default notranslate"><div class="highlight"><pre>heap type: ┌───────────────────┬──────────────┬───┬───┬╌╌╌╌┐ │ PyObject_VAR_HEAD │Heap type data│var. data │ └───────────────────┴──────────────┴───┴───┴╌╌╌╌┘ type subclass: ┌───────────────────┬──────────────┬─────────────┬───┬───┬╌╌╌╌┐ │ PyObject_VAR_HEAD │Heap type data│subclass data│var. data │ └───────────────────┴──────────────┴─────────────┴───┴───┴╌╌╌╌┘ </pre></div> </div> The first layout enables fast access to the items array. The second allows subclasses to ignore the variable-sized array (assuming they use offsets from the start of the object to access their data). Since this PEP focuses on <code class="docutils literal notranslate">PyHeapTypeObject</code>, it proposes an API to allow subclassing for the second variant. Support for the first can be added later as an API-compatible change (though your PEP author doubts it’d be worth the effort). </section> <section id="extending-classes-with-the-pyheaptypeobject-like-layout"> <h4><a class="toc-backref" href="#extending-classes-with-the-pyheaptypeobject-like-layout" role="doc-backlink">Extending classes with the <code class="docutils literal notranslate">PyHeapTypeObject</code>-like layout</a></h4> This PEP proposes a type flag, <code class="docutils literal notranslate">Py_TPFLAGS_ITEMS_AT_END</code>, which will indicate the <code class="docutils literal notranslate">PyHeapTypeObject</code>-like layout. This can be set in two ways: <ul class="simple"> <li>the superclass can set the flag, allowing subclass authors to not care about the fact that <code class="docutils literal notranslate">itemsize</code> is involved, or</li> <li>the new subclass sets the flag, asserting that the author knows the superclass is suitable (but perhaps hasn’t been updated to use the flag yet).</li> </ul> This flag will be necessary to extend a variable-sized type using negative <code class="docutils literal notranslate">basicsize</code>. An alternative to a flag would be to require subclass authors to know that the base uses a compatible layout (e.g. from documentation). A past version of this PEP proposed a new <code class="docutils literal notranslate">PyType_Slot</code> for it. This turned out to be hard to explain, and goes against the idea of decoupling the subclass from the base layout. The new flag will be used to allow safely extending variable-sized types: creating a type with <code class="docutils literal notranslate">spec->basicsize < 0</code> and <code class="docutils literal notranslate">base->tp_itemsize > 0</code> will require the flag. Additionally, this PEP proposes a helper function to get the variable-sized data of a given instance, if it uses the new <code class="docutils literal notranslate">Py_TPFLAGS_ITEMS_AT_END</code> flag. This hides the necessary pointer arithmetic behind an API that can potentially be adapted to other layouts in the future (including, potentially, a VM-managed layout). </section> <section id="big-picture"> <h4><a class="toc-backref" href="#big-picture" role="doc-backlink">Big picture</a></h4> To make it easier to verify that all cases are covered, here’s a scary-looking big-picture decision tree. <div class="admonition note"> Note The individual cases are easier to explain in isolation (see the <a class="reference internal" href="#reference-implementation">reference implementation</a> for draft docs). </div> <ul class="simple"> <li><code class="docutils literal notranslate">spec->basicsize > 0</code>: No change to the status quo. (The base class layout is known.)</li> <li><code class="docutils literal notranslate">spec->basicsize == 0</code>: (Inheriting the basicsize)<ul> <li><code class="docutils literal notranslate">base->tp_itemsize == 0</code>: The item size is set to <code class="docutils literal notranslate">spec->tp_itemsize</code>. (No change to status quo.)</li> <li><code class="docutils literal notranslate">base->tp_itemsize > 0</code>: (Extending a variable-size class)<ul> <li><code class="docutils literal notranslate">spec->itemsize == 0</code>: The item size is inherited. (No change to status quo.)</li> <li><code class="docutils literal notranslate">spec->itemsize > 0</code>: The item size is set. (This is hard to use safely, but it’s CPython’s current behavior.)</li> </ul> </li> </ul> </li> <li><code class="docutils literal notranslate">spec->basicsize < 0</code>: (Extending the basicsize)<ul> <li><code class="docutils literal notranslate">base->tp_itemsize == 0</code>: (Extending a fixed-size class)<ul> <li><code class="docutils literal notranslate">spec->itemsize == 0</code>: The item size is set to 0.</li> <li><code class="docutils literal notranslate">spec->itemsize > 0</code>: Fail. (We’d need to add an <code class="docutils literal notranslate">ob_size</code>, which is only possible for trivial types – and the trivial layout must be known.)</li> </ul> </li> <li><code class="docutils literal notranslate">base->tp_itemsize > 0</code>: (Extending a variable-size class)<ul> <li><code class="docutils literal notranslate">spec->itemsize == 0</code>: (Inheriting the itemsize)<ul> <li><code class="docutils literal notranslate">Py_TPFLAGS_ITEMS_AT_END</code> used: itemsize is inherited.</li> <li><code class="docutils literal notranslate">Py_TPFLAGS_ITEMS_AT_END</code> not used: Fail. (Possible conflict.)</li> </ul> </li> <li><code class="docutils literal notranslate">spec->itemsize > 0</code>: Fail. (Changing/extending the item size can’t be done safely.)</li> </ul> </li> </ul> </li> </ul> Setting <code class="docutils literal notranslate">spec->itemsize < 0</code> is always an error. This PEP does not propose any mechanism to extend <code class="docutils literal notranslate">tp->itemsize</code> rather than just inherit it. </section> </section> <section id="relative-member-offsets"> <h3><a class="toc-backref" href="#relative-member-offsets" role="doc-backlink">Relative member offsets</a></h3> One more piece of the puzzle is <code class="docutils literal notranslate">PyMemberDef.offset</code>. Extensions that use a subclass-specific <code class="docutils literal notranslate">struct</code> (<code class="docutils literal notranslate">SubListState</code> above) will get a way to specify “relative” offsets (offsets based from this <code class="docutils literal notranslate">struct</code>) rather than “absolute” ones (based off the <code class="docutils literal notranslate">PyObject</code> struct). One way to do it would be to automatically assume “relative” offsets when creating a class using the new API. However, this implicit assumption would be too surprising. To be more explicit, this PEP proposes a new flag for “relative” offsets. At least initially, this flag will serve only as a check against misuse (and a hint for reviewers). It must be present if used with the new API, and must not be used otherwise. </section> </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> In the code blocks below, only function headers are part of the specification. Other code (the size/offset calculations) are details of the initial CPython implementation, and subject to change. <section id="relative-basicsize"> <h3><a class="toc-backref" href="#relative-basicsize" role="doc-backlink">Relative <code class="docutils literal notranslate">basicsize</code></a></h3> The <code class="docutils literal notranslate">basicsize</code> member of <code class="docutils literal notranslate">PyType_Spec</code> will be allowed to be zero or negative. In that case, its absolute value will specify how much extra storage space instances of the new class require, in addition to the basicsize of the base class. That is, the basicsize of the resulting class will be: <div class="highlight-c notranslate"><div class="highlight"><pre>type->tp_basicsize = _align(base->tp_basicsize) + _align(-spec->basicsize); </pre></div> </div> where <code class="docutils literal notranslate">_align</code> rounds up to a multiple of <code class="docutils literal notranslate">alignof(max_align_t)</code>. When <code class="docutils literal notranslate">spec->basicsize</code> is zero, basicsize will be inherited directly instead, i.e. set to <code class="docutils literal notranslate">base->tp_basicsize</code> without aligning. (This already works; explicit tests and documentation will be added.) On an instance, the memory area specific to a subclass – that is, the “extra space” that subclass reserves in addition its base – will be available through a new function, <code class="docutils literal notranslate">PyObject_GetTypeData</code>. In CPython, this function will be defined as: <div class="highlight-c notranslate"><div class="highlight"><pre>void * PyObject_GetTypeData(PyObject *obj, PyTypeObject *cls) { return (char *)obj + _align(cls->tp_base->tp_basicsize); } </pre></div> </div> Another function will be added to retrieve the size of this memory area: <div class="highlight-c notranslate"><div class="highlight"><pre>Py_ssize_t PyType_GetTypeDataSize(PyTypeObject *cls) { return cls->tp_basicsize - _align(cls->tp_base->tp_basicsize); } </pre></div> </div> The result may be higher than requested by <code class="docutils literal notranslate">-basicsize</code>. It is safe to use all of it (e.g. with <code class="docutils literal notranslate">memset</code>). The new <code class="docutils literal notranslate">*Get*</code> functions come with an important caveat, which will be pointed out in documentation: They may only be used for classes created using negative <code class="docutils literal notranslate">PyType_Spec.basicsize</code>. For other classes, their behavior is undefined. (Note that this allows the above code to assume <code class="docutils literal notranslate">cls->tp_base</code> is not <code class="docutils literal notranslate">NULL</code>.) </section> <section id="inheriting-itemsize"> <h3><a class="toc-backref" href="#inheriting-itemsize" role="doc-backlink">Inheriting <code class="docutils literal notranslate">itemsize</code></a></h3> When <code class="docutils literal notranslate">spec->itemsize</code> is zero, <code class="docutils literal notranslate">tp_itemsize</code> will be inherited from the base. (This already works; explicit tests and documentation will be added.) A new type flag, <code class="docutils literal notranslate">Py_TPFLAGS_ITEMS_AT_END</code>, will be added. This flag can only be set on types with non-zero <code class="docutils literal notranslate">tp_itemsize</code>. It indicates that the variable-sized portion of an instance is stored at the end of the instance’s memory. The default metatype (<code class="docutils literal notranslate">PyType_Type</code>) will set this flag. A new function, <code class="docutils literal notranslate">PyObject_GetItemData</code>, will be added to access the memory reserved for variable-sized content of types with the new flag. In CPython it will be defined as: <div class="highlight-c notranslate"><div class="highlight"><pre>void * PyObject_GetItemData(PyObject *obj) { if (!PyType_HasFeature(Py_TYPE(obj), Py_TPFLAGS_ITEMS_AT_END) { <fail with TypeError> } return (char *)obj + Py_TYPE(obj)->tp_basicsize; } </pre></div> </div> This function will initially not be added to the Limited API. Extending a class with positive <code class="docutils literal notranslate">base->itemsize</code> using negative <code class="docutils literal notranslate">spec->basicsize</code> will fail unless <code class="docutils literal notranslate">Py_TPFLAGS_ITEMS_AT_END</code> is set, either on the base or in <code class="docutils literal notranslate">spec->flags</code>. (See <a class="reference internal" href="#extending-variable-size-objects">Extending variable-size objects</a> for a full explanation.) Extending a class with positive <code class="docutils literal notranslate">spec->itemsize</code> using negative <code class="docutils literal notranslate">spec->basicsize</code> will fail. </section> <section id="id2"> <h3><a class="toc-backref" href="#id2" role="doc-backlink">Relative member offsets</a></h3> In types defined using negative <code class="docutils literal notranslate">PyType_Spec.basicsize</code>, the offsets of members defined via <code class="docutils literal notranslate">Py_tp_members</code> must be relative to the extra subclass data, rather than the full <code class="docutils literal notranslate">PyObject</code> struct. This will be indicated by a new flag in <code class="docutils literal notranslate">PyMemberDef.flags</code>: <code class="docutils literal notranslate">Py_RELATIVE_OFFSET</code>. In the initial implementation, the new flag will be redundant. It only serves to make the offset’s changed meaning clear, and to help avoid mistakes. It will be an error to not use <code class="docutils literal notranslate">Py_RELATIVE_OFFSET</code> with negative <code class="docutils literal notranslate">basicsize</code>, and it will be an error to use it in any other context (i.e. direct or indirect calls to <code class="docutils literal notranslate">PyDescr_NewMember</code>, <code class="docutils literal notranslate">PyMember_GetOne</code>, <code class="docutils literal notranslate">PyMember_SetOne</code>). CPython will adjust the offset and clear the <code class="docutils literal notranslate">Py_RELATIVE_OFFSET</code> flag when initializing a type. This means that: <ul class="simple"> <li>the created type’s <code class="docutils literal notranslate">tp_members</code> will not match the input definition’s <code class="docutils literal notranslate">Py_tp_members</code> slot, and</li> <li>any code that reads <code class="docutils literal notranslate">tp_members</code> will not need to handle the flag.</li> </ul> </section> </section> <section id="list-of-new-api"> <h2><a class="toc-backref" href="#list-of-new-api" role="doc-backlink">List of new API</a></h2> The following new functions/values are proposed. These will be added to the Limited API/Stable ABI: <ul class="simple"> <li><code class="docutils literal notranslate">void * PyObject_GetTypeData(PyObject *obj, PyTypeObject *cls)</code></li> <li><code class="docutils literal notranslate">Py_ssize_t PyType_GetTypeDataSize(PyTypeObject *cls)</code></li> <li><code class="docutils literal notranslate">Py_TPFLAGS_ITEMS_AT_END</code> flag for <code class="docutils literal notranslate">PyTypeObject.tp_flags</code></li> <li><code class="docutils literal notranslate">Py_RELATIVE_OFFSET</code> flag for <code class="docutils literal notranslate">PyMemberDef.flags</code></li> </ul> These will be added to the public C API only: <ul class="simple"> <li><code class="docutils literal notranslate">void *PyObject_GetItemData(PyObject *obj)</code></li> </ul> </section> <section id="backwards-compatibility"> <h2><a class="toc-backref" href="#backwards-compatibility" role="doc-backlink">Backwards Compatibility</a></h2> No backwards compatibility concerns are known. </section> <section id="assumptions"> <h2><a class="toc-backref" href="#assumptions" role="doc-backlink">Assumptions</a></h2> The implementation assumes that an instance’s memory between <code class="docutils literal notranslate">type->tp_base->tp_basicsize</code> and <code class="docutils literal notranslate">type->tp_basicsize</code> offsets “belongs” to <code class="docutils literal notranslate">type</code> (except variable-length types). This is not documented explicitly, but CPython up to version 3.11 relied on it when adding <code class="docutils literal notranslate">__dict__</code> to subclasses, so it should be safe. </section> <section id="security-implications"> <h2><a class="toc-backref" href="#security-implications" role="doc-backlink">Security Implications</a></h2> None known. </section> <section id="endorsements"> <h2><a class="toc-backref" href="#endorsements" role="doc-backlink">Endorsements</a></h2> The author of <code class="docutils literal notranslate">pybind11</code> originally requested solving the issue (see point 2 in <a class="reference external" href="https://discuss.python.org/t/15993">this list</a>), and <a class="reference external" href="https://discuss.python.org/t/19743/14">has been verifying the implementation</a>. Florian from the HPy project <a class="reference external" href="https://discuss.python.org/t/19743/3">said</a> that the API looks good in general. (See <a class="reference internal" href="#alignment-performance">below</a> for a possible solution to performance concerns.) </section> <section id="how-to-teach-this"> <h2><a class="toc-backref" href="#how-to-teach-this" role="doc-backlink">How to Teach This</a></h2> The initial implementation will include reference documentation and a What’s New entry, which should be enough for the target audience – authors of C extension libraries. </section> <section id="reference-implementation"> <h2><a class="toc-backref" href="#reference-implementation" role="doc-backlink">Reference Implementation</a></h2> A reference implementation is in the <a class="reference external" href="https://github.com/python/cpython/compare/main...encukou:cpython:extend-opaque">extend-opaque branch</a> in the <code class="docutils literal notranslate">encukou/cpython</code> GitHub repo. </section> <section id="possible-future-enhancements"> <h2><a class="toc-backref" href="#possible-future-enhancements" role="doc-backlink">Possible Future Enhancements</a></h2> <section id="alignment-performance"> <h3><a class="toc-backref" href="#alignment-performance" role="doc-backlink">Alignment & Performance</a></h3> The proposed implementation may waste some space if instance structs need smaller alignment than <code class="docutils literal notranslate">alignof(max_align_t)</code>. Also, dealing with alignment makes the calculation slower than it could be if we could rely on <code class="docutils literal notranslate">base->tp_basicsize</code> being properly aligned for the subtype. In other words, the proposed implementation focuses on safety and ease of use, and trades space and time for it. If it turns out that this is a problem, the implementation can be adjusted without breaking the API: <ul class="simple"> <li>The offset to the type-specific buffer can be stored, so <code class="docutils literal notranslate">PyObject_GetTypeData</code> effectively becomes <code class="docutils literal notranslate">(char *)obj + cls->ht_typedataoffset</code>, possibly speeding things up at the cost of an extra pointer in the class.</li> <li>Then, a new <code class="docutils literal notranslate">PyType_Slot</code> can specify the desired alignment, to reduce space requirements for instances.</li> </ul> </section> <section id="other-layouts-for-variable-size-types"> <h3><a class="toc-backref" href="#other-layouts-for-variable-size-types" role="doc-backlink">Other layouts for variable-size types</a></h3> A flag like <code class="docutils literal notranslate">Py_TPFLAGS_ITEMS_AT_END</code> could be added to signal the “tuple-like” layout described in <a class="reference internal" href="#extending-variable-size-objects">Extending variable-size objects</a>, and all mechanisms this PEP proposes could be adapted to support it. Other layouts could be added as well. However, it seems there’d be very little practical benefit, so it’s just a theoretical possibility. </section> </section> <section id="rejected-ideas"> <h2><a class="toc-backref" href="#rejected-ideas" role="doc-backlink">Rejected Ideas</a></h2> Instead of a negative <code class="docutils literal notranslate">spec->basicsize</code>, a new <code class="docutils literal notranslate">PyType_Spec</code> flag could’ve been added. The effect would be the same to any existing code accessing these internals without up to date knowledge of the change as the meaning of the field value is changing in this situation. </section> <section id="footnotes"> <h2><a class="toc-backref" href="#footnotes" role="doc-backlink">Footnotes</a></h2> <aside class="footnote-list brackets"> <aside class="footnote brackets" id="f1" role="doc-footnote"> <dt class="label" id="f1">[<a href="#id1">1</a>]</dt> <dd>This PEP does not make it “safe” to subclass NumPy arrays specifically. NumPy publishes <a class="reference external" href="https://numpy.org/doc/1.23/user/basics.subclassing.html">an extensive list of caveats</a> for subclassing its arrays from Python, and extending in C might need a similar list.</aside> </aside> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0697.rst">https://github.com/python/peps/blob/main/peps/pep-0697.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0697.rst">2024-08-20 10:29:32 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#extending-opaque-types">Extending opaque types</a></li> <li><a class="reference internal" href="#extending-variable-size-objects">Extending variable-size objects</a><ul> <li><a class="reference internal" href="#variable-size-layouts">Variable-size layouts</a></li> <li><a class="reference internal" href="#extending-classes-with-the-pyheaptypeobject-like-layout">Extending classes with the <code class="docutils literal notranslate">PyHeapTypeObject</code>-like layout</a></li> <li><a class="reference internal" href="#big-picture">Big picture</a></li> </ul> </li> <li><a class="reference internal" href="#relative-member-offsets">Relative member offsets</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#relative-basicsize">Relative <code class="docutils literal notranslate">basicsize</code></a></li> <li><a class="reference internal" href="#inheriting-itemsize">Inheriting <code class="docutils literal notranslate">itemsize</code></a></li> <li><a class="reference internal" href="#id2">Relative member offsets</a></li> </ul> </li> <li><a class="reference internal" href="#list-of-new-api">List of new API</a></li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a></li> <li><a class="reference internal" href="#assumptions">Assumptions</a></li> <li><a class="reference internal" href="#security-implications">Security Implications</a></li> <li><a class="reference internal" href="#endorsements">Endorsements</a></li> <li><a class="reference internal" href="#how-to-teach-this">How to Teach This</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#possible-future-enhancements">Possible Future Enhancements</a><ul> <li><a class="reference internal" href="#alignment-performance">Alignment & Performance</a></li> <li><a class="reference internal" href="#other-layouts-for-variable-size-types">Other layouts for variable-size types</a></li> </ul> </li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a></li> <li><a class="reference internal" href="#footnotes">Footnotes</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0697.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>