CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 674 – Disallow using macros as l-values | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0674/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 674 – Disallow using macros as l-values | peps.python.org'> <meta property="og:description" content="Disallow using macros as l-values. For example, Py_TYPE(obj) = new_type now fails with a compiler error."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0674/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="Disallow using macros as l-values. For example, Py_TYPE(obj) = new_type now fails with a compiler error."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 674</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> <span class="visually-hidden">Toggle light / dark / auto colour theme</span> </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 674 – Disallow using macros as l-values</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author<span class="colon">:</span></dt> <dd class="field-odd">Victor Stinner <vstinner at python.org></dd> <dt class="field-even">Status<span class="colon">:</span></dt> <dd class="field-even"><abbr title="Inactive draft that may be taken up again at a later time">Deferred</abbr></dd> <dt class="field-odd">Type<span class="colon">:</span></dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created<span class="colon">:</span></dt> <dd class="field-even">30-Nov-2021</dd> <dt class="field-odd">Python-Version<span class="colon">:</span></dt> <dd class="field-odd">3.12</dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#pep-deferral">PEP Deferral</a></li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#using-a-macro-as-a-an-l-value">Using a macro as a an l-value</a></li> <li><a class="reference internal" href="#cpython-nogil-fork">CPython nogil fork</a></li> <li><a class="reference internal" href="#hpy-project">HPy project</a></li> <li><a class="reference internal" href="#graalvm-python">GraalVM Python</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#disallow-using-macros-as-l-values">Disallow using macros as l-values</a><ul> <li><a class="reference internal" href="#pyobject-and-pyvarobject-macros">PyObject and PyVarObject macros</a></li> <li><a class="reference internal" href="#get-macros">GET macros</a></li> <li><a class="reference internal" href="#as-macros">AS macros</a></li> <li><a class="reference internal" href="#pyunicode-macros">PyUnicode macros</a></li> <li><a class="reference internal" href="#pydatetime-get-macros">PyDateTime GET macros</a></li> </ul> </li> <li><a class="reference internal" href="#port-c-extensions-to-python-3-11">Port C extensions to Python 3.11</a></li> <li><a class="reference internal" href="#pytuple-get-item-and-pylist-get-item-are-left-unchanged">PyTuple_GET_ITEM() and PyList_GET_ITEM() are left unchanged</a></li> <li><a class="reference internal" href="#pydescr-name-and-pydescr-type-are-left-unchanged">PyDescr_NAME() and PyDescr_TYPE() are left unchanged</a></li> </ul> </li> <li><a class="reference internal" href="#implementation">Implementation</a><ul> <li><a class="reference internal" href="#py-type-and-py-size-macros">Py_TYPE() and Py_SIZE() macros</a></li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a><ul> <li><a class="reference internal" href="#statistics">Statistics</a></li> <li><a class="reference internal" href="#top-5000-pypi">Top 5000 PyPI</a></li> <li><a class="reference internal" href="#other-affected-projects">Other affected projects</a></li> </ul> </li> <li><a class="reference internal" href="#relationship-with-the-hpy-project">Relationship with the HPy project</a><ul> <li><a class="reference internal" href="#the-hpy-project">The HPy project</a></li> <li><a class="reference internal" href="#the-c-api-is-here-is-stay-for-a-few-more-years">The C API is here is stay for a few more years</a></li> </ul> </li> <li><a class="reference internal" href="#rejected-idea-leave-the-macros-as-they-are">Rejected Idea: Leave the macros as they are</a></li> <li><a class="reference internal" href="#macros-already-modified">Macros already modified</a></li> <li><a class="reference internal" href="#post-history">Post History</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#version-history">Version History</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> <p>Disallow using macros as l-values. For example, <code class="docutils literal notranslate"><span class="pre">Py_TYPE(obj)</span> <span class="pre">=</span> <span class="pre">new_type</span></code> now fails with a compiler error.</p> <p>In practice, the majority of affected projects only have to make two changes:</p> <ul class="simple"> <li>Replace <code class="docutils literal notranslate"><span class="pre">Py_TYPE(obj)</span> <span class="pre">=</span> <span class="pre">new_type</span></code> with <code class="docutils literal notranslate"><span class="pre">Py_SET_TYPE(obj,</span> <span class="pre">new_type)</span></code>.</li> <li>Replace <code class="docutils literal notranslate"><span class="pre">Py_SIZE(obj)</span> <span class="pre">=</span> <span class="pre">new_size</span></code> with <code class="docutils literal notranslate"><span class="pre">Py_SET_SIZE(obj,</span> <span class="pre">new_size)</span></code>.</li> </ul> </section> <section id="pep-deferral"> <h2><a class="toc-backref" href="#pep-deferral" role="doc-backlink">PEP Deferral</a></h2> <p>See <a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/message/CV6KWDRHV5WP6TIDK3Z46PW7HNSHYOWG/">SC reply to PEP 674 – Disallow using macros as l-values</a> (February 2022).</p> </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> <section id="using-a-macro-as-a-an-l-value"> <h3><a class="toc-backref" href="#using-a-macro-as-a-an-l-value" role="doc-backlink">Using a macro as a an l-value</a></h3> <p>In the Python C API, some functions are implemented as macro because writing a macro is simpler than writing a regular function. If a macro exposes directly a structure member, it is technically possible to use this macro to not only get the structure member but also set it.</p> <p>Example with the Python 3.10 <code class="docutils literal notranslate"><span class="pre">Py_TYPE()</span></code> macro:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1">#define Py_TYPE(ob) (((PyObject *)(ob))->ob_type)</span> </pre></div> </div> <p>This macro can be used as a <strong>r-value</strong> to <strong>get</strong> an object type:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="nb">type</span> <span class="o">=</span> <span class="n">Py_TYPE</span><span class="p">(</span><span class="nb">object</span><span class="p">);</span> </pre></div> </div> <p>It can also be used as an <strong>l-value</strong> to <strong>set</strong> an object type:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Py_TYPE</span><span class="p">(</span><span class="nb">object</span><span class="p">)</span> <span class="o">=</span> <span class="n">new_type</span><span class="p">;</span> </pre></div> </div> <p>It is also possible to set an object reference count and an object size using <code class="docutils literal notranslate"><span class="pre">Py_REFCNT()</span></code> and <code class="docutils literal notranslate"><span class="pre">Py_SIZE()</span></code> macros.</p> <p>Setting directly an object attribute relies on the current exact CPython implementation. Implementing this feature in other Python implementations can make their C API implementation less efficient.</p> </section> <section id="cpython-nogil-fork"> <h3><a class="toc-backref" href="#cpython-nogil-fork" role="doc-backlink">CPython nogil fork</a></h3> <p>Sam Gross forked Python 3.9 to remove the GIL: the <a class="reference external" href="https://github.com/colesbury/nogil/">nogil branch</a>. This fork has no <code class="docutils literal notranslate"><span class="pre">PyObject.ob_refcnt</span></code> member, but a more elaborated implementation for reference counting, and so the <code class="docutils literal notranslate"><span class="pre">Py_REFCNT(obj)</span> <span class="pre">=</span> <span class="pre">new_refcnt;</span></code> code fails with a compiler error.</p> <p>Merging the nogil fork into the upstream CPython main branch requires first to fix this C API compatibility issue. It is a concrete example of a Python optimization blocked indirectly by the C API.</p> <p>This issue was already fixed in Python 3.10: the <code class="docutils literal notranslate"><span class="pre">Py_REFCNT()</span></code> macro has been already modified to disallow using it as an l-value.</p> <p>These statements are endorsed by Sam Gross (nogil developer).</p> </section> <section id="hpy-project"> <h3><a class="toc-backref" href="#hpy-project" role="doc-backlink">HPy project</a></h3> <p>The <a class="reference external" href="https://hpyproject.org/">HPy project</a> is a brand new C API for Python using only handles and function calls: handles are opaque, structure members cannot be accessed directly, and pointers cannot be dereferenced.</p> <p>Searching and replacing <code class="docutils literal notranslate"><span class="pre">Py_SET_SIZE()</span></code> is easier and safer than searching and replacing some strange macro uses of <code class="docutils literal notranslate"><span class="pre">Py_SIZE()</span></code>. <code class="docutils literal notranslate"><span class="pre">Py_SIZE()</span></code> can be semi-mechanically replaced by <code class="docutils literal notranslate"><span class="pre">HPy_Length()</span></code>, whereas seeing <code class="docutils literal notranslate"><span class="pre">Py_SET_SIZE()</span></code> would immediately make clear that the code needs bigger changes in order to be ported to HPy (for example by using <code class="docutils literal notranslate"><span class="pre">HPyTupleBuilder</span></code> or <code class="docutils literal notranslate"><span class="pre">HPyListBuilder</span></code>).</p> <p>The fewer internal details exposed via macros, the easier it will be for HPy to provide direct equivalents. Any macro that references “non-public” interfaces effectively exposes those interfaces publicly.</p> <p>These statements are endorsed by Antonio Cuni (HPy developer).</p> </section> <section id="graalvm-python"> <h3><a class="toc-backref" href="#graalvm-python" role="doc-backlink">GraalVM Python</a></h3> <p>In GraalVM, when a Python object is accessed by the Python C API, the C API emulation layer has to wrap the GraalVM objects into wrappers that expose the internal structure of the CPython structures (PyObject, PyLongObject, PyTypeObject, etc). This is because when the C code accesses it directly or via macros, all GraalVM can intercept is a read at the struct offset, which has to be mapped back to the representation in GraalVM. The smaller the “effective” number of exposed struct members (by replacing macros with functions), the simpler GraalVM wrappers can be.</p> <p>This PEP alone is not enough to get rid of the wrappers in GraalVM, but it is a step towards this long term goal. GraalVM already supports HPy which is a better solution in the long term.</p> <p>These statements are endorsed by Tim Felgentreff (GraalVM Python developer).</p> </section> </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> <section id="disallow-using-macros-as-l-values"> <h3><a class="toc-backref" href="#disallow-using-macros-as-l-values" role="doc-backlink">Disallow using macros as l-values</a></h3> <p>The following 65 macros are modified to disallow using them as l-values.</p> <section id="pyobject-and-pyvarobject-macros"> <h4><a class="toc-backref" href="#pyobject-and-pyvarobject-macros" role="doc-backlink">PyObject and PyVarObject macros</a></h4> <ul class="simple"> <li><code class="docutils literal notranslate"><span class="pre">Py_TYPE()</span></code>: <code class="docutils literal notranslate"><span class="pre">Py_SET_TYPE()</span></code> must be used instead</li> <li><code class="docutils literal notranslate"><span class="pre">Py_SIZE()</span></code>: <code class="docutils literal notranslate"><span class="pre">Py_SET_SIZE()</span></code> must be used instead</li> </ul> </section> <section id="get-macros"> <h4><a class="toc-backref" href="#get-macros" role="doc-backlink">GET macros</a></h4> <ul class="simple"> <li><code class="docutils literal notranslate"><span class="pre">PyByteArray_GET_SIZE()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyBytes_GET_SIZE()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyCFunction_GET_CLASS()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyCFunction_GET_FLAGS()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyCFunction_GET_FUNCTION()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyCFunction_GET_SELF()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyCell_GET()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyCode_GetNumFree()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyDict_GET_SIZE()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyFunction_GET_ANNOTATIONS()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyFunction_GET_CLOSURE()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyFunction_GET_CODE()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyFunction_GET_DEFAULTS()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyFunction_GET_GLOBALS()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyFunction_GET_KW_DEFAULTS()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyFunction_GET_MODULE()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyHeapType_GET_MEMBERS()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyInstanceMethod_GET_FUNCTION()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyList_GET_SIZE()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyMemoryView_GET_BASE()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyMemoryView_GET_BUFFER()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyMethod_GET_FUNCTION()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyMethod_GET_SELF()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PySet_GET_SIZE()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyTuple_GET_SIZE()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyUnicode_GET_DATA_SIZE()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyUnicode_GET_LENGTH()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyUnicode_GET_LENGTH()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyUnicode_GET_SIZE()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyWeakref_GET_OBJECT()</span></code></li> </ul> </section> <section id="as-macros"> <h4><a class="toc-backref" href="#as-macros" role="doc-backlink">AS macros</a></h4> <ul class="simple"> <li><code class="docutils literal notranslate"><span class="pre">PyByteArray_AS_STRING()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyBytes_AS_STRING()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyFloat_AS_DOUBLE()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyUnicode_AS_DATA()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyUnicode_AS_UNICODE()</span></code></li> </ul> </section> <section id="pyunicode-macros"> <h4><a class="toc-backref" href="#pyunicode-macros" role="doc-backlink">PyUnicode macros</a></h4> <ul class="simple"> <li><code class="docutils literal notranslate"><span class="pre">PyUnicode_1BYTE_DATA()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyUnicode_2BYTE_DATA()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyUnicode_4BYTE_DATA()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyUnicode_DATA()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyUnicode_IS_ASCII()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyUnicode_IS_COMPACT()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyUnicode_IS_READY()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyUnicode_KIND()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyUnicode_READ()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyUnicode_READ_CHAR()</span></code></li> </ul> </section> <section id="pydatetime-get-macros"> <h4><a class="toc-backref" href="#pydatetime-get-macros" role="doc-backlink">PyDateTime GET macros</a></h4> <ul class="simple"> <li><code class="docutils literal notranslate"><span class="pre">PyDateTime_DATE_GET_FOLD()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyDateTime_DATE_GET_HOUR()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyDateTime_DATE_GET_MICROSECOND()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyDateTime_DATE_GET_MINUTE()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyDateTime_DATE_GET_SECOND()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyDateTime_DATE_GET_TZINFO()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyDateTime_DELTA_GET_DAYS()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyDateTime_DELTA_GET_MICROSECONDS()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyDateTime_DELTA_GET_SECONDS()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyDateTime_GET_DAY()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyDateTime_GET_MONTH()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyDateTime_GET_YEAR()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyDateTime_TIME_GET_FOLD()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyDateTime_TIME_GET_HOUR()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyDateTime_TIME_GET_MICROSECOND()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyDateTime_TIME_GET_MINUTE()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyDateTime_TIME_GET_SECOND()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyDateTime_TIME_GET_TZINFO()</span></code></li> </ul> </section> </section> <section id="port-c-extensions-to-python-3-11"> <h3><a class="toc-backref" href="#port-c-extensions-to-python-3-11" role="doc-backlink">Port C extensions to Python 3.11</a></h3> <p>In practice, the majority of projects affected by these PEP only have to make two changes:</p> <ul class="simple"> <li>Replace <code class="docutils literal notranslate"><span class="pre">Py_TYPE(obj)</span> <span class="pre">=</span> <span class="pre">new_type</span></code> with <code class="docutils literal notranslate"><span class="pre">Py_SET_TYPE(obj,</span> <span class="pre">new_type)</span></code>.</li> <li>Replace <code class="docutils literal notranslate"><span class="pre">Py_SIZE(obj)</span> <span class="pre">=</span> <span class="pre">new_size</span></code> with <code class="docutils literal notranslate"><span class="pre">Py_SET_SIZE(obj,</span> <span class="pre">new_size)</span></code>.</li> </ul> <p>The <a class="reference external" href="https://github.com/pythoncapi/pythoncapi_compat">pythoncapi_compat project</a> can be used to update automatically C extensions: add Python 3.11 support without losing support with older Python versions. The project provides a header file which provides <code class="docutils literal notranslate"><span class="pre">Py_SET_REFCNT()</span></code>, <code class="docutils literal notranslate"><span class="pre">Py_SET_TYPE()</span></code> and <code class="docutils literal notranslate"><span class="pre">Py_SET_SIZE()</span></code> functions to Python 3.8 and older.</p> </section> <section id="pytuple-get-item-and-pylist-get-item-are-left-unchanged"> <h3><a class="toc-backref" href="#pytuple-get-item-and-pylist-get-item-are-left-unchanged" role="doc-backlink">PyTuple_GET_ITEM() and PyList_GET_ITEM() are left unchanged</a></h3> <p>The <code class="docutils literal notranslate"><span class="pre">PyTuple_GET_ITEM()</span></code> and <code class="docutils literal notranslate"><span class="pre">PyList_GET_ITEM()</span></code> macros are left unchanged.</p> <p>The code patterns <code class="docutils literal notranslate"><span class="pre">&PyTuple_GET_ITEM(tuple,</span> <span class="pre">0)</span></code> and <code class="docutils literal notranslate"><span class="pre">&PyList_GET_ITEM(list,</span> <span class="pre">0)</span></code> are still commonly used to get access to the inner <code class="docutils literal notranslate"><span class="pre">PyObject**</span></code> array.</p> <p>Changing these macros is out of the scope of this PEP.</p> </section> <section id="pydescr-name-and-pydescr-type-are-left-unchanged"> <h3><a class="toc-backref" href="#pydescr-name-and-pydescr-type-are-left-unchanged" role="doc-backlink">PyDescr_NAME() and PyDescr_TYPE() are left unchanged</a></h3> <p>The <code class="docutils literal notranslate"><span class="pre">PyDescr_NAME()</span></code> and <code class="docutils literal notranslate"><span class="pre">PyDescr_TYPE()</span></code> macros are left unchanged.</p> <p>These macros give access to <code class="docutils literal notranslate"><span class="pre">PyDescrObject.d_name</span></code> and <code class="docutils literal notranslate"><span class="pre">PyDescrObject.d_type</span></code> members. They can be used as l-values to set these members.</p> <p>The SWIG project uses these macros as l-values to set these members. It would be possible to modify SWIG to prevent setting <code class="docutils literal notranslate"><span class="pre">PyDescrObject</span></code> structure members directly, but it is not really worth it since the <code class="docutils literal notranslate"><span class="pre">PyDescrObject</span></code> structure is not performance critical and is unlikely to change soon.</p> <p>See the <a class="reference external" href="https://bugs.python.org/issue46538">bpo-46538</a> “[C API] Make the PyDescrObject structure opaque: PyDescr_NAME() and PyDescr_TYPE()” issue for more details.</p> </section> </section> <section id="implementation"> <h2><a class="toc-backref" href="#implementation" role="doc-backlink">Implementation</a></h2> <p>The implementation is tracked by <a class="reference external" href="https://bugs.python.org/issue45476">bpo-45476: [C API] PEP 674: Disallow using macros as l-values</a>.</p> <section id="py-type-and-py-size-macros"> <h3><a class="toc-backref" href="#py-type-and-py-size-macros" role="doc-backlink">Py_TYPE() and Py_SIZE() macros</a></h3> <p>In May 2020, the <code class="docutils literal notranslate"><span class="pre">Py_TYPE()</span></code> and <code class="docutils literal notranslate"><span class="pre">Py_SIZE()</span></code> macros have been modified to disallow using them as l-values (<a class="reference external" href="https://github.com/python/cpython/commit/ad3252bad905d41635bcbb4b76db30d570cf0087">Py_TYPE</a>, <a class="reference external" href="https://github.com/python/cpython/commit/fe2978b3b940fe2478335e3a2ca5ad22338cdf9c">Py_SIZE</a>).</p> <p>In November 2020, the change was <a class="reference external" href="https://github.com/python/cpython/commit/0e2ac21dd4960574e89561243763eabba685296a">reverted</a>, since it broke too many third party projects.</p> <p>In June 2021, once most third party projects were updated, a <a class="reference external" href="https://github.com/python/cpython/commit/f3fa63ec75fdbb4a08a10957a5c631bf0c4a5970">second attempt</a> was done, but had to be <a class="reference external" href="https://github.com/python/cpython/commit/6d518bb3a11f9b16098f45b21a13ebe8f537f045">reverted again</a> , since it broke test_exceptions on Windows.</p> <p>In September 2021, once <a class="reference external" href="https://github.com/python/cpython/commit/fb305092a5d7894b41f122c1a1117b3abf4c567e">test_exceptions has been fixed</a>, Py_TYPE() and Py_SIZE() were finally <a class="reference external" href="https://github.com/python/cpython/commit/f3fa63ec75fdbb4a08a10957a5c631bf0c4a5970">changed</a>.</p> <p>In November 2021, this backward incompatible change got a <a class="reference external" href="https://github.com/python/steering-council/issues/79#issuecomment-981153173">Steering Council exception</a>.</p> <p>In October 2022, Python 3.11 got released with Py_TYPE() and Py_SIZE() incompatible changes.</p> </section> </section> <section id="backwards-compatibility"> <h2><a class="toc-backref" href="#backwards-compatibility" role="doc-backlink">Backwards Compatibility</a></h2> <p>The proposed C API changes are backward incompatible on purpose.</p> <p>In practice, only <code class="docutils literal notranslate"><span class="pre">Py_TYPE()</span></code> and <code class="docutils literal notranslate"><span class="pre">Py_SIZE()</span></code> macros are used as l-values.</p> <p>This change does not follow the <a class="pep reference internal" href="../pep-0387/" title="PEP 387 – Backwards Compatibility Policy">PEP 387</a> deprecation process. There is no known way to emit a deprecation warning only when a macro is used as an l-value, but not when it’s used differently (ex: as a r-value).</p> <p>The following 4 macros are left unchanged to reduce the number of affected projects: <code class="docutils literal notranslate"><span class="pre">PyDescr_NAME()</span></code>, <code class="docutils literal notranslate"><span class="pre">PyDescr_TYPE()</span></code>, <code class="docutils literal notranslate"><span class="pre">PyList_GET_ITEM()</span></code> and <code class="docutils literal notranslate"><span class="pre">PyTuple_GET_ITEM()</span></code>.</p> <section id="statistics"> <h3><a class="toc-backref" href="#statistics" role="doc-backlink">Statistics</a></h3> <p>In total (projects on PyPI and not on PyPI), 34 projects are known to be affected by this PEP:</p> <ul class="simple"> <li>16 projects (47%) are already fixed</li> <li>18 projects (53%) are not fixed yet (pending fix or have to regenerate their Cython code)</li> </ul> <p>On September 1, 2022, the PEP affects 18 projects (0.4%) of the top 5000 PyPI projects:</p> <ul class="simple"> <li>15 projects (0.3%) have to regenerate their Cython code</li> <li>3 projects (0.1%) have a pending fix</li> </ul> </section> <section id="top-5000-pypi"> <h3><a class="toc-backref" href="#top-5000-pypi" role="doc-backlink">Top 5000 PyPI</a></h3> <p>Projects with a pending fix (3):</p> <ul class="simple"> <li>datatable (1.0.0): <a class="reference external" href="https://github.com/h2oai/datatable/commit/02f13114828ed4567e4410f5bac579895e20355a">fixed</a></li> <li>guppy3 (3.1.2): <a class="reference external" href="https://github.com/zhuyifei1999/guppy3/commit/4cb9fcb5d75327544a6875b6caabfdffb70a7e29">fixed</a></li> <li>scipy (1.9.3): need to update boost python</li> </ul> <p>Moreover, 15 projects have to regenerate their Cython code.</p> <p>Projects released with a fix (12):</p> <ul class="simple"> <li>bitarray (1.6.2): <a class="reference external" href="https://github.com/ilanschnell/bitarray/commit/a0cca9f2986ec796df74ca8f42aff56c4c7103ba">commit</a></li> <li>Cython (0.29.20): <a class="reference external" href="https://github.com/cython/cython/commit/d8e93b332fe7d15459433ea74cd29178c03186bd">commit</a></li> <li>immutables (0.15): <a class="reference external" href="https://github.com/MagicStack/immutables/commit/45105ecd8b56a4d88dbcb380fcb8ff4b9cc7b19c">commit</a></li> <li>mercurial (5.7): <a class="reference external" href="https://www.mercurial-scm.org/repo/hg/rev/e92ca942ddca">commit</a>, <a class="reference external" href="https://bz.mercurial-scm.org/show_bug.cgi?id=6451">bug report</a></li> <li>mypy (v0.930): <a class="reference external" href="https://github.com/python/mypy/commit/2b7e2df923f7e4a3a199915b3c8563f45bc69dfa">commit</a></li> <li>numpy (1.22.1): <a class="reference external" href="https://github.com/numpy/numpy/commit/a96b18e3d4d11be31a321999cda4b795ea9eccaa">commit</a>, <a class="reference external" href="https://github.com/numpy/numpy/commit/f1671076c80bd972421751f2d48186ee9ac808aa">commit 2</a></li> <li>pycurl (7.44.1): <a class="reference external" href="https://github.com/pycurl/pycurl/commit/e633f9a1ac4df5e249e78c218d5fbbd848219042">commit</a></li> <li>PyGObject (3.42.0)</li> <li>pyside2 (5.15.1): <a class="reference external" href="https://bugreports.qt.io/browse/PYSIDE-1436">bug report</a></li> <li>python-snappy (0.6.1): <a class="reference external" href="https://github.com/andrix/python-snappy/commit/1a539d71d5b1ceaf9a2291f21f686cf53a46d707">fixed</a></li> <li>recordclass (0.17.2): <a class="reference external" href="https://bitbucket.org/intellimath/recordclass/commits/d20d72fa3cdbdcf96c72941560041460adeecff1">fixed</a></li> <li>zstd (1.5.0.3): <a class="reference external" href="https://github.com/sergey-dryabzhinsky/python-zstd/commit/8aa6d7a4b250e1f0a4e27b4107c39dc516c87f96">commit</a></li> </ul> <p>There are also two backport projects which are affected by this PEP:</p> <ul class="simple"> <li>pickle5 (0.0.12): backport for Python <= 3.7</li> <li>pysha3 (1.0.2): backport for Python <= 3.5</li> </ul> <p>They must not be used and cannot be used on Python 3.11.</p> </section> <section id="other-affected-projects"> <h3><a class="toc-backref" href="#other-affected-projects" role="doc-backlink">Other affected projects</a></h3> <p>Other projects released with a fix (4):</p> <ul class="simple"> <li>boost (1.78.0): <a class="reference external" href="https://github.com/boostorg/python/commit/500194edb7833d0627ce7a2595fec49d0aae2484">commit</a></li> <li>breezy (3.2.1): <a class="reference external" href="https://bugs.launchpad.net/brz/+bug/1904868">bug report</a></li> <li>duplicity (0.8.18): <a class="reference external" href="https://git.launchpad.net/duplicity/commit/duplicity/_librsyncmodule.c?id=bbaae91b5ac6ef7e295968e508522884609fbf84">commit</a></li> <li>gobject-introspection (1.70.0): <a class="reference external" href="https://gitlab.gnome.org/GNOME/gobject-introspection/-/merge_requests/243">MR</a></li> </ul> </section> </section> <section id="relationship-with-the-hpy-project"> <h2><a class="toc-backref" href="#relationship-with-the-hpy-project" role="doc-backlink">Relationship with the HPy project</a></h2> <section id="the-hpy-project"> <h3><a class="toc-backref" href="#the-hpy-project" role="doc-backlink">The HPy project</a></h3> <p>The hope with the HPy project is to provide a C API that is close to the original API—to make porting easy—and have it perform as close to the existing API as possible. At the same time, HPy is sufficiently removed to be a good “C extension API” (as opposed to a stable subset of the CPython implementation API) that does not leak implementation details. To ensure this latter property, the HPy project tries to develop everything in parallel for CPython, PyPy, and GraalVM Python.</p> <p>HPy is still evolving very fast. Issues are still being solved while migrating NumPy, and work has begun on adding support for HPy to Cython. Work on pybind11 is starting soon. Tim Felgentreff believes by the time HPy has these users of the existing C API working, HPy should be in a state where it is generally useful and can be deemed stable enough that further development can follow a more stable process.</p> <p>In the long run the HPy project would like to become a promoted API to write Python C extensions.</p> <p>The HPy project is a good solution for the long term. It has the advantage of being developed outside Python and it doesn’t require any C API change.</p> </section> <section id="the-c-api-is-here-is-stay-for-a-few-more-years"> <h3><a class="toc-backref" href="#the-c-api-is-here-is-stay-for-a-few-more-years" role="doc-backlink">The C API is here is stay for a few more years</a></h3> <p>The first concern about HPy is that right now, HPy is not mature nor widely used, and CPython still has to continue supporting a large amount of C extensions which are not likely to be ported to HPy soon.</p> <p>The second concern is the inability to evolve CPython internals to implement new optimizations, and the inefficient implementation of the current C API in PyPy, GraalPython, etc. Sadly, HPy will only solve these problems when most C extensions will be fully ported to HPy: when it will become reasonable to consider dropping the “legacy” Python C API.</p> <p>While porting a C extension to HPy can be done incrementally on CPython, it requires to modify a lot of code and takes time. Porting most C extensions to HPy is expected to take a few years.</p> <p>This PEP proposes to make the C API “less bad” by fixing one problem which is clearily identified as causing practical issues: macros used as l-values. This PEP only requires updating a minority of C extensions, and usually only a few lines need to be changed in impacted extensions.</p> <p>For example, NumPy 1.22 is made of 307,300 lines of C code, and adapting NumPy to the this PEP only modified 11 lines (use Py_SET_TYPE and Py_SET_SIZE) and adding 4 lines (to define Py_SET_TYPE and Py_SET_SIZE for Python 3.8 and older). The beginnings of the NumPy port to HPy already required modifying more lines than that.</p> <p>Right now, it’s hard to bet which approach is the best: fixing the current C API, or focusing on HPy. It would be risky to only focus on HPy.</p> </section> </section> <section id="rejected-idea-leave-the-macros-as-they-are"> <h2><a class="toc-backref" href="#rejected-idea-leave-the-macros-as-they-are" role="doc-backlink">Rejected Idea: Leave the macros as they are</a></h2> <p>The documentation of each function can discourage developers to use macros to modify Python objects.</p> <p>If these is a need to make an assignment, a setter function can be added and the macro documentation can require to use the setter function. For example, a <code class="docutils literal notranslate"><span class="pre">Py_SET_TYPE()</span></code> function has been added to Python 3.9 and the <code class="docutils literal notranslate"><span class="pre">Py_TYPE()</span></code> documentation now requires to use the <code class="docutils literal notranslate"><span class="pre">Py_SET_TYPE()</span></code> function to set an object type.</p> <p>If developers use macros as an l-value, it’s their responsibility when their code breaks, not Python’s responsibility. We are operating under the consenting adults principle: we expect users of the Python C API to use it as documented and expect them to take care of the fallout, if things break when they don’t.</p> <p>This idea was rejected because only few developers read the documentation, and only a minority is tracking changes of the Python C API documentation. The majority of developers are only using CPython and so are not aware of compatibility issues with other Python implementations.</p> <p>Moreover, continuing to allow using macros as an l-value does not help the HPy project, and leaves the burden of emulating them on GraalVM’s Python implementation.</p> </section> <section id="macros-already-modified"> <h2><a class="toc-backref" href="#macros-already-modified" role="doc-backlink">Macros already modified</a></h2> <p>The following C API macros have already been modified to disallow using them as l-value:</p> <ul class="simple"> <li><code class="docutils literal notranslate"><span class="pre">PyCell_SET()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyList_SET_ITEM()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">PyTuple_SET_ITEM()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">Py_REFCNT()</span></code> (Python 3.10): <code class="docutils literal notranslate"><span class="pre">Py_SET_REFCNT()</span></code> must be used</li> <li><code class="docutils literal notranslate"><span class="pre">_PyGCHead_SET_FINALIZED()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">_PyGCHead_SET_NEXT()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">asdl_seq_GET()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">asdl_seq_GET_UNTYPED()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">asdl_seq_LEN()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">asdl_seq_SET()</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">asdl_seq_SET_UNTYPED()</span></code></li> </ul> <p>For example, <code class="docutils literal notranslate"><span class="pre">PyList_SET_ITEM(list,</span> <span class="pre">0,</span> <span class="pre">item)</span> <span class="pre"><</span> <span class="pre">0</span></code> now fails with a compiler error as expected.</p> </section> <section id="post-history"> <h2><a class="toc-backref" href="#post-history" role="doc-backlink">Post History</a></h2> <ul class="simple"> <li><a class="reference external" href="https://discuss.python.org/t/pep-674-disallow-using-macros-as-l-values-and-python-3-11/18297">PEP 674 “Disallow using macros as l-values” and Python 3.11</a> (August 18, 2022)</li> <li><a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/CV6KWDRHV5WP6TIDK3Z46PW7HNSHYOWG/">SC reply to PEP 674 – Disallow using macros as l-values</a> (February 22, 2022)</li> <li><a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/J7SXC2YQGP37UYIEULISLUTKW5FHN3Z7/">PEP 674: Disallow using macros as l-value (version 2)</a> (Jan 18, 2022)</li> <li><a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/KPIJPPJ6XVNOLGZQD2PFGMT7LBJMTTCO/">PEP 674: Disallow using macros as l-value</a> (Nov 30, 2021)</li> </ul> </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <ul class="simple"> <li><a class="reference external" href="https://vstinner.github.io/c-api-abstract-pyobject.html">Python C API: Add functions to access PyObject</a> (October 2021) article by Victor Stinner</li> <li><a class="reference external" href="https://mail.python.org/archives/list/capi-sig@python.org/thread/WGRLTHTHC32DQTACPPX36TPR2GLJAFRB/">[capi-sig] Py_TYPE() and Py_SIZE() become static inline functions</a> (September 2021)</li> <li><a class="reference external" href="https://bugs.python.org/issue39573">[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type</a> (February 2020)</li> <li><a class="reference external" href="https://bugs.python.org/issue30459">bpo-30459: PyList_SET_ITEM could be safer</a> (May 2017)</li> </ul> </section> <section id="version-history"> <h2><a class="toc-backref" href="#version-history" role="doc-backlink">Version History</a></h2> <ul class="simple"> <li>Version 3: No longer change PyDescr_TYPE() and PyDescr_NAME() macros</li> <li>Version 2: Add “Relationship with the HPy project” section, remove the PyPy section</li> <li>Version 1: First public version</li> </ul> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> <p>This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.</p> </section> </section> <hr class="docutils" /> <p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0674.rst">https://github.com/python/peps/blob/main/peps/pep-0674.rst</a></p> <p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0674.rst">2023-09-09 17:39:29 GMT</a></p> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#pep-deferral">PEP Deferral</a></li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#using-a-macro-as-a-an-l-value">Using a macro as a an l-value</a></li> <li><a class="reference internal" href="#cpython-nogil-fork">CPython nogil fork</a></li> <li><a class="reference internal" href="#hpy-project">HPy project</a></li> <li><a class="reference internal" href="#graalvm-python">GraalVM Python</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#disallow-using-macros-as-l-values">Disallow using macros as l-values</a><ul> <li><a class="reference internal" href="#pyobject-and-pyvarobject-macros">PyObject and PyVarObject macros</a></li> <li><a class="reference internal" href="#get-macros">GET macros</a></li> <li><a class="reference internal" href="#as-macros">AS macros</a></li> <li><a class="reference internal" href="#pyunicode-macros">PyUnicode macros</a></li> <li><a class="reference internal" href="#pydatetime-get-macros">PyDateTime GET macros</a></li> </ul> </li> <li><a class="reference internal" href="#port-c-extensions-to-python-3-11">Port C extensions to Python 3.11</a></li> <li><a class="reference internal" href="#pytuple-get-item-and-pylist-get-item-are-left-unchanged">PyTuple_GET_ITEM() and PyList_GET_ITEM() are left unchanged</a></li> <li><a class="reference internal" href="#pydescr-name-and-pydescr-type-are-left-unchanged">PyDescr_NAME() and PyDescr_TYPE() are left unchanged</a></li> </ul> </li> <li><a class="reference internal" href="#implementation">Implementation</a><ul> <li><a class="reference internal" href="#py-type-and-py-size-macros">Py_TYPE() and Py_SIZE() macros</a></li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a><ul> <li><a class="reference internal" href="#statistics">Statistics</a></li> <li><a class="reference internal" href="#top-5000-pypi">Top 5000 PyPI</a></li> <li><a class="reference internal" href="#other-affected-projects">Other affected projects</a></li> </ul> </li> <li><a class="reference internal" href="#relationship-with-the-hpy-project">Relationship with the HPy project</a><ul> <li><a class="reference internal" href="#the-hpy-project">The HPy project</a></li> <li><a class="reference internal" href="#the-c-api-is-here-is-stay-for-a-few-more-years">The C API is here is stay for a few more years</a></li> </ul> </li> <li><a class="reference internal" href="#rejected-idea-leave-the-macros-as-they-are">Rejected Idea: Leave the macros as they are</a></li> <li><a class="reference internal" href="#macros-already-modified">Macros already modified</a></li> <li><a class="reference internal" href="#post-history">Post History</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#version-history">Version History</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <br> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0674.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>