CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 442 – Safe object finalization | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0442/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 442 – Safe object finalization | peps.python.org'> <meta property="og:description" content="This PEP proposes to deal with the current limitations of object finalization. The goal is to be able to define and run finalizers for any object, regardless of their position in the object graph."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0442/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This PEP proposes to deal with the current limitations of object finalization. The goal is to be able to define and run finalizers for any object, regardless of their position in the object graph."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 442</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> <span class="visually-hidden">Toggle light / dark / auto colour theme</span> </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 442 – Safe object finalization</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author<span class="colon">:</span></dt> <dd class="field-odd">Antoine Pitrou <solipsis at pitrou.net></dd> <dt class="field-even">BDFL-Delegate<span class="colon">:</span></dt> <dd class="field-even">Benjamin Peterson <benjamin at python.org></dd> <dt class="field-odd">Status<span class="colon">:</span></dt> <dd class="field-odd"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-even">Type<span class="colon">:</span></dt> <dd class="field-even"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-odd">Created<span class="colon">:</span></dt> <dd class="field-odd">18-May-2013</dd> <dt class="field-even">Python-Version<span class="colon">:</span></dt> <dd class="field-even">3.4</dd> <dt class="field-odd">Post-History<span class="colon">:</span></dt> <dd class="field-odd">18-May-2013</dd> <dt class="field-even">Resolution<span class="colon">:</span></dt> <dd class="field-even"><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2013-June/126746.html">Python-Dev message</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#definitions">Definitions</a></li> <li><a class="reference internal" href="#impact">Impact</a></li> <li><a class="reference internal" href="#benefits">Benefits</a></li> <li><a class="reference internal" href="#description">Description</a><ul> <li><a class="reference internal" href="#reference-counted-disposal">Reference-counted disposal</a></li> <li><a class="reference internal" href="#disposal-of-cyclic-isolates">Disposal of cyclic isolates</a></li> </ul> </li> <li><a class="reference internal" href="#c-level-changes">C-level changes</a></li> <li><a class="reference internal" href="#discussion">Discussion</a><ul> <li><a class="reference internal" href="#predictability">Predictability</a></li> <li><a class="reference internal" href="#safety">Safety</a></li> </ul> </li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#validation">Validation</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> <p>This PEP proposes to deal with the current limitations of object finalization. The goal is to be able to define and run finalizers for any object, regardless of their position in the object graph.</p> <p>This PEP doesn’t call for any change in Python code. Objects with existing finalizers will benefit automatically.</p> </section> <section id="definitions"> <h2><a class="toc-backref" href="#definitions" role="doc-backlink">Definitions</a></h2> <dl class="simple"> <dt>Reference</dt><dd>A directional link from an object to another. The target of the reference is kept alive by the reference, as long as the source is itself alive and the reference isn’t cleared.</dd> <dt>Weak reference</dt><dd>A directional link from an object to another, which doesn’t keep alive its target. This PEP focuses on non-weak references.</dd> <dt>Reference cycle</dt><dd>A cyclic subgraph of directional links between objects, which keeps those objects from being collected in a pure reference-counting scheme.</dd> <dt>Cyclic isolate (CI)</dt><dd>A standalone subgraph of objects in which no object is referenced from the outside, containing one or several reference cycles, <em>and</em> whose objects are still in a usable, non-broken state: they can access each other from their respective finalizers.</dd> <dt>Cyclic garbage collector (GC)</dt><dd>A device able to detect cyclic isolates and turn them into cyclic trash. Objects in cyclic trash are eventually disposed of by the natural effect of the references being cleared and their reference counts dropping to zero.</dd> <dt>Cyclic trash (CT)</dt><dd>A former cyclic isolate whose objects have started being cleared by the GC. Objects in cyclic trash are potential zombies; if they are accessed by Python code, the symptoms can vary from weird AttributeErrors to crashes.</dd> <dt>Zombie / broken object</dt><dd>An object part of cyclic trash. The term stresses that the object is not safe: its outgoing references may have been cleared, or one of the objects it references may be zombie. Therefore, it should not be accessed by arbitrary code (such as finalizers).</dd> <dt>Finalizer</dt><dd>A function or method called when an object is intended to be disposed of. The finalizer can access the object and release any resource held by the object (for example mutexes or file descriptors). An example is a <code class="docutils literal notranslate"><span class="pre">__del__</span></code> method.</dd> <dt>Resurrection</dt><dd>The process by which a finalizer creates a new reference to an object in a CI. This can happen as a quirky but supported side-effect of <code class="docutils literal notranslate"><span class="pre">__del__</span></code> methods.</dd> </dl> </section> <section id="impact"> <h2><a class="toc-backref" href="#impact" role="doc-backlink">Impact</a></h2> <p>While this PEP discusses CPython-specific implementation details, the change in finalization semantics is expected to affect the Python ecosystem as a whole. In particular, this PEP obsoletes the current guideline that “objects with a <code class="docutils literal notranslate"><span class="pre">__del__</span></code> method should not be part of a reference cycle”.</p> </section> <section id="benefits"> <h2><a class="toc-backref" href="#benefits" role="doc-backlink">Benefits</a></h2> <p>The primary benefits of this PEP regard objects with finalizers, such as objects with a <code class="docutils literal notranslate"><span class="pre">__del__</span></code> method and generators with a <code class="docutils literal notranslate"><span class="pre">finally</span></code> block. Those objects can now be reclaimed when they are part of a reference cycle.</p> <p>The PEP also paves the way for further benefits:</p> <ul class="simple"> <li>The module shutdown procedure may not need to set global variables to None anymore. This could solve a well-known class of irritating issues.</li> </ul> <p>The PEP doesn’t change the semantics of:</p> <ul class="simple"> <li>Weak references caught in reference cycles.</li> <li>C extension types with a custom <code class="docutils literal notranslate"><span class="pre">tp_dealloc</span></code> function.</li> </ul> </section> <section id="description"> <h2><a class="toc-backref" href="#description" role="doc-backlink">Description</a></h2> <section id="reference-counted-disposal"> <h3><a class="toc-backref" href="#reference-counted-disposal" role="doc-backlink">Reference-counted disposal</a></h3> <p>In normal reference-counted disposal, an object’s finalizer is called just before the object is deallocated. If the finalizer resurrects the object, deallocation is aborted.</p> <p><em>However</em>, if the object was already finalized, then the finalizer isn’t called. This prevents us from finalizing zombies (see below).</p> </section> <section id="disposal-of-cyclic-isolates"> <h3><a class="toc-backref" href="#disposal-of-cyclic-isolates" role="doc-backlink">Disposal of cyclic isolates</a></h3> <p>Cyclic isolates are first detected by the garbage collector, and then disposed of. The detection phase doesn’t change and won’t be described here. Disposal of a CI traditionally works in the following order:</p> <ol class="arabic simple"> <li>Weakrefs to CI objects are cleared, and their callbacks called. At this point, the objects are still safe to use.</li> <li>The CI becomes a CT as the GC systematically breaks all known references inside it (using the <code class="docutils literal notranslate"><span class="pre">tp_clear</span></code> function).</li> <li>Nothing. All CT objects should have been disposed of in step 2 (as a side-effect of clearing references); this collection is finished.</li> </ol> <p>This PEP proposes to turn CI disposal into the following sequence (new steps are in bold):</p> <ol class="arabic simple"> <li>Weakrefs to CI objects are cleared, and their callbacks called. At this point, the objects are still safe to use.</li> <li><strong>The finalizers of all CI objects are called.</strong></li> <li><strong>The CI is traversed again to determine if it is still isolated. If it is determined that at least one object in CI is now reachable from outside the CI, this collection is aborted and the whole CI is resurrected. Otherwise, proceed.</strong></li> <li>The CI becomes a CT as the GC systematically breaks all known references inside it (using the <code class="docutils literal notranslate"><span class="pre">tp_clear</span></code> function).</li> <li>Nothing. All CT objects should have been disposed of in step 4 (as a side-effect of clearing references); this collection is finished.</li> </ol> <div class="admonition note"> <p class="admonition-title">Note</p> <p>The GC doesn’t recalculate the CI after step 2 above, hence the need for step 3 to check that the whole subgraph is still isolated.</p> </div> </section> </section> <section id="c-level-changes"> <h2><a class="toc-backref" href="#c-level-changes" role="doc-backlink">C-level changes</a></h2> <p>Type objects get a new <code class="docutils literal notranslate"><span class="pre">tp_finalize</span></code> slot to which <code class="docutils literal notranslate"><span class="pre">__del__</span></code> methods are mapped (and reciprocally). Generators are modified to use this slot, rather than <code class="docutils literal notranslate"><span class="pre">tp_del</span></code>. A <code class="docutils literal notranslate"><span class="pre">tp_finalize</span></code> function is a normal C function which will be called with a valid and alive <code class="docutils literal notranslate"><span class="pre">PyObject</span></code> as its only argument. It doesn’t need to manipulate the object’s reference count, as this will be done by the caller. However, it must ensure that the original exception state is restored before returning to the caller.</p> <p>For compatibility, <code class="docutils literal notranslate"><span class="pre">tp_del</span></code> is kept in the type structure. Handling of objects with a non-NULL <code class="docutils literal notranslate"><span class="pre">tp_del</span></code> is unchanged: when part of a CI, they are not finalized and end up in <code class="docutils literal notranslate"><span class="pre">gc.garbage</span></code>. However, a non-NULL <code class="docutils literal notranslate"><span class="pre">tp_del</span></code> is not encountered anymore in the CPython source tree (except for testing purposes).</p> <p>Two new C API functions are provided to ease calling of <code class="docutils literal notranslate"><span class="pre">tp_finalize</span></code>, especially from custom deallocators.</p> <p>On the internal side, a bit is reserved in the GC header for GC-managed objects to signal that they were finalized. This helps avoid finalizing an object twice (and, especially, finalizing a CT object after it was broken by the GC).</p> <div class="admonition note"> <p class="admonition-title">Note</p> <p>Objects which are not GC-enabled can also have a <code class="docutils literal notranslate"><span class="pre">tp_finalize</span></code> slot. They don’t need the additional bit since their <code class="docutils literal notranslate"><span class="pre">tp_finalize</span></code> function can only be called from the deallocator: it therefore cannot be called twice, except when resurrected.</p> </div> </section> <section id="discussion"> <h2><a class="toc-backref" href="#discussion" role="doc-backlink">Discussion</a></h2> <section id="predictability"> <h3><a class="toc-backref" href="#predictability" role="doc-backlink">Predictability</a></h3> <p>Following this scheme, an object’s finalizer is always called exactly once, even if it was resurrected afterwards.</p> <p>For CI objects, the order in which finalizers are called (step 2 above) is undefined.</p> </section> <section id="safety"> <h3><a class="toc-backref" href="#safety" role="doc-backlink">Safety</a></h3> <p>It is important to explain why the proposed change is safe. There are two aspects to be discussed:</p> <ul class="simple"> <li>Can a finalizer access zombie objects (including the object being finalized)?</li> <li>What happens if a finalizer mutates the object graph so as to impact the CI?</li> </ul> <p>Let’s discuss the first issue. We will divide possible cases in two categories:</p> <ul class="simple"> <li>If the object being finalized is part of the CI: by construction, no objects in CI are zombies yet, since CI finalizers are called before any reference breaking is done. Therefore, the finalizer cannot access zombie objects, which don’t exist.</li> <li>If the object being finalized is not part of the CI/CT: by definition, objects in the CI/CT don’t have any references pointing to them from outside the CI/CT. Therefore, the finalizer cannot reach any zombie object (that is, even if the object being finalized was itself referenced from a zombie object).</li> </ul> <p>Now for the second issue. There are three potential cases:</p> <ul class="simple"> <li>The finalizer clears an existing reference to a CI object. The CI object may be disposed of before the GC tries to break it, which is fine (the GC simply has to be aware of this possibility).</li> <li>The finalizer creates a new reference to a CI object. This can only happen from a CI object’s finalizer (see above why). Therefore, the new reference will be detected by the GC after all CI finalizers are called (step 3 above), and collection will be aborted without any objects being broken.</li> <li>The finalizer clears or creates a reference to a non-CI object. By construction, this is not a problem.</li> </ul> </section> </section> <section id="implementation"> <h2><a class="toc-backref" href="#implementation" role="doc-backlink">Implementation</a></h2> <p>An implementation is available in branch <code class="docutils literal notranslate"><span class="pre">finalize</span></code> of the repository at <a class="reference external" href="http://hg.python.org/features/finalize/">http://hg.python.org/features/finalize/</a>.</p> </section> <section id="validation"> <h2><a class="toc-backref" href="#validation" role="doc-backlink">Validation</a></h2> <p>Besides running the normal Python test suite, the implementation adds test cases for various finalization possibilities including reference cycles, object resurrection and legacy <code class="docutils literal notranslate"><span class="pre">tp_del</span></code> slots.</p> <p>The implementation has also been checked to not produce any regressions on the following test suites:</p> <ul class="simple"> <li><a class="reference external" href="http://code.google.com/p/tulip/">Tulip</a>, which makes an extensive use of generators</li> <li><a class="reference external" href="http://www.tornadoweb.org">Tornado</a></li> <li><a class="reference external" href="http://www.sqlalchemy.org/">SQLAlchemy</a></li> <li><a class="reference external" href="https://www.djangoproject.com/">Django</a></li> <li><a class="reference external" href="http://pypi.python.org/pypi/zope.interface">zope.interface</a></li> </ul> </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <p>Notes about reference cycle collection and weak reference callbacks: <a class="reference external" href="http://hg.python.org/cpython/file/4e687d53b645/Modules/gc_weakref.txt">http://hg.python.org/cpython/file/4e687d53b645/Modules/gc_weakref.txt</a></p> <p>Generator memory leak: <a class="reference external" href="http://bugs.python.org/issue17468">http://bugs.python.org/issue17468</a></p> <p>Allow objects to decide if they can be collected by GC: <a class="reference external" href="http://bugs.python.org/issue9141">http://bugs.python.org/issue9141</a></p> <p>Module shutdown procedure based on GC <a class="reference external" href="http://bugs.python.org/issue812369">http://bugs.python.org/issue812369</a></p> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> <p>This document has been placed in the public domain.</p> </section> </section> <hr class="docutils" /> <p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0442.rst">https://github.com/python/peps/blob/main/peps/pep-0442.rst</a></p> <p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0442.rst">2025-02-01 08:59:27 GMT</a></p> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#definitions">Definitions</a></li> <li><a class="reference internal" href="#impact">Impact</a></li> <li><a class="reference internal" href="#benefits">Benefits</a></li> <li><a class="reference internal" href="#description">Description</a><ul> <li><a class="reference internal" href="#reference-counted-disposal">Reference-counted disposal</a></li> <li><a class="reference internal" href="#disposal-of-cyclic-isolates">Disposal of cyclic isolates</a></li> </ul> </li> <li><a class="reference internal" href="#c-level-changes">C-level changes</a></li> <li><a class="reference internal" href="#discussion">Discussion</a><ul> <li><a class="reference internal" href="#predictability">Predictability</a></li> <li><a class="reference internal" href="#safety">Safety</a></li> </ul> </li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#validation">Validation</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <br> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0442.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>