CINXE.COM
PEP 419 – Protecting cleanup statements from interruptions | peps.python.org
<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 419 – Protecting cleanup statements from interruptions | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0419/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 419 – Protecting cleanup statements from interruptions | peps.python.org'> <meta property="og:description" content="This PEP proposes a way to protect Python code from being interrupted inside a finally clause or during context manager cleanup."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0419/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This PEP proposes a way to protect Python code from being interrupted inside a finally clause or during context manager cleanup."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 419</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> <span class="visually-hidden">Toggle light / dark / auto colour theme</span> </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 419 – Protecting cleanup statements from interruptions</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author<span class="colon">:</span></dt> <dd class="field-odd">Paul Colomiets <paul at colomiets.name></dd> <dt class="field-even">Status<span class="colon">:</span></dt> <dd class="field-even"><abbr title="Inactive draft that may be taken up again at a later time">Deferred</abbr></dd> <dt class="field-odd">Type<span class="colon">:</span></dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created<span class="colon">:</span></dt> <dd class="field-even">06-Apr-2012</dd> <dt class="field-odd">Python-Version<span class="colon">:</span></dt> <dd class="field-odd">3.3</dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#pep-deferral">PEP Deferral</a></li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#coroutine-use-case">Coroutine Use Case</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#frame-flag-f-in-cleanup">Frame Flag ‘f_in_cleanup’</a></li> <li><a class="reference internal" href="#function-sys-setcleanuphook">Function ‘sys.setcleanuphook’</a></li> <li><a class="reference internal" href="#inspect-module-enhancements">Inspect Module Enhancements</a></li> </ul> </li> <li><a class="reference internal" href="#example">Example</a></li> <li><a class="reference internal" href="#unresolved-issues">Unresolved Issues</a><ul> <li><a class="reference internal" href="#interruption-inside-with-statement-expression">Interruption Inside With Statement Expression</a></li> <li><a class="reference internal" href="#exception-propagation">Exception Propagation</a></li> <li><a class="reference internal" href="#interruption-between-acquiring-resource-and-try-block">Interruption Between Acquiring Resource and Try Block</a></li> <li><a class="reference internal" href="#handling-eintr-inside-a-finally">Handling EINTR Inside a Finally</a></li> <li><a class="reference internal" href="#setting-interruption-context-inside-finally-itself">Setting Interruption Context Inside Finally Itself</a></li> <li><a class="reference internal" href="#modifying-keyboardinterrupt">Modifying KeyboardInterrupt</a></li> </ul> </li> <li><a class="reference internal" href="#alternative-python-implementations-support">Alternative Python Implementations Support</a></li> <li><a class="reference internal" href="#alternative-names">Alternative Names</a></li> <li><a class="reference internal" href="#alternative-proposals">Alternative Proposals</a><ul> <li><a class="reference internal" href="#propagating-f-in-cleanup-flag-automatically">Propagating ‘f_in_cleanup’ Flag Automatically</a></li> <li><a class="reference internal" href="#add-bytecodes-incr-cleanup-decr-cleanup">Add Bytecodes ‘INCR_CLEANUP’, ‘DECR_CLEANUP’</a></li> <li><a class="reference internal" href="#expose-f-in-cleanup-as-a-counter">Expose ‘f_in_cleanup’ as a Counter</a></li> <li><a class="reference internal" href="#add-code-object-flag-co-cleanup">Add code object flag ‘CO_CLEANUP’</a></li> <li><a class="reference internal" href="#have-cleanup-callback-on-frame-object-itself">Have Cleanup Callback on Frame Object Itself</a></li> <li><a class="reference internal" href="#no-cleanup-hook">No Cleanup Hook</a></li> </ul> </li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> <p>This PEP proposes a way to protect Python code from being interrupted inside a finally clause or during context manager cleanup.</p> </section> <section id="pep-deferral"> <h2><a class="toc-backref" href="#pep-deferral" role="doc-backlink">PEP Deferral</a></h2> <p>Further exploration of the concepts covered in this PEP has been deferred for lack of a current champion interested in promoting the goals of the PEP and collecting and incorporating feedback, and with sufficient available time to do so effectively.</p> </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> <p>Python has two nice ways to do cleanup. One is a <code class="docutils literal notranslate"><span class="pre">finally</span></code> statement and the other is a context manager (usually called using a <code class="docutils literal notranslate"><span class="pre">with</span></code> statement). However, neither is protected from interruption by <code class="docutils literal notranslate"><span class="pre">KeyboardInterrupt</span></code> or <code class="docutils literal notranslate"><span class="pre">GeneratorExit</span></code> caused by <code class="docutils literal notranslate"><span class="pre">generator.throw()</span></code>. For example:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">lock</span><span class="o">.</span><span class="n">acquire</span><span class="p">()</span> <span class="k">try</span><span class="p">:</span> <span class="nb">print</span><span class="p">(</span><span class="s1">'starting'</span><span class="p">)</span> <span class="n">do_something</span><span class="p">()</span> <span class="k">finally</span><span class="p">:</span> <span class="nb">print</span><span class="p">(</span><span class="s1">'finished'</span><span class="p">)</span> <span class="n">lock</span><span class="o">.</span><span class="n">release</span><span class="p">()</span> </pre></div> </div> <p>If <code class="docutils literal notranslate"><span class="pre">KeyboardInterrupt</span></code> occurs just after the second <code class="docutils literal notranslate"><span class="pre">print()</span></code> call, the lock will not be released. Similarly, the following code using the <code class="docutils literal notranslate"><span class="pre">with</span></code> statement is affected:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">threading</span><span class="w"> </span><span class="kn">import</span> <span class="n">Lock</span> <span class="k">class</span><span class="w"> </span><span class="nc">MyLock</span><span class="p">:</span> <span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">_lock_impl</span> <span class="o">=</span> <span class="n">Lock</span><span class="p">()</span> <span class="k">def</span><span class="w"> </span><span class="fm">__enter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">_lock_impl</span><span class="o">.</span><span class="n">acquire</span><span class="p">()</span> <span class="nb">print</span><span class="p">(</span><span class="s2">"LOCKED"</span><span class="p">)</span> <span class="k">def</span><span class="w"> </span><span class="fm">__exit__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="nb">print</span><span class="p">(</span><span class="s2">"UNLOCKING"</span><span class="p">)</span> <span class="bp">self</span><span class="o">.</span><span class="n">_lock_impl</span><span class="o">.</span><span class="n">release</span><span class="p">()</span> <span class="n">lock</span> <span class="o">=</span> <span class="n">MyLock</span><span class="p">()</span> <span class="k">with</span> <span class="n">lock</span><span class="p">:</span> <span class="n">do_something</span> </pre></div> </div> <p>If <code class="docutils literal notranslate"><span class="pre">KeyboardInterrupt</span></code> occurs near any of the <code class="docutils literal notranslate"><span class="pre">print()</span></code> calls, the lock will never be released.</p> <section id="coroutine-use-case"> <h3><a class="toc-backref" href="#coroutine-use-case" role="doc-backlink">Coroutine Use Case</a></h3> <p>A similar case occurs with coroutines. Usually coroutine libraries want to interrupt the coroutine with a timeout. The <code class="docutils literal notranslate"><span class="pre">generator.throw()</span></code> method works for this use case, but there is no way of knowing if the coroutine is currently suspended from inside a <code class="docutils literal notranslate"><span class="pre">finally</span></code> clause.</p> <p>An example that uses yield-based coroutines follows. The code looks similar using any of the popular coroutine libraries Monocle <a class="footnote-reference brackets" href="#id4" id="id1">[1]</a>, Bluelet <a class="footnote-reference brackets" href="#id5" id="id2">[2]</a>, or Twisted <a class="footnote-reference brackets" href="#id6" id="id3">[3]</a>.</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">run_locked</span><span class="p">():</span> <span class="k">yield</span> <span class="n">connection</span><span class="o">.</span><span class="n">sendall</span><span class="p">(</span><span class="s1">'LOCK'</span><span class="p">)</span> <span class="k">try</span><span class="p">:</span> <span class="k">yield</span> <span class="n">do_something</span><span class="p">()</span> <span class="k">yield</span> <span class="n">do_something_else</span><span class="p">()</span> <span class="k">finally</span><span class="p">:</span> <span class="k">yield</span> <span class="n">connection</span><span class="o">.</span><span class="n">sendall</span><span class="p">(</span><span class="s1">'UNLOCK'</span><span class="p">)</span> <span class="k">with</span> <span class="n">timeout</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span> <span class="k">yield</span> <span class="n">run_locked</span><span class="p">()</span> </pre></div> </div> <p>In the example above, <code class="docutils literal notranslate"><span class="pre">yield</span> <span class="pre">something</span></code> means to pause executing the current coroutine and to execute coroutine <code class="docutils literal notranslate"><span class="pre">something</span></code> until it finishes execution. Therefore, the coroutine library itself needs to maintain a stack of generators. The <code class="docutils literal notranslate"><span class="pre">connection.sendall()</span></code> call waits until the socket is writable and does a similar thing to what <code class="docutils literal notranslate"><span class="pre">socket.sendall()</span></code> does.</p> <p>The <code class="docutils literal notranslate"><span class="pre">with</span></code> statement ensures that all code is executed within 5 seconds timeout. It does so by registering a callback in the main loop, which calls <code class="docutils literal notranslate"><span class="pre">generator.throw()</span></code> on the top-most frame in the coroutine stack when a timeout happens.</p> <p>The <code class="docutils literal notranslate"><span class="pre">greenlets</span></code> extension works in a similar way, except that it doesn’t need <code class="docutils literal notranslate"><span class="pre">yield</span></code> to enter a new stack frame. Otherwise considerations are similar.</p> </section> </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> <section id="frame-flag-f-in-cleanup"> <h3><a class="toc-backref" href="#frame-flag-f-in-cleanup" role="doc-backlink">Frame Flag ‘f_in_cleanup’</a></h3> <p>A new flag on the frame object is proposed. It is set to <code class="docutils literal notranslate"><span class="pre">True</span></code> if this frame is currently executing a <code class="docutils literal notranslate"><span class="pre">finally</span></code> clause. Internally, the flag must be implemented as a counter of nested finally statements currently being executed.</p> <p>The internal counter also needs to be incremented during execution of the <code class="docutils literal notranslate"><span class="pre">SETUP_WITH</span></code> and <code class="docutils literal notranslate"><span class="pre">WITH_CLEANUP</span></code> bytecodes, and decremented when execution for these bytecodes is finished. This allows to also protect <code class="docutils literal notranslate"><span class="pre">__enter__()</span></code> and <code class="docutils literal notranslate"><span class="pre">__exit__()</span></code> methods.</p> </section> <section id="function-sys-setcleanuphook"> <h3><a class="toc-backref" href="#function-sys-setcleanuphook" role="doc-backlink">Function ‘sys.setcleanuphook’</a></h3> <p>A new function for the <code class="docutils literal notranslate"><span class="pre">sys</span></code> module is proposed. This function sets a callback which is executed every time <code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code> becomes false. Callbacks get a frame object as their sole argument, so that they can figure out where they are called from.</p> <p>The setting is thread local and must be stored in the <code class="docutils literal notranslate"><span class="pre">PyThreadState</span></code> structure.</p> </section> <section id="inspect-module-enhancements"> <h3><a class="toc-backref" href="#inspect-module-enhancements" role="doc-backlink">Inspect Module Enhancements</a></h3> <p>Two new functions are proposed for the <code class="docutils literal notranslate"><span class="pre">inspect</span></code> module: <code class="docutils literal notranslate"><span class="pre">isframeincleanup()</span></code> and <code class="docutils literal notranslate"><span class="pre">getcleanupframe()</span></code>.</p> <p><code class="docutils literal notranslate"><span class="pre">isframeincleanup()</span></code>, given a frame or generator object as its sole argument, returns the value of the <code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code> attribute of a frame itself or of the <code class="docutils literal notranslate"><span class="pre">gi_frame</span></code> attribute of a generator.</p> <p><code class="docutils literal notranslate"><span class="pre">getcleanupframe()</span></code>, given a frame object as its sole argument, returns the innermost frame which has a true value of <code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code>, or <code class="docutils literal notranslate"><span class="pre">None</span></code> if no frames in the stack have a nonzero value for that attribute. It starts to inspect from the specified frame and walks to outer frames using <code class="docutils literal notranslate"><span class="pre">f_back</span></code> pointers, just like <code class="docutils literal notranslate"><span class="pre">getouterframes()</span></code> does.</p> </section> </section> <section id="example"> <h2><a class="toc-backref" href="#example" role="doc-backlink">Example</a></h2> <p>An example implementation of a SIGINT handler that interrupts safely might look like:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">inspect</span><span class="o">,</span><span class="w"> </span><span class="nn">sys</span><span class="o">,</span><span class="w"> </span><span class="nn">functools</span> <span class="k">def</span><span class="w"> </span><span class="nf">sigint_handler</span><span class="p">(</span><span class="n">sig</span><span class="p">,</span> <span class="n">frame</span><span class="p">):</span> <span class="k">if</span> <span class="n">inspect</span><span class="o">.</span><span class="n">getcleanupframe</span><span class="p">(</span><span class="n">frame</span><span class="p">)</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span> <span class="k">raise</span> <span class="ne">KeyboardInterrupt</span><span class="p">()</span> <span class="n">sys</span><span class="o">.</span><span class="n">setcleanuphook</span><span class="p">(</span><span class="n">functools</span><span class="o">.</span><span class="n">partial</span><span class="p">(</span><span class="n">sigint_handler</span><span class="p">,</span> <span class="mi">0</span><span class="p">))</span> </pre></div> </div> <p>A coroutine example is out of scope of this document, because its implementation depends very much on a trampoline (or main loop) used by coroutine library.</p> </section> <section id="unresolved-issues"> <h2><a class="toc-backref" href="#unresolved-issues" role="doc-backlink">Unresolved Issues</a></h2> <section id="interruption-inside-with-statement-expression"> <h3><a class="toc-backref" href="#interruption-inside-with-statement-expression" role="doc-backlink">Interruption Inside With Statement Expression</a></h3> <p>Given the statement</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">):</span> <span class="n">do_something</span><span class="p">()</span> </pre></div> </div> <p>Python can be interrupted after <code class="docutils literal notranslate"><span class="pre">open()</span></code> is called, but before the <code class="docutils literal notranslate"><span class="pre">SETUP_WITH</span></code> bytecode is executed. There are two possible decisions:</p> <ul> <li>Protect <code class="docutils literal notranslate"><span class="pre">with</span></code> expressions. This would require another bytecode, since currently there is no way of recognizing the start of the <code class="docutils literal notranslate"><span class="pre">with</span></code> expression.</li> <li>Let the user write a wrapper if he considers it important for the use-case. A safe wrapper might look like this:<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">class</span><span class="w"> </span><span class="nc">FileWrapper</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">mode</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">filename</span> <span class="o">=</span> <span class="n">filename</span> <span class="bp">self</span><span class="o">.</span><span class="n">mode</span> <span class="o">=</span> <span class="n">mode</span> <span class="k">def</span><span class="w"> </span><span class="fm">__enter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">file</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">mode</span><span class="p">)</span> <span class="k">def</span><span class="w"> </span><span class="fm">__exit__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="bp">self</span><span class="o">.</span><span class="n">file</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> </pre></div> </div> <p>Alternatively it can be written using the <code class="docutils literal notranslate"><span class="pre">contextmanager()</span></code> decorator:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="nd">@contextmanager</span> <span class="k">def</span><span class="w"> </span><span class="nf">open_wrapper</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">mode</span><span class="p">):</span> <span class="n">file</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">mode</span><span class="p">)</span> <span class="k">try</span><span class="p">:</span> <span class="k">yield</span> <span class="n">file</span> <span class="k">finally</span><span class="p">:</span> <span class="n">file</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> </pre></div> </div> <p>This code is safe, as the first part of the generator (before yield) is executed inside the <code class="docutils literal notranslate"><span class="pre">SETUP_WITH</span></code> bytecode of the caller.</p> </li> </ul> </section> <section id="exception-propagation"> <h3><a class="toc-backref" href="#exception-propagation" role="doc-backlink">Exception Propagation</a></h3> <p>Sometimes a <code class="docutils literal notranslate"><span class="pre">finally</span></code> clause or an <code class="docutils literal notranslate"><span class="pre">__enter__()</span></code>/<code class="docutils literal notranslate"><span class="pre">__exit__()</span></code> method can raise an exception. Usually this is not a problem, since more important exceptions like <code class="docutils literal notranslate"><span class="pre">KeyboardInterrupt</span></code> or <code class="docutils literal notranslate"><span class="pre">SystemExit</span></code> should be raised instead. But it may be nice to be able to keep the original exception inside a <code class="docutils literal notranslate"><span class="pre">__context__</span></code> attribute. So the cleanup hook signature may grow an exception argument:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">sigint_handler</span><span class="p">(</span><span class="n">sig</span><span class="p">,</span> <span class="n">frame</span><span class="p">)</span> <span class="k">if</span> <span class="n">inspect</span><span class="o">.</span><span class="n">getcleanupframe</span><span class="p">(</span><span class="n">frame</span><span class="p">)</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span> <span class="k">raise</span> <span class="ne">KeyboardInterrupt</span><span class="p">()</span> <span class="n">sys</span><span class="o">.</span><span class="n">setcleanuphook</span><span class="p">(</span><span class="n">retry_sigint</span><span class="p">)</span> <span class="k">def</span><span class="w"> </span><span class="nf">retry_sigint</span><span class="p">(</span><span class="n">frame</span><span class="p">,</span> <span class="n">exception</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span> <span class="k">if</span> <span class="n">inspect</span><span class="o">.</span><span class="n">getcleanupframe</span><span class="p">(</span><span class="n">frame</span><span class="p">)</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span> <span class="k">raise</span> <span class="ne">KeyboardInterrupt</span><span class="p">()</span> <span class="kn">from</span><span class="w"> </span><span class="nn">exception</span> </pre></div> </div> <div class="admonition note"> <p class="admonition-title">Note</p> <p>There is no need to have three arguments like in the <code class="docutils literal notranslate"><span class="pre">__exit__</span></code> method since there is a <code class="docutils literal notranslate"><span class="pre">__traceback__</span></code> attribute in exception in Python 3.</p> </div> <p>However, this will set the <code class="docutils literal notranslate"><span class="pre">__cause__</span></code> for the exception, which is not exactly what’s intended. So some hidden interpreter logic may be used to put a <code class="docutils literal notranslate"><span class="pre">__context__</span></code> attribute on every exception raised in a cleanup hook.</p> </section> <section id="interruption-between-acquiring-resource-and-try-block"> <h3><a class="toc-backref" href="#interruption-between-acquiring-resource-and-try-block" role="doc-backlink">Interruption Between Acquiring Resource and Try Block</a></h3> <p>The example from the first section is not totally safe. Let’s take a closer look:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">lock</span><span class="o">.</span><span class="n">acquire</span><span class="p">()</span> <span class="k">try</span><span class="p">:</span> <span class="n">do_something</span><span class="p">()</span> <span class="k">finally</span><span class="p">:</span> <span class="n">lock</span><span class="o">.</span><span class="n">release</span><span class="p">()</span> </pre></div> </div> <p>The problem might occur if the code is interrupted just after <code class="docutils literal notranslate"><span class="pre">lock.acquire()</span></code> is executed but before the <code class="docutils literal notranslate"><span class="pre">try</span></code> block is entered.</p> <p>There is no way the code can be fixed unmodified. The actual fix depends very much on the use case. Usually code can be fixed using a <code class="docutils literal notranslate"><span class="pre">with</span></code> statement:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">lock</span><span class="p">:</span> <span class="n">do_something</span><span class="p">()</span> </pre></div> </div> <p>However, for coroutines one usually can’t use the <code class="docutils literal notranslate"><span class="pre">with</span></code> statement because you need to <code class="docutils literal notranslate"><span class="pre">yield</span></code> for both the acquire and release operations. So the code might be rewritten like this:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">try</span><span class="p">:</span> <span class="k">yield</span> <span class="n">lock</span><span class="o">.</span><span class="n">acquire</span><span class="p">()</span> <span class="n">do_something</span><span class="p">()</span> <span class="k">finally</span><span class="p">:</span> <span class="k">yield</span> <span class="n">lock</span><span class="o">.</span><span class="n">release</span><span class="p">()</span> </pre></div> </div> <p>The actual locking code might need more code to support this use case, but the implementation is usually trivial, like this: check if the lock has been acquired and unlock if it is.</p> </section> <section id="handling-eintr-inside-a-finally"> <h3><a class="toc-backref" href="#handling-eintr-inside-a-finally" role="doc-backlink">Handling EINTR Inside a Finally</a></h3> <p>Even if a signal handler is prepared to check the <code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code> flag, <code class="docutils literal notranslate"><span class="pre">InterruptedError</span></code> might be raised in the cleanup handler, because the respective system call returned an <code class="docutils literal notranslate"><span class="pre">EINTR</span></code> error. The primary use cases are prepared to handle this:</p> <ul class="simple"> <li>Posix mutexes never return <code class="docutils literal notranslate"><span class="pre">EINTR</span></code></li> <li>Networking libraries are always prepared to handle <code class="docutils literal notranslate"><span class="pre">EINTR</span></code></li> <li>Coroutine libraries are usually interrupted with the <code class="docutils literal notranslate"><span class="pre">throw()</span></code> method, not with a signal</li> </ul> <p>The platform-specific function <code class="docutils literal notranslate"><span class="pre">siginterrupt()</span></code> might be used to remove the need to handle <code class="docutils literal notranslate"><span class="pre">EINTR</span></code>. However, it may have hardly predictable consequences, for example <code class="docutils literal notranslate"><span class="pre">SIGINT</span></code> a handler is never called if the main thread is stuck inside an IO routine.</p> <p>A better approach would be to have the code, which is usually used in cleanup handlers, be prepared to handle <code class="docutils literal notranslate"><span class="pre">InterruptedError</span></code> explicitly. An example of such code might be a file-based lock implementation.</p> <p><code class="docutils literal notranslate"><span class="pre">signal.pthread_sigmask</span></code> can be used to block signals inside cleanup handlers which can be interrupted with <code class="docutils literal notranslate"><span class="pre">EINTR</span></code>.</p> </section> <section id="setting-interruption-context-inside-finally-itself"> <h3><a class="toc-backref" href="#setting-interruption-context-inside-finally-itself" role="doc-backlink">Setting Interruption Context Inside Finally Itself</a></h3> <p>Some coroutine libraries may need to set a timeout for the finally clause itself. For example:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">try</span><span class="p">:</span> <span class="n">do_something</span><span class="p">()</span> <span class="k">finally</span><span class="p">:</span> <span class="k">with</span> <span class="n">timeout</span><span class="p">(</span><span class="mf">0.5</span><span class="p">):</span> <span class="k">try</span><span class="p">:</span> <span class="k">yield</span> <span class="n">do_slow_cleanup</span><span class="p">()</span> <span class="k">finally</span><span class="p">:</span> <span class="k">yield</span> <span class="n">do_fast_cleanup</span><span class="p">()</span> </pre></div> </div> <p>With current semantics, timeout will either protect the whole <code class="docutils literal notranslate"><span class="pre">with</span></code> block or nothing at all, depending on the implementation of each library. What the author intended is to treat <code class="docutils literal notranslate"><span class="pre">do_slow_cleanup</span></code> as ordinary code, and <code class="docutils literal notranslate"><span class="pre">do_fast_cleanup</span></code> as a cleanup (a non-interruptible one).</p> <p>A similar case might occur when using greenlets or tasklets.</p> <p>This case can be fixed by exposing <code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code> as a counter, and by calling a cleanup hook on each decrement. A coroutine library may then remember the value at timeout start, and compare it on each hook execution.</p> <p>But in practice, the example is considered to be too obscure to take into account.</p> </section> <section id="modifying-keyboardinterrupt"> <h3><a class="toc-backref" href="#modifying-keyboardinterrupt" role="doc-backlink">Modifying KeyboardInterrupt</a></h3> <p>It should be decided if the default <code class="docutils literal notranslate"><span class="pre">SIGINT</span></code> handler should be modified to use the described mechanism. The initial proposition is to keep old behavior, for two reasons:</p> <ul class="simple"> <li>Most application do not care about cleanup on exit (either they do not have external state, or they modify it in crash-safe way).</li> <li>Cleanup may take too much time, not giving user a chance to interrupt an application.</li> </ul> <p>The latter case can be fixed by allowing an unsafe break if a <code class="docutils literal notranslate"><span class="pre">SIGINT</span></code> handler is called twice, but it seems not worth the complexity.</p> </section> </section> <section id="alternative-python-implementations-support"> <h2><a class="toc-backref" href="#alternative-python-implementations-support" role="doc-backlink">Alternative Python Implementations Support</a></h2> <p>We consider <code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code> an implementation detail. The actual implementation may have some fake frame-like object passed to signal handler, cleanup hook and returned from <code class="docutils literal notranslate"><span class="pre">getcleanupframe()</span></code>. The only requirement is that the <code class="docutils literal notranslate"><span class="pre">inspect</span></code> module functions work as expected on these objects. For this reason, we also allow to pass a generator object to the <code class="docutils literal notranslate"><span class="pre">isframeincleanup()</span></code> function, which removes the need to use the <code class="docutils literal notranslate"><span class="pre">gi_frame</span></code> attribute.</p> <p>It might be necessary to specify that <code class="docutils literal notranslate"><span class="pre">getcleanupframe()</span></code> must return the same object that will be passed to cleanup hook at the next invocation.</p> </section> <section id="alternative-names"> <h2><a class="toc-backref" href="#alternative-names" role="doc-backlink">Alternative Names</a></h2> <p>The original proposal had a <code class="docutils literal notranslate"><span class="pre">f_in_finally</span></code> frame attribute, as the original intention was to protect <code class="docutils literal notranslate"><span class="pre">finally</span></code> clauses. But as it grew up to protecting <code class="docutils literal notranslate"><span class="pre">__enter__</span></code> and <code class="docutils literal notranslate"><span class="pre">__exit__</span></code> methods too, the <code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code> name seems better. Although the <code class="docutils literal notranslate"><span class="pre">__enter__</span></code> method is not a cleanup routine, it at least relates to cleanup done by context managers.</p> <p><code class="docutils literal notranslate"><span class="pre">setcleanuphook</span></code>, <code class="docutils literal notranslate"><span class="pre">isframeincleanup</span></code> and <code class="docutils literal notranslate"><span class="pre">getcleanupframe</span></code> can be unobscured to <code class="docutils literal notranslate"><span class="pre">set_cleanup_hook</span></code>, <code class="docutils literal notranslate"><span class="pre">is_frame_in_cleanup</span></code> and <code class="docutils literal notranslate"><span class="pre">get_cleanup_frame</span></code>, although they follow the naming convention of their respective modules.</p> </section> <section id="alternative-proposals"> <h2><a class="toc-backref" href="#alternative-proposals" role="doc-backlink">Alternative Proposals</a></h2> <section id="propagating-f-in-cleanup-flag-automatically"> <h3><a class="toc-backref" href="#propagating-f-in-cleanup-flag-automatically" role="doc-backlink">Propagating ‘f_in_cleanup’ Flag Automatically</a></h3> <p>This can make <code class="docutils literal notranslate"><span class="pre">getcleanupframe()</span></code> unnecessary. But for yield-based coroutines you need to propagate it yourself. Making it writable leads to somewhat unpredictable behavior of <code class="docutils literal notranslate"><span class="pre">setcleanuphook()</span></code>.</p> </section> <section id="add-bytecodes-incr-cleanup-decr-cleanup"> <h3><a class="toc-backref" href="#add-bytecodes-incr-cleanup-decr-cleanup" role="doc-backlink">Add Bytecodes ‘INCR_CLEANUP’, ‘DECR_CLEANUP’</a></h3> <p>These bytecodes can be used to protect the expression inside the <code class="docutils literal notranslate"><span class="pre">with</span></code> statement, as well as making counter increments more explicit and easy to debug (visible inside a disassembly). Some middle ground might be chosen, like <code class="docutils literal notranslate"><span class="pre">END_FINALLY</span></code> and <code class="docutils literal notranslate"><span class="pre">SETUP_WITH</span></code> implicitly decrementing the counter (<code class="docutils literal notranslate"><span class="pre">END_FINALLY</span></code> is present at end of every <code class="docutils literal notranslate"><span class="pre">with</span></code> suite).</p> <p>However, adding new bytecodes must be considered very carefully.</p> </section> <section id="expose-f-in-cleanup-as-a-counter"> <h3><a class="toc-backref" href="#expose-f-in-cleanup-as-a-counter" role="doc-backlink">Expose ‘f_in_cleanup’ as a Counter</a></h3> <p>The original intention was to expose a minimum of needed functionality. However, as we consider the frame flag <code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code> an implementation detail, we may expose it as a counter.</p> <p>Similarly, if we have a counter we may need to have the cleanup hook called on every counter decrement. It’s unlikely to have much performance impact as nested finally clauses are an uncommon case.</p> </section> <section id="add-code-object-flag-co-cleanup"> <h3><a class="toc-backref" href="#add-code-object-flag-co-cleanup" role="doc-backlink">Add code object flag ‘CO_CLEANUP’</a></h3> <p>As an alternative to set the flag inside the <code class="docutils literal notranslate"><span class="pre">SETUP_WITH</span></code> and <code class="docutils literal notranslate"><span class="pre">WITH_CLEANUP</span></code> bytecodes, we can introduce a flag <code class="docutils literal notranslate"><span class="pre">CO_CLEANUP</span></code>. When the interpreter starts to execute code with <code class="docutils literal notranslate"><span class="pre">CO_CLEANUP</span></code> set, it sets <code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code> for the whole function body. This flag is set for code objects of <code class="docutils literal notranslate"><span class="pre">__enter__</span></code> and <code class="docutils literal notranslate"><span class="pre">__exit__</span></code> special methods. Technically it might be set on functions called <code class="docutils literal notranslate"><span class="pre">__enter__</span></code> and <code class="docutils literal notranslate"><span class="pre">__exit__</span></code>.</p> <p>This seems to be less clear solution. It also covers the case where <code class="docutils literal notranslate"><span class="pre">__enter__</span></code> and <code class="docutils literal notranslate"><span class="pre">__exit__</span></code> are called manually. This may be accepted either as a feature or as an unnecessary side-effect (or, though unlikely, as a bug).</p> <p>It may also impose a problem when <code class="docutils literal notranslate"><span class="pre">__enter__</span></code> or <code class="docutils literal notranslate"><span class="pre">__exit__</span></code> functions are implemented in C, as there is no code object to check for the <code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code> flag.</p> </section> <section id="have-cleanup-callback-on-frame-object-itself"> <h3><a class="toc-backref" href="#have-cleanup-callback-on-frame-object-itself" role="doc-backlink">Have Cleanup Callback on Frame Object Itself</a></h3> <p>The frame object may be extended to have a <code class="docutils literal notranslate"><span class="pre">f_cleanup_callback</span></code> member which is called when <code class="docutils literal notranslate"><span class="pre">f_in_cleanup</span></code> is reset to 0. This would help to register different callbacks to different coroutines.</p> <p>Despite its apparent beauty, this solution doesn’t add anything, as the two primary use cases are:</p> <ul class="simple"> <li>Setting the callback in a signal handler. The callback is inherently a single one for this case.</li> <li>Use a single callback per loop for the coroutine use case. Here, in almost all cases, there is only one loop per thread.</li> </ul> </section> <section id="no-cleanup-hook"> <h3><a class="toc-backref" href="#no-cleanup-hook" role="doc-backlink">No Cleanup Hook</a></h3> <p>The original proposal included no cleanup hook specification, as there are a few ways to achieve the same using current tools:</p> <ul class="simple"> <li>Using <code class="docutils literal notranslate"><span class="pre">sys.settrace()</span></code> and the <code class="docutils literal notranslate"><span class="pre">f_trace</span></code> callback. This may impose some problem to debugging, and has a big performance impact (although interrupting doesn’t happen very often).</li> <li>Sleeping a bit more and trying again. For a coroutine library this is easy. For signals it may be achieved using <code class="docutils literal notranslate"><span class="pre">signal.alert</span></code>.</li> </ul> <p>Both methods are considered too impractical and a way to catch exit from <code class="docutils literal notranslate"><span class="pre">finally</span></code> clauses is proposed.</p> </section> </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <aside class="footnote-list brackets"> <aside class="footnote brackets" id="id4" role="doc-footnote"> <dt class="label" id="id4">[<a href="#id1">1</a>]</dt> <dd>Monocle <a class="reference external" href="https://github.com/saucelabs/monocle">https://github.com/saucelabs/monocle</a></aside> <aside class="footnote brackets" id="id5" role="doc-footnote"> <dt class="label" id="id5">[<a href="#id2">2</a>]</dt> <dd>Bluelet <a class="reference external" href="https://github.com/sampsyo/bluelet">https://github.com/sampsyo/bluelet</a></aside> <aside class="footnote brackets" id="id6" role="doc-footnote"> <dt class="label" id="id6">[<a href="#id3">3</a>]</dt> <dd>Twisted: inlineCallbacks <a class="reference external" href="https://twisted.org/documents/8.1.0/api/twisted.internet.defer.html">https://twisted.org/documents/8.1.0/api/twisted.internet.defer.html</a></aside> </aside> <p>[4] Original discussion <a class="reference external" href="https://mail.python.org/pipermail/python-ideas/2012-April/014705.html">https://mail.python.org/pipermail/python-ideas/2012-April/014705.html</a></p> <p>[5] Implementation of PEP 419 <a class="reference external" href="https://github.com/python/cpython/issues/58935">https://github.com/python/cpython/issues/58935</a></p> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> <p>This document has been placed in the public domain.</p> </section> </section> <hr class="docutils" /> <p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0419.rst">https://github.com/python/peps/blob/main/peps/pep-0419.rst</a></p> <p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0419.rst">2025-02-01 08:59:27 GMT</a></p> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#pep-deferral">PEP Deferral</a></li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#coroutine-use-case">Coroutine Use Case</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#frame-flag-f-in-cleanup">Frame Flag ‘f_in_cleanup’</a></li> <li><a class="reference internal" href="#function-sys-setcleanuphook">Function ‘sys.setcleanuphook’</a></li> <li><a class="reference internal" href="#inspect-module-enhancements">Inspect Module Enhancements</a></li> </ul> </li> <li><a class="reference internal" href="#example">Example</a></li> <li><a class="reference internal" href="#unresolved-issues">Unresolved Issues</a><ul> <li><a class="reference internal" href="#interruption-inside-with-statement-expression">Interruption Inside With Statement Expression</a></li> <li><a class="reference internal" href="#exception-propagation">Exception Propagation</a></li> <li><a class="reference internal" href="#interruption-between-acquiring-resource-and-try-block">Interruption Between Acquiring Resource and Try Block</a></li> <li><a class="reference internal" href="#handling-eintr-inside-a-finally">Handling EINTR Inside a Finally</a></li> <li><a class="reference internal" href="#setting-interruption-context-inside-finally-itself">Setting Interruption Context Inside Finally Itself</a></li> <li><a class="reference internal" href="#modifying-keyboardinterrupt">Modifying KeyboardInterrupt</a></li> </ul> </li> <li><a class="reference internal" href="#alternative-python-implementations-support">Alternative Python Implementations Support</a></li> <li><a class="reference internal" href="#alternative-names">Alternative Names</a></li> <li><a class="reference internal" href="#alternative-proposals">Alternative Proposals</a><ul> <li><a class="reference internal" href="#propagating-f-in-cleanup-flag-automatically">Propagating ‘f_in_cleanup’ Flag Automatically</a></li> <li><a class="reference internal" href="#add-bytecodes-incr-cleanup-decr-cleanup">Add Bytecodes ‘INCR_CLEANUP’, ‘DECR_CLEANUP’</a></li> <li><a class="reference internal" href="#expose-f-in-cleanup-as-a-counter">Expose ‘f_in_cleanup’ as a Counter</a></li> <li><a class="reference internal" href="#add-code-object-flag-co-cleanup">Add code object flag ‘CO_CLEANUP’</a></li> <li><a class="reference internal" href="#have-cleanup-callback-on-frame-object-itself">Have Cleanup Callback on Frame Object Itself</a></li> <li><a class="reference internal" href="#no-cleanup-hook">No Cleanup Hook</a></li> </ul> </li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <br> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0419.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>