CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 419 – Protecting cleanup statements from interruptions | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0419/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 419 – Protecting cleanup statements from interruptions | peps.python.org'> <meta property="og:description" content="This PEP proposes a way to protect Python code from being interrupted inside a finally clause or during context manager cleanup."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0419/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This PEP proposes a way to protect Python code from being interrupted inside a finally clause or during context manager cleanup."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 419</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 419 – Protecting cleanup statements from interruptions</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Paul Colomiets <paul at colomiets.name></dd> <dt class="field-even">Status:</dt> <dd class="field-even"><abbr title="Inactive draft that may be taken up again at a later time">Deferred</abbr></dd> <dt class="field-odd">Type:</dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created:</dt> <dd class="field-even">06-Apr-2012</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">3.3</dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#pep-deferral">PEP Deferral</a></li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#coroutine-use-case">Coroutine Use Case</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#frame-flag-f-in-cleanup">Frame Flag ‘f_in_cleanup’</a></li> <li><a class="reference internal" href="#function-sys-setcleanuphook">Function ‘sys.setcleanuphook’</a></li> <li><a class="reference internal" href="#inspect-module-enhancements">Inspect Module Enhancements</a></li> </ul> </li> <li><a class="reference internal" href="#example">Example</a></li> <li><a class="reference internal" href="#unresolved-issues">Unresolved Issues</a><ul> <li><a class="reference internal" href="#interruption-inside-with-statement-expression">Interruption Inside With Statement Expression</a></li> <li><a class="reference internal" href="#exception-propagation">Exception Propagation</a></li> <li><a class="reference internal" href="#interruption-between-acquiring-resource-and-try-block">Interruption Between Acquiring Resource and Try Block</a></li> <li><a class="reference internal" href="#handling-eintr-inside-a-finally">Handling EINTR Inside a Finally</a></li> <li><a class="reference internal" href="#setting-interruption-context-inside-finally-itself">Setting Interruption Context Inside Finally Itself</a></li> <li><a class="reference internal" href="#modifying-keyboardinterrupt">Modifying KeyboardInterrupt</a></li> </ul> </li> <li><a class="reference internal" href="#alternative-python-implementations-support">Alternative Python Implementations Support</a></li> <li><a class="reference internal" href="#alternative-names">Alternative Names</a></li> <li><a class="reference internal" href="#alternative-proposals">Alternative Proposals</a><ul> <li><a class="reference internal" href="#propagating-f-in-cleanup-flag-automatically">Propagating ‘f_in_cleanup’ Flag Automatically</a></li> <li><a class="reference internal" href="#add-bytecodes-incr-cleanup-decr-cleanup">Add Bytecodes ‘INCR_CLEANUP’, ‘DECR_CLEANUP’</a></li> <li><a class="reference internal" href="#expose-f-in-cleanup-as-a-counter">Expose ‘f_in_cleanup’ as a Counter</a></li> <li><a class="reference internal" href="#add-code-object-flag-co-cleanup">Add code object flag ‘CO_CLEANUP’</a></li> <li><a class="reference internal" href="#have-cleanup-callback-on-frame-object-itself">Have Cleanup Callback on Frame Object Itself</a></li> <li><a class="reference internal" href="#no-cleanup-hook">No Cleanup Hook</a></li> </ul> </li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> This PEP proposes a way to protect Python code from being interrupted inside a finally clause or during context manager cleanup. </section> <section id="pep-deferral"> <h2><a class="toc-backref" href="#pep-deferral" role="doc-backlink">PEP Deferral</a></h2> Further exploration of the concepts covered in this PEP has been deferred for lack of a current champion interested in promoting the goals of the PEP and collecting and incorporating feedback, and with sufficient available time to do so effectively. </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> Python has two nice ways to do cleanup. One is a <code class="docutils literal notranslate">finally</code> statement and the other is a context manager (usually called using a <code class="docutils literal notranslate">with</code> statement). However, neither is protected from interruption by <code class="docutils literal notranslate">KeyboardInterrupt</code> or <code class="docutils literal notranslate">GeneratorExit</code> caused by <code class="docutils literal notranslate">generator.throw()</code>. For example: <div class="highlight-default notranslate"><div class="highlight"><pre>lock.acquire() try: print('starting') do_something() finally: print('finished') lock.release() </pre></div> </div> If <code class="docutils literal notranslate">KeyboardInterrupt</code> occurs just after the second <code class="docutils literal notranslate">print()</code> call, the lock will not be released. Similarly, the following code using the <code class="docutils literal notranslate">with</code> statement is affected: <div class="highlight-default notranslate"><div class="highlight"><pre>from threading import Lock class MyLock: def __init__(self): self._lock_impl = Lock() def __enter__(self): self._lock_impl.acquire() print("LOCKED") def __exit__(self): print("UNLOCKING") self._lock_impl.release() lock = MyLock() with lock: do_something </pre></div> </div> If <code class="docutils literal notranslate">KeyboardInterrupt</code> occurs near any of the <code class="docutils literal notranslate">print()</code> calls, the lock will never be released. <section id="coroutine-use-case"> <h3><a class="toc-backref" href="#coroutine-use-case" role="doc-backlink">Coroutine Use Case</a></h3> A similar case occurs with coroutines. Usually coroutine libraries want to interrupt the coroutine with a timeout. The <code class="docutils literal notranslate">generator.throw()</code> method works for this use case, but there is no way of knowing if the coroutine is currently suspended from inside a <code class="docutils literal notranslate">finally</code> clause. An example that uses yield-based coroutines follows. The code looks similar using any of the popular coroutine libraries Monocle <a class="footnote-reference brackets" href="#id4" id="id1">[1]</a>, Bluelet <a class="footnote-reference brackets" href="#id5" id="id2">[2]</a>, or Twisted <a class="footnote-reference brackets" href="#id6" id="id3">[3]</a>. <div class="highlight-default notranslate"><div class="highlight"><pre>def run_locked(): yield connection.sendall('LOCK') try: yield do_something() yield do_something_else() finally: yield connection.sendall('UNLOCK') with timeout(5): yield run_locked() </pre></div> </div> In the example above, <code class="docutils literal notranslate">yield something</code> means to pause executing the current coroutine and to execute coroutine <code class="docutils literal notranslate">something</code> until it finishes execution. Therefore, the coroutine library itself needs to maintain a stack of generators. The <code class="docutils literal notranslate">connection.sendall()</code> call waits until the socket is writable and does a similar thing to what <code class="docutils literal notranslate">socket.sendall()</code> does. The <code class="docutils literal notranslate">with</code> statement ensures that all code is executed within 5 seconds timeout. It does so by registering a callback in the main loop, which calls <code class="docutils literal notranslate">generator.throw()</code> on the top-most frame in the coroutine stack when a timeout happens. The <code class="docutils literal notranslate">greenlets</code> extension works in a similar way, except that it doesn’t need <code class="docutils literal notranslate">yield</code> to enter a new stack frame. Otherwise considerations are similar. </section> </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> <section id="frame-flag-f-in-cleanup"> <h3><a class="toc-backref" href="#frame-flag-f-in-cleanup" role="doc-backlink">Frame Flag ‘f_in_cleanup’</a></h3> A new flag on the frame object is proposed. It is set to <code class="docutils literal notranslate">True</code> if this frame is currently executing a <code class="docutils literal notranslate">finally</code> clause. Internally, the flag must be implemented as a counter of nested finally statements currently being executed. The internal counter also needs to be incremented during execution of the <code class="docutils literal notranslate">SETUP_WITH</code> and <code class="docutils literal notranslate">WITH_CLEANUP</code> bytecodes, and decremented when execution for these bytecodes is finished. This allows to also protect <code class="docutils literal notranslate">__enter__()</code> and <code class="docutils literal notranslate">__exit__()</code> methods. </section> <section id="function-sys-setcleanuphook"> <h3><a class="toc-backref" href="#function-sys-setcleanuphook" role="doc-backlink">Function ‘sys.setcleanuphook’</a></h3> A new function for the <code class="docutils literal notranslate">sys</code> module is proposed. This function sets a callback which is executed every time <code class="docutils literal notranslate">f_in_cleanup</code> becomes false. Callbacks get a frame object as their sole argument, so that they can figure out where they are called from. The setting is thread local and must be stored in the <code class="docutils literal notranslate">PyThreadState</code> structure. </section> <section id="inspect-module-enhancements"> <h3><a class="toc-backref" href="#inspect-module-enhancements" role="doc-backlink">Inspect Module Enhancements</a></h3> Two new functions are proposed for the <code class="docutils literal notranslate">inspect</code> module: <code class="docutils literal notranslate">isframeincleanup()</code> and <code class="docutils literal notranslate">getcleanupframe()</code>. <code class="docutils literal notranslate">isframeincleanup()</code>, given a frame or generator object as its sole argument, returns the value of the <code class="docutils literal notranslate">f_in_cleanup</code> attribute of a frame itself or of the <code class="docutils literal notranslate">gi_frame</code> attribute of a generator. <code class="docutils literal notranslate">getcleanupframe()</code>, given a frame object as its sole argument, returns the innermost frame which has a true value of <code class="docutils literal notranslate">f_in_cleanup</code>, or <code class="docutils literal notranslate">None</code> if no frames in the stack have a nonzero value for that attribute. It starts to inspect from the specified frame and walks to outer frames using <code class="docutils literal notranslate">f_back</code> pointers, just like <code class="docutils literal notranslate">getouterframes()</code> does. </section> </section> <section id="example"> <h2><a class="toc-backref" href="#example" role="doc-backlink">Example</a></h2> An example implementation of a SIGINT handler that interrupts safely might look like: <div class="highlight-default notranslate"><div class="highlight"><pre>import inspect, sys, functools def sigint_handler(sig, frame): if inspect.getcleanupframe(frame) is None: raise KeyboardInterrupt() sys.setcleanuphook(functools.partial(sigint_handler, 0)) </pre></div> </div> A coroutine example is out of scope of this document, because its implementation depends very much on a trampoline (or main loop) used by coroutine library. </section> <section id="unresolved-issues"> <h2><a class="toc-backref" href="#unresolved-issues" role="doc-backlink">Unresolved Issues</a></h2> <section id="interruption-inside-with-statement-expression"> <h3><a class="toc-backref" href="#interruption-inside-with-statement-expression" role="doc-backlink">Interruption Inside With Statement Expression</a></h3> Given the statement <div class="highlight-default notranslate"><div class="highlight"><pre>with open(filename): do_something() </pre></div> </div> Python can be interrupted after <code class="docutils literal notranslate">open()</code> is called, but before the <code class="docutils literal notranslate">SETUP_WITH</code> bytecode is executed. There are two possible decisions: <ul> <li>Protect <code class="docutils literal notranslate">with</code> expressions. This would require another bytecode, since currently there is no way of recognizing the start of the <code class="docutils literal notranslate">with</code> expression.</li> <li>Let the user write a wrapper if he considers it important for the use-case. A safe wrapper might look like this:<div class="highlight-default notranslate"><div class="highlight"><pre>class FileWrapper(object): def __init__(self, filename, mode): self.filename = filename self.mode = mode def __enter__(self): self.file = open(self.filename, self.mode) def __exit__(self): self.file.close() </pre></div> </div> Alternatively it can be written using the <code class="docutils literal notranslate">contextmanager()</code> decorator: <div class="highlight-default notranslate"><div class="highlight"><pre>@contextmanager def open_wrapper(filename, mode): file = open(filename, mode) try: yield file finally: file.close() </pre></div> </div> This code is safe, as the first part of the generator (before yield) is executed inside the <code class="docutils literal notranslate">SETUP_WITH</code> bytecode of the caller. </li> </ul> </section> <section id="exception-propagation"> <h3><a class="toc-backref" href="#exception-propagation" role="doc-backlink">Exception Propagation</a></h3> Sometimes a <code class="docutils literal notranslate">finally</code> clause or an <code class="docutils literal notranslate">__enter__()</code>/<code class="docutils literal notranslate">__exit__()</code> method can raise an exception. Usually this is not a problem, since more important exceptions like <code class="docutils literal notranslate">KeyboardInterrupt</code> or <code class="docutils literal notranslate">SystemExit</code> should be raised instead. But it may be nice to be able to keep the original exception inside a <code class="docutils literal notranslate">__context__</code> attribute. So the cleanup hook signature may grow an exception argument: <div class="highlight-default notranslate"><div class="highlight"><pre>def sigint_handler(sig, frame) if inspect.getcleanupframe(frame) is None: raise KeyboardInterrupt() sys.setcleanuphook(retry_sigint) def retry_sigint(frame, exception=None): if inspect.getcleanupframe(frame) is None: raise KeyboardInterrupt() from exception </pre></div> </div> <div class="admonition note"> Note There is no need to have three arguments like in the <code class="docutils literal notranslate">__exit__</code> method since there is a <code class="docutils literal notranslate">__traceback__</code> attribute in exception in Python 3. </div> However, this will set the <code class="docutils literal notranslate">__cause__</code> for the exception, which is not exactly what’s intended. So some hidden interpreter logic may be used to put a <code class="docutils literal notranslate">__context__</code> attribute on every exception raised in a cleanup hook. </section> <section id="interruption-between-acquiring-resource-and-try-block"> <h3><a class="toc-backref" href="#interruption-between-acquiring-resource-and-try-block" role="doc-backlink">Interruption Between Acquiring Resource and Try Block</a></h3> The example from the first section is not totally safe. Let’s take a closer look: <div class="highlight-default notranslate"><div class="highlight"><pre>lock.acquire() try: do_something() finally: lock.release() </pre></div> </div> The problem might occur if the code is interrupted just after <code class="docutils literal notranslate">lock.acquire()</code> is executed but before the <code class="docutils literal notranslate">try</code> block is entered. There is no way the code can be fixed unmodified. The actual fix depends very much on the use case. Usually code can be fixed using a <code class="docutils literal notranslate">with</code> statement: <div class="highlight-default notranslate"><div class="highlight"><pre>with lock: do_something() </pre></div> </div> However, for coroutines one usually can’t use the <code class="docutils literal notranslate">with</code> statement because you need to <code class="docutils literal notranslate">yield</code> for both the acquire and release operations. So the code might be rewritten like this: <div class="highlight-default notranslate"><div class="highlight"><pre>try: yield lock.acquire() do_something() finally: yield lock.release() </pre></div> </div> The actual locking code might need more code to support this use case, but the implementation is usually trivial, like this: check if the lock has been acquired and unlock if it is. </section> <section id="handling-eintr-inside-a-finally"> <h3><a class="toc-backref" href="#handling-eintr-inside-a-finally" role="doc-backlink">Handling EINTR Inside a Finally</a></h3> Even if a signal handler is prepared to check the <code class="docutils literal notranslate">f_in_cleanup</code> flag, <code class="docutils literal notranslate">InterruptedError</code> might be raised in the cleanup handler, because the respective system call returned an <code class="docutils literal notranslate">EINTR</code> error. The primary use cases are prepared to handle this: <ul class="simple"> <li>Posix mutexes never return <code class="docutils literal notranslate">EINTR</code></li> <li>Networking libraries are always prepared to handle <code class="docutils literal notranslate">EINTR</code></li> <li>Coroutine libraries are usually interrupted with the <code class="docutils literal notranslate">throw()</code> method, not with a signal</li> </ul> The platform-specific function <code class="docutils literal notranslate">siginterrupt()</code> might be used to remove the need to handle <code class="docutils literal notranslate">EINTR</code>. However, it may have hardly predictable consequences, for example <code class="docutils literal notranslate">SIGINT</code> a handler is never called if the main thread is stuck inside an IO routine. A better approach would be to have the code, which is usually used in cleanup handlers, be prepared to handle <code class="docutils literal notranslate">InterruptedError</code> explicitly. An example of such code might be a file-based lock implementation. <code class="docutils literal notranslate">signal.pthread_sigmask</code> can be used to block signals inside cleanup handlers which can be interrupted with <code class="docutils literal notranslate">EINTR</code>. </section> <section id="setting-interruption-context-inside-finally-itself"> <h3><a class="toc-backref" href="#setting-interruption-context-inside-finally-itself" role="doc-backlink">Setting Interruption Context Inside Finally Itself</a></h3> Some coroutine libraries may need to set a timeout for the finally clause itself. For example: <div class="highlight-default notranslate"><div class="highlight"><pre>try: do_something() finally: with timeout(0.5): try: yield do_slow_cleanup() finally: yield do_fast_cleanup() </pre></div> </div> With current semantics, timeout will either protect the whole <code class="docutils literal notranslate">with</code> block or nothing at all, depending on the implementation of each library. What the author intended is to treat <code class="docutils literal notranslate">do_slow_cleanup</code> as ordinary code, and <code class="docutils literal notranslate">do_fast_cleanup</code> as a cleanup (a non-interruptible one). A similar case might occur when using greenlets or tasklets. This case can be fixed by exposing <code class="docutils literal notranslate">f_in_cleanup</code> as a counter, and by calling a cleanup hook on each decrement. A coroutine library may then remember the value at timeout start, and compare it on each hook execution. But in practice, the example is considered to be too obscure to take into account. </section> <section id="modifying-keyboardinterrupt"> <h3><a class="toc-backref" href="#modifying-keyboardinterrupt" role="doc-backlink">Modifying KeyboardInterrupt</a></h3> It should be decided if the default <code class="docutils literal notranslate">SIGINT</code> handler should be modified to use the described mechanism. The initial proposition is to keep old behavior, for two reasons: <ul class="simple"> <li>Most application do not care about cleanup on exit (either they do not have external state, or they modify it in crash-safe way).</li> <li>Cleanup may take too much time, not giving user a chance to interrupt an application.</li> </ul> The latter case can be fixed by allowing an unsafe break if a <code class="docutils literal notranslate">SIGINT</code> handler is called twice, but it seems not worth the complexity. </section> </section> <section id="alternative-python-implementations-support"> <h2><a class="toc-backref" href="#alternative-python-implementations-support" role="doc-backlink">Alternative Python Implementations Support</a></h2> We consider <code class="docutils literal notranslate">f_in_cleanup</code> an implementation detail. The actual implementation may have some fake frame-like object passed to signal handler, cleanup hook and returned from <code class="docutils literal notranslate">getcleanupframe()</code>. The only requirement is that the <code class="docutils literal notranslate">inspect</code> module functions work as expected on these objects. For this reason, we also allow to pass a generator object to the <code class="docutils literal notranslate">isframeincleanup()</code> function, which removes the need to use the <code class="docutils literal notranslate">gi_frame</code> attribute. It might be necessary to specify that <code class="docutils literal notranslate">getcleanupframe()</code> must return the same object that will be passed to cleanup hook at the next invocation. </section> <section id="alternative-names"> <h2><a class="toc-backref" href="#alternative-names" role="doc-backlink">Alternative Names</a></h2> The original proposal had a <code class="docutils literal notranslate">f_in_finally</code> frame attribute, as the original intention was to protect <code class="docutils literal notranslate">finally</code> clauses. But as it grew up to protecting <code class="docutils literal notranslate">__enter__</code> and <code class="docutils literal notranslate">__exit__</code> methods too, the <code class="docutils literal notranslate">f_in_cleanup</code> name seems better. Although the <code class="docutils literal notranslate">__enter__</code> method is not a cleanup routine, it at least relates to cleanup done by context managers. <code class="docutils literal notranslate">setcleanuphook</code>, <code class="docutils literal notranslate">isframeincleanup</code> and <code class="docutils literal notranslate">getcleanupframe</code> can be unobscured to <code class="docutils literal notranslate">set_cleanup_hook</code>, <code class="docutils literal notranslate">is_frame_in_cleanup</code> and <code class="docutils literal notranslate">get_cleanup_frame</code>, although they follow the naming convention of their respective modules. </section> <section id="alternative-proposals"> <h2><a class="toc-backref" href="#alternative-proposals" role="doc-backlink">Alternative Proposals</a></h2> <section id="propagating-f-in-cleanup-flag-automatically"> <h3><a class="toc-backref" href="#propagating-f-in-cleanup-flag-automatically" role="doc-backlink">Propagating ‘f_in_cleanup’ Flag Automatically</a></h3> This can make <code class="docutils literal notranslate">getcleanupframe()</code> unnecessary. But for yield-based coroutines you need to propagate it yourself. Making it writable leads to somewhat unpredictable behavior of <code class="docutils literal notranslate">setcleanuphook()</code>. </section> <section id="add-bytecodes-incr-cleanup-decr-cleanup"> <h3><a class="toc-backref" href="#add-bytecodes-incr-cleanup-decr-cleanup" role="doc-backlink">Add Bytecodes ‘INCR_CLEANUP’, ‘DECR_CLEANUP’</a></h3> These bytecodes can be used to protect the expression inside the <code class="docutils literal notranslate">with</code> statement, as well as making counter increments more explicit and easy to debug (visible inside a disassembly). Some middle ground might be chosen, like <code class="docutils literal notranslate">END_FINALLY</code> and <code class="docutils literal notranslate">SETUP_WITH</code> implicitly decrementing the counter (<code class="docutils literal notranslate">END_FINALLY</code> is present at end of every <code class="docutils literal notranslate">with</code> suite). However, adding new bytecodes must be considered very carefully. </section> <section id="expose-f-in-cleanup-as-a-counter"> <h3><a class="toc-backref" href="#expose-f-in-cleanup-as-a-counter" role="doc-backlink">Expose ‘f_in_cleanup’ as a Counter</a></h3> The original intention was to expose a minimum of needed functionality. However, as we consider the frame flag <code class="docutils literal notranslate">f_in_cleanup</code> an implementation detail, we may expose it as a counter. Similarly, if we have a counter we may need to have the cleanup hook called on every counter decrement. It’s unlikely to have much performance impact as nested finally clauses are an uncommon case. </section> <section id="add-code-object-flag-co-cleanup"> <h3><a class="toc-backref" href="#add-code-object-flag-co-cleanup" role="doc-backlink">Add code object flag ‘CO_CLEANUP’</a></h3> As an alternative to set the flag inside the <code class="docutils literal notranslate">SETUP_WITH</code> and <code class="docutils literal notranslate">WITH_CLEANUP</code> bytecodes, we can introduce a flag <code class="docutils literal notranslate">CO_CLEANUP</code>. When the interpreter starts to execute code with <code class="docutils literal notranslate">CO_CLEANUP</code> set, it sets <code class="docutils literal notranslate">f_in_cleanup</code> for the whole function body. This flag is set for code objects of <code class="docutils literal notranslate">__enter__</code> and <code class="docutils literal notranslate">__exit__</code> special methods. Technically it might be set on functions called <code class="docutils literal notranslate">__enter__</code> and <code class="docutils literal notranslate">__exit__</code>. This seems to be less clear solution. It also covers the case where <code class="docutils literal notranslate">__enter__</code> and <code class="docutils literal notranslate">__exit__</code> are called manually. This may be accepted either as a feature or as an unnecessary side-effect (or, though unlikely, as a bug). It may also impose a problem when <code class="docutils literal notranslate">__enter__</code> or <code class="docutils literal notranslate">__exit__</code> functions are implemented in C, as there is no code object to check for the <code class="docutils literal notranslate">f_in_cleanup</code> flag. </section> <section id="have-cleanup-callback-on-frame-object-itself"> <h3><a class="toc-backref" href="#have-cleanup-callback-on-frame-object-itself" role="doc-backlink">Have Cleanup Callback on Frame Object Itself</a></h3> The frame object may be extended to have a <code class="docutils literal notranslate">f_cleanup_callback</code> member which is called when <code class="docutils literal notranslate">f_in_cleanup</code> is reset to 0. This would help to register different callbacks to different coroutines. Despite its apparent beauty, this solution doesn’t add anything, as the two primary use cases are: <ul class="simple"> <li>Setting the callback in a signal handler. The callback is inherently a single one for this case.</li> <li>Use a single callback per loop for the coroutine use case. Here, in almost all cases, there is only one loop per thread.</li> </ul> </section> <section id="no-cleanup-hook"> <h3><a class="toc-backref" href="#no-cleanup-hook" role="doc-backlink">No Cleanup Hook</a></h3> The original proposal included no cleanup hook specification, as there are a few ways to achieve the same using current tools: <ul class="simple"> <li>Using <code class="docutils literal notranslate">sys.settrace()</code> and the <code class="docutils literal notranslate">f_trace</code> callback. This may impose some problem to debugging, and has a big performance impact (although interrupting doesn’t happen very often).</li> <li>Sleeping a bit more and trying again. For a coroutine library this is easy. For signals it may be achieved using <code class="docutils literal notranslate">signal.alert</code>.</li> </ul> Both methods are considered too impractical and a way to catch exit from <code class="docutils literal notranslate">finally</code> clauses is proposed. </section> </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <aside class="footnote-list brackets"> <aside class="footnote brackets" id="id4" role="doc-footnote"> <dt class="label" id="id4">[<a href="#id1">1</a>]</dt> <dd>Monocle <a class="reference external" href="https://github.com/saucelabs/monocle">https://github.com/saucelabs/monocle</a></aside> <aside class="footnote brackets" id="id5" role="doc-footnote"> <dt class="label" id="id5">[<a href="#id2">2</a>]</dt> <dd>Bluelet <a class="reference external" href="https://github.com/sampsyo/bluelet">https://github.com/sampsyo/bluelet</a></aside> <aside class="footnote brackets" id="id6" role="doc-footnote"> <dt class="label" id="id6">[<a href="#id3">3</a>]</dt> <dd>Twisted: inlineCallbacks <a class="reference external" href="https://twisted.org/documents/8.1.0/api/twisted.internet.defer.html">https://twisted.org/documents/8.1.0/api/twisted.internet.defer.html</a></aside> </aside> [4] Original discussion <a class="reference external" href="https://mail.python.org/pipermail/python-ideas/2012-April/014705.html">https://mail.python.org/pipermail/python-ideas/2012-April/014705.html</a> [5] Implementation of PEP 419 <a class="reference external" href="https://github.com/python/cpython/issues/58935">https://github.com/python/cpython/issues/58935</a> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0419.rst">https://github.com/python/peps/blob/main/peps/pep-0419.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0419.rst">2025-02-01 08:59:27 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#pep-deferral">PEP Deferral</a></li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#coroutine-use-case">Coroutine Use Case</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#frame-flag-f-in-cleanup">Frame Flag ‘f_in_cleanup’</a></li> <li><a class="reference internal" href="#function-sys-setcleanuphook">Function ‘sys.setcleanuphook’</a></li> <li><a class="reference internal" href="#inspect-module-enhancements">Inspect Module Enhancements</a></li> </ul> </li> <li><a class="reference internal" href="#example">Example</a></li> <li><a class="reference internal" href="#unresolved-issues">Unresolved Issues</a><ul> <li><a class="reference internal" href="#interruption-inside-with-statement-expression">Interruption Inside With Statement Expression</a></li> <li><a class="reference internal" href="#exception-propagation">Exception Propagation</a></li> <li><a class="reference internal" href="#interruption-between-acquiring-resource-and-try-block">Interruption Between Acquiring Resource and Try Block</a></li> <li><a class="reference internal" href="#handling-eintr-inside-a-finally">Handling EINTR Inside a Finally</a></li> <li><a class="reference internal" href="#setting-interruption-context-inside-finally-itself">Setting Interruption Context Inside Finally Itself</a></li> <li><a class="reference internal" href="#modifying-keyboardinterrupt">Modifying KeyboardInterrupt</a></li> </ul> </li> <li><a class="reference internal" href="#alternative-python-implementations-support">Alternative Python Implementations Support</a></li> <li><a class="reference internal" href="#alternative-names">Alternative Names</a></li> <li><a class="reference internal" href="#alternative-proposals">Alternative Proposals</a><ul> <li><a class="reference internal" href="#propagating-f-in-cleanup-flag-automatically">Propagating ‘f_in_cleanup’ Flag Automatically</a></li> <li><a class="reference internal" href="#add-bytecodes-incr-cleanup-decr-cleanup">Add Bytecodes ‘INCR_CLEANUP’, ‘DECR_CLEANUP’</a></li> <li><a class="reference internal" href="#expose-f-in-cleanup-as-a-counter">Expose ‘f_in_cleanup’ as a Counter</a></li> <li><a class="reference internal" href="#add-code-object-flag-co-cleanup">Add code object flag ‘CO_CLEANUP’</a></li> <li><a class="reference internal" href="#have-cleanup-callback-on-frame-object-itself">Have Cleanup Callback on Frame Object Itself</a></li> <li><a class="reference internal" href="#no-cleanup-hook">No Cleanup Hook</a></li> </ul> </li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0419.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>