CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 377 – Allow __enter__() methods to skip the statement body | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0377/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 377 – Allow __enter__() methods to skip the statement body | peps.python.org'> <meta property="og:description" content="This PEP proposes a backwards compatible mechanism that allows __enter__() methods to skip the body of the associated with statement. The lack of this ability currently means the contextlib.contextmanager decorator is unable to fulfil its specification ..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0377/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This PEP proposes a backwards compatible mechanism that allows __enter__() methods to skip the body of the associated with statement. The lack of this ability currently means the contextlib.contextmanager decorator is unable to fulfil its specification ..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 377</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 377 – Allow __enter__() methods to skip the statement body</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Alyssa Coghlan <ncoghlan at gmail.com></dd> <dt class="field-even">Status:</dt> <dd class="field-even"><abbr title="Formally declined and will not be accepted">Rejected</abbr></dd> <dt class="field-odd">Type:</dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created:</dt> <dd class="field-even">08-Mar-2009</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">2.7, 3.1</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even">08-Mar-2009</dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#pep-rejection">PEP Rejection</a></li> <li><a class="reference internal" href="#proposed-change">Proposed Change</a></li> <li><a class="reference internal" href="#rationale-for-change">Rationale for Change</a></li> <li><a class="reference internal" href="#performance-impact">Performance Impact</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#acknowledgements">Acknowledgements</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> This PEP proposes a backwards compatible mechanism that allows <code class="docutils literal notranslate">__enter__()</code> methods to skip the body of the associated <code class="docutils literal notranslate">with</code> statement. The lack of this ability currently means the <code class="docutils literal notranslate">contextlib.contextmanager</code> decorator is unable to fulfil its specification of being able to turn arbitrary code into a context manager by moving it into a generator function with a yield in the appropriate location. One symptom of this is that <code class="docutils literal notranslate">contextlib.nested</code> will currently raise <code class="docutils literal notranslate">RuntimeError</code> in situations where writing out the corresponding nested <code class="docutils literal notranslate">with</code> statements would not <a class="footnote-reference brackets" href="#id5" id="id1">[1]</a>. The proposed change is to introduce a new flow control exception <code class="docutils literal notranslate">SkipStatement</code>, and skip the execution of the <code class="docutils literal notranslate">with</code> statement body if <code class="docutils literal notranslate">__enter__()</code> raises this exception. </section> <section id="pep-rejection"> <h2><a class="toc-backref" href="#pep-rejection" role="doc-backlink">PEP Rejection</a></h2> This PEP was rejected by Guido <a class="footnote-reference brackets" href="#id7" id="id2">[4]</a> as it imposes too great an increase in complexity without a proportional increase in expressiveness and correctness. In the absence of compelling use cases that need the more complex semantics proposed by this PEP the existing behaviour is considered acceptable. </section> <section id="proposed-change"> <h2><a class="toc-backref" href="#proposed-change" role="doc-backlink">Proposed Change</a></h2> The semantics of the <code class="docutils literal notranslate">with</code> statement will be changed to include a new <code class="docutils literal notranslate">try</code>/<code class="docutils literal notranslate">except</code>/<code class="docutils literal notranslate">else</code> block around the call to <code class="docutils literal notranslate">__enter__()</code>. If <code class="docutils literal notranslate">SkipStatement</code> is raised by the <code class="docutils literal notranslate">__enter__()</code> method, then the main section of the <code class="docutils literal notranslate">with</code> statement (now located in the <code class="docutils literal notranslate">else</code> clause) will not be executed. To avoid leaving the names in any <code class="docutils literal notranslate">as</code> clause unbound in this case, a new <code class="docutils literal notranslate">StatementSkipped</code> singleton (similar to the existing <code class="docutils literal notranslate">NotImplemented</code> singleton) will be assigned to all names that appear in the <code class="docutils literal notranslate">as</code> clause. The components of the <code class="docutils literal notranslate">with</code> statement remain as described in <a class="pep reference internal" href="../pep-0343/" title="PEP 343 – The “with” Statement">PEP 343</a>: <div class="highlight-default notranslate"><div class="highlight"><pre>with EXPR as VAR: BLOCK </pre></div> </div> After the modification, the <code class="docutils literal notranslate">with</code> statement semantics would be as follows: <div class="highlight-default notranslate"><div class="highlight"><pre>mgr = (EXPR) exit = mgr.__exit__ # Not calling it yet try: value = mgr.__enter__() except SkipStatement: VAR = StatementSkipped # Only if "as VAR" is present and # VAR is a single name # If VAR is a tuple of names, then StatementSkipped # will be assigned to each name in the tuple else: exc = True try: try: VAR = value # Only if "as VAR" is present BLOCK except: # The exceptional case is handled here exc = False if not exit(*sys.exc_info()): raise # The exception is swallowed if exit() returns true finally: # The normal and non-local-goto cases are handled here if exc: exit(None, None, None) </pre></div> </div> With the above change in place for the <code class="docutils literal notranslate">with</code> statement semantics, <code class="docutils literal notranslate">contextlib.contextmanager()</code> will then be modified to raise <code class="docutils literal notranslate">SkipStatement</code> instead of <code class="docutils literal notranslate">RuntimeError</code> when the underlying generator doesn’t yield. </section> <section id="rationale-for-change"> <h2><a class="toc-backref" href="#rationale-for-change" role="doc-backlink">Rationale for Change</a></h2> Currently, some apparently innocuous context managers may raise <code class="docutils literal notranslate">RuntimeError</code> when executed. This occurs when the context manager’s <code class="docutils literal notranslate">__enter__()</code> method encounters a situation where the written out version of the code corresponding to the context manager would skip the code that is now the body of the <code class="docutils literal notranslate">with</code> statement. Since the <code class="docutils literal notranslate">__enter__()</code> method has no mechanism available to signal this to the interpreter, it is instead forced to raise an exception that not only skips the body of the <code class="docutils literal notranslate">with</code> statement, but also jumps over all code until the nearest exception handler. This goes against one of the design goals of the <code class="docutils literal notranslate">with</code> statement, which was to be able to factor out arbitrary common exception handling code into a single context manager by putting into a generator function and replacing the variant part of the code with a <code class="docutils literal notranslate">yield</code> statement. Specifically, the following examples behave differently if <code class="docutils literal notranslate">cmB().__enter__()</code> raises an exception which <code class="docutils literal notranslate">cmA().__exit__()</code> then handles and suppresses: <div class="highlight-default notranslate"><div class="highlight"><pre>with cmA(): with cmB(): do_stuff() # This will resume here without executing "do_stuff()" @contextlib.contextmanager def combined(): with cmA(): with cmB(): yield with combined(): do_stuff() # This will raise a RuntimeError complaining that the context # manager's underlying generator didn't yield with contextlib.nested(cmA(), cmB()): do_stuff() # This will raise the same RuntimeError as the contextmanager() # example (unsurprising, given that the nested() implementation # uses contextmanager()) # The following class based version shows that the issue isn't # specific to contextlib.contextmanager() (it also shows how # much simpler it is to write context managers as generators # instead of as classes!) class CM(object): def __init__(self): self.cmA = None self.cmB = None def __enter__(self): if self.cmA is not None: raise RuntimeError("Can't re-use this CM") self.cmA = cmA() self.cmA.__enter__() try: self.cmB = cmB() self.cmB.__enter__() except: self.cmA.__exit__(*sys.exc_info()) # Can't suppress in __enter__(), so must raise raise def __exit__(self, *args): suppress = False try: if self.cmB is not None: suppress = self.cmB.__exit__(*args) except: suppress = self.cmA.__exit__(*sys.exc_info()): if not suppress: # Exception has changed, so reraise explicitly raise else: if suppress: # cmB already suppressed the exception, # so don't pass it to cmA suppress = self.cmA.__exit__(None, None, None): else: suppress = self.cmA.__exit__(*args): return suppress </pre></div> </div> With the proposed semantic change in place, the contextlib based examples above would then “just work”, but the class based version would need a small adjustment to take advantage of the new semantics: <div class="highlight-default notranslate"><div class="highlight"><pre>class CM(object): def __init__(self): self.cmA = None self.cmB = None def __enter__(self): if self.cmA is not None: raise RuntimeError("Can't re-use this CM") self.cmA = cmA() self.cmA.__enter__() try: self.cmB = cmB() self.cmB.__enter__() except: if self.cmA.__exit__(*sys.exc_info()): # Suppress the exception, but don't run # the body of the with statement either raise SkipStatement raise def __exit__(self, *args): suppress = False try: if self.cmB is not None: suppress = self.cmB.__exit__(*args) except: suppress = self.cmA.__exit__(*sys.exc_info()): if not suppress: # Exception has changed, so reraise explicitly raise else: if suppress: # cmB already suppressed the exception, # so don't pass it to cmA suppress = self.cmA.__exit__(None, None, None): else: suppress = self.cmA.__exit__(*args): return suppress </pre></div> </div> There is currently a tentative suggestion <a class="footnote-reference brackets" href="#id6" id="id3">[3]</a> to add import-style syntax to the <code class="docutils literal notranslate">with</code> statement to allow multiple context managers to be included in a single <code class="docutils literal notranslate">with</code> statement without needing to use <code class="docutils literal notranslate">contextlib.nested</code>. In that case the compiler has the option of simply emitting multiple <code class="docutils literal notranslate">with</code> statements at the AST level, thus allowing the semantics of actual nested <code class="docutils literal notranslate">with</code> statements to be reproduced accurately. However, such a change would highlight rather than alleviate the problem the current PEP aims to address: it would not be possible to use <code class="docutils literal notranslate">contextlib.contextmanager</code> to reliably factor out such <code class="docutils literal notranslate">with</code> statements, as they would exhibit exactly the same semantic differences as are seen with the <code class="docutils literal notranslate">combined()</code> context manager in the above example. </section> <section id="performance-impact"> <h2><a class="toc-backref" href="#performance-impact" role="doc-backlink">Performance Impact</a></h2> Implementing the new semantics makes it necessary to store the references to the <code class="docutils literal notranslate">__enter__</code> and <code class="docutils literal notranslate">__exit__</code> methods in temporary variables instead of on the stack. This results in a slight regression in <code class="docutils literal notranslate">with</code> statement speed relative to Python 2.6/3.1. However, implementing a custom <code class="docutils literal notranslate">SETUP_WITH</code> opcode would negate any differences between the two approaches (as well as dramatically improving speed by eliminating more than a dozen unnecessary trips around the eval loop). </section> <section id="reference-implementation"> <h2><a class="toc-backref" href="#reference-implementation" role="doc-backlink">Reference Implementation</a></h2> Patch attached to Issue 5251 <a class="footnote-reference brackets" href="#id5" id="id4">[1]</a>. That patch uses only existing opcodes (i.e. no <code class="docutils literal notranslate">SETUP_WITH</code>). </section> <section id="acknowledgements"> <h2><a class="toc-backref" href="#acknowledgements" role="doc-backlink">Acknowledgements</a></h2> James William Pye both raised the issue and suggested the basic outline of the solution described in this PEP. </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <aside class="footnote-list brackets"> <aside class="footnote brackets" id="id5" role="doc-footnote"> <dt class="label" id="id5">[1] (<a href='#id1'>1</a>, <a href='#id4'>2</a>) </dt> <dd>Issue 5251: contextlib.nested inconsistent with nested with statements (<a class="reference external" href="http://bugs.python.org/issue5251">http://bugs.python.org/issue5251</a>)</aside> <aside class="footnote brackets" id="id6" role="doc-footnote"> <dt class="label" id="id6">[<a href="#id3">3</a>]</dt> <dd>Import-style syntax to reduce indentation of nested with statements (<a class="reference external" href="https://mail.python.org/pipermail/python-ideas/2009-March/003188.html">https://mail.python.org/pipermail/python-ideas/2009-March/003188.html</a>)</aside> <aside class="footnote brackets" id="id7" role="doc-footnote"> <dt class="label" id="id7">[<a href="#id2">4</a>]</dt> <dd>Guido’s rejection of the PEP (<a class="reference external" href="https://mail.python.org/pipermail/python-dev/2009-March/087263.html">https://mail.python.org/pipermail/python-dev/2009-March/087263.html</a>)</aside> </aside> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0377.rst">https://github.com/python/peps/blob/main/peps/pep-0377.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0377.rst">2023-10-11 12:05:51 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#pep-rejection">PEP Rejection</a></li> <li><a class="reference internal" href="#proposed-change">Proposed Change</a></li> <li><a class="reference internal" href="#rationale-for-change">Rationale for Change</a></li> <li><a class="reference internal" href="#performance-impact">Performance Impact</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#acknowledgements">Acknowledgements</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0377.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>