CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 533 – Deterministic cleanup for iterators | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0533/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 533 – Deterministic cleanup for iterators | peps.python.org'> <meta property="og:description" content="We propose to extend the iterator protocol with a new __(a)iterclose__ slot, which is called automatically on exit from (async) for loops, regardless of how they exit. This allows for convenient, deterministic cleanup of resources held by iterators with..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0533/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="We propose to extend the iterator protocol with a new __(a)iterclose__ slot, which is called automatically on exit from (async) for loops, regardless of how they exit. This allows for convenient, deterministic cleanup of resources held by iterators with..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 533</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 533 – Deterministic cleanup for iterators</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Nathaniel J. Smith</dd> <dt class="field-even">BDFL-Delegate:</dt> <dd class="field-even">Yury Selivanov <yury at edgedb.com></dd> <dt class="field-odd">Status:</dt> <dd class="field-odd"><abbr title="Inactive draft that may be taken up again at a later time">Deferred</abbr></dd> <dt class="field-even">Type:</dt> <dd class="field-even"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-odd">Created:</dt> <dd class="field-odd">18-Oct-2016</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even">18-Oct-2016</dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#note-on-timing">Note on timing</a></li> <li><a class="reference internal" href="#background-and-motivation">Background and motivation</a></li> <li><a class="reference internal" href="#alternatives">Alternatives</a><ul> <li><a class="reference internal" href="#pep-525-asyncgen-hooks">PEP 525 asyncgen hooks</a></li> <li><a class="reference internal" href="#always-inject-resources-and-do-all-cleanup-at-the-top-level">Always inject resources, and do all cleanup at the top level</a></li> <li><a class="reference internal" href="#more-complex-variants-of-a-iterclose">More complex variants of __(a)iterclose__</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#guiding-principles">Guiding principles</a></li> <li><a class="reference internal" href="#changes-to-iteration">Changes to iteration</a></li> <li><a class="reference internal" href="#changes-to-async-iteration">Changes to async iteration</a></li> <li><a class="reference internal" href="#modifications-to-basic-iterator-types">Modifications to basic iterator types</a></li> <li><a class="reference internal" href="#new-convenience-functions">New convenience functions</a></li> <li><a class="reference internal" href="#iterclose-implementations-for-iterator-wrappers">__iterclose__ implementations for iterator wrappers</a></li> <li><a class="reference internal" href="#example-rationale">Example / Rationale</a></li> </ul> </li> <li><a class="reference internal" href="#transition-plan">Transition plan</a></li> <li><a class="reference internal" href="#acknowledgements">Acknowledgements</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> We propose to extend the iterator protocol with a new <code class="docutils literal notranslate">__(a)iterclose__</code> slot, which is called automatically on exit from <code class="docutils literal notranslate">(async) for</code> loops, regardless of how they exit. This allows for convenient, deterministic cleanup of resources held by iterators without reliance on the garbage collector. This is especially valuable for asynchronous generators. </section> <section id="note-on-timing"> <h2><a class="toc-backref" href="#note-on-timing" role="doc-backlink">Note on timing</a></h2> In practical terms, the proposal here is divided into two separate parts: the handling of async iterators, which should ideally be implemented ASAP, and the handling of regular iterators, which is a larger but more relaxed project that can’t start until 3.7 at the earliest. But since the changes are closely related, and we probably don’t want to end up with async iterators and regular iterators diverging in the long run, it seems useful to look at them together. </section> <section id="background-and-motivation"> <h2><a class="toc-backref" href="#background-and-motivation" role="doc-backlink">Background and motivation</a></h2> Python iterables often hold resources which require cleanup. For example: <code class="docutils literal notranslate">file</code> objects need to be closed; the <a class="pep reference internal" href="../pep-0333/" title="PEP 333 – Python Web Server Gateway Interface v1.0">WSGI spec</a> adds a <code class="docutils literal notranslate">close</code> method on top of the regular iterator protocol and demands that consumers call it at the appropriate time (though forgetting to do so is a <a class="reference external" href="http://blog.dscpl.com.au/2012/10/obligations-for-calling-close-on.html">frequent source of bugs</a>); and <a class="pep reference internal" href="../pep-0342/" title="PEP 342 – Coroutines via Enhanced Generators">PEP 342</a> (based on <a class="pep reference internal" href="../pep-0325/" title="PEP 325 – Resource-Release Support for Generators">PEP 325</a>) extended generator objects to add a <code class="docutils literal notranslate">close</code> method to allow generators to clean up after themselves. Generally, objects that need to clean up after themselves also define a <code class="docutils literal notranslate">__del__</code> method to ensure that this cleanup will happen eventually, when the object is garbage collected. However, relying on the garbage collector for cleanup like this causes serious problems in several cases: <ul class="simple"> <li>In Python implementations that do not use reference counting (e.g. PyPy, Jython), calls to <code class="docutils literal notranslate">__del__</code> may be arbitrarily delayed – yet many situations require prompt cleanup of resources. Delayed cleanup produces problems like crashes due to file descriptor exhaustion, or WSGI timing middleware that collects bogus times.</li> <li>Async generators (<a class="pep reference internal" href="../pep-0525/" title="PEP 525 – Asynchronous Generators">PEP 525</a>) can only perform cleanup under the supervision of the appropriate coroutine runner. <code class="docutils literal notranslate">__del__</code> doesn’t have access to the coroutine runner; indeed, the coroutine runner might be garbage collected before the generator object. So relying on the garbage collector is effectively impossible without some kind of language extension. (<a class="pep reference internal" href="../pep-0525/" title="PEP 525 – Asynchronous Generators">PEP 525</a> does provide such an extension, but it has a number of limitations that this proposal fixes; see the “alternatives” section below for discussion.)</li> </ul> Fortunately, Python provides a standard tool for doing resource cleanup in a more structured way: <code class="docutils literal notranslate">with</code> blocks. For example, this code opens a file but relies on the garbage collector to close it: <div class="highlight-default notranslate"><div class="highlight"><pre>def read_newline_separated_json(path): for line in open(path): yield json.loads(line) for document in read_newline_separated_json(path): ... </pre></div> </div> and recent versions of CPython will point this out by issuing a <code class="docutils literal notranslate">ResourceWarning</code>, nudging us to fix it by adding a <code class="docutils literal notranslate">with</code> block: <div class="highlight-default notranslate"><div class="highlight"><pre>def read_newline_separated_json(path): with open(path) as file_handle: # <-- with block for line in file_handle: yield json.loads(line) for document in read_newline_separated_json(path): # <-- outer for loop ... </pre></div> </div> But there’s a subtlety here, caused by the interaction of <code class="docutils literal notranslate">with</code> blocks and generators. <code class="docutils literal notranslate">with</code> blocks are Python’s main tool for managing cleanup, and they’re a powerful one, because they pin the lifetime of a resource to the lifetime of a stack frame. But this assumes that someone will take care of cleaning up the stack frame… and for generators, this requires that someone <code class="docutils literal notranslate">close</code> them. In this case, adding the <code class="docutils literal notranslate">with</code> block is enough to shut up the <code class="docutils literal notranslate">ResourceWarning</code>, but this is misleading – the file object cleanup here is still dependent on the garbage collector. The <code class="docutils literal notranslate">with</code> block will only be unwound when the <code class="docutils literal notranslate">read_newline_separated_json</code> generator is closed. If the outer <code class="docutils literal notranslate">for</code> loop runs to completion then the cleanup will happen immediately; but if this loop is terminated early by a <code class="docutils literal notranslate">break</code> or an exception, then the <code class="docutils literal notranslate">with</code> block won’t fire until the generator object is garbage collected. The correct solution requires that all users of this API wrap every <code class="docutils literal notranslate">for</code> loop in its own <code class="docutils literal notranslate">with</code> block: <div class="highlight-default notranslate"><div class="highlight"><pre>with closing(read_newline_separated_json(path)) as genobj: for document in genobj: ... </pre></div> </div> This gets even worse if we consider the idiom of decomposing a complex pipeline into multiple nested generators: <div class="highlight-default notranslate"><div class="highlight"><pre>def read_users(path): with closing(read_newline_separated_json(path)) as gen: for document in gen: yield User.from_json(document) def users_in_group(path, group): with closing(read_users(path)) as gen: for user in gen: if user.group == group: yield user </pre></div> </div> In general if you have N nested generators then you need N+1 <code class="docutils literal notranslate">with</code> blocks to clean up 1 file. And good defensive programming would suggest that any time we use a generator, we should assume the possibility that there could be at least one <code class="docutils literal notranslate">with</code> block somewhere in its (potentially transitive) call stack, either now or in the future, and thus always wrap it in a <code class="docutils literal notranslate">with</code>. But in practice, basically nobody does this, because programmers would rather write buggy code than tiresome repetitive code. In simple cases like this there are some workarounds that good Python developers know (e.g. in this simple case it would be idiomatic to pass in a file handle instead of a path and move the resource management to the top level), but in general we cannot avoid the use of <code class="docutils literal notranslate">with</code>/<code class="docutils literal notranslate">finally</code> inside of generators, and thus dealing with this problem one way or another. When beauty and correctness fight then beauty tends to win, so it’s important to make correct code beautiful. Still, is this worth fixing? Until async generators came along I would have argued yes, but that it was a low priority, since everyone seems to be muddling along okay – but async generators make it much more urgent. Async generators cannot do cleanup at all without some mechanism for deterministic cleanup that people will actually use, and async generators are particularly likely to hold resources like file descriptors. (After all, if they weren’t doing I/O, they’d be generators, not async generators.) So we have to do something, and it might as well be a comprehensive fix to the underlying problem. And it’s much easier to fix this now when async generators are first rolling out, than it will be to fix it later. The proposal itself is simple in concept: add a <code class="docutils literal notranslate">__(a)iterclose__</code> method to the iterator protocol, and have (async) <code class="docutils literal notranslate">for</code> loops call it when the loop is exited, even if this occurs via <code class="docutils literal notranslate">break</code> or exception unwinding. Effectively, we’re taking the current cumbersome idiom (<code class="docutils literal notranslate">with</code> block + <code class="docutils literal notranslate">for</code> loop) and merging them together into a fancier <code class="docutils literal notranslate">for</code>. This may seem non-orthogonal, but makes sense when you consider that the existence of generators means that <code class="docutils literal notranslate">with</code> blocks actually depend on iterator cleanup to work reliably, plus experience showing that iterator cleanup is often a desirable feature in its own right. </section> <section id="alternatives"> <h2><a class="toc-backref" href="#alternatives" role="doc-backlink">Alternatives</a></h2> <section id="pep-525-asyncgen-hooks"> <h3><a class="toc-backref" href="#pep-525-asyncgen-hooks" role="doc-backlink">PEP 525 asyncgen hooks</a></h3> <a class="pep reference internal" href="../pep-0525/#finalization" title="PEP 525 – Asynchronous Generators § Finalization">PEP 525 proposes a set of global thread-local hooks</a> managed by new <code class="docutils literal notranslate">sys.{get/set}_asyncgen_hooks()</code> functions, which allow event loops to integrate with the garbage collector to run cleanup for async generators. In principle, this proposal and <a class="pep reference internal" href="../pep-0525/" title="PEP 525 – Asynchronous Generators">PEP 525</a> are complementary, in the same way that <code class="docutils literal notranslate">with</code> blocks and <code class="docutils literal notranslate">__del__</code> are complementary: this proposal takes care of ensuring deterministic cleanup in most cases, while <a class="pep reference internal" href="../pep-0525/" title="PEP 525 – Asynchronous Generators">PEP 525</a>’s GC hooks clean up anything that gets missed. But <code class="docutils literal notranslate">__aiterclose__</code> provides a number of advantages over GC hooks alone: <ul> <li>The GC hook semantics aren’t part of the abstract async iterator protocol, but are instead restricted <a class="reference external" href="https://mail.python.org/pipermail/python-dev/2016-September/146129.html">specifically to the async generator concrete type</a>. If you have an async iterator implemented using a class, like:<div class="highlight-default notranslate"><div class="highlight"><pre>class MyAsyncIterator: async def __anext__(): ... </pre></div> </div> then you can’t refactor this into an async generator without changing its semantics, and vice-versa. This seems very unpythonic. (It also leaves open the question of what exactly class-based async iterators are supposed to do, given that they face exactly the same cleanup problems as async generators.) <code class="docutils literal notranslate">__aiterclose__</code>, on the other hand, is defined at the protocol level, so it’s duck-type friendly and works for all iterators, not just generators. </li> <li>Code that wants to work on non-CPython implementations like PyPy cannot in general rely on GC for cleanup. Without <code class="docutils literal notranslate">__aiterclose__</code>, it’s more or less guaranteed that developers who develop and test on CPython will produce libraries that leak resources when used on PyPy. Developers who do want to target alternative implementations will either have to take the defensive approach of wrapping every <code class="docutils literal notranslate">for</code> loop in a <code class="docutils literal notranslate">with</code> block, or else carefully audit their code to figure out which generators might possibly contain cleanup code and add <code class="docutils literal notranslate">with</code> blocks around those only. With <code class="docutils literal notranslate">__aiterclose__</code>, writing portable code becomes easy and natural.</li> <li>An important part of building robust software is making sure that exceptions always propagate correctly without being lost. One of the most exciting things about async/await compared to traditional callback-based systems is that instead of requiring manual chaining, the runtime can now do the heavy lifting of propagating errors, making it much easier to write robust code. But, this beautiful new picture has one major gap: if we rely on the GC for generator cleanup, then exceptions raised during cleanup are lost. So, again, with <code class="docutils literal notranslate">__aiterclose__</code>, developers who care about this kind of robustness will either have to take the defensive approach of wrapping every <code class="docutils literal notranslate">for</code> loop in a <code class="docutils literal notranslate">with</code> block, or else carefully audit their code to figure out which generators might possibly contain cleanup code. <code class="docutils literal notranslate">__aiterclose__</code> plugs this hole by performing cleanup in the caller’s context, so writing more robust code becomes the path of least resistance.</li> <li>The WSGI experience suggests that there exist important iterator-based APIs that need prompt cleanup and cannot rely on the GC, even in CPython. For example, consider a hypothetical WSGI-like API based around async/await and async iterators, where a response handler is an async generator that takes request headers + an async iterator over the request body, and yields response headers + the response body. (This is actually the use case that got me interested in async generators in the first place, i.e. this isn’t hypothetical.) If we follow WSGI in requiring that child iterators must be closed properly, then without <code class="docutils literal notranslate">__aiterclose__</code> the absolute most minimalistic middleware in our system looks something like:<div class="highlight-default notranslate"><div class="highlight"><pre>async def noop_middleware(handler, request_header, request_body): async with aclosing(handler(request_body, request_body)) as aiter: async for response_item in aiter: yield response_item </pre></div> </div> Arguably in regular code one can get away with skipping the <code class="docutils literal notranslate">with</code> block around <code class="docutils literal notranslate">for</code> loops, depending on how confident one is that one understands the internal implementation of the generator. But here we have to cope with arbitrary response handlers, so without <code class="docutils literal notranslate">__aiterclose__</code>, this <code class="docutils literal notranslate">with</code> construction is a mandatory part of every middleware. <code class="docutils literal notranslate">__aiterclose__</code> allows us to eliminate the mandatory boilerplate and an extra level of indentation from every middleware: <div class="highlight-default notranslate"><div class="highlight"><pre>async def noop_middleware(handler, request_header, request_body): async for response_item in handler(request_header, request_body): yield response_item </pre></div> </div> </li> </ul> So the <code class="docutils literal notranslate">__aiterclose__</code> approach provides substantial advantages over GC hooks. This leaves open the question of whether we want a combination of GC hooks + <code class="docutils literal notranslate">__aiterclose__</code>, or just <code class="docutils literal notranslate">__aiterclose__</code> alone. Since the vast majority of generators are iterated over using a <code class="docutils literal notranslate">for</code> loop or equivalent, <code class="docutils literal notranslate">__aiterclose__</code> handles most situations before the GC has a chance to get involved. The case where GC hooks provide additional value is in code that does manual iteration, e.g.: <div class="highlight-default notranslate"><div class="highlight"><pre>agen = fetch_newline_separated_json_from_url(...) while True: document = await type(agen).__anext__(agen) if document["id"] == needle: break # doesn't do 'await agen.aclose()' </pre></div> </div> If we go with the GC-hooks + <code class="docutils literal notranslate">__aiterclose__</code> approach, this generator will eventually be cleaned up by GC calling the generator <code class="docutils literal notranslate">__del__</code> method, which then will use the hooks to call back into the event loop to run the cleanup code. If we go with the no-GC-hooks approach, this generator will eventually be garbage collected, with the following effects: <ul class="simple"> <li>its <code class="docutils literal notranslate">__del__</code> method will issue a warning that the generator was not closed (similar to the existing “coroutine never awaited” warning).</li> <li>The underlying resources involved will still be cleaned up, because the generator frame will still be garbage collected, causing it to drop references to any file handles or sockets it holds, and then those objects’s <code class="docutils literal notranslate">__del__</code> methods will release the actual operating system resources.</li> <li>But, any cleanup code inside the generator itself (e.g. logging, buffer flushing) will not get a chance to run.</li> </ul> The solution here – as the warning would indicate – is to fix the code so that it calls <code class="docutils literal notranslate">__aiterclose__</code>, e.g. by using a <code class="docutils literal notranslate">with</code> block: <div class="highlight-default notranslate"><div class="highlight"><pre>async with aclosing(fetch_newline_separated_json_from_url(...)) as agen: while True: document = await type(agen).__anext__(agen) if document["id"] == needle: break </pre></div> </div> Basically in this approach, the rule would be that if you want to manually implement the iterator protocol, then it’s your responsibility to implement all of it, and that now includes <code class="docutils literal notranslate">__(a)iterclose__</code>. GC hooks add non-trivial complexity in the form of (a) new global interpreter state, (b) a somewhat complicated control flow (e.g., async generator GC always involves resurrection, so the details of PEP 442 are important), and (c) a new public API in asyncio (<code class="docutils literal notranslate">await loop.shutdown_asyncgens()</code>) that users have to remember to call at the appropriate time. (This last point in particular somewhat undermines the argument that GC hooks provide a safe backup to guarantee cleanup, since if <code class="docutils literal notranslate">shutdown_asyncgens()</code> isn’t called correctly then I think it’s possible for generators to be silently discarded without their cleanup code being called; compare this to the <code class="docutils literal notranslate">__aiterclose__</code>-only approach where in the worst case we still at least get a warning printed. This might be fixable.) All this considered, GC hooks arguably aren’t worth it, given that the only people they help are those who want to manually call <code class="docutils literal notranslate">__anext__</code> yet don’t want to manually call <code class="docutils literal notranslate">__aiterclose__</code>. But Yury disagrees with me on this :-). And both options are viable. </section> <section id="always-inject-resources-and-do-all-cleanup-at-the-top-level"> <h3><a class="toc-backref" href="#always-inject-resources-and-do-all-cleanup-at-the-top-level" role="doc-backlink">Always inject resources, and do all cleanup at the top level</a></h3> Several commentators on python-dev and python-ideas have suggested that a pattern to avoid these problems is to always pass resources in from above, e.g. <code class="docutils literal notranslate">read_newline_separated_json</code> should take a file object rather than a path, with cleanup handled at the top level: <div class="highlight-default notranslate"><div class="highlight"><pre>def read_newline_separated_json(file_handle): for line in file_handle: yield json.loads(line) def read_users(file_handle): for document in read_newline_separated_json(file_handle): yield User.from_json(document) with open(path) as file_handle: for user in read_users(file_handle): ... </pre></div> </div> This works well in simple cases; here it lets us avoid the “N+1 <code class="docutils literal notranslate">with</code> blocks problem”. But unfortunately, it breaks down quickly when things get more complex. Consider if instead of reading from a file, our generator was reading from a streaming HTTP GET request – while handling redirects and authentication via OAUTH. Then we’d really want the sockets to be managed down inside our HTTP client library, not at the top level. Plus there are other cases where <code class="docutils literal notranslate">finally</code> blocks embedded inside generators are important in their own right: db transaction management, emitting logging information during cleanup (one of the major motivating use cases for WSGI <code class="docutils literal notranslate">close</code>), and so forth. So this is really a workaround for simple cases, not a general solution. </section> <section id="more-complex-variants-of-a-iterclose"> <h3><a class="toc-backref" href="#more-complex-variants-of-a-iterclose" role="doc-backlink">More complex variants of __(a)iterclose__</a></h3> The semantics of <code class="docutils literal notranslate">__(a)iterclose__</code> are somewhat inspired by <code class="docutils literal notranslate">with</code> blocks, but context managers are more powerful: <code class="docutils literal notranslate">__(a)exit__</code> can distinguish between a normal exit versus exception unwinding, and in the case of an exception it can examine the exception details and optionally suppress propagation. <code class="docutils literal notranslate">__(a)iterclose__</code> as proposed here does not have these powers, but one can imagine an alternative design where it did. However, this seems like unwarranted complexity: experience suggests that it’s common for iterables to have <code class="docutils literal notranslate">close</code> methods, and even to have <code class="docutils literal notranslate">__exit__</code> methods that call <code class="docutils literal notranslate">self.close()</code>, but I’m not aware of any common cases that make use of <code class="docutils literal notranslate">__exit__</code>’s full power. I also can’t think of any examples where this would be useful. And it seems unnecessarily confusing to allow iterators to affect flow control by swallowing exceptions – if you’re in a situation where you really want that, then you should probably use a real <code class="docutils literal notranslate">with</code> block anyway. </section> </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> This section describes where we want to eventually end up, though there are some backwards compatibility issues that mean we can’t jump directly here. A later section describes the transition plan. <section id="guiding-principles"> <h3><a class="toc-backref" href="#guiding-principles" role="doc-backlink">Guiding principles</a></h3> Generally, <code class="docutils literal notranslate">__(a)iterclose__</code> implementations should: <ul class="simple"> <li>be idempotent,</li> <li>perform any cleanup that is appropriate on the assumption that the iterator will not be used again after <code class="docutils literal notranslate">__(a)iterclose__</code> is called. In particular, once <code class="docutils literal notranslate">__(a)iterclose__</code> has been called then calling <code class="docutils literal notranslate">__(a)next__</code> produces undefined behavior.</li> </ul> And generally, any code which starts iterating through an iterable with the intention of exhausting it, should arrange to make sure that <code class="docutils literal notranslate">__(a)iterclose__</code> is eventually called, whether or not the iterator is actually exhausted. </section> <section id="changes-to-iteration"> <h3><a class="toc-backref" href="#changes-to-iteration" role="doc-backlink">Changes to iteration</a></h3> The core proposal is the change in behavior of <code class="docutils literal notranslate">for</code> loops. Given this Python code: <div class="highlight-default notranslate"><div class="highlight"><pre>for VAR in ITERABLE: LOOP-BODY else: ELSE-BODY </pre></div> </div> we desugar to the equivalent of: <div class="highlight-default notranslate"><div class="highlight"><pre>_iter = iter(ITERABLE) _iterclose = getattr(type(_iter), "__iterclose__", lambda: None) try: traditional-for VAR in _iter: LOOP-BODY else: ELSE-BODY finally: _iterclose(_iter) </pre></div> </div> where the “traditional-for statement” here is meant as a shorthand for the classic 3.5-and-earlier <code class="docutils literal notranslate">for</code> loop semantics. Besides the top-level <code class="docutils literal notranslate">for</code> statement, Python also contains several other places where iterators are consumed. For consistency, these should call <code class="docutils literal notranslate">__iterclose__</code> as well using semantics equivalent to the above. This includes: <ul class="simple"> <li><code class="docutils literal notranslate">for</code> loops inside comprehensions</li> <li><code class="docutils literal notranslate">*</code> unpacking</li> <li>functions which accept and fully consume iterables, like <code class="docutils literal notranslate">list(it)</code>, <code class="docutils literal notranslate">tuple(it)</code>, <code class="docutils literal notranslate">itertools.product(it1, it2, ...)</code>, and others.</li> </ul> In addition, a <code class="docutils literal notranslate">yield from</code> that successfully exhausts the called generator should as a last step call its <code class="docutils literal notranslate">__iterclose__</code> method. (Rationale: <code class="docutils literal notranslate">yield from</code> already links the lifetime of the calling generator to the called generator; if the calling generator is closed when half-way through a <code class="docutils literal notranslate">yield from</code>, then this will already automatically close the called generator.) </section> <section id="changes-to-async-iteration"> <h3><a class="toc-backref" href="#changes-to-async-iteration" role="doc-backlink">Changes to async iteration</a></h3> We also make the analogous changes to async iteration constructs, except that the new slot is called <code class="docutils literal notranslate">__aiterclose__</code>, and it’s an async method that gets <code class="docutils literal notranslate">await</code>ed. </section> <section id="modifications-to-basic-iterator-types"> <h3><a class="toc-backref" href="#modifications-to-basic-iterator-types" role="doc-backlink">Modifications to basic iterator types</a></h3> Generator objects (including those created by generator comprehensions): <ul class="simple"> <li><code class="docutils literal notranslate">__iterclose__</code> calls <code class="docutils literal notranslate">self.close()</code></li> <li><code class="docutils literal notranslate">__del__</code> calls <code class="docutils literal notranslate">self.close()</code> (same as now), and additionally issues a <code class="docutils literal notranslate">ResourceWarning</code> if the generator wasn’t exhausted. This warning is hidden by default, but can be enabled for those who want to make sure they aren’t inadvertently relying on CPython-specific GC semantics.</li> </ul> Async generator objects (including those created by async generator comprehensions): <ul class="simple"> <li><code class="docutils literal notranslate">__aiterclose__</code> calls <code class="docutils literal notranslate">self.aclose()</code></li> <li><code class="docutils literal notranslate">__del__</code> issues a <code class="docutils literal notranslate">RuntimeWarning</code> if <code class="docutils literal notranslate">aclose</code> has not been called, since this probably indicates a latent bug, similar to the “coroutine never awaited” warning.</li> </ul> QUESTION: should file objects implement <code class="docutils literal notranslate">__iterclose__</code> to close the file? On the one hand this would make this change more disruptive; on the other hand people really like writing <code class="docutils literal notranslate">for line in open(...): ...</code>, and if we get used to iterators taking care of their own cleanup then it might become very weird if files don’t. </section> <section id="new-convenience-functions"> <h3><a class="toc-backref" href="#new-convenience-functions" role="doc-backlink">New convenience functions</a></h3> The <code class="docutils literal notranslate">operator</code> module gains two new functions, with semantics equivalent to the following: <div class="highlight-default notranslate"><div class="highlight"><pre>def iterclose(it): if not isinstance(it, collections.abc.Iterator): raise TypeError("not an iterator") if hasattr(type(it), "__iterclose__"): type(it).__iterclose__(it) async def aiterclose(ait): if not isinstance(it, collections.abc.AsyncIterator): raise TypeError("not an iterator") if hasattr(type(ait), "__aiterclose__"): await type(ait).__aiterclose__(ait) </pre></div> </div> The <code class="docutils literal notranslate">itertools</code> module gains a new iterator wrapper that can be used to selectively disable the new <code class="docutils literal notranslate">__iterclose__</code> behavior: <div class="highlight-default notranslate"><div class="highlight"><pre># QUESTION: I feel like there might be a better name for this one? class preserve(iterable): def __init__(self, iterable): self._it = iter(iterable) def __iter__(self): return self def __next__(self): return next(self._it) def __iterclose__(self): # Swallow __iterclose__ without passing it on pass </pre></div> </div> Example usage (assuming that file objects implements <code class="docutils literal notranslate">__iterclose__</code>): <div class="highlight-default notranslate"><div class="highlight"><pre>with open(...) as handle: # Iterate through the same file twice: for line in itertools.preserve(handle): ... handle.seek(0) for line in itertools.preserve(handle): ... </pre></div> </div> <div class="highlight-default notranslate"><div class="highlight"><pre>@contextlib.contextmanager def iterclosing(iterable): it = iter(iterable) try: yield preserve(it) finally: iterclose(it) </pre></div> </div> </section> <section id="iterclose-implementations-for-iterator-wrappers"> <h3><a class="toc-backref" href="#iterclose-implementations-for-iterator-wrappers" role="doc-backlink">__iterclose__ implementations for iterator wrappers</a></h3> Python ships a number of iterator types that act as wrappers around other iterators: <code class="docutils literal notranslate">map</code>, <code class="docutils literal notranslate">zip</code>, <code class="docutils literal notranslate">itertools.accumulate</code>, <code class="docutils literal notranslate">csv.reader</code>, and others. These iterators should define a <code class="docutils literal notranslate">__iterclose__</code> method which calls <code class="docutils literal notranslate">__iterclose__</code> in turn on their underlying iterators. For example, <code class="docutils literal notranslate">map</code> could be implemented as: <div class="highlight-default notranslate"><div class="highlight"><pre># Helper function map_chaining_exceptions(fn, items, last_exc=None): for item in items: try: fn(item) except BaseException as new_exc: if new_exc.__context__ is None: new_exc.__context__ = last_exc last_exc = new_exc if last_exc is not None: raise last_exc class map: def __init__(self, fn, *iterables): self._fn = fn self._iters = [iter(iterable) for iterable in iterables] def __iter__(self): return self def __next__(self): return self._fn(*[next(it) for it in self._iters]) def __iterclose__(self): map_chaining_exceptions(operator.iterclose, self._iters) def chain(*iterables): try: while iterables: for element in iterables.pop(0): yield element except BaseException as e: def iterclose_iterable(iterable): operations.iterclose(iter(iterable)) map_chaining_exceptions(iterclose_iterable, iterables, last_exc=e) </pre></div> </div> In some cases this requires some subtlety; for example, <a class="reference external" href="https://docs.python.org/3/library/itertools.html#itertools.tee">itertools.tee</a> should not call <code class="docutils literal notranslate">__iterclose__</code> on the underlying iterator until it has been called on all of the clone iterators. </section> <section id="example-rationale"> <h3><a class="toc-backref" href="#example-rationale" role="doc-backlink">Example / Rationale</a></h3> The payoff for all this is that we can now write straightforward code like: <div class="highlight-default notranslate"><div class="highlight"><pre>def read_newline_separated_json(path): for line in open(path): yield json.loads(line) </pre></div> </div> and be confident that the file will receive deterministic cleanup without the end-user having to take any special effort, even in complex cases. For example, consider this silly pipeline: <div class="highlight-default notranslate"><div class="highlight"><pre>list(map(lambda key: key.upper(), doc["key"] for doc in read_newline_separated_json(path))) </pre></div> </div> If our file contains a document where <code class="docutils literal notranslate">doc["key"]</code> turns out to be an integer, then the following sequence of events will happen: <ol class="arabic simple"> <li><code class="docutils literal notranslate">key.upper()</code> raises an <code class="docutils literal notranslate">AttributeError</code>, which propagates out of the <code class="docutils literal notranslate">map</code> and triggers the implicit <code class="docutils literal notranslate">finally</code> block inside <code class="docutils literal notranslate">list</code>.</li> <li>The <code class="docutils literal notranslate">finally</code> block in <code class="docutils literal notranslate">list</code> calls <code class="docutils literal notranslate">__iterclose__()</code> on the map object.</li> <li><code class="docutils literal notranslate">map.__iterclose__()</code> calls <code class="docutils literal notranslate">__iterclose__()</code> on the generator comprehension object.</li> <li>This injects a <code class="docutils literal notranslate">GeneratorExit</code> exception into the generator comprehension body, which is currently suspended inside the comprehension’s <code class="docutils literal notranslate">for</code> loop body.</li> <li>The exception propagates out of the <code class="docutils literal notranslate">for</code> loop, triggering the <code class="docutils literal notranslate">for</code> loop’s implicit <code class="docutils literal notranslate">finally</code> block, which calls <code class="docutils literal notranslate">__iterclose__</code> on the generator object representing the call to <code class="docutils literal notranslate">read_newline_separated_json</code>.</li> <li>This injects an inner <code class="docutils literal notranslate">GeneratorExit</code> exception into the body of <code class="docutils literal notranslate">read_newline_separated_json</code>, currently suspended at the <code class="docutils literal notranslate">yield</code>.</li> <li>The inner <code class="docutils literal notranslate">GeneratorExit</code> propagates out of the <code class="docutils literal notranslate">for</code> loop, triggering the <code class="docutils literal notranslate">for</code> loop’s implicit <code class="docutils literal notranslate">finally</code> block, which calls <code class="docutils literal notranslate">__iterclose__()</code> on the file object.</li> <li>The file object is closed.</li> <li>The inner <code class="docutils literal notranslate">GeneratorExit</code> resumes propagating, hits the boundary of the generator function, and causes <code class="docutils literal notranslate">read_newline_separated_json</code>’s <code class="docutils literal notranslate">__iterclose__()</code> method to return successfully.</li> <li>Control returns to the generator comprehension body, and the outer <code class="docutils literal notranslate">GeneratorExit</code> continues propagating, allowing the comprehension’s <code class="docutils literal notranslate">__iterclose__()</code> to return successfully.</li> <li>The rest of the <code class="docutils literal notranslate">__iterclose__()</code> calls unwind without incident, back into the body of <code class="docutils literal notranslate">list</code>.</li> <li>The original <code class="docutils literal notranslate">AttributeError</code> resumes propagating.</li> </ol> (The details above assume that we implement <code class="docutils literal notranslate">file.__iterclose__</code>; if not then add a <code class="docutils literal notranslate">with</code> block to <code class="docutils literal notranslate">read_newline_separated_json</code> and essentially the same logic goes through.) Of course, from the user’s point of view, this can be simplified down to just: 1. <code class="docutils literal notranslate">int.upper()</code> raises an <code class="docutils literal notranslate">AttributeError</code> 1. The file object is closed. 2. The <code class="docutils literal notranslate">AttributeError</code> propagates out of <code class="docutils literal notranslate">list</code> So we’ve accomplished our goal of making this “just work” without the user having to think about it. </section> </section> <section id="transition-plan"> <h2><a class="toc-backref" href="#transition-plan" role="doc-backlink">Transition plan</a></h2> While the majority of existing <code class="docutils literal notranslate">for</code> loops will continue to produce identical results, the proposed changes will produce backwards-incompatible behavior in some cases. Example: <div class="highlight-default notranslate"><div class="highlight"><pre>def read_csv_with_header(lines_iterable): lines_iterator = iter(lines_iterable) for line in lines_iterator: column_names = line.strip().split("\t") break for line in lines_iterator: values = line.strip().split("\t") record = dict(zip(column_names, values)) yield record </pre></div> </div> This code used to be correct, but after this proposal is implemented will require an <code class="docutils literal notranslate">itertools.preserve</code> call added to the first <code class="docutils literal notranslate">for</code> loop. [QUESTION: currently, if you close a generator and then try to iterate over it then it just raises <code class="docutils literal notranslate">Stop(Async)Iteration</code>, so code the passes the same generator object to multiple <code class="docutils literal notranslate">for</code> loops but forgets to use <code class="docutils literal notranslate">itertools.preserve</code> won’t see an obvious error – the second <code class="docutils literal notranslate">for</code> loop will just exit immediately. Perhaps it would be better if iterating a closed generator raised a <code class="docutils literal notranslate">RuntimeError</code>? Note that files don’t have this problem – attempting to iterate a closed file object already raises <code class="docutils literal notranslate">ValueError</code>.] Specifically, the incompatibility happens when all of these factors come together: <ul class="simple"> <li>The automatic calling of <code class="docutils literal notranslate">__(a)iterclose__</code> is enabled</li> <li>The iterable did not previously define <code class="docutils literal notranslate">__(a)iterclose__</code></li> <li>The iterable does now define <code class="docutils literal notranslate">__(a)iterclose__</code></li> <li>The iterable is re-used after the <code class="docutils literal notranslate">for</code> loop exits</li> </ul> So the problem is how to manage this transition, and those are the levers we have to work with. First, observe that the only async iterables where we propose to add <code class="docutils literal notranslate">__aiterclose__</code> are async generators, and there is currently no existing code using async generators (though this will start changing very soon), so the async changes do not produce any backwards incompatibilities. (There is existing code using async iterators, but using the new async for loop on an old async iterator is harmless, because old async iterators don’t have <code class="docutils literal notranslate">__aiterclose__</code>.) In addition, <a class="pep reference internal" href="../pep-0525/" title="PEP 525 – Asynchronous Generators">PEP 525</a> was accepted on a provisional basis, and async generators are by far the biggest beneficiary of this PEP’s proposed changes. Therefore, I think we should strongly consider enabling <code class="docutils literal notranslate">__aiterclose__</code> for <code class="docutils literal notranslate">async for</code> loops and async generators ASAP, ideally for 3.6.0 or 3.6.1. For the non-async world, things are harder, but here’s a potential transition path: In 3.7: Our goal is that existing unsafe code will start emitting warnings, while those who want to opt-in to the future can do that immediately: <ul class="simple"> <li>We immediately add all the <code class="docutils literal notranslate">__iterclose__</code> methods described above.</li> <li>If <code class="docutils literal notranslate">from __future__ import iterclose</code> is in effect, then <code class="docutils literal notranslate">for</code> loops and <code class="docutils literal notranslate">*</code> unpacking call <code class="docutils literal notranslate">__iterclose__</code> as specified above.</li> <li>If the future is not enabled, then <code class="docutils literal notranslate">for</code> loops and <code class="docutils literal notranslate">*</code> unpacking do not call <code class="docutils literal notranslate">__iterclose__</code>. But they do call some other method instead, e.g. <code class="docutils literal notranslate">__iterclose_warning__</code>.</li> <li>Similarly, functions like <code class="docutils literal notranslate">list</code> use stack introspection (!!) to check whether their direct caller has <code class="docutils literal notranslate">__future__.iterclose</code> enabled, and use this to decide whether to call <code class="docutils literal notranslate">__iterclose__</code> or <code class="docutils literal notranslate">__iterclose_warning__</code>.</li> <li>For all the wrapper iterators, we also add <code class="docutils literal notranslate">__iterclose_warning__</code> methods that forward to the <code class="docutils literal notranslate">__iterclose_warning__</code> method of the underlying iterator or iterators.</li> <li>For generators (and files, if we decide to do that), <code class="docutils literal notranslate">__iterclose_warning__</code> is defined to set an internal flag, and other methods on the object are modified to check for this flag. If they find the flag set, they issue a <code class="docutils literal notranslate">PendingDeprecationWarning</code> to inform the user that in the future this sequence would have led to a use-after-close situation and the user should use <code class="docutils literal notranslate">preserve()</code>.</li> </ul> In 3.8: <ul class="simple"> <li>Switch from <code class="docutils literal notranslate">PendingDeprecationWarning</code> to <code class="docutils literal notranslate">DeprecationWarning</code></li> </ul> In 3.9: <ul class="simple"> <li>Enable the <code class="docutils literal notranslate">__future__</code> unconditionally and remove all the <code class="docutils literal notranslate">__iterclose_warning__</code> stuff.</li> </ul> I believe that this satisfies the normal requirements for this kind of transition – opt-in initially, with warnings targeted precisely to the cases that will be effected, and a long deprecation cycle. Probably the most controversial / risky part of this is the use of stack introspection to make the iterable-consuming functions sensitive to a <code class="docutils literal notranslate">__future__</code> setting, though I haven’t thought of any situation where it would actually go wrong yet… </section> <section id="acknowledgements"> <h2><a class="toc-backref" href="#acknowledgements" role="doc-backlink">Acknowledgements</a></h2> Thanks to Yury Selivanov, Armin Rigo, and Carl Friedrich Bolz for helpful discussion on earlier versions of this idea. </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0533.rst">https://github.com/python/peps/blob/main/peps/pep-0533.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0533.rst">2025-02-01 08:59:27 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#note-on-timing">Note on timing</a></li> <li><a class="reference internal" href="#background-and-motivation">Background and motivation</a></li> <li><a class="reference internal" href="#alternatives">Alternatives</a><ul> <li><a class="reference internal" href="#pep-525-asyncgen-hooks">PEP 525 asyncgen hooks</a></li> <li><a class="reference internal" href="#always-inject-resources-and-do-all-cleanup-at-the-top-level">Always inject resources, and do all cleanup at the top level</a></li> <li><a class="reference internal" href="#more-complex-variants-of-a-iterclose">More complex variants of __(a)iterclose__</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#guiding-principles">Guiding principles</a></li> <li><a class="reference internal" href="#changes-to-iteration">Changes to iteration</a></li> <li><a class="reference internal" href="#changes-to-async-iteration">Changes to async iteration</a></li> <li><a class="reference internal" href="#modifications-to-basic-iterator-types">Modifications to basic iterator types</a></li> <li><a class="reference internal" href="#new-convenience-functions">New convenience functions</a></li> <li><a class="reference internal" href="#iterclose-implementations-for-iterator-wrappers">__iterclose__ implementations for iterator wrappers</a></li> <li><a class="reference internal" href="#example-rationale">Example / Rationale</a></li> </ul> </li> <li><a class="reference internal" href="#transition-plan">Transition plan</a></li> <li><a class="reference internal" href="#acknowledgements">Acknowledgements</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0533.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>