CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 342 – Coroutines via Enhanced Generators | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0342/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 342 – Coroutines via Enhanced Generators | peps.python.org'> <meta property="og:description" content="This PEP proposes some enhancements to the API and syntax of generators, to make them usable as simple coroutines. It is basically a combination of ideas from these two PEPs, which may be considered redundant if this PEP is accepted:"> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0342/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This PEP proposes some enhancements to the API and syntax of generators, to make them usable as simple coroutines. It is basically a combination of ideas from these two PEPs, which may be considered redundant if this PEP is accepted:"> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 342</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 342 – Coroutines via Enhanced Generators</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Guido van Rossum, Phillip J. Eby</dd> <dt class="field-even">Status:</dt> <dd class="field-even"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-odd">Type:</dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created:</dt> <dd class="field-even">10-May-2005</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">2.5</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even"></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#introduction">Introduction</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#specification-summary">Specification Summary</a></li> <li><a class="reference internal" href="#specification-sending-values-into-generators">Specification: Sending Values into Generators</a><ul> <li><a class="reference internal" href="#new-generator-method-send-value">New generator method: <code class="docutils literal notranslate">send(value)</code></a></li> <li><a class="reference internal" href="#new-syntax-yield-expressions">New syntax: Yield Expressions</a></li> </ul> </li> <li><a class="reference internal" href="#specification-exceptions-and-cleanup">Specification: Exceptions and Cleanup</a><ul> <li><a class="reference internal" href="#new-syntax-yield-allowed-inside-try-finally">New syntax: <code class="docutils literal notranslate">yield</code> allowed inside <code class="docutils literal notranslate">try-finally</code></a></li> <li><a class="reference internal" href="#new-generator-method-throw-type-value-none-traceback-none">New generator method: <code class="docutils literal notranslate">throw(type, value=None, traceback=None)</code></a></li> <li><a class="reference internal" href="#new-standard-exception-generatorexit">New standard exception: <code class="docutils literal notranslate">GeneratorExit</code></a></li> <li><a class="reference internal" href="#new-generator-method-close">New generator method: <code class="docutils literal notranslate">close()</code></a></li> <li><a class="reference internal" href="#new-generator-method-del">New generator method: __del__()</a></li> </ul> </li> <li><a class="reference internal" href="#optional-extensions">Optional Extensions</a><ul> <li><a class="reference internal" href="#the-extended-continue-statement">The Extended <code class="docutils literal notranslate">continue</code> Statement</a></li> </ul> </li> <li><a class="reference internal" href="#open-issues">Open Issues</a></li> <li><a class="reference internal" href="#examples">Examples</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#acknowledgements">Acknowledgements</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="introduction"> <h2><a class="toc-backref" href="#introduction" role="doc-backlink">Introduction</a></h2> This PEP proposes some enhancements to the API and syntax of generators, to make them usable as simple coroutines. It is basically a combination of ideas from these two PEPs, which may be considered redundant if this PEP is accepted: <ul class="simple"> <li><a class="pep reference internal" href="../pep-0288/" title="PEP 288 – Generators Attributes and Exceptions">PEP 288</a>, Generators Attributes and Exceptions. The current PEP covers its second half, generator exceptions (in fact the <code class="docutils literal notranslate">throw()</code> method name was taken from <a class="pep reference internal" href="../pep-0288/" title="PEP 288 – Generators Attributes and Exceptions">PEP 288</a>). <a class="pep reference internal" href="../pep-0342/" title="PEP 342 – Coroutines via Enhanced Generators">PEP 342</a> replaces generator attributes, however, with a concept from an earlier revision of <a class="pep reference internal" href="../pep-0288/" title="PEP 288 – Generators Attributes and Exceptions">PEP 288</a>, the yield expression.</li> <li><a class="pep reference internal" href="../pep-0325/" title="PEP 325 – Resource-Release Support for Generators">PEP 325</a>, Resource-Release Support for Generators. <a class="pep reference internal" href="../pep-0342/" title="PEP 342 – Coroutines via Enhanced Generators">PEP 342</a> ties up a few loose ends in the <a class="pep reference internal" href="../pep-0325/" title="PEP 325 – Resource-Release Support for Generators">PEP 325</a> spec, to make it suitable for actual implementation.</li> </ul> </section> <section id="motivation"> <h2><a class="toc-backref" href="#motivation" role="doc-backlink">Motivation</a></h2> Coroutines are a natural way of expressing many algorithms, such as simulations, games, asynchronous I/O, and other forms of event-driven programming or co-operative multitasking. Python’s generator functions are almost coroutines – but not quite – in that they allow pausing execution to produce a value, but do not provide for values or exceptions to be passed in when execution resumes. They also do not allow execution to be paused within the <code class="docutils literal notranslate">try</code> portion of <code class="docutils literal notranslate">try/finally</code> blocks, and therefore make it difficult for an aborted coroutine to clean up after itself. Also, generators cannot yield control while other functions are executing, unless those functions are themselves expressed as generators, and the outer generator is written to yield in response to values yielded by the inner generator. This complicates the implementation of even relatively simple use cases like asynchronous communications, because calling any functions either requires the generator to block (i.e. be unable to yield control), or else a lot of boilerplate looping code must be added around every needed function call. However, if it were possible to pass values or exceptions into a generator at the point where it was suspended, a simple co-routine scheduler or trampoline function would let coroutines call each other without blocking – a tremendous boon for asynchronous applications. Such applications could then write co-routines to do non-blocking socket I/O by yielding control to an I/O scheduler until data has been sent or becomes available. Meanwhile, code that performs the I/O would simply do something like this: <div class="highlight-default notranslate"><div class="highlight"><pre>data = (yield nonblocking_read(my_socket, nbytes)) </pre></div> </div> in order to pause execution until the <code class="docutils literal notranslate">nonblocking_read()</code> coroutine produced a value. In other words, with a few relatively minor enhancements to the language and to the implementation of the generator-iterator type, Python will be able to support performing asynchronous operations without needing to write the entire application as a series of callbacks, and without requiring the use of resource-intensive threads for programs that need hundreds or even thousands of co-operatively multitasking pseudothreads. Thus, these enhancements will give standard Python many of the benefits of the Stackless Python fork, without requiring any significant modification to the CPython core or its APIs. In addition, these enhancements should be readily implementable by any Python implementation (such as Jython) that already supports generators. </section> <section id="specification-summary"> <h2><a class="toc-backref" href="#specification-summary" role="doc-backlink">Specification Summary</a></h2> By adding a few simple methods to the generator-iterator type, and with two minor syntax adjustments, Python developers will be able to use generator functions to implement co-routines and other forms of co-operative multitasking. These methods and adjustments are: <ol class="arabic simple"> <li>Redefine <code class="docutils literal notranslate">yield</code> to be an expression, rather than a statement. The current yield statement would become a yield expression whose value is thrown away. A yield expression’s value is <code class="docutils literal notranslate">None</code> whenever the generator is resumed by a normal <code class="docutils literal notranslate">next()</code> call.</li> <li>Add a new <code class="docutils literal notranslate">send()</code> method for generator-iterators, which resumes the generator and sends a value that becomes the result of the current yield-expression. The <code class="docutils literal notranslate">send()</code> method returns the next value yielded by the generator, or raises <code class="docutils literal notranslate">StopIteration</code> if the generator exits without yielding another value.</li> <li>Add a new <code class="docutils literal notranslate">throw()</code> method for generator-iterators, which raises an exception at the point where the generator was paused, and which returns the next value yielded by the generator, raising <code class="docutils literal notranslate">StopIteration</code> if the generator exits without yielding another value. (If the generator does not catch the passed-in exception, or raises a different exception, then that exception propagates to the caller.)</li> <li>Add a <code class="docutils literal notranslate">close()</code> method for generator-iterators, which raises <code class="docutils literal notranslate">GeneratorExit</code> at the point where the generator was paused. If the generator then raises <code class="docutils literal notranslate">StopIteration</code> (by exiting normally, or due to already being closed) or <code class="docutils literal notranslate">GeneratorExit</code> (by not catching the exception), <code class="docutils literal notranslate">close()</code> returns to its caller. If the generator yields a value, a <code class="docutils literal notranslate">RuntimeError</code> is raised. If the generator raises any other exception, it is propagated to the caller. <code class="docutils literal notranslate">close()</code> does nothing if the generator has already exited due to an exception or normal exit.</li> <li>Add support to ensure that <code class="docutils literal notranslate">close()</code> is called when a generator iterator is garbage-collected.</li> <li>Allow <code class="docutils literal notranslate">yield</code> to be used in <code class="docutils literal notranslate">try/finally</code> blocks, since garbage collection or an explicit <code class="docutils literal notranslate">close()</code> call would now allow the <code class="docutils literal notranslate">finally</code> clause to execute.</li> </ol> A prototype patch implementing all of these changes against the current Python CVS HEAD is available as SourceForge patch #1223381 (<a class="reference external" href="https://bugs.python.org/issue1223381">https://bugs.python.org/issue1223381</a>). </section> <section id="specification-sending-values-into-generators"> <h2><a class="toc-backref" href="#specification-sending-values-into-generators" role="doc-backlink">Specification: Sending Values into Generators</a></h2> <section id="new-generator-method-send-value"> <h3><a class="toc-backref" href="#new-generator-method-send-value" role="doc-backlink">New generator method: <code class="docutils literal notranslate">send(value)</code></a></h3> A new method for generator-iterators is proposed, called <code class="docutils literal notranslate">send()</code>. It takes exactly one argument, which is the value that should be sent in to the generator. Calling <code class="docutils literal notranslate">send(None)</code> is exactly equivalent to calling a generator’s <code class="docutils literal notranslate">next()</code> method. Calling <code class="docutils literal notranslate">send()</code> with any other value is the same, except that the value produced by the generator’s current yield expression will be different. Because generator-iterators begin execution at the top of the generator’s function body, there is no yield expression to receive a value when the generator has just been created. Therefore, calling <code class="docutils literal notranslate">send()</code> with a non-<code class="docutils literal notranslate">None</code> argument is prohibited when the generator iterator has just started, and a <code class="docutils literal notranslate">TypeError</code> is raised if this occurs (presumably due to a logic error of some kind). Thus, before you can communicate with a coroutine you must first call <code class="docutils literal notranslate">next()</code> or <code class="docutils literal notranslate">send(None)</code> to advance its execution to the first yield expression. As with the <code class="docutils literal notranslate">next()</code> method, the <code class="docutils literal notranslate">send()</code> method returns the next value yielded by the generator-iterator, or raises <code class="docutils literal notranslate">StopIteration</code> if the generator exits normally, or has already exited. If the generator raises an uncaught exception, it is propagated to <code class="docutils literal notranslate">send()</code>’s caller. </section> <section id="new-syntax-yield-expressions"> <h3><a class="toc-backref" href="#new-syntax-yield-expressions" role="doc-backlink">New syntax: Yield Expressions</a></h3> The yield-statement will be allowed to be used on the right-hand side of an assignment; in that case it is referred to as yield-expression. The value of this yield-expression is <code class="docutils literal notranslate">None</code> unless <code class="docutils literal notranslate">send()</code> was called with a non-<code class="docutils literal notranslate">None</code> argument; see below. A yield-expression must always be parenthesized except when it occurs at the top-level expression on the right-hand side of an assignment. So <div class="highlight-default notranslate"><div class="highlight"><pre>x = yield 42 x = yield x = 12 + (yield 42) x = 12 + (yield) foo(yield 42) foo(yield) </pre></div> </div> are all legal, but <div class="highlight-default notranslate"><div class="highlight"><pre>x = 12 + yield 42 x = 12 + yield foo(yield 42, 12) foo(yield, 12) </pre></div> </div> are all illegal. (Some of the edge cases are motivated by the current legality of <code class="docutils literal notranslate">yield 12, 42</code>.) Note that a yield-statement or yield-expression without an expression is now legal. This makes sense: when the information flow in the <code class="docutils literal notranslate">next()</code> call is reversed, it should be possible to yield without passing an explicit value (<code class="docutils literal notranslate">yield</code> is of course equivalent to <code class="docutils literal notranslate">yield None</code>). When <code class="docutils literal notranslate">send(value)</code> is called, the yield-expression that it resumes will return the passed-in value. When <code class="docutils literal notranslate">next()</code> is called, the resumed yield-expression will return <code class="docutils literal notranslate">None</code>. If the yield-expression is a yield-statement, this returned value is ignored, similar to ignoring the value returned by a function call used as a statement. In effect, a yield-expression is like an inverted function call; the argument to yield is in fact returned (yielded) from the currently executing function, and the return value of yield is the argument passed in via <code class="docutils literal notranslate">send()</code>. Note: the syntactic extensions to yield make its use very similar to that in Ruby. This is intentional. Do note that in Python the block passes a value to the generator using <code class="docutils literal notranslate">send(EXPR)</code> rather than <code class="docutils literal notranslate">return EXPR</code>, and the underlying mechanism whereby control is passed between the generator and the block is completely different. Blocks in Python are not compiled into thunks; rather, <code class="docutils literal notranslate">yield</code> suspends execution of the generator’s frame. Some edge cases work differently; in Python, you cannot save the block for later use, and you cannot test whether there is a block or not. (XXX - this stuff about blocks seems out of place now, perhaps Guido can edit to clarify.) </section> </section> <section id="specification-exceptions-and-cleanup"> <h2><a class="toc-backref" href="#specification-exceptions-and-cleanup" role="doc-backlink">Specification: Exceptions and Cleanup</a></h2> Let a generator object be the iterator produced by calling a generator function. Below, g always refers to a generator object. <section id="new-syntax-yield-allowed-inside-try-finally"> <h3><a class="toc-backref" href="#new-syntax-yield-allowed-inside-try-finally" role="doc-backlink">New syntax: <code class="docutils literal notranslate">yield</code> allowed inside <code class="docutils literal notranslate">try-finally</code></a></h3> The syntax for generator functions is extended to allow a yield-statement inside a <code class="docutils literal notranslate">try-finally</code> statement. </section> <section id="new-generator-method-throw-type-value-none-traceback-none"> <h3><a class="toc-backref" href="#new-generator-method-throw-type-value-none-traceback-none" role="doc-backlink">New generator method: <code class="docutils literal notranslate">throw(type, value=None, traceback=None)</code></a></h3> <code class="docutils literal notranslate">g.throw(type, value, traceback)</code> causes the specified exception to be thrown at the point where the generator g is currently suspended (i.e. at a yield-statement, or at the start of its function body if <code class="docutils literal notranslate">next()</code> has not been called yet). If the generator catches the exception and yields another value, that is the return value of <code class="docutils literal notranslate">g.throw()</code>. If it doesn’t catch the exception, the <code class="docutils literal notranslate">throw()</code> appears to raise the same exception passed it (it falls through). If the generator raises another exception (this includes the <code class="docutils literal notranslate">StopIteration</code> produced when it returns) that exception is raised by the <code class="docutils literal notranslate">throw()</code> call. In summary, <code class="docutils literal notranslate">throw()</code> behaves like <code class="docutils literal notranslate">next()</code> or <code class="docutils literal notranslate">send()</code>, except it raises an exception at the suspension point. If the generator is already in the closed state, <code class="docutils literal notranslate">throw()</code> just raises the exception it was passed without executing any of the generator’s code. The effect of raising the exception is exactly as if the statement: <div class="highlight-default notranslate"><div class="highlight"><pre>raise type, value, traceback </pre></div> </div> was executed at the suspension point. The type argument must not be <code class="docutils literal notranslate">None</code>, and the type and value must be compatible. If the value is not an instance of the type, a new exception instance is created using the value, following the same rules that the <code class="docutils literal notranslate">raise</code> statement uses to create an exception instance. The traceback, if supplied, must be a valid Python traceback object, or a <code class="docutils literal notranslate">TypeError</code> occurs. Note: The name of the <code class="docutils literal notranslate">throw()</code> method was selected for several reasons. <code class="docutils literal notranslate">Raise</code> is a keyword and so cannot be used as a method name. Unlike <code class="docutils literal notranslate">raise</code> (which immediately raises an exception from the current execution point), <code class="docutils literal notranslate">throw()</code> first resumes the generator, and only then raises the exception. The word throw is suggestive of putting the exception in another location, and is already associated with exceptions in other languages. Alternative method names were considered: <code class="docutils literal notranslate">resolve()</code>, <code class="docutils literal notranslate">signal()</code>, <code class="docutils literal notranslate">genraise()</code>, <code class="docutils literal notranslate">raiseinto()</code>, and <code class="docutils literal notranslate">flush()</code>. None of these seem to fit as well as <code class="docutils literal notranslate">throw()</code>. </section> <section id="new-standard-exception-generatorexit"> <h3><a class="toc-backref" href="#new-standard-exception-generatorexit" role="doc-backlink">New standard exception: <code class="docutils literal notranslate">GeneratorExit</code></a></h3> A new standard exception is defined, <code class="docutils literal notranslate">GeneratorExit</code>, inheriting from <code class="docutils literal notranslate">Exception</code>. A generator should handle this by re-raising it (or just not catching it) or by raising <code class="docutils literal notranslate">StopIteration</code>. </section> <section id="new-generator-method-close"> <h3><a class="toc-backref" href="#new-generator-method-close" role="doc-backlink">New generator method: <code class="docutils literal notranslate">close()</code></a></h3> <code class="docutils literal notranslate">g.close()</code> is defined by the following pseudo-code: <div class="highlight-default notranslate"><div class="highlight"><pre>def close(self): try: self.throw(GeneratorExit) except (GeneratorExit, StopIteration): pass else: raise RuntimeError("generator ignored GeneratorExit") # Other exceptions are not caught </pre></div> </div> </section> <section id="new-generator-method-del"> <h3><a class="toc-backref" href="#new-generator-method-del" role="doc-backlink">New generator method: __del__()</a></h3> <code class="docutils literal notranslate">g.__del__()</code> is a wrapper for <code class="docutils literal notranslate">g.close()</code>. This will be called when the generator object is garbage-collected (in CPython, this is when its reference count goes to zero). If <code class="docutils literal notranslate">close()</code> raises an exception, a traceback for the exception is printed to <code class="docutils literal notranslate">sys.stderr</code> and further ignored; it is not propagated back to the place that triggered the garbage collection. This is consistent with the handling of exceptions in <code class="docutils literal notranslate">__del__()</code> methods on class instances. If the generator object participates in a cycle, <code class="docutils literal notranslate">g.__del__()</code> may not be called. This is the behavior of CPython’s current garbage collector. The reason for the restriction is that the GC code needs to break a cycle at an arbitrary point in order to collect it, and from then on no Python code should be allowed to see the objects that formed the cycle, as they may be in an invalid state. Objects hanging off a cycle are not subject to this restriction. Note that it is unlikely to see a generator object participate in a cycle in practice. However, storing a generator object in a global variable creates a cycle via the generator frame’s <code class="docutils literal notranslate">f_globals</code> pointer. Another way to create a cycle would be to store a reference to the generator object in a data structure that is passed to the generator as an argument (e.g., if an object has a method that’s a generator, and keeps a reference to a running iterator created by that method). Neither of these cases are very likely given the typical patterns of generator use. Also, in the CPython implementation of this PEP, the frame object used by the generator should be released whenever its execution is terminated due to an error or normal exit. This will ensure that generators that cannot be resumed do not remain part of an uncollectable reference cycle. This allows other code to potentially use <code class="docutils literal notranslate">close()</code> in a <code class="docutils literal notranslate">try/finally</code> or <code class="docutils literal notranslate">with</code> block (per <a class="pep reference internal" href="../pep-0343/" title="PEP 343 – The “with” Statement">PEP 343</a>) to ensure that a given generator is properly finalized. </section> </section> <section id="optional-extensions"> <h2><a class="toc-backref" href="#optional-extensions" role="doc-backlink">Optional Extensions</a></h2> <section id="the-extended-continue-statement"> <h3><a class="toc-backref" href="#the-extended-continue-statement" role="doc-backlink">The Extended <code class="docutils literal notranslate">continue</code> Statement</a></h3> An earlier draft of this PEP proposed a new <code class="docutils literal notranslate">continue EXPR</code> syntax for use in for-loops (carried over from <a class="pep reference internal" href="../pep-0340/" title="PEP 340 – Anonymous Block Statements">PEP 340</a>), that would pass the value of EXPR into the iterator being looped over. This feature has been withdrawn for the time being, because the scope of this PEP has been narrowed to focus only on passing values into generator-iterators, and not other kinds of iterators. It was also felt by some on the Python-Dev list that adding new syntax for this particular feature would be premature at best. </section> </section> <section id="open-issues"> <h2><a class="toc-backref" href="#open-issues" role="doc-backlink">Open Issues</a></h2> Discussion on python-dev has revealed some open issues. I list them here, with my preferred resolution and its motivation. The PEP as currently written reflects this preferred resolution. <ol class="arabic"> <li>What exception should be raised by <code class="docutils literal notranslate">close()</code> when the generator yields another value as a response to the <code class="docutils literal notranslate">GeneratorExit</code> exception?I originally chose <code class="docutils literal notranslate">TypeError</code> because it represents gross misbehavior of the generator function, which should be fixed by changing the code. But the <code class="docutils literal notranslate">with_template</code> decorator class in <a class="pep reference internal" href="../pep-0343/" title="PEP 343 – The “with” Statement">PEP 343</a> uses <code class="docutils literal notranslate">RuntimeError</code> for similar offenses. Arguably they should all use the same exception. I’d rather not introduce a new exception class just for this purpose, since it’s not an exception that I want people to catch: I want it to turn into a traceback which is seen by the programmer who then fixes the code. So now I believe they should both raise <code class="docutils literal notranslate">RuntimeError</code>. There are some precedents for that: it’s raised by the core Python code in situations where endless recursion is detected, and for uninitialized objects (and for a variety of miscellaneous conditions). </li> <li>Oren Tirosh has proposed renaming the <code class="docutils literal notranslate">send()</code> method to <code class="docutils literal notranslate">feed()</code>, for compatibility with the consumer interface (see <a class="reference external" href="http://effbot.org/zone/consumer.htm">http://effbot.org/zone/consumer.htm</a> for the specification.)However, looking more closely at the consumer interface, it seems that the desired semantics for <code class="docutils literal notranslate">feed()</code> are different than for <code class="docutils literal notranslate">send()</code>, because <code class="docutils literal notranslate">send()</code> can’t be meaningfully called on a just-started generator. Also, the consumer interface as currently defined doesn’t include handling for <code class="docutils literal notranslate">StopIteration</code>. Therefore, it seems like it would probably be more useful to create a simple decorator that wraps a generator function to make it conform to the consumer interface. For example, it could warm up the generator with an initial <code class="docutils literal notranslate">next()</code> call, trap StopIteration, and perhaps even provide <code class="docutils literal notranslate">reset()</code> by re-invoking the generator function. </li> </ol> </section> <section id="examples"> <h2><a class="toc-backref" href="#examples" role="doc-backlink">Examples</a></h2> <ol class="arabic"> <li>A simple consumer decorator that makes a generator function automatically advance to its first yield point when initially called:<div class="highlight-default notranslate"><div class="highlight"><pre>def consumer(func): def wrapper(*args,**kw): gen = func(*args, **kw) gen.next() return gen wrapper.__name__ = func.__name__ wrapper.__dict__ = func.__dict__ wrapper.__doc__ = func.__doc__ return wrapper </pre></div> </div> </li> <li>An example of using the consumer decorator to create a reverse generator that receives images and creates thumbnail pages, sending them on to another consumer. Functions like this can be chained together to form efficient processing pipelines of consumers that each can have complex internal state:<div class="highlight-default notranslate"><div class="highlight"><pre>@consumer def thumbnail_pager(pagesize, thumbsize, destination): while True: page = new_image(pagesize) rows, columns = pagesize / thumbsize pending = False try: for row in xrange(rows): for column in xrange(columns): thumb = create_thumbnail((yield), thumbsize) page.write( thumb, col*thumbsize.x, row*thumbsize.y ) pending = True except GeneratorExit: # close() was called, so flush any pending output if pending: destination.send(page) # then close the downstream consumer, and exit destination.close() return else: # we finished a page full of thumbnails, so send it # downstream and keep on looping destination.send(page) @consumer def jpeg_writer(dirname): fileno = 1 while True: filename = os.path.join(dirname,"page%04d.jpg" % fileno) write_jpeg((yield), filename) fileno += 1 # Put them together to make a function that makes thumbnail # pages from a list of images and other parameters. # def write_thumbnails(pagesize, thumbsize, images, output_dir): pipeline = thumbnail_pager( pagesize, thumbsize, jpeg_writer(output_dir) ) for image in images: pipeline.send(image) pipeline.close() </pre></div> </div> </li> <li>A simple co-routine scheduler or trampoline that lets coroutines call other coroutines by yielding the coroutine they wish to invoke. Any non-generator value yielded by a coroutine is returned to the coroutine that called the one yielding the value. Similarly, if a coroutine raises an exception, the exception is propagated to its caller. In effect, this example emulates simple tasklets as are used in Stackless Python, as long as you use a yield expression to invoke routines that would otherwise block. This is only a very simple example, and far more sophisticated schedulers are possible. (For example, the existing GTasklet framework for Python (<a class="reference external" href="http://www.gnome.org/~gjc/gtasklet/gtasklets.html">http://www.gnome.org/~gjc/gtasklet/gtasklets.html</a>) and the peak.events framework (<a class="reference external" href="http://peak.telecommunity.com/">http://peak.telecommunity.com/</a>) already implement similar scheduling capabilities, but must currently use awkward workarounds for the inability to pass values or exceptions into generators.)<div class="highlight-default notranslate"><div class="highlight"><pre>import collections class Trampoline: """Manage communications between coroutines""" running = False def __init__(self): self.queue = collections.deque() def add(self, coroutine): """Request that a coroutine be executed""" self.schedule(coroutine) def run(self): result = None self.running = True try: while self.running and self.queue: func = self.queue.popleft() result = func() return result finally: self.running = False def stop(self): self.running = False def schedule(self, coroutine, stack=(), val=None, *exc): def resume(): value = val try: if exc: value = coroutine.throw(value,*exc) else: value = coroutine.send(value) except: if stack: # send the error back to the "caller" self.schedule( stack[0], stack[1], *sys.exc_info() ) else: # Nothing left in this pseudothread to # handle it, let it propagate to the # run loop raise if isinstance(value, types.GeneratorType): # Yielded to a specific coroutine, push the # current one on the stack, and call the new # one with no args self.schedule(value, (coroutine,stack)) elif stack: # Yielded a result, pop the stack and send the # value to the caller self.schedule(stack[0], stack[1], value) # else: this pseudothread has ended self.queue.append(resume) </pre></div> </div> </li> <li>A simple echo server, and code to run it using a trampoline (presumes the existence of <code class="docutils literal notranslate">nonblocking_read</code>, <code class="docutils literal notranslate">nonblocking_write</code>, and other I/O coroutines, that e.g. raise <code class="docutils literal notranslate">ConnectionLost</code> if the connection is closed):<div class="highlight-default notranslate"><div class="highlight"><pre># coroutine function that echos data back on a connected # socket # def echo_handler(sock): while True: try: data = yield nonblocking_read(sock) yield nonblocking_write(sock, data) except ConnectionLost: pass # exit normally if connection lost # coroutine function that listens for connections on a # socket, and then launches a service "handler" coroutine # to service the connection # def listen_on(trampoline, sock, handler): while True: # get the next incoming connection connected_socket = yield nonblocking_accept(sock) # start another coroutine to handle the connection trampoline.add( handler(connected_socket) ) # Create a scheduler to manage all our coroutines t = Trampoline() # Create a coroutine instance to run the echo_handler on # incoming connections # server = listen_on( t, listening_socket("localhost","echo"), echo_handler ) # Add the coroutine to the scheduler t.add(server) # loop forever, accepting connections and servicing them # "in parallel" # t.run() </pre></div> </div> </li> </ol> </section> <section id="reference-implementation"> <h2><a class="toc-backref" href="#reference-implementation" role="doc-backlink">Reference Implementation</a></h2> A prototype patch implementing all of the features described in this PEP is available as SourceForge patch #1223381 (<a class="reference external" href="https://bugs.python.org/issue1223381">https://bugs.python.org/issue1223381</a>). This patch was committed to CVS 01-02 August 2005. </section> <section id="acknowledgements"> <h2><a class="toc-backref" href="#acknowledgements" role="doc-backlink">Acknowledgements</a></h2> Raymond Hettinger (<a class="pep reference internal" href="../pep-0288/" title="PEP 288 – Generators Attributes and Exceptions">PEP 288</a>) and Samuele Pedroni (<a class="pep reference internal" href="../pep-0325/" title="PEP 325 – Resource-Release Support for Generators">PEP 325</a>) first formally proposed the ideas of communicating values or exceptions into generators, and the ability to close generators. Timothy Delaney suggested the title of this PEP, and Steven Bethard helped edit a previous version. See also the Acknowledgements section of <a class="pep reference internal" href="../pep-0340/" title="PEP 340 – Anonymous Block Statements">PEP 340</a>. </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> TBD. </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0342.rst">https://github.com/python/peps/blob/main/peps/pep-0342.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0342.rst">2025-02-01 08:59:27 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#introduction">Introduction</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#specification-summary">Specification Summary</a></li> <li><a class="reference internal" href="#specification-sending-values-into-generators">Specification: Sending Values into Generators</a><ul> <li><a class="reference internal" href="#new-generator-method-send-value">New generator method: <code class="docutils literal notranslate">send(value)</code></a></li> <li><a class="reference internal" href="#new-syntax-yield-expressions">New syntax: Yield Expressions</a></li> </ul> </li> <li><a class="reference internal" href="#specification-exceptions-and-cleanup">Specification: Exceptions and Cleanup</a><ul> <li><a class="reference internal" href="#new-syntax-yield-allowed-inside-try-finally">New syntax: <code class="docutils literal notranslate">yield</code> allowed inside <code class="docutils literal notranslate">try-finally</code></a></li> <li><a class="reference internal" href="#new-generator-method-throw-type-value-none-traceback-none">New generator method: <code class="docutils literal notranslate">throw(type, value=None, traceback=None)</code></a></li> <li><a class="reference internal" href="#new-standard-exception-generatorexit">New standard exception: <code class="docutils literal notranslate">GeneratorExit</code></a></li> <li><a class="reference internal" href="#new-generator-method-close">New generator method: <code class="docutils literal notranslate">close()</code></a></li> <li><a class="reference internal" href="#new-generator-method-del">New generator method: __del__()</a></li> </ul> </li> <li><a class="reference internal" href="#optional-extensions">Optional Extensions</a><ul> <li><a class="reference internal" href="#the-extended-continue-statement">The Extended <code class="docutils literal notranslate">continue</code> Statement</a></li> </ul> </li> <li><a class="reference internal" href="#open-issues">Open Issues</a></li> <li><a class="reference internal" href="#examples">Examples</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#acknowledgements">Acknowledgements</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0342.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>