CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 525 – Asynchronous Generators | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0525/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 525 – Asynchronous Generators | peps.python.org'> <meta property="og:description" content="PEP 492 introduced support for native coroutines and async/await syntax to Python 3.5. It is proposed here to extend Python’s asynchronous capabilities by adding support for asynchronous generators."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0525/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="PEP 492 introduced support for native coroutines and async/await syntax to Python 3.5. It is proposed here to extend Python’s asynchronous capabilities by adding support for asynchronous generators."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 525</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 525 – Asynchronous Generators</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Yury Selivanov <yury at edgedb.com></dd> <dt class="field-even">Discussions-To:</dt> <dd class="field-even"><a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/">Python-Dev list</a></dd> <dt class="field-odd">Status:</dt> <dd class="field-odd"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-even">Type:</dt> <dd class="field-even"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-odd">Created:</dt> <dd class="field-odd">28-Jul-2016</dd> <dt class="field-even">Python-Version:</dt> <dd class="field-even">3.6</dd> <dt class="field-odd">Post-History:</dt> <dd class="field-odd">02-Aug-2016, 23-Aug-2016, 01-Sep-2016, 06-Sep-2016</dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale-and-goals">Rationale and Goals</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#asynchronous-generators">Asynchronous Generators</a></li> <li><a class="reference internal" href="#support-for-asynchronous-iteration-protocol">Support for Asynchronous Iteration Protocol</a></li> <li><a class="reference internal" href="#finalization">Finalization</a></li> <li><a class="reference internal" href="#asyncio">asyncio</a></li> <li><a class="reference internal" href="#asynchronous-generator-object">Asynchronous Generator Object</a></li> <li><a class="reference internal" href="#implementation-details">Implementation Details</a><ul> <li><a class="reference internal" href="#pyasyncgenasend-and-pyasyncgenathrow">PyAsyncGenASend and PyAsyncGenAThrow</a></li> </ul> </li> <li><a class="reference internal" href="#new-standard-library-functions-and-types">New Standard Library Functions and Types</a></li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a></li> </ul> </li> <li><a class="reference internal" href="#performance">Performance</a><ul> <li><a class="reference internal" href="#regular-generators">Regular Generators</a></li> <li><a class="reference internal" href="#improvements-over-asynchronous-iterators">Improvements over asynchronous iterators</a></li> </ul> </li> <li><a class="reference internal" href="#design-considerations">Design Considerations</a><ul> <li><a class="reference internal" href="#aiter-and-anext-builtins"><code class="docutils literal notranslate">aiter()</code> and <code class="docutils literal notranslate">anext()</code> builtins</a></li> <li><a class="reference internal" href="#asynchronous-list-dict-set-comprehensions">Asynchronous list/dict/set comprehensions</a></li> <li><a class="reference internal" href="#asynchronous-yield-from">Asynchronous <code class="docutils literal notranslate">yield from</code></a></li> <li><a class="reference internal" href="#why-the-asend-and-athrow-methods-are-necessary">Why the <code class="docutils literal notranslate">asend()</code> and <code class="docutils literal notranslate">athrow()</code> methods are necessary</a></li> </ul> </li> <li><a class="reference internal" href="#example">Example</a></li> <li><a class="reference internal" href="#acceptance">Acceptance</a></li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#acknowledgments">Acknowledgments</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> <a class="pep reference internal" href="../pep-0492/" title="PEP 492 – Coroutines with async and await syntax">PEP 492</a> introduced support for native coroutines and <code class="docutils literal notranslate">async</code>/<code class="docutils literal notranslate">await</code> syntax to Python 3.5. It is proposed here to extend Python’s asynchronous capabilities by adding support for asynchronous generators. </section> <section id="rationale-and-goals"> <h2><a class="toc-backref" href="#rationale-and-goals" role="doc-backlink">Rationale and Goals</a></h2> Regular generators (introduced in <a class="pep reference internal" href="../pep-0255/" title="PEP 255 – Simple Generators">PEP 255</a>) enabled an elegant way of writing complex data producers and have them behave like an iterator. However, currently there is no equivalent concept for the asynchronous iteration protocol (<code class="docutils literal notranslate">async for</code>). This makes writing asynchronous data producers unnecessarily complex, as one must define a class that implements <code class="docutils literal notranslate">__aiter__</code> and <code class="docutils literal notranslate">__anext__</code> to be able to use it in an <code class="docutils literal notranslate">async for</code> statement. Essentially, the goals and rationale for <a class="pep reference internal" href="../pep-0255/" title="PEP 255 – Simple Generators">PEP 255</a>, applied to the asynchronous execution case, hold true for this proposal as well. Performance is an additional point for this proposal: in our testing of the reference implementation, asynchronous generators are 2x faster than an equivalent implemented as an asynchronous iterator. As an illustration of the code quality improvement, consider the following class that prints numbers with a given delay once iterated: <div class="highlight-default notranslate"><div class="highlight"><pre>class Ticker: """Yield numbers from 0 to `to` every `delay` seconds.""" def __init__(self, delay, to): self.delay = delay self.i = 0 self.to = to def __aiter__(self): return self async def __anext__(self): i = self.i if i >= self.to: raise StopAsyncIteration self.i += 1 if i: await asyncio.sleep(self.delay) return i </pre></div> </div> The same can be implemented as a much simpler asynchronous generator: <div class="highlight-default notranslate"><div class="highlight"><pre>async def ticker(delay, to): """Yield numbers from 0 to `to` every `delay` seconds.""" for i in range(to): yield i await asyncio.sleep(delay) </pre></div> </div> </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> This proposal introduces the concept of asynchronous generators to Python. This specification presumes knowledge of the implementation of generators and coroutines in Python (<a class="pep reference internal" href="../pep-0342/" title="PEP 342 – Coroutines via Enhanced Generators">PEP 342</a>, <a class="pep reference internal" href="../pep-0380/" title="PEP 380 – Syntax for Delegating to a Subgenerator">PEP 380</a> and <a class="pep reference internal" href="../pep-0492/" title="PEP 492 – Coroutines with async and await syntax">PEP 492</a>). <section id="asynchronous-generators"> <h3><a class="toc-backref" href="#asynchronous-generators" role="doc-backlink">Asynchronous Generators</a></h3> A Python generator is any function containing one or more <code class="docutils literal notranslate">yield</code> expressions: <div class="highlight-default notranslate"><div class="highlight"><pre>def func(): # a function return def genfunc(): # a generator function yield </pre></div> </div> We propose to use the same approach to define asynchronous generators: <div class="highlight-default notranslate"><div class="highlight"><pre>async def coro(): # a coroutine function await smth() async def asyncgen(): # an asynchronous generator function await smth() yield 42 </pre></div> </div> The result of calling an asynchronous generator function is an asynchronous generator object, which implements the asynchronous iteration protocol defined in <a class="pep reference internal" href="../pep-0492/" title="PEP 492 – Coroutines with async and await syntax">PEP 492</a>. It is a <code class="docutils literal notranslate">SyntaxError</code> to have a non-empty <code class="docutils literal notranslate">return</code> statement in an asynchronous generator. </section> <section id="support-for-asynchronous-iteration-protocol"> <h3><a class="toc-backref" href="#support-for-asynchronous-iteration-protocol" role="doc-backlink">Support for Asynchronous Iteration Protocol</a></h3> The protocol requires two special methods to be implemented: <ol class="arabic simple"> <li>An <code class="docutils literal notranslate">__aiter__</code> method returning an asynchronous iterator.</li> <li>An <code class="docutils literal notranslate">__anext__</code> method returning an awaitable object, which uses <code class="docutils literal notranslate">StopIteration</code> exception to “yield” values, and <code class="docutils literal notranslate">StopAsyncIteration</code> exception to signal the end of the iteration.</li> </ol> Asynchronous generators define both of these methods. Let’s manually iterate over a simple asynchronous generator: <div class="highlight-default notranslate"><div class="highlight"><pre>async def genfunc(): yield 1 yield 2 gen = genfunc() assert gen.__aiter__() is gen assert await gen.__anext__() == 1 assert await gen.__anext__() == 2 await gen.__anext__() # This line will raise StopAsyncIteration. </pre></div> </div> </section> <section id="finalization"> <h3><a class="toc-backref" href="#finalization" role="doc-backlink">Finalization</a></h3> <a class="pep reference internal" href="../pep-0492/" title="PEP 492 – Coroutines with async and await syntax">PEP 492</a> requires an event loop or a scheduler to run coroutines. Because asynchronous generators are meant to be used from coroutines, they also require an event loop to run and finalize them. Asynchronous generators can have <code class="docutils literal notranslate">try..finally</code> blocks, as well as <code class="docutils literal notranslate">async with</code>. It is important to provide a guarantee that, even when partially iterated, and then garbage collected, generators can be safely finalized. For example: <div class="highlight-default notranslate"><div class="highlight"><pre>async def square_series(con, to): async with con.transaction(): cursor = con.cursor( 'SELECT generate_series(0, $1) AS i', to) async for row in cursor: yield row['i'] ** 2 async for i in square_series(con, 1000): if i == 100: break </pre></div> </div> The above code defines an asynchronous generator that uses <code class="docutils literal notranslate">async with</code> to iterate over a database cursor in a transaction. The generator is then iterated over with <code class="docutils literal notranslate">async for</code>, which interrupts the iteration at some point. The <code class="docutils literal notranslate">square_series()</code> generator will then be garbage collected, and without a mechanism to asynchronously close the generator, Python interpreter would not be able to do anything. To solve this problem we propose to do the following: <ol class="arabic"> <li>Implement an <code class="docutils literal notranslate">aclose</code> method on asynchronous generators returning a special awaitable. When awaited it throws a <code class="docutils literal notranslate">GeneratorExit</code> into the suspended generator and iterates over it until either a <code class="docutils literal notranslate">GeneratorExit</code> or a <code class="docutils literal notranslate">StopAsyncIteration</code> occur.This is very similar to what the <code class="docutils literal notranslate">close()</code> method does to regular Python generators, except that an event loop is required to execute <code class="docutils literal notranslate">aclose()</code>. </li> <li>Raise a <code class="docutils literal notranslate">RuntimeError</code>, when an asynchronous generator executes a <code class="docutils literal notranslate">yield</code> expression in its <code class="docutils literal notranslate">finally</code> block (using <code class="docutils literal notranslate">await</code> is fine, though):<div class="highlight-default notranslate"><div class="highlight"><pre>async def gen(): try: yield finally: await asyncio.sleep(1) # Can use 'await'. yield # Cannot use 'yield', # this line will trigger a # RuntimeError. </pre></div> </div> </li> <li>Add two new methods to the <code class="docutils literal notranslate">sys</code> module: <code class="docutils literal notranslate">set_asyncgen_hooks()</code> and <code class="docutils literal notranslate">get_asyncgen_hooks()</code>.</li> </ol> The idea behind <code class="docutils literal notranslate">sys.set_asyncgen_hooks()</code> is to allow event loops to intercept asynchronous generators iteration and finalization, so that the end user does not need to care about the finalization problem, and everything just works. <code class="docutils literal notranslate">sys.set_asyncgen_hooks()</code> accepts two arguments: <ul class="simple"> <li><code class="docutils literal notranslate">firstiter</code>: a callable which will be called when an asynchronous generator is iterated for the first time.</li> <li><code class="docutils literal notranslate">finalizer</code>: a callable which will be called when an asynchronous generator is about to be GCed.</li> </ul> When an asynchronous generator is iterated for the first time, it stores a reference to the current finalizer. When an asynchronous generator is about to be garbage collected, it calls its cached finalizer. The assumption is that the finalizer will schedule an <code class="docutils literal notranslate">aclose()</code> call with the loop that was active when the iteration started. For instance, here is how asyncio is modified to allow safe finalization of asynchronous generators: <div class="highlight-default notranslate"><div class="highlight"><pre># asyncio/base_events.py class BaseEventLoop: def run_forever(self): ... old_hooks = sys.get_asyncgen_hooks() sys.set_asyncgen_hooks(finalizer=self._finalize_asyncgen) try: ... finally: sys.set_asyncgen_hooks(*old_hooks) ... def _finalize_asyncgen(self, gen): self.create_task(gen.aclose()) </pre></div> </div> The second argument, <code class="docutils literal notranslate">firstiter</code>, allows event loops to maintain a weak set of asynchronous generators instantiated under their control. This makes it possible to implement “shutdown” mechanisms to safely finalize all open generators and close the event loop. <code class="docutils literal notranslate">sys.set_asyncgen_hooks()</code> is thread-specific, so several event loops running in parallel threads can use it safely. <code class="docutils literal notranslate">sys.get_asyncgen_hooks()</code> returns a namedtuple-like structure with <code class="docutils literal notranslate">firstiter</code> and <code class="docutils literal notranslate">finalizer</code> fields. </section> <section id="asyncio"> <h3><a class="toc-backref" href="#asyncio" role="doc-backlink">asyncio</a></h3> The asyncio event loop will use <code class="docutils literal notranslate">sys.set_asyncgen_hooks()</code> API to maintain a weak set of all scheduled asynchronous generators, and to schedule their <code class="docutils literal notranslate">aclose()</code> coroutine methods when it is time for generators to be GCed. To make sure that asyncio programs can finalize all scheduled asynchronous generators reliably, we propose to add a new event loop coroutine method <code class="docutils literal notranslate">loop.shutdown_asyncgens()</code>. The method will schedule all currently open asynchronous generators to close with an <code class="docutils literal notranslate">aclose()</code> call. After calling the <code class="docutils literal notranslate">loop.shutdown_asyncgens()</code> method, the event loop will issue a warning whenever a new asynchronous generator is iterated for the first time. The idea is that after requesting all asynchronous generators to be shutdown, the program should not execute code that iterates over new asynchronous generators. An example of how <code class="docutils literal notranslate">shutdown_asyncgens</code> coroutine should be used: <div class="highlight-default notranslate"><div class="highlight"><pre>try: loop.run_forever() finally: loop.run_until_complete(loop.shutdown_asyncgens()) loop.close() </pre></div> </div> </section> <section id="asynchronous-generator-object"> <h3><a class="toc-backref" href="#asynchronous-generator-object" role="doc-backlink">Asynchronous Generator Object</a></h3> The object is modeled after the standard Python generator object. Essentially, the behaviour of asynchronous generators is designed to replicate the behaviour of synchronous generators, with the only difference in that the API is asynchronous. The following methods and properties are defined: <ol class="arabic"> <li><code class="docutils literal notranslate">agen.__aiter__()</code>: Returns <code class="docutils literal notranslate">agen</code>.</li> <li><code class="docutils literal notranslate">agen.__anext__()</code>: Returns an awaitable, that performs one asynchronous generator iteration when awaited.</li> <li><code class="docutils literal notranslate">agen.asend(val)</code>: Returns an awaitable, that pushes the <code class="docutils literal notranslate">val</code> object in the <code class="docutils literal notranslate">agen</code> generator. When the <code class="docutils literal notranslate">agen</code> has not yet been iterated, <code class="docutils literal notranslate">val</code> must be <code class="docutils literal notranslate">None</code>.Example: <div class="highlight-default notranslate"><div class="highlight"><pre>async def gen(): await asyncio.sleep(0.1) v = yield 42 print(v) await asyncio.sleep(0.2) g = gen() await g.asend(None) # Will return 42 after sleeping # for 0.1 seconds. await g.asend('hello') # Will print 'hello' and # raise StopAsyncIteration # (after sleeping for 0.2 seconds.) </pre></div> </div> </li> <li><code class="docutils literal notranslate">agen.athrow(typ, [val, [tb]])</code>: Returns an awaitable, that throws an exception into the <code class="docutils literal notranslate">agen</code> generator.Example: <div class="highlight-default notranslate"><div class="highlight"><pre>async def gen(): try: await asyncio.sleep(0.1) yield 'hello' except ZeroDivisionError: await asyncio.sleep(0.2) yield 'world' g = gen() v = await g.asend(None) print(v) # Will print 'hello' after # sleeping for 0.1 seconds. v = await g.athrow(ZeroDivisionError) print(v) # Will print 'world' after # sleeping 0.2 seconds. </pre></div> </div> </li> <li><code class="docutils literal notranslate">agen.aclose()</code>: Returns an awaitable, that throws a <code class="docutils literal notranslate">GeneratorExit</code> exception into the generator. The awaitable can either return a yielded value, if <code class="docutils literal notranslate">agen</code> handled the exception, or <code class="docutils literal notranslate">agen</code> will be closed and the exception will propagate back to the caller.</li> <li><code class="docutils literal notranslate">agen.__name__</code> and <code class="docutils literal notranslate">agen.__qualname__</code>: readable and writable name and qualified name attributes.</li> <li><code class="docutils literal notranslate">agen.ag_await</code>: The object that <code class="docutils literal notranslate">agen</code> is currently awaiting on, or <code class="docutils literal notranslate">None</code>. This is similar to the currently available <code class="docutils literal notranslate">gi_yieldfrom</code> for generators and <code class="docutils literal notranslate">cr_await</code> for coroutines.</li> <li><code class="docutils literal notranslate">agen.ag_frame</code>, <code class="docutils literal notranslate">agen.ag_running</code>, and <code class="docutils literal notranslate">agen.ag_code</code>: defined in the same way as similar attributes of standard generators.</li> </ol> <code class="docutils literal notranslate">StopIteration</code> and <code class="docutils literal notranslate">StopAsyncIteration</code> are not propagated out of asynchronous generators, and are replaced with a <code class="docutils literal notranslate">RuntimeError</code>. </section> <section id="implementation-details"> <h3><a class="toc-backref" href="#implementation-details" role="doc-backlink">Implementation Details</a></h3> Asynchronous generator object (<code class="docutils literal notranslate">PyAsyncGenObject</code>) shares the struct layout with <code class="docutils literal notranslate">PyGenObject</code>. In addition to that, the reference implementation introduces three new objects: <ol class="arabic simple"> <li><code class="docutils literal notranslate">PyAsyncGenASend</code>: the awaitable object that implements <code class="docutils literal notranslate">__anext__</code> and <code class="docutils literal notranslate">asend()</code> methods.</li> <li><code class="docutils literal notranslate">PyAsyncGenAThrow</code>: the awaitable object that implements <code class="docutils literal notranslate">athrow()</code> and <code class="docutils literal notranslate">aclose()</code> methods.</li> <li><code class="docutils literal notranslate">_PyAsyncGenWrappedValue</code>: every directly yielded object from an asynchronous generator is implicitly boxed into this structure. This is how the generator implementation can separate objects that are yielded using regular iteration protocol from objects that are yielded using asynchronous iteration protocol.</li> </ol> <code class="docutils literal notranslate">PyAsyncGenASend</code> and <code class="docutils literal notranslate">PyAsyncGenAThrow</code> are awaitables (they have <code class="docutils literal notranslate">__await__</code> methods returning <code class="docutils literal notranslate">self</code>) and are coroutine-like objects (implementing <code class="docutils literal notranslate">__iter__</code>, <code class="docutils literal notranslate">__next__</code>, <code class="docutils literal notranslate">send()</code> and <code class="docutils literal notranslate">throw()</code> methods). Essentially, they control how asynchronous generators are iterated: <a class="invert-in-dark-mode reference internal image-reference" href="../_images/pep-0525-1.png"><img alt="../_images/pep-0525-1.png" class="invert-in-dark-mode align-center" src="../_images/pep-0525-1.png" style="width: 80%;" /> </a> <section id="pyasyncgenasend-and-pyasyncgenathrow"> <h4><a class="toc-backref" href="#pyasyncgenasend-and-pyasyncgenathrow" role="doc-backlink">PyAsyncGenASend and PyAsyncGenAThrow</a></h4> <code class="docutils literal notranslate">PyAsyncGenASend</code> is a coroutine-like object that drives <code class="docutils literal notranslate">__anext__</code> and <code class="docutils literal notranslate">asend()</code> methods and implements the asynchronous iteration protocol. <code class="docutils literal notranslate">agen.asend(val)</code> and <code class="docutils literal notranslate">agen.__anext__()</code> return instances of <code class="docutils literal notranslate">PyAsyncGenASend</code> (which hold references back to the parent <code class="docutils literal notranslate">agen</code> object.) The data flow is defined as follows: <ol class="arabic"> <li>When <code class="docutils literal notranslate">PyAsyncGenASend.send(val)</code> is called for the first time, <code class="docutils literal notranslate">val</code> is pushed to the parent <code class="docutils literal notranslate">agen</code> object (using existing facilities of <code class="docutils literal notranslate">PyGenObject</code>.)Subsequent iterations over the <code class="docutils literal notranslate">PyAsyncGenASend</code> objects, push <code class="docutils literal notranslate">None</code> to <code class="docutils literal notranslate">agen</code>. When a <code class="docutils literal notranslate">_PyAsyncGenWrappedValue</code> object is yielded, it is unboxed, and a <code class="docutils literal notranslate">StopIteration</code> exception is raised with the unwrapped value as an argument. </li> <li>When <code class="docutils literal notranslate">PyAsyncGenASend.throw(*exc)</code> is called for the first time, <code class="docutils literal notranslate">*exc</code> is thrown into the parent <code class="docutils literal notranslate">agen</code> object.Subsequent iterations over the <code class="docutils literal notranslate">PyAsyncGenASend</code> objects, push <code class="docutils literal notranslate">None</code> to <code class="docutils literal notranslate">agen</code>. When a <code class="docutils literal notranslate">_PyAsyncGenWrappedValue</code> object is yielded, it is unboxed, and a <code class="docutils literal notranslate">StopIteration</code> exception is raised with the unwrapped value as an argument. </li> <li><code class="docutils literal notranslate">return</code> statements in asynchronous generators raise <code class="docutils literal notranslate">StopAsyncIteration</code> exception, which is propagated through <code class="docutils literal notranslate">PyAsyncGenASend.send()</code> and <code class="docutils literal notranslate">PyAsyncGenASend.throw()</code> methods.</li> </ol> <code class="docutils literal notranslate">PyAsyncGenAThrow</code> is very similar to <code class="docutils literal notranslate">PyAsyncGenASend</code>. The only difference is that <code class="docutils literal notranslate">PyAsyncGenAThrow.send()</code>, when called first time, throws an exception into the parent <code class="docutils literal notranslate">agen</code> object (instead of pushing a value into it.) </section> </section> <section id="new-standard-library-functions-and-types"> <h3><a class="toc-backref" href="#new-standard-library-functions-and-types" role="doc-backlink">New Standard Library Functions and Types</a></h3> <ol class="arabic simple"> <li><code class="docutils literal notranslate">types.AsyncGeneratorType</code> – type of asynchronous generator object.</li> <li><code class="docutils literal notranslate">sys.set_asyncgen_hooks()</code> and <code class="docutils literal notranslate">sys.get_asyncgen_hooks()</code> methods to set up asynchronous generators finalizers and iteration interceptors in event loops.</li> <li><code class="docutils literal notranslate">inspect.isasyncgen()</code> and <code class="docutils literal notranslate">inspect.isasyncgenfunction()</code> introspection functions.</li> <li>New method for asyncio event loop: <code class="docutils literal notranslate">loop.shutdown_asyncgens()</code>.</li> <li>New <code class="docutils literal notranslate">collections.abc.AsyncGenerator</code> abstract base class.</li> </ol> </section> <section id="backwards-compatibility"> <h3><a class="toc-backref" href="#backwards-compatibility" role="doc-backlink">Backwards Compatibility</a></h3> The proposal is fully backwards compatible. In Python 3.5 it is a <code class="docutils literal notranslate">SyntaxError</code> to define an <code class="docutils literal notranslate">async def</code> function with a <code class="docutils literal notranslate">yield</code> expression inside, therefore it’s safe to introduce asynchronous generators in 3.6. </section> </section> <section id="performance"> <h2><a class="toc-backref" href="#performance" role="doc-backlink">Performance</a></h2> <section id="regular-generators"> <h3><a class="toc-backref" href="#regular-generators" role="doc-backlink">Regular Generators</a></h3> There is no performance degradation for regular generators. The following micro benchmark runs at the same speed on CPython with and without asynchronous generators: <div class="highlight-default notranslate"><div class="highlight"><pre>def gen(): i = 0 while i < 100000000: yield i i += 1 list(gen()) </pre></div> </div> </section> <section id="improvements-over-asynchronous-iterators"> <h3><a class="toc-backref" href="#improvements-over-asynchronous-iterators" role="doc-backlink">Improvements over asynchronous iterators</a></h3> The following micro-benchmark shows that asynchronous generators are about 2.3x faster than asynchronous iterators implemented in pure Python: <div class="highlight-default notranslate"><div class="highlight"><pre>N = 10 ** 7 async def agen(): for i in range(N): yield i class AIter: def __init__(self): self.i = 0 def __aiter__(self): return self async def __anext__(self): i = self.i if i >= N: raise StopAsyncIteration self.i += 1 return i </pre></div> </div> </section> </section> <section id="design-considerations"> <h2><a class="toc-backref" href="#design-considerations" role="doc-backlink">Design Considerations</a></h2> <section id="aiter-and-anext-builtins"> <h3><a class="toc-backref" href="#aiter-and-anext-builtins" role="doc-backlink"><code class="docutils literal notranslate">aiter()</code> and <code class="docutils literal notranslate">anext()</code> builtins</a></h3> Originally, <a class="pep reference internal" href="../pep-0492/" title="PEP 492 – Coroutines with async and await syntax">PEP 492</a> defined <code class="docutils literal notranslate">__aiter__</code> as a method that should return an awaitable object, resulting in an asynchronous iterator. However, in CPython 3.5.2, <code class="docutils literal notranslate">__aiter__</code> was redefined to return asynchronous iterators directly. To avoid breaking backwards compatibility, it was decided that Python 3.6 will support both ways: <code class="docutils literal notranslate">__aiter__</code> can still return an awaitable with a <code class="docutils literal notranslate">DeprecationWarning</code> being issued. Because of this dual nature of <code class="docutils literal notranslate">__aiter__</code> in Python 3.6, we cannot add a synchronous implementation of <code class="docutils literal notranslate">aiter()</code> built-in. Therefore, it is proposed to wait until Python 3.7. </section> <section id="asynchronous-list-dict-set-comprehensions"> <h3><a class="toc-backref" href="#asynchronous-list-dict-set-comprehensions" role="doc-backlink">Asynchronous list/dict/set comprehensions</a></h3> Syntax for asynchronous comprehensions is unrelated to the asynchronous generators machinery, and should be considered in a separate PEP. </section> <section id="asynchronous-yield-from"> <h3><a class="toc-backref" href="#asynchronous-yield-from" role="doc-backlink">Asynchronous <code class="docutils literal notranslate">yield from</code></a></h3> While it is theoretically possible to implement <code class="docutils literal notranslate">yield from</code> support for asynchronous generators, it would require a serious redesign of the generators implementation. <code class="docutils literal notranslate">yield from</code> is also less critical for asynchronous generators, since there is no need provide a mechanism of implementing another coroutines protocol on top of coroutines. And to compose asynchronous generators a simple <code class="docutils literal notranslate">async for</code> loop can be used: <div class="highlight-default notranslate"><div class="highlight"><pre>async def g1(): yield 1 yield 2 async def g2(): async for v in g1(): yield v </pre></div> </div> </section> <section id="why-the-asend-and-athrow-methods-are-necessary"> <h3><a class="toc-backref" href="#why-the-asend-and-athrow-methods-are-necessary" role="doc-backlink">Why the <code class="docutils literal notranslate">asend()</code> and <code class="docutils literal notranslate">athrow()</code> methods are necessary</a></h3> They make it possible to implement concepts similar to <code class="docutils literal notranslate">contextlib.contextmanager</code> using asynchronous generators. For instance, with the proposed design, it is possible to implement the following pattern: <div class="highlight-default notranslate"><div class="highlight"><pre>@async_context_manager async def ctx(): await open() try: yield finally: await close() async with ctx(): await ... </pre></div> </div> Another reason is that it is possible to push data and throw exceptions into asynchronous generators using the object returned from <code class="docutils literal notranslate">__anext__</code> object, but it is hard to do that correctly. Adding explicit <code class="docutils literal notranslate">asend()</code> and <code class="docutils literal notranslate">athrow()</code> will pave a safe way to accomplish that. In terms of implementation, <code class="docutils literal notranslate">asend()</code> is a slightly more generic version of <code class="docutils literal notranslate">__anext__</code>, and <code class="docutils literal notranslate">athrow()</code> is very similar to <code class="docutils literal notranslate">aclose()</code>. Therefore, having these methods defined for asynchronous generators does not add any extra complexity. </section> </section> <section id="example"> <h2><a class="toc-backref" href="#example" role="doc-backlink">Example</a></h2> A working example with the current reference implementation (will print numbers from 0 to 9 with one second delay): <div class="highlight-default notranslate"><div class="highlight"><pre>async def ticker(delay, to): for i in range(to): yield i await asyncio.sleep(delay) async def run(): async for i in ticker(1, 10): print(i) import asyncio loop = asyncio.get_event_loop() try: loop.run_until_complete(run()) finally: loop.close() </pre></div> </div> </section> <section id="acceptance"> <h2><a class="toc-backref" href="#acceptance" role="doc-backlink">Acceptance</a></h2> <a class="pep reference internal" href="../pep-0525/" title="PEP 525 – Asynchronous Generators">PEP 525</a> was accepted by Guido, September 6, 2016 <a class="footnote-reference brackets" href="#id5" id="id1">[2]</a>. </section> <section id="implementation"> <h2><a class="toc-backref" href="#implementation" role="doc-backlink">Implementation</a></h2> The implementation is tracked in issue 28003 <a class="footnote-reference brackets" href="#id6" id="id2">[3]</a>. The reference implementation git repository is available at <a class="footnote-reference brackets" href="#id4" id="id3">[1]</a>. </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <aside class="footnote-list brackets"> <aside class="footnote brackets" id="id4" role="doc-footnote"> <dt class="label" id="id4">[<a href="#id3">1</a>]</dt> <dd><a class="reference external" href="https://github.com/1st1/cpython/tree/async_gen">https://github.com/1st1/cpython/tree/async_gen</a></aside> <aside class="footnote brackets" id="id5" role="doc-footnote"> <dt class="label" id="id5">[<a href="#id1">2</a>]</dt> <dd><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2016-September/146267.html">https://mail.python.org/pipermail/python-dev/2016-September/146267.html</a></aside> <aside class="footnote brackets" id="id6" role="doc-footnote"> <dt class="label" id="id6">[<a href="#id2">3</a>]</dt> <dd><a class="reference external" href="http://bugs.python.org/issue28003">http://bugs.python.org/issue28003</a></aside> </aside> </section> <section id="acknowledgments"> <h2><a class="toc-backref" href="#acknowledgments" role="doc-backlink">Acknowledgments</a></h2> I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Nathaniel Smith, Łukasz Langa, Andrew Svetlov and many others for their feedback, code reviews, and discussions around this PEP. </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0525.rst">https://github.com/python/peps/blob/main/peps/pep-0525.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0525.rst">2025-02-01 08:59:27 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale-and-goals">Rationale and Goals</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#asynchronous-generators">Asynchronous Generators</a></li> <li><a class="reference internal" href="#support-for-asynchronous-iteration-protocol">Support for Asynchronous Iteration Protocol</a></li> <li><a class="reference internal" href="#finalization">Finalization</a></li> <li><a class="reference internal" href="#asyncio">asyncio</a></li> <li><a class="reference internal" href="#asynchronous-generator-object">Asynchronous Generator Object</a></li> <li><a class="reference internal" href="#implementation-details">Implementation Details</a><ul> <li><a class="reference internal" href="#pyasyncgenasend-and-pyasyncgenathrow">PyAsyncGenASend and PyAsyncGenAThrow</a></li> </ul> </li> <li><a class="reference internal" href="#new-standard-library-functions-and-types">New Standard Library Functions and Types</a></li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a></li> </ul> </li> <li><a class="reference internal" href="#performance">Performance</a><ul> <li><a class="reference internal" href="#regular-generators">Regular Generators</a></li> <li><a class="reference internal" href="#improvements-over-asynchronous-iterators">Improvements over asynchronous iterators</a></li> </ul> </li> <li><a class="reference internal" href="#design-considerations">Design Considerations</a><ul> <li><a class="reference internal" href="#aiter-and-anext-builtins"><code class="docutils literal notranslate">aiter()</code> and <code class="docutils literal notranslate">anext()</code> builtins</a></li> <li><a class="reference internal" href="#asynchronous-list-dict-set-comprehensions">Asynchronous list/dict/set comprehensions</a></li> <li><a class="reference internal" href="#asynchronous-yield-from">Asynchronous <code class="docutils literal notranslate">yield from</code></a></li> <li><a class="reference internal" href="#why-the-asend-and-athrow-methods-are-necessary">Why the <code class="docutils literal notranslate">asend()</code> and <code class="docutils literal notranslate">athrow()</code> methods are necessary</a></li> </ul> </li> <li><a class="reference internal" href="#example">Example</a></li> <li><a class="reference internal" href="#acceptance">Acceptance</a></li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#acknowledgments">Acknowledgments</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0525.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>