CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 492 – Coroutines with async and await syntax | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0492/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 492 – Coroutines with async and await syntax | peps.python.org'> <meta property="og:description" content="The growth of Internet and general connectivity has triggered the proportionate need for responsive and scalable code. This proposal aims to answer that need by making writing explicitly asynchronous, concurrent Python code easier and more Pythonic."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0492/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="The growth of Internet and general connectivity has triggered the proportionate need for responsive and scalable code. This proposal aims to answer that need by making writing explicitly asynchronous, concurrent Python code easier and more Pythonic."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 492</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 492 – Coroutines with async and await syntax</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Yury Selivanov <yury at edgedb.com></dd> <dt class="field-even">Discussions-To:</dt> <dd class="field-even"><a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/">Python-Dev list</a></dd> <dt class="field-odd">Status:</dt> <dd class="field-odd"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-even">Type:</dt> <dd class="field-even"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-odd">Created:</dt> <dd class="field-odd">09-Apr-2015</dd> <dt class="field-even">Python-Version:</dt> <dd class="field-even">3.5</dd> <dt class="field-odd">Post-History:</dt> <dd class="field-odd">17-Apr-2015, 21-Apr-2015, 27-Apr-2015, 29-Apr-2015, 05-May-2015</dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#api-design-and-implementation-revisions">API Design and Implementation Revisions</a></li> <li><a class="reference internal" href="#rationale-and-goals">Rationale and Goals</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#new-coroutine-declaration-syntax">New Coroutine Declaration Syntax</a></li> <li><a class="reference internal" href="#types-coroutine">types.coroutine()</a></li> <li><a class="reference internal" href="#await-expression">Await Expression</a><ul> <li><a class="reference internal" href="#updated-operator-precedence-table">Updated operator precedence table</a></li> <li><a class="reference internal" href="#examples-of-await-expressions">Examples of “await” expressions</a></li> </ul> </li> <li><a class="reference internal" href="#asynchronous-context-managers-and-async-with">Asynchronous Context Managers and “async with”</a><ul> <li><a class="reference internal" href="#new-syntax">New Syntax</a></li> <li><a class="reference internal" href="#example">Example</a></li> </ul> </li> <li><a class="reference internal" href="#asynchronous-iterators-and-async-for">Asynchronous Iterators and “async for”</a><ul> <li><a class="reference internal" href="#id12">New Syntax</a></li> <li><a class="reference internal" href="#example-1">Example 1</a></li> <li><a class="reference internal" href="#example-2">Example 2</a></li> <li><a class="reference internal" href="#why-stopasynciteration">Why StopAsyncIteration?</a></li> </ul> </li> <li><a class="reference internal" href="#coroutine-objects">Coroutine objects</a><ul> <li><a class="reference internal" href="#differences-from-generators">Differences from generators</a></li> <li><a class="reference internal" href="#coroutine-object-methods">Coroutine object methods</a></li> </ul> </li> <li><a class="reference internal" href="#debugging-features">Debugging Features</a></li> <li><a class="reference internal" href="#new-standard-library-functions">New Standard Library Functions</a></li> <li><a class="reference internal" href="#new-abstract-base-classes">New Abstract Base Classes</a></li> </ul> </li> <li><a class="reference internal" href="#glossary">Glossary</a></li> <li><a class="reference internal" href="#transition-plan">Transition Plan</a><ul> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a><ul> <li><a class="reference internal" href="#asyncio">asyncio</a></li> <li><a class="reference internal" href="#asyncio-migration-strategy">asyncio migration strategy</a></li> <li><a class="reference internal" href="#async-await-in-cpython-code-base">async/await in CPython code base</a></li> </ul> </li> <li><a class="reference internal" href="#grammar-updates">Grammar Updates</a></li> <li><a class="reference internal" href="#deprecation-plans">Deprecation Plans</a></li> </ul> </li> <li><a class="reference internal" href="#design-considerations">Design Considerations</a><ul> <li><a class="reference internal" href="#pep-3152">PEP 3152</a></li> <li><a class="reference internal" href="#coroutine-generators">Coroutine-generators</a></li> <li><a class="reference internal" href="#why-async-and-await-keywords">Why “async” and “await” keywords</a></li> <li><a class="reference internal" href="#why-aiter-does-not-return-an-awaitable">Why “__aiter__” does not return an awaitable</a></li> <li><a class="reference internal" href="#importance-of-async-keyword">Importance of “async” keyword</a></li> <li><a class="reference internal" href="#why-async-def">Why “async def”</a></li> <li><a class="reference internal" href="#why-not-await-for-and-await-with">Why not “await for” and “await with”</a></li> <li><a class="reference internal" href="#why-async-def-and-not-def-async">Why “async def” and not “def async”</a></li> <li><a class="reference internal" href="#why-not-a-future-import">Why not a __future__ import</a></li> <li><a class="reference internal" href="#why-magic-methods-start-with-a">Why magic methods start with “a”</a></li> <li><a class="reference internal" href="#why-not-reuse-existing-magic-names">Why not reuse existing magic names</a></li> <li><a class="reference internal" href="#why-not-reuse-existing-for-and-with-statements">Why not reuse existing “for” and “with” statements</a></li> <li><a class="reference internal" href="#comprehensions">Comprehensions</a></li> <li><a class="reference internal" href="#async-lambda-functions">Async lambda functions</a></li> </ul> </li> <li><a class="reference internal" href="#performance">Performance</a><ul> <li><a class="reference internal" href="#overall-impact">Overall Impact</a></li> <li><a class="reference internal" href="#tokenizer-modifications">Tokenizer modifications</a></li> <li><a class="reference internal" href="#async-await">async/await</a></li> </ul> </li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a><ul> <li><a class="reference internal" href="#list-of-high-level-changes-and-new-protocols">List of high-level changes and new protocols</a></li> <li><a class="reference internal" href="#working-example">Working example</a></li> </ul> </li> <li><a class="reference internal" href="#acceptance">Acceptance</a></li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#acknowledgments">Acknowledgments</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> The growth of Internet and general connectivity has triggered the proportionate need for responsive and scalable code. This proposal aims to answer that need by making writing explicitly asynchronous, concurrent Python code easier and more Pythonic. It is proposed to make coroutines a proper standalone concept in Python, and introduce new supporting syntax. The ultimate goal is to help establish a common, easily approachable, mental model of asynchronous programming in Python and make it as close to synchronous programming as possible. This PEP assumes that the asynchronous tasks are scheduled and coordinated by an Event Loop similar to that of stdlib module <code class="docutils literal notranslate">asyncio.events.AbstractEventLoop</code>. While the PEP is not tied to any specific Event Loop implementation, it is relevant only to the kind of coroutine that uses <code class="docutils literal notranslate">yield</code> as a signal to the scheduler, indicating that the coroutine will be waiting until an event (such as IO) is completed. We believe that the changes proposed here will help keep Python relevant and competitive in a quickly growing area of asynchronous programming, as many other languages have adopted, or are planning to adopt, similar features: <a class="footnote-reference brackets" href="#id32" id="id1">[2]</a>, <a class="footnote-reference brackets" href="#id35" id="id2">[5]</a>, <a class="footnote-reference brackets" href="#id36" id="id3">[6]</a>, <a class="footnote-reference brackets" href="#id37" id="id4">[7]</a>, <a class="footnote-reference brackets" href="#id38" id="id5">[8]</a>, <a class="footnote-reference brackets" href="#id40" id="id6">[10]</a>. </section> <section id="api-design-and-implementation-revisions"> <h2><a class="toc-backref" href="#api-design-and-implementation-revisions" role="doc-backlink">API Design and Implementation Revisions</a></h2> <ol class="arabic"> <li>Feedback on the initial beta release of Python 3.5 resulted in a redesign of the object model supporting this PEP to more clearly separate native coroutines from generators - rather than being a new kind of generator, native coroutines are now their own completely distinct type (implemented in <a class="footnote-reference brackets" href="#id47" id="id7">[17]</a>).This change was implemented based primarily due to problems encountered attempting to integrate support for native coroutines into the Tornado web server (reported in <a class="footnote-reference brackets" href="#id48" id="id8">[18]</a>). </li> <li>In CPython 3.5.2, the <code class="docutils literal notranslate">__aiter__</code> protocol was updated.Before 3.5.2, <code class="docutils literal notranslate">__aiter__</code> was expected to return an awaitable resolving to an asynchronous iterator. Starting with 3.5.2, <code class="docutils literal notranslate">__aiter__</code> should return asynchronous iterators directly. If the old protocol is used in 3.5.2, Python will raise a <code class="docutils literal notranslate">PendingDeprecationWarning</code>. In CPython 3.6, the old <code class="docutils literal notranslate">__aiter__</code> protocol will still be supported with a <code class="docutils literal notranslate">DeprecationWarning</code> being raised. In CPython 3.7, the old <code class="docutils literal notranslate">__aiter__</code> protocol will no longer be supported: a <code class="docutils literal notranslate">RuntimeError</code> will be raised if <code class="docutils literal notranslate">__aiter__</code> returns anything but an asynchronous iterator. See <a class="footnote-reference brackets" href="#id49" id="id9">[19]</a> and <a class="footnote-reference brackets" href="#id50" id="id10">[20]</a> for more details. </li> </ol> </section> <section id="rationale-and-goals"> <h2><a class="toc-backref" href="#rationale-and-goals" role="doc-backlink">Rationale and Goals</a></h2> Current Python supports implementing coroutines via generators (PEP 342), further enhanced by the <code class="docutils literal notranslate">yield from</code> syntax introduced in PEP 380. This approach has a number of shortcomings: <ul class="simple"> <li>It is easy to confuse coroutines with regular generators, since they share the same syntax; this is especially true for new developers.</li> <li>Whether or not a function is a coroutine is determined by a presence of <code class="docutils literal notranslate">yield</code> or <code class="docutils literal notranslate">yield from</code> statements in its body, which can lead to unobvious errors when such statements appear in or disappear from function body during refactoring.</li> <li>Support for asynchronous calls is limited to expressions where <code class="docutils literal notranslate">yield</code> is allowed syntactically, limiting the usefulness of syntactic features, such as <code class="docutils literal notranslate">with</code> and <code class="docutils literal notranslate">for</code> statements.</li> </ul> This proposal makes coroutines a native Python language feature, and clearly separates them from generators. This removes generator/coroutine ambiguity, and makes it possible to reliably define coroutines without reliance on a specific library. This also enables linters and IDEs to improve static code analysis and refactoring. Native coroutines and the associated new syntax features make it possible to define context manager and iteration protocols in asynchronous terms. As shown later in this proposal, the new <code class="docutils literal notranslate">async with</code> statement lets Python programs perform asynchronous calls when entering and exiting a runtime context, and the new <code class="docutils literal notranslate">async for</code> statement makes it possible to perform asynchronous calls in iterators. </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> This proposal introduces new syntax and semantics to enhance coroutine support in Python. This specification presumes knowledge of the implementation of coroutines in Python (<a class="pep reference internal" href="../pep-0342/" title="PEP 342 – Coroutines via Enhanced Generators">PEP 342</a> and <a class="pep reference internal" href="../pep-0380/" title="PEP 380 – Syntax for Delegating to a Subgenerator">PEP 380</a>). Motivation for the syntax changes proposed here comes from the asyncio framework (<a class="pep reference internal" href="../pep-3156/" title="PEP 3156 – Asynchronous IO Support Rebooted: the “asyncio” Module">PEP 3156</a>) and the “Cofunctions” proposal (<a class="pep reference internal" href="../pep-3152/" title="PEP 3152 – Cofunctions">PEP 3152</a>, now rejected in favor of this specification). From this point in this document we use the word native coroutine to refer to functions declared using the new syntax. generator-based coroutine is used where necessary to refer to coroutines that are based on generator syntax. coroutine is used in contexts where both definitions are applicable. <section id="new-coroutine-declaration-syntax"> <h3><a class="toc-backref" href="#new-coroutine-declaration-syntax" role="doc-backlink">New Coroutine Declaration Syntax</a></h3> The following new syntax is used to declare a native coroutine: <div class="highlight-default notranslate"><div class="highlight"><pre>async def read_data(db): pass </pre></div> </div> Key properties of coroutines: <ul class="simple"> <li><code class="docutils literal notranslate">async def</code> functions are always coroutines, even if they do not contain <code class="docutils literal notranslate">await</code> expressions.</li> <li>It is a <code class="docutils literal notranslate">SyntaxError</code> to have <code class="docutils literal notranslate">yield</code> or <code class="docutils literal notranslate">yield from</code> expressions in an <code class="docutils literal notranslate">async</code> function.</li> <li>Internally, two new code object flags were introduced:<ul> <li><code class="docutils literal notranslate">CO_COROUTINE</code> is used to mark native coroutines (defined with new syntax).</li> <li><code class="docutils literal notranslate">CO_ITERABLE_COROUTINE</code> is used to make generator-based coroutines compatible with native coroutines (set by <a class="reference internal" href="#types-coroutine">types.coroutine()</a> function).</li> </ul> </li> <li>Regular generators, when called, return a generator object; similarly, coroutines return a coroutine object.</li> <li><code class="docutils literal notranslate">StopIteration</code> exceptions are not propagated out of coroutines, and are replaced with a <code class="docutils literal notranslate">RuntimeError</code>. For regular generators such behavior requires a future import (see <a class="pep reference internal" href="../pep-0479/" title="PEP 479 – Change StopIteration handling inside generators">PEP 479</a>).</li> <li>When a native coroutine is garbage collected, a <code class="docutils literal notranslate">RuntimeWarning</code> is raised if it was never awaited on (see also <a class="reference internal" href="#debugging-features">Debugging Features</a>).</li> <li>See also <a class="reference internal" href="#coroutine-objects">Coroutine objects</a> section.</li> </ul> </section> <section id="types-coroutine"> <h3><a class="toc-backref" href="#types-coroutine" role="doc-backlink">types.coroutine()</a></h3> A new function <code class="docutils literal notranslate">coroutine(fn)</code> is added to the <code class="docutils literal notranslate">types</code> module. It allows interoperability between existing generator-based coroutines in asyncio and native coroutines introduced by this PEP: <div class="highlight-default notranslate"><div class="highlight"><pre>@types.coroutine def process_data(db): data = yield from read_data(db) ... </pre></div> </div> The function applies <code class="docutils literal notranslate">CO_ITERABLE_COROUTINE</code> flag to generator-function’s code object, making it return a coroutine object. If <code class="docutils literal notranslate">fn</code> is not a generator function, it is wrapped. If it returns a generator, it will be wrapped in an awaitable proxy object (see below the definition of awaitable objects). Note, that the <code class="docutils literal notranslate">CO_COROUTINE</code> flag is not applied by <code class="docutils literal notranslate">types.coroutine()</code> to make it possible to separate native coroutines defined with new syntax, from generator-based coroutines. </section> <section id="await-expression"> <h3><a class="toc-backref" href="#await-expression" role="doc-backlink">Await Expression</a></h3> The following new <code class="docutils literal notranslate">await</code> expression is used to obtain a result of coroutine execution: <div class="highlight-default notranslate"><div class="highlight"><pre>async def read_data(db): data = await db.fetch('SELECT ...') ... </pre></div> </div> <code class="docutils literal notranslate">await</code>, similarly to <code class="docutils literal notranslate">yield from</code>, suspends execution of <code class="docutils literal notranslate">read_data</code> coroutine until <code class="docutils literal notranslate">db.fetch</code> awaitable completes and returns the result data. It uses the <code class="docutils literal notranslate">yield from</code> implementation with an extra step of validating its argument. <code class="docutils literal notranslate">await</code> only accepts an awaitable, which can be one of: <ul> <li>A native coroutine object returned from a native coroutine function.</li> <li>A generator-based coroutine object returned from a function decorated with <code class="docutils literal notranslate">types.coroutine()</code>.</li> <li>An object with an <code class="docutils literal notranslate">__await__</code> method returning an iterator.Any <code class="docutils literal notranslate">yield from</code> chain of calls ends with a <code class="docutils literal notranslate">yield</code>. This is a fundamental mechanism of how Futures are implemented. Since, internally, coroutines are a special kind of generators, every <code class="docutils literal notranslate">await</code> is suspended by a <code class="docutils literal notranslate">yield</code> somewhere down the chain of <code class="docutils literal notranslate">await</code> calls (please refer to <a class="pep reference internal" href="../pep-3156/" title="PEP 3156 – Asynchronous IO Support Rebooted: the “asyncio” Module">PEP 3156</a> for a detailed explanation). To enable this behavior for coroutines, a new magic method called <code class="docutils literal notranslate">__await__</code> is added. In asyncio, for instance, to enable Future objects in <code class="docutils literal notranslate">await</code> statements, the only change is to add <code class="docutils literal notranslate">__await__ = __iter__</code> line to <code class="docutils literal notranslate">asyncio.Future</code> class. Objects with <code class="docutils literal notranslate">__await__</code> method are called Future-like objects in the rest of this PEP. It is a <code class="docutils literal notranslate">TypeError</code> if <code class="docutils literal notranslate">__await__</code> returns anything but an iterator. </li> <li>Objects defined with CPython C API with a <code class="docutils literal notranslate">tp_as_async.am_await</code> function, returning an iterator (similar to <code class="docutils literal notranslate">__await__</code> method).</li> </ul> It is a <code class="docutils literal notranslate">SyntaxError</code> to use <code class="docutils literal notranslate">await</code> outside of an <code class="docutils literal notranslate">async def</code> function (like it is a <code class="docutils literal notranslate">SyntaxError</code> to use <code class="docutils literal notranslate">yield</code> outside of <code class="docutils literal notranslate">def</code> function). It is a <code class="docutils literal notranslate">TypeError</code> to pass anything other than an awaitable object to an <code class="docutils literal notranslate">await</code> expression. <section id="updated-operator-precedence-table"> <h4><a class="toc-backref" href="#updated-operator-precedence-table" role="doc-backlink">Updated operator precedence table</a></h4> <code class="docutils literal notranslate">await</code> keyword is defined as follows: <div class="highlight-default notranslate"><div class="highlight"><pre>power ::= await ["**" u_expr] await ::= ["await"] primary </pre></div> </div> where “primary” represents the most tightly bound operations of the language. Its syntax is: <div class="highlight-default notranslate"><div class="highlight"><pre>primary ::= atom | attributeref | subscription | slicing | call </pre></div> </div> See Python Documentation <a class="footnote-reference brackets" href="#id42" id="id11">[12]</a> and <a class="reference internal" href="#grammar-updates">Grammar Updates</a> section of this proposal for details. The key <code class="docutils literal notranslate">await</code> difference from <code class="docutils literal notranslate">yield</code> and <code class="docutils literal notranslate">yield from</code> operators is that await expressions do not require parentheses around them most of the times. Also, <code class="docutils literal notranslate">yield from</code> allows any expression as its argument, including expressions like <code class="docutils literal notranslate">yield from a() + b()</code>, that would be parsed as <code class="docutils literal notranslate">yield from (a() + b())</code>, which is almost always a bug. In general, the result of any arithmetic operation is not an awaitable object. To avoid this kind of mistakes, it was decided to make <code class="docutils literal notranslate">await</code> precedence lower than <code class="docutils literal notranslate">[]</code>, <code class="docutils literal notranslate">()</code>, and <code class="docutils literal notranslate">.</code>, but higher than <code class="docutils literal notranslate">**</code> operators. <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Operator</th> <th class="head">Description</th> </tr> </thead> <tbody> <tr class="row-even"><td><code class="docutils literal notranslate">yield</code> <code class="docutils literal notranslate">x</code>, <code class="docutils literal notranslate">yield from</code> <code class="docutils literal notranslate">x</code></td> <td>Yield expression</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">lambda</code></td> <td>Lambda expression</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">if</code> – <code class="docutils literal notranslate">else</code></td> <td>Conditional expression</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">or</code></td> <td>Boolean OR</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">and</code></td> <td>Boolean AND</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">not</code> <code class="docutils literal notranslate">x</code></td> <td>Boolean NOT</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">in</code>, <code class="docutils literal notranslate">not in</code>, <code class="docutils literal notranslate">is</code>, <code class="docutils literal notranslate">is not</code>, <code class="docutils literal notranslate"><</code>, <code class="docutils literal notranslate"><=</code>, <code class="docutils literal notranslate">></code>, <code class="docutils literal notranslate">>=</code>, <code class="docutils literal notranslate">!=</code>, <code class="docutils literal notranslate">==</code></td> <td>Comparisons, including membership tests and identity tests</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">|</code></td> <td>Bitwise OR</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">^</code></td> <td>Bitwise XOR</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">&</code></td> <td>Bitwise AND</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate"><<</code>, <code class="docutils literal notranslate">>></code></td> <td>Shifts</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">+</code>, <code class="docutils literal notranslate">-</code></td> <td>Addition and subtraction</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">*</code>, <code class="docutils literal notranslate">@</code>, <code class="docutils literal notranslate">/</code>, <code class="docutils literal notranslate">//</code>, <code class="docutils literal notranslate">%</code></td> <td>Multiplication, matrix multiplication, division, remainder</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">+x</code>, <code class="docutils literal notranslate">-x</code>, <code class="docutils literal notranslate">~x</code></td> <td>Positive, negative, bitwise NOT</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">**</code></td> <td>Exponentiation</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">await</code> <code class="docutils literal notranslate">x</code></td> <td>Await expression</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">x[index]</code>, <code class="docutils literal notranslate">x[index:index]</code>, <code class="docutils literal notranslate">x(arguments...)</code>, <code class="docutils literal notranslate">x.attribute</code></td> <td>Subscription, slicing, call, attribute reference</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">(expressions...)</code>, <code class="docutils literal notranslate">[expressions...]</code>, <code class="docutils literal notranslate">{key: value...}</code>, <code class="docutils literal notranslate">{expressions...}</code></td> <td>Binding or tuple display, list display, dictionary display, set display</td> </tr> </tbody> </table> </section> <section id="examples-of-await-expressions"> <h4><a class="toc-backref" href="#examples-of-await-expressions" role="doc-backlink">Examples of “await” expressions</a></h4> Valid syntax examples: <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Expression</th> <th class="head">Will be parsed as</th> </tr> </thead> <tbody> <tr class="row-even"><td><code class="docutils literal notranslate">if await fut: pass</code></td> <td><code class="docutils literal notranslate">if (await fut): pass</code></td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">if await fut + 1: pass</code></td> <td><code class="docutils literal notranslate">if (await fut) + 1: pass</code></td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">pair = await fut, 'spam'</code></td> <td><code class="docutils literal notranslate">pair = (await fut), 'spam'</code></td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">with await fut, open(): pass</code></td> <td><code class="docutils literal notranslate">with (await fut), open(): pass</code></td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">await foo()['spam'].baz()()</code></td> <td><code class="docutils literal notranslate">await ( foo()['spam'].baz()() )</code></td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">return await coro()</code></td> <td><code class="docutils literal notranslate">return ( await coro() )</code></td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">res = await coro() ** 2</code></td> <td><code class="docutils literal notranslate">res = (await coro()) ** 2</code></td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">func(a1=await coro(), a2=0)</code></td> <td><code class="docutils literal notranslate">func(a1=(await coro()), a2=0)</code></td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">await foo() + await bar()</code></td> <td><code class="docutils literal notranslate">(await foo()) + (await bar())</code></td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">-await foo()</code></td> <td><code class="docutils literal notranslate">-(await foo())</code></td> </tr> </tbody> </table> Invalid syntax examples: <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Expression</th> <th class="head">Should be written as</th> </tr> </thead> <tbody> <tr class="row-even"><td><code class="docutils literal notranslate">await await coro()</code></td> <td><code class="docutils literal notranslate">await (await coro())</code></td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">await -coro()</code></td> <td><code class="docutils literal notranslate">await (-coro())</code></td> </tr> </tbody> </table> </section> </section> <section id="asynchronous-context-managers-and-async-with"> <h3><a class="toc-backref" href="#asynchronous-context-managers-and-async-with" role="doc-backlink">Asynchronous Context Managers and “async with”</a></h3> An asynchronous context manager is a context manager that is able to suspend execution in its enter and exit methods. To make this possible, a new protocol for asynchronous context managers is proposed. Two new magic methods are added: <code class="docutils literal notranslate">__aenter__</code> and <code class="docutils literal notranslate">__aexit__</code>. Both must return an awaitable. An example of an asynchronous context manager: <div class="highlight-default notranslate"><div class="highlight"><pre>class AsyncContextManager: async def __aenter__(self): await log('entering context') async def __aexit__(self, exc_type, exc, tb): await log('exiting context') </pre></div> </div> <section id="new-syntax"> <h4><a class="toc-backref" href="#new-syntax" role="doc-backlink">New Syntax</a></h4> A new statement for asynchronous context managers is proposed: <div class="highlight-default notranslate"><div class="highlight"><pre>async with EXPR as VAR: BLOCK </pre></div> </div> which is semantically equivalent to: <div class="highlight-default notranslate"><div class="highlight"><pre>mgr = (EXPR) aexit = type(mgr).__aexit__ aenter = type(mgr).__aenter__ VAR = await aenter(mgr) try: BLOCK except: if not await aexit(mgr, *sys.exc_info()): raise else: await aexit(mgr, None, None, None) </pre></div> </div> As with regular <code class="docutils literal notranslate">with</code> statements, it is possible to specify multiple context managers in a single <code class="docutils literal notranslate">async with</code> statement. It is an error to pass a regular context manager without <code class="docutils literal notranslate">__aenter__</code> and <code class="docutils literal notranslate">__aexit__</code> methods to <code class="docutils literal notranslate">async with</code>. It is a <code class="docutils literal notranslate">SyntaxError</code> to use <code class="docutils literal notranslate">async with</code> outside of an <code class="docutils literal notranslate">async def</code> function. </section> <section id="example"> <h4><a class="toc-backref" href="#example" role="doc-backlink">Example</a></h4> With asynchronous context managers it is easy to implement proper database transaction managers for coroutines: <div class="highlight-default notranslate"><div class="highlight"><pre>async def commit(session, data): ... async with session.transaction(): ... await session.update(data) ... </pre></div> </div> Code that needs locking also looks lighter: <div class="highlight-default notranslate"><div class="highlight"><pre>async with lock: ... </pre></div> </div> instead of: <div class="highlight-default notranslate"><div class="highlight"><pre>with (yield from lock): ... </pre></div> </div> </section> </section> <section id="asynchronous-iterators-and-async-for"> <h3><a class="toc-backref" href="#asynchronous-iterators-and-async-for" role="doc-backlink">Asynchronous Iterators and “async for”</a></h3> An asynchronous iterable is able to call asynchronous code in its iter implementation, and asynchronous iterator can call asynchronous code in its next method. To support asynchronous iteration: <ol class="arabic simple"> <li>An object must implement an <code class="docutils literal notranslate">__aiter__</code> method (or, if defined with CPython C API, <code class="docutils literal notranslate">tp_as_async.am_aiter</code> slot) returning an asynchronous iterator object.</li> <li>An asynchronous iterator object must implement an <code class="docutils literal notranslate">__anext__</code> method (or, if defined with CPython C API, <code class="docutils literal notranslate">tp_as_async.am_anext</code> slot) returning an awaitable.</li> <li>To stop iteration <code class="docutils literal notranslate">__anext__</code> must raise a <code class="docutils literal notranslate">StopAsyncIteration</code> exception.</li> </ol> An example of asynchronous iterable: <div class="highlight-default notranslate"><div class="highlight"><pre>class AsyncIterable: def __aiter__(self): return self async def __anext__(self): data = await self.fetch_data() if data: return data else: raise StopAsyncIteration async def fetch_data(self): ... </pre></div> </div> <section id="id12"> <h4><a class="toc-backref" href="#id12" role="doc-backlink">New Syntax</a></h4> A new statement for iterating through asynchronous iterators is proposed: <div class="highlight-default notranslate"><div class="highlight"><pre>async for TARGET in ITER: BLOCK else: BLOCK2 </pre></div> </div> which is semantically equivalent to: <div class="highlight-default notranslate"><div class="highlight"><pre>iter = (ITER) iter = type(iter).__aiter__(iter) running = True while running: try: TARGET = await type(iter).__anext__(iter) except StopAsyncIteration: running = False else: BLOCK else: BLOCK2 </pre></div> </div> It is a <code class="docutils literal notranslate">TypeError</code> to pass a regular iterable without <code class="docutils literal notranslate">__aiter__</code> method to <code class="docutils literal notranslate">async for</code>. It is a <code class="docutils literal notranslate">SyntaxError</code> to use <code class="docutils literal notranslate">async for</code> outside of an <code class="docutils literal notranslate">async def</code> function. As for with regular <code class="docutils literal notranslate">for</code> statement, <code class="docutils literal notranslate">async for</code> has an optional <code class="docutils literal notranslate">else</code> clause. </section> <section id="example-1"> <h4><a class="toc-backref" href="#example-1" role="doc-backlink">Example 1</a></h4> With asynchronous iteration protocol it is possible to asynchronously buffer data during iteration: <div class="highlight-default notranslate"><div class="highlight"><pre>async for data in cursor: ... </pre></div> </div> Where <code class="docutils literal notranslate">cursor</code> is an asynchronous iterator that prefetches <code class="docutils literal notranslate">N</code> rows of data from a database after every <code class="docutils literal notranslate">N</code> iterations. The following code illustrates new asynchronous iteration protocol: <div class="highlight-default notranslate"><div class="highlight"><pre>class Cursor: def __init__(self): self.buffer = collections.deque() async def _prefetch(self): ... def __aiter__(self): return self async def __anext__(self): if not self.buffer: self.buffer = await self._prefetch() if not self.buffer: raise StopAsyncIteration return self.buffer.popleft() </pre></div> </div> then the <code class="docutils literal notranslate">Cursor</code> class can be used as follows: <div class="highlight-default notranslate"><div class="highlight"><pre>async for row in Cursor(): print(row) </pre></div> </div> which would be equivalent to the following code: <div class="highlight-default notranslate"><div class="highlight"><pre>i = Cursor().__aiter__() while True: try: row = await i.__anext__() except StopAsyncIteration: break else: print(row) </pre></div> </div> </section> <section id="example-2"> <h4><a class="toc-backref" href="#example-2" role="doc-backlink">Example 2</a></h4> The following is a utility class that transforms a regular iterable to an asynchronous one. While this is not a very useful thing to do, the code illustrates the relationship between regular and asynchronous iterators. <div class="highlight-python notranslate"><div class="highlight"><pre>class AsyncIteratorWrapper: def __init__(self, obj): self._it = iter(obj) def __aiter__(self): return self async def __anext__(self): try: value = next(self._it) except StopIteration: raise StopAsyncIteration return value async for letter in AsyncIteratorWrapper("abc"): print(letter) </pre></div> </div> </section> <section id="why-stopasynciteration"> <h4><a class="toc-backref" href="#why-stopasynciteration" role="doc-backlink">Why StopAsyncIteration?</a></h4> Coroutines are still based on generators internally. So, before PEP 479, there was no fundamental difference between <div class="highlight-python notranslate"><div class="highlight"><pre>def g1(): yield from fut return 'spam' </pre></div> </div> and <div class="highlight-python notranslate"><div class="highlight"><pre>def g2(): yield from fut raise StopIteration('spam') </pre></div> </div> And since <a class="pep reference internal" href="../pep-0479/" title="PEP 479 – Change StopIteration handling inside generators">PEP 479</a> is accepted and enabled by default for coroutines, the following example will have its <code class="docutils literal notranslate">StopIteration</code> wrapped into a <code class="docutils literal notranslate">RuntimeError</code> <div class="highlight-python notranslate"><div class="highlight"><pre>async def a1(): await fut raise StopIteration('spam') </pre></div> </div> The only way to tell the outside code that the iteration has ended is to raise something other than <code class="docutils literal notranslate">StopIteration</code>. Therefore, a new built-in exception class <code class="docutils literal notranslate">StopAsyncIteration</code> was added. Moreover, with semantics from <a class="pep reference internal" href="../pep-0479/" title="PEP 479 – Change StopIteration handling inside generators">PEP 479</a>, all <code class="docutils literal notranslate">StopIteration</code> exceptions raised in coroutines are wrapped in <code class="docutils literal notranslate">RuntimeError</code>. </section> </section> <section id="coroutine-objects"> <h3><a class="toc-backref" href="#coroutine-objects" role="doc-backlink">Coroutine objects</a></h3> <section id="differences-from-generators"> <h4><a class="toc-backref" href="#differences-from-generators" role="doc-backlink">Differences from generators</a></h4> This section applies only to native coroutines with <code class="docutils literal notranslate">CO_COROUTINE</code> flag, i.e. defined with the new <code class="docutils literal notranslate">async def</code> syntax. The behavior of existing *generator-based coroutines* in asyncio remains unchanged. Great effort has been made to make sure that coroutines and generators are treated as distinct concepts: <ol class="arabic"> <li>Native coroutine objects do not implement <code class="docutils literal notranslate">__iter__</code> and <code class="docutils literal notranslate">__next__</code> methods. Therefore, they cannot be iterated over or passed to <code class="docutils literal notranslate">iter()</code>, <code class="docutils literal notranslate">list()</code>, <code class="docutils literal notranslate">tuple()</code> and other built-ins. They also cannot be used in a <code class="docutils literal notranslate">for..in</code> loop.An attempt to use <code class="docutils literal notranslate">__iter__</code> or <code class="docutils literal notranslate">__next__</code> on a native coroutine object will result in a <code class="docutils literal notranslate">TypeError</code>. </li> <li>Plain generators cannot <code class="docutils literal notranslate">yield from</code> native coroutines: doing so will result in a <code class="docutils literal notranslate">TypeError</code>.</li> <li>generator-based coroutines (for asyncio code must be decorated with <code class="docutils literal notranslate">@asyncio.coroutine</code> <a class="footnote-reference brackets" href="#id31" id="id13">[1]</a>) can <code class="docutils literal notranslate">yield from</code> native coroutine objects.</li> <li><code class="docutils literal notranslate">inspect.isgenerator()</code> and <code class="docutils literal notranslate">inspect.isgeneratorfunction()</code> return <code class="docutils literal notranslate">False</code> for native coroutine objects and native coroutine functions.</li> </ol> </section> <section id="coroutine-object-methods"> <h4><a class="toc-backref" href="#coroutine-object-methods" role="doc-backlink">Coroutine object methods</a></h4> Coroutines are based on generators internally, thus they share the implementation. Similarly to generator objects, coroutines have <code class="docutils literal notranslate">throw()</code>, <code class="docutils literal notranslate">send()</code> and <code class="docutils literal notranslate">close()</code> methods. <code class="docutils literal notranslate">StopIteration</code> and <code class="docutils literal notranslate">GeneratorExit</code> play the same role for coroutines (although <a class="pep reference internal" href="../pep-0479/" title="PEP 479 – Change StopIteration handling inside generators">PEP 479</a> is enabled by default for coroutines). See <a class="pep reference internal" href="../pep-0342/" title="PEP 342 – Coroutines via Enhanced Generators">PEP 342</a>, <a class="pep reference internal" href="../pep-0380/" title="PEP 380 – Syntax for Delegating to a Subgenerator">PEP 380</a>, and Python Documentation <a class="footnote-reference brackets" href="#id41" id="id14">[11]</a> for details. <code class="docutils literal notranslate">throw()</code>, <code class="docutils literal notranslate">send()</code> methods for coroutines are used to push values and raise errors into Future-like objects. </section> </section> <section id="debugging-features"> <h3><a class="toc-backref" href="#debugging-features" role="doc-backlink">Debugging Features</a></h3> A common beginner mistake is forgetting to use <code class="docutils literal notranslate">yield from</code> on coroutines: <div class="highlight-default notranslate"><div class="highlight"><pre>@asyncio.coroutine def useful(): asyncio.sleep(1) # this will do nothing without 'yield from' </pre></div> </div> For debugging this kind of mistakes there is a special debug mode in asyncio, in which <code class="docutils literal notranslate">@coroutine</code> decorator wraps all functions with a special object with a destructor logging a warning. Whenever a wrapped generator gets garbage collected, a detailed logging message is generated with information about where exactly the decorator function was defined, stack trace of where it was collected, etc. Wrapper object also provides a convenient <code class="docutils literal notranslate">__repr__</code> function with detailed information about the generator. The only problem is how to enable these debug capabilities. Since debug facilities should be a no-op in production mode, <code class="docutils literal notranslate">@coroutine</code> decorator makes the decision of whether to wrap or not to wrap based on an OS environment variable <code class="docutils literal notranslate">PYTHONASYNCIODEBUG</code>. This way it is possible to run asyncio programs with asyncio’s own functions instrumented. <code class="docutils literal notranslate">EventLoop.set_debug</code>, a different debug facility, has no impact on <code class="docutils literal notranslate">@coroutine</code> decorator’s behavior. With this proposal, coroutines is a native, distinct from generators, concept. In addition to a <code class="docutils literal notranslate">RuntimeWarning</code> being raised on coroutines that were never awaited, it is proposed to add two new functions to the <code class="docutils literal notranslate">sys</code> module: <code class="docutils literal notranslate">set_coroutine_wrapper</code> and <code class="docutils literal notranslate">get_coroutine_wrapper</code>. This is to enable advanced debugging facilities in asyncio and other frameworks (such as displaying where exactly coroutine was created, and a more detailed stack trace of where it was garbage collected). </section> <section id="new-standard-library-functions"> <h3><a class="toc-backref" href="#new-standard-library-functions" role="doc-backlink">New Standard Library Functions</a></h3> <ul class="simple"> <li><code class="docutils literal notranslate">types.coroutine(gen)</code>. See <a class="reference internal" href="#types-coroutine">types.coroutine()</a> section for details.</li> <li><code class="docutils literal notranslate">inspect.iscoroutine(obj)</code> returns <code class="docutils literal notranslate">True</code> if <code class="docutils literal notranslate">obj</code> is a native coroutine object.</li> <li><code class="docutils literal notranslate">inspect.iscoroutinefunction(obj)</code> returns <code class="docutils literal notranslate">True</code> if <code class="docutils literal notranslate">obj</code> is a native coroutine function.</li> <li><code class="docutils literal notranslate">inspect.isawaitable(obj)</code> returns <code class="docutils literal notranslate">True</code> if <code class="docutils literal notranslate">obj</code> is an awaitable.</li> <li><code class="docutils literal notranslate">inspect.getcoroutinestate(coro)</code> returns the current state of a native coroutine object (mirrors <code class="docutils literal notranslate">inspect.getfgeneratorstate(gen)</code>).</li> <li><code class="docutils literal notranslate">inspect.getcoroutinelocals(coro)</code> returns the mapping of a native coroutine object’s local variables to their values (mirrors <code class="docutils literal notranslate">inspect.getgeneratorlocals(gen)</code>).</li> <li><code class="docutils literal notranslate">sys.set_coroutine_wrapper(wrapper)</code> allows to intercept creation of native coroutine objects. <code class="docutils literal notranslate">wrapper</code> must be either a callable that accepts one argument (a coroutine object), or <code class="docutils literal notranslate">None</code>. <code class="docutils literal notranslate">None</code> resets the wrapper. If called twice, the new wrapper replaces the previous one. The function is thread-specific. See <a class="reference internal" href="#debugging-features">Debugging Features</a> for more details.</li> <li><code class="docutils literal notranslate">sys.get_coroutine_wrapper()</code> returns the current wrapper object. Returns <code class="docutils literal notranslate">None</code> if no wrapper was set. The function is thread-specific. See <a class="reference internal" href="#debugging-features">Debugging Features</a> for more details.</li> </ul> </section> <section id="new-abstract-base-classes"> <h3><a class="toc-backref" href="#new-abstract-base-classes" role="doc-backlink">New Abstract Base Classes</a></h3> In order to allow better integration with existing frameworks (such as Tornado, see <a class="footnote-reference brackets" href="#id43" id="id15">[13]</a>) and compilers (such as Cython, see <a class="footnote-reference brackets" href="#id46" id="id16">[16]</a>), two new Abstract Base Classes (ABC) are added: <ul> <li><code class="docutils literal notranslate">collections.abc.Awaitable</code> ABC for Future-like classes, that implement <code class="docutils literal notranslate">__await__</code> method.</li> <li><code class="docutils literal notranslate">collections.abc.Coroutine</code> ABC for coroutine objects, that implement <code class="docutils literal notranslate">send(value)</code>, <code class="docutils literal notranslate">throw(type, exc, tb)</code>, <code class="docutils literal notranslate">close()</code> and <code class="docutils literal notranslate">__await__()</code> methods.Note that generator-based coroutines with <code class="docutils literal notranslate">CO_ITERABLE_COROUTINE</code> flag do not implement <code class="docutils literal notranslate">__await__</code> method, and therefore are not instances of <code class="docutils literal notranslate">collections.abc.Coroutine</code> and <code class="docutils literal notranslate">collections.abc.Awaitable</code> ABCs: <div class="highlight-default notranslate"><div class="highlight"><pre>@types.coroutine def gencoro(): yield assert not isinstance(gencoro(), collections.abc.Coroutine) # however: assert inspect.isawaitable(gencoro()) </pre></div> </div> </li> </ul> To allow easy testing if objects support asynchronous iteration, two more ABCs are added: <ul class="simple"> <li><code class="docutils literal notranslate">collections.abc.AsyncIterable</code> – tests for <code class="docutils literal notranslate">__aiter__</code> method.</li> <li><code class="docutils literal notranslate">collections.abc.AsyncIterator</code> – tests for <code class="docutils literal notranslate">__aiter__</code> and <code class="docutils literal notranslate">__anext__</code> methods.</li> </ul> </section> </section> <section id="glossary"> <h2><a class="toc-backref" href="#glossary" role="doc-backlink">Glossary</a></h2> <dl class="simple"> <dt>Native coroutine function</dt><dd>A coroutine function is declared with <code class="docutils literal notranslate">async def</code>. It uses <code class="docutils literal notranslate">await</code> and <code class="docutils literal notranslate">return value</code>; see <a class="reference internal" href="#new-coroutine-declaration-syntax">New Coroutine Declaration Syntax</a> for details.</dd> <dt>Native coroutine</dt><dd>Returned from a native coroutine function. See <a class="reference internal" href="#await-expression">Await Expression</a> for details.</dd> <dt>Generator-based coroutine function</dt><dd>Coroutines based on generator syntax. Most common example are functions decorated with <code class="docutils literal notranslate">@asyncio.coroutine</code>.</dd> <dt>Generator-based coroutine</dt><dd>Returned from a generator-based coroutine function.</dd> <dt>Coroutine</dt><dd>Either native coroutine or generator-based coroutine.</dd> <dt>Coroutine object</dt><dd>Either native coroutine object or generator-based coroutine object.</dd> <dt>Future-like object</dt><dd>An object with an <code class="docutils literal notranslate">__await__</code> method, or a C object with <code class="docutils literal notranslate">tp_as_async->am_await</code> function, returning an iterator. Can be consumed by an <code class="docutils literal notranslate">await</code> expression in a coroutine. A coroutine waiting for a Future-like object is suspended until the Future-like object’s <code class="docutils literal notranslate">__await__</code> completes, and returns the result. See <a class="reference internal" href="#await-expression">Await Expression</a> for details.</dd> <dt>Awaitable</dt><dd>A Future-like object or a coroutine object. See <a class="reference internal" href="#await-expression">Await Expression</a> for details.</dd> <dt>Asynchronous context manager</dt><dd>An asynchronous context manager has <code class="docutils literal notranslate">__aenter__</code> and <code class="docutils literal notranslate">__aexit__</code> methods and can be used with <code class="docutils literal notranslate">async with</code>. See <a class="reference internal" href="#asynchronous-context-managers-and-async-with">Asynchronous Context Managers and “async with”</a> for details.</dd> <dt>Asynchronous iterable</dt><dd>An object with an <code class="docutils literal notranslate">__aiter__</code> method, which must return an asynchronous iterator object. Can be used with <code class="docutils literal notranslate">async for</code>. See <a class="reference internal" href="#asynchronous-iterators-and-async-for">Asynchronous Iterators and “async for”</a> for details.</dd> <dt>Asynchronous iterator</dt><dd>An asynchronous iterator has an <code class="docutils literal notranslate">__anext__</code> method. See <a class="reference internal" href="#asynchronous-iterators-and-async-for">Asynchronous Iterators and “async for”</a> for details.</dd> </dl> </section> <section id="transition-plan"> <h2><a class="toc-backref" href="#transition-plan" role="doc-backlink">Transition Plan</a></h2> To avoid backwards compatibility issues with <code class="docutils literal notranslate">async</code> and <code class="docutils literal notranslate">await</code> keywords, it was decided to modify <code class="docutils literal notranslate">tokenizer.c</code> in such a way, that it: <ul class="simple"> <li>recognizes <code class="docutils literal notranslate">async def</code> <code class="docutils literal notranslate">NAME</code> tokens combination;</li> <li>while tokenizing <code class="docutils literal notranslate">async def</code> block, it replaces <code class="docutils literal notranslate">'async'</code> <code class="docutils literal notranslate">NAME</code> token with <code class="docutils literal notranslate">ASYNC</code>, and <code class="docutils literal notranslate">'await'</code> <code class="docutils literal notranslate">NAME</code> token with <code class="docutils literal notranslate">AWAIT</code>;</li> <li>while tokenizing <code class="docutils literal notranslate">def</code> block, it yields <code class="docutils literal notranslate">'async'</code> and <code class="docutils literal notranslate">'await'</code> <code class="docutils literal notranslate">NAME</code> tokens as is.</li> </ul> This approach allows for seamless combination of new syntax features (all of them available only in <code class="docutils literal notranslate">async</code> functions) with any existing code. An example of having “async def” and “async” attribute in one piece of code: <div class="highlight-default notranslate"><div class="highlight"><pre>class Spam: async = 42 async def ham(): print(getattr(Spam, 'async')) # The coroutine can be executed and will print '42' </pre></div> </div> <section id="backwards-compatibility"> <h3><a class="toc-backref" href="#backwards-compatibility" role="doc-backlink">Backwards Compatibility</a></h3> This proposal preserves 100% backwards compatibility. <section id="asyncio"> <h4><a class="toc-backref" href="#asyncio" role="doc-backlink">asyncio</a></h4> <code class="docutils literal notranslate">asyncio</code> module was adapted and tested to work with coroutines and new statements. Backwards compatibility is 100% preserved, i.e. all existing code will work as-is. The required changes are mainly: <ol class="arabic simple"> <li>Modify <code class="docutils literal notranslate">@asyncio.coroutine</code> decorator to use new <code class="docutils literal notranslate">types.coroutine()</code> function.</li> <li>Add <code class="docutils literal notranslate">__await__ = __iter__</code> line to <code class="docutils literal notranslate">asyncio.Future</code> class.</li> <li>Add <code class="docutils literal notranslate">ensure_future()</code> as an alias for <code class="docutils literal notranslate">async()</code> function. Deprecate <code class="docutils literal notranslate">async()</code> function.</li> </ol> </section> <section id="asyncio-migration-strategy"> <h4><a class="toc-backref" href="#asyncio-migration-strategy" role="doc-backlink">asyncio migration strategy</a></h4> Because plain generators cannot <code class="docutils literal notranslate">yield from</code> native coroutine objects (see <a class="reference internal" href="#differences-from-generators">Differences from generators</a> section for more details), it is advised to make sure that all generator-based coroutines are decorated with <code class="docutils literal notranslate">@asyncio.coroutine</code> before starting to use the new syntax. </section> <section id="async-await-in-cpython-code-base"> <h4><a class="toc-backref" href="#async-await-in-cpython-code-base" role="doc-backlink">async/await in CPython code base</a></h4> There is no use of <code class="docutils literal notranslate">await</code> names in CPython. <code class="docutils literal notranslate">async</code> is mostly used by asyncio. We are addressing this by renaming <code class="docutils literal notranslate">async()</code> function to <code class="docutils literal notranslate">ensure_future()</code> (see <a class="reference internal" href="#asyncio">asyncio</a> section for details). Another use of <code class="docutils literal notranslate">async</code> keyword is in <code class="docutils literal notranslate">Lib/xml/dom/xmlbuilder.py</code>, to define an <code class="docutils literal notranslate">async = False</code> attribute for <code class="docutils literal notranslate">DocumentLS</code> class. There is no documentation or tests for it, it is not used anywhere else in CPython. It is replaced with a getter, that raises a <code class="docutils literal notranslate">DeprecationWarning</code>, advising to use <code class="docutils literal notranslate">async_</code> attribute instead. ‘async’ attribute is not documented and is not used in CPython code base. </section> </section> <section id="grammar-updates"> <h3><a class="toc-backref" href="#grammar-updates" role="doc-backlink">Grammar Updates</a></h3> Grammar changes are fairly minimal: <div class="highlight-default notranslate"><div class="highlight"><pre>decorated: decorators (classdef | funcdef | async_funcdef) async_funcdef: ASYNC funcdef compound_stmt: (if_stmt | while_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef | decorated | async_stmt) async_stmt: ASYNC (funcdef | with_stmt | for_stmt) power: atom_expr ['**' factor] atom_expr: [AWAIT] atom trailer* </pre></div> </div> </section> <section id="deprecation-plans"> <h3><a class="toc-backref" href="#deprecation-plans" role="doc-backlink">Deprecation Plans</a></h3> <code class="docutils literal notranslate">async</code> and <code class="docutils literal notranslate">await</code> names will be softly deprecated in CPython 3.5 and 3.6. In 3.7 we will transform them to proper keywords. Making <code class="docutils literal notranslate">async</code> and <code class="docutils literal notranslate">await</code> proper keywords before 3.7 might make it harder for people to port their code to Python 3. </section> </section> <section id="design-considerations"> <h2><a class="toc-backref" href="#design-considerations" role="doc-backlink">Design Considerations</a></h2> <section id="pep-3152"> <h3><a class="toc-backref" href="#pep-3152" role="doc-backlink">PEP 3152</a></h3> <a class="pep reference internal" href="../pep-3152/" title="PEP 3152 – Cofunctions">PEP 3152</a> by Gregory Ewing proposes a different mechanism for coroutines (called “cofunctions”). Some key points: <ol class="arabic"> <li>A new keyword <code class="docutils literal notranslate">codef</code> to declare a cofunction. Cofunction is always a generator, even if there is no <code class="docutils literal notranslate">cocall</code> expressions inside it. Maps to <code class="docutils literal notranslate">async def</code> in this proposal.</li> <li>A new keyword <code class="docutils literal notranslate">cocall</code> to call a cofunction. Can only be used inside a cofunction. Maps to <code class="docutils literal notranslate">await</code> in this proposal (with some differences, see below).</li> <li>It is not possible to call a cofunction without a <code class="docutils literal notranslate">cocall</code> keyword.</li> <li><code class="docutils literal notranslate">cocall</code> grammatically requires parentheses after it:<div class="highlight-default notranslate"><div class="highlight"><pre>atom: cocall | <existing alternatives for atom> cocall: 'cocall' atom cotrailer* '(' [arglist] ')' cotrailer: '[' subscriptlist ']' | '.' NAME </pre></div> </div> </li> <li><code class="docutils literal notranslate">cocall f(*args, **kwds)</code> is semantically equivalent to <code class="docutils literal notranslate">yield from f.__cocall__(*args, **kwds)</code>.</li> </ol> Differences from this proposal: <ol class="arabic"> <li>There is no equivalent of <code class="docutils literal notranslate">__cocall__</code> in this PEP, which is called and its result is passed to <code class="docutils literal notranslate">yield from</code> in the <code class="docutils literal notranslate">cocall</code> expression. <code class="docutils literal notranslate">await</code> keyword expects an awaitable object, validates the type, and executes <code class="docutils literal notranslate">yield from</code> on it. Although, <code class="docutils literal notranslate">__await__</code> method is similar to <code class="docutils literal notranslate">__cocall__</code>, but is only used to define Future-like objects.</li> <li><code class="docutils literal notranslate">await</code> is defined in almost the same way as <code class="docutils literal notranslate">yield from</code> in the grammar (it is later enforced that <code class="docutils literal notranslate">await</code> can only be inside <code class="docutils literal notranslate">async def</code>). It is possible to simply write <code class="docutils literal notranslate">await future</code>, whereas <code class="docutils literal notranslate">cocall</code> always requires parentheses.</li> <li>To make asyncio work with <a class="pep reference internal" href="../pep-3152/" title="PEP 3152 – Cofunctions">PEP 3152</a> it would be required to modify <code class="docutils literal notranslate">@asyncio.coroutine</code> decorator to wrap all functions in an object with a <code class="docutils literal notranslate">__cocall__</code> method, or to implement <code class="docutils literal notranslate">__cocall__</code> on generators. To call cofunctions from existing generator-based coroutines it would be required to use <code class="docutils literal notranslate">costart(cofunc, *args, **kwargs)</code> built-in.</li> <li>Since it is impossible to call a cofunction without a <code class="docutils literal notranslate">cocall</code> keyword, it automatically prevents the common mistake of forgetting to use <code class="docutils literal notranslate">yield from</code> on generator-based coroutines. This proposal addresses this problem with a different approach, see <a class="reference internal" href="#debugging-features">Debugging Features</a>.</li> <li>A shortcoming of requiring a <code class="docutils literal notranslate">cocall</code> keyword to call a coroutine is that if is decided to implement coroutine-generators – coroutines with <code class="docutils literal notranslate">yield</code> or <code class="docutils literal notranslate">async yield</code> expressions – we wouldn’t need a <code class="docutils literal notranslate">cocall</code> keyword to call them. So we’ll end up having <code class="docutils literal notranslate">__cocall__</code> and no <code class="docutils literal notranslate">__call__</code> for regular coroutines, and having <code class="docutils literal notranslate">__call__</code> and no <code class="docutils literal notranslate">__cocall__</code> for coroutine-generators.</li> <li>Requiring parentheses grammatically also introduces a whole lot of new problems.The following code: <div class="highlight-default notranslate"><div class="highlight"><pre>await fut await function_returning_future() await asyncio.gather(coro1(arg1, arg2), coro2(arg1, arg2)) </pre></div> </div> would look like: <div class="highlight-default notranslate"><div class="highlight"><pre>cocall fut() # or cocall costart(fut) cocall (function_returning_future())() cocall asyncio.gather(costart(coro1, arg1, arg2), costart(coro2, arg1, arg2)) </pre></div> </div> </li> <li>There are no equivalents of <code class="docutils literal notranslate">async for</code> and <code class="docutils literal notranslate">async with</code> in PEP 3152.</li> </ol> </section> <section id="coroutine-generators"> <h3><a class="toc-backref" href="#coroutine-generators" role="doc-backlink">Coroutine-generators</a></h3> With <code class="docutils literal notranslate">async for</code> keyword it is desirable to have a concept of a coroutine-generator – a coroutine with <code class="docutils literal notranslate">yield</code> and <code class="docutils literal notranslate">yield from</code> expressions. To avoid any ambiguity with regular generators, we would likely require to have an <code class="docutils literal notranslate">async</code> keyword before <code class="docutils literal notranslate">yield</code>, and <code class="docutils literal notranslate">async yield from</code> would raise a <code class="docutils literal notranslate">StopAsyncIteration</code> exception. While it is possible to implement coroutine-generators, we believe that they are out of scope of this proposal. It is an advanced concept that should be carefully considered and balanced, with a non-trivial changes in the implementation of current generator objects. This is a matter for a separate PEP. </section> <section id="why-async-and-await-keywords"> <h3><a class="toc-backref" href="#why-async-and-await-keywords" role="doc-backlink">Why “async” and “await” keywords</a></h3> async/await is not a new concept in programming languages: <ul class="simple"> <li>C# has it since long time ago <a class="footnote-reference brackets" href="#id35" id="id17">[5]</a>;</li> <li>proposal to add async/await in ECMAScript 7 <a class="footnote-reference brackets" href="#id32" id="id18">[2]</a>; see also Traceur project <a class="footnote-reference brackets" href="#id39" id="id19">[9]</a>;</li> <li>Facebook’s Hack/HHVM <a class="footnote-reference brackets" href="#id36" id="id20">[6]</a>;</li> <li>Google’s Dart language <a class="footnote-reference brackets" href="#id37" id="id21">[7]</a>;</li> <li>Scala <a class="footnote-reference brackets" href="#id38" id="id22">[8]</a>;</li> <li>proposal to add async/await to C++ <a class="footnote-reference brackets" href="#id40" id="id23">[10]</a>;</li> <li>and many other less popular languages.</li> </ul> This is a huge benefit, as some users already have experience with async/await, and because it makes working with many languages in one project easier (Python with ECMAScript 7 for instance). </section> <section id="why-aiter-does-not-return-an-awaitable"> <h3><a class="toc-backref" href="#why-aiter-does-not-return-an-awaitable" role="doc-backlink">Why “__aiter__” does not return an awaitable</a></h3> <a class="pep reference internal" href="../pep-0492/" title="PEP 492 – Coroutines with async and await syntax">PEP 492</a> was accepted in CPython 3.5.0 with <code class="docutils literal notranslate">__aiter__</code> defined as a method, that was expected to return an awaitable resolving to an asynchronous iterator. In 3.5.2 (as <a class="pep reference internal" href="../pep-0492/" title="PEP 492 – Coroutines with async and await syntax">PEP 492</a> was accepted on a provisional basis) the <code class="docutils literal notranslate">__aiter__</code> protocol was updated to return asynchronous iterators directly. The motivation behind this change is to make it possible to implement asynchronous generators in Python. See <a class="footnote-reference brackets" href="#id49" id="id24">[19]</a> and <a class="footnote-reference brackets" href="#id50" id="id25">[20]</a> for more details. </section> <section id="importance-of-async-keyword"> <h3><a class="toc-backref" href="#importance-of-async-keyword" role="doc-backlink">Importance of “async” keyword</a></h3> While it is possible to just implement <code class="docutils literal notranslate">await</code> expression and treat all functions with at least one <code class="docutils literal notranslate">await</code> as coroutines, this approach makes APIs design, code refactoring and its long time support harder. Let’s pretend that Python only has <code class="docutils literal notranslate">await</code> keyword: <div class="highlight-default notranslate"><div class="highlight"><pre>def useful(): ... await log(...) ... def important(): await useful() </pre></div> </div> If <code class="docutils literal notranslate">useful()</code> function is refactored and someone removes all <code class="docutils literal notranslate">await</code> expressions from it, it would become a regular python function, and all code that depends on it, including <code class="docutils literal notranslate">important()</code> would be broken. To mitigate this issue a decorator similar to <code class="docutils literal notranslate">@asyncio.coroutine</code> has to be introduced. </section> <section id="why-async-def"> <h3><a class="toc-backref" href="#why-async-def" role="doc-backlink">Why “async def”</a></h3> For some people bare <code class="docutils literal notranslate">async name(): pass</code> syntax might look more appealing than <code class="docutils literal notranslate">async def name(): pass</code>. It is certainly easier to type. But on the other hand, it breaks the symmetry between <code class="docutils literal notranslate">async def</code>, <code class="docutils literal notranslate">async with</code> and <code class="docutils literal notranslate">async for</code>, where <code class="docutils literal notranslate">async</code> is a modifier, stating that the statement is asynchronous. It is also more consistent with the existing grammar. </section> <section id="why-not-await-for-and-await-with"> <h3><a class="toc-backref" href="#why-not-await-for-and-await-with" role="doc-backlink">Why not “await for” and “await with”</a></h3> <code class="docutils literal notranslate">async</code> is an adjective, and hence it is a better choice for a statement qualifier keyword. <code class="docutils literal notranslate">await for/with</code> would imply that something is awaiting for a completion of a <code class="docutils literal notranslate">for</code> or <code class="docutils literal notranslate">with</code> statement. </section> <section id="why-async-def-and-not-def-async"> <h3><a class="toc-backref" href="#why-async-def-and-not-def-async" role="doc-backlink">Why “async def” and not “def async”</a></h3> <code class="docutils literal notranslate">async</code> keyword is a statement qualifier. A good analogy to it are “static”, “public”, “unsafe” keywords from other languages. “async for” is an asynchronous “for” statement, “async with” is an asynchronous “with” statement, “async def” is an asynchronous function. Having “async” after the main statement keyword might introduce some confusion, like “for async item in iterator” can be read as “for each asynchronous item in iterator”. Having <code class="docutils literal notranslate">async</code> keyword before <code class="docutils literal notranslate">def</code>, <code class="docutils literal notranslate">with</code> and <code class="docutils literal notranslate">for</code> also makes the language grammar simpler. And “async def” better separates coroutines from regular functions visually. </section> <section id="why-not-a-future-import"> <h3><a class="toc-backref" href="#why-not-a-future-import" role="doc-backlink">Why not a __future__ import</a></h3> <a class="reference internal" href="#transition-plan">Transition Plan</a> section explains how tokenizer is modified to treat <code class="docutils literal notranslate">async</code> and <code class="docutils literal notranslate">await</code> as keywords only in <code class="docutils literal notranslate">async def</code> blocks. Hence <code class="docutils literal notranslate">async def</code> fills the role that a module level compiler declaration like <code class="docutils literal notranslate">from __future__ import async_await</code> would otherwise fill. </section> <section id="why-magic-methods-start-with-a"> <h3><a class="toc-backref" href="#why-magic-methods-start-with-a" role="doc-backlink">Why magic methods start with “a”</a></h3> New asynchronous magic methods <code class="docutils literal notranslate">__aiter__</code>, <code class="docutils literal notranslate">__anext__</code>, <code class="docutils literal notranslate">__aenter__</code>, and <code class="docutils literal notranslate">__aexit__</code> all start with the same prefix “a”. An alternative proposal is to use “async” prefix, so that <code class="docutils literal notranslate">__anext__</code> becomes <code class="docutils literal notranslate">__async_next__</code>. However, to align new magic methods with the existing ones, such as <code class="docutils literal notranslate">__radd__</code> and <code class="docutils literal notranslate">__iadd__</code> it was decided to use a shorter version. </section> <section id="why-not-reuse-existing-magic-names"> <h3><a class="toc-backref" href="#why-not-reuse-existing-magic-names" role="doc-backlink">Why not reuse existing magic names</a></h3> An alternative idea about new asynchronous iterators and context managers was to reuse existing magic methods, by adding an <code class="docutils literal notranslate">async</code> keyword to their declarations: <div class="highlight-default notranslate"><div class="highlight"><pre>class CM: async def __enter__(self): # instead of __aenter__ ... </pre></div> </div> This approach has the following downsides: <ul class="simple"> <li>it would not be possible to create an object that works in both <code class="docutils literal notranslate">with</code> and <code class="docutils literal notranslate">async with</code> statements;</li> <li>it would break backwards compatibility, as nothing prohibits from returning a Future-like objects from <code class="docutils literal notranslate">__enter__</code> and/or <code class="docutils literal notranslate">__exit__</code> in Python <= 3.4;</li> <li>one of the main points of this proposal is to make native coroutines as simple and foolproof as possible, hence the clear separation of the protocols.</li> </ul> </section> <section id="why-not-reuse-existing-for-and-with-statements"> <h3><a class="toc-backref" href="#why-not-reuse-existing-for-and-with-statements" role="doc-backlink">Why not reuse existing “for” and “with” statements</a></h3> The vision behind existing generator-based coroutines and this proposal is to make it easy for users to see where the code might be suspended. Making existing “for” and “with” statements to recognize asynchronous iterators and context managers will inevitably create implicit suspend points, making it harder to reason about the code. </section> <section id="comprehensions"> <h3><a class="toc-backref" href="#comprehensions" role="doc-backlink">Comprehensions</a></h3> Syntax for asynchronous comprehensions could be provided, but this construct is outside of the scope of this PEP. </section> <section id="async-lambda-functions"> <h3><a class="toc-backref" href="#async-lambda-functions" role="doc-backlink">Async lambda functions</a></h3> Syntax for asynchronous lambda functions could be provided, but this construct is outside of the scope of this PEP. </section> </section> <section id="performance"> <h2><a class="toc-backref" href="#performance" role="doc-backlink">Performance</a></h2> <section id="overall-impact"> <h3><a class="toc-backref" href="#overall-impact" role="doc-backlink">Overall Impact</a></h3> This proposal introduces no observable performance impact. Here is an output of python’s official set of benchmarks <a class="footnote-reference brackets" href="#id34" id="id26">[4]</a>: <div class="highlight-text notranslate"><div class="highlight"><pre>python perf.py -r -b default ../cpython/python.exe ../cpython-aw/python.exe [skipped] Report on Darwin ysmac 14.3.0 Darwin Kernel Version 14.3.0: Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 x86_64 i386 Total CPU cores: 8 ### etree_iterparse ### Min: 0.365359 -> 0.349168: 1.05x faster Avg: 0.396924 -> 0.379735: 1.05x faster Significant (t=9.71) Stddev: 0.01225 -> 0.01277: 1.0423x larger The following not significant results are hidden, use -v to show them: django_v2, 2to3, etree_generate, etree_parse, etree_process, fastpickle, fastunpickle, json_dump_v2, json_load, nbody, regex_v8, tornado_http. </pre></div> </div> </section> <section id="tokenizer-modifications"> <h3><a class="toc-backref" href="#tokenizer-modifications" role="doc-backlink">Tokenizer modifications</a></h3> There is no observable slowdown of parsing python files with the modified tokenizer: parsing of one 12Mb file (<code class="docutils literal notranslate">Lib/test/test_binop.py</code> repeated 1000 times) takes the same amount of time. </section> <section id="async-await"> <h3><a class="toc-backref" href="#async-await" role="doc-backlink">async/await</a></h3> The following micro-benchmark was used to determine performance difference between “async” functions and generators: <div class="highlight-default notranslate"><div class="highlight"><pre>import sys import time def binary(n): if n <= 0: return 1 l = yield from binary(n - 1) r = yield from binary(n - 1) return l + 1 + r async def abinary(n): if n <= 0: return 1 l = await abinary(n - 1) r = await abinary(n - 1) return l + 1 + r def timeit(func, depth, repeat): t0 = time.time() for _ in range(repeat): o = func(depth) try: while True: o.send(None) except StopIteration: pass t1 = time.time() print('{}({}) * {}: total {:.3f}s'.format( func.__name__, depth, repeat, t1-t0)) </pre></div> </div> The result is that there is no observable performance difference: <div class="highlight-default notranslate"><div class="highlight"><pre>binary(19) * 30: total 53.321s abinary(19) * 30: total 55.073s binary(19) * 30: total 53.361s abinary(19) * 30: total 51.360s binary(19) * 30: total 49.438s abinary(19) * 30: total 51.047s </pre></div> </div> Note that depth of 19 means 1,048,575 calls. </section> </section> <section id="reference-implementation"> <h2><a class="toc-backref" href="#reference-implementation" role="doc-backlink">Reference Implementation</a></h2> The reference implementation can be found here: <a class="footnote-reference brackets" href="#id33" id="id27">[3]</a>. <section id="list-of-high-level-changes-and-new-protocols"> <h3><a class="toc-backref" href="#list-of-high-level-changes-and-new-protocols" role="doc-backlink">List of high-level changes and new protocols</a></h3> <ol class="arabic simple"> <li>New syntax for defining coroutines: <code class="docutils literal notranslate">async def</code> and new <code class="docutils literal notranslate">await</code> keyword.</li> <li>New <code class="docutils literal notranslate">__await__</code> method for Future-like objects, and new <code class="docutils literal notranslate">tp_as_async.am_await</code> slot in <code class="docutils literal notranslate">PyTypeObject</code>.</li> <li>New syntax for asynchronous context managers: <code class="docutils literal notranslate">async with</code>. And associated protocol with <code class="docutils literal notranslate">__aenter__</code> and <code class="docutils literal notranslate">__aexit__</code> methods.</li> <li>New syntax for asynchronous iteration: <code class="docutils literal notranslate">async for</code>. And associated protocol with <code class="docutils literal notranslate">__aiter__</code>, <code class="docutils literal notranslate">__aexit__</code> and new built-in exception <code class="docutils literal notranslate">StopAsyncIteration</code>. New <code class="docutils literal notranslate">tp_as_async.am_aiter</code> and <code class="docutils literal notranslate">tp_as_async.am_anext</code> slots in <code class="docutils literal notranslate">PyTypeObject</code>.</li> <li>New AST nodes: <code class="docutils literal notranslate">AsyncFunctionDef</code>, <code class="docutils literal notranslate">AsyncFor</code>, <code class="docutils literal notranslate">AsyncWith</code>, <code class="docutils literal notranslate">Await</code>.</li> <li>New functions: <code class="docutils literal notranslate">sys.set_coroutine_wrapper(callback)</code>, <code class="docutils literal notranslate">sys.get_coroutine_wrapper()</code>, <code class="docutils literal notranslate">types.coroutine(gen)</code>, <code class="docutils literal notranslate">inspect.iscoroutinefunction(func)</code>, <code class="docutils literal notranslate">inspect.iscoroutine(obj)</code>, <code class="docutils literal notranslate">inspect.isawaitable(obj)</code>, <code class="docutils literal notranslate">inspect.getcoroutinestate(coro)</code>, and <code class="docutils literal notranslate">inspect.getcoroutinelocals(coro)</code>.</li> <li>New <code class="docutils literal notranslate">CO_COROUTINE</code> and <code class="docutils literal notranslate">CO_ITERABLE_COROUTINE</code> bit flags for code objects.</li> <li>New ABCs: <code class="docutils literal notranslate">collections.abc.Awaitable</code>, <code class="docutils literal notranslate">collections.abc.Coroutine</code>, <code class="docutils literal notranslate">collections.abc.AsyncIterable</code>, and <code class="docutils literal notranslate">collections.abc.AsyncIterator</code>.</li> <li>C API changes: new <code class="docutils literal notranslate">PyCoro_Type</code> (exposed to Python as <code class="docutils literal notranslate">types.CoroutineType</code>) and <code class="docutils literal notranslate">PyCoroObject</code>. <code class="docutils literal notranslate">PyCoro_CheckExact(*o)</code> to test if <code class="docutils literal notranslate">o</code> is a native coroutine.</li> </ol> While the list of changes and new things is not short, it is important to understand, that most users will not use these features directly. It is intended to be used in frameworks and libraries to provide users with convenient to use and unambiguous APIs with <code class="docutils literal notranslate">async def</code>, <code class="docutils literal notranslate">await</code>, <code class="docutils literal notranslate">async for</code> and <code class="docutils literal notranslate">async with</code> syntax. </section> <section id="working-example"> <h3><a class="toc-backref" href="#working-example" role="doc-backlink">Working example</a></h3> All concepts proposed in this PEP are implemented <a class="footnote-reference brackets" href="#id33" id="id28">[3]</a> and can be tested. <div class="highlight-python notranslate"><div class="highlight"><pre>import asyncio async def echo_server(): print('Serving on localhost:8000') await asyncio.start_server(handle_connection, 'localhost', 8000) async def handle_connection(reader, writer): print('New connection...') while True: data = await reader.read(8192) if not data: break print('Sending {:.10}... back'.format(repr(data))) writer.write(data) loop = asyncio.get_event_loop() loop.run_until_complete(echo_server()) try: loop.run_forever() finally: loop.close() </pre></div> </div> </section> </section> <section id="acceptance"> <h2><a class="toc-backref" href="#acceptance" role="doc-backlink">Acceptance</a></h2> <a class="pep reference internal" href="../pep-0492/" title="PEP 492 – Coroutines with async and await syntax">PEP 492</a> was accepted by Guido, Tuesday, May 5, 2015 <a class="footnote-reference brackets" href="#id44" id="id29">[14]</a>. </section> <section id="implementation"> <h2><a class="toc-backref" href="#implementation" role="doc-backlink">Implementation</a></h2> The implementation is tracked in issue 24017 <a class="footnote-reference brackets" href="#id45" id="id30">[15]</a>. It was committed on May 11, 2015. </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <aside class="footnote-list brackets"> <aside class="footnote brackets" id="id31" role="doc-footnote"> <dt class="label" id="id31">[<a href="#id13">1</a>]</dt> <dd><a class="reference external" href="https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine">https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine</a></aside> <aside class="footnote brackets" id="id32" role="doc-footnote"> <dt class="label" id="id32">[2] (<a href='#id1'>1</a>, <a href='#id18'>2</a>) </dt> <dd><a class="reference external" href="http://wiki.ecmascript.org/doku.php?id=strawman:async_functions">http://wiki.ecmascript.org/doku.php?id=strawman:async_functions</a></aside> <aside class="footnote brackets" id="id33" role="doc-footnote"> <dt class="label" id="id33">[3] (<a href='#id27'>1</a>, <a href='#id28'>2</a>) </dt> <dd><a class="reference external" href="https://github.com/1st1/cpython/tree/await">https://github.com/1st1/cpython/tree/await</a></aside> <aside class="footnote brackets" id="id34" role="doc-footnote"> <dt class="label" id="id34">[<a href="#id26">4</a>]</dt> <dd><a class="reference external" href="https://hg.python.org/benchmarks">https://hg.python.org/benchmarks</a></aside> <aside class="footnote brackets" id="id35" role="doc-footnote"> <dt class="label" id="id35">[5] (<a href='#id2'>1</a>, <a href='#id17'>2</a>) </dt> <dd><a class="reference external" href="https://msdn.microsoft.com/en-us/library/hh191443.aspx">https://msdn.microsoft.com/en-us/library/hh191443.aspx</a></aside> <aside class="footnote brackets" id="id36" role="doc-footnote"> <dt class="label" id="id36">[6] (<a href='#id3'>1</a>, <a href='#id20'>2</a>) </dt> <dd><a class="reference external" href="http://docs.hhvm.com/manual/en/hack.async.php">http://docs.hhvm.com/manual/en/hack.async.php</a></aside> <aside class="footnote brackets" id="id37" role="doc-footnote"> <dt class="label" id="id37">[7] (<a href='#id4'>1</a>, <a href='#id21'>2</a>) </dt> <dd><a class="reference external" href="https://www.dartlang.org/articles/await-async/">https://www.dartlang.org/articles/await-async/</a></aside> <aside class="footnote brackets" id="id38" role="doc-footnote"> <dt class="label" id="id38">[8] (<a href='#id5'>1</a>, <a href='#id22'>2</a>) </dt> <dd><a class="reference external" href="http://docs.scala-lang.org/sips/pending/async.html">http://docs.scala-lang.org/sips/pending/async.html</a></aside> <aside class="footnote brackets" id="id39" role="doc-footnote"> <dt class="label" id="id39">[<a href="#id19">9</a>]</dt> <dd><a class="reference external" href="https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-functions-experimental">https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-functions-experimental</a></aside> <aside class="footnote brackets" id="id40" role="doc-footnote"> <dt class="label" id="id40">[10] (<a href='#id6'>1</a>, <a href='#id23'>2</a>) </dt> <dd><a class="reference external" href="http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf">http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf</a> (PDF)</aside> <aside class="footnote brackets" id="id41" role="doc-footnote"> <dt class="label" id="id41">[<a href="#id14">11</a>]</dt> <dd><a class="reference external" href="https://docs.python.org/3/reference/expressions.html#generator-iterator-methods">https://docs.python.org/3/reference/expressions.html#generator-iterator-methods</a></aside> <aside class="footnote brackets" id="id42" role="doc-footnote"> <dt class="label" id="id42">[<a href="#id11">12</a>]</dt> <dd><a class="reference external" href="https://docs.python.org/3/reference/expressions.html#primaries">https://docs.python.org/3/reference/expressions.html#primaries</a></aside> <aside class="footnote brackets" id="id43" role="doc-footnote"> <dt class="label" id="id43">[<a href="#id15">13</a>]</dt> <dd><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2015-May/139851.html">https://mail.python.org/pipermail/python-dev/2015-May/139851.html</a></aside> <aside class="footnote brackets" id="id44" role="doc-footnote"> <dt class="label" id="id44">[<a href="#id29">14</a>]</dt> <dd><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2015-May/139844.html">https://mail.python.org/pipermail/python-dev/2015-May/139844.html</a></aside> <aside class="footnote brackets" id="id45" role="doc-footnote"> <dt class="label" id="id45">[<a href="#id30">15</a>]</dt> <dd><a class="reference external" href="http://bugs.python.org/issue24017">http://bugs.python.org/issue24017</a></aside> <aside class="footnote brackets" id="id46" role="doc-footnote"> <dt class="label" id="id46">[<a href="#id16">16</a>]</dt> <dd><a class="reference external" href="https://github.com/python/asyncio/issues/233">https://github.com/python/asyncio/issues/233</a></aside> <aside class="footnote brackets" id="id47" role="doc-footnote"> <dt class="label" id="id47">[<a href="#id7">17</a>]</dt> <dd><a class="reference external" href="https://hg.python.org/cpython/rev/7a0a1a4ac639">https://hg.python.org/cpython/rev/7a0a1a4ac639</a></aside> <aside class="footnote brackets" id="id48" role="doc-footnote"> <dt class="label" id="id48">[<a href="#id8">18</a>]</dt> <dd><a class="reference external" href="http://bugs.python.org/issue24400">http://bugs.python.org/issue24400</a></aside> <aside class="footnote brackets" id="id49" role="doc-footnote"> <dt class="label" id="id49">[19] (<a href='#id9'>1</a>, <a href='#id24'>2</a>) </dt> <dd><a class="reference external" href="http://bugs.python.org/issue27243">http://bugs.python.org/issue27243</a></aside> <aside class="footnote brackets" id="id50" role="doc-footnote"> <dt class="label" id="id50">[20] (<a href='#id10'>1</a>, <a href='#id25'>2</a>) </dt> <dd><a class="reference external" href="https://docs.python.org/3/reference/datamodel.html#async-iterators">https://docs.python.org/3/reference/datamodel.html#async-iterators</a></aside> </aside> </section> <section id="acknowledgments"> <h2><a class="toc-backref" href="#acknowledgments" role="doc-backlink">Acknowledgments</a></h2> I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew Svetlov, Łukasz Langa, Greg Ewing, Stephen J. Turnbull, Jim J. Jewett, Brett Cannon, Alyssa Coghlan, Steven D’Aprano, Paul Moore, Nathaniel Smith, Ethan Furman, Stefan Behnel, Paul Sokolovsky, Victor Petrovykh, and many others for their feedback, ideas, edits, criticism, code reviews, and discussions around this PEP. </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0492.rst">https://github.com/python/peps/blob/main/peps/pep-0492.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0492.rst">2025-02-01 08:55:40 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#api-design-and-implementation-revisions">API Design and Implementation Revisions</a></li> <li><a class="reference internal" href="#rationale-and-goals">Rationale and Goals</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#new-coroutine-declaration-syntax">New Coroutine Declaration Syntax</a></li> <li><a class="reference internal" href="#types-coroutine">types.coroutine()</a></li> <li><a class="reference internal" href="#await-expression">Await Expression</a><ul> <li><a class="reference internal" href="#updated-operator-precedence-table">Updated operator precedence table</a></li> <li><a class="reference internal" href="#examples-of-await-expressions">Examples of “await” expressions</a></li> </ul> </li> <li><a class="reference internal" href="#asynchronous-context-managers-and-async-with">Asynchronous Context Managers and “async with”</a><ul> <li><a class="reference internal" href="#new-syntax">New Syntax</a></li> <li><a class="reference internal" href="#example">Example</a></li> </ul> </li> <li><a class="reference internal" href="#asynchronous-iterators-and-async-for">Asynchronous Iterators and “async for”</a><ul> <li><a class="reference internal" href="#id12">New Syntax</a></li> <li><a class="reference internal" href="#example-1">Example 1</a></li> <li><a class="reference internal" href="#example-2">Example 2</a></li> <li><a class="reference internal" href="#why-stopasynciteration">Why StopAsyncIteration?</a></li> </ul> </li> <li><a class="reference internal" href="#coroutine-objects">Coroutine objects</a><ul> <li><a class="reference internal" href="#differences-from-generators">Differences from generators</a></li> <li><a class="reference internal" href="#coroutine-object-methods">Coroutine object methods</a></li> </ul> </li> <li><a class="reference internal" href="#debugging-features">Debugging Features</a></li> <li><a class="reference internal" href="#new-standard-library-functions">New Standard Library Functions</a></li> <li><a class="reference internal" href="#new-abstract-base-classes">New Abstract Base Classes</a></li> </ul> </li> <li><a class="reference internal" href="#glossary">Glossary</a></li> <li><a class="reference internal" href="#transition-plan">Transition Plan</a><ul> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a><ul> <li><a class="reference internal" href="#asyncio">asyncio</a></li> <li><a class="reference internal" href="#asyncio-migration-strategy">asyncio migration strategy</a></li> <li><a class="reference internal" href="#async-await-in-cpython-code-base">async/await in CPython code base</a></li> </ul> </li> <li><a class="reference internal" href="#grammar-updates">Grammar Updates</a></li> <li><a class="reference internal" href="#deprecation-plans">Deprecation Plans</a></li> </ul> </li> <li><a class="reference internal" href="#design-considerations">Design Considerations</a><ul> <li><a class="reference internal" href="#pep-3152">PEP 3152</a></li> <li><a class="reference internal" href="#coroutine-generators">Coroutine-generators</a></li> <li><a class="reference internal" href="#why-async-and-await-keywords">Why “async” and “await” keywords</a></li> <li><a class="reference internal" href="#why-aiter-does-not-return-an-awaitable">Why “__aiter__” does not return an awaitable</a></li> <li><a class="reference internal" href="#importance-of-async-keyword">Importance of “async” keyword</a></li> <li><a class="reference internal" href="#why-async-def">Why “async def”</a></li> <li><a class="reference internal" href="#why-not-await-for-and-await-with">Why not “await for” and “await with”</a></li> <li><a class="reference internal" href="#why-async-def-and-not-def-async">Why “async def” and not “def async”</a></li> <li><a class="reference internal" href="#why-not-a-future-import">Why not a __future__ import</a></li> <li><a class="reference internal" href="#why-magic-methods-start-with-a">Why magic methods start with “a”</a></li> <li><a class="reference internal" href="#why-not-reuse-existing-magic-names">Why not reuse existing magic names</a></li> <li><a class="reference internal" href="#why-not-reuse-existing-for-and-with-statements">Why not reuse existing “for” and “with” statements</a></li> <li><a class="reference internal" href="#comprehensions">Comprehensions</a></li> <li><a class="reference internal" href="#async-lambda-functions">Async lambda functions</a></li> </ul> </li> <li><a class="reference internal" href="#performance">Performance</a><ul> <li><a class="reference internal" href="#overall-impact">Overall Impact</a></li> <li><a class="reference internal" href="#tokenizer-modifications">Tokenizer modifications</a></li> <li><a class="reference internal" href="#async-await">async/await</a></li> </ul> </li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a><ul> <li><a class="reference internal" href="#list-of-high-level-changes-and-new-protocols">List of high-level changes and new protocols</a></li> <li><a class="reference internal" href="#working-example">Working example</a></li> </ul> </li> <li><a class="reference internal" href="#acceptance">Acceptance</a></li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#acknowledgments">Acknowledgments</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0492.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>