CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 555 – Context-local variables (contextvars) | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0555/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 555 – Context-local variables (contextvars) | peps.python.org'> <meta property="og:description" content="Sometimes, in special cases, it is desired that code can pass information down the function call chain to the callees without having to explicitly pass the information as arguments to each function in the call chain. This proposal describes a construct ..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0555/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="Sometimes, in special cases, it is desired that code can pass information down the function call chain to the callees without having to explicitly pass the information as arguments to each function in the call chain. This proposal describes a construct ..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 555</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 555 – Context-local variables (contextvars)</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Koos Zevenhoven</dd> <dt class="field-even">Status:</dt> <dd class="field-even"><abbr title="Removed from consideration by sponsor or authors">Withdrawn</abbr></dd> <dt class="field-odd">Type:</dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created:</dt> <dd class="field-even">06-Sep-2017</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">3.7</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even">06-Sep-2017</dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#proposal">Proposal</a><ul> <li><a class="reference internal" href="#semantics-and-higher-level-api">Semantics and higher-level API</a><ul> <li><a class="reference internal" href="#core-concept">Core concept</a></li> <li><a class="reference internal" href="#refactoring-into-subroutines">Refactoring into subroutines</a></li> <li><a class="reference internal" href="#semantics-for-generators-and-generator-based-coroutines">Semantics for generators and generator-based coroutines</a></li> </ul> </li> <li><a class="reference internal" href="#special-functionality-for-framework-authors">Special functionality for framework authors</a><ul> <li><a class="reference internal" href="#leaking-yields">Leaking yields</a></li> <li><a class="reference internal" href="#capturing-contextvar-assignments">Capturing contextvar assignments</a></li> <li><a class="reference internal" href="#getting-a-snapshot-of-context-state">Getting a snapshot of context state</a></li> <li><a class="reference internal" href="#running-code-in-a-clean-state">Running code in a clean state</a></li> </ul> </li> <li><a class="reference internal" href="#implementation">Implementation</a><ul> <li><a class="reference internal" href="#data-structures-and-implementation-of-the-core-concept">Data structures and implementation of the core concept</a></li> <li><a class="reference internal" href="#implementation-of-generator-and-coroutine-semantics">Implementation of generator and coroutine semantics</a></li> <li><a class="reference internal" href="#more-on-implementation">More on implementation</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards compatibility</a></li> <li><a class="reference internal" href="#open-issues">Open Issues</a><ul> <li><a class="reference internal" href="#out-of-order-de-assignments">Out-of-order de-assignments</a></li> </ul> </li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a><ul> <li><a class="reference internal" href="#dynamic-scoping-linked-to-subroutine-scopes">Dynamic scoping linked to subroutine scopes</a></li> </ul> </li> <li><a class="reference internal" href="#acknowledgements">Acknowledgements</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> Sometimes, in special cases, it is desired that code can pass information down the function call chain to the callees without having to explicitly pass the information as arguments to each function in the call chain. This proposal describes a construct which allows code to explicitly switch in and out of a context where a certain context variable has a given value assigned to it. This is a modern alternative to some uses of things like global variables in traditional single-threaded (or thread-unsafe) code and of thread-local storage in traditional concurrency-unsafe code (single- or multi-threaded). In particular, the proposed mechanism can also be used with more modern concurrent execution mechanisms such as asynchronously executed coroutines, without the concurrently executed call chains interfering with each other’s contexts. The “call chain” can consist of normal functions, awaited coroutines, or generators. The semantics of context variable scope are equivalent in all cases, allowing code to be refactored freely into subroutines (which here refers to functions, sub-generators or sub-coroutines) without affecting the semantics of context variables. Regarding implementation, this proposal aims at simplicity and minimum changes to the CPython interpreter and to other Python interpreters. </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> Consider a modern Python call chain (or call tree), which in this proposal refers to any chained (nested) execution of subroutines, using any possible combinations of normal function calls, or expressions using <code class="docutils literal notranslate">await</code> or <code class="docutils literal notranslate">yield from</code>. In some cases, passing necessary information down the call chain as arguments can substantially complicate the required function signatures, or it can even be impossible to achieve in practice. In these cases, one may search for another place to store this information. Let us look at some historical examples. The most naive option is to assign the value to a global variable or similar, where the code down the call chain can access it. However, this immediately makes the code thread-unsafe, because with multiple threads, all threads assign to the same global variable, and another thread can interfere at any point in the call chain. Sooner or later, someone will probably find a reason to run the same code in parallel threads. A somewhat less naive option is to store the information as per-thread information in thread-local storage, where each thread has its own “copy” of the variable which other threads cannot interfere with. Although non-ideal, this has been the best solution in many cases. However, thanks to generators and coroutines, the execution of the call chain can be suspended and resumed, allowing code in other contexts to run concurrently. Therefore, using thread-local storage is concurrency-unsafe, because other call chains in other contexts may interfere with the thread-local variable. Note that in the above two historical approaches, the stored information has the widest available scope without causing problems. For a third solution along the same path, one would first define an equivalent of a “thread” for asynchronous execution and concurrency. This could be seen as the largest amount of code and nested calls that is guaranteed to be executed sequentially without ambiguity in execution order. This might be referred to as concurrency-local or task-local storage. In this meaning of “task”, there is no ambiguity in the order of execution of the code within one task. (This concept of a task is close to equivalent to a <code class="docutils literal notranslate">Task</code> in <code class="docutils literal notranslate">asyncio</code>, but not exactly.) In such concurrency-locals, it is possible to pass information down the call chain to callees without another code path interfering with the value in the background. Common to the above approaches is that they indeed use variables with a wide but just-narrow-enough scope. Thread-locals could also be called thread-wide globals—in single-threaded code, they are indeed truly global. And task-locals could be called task-wide globals, because tasks can be very big. The issue here is that neither global variables, thread-locals nor task-locals are really meant to be used for this purpose of passing information of the execution context down the call chain. Instead of the widest possible variable scope, the scope of the variables should be controlled by the programmer, typically of a library, to have the desired scope—not wider. In other words, task-local variables (and globals and thread-locals) have nothing to do with the kind of context-bound information passing that this proposal intends to enable, even if task-locals can be used to emulate the desired semantics. Therefore, in the following, this proposal describes the semantics and the outlines of an implementation for context-local variables (or context variables, contextvars). In fact, as a side effect of this PEP, an async framework can use the proposed feature to implement task-local variables. </section> <section id="proposal"> <h2><a class="toc-backref" href="#proposal" role="doc-backlink">Proposal</a></h2> Because the proposed semantics are not a direct extension to anything already available in Python, this proposal is first described in terms of semantics and API at a fairly high level. In particular, Python <code class="docutils literal notranslate">with</code> statements are heavily used in the description, as they are a good match with the proposed semantics. However, the underlying <code class="docutils literal notranslate">__enter__</code> and <code class="docutils literal notranslate">__exit__</code> methods correspond to functions in the lower-level speed-optimized (C) API. For clarity of this document, the lower-level functions are not explicitly named in the definition of the semantics. After describing the semantics and high-level API, the implementation is described, going to a lower level. <section id="semantics-and-higher-level-api"> <h3><a class="toc-backref" href="#semantics-and-higher-level-api" role="doc-backlink">Semantics and higher-level API</a></h3> <section id="core-concept"> <h4><a class="toc-backref" href="#core-concept" role="doc-backlink">Core concept</a></h4> A context-local variable is represented by a single instance of <code class="docutils literal notranslate">contextvars.Var</code>, say <code class="docutils literal notranslate">cvar</code>. Any code that has access to the <code class="docutils literal notranslate">cvar</code> object can ask for its value with respect to the current context. In the high-level API, this value is given by the <code class="docutils literal notranslate">cvar.value</code> property: <div class="highlight-default notranslate"><div class="highlight"><pre>cvar = contextvars.Var(default="the default value", description="example context variable") assert cvar.value == "the default value" # default still applies # In code examples, all ``assert`` statements should # succeed according to the proposed semantics. </pre></div> </div> No assignments to <code class="docutils literal notranslate">cvar</code> have been applied for this context, so <code class="docutils literal notranslate">cvar.value</code> gives the default value. Assigning new values to contextvars is done in a highly scope-aware manner: <div class="highlight-default notranslate"><div class="highlight"><pre>with cvar.assign(new_value): assert cvar.value is new_value # Any code here, or down the call chain from here, sees: # cvar.value is new_value # unless another value has been assigned in a # nested context assert cvar.value is new_value # the assignment of ``cvar`` to ``new_value`` is no longer visible assert cvar.value == "the default value" </pre></div> </div> Here, <code class="docutils literal notranslate">cvar.assign(value)</code> returns another object, namely <code class="docutils literal notranslate">contextvars.Assignment(cvar, new_value)</code>. The essential part here is that applying a context variable assignment (<code class="docutils literal notranslate">Assignment.__enter__</code>) is paired with a de-assignment (<code class="docutils literal notranslate">Assignment.__exit__</code>). These operations set the bounds for the scope of the assigned value. Assignments to the same context variable can be nested to override the outer assignment in a narrower context: <div class="highlight-default notranslate"><div class="highlight"><pre>assert cvar.value == "the default value" with cvar.assign("outer"): assert cvar.value == "outer" with cvar.assign("inner"): assert cvar.value == "inner" assert cvar.value == "outer" assert cvar.value == "the default value" </pre></div> </div> Also multiple variables can be assigned to in a nested manner without affecting each other: <div class="highlight-default notranslate"><div class="highlight"><pre>cvar1 = contextvars.Var() cvar2 = contextvars.Var() assert cvar1.value is None # default is None by default assert cvar2.value is None with cvar1.assign(value1): assert cvar1.value is value1 assert cvar2.value is None with cvar2.assign(value2): assert cvar1.value is value1 assert cvar2.value is value2 assert cvar1.value is value1 assert cvar2.value is None assert cvar1.value is None assert cvar2.value is None </pre></div> </div> Or with more convenient Python syntax: <div class="highlight-default notranslate"><div class="highlight"><pre>with cvar1.assign(value1), cvar2.assign(value2): assert cvar1.value is value1 assert cvar2.value is value2 </pre></div> </div> In another context, in another thread or otherwise concurrently executed task or code path, the context variables can have a completely different state. The programmer thus only needs to worry about the context at hand. </section> <section id="refactoring-into-subroutines"> <h4><a class="toc-backref" href="#refactoring-into-subroutines" role="doc-backlink">Refactoring into subroutines</a></h4> Code using contextvars can be refactored into subroutines without affecting the semantics. For instance: <div class="highlight-default notranslate"><div class="highlight"><pre>assi = cvar.assign(new_value) def apply(): assi.__enter__() assert cvar.value == "the default value" apply() assert cvar.value is new_value assi.__exit__() assert cvar.value == "the default value" </pre></div> </div> Or similarly in an asynchronous context where <code class="docutils literal notranslate">await</code> expressions are used. The subroutine can now be a coroutine: <div class="highlight-default notranslate"><div class="highlight"><pre>assi = cvar.assign(new_value) async def apply(): assi.__enter__() assert cvar.value == "the default value" await apply() assert cvar.value is new_value assi.__exit__() assert cvar.value == "the default value" </pre></div> </div> Or when the subroutine is a generator: <div class="highlight-default notranslate"><div class="highlight"><pre>def apply(): yield assi.__enter__() </pre></div> </div> which is called using <code class="docutils literal notranslate">yield from apply()</code> or with calls to <code class="docutils literal notranslate">next</code> or <code class="docutils literal notranslate">.send</code>. This is discussed further in later sections. </section> <section id="semantics-for-generators-and-generator-based-coroutines"> <h4><a class="toc-backref" href="#semantics-for-generators-and-generator-based-coroutines" role="doc-backlink">Semantics for generators and generator-based coroutines</a></h4> Generators, coroutines and async generators act as subroutines in much the same way that normal functions do. However, they have the additional possibility of being suspended by <code class="docutils literal notranslate">yield</code> expressions. Assignment contexts entered inside a generator are normally preserved across yields: <div class="highlight-default notranslate"><div class="highlight"><pre>def genfunc(): with cvar.assign(new_value): assert cvar.value is new_value yield assert cvar.value is new_value g = genfunc() next(g) assert cvar.value == "the default value" with cvar.assign(another_value): next(g) </pre></div> </div> However, the outer context visible to the generator may change state across yields: <div class="highlight-default notranslate"><div class="highlight"><pre>def genfunc(): assert cvar.value is value2 yield assert cvar.value is value1 yield with cvar.assign(value3): assert cvar.value is value3 with cvar.assign(value1): g = genfunc() with cvar.assign(value2): next(g) next(g) next(g) assert cvar.value is value1 </pre></div> </div> Similar semantics apply to async generators defined by <code class="docutils literal notranslate">async def ... yield ...</code> ). By default, values assigned inside a generator do not leak through yields to the code that drives the generator. However, the assignment contexts entered and left open inside the generator do become visible outside the generator after the generator has finished with a <code class="docutils literal notranslate">StopIteration</code> or another exception: <div class="highlight-default notranslate"><div class="highlight"><pre>assi = cvar.assign(new_value) def genfunc(): yield assi.__enter__(): yield g = genfunc() assert cvar.value == "the default value" next(g) assert cvar.value == "the default value" next(g) # assi.__enter__() is called here assert cvar.value == "the default value" next(g) assert cvar.value is new_value assi.__exit__() </pre></div> </div> </section> </section> <section id="special-functionality-for-framework-authors"> <h3><a class="toc-backref" href="#special-functionality-for-framework-authors" role="doc-backlink">Special functionality for framework authors</a></h3> Frameworks, such as <code class="docutils literal notranslate">asyncio</code> or third-party libraries, can use additional functionality in <code class="docutils literal notranslate">contextvars</code> to achieve the desired semantics in cases which are not determined by the Python interpreter. Some of the semantics described in this section are also afterwards used to describe the internal implementation. <section id="leaking-yields"> <h4><a class="toc-backref" href="#leaking-yields" role="doc-backlink">Leaking yields</a></h4> Using the <code class="docutils literal notranslate">contextvars.leaking_yields</code> decorator, one can choose to leak the context through <code class="docutils literal notranslate">yield</code> expressions into the outer context that drives the generator: <div class="highlight-default notranslate"><div class="highlight"><pre>@contextvars.leaking_yields def genfunc(): assert cvar.value == "outer" with cvar.assign("inner"): yield assert cvar.value == "inner" assert cvar.value == "outer" g = genfunc(): with cvar.assign("outer"): assert cvar.value == "outer" next(g) assert cvar.value == "inner" next(g) assert cvar.value == "outer" </pre></div> </div> </section> <section id="capturing-contextvar-assignments"> <h4><a class="toc-backref" href="#capturing-contextvar-assignments" role="doc-backlink">Capturing contextvar assignments</a></h4> Using <code class="docutils literal notranslate">contextvars.capture()</code>, one can capture the assignment contexts that are entered by a block of code. The changes applied by the block of code can then be reverted and subsequently reapplied, even in another context: <div class="highlight-default notranslate"><div class="highlight"><pre>assert cvar1.value is None # default assert cvar2.value is None # default assi1 = cvar1.assign(value1) assi2 = cvar1.assign(value2) with contextvars.capture() as delta: assi1.__enter__() with cvar2.assign("not captured"): assert cvar2.value is "not captured" assi2.__enter__() assert cvar1.value is value2 delta.revert() assert cvar1.value is None assert cvar2.value is None ... with cvar1.assign(1), cvar2.assign(2): delta.reapply() assert cvar1.value is value2 assert cvar2.value == 2 </pre></div> </div> However, reapplying the “delta” if its net contents include deassignments may not be possible (see also Implementation and Open Issues). </section> <section id="getting-a-snapshot-of-context-state"> <h4><a class="toc-backref" href="#getting-a-snapshot-of-context-state" role="doc-backlink">Getting a snapshot of context state</a></h4> The function <code class="docutils literal notranslate">contextvars.get_local_state()</code> returns an object representing the applied assignments to all context-local variables in the context where the function is called. This can be seen as equivalent to using <code class="docutils literal notranslate">contextvars.capture()</code> to capture all context changes from the beginning of execution. The returned object supports methods <code class="docutils literal notranslate">.revert()</code> and <code class="docutils literal notranslate">reapply()</code> as above. </section> <section id="running-code-in-a-clean-state"> <h4><a class="toc-backref" href="#running-code-in-a-clean-state" role="doc-backlink">Running code in a clean state</a></h4> Although it is possible to revert all applied context changes using the above primitives, a more convenient way to run a block of code in a clean context is provided: <div class="highlight-default notranslate"><div class="highlight"><pre>with context_vars.clean_context(): # here, all context vars start off with their default values # here, the state is back to what it was before the with block. </pre></div> </div> </section> </section> <section id="implementation"> <h3><a class="toc-backref" href="#implementation" role="doc-backlink">Implementation</a></h3> This section describes to a variable level of detail how the described semantics can be implemented. At present, an implementation aimed at simplicity but sufficient features is described. More details will be added later. Alternatively, a somewhat more complicated implementation offers minor additional features while adding some performance overhead and requiring more code in the implementation. <section id="data-structures-and-implementation-of-the-core-concept"> <h4><a class="toc-backref" href="#data-structures-and-implementation-of-the-core-concept" role="doc-backlink">Data structures and implementation of the core concept</a></h4> Each thread of the Python interpreter keeps its own stack of <code class="docutils literal notranslate">contextvars.Assignment</code> objects, each having a pointer to the previous (outer) assignment like in a linked list. The local state (also returned by <code class="docutils literal notranslate">contextvars.get_local_state()</code>) then consists of a reference to the top of the stack and a pointer/weak reference to the bottom of the stack. This allows efficient stack manipulations. An object produced by <code class="docutils literal notranslate">contextvars.capture()</code> is similar, but refers to only a part of the stack with the bottom reference pointing to the top of the stack as it was in the beginning of the capture block. Now, the stack evolves according to the assignment <code class="docutils literal notranslate">__enter__</code> and <code class="docutils literal notranslate">__exit__</code> methods. For example: <div class="highlight-default notranslate"><div class="highlight"><pre>cvar1 = contextvars.Var() cvar2 = contextvars.Var() # stack: [] assert cvar1.value is None assert cvar2.value is None with cvar1.assign("outer"): # stack: [Assignment(cvar1, "outer")] assert cvar1.value == "outer" with cvar1.assign("inner"): # stack: [Assignment(cvar1, "outer"), # Assignment(cvar1, "inner")] assert cvar1.value == "inner" with cvar2.assign("hello"): # stack: [Assignment(cvar1, "outer"), # Assignment(cvar1, "inner"), # Assignment(cvar2, "hello")] assert cvar2.value == "hello" # stack: [Assignment(cvar1, "outer"), # Assignment(cvar1, "inner")] assert cvar1.value == "inner" assert cvar2.value is None # stack: [Assignment(cvar1, "outer")] assert cvar1.value == "outer" # stack: [] assert cvar1.value is None assert cvar2.value is None </pre></div> </div> Getting a value from the context using <code class="docutils literal notranslate">cvar1.value</code> can be implemented as finding the topmost occurrence of a <code class="docutils literal notranslate">cvar1</code> assignment on the stack and returning the value there, or the default value if no assignment is found on the stack. However, this can be optimized to instead be an O(1) operation in most cases. Still, even searching through the stack may be reasonably fast since these stacks are not intended to grow very large. The above description is already sufficient for implementing the core concept. Suspendable frames require some additional attention, as explained in the following. </section> <section id="implementation-of-generator-and-coroutine-semantics"> <h4><a class="toc-backref" href="#implementation-of-generator-and-coroutine-semantics" role="doc-backlink">Implementation of generator and coroutine semantics</a></h4> Within generators, coroutines and async generators, assignments and deassignments are handled in exactly the same way as anywhere else. However, some changes are needed in the builtin generator methods <code class="docutils literal notranslate">send</code>, <code class="docutils literal notranslate">__next__</code>, <code class="docutils literal notranslate">throw</code> and <code class="docutils literal notranslate">close</code>. Here is the Python equivalent of the changes needed in <code class="docutils literal notranslate">send</code> for a generator (here <code class="docutils literal notranslate">_old_send</code> refers to the behavior in Python 3.6): <div class="highlight-default notranslate"><div class="highlight"><pre>def send(self, value): if self.gi_contextvars is LEAK: # If decorated with contextvars.leaking_yields. # Nothing needs to be done to leak context through yields :) return self._old_send(value) try: with contextvars.capture() as delta: if self.gi_contextvars: # non-zero captured content from previous iteration self.gi_contextvars.reapply() ret = self._old_send(value) except Exception: raise # back to the calling frame (e.g. StopIteration) else: # suspending, revert context changes but save them for later delta.revert() self.gi_contextvars = delta return ret </pre></div> </div> The corresponding modifications to the other methods is essentially identical. The same applies to coroutines and async generators. For code that does not use <code class="docutils literal notranslate">contextvars</code>, the additions are O(1) and essentially reduce to a couple of pointer comparisons. For code that does use <code class="docutils literal notranslate">contextvars</code>, the additions are still O(1) in most cases. </section> <section id="more-on-implementation"> <h4><a class="toc-backref" href="#more-on-implementation" role="doc-backlink">More on implementation</a></h4> The rest of the functionality, including <code class="docutils literal notranslate">contextvars.leaking_yields</code>, <code class="docutils literal notranslate">contextvars.capture()</code>, <code class="docutils literal notranslate">contextvars.get_local_state()</code> and <code class="docutils literal notranslate">contextvars.clean_context()</code> are in fact quite straightforward to implement, but their implementation will be discussed further in later versions of this proposal. Caching of assigned values is somewhat more complicated, and will be discussed later, but it seems that most cases should achieve O(1) complexity. </section> </section> </section> <section id="backwards-compatibility"> <h2><a class="toc-backref" href="#backwards-compatibility" role="doc-backlink">Backwards compatibility</a></h2> There are no direct backwards-compatibility concerns, since a completely new feature is proposed. However, various traditional uses of thread-local storage may need a smooth transition to <code class="docutils literal notranslate">contextvars</code> so they can be concurrency-safe. There are several approaches to this, including emulating task-local storage with a little bit of help from async frameworks. A fully general implementation cannot be provided, because the desired semantics may depend on the design of the framework. Another way to deal with the transition is for code to first look for a context created using <code class="docutils literal notranslate">contextvars</code>. If that fails because a new-style context has not been set or because the code runs on an older Python version, a fallback to thread-local storage is used. </section> <section id="open-issues"> <h2><a class="toc-backref" href="#open-issues" role="doc-backlink">Open Issues</a></h2> <section id="out-of-order-de-assignments"> <h3><a class="toc-backref" href="#out-of-order-de-assignments" role="doc-backlink">Out-of-order de-assignments</a></h3> In this proposal, all variable deassignments are made in the opposite order compared to the preceding assignments. This has two useful properties: it encourages using <code class="docutils literal notranslate">with</code> statements to define assignment scope and has a tendency to catch errors early (forgetting a <code class="docutils literal notranslate">.__exit__()</code> call often results in a meaningful error. To have this as a requirement is beneficial also in terms of implementation simplicity and performance. Nevertheless, allowing out-of-order context exits is not completely out of the question, and reasonable implementation strategies for that do exist. </section> </section> <section id="rejected-ideas"> <h2><a class="toc-backref" href="#rejected-ideas" role="doc-backlink">Rejected Ideas</a></h2> <section id="dynamic-scoping-linked-to-subroutine-scopes"> <h3><a class="toc-backref" href="#dynamic-scoping-linked-to-subroutine-scopes" role="doc-backlink">Dynamic scoping linked to subroutine scopes</a></h3> The scope of value visibility should not be determined by the way the code is refactored into subroutines. It is necessary to have per-variable control of the assignment scope. </section> </section> <section id="acknowledgements"> <h2><a class="toc-backref" href="#acknowledgements" role="doc-backlink">Acknowledgements</a></h2> To be added. </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> To be added. </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0555.rst">https://github.com/python/peps/blob/main/peps/pep-0555.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0555.rst">2025-02-01 08:59:27 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#proposal">Proposal</a><ul> <li><a class="reference internal" href="#semantics-and-higher-level-api">Semantics and higher-level API</a><ul> <li><a class="reference internal" href="#core-concept">Core concept</a></li> <li><a class="reference internal" href="#refactoring-into-subroutines">Refactoring into subroutines</a></li> <li><a class="reference internal" href="#semantics-for-generators-and-generator-based-coroutines">Semantics for generators and generator-based coroutines</a></li> </ul> </li> <li><a class="reference internal" href="#special-functionality-for-framework-authors">Special functionality for framework authors</a><ul> <li><a class="reference internal" href="#leaking-yields">Leaking yields</a></li> <li><a class="reference internal" href="#capturing-contextvar-assignments">Capturing contextvar assignments</a></li> <li><a class="reference internal" href="#getting-a-snapshot-of-context-state">Getting a snapshot of context state</a></li> <li><a class="reference internal" href="#running-code-in-a-clean-state">Running code in a clean state</a></li> </ul> </li> <li><a class="reference internal" href="#implementation">Implementation</a><ul> <li><a class="reference internal" href="#data-structures-and-implementation-of-the-core-concept">Data structures and implementation of the core concept</a></li> <li><a class="reference internal" href="#implementation-of-generator-and-coroutine-semantics">Implementation of generator and coroutine semantics</a></li> <li><a class="reference internal" href="#more-on-implementation">More on implementation</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards compatibility</a></li> <li><a class="reference internal" href="#open-issues">Open Issues</a><ul> <li><a class="reference internal" href="#out-of-order-de-assignments">Out-of-order de-assignments</a></li> </ul> </li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a><ul> <li><a class="reference internal" href="#dynamic-scoping-linked-to-subroutine-scopes">Dynamic scoping linked to subroutine scopes</a></li> </ul> </li> <li><a class="reference internal" href="#acknowledgements">Acknowledgements</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0555.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>