CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 649 – Deferred Evaluation Of Annotations Using Descriptors | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0649/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 649 – Deferred Evaluation Of Annotations Using Descriptors | peps.python.org'> <meta property="og:description" content="Annotations are a Python technology that allows expressing type information and other metadata about Python functions, classes, and modules. But Python’s original semantics for annotations required them to be eagerly evaluated, at the time the annotate..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0649/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="Annotations are a Python technology that allows expressing type information and other metadata about Python functions, classes, and modules. But Python’s original semantics for annotations required them to be eagerly evaluated, at the time the annotate..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 649</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 649 – Deferred Evaluation Of Annotations Using Descriptors</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Larry Hastings <larry at hastings.org></dd> <dt class="field-even">Discussions-To:</dt> <dd class="field-even"><a class="reference external" href="https://discuss.python.org/t/pep-649-deferred-evaluation-of-annotations-tentatively-accepted/21331/">Discourse thread</a></dd> <dt class="field-odd">Status:</dt> <dd class="field-odd"><abbr title="Normative proposal accepted for implementation">Accepted</abbr></dd> <dt class="field-even">Type:</dt> <dd class="field-even"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-odd">Topic:</dt> <dd class="field-odd"><a class="reference external" href="../topic/typing/">Typing</a></dd> <dt class="field-even">Created:</dt> <dd class="field-even">11-Jan-2021</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">3.14</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even"><a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/5QMMCRF4HTRRNJV56CGHVI5GRHVBDGQO/" title="Python-Dev thread">11-Jan-2021</a>, <a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/QSASX6PZ3LIIFIANHQQFS752BJYFUFPY/" title="Python-Dev thread">12-Apr-2021</a>, <a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/WUZGTGE43T7XV3EUGT6AN2N52OD3U7AE/" title="Python-Dev thread">18-Apr-2021</a>, <a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/2MEOWHCVDLPABOBLYUGRXVOOOBYOLLU6/" title="Python-Dev thread">09-Aug-2021</a>, <a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/SZLWVYV2HPLU6AH7DOUD7DWFUGBJGQAY/" title="Python-Dev thread">20-Oct-2021</a>, <a class="reference external" href="https://discuss.python.org/t/type-annotations-pep-649-and-pep-563/11363" title="Discourse thread">20-Oct-2021</a>, <a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/VIZEBX5EYMSYIJNDBF6DMUMZOCWHARSO/" title="Python-Dev thread">17-Nov-2021</a>, <a class="reference external" href="https://discuss.python.org/t/finding-edge-cases-for-peps-484-563-and-649-type-annotations/14314" title="Discourse thread">15-Mar-2022</a>, <a class="reference external" href="https://discuss.python.org/t/pep-649-deferred-evaluation-of-annotations-tentatively-accepted/21331" title="Discourse thread">23-Nov-2022</a>, <a class="reference external" href="https://discuss.python.org/t/two-polls-on-how-to-revise-pep-649/23628" title="Discourse thread">07-Feb-2023</a>, <a class="reference external" href="https://discuss.python.org/t/a-massive-pep-649-update-with-some-major-course-corrections/25672" title="Discourse thread">11-Apr-2023</a></dd> <dt class="field-odd">Replaces:</dt> <dd class="field-odd"><a class="reference external" href="../pep-0563/">563</a></dd> <dt class="field-even">Resolution:</dt> <dd class="field-even"><a class="reference external" href="https://discuss.python.org/t/pep-649-deferred-evaluation-of-annotations-tentatively-accepted/21331/43">08-May-2023</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#overview">Overview</a><ul> <li><a class="reference internal" href="#comparison-of-annotation-semantics">Comparison Of Annotation Semantics</a></li> <li><a class="reference internal" href="#mistaken-rejection-of-this-approach-in-november-2017">Mistaken Rejection Of This Approach In November 2017</a></li> </ul> </li> <li><a class="reference internal" href="#motivation">Motivation</a><ul> <li><a class="reference internal" href="#a-history-of-annotations">A History Of Annotations</a></li> <li><a class="reference internal" href="#the-current-state-of-annotation-use-cases">The Current State Of Annotation Use Cases</a><ul> <li><a class="reference internal" href="#static-typing-users">Static typing users</a></li> <li><a class="reference internal" href="#runtime-annotation-users">Runtime annotation users</a></li> <li><a class="reference internal" href="#wrappers">Wrappers</a></li> <li><a class="reference internal" href="#documentation">Documentation</a></li> </ul> </li> <li><a class="reference internal" href="#motivation-for-this-pep">Motivation For This PEP</a></li> </ul> </li> <li><a class="reference internal" href="#implementation">Implementation</a><ul> <li><a class="reference internal" href="#observed-semantics-for-annotations-expressions">Observed semantics for annotations expressions</a></li> <li><a class="reference internal" href="#annotate-and-annotations">__annotate__ and __annotations__</a></li> <li><a class="reference internal" href="#changes-to-allowable-annotations-syntax">Changes to allowable annotations syntax</a></li> <li><a class="reference internal" href="#changes-to-inspect-get-annotations-and-typing-get-type-hints">Changes to <code class="docutils literal notranslate">inspect.get_annotations</code> and <code class="docutils literal notranslate">typing.get_type_hints</code></a></li> <li><a class="reference internal" href="#the-stringizer-and-the-fake-globals-environment">The <code class="docutils literal notranslate">stringizer</code> and the <code class="docutils literal notranslate">fake globals</code> environment</a></li> <li><a class="reference internal" href="#compiler-generated-annotate-functions">Compiler-generated <code class="docutils literal notranslate">__annotate__</code> functions</a></li> <li><a class="reference internal" href="#third-party-annotate-functions">Third-party <code class="docutils literal notranslate">__annotate__</code> functions</a></li> <li><a class="reference internal" href="#pseudocode">Pseudocode</a></li> <li><a class="reference internal" href="#other-modifications-to-the-python-runtime">Other modifications to the Python runtime</a></li> <li><a class="reference internal" href="#interactive-repl-shell">Interactive REPL Shell</a></li> <li><a class="reference internal" href="#annotations-on-local-variables-inside-functions">Annotations On Local Variables Inside Functions</a></li> <li><a class="reference internal" href="#prototype">Prototype</a></li> <li><a class="reference internal" href="#performance-comparison">Performance Comparison</a></li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a><ul> <li><a class="reference internal" href="#backwards-compatibility-with-stock-semantics">Backwards Compatibility With Stock Semantics</a></li> <li><a class="reference internal" href="#backwards-compatibility-with-pep-563-semantics">Backwards Compatibility With PEP 563 Semantics</a></li> </ul> </li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a><ul> <li><a class="reference internal" href="#just-store-the-strings">“Just store the strings”</a></li> </ul> </li> <li><a class="reference internal" href="#acknowledgements">Acknowledgements</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> Annotations are a Python technology that allows expressing type information and other metadata about Python functions, classes, and modules. But Python’s original semantics for annotations required them to be eagerly evaluated, at the time the annotated object was bound. This caused chronic problems for static type analysis users using “type hints”, due to forward-reference and circular-reference problems. Python solved this by accepting <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a>, incorporating a new approach called “stringized annotations” in which annotations were automatically converted into strings by Python. This solved the forward-reference and circular-reference problems, and also fostered intriguing new uses for annotation metadata. But stringized annotations in turn caused chronic problems for runtime users of annotations. This PEP proposes a new and comprehensive third approach for representing and computing annotations. It adds a new internal mechanism for lazily computing annotations on demand, via a new object method called <code class="docutils literal notranslate">__annotate__</code>. This approach, when combined with a novel technique for coercing annotation values into alternative formats, solves all the above problems, supports all existing use cases, and should foster future innovations in annotations. </section> <section id="overview"> <h2><a class="toc-backref" href="#overview" role="doc-backlink">Overview</a></h2> This PEP adds a new dunder attribute to the objects that support annotations–functions, classes, and modules. The new attribute is called <code class="docutils literal notranslate">__annotate__</code>, and is a reference to a function which computes and returns that object’s annotations dict. At compile time, if the definition of an object includes annotations, the Python compiler will write the expressions computing the annotations into its own function. When run, the function will return the annotations dict. The Python compiler then stores a reference to this function in <code class="docutils literal notranslate">__annotate__</code> on the object. Furthermore, <code class="docutils literal notranslate">__annotations__</code> is redefined to be a “data descriptor” which calls this annotation function once and caches the result. This mechanism delays the evaluation of annotations expressions until the annotations are examined, which solves many circular reference problems. This PEP also defines new functionality for two functions in the Python standard library: <code class="docutils literal notranslate">inspect.get_annotations</code> and <code class="docutils literal notranslate">typing.get_type_hints</code>. The functionality is accessed via a new keyword-only parameter, <code class="docutils literal notranslate">format</code>. <code class="docutils literal notranslate">format</code> allows the user to request the annotations from these functions in a specific format. Format identifiers are always predefined integer values. The formats defined by this PEP are: <ul> <li><code class="docutils literal notranslate">inspect.VALUE = 1</code>The default value. The function will return the conventional Python values for the annotations. This format is identical to the return value for these functions under Python 3.11. </li> <li><code class="docutils literal notranslate">inspect.FORWARDREF = 2</code>The function will attempt to return the conventional Python values for the annotations. However, if it encounters an undefined name, or a free variable that has not yet been associated with a value, it dynamically creates a proxy object (a <code class="docutils literal notranslate">ForwardRef</code>) that substitutes for that value in the expression, then continues evaluation. The resulting dict may contain a mixture of proxies and real values. If all real values are defined at the time the function is called, <code class="docutils literal notranslate">inspect.FORWARDREF</code> and <code class="docutils literal notranslate">inspect.VALUE</code> produce identical results. </li> <li><code class="docutils literal notranslate">inspect.SOURCE = 3</code>The function will produce an annotation dictionary where the values have been replaced by strings containing the original source code for the annotation expressions. These strings may only be approximate, as they may be reverse-engineered from another format, rather than preserving the original source code, but the differences will be minor. </li> </ul> If accepted, this PEP would supersede <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a>, and <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a>’s behavior would be deprecated and eventually removed. <section id="comparison-of-annotation-semantics"> <h3><a class="toc-backref" href="#comparison-of-annotation-semantics" role="doc-backlink">Comparison Of Annotation Semantics</a></h3> <div class="admonition note"> Note The code presented in this section is simplified for clarity, and is intentionally inaccurate in some critical aspects. This example is intended merely to communicate the high-level concepts involved without getting lost in the details. But readers should note that the actual implementation is quite different in several important ways. See the <a class="reference internal" href="#implementation">Implementation</a> section later in this PEP for a far more accurate description of what this PEP proposes from a technical level. </div> Consider this example code: <div class="highlight-default notranslate"><div class="highlight"><pre>def foo(x: int = 3, y: MyType = None) -> float: ... class MyType: ... foo_y_annotation = foo.__annotations__['y'] </pre></div> </div> As we see here, annotations are available at runtime through an <code class="docutils literal notranslate">__annotations__</code> attribute on functions, classes, and modules. When annotations are specified on one of these objects, <code class="docutils literal notranslate">__annotations__</code> is a dictionary mapping the names of the fields to the value specified as that field’s annotation. The default behavior in Python is to evaluate the expressions for the annotations, and build the annotations dict, at the time the function, class, or module is bound. At runtime the above code actually works something like this: <div class="highlight-default notranslate"><div class="highlight"><pre>annotations = {'x': int, 'y': MyType, 'return': float} def foo(x = 3, y = "abc"): ... foo.__annotations__ = annotations class MyType: ... foo_y_annotation = foo.__annotations__['y'] </pre></div> </div> The crucial detail here is that the values <code class="docutils literal notranslate">int</code>, <code class="docutils literal notranslate">MyType</code>, and <code class="docutils literal notranslate">float</code> are looked up at the time the function object is bound, and these values are stored in the annotations dict. But this code doesn’t run—it throws a <code class="docutils literal notranslate">NameError</code> on the first line, because <code class="docutils literal notranslate">MyType</code> hasn’t been defined yet. <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a>’s solution is to decompile the expressions back into strings during compilation and store those strings as the values in the annotations dict. The equivalent runtime code would look something like this: <div class="highlight-default notranslate"><div class="highlight"><pre>annotations = {'x': 'int', 'y': 'MyType', 'return': 'float'} def foo(x = 3, y = "abc"): ... foo.__annotations__ = annotations class MyType: ... foo_y_annotation = foo.__annotations__['y'] </pre></div> </div> This code now runs successfully. However, <code class="docutils literal notranslate">foo_y_annotation</code> is no longer a reference to <code class="docutils literal notranslate">MyType</code>, it is the string <code class="docutils literal notranslate">'MyType'</code>. To turn the string into the real value <code class="docutils literal notranslate">MyType</code>, the user would need to evaluate the string using <code class="docutils literal notranslate">eval</code>, <code class="docutils literal notranslate">inspect.get_annotations</code>, or <code class="docutils literal notranslate">typing.get_type_hints</code>. This PEP proposes a third approach, delaying the evaluation of the annotations by computing them in their own function. If this PEP was active, the generated code would work something like this: <div class="highlight-default notranslate"><div class="highlight"><pre>class function: # __annotations__ on a function object is already a # "data descriptor" in Python, we're just changing # what it does @property def __annotations__(self): return self.__annotate__() # ... def annotate_foo(): return {'x': int, 'y': MyType, 'return': float} def foo(x = 3, y = "abc"): ... foo.__annotate__ = annotate_foo class MyType: ... foo_y_annotation = foo.__annotations__['y'] </pre></div> </div> The important change is that the code constructing the annotations dict now lives in a function—here, called <code class="docutils literal notranslate">annotate_foo()</code>. But this function isn’t called until we ask for the value of <code class="docutils literal notranslate">foo.__annotations__</code>, and we don’t do that until after the definition of <code class="docutils literal notranslate">MyType</code>. So this code also runs successfully, and <code class="docutils literal notranslate">foo_y_annotation</code> now has the correct value–the class <code class="docutils literal notranslate">MyType</code>–even though <code class="docutils literal notranslate">MyType</code> wasn’t defined until after the annotation was defined. </section> <section id="mistaken-rejection-of-this-approach-in-november-2017"> <h3><a class="toc-backref" href="#mistaken-rejection-of-this-approach-in-november-2017" role="doc-backlink">Mistaken Rejection Of This Approach In November 2017</a></h3> During the early days of discussion around <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a>, in a November 2017 thread in <code class="docutils literal notranslate">comp.lang.python-dev</code>, the idea of using code to delay the evaluation of annotations was briefly discussed. At the time the technique was termed an “implicit lambda expression”. Guido van Rossum—Python’s BDFL at the time—replied, asserting that these “implicit lambda expression” wouldn’t work, because they’d only be able to resolve symbols at module-level scope: <blockquote> <div>IMO the inability of referencing class-level definitions from annotations on methods pretty much kills this idea.</div></blockquote> <a class="reference external" href="https://mail.python.org/pipermail/python-dev/2017-November/150109.html">https://mail.python.org/pipermail/python-dev/2017-November/150109.html</a> This led to a short discussion about extending lambda-ized annotations for methods to be able to refer to class-level definitions, by maintaining a reference to the class-level scope. This idea, too, was quickly rejected. <a class="pep reference internal" href="../pep-0563/#keeping-the-ability-to-use-function-local-state-when-defining-annotations" title="PEP 563 – Postponed Evaluation of Annotations § Keeping the ability to use function local state when defining annotations">PEP 563 summarizes the above discussion</a> The approach taken by this PEP doesn’t suffer from these restrictions. Annotations can access module-level definitions, class-level definitions, and even local and free variables. </section> </section> <section id="motivation"> <h2><a class="toc-backref" href="#motivation" role="doc-backlink">Motivation</a></h2> <section id="a-history-of-annotations"> <h3><a class="toc-backref" href="#a-history-of-annotations" role="doc-backlink">A History Of Annotations</a></h3> Python 3.0 shipped with a new syntax feature, “annotations”, defined in <a class="pep reference internal" href="../pep-3107/" title="PEP 3107 – Function Annotations">PEP 3107</a>. This allowed specifying a Python value that would be associated with a parameter of a Python function, or with the value that function returns. Said another way, annotations gave Python users an interface to provide rich metadata about a function parameter or return value, for example type information. All the annotations for a function were stored together in a new attribute <code class="docutils literal notranslate">__annotations__</code>, in an “annotation dict” that mapped parameter names (or, in the case of the return annotation, using the name <code class="docutils literal notranslate">'return'</code>) to their Python value. In an effort to foster experimentation, Python intentionally didn’t define what form this metadata should take, or what values should be used. User code began experimenting with this new facility almost immediately. But popular libraries that make use of this functionality were slow to emerge. After years of little progress, the BDFL chose a particular approach for expressing static type information, called type hints, as defined in <a class="pep reference internal" href="../pep-0484/" title="PEP 484 – Type Hints">PEP 484</a>. Python 3.5 shipped with a new <a class="reference external" href="https://docs.python.org/3/library/typing.html#module-typing" title="(in Python v3.13)"><code class="xref py py-mod docutils literal notranslate">typing</code></a> module which quickly became very popular. Python 3.6 added syntax to annotate local variables, class attributes, and module attributes, using the approach proposed in <a class="pep reference internal" href="../pep-0526/" title="PEP 526 – Syntax for Variable Annotations">PEP 526</a>. Static type analysis continued to grow in popularity. However, static type analysis users were increasingly frustrated by an inconvenient problem: forward references. In classic Python, if a class C depends on a later-defined class D, it’s normally not a problem, because user code will usually wait until both are defined before trying to use either. But annotations added a new complication, because they were computed at the time the annotated object (function, class, or module) was bound. If methods on class C are annotated with type D, and these annotation expressions are computed at the time that the method is bound, D may not be defined yet. And if methods in D are also annotated with type C, you now have an unresolvable circular reference problem. Initially, static type users worked around this problem by defining their problematic annotations as strings. This worked because a string containing the type hint was just as usable for the static type analysis tool. And users of static type analysis tools rarely examine the annotations at runtime, so this representation wasn’t itself an inconvenience. But manually stringizing type hints was clumsy and error-prone. Also, code bases were adding more and more annotations, which consumed more and more CPU time to create and bind. To solve these problems, the BDFL accepted <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a>, which added a new feature to Python 3.7: “stringized annotations”. It was activated with a future import: <div class="highlight-default notranslate"><div class="highlight"><pre>from __future__ import annotations </pre></div> </div> Normally, annotation expressions were evaluated at the time the object was bound, with their values being stored in the annotations dict. When stringized annotations were active, these semantics changed: instead, at compile time, the compiler converted all annotations in that module into string representations of their source code–thus, automatically turning the users’s annotations into strings, obviating the need to manually stringize them as before. <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> suggested users could evaluate this string with <code class="docutils literal notranslate">eval</code> if the actual value was needed at runtime. (From here on out, this PEP will refer to the classic semantics of <a class="pep reference internal" href="../pep-3107/" title="PEP 3107 – Function Annotations">PEP 3107</a> and <a class="pep reference internal" href="../pep-0526/" title="PEP 526 – Syntax for Variable Annotations">PEP 526</a>, where the values of annotation expressions are computed at the time the object is bound, as “stock” semantics, to differentiate them from the new <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> “stringized” annotation semantics.) </section> <section id="the-current-state-of-annotation-use-cases"> <h3><a class="toc-backref" href="#the-current-state-of-annotation-use-cases" role="doc-backlink">The Current State Of Annotation Use Cases</a></h3> Although there are many specific use cases for annotations, annotation users in the discussion around this PEP tended to fall into one of these four categories. <section id="static-typing-users"> <h4><a class="toc-backref" href="#static-typing-users" role="doc-backlink">Static typing users</a></h4> Static typing users use annotations to add type information to their code. But they largely don’t examine the annotations at runtime. Instead, they use static type analysis tools (mypy, pytype) to examine their source tree and determine whether or not their code is using types consistently. This is almost certainly the most popular use case for annotations today. Many of the annotations use type hints, a la <a class="pep reference internal" href="../pep-0484/" title="PEP 484 – Type Hints">PEP 484</a> (and many subsequent PEPs). Type hints are passive objects, mere representation of type information; they don’t do any actual work. Type hints are often parameterized with other types or other type hints. Since they’re agnostic about what these actual values are, type hints work fine with <code class="docutils literal notranslate">ForwardRef</code> proxy objects. Users of static type hints discovered that extensive type hinting under stock semantics often created large-scale circular reference and circular import problems that could be difficult to solve. <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> was designed specifically to solve this problem, and the solution worked great for these users. The difficulty of rendering stringized annotations into real values largely didn’t inconvenience these users because of how infrequently they examine annotations at runtime. Static typing users often combine <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> with the <code class="docutils literal notranslate">if typing.TYPE_CHECKING</code> idiom to prevent their type hints from being loaded at runtime. This means they often aren’t able to evaluate their stringized annotations and produce real values at runtime. On the rare occasion that they do examine annotations at runtime, they often forgo <code class="docutils literal notranslate">eval</code>, instead using lexical analysis directly on the stringized annotations. Under this PEP, static typing users will probably prefer <code class="docutils literal notranslate">FORWARDREF</code> or <code class="docutils literal notranslate">SOURCE</code> format. </section> <section id="runtime-annotation-users"> <h4><a class="toc-backref" href="#runtime-annotation-users" role="doc-backlink">Runtime annotation users</a></h4> Runtime annotation users use annotations as a means of expressing rich metadata about their functions and classes, which they use as input to runtime behavior. Specific use cases include runtime type verification (Pydantic) and glue logic to expose Python APIs in another domain (FastAPI, Typer). The annotations may or may not be type hints. As runtime annotation users examine annotations at runtime, they were traditionally better served with stock semantics. This use case is largely incompatible with <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a>, particularly with the <code class="docutils literal notranslate">if typing.TYPE_CHECKING</code> idiom. Under this PEP, runtime annotation users will most likely prefer <code class="docutils literal notranslate">VALUE</code> format, though some (e.g. if they evaluate annotations eagerly in a decorator and want to support forward references) may also use <code class="docutils literal notranslate">FORWARDREF</code> format. </section> <section id="wrappers"> <h4><a class="toc-backref" href="#wrappers" role="doc-backlink">Wrappers</a></h4> Wrappers are functions or classes that wrap user functions or classes and add functionality. Examples of this would be <a class="reference external" href="https://docs.python.org/3/library/dataclasses.html#dataclasses.dataclass" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">dataclass()</code></a>, <a class="reference external" href="https://docs.python.org/3/library/functools.html#functools.partial" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">functools.partial()</code></a>, <code class="docutils literal notranslate">attrs</code>, and <code class="docutils literal notranslate">wrapt</code>. Wrappers are a distinct subcategory of runtime annotation users. Although they do use annotations at runtime, they may or may not actually examine the annotations of the objects they wrap–it depends on the functionality the wrapper provides. As a rule they should propagate the annotations of the wrapped object to the wrapper they create, although it’s possible they may modify those annotations. Wrappers were generally designed to work well under stock semantics. Whether or not they work well under <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> semantics depends on the degree to which they examine the wrapped object’s annotations. Often wrappers don’t care about the value per se, only needing specific information about the annotations. Even so, <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> and the <code class="docutils literal notranslate">if typing.TYPE_CHECKING</code> idiom can make it difficult for wrappers to reliably determine the information they need at runtime. This is an ongoing, chronic problem. Under this PEP, wrappers will probably prefer <code class="docutils literal notranslate">FORWARDREF</code> format for their internal logic. But the wrapped objects need to support all formats for their users. </section> <section id="documentation"> <h4><a class="toc-backref" href="#documentation" role="doc-backlink">Documentation</a></h4> <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> stringized annotations were a boon for tools that mechanically construct documentation. Stringized type hints make for excellent documentation; type hints as expressed in source code are often succinct and readable. However, at runtime these same type hints can produce value at runtime whose repr is a sprawling, nested, unreadable mess. Thus documentation users were well-served by <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> but poorly served with stock semantics. Under this PEP, documentation users are expected to use <code class="docutils literal notranslate">SOURCE</code> format. </section> </section> <section id="motivation-for-this-pep"> <h3><a class="toc-backref" href="#motivation-for-this-pep" role="doc-backlink">Motivation For This PEP</a></h3> Python’s original semantics for annotations made its use for static type analysis painful due to forward reference problems. <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> solved the forward reference problem, and many static type analysis users became happy early adopters of it. But its unconventional solution created new problems for two of the above cited use cases: runtime annotation users, and wrappers. First, stringized annotations didn’t permit referencing local or free variables, which meant many useful, reasonable approaches to creating annotations were no longer viable. This was particularly inconvenient for decorators that wrap existing functions and classes, as these decorators often use closures. Second, in order for <code class="docutils literal notranslate">eval</code> to correctly look up globals in a stringized annotation, you must first obtain a reference to the correct module. But class objects don’t retain a reference to their globals. <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> suggests looking up a class’s module by name in <code class="docutils literal notranslate">sys.modules</code>—a surprising requirement for a language-level feature. Additionally, complex but legitimate constructions can make it difficult to determine the correct globals and locals dicts to give to <code class="docutils literal notranslate">eval</code> to properly evaluate a stringized annotation. Even worse, in some situations it may simply be infeasible. For example, some libraries (e.g. <code class="docutils literal notranslate">typing.TypedDict</code>, <a class="reference external" href="https://docs.python.org/3/library/dataclasses.html#module-dataclasses" title="(in Python v3.13)"><code class="xref py py-mod docutils literal notranslate">dataclasses</code></a>) wrap a user class, then merge all the annotations from all that class’s base classes together into one cumulative annotations dict. If those annotations were stringized, calling <code class="docutils literal notranslate">eval</code> on them later may not work properly, because the globals dictionary used for the <code class="docutils literal notranslate">eval</code> will be the module where the user class was defined, which may not be the same module where the annotation was defined. However, if the annotations were stringized because of forward-reference problems, calling <code class="docutils literal notranslate">eval</code> on them early may not work either, due to the forward reference not being resolvable yet. This has proved to be difficult to reconcile; of the three bug reports linked to below, only one has been marked as fixed. <ul class="simple"> <li><a class="reference external" href="https://github.com/python/cpython/issues/89687">https://github.com/python/cpython/issues/89687</a></li> <li><a class="reference external" href="https://github.com/python/cpython/issues/85421">https://github.com/python/cpython/issues/85421</a></li> <li><a class="reference external" href="https://github.com/python/cpython/issues/90531">https://github.com/python/cpython/issues/90531</a></li> </ul> Even with proper globals and locals, <code class="docutils literal notranslate">eval</code> can be unreliable on stringized annotations. <code class="docutils literal notranslate">eval</code> can only succeed if all the symbols referenced in an annotations are defined. If a stringized annotation refers to a mixture of defined and undefined symbols, a simple <code class="docutils literal notranslate">eval</code> of that string will fail. This is a problem for libraries with that need to examine the annotation, because they can’t reliably convert these stringized annotations into real values. <ul class="simple"> <li>Some libraries (e.g. <a class="reference external" href="https://docs.python.org/3/library/dataclasses.html#module-dataclasses" title="(in Python v3.13)"><code class="xref py py-mod docutils literal notranslate">dataclasses</code></a>) solved this by foregoing real values and performing lexical analysis of the stringized annotation, which requires a lot of work to get right.</li> <li>Other libraries still suffer with this problem, which can produce surprising runtime behavior. <a class="reference external" href="https://github.com/python/cpython/issues/97727">https://github.com/python/cpython/issues/97727</a></li> </ul> Also, <code class="docutils literal notranslate">eval()</code> is slow, and it isn’t always available; it’s sometimes removed for space reasons on certain platforms. <code class="docutils literal notranslate">eval()</code> on MicroPython doesn’t support the <code class="docutils literal notranslate">locals</code> argument, which makes converting stringized annotations into real values at runtime even harder. Finally, <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> requires Python implementations to stringize their annotations. This is surprising behavior—unprecedented for a language-level feature, with a complicated implementation, that must be updated whenever a new operator is added to the language. These problems motivated the research into finding a new approach to solve the problems facing annotations users, resulting in this PEP. </section> </section> <section id="implementation"> <h2><a class="toc-backref" href="#implementation" role="doc-backlink">Implementation</a></h2> <section id="observed-semantics-for-annotations-expressions"> <h3><a class="toc-backref" href="#observed-semantics-for-annotations-expressions" role="doc-backlink">Observed semantics for annotations expressions</a></h3> For any object <code class="docutils literal notranslate">o</code> that supports annotations, provided that all names evaluated in the annotations expressions are bound before <code class="docutils literal notranslate">o</code> is defined and never subsequently rebound, <code class="docutils literal notranslate">o.__annotations__</code> will produce an identical annotations dict both when “stock” semantics are active and when this PEP is active. In particular, name resolution will be performed identically in both scenarios. When this PEP is active, the value of <code class="docutils literal notranslate">o.__annotations__</code> won’t be calculated until the first time <code class="docutils literal notranslate">o.__annotations__</code> itself is evaluated. All evaluation of the annotation expressions is delayed until this moment, which also means that <ul class="simple"> <li>names referenced in the annotations expressions will use their current value at this moment, and</li> <li>if evaluating the annotations expressions raises an exception, that exception will be raised at this moment.</li> </ul> Once <code class="docutils literal notranslate">o.__annotations__</code> is successfully calculated for the first time, this value is cached and will be returned by future requests for <code class="docutils literal notranslate">o.__annotations__</code>. </section> <section id="annotate-and-annotations"> <h3><a class="toc-backref" href="#annotate-and-annotations" role="doc-backlink">__annotate__ and __annotations__</a></h3> Python supports annotations on three different types: functions, classes, and modules. This PEP modifies the semantics on all three of these types in a similar way. First, this PEP adds a new “dunder” attribute, <code class="docutils literal notranslate">__annotate__</code>. <code class="docutils literal notranslate">__annotate__</code> must be a “data descriptor”, implementing all three actions: get, set, and delete. The <code class="docutils literal notranslate">__annotate__</code> attribute is always defined, and may only be set to either <code class="docutils literal notranslate">None</code> or to a callable. (<code class="docutils literal notranslate">__annotate__</code> cannot be deleted.) If an object has no annotations, <code class="docutils literal notranslate">__annotate__</code> should be initialized to <code class="docutils literal notranslate">None</code>, rather than to a function that returns an empty dict. The <code class="docutils literal notranslate">__annotate__</code> data descriptor must have dedicated storage inside the object to store the reference to its value. The location of this storage at runtime is an implementation detail. Even if it’s visible to Python code, it should still be considered an internal implementation detail, and Python code should prefer to interact with it only via the <code class="docutils literal notranslate">__annotate__</code> attribute. The callable stored in <code class="docutils literal notranslate">__annotate__</code> must accept a single required positional argument called <code class="docutils literal notranslate">format</code>, which will always be an <code class="docutils literal notranslate">int</code> (or a subclass of <code class="docutils literal notranslate">int</code>). It must either return a dict (or subclass of dict) or raise <code class="docutils literal notranslate">NotImplementedError()</code>. Here’s a formal definition of <code class="docutils literal notranslate">__annotate__</code>, as it will appear in the “Magic methods” section of the Python Language Reference: <blockquote> <div><code class="docutils literal notranslate">__annotate__(format: int) -> dict</code>Returns a new dictionary object mapping attribute/parameter names to their annotation values. Takes a <code class="docutils literal notranslate">format</code> parameter specifying the format in which annotations values should be provided. Must be one of the following: <code class="docutils literal notranslate">inspect.VALUE</code> (equivalent to the <code class="docutils literal notranslate">int</code> constant <code class="docutils literal notranslate">1</code>) <blockquote> <div>Values are the result of evaluating the annotation expressions.</div></blockquote> <code class="docutils literal notranslate">inspect.FORWARDREF</code> (equivalent to the <code class="docutils literal notranslate">int</code> constant <code class="docutils literal notranslate">2</code>) <blockquote> <div>Values are real annotation values (as per <code class="docutils literal notranslate">inspect.VALUE</code> format) for defined values, and <code class="docutils literal notranslate">ForwardRef</code> proxies for undefined values. Real objects may be exposed to, or contain references to, <code class="docutils literal notranslate">ForwardRef</code> proxy objects.</div></blockquote> <code class="docutils literal notranslate">inspect.SOURCE</code> (equivalent to the <code class="docutils literal notranslate">int</code> constant <code class="docutils literal notranslate">3</code>) <blockquote> <div>Values are the text string of the annotation as it appears in the source code. May only be approximate; whitespace may be normalized, and constant values may be optimized. It’s possible the exact values of these strings could change in future version of Python.</div></blockquote> If an <code class="docutils literal notranslate">__annotate__</code> function doesn’t support the requested format, it must raise <code class="docutils literal notranslate">NotImplementedError()</code>. <code class="docutils literal notranslate">__annotate__</code> functions must always support <code class="docutils literal notranslate">1</code> (<code class="docutils literal notranslate">inspect.VALUE</code>) format; they must not raise <code class="docutils literal notranslate">NotImplementedError()</code> when called with <code class="docutils literal notranslate">format=1</code>. When called with <code class="docutils literal notranslate">format=1</code>, an <code class="docutils literal notranslate">__annotate__</code> function may raise <code class="docutils literal notranslate">NameError</code>; it must not raise <code class="docutils literal notranslate">NameError</code> when called requesting any other format. If an object doesn’t have any annotations, <code class="docutils literal notranslate">__annotate__</code> should preferably be set to <code class="docutils literal notranslate">None</code> (it can’t be deleted), rather than set to a function that returns an empty dict. </div></blockquote> When the Python compiler compiles an object with annotations, it simultaneously compiles the appropriate annotate function. This function, called with the single positional argument <code class="docutils literal notranslate">inspect.VALUE</code>, computes and returns the annotations dict as defined on that object. The Python compiler and runtime work in concert to ensure that the function is bound to the appropriate namespaces: <ul class="simple"> <li>For functions and classes, the globals dictionary will be the module where the object was defined. If the object is itself a module, its globals dictionary will be its own dict.</li> <li>For methods on classes, and for classes, the locals dictionary will be the class dictionary.</li> <li>If the annotations refer to free variables, the closure will be the appropriate closure tuple containing cells for free variables.</li> </ul> Second, this PEP requires that the existing <code class="docutils literal notranslate">__annotations__</code> must be a “data descriptor”, implementing all three actions: get, set, and delete. <code class="docutils literal notranslate">__annotations__</code> must also have its own internal storage it uses to cache a reference to the annotations dict: <ul class="simple"> <li>Class and module objects must cache the annotations dict in their <code class="docutils literal notranslate">__dict__</code>, using the key <code class="docutils literal notranslate">__annotations__</code>. This is required for backwards compatibility reasons.</li> <li>For function objects, storage for the annotations dict cache is an implementation detail. It’s preferably internal to the function object and not visible in Python.</li> </ul> This PEP defines semantics on how <code class="docutils literal notranslate">__annotations__</code> and <code class="docutils literal notranslate">__annotate__</code> interact, for all three types that implement them. In the following examples, <code class="docutils literal notranslate">fn</code> represents a function, <code class="docutils literal notranslate">cls</code> represents a class, <code class="docutils literal notranslate">mod</code> represents a module, and <code class="docutils literal notranslate">o</code> represents an object of any of these three types: <ul class="simple"> <li>When <code class="docutils literal notranslate">o.__annotations__</code> is evaluated, and the internal storage for <code class="docutils literal notranslate">o.__annotations__</code> is unset, and <code class="docutils literal notranslate">o.__annotate__</code> is set to a callable, the getter for <code class="docutils literal notranslate">o.__annotations__</code> calls <code class="docutils literal notranslate">o.__annotate__(1)</code>, then caches the result in its internal storage and returns the result.<ul> <li>To explicitly clarify one question that has come up multiple times: this <code class="docutils literal notranslate">o.__annotations__</code> cache is the only caching mechanism defined in this PEP. There are no other caching mechanisms defined in this PEP. The <code class="docutils literal notranslate">__annotate__</code> functions generated by the Python compiler explicitly don’t cache any of the values they compute.</li> </ul> </li> <li>Setting <code class="docutils literal notranslate">o.__annotate__</code> to a callable invalidates the cached annotations dict.</li> <li>Setting <code class="docutils literal notranslate">o.__annotate__</code> to <code class="docutils literal notranslate">None</code> has no effect on the cached annotations dict.</li> <li>Deleting <code class="docutils literal notranslate">o.__annotate__</code> raises <code class="docutils literal notranslate">TypeError</code>. <code class="docutils literal notranslate">__annotate__</code> must always be set; this prevents unannotated subclasses from inheriting the <code class="docutils literal notranslate">__annotate__</code> method of one of their base classes.</li> <li>Setting <code class="docutils literal notranslate">o.__annotations__</code> to a legal value automatically sets <code class="docutils literal notranslate">o.__annotate__</code> to <code class="docutils literal notranslate">None</code>.<ul> <li>Setting <code class="docutils literal notranslate">cls.__annotations__</code> or <code class="docutils literal notranslate">mod.__annotations__</code> to <code class="docutils literal notranslate">None</code> otherwise works like any other attribute; the attribute is set to <code class="docutils literal notranslate">None</code>.</li> <li>Setting <code class="docutils literal notranslate">fn.__annotations__</code> to <code class="docutils literal notranslate">None</code> invalidates the cached annotations dict. If <code class="docutils literal notranslate">fn.__annotations__</code> doesn’t have a cached annotations value, and <code class="docutils literal notranslate">fn.__annotate__</code> is <code class="docutils literal notranslate">None</code>, the <code class="docutils literal notranslate">fn.__annotations__</code> data descriptor creates, caches, and returns a new empty dict. (This is for backwards compatibility with <a class="pep reference internal" href="../pep-3107/" title="PEP 3107 – Function Annotations">PEP 3107</a> semantics.)</li> </ul> </li> </ul> </section> <section id="changes-to-allowable-annotations-syntax"> <h3><a class="toc-backref" href="#changes-to-allowable-annotations-syntax" role="doc-backlink">Changes to allowable annotations syntax</a></h3> <code class="docutils literal notranslate">__annotate__</code> now delays the evaluation of annotations until <code class="docutils literal notranslate">__annotations__</code> is referenced in the future. It also means annotations are evaluated in a new function, rather than in the original context where the object they were defined on was bound. There are four operators with significant runtime side-effects that were permitted in stock semantics, but are disallowed when <code class="docutils literal notranslate">from __future__ import annotations</code> is active, and will have to be disallowed when this PEP is active: <ul class="simple"> <li><code class="docutils literal notranslate">:=</code></li> <li><code class="docutils literal notranslate">yield</code></li> <li><code class="docutils literal notranslate">yield from</code></li> <li><code class="docutils literal notranslate">await</code></li> </ul> </section> <section id="changes-to-inspect-get-annotations-and-typing-get-type-hints"> <h3><a class="toc-backref" href="#changes-to-inspect-get-annotations-and-typing-get-type-hints" role="doc-backlink">Changes to <code class="docutils literal notranslate">inspect.get_annotations</code> and <code class="docutils literal notranslate">typing.get_type_hints</code></a></h3> (This PEP makes frequent reference to these two functions. In the future it will refer to them collectively as “the helper functions”, as they help user code work with annotations.) These two functions extract and return the annotations from an object. <code class="docutils literal notranslate">inspect.get_annotations</code> returns the annotations unchanged; for the convenience of static typing users, <code class="docutils literal notranslate">typing.get_type_hints</code> makes some modifications to the annotations before it returns them. This PEP adds a new keyword-only parameter to these two functions, <code class="docutils literal notranslate">format</code>. <code class="docutils literal notranslate">format</code> specifies what format the values in the annotations dict should be returned in. The <code class="docutils literal notranslate">format</code> parameter on these two functions accepts the same values as the <code class="docutils literal notranslate">format</code> parameter on the <code class="docutils literal notranslate">__annotate__</code> magic method defined above; however, these <code class="docutils literal notranslate">format</code> parameters also have a default value of <code class="docutils literal notranslate">inspect.VALUE</code>. When either <code class="docutils literal notranslate">__annotations__</code> or <code class="docutils literal notranslate">__annotate__</code> is updated on an object, the other of those two attributes is now out-of-date and should also either be updated or deleted (set to <code class="docutils literal notranslate">None</code>, in the case of <code class="docutils literal notranslate">__annotate__</code> which cannot be deleted). In general, the semantics established in the previous section ensure that this happens automatically. However, there’s one case which for all practical purposes can’t be handled automatically: when the dict cached by <code class="docutils literal notranslate">o.__annotations__</code> is itself modified, or when mutable values inside that dict are modified. Since this can’t be handled in code, it must be handled in documentation. This PEP proposes amending the documentation for <code class="docutils literal notranslate">inspect.get_annotations</code> (and similarly for <code class="docutils literal notranslate">typing.get_type_hints</code>) as follows: <blockquote> <div>If you directly modify the <code class="docutils literal notranslate">__annotations__</code> dict on an object, by default these changes may not be reflected in the dictionary returned by <code class="docutils literal notranslate">inspect.get_annotations</code> when requesting either <code class="docutils literal notranslate">SOURCE</code> or <code class="docutils literal notranslate">FORWARDREF</code> format on that object. Rather than modifying the <code class="docutils literal notranslate">__annotations__</code> dict directly, consider replacing that object’s <code class="docutils literal notranslate">__annotate__</code> method with a function computing the annotations dict with your desired values. Failing that, it’s best to overwrite the object’s <code class="docutils literal notranslate">__annotate__</code> method with <code class="docutils literal notranslate">None</code> to prevent <code class="docutils literal notranslate">inspect.get_annotations</code> from generating stale results for <code class="docutils literal notranslate">SOURCE</code> and <code class="docutils literal notranslate">FORWARDREF</code> formats.</div></blockquote> </section> <section id="the-stringizer-and-the-fake-globals-environment"> <h3><a class="toc-backref" href="#the-stringizer-and-the-fake-globals-environment" role="doc-backlink">The <code class="docutils literal notranslate">stringizer</code> and the <code class="docutils literal notranslate">fake globals</code> environment</a></h3> As originally proposed, this PEP supported many runtime annotation user use cases, and many static type user use cases. But this was insufficient–this PEP could not be accepted until it satisfied all extant use cases. This became a longtime blocker of this PEP until Carl Meyer proposed the “stringizer” and the “fake globals” environment as described below. These techniques allow this PEP to support both the <code class="docutils literal notranslate">FORWARDREF</code> and <code class="docutils literal notranslate">SOURCE</code> formats, ably satisfying all remaining uses cases. In a nutshell, this technique involves running a Python-compiler-generated <code class="docutils literal notranslate">__annotate__</code> function in an exotic runtime environment. Its normal <code class="docutils literal notranslate">globals</code> dict is replaced with what’s called a “fake globals” dict. A “fake globals” dict is a dict with one important difference: every time you “get” a key from it that isn’t mapped, it creates, caches, and returns a new value for that key (as per the <code class="docutils literal notranslate">__missing__</code> callback for a dictionary). That value is a an instance of a novel type referred to as a “stringizer”. A “stringizer” is a Python class with highly unusual behavior. Every stringizer is initialized with its “value”, initially the name of the missing key in the “fake globals” dict. The stringizer then implements every Python “dunder” method used to implement operators, and the value returned by that method is a new stringizer whose value is a text representation of that operation. When these stringizers are used in expressions, the result of the expression is a new stringizer whose name textually represents that expression. For example, let’s say you have a variable <code class="docutils literal notranslate">f</code>, which is a reference to a stringizer initialized with the value <code class="docutils literal notranslate">'f'</code>. Here are some examples of operations you could perform on <code class="docutils literal notranslate">f</code> and the values they would return: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> f Stringizer('f') >>> f + 3 Stringizer('f + 3') >> f["key"] Stringizer('f["key"]') </pre></div> </div> Bringing it all together: if we run a Python-generated <code class="docutils literal notranslate">__annotate__</code> function, but we replace its globals with a “fake globals” dict, all undefined symbols it references will be replaced with stringizer proxy objects representing those symbols, and any operations performed on those proxies will in turn result in proxies representing that expression. This allows <code class="docutils literal notranslate">__annotate__</code> to complete, and to return an annotations dict, with stringizer instances standing in for names and entire expressions that could not have otherwise been evaluated. In practice, the “stringizer” functionality will be implemented in the <code class="docutils literal notranslate">ForwardRef</code> object currently defined in the <code class="docutils literal notranslate">typing</code> module. <code class="docutils literal notranslate">ForwardRef</code> will be extended to implement all stringizer functionality; it will also be extended to support evaluating the string it contains, to produce the real value (assuming all symbols referenced are defined). This means the <code class="docutils literal notranslate">ForwardRef</code> object will retain references to the appropriate “globals”, “locals”, and even “closure” information needed to evaluate the expression. This technique is the core of how <code class="docutils literal notranslate">inspect.get_annotations</code> supports <code class="docutils literal notranslate">FORWARDREF</code> and <code class="docutils literal notranslate">SOURCE</code> formats. Initially, <code class="docutils literal notranslate">inspect.get_annotations</code> will call the object’s <code class="docutils literal notranslate">__annotate__</code> method requesting the desired format. If that raises <code class="docutils literal notranslate">NotImplementedError</code>, <code class="docutils literal notranslate">inspect.get_annotations</code> will construct a “fake globals” environment, then call the object’s <code class="docutils literal notranslate">__annotate__</code> method. <ul class="simple"> <li><code class="docutils literal notranslate">inspect.get_annotations</code> produces <code class="docutils literal notranslate">SOURCE</code> format by creating a new empty “fake globals” dict, binding it to the object’s <code class="docutils literal notranslate">__annotate__</code> method, calling that requesting <code class="docutils literal notranslate">VALUE</code> format, and then extracting the string “value” from each <code class="docutils literal notranslate">ForwardRef</code> object in the resulting dict.</li> <li><code class="docutils literal notranslate">inspect.get_annotations</code> produces <code class="docutils literal notranslate">FORWARDREF</code> format by creating a new empty “fake globals” dict, pre-populating it with the current contents of the <code class="docutils literal notranslate">__annotate__</code> method’s globals dict, binding the “fake globals” dict to the object’s <code class="docutils literal notranslate">__annotate__</code> method, calling that requesting <code class="docutils literal notranslate">VALUE</code> format, and returning the result.</li> </ul> This entire technique works because the <code class="docutils literal notranslate">__annotate__</code> functions generated by the compiler are controlled by Python itself, and are simple and predictable. They’re effectively a single <code class="docutils literal notranslate">return</code> statement, computing and returning the annotations dict. Since most operations needed to compute an annotation are implemented in Python using dunder methods, and the stringizer supports all the relevant dunder methods, this approach is a reliable, practical solution. However, it’s not reasonable to attempt this technique with just any <code class="docutils literal notranslate">__annotate__</code> method. This PEP assumes that third-party libraries may implement their own <code class="docutils literal notranslate">__annotate__</code> methods, and those functions would almost certainly work incorrectly when run in this “fake globals” environment. For that reason, this PEP allocates a flag on code objects, one of the unused bits in <code class="docutils literal notranslate">co_flags</code>, to mean “This code object can be run in a ‘fake globals’ environment.” This makes the “fake globals” environment strictly opt-in, and it’s expected that only <code class="docutils literal notranslate">__annotate__</code> methods generated by the Python compiler will set it. The weakness in this technique is in handling operators which don’t directly map to dunder methods on an object. These are all operators that implement some manner of flow control, either branching or iteration: <ul class="simple"> <li>Short-circuiting <code class="docutils literal notranslate">or</code></li> <li>Short-circuiting <code class="docutils literal notranslate">and</code></li> <li>Ternary operator (the <code class="docutils literal notranslate">if</code> / <code class="docutils literal notranslate">then</code> operator)</li> <li>Generator expressions</li> <li>List / dict / set comprehensions</li> <li>Iterable unpacking</li> </ul> As a rule these techniques aren’t used in annotations, so it doesn’t pose a problem in practice. However, the recent addition of <code class="docutils literal notranslate">TypeVarTuple</code> to Python does use iterable unpacking. The dunder methods involved (<code class="docutils literal notranslate">__iter__</code> and <code class="docutils literal notranslate">__next__</code>) don’t permit distinguishing between iteration use cases; in order to correctly detect which use case was involved, mere “fake globals” and a “stringizer” wouldn’t be sufficient; this would require a custom bytecode interpreter designed specifically around producing <code class="docutils literal notranslate">SOURCE</code> and <code class="docutils literal notranslate">FORWARDREF</code> formats. Thankfully there’s a shortcut that will work fine: the stringizer will simply assume that when its iteration dunder methods are called, it’s in service of iterator unpacking being performed by <code class="docutils literal notranslate">TypeVarTuple</code>. It will hard-code this behavior. This means no other technique using iteration will work, but in practice this won’t inconvenience real-world use cases. Finally, note that the “fake globals” environment will also require constructing a matching “fake locals” dictionary, which for <code class="docutils literal notranslate">FORWARDREF</code> format will be pre-populated with the relevant locals dict. The “fake globals” environment will also have to create a fake “closure”, a tuple of <code class="docutils literal notranslate">ForwardRef</code> objects pre-created with the names of the free variables referenced by the <code class="docutils literal notranslate">__annotate__</code> method. <code class="docutils literal notranslate">ForwardRef</code> proxies created from <code class="docutils literal notranslate">__annotate__</code> methods that reference free variables will map the names and closure values of those free variables into the locals dictionary, to ensure that <code class="docutils literal notranslate">eval</code> uses the correct values for those names. </section> <section id="compiler-generated-annotate-functions"> <h3><a class="toc-backref" href="#compiler-generated-annotate-functions" role="doc-backlink">Compiler-generated <code class="docutils literal notranslate">__annotate__</code> functions</a></h3> As mentioned in the previous section, the <code class="docutils literal notranslate">__annotate__</code> functions generated by the compiler are simple. They’re mainly a single <code class="docutils literal notranslate">return</code> statement, computing and returning the annotations dict. However, the protocol for <code class="docutils literal notranslate">inspect.get_annotations</code> to request either <code class="docutils literal notranslate">FORWARDREF</code> or <code class="docutils literal notranslate">SOURCE</code> format requires first asking the <code class="docutils literal notranslate">__annotate__</code> method to produce it. <code class="docutils literal notranslate">__annotate__</code> methods generated by the Python compiler won’t support either of these formats and will raise <code class="docutils literal notranslate">NotImplementedError()</code>. </section> <section id="third-party-annotate-functions"> <h3><a class="toc-backref" href="#third-party-annotate-functions" role="doc-backlink">Third-party <code class="docutils literal notranslate">__annotate__</code> functions</a></h3> Third-party classes and functions will likely need to implement their own <code class="docutils literal notranslate">__annotate__</code> methods, so that downstream users of those objects can take full advantage of annotations. In particular, wrappers will likely need to transform the annotation dicts produced by the wrapped object: adding, removing, or modifying the dictionary in some way. Most of the time, third-party code will implement their <code class="docutils literal notranslate">__annotate__</code> methods by calling <code class="docutils literal notranslate">inspect.get_annotations</code> on some existing upstream object. For example, wrappers will likely request the annotations dict for their wrapped object, in the format that was requested from them, then modify the returned annotations dict as appropriate and return that. This allows third-party code to leverage the “fake globals” technique without having to understand or participate in it. Third-party libraries that support both pre- and post-PEP-649 versions of Python will have to innovate their own best practices on how to support both. One sensible approach would be for their wrapper to always support <code class="docutils literal notranslate">__annotate__</code>, then call it requesting <code class="docutils literal notranslate">VALUE</code> format and store the result as the <code class="docutils literal notranslate">__annotations__</code> on their wrapper object. This would support pre-649 Python semantics, and be forward-compatible with post-649 semantics. </section> <section id="pseudocode"> <h3><a class="toc-backref" href="#pseudocode" role="doc-backlink">Pseudocode</a></h3> Here’s high-level pseudocode for <code class="docutils literal notranslate">inspect.get_annotations</code>: <div class="highlight-default notranslate"><div class="highlight"><pre>def get_annotations(o, format): if format == VALUE: return dict(o.__annotations__) if format == FORWARDREF: try: return dict(o.__annotations__) except NameError: pass if not hasattr(o.__annotate__): return {} c_a = o.__annotate__ try: return c_a(format) except NotImplementedError: if not can_be_called_with_fake_globals(c_a): return {} c_a_with_fake_globals = make_fake_globals_version(c_a, format) return c_a_with_fake_globals(VALUE) </pre></div> </div> Here’s what a Python compiler-generated <code class="docutils literal notranslate">__annotate__</code> method might look like if it was written in Python: <div class="highlight-default notranslate"><div class="highlight"><pre>def __annotate__(self, format): if format != 1: raise NotImplementedError() return { ... } </pre></div> </div> Here’s how a third-party wrapper class might implement <code class="docutils literal notranslate">__annotate__</code>. In this example, the wrapper works like <code class="docutils literal notranslate">functools.partial</code>, pre-binding one parameter of the wrapped callable, which for simplicity must be named <code class="docutils literal notranslate">arg</code>: <div class="highlight-default notranslate"><div class="highlight"><pre>def __annotate__(self, format): ann = inspect.get_annotations(self.wrapped_fn, format) if 'arg' in ann: del ann['arg'] return ann </pre></div> </div> </section> <section id="other-modifications-to-the-python-runtime"> <h3><a class="toc-backref" href="#other-modifications-to-the-python-runtime" role="doc-backlink">Other modifications to the Python runtime</a></h3> This PEP does not dictate exactly how it should be implemented; that is left up to the language implementation maintainers. However, the best implementation of this PEP may require adding additional information to existing Python objects, which is implicitly condoned by the acceptance of this PEP. For example, it may be necessary to add a <code class="docutils literal notranslate">__globals__</code> attribute to class objects, so that the <code class="docutils literal notranslate">__annotate__</code> function for that class can be lazily bound, only on demand. Also, <code class="docutils literal notranslate">__annotate__</code> functions defined on methods defined in a class may need to retain a reference to the class’s <code class="docutils literal notranslate">__dict__</code>, in order to correctly evaluate names bound in that class. It’s expected that the CPython implementation of this PEP will include both those new attributes. All such new information added to existing Python objects should be done with “dunder” attributes, as they will of course be implementation details. </section> <section id="interactive-repl-shell"> <h3><a class="toc-backref" href="#interactive-repl-shell" role="doc-backlink">Interactive REPL Shell</a></h3> The semantics established in this PEP also hold true when executing code in Python’s interactive REPL shell, except for module annotations in the interactive module (<code class="docutils literal notranslate">__main__</code>) itself. Since that module is never “finished”, there’s no specific point where we can compile the <code class="docutils literal notranslate">__annotate__</code> function. For the sake of simplicity, in this case we forego delayed evaluation. Module-level annotations in the REPL shell will continue to work exactly as they do with “stock semantics”, evaluating immediately and setting the result directly inside the <code class="docutils literal notranslate">__annotations__</code> dict. </section> <section id="annotations-on-local-variables-inside-functions"> <h3><a class="toc-backref" href="#annotations-on-local-variables-inside-functions" role="doc-backlink">Annotations On Local Variables Inside Functions</a></h3> Python supports syntax for local variable annotations inside functions. However, these annotations have no runtime effect–they’re discarded at compile-time. Therefore, this PEP doesn’t need to do anything to support them, the same as stock semantics and <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a>. </section> <section id="prototype"> <h3><a class="toc-backref" href="#prototype" role="doc-backlink">Prototype</a></h3> The original prototype implementation of this PEP can be found here: <a class="reference external" href="https://github.com/larryhastings/co_annotations/">https://github.com/larryhastings/co_annotations/</a> As of this writing, the implementation is severely out of date; it’s based on Python 3.10 and implements the semantics of the first draft of this PEP, from early 2021. It will be updated shortly. </section> <section id="performance-comparison"> <h3><a class="toc-backref" href="#performance-comparison" role="doc-backlink">Performance Comparison</a></h3> Performance with this PEP is generally favorable. There are four scenarios to consider: <ul class="simple"> <li>the runtime cost when annotations aren’t defined,</li> <li>the runtime cost when annotations are defined but not referenced, and</li> <li>the runtime cost when annotations are defined and referenced as objects.</li> <li>the runtime cost when annotations are defined and referenced as strings.</li> </ul> We’ll examine each of these scenarios in the context of all three semantics for annotations: stock, <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a>, and this PEP. When there are no annotations, all three semantics have the same runtime cost: zero. No annotations dict is created and no code is generated for it. This requires no runtime processor time and consumes no memory. When annotations are defined but not referenced, the runtime cost of Python with this PEP is roughly the same as <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a>, and improved over stock. The specifics depend on the object being annotated: <ul class="simple"> <li>With stock semantics, the annotations dict is always built, and set as an attribute of the object being annotated.</li> <li>In <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> semantics, for function objects, a precompiled constant (a specially constructed tuple) is set as an attribute of the function. For class and module objects, the annotations dict is always built and set as an attribute of the class or module.</li> <li>With this PEP, a single object is set as an attribute of the object being annotated. Most of the time, this object is a constant (a code object), but when the annotations require a class namespace or closure, this object will be a tuple constructed at binding time.</li> </ul> When annotations are both defined and referenced as objects, code using this PEP should be much faster than <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a>, and be as fast or faster than stock. <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> semantics requires invoking <code class="docutils literal notranslate">eval()</code> for every value inside an annotations dict which is enormously slow. And the implementation of this PEP generates measurably more efficient bytecode for class and module annotations than stock semantics; for function annotations, this PEP and stock semantics should be about the same speed. The one case where this PEP will be noticeably slower than <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> is when annotations are requested as strings; it’s hard to beat “they are already strings.” But stringified annotations are intended for online documentation use cases, where performance is less likely to be a key factor. Memory use should also be comparable in all three scenarios across all three semantic contexts. In the first and third scenarios, memory usage should be roughly equivalent in all cases. In the second scenario, when annotations are defined but not referenced, using this PEP’s semantics will mean the function/class/module will store one unused code object (possibly bound to an unused function object); with the other two semantics, they’ll store one unused dictionary or constant tuple. </section> </section> <section id="backwards-compatibility"> <h2><a class="toc-backref" href="#backwards-compatibility" role="doc-backlink">Backwards Compatibility</a></h2> <section id="backwards-compatibility-with-stock-semantics"> <h3><a class="toc-backref" href="#backwards-compatibility-with-stock-semantics" role="doc-backlink">Backwards Compatibility With Stock Semantics</a></h3> This PEP preserves nearly all existing behavior of annotations from stock semantics: <ul class="simple"> <li>The format of the annotations dict stored in the <code class="docutils literal notranslate">__annotations__</code> attribute is unchanged. Annotations dicts contain real values, not strings as per <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a>.</li> <li>Annotations dicts are mutable, and any changes to them are preserved.</li> <li>The <code class="docutils literal notranslate">__annotations__</code> attribute can be explicitly set, and any legal value set this way will be preserved.</li> <li>The <code class="docutils literal notranslate">__annotations__</code> attribute can be deleted using the <code class="docutils literal notranslate">del</code> statement.</li> </ul> Most code that works with stock semantics should continue to work when this PEP is active without any modification necessary. But there are exceptions, as follows. First, there’s a well-known idiom for accessing class annotations which may not work correctly when this PEP is active. The original implementation of class annotations had what can only be called a bug: if a class didn’t define any annotations of its own, but one of its base classes did define annotations, the class would “inherit” those annotations. This behavior was never desirable, so user code found a workaround: instead of accessing the annotations on the class directly via <code class="docutils literal notranslate">cls.__annotations__</code>, code would access the class’s annotations via its dict as in <code class="docutils literal notranslate">cls.__dict__.get("__annotations__", {})</code>. This idiom worked because classes stored their annotations in their <code class="docutils literal notranslate">__dict__</code>, and accessing them this way avoided the lookups in the base classes. The technique relied on implementation details of CPython, so it was never supported behavior–though it was necessary. However, when this PEP is active, a class may have annotations defined but hasn’t yet called <code class="docutils literal notranslate">__annotate__</code> and cached the result, in which case this approach would lead to mistakenly assuming the class didn’t have annotations. In any case, the bug was fixed as of Python 3.10, and the idiom should no longer be used. Also as of Python 3.10, there’s an <a class="reference external" href="https://docs.python.org/3/howto/annotations.html">Annotations HOWTO</a> that defines best practices for working with annotations; code that follows these guidelines will work correctly even when this PEP is active, because it suggests using different approaches to get annotations from class objects based on the Python version the code runs under. Since delaying the evaluation of annotations until they are introspected changes the semantics of the language, it’s observable from within the language. Therefore it’s possible to write code that behaves differently based on whether annotations are evaluated at binding time or at access time, e.g. <div class="highlight-default notranslate"><div class="highlight"><pre>mytype = str def foo(a:mytype): pass mytype = int print(foo.__annotations__['a']) </pre></div> </div> This will print <code class="docutils literal notranslate"><class 'str'></code> with stock semantics and <code class="docutils literal notranslate"><class 'int'></code> when this PEP is active. This is therefore a backwards-incompatible change. However, this example is poor programming style, so this change seems acceptable. There are two uncommon interactions possible with class and module annotations that work with stock semantics that would no longer work when this PEP was active. These two interactions would have to be prohibited. The good news is, neither is common, and neither is considered good practice. In fact, they’re rarely seen outside of Python’s own regression test suite. They are: <ul class="simple"> <li>Code that sets annotations on module or class attributes from inside any kind of flow control statement. It’s currently possible to set module and class attributes with annotations inside an <code class="docutils literal notranslate">if</code> or <code class="docutils literal notranslate">try</code> statement, and it works as one would expect. It’s untenable to support this behavior when this PEP is active.</li> <li>Code in module or class scope that references or modifies the local <code class="docutils literal notranslate">__annotations__</code> dict directly. Currently, when setting annotations on module or class attributes, the generated code simply creates a local <code class="docutils literal notranslate">__annotations__</code> dict, then adds mappings to it as needed. It’s possible for user code to directly modify this dict, though this doesn’t seem to be an intentional feature. Although it would be possible to support this after a fashion once this PEP was active, the semantics would likely be surprising and wouldn’t make anyone happy.</li> </ul> Note that these are both also pain points for static type checkers, and are unsupported by those tools. It seems reasonable to declare that both are at the very least unsupported, and their use results in undefined behavior. It might be worth making a small effort to explicitly prohibit them with compile-time checks. Finally, if this PEP is active, annotation values shouldn’t use the <code class="docutils literal notranslate">if / else</code> ternary operator. Although this will work correctly when accessing <code class="docutils literal notranslate">o.__annotations__</code> or requesting <code class="docutils literal notranslate">inspect.VALUE</code> from a helper function, the boolean expression may not compute correctly with <code class="docutils literal notranslate">inspect.FORWARDREF</code> when some names are defined, and would be far less correct with <code class="docutils literal notranslate">inspect.SOURCE</code>. </section> <section id="backwards-compatibility-with-pep-563-semantics"> <h3><a class="toc-backref" href="#backwards-compatibility-with-pep-563-semantics" role="doc-backlink">Backwards Compatibility With PEP 563 Semantics</a></h3> <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> changed the semantics of annotations. When its semantics are active, annotations must assume they will be evaluated in module-level or class-level scope. They may no longer refer directly to local variables in the current function or an enclosing function. This PEP removes that restriction, and annotations may refer any local variable. <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> requires using <code class="docutils literal notranslate">eval</code> (or a helper function like <code class="docutils literal notranslate">typing.get_type_hints</code> or <code class="docutils literal notranslate">inspect.get_annotations</code> that uses <code class="docutils literal notranslate">eval</code> for you) to convert stringized annotations into their “real” values. Existing code that activates stringized annotations, and calls <code class="docutils literal notranslate">eval()</code> directly to convert the strings back into real values, can simply remove the <code class="docutils literal notranslate">eval()</code> call. Existing code using a helper function would continue to work unchanged, though use of those functions may become optional. Static typing users often have modules that only contain inert type hint definitions–but no live code. These modules are only needed when running static type checking; they aren’t used at runtime. But under stock semantics, these modules have to be imported in order for the runtime to evaluate and compute the annotations. Meanwhile, these modules often caused circular import problems that could be difficult or even impossible to solve. <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> allowed users to solve these circular import problems by doing two things. First, they activated <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> in their modules, which meant annotations were constant strings, and didn’t require the real symbols to be defined in order for the annotations to be computable. Second, this permitted users to only import the problematic modules in an <code class="docutils literal notranslate">if typing.TYPE_CHECKING</code> block. This allowed the static type checkers to import the modules and the type definitions inside, but they wouldn’t be imported at runtime. So far, this approach will work unchanged when this PEP is active; <code class="docutils literal notranslate">if typing.TYPE_CHECKING</code> is supported behavior. However, some codebases actually did examine their annotations at runtime, even when using the <code class="docutils literal notranslate">if typing.TYPE_CHECKING</code> technique and not importing definitions used in their annotations. These codebases examined the annotation strings without evaluating them, instead relying on identity checks or simple lexical analysis on the strings. This PEP supports these techniques too. But users will need to port their code to it. First, user code will need to use <code class="docutils literal notranslate">inspect.get_annotations</code> or <code class="docutils literal notranslate">typing.get_type_hints</code> to access the annotations; they won’t be able to simply get the <code class="docutils literal notranslate">__annotations__</code> attribute from their object. Second, they will need to specify either <code class="docutils literal notranslate">inspect.FORWARDREF</code> or <code class="docutils literal notranslate">inspect.SOURCE</code> for the <code class="docutils literal notranslate">format</code> when calling that function. This means the helper function can succeed in producing the annotations dict, even when not all the symbols are defined. Code expecting stringized annotations should work unmodified with <code class="docutils literal notranslate">inspect.SOURCE</code> formatted annotations dicts; however, users should consider switching to <code class="docutils literal notranslate">inspect.FORWARDREF</code>, as it may make their analysis easier. Similarly, <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> permitted use of class decorators on annotated classes in a way that hadn’t previously been possible. Some class decorators (e.g. <a class="reference external" href="https://docs.python.org/3/library/dataclasses.html#module-dataclasses" title="(in Python v3.13)"><code class="xref py py-mod docutils literal notranslate">dataclasses</code></a>) examine the annotations on the class. Because class decorators using the <code class="docutils literal notranslate">@</code> decorator syntax are run before the class name is bound, they can cause unsolvable circular-definition problems. If you annotate attributes of a class with references to the class itself, or annotate attributes in multiple classes with circular references to each other, you can’t decorate those classes with the <code class="docutils literal notranslate">@</code> decorator syntax using decorators that examine the annotations. <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> allowed this to work, as long as the decorators examined the strings lexically and didn’t use <code class="docutils literal notranslate">eval</code> to evaluate them (or handled the <code class="docutils literal notranslate">NameError</code> with further workarounds). When this PEP is active, decorators will be able to compute the annotations dict in <code class="docutils literal notranslate">inspect.SOURCE</code> or <code class="docutils literal notranslate">inspect.FORWARDREF</code> format using the helper functions. This will permit them to analyze annotations containing undefined symbols, in the format they prefer. Early adopters of <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> discovered that “stringized” annotations were useful for automatically-generated documentation. Users experimented with this use case, and Python’s <code class="docutils literal notranslate">pydoc</code> has expressed some interest in this technique. This PEP supports this use case; the code generating the documentation will have to be updated to use a helper function to access the annotations in <code class="docutils literal notranslate">inspect.SOURCE</code> format. Finally, the warnings about using the <code class="docutils literal notranslate">if / else</code> ternary operator in annotations apply equally to users of <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a>. It currently works for them, but could produce incorrect results when requesting some formats from the helper functions. If this PEP is accepted, <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> will be deprecated and eventually removed. To facilitate this transition for early adopters of <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a>, who now depend on its semantics, <code class="docutils literal notranslate">inspect.get_annotations</code> and <code class="docutils literal notranslate">typing.get_type_hints</code> will implement a special affordance. The Python compiler won’t generate annotation code objects for objects defined in a module where <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> semantics are active, even if this PEP is accepted. So, under normal circumstances, requesting <code class="docutils literal notranslate">inspect.SOURCE</code> format from a helper function would return an empty dict. As an affordance, to facilitate the transition, if the helper functions detect that an object was defined in a module with <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> active, and the user requests <code class="docutils literal notranslate">inspect.SOURCE</code> format, they’ll return the current value of the <code class="docutils literal notranslate">__annotations__</code> dict, which in this case will be the stringized annotations. This will allow <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> users who lexically analyze stringized annotations to immediately change over to requesting <code class="docutils literal notranslate">inspect.SOURCE</code> format from the helper functions, which will hopefully smooth their transition away from <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a>. </section> </section> <section id="rejected-ideas"> <h2><a class="toc-backref" href="#rejected-ideas" role="doc-backlink">Rejected Ideas</a></h2> <section id="just-store-the-strings"> <h3><a class="toc-backref" href="#just-store-the-strings" role="doc-backlink">“Just store the strings”</a></h3> One proposed idea for supporting <code class="docutils literal notranslate">SOURCE</code> format was for the Python compiler to emit the actual source code for the annotation values somewhere, and to furnish that when the user requested <code class="docutils literal notranslate">SOURCE</code> format. This idea wasn’t rejected so much as categorized as “not yet”. We already know we need to support <code class="docutils literal notranslate">FORWARDREF</code> format, and that technique can be adapted to support <code class="docutils literal notranslate">SOURCE</code> format in just a few lines. There are many unanswered questions about this approach: <ul class="simple"> <li>Where would we store the strings? Would they always be loaded when the annotated object was created, or would they be lazy-loaded on demand? If so, how would the lazy-loading work?</li> <li>Would the “source code” include the newlines and comments of the original? Would it preserve all whitespace, including indents and extra spaces used purely for formatting?</li> </ul> It’s possible we’ll revisit this topic in the future, if improving the fidelity of <code class="docutils literal notranslate">SOURCE</code> values to the original source code is judged sufficiently important. </section> </section> <section id="acknowledgements"> <h2><a class="toc-backref" href="#acknowledgements" role="doc-backlink">Acknowledgements</a></h2> Thanks to Carl Meyer, Barry Warsaw, Eric V. Smith, Mark Shannon, Jelle Ziljstra, and Guido van Rossum for ongoing feedback and encouragement. Particular thanks to several individuals who contributed key ideas that became some of the best aspects of this proposal: <ul class="simple"> <li>Carl Meyer suggested the “stringizer” technique that made <code class="docutils literal notranslate">FORWARDREF</code> and <code class="docutils literal notranslate">SOURCE</code> formats possible, which allowed making forward progress on this PEP possible after a year of languishing due to seemingly-unfixable problems. He also suggested the affordance for <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> users where <code class="docutils literal notranslate">inspect.SOURCE</code> will return the stringized annotations, and many more suggestions besides. Carl was also the primary correspondent in private email threads discussing this PEP, and was a tireless resource and voice of sanity. This PEP would almost certainly not have been accepted it were it not for Carl’s contributions.</li> <li>Mark Shannon suggested building the entire annotations dict inside a single code object, and only binding it to a function on demand.</li> <li>Guido van Rossum suggested that <code class="docutils literal notranslate">__annotate__</code> functions should duplicate the name visibility rules of annotations under “stock” semantics.</li> <li>Jelle Zijlstra contributed not only feedback–but code!</li> </ul> </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <ul class="simple"> <li><a class="reference external" href="https://github.com/larryhastings/co_annotations/issues">https://github.com/larryhastings/co_annotations/issues</a></li> <li><a class="reference external" href="https://discuss.python.org/t/two-polls-on-how-to-revise-pep-649/23628">https://discuss.python.org/t/two-polls-on-how-to-revise-pep-649/23628</a></li> <li><a class="reference external" href="https://discuss.python.org/t/a-massive-pep-649-update-with-some-major-course-corrections/25672">https://discuss.python.org/t/a-massive-pep-649-update-with-some-major-course-corrections/25672</a></li> </ul> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0649.rst">https://github.com/python/peps/blob/main/peps/pep-0649.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0649.rst">2025-02-01 08:55:40 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#overview">Overview</a><ul> <li><a class="reference internal" href="#comparison-of-annotation-semantics">Comparison Of Annotation Semantics</a></li> <li><a class="reference internal" href="#mistaken-rejection-of-this-approach-in-november-2017">Mistaken Rejection Of This Approach In November 2017</a></li> </ul> </li> <li><a class="reference internal" href="#motivation">Motivation</a><ul> <li><a class="reference internal" href="#a-history-of-annotations">A History Of Annotations</a></li> <li><a class="reference internal" href="#the-current-state-of-annotation-use-cases">The Current State Of Annotation Use Cases</a><ul> <li><a class="reference internal" href="#static-typing-users">Static typing users</a></li> <li><a class="reference internal" href="#runtime-annotation-users">Runtime annotation users</a></li> <li><a class="reference internal" href="#wrappers">Wrappers</a></li> <li><a class="reference internal" href="#documentation">Documentation</a></li> </ul> </li> <li><a class="reference internal" href="#motivation-for-this-pep">Motivation For This PEP</a></li> </ul> </li> <li><a class="reference internal" href="#implementation">Implementation</a><ul> <li><a class="reference internal" href="#observed-semantics-for-annotations-expressions">Observed semantics for annotations expressions</a></li> <li><a class="reference internal" href="#annotate-and-annotations">__annotate__ and __annotations__</a></li> <li><a class="reference internal" href="#changes-to-allowable-annotations-syntax">Changes to allowable annotations syntax</a></li> <li><a class="reference internal" href="#changes-to-inspect-get-annotations-and-typing-get-type-hints">Changes to <code class="docutils literal notranslate">inspect.get_annotations</code> and <code class="docutils literal notranslate">typing.get_type_hints</code></a></li> <li><a class="reference internal" href="#the-stringizer-and-the-fake-globals-environment">The <code class="docutils literal notranslate">stringizer</code> and the <code class="docutils literal notranslate">fake globals</code> environment</a></li> <li><a class="reference internal" href="#compiler-generated-annotate-functions">Compiler-generated <code class="docutils literal notranslate">__annotate__</code> functions</a></li> <li><a class="reference internal" href="#third-party-annotate-functions">Third-party <code class="docutils literal notranslate">__annotate__</code> functions</a></li> <li><a class="reference internal" href="#pseudocode">Pseudocode</a></li> <li><a class="reference internal" href="#other-modifications-to-the-python-runtime">Other modifications to the Python runtime</a></li> <li><a class="reference internal" href="#interactive-repl-shell">Interactive REPL Shell</a></li> <li><a class="reference internal" href="#annotations-on-local-variables-inside-functions">Annotations On Local Variables Inside Functions</a></li> <li><a class="reference internal" href="#prototype">Prototype</a></li> <li><a class="reference internal" href="#performance-comparison">Performance Comparison</a></li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a><ul> <li><a class="reference internal" href="#backwards-compatibility-with-stock-semantics">Backwards Compatibility With Stock Semantics</a></li> <li><a class="reference internal" href="#backwards-compatibility-with-pep-563-semantics">Backwards Compatibility With PEP 563 Semantics</a></li> </ul> </li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a><ul> <li><a class="reference internal" href="#just-store-the-strings">“Just store the strings”</a></li> </ul> </li> <li><a class="reference internal" href="#acknowledgements">Acknowledgements</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0649.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>