CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 690 – Lazy Imports | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0690/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 690 – Lazy Imports | peps.python.org'> <meta property="og:description" content="This PEP proposes a feature to transparently defer the finding and execution of imported modules until the moment when an imported object is first used. Since Python programs commonly import many more modules than a single invocation of the program is ..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0690/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This PEP proposes a feature to transparently defer the finding and execution of imported modules until the moment when an imported object is first used. Since Python programs commonly import many more modules than a single invocation of the program is ..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 690</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 690 – Lazy Imports</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Germán Méndez Bravo <german.mb at gmail.com>, Carl Meyer <carl at oddbird.net></dd> <dt class="field-even">Sponsor:</dt> <dd class="field-even">Barry Warsaw <barry at python.org></dd> <dt class="field-odd">Discussions-To:</dt> <dd class="field-odd"><a class="reference external" href="https://discuss.python.org/t/pep-690-lazy-imports/15474">Discourse thread</a></dd> <dt class="field-even">Status:</dt> <dd class="field-even"><abbr title="Formally declined and will not be accepted">Rejected</abbr></dd> <dt class="field-odd">Type:</dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created:</dt> <dd class="field-even">29-Apr-2022</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">3.12</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even"><a class="reference external" href="https://discuss.python.org/t/pep-690-lazy-imports/15474" title="Discourse thread">03-May-2022</a>, <a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/IHOSWMIBKCXVB46FI7NGOC2F34RUYZ5Z/" title="Python-Dev thread">03-May-2022</a></dd> <dt class="field-odd">Resolution:</dt> <dd class="field-odd"><a class="reference external" href="https://discuss.python.org/t/pep-690-lazy-imports-again/19661/26">Discourse message</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#example">Example</a></li> <li><a class="reference internal" href="#intended-usage">Intended usage</a></li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#debugging">Debugging</a></li> <li><a class="reference internal" href="#per-module-opt-out">Per-module opt-out</a></li> <li><a class="reference internal" href="#testing">Testing</a></li> <li><a class="reference internal" href="#c-api">C API</a></li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a><ul> <li><a class="reference internal" href="#import-side-effects">Import Side Effects</a></li> <li><a class="reference internal" href="#dynamic-paths">Dynamic Paths</a></li> <li><a class="reference internal" href="#deferred-exceptions">Deferred Exceptions</a></li> </ul> </li> <li><a class="reference internal" href="#drawbacks">Drawbacks</a></li> <li><a class="reference internal" href="#security-implications">Security Implications</a></li> <li><a class="reference internal" href="#performance-impact">Performance Impact</a></li> <li><a class="reference internal" href="#how-to-teach-this">How to Teach This</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a><ul> <li><a class="reference internal" href="#wrapping-deferred-exceptions">Wrapping deferred exceptions</a></li> <li><a class="reference internal" href="#per-module-opt-in">Per-module opt-in</a></li> <li><a class="reference internal" href="#explicit-syntax-for-individual-lazy-imports">Explicit syntax for individual lazy imports</a></li> <li><a class="reference internal" href="#environment-variable-to-enable-lazy-imports">Environment variable to enable lazy imports</a></li> <li><a class="reference internal" href="#removing-the-l-flag">Removing the <code class="docutils literal notranslate">-L</code> flag</a></li> <li><a class="reference internal" href="#half-lazy-imports">Half-lazy imports</a></li> <li><a class="reference internal" href="#lazy-dynamic-imports">Lazy dynamic imports</a></li> <li><a class="reference internal" href="#deep-eager-imports-override">Deep eager-imports override</a></li> <li><a class="reference internal" href="#making-lazy-imports-the-default-behavior">Making lazy imports the default behavior</a></li> </ul> </li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> This PEP proposes a feature to transparently defer the finding and execution of imported modules until the moment when an imported object is first used. Since Python programs commonly import many more modules than a single invocation of the program is likely to use in practice, lazy imports can greatly reduce the overall number of modules loaded, improving startup time and memory usage. Lazy imports also mostly eliminate the risk of import cycles. </section> <section id="motivation"> <h2><a class="toc-backref" href="#motivation" role="doc-backlink">Motivation</a></h2> Common Python code style <a class="pep reference internal" href="../pep-0008/#imports" title="PEP 8 – Style Guide for Python Code § Imports">prefers</a> imports at module level, so they don’t have to be repeated within each scope the imported object is used in, and to avoid the inefficiency of repeated execution of the import system at runtime. This means that importing the main module of a program typically results in an immediate cascade of imports of most or all of the modules that may ever be needed by the program. Consider the example of a Python command line program (CLI) with a number of subcommands. Each subcommand may perform different tasks, requiring the import of different dependencies. But a given invocation of the program will only execute a single subcommand, or possibly none (i.e. if just <code class="docutils literal notranslate">--help</code> usage info is requested). Top-level eager imports in such a program will result in the import of many modules that will never be used at all; the time spent (possibly compiling and) executing these modules is pure waste. To improve startup time, some large Python CLIs make imports lazy by manually placing imports inline into functions to delay imports of expensive subsystems. This manual approach is labor-intensive and fragile; one misplaced import or refactor can easily undo painstaking optimization work. The Python standard library already includes built-in support for lazy imports, via <a class="reference external" href="https://docs.python.org/3/library/importlib.html#importlib.util.LazyLoader">importlib.util.LazyLoader</a>. There are also third-party packages such as <a class="reference external" href="https://github.com/bwesterb/py-demandimport/">demandimport</a>. These provide a “lazy module object” which delays its own import until first attribute access. This is not sufficient to make all imports lazy: imports such as <code class="docutils literal notranslate">from foo import a, b</code> will still eagerly import the module <code class="docutils literal notranslate">foo</code> since they immediately access an attribute from it. It also imposes noticeable runtime overhead on every module attribute access, since it requires a Python-level <code class="docutils literal notranslate">__getattr__</code> or <code class="docutils literal notranslate">__getattribute__</code> implementation. Authors of scientific Python packages have also made extensive use of lazy imports to allow users to write e.g. <code class="docutils literal notranslate">import scipy as sp</code> and then easily access many different submodules with e.g. <code class="docutils literal notranslate">sp.linalg</code>, without requiring all the many submodules to be imported up-front. <a class="reference external" href="https://scientific-python.org/specs/spec-0001/">SPEC 1</a> codifies this practice in the form of a <code class="docutils literal notranslate">lazy_loader</code> library that can be used explicitly in a package <code class="docutils literal notranslate">__init__.py</code> to provide lazily accessible submodules. Users of static typing also have to import names for use in type annotations that may never be used at runtime (if <a class="pep reference internal" href="../pep-0563/" title="PEP 563 – Postponed Evaluation of Annotations">PEP 563</a> or possibly in future <a class="pep reference internal" href="../pep-0649/" title="PEP 649 – Deferred Evaluation Of Annotations Using Descriptors">PEP 649</a> are used to avoid eager runtime evaluation of annotations). Lazy imports are very attractive in this scenario to avoid overhead of unneeded imports. This PEP proposes a more general and comprehensive solution for lazy imports that can encompass all of the above use cases and does not impose detectable overhead in real-world use. The implementation in this PEP has already <a class="reference external" href="https://github.com/facebookincubator/cinder/blob/cinder/3.8/CinderDoc/lazy_imports.rst">demonstrated</a> startup time improvements up to 70% and memory-use reductions up to 40% on real-world Python CLIs. Lazy imports also eliminate most import cycles. With eager imports, “false cycles” can easily occur which are fixed by simply moving an import to the bottom of a module or inline into a function, or switching from <code class="docutils literal notranslate">from foo import bar</code> to <code class="docutils literal notranslate">import foo</code>. With lazy imports, these “cycles” just work. The only cycles which will remain are those where two modules actually each use a name from the other at module level; these “true” cycles are only fixable by refactoring the classes or functions involved. </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> The aim of this feature is to make imports transparently lazy. “Lazy” means that the import of a module (execution of the module body and addition of the module object to <code class="docutils literal notranslate">sys.modules</code>) should not occur until the module (or a name imported from it) is actually referenced during execution. “Transparent” means that besides the delayed import (and necessarily observable effects of that, such as delayed import side effects and changes to <code class="docutils literal notranslate">sys.modules</code>), there is no other observable change in behavior: the imported object is present in the module namespace as normal and is transparently loaded whenever first used: its status as a “lazy imported object” is not directly observable from Python or from C extension code. The requirement that the imported object be present in the module namespace as usual, even before the import has actually occurred, means that we need some kind of “lazy object” placeholder to represent the not-yet-imported object. The transparency requirement dictates that this placeholder must never be visible to Python code; any reference to it must trigger the import and replace it with the real imported object. Given the possibility that Python (or C extension) code may pull objects directly out of a module <code class="docutils literal notranslate">__dict__</code>, the only way to reliably prevent accidental leakage of lazy objects is to have the dictionary itself be responsible to ensure resolution of lazy objects on lookup. When a lookup finds that the key references a lazy object, it resolves the lazy object immediately before returning it. To avoid side effects mutating dictionaries midway through iteration, all lazy objects in a dictionary are resolved prior to starting an iteration; this could incur a performance penalty when using bulk iterations (<code class="docutils literal notranslate">iter(dict)</code>, <code class="docutils literal notranslate">reversed(dict)</code>, <code class="docutils literal notranslate">dict.__reversed__()</code>, <code class="docutils literal notranslate">dict.keys()</code>, <code class="docutils literal notranslate">iter(dict.keys())</code> and <code class="docutils literal notranslate">reversed(dict.keys())</code>). To avoid this performance penalty on the vast majority of dictionaries, which never contain any lazy objects, we steal a bit from the <code class="docutils literal notranslate">dk_kind</code> field for a new <code class="docutils literal notranslate">dk_lazy_imports</code> flag to keep track of whether a dictionary may contain lazy objects or not. This implementation comprehensively prevents leakage of lazy objects, ensuring they are always resolved to the real imported object before anyone can get hold of them for any use, while avoiding any significant performance impact on dictionaries in general. </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> Lazy imports are opt-in, and they can be globally enabled either via a new <code class="docutils literal notranslate">-L</code> flag to the Python interpreter, or via a call to a new <code class="docutils literal notranslate">importlib.set_lazy_imports()</code> function. This function takes two arguments, a boolean <code class="docutils literal notranslate">enabled</code> and an <code class="docutils literal notranslate">excluding</code> container. If <code class="docutils literal notranslate">enabled</code> is true, lazy imports will be turned on from that point forward. If it is false, they will be turned off from that point forward. (Use of the <code class="docutils literal notranslate">excluding</code> keyword is discussed below under “Per-module opt-out.”) When the flag <code class="docutils literal notranslate">-L</code> is passed to the Python interpreter, a new <code class="docutils literal notranslate">sys.flags.lazy_imports</code> is set to <code class="docutils literal notranslate">True</code>, otherwise it exists as <code class="docutils literal notranslate">False</code>. This flag is used to propagate <code class="docutils literal notranslate">-L</code> to new Python subprocesses. The flag in <code class="docutils literal notranslate">sys.flags.lazy_imports</code> does not necessarily reflect the current status of lazy imports, only whether the interpreter was started with the <code class="docutils literal notranslate">-L</code> option. Actual current status of whether lazy imports are enabled or not at any moment can be retrieved using <code class="docutils literal notranslate">importlib.is_lazy_imports_enabled()</code>, which will return <code class="docutils literal notranslate">True</code> if lazy imports are enabled at the call point or <code class="docutils literal notranslate">False</code> otherwise. When lazy imports are enabled, the loading and execution of all (and only) top-level imports is deferred until the imported name is first used. This could happen immediately (e.g. on the very next line after the import statement) or much later (e.g. while using the name inside a function being called by some other code at some later time.) For these top level imports, there are two contexts which will make them eager (not lazy): imports inside <code class="docutils literal notranslate">try</code> / <code class="docutils literal notranslate">except</code> / <code class="docutils literal notranslate">finally</code> or <code class="docutils literal notranslate">with</code> blocks, and star imports (<code class="docutils literal notranslate">from foo import *</code>.) Imports inside exception-handling blocks (this includes <code class="docutils literal notranslate">with</code> blocks, since those can also “catch” and handle exceptions) remain eager so that any exceptions arising from the import can be handled. Star imports must remain eager since performing the import is the only way to know which names should be added to the namespace. Imports inside class definitions or inside functions/methods are not “top level” and are never lazy. Dynamic imports using <code class="docutils literal notranslate">__import__()</code> or <code class="docutils literal notranslate">importlib.import_module()</code> are also never lazy. Lazy imports state (i.e. whether they have been enabled, and any excluded modules; see below) is per-interpreter, but global within the interpreter (i.e. all threads will be affected). <section id="example"> <h3><a class="toc-backref" href="#example" role="doc-backlink">Example</a></h3> Say we have a module <code class="docutils literal notranslate">spam.py</code>: <div class="highlight-default notranslate"><div class="highlight"><pre># simulate some work import time time.sleep(10) print("spam loaded") </pre></div> </div> And a module <code class="docutils literal notranslate">eggs.py</code> which imports it: <div class="highlight-default notranslate"><div class="highlight"><pre>import spam print("imports done") </pre></div> </div> If we run <code class="docutils literal notranslate">python -L eggs.py</code>, the <code class="docutils literal notranslate">spam</code> module will never be imported (because it is never referenced after the import), <code class="docutils literal notranslate">"spam loaded"</code> will never be printed, and there will be no 10 second delay. But if <code class="docutils literal notranslate">eggs.py</code> simply references the name <code class="docutils literal notranslate">spam</code> after importing it, that will be enough to trigger the import of <code class="docutils literal notranslate">spam.py</code>: <div class="highlight-default notranslate"><div class="highlight"><pre>import spam print("imports done") spam </pre></div> </div> Now if we run <code class="docutils literal notranslate">python -L eggs.py</code>, we will see the output <code class="docutils literal notranslate">"imports done"</code> printed first, then a 10 second delay, and then <code class="docutils literal notranslate">"spam loaded"</code> printed after that. Of course, in real use cases (especially with lazy imports), it’s not recommended to rely on import side effects like this to trigger real work. This example is just to clarify the behavior of lazy imports. Another way to explain the effect of lazy imports is that it is as if each lazy import statement had instead been written inline in the source code immediately before each use of the imported name. So one can think of lazy imports as similar to transforming this code: <div class="highlight-default notranslate"><div class="highlight"><pre>import foo def func1(): return foo.bar() def func2(): return foo.baz() </pre></div> </div> To this: <div class="highlight-default notranslate"><div class="highlight"><pre>def func1(): import foo return foo.bar() def func2(): import foo return foo.baz() </pre></div> </div> This gives a good sense of when the import of <code class="docutils literal notranslate">foo</code> will occur under lazy imports, but lazy import is not really equivalent to this code transformation. There are several notable differences: <ul class="simple"> <li>Unlike in the latter code, under lazy imports the name <code class="docutils literal notranslate">foo</code> still does exist in the module’s global namespace, and can be imported or referenced by other modules that import this one. (Such references would also trigger the import.)</li> <li>The runtime overhead of lazy imports is much lower than the latter code; after the first reference to the name <code class="docutils literal notranslate">foo</code> which triggers the import, subsequent references will have zero import system overhead; they are indistinguishable from a normal name reference.</li> </ul> In a sense, lazy imports turn the import statement into just a declaration of an imported name or names, to later be fully resolved when referenced. An import in the style <code class="docutils literal notranslate">from foo import bar</code> can also be made lazy. When the import occurs, the name <code class="docutils literal notranslate">bar</code> will be added to the module namespace as a lazy import. The first reference to <code class="docutils literal notranslate">bar</code> will import <code class="docutils literal notranslate">foo</code> and resolve <code class="docutils literal notranslate">bar</code> to <code class="docutils literal notranslate">foo.bar</code>. </section> <section id="intended-usage"> <h3><a class="toc-backref" href="#intended-usage" role="doc-backlink">Intended usage</a></h3> Since lazy imports are a potentially-breaking semantic change, they should be enabled only by the author or maintainer of a Python application, who is prepared to thoroughly test the application under the new semantics, ensure it behaves as expected, and opt-out any specific imports as needed (see below). Lazy imports should not be enabled speculatively by the end user of a Python application with any expectation of success. It is the responsibility of the application developer enabling lazy imports for their application to opt-out any library imports that turn out to need to be eager for their application to work correctly; it is not the responsibility of library authors to ensure that their library behaves exactly the same under lazy imports. The documentation of the feature, the <code class="docutils literal notranslate">-L</code> flag, and the new <code class="docutils literal notranslate">importlib</code> APIs will be clear about the intended usage and the risks of adoption without testing. </section> <section id="implementation"> <h3><a class="toc-backref" href="#implementation" role="doc-backlink">Implementation</a></h3> Lazy imports are represented internally by a “lazy import” object. When a lazy import occurs (say <code class="docutils literal notranslate">import foo</code> or <code class="docutils literal notranslate">from foo import bar</code>), the key <code class="docutils literal notranslate">"foo"</code> or <code class="docutils literal notranslate">"bar"</code> is immediately added to the module namespace dictionary, but with its value set to an internal-only “lazy import” object that preserves all the necessary metadata to execute the import later. A new boolean flag in <code class="docutils literal notranslate">PyDictKeysObject</code> (<code class="docutils literal notranslate">dk_lazy_imports</code>) is set to signal that this particular dictionary may contain lazy import objects. This flag is only used to efficiently resolve all lazy objects in “bulk” operations, when a dictionary may contain lazy objects. Anytime a key is looked up in a dictionary to extract its value, the value is checked to see if it is a lazy import object. If so, the lazy object is immediately resolved, the relevant imported modules executed, the lazy import object is replaced in the dictionary (if possible) by the actual imported value, and the resolved value is returned from the lookup function. A dictionary could mutate as part of an import side effect while resolving a lazy import object. In this case it is not possible to efficiently replace the key value with the resolved object. In this case, the lazy import object will gain a cached pointer to the resolved object. On next access that cached reference will be returned and the lazy import object will be replaced in the dict with the resolved value. Because this is all handled internally by the dictionary implementation, lazy import objects can never escape from the module namespace to become visible to Python code; they are always resolved at their first reference. No stub, dummy or thunk objects are ever visible to Python code or placed in <code class="docutils literal notranslate">sys.modules</code>. If a module is imported lazily, no entry for it will appear in <code class="docutils literal notranslate">sys.modules</code> at all until it is actually imported on first reference. If two different modules (<code class="docutils literal notranslate">moda</code> and <code class="docutils literal notranslate">modb</code>) both contain a lazy <code class="docutils literal notranslate">import foo</code>, each module’s namespace dictionary will have an independent lazy import object under the key <code class="docutils literal notranslate">"foo"</code>, delaying import of the same <code class="docutils literal notranslate">foo</code> module. This is not a problem. When there is first a reference to, say, <code class="docutils literal notranslate">moda.foo</code>, the module <code class="docutils literal notranslate">foo</code> will be imported and placed in <code class="docutils literal notranslate">sys.modules</code> as usual, and the lazy object under the key <code class="docutils literal notranslate">moda.__dict__["foo"]</code> will be replaced by the actual module <code class="docutils literal notranslate">foo</code>. At this point <code class="docutils literal notranslate">modb.__dict__["foo"]</code> will remain a lazy import object. When <code class="docutils literal notranslate">modb.foo</code> is later referenced, it will also try to <code class="docutils literal notranslate">import foo</code>. This import will find the module already present in <code class="docutils literal notranslate">sys.modules</code>, as is normal for subsequent imports of the same module in Python, and at this point will replace the lazy import object at <code class="docutils literal notranslate">modb.__dict__["foo"]</code> with the actual module <code class="docutils literal notranslate">foo</code>. There are two cases in which a lazy import object can escape a dictionary: <ul class="simple"> <li>Into another dictionary: to preserve the performance of bulk-copy operations like <code class="docutils literal notranslate">dict.update()</code> and <code class="docutils literal notranslate">dict.copy()</code>, they do not check for or resolve lazy import objects. However, if the source dict has the <code class="docutils literal notranslate">dk_lazy_imports</code> flag set that indicates it may contain lazy objects, that flag will be passed on to the updated/copied dictionary. This still ensures that the lazy import object can’t escape into Python code without being resolved.</li> <li>Through the garbage collector: lazy imported objects are still Python objects and live within the garbage collector; as such, they can be collected and seen via e.g. <code class="docutils literal notranslate">gc.get_objects()</code>. If a lazy object becomes visible to Python code in this way, it is opaque and inert; it has no useful methods or attributes. A <code class="docutils literal notranslate">repr()</code> of it would be shown as something like: <code class="docutils literal notranslate"><lazy_object 'fully.qualified.name'></code>.</li> </ul> When a lazy object is added to a dictionary, the flag <code class="docutils literal notranslate">dk_lazy_imports</code> is set. Once set, the flag is only cleared if all lazy import objects in the dictionary are resolved, e.g. prior to dictionary iteration. All dictionary iteration methods involving values (such as <code class="docutils literal notranslate">dict.items()</code>, <code class="docutils literal notranslate">dict.values()</code>, <code class="docutils literal notranslate">PyDict_Next()</code> etc.) will attempt to resolve all lazy import objects in the dictionary prior to starting the iteration. Since only (some) module namespace dictionaries will ever have <code class="docutils literal notranslate">dk_lazy_imports</code> set, the extra overhead of resolving all lazy import objects inside a dictionary is only paid by those dictionaries that need it. Minimizing the overhead on normal non-lazy dictionaries is the sole purpose of the <code class="docutils literal notranslate">dk_lazy_imports</code> flag. <code class="docutils literal notranslate">PyDict_Next</code> will attempt to resolve all lazy import objects the first time position <code class="docutils literal notranslate">0</code> is accessed, and those imports could fail with exceptions. Since <code class="docutils literal notranslate">PyDict_Next</code> cannot set an exception, <code class="docutils literal notranslate">PyDict_Next</code> will return <code class="docutils literal notranslate">0</code> immediately in this case, and any exception will be printed to stderr as an unraisable exception. For this reason, this PEP introduces <code class="docutils literal notranslate">PyDict_NextWithError</code>, which works in the same way as <code class="docutils literal notranslate">PyDict_Next</code>, but which can set an error when returning <code class="docutils literal notranslate">0</code> and this should be checked via <code class="docutils literal notranslate">PyErr_Occurred()</code> after the call. The eagerness of imports within <code class="docutils literal notranslate">try</code> / <code class="docutils literal notranslate">except</code> / <code class="docutils literal notranslate">with</code> blocks or within class or function bodies is handled in the compiler via a new <code class="docutils literal notranslate">EAGER_IMPORT_NAME</code> opcode that always imports eagerly. Top-level imports use <code class="docutils literal notranslate">IMPORT_NAME</code>, which may be lazy or eager depending on <code class="docutils literal notranslate">-L</code> and/or <code class="docutils literal notranslate">importlib.set_lazy_imports()</code>. </section> <section id="debugging"> <h3><a class="toc-backref" href="#debugging" role="doc-backlink">Debugging</a></h3> Debug logging from <code class="docutils literal notranslate">python -v</code> will include logging whenever an import statement has been encountered but execution of the import will be deferred. Python’s <code class="docutils literal notranslate">-X importtime</code> feature for profiling import costs adapts naturally to lazy imports; the profiled time is the time spent actually importing. Although lazy import objects are not generally visible to Python code, in some debugging cases it may be useful to check from Python code whether the value at a given key in a given dictionary is a lazy import object, without triggering its resolution. For this purpose, <code class="docutils literal notranslate">importlib.is_lazy_import()</code> can be used: <div class="highlight-default notranslate"><div class="highlight"><pre>from importlib import is_lazy_import import foo is_lazy_import(globals(), "foo") foo is_lazy_import(globals(), "foo") </pre></div> </div> In this example, if lazy imports have been enabled the first call to <code class="docutils literal notranslate">is_lazy_import</code> will return <code class="docutils literal notranslate">True</code> and the second will return <code class="docutils literal notranslate">False</code>. </section> <section id="per-module-opt-out"> <h3><a class="toc-backref" href="#per-module-opt-out" role="doc-backlink">Per-module opt-out</a></h3> Due to the backwards compatibility issues mentioned below, it may be necessary for an application using lazy imports to force some imports to be eager. In first-party code, since imports inside a <code class="docutils literal notranslate">try</code> or <code class="docutils literal notranslate">with</code> block are never lazy, this can be easily accomplished: <div class="highlight-default notranslate"><div class="highlight"><pre>try: # force these imports to be eager import foo import bar finally: pass </pre></div> </div> This PEP proposes to add a new <code class="docutils literal notranslate">importlib.eager_imports()</code> context manager, so the above technique can be less verbose and doesn’t require comments to clarify its intent: <div class="highlight-default notranslate"><div class="highlight"><pre>from importlib import eager_imports with eager_imports(): import foo import bar </pre></div> </div> Since imports within context managers are always eager, the <code class="docutils literal notranslate">eager_imports()</code> context manager can just be an alias to a null context manager. The context manager’s effect is not transitive: <code class="docutils literal notranslate">foo</code> and <code class="docutils literal notranslate">bar</code> will be imported eagerly, but imports within those modules will still follow the usual laziness rules. The more difficult case can occur if an import in third-party code that can’t easily be modified must be forced to be eager. For this purpose, <code class="docutils literal notranslate">importlib.set_lazy_imports()</code> takes a second optional keyword-only <code class="docutils literal notranslate">excluding</code> argument, which can be set to a container of module names within which all imports will be eager: <div class="highlight-default notranslate"><div class="highlight"><pre>from importlib import set_lazy_imports set_lazy_imports(excluding=["one.mod", "another"]) </pre></div> </div> The effect of this is also shallow: all imports within <code class="docutils literal notranslate">one.mod</code> will be eager, but not imports in all modules imported by <code class="docutils literal notranslate">one.mod</code>. The <code class="docutils literal notranslate">excluding</code> parameter of <code class="docutils literal notranslate">set_lazy_imports()</code> can be a container of any kind that will be checked to see whether it contains a module name. If the module name is contained in the object, imports within it will be eager. Thus, arbitrary opt-out logic can be encoded in a <code class="docutils literal notranslate">__contains__</code> method: <div class="highlight-default notranslate"><div class="highlight"><pre>import re from importlib import set_lazy_imports class Checker: def __contains__(self, name): return re.match(r"foo\.[^.]+\.logger", name) set_lazy_imports(excluding=Checker()) </pre></div> </div> If Python was executed with the <code class="docutils literal notranslate">-L</code> flag, then lazy imports will already be globally enabled, and the only effect of calling <code class="docutils literal notranslate">set_lazy_imports(True, excluding=...)</code> will be to globally set the eager module names/callback. If <code class="docutils literal notranslate">set_lazy_imports(True)</code> is called with no <code class="docutils literal notranslate">excluding</code> argument, the exclusion list/callback will be cleared and all eligible imports (module-level imports not in <code class="docutils literal notranslate">try/except/with</code>, and not <code class="docutils literal notranslate">import *</code>) will be lazy from that point forward. This opt-out system is designed to maintain the possibility of local reasoning about the laziness of an import. You only need to see the code of one module, and the <code class="docutils literal notranslate">excluding</code> argument to <code class="docutils literal notranslate">set_lazy_imports</code>, if any, to know whether a given import will be eager or lazy. </section> <section id="testing"> <h3><a class="toc-backref" href="#testing" role="doc-backlink">Testing</a></h3> The CPython test suite will pass with lazy imports enabled (with some tests skipped). One buildbot should run the test suite with lazy imports enabled. </section> <section id="c-api"> <h3><a class="toc-backref" href="#c-api" role="doc-backlink">C API</a></h3> For authors of C extension modules, the proposed public C API is as follows: <table class="docutils align-default"> <colgroup> <col style="width: 50.0%" /> <col style="width: 50.0%" /> </colgroup> <thead> <tr class="row-odd"><th class="head">C API</th> <th class="head">Python API</th> </tr> </thead> <tbody> <tr class="row-even"><td><code class="docutils literal notranslate">PyObject *PyImport_SetLazyImports(PyObject *enabled, PyObject *excluding)</code></td> <td><code class="docutils literal notranslate">importlib.set_lazy_imports(enabled: bool = True, *, excluding: typing.Container[str] | None = None)</code></td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">int PyDict_IsLazyImport(PyObject *dict, PyObject *name)</code></td> <td><code class="docutils literal notranslate">importlib.is_lazy_import(dict: typing.Dict[str, object], name: str) -> bool</code></td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">int PyImport_IsLazyImportsEnabled()</code></td> <td><code class="docutils literal notranslate">importlib.is_lazy_imports_enabled() -> bool</code></td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">void PyDict_ResolveLazyImports(PyObject *dict)</code></td> <td></td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">PyDict_NextWithError()</code></td> <td></td> </tr> </tbody> </table> <ul class="simple"> <li><code class="docutils literal notranslate">void PyDict_ResolveLazyImports(PyObject *dict)</code> resolves all lazy objects in a dictionary, if any. To be used prior calling <code class="docutils literal notranslate">PyDict_NextWithError()</code> or <code class="docutils literal notranslate">PyDict_Next()</code>.</li> <li><code class="docutils literal notranslate">PyDict_NextWithError()</code>, works the same way as <code class="docutils literal notranslate">PyDict_Next()</code>, with the exception it propagates any errors to the caller by returning <code class="docutils literal notranslate">0</code> and setting an exception. The caller should use <code class="docutils literal notranslate">PyErr_Occurred()</code> to check for any errors.</li> </ul> </section> </section> <section id="backwards-compatibility"> <h2><a class="toc-backref" href="#backwards-compatibility" role="doc-backlink">Backwards Compatibility</a></h2> This proposal preserves full backwards compatibility when the feature is disabled, which is the default. Even when enabled, most code will continue to work normally without any observable change (other than improved startup time and memory usage.) Namespace packages are not affected: they work just as they do currently, except lazily. In some existing code, lazy imports could produce currently unexpected results and behaviors. The problems that we may see when enabling lazy imports in an existing codebase are related to: <section id="import-side-effects"> <h3><a class="toc-backref" href="#import-side-effects" role="doc-backlink">Import Side Effects</a></h3> Import side effects that would otherwise be produced by the execution of imported modules during the execution of import statements will be deferred until the imported objects are used. These import side effects may include: <ul class="simple"> <li>code executing any side-effecting logic during import;</li> <li>relying on imported submodules being set as attributes in the parent module.</li> </ul> A relevant and typical affected case is the <a class="reference external" href="https://click.palletsprojects.com/">click</a> library for building Python command-line interfaces. If e.g. <code class="docutils literal notranslate">cli = click.group()</code> is defined in <code class="docutils literal notranslate">main.py</code>, and <code class="docutils literal notranslate">sub.py</code> imports <code class="docutils literal notranslate">cli</code> from <code class="docutils literal notranslate">main</code> and adds subcommands to it via decorator (<code class="docutils literal notranslate">@cli.command(...)</code>), but the actual <code class="docutils literal notranslate">cli()</code> call is in <code class="docutils literal notranslate">main.py</code>, then lazy imports may prevent the subcommands from being registered, since in this case Click is depending on side effects of the import of <code class="docutils literal notranslate">sub.py</code>. In this case the fix is to ensure the import of <code class="docutils literal notranslate">sub.py</code> is eager, e.g. by using the <code class="docutils literal notranslate">importlib.eager_imports()</code> context manager. </section> <section id="dynamic-paths"> <h3><a class="toc-backref" href="#dynamic-paths" role="doc-backlink">Dynamic Paths</a></h3> There could be issues related to dynamic Python import paths; particularly, adding (and then removing after the import) paths from <code class="docutils literal notranslate">sys.path</code>: <div class="highlight-default notranslate"><div class="highlight"><pre>sys.path.insert(0, "/path/to/foo/module") import foo del sys.path[0] foo.Bar() </pre></div> </div> In this case, with lazy imports enabled, the import of <code class="docutils literal notranslate">foo</code> will not actually occur while the addition to <code class="docutils literal notranslate">sys.path</code> is present. An easy fix for this (which also improves the code style and ensures cleanup) would be to place the <code class="docutils literal notranslate">sys.path</code> modifications in a context manager. This resolves the issue, since imports inside a <code class="docutils literal notranslate">with</code> block are always eager. </section> <section id="deferred-exceptions"> <h3><a class="toc-backref" href="#deferred-exceptions" role="doc-backlink">Deferred Exceptions</a></h3> Exceptions that occur during a lazy import bubble up and erase the partially-constructed module(s) from <code class="docutils literal notranslate">sys.modules</code>, just as exceptions during normal import do. Since errors raised during a lazy import will occur later than they would if the import were eager (i.e. wherever the name is first referenced), it is also possible that they could be accidentally caught by exception handlers that did not expect the import to be running within their <code class="docutils literal notranslate">try</code> block, leading to confusion. </section> </section> <section id="drawbacks"> <h2><a class="toc-backref" href="#drawbacks" role="doc-backlink">Drawbacks</a></h2> Downsides of this PEP include: <ul class="simple"> <li>It provides a subtly incompatible semantics for the behavior of Python imports. This is a potential burden on library authors who may be asked by their users to support both semantics, and is one more possibility for Python users/readers to be aware of.</li> <li>Some popular Python coding patterns (notably centralized registries populated by a decorator) rely on import side effects and may require explicit opt-out to work as expected with lazy imports.</li> <li>Exceptions can be raised at any point while accessing names representing lazy imports, this could lead to confusion and debugging of unexpected exceptions.</li> </ul> Lazy import semantics are already possible and even supported today in the Python standard library, so these drawbacks are not newly introduced by this PEP. So far, existing usage of lazy imports by some applications has not proven a problem. But this PEP could make the usage of lazy imports more popular, potentially exacerbating these drawbacks. These drawbacks must be weighed against the significant benefits offered by this PEP’s implementation of lazy imports. Ultimately these costs will be higher if the feature is widely used; but wide usage also indicates the feature provides a lot of value, perhaps justifying the costs. </section> <section id="security-implications"> <h2><a class="toc-backref" href="#security-implications" role="doc-backlink">Security Implications</a></h2> Deferred execution of code could produce security concerns if process owner, shell path, <code class="docutils literal notranslate">sys.path</code>, or other sensitive environment or contextual states change between the time the <code class="docutils literal notranslate">import</code> statement is executed and the time the imported object is first referenced. </section> <section id="performance-impact"> <h2><a class="toc-backref" href="#performance-impact" role="doc-backlink">Performance Impact</a></h2> The reference implementation has shown that the feature has negligible performance impact on existing real-world codebases (Instagram Server, several CLI programs at Meta, Jupyter notebooks used by Meta researchers), while providing substantial improvements to startup time and memory usage. The reference implementation shows <a class="reference external" href="https://gist.github.com/ericsnowcurrently/d027ff4130dedec3b58ab1f55be11e8c">no measurable change</a> in aggregate performance on the <a class="reference external" href="https://github.com/python/pyperformance">pyperformance benchmark suite</a>. </section> <section id="how-to-teach-this"> <h2><a class="toc-backref" href="#how-to-teach-this" role="doc-backlink">How to Teach This</a></h2> Since the feature is opt-in, beginners should not encounter it by default. Documentation of the <code class="docutils literal notranslate">-L</code> flag and <code class="docutils literal notranslate">importlib.set_lazy_imports()</code> can clarify the behavior of lazy imports. The documentation should also clarify that opting into lazy imports is opting into a non-standard semantics for Python imports, which could cause Python libraries to break in unexpected ways. The responsibility to identify these breakages and work around them with an opt-out (or stop using lazy imports) rests entirely with the person choosing to enable lazy imports for their application, not with the library author. Python libraries are under no obligation to support lazy import semantics. Politely reporting an incompatibility may be useful to the library author, but they may choose to simply say their library does not support use with lazy imports, and this is a valid choice. Some best practices to deal with some of the issues that could arise and to better take advantage of lazy imports are: <ul class="simple"> <li>Avoid relying on import side effects. Perhaps the most common reliance on import side effects is the registry pattern, where population of some external registry happens implicitly during the importing of modules, often via decorators. Instead, the registry should be built via an explicit call that does a discovery process to find decorated functions or classes in explicitly nominated modules.</li> <li>Always import needed submodules explicitly, don’t rely on some other import to ensure a module has its submodules as attributes. That is, unless there is an explicit <code class="docutils literal notranslate">from . import bar</code> in <code class="docutils literal notranslate">foo/__init__.py</code>, always do <code class="docutils literal notranslate">import foo.bar; foo.bar.Baz</code>, not <code class="docutils literal notranslate">import foo; foo.bar.Baz</code>. The latter only works (unreliably) because the attribute <code class="docutils literal notranslate">foo.bar</code> is added as a side effect of <code class="docutils literal notranslate">foo.bar</code> being imported somewhere else. With lazy imports this may not always happen in time.</li> <li>Avoid using star imports, as those are always eager.</li> </ul> </section> <section id="reference-implementation"> <h2><a class="toc-backref" href="#reference-implementation" role="doc-backlink">Reference Implementation</a></h2> The initial implementation is available as part of <a class="reference external" href="https://github.com/facebookincubator/cinder">Cinder</a>. This reference implementation is in use within Meta and has proven to achieve improvements in startup time (and total runtime for some applications) in the range of 40%-70%, as well as significant reduction in memory footprint (up to 40%), thanks to not needing to execute imports that end up being unused in the common flow. An <a class="reference external" href="https://github.com/Kronuz/cpython/pull/17">updated reference implementation based on CPython main branch</a> is also available. </section> <section id="rejected-ideas"> <h2><a class="toc-backref" href="#rejected-ideas" role="doc-backlink">Rejected Ideas</a></h2> <section id="wrapping-deferred-exceptions"> <h3><a class="toc-backref" href="#wrapping-deferred-exceptions" role="doc-backlink">Wrapping deferred exceptions</a></h3> To reduce the potential for confusion, exceptions raised in the course of executing a lazy import could be replaced by a <code class="docutils literal notranslate">LazyImportError</code> exception (a subclass of <code class="docutils literal notranslate">ImportError</code>), with a <code class="docutils literal notranslate">__cause__</code> set to the original exception. Ensuring that all lazy import errors are raised as <code class="docutils literal notranslate">LazyImportError</code> would reduce the likelihood that they would be accidentally caught and mistaken for a different expected exception. However, in practice we have seen cases, e.g. inside tests, where failing modules raise <code class="docutils literal notranslate">unittest.SkipTest</code> exception and this too would end up being wrapped in <code class="docutils literal notranslate">LazyImportError</code>, making such tests fail because the true exception type is hidden. The drawbacks here seem to outweigh the hypothetical case where unexpected deferred exceptions are caught by mistake. </section> <section id="per-module-opt-in"> <h3><a class="toc-backref" href="#per-module-opt-in" role="doc-backlink">Per-module opt-in</a></h3> A per-module opt-in using future imports (i.e. <code class="docutils literal notranslate">from __future__ import lazy_imports</code>) does not make sense because <code class="docutils literal notranslate">__future__</code> imports are not feature flags, they are for transition to behaviors which will become default in the future. It is not clear if lazy imports will ever make sense as the default behavior, so we should not promise this with a <code class="docutils literal notranslate">__future__</code> import. There are other cases where a library might desire to locally opt-in to lazy imports for a particular module; e.g. a lazy top-level <code class="docutils literal notranslate">__init__.py</code> for a large library, to make its subcomponents accessible as lazy attributes. For now, to keep the feature simpler, this PEP chooses to focus on the “application” use case and does not address the library use case. The underlying laziness mechanism introduced in this PEP could be used in the future to address this use case as well. </section> <section id="explicit-syntax-for-individual-lazy-imports"> <h3><a class="toc-backref" href="#explicit-syntax-for-individual-lazy-imports" role="doc-backlink">Explicit syntax for individual lazy imports</a></h3> If the primary objective of lazy imports were solely to work around import cycles and forward references, an explicitly-marked syntax for particular targeted imports to be lazy would make a lot of sense. But in practice it would be very hard to get robust startup time or memory use benefits from this approach, since it would require converting most imports within your code base (and in third-party dependencies) to use the lazy import syntax. It would be possible to aim for a “shallow” laziness where only the top-level imports of subsystems from the main module are made explicitly lazy, but then imports within the subsystems are all eager. This is extremely fragile, though – it only takes one mis-placed import to undo the carefully constructed shallow laziness. Globally enabling lazy imports, on the other hand, provides in-depth robust laziness where you always pay only for the imports you use. There may be use cases (e.g. for static typing) where individually-marked lazy imports are desirable to avoid forward references, but the perf/memory benefits of globally lazy imports are not needed. Since this is a different set of motivating use cases and requires new syntax, we prefer not to include it in this PEP. Another PEP could build on top of this implementation and propose the additional syntax. </section> <section id="environment-variable-to-enable-lazy-imports"> <h3><a class="toc-backref" href="#environment-variable-to-enable-lazy-imports" role="doc-backlink">Environment variable to enable lazy imports</a></h3> Providing an environment variable opt-in lends itself too easily to abuse of the feature. It may seem tempting for a Python user to, for instance, globally set the environment variable in their shell in the hopes of speeding up all the Python programs they run. This usage with untested programs is likely to lead to spurious bug reports and maintenance burden for the authors of those tools. To avoid this, we choose not to provide an environment variable opt-in at all. </section> <section id="removing-the-l-flag"> <h3><a class="toc-backref" href="#removing-the-l-flag" role="doc-backlink">Removing the <code class="docutils literal notranslate">-L</code> flag</a></h3> We do provide the <code class="docutils literal notranslate">-L</code> CLI flag, which could in theory be abused in a similar way by an end user running an individual Python program that is run with <code class="docutils literal notranslate">python somescript.py</code> or <code class="docutils literal notranslate">python -m somescript</code> (rather than distributed via Python packaging tools). But the potential scope for misuse is much less with <code class="docutils literal notranslate">-L</code> than an environment variable, and <code class="docutils literal notranslate">-L</code> is valuable for some applications to maximize startup time benefits by ensuring that all imports from the start of a process will be lazy, so we choose to keep it. It is already the case that running arbitrary Python programs with command line flags they weren’t intended to be used with (e.g. <code class="docutils literal notranslate">-s</code>, <code class="docutils literal notranslate">-S</code>, <code class="docutils literal notranslate">-E</code>, or <code class="docutils literal notranslate">-I</code>) can have unexpected and breaking results. <code class="docutils literal notranslate">-L</code> is nothing new in this regard. </section> <section id="half-lazy-imports"> <h3><a class="toc-backref" href="#half-lazy-imports" role="doc-backlink">Half-lazy imports</a></h3> It would be possible to eagerly run the import loader to the point of finding the module source, but then defer the actual execution of the module and creation of the module object. The advantage of this would be that certain classes of import errors (e.g. a simple typo in the module name) would be caught eagerly instead of being deferred to the use of an imported name. The disadvantage would be that the startup time benefits of lazy imports would be significantly reduced, since unused imports would still require a filesystem <code class="docutils literal notranslate">stat()</code> call, at least. It would also introduce a possibly non-obvious split between which import errors are raised eagerly and which are delayed, when lazy imports are enabled. This idea is rejected for now on the basis that in practice, confusion about import typos has not been an observed problem with the reference implementation. Generally delayed imports are not delayed forever, and errors show up soon enough to be caught and fixed (unless the import is truly unused.) Another possible motivation for half-lazy imports would be to allow modules themselves to control via some flag whether they are imported lazily or eagerly. This is rejected both on the basis that it requires half-lazy imports, giving up some of the performance benefits of import laziness, and because in general modules do not decide how or when they are imported, the module importing them decides that. There isn’t clear rationale for this PEP to invert that control; instead it just provides more options for the importing code to make the decision. </section> <section id="lazy-dynamic-imports"> <h3><a class="toc-backref" href="#lazy-dynamic-imports" role="doc-backlink">Lazy dynamic imports</a></h3> It would be possible to add a <code class="docutils literal notranslate">lazy=True</code> or similar option to <code class="docutils literal notranslate">__import__()</code> and/or <code class="docutils literal notranslate">importlib.import_module()</code>, to enable them to perform lazy imports. That idea is rejected in this PEP for lack of a clear use case. Dynamic imports are already far outside the <a class="pep reference internal" href="../pep-0008/" title="PEP 8 – Style Guide for Python Code">PEP 8</a> code style recommendations for imports, and can easily be made precisely as lazy as desired by placing them at the desired point in the code flow. These aren’t commonly used at module top level, which is where lazy imports applies. </section> <section id="deep-eager-imports-override"> <h3><a class="toc-backref" href="#deep-eager-imports-override" role="doc-backlink">Deep eager-imports override</a></h3> The proposed <code class="docutils literal notranslate">importlib.eager_imports()</code> context manager and excluded modules in the <code class="docutils literal notranslate">importlib.set_lazy_imports(excluding=...)</code> override all have shallow effects: they only force eagerness for the location they are applied to, not transitively. It would be possible to provide a deep/transitive version of one or both. That idea is rejected in this PEP because the implementation would be complex (taking into account threads and async code), experience with the reference implementation has not shown it to be necessary, and because it prevents local reasoning about laziness of imports. A deep override can lead to confusing behavior because the transitively-imported modules may be imported from multiple locations, some of which use the “deep eager override” and some of which don’t. Thus those modules may still be imported lazily initially, if they are first imported from a location that doesn’t have the override. With deep overrides it is not possible to locally reason about whether a given import will be lazy or eager. With the behavior specified in this PEP, such local reasoning is possible. </section> <section id="making-lazy-imports-the-default-behavior"> <h3><a class="toc-backref" href="#making-lazy-imports-the-default-behavior" role="doc-backlink">Making lazy imports the default behavior</a></h3> Making lazy imports the default/sole behavior of Python imports, instead of opt-in, would have some long-term benefits, in that library authors would (eventually) no longer need to consider the possibility of both semantics. However, the backwards-incompatibilies are such that this could only be considered over a long time frame, with a <code class="docutils literal notranslate">__future__</code> import. It is not at all clear that lazy imports should become the default import semantics for Python. This PEP takes the position that the Python community needs more experience with lazy imports before considering making it the default behavior, so that is entirely left to a possible future PEP. </section> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0690.rst">https://github.com/python/peps/blob/main/peps/pep-0690.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0690.rst">2025-02-01 08:55:40 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#example">Example</a></li> <li><a class="reference internal" href="#intended-usage">Intended usage</a></li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#debugging">Debugging</a></li> <li><a class="reference internal" href="#per-module-opt-out">Per-module opt-out</a></li> <li><a class="reference internal" href="#testing">Testing</a></li> <li><a class="reference internal" href="#c-api">C API</a></li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a><ul> <li><a class="reference internal" href="#import-side-effects">Import Side Effects</a></li> <li><a class="reference internal" href="#dynamic-paths">Dynamic Paths</a></li> <li><a class="reference internal" href="#deferred-exceptions">Deferred Exceptions</a></li> </ul> </li> <li><a class="reference internal" href="#drawbacks">Drawbacks</a></li> <li><a class="reference internal" href="#security-implications">Security Implications</a></li> <li><a class="reference internal" href="#performance-impact">Performance Impact</a></li> <li><a class="reference internal" href="#how-to-teach-this">How to Teach This</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a><ul> <li><a class="reference internal" href="#wrapping-deferred-exceptions">Wrapping deferred exceptions</a></li> <li><a class="reference internal" href="#per-module-opt-in">Per-module opt-in</a></li> <li><a class="reference internal" href="#explicit-syntax-for-individual-lazy-imports">Explicit syntax for individual lazy imports</a></li> <li><a class="reference internal" href="#environment-variable-to-enable-lazy-imports">Environment variable to enable lazy imports</a></li> <li><a class="reference internal" href="#removing-the-l-flag">Removing the <code class="docutils literal notranslate">-L</code> flag</a></li> <li><a class="reference internal" href="#half-lazy-imports">Half-lazy imports</a></li> <li><a class="reference internal" href="#lazy-dynamic-imports">Lazy dynamic imports</a></li> <li><a class="reference internal" href="#deep-eager-imports-override">Deep eager-imports override</a></li> <li><a class="reference internal" href="#making-lazy-imports-the-default-behavior">Making lazy imports the default behavior</a></li> </ul> </li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0690.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>