CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 406 – Improved Encapsulation of Import State | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0406/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 406 – Improved Encapsulation of Import State | peps.python.org'> <meta property="og:description" content="This PEP proposes the introduction of a new ‘ImportEngine’ class as part of importlib which would encapsulate all state related to importing modules into a single object. Creating new instances of this object would then provide an alternative to complet..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0406/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This PEP proposes the introduction of a new ‘ImportEngine’ class as part of importlib which would encapsulate all state related to importing modules into a single object. Creating new instances of this object would then provide an alternative to complet..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 406</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> <span class="visually-hidden">Toggle light / dark / auto colour theme</span> </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 406 – Improved Encapsulation of Import State</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author<span class="colon">:</span></dt> <dd class="field-odd">Alyssa Coghlan <ncoghlan at gmail.com>, Greg Slodkowicz <jergosh at gmail.com></dd> <dt class="field-even">Status<span class="colon">:</span></dt> <dd class="field-even"><abbr title="Removed from consideration by sponsor or authors">Withdrawn</abbr></dd> <dt class="field-odd">Type<span class="colon">:</span></dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created<span class="colon">:</span></dt> <dd class="field-even">04-Jul-2011</dd> <dt class="field-odd">Python-Version<span class="colon">:</span></dt> <dd class="field-odd">3.4</dd> <dt class="field-even">Post-History<span class="colon">:</span></dt> <dd class="field-even">31-Jul-2011, 13-Nov-2011, 04-Dec-2011</dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#pep-withdrawal">PEP Withdrawal</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#proposal">Proposal</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#importengine-api">ImportEngine API</a></li> <li><a class="reference internal" href="#global-variables">Global variables</a></li> <li><a class="reference internal" href="#no-changes-to-finder-loader-interfaces">No changes to finder/loader interfaces</a></li> </ul> </li> <li><a class="reference internal" href="#open-issues">Open Issues</a><ul> <li><a class="reference internal" href="#api-design-for-falling-back-to-global-import-state">API design for falling back to global import state</a></li> <li><a class="reference internal" href="#builtin-and-extension-modules-must-be-process-global">Builtin and extension modules must be process global</a></li> <li><a class="reference internal" href="#scope-of-substitution">Scope of substitution</a></li> </ul> </li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> <p>This PEP proposes the introduction of a new ‘ImportEngine’ class as part of <code class="docutils literal notranslate"><span class="pre">importlib</span></code> which would encapsulate all state related to importing modules into a single object. Creating new instances of this object would then provide an alternative to completely replacing the built-in implementation of the import statement, by overriding the <code class="docutils literal notranslate"><span class="pre">__import__()</span></code> function. To work with the builtin import functionality and importing via import engine objects, this PEP proposes a context management based approach to temporarily replacing the global import state.</p> <p>The PEP also proposes inclusion of a <code class="docutils literal notranslate"><span class="pre">GlobalImportEngine</span></code> subclass and a globally accessible instance of that class, which “writes through” to the process global state. This provides a backwards compatible bridge between the proposed encapsulated API and the legacy process global state, and allows straightforward support for related state updates (e.g. selectively invalidating path cache entries when <code class="docutils literal notranslate"><span class="pre">sys.path</span></code> is modified).</p> </section> <section id="pep-withdrawal"> <h2><a class="toc-backref" href="#pep-withdrawal" role="doc-backlink">PEP Withdrawal</a></h2> <p>The import system has seen substantial changes since this PEP was originally written, as part of <a class="pep reference internal" href="../pep-0420/" title="PEP 420 – Implicit Namespace Packages">PEP 420</a> in Python 3.3 and <a class="pep reference internal" href="../pep-0451/" title="PEP 451 – A ModuleSpec Type for the Import System">PEP 451</a> in Python 3.4.</p> <p>While providing an encapsulation of the import state is still highly desirable, it is better tackled in a new PEP using <a class="pep reference internal" href="../pep-0451/" title="PEP 451 – A ModuleSpec Type for the Import System">PEP 451</a> as a foundation, and permitting only the use of <a class="pep reference internal" href="../pep-0451/" title="PEP 451 – A ModuleSpec Type for the Import System">PEP 451</a> compatible finders and loaders (as those avoid many of the issues of direct manipulation of global state associated with the previous loader API).</p> </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> <p>Currently, most state related to the import system is stored as module level attributes in the <code class="docutils literal notranslate"><span class="pre">sys</span></code> module. The one exception is the import lock, which is not accessible directly, but only via the related functions in the <code class="docutils literal notranslate"><span class="pre">imp</span></code> module. The current process global import state comprises:</p> <ul class="simple"> <li>sys.modules</li> <li>sys.path</li> <li>sys.path_hooks</li> <li>sys.meta_path</li> <li>sys.path_importer_cache</li> <li>the import lock (imp.lock_held()/acquire_lock()/release_lock())</li> </ul> <p>Isolating this state would allow multiple import states to be conveniently stored within a process. Placing the import functionality in a self-contained object would also allow subclassing to add additional features (e.g. module import notifications or fine-grained control over which modules can be imported). The engine would also be subclassed to make it possible to use the import engine API to interact with the existing process-global state.</p> <p>The namespace PEPs (especially <a class="pep reference internal" href="../pep-0402/" title="PEP 402 – Simplified Package Layout and Partitioning">PEP 402</a>) raise a potential need for <em>additional</em> process global state, in order to correctly update package paths as <code class="docutils literal notranslate"><span class="pre">sys.path</span></code> is modified.</p> <p>Finally, providing a coherent object for all this state makes it feasible to also provide context management features that allow the import state to be temporarily substituted.</p> </section> <section id="proposal"> <h2><a class="toc-backref" href="#proposal" role="doc-backlink">Proposal</a></h2> <p>We propose introducing an ImportEngine class to encapsulate import functionality. This includes an <code class="docutils literal notranslate"><span class="pre">__import__()</span></code> method which can be used as an alternative to the built-in <code class="docutils literal notranslate"><span class="pre">__import__()</span></code> when desired and also an <code class="docutils literal notranslate"><span class="pre">import_module()</span></code> method, equivalent to <code class="docutils literal notranslate"><span class="pre">importlib.import_module()</span></code> <a class="footnote-reference brackets" href="#id6" id="id1">[3]</a>.</p> <p>Since there are global import state invariants that are assumed and should be maintained, we introduce a <code class="docutils literal notranslate"><span class="pre">GlobalImportState</span></code> class with an interface identical to <code class="docutils literal notranslate"><span class="pre">ImportEngine</span></code> but directly accessing the current global import state. This can be easily implemented using class properties.</p> </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> <section id="importengine-api"> <h3><a class="toc-backref" href="#importengine-api" role="doc-backlink">ImportEngine API</a></h3> <p>The proposed extension consists of the following objects:</p> <p><code class="docutils literal notranslate"><span class="pre">importlib.engine.ImportEngine</span></code></p> <blockquote> <div><code class="docutils literal notranslate"><span class="pre">from_engine(self,</span> <span class="pre">other)</span></code><blockquote> <div>Create a new import object from another ImportEngine instance. The new object is initialised with a copy of the state in <code class="docutils literal notranslate"><span class="pre">other</span></code>. When called on <code class="docutils literal notranslate"><span class="pre">importlib</span> <span class="pre">engine.sysengine</span></code>, <code class="docutils literal notranslate"><span class="pre">from_engine()</span></code> can be used to create an <code class="docutils literal notranslate"><span class="pre">ImportEngine</span></code> object with a <strong>copy</strong> of the global import state.</div></blockquote> <p><code class="docutils literal notranslate"><span class="pre">__import__(self,</span> <span class="pre">name,</span> <span class="pre">globals={},</span> <span class="pre">locals={},</span> <span class="pre">fromlist=[],</span> <span class="pre">level=0)</span></code></p> <blockquote> <div>Reimplementation of the builtin <code class="docutils literal notranslate"><span class="pre">__import__()</span></code> function. The import of a module will proceed using the state stored in the ImportEngine instance rather than the global import state. For full documentation of <code class="docutils literal notranslate"><span class="pre">__import__</span></code> functionality, see <a class="footnote-reference brackets" href="#id5" id="id2">[2]</a> . <code class="docutils literal notranslate"><span class="pre">__import__()</span></code> from <code class="docutils literal notranslate"><span class="pre">ImportEngine</span></code> and its subclasses can be used to customise the behaviour of the <code class="docutils literal notranslate"><span class="pre">import</span></code> statement by replacing <code class="docutils literal notranslate"><span class="pre">__builtin__.__import__</span></code> with <code class="docutils literal notranslate"><span class="pre">ImportEngine().__import__</span></code>.</div></blockquote> <p><code class="docutils literal notranslate"><span class="pre">import_module(name,</span> <span class="pre">package=None)</span></code></p> <blockquote> <div>A reimplementation of <code class="docutils literal notranslate"><span class="pre">importlib.import_module()</span></code> which uses the import state stored in the ImportEngine instance. See <a class="footnote-reference brackets" href="#id6" id="id3">[3]</a> for a full reference.</div></blockquote> <p><code class="docutils literal notranslate"><span class="pre">modules,</span> <span class="pre">path,</span> <span class="pre">path_hooks,</span> <span class="pre">meta_path,</span> <span class="pre">path_importer_cache</span></code></p> <blockquote> <div>Instance-specific versions of their process global <code class="docutils literal notranslate"><span class="pre">sys</span></code> equivalents</div></blockquote> </div></blockquote> <p><code class="docutils literal notranslate"><span class="pre">importlib.engine.GlobalImportEngine(ImportEngine)</span></code></p> <blockquote> <div>Convenience class to provide engine-like access to the global state. Provides <code class="docutils literal notranslate"><span class="pre">__import__()</span></code>, <code class="docutils literal notranslate"><span class="pre">import_module()</span></code> and <code class="docutils literal notranslate"><span class="pre">from_engine()</span></code> methods like <code class="docutils literal notranslate"><span class="pre">ImportEngine</span></code> but writes through to the global state in <code class="docutils literal notranslate"><span class="pre">sys</span></code>.</div></blockquote> <p>To support various namespace package mechanisms, when <code class="docutils literal notranslate"><span class="pre">sys.path</span></code> is altered, tools like <code class="docutils literal notranslate"><span class="pre">pkgutil.extend_path</span></code> should be used to also modify other parts of the import state (in this case, package <code class="docutils literal notranslate"><span class="pre">__path__</span></code> attributes). The path importer cache should also be invalidated when a variety of changes are made.</p> <p>The <code class="docutils literal notranslate"><span class="pre">ImportEngine</span></code> API will provide convenience methods that automatically make related import state updates as part of a single operation.</p> </section> <section id="global-variables"> <h3><a class="toc-backref" href="#global-variables" role="doc-backlink">Global variables</a></h3> <p><code class="docutils literal notranslate"><span class="pre">importlib.engine.sysengine</span></code></p> <blockquote> <div>A precreated instance of <code class="docutils literal notranslate"><span class="pre">GlobalImportEngine</span></code>. Intended for use by importers and loaders that have been updated to accept optional <code class="docutils literal notranslate"><span class="pre">engine</span></code> parameters and with <code class="docutils literal notranslate"><span class="pre">ImportEngine.from_engine(sysengine)</span></code> to start with a copy of the process global import state.</div></blockquote> </section> <section id="no-changes-to-finder-loader-interfaces"> <h3><a class="toc-backref" href="#no-changes-to-finder-loader-interfaces" role="doc-backlink">No changes to finder/loader interfaces</a></h3> <p>Rather than attempting to update the <a class="pep reference internal" href="../pep-0302/" title="PEP 302 – New Import Hooks">PEP 302</a> APIs to accept additional state, this PEP proposes that <code class="docutils literal notranslate"><span class="pre">ImportEngine</span></code> support the content management protocol (similar to the context substitution mechanisms in the <code class="docutils literal notranslate"><span class="pre">decimal</span></code> module).</p> <p>The context management mechanism for <code class="docutils literal notranslate"><span class="pre">ImportEngine</span></code> would:</p> <ul class="simple"> <li>On entry: * Acquire the import lock * Substitute the global import state with the import engine’s own state</li> <li>On exit: * Restore the previous global import state * Release the import lock</li> </ul> <p>The precise API for this is TBD (but will probably use a distinct context management object, along the lines of that created by <code class="docutils literal notranslate"><span class="pre">decimal.localcontext</span></code>).</p> </section> </section> <section id="open-issues"> <h2><a class="toc-backref" href="#open-issues" role="doc-backlink">Open Issues</a></h2> <section id="api-design-for-falling-back-to-global-import-state"> <h3><a class="toc-backref" href="#api-design-for-falling-back-to-global-import-state" role="doc-backlink">API design for falling back to global import state</a></h3> <p>The current proposal relies on the <code class="docutils literal notranslate"><span class="pre">from_engine()</span></code> API to fall back to the global import state. It may be desirable to offer a variant that instead falls back to the global import state dynamically.</p> <p>However, one big advantage of starting with an “as isolated as possible” design is that it becomes possible to experiment with subclasses that blur the boundaries between the engine instance state and the process global state in various ways.</p> </section> <section id="builtin-and-extension-modules-must-be-process-global"> <h3><a class="toc-backref" href="#builtin-and-extension-modules-must-be-process-global" role="doc-backlink">Builtin and extension modules must be process global</a></h3> <p>Due to platform limitations, only one copy of each builtin and extension module can readily exist in each process. Accordingly, it is impossible for each <code class="docutils literal notranslate"><span class="pre">ImportEngine</span></code> instance to load such modules independently.</p> <p>The simplest solution is for <code class="docutils literal notranslate"><span class="pre">ImportEngine</span></code> to refuse to load such modules, raising <code class="docutils literal notranslate"><span class="pre">ImportError</span></code>. <code class="docutils literal notranslate"><span class="pre">GlobalImportEngine</span></code> would be able to load them normally.</p> <p><code class="docutils literal notranslate"><span class="pre">ImportEngine</span></code> will still return such modules from a prepopulated module cache - it’s only loading them directly which causes problems.</p> </section> <section id="scope-of-substitution"> <h3><a class="toc-backref" href="#scope-of-substitution" role="doc-backlink">Scope of substitution</a></h3> <p>Related to the previous open issue is the question of what state to substitute when using the context management API. It is currently the case that replacing <code class="docutils literal notranslate"><span class="pre">sys.modules</span></code> can be unreliable due to cached references and there’s the underlying fact that having independent copies of some modules is simply impossible due to platform limitations.</p> <p>As part of this PEP, it will be necessary to document explicitly:</p> <ul class="simple"> <li>Which parts of the global import state can be substituted (and declare code which caches references to that state without dealing with the substitution case buggy)</li> <li>Which parts must be modified in-place (and hence are not substituted by the <code class="docutils literal notranslate"><span class="pre">ImportEngine</span></code> context management API, or otherwise scoped to <code class="docutils literal notranslate"><span class="pre">ImportEngine</span></code> instances)</li> </ul> </section> </section> <section id="reference-implementation"> <h2><a class="toc-backref" href="#reference-implementation" role="doc-backlink">Reference Implementation</a></h2> <p>A reference implementation <a class="footnote-reference brackets" href="#id7" id="id4">[4]</a> for an earlier draft of this PEP, based on Brett Cannon’s importlib has been developed by Greg Slodkowicz as part of the 2011 Google Summer of Code. Note that the current implementation avoids modifying existing code, and hence duplicates a lot of things unnecessarily. An actual implementation would just modify any such affected code in place.</p> <p>That earlier draft of the PEP proposed change the <a class="pep reference internal" href="../pep-0302/" title="PEP 302 – New Import Hooks">PEP 302</a> APIs to support passing in an optional engine instance. This had the (serious) downside of not correctly affecting further imports from the imported module, hence the change to the context management based proposal for substituting the global state.</p> </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <aside class="footnote-list brackets"> <aside class="footnote brackets" id="id5" role="doc-footnote"> <dt class="label" id="id5">[<a href="#id2">2</a>]</dt> <dd>__import__() builtin function, The Python Standard Library documentation (<a class="reference external" href="http://docs.python.org/library/functions.html#__import__">http://docs.python.org/library/functions.html#__import__</a>)</aside> <aside class="footnote brackets" id="id6" role="doc-footnote"> <dt class="label" id="id6">[3]<em> (<a href='#id1'>1</a>, <a href='#id3'>2</a>) </em></dt> <dd>Importlib documentation, Cannon (<a class="reference external" href="http://docs.python.org/dev/library/importlib">http://docs.python.org/dev/library/importlib</a>)</aside> <aside class="footnote brackets" id="id7" role="doc-footnote"> <dt class="label" id="id7">[<a href="#id4">4</a>]</dt> <dd>Reference implementation (<a class="reference external" href="https://bitbucket.org/jergosh/gsoc_import_engine/src/default/Lib/importlib/engine.py">https://bitbucket.org/jergosh/gsoc_import_engine/src/default/Lib/importlib/engine.py</a>)</aside> </aside> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> <p>This document has been placed in the public domain.</p> </section> </section> <hr class="docutils" /> <p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0406.rst">https://github.com/python/peps/blob/main/peps/pep-0406.rst</a></p> <p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0406.rst">2025-02-01 08:59:27 GMT</a></p> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#pep-withdrawal">PEP Withdrawal</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#proposal">Proposal</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#importengine-api">ImportEngine API</a></li> <li><a class="reference internal" href="#global-variables">Global variables</a></li> <li><a class="reference internal" href="#no-changes-to-finder-loader-interfaces">No changes to finder/loader interfaces</a></li> </ul> </li> <li><a class="reference internal" href="#open-issues">Open Issues</a><ul> <li><a class="reference internal" href="#api-design-for-falling-back-to-global-import-state">API design for falling back to global import state</a></li> <li><a class="reference internal" href="#builtin-and-extension-modules-must-be-process-global">Builtin and extension modules must be process global</a></li> <li><a class="reference internal" href="#scope-of-substitution">Scope of substitution</a></li> </ul> </li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <br> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0406.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>