CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 420 – Implicit Namespace Packages | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0420/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 420 – Implicit Namespace Packages | peps.python.org'> <meta property="og:description" content="Namespace packages are a mechanism for splitting a single Python package across multiple directories on disk. In current Python versions, an algorithm to compute the packages __path__ must be formulated. With the enhancement proposed here, the import ..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0420/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="Namespace packages are a mechanism for splitting a single Python package across multiple directories on disk. In current Python versions, an algorithm to compute the packages __path__ must be formulated. With the enhancement proposed here, the import ..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 420</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 420 – Implicit Namespace Packages</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Eric V. Smith <eric at trueblade.com></dd> <dt class="field-even">Status:</dt> <dd class="field-even"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-odd">Type:</dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created:</dt> <dd class="field-even">19-Apr-2012</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">3.3</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even"></dd> <dt class="field-odd">Resolution:</dt> <dd class="field-odd"><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2012-May/119651.html">Python-Dev message</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#terminology">Terminology</a></li> <li><a class="reference internal" href="#namespace-packages-today">Namespace packages today</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#dynamic-path-computation">Dynamic path computation</a></li> <li><a class="reference internal" href="#impact-on-import-finders-and-loaders">Impact on import finders and loaders</a></li> <li><a class="reference internal" href="#differences-between-namespace-packages-and-regular-packages">Differences between namespace packages and regular packages</a></li> <li><a class="reference internal" href="#namespace-packages-in-the-standard-library">Namespace packages in the standard library</a></li> <li><a class="reference internal" href="#migrating-from-legacy-namespace-packages">Migrating from legacy namespace packages</a></li> </ul> </li> <li><a class="reference internal" href="#packaging-implications">Packaging Implications</a></li> <li><a class="reference internal" href="#examples">Examples</a><ul> <li><a class="reference internal" href="#nested-namespace-packages">Nested namespace packages</a></li> <li><a class="reference internal" href="#id2">Dynamic path computation</a></li> </ul> </li> <li><a class="reference internal" href="#discussion">Discussion</a><ul> <li><a class="reference internal" href="#find-module-versus-find-loader"><code class="docutils literal notranslate">find_module</code> versus <code class="docutils literal notranslate">find_loader</code></a></li> <li><a class="reference internal" href="#id8">Dynamic path computation</a></li> </ul> </li> <li><a class="reference internal" href="#module-reprs">Module reprs</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> Namespace packages are a mechanism for splitting a single Python package across multiple directories on disk. In current Python versions, an algorithm to compute the packages <code class="docutils literal notranslate">__path__</code> must be formulated. With the enhancement proposed here, the import machinery itself will construct the list of directories that make up the package. This PEP builds upon previous work, documented in <a class="pep reference internal" href="../pep-0382/" title="PEP 382 – Namespace Packages">PEP 382</a> and <a class="pep reference internal" href="../pep-0402/" title="PEP 402 – Simplified Package Layout and Partitioning">PEP 402</a>. Those PEPs have since been rejected in favor of this one. An implementation of this PEP is at <a class="footnote-reference brackets" href="#id11" id="id1">[1]</a>. </section> <section id="terminology"> <h2><a class="toc-backref" href="#terminology" role="doc-backlink">Terminology</a></h2> Within this PEP: <ul class="simple"> <li>“package” refers to Python packages as defined by Python’s import statement.</li> <li>“distribution” refers to separately installable sets of Python modules as stored in the Python package index, and installed by distutils or setuptools.</li> <li>“vendor package” refers to groups of files installed by an operating system’s packaging mechanism (e.g. Debian or Redhat packages install on Linux systems).</li> <li>“regular package” refers to packages as they are implemented in Python 3.2 and earlier.</li> <li>“portion” refers to a set of files in a single directory (possibly stored in a zip file) that contribute to a namespace package.</li> <li>“legacy portion” refers to a portion that uses <code class="docutils literal notranslate">__path__</code> manipulation in order to implement namespace packages.</li> </ul> This PEP defines a new type of package, the “namespace package”. </section> <section id="namespace-packages-today"> <h2><a class="toc-backref" href="#namespace-packages-today" role="doc-backlink">Namespace packages today</a></h2> Python currently provides <code class="docutils literal notranslate">pkgutil.extend_path</code> to denote a package as a namespace package. The recommended way of using it is to put: <div class="highlight-default notranslate"><div class="highlight"><pre>from pkgutil import extend_path __path__ = extend_path(__path__, __name__) </pre></div> </div> in the package’s <code class="docutils literal notranslate">__init__.py</code>. Every distribution needs to provide the same contents in its <code class="docutils literal notranslate">__init__.py</code>, so that <code class="docutils literal notranslate">extend_path</code> is invoked independent of which portion of the package gets imported first. As a consequence, the package’s <code class="docutils literal notranslate">__init__.py</code> cannot practically define any names as it depends on the order of the package fragments on <code class="docutils literal notranslate">sys.path</code> to determine which portion is imported first. As a special feature, <code class="docutils literal notranslate">extend_path</code> reads files named <code class="docutils literal notranslate"><packagename>.pkg</code> which allows declaration of additional portions. setuptools provides a similar function named <code class="docutils literal notranslate">pkg_resources.declare_namespace</code> that is used in the form: <div class="highlight-default notranslate"><div class="highlight"><pre>import pkg_resources pkg_resources.declare_namespace(__name__) </pre></div> </div> In the portion’s <code class="docutils literal notranslate">__init__.py</code>, no assignment to <code class="docutils literal notranslate">__path__</code> is necessary, as <code class="docutils literal notranslate">declare_namespace</code> modifies the package <code class="docutils literal notranslate">__path__</code> through <code class="docutils literal notranslate">sys.modules</code>. As a special feature, <code class="docutils literal notranslate">declare_namespace</code> also supports zip files, and registers the package name internally so that future additions to <code class="docutils literal notranslate">sys.path</code> by setuptools can properly add additional portions to each package. setuptools allows declaring namespace packages in a distribution’s <code class="docutils literal notranslate">setup.py</code>, so that distribution developers don’t need to put the magic <code class="docutils literal notranslate">__path__</code> modification into <code class="docutils literal notranslate">__init__.py</code> themselves. See <a class="pep reference internal" href="../pep-0402/" title="PEP 402 – Simplified Package Layout and Partitioning">PEP 402</a>’s <a class="pep reference internal" href="../pep-0402/#the-problem" title="PEP 402 – Simplified Package Layout and Partitioning § The Problem">“The Problem”</a> section for additional motivations for namespace packages. Note that <a class="pep reference internal" href="../pep-0402/" title="PEP 402 – Simplified Package Layout and Partitioning">PEP 402</a> has been rejected, but the motivating use cases are still valid. </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> The current imperative approach to namespace packages has led to multiple slightly-incompatible mechanisms for providing namespace packages. For example, pkgutil supports <code class="docutils literal notranslate">*.pkg</code> files; setuptools doesn’t. Likewise, setuptools supports inspecting zip files, and supports adding portions to its <code class="docutils literal notranslate">_namespace_packages</code> variable, whereas pkgutil doesn’t. Namespace packages are designed to support being split across multiple directories (and hence found via multiple <code class="docutils literal notranslate">sys.path</code> entries). In this configuration, it doesn’t matter if multiple portions all provide an <code class="docutils literal notranslate">__init__.py</code> file, so long as each portion correctly initializes the namespace package. However, Linux distribution vendors (amongst others) prefer to combine the separate portions and install them all into the same file system directory. This creates a potential for conflict, as the portions are now attempting to provide the same file on the target system - something that is not allowed by many package managers. Allowing implicit namespace packages means that the requirement to provide an <code class="docutils literal notranslate">__init__.py</code> file can be dropped completely, and affected portions can be installed into a common directory or split across multiple directories as distributions see fit. A namespace package will not be constrained by a fixed <code class="docutils literal notranslate">__path__</code>, computed from the parent path at namespace package creation time. Consider the standard library <code class="docutils literal notranslate">encodings</code> package: <ol class="arabic simple"> <li>Suppose that <code class="docutils literal notranslate">encodings</code> becomes a namespace package.</li> <li>It sometimes gets imported during interpreter startup to initialize the standard io streams.</li> <li>An application modifies <code class="docutils literal notranslate">sys.path</code> after startup and wants to contribute additional encodings from new path entries.</li> <li>An attempt is made to import an encoding from an <code class="docutils literal notranslate">encodings</code> portion that is found on a path entry added in step 3.</li> </ol> If the import system was restricted to only finding portions along the value of <code class="docutils literal notranslate">sys.path</code> that existed at the time the <code class="docutils literal notranslate">encodings</code> namespace package was created, the additional paths added in step 3 would never be searched for the additional portions imported in step 4. In addition, if step 2 were sometimes skipped (due to some runtime flag or other condition), then the path items added in step 3 would indeed be used the first time a portion was imported. Thus this PEP requires that the list of path entries be dynamically computed when each portion is loaded. It is expected that the import machinery will do this efficiently by caching <code class="docutils literal notranslate">__path__</code> values and only refreshing them when it detects that the parent path has changed. In the case of a top-level package like <code class="docutils literal notranslate">encodings</code>, this parent path would be <code class="docutils literal notranslate">sys.path</code>. </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> Regular packages will continue to have an <code class="docutils literal notranslate">__init__.py</code> and will reside in a single directory. Namespace packages cannot contain an <code class="docutils literal notranslate">__init__.py</code>. As a consequence, <code class="docutils literal notranslate">pkgutil.extend_path</code> and <code class="docutils literal notranslate">pkg_resources.declare_namespace</code> become obsolete for purposes of namespace package creation. There will be no marker file or directory for specifying a namespace package. During import processing, the import machinery will continue to iterate over each directory in the parent path as it does in Python 3.2. While looking for a module or package named “foo”, for each directory in the parent path: <ul class="simple"> <li>If <code class="docutils literal notranslate"><directory>/foo/__init__.py</code> is found, a regular package is imported and returned.</li> <li>If not, but <code class="docutils literal notranslate"><directory>/foo.{py,pyc,so,pyd}</code> is found, a module is imported and returned. The exact list of extension varies by platform and whether the -O flag is specified. The list here is representative.</li> <li>If not, but <code class="docutils literal notranslate"><directory>/foo</code> is found and is a directory, it is recorded and the scan continues with the next directory in the parent path.</li> <li>Otherwise the scan continues with the next directory in the parent path.</li> </ul> If the scan completes without returning a module or package, and at least one directory was recorded, then a namespace package is created. The new namespace package: <ul class="simple"> <li>Has a <code class="docutils literal notranslate">__path__</code> attribute set to an iterable of the path strings that were found and recorded during the scan.</li> <li>Does not have a <code class="docutils literal notranslate">__file__</code> attribute.</li> </ul> Note that if “import foo” is executed and “foo” is found as a namespace package (using the above rules), then “foo” is immediately created as a package. The creation of the namespace package is not deferred until a sub-level import occurs. A namespace package is not fundamentally different from a regular package. It is just a different way of creating packages. Once a namespace package is created, there is no functional difference between it and a regular package. <section id="dynamic-path-computation"> <h3><a class="toc-backref" href="#dynamic-path-computation" role="doc-backlink">Dynamic path computation</a></h3> The import machinery will behave as if a namespace package’s <code class="docutils literal notranslate">__path__</code> is recomputed before each portion is loaded. For performance reasons, it is expected that this will be achieved by detecting that the parent path has changed. If no change has taken place, then no <code class="docutils literal notranslate">__path__</code> recomputation is required. The implementation must ensure that changes to the contents of the parent path are detected, as well as detecting the replacement of the parent path with a new path entry list object. </section> <section id="impact-on-import-finders-and-loaders"> <h3><a class="toc-backref" href="#impact-on-import-finders-and-loaders" role="doc-backlink">Impact on import finders and loaders</a></h3> <a class="pep reference internal" href="../pep-0302/" title="PEP 302 – New Import Hooks">PEP 302</a> defines “finders” that are called to search path elements. These finders’ <code class="docutils literal notranslate">find_module</code> methods return either a “loader” object or <code class="docutils literal notranslate">None</code>. For a finder to contribute to namespace packages, it must implement a new <code class="docutils literal notranslate">find_loader(fullname)</code> method. <code class="docutils literal notranslate">fullname</code> has the same meaning as for <code class="docutils literal notranslate">find_module</code>. <code class="docutils literal notranslate">find_loader</code> always returns a 2-tuple of <code class="docutils literal notranslate">(loader, <iterable-of-path-entries>)</code>. <code class="docutils literal notranslate">loader</code> may be <code class="docutils literal notranslate">None</code>, in which case <code class="docutils literal notranslate"><iterable-of-path-entries></code> (which may be empty) is added to the list of recorded path entries and path searching continues. If <code class="docutils literal notranslate">loader</code> is not <code class="docutils literal notranslate">None</code>, it is immediately used to load a module or regular package. Even if <code class="docutils literal notranslate">loader</code> is returned and is not <code class="docutils literal notranslate">None</code>, <code class="docutils literal notranslate"><iterable-of-path-entries></code> must still contain the path entries for the package. This allows code such as <code class="docutils literal notranslate">pkgutil.extend_path()</code> to compute path entries for packages that it does not load. Note that multiple path entries per finder are allowed. This is to support the case where a finder discovers multiple namespace portions for a given <code class="docutils literal notranslate">fullname</code>. Many finders will support only a single namespace package portion per <code class="docutils literal notranslate">find_loader</code> call, in which case this iterable will contain only a single string. The import machinery will call <code class="docutils literal notranslate">find_loader</code> if it exists, else fall back to <code class="docutils literal notranslate">find_module</code>. Legacy finders which implement <code class="docutils literal notranslate">find_module</code> but not <code class="docutils literal notranslate">find_loader</code> will be unable to contribute portions to a namespace package. The specification expands <a class="pep reference internal" href="../pep-0302/" title="PEP 302 – New Import Hooks">PEP 302</a> loaders to include an optional method called <code class="docutils literal notranslate">module_repr()</code> which if present, is used to generate module object reprs. See the section below for further details. </section> <section id="differences-between-namespace-packages-and-regular-packages"> <h3><a class="toc-backref" href="#differences-between-namespace-packages-and-regular-packages" role="doc-backlink">Differences between namespace packages and regular packages</a></h3> Namespace packages and regular packages are very similar. The differences are: <ul class="simple"> <li>Portions of namespace packages need not all come from the same directory structure, or even from the same loader. Regular packages are self-contained: all parts live in the same directory hierarchy.</li> <li>Namespace packages have no <code class="docutils literal notranslate">__file__</code> attribute.</li> <li>Namespace packages’ <code class="docutils literal notranslate">__path__</code> attribute is a read-only iterable of strings, which is automatically updated when the parent path is modified.</li> <li>Namespace packages have no <code class="docutils literal notranslate">__init__.py</code> module.</li> <li>Namespace packages have a different type of object for their <code class="docutils literal notranslate">__loader__</code> attribute.</li> </ul> </section> <section id="namespace-packages-in-the-standard-library"> <h3><a class="toc-backref" href="#namespace-packages-in-the-standard-library" role="doc-backlink">Namespace packages in the standard library</a></h3> It is possible, and this PEP explicitly allows, that parts of the standard library be implemented as namespace packages. When and if any standard library packages become namespace packages is outside the scope of this PEP. </section> <section id="migrating-from-legacy-namespace-packages"> <h3><a class="toc-backref" href="#migrating-from-legacy-namespace-packages" role="doc-backlink">Migrating from legacy namespace packages</a></h3> As described above, prior to this PEP <code class="docutils literal notranslate">pkgutil.extend_path()</code> was used by legacy portions to create namespace packages. Because it is likely not practical for all existing portions of a namespace package to be migrated to this PEP at once, <code class="docutils literal notranslate">extend_path()</code> will be modified to also recognize <a class="pep reference internal" href="../pep-0420/" title="PEP 420 – Implicit Namespace Packages">PEP 420</a> namespace packages. This will allow some portions of a namespace to be legacy portions while others are migrated to <a class="pep reference internal" href="../pep-0420/" title="PEP 420 – Implicit Namespace Packages">PEP 420</a>. These hybrid namespace packages will not have the dynamic path computation that normal namespace packages have, since <code class="docutils literal notranslate">extend_path()</code> never provided this functionality in the past. </section> </section> <section id="packaging-implications"> <h2><a class="toc-backref" href="#packaging-implications" role="doc-backlink">Packaging Implications</a></h2> Multiple portions of a namespace package can be installed into the same directory, or into separate directories. For this section, suppose there are two portions which define “foo.bar” and “foo.baz”. “foo” itself is a namespace package. If these are installed in the same location, a single directory “foo” would be in a directory that is on <code class="docutils literal notranslate">sys.path</code>. Inside “foo” would be two directories, “bar” and “baz”. If “foo.bar” is removed (perhaps by an OS package manager), care must be taken not to remove the “foo/baz” or “foo” directories. Note that in this case “foo” will be a namespace package (because it lacks an <code class="docutils literal notranslate">__init__.py</code>), even though all of its portions are in the same directory. Note that “foo.bar” and “foo.baz” can be installed into the same “foo” directory because they will not have any files in common. If the portions are installed in different locations, two different “foo” directories would be in directories that are on <code class="docutils literal notranslate">sys.path</code>. “foo/bar” would be in one of these sys.path entries, and “foo/baz” would be in the other. Upon removal of “foo.bar”, the “foo/bar” and corresponding “foo” directories can be completely removed. But “foo/baz” and its corresponding “foo” directory cannot be removed. It is also possible to have the “foo.bar” portion installed in a directory on <code class="docutils literal notranslate">sys.path</code>, and have the “foo.baz” portion provided in a zip file, also on <code class="docutils literal notranslate">sys.path</code>. </section> <section id="examples"> <h2><a class="toc-backref" href="#examples" role="doc-backlink">Examples</a></h2> <section id="nested-namespace-packages"> <h3><a class="toc-backref" href="#nested-namespace-packages" role="doc-backlink">Nested namespace packages</a></h3> This example uses the following directory structure: <div class="highlight-default notranslate"><div class="highlight"><pre>Lib/test/namespace_pkgs project1 parent child one.py project2 parent child two.py </pre></div> </div> Here, both parent and child are namespace packages: Portions of them exist in different directories, and they do not have <code class="docutils literal notranslate">__init__.py</code> files. Here we add the parent directories to <code class="docutils literal notranslate">sys.path</code>, and show that the portions are correctly found: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> import sys >>> sys.path += ['Lib/test/namespace_pkgs/project1', 'Lib/test/namespace_pkgs/project2'] >>> import parent.child.one >>> parent.__path__ _NamespacePath(['Lib/test/namespace_pkgs/project1/parent', 'Lib/test/namespace_pkgs/project2/parent']) >>> parent.child.__path__ _NamespacePath(['Lib/test/namespace_pkgs/project1/parent/child', 'Lib/test/namespace_pkgs/project2/parent/child']) >>> import parent.child.two >>> </pre></div> </div> </section> <section id="id2"> <h3><a class="toc-backref" href="#id2" role="doc-backlink">Dynamic path computation</a></h3> This example uses a similar directory structure, but adds a third portion: <div class="highlight-default notranslate"><div class="highlight"><pre>Lib/test/namespace_pkgs project1 parent child one.py project2 parent child two.py project3 parent child three.py </pre></div> </div> We add <code class="docutils literal notranslate">project1</code> and <code class="docutils literal notranslate">project2</code> to <code class="docutils literal notranslate">sys.path</code>, then import <code class="docutils literal notranslate">parent.child.one</code> and <code class="docutils literal notranslate">parent.child.two</code>. Then we add the <code class="docutils literal notranslate">project3</code> to <code class="docutils literal notranslate">sys.path</code> and when <code class="docutils literal notranslate">parent.child.three</code> is imported, <code class="docutils literal notranslate">project3/parent</code> is automatically added to <code class="docutils literal notranslate">parent.__path__</code>: <div class="highlight-default notranslate"><div class="highlight"><pre># add the first two parent paths to sys.path >>> import sys >>> sys.path += ['Lib/test/namespace_pkgs/project1', 'Lib/test/namespace_pkgs/project2'] # parent.child.one can be imported, because project1 was added to sys.path: >>> import parent.child.one >>> parent.__path__ _NamespacePath(['Lib/test/namespace_pkgs/project1/parent', 'Lib/test/namespace_pkgs/project2/parent']) # parent.child.__path__ contains project1/parent/child and project2/parent/child, but not project3/parent/child: >>> parent.child.__path__ _NamespacePath(['Lib/test/namespace_pkgs/project1/parent/child', 'Lib/test/namespace_pkgs/project2/parent/child']) # parent.child.two can be imported, because project2 was added to sys.path: >>> import parent.child.two # we cannot import parent.child.three, because project3 is not in the path: >>> import parent.child.three Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<frozen importlib._bootstrap>", line 1286, in _find_and_load File "<frozen importlib._bootstrap>", line 1250, in _find_and_load_unlocked ImportError: No module named 'parent.child.three' # now add project3 to sys.path: >>> sys.path.append('Lib/test/namespace_pkgs/project3') # and now parent.child.three can be imported: >>> import parent.child.three # project3/parent has been added to parent.__path__: >>> parent.__path__ _NamespacePath(['Lib/test/namespace_pkgs/project1/parent', 'Lib/test/namespace_pkgs/project2/parent', 'Lib/test/namespace_pkgs/project3/parent']) # and project3/parent/child has been added to parent.child.__path__ >>> parent.child.__path__ _NamespacePath(['Lib/test/namespace_pkgs/project1/parent/child', 'Lib/test/namespace_pkgs/project2/parent/child', 'Lib/test/namespace_pkgs/project3/parent/child']) >>> </pre></div> </div> </section> </section> <section id="discussion"> <h2><a class="toc-backref" href="#discussion" role="doc-backlink">Discussion</a></h2> At PyCon 2012, we had a discussion about namespace packages at which <a class="pep reference internal" href="../pep-0382/" title="PEP 382 – Namespace Packages">PEP 382</a> and <a class="pep reference internal" href="../pep-0402/" title="PEP 402 – Simplified Package Layout and Partitioning">PEP 402</a> were rejected, to be replaced by this PEP <a class="footnote-reference brackets" href="#id12" id="id3">[3]</a>. There is no intention to remove support of regular packages. If a developer knows that her package will never be a portion of a namespace package, then there is a performance advantage to it being a regular package (with an <code class="docutils literal notranslate">__init__.py</code>). Creation and loading of a regular package can take place immediately when it is located along the path. With namespace packages, all entries in the path must be scanned before the package is created. Note that an ImportWarning will no longer be raised for a directory lacking an <code class="docutils literal notranslate">__init__.py</code> file. Such a directory will now be imported as a namespace package, whereas in prior Python versions an ImportWarning would be raised. Alyssa (Nick) Coghlan presented a list of her objections to this proposal <a class="footnote-reference brackets" href="#id13" id="id4">[4]</a>. They are: <ol class="arabic simple"> <li>Implicit package directories go against the Zen of Python.</li> <li>Implicit package directories pose awkward backwards compatibility challenges.</li> <li>Implicit package directories introduce ambiguity into file system layouts.</li> <li>Implicit package directories will permanently entrench current newbie-hostile behavior in <code class="docutils literal notranslate">__main__</code>.</li> </ol> Alyssa later gave a detailed response to her own objections <a class="footnote-reference brackets" href="#id14" id="id5">[5]</a>, which is summarized here: <ol class="arabic simple"> <li>The practicality of this PEP wins over other proposals and the status quo.</li> <li>Minor backward compatibility issues are okay, as long as they are properly documented.</li> <li>This will be addressed in <a class="pep reference internal" href="../pep-0395/" title="PEP 395 – Qualified Names for Modules">PEP 395</a>.</li> <li>This will also be addressed in <a class="pep reference internal" href="../pep-0395/" title="PEP 395 – Qualified Names for Modules">PEP 395</a>.</li> </ol> The inclusion of namespace packages in the standard library was motivated by Martin v. Löwis, who wanted the <code class="docutils literal notranslate">encodings</code> package to become a namespace package <a class="footnote-reference brackets" href="#id15" id="id6">[6]</a>. While this PEP allows for standard library packages to become namespaces, it defers a decision on <code class="docutils literal notranslate">encodings</code>. <section id="find-module-versus-find-loader"> <h3><a class="toc-backref" href="#find-module-versus-find-loader" role="doc-backlink"><code class="docutils literal notranslate">find_module</code> versus <code class="docutils literal notranslate">find_loader</code></a></h3> An early draft of this PEP specified a change to the <code class="docutils literal notranslate">find_module</code> method in order to support namespace packages. It would be modified to return a string in the case where a namespace package portion was discovered. However, this caused a problem with existing code outside of the standard library which calls <code class="docutils literal notranslate">find_module</code>. Because this code would not be upgraded in concert with changes required by this PEP, it would fail when it would receive unexpected return values from <code class="docutils literal notranslate">find_module</code>. Because of this incompatibility, this PEP now specifies that finders that want to provide namespace portions must implement the <code class="docutils literal notranslate">find_loader</code> method, described above. The use case for supporting multiple portions per <code class="docutils literal notranslate">find_loader</code> call is given in <a class="footnote-reference brackets" href="#id16" id="id7">[7]</a>. </section> <section id="id8"> <h3><a class="toc-backref" href="#id8" role="doc-backlink">Dynamic path computation</a></h3> Guido raised a concern that automatic dynamic path computation was an unnecessary feature <a class="footnote-reference brackets" href="#id17" id="id9">[8]</a>. Later in that thread, PJ Eby and Alyssa Coghlan presented arguments as to why dynamic computation would minimize surprise to Python users. The conclusion of that discussion has been included in this PEP’s Rationale section. An earlier version of this PEP required that dynamic path computation could only take affect if the parent path object were modified in-place. That is, this would work: <div class="highlight-default notranslate"><div class="highlight"><pre>sys.path.append('new-dir') </pre></div> </div> But this would not: <div class="highlight-default notranslate"><div class="highlight"><pre>sys.path = sys.path + ['new-dir'] </pre></div> </div> In the same thread <a class="footnote-reference brackets" href="#id17" id="id10">[8]</a>, it was pointed out that this restriction is not required. If the parent path is looked up by name instead of by holding a reference to it, then there is no restriction on how the parent path is modified or replaced. For a top-level namespace package, the lookup would be the module named <code class="docutils literal notranslate">"sys"</code> then its attribute <code class="docutils literal notranslate">"path"</code>. For a namespace package nested inside a package <code class="docutils literal notranslate">foo</code>, the lookup would be for the module named <code class="docutils literal notranslate">"foo"</code> then its attribute <code class="docutils literal notranslate">"__path__"</code>. </section> </section> <section id="module-reprs"> <h2><a class="toc-backref" href="#module-reprs" role="doc-backlink">Module reprs</a></h2> Previously, module reprs were hard coded based on assumptions about a module’s <code class="docutils literal notranslate">__file__</code> attribute. If this attribute existed and was a string, it was assumed to be a file system path, and the module object’s repr would include this in its value. The only exception was that <a class="pep reference internal" href="../pep-0302/" title="PEP 302 – New Import Hooks">PEP 302</a> reserved missing <code class="docutils literal notranslate">__file__</code> attributes to built-in modules, and in CPython, this assumption was baked into the module object’s implementation. Because of this restriction, some modules contained contrived <code class="docutils literal notranslate">__file__</code> values that did not reflect file system paths, and which could cause unexpected problems later (e.g. <code class="docutils literal notranslate">os.path.join()</code> on a non-path <code class="docutils literal notranslate">__file__</code> would return gibberish). This PEP relaxes this constraint, and leaves the setting of <code class="docutils literal notranslate">__file__</code> to the purview of the loader producing the module. Loaders may opt to leave <code class="docutils literal notranslate">__file__</code> unset if no file system path is appropriate. Loaders may also set additional reserved attributes on the module if useful. This means that the definitive way to determine the origin of a module is to check its <code class="docutils literal notranslate">__loader__</code> attribute. For example, namespace packages as described in this PEP will have no <code class="docutils literal notranslate">__file__</code> attribute because no corresponding file exists. In order to provide flexibility and descriptiveness in the reprs of such modules, a new optional protocol is added to <a class="pep reference internal" href="../pep-0302/" title="PEP 302 – New Import Hooks">PEP 302</a> loaders. Loaders can implement a <code class="docutils literal notranslate">module_repr()</code> method which takes a single argument, the module object. This method should return the string to be used verbatim as the repr of the module. The rules for producing a module repr are now standardized as: <ul class="simple"> <li>If the module has an <code class="docutils literal notranslate">__loader__</code> and that loader has a <code class="docutils literal notranslate">module_repr()</code> method, call it with a single argument, which is the module object. The value returned is used as the module’s repr.</li> <li>If an exception occurs in <code class="docutils literal notranslate">module_repr()</code>, the exception is caught and discarded, and the calculation of the module’s repr continues as if <code class="docutils literal notranslate">module_repr()</code> did not exist.</li> <li>If the module has an <code class="docutils literal notranslate">__file__</code> attribute, this is used as part of the module’s repr.</li> <li>If the module has no <code class="docutils literal notranslate">__file__</code> but does have an <code class="docutils literal notranslate">__loader__</code>, then the loader’s repr is used as part of the module’s repr.</li> <li>Otherwise, just use the module’s <code class="docutils literal notranslate">__name__</code> in the repr.</li> </ul> Here is a snippet showing how namespace module reprs are calculated from its loader: <div class="highlight-default notranslate"><div class="highlight"><pre>class NamespaceLoader: @classmethod def module_repr(cls, module): return "<module '{}' (namespace)>".format(module.__name__) </pre></div> </div> Built-in module reprs would no longer need to be hard-coded, but instead would come from their loader as well: <div class="highlight-default notranslate"><div class="highlight"><pre>class BuiltinImporter: @classmethod def module_repr(cls, module): return "<module '{}' (built-in)>".format(module.__name__) </pre></div> </div> Here are some example reprs of different types of modules with different sets of the related attributes: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> import email >>> email <module 'email' from '/home/barry/projects/python/pep-420/Lib/email/__init__.py'> >>> m = type(email)('foo') >>> m <module 'foo'> >>> m.__file__ = 'zippy:/de/do/dah' >>> m <module 'foo' from 'zippy:/de/do/dah'> >>> class Loader: pass ... >>> m.__loader__ = Loader >>> del m.__file__ >>> m <module 'foo' (<class '__main__.Loader'>)> >>> class NewLoader: ... @classmethod ... def module_repr(cls, module): ... return '<mystery module!>' ... >>> m.__loader__ = NewLoader >>> m <mystery module!> >>> </pre></div> </div> </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <aside class="footnote-list brackets"> <aside class="footnote brackets" id="id11" role="doc-footnote"> <dt class="label" id="id11">[<a href="#id1">1</a>]</dt> <dd>PEP 420 branch (<a class="reference external" href="http://hg.python.org/features/pep-420">http://hg.python.org/features/pep-420</a>)</aside> <aside class="footnote brackets" id="id12" role="doc-footnote"> <dt class="label" id="id12">[<a href="#id3">3</a>]</dt> <dd>PyCon 2012 Namespace Package discussion outcome (<a class="reference external" href="https://mail.python.org/pipermail/import-sig/2012-March/000421.html">https://mail.python.org/pipermail/import-sig/2012-March/000421.html</a>)</aside> <aside class="footnote brackets" id="id13" role="doc-footnote"> <dt class="label" id="id13">[<a href="#id4">4</a>]</dt> <dd>Alyssa Coghlan’s objection to the lack of marker files or directories (<a class="reference external" href="https://mail.python.org/pipermail/import-sig/2012-March/000423.html">https://mail.python.org/pipermail/import-sig/2012-March/000423.html</a>)</aside> <aside class="footnote brackets" id="id14" role="doc-footnote"> <dt class="label" id="id14">[<a href="#id5">5</a>]</dt> <dd>Alyssa Coghlan’s response to her initial objections (<a class="reference external" href="https://mail.python.org/pipermail/import-sig/2012-April/000464.html">https://mail.python.org/pipermail/import-sig/2012-April/000464.html</a>)</aside> <aside class="footnote brackets" id="id15" role="doc-footnote"> <dt class="label" id="id15">[<a href="#id6">6</a>]</dt> <dd>Martin v. Löwis’s suggestion to make <code class="docutils literal notranslate">encodings</code> a namespace package (<a class="reference external" href="https://mail.python.org/pipermail/import-sig/2012-May/000540.html">https://mail.python.org/pipermail/import-sig/2012-May/000540.html</a>)</aside> <aside class="footnote brackets" id="id16" role="doc-footnote"> <dt class="label" id="id16">[<a href="#id7">7</a>]</dt> <dd>Use case for multiple portions per <code class="docutils literal notranslate">find_loader</code> call (<a class="reference external" href="https://mail.python.org/pipermail/import-sig/2012-May/000585.html">https://mail.python.org/pipermail/import-sig/2012-May/000585.html</a>)</aside> <aside class="footnote brackets" id="id17" role="doc-footnote"> <dt class="label" id="id17">[8] (<a href='#id9'>1</a>, <a href='#id10'>2</a>) </dt> <dd>Discussion about dynamic path computation (<a class="reference external" href="https://mail.python.org/pipermail/python-dev/2012-May/119560.html">https://mail.python.org/pipermail/python-dev/2012-May/119560.html</a>)</aside> </aside> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0420.rst">https://github.com/python/peps/blob/main/peps/pep-0420.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0420.rst">2023-10-11 12:05:51 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#terminology">Terminology</a></li> <li><a class="reference internal" href="#namespace-packages-today">Namespace packages today</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#dynamic-path-computation">Dynamic path computation</a></li> <li><a class="reference internal" href="#impact-on-import-finders-and-loaders">Impact on import finders and loaders</a></li> <li><a class="reference internal" href="#differences-between-namespace-packages-and-regular-packages">Differences between namespace packages and regular packages</a></li> <li><a class="reference internal" href="#namespace-packages-in-the-standard-library">Namespace packages in the standard library</a></li> <li><a class="reference internal" href="#migrating-from-legacy-namespace-packages">Migrating from legacy namespace packages</a></li> </ul> </li> <li><a class="reference internal" href="#packaging-implications">Packaging Implications</a></li> <li><a class="reference internal" href="#examples">Examples</a><ul> <li><a class="reference internal" href="#nested-namespace-packages">Nested namespace packages</a></li> <li><a class="reference internal" href="#id2">Dynamic path computation</a></li> </ul> </li> <li><a class="reference internal" href="#discussion">Discussion</a><ul> <li><a class="reference internal" href="#find-module-versus-find-loader"><code class="docutils literal notranslate">find_module</code> versus <code class="docutils literal notranslate">find_loader</code></a></li> <li><a class="reference internal" href="#id8">Dynamic path computation</a></li> </ul> </li> <li><a class="reference internal" href="#module-reprs">Module reprs</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0420.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>