CINXE.COM
PEP 382 – Namespace Packages | peps.python.org
<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 382 – Namespace Packages | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0382/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 382 – Namespace Packages | peps.python.org'> <meta property="og:description" content="Namespace packages are a mechanism for splitting a single Python package across multiple directories on disk. In current Python versions, an algorithm to compute the packages __path__ must be formulated. With the enhancement proposed here, the import ma..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0382/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="Namespace packages are a mechanism for splitting a single Python package across multiple directories on disk. In current Python versions, an algorithm to compute the packages __path__ must be formulated. With the enhancement proposed here, the import ma..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 382</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> <span class="visually-hidden">Toggle light / dark / auto colour theme</span> </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 382 – Namespace Packages</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author<span class="colon">:</span></dt> <dd class="field-odd">Martin von Löwis <martin at v.loewis.de></dd> <dt class="field-even">Status<span class="colon">:</span></dt> <dd class="field-even"><abbr title="Formally declined and will not be accepted">Rejected</abbr></dd> <dt class="field-odd">Type<span class="colon">:</span></dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created<span class="colon">:</span></dt> <dd class="field-even">02-Apr-2009</dd> <dt class="field-odd">Python-Version<span class="colon">:</span></dt> <dd class="field-odd">3.2</dd> <dt class="field-even">Post-History<span class="colon">:</span></dt> <dd class="field-even"><p></p></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#rejection-notice">Rejection Notice</a></li> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#terminology">Terminology</a></li> <li><a class="reference internal" href="#namespace-packages-today">Namespace packages today</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#impact-on-import-hooks">Impact on Import Hooks</a></li> </ul> </li> <li><a class="reference internal" href="#discussion">Discussion</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="rejection-notice"> <h2><a class="toc-backref" href="#rejection-notice" role="doc-backlink">Rejection Notice</a></h2> <p>On the first day of sprints at US PyCon 2012 we had a long and fruitful discussion about <a class="pep reference internal" href="../pep-0382/" title="PEP 382 – Namespace Packages">PEP 382</a> and <a class="pep reference internal" href="../pep-0402/" title="PEP 402 – Simplified Package Layout and Partitioning">PEP 402</a>. We ended up rejecting both but a new PEP will be written to carry on in the spirit of PEP 402. Martin von Löwis wrote up a summary: <a class="footnote-reference brackets" href="#id4" id="id1">[2]</a>.</p> </section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> <p>Namespace packages are a mechanism for splitting a single Python package across multiple directories on disk. In current Python versions, an algorithm to compute the packages __path__ must be formulated. With the enhancement proposed here, the import machinery itself will construct the list of directories that make up the package. An implementation of this PEP is available at <a class="footnote-reference brackets" href="#id3" id="id2">[1]</a>.</p> </section> <section id="terminology"> <h2><a class="toc-backref" href="#terminology" role="doc-backlink">Terminology</a></h2> <p>Within this PEP, the term package refers to Python packages as defined by Python’s import statement. The term distribution refers to separately installable sets of Python modules as stored in the Python package index, and installed by distutils or setuptools. The term vendor package refers to groups of files installed by an operating system’s packaging mechanism (e.g. Debian or Redhat packages install on Linux systems).</p> <p>The term portion refers to a set of files in a single directory (possibly stored in a zip file) that contribute to a namespace package.</p> </section> <section id="namespace-packages-today"> <h2><a class="toc-backref" href="#namespace-packages-today" role="doc-backlink">Namespace packages today</a></h2> <p>Python currently provides the pkgutil.extend_path to denote a package as a namespace package. The recommended way of using it is to put:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">pkgutil</span><span class="w"> </span><span class="kn">import</span> <span class="n">extend_path</span> <span class="n">__path__</span> <span class="o">=</span> <span class="n">extend_path</span><span class="p">(</span><span class="n">__path__</span><span class="p">,</span> <span class="vm">__name__</span><span class="p">)</span> </pre></div> </div> <p>in the package’s <code class="docutils literal notranslate"><span class="pre">__init__.py</span></code>. Every distribution needs to provide the same contents in its <code class="docutils literal notranslate"><span class="pre">__init__.py</span></code>, so that extend_path is invoked independent of which portion of the package gets imported first. As a consequence, the package’s <code class="docutils literal notranslate"><span class="pre">__init__.py</span></code> cannot practically define any names as it depends on the order of the package fragments on sys.path which portion is imported first. As a special feature, extend_path reads files named <code class="docutils literal notranslate"><span class="pre"><packagename>.pkg</span></code> which allow to declare additional portions.</p> <p>setuptools provides a similar function pkg_resources.declare_namespace that is used in the form:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">pkg_resources</span> <span class="n">pkg_resources</span><span class="o">.</span><span class="n">declare_namespace</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span> </pre></div> </div> <p>In the portion’s __init__.py, no assignment to __path__ is necessary, as declare_namespace modifies the package __path__ through sys.modules. As a special feature, declare_namespace also supports zip files, and registers the package name internally so that future additions to sys.path by setuptools can properly add additional portions to each package.</p> <p>setuptools allows declaring namespace packages in a distribution’s setup.py, so that distribution developers don’t need to put the magic __path__ modification into __init__.py themselves.</p> </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> <p>The current imperative approach to namespace packages has lead to multiple slightly-incompatible mechanisms for providing namespace packages. For example, pkgutil supports <code class="docutils literal notranslate"><span class="pre">*.pkg</span></code> files; setuptools doesn’t. Likewise, setuptools supports inspecting zip files, and supports adding portions to its _namespace_packages variable, whereas pkgutil doesn’t.</p> <p>In addition, the current approach causes problems for system vendors. Vendor packages typically must not provide overlapping files, and an attempt to install a vendor package that has a file already on disk will fail or cause unpredictable behavior. As vendors might chose to package distributions such that they will end up all in a single directory for the namespace package, all portions would contribute conflicting __init__.py files.</p> </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> <p>Rather than using an imperative mechanism for importing packages, a declarative approach is proposed here: A directory whose name ends with <code class="docutils literal notranslate"><span class="pre">.pyp</span></code> (for Python package) contains a portion of a package.</p> <p>The import statement is extended so that computes the package’s <code class="docutils literal notranslate"><span class="pre">__path__</span></code> attribute for a package named <code class="docutils literal notranslate"><span class="pre">P</span></code> as consisting of optionally a single directory name <code class="docutils literal notranslate"><span class="pre">P</span></code> containing a file <code class="docutils literal notranslate"><span class="pre">__init__.py</span></code>, plus all directories named <code class="docutils literal notranslate"><span class="pre">P.pyp</span></code>, in the order in which they are found in the parent’s package <code class="docutils literal notranslate"><span class="pre">__path__</span></code> (or <code class="docutils literal notranslate"><span class="pre">sys.path</span></code>). If either of these are found, search for additional portions of the package continues.</p> <p>A directory may contain both a package in the <code class="docutils literal notranslate"><span class="pre">P/__init__.py</span></code> and the <code class="docutils literal notranslate"><span class="pre">P.pyp</span></code> form.</p> <p>No other change to the importing mechanism is made; searching modules (including __init__.py) will continue to stop at the first module encountered. In summary, the process import a package foo works like this:</p> <ol class="arabic simple"> <li>sys.path is searched for directories foo or foo.pyp, or a file foo.<ext>. If a file is found and no directory, it is treated as a module, and imported.</li> <li>If a directory foo is found, a check is made whether it contains __init__.py. If so, the location of the __init__.py is remembered. Otherwise, the directory is skipped. Once an __init__.py is found, further directories called foo are skipped.</li> <li>For both directories foo and foo.pyp, the directories are added to the package’s __path__.</li> <li>If an __init__ module was found, it is imported, with __path__ being initialized to the path computed all <code class="docutils literal notranslate"><span class="pre">.pyp</span></code> directories.</li> </ol> <section id="impact-on-import-hooks"> <h3><a class="toc-backref" href="#impact-on-import-hooks" role="doc-backlink">Impact on Import Hooks</a></h3> <p>Both loaders and finders as defined in <a class="pep reference internal" href="../pep-0302/" title="PEP 302 – New Import Hooks">PEP 302</a> will need to be changed to support namespace packages. Failure to conform to the protocol below might cause a package not being recognized as a namespace package; loaders and finders not supporting this protocol must raise AttributeError when the functions below get accessed.</p> <p>Finders need to support looking for *.pth files in step 1 of above algorithm. To do so, a finder used as a path hook must support a method:</p> <blockquote> <div>finder.find_package_portion(fullname)</div></blockquote> <p>This method will be called in the same manner as find_module, and it must return a string to be added to the package’s <code class="docutils literal notranslate"><span class="pre">__path__</span></code>. If the finder doesn’t find a portion of the package, it shall return <code class="docutils literal notranslate"><span class="pre">None</span></code>. Raising <code class="docutils literal notranslate"><span class="pre">AttributeError</span></code> from above call will be treated as non-conformance with this PEP, and the exception will be ignored. All other exceptions are reported.</p> <p>A finder may report both success from <code class="docutils literal notranslate"><span class="pre">find_module</span></code> and from <code class="docutils literal notranslate"><span class="pre">find_package_portion</span></code>, allowing for both a package containing an <code class="docutils literal notranslate"><span class="pre">__init__.py</span></code> and a portion of the same package.</p> <p>All strings returned from <code class="docutils literal notranslate"><span class="pre">find_package_portion</span></code>, along with all path names of <code class="docutils literal notranslate"><span class="pre">.pyp</span></code> directories are added to the new package’s <code class="docutils literal notranslate"><span class="pre">__path__</span></code>.</p> </section> </section> <section id="discussion"> <h2><a class="toc-backref" href="#discussion" role="doc-backlink">Discussion</a></h2> <p>Original versions of this specification proposed the addition of <code class="docutils literal notranslate"><span class="pre">*.pth</span></code> files, similar to the way those files are used on sys.path. With a wildcard marker (<code class="docutils literal notranslate"><span class="pre">*</span></code>), a package could indicate that the entire path is derived by looking at the parent path, searching for properly-named subdirectories.</p> <p>People then observed that the support for the full .pth syntax is inappropriate, and the .pth files were changed to be mere marker files, indicating that a directories is a package. Peter Tröger suggested that .pth is an unsuitable file extension, as all file extensions related to Python should start with <code class="docutils literal notranslate"><span class="pre">.py</span></code>. Therefore, the marker file was renamed to be <code class="docutils literal notranslate"><span class="pre">.pyp</span></code>.</p> <p>Dinu Gherman then observed that using a marker file is not necessary, and that a directory extension could well serve as a such as a marker. This is what this PEP currently proposes.</p> <p>Phillip Eby designed <a class="pep reference internal" href="../pep-0402/" title="PEP 402 – Simplified Package Layout and Partitioning">PEP 402</a> as an alternative approach to this PEP, after comparing Python’s package syntax with that found in other languages. <a class="pep reference internal" href="../pep-0402/" title="PEP 402 – Simplified Package Layout and Partitioning">PEP 402</a> proposes not to use a marker file at all. At the discussion at PyCon DE 2011, people remarked that having an explicit declaration of a directory as contributing to a package is a desirable property, rather than an obstacle. In particular, Jython developers noticed that Jython could easily mistake a directory that is a Java package as being a Python package, if there is no need to declare Python packages.</p> <p>Packages can stop filling out the namespace package’s __init__.py. As a consequence, extend_path and declare_namespace become obsolete.</p> <p>Namespace packages can start providing non-trivial __init__.py implementations; to do so, it is recommended that a single distribution provides a portion with just the namespace package’s __init__.py (and potentially other modules that belong to the namespace package proper).</p> <p>The mechanism is mostly compatible with the existing namespace mechanisms. extend_path will be adjusted to this specification; any other mechanism might cause portions to get added twice to __path__.</p> </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <aside class="footnote-list brackets"> <aside class="footnote brackets" id="id3" role="doc-footnote"> <dt class="label" id="id3">[<a href="#id2">1</a>]</dt> <dd>PEP 382 branch (<a class="reference external" href="http://hg.python.org/features/pep-382-2#pep-382">http://hg.python.org/features/pep-382-2#pep-382</a>)</aside> <aside class="footnote brackets" id="id4" role="doc-footnote"> <dt class="label" id="id4">[<a href="#id1">2</a>]</dt> <dd>Namespace Packages resolution (<a class="reference external" href="https://mail.python.org/pipermail/import-sig/2012-March/000421.html">https://mail.python.org/pipermail/import-sig/2012-March/000421.html</a>)</aside> </aside> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> <p>This document has been placed in the public domain.</p> </section> </section> <hr class="docutils" /> <p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0382.rst">https://github.com/python/peps/blob/main/peps/pep-0382.rst</a></p> <p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0382.rst">2025-02-01 08:55:40 GMT</a></p> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#rejection-notice">Rejection Notice</a></li> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#terminology">Terminology</a></li> <li><a class="reference internal" href="#namespace-packages-today">Namespace packages today</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#impact-on-import-hooks">Impact on Import Hooks</a></li> </ul> </li> <li><a class="reference internal" href="#discussion">Discussion</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <br> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0382.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>