CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 395 – Qualified Names for Modules | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0395/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 395 – Qualified Names for Modules | peps.python.org'> <meta property="og:description" content="This PEP proposes new mechanisms that eliminate some longstanding traps for the unwary when dealing with Python’s import system, as well as serialisation and introspection of functions and classes."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0395/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This PEP proposes new mechanisms that eliminate some longstanding traps for the unwary when dealing with Python’s import system, as well as serialisation and introspection of functions and classes."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 395</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 395 – Qualified Names for Modules</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Alyssa Coghlan <ncoghlan at gmail.com></dd> <dt class="field-even">Status:</dt> <dd class="field-even"><abbr title="Removed from consideration by sponsor or authors">Withdrawn</abbr></dd> <dt class="field-odd">Type:</dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created:</dt> <dd class="field-even">04-Mar-2011</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">3.4</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even">05-Mar-2011, 19-Nov-2011</dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#pep-withdrawal">PEP Withdrawal</a></li> <li><a class="reference internal" href="#abstract">Abstract</a><ul> <li><a class="reference internal" href="#relationship-with-other-peps">Relationship with Other PEPs</a></li> </ul> </li> <li><a class="reference internal" href="#what-s-in-a-name">What’s in a <code class="docutils literal notranslate">__name__</code>?</a></li> <li><a class="reference internal" href="#traps-for-the-unwary">Traps for the Unwary</a><ul> <li><a class="reference internal" href="#why-are-my-imports-broken">Why are my imports broken?</a></li> <li><a class="reference internal" href="#importing-the-main-module-twice">Importing the main module twice</a></li> <li><a class="reference internal" href="#in-a-bit-of-a-pickle">In a bit of a pickle</a></li> <li><a class="reference internal" href="#where-s-the-source">Where’s the source?</a></li> <li><a class="reference internal" href="#forkless-windows">Forkless Windows</a></li> </ul> </li> <li><a class="reference internal" href="#qualified-names-for-modules">Qualified Names for Modules</a><ul> <li><a class="reference internal" href="#alternative-names">Alternative Names</a></li> </ul> </li> <li><a class="reference internal" href="#eliminating-the-traps">Eliminating the Traps</a><ul> <li><a class="reference internal" href="#fixing-main-module-imports-inside-packages">Fixing main module imports inside packages</a><ul> <li><a class="reference internal" href="#optional-addition-command-line-relative-imports">Optional addition: command line relative imports</a></li> <li><a class="reference internal" href="#compatibility-with-pep-382">Compatibility with PEP 382</a></li> <li><a class="reference internal" href="#incompatibility-with-pep-402">Incompatibility with PEP 402</a></li> <li><a class="reference internal" href="#potential-incompatibilities-with-scripts-stored-in-packages">Potential incompatibilities with scripts stored in packages</a></li> </ul> </li> <li><a class="reference internal" href="#fixing-dual-imports-of-the-main-module">Fixing dual imports of the main module</a></li> <li><a class="reference internal" href="#fixing-pickling-without-breaking-introspection">Fixing pickling without breaking introspection</a></li> <li><a class="reference internal" href="#fixing-multiprocessing-on-windows">Fixing multiprocessing on Windows</a></li> </ul> </li> <li><a class="reference internal" href="#explicit-relative-imports">Explicit relative imports</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="pep-withdrawal"> <h2><a class="toc-backref" href="#pep-withdrawal" role="doc-backlink">PEP Withdrawal</a></h2> This PEP was withdrawn by the author in December 2013, as other significant changes in the time since it was written have rendered several aspects obsolete. Most notably <a class="pep reference internal" href="../pep-0420/" title="PEP 420 – Implicit Namespace Packages">PEP 420</a> namespace packages rendered some of the proposals related to package detection unworkable and <a class="pep reference internal" href="../pep-0451/" title="PEP 451 – A ModuleSpec Type for the Import System">PEP 451</a> module specifications resolved the multiprocessing issues and provide a possible means to tackle the pickle compatibility issues. A future PEP to resolve the remaining issues would still be appropriate, but it’s worth starting any such effort as a fresh PEP restating the remaining problems in an updated context rather than trying to build on this one directly. </section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> This PEP proposes new mechanisms that eliminate some longstanding traps for the unwary when dealing with Python’s import system, as well as serialisation and introspection of functions and classes. It builds on the “Qualified Name” concept defined in <a class="pep reference internal" href="../pep-3155/" title="PEP 3155 – Qualified name for classes and functions">PEP 3155</a>. <section id="relationship-with-other-peps"> <h3><a class="toc-backref" href="#relationship-with-other-peps" role="doc-backlink">Relationship with Other PEPs</a></h3> Most significantly, this PEP is currently deferred as it requires significant changes in order to be made compatible with the removal of mandatory __init__.py files in <a class="pep reference internal" href="../pep-0420/" title="PEP 420 – Implicit Namespace Packages">PEP 420</a> (which has been implemented and released in Python 3.3). This PEP builds on the “qualified name” concept introduced by <a class="pep reference internal" href="../pep-3155/" title="PEP 3155 – Qualified name for classes and functions">PEP 3155</a>, and also shares in that PEP’s aim of fixing some ugly corner cases when dealing with serialisation of arbitrary functions and classes. It also builds on <a class="pep reference internal" href="../pep-0366/" title="PEP 366 – Main module explicit relative imports">PEP 366</a>, which took initial tentative steps towards making explicit relative imports from the main module work correctly in at least some circumstances. Finally, <a class="pep reference internal" href="../pep-0328/" title="PEP 328 – Imports: Multi-Line and Absolute/Relative">PEP 328</a> eliminated implicit relative imports from imported modules. This PEP proposes that the de facto implicit relative imports from main modules that are provided by the current initialisation behaviour for <code class="docutils literal notranslate">sys.path[0]</code> also be eliminated. </section> </section> <section id="what-s-in-a-name"> <h2><a class="toc-backref" href="#what-s-in-a-name" role="doc-backlink">What’s in a <code class="docutils literal notranslate">__name__</code>?</a></h2> Over time, a module’s <code class="docutils literal notranslate">__name__</code> attribute has come to be used to handle a number of different tasks. The key use cases identified for this module attribute are: <ol class="arabic simple"> <li>Flagging the main module in a program, using the <code class="docutils literal notranslate">if __name__ == "__main__":</code> convention.</li> <li>As the starting point for relative imports</li> <li>To identify the location of function and class definitions within the running application</li> <li>To identify the location of classes for serialisation into pickle objects which may be shared with other interpreter instances</li> </ol> </section> <section id="traps-for-the-unwary"> <h2><a class="toc-backref" href="#traps-for-the-unwary" role="doc-backlink">Traps for the Unwary</a></h2> The overloading of the semantics of <code class="docutils literal notranslate">__name__</code>, along with some historically associated behaviour in the initialisation of <code class="docutils literal notranslate">sys.path[0]</code>, has resulted in several traps for the unwary. These traps can be quite annoying in practice, as they are highly unobvious (especially to beginners) and can cause quite confusing behaviour. <section id="why-are-my-imports-broken"> <h3><a class="toc-backref" href="#why-are-my-imports-broken" role="doc-backlink">Why are my imports broken?</a></h3> There’s a general principle that applies when modifying <code class="docutils literal notranslate">sys.path</code>: never put a package directory directly on <code class="docutils literal notranslate">sys.path</code>. The reason this is problematic is that every module in that directory is now potentially accessible under two different names: as a top level module (since the package directory is on <code class="docutils literal notranslate">sys.path</code>) and as a submodule of the package (if the higher level directory containing the package itself is also on <code class="docutils literal notranslate">sys.path</code>). As an example, Django (up to and including version 1.3) is guilty of setting up exactly this situation for site-specific applications - the application ends up being accessible as both <code class="docutils literal notranslate">app</code> and <code class="docutils literal notranslate">site.app</code> in the module namespace, and these are actually two different copies of the module. This is a recipe for confusion if there is any meaningful mutable module level state, so this behaviour is being eliminated from the default site set up in version 1.4 (site-specific apps will always be fully qualified with the site name). However, it’s hard to blame Django for this, when the same part of Python responsible for setting <code class="docutils literal notranslate">__name__ = "__main__"</code> in the main module commits the exact same error when determining the value for <code class="docutils literal notranslate">sys.path[0]</code>. The impact of this can be seen relatively frequently if you follow the “python” and “import” tags on Stack Overflow. When I had the time to follow it myself, I regularly encountered people struggling to understand the behaviour of straightforward package layouts like the following (I actually use package layouts along these lines in my own projects): <div class="highlight-default notranslate"><div class="highlight"><pre>project/ setup.py example/ __init__.py foo.py tests/ __init__.py test_foo.py </pre></div> </div> While I would often see it without the <code class="docutils literal notranslate">__init__.py</code> files first, that’s a trivial fix to explain. What’s hard to explain is that all of the following ways to invoke <code class="docutils literal notranslate">test_foo.py</code> probably won’t work due to broken imports (either failing to find <code class="docutils literal notranslate">example</code> for absolute imports, complaining about relative imports in a non-package or beyond the toplevel package for explicit relative imports, or issuing even more obscure errors if some other submodule happens to shadow the name of a top-level module, such as an <code class="docutils literal notranslate">example.json</code> module that handled serialisation or an <code class="docutils literal notranslate">example.tests.unittest</code> test runner): <div class="highlight-default notranslate"><div class="highlight"><pre># These commands will most likely *FAIL*, even if the code is correct # working directory: project/example/tests ./test_foo.py python test_foo.py python -m package.tests.test_foo python -c "from package.tests.test_foo import main; main()" # working directory: project/package tests/test_foo.py python tests/test_foo.py python -m package.tests.test_foo python -c "from package.tests.test_foo import main; main()" # working directory: project example/tests/test_foo.py python example/tests/test_foo.py # working directory: project/.. project/example/tests/test_foo.py python project/example/tests/test_foo.py # The -m and -c approaches don't work from here either, but the failure # to find 'package' correctly is easier to explain in this case </pre></div> </div> That’s right, that long list is of all the methods of invocation that will almost certainly break if you try them, and the error messages won’t make any sense if you’re not already intimately familiar not only with the way Python’s import system works, but also with how it gets initialised. For a long time, the only way to get <code class="docutils literal notranslate">sys.path</code> right with that kind of setup was to either set it manually in <code class="docutils literal notranslate">test_foo.py</code> itself (hardly something a novice, or even many veteran, Python programmers are going to know how to do) or else to make sure to import the module instead of executing it directly: <div class="highlight-default notranslate"><div class="highlight"><pre># working directory: project python -c "from package.tests.test_foo import main; main()" </pre></div> </div> Since the implementation of <a class="pep reference internal" href="../pep-0366/" title="PEP 366 – Main module explicit relative imports">PEP 366</a> (which defined a mechanism that allows relative imports to work correctly when a module inside a package is executed via the <code class="docutils literal notranslate">-m</code> switch), the following also works properly: <div class="highlight-default notranslate"><div class="highlight"><pre># working directory: project python -m package.tests.test_foo </pre></div> </div> The fact that most methods of invoking Python code from the command line break when that code is inside a package, and the two that do work are highly sensitive to the current working directory is all thoroughly confusing for a beginner. I personally believe it is one of the key factors leading to the perception that Python packages are complicated and hard to get right. This problem isn’t even limited to the command line - if <code class="docutils literal notranslate">test_foo.py</code> is open in Idle and you attempt to run it by pressing F5, or if you try to run it by clicking on it in a graphical filebrowser, then it will fail in just the same way it would if run directly from the command line. There’s a reason the general “no package directories on <code class="docutils literal notranslate">sys.path</code>” guideline exists, and the fact that the interpreter itself doesn’t follow it when determining <code class="docutils literal notranslate">sys.path[0]</code> is the root cause of all sorts of grief. In the past, this couldn’t be fixed due to backwards compatibility concerns. However, scripts potentially affected by this problem will already require fixes when porting to the Python 3.x (due to the elimination of implicit relative imports when importing modules normally). This provides a convenient opportunity to implement a corresponding change in the initialisation semantics for <code class="docutils literal notranslate">sys.path[0]</code>. </section> <section id="importing-the-main-module-twice"> <h3><a class="toc-backref" href="#importing-the-main-module-twice" role="doc-backlink">Importing the main module twice</a></h3> Another venerable trap is the issue of importing <code class="docutils literal notranslate">__main__</code> twice. This occurs when the main module is also imported under its real name, effectively creating two instances of the same module under different names. If the state stored in <code class="docutils literal notranslate">__main__</code> is significant to the correct operation of the program, or if there is top-level code in the main module that has non-idempotent side effects, then this duplication can cause obscure and surprising errors. </section> <section id="in-a-bit-of-a-pickle"> <h3><a class="toc-backref" href="#in-a-bit-of-a-pickle" role="doc-backlink">In a bit of a pickle</a></h3> Something many users may not realise is that the <code class="docutils literal notranslate">pickle</code> module sometimes relies on the <code class="docutils literal notranslate">__module__</code> attribute when serialising instances of arbitrary classes. So instances of classes defined in <code class="docutils literal notranslate">__main__</code> are pickled that way, and won’t be unpickled correctly by another python instance that only imported that module instead of running it directly. This behaviour is the underlying reason for the advice from many Python veterans to do as little as possible in the <code class="docutils literal notranslate">__main__</code> module in any application that involves any form of object serialisation and persistence. Similarly, when creating a pseudo-module (see next paragraph), pickles rely on the name of the module where a class is actually defined, rather than the officially documented location for that class in the module hierarchy. For the purposes of this PEP, a “pseudo-module” is a package designed like the Python 3.2 <code class="docutils literal notranslate">unittest</code> and <code class="docutils literal notranslate">concurrent.futures</code> packages. These packages are documented as if they were single modules, but are in fact internally implemented as a package. This is supposed to be an implementation detail that users and other implementations don’t need to worry about, but, thanks to <code class="docutils literal notranslate">pickle</code> (and serialisation in general), the details are often exposed and can effectively become part of the public API. While this PEP focuses specifically on <code class="docutils literal notranslate">pickle</code> as the principal serialisation scheme in the standard library, this issue may also affect other mechanisms that support serialisation of arbitrary class instances and rely on <code class="docutils literal notranslate">__module__</code> attributes to determine how to handle deserialisation. </section> <section id="where-s-the-source"> <h3><a class="toc-backref" href="#where-s-the-source" role="doc-backlink">Where’s the source?</a></h3> Some sophisticated users of the pseudo-module technique described above recognise the problem with implementation details leaking out via the <code class="docutils literal notranslate">pickle</code> module, and choose to address it by altering <code class="docutils literal notranslate">__name__</code> to refer to the public location for the module before defining any functions or classes (or else by modifying the <code class="docutils literal notranslate">__module__</code> attributes of those objects after they have been defined). This approach is effective at eliminating the leakage of information via pickling, but comes at the cost of breaking introspection for functions and classes (as their <code class="docutils literal notranslate">__module__</code> attribute now points to the wrong place). </section> <section id="forkless-windows"> <h3><a class="toc-backref" href="#forkless-windows" role="doc-backlink">Forkless Windows</a></h3> To get around the lack of <code class="docutils literal notranslate">os.fork</code> on Windows, the <code class="docutils literal notranslate">multiprocessing</code> module attempts to re-execute Python with the same main module, but skipping over any code guarded by <code class="docutils literal notranslate">if __name__ == "__main__":</code> checks. It does the best it can with the information it has, but is forced to make assumptions that simply aren’t valid whenever the main module isn’t an ordinary directly executed script or top-level module. Packages and non-top-level modules executed via the <code class="docutils literal notranslate">-m</code> switch, as well as directly executed zipfiles or directories, are likely to make multiprocessing on Windows do the wrong thing (either quietly or noisily, depending on application details) when spawning a new process. While this issue currently only affects Windows directly, it also impacts any proposals to provide Windows-style “clean process” invocation via the multiprocessing module on other platforms. </section> </section> <section id="qualified-names-for-modules"> <h2><a class="toc-backref" href="#qualified-names-for-modules" role="doc-backlink">Qualified Names for Modules</a></h2> To make it feasible to fix these problems once and for all, it is proposed to add a new module level attribute: <code class="docutils literal notranslate">__qualname__</code>. This abbreviation of “qualified name” is taken from <a class="pep reference internal" href="../pep-3155/" title="PEP 3155 – Qualified name for classes and functions">PEP 3155</a>, where it is used to store the naming path to a nested class or function definition relative to the top level module. For modules, <code class="docutils literal notranslate">__qualname__</code> will normally be the same as <code class="docutils literal notranslate">__name__</code>, just as it is for top-level functions and classes in <a class="pep reference internal" href="../pep-3155/" title="PEP 3155 – Qualified name for classes and functions">PEP 3155</a>. However, it will differ in some situations so that the above problems can be addressed. Specifically, whenever <code class="docutils literal notranslate">__name__</code> is modified for some other purpose (such as to denote the main module), then <code class="docutils literal notranslate">__qualname__</code> will remain unchanged, allowing code that needs it to access the original unmodified value. If a module loader does not initialise <code class="docutils literal notranslate">__qualname__</code> itself, then the import system will add it automatically (setting it to the same value as <code class="docutils literal notranslate">__name__</code>). <section id="alternative-names"> <h3><a class="toc-backref" href="#alternative-names" role="doc-backlink">Alternative Names</a></h3> Two alternative names were also considered for the new attribute: “full name” (<code class="docutils literal notranslate">__fullname__</code>) and “implementation name” (<code class="docutils literal notranslate">__implname__</code>). Either of those would actually be valid for the use case in this PEP. However, as a meta-issue, <a class="pep reference internal" href="../pep-3155/" title="PEP 3155 – Qualified name for classes and functions">PEP 3155</a> is also adding a new attribute (for functions and classes) that is “like <code class="docutils literal notranslate">__name__</code>, but different in some cases where <code class="docutils literal notranslate">__name__</code> is missing necessary information” and those terms aren’t accurate for the <a class="pep reference internal" href="../pep-3155/" title="PEP 3155 – Qualified name for classes and functions">PEP 3155</a> function and class use case. <a class="pep reference internal" href="../pep-3155/" title="PEP 3155 – Qualified name for classes and functions">PEP 3155</a> deliberately omits the module information, so the term “full name” is simply untrue, and “implementation name” implies that it may specify an object other than that specified by <code class="docutils literal notranslate">__name__</code>, and that is never the case for <a class="pep reference internal" href="../pep-3155/" title="PEP 3155 – Qualified name for classes and functions">PEP 3155</a> (in that PEP, <code class="docutils literal notranslate">__name__</code> and <code class="docutils literal notranslate">__qualname__</code> always refer to the same function or class, it’s just that <code class="docutils literal notranslate">__name__</code> is insufficient to accurately identify nested functions and classes). Since it seems needlessly inconsistent to add two new terms for attributes that only exist because backwards compatibility concerns keep us from changing the behaviour of <code class="docutils literal notranslate">__name__</code> itself, this PEP instead chose to adopt the <a class="pep reference internal" href="../pep-3155/" title="PEP 3155 – Qualified name for classes and functions">PEP 3155</a> terminology. If the relative inscrutability of “qualified name” and <code class="docutils literal notranslate">__qualname__</code> encourages interested developers to look them up at least once rather than assuming they know what they mean just from the name and guessing wrong, that’s not necessarily a bad outcome. Besides, 99% of Python developers should never need to even care these extra attributes exist - they’re really an implementation detail to let us fix a few problematic behaviours exhibited by imports, pickling and introspection, not something people are going to be dealing with on a regular basis. </section> </section> <section id="eliminating-the-traps"> <h2><a class="toc-backref" href="#eliminating-the-traps" role="doc-backlink">Eliminating the Traps</a></h2> The following changes are interrelated and make the most sense when considered together. They collectively either completely eliminate the traps for the unwary noted above, or else provide straightforward mechanisms for dealing with them. A rough draft of some of the concepts presented here was first posted on the python-ideas list (<a class="footnote-reference brackets" href="#id6" id="id1">[1]</a>), but they have evolved considerably since first being discussed in that thread. Further discussion has subsequently taken place on the import-sig mailing list (<a class="footnote-reference brackets" href="#id7" id="id2">[2]</a>. <a class="footnote-reference brackets" href="#id8" id="id3">[3]</a>). <section id="fixing-main-module-imports-inside-packages"> <h3><a class="toc-backref" href="#fixing-main-module-imports-inside-packages" role="doc-backlink">Fixing main module imports inside packages</a></h3> To eliminate this trap, it is proposed that an additional filesystem check be performed when determining a suitable value for <code class="docutils literal notranslate">sys.path[0]</code>. This check will look for Python’s explicit package directory markers and use them to find the appropriate directory to add to <code class="docutils literal notranslate">sys.path</code>. The current algorithm for setting <code class="docutils literal notranslate">sys.path[0]</code> in relevant cases is roughly as follows: <div class="highlight-default notranslate"><div class="highlight"><pre># Interactive prompt, -m switch, -c switch sys.path.insert(0, '') </pre></div> </div> <div class="highlight-default notranslate"><div class="highlight"><pre># Valid sys.path entry execution (i.e. directory and zip execution) sys.path.insert(0, sys.argv[0]) </pre></div> </div> <div class="highlight-default notranslate"><div class="highlight"><pre># Direct script execution sys.path.insert(0, os.path.dirname(sys.argv[0])) </pre></div> </div> It is proposed that this initialisation process be modified to take package details stored on the filesystem into account: <div class="highlight-default notranslate"><div class="highlight"><pre># Interactive prompt, -m switch, -c switch in_package, path_entry, _ignored = split_path_module(os.getcwd(), '') if in_package: sys.path.insert(0, path_entry) else: sys.path.insert(0, '') # Start interactive prompt or run -c command as usual # __main__.__qualname__ is set to "__main__" # The -m switches uses the same sys.path[0] calculation, but: # modname is the argument to the -m switch # modname is passed to ``runpy._run_module_as_main()`` as usual # __main__.__qualname__ is set to modname </pre></div> </div> <div class="highlight-default notranslate"><div class="highlight"><pre># Valid sys.path entry execution (i.e. directory and zip execution) modname = "__main__" path_entry, modname = split_path_module(sys.argv[0], modname) sys.path.insert(0, path_entry) # modname (possibly adjusted) is passed to ``runpy._run_module_as_main()`` # __main__.__qualname__ is set to modname </pre></div> </div> <div class="highlight-default notranslate"><div class="highlight"><pre># Direct script execution in_package, path_entry, modname = split_path_module(sys.argv[0]) sys.path.insert(0, path_entry) if in_package: # Pass modname to ``runpy._run_module_as_main()`` else: # Run script directly # __main__.__qualname__ is set to modname </pre></div> </div> The <code class="docutils literal notranslate">split_path_module()</code> supporting function used in the above pseudo-code would have the following semantics: <div class="highlight-default notranslate"><div class="highlight"><pre>def _splitmodname(fspath): path_entry, fname = os.path.split(fspath) modname = os.path.splitext(fname)[0] return path_entry, modname def _is_package_dir(fspath): return any(os.exists("__init__" + info[0]) for info in imp.get_suffixes()) def split_path_module(fspath, modname=None): """Given a filesystem path and a relative module name, determine an appropriate sys.path entry and a fully qualified module name. Returns a 3-tuple of (package_depth, fspath, modname). A reported package depth of 0 indicates that this would be a top level import. If no relative module name is given, it is derived from the final component in the supplied path with the extension stripped. """ if modname is None: fspath, modname = _splitmodname(fspath) package_depth = 0 while _is_package_dir(fspath): fspath, pkg = _splitmodname(fspath) modname = pkg + '.' + modname return package_depth, fspath, modname </pre></div> </div> This PEP also proposes that the <code class="docutils literal notranslate">split_path_module()</code> functionality be exposed directly to Python users via the <code class="docutils literal notranslate">runpy</code> module. With this fix in place, and the same simple package layout described earlier, all of the following commands would invoke the test suite correctly: <div class="highlight-default notranslate"><div class="highlight"><pre># working directory: project/example/tests ./test_foo.py python test_foo.py python -m package.tests.test_foo python -c "from .test_foo import main; main()" python -c "from ..tests.test_foo import main; main()" python -c "from package.tests.test_foo import main; main()" # working directory: project/package tests/test_foo.py python tests/test_foo.py python -m package.tests.test_foo python -c "from .tests.test_foo import main; main()" python -c "from package.tests.test_foo import main; main()" # working directory: project example/tests/test_foo.py python example/tests/test_foo.py python -m package.tests.test_foo python -c "from package.tests.test_foo import main; main()" # working directory: project/.. project/example/tests/test_foo.py python project/example/tests/test_foo.py # The -m and -c approaches still don't work from here, but the failure # to find 'package' correctly is pretty easy to explain in this case </pre></div> </div> With these changes, clicking Python modules in a graphical file browser should always execute them correctly, even if they live inside a package. Depending on the details of how it invokes the script, Idle would likely also be able to run <code class="docutils literal notranslate">test_foo.py</code> correctly with F5, without needing any Idle specific fixes. <section id="optional-addition-command-line-relative-imports"> <h4><a class="toc-backref" href="#optional-addition-command-line-relative-imports" role="doc-backlink">Optional addition: command line relative imports</a></h4> With the above changes in place, it would be a fairly minor addition to allow explicit relative imports as arguments to the <code class="docutils literal notranslate">-m</code> switch: <div class="highlight-default notranslate"><div class="highlight"><pre># working directory: project/example/tests python -m .test_foo python -m ..tests.test_foo # working directory: project/example/ python -m .tests.test_foo </pre></div> </div> With this addition, system initialisation for the <code class="docutils literal notranslate">-m</code> switch would change as follows: <div class="highlight-default notranslate"><div class="highlight"><pre># -m switch (permitting explicit relative imports) in_package, path_entry, pkg_name = split_path_module(os.getcwd(), '') qualname= <<arguments to -m switch>> if qualname.startswith('.'): modname = qualname while modname.startswith('.'): modname = modname[1:] pkg_name, sep, _ignored = pkg_name.rpartition('.') if not sep: raise ImportError("Attempted relative import beyond top level package") qualname = pkg_name + '.' modname if in_package: sys.path.insert(0, path_entry) else: sys.path.insert(0, '') # qualname is passed to ``runpy._run_module_as_main()`` # _main__.__qualname__ is set to qualname </pre></div> </div> </section> <section id="compatibility-with-pep-382"> <h4><a class="toc-backref" href="#compatibility-with-pep-382" role="doc-backlink">Compatibility with PEP 382</a></h4> Making this proposal compatible with the <a class="pep reference internal" href="../pep-0382/" title="PEP 382 – Namespace Packages">PEP 382</a> namespace packaging PEP is trivial. The semantics of <code class="docutils literal notranslate">_is_package_dir()</code> are merely changed to be: <div class="highlight-default notranslate"><div class="highlight"><pre>def _is_package_dir(fspath): return (fspath.endswith(".pyp") or any(os.exists("__init__" + info[0]) for info in imp.get_suffixes())) </pre></div> </div> </section> <section id="incompatibility-with-pep-402"> <h4><a class="toc-backref" href="#incompatibility-with-pep-402" role="doc-backlink">Incompatibility with PEP 402</a></h4> <a class="pep reference internal" href="../pep-0402/" title="PEP 402 – Simplified Package Layout and Partitioning">PEP 402</a> proposes the elimination of explicit markers in the file system for Python packages. This fundamentally breaks the proposed concept of being able to take a filesystem path and a Python module name and work out an unambiguous mapping to the Python module namespace. Instead, the appropriate mapping would depend on the current values in <code class="docutils literal notranslate">sys.path</code>, rendering it impossible to ever fix the problems described above with the calculation of <code class="docutils literal notranslate">sys.path[0]</code> when the interpreter is initialised. While some aspects of this PEP could probably be salvaged if <a class="pep reference internal" href="../pep-0402/" title="PEP 402 – Simplified Package Layout and Partitioning">PEP 402</a> were adopted, the core concept of making import semantics from main and other modules more consistent would no longer be feasible. This incompatibility is discussed in more detail in the relevant import-sig threads (<a class="footnote-reference brackets" href="#id7" id="id4">[2]</a>, <a class="footnote-reference brackets" href="#id8" id="id5">[3]</a>). </section> <section id="potential-incompatibilities-with-scripts-stored-in-packages"> <h4><a class="toc-backref" href="#potential-incompatibilities-with-scripts-stored-in-packages" role="doc-backlink">Potential incompatibilities with scripts stored in packages</a></h4> The proposed change to <code class="docutils literal notranslate">sys.path[0]</code> initialisation may break some existing code. Specifically, it will break scripts stored in package directories that rely on the implicit relative imports from <code class="docutils literal notranslate">__main__</code> in order to run correctly under Python 3. While such scripts could be imported in Python 2 (due to implicit relative imports) it is already the case that they cannot be imported in Python 3, as implicit relative imports are no longer permitted when a module is imported. By disallowing implicit relatives imports from the main module as well, such modules won’t even work as scripts with this PEP. Switching them over to explicit relative imports will then get them working again as both executable scripts and as importable modules. To support earlier versions of Python, a script could be written to use different forms of import based on the Python version: <div class="highlight-default notranslate"><div class="highlight"><pre>if __name__ == "__main__" and sys.version_info < (3, 3): import peer # Implicit relative import else: from . import peer # explicit relative import </pre></div> </div> </section> </section> <section id="fixing-dual-imports-of-the-main-module"> <h3><a class="toc-backref" href="#fixing-dual-imports-of-the-main-module" role="doc-backlink">Fixing dual imports of the main module</a></h3> Given the above proposal to get <code class="docutils literal notranslate">__qualname__</code> consistently set correctly in the main module, one simple change is proposed to eliminate the problem of dual imports of the main module: the addition of a <code class="docutils literal notranslate">sys.metapath</code> hook that detects attempts to import <code class="docutils literal notranslate">__main__</code> under its real name and returns the original main module instead: <div class="highlight-default notranslate"><div class="highlight"><pre>class AliasImporter: def __init__(self, module, alias): self.module = module self.alias = alias def __repr__(self): fmt = "{0.__class__.__name__}({0.module.__name__}, {0.alias})" return fmt.format(self) def find_module(self, fullname, path=None): if path is None and fullname == self.alias: return self return None def load_module(self, fullname): if fullname != self.alias: raise ImportError("{!r} cannot load {!r}".format(self, fullname)) return self.main_module </pre></div> </div> This metapath hook would be added automatically during import system initialisation based on the following logic: <div class="highlight-default notranslate"><div class="highlight"><pre>main = sys.modules["__main__"] if main.__name__ != main.__qualname__: sys.metapath.append(AliasImporter(main, main.__qualname__)) </pre></div> </div> This is probably the least important proposal in the PEP - it just closes off the last mechanism that is likely to lead to module duplication after the configuration of <code class="docutils literal notranslate">sys.path[0]</code> at interpreter startup is addressed. </section> <section id="fixing-pickling-without-breaking-introspection"> <h3><a class="toc-backref" href="#fixing-pickling-without-breaking-introspection" role="doc-backlink">Fixing pickling without breaking introspection</a></h3> To fix this problem, it is proposed to make use of the new module level <code class="docutils literal notranslate">__qualname__</code> attributes to determine the real module location when <code class="docutils literal notranslate">__name__</code> has been modified for any reason. In the main module, <code class="docutils literal notranslate">__qualname__</code> will automatically be set to the main module’s “real” name (as described above) by the interpreter. Pseudo-modules that adjust <code class="docutils literal notranslate">__name__</code> to point to the public namespace will leave <code class="docutils literal notranslate">__qualname__</code> untouched, so the implementation location remains readily accessible for introspection. If <code class="docutils literal notranslate">__name__</code> is adjusted at the top of a module, then this will automatically adjust the <code class="docutils literal notranslate">__module__</code> attribute for all functions and classes subsequently defined in that module. Since multiple submodules may be set to use the same “public” namespace, functions and classes will be given a new <code class="docutils literal notranslate">__qualmodule__</code> attribute that refers to the <code class="docutils literal notranslate">__qualname__</code> of their module. This isn’t strictly necessary for functions (you could find out their module’s qualified name by looking in their globals dictionary), but it is needed for classes, since they don’t hold a reference to the globals of their defining module. Once a new attribute is added to classes, it is more convenient to keep the API consistent and add a new attribute to functions as well. These changes mean that adjusting <code class="docutils literal notranslate">__name__</code> (and, either directly or indirectly, the corresponding function and class <code class="docutils literal notranslate">__module__</code> attributes) becomes the officially sanctioned way to implement a namespace as a package, while exposing the API as if it were still a single module. All serialisation code that currently uses <code class="docutils literal notranslate">__name__</code> and <code class="docutils literal notranslate">__module__</code> attributes will then avoid exposing implementation details by default. To correctly handle serialisation of items from the main module, the class and function definition logic will be updated to also use <code class="docutils literal notranslate">__qualname__</code> for the <code class="docutils literal notranslate">__module__</code> attribute in the case where <code class="docutils literal notranslate">__name__ == "__main__"</code>. With <code class="docutils literal notranslate">__name__</code> and <code class="docutils literal notranslate">__module__</code> being officially blessed as being used for the public names of things, the introspection tools in the standard library will be updated to use <code class="docutils literal notranslate">__qualname__</code> and <code class="docutils literal notranslate">__qualmodule__</code> where appropriate. For example: <ul class="simple"> <li><code class="docutils literal notranslate">pydoc</code> will report both public and qualified names for modules</li> <li><code class="docutils literal notranslate">inspect.getsource()</code> (and similar tools) will use the qualified names that point to the implementation of the code</li> <li>additional <code class="docutils literal notranslate">pydoc</code> and/or <code class="docutils literal notranslate">inspect</code> APIs may be provided that report all modules with a given public <code class="docutils literal notranslate">__name__</code>.</li> </ul> </section> <section id="fixing-multiprocessing-on-windows"> <h3><a class="toc-backref" href="#fixing-multiprocessing-on-windows" role="doc-backlink">Fixing multiprocessing on Windows</a></h3> With <code class="docutils literal notranslate">__qualname__</code> now available to tell <code class="docutils literal notranslate">multiprocessing</code> the real name of the main module, it will be able to simply include it in the serialised information passed to the child process, eliminating the need for the current dubious introspection of the <code class="docutils literal notranslate">__file__</code> attribute. For older Python versions, <code class="docutils literal notranslate">multiprocessing</code> could be improved by applying the <code class="docutils literal notranslate">split_path_module()</code> algorithm described above when attempting to work out how to execute the main module based on its <code class="docutils literal notranslate">__file__</code> attribute. </section> </section> <section id="explicit-relative-imports"> <h2><a class="toc-backref" href="#explicit-relative-imports" role="doc-backlink">Explicit relative imports</a></h2> This PEP proposes that <code class="docutils literal notranslate">__package__</code> be unconditionally defined in the main module as <code class="docutils literal notranslate">__qualname__.rpartition('.')[0]</code>. Aside from that, it proposes that the behaviour of explicit relative imports be left alone. In particular, if <code class="docutils literal notranslate">__package__</code> is not set in a module when an explicit relative import occurs, the automatically cached value will continue to be derived from <code class="docutils literal notranslate">__name__</code> rather than <code class="docutils literal notranslate">__qualname__</code>. This minimises any backwards incompatibilities with existing code that deliberately manipulates relative imports by adjusting <code class="docutils literal notranslate">__name__</code> rather than setting <code class="docutils literal notranslate">__package__</code> directly. This PEP does not propose that <code class="docutils literal notranslate">__package__</code> be deprecated. While it is technically redundant following the introduction of <code class="docutils literal notranslate">__qualname__</code>, it just isn’t worth the hassle of deprecating it within the lifetime of Python 3.x. </section> <section id="reference-implementation"> <h2><a class="toc-backref" href="#reference-implementation" role="doc-backlink">Reference Implementation</a></h2> None as yet. </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <aside class="footnote-list brackets"> <aside class="footnote brackets" id="id6" role="doc-footnote"> <dt class="label" id="id6">[<a href="#id1">1</a>]</dt> <dd><a class="reference external" href="https://mail.python.org/pipermail/python-ideas/2011-January/008983.html">Module aliases and/or “real names”</a></aside> <aside class="footnote brackets" id="id7" role="doc-footnote"> <dt class="label" id="id7">[2] (<a href='#id2'>1</a>, <a href='#id4'>2</a>) </dt> <dd><a class="reference external" href="https://mail.python.org/pipermail/import-sig/2011-November/000382.html">PEP 395 (Module aliasing) and the namespace PEPs</a></aside> <aside class="footnote brackets" id="id8" role="doc-footnote"> <dt class="label" id="id8">[3] (<a href='#id3'>1</a>, <a href='#id5'>2</a>) </dt> <dd><a class="reference external" href="https://mail.python.org/pipermail/import-sig/2011-November/000397.html">Updated PEP 395 (aka “Implicit Relative Imports Must Die!”)</a></aside> </aside> <ul class="simple"> <li><a class="reference external" href="https://mail.python.org/pipermail/import-sig/2011-November/000403.html">Elaboration of compatibility problems between this PEP and PEP 402</a></li> </ul> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0395.rst">https://github.com/python/peps/blob/main/peps/pep-0395.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0395.rst">2025-02-01 08:59:27 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#pep-withdrawal">PEP Withdrawal</a></li> <li><a class="reference internal" href="#abstract">Abstract</a><ul> <li><a class="reference internal" href="#relationship-with-other-peps">Relationship with Other PEPs</a></li> </ul> </li> <li><a class="reference internal" href="#what-s-in-a-name">What’s in a <code class="docutils literal notranslate">__name__</code>?</a></li> <li><a class="reference internal" href="#traps-for-the-unwary">Traps for the Unwary</a><ul> <li><a class="reference internal" href="#why-are-my-imports-broken">Why are my imports broken?</a></li> <li><a class="reference internal" href="#importing-the-main-module-twice">Importing the main module twice</a></li> <li><a class="reference internal" href="#in-a-bit-of-a-pickle">In a bit of a pickle</a></li> <li><a class="reference internal" href="#where-s-the-source">Where’s the source?</a></li> <li><a class="reference internal" href="#forkless-windows">Forkless Windows</a></li> </ul> </li> <li><a class="reference internal" href="#qualified-names-for-modules">Qualified Names for Modules</a><ul> <li><a class="reference internal" href="#alternative-names">Alternative Names</a></li> </ul> </li> <li><a class="reference internal" href="#eliminating-the-traps">Eliminating the Traps</a><ul> <li><a class="reference internal" href="#fixing-main-module-imports-inside-packages">Fixing main module imports inside packages</a><ul> <li><a class="reference internal" href="#optional-addition-command-line-relative-imports">Optional addition: command line relative imports</a></li> <li><a class="reference internal" href="#compatibility-with-pep-382">Compatibility with PEP 382</a></li> <li><a class="reference internal" href="#incompatibility-with-pep-402">Incompatibility with PEP 402</a></li> <li><a class="reference internal" href="#potential-incompatibilities-with-scripts-stored-in-packages">Potential incompatibilities with scripts stored in packages</a></li> </ul> </li> <li><a class="reference internal" href="#fixing-dual-imports-of-the-main-module">Fixing dual imports of the main module</a></li> <li><a class="reference internal" href="#fixing-pickling-without-breaking-introspection">Fixing pickling without breaking introspection</a></li> <li><a class="reference internal" href="#fixing-multiprocessing-on-windows">Fixing multiprocessing on Windows</a></li> </ul> </li> <li><a class="reference internal" href="#explicit-relative-imports">Explicit relative imports</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0395.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>