CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 662 – Editable installs via virtual wheels | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0662/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 662 – Editable installs via virtual wheels | peps.python.org'> <meta property="og:description" content="This document describes extensions to the build backend and frontend communication (as introduced by PEP 517) to allow projects to be installed in editable mode by introducing virtual wheels."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0662/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This document describes extensions to the build backend and frontend communication (as introduced by PEP 517) to allow projects to be installed in editable mode by introducing virtual wheels."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 662</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 662 – Editable installs via virtual wheels</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Bernát Gábor <gaborjbernat at gmail.com></dd> <dt class="field-even">Sponsor:</dt> <dd class="field-even">Brett Cannon <brett at python.org></dd> <dt class="field-odd">Discussions-To:</dt> <dd class="field-odd"><a class="reference external" href="https://discuss.python.org/t/discuss-tbd-editable-installs-by-gaborbernat/9071">Discourse thread</a></dd> <dt class="field-even">Status:</dt> <dd class="field-even"><abbr title="Formally declined and will not be accepted">Rejected</abbr></dd> <dt class="field-odd">Type:</dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Topic:</dt> <dd class="field-even"><a class="reference external" href="../topic/packaging/">Packaging</a></dd> <dt class="field-odd">Created:</dt> <dd class="field-odd">28-May-2021</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even"></dd> <dt class="field-odd">Resolution:</dt> <dd class="field-odd"><a class="reference external" href="https://discuss.python.org/t/pronouncement-on-peps-660-and-662-editable-installs/9450">Discourse thread</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#terminology-and-goals">Terminology and goals</a></li> <li><a class="reference internal" href="#the-mechanism">The Mechanism</a><ul> <li><a class="reference internal" href="#get-requires-for-build-editable"><code class="docutils literal notranslate">get_requires_for_build_editable</code></a></li> <li><a class="reference internal" href="#prepare-metadata-for-build-editable"><code class="docutils literal notranslate">prepare_metadata_for_build_editable</code></a></li> <li><a class="reference internal" href="#build-editable"><code class="docutils literal notranslate">build_editable</code></a></li> <li><a class="reference internal" href="#build-frontend-requirements">Build frontend requirements</a></li> <li><a class="reference internal" href="#frontend-requirements">Frontend requirements</a></li> <li><a class="reference internal" href="#example-editable-implementations">Example editable implementations</a><ul> <li><a class="reference internal" href="#add-the-source-tree-as-is-to-the-interpreter">Add the source tree as is to the interpreter</a></li> <li><a class="reference internal" href="#using-custom-importers">Using custom importers</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#rejected-ideas">Rejected ideas</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> This document describes extensions to the build backend and frontend communication (as introduced by <a class="pep reference internal" href="../pep-0517/" title="PEP 517 – A build-system independent format for source trees">PEP 517</a>) to allow projects to be installed in editable mode by introducing virtual wheels. </section> <section id="motivation"> <h2><a class="toc-backref" href="#motivation" role="doc-backlink">Motivation</a></h2> During development, many Python users prefer to install their libraries so that changes to the underlying source code and resources are automatically reflected in subsequent interpreter invocations without an additional installation step. This mode is usually called “development mode” or “editable installs”. Currently, there is no standardized way to accomplish this, as it was explicitly left out of <a class="pep reference internal" href="../pep-0517/" title="PEP 517 – A build-system independent format for source trees">PEP 517</a> due to the complexity of the actual observed behaviors. At the moment, users to get this behaviour perform one of the following: <ul class="simple"> <li>For just Python code by adding the relevant source directories to <code class="docutils literal notranslate">sys.path</code> (configurable from the command line interface via the <code class="docutils literal notranslate">PYTHONPATH</code> environment variable). Note in this case, the users have to install the project dependencies themselves, and entry points or project metadata are not generated.</li> <li><a class="reference external" href="https://setuptools.readthedocs.io/en/latest/">setuptools</a> provides the <a class="reference external" href="https://setuptools.readthedocs.io/en/latest/userguide/commands.html#develop-deploy-the-project-source-in-development-mode">setup.py develop</a> mechanism: that installs a <code class="docutils literal notranslate">pth</code> file that injects the project root onto the <code class="docutils literal notranslate">sys.path</code> at interpreter startup time, generates the project metadata, and also installs project dependencies. <a class="reference external" href="https://pip.pypa.io">pip</a> exposes calling this mechanism via the <a class="reference external" href="project_directory">pip install -e</a> command-line interface.</li> <li><a class="reference external" href="https://flit.readthedocs.io/en/latest/index.html">flit</a> provides the <a class="reference external" href="https://flit.readthedocs.io/en/latest/cmdline.html#cmdoption-flit-install-s">flit install –symlink</a> command that symlinks the project files into the interpreters <code class="docutils literal notranslate">purelib</code> folder, generates the project metadata, and also installs dependencies. Note, this allows supporting resource files too.</li> </ul> As these examples shows an editable install can be achieved in multiple ways and at the moment there’s no standard way of doing it. Furthermore, it’s not clear whose responsibility it is to achieve and define what an editable installation is: <ol class="arabic simple"> <li>allow the build backend to define and materialize it,</li> <li>allow the build frontend to define and materialize it,</li> <li>explicitly define and standardize one method from the possible options.</li> </ol> The author of this PEP believes there’s no one size fits all solution here, each method of achieving editable effect has its pros and cons. Therefore this PEP rejects option three as it’s unlikely for the community to agree on a single solution. Furthermore, question remains as to whether the frontend or the build backend should own this responsibility. <a class="pep reference internal" href="../pep-0660/" title="PEP 660 – Editable installs for pyproject.toml based builds (wheel based)">PEP 660</a> proposes the build backend to own this, while the current PEP proposes primarily the frontend, but still allows the backend to take take control if it wants to do so. </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> <a class="pep reference internal" href="../pep-0517/" title="PEP 517 – A build-system independent format for source trees">PEP 517</a> deferred “Editable installs” because this would have delayed further its adoption, and there wasn’t an agreement on how editable installs should be achieved. Due to the popularity of the <a class="reference external" href="https://setuptools.readthedocs.io/en/latest/">setuptools</a> and <a class="reference external" href="https://pip.pypa.io">pip</a> projects, the status quo prevailed, and the backend could achieve editable mode by providing a <code class="docutils literal notranslate">setup.py develop</code> implementation, which the user could trigger via <a class="reference external" href="project_directory">pip install -e</a>. By defining an editable interface between the build backend and frontend, we can eliminate the <code class="docutils literal notranslate">setup.py</code> file and their current communication method. </section> <section id="terminology-and-goals"> <h2><a class="toc-backref" href="#terminology-and-goals" role="doc-backlink">Terminology and goals</a></h2> This PEP aims to delineate the frontend and the backend roles clearly and give the developers of each the maximum ability to provide valuable features to their users. In this proposal, the backend’s role is to prepare the project for an editable installation, and then provide enough information to the frontend so that the frontend can manifest and enforce the editable installation. The information the backend provides to the frontend is a wheel that follows the existing specification within <a class="pep reference internal" href="../pep-0427/" title="PEP 427 – The Wheel Binary Package Format 1.0">PEP 427</a>. The wheel metadata about the archive itself (<code class="docutils literal notranslate">{distribution}-{version}.dist-info/WHEEL</code>) must also contain the key <code class="docutils literal notranslate">Editable</code> with value of <code class="docutils literal notranslate">true</code>. However, instead of providing the project files within the wheel, it must provide an <code class="docutils literal notranslate">editable.json</code> file (at the root level of the wheel) that defines the files to be exposed by the frontend. The content of this file is formulated as a mapping of absolute source tree paths to relative target interpreter destination paths within a scheme mapping. A wheel that satisfies the previous two paragraphs is a virtual wheel. The frontend’s role is to take the virtual wheel and install the project in editable mode. The way it achieves this is entirely up to the frontend and is considered implementation detail. The editable installation mode implies that the source code of the project being installed is available in a local directory. Once the project is installed in editable mode, some changes to the project code in the local source tree will become effective without the need for a new installation step. At a minimum, changes to the text of non-generated files that existed at the installation time should be reflected upon the subsequent import of the package. Some kinds of changes, such as adding or modifying entry points or new dependencies, require a new installation step to become effective. These changes are typically made in build backend configuration files (such as <code class="docutils literal notranslate">pyproject.toml</code>). This requirement is consistent with the general user expectation that such modifications will only become effective after re-installation. While users expect editable installations to behave identically to standard installations, this may not always be possible and may be in tension with other user expectations. Depending on how a frontend implements the editable mode, some differences may be visible, such as the presence of additional files (compared to a typical installation), either in the source tree or the interpreter’s installation path. Frontends should seek to minimize differences between the behavior of editable and standard installations and document known differences. For reference, a non-editable installation works as follows: <ol class="arabic simple"> <li>The developer is using a tool, we’ll call it here the frontend, to drive the project development (e.g., <a class="reference external" href="https://pip.pypa.io">pip</a>). When the user wants to trigger a package build and installation of a project, they’ll communicate with the frontend.</li> <li>The frontend uses a build frontend to trigger the build of a wheel (e.g., <a class="reference external" href="https://pypa-build.readthedocs.io">build</a>). The build frontend uses <a class="pep reference internal" href="../pep-0517/" title="PEP 517 – A build-system independent format for source trees">PEP 517</a> to communicate with the build backend (e.g. <a class="reference external" href="https://setuptools.readthedocs.io/en/latest/">setuptools</a>) - with the build backend installed into a <a class="pep reference internal" href="../pep-0518/" title="PEP 518 – Specifying Minimum Build System Requirements for Python Projects">PEP 518</a> environment. Once invoked, the backend returns a wheel.</li> <li>The frontend takes the wheel and feeds it to an installer (e.g., <a class="reference external" href="https://pypi.org/project/installer">installer</a>) to install the wheel into the target Python interpreter.</li> </ol> </section> <section id="the-mechanism"> <h2><a class="toc-backref" href="#the-mechanism" role="doc-backlink">The Mechanism</a></h2> This PEP adds two optional hooks to the <a class="pep reference internal" href="../pep-0517/" title="PEP 517 – A build-system independent format for source trees">PEP 517</a> backend interface. One of the hooks is used to specify the build dependencies of an editable install. The other hook returns the necessary information via the build frontend the frontend needs to create an editable install. <section id="get-requires-for-build-editable"> <h3><a class="toc-backref" href="#get-requires-for-build-editable" role="doc-backlink"><code class="docutils literal notranslate">get_requires_for_build_editable</code></a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre>def get_requires_for_build_editable(config_settings=None): ... </pre></div> </div> This hook MUST return an additional sequence of strings containing <a class="pep reference internal" href="../pep-0508/" title="PEP 508 – Dependency specification for Python Software Packages">PEP 508</a> dependency specifications, above and beyond those specified in the <code class="docutils literal notranslate">pyproject.toml</code> file. The frontend must ensure that these dependencies are available in the build environment in which the <code class="docutils literal notranslate">build_editable</code> hook is called. If not defined, the default implementation is equivalent to returning <code class="docutils literal notranslate">[]</code>. </section> <section id="prepare-metadata-for-build-editable"> <h3><a class="toc-backref" href="#prepare-metadata-for-build-editable" role="doc-backlink"><code class="docutils literal notranslate">prepare_metadata_for_build_editable</code></a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre>def prepare_metadata_for_build_editable(metadata_directory, config_settings=None): ... </pre></div> </div> Must create a <code class="docutils literal notranslate">.dist-info</code> directory containing wheel metadata inside the specified <code class="docutils literal notranslate">metadata_directory</code> (i.e., creates a directory like <code class="docutils literal notranslate">{metadata_directory}/{package}-{version}.dist-info/</code>). This directory MUST be a valid <code class="docutils literal notranslate">.dist-info</code> directory as defined in the wheel specification, except that it need not contain <code class="docutils literal notranslate">RECORD</code> or signatures. The hook MAY also create other files inside this directory, and a build frontend MUST preserve, but otherwise ignore, such files; the intention here is that in cases where the metadata depends on build-time decisions, the build backend may need to record these decisions in some convenient format for re-use by the actual wheel-building step. This must return the basename (not the full path) of the <code class="docutils literal notranslate">.dist-info</code> directory it creates, as a unicode string. If a build frontend needs this information and the method is not defined, it should call <code class="docutils literal notranslate">build_editable</code> and look at the resulting metadata directly. </section> <section id="build-editable"> <h3><a class="toc-backref" href="#build-editable" role="doc-backlink"><code class="docutils literal notranslate">build_editable</code></a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre>def build_editable(self, wheel_directory, config_settings=None, metadata_directory=None): ... </pre></div> </div> Must build a .whl file, and place it in the specified <code class="docutils literal notranslate">wheel_directory</code>. It must return the basename (not the full path) of the <code class="docutils literal notranslate">.whl</code> file it creates, as a unicode string. The wheel file must be of type virtual wheel as defined under the terminology section. If the build frontend has previously called <code class="docutils literal notranslate">prepare_metadata_for_build_editable</code> and depends on the wheel resulting from this call to have metadata matching this earlier call, then it should provide the path to the created <code class="docutils literal notranslate">.dist-info</code> directory as the <code class="docutils literal notranslate">metadata_directory</code> argument. If this argument is provided, then <code class="docutils literal notranslate">build_editable</code> MUST produce a wheel with identical metadata. The directory passed in by the build frontend MUST be identical to the directory created by <code class="docutils literal notranslate">prepare_metadata_for_build_editable</code>, including any unrecognized files it created. Backends which do not provide the <code class="docutils literal notranslate">prepare_metadata_for_build_editable</code> hook may either silently ignore the <code class="docutils literal notranslate">metadata_directory</code> parameter to <code class="docutils literal notranslate">build_editable</code>, or else raise an exception when it is set to anything other than <code class="docutils literal notranslate">None</code>. The source directory may be read-only, in such cases the backend may raise an error that the frontend can display to the user. The backend may store intermediate artifacts in cache locations or temporary directories. The presence or absence of any caches should not make a material difference to the final result of the build. The content of the <code class="docutils literal notranslate">editable.json</code> MUST pass against the following JSON schema: <div class="code highlight-default notranslate"><div class="highlight"><pre>{ "$schema": "http://json-schema.org/draft-07/schema", "$id": "http://pypa.io/editables.json", "type": "object", "title": "Virtual wheel editable schema.", "required": ["version", "scheme"], "properties": { "version": { "$id": "#/properties/version", "type": "integer", "minimum": 1, "maximum": 1, "title": "The version of the schema." }, "scheme": { "$id": "#/properties/scheme", "type": "object", "title": "Files to expose.", "required": ["purelib", "platlib", "data", "headers", "scripts"], "properties": { "purelib": { "$ref": "#/$defs/mapping" }, "platlib": { "$ref": "#/$defs/mapping" }, "data": { "$ref": "#/$defs/mapping" }, "headers": { "$ref": "#/$defs/mapping" }, "scripts": { "$ref": "#/$defs/mapping" } }, "additionalProperties": true } }, "additionalProperties": true, "$defs": { "mapping": { "type": "object", "description": "A mapping of source to target paths. The source is absolute path, the destination is relative path.", "additionalProperties": true } } } </pre></div> </div> For example: <div class="highlight-default notranslate"><div class="highlight"><pre>{ "version": 1, "scheme": { "purelib": {"/src/tree/a.py": "tree/a.py"}, "platlib": {}, "data": {"/src/tree/py.typed": "tree/py.typed"}, "headers": {}, "scripts": {} } } </pre></div> </div> The scheme paths map from project source absolute paths to target directory relative paths. We allow backends to change the project layout from the project source directory to what the interpreter will see by using the mapping. For example if the backend returns <code class="docutils literal notranslate">"purelib": {"/me/project/src": ""}</code> this would mean that expose all files and modules within <code class="docutils literal notranslate">/me/project/src</code> at the root of the <code class="docutils literal notranslate">purelib</code> path within the target interpreter. </section> <section id="build-frontend-requirements"> <h3><a class="toc-backref" href="#build-frontend-requirements" role="doc-backlink">Build frontend requirements</a></h3> The build frontend is responsible for setting up the environment for the build backend to generate the virtual wheel. All recommendations from <a class="pep reference internal" href="../pep-0517/" title="PEP 517 – A build-system independent format for source trees">PEP 517</a> for the build wheel hook applies here too. </section> <section id="frontend-requirements"> <h3><a class="toc-backref" href="#frontend-requirements" role="doc-backlink">Frontend requirements</a></h3> The frontend must install the virtual wheel exactly as defined within <a class="pep reference internal" href="../pep-0427/" title="PEP 427 – The Wheel Binary Package Format 1.0">PEP 427</a>. Furthermore is responsible for also installing the files defined within the <code class="docutils literal notranslate">editable.json</code> file. The manner in which it does is left up to the frontend, and is encouraged for the frontend to communicate with the user exactly the method chosen, and what limitations that solution will have. The frontend must create a <code class="docutils literal notranslate">direct_url.json</code> file in the <code class="docutils literal notranslate">.dist-info</code> directory of the installed distribution, in compliance with <a class="pep reference internal" href="../pep-0610/" title="PEP 610 – Recording the Direct URL Origin of installed distributions">PEP 610</a>. The <code class="docutils literal notranslate">url</code> value must be a <code class="docutils literal notranslate">file://</code> URL pointing to the project directory (i.e., the directory containing <code class="docutils literal notranslate">pyproject.toml</code>), and the <code class="docutils literal notranslate">dir_info</code> value must be <code class="docutils literal notranslate">{'editable': true}</code>. The frontend can rely on the <code class="docutils literal notranslate">prepare_metadata_for_build_editable</code> hook when installing in editable mode. If the frontend concludes it cannot achieve an editable installation with the information provided by the build backend it should fail and raise an error to clarify to the user why not. The frontend might implement one or more editable installation mechanisms and can leave it up to the user the choose one that its optimal to the use case of the user. For example, pip could add an editable mode flag, and allow the user to choose between <code class="docutils literal notranslate">pth</code> files or symlinks ( <code class="docutils literal notranslate">pip install -e . --editable-mode=pth</code> vs <code class="docutils literal notranslate">pip install -e . --editable-mode=symlink</code>). </section> <section id="example-editable-implementations"> <h3><a class="toc-backref" href="#example-editable-implementations" role="doc-backlink">Example editable implementations</a></h3> To show how this PEP might be used, we’ll now present a few case studies. Note the offered solutions are purely for illustration purpose and are not normative for the frontend/backend. <section id="add-the-source-tree-as-is-to-the-interpreter"> <h4><a class="toc-backref" href="#add-the-source-tree-as-is-to-the-interpreter" role="doc-backlink">Add the source tree as is to the interpreter</a></h4> This is one of the simplest implementations, it will add the source tree as is into the interpreters scheme paths, the <code class="docutils literal notranslate">editable.json</code> within the virtual wheel might look like: <div class="highlight-default notranslate"><div class="highlight"><pre>{ {"version": 1, "scheme": {"purelib": {"<project dir>": "<project dir>"}}} } </pre></div> </div> The frontend then could either: <ul class="simple"> <li>Add the source directory onto the target interpreters <code class="docutils literal notranslate">sys.path</code> during startup of it. This is done by creating a <code class="docutils literal notranslate">pth</code> file into the target interpreters <code class="docutils literal notranslate">purelib</code> folder. <a class="reference external" href="https://setuptools.readthedocs.io/en/latest/">setuptools</a> does this today and is what <a class="reference external" href="project_directory">pip install -e</a> translate too. This solution is fast and cross-platform compatible. However, this puts the entire source tree onto the system, potentially exposing modules that would not be available in a standard installation case.</li> <li>Symlink the folder, or the individual files within it. This method is what flit does via its <a class="reference external" href="https://flit.readthedocs.io/en/latest/cmdline.html#cmdoption-flit-install-s">flit install –symlink</a>. This solution requires the current platform to support symlinks. Still, it allows potentially to symlink individual files, which could solve the problem of including files that should be excluded from the source tree.</li> </ul> </section> <section id="using-custom-importers"> <h4><a class="toc-backref" href="#using-custom-importers" role="doc-backlink">Using custom importers</a></h4> For a more robust and more dynamic collaboration between the build backend and the target interpreter, we can take advantage of the import system allowing the registration of custom importers. See <a class="pep reference internal" href="../pep-0302/" title="PEP 302 – New Import Hooks">PEP 302</a> for more details and <a class="reference external" href="https://pypi.org/project/editables">editables</a> as an example of this. The backend can generate a new importer during the editable build (or install it as an additional dependency) and register it at interpreter startup by adding a <code class="docutils literal notranslate">pth</code> file. <div class="highlight-default notranslate"><div class="highlight"><pre>{ "version": 1, "scheme": { "purelib": { "<project dir>/.editable/_register_importer.pth": "<project dir>/_register_importer.pth". "<project dir>/.editable/_editable_importer.py": "<project dir>/_editable_importer.py" } } } } </pre></div> </div> The backend here registered a hook that is called whenever a new module is imported, allowing dynamic and on-demand functionality. Potential use cases where this is useful: <ul class="simple"> <li>Expose a source folder, but honor module excludes: the backend may generate an import hook that consults the exclusion table before allowing a source file loader to discover a file in the source directory or not.</li> <li>For a project, let there be two modules, <code class="docutils literal notranslate">A.py</code> and <code class="docutils literal notranslate">B.py</code>. These are two separate files in the source directory; however, while building a wheel, they are merged into one mega file <code class="docutils literal notranslate">project.py</code>. In this case, with this PEP, the backend could generate an import hook that reads the source files at import time and merges them in memory before materializing it as a module.</li> <li>Automatically update out-of-date C-extensions: the backend may generate an import hook that checks the last modified timestamp for a C-extension source file. If it is greater than the current C-extension binary, trigger an update by calling the compiler before import.</li> </ul> </section> </section> </section> <section id="rejected-ideas"> <h2><a class="toc-backref" href="#rejected-ideas" role="doc-backlink">Rejected ideas</a></h2> This PEP competes with <a class="pep reference internal" href="../pep-0660/" title="PEP 660 – Editable installs for pyproject.toml based builds (wheel based)">PEP 660</a> and rejects that proposal because we think the mechanism of achieving an editable installation should be within the frontend rather than the build backend. Furthermore, this approach allows the ecosystem to use alternative means to accomplish the editable installation effect (e.g., insert path on <code class="docutils literal notranslate">sys.path</code> or symlinks instead of just implying the loose wheel mode from the backend described by that PEP). Prominently, <a class="pep reference internal" href="../pep-0660/" title="PEP 660 – Editable installs for pyproject.toml based builds (wheel based)">PEP 660</a> does not allow using symlinks to expose code and data files without also extending the wheel file standard with symlink support. It’s not clear how the wheel format could be extended to support symlinks that refer not to files within the wheel itself, but files only available on the local disk. It’s important to note that the backend itself (or backend generated code) must not generate these symlinks (e.g., at interpreter startup time) as that would conflict with the frontends book keeping of what files need to be uninstalled. Finally, <a class="pep reference internal" href="../pep-0660/" title="PEP 660 – Editable installs for pyproject.toml based builds (wheel based)">PEP 660</a> adds support only for <code class="docutils literal notranslate">purelib</code> and <code class="docutils literal notranslate">platlib</code> files. It purposefully avoids supporting other types of information that the wheel format supports: <code class="docutils literal notranslate">include</code>, <code class="docutils literal notranslate">data</code> and <code class="docutils literal notranslate">scripts</code>. With this path the frontend can support these on a best effort basis via the symlinks mechanism (though this feature is not universally available - on Windows require enablement). We believe its beneficial to add best effort support for these file types, rather than exclude the possibility of supporting them at all. </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0662.rst">https://github.com/python/peps/blob/main/peps/pep-0662.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0662.rst">2023-09-09 17:39:29 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#terminology-and-goals">Terminology and goals</a></li> <li><a class="reference internal" href="#the-mechanism">The Mechanism</a><ul> <li><a class="reference internal" href="#get-requires-for-build-editable"><code class="docutils literal notranslate">get_requires_for_build_editable</code></a></li> <li><a class="reference internal" href="#prepare-metadata-for-build-editable"><code class="docutils literal notranslate">prepare_metadata_for_build_editable</code></a></li> <li><a class="reference internal" href="#build-editable"><code class="docutils literal notranslate">build_editable</code></a></li> <li><a class="reference internal" href="#build-frontend-requirements">Build frontend requirements</a></li> <li><a class="reference internal" href="#frontend-requirements">Frontend requirements</a></li> <li><a class="reference internal" href="#example-editable-implementations">Example editable implementations</a><ul> <li><a class="reference internal" href="#add-the-source-tree-as-is-to-the-interpreter">Add the source tree as is to the interpreter</a></li> <li><a class="reference internal" href="#using-custom-importers">Using custom importers</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#rejected-ideas">Rejected ideas</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0662.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>