CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 517 – A build-system independent format for source trees | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0517/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 517 – A build-system independent format for source trees | peps.python.org'> <meta property="og:description" content="While distutils / setuptools have taken us a long way, they suffer from three serious problems: (a) they’re missing important features like usable build-time dependency declaration, autoconfiguration, and even basic ergonomic niceties like DRY-compliant..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0517/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="While distutils / setuptools have taken us a long way, they suffer from three serious problems: (a) they’re missing important features like usable build-time dependency declaration, autoconfiguration, and even basic ergonomic niceties like DRY-compliant..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 517</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 517 – A build-system independent format for source trees</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Nathaniel J. Smith <njs at pobox.com>, Thomas Kluyver <thomas at kluyver.me.uk></dd> <dt class="field-even">BDFL-Delegate:</dt> <dd class="field-even">Alyssa Coghlan <ncoghlan at gmail.com></dd> <dt class="field-odd">Discussions-To:</dt> <dd class="field-odd"><a class="reference external" href="https://mail.python.org/archives/list/distutils-sig@python.org/">Distutils-SIG list</a></dd> <dt class="field-even">Status:</dt> <dd class="field-even"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-odd">Type:</dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Topic:</dt> <dd class="field-even"><a class="reference external" href="../topic/packaging/">Packaging</a></dd> <dt class="field-odd">Created:</dt> <dd class="field-odd">30-Sep-2015</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even">01-Oct-2015, 25-Oct-2015, 19-May-2017, 11-Sep-2017</dd> <dt class="field-odd">Resolution:</dt> <dd class="field-odd"><a class="reference external" href="https://mail.python.org/pipermail/distutils-sig/2017-September/031548.html">Distutils-SIG message</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#terminology-and-goals">Terminology and goals</a></li> <li><a class="reference internal" href="#source-trees">Source trees</a><ul> <li><a class="reference internal" href="#build-requirements">Build requirements</a></li> </ul> </li> <li><a class="reference internal" href="#build-backend-interface">Build backend interface</a><ul> <li><a class="reference internal" href="#mandatory-hooks">Mandatory hooks</a><ul> <li><a class="reference internal" href="#build-wheel">build_wheel</a></li> <li><a class="reference internal" href="#build-sdist">build_sdist</a></li> </ul> </li> <li><a class="reference internal" href="#optional-hooks">Optional hooks</a><ul> <li><a class="reference internal" href="#get-requires-for-build-wheel">get_requires_for_build_wheel</a></li> <li><a class="reference internal" href="#prepare-metadata-for-build-wheel">prepare_metadata_for_build_wheel</a></li> <li><a class="reference internal" href="#get-requires-for-build-sdist">get_requires_for_build_sdist</a></li> </ul> </li> <li><a class="reference internal" href="#config-settings">Config settings</a></li> <li><a class="reference internal" href="#build-environment">Build environment</a><ul> <li><a class="reference internal" href="#recommendations-for-build-frontends-non-normative">Recommendations for build frontends (non-normative)</a></li> </ul> </li> <li><a class="reference internal" href="#in-tree-build-backends">In-tree build backends</a></li> </ul> </li> <li><a class="reference internal" href="#source-distributions">Source distributions</a></li> <li><a class="reference internal" href="#evolutionary-notes">Evolutionary notes</a></li> <li><a class="reference internal" href="#rejected-options">Rejected options</a></li> <li><a class="reference internal" href="#summary-of-changes-to-pep-517">Summary of changes to PEP 517</a></li> <li><a class="reference internal" href="#appendix-a-comparison-to-pep-516">Appendix A: Comparison to PEP 516</a><ul> <li><a class="reference internal" href="#other-differences">Other differences</a></li> </ul> </li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> While <code class="docutils literal notranslate">distutils</code> / <code class="docutils literal notranslate">setuptools</code> have taken us a long way, they suffer from three serious problems: (a) they’re missing important features like usable build-time dependency declaration, autoconfiguration, and even basic ergonomic niceties like <a class="reference external" href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself">DRY</a>-compliant version number management, and (b) extending them is difficult, so while there do exist various solutions to the above problems, they’re often quirky, fragile, and expensive to maintain, and yet (c) it’s very difficult to use anything else, because distutils/setuptools provide the standard interface for installing packages expected by both users and installation tools like <code class="docutils literal notranslate">pip</code>. Previous efforts (e.g. distutils2 or setuptools itself) have attempted to solve problems (a) and/or (b). This proposal aims to solve (c). The goal of this PEP is get distutils-sig out of the business of being a gatekeeper for Python build systems. If you want to use distutils, great; if you want to use something else, then that should be easy to do using standardized methods. The difficulty of interfacing with distutils means that there aren’t many such systems right now, but to give a sense of what we’re thinking about see <a class="reference external" href="https://github.com/takluyver/flit">flit</a> or <a class="reference external" href="https://cournape.github.io/Bento/">bento</a>. Fortunately, wheels have now solved many of the hard problems here – e.g. it’s no longer necessary that a build system also know about every possible installation configuration – so pretty much all we really need from a build system is that it have some way to spit out standard-compliant wheels and sdists. We therefore propose a new, relatively minimal interface for installation tools like <code class="docutils literal notranslate">pip</code> to interact with package source trees and source distributions. </section> <section id="terminology-and-goals"> <h2><a class="toc-backref" href="#terminology-and-goals" role="doc-backlink">Terminology and goals</a></h2> A source tree is something like a VCS checkout. We need a standard interface for installing from this format, to support usages like <code class="docutils literal notranslate">pip install some-directory/</code>. A source distribution is a static snapshot representing a particular release of some source code, like <code class="docutils literal notranslate">lxml-3.4.4.tar.gz</code>. Source distributions serve many purposes: they form an archival record of releases, they provide a stupid-simple de facto standard for tools that want to ingest and process large corpora of code, possibly written in many languages (e.g. code search), they act as the input to downstream packaging systems like Debian/Fedora/Conda/…, and so forth. In the Python ecosystem they additionally have a particularly important role to play, because packaging tools like <code class="docutils literal notranslate">pip</code> are able to use source distributions to fulfill binary dependencies, e.g. if there is a distribution <code class="docutils literal notranslate">foo.whl</code> which declares a dependency on <code class="docutils literal notranslate">bar</code>, then we need to support the case where <code class="docutils literal notranslate">pip install bar</code> or <code class="docutils literal notranslate">pip install foo</code> automatically locates the sdist for <code class="docutils literal notranslate">bar</code>, downloads it, builds it, and installs the resulting package. Source distributions are also known as sdists for short. A build frontend is a tool that users might run that takes arbitrary source trees or source distributions and builds wheels from them. The actual building is done by each source tree’s build backend. In a command like <code class="docutils literal notranslate">pip wheel some-directory/</code>, pip is acting as a build frontend. An integration frontend is a tool that users might run that takes a set of package requirements (e.g. a requirements.txt file) and attempts to update a working environment to satisfy those requirements. This may require locating, building, and installing a combination of wheels and sdists. In a command like <code class="docutils literal notranslate">pip install lxml==2.4.0</code>, pip is acting as an integration frontend. </section> <section id="source-trees"> <h2><a class="toc-backref" href="#source-trees" role="doc-backlink">Source trees</a></h2> There is an existing, legacy source tree format involving <code class="docutils literal notranslate">setup.py</code>. We don’t try to specify it further; its de facto specification is encoded in the source code and documentation of <code class="docutils literal notranslate">distutils</code>, <code class="docutils literal notranslate">setuptools</code>, <code class="docutils literal notranslate">pip</code>, and other tools. We’ll refer to it as the <code class="docutils literal notranslate">setup.py</code>-style. Here we define a new style of source tree based around the <code class="docutils literal notranslate">pyproject.toml</code> file defined in <a class="pep reference internal" href="../pep-0518/" title="PEP 518 – Specifying Minimum Build System Requirements for Python Projects">PEP 518</a>, extending the <code class="docutils literal notranslate">[build-system]</code> table in that file with one additional key, <code class="docutils literal notranslate">build-backend</code>. Here’s an example of how it would look: <div class="highlight-default notranslate"><div class="highlight"><pre>[build-system] # Defined by PEP 518: requires = ["flit"] # Defined by this PEP: build-backend = "flit.api:main" </pre></div> </div> <code class="docutils literal notranslate">build-backend</code> is a string naming a Python object that will be used to perform the build (see below for details). This is formatted following the same <code class="docutils literal notranslate">module:object</code> syntax as a <code class="docutils literal notranslate">setuptools</code> entry point. For instance, if the string is <code class="docutils literal notranslate">"flit.api:main"</code> as in the example above, this object would be looked up by executing the equivalent of: <div class="highlight-default notranslate"><div class="highlight"><pre>import flit.api backend = flit.api.main </pre></div> </div> It’s also legal to leave out the <code class="docutils literal notranslate">:object</code> part, e.g. <div class="highlight-default notranslate"><div class="highlight"><pre>build-backend = "flit.api" </pre></div> </div> which acts like: <div class="highlight-default notranslate"><div class="highlight"><pre>import flit.api backend = flit.api </pre></div> </div> Formally, the string should satisfy this grammar: <div class="highlight-default notranslate"><div class="highlight"><pre>identifier = (letter | '_') (letter | '_' | digit)* module_path = identifier ('.' identifier)* object_path = identifier ('.' identifier)* entry_point = module_path (':' object_path)? </pre></div> </div> And we import <code class="docutils literal notranslate">module_path</code> and then lookup <code class="docutils literal notranslate">module_path.object_path</code> (or just <code class="docutils literal notranslate">module_path</code> if <code class="docutils literal notranslate">object_path</code> is missing). When importing the module path, we do not look in the directory containing the source tree, unless that would be on <code class="docutils literal notranslate">sys.path</code> anyway (e.g. because it is specified in PYTHONPATH). Although Python automatically adds the working directory to <code class="docutils literal notranslate">sys.path</code> in some situations, code to resolve the backend should not be affected by this. If the <code class="docutils literal notranslate">pyproject.toml</code> file is absent, or the <code class="docutils literal notranslate">build-backend</code> key is missing, the source tree is not using this specification, and tools should revert to the legacy behaviour of running <code class="docutils literal notranslate">setup.py</code> (either directly, or by implicitly invoking the <code class="docutils literal notranslate">setuptools.build_meta:__legacy__</code> backend). Where the <code class="docutils literal notranslate">build-backend</code> key exists, this takes precedence and the source tree follows the format and conventions of the specified backend (as such no <code class="docutils literal notranslate">setup.py</code> is needed unless the backend requires it). Projects may still wish to include a <code class="docutils literal notranslate">setup.py</code> for compatibility with tools that do not use this spec. This PEP also defines a <code class="docutils literal notranslate">backend-path</code> key for use in <code class="docutils literal notranslate">pyproject.toml</code>, see the “In-Tree Build Backends” section below. This key would be used as follows: <div class="highlight-default notranslate"><div class="highlight"><pre>[build-system] # Defined by PEP 518: requires = ["flit"] # Defined by this PEP: build-backend = "local_backend" backend-path = ["backend"] </pre></div> </div> <section id="build-requirements"> <h3><a class="toc-backref" href="#build-requirements" role="doc-backlink">Build requirements</a></h3> This PEP places a number of additional requirements on the “build requirements” section of <code class="docutils literal notranslate">pyproject.toml</code>. These are intended to ensure that projects do not create impossible to satisfy conditions with their build requirements. <ul class="simple"> <li>Project build requirements will define a directed graph of requirements (project A needs B to build, B needs C and D, etc.) This graph MUST NOT contain cycles. If (due to lack of co-ordination between projects, for example) a cycle is present, front ends MAY refuse to build the project.</li> <li>Where build requirements are available as wheels, front ends SHOULD use these where practical, to avoid deeply nested builds. However front ends MAY have modes where they do not consider wheels when locating build requirements, and so projects MUST NOT assume that publishing wheels is sufficient to break a requirement cycle.</li> <li>Front ends SHOULD check explicitly for requirement cycles, and terminate the build with an informative message if one is found.</li> </ul> Note in particular that the requirement for no requirement cycles means that backends wishing to self-host (i.e., building a wheel for a backend uses that backend for the build) need to make special provision to avoid causing cycles. Typically this will involve specifying themselves as an in-tree backend, and avoiding external build dependencies (usually by vendoring them). </section> </section> <section id="build-backend-interface"> <h2><a class="toc-backref" href="#build-backend-interface" role="doc-backlink">Build backend interface</a></h2> The build backend object is expected to have attributes which provide some or all of the following hooks. The common <code class="docutils literal notranslate">config_settings</code> argument is described after the individual hooks. <section id="mandatory-hooks"> <h3><a class="toc-backref" href="#mandatory-hooks" role="doc-backlink">Mandatory hooks</a></h3> <section id="build-wheel"> <h4><a class="toc-backref" href="#build-wheel" role="doc-backlink">build_wheel</a></h4> <div class="highlight-default notranslate"><div class="highlight"><pre>def build_wheel(wheel_directory, config_settings=None, metadata_directory=None): ... </pre></div> </div> Must build a .whl file, and place it in the specified <code class="docutils literal notranslate">wheel_directory</code>. It must return the basename (not the full path) of the <code class="docutils literal notranslate">.whl</code> file it creates, as a unicode string. If the build frontend has previously called <code class="docutils literal notranslate">prepare_metadata_for_build_wheel</code> and depends on the wheel resulting from this call to have metadata matching this earlier call, then it should provide the path to the created <code class="docutils literal notranslate">.dist-info</code> directory as the <code class="docutils literal notranslate">metadata_directory</code> argument. If this argument is provided, then <code class="docutils literal notranslate">build_wheel</code> MUST produce a wheel with identical metadata. The directory passed in by the build frontend MUST be identical to the directory created by <code class="docutils literal notranslate">prepare_metadata_for_build_wheel</code>, including any unrecognized files it created. Backends which do not provide the <code class="docutils literal notranslate">prepare_metadata_for_build_wheel</code> hook may either silently ignore the <code class="docutils literal notranslate">metadata_directory</code> parameter to <code class="docutils literal notranslate">build_wheel</code>, or else raise an exception when it is set to anything other than <code class="docutils literal notranslate">None</code>. To ensure that wheels from different sources are built the same way, frontends may call <code class="docutils literal notranslate">build_sdist</code> first, and then call <code class="docutils literal notranslate">build_wheel</code> in the unpacked sdist. But if the backend indicates that it is missing some requirements for creating an sdist (see below), the frontend will fall back to calling <code class="docutils literal notranslate">build_wheel</code> in the source directory. The source directory may be read-only. Backends should therefore be prepared to build without creating or modifying any files in the source directory, but they may opt not to handle this case, in which case failures will be visible to the user. Frontends are not responsible for any special handling of read-only source directories. The backend may store intermediate artifacts in cache locations or temporary directories. The presence or absence of any caches should not make a material difference to the final result of the build. </section> <section id="build-sdist"> <h4><a class="toc-backref" href="#build-sdist" role="doc-backlink">build_sdist</a></h4> <div class="highlight-default notranslate"><div class="highlight"><pre>def build_sdist(sdist_directory, config_settings=None): ... </pre></div> </div> Must build a .tar.gz source distribution and place it in the specified <code class="docutils literal notranslate">sdist_directory</code>. It must return the basename (not the full path) of the <code class="docutils literal notranslate">.tar.gz</code> file it creates, as a unicode string. A .tar.gz source distribution (sdist) contains a single top-level directory called <code class="docutils literal notranslate">{name}-{version}</code> (e.g. <code class="docutils literal notranslate">foo-1.0</code>), containing the source files of the package. This directory must also contain the <code class="docutils literal notranslate">pyproject.toml</code> from the build directory, and a PKG-INFO file containing metadata in the format described in <a class="pep reference internal" href="../pep-0345/" title="PEP 345 – Metadata for Python Software Packages 1.2">PEP 345</a>. Although historically zip files have also been used as sdists, this hook should produce a gzipped tarball. This is already the more common format for sdists, and having a consistent format makes for simpler tooling. The generated tarball should use the modern POSIX.1-2001 pax tar format, which specifies UTF-8 based file names. This is not yet the default for the tarfile module shipped with Python 3.6, so backends using the tarfile module need to explicitly pass <code class="docutils literal notranslate">format=tarfile.PAX_FORMAT</code>. Some backends may have extra requirements for creating sdists, such as version control tools. However, some frontends may prefer to make intermediate sdists when producing wheels, to ensure consistency. If the backend cannot produce an sdist because a dependency is missing, or for another well understood reason, it should raise an exception of a specific type which it makes available as <code class="docutils literal notranslate">UnsupportedOperation</code> on the backend object. If the frontend gets this exception while building an sdist as an intermediate for a wheel, it should fall back to building a wheel directly. The backend does not need to define this exception type if it would never raise it. </section> </section> <section id="optional-hooks"> <h3><a class="toc-backref" href="#optional-hooks" role="doc-backlink">Optional hooks</a></h3> <section id="get-requires-for-build-wheel"> <h4><a class="toc-backref" href="#get-requires-for-build-wheel" role="doc-backlink">get_requires_for_build_wheel</a></h4> <div class="highlight-default notranslate"><div class="highlight"><pre>def get_requires_for_build_wheel(config_settings=None): ... </pre></div> </div> This hook MUST return an additional list of strings containing <a class="pep reference internal" href="../pep-0508/" title="PEP 508 – Dependency specification for Python Software Packages">PEP 508</a> dependency specifications, above and beyond those specified in the <code class="docutils literal notranslate">pyproject.toml</code> file, to be installed when calling the <code class="docutils literal notranslate">build_wheel</code> or <code class="docutils literal notranslate">prepare_metadata_for_build_wheel</code> hooks. Example: <div class="highlight-default notranslate"><div class="highlight"><pre>def get_requires_for_build_wheel(config_settings): return ["wheel >= 0.25", "setuptools"] </pre></div> </div> If not defined, the default implementation is equivalent to <code class="docutils literal notranslate">return []</code>. </section> <section id="prepare-metadata-for-build-wheel"> <h4><a class="toc-backref" href="#prepare-metadata-for-build-wheel" role="doc-backlink">prepare_metadata_for_build_wheel</a></h4> <div class="highlight-default notranslate"><div class="highlight"><pre>def prepare_metadata_for_build_wheel(metadata_directory, config_settings=None): ... </pre></div> </div> Must create a <code class="docutils literal notranslate">.dist-info</code> directory containing wheel metadata inside the specified <code class="docutils literal notranslate">metadata_directory</code> (i.e., creates a directory like <code class="docutils literal notranslate">{metadata_directory}/{package}-{version}.dist-info/</code>). This directory MUST be a valid <code class="docutils literal notranslate">.dist-info</code> directory as defined in the wheel specification, except that it need not contain <code class="docutils literal notranslate">RECORD</code> or signatures. The hook MAY also create other files inside this directory, and a build frontend MUST preserve, but otherwise ignore, such files; the intention here is that in cases where the metadata depends on build-time decisions, the build backend may need to record these decisions in some convenient format for re-use by the actual wheel-building step. This must return the basename (not the full path) of the <code class="docutils literal notranslate">.dist-info</code> directory it creates, as a unicode string. If a build frontend needs this information and the method is not defined, it should call <code class="docutils literal notranslate">build_wheel</code> and look at the resulting metadata directly. </section> <section id="get-requires-for-build-sdist"> <h4><a class="toc-backref" href="#get-requires-for-build-sdist" role="doc-backlink">get_requires_for_build_sdist</a></h4> <div class="highlight-default notranslate"><div class="highlight"><pre>def get_requires_for_build_sdist(config_settings=None): ... </pre></div> </div> This hook MUST return an additional list of strings containing <a class="pep reference internal" href="../pep-0508/" title="PEP 508 – Dependency specification for Python Software Packages">PEP 508</a> dependency specifications, above and beyond those specified in the <code class="docutils literal notranslate">pyproject.toml</code> file. These dependencies will be installed when calling the <code class="docutils literal notranslate">build_sdist</code> hook. If not defined, the default implementation is equivalent to <code class="docutils literal notranslate">return []</code>. <div class="admonition note"> Note Editable installs This PEP originally specified another hook, <code class="docutils literal notranslate">install_editable</code>, to do an editable install (as with <code class="docutils literal notranslate">pip install -e</code>). It was removed due to the complexity of the topic, but may be specified in a later PEP. Briefly, the questions to be answered include: what reasonable ways existing of implementing an ‘editable install’? Should the backend or the frontend pick how to make an editable install? And if the frontend does, what does it need from the backend to do so. </div> </section> </section> <section id="config-settings"> <h3><a class="toc-backref" href="#config-settings" role="doc-backlink">Config settings</a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre>config_settings </pre></div> </div> This argument, which is passed to all hooks, is an arbitrary dictionary provided as an “escape hatch” for users to pass ad-hoc configuration into individual package builds. Build backends MAY assign any semantics they like to this dictionary. Build frontends SHOULD provide some mechanism for users to specify arbitrary string-key/string-value pairs to be placed in this dictionary. For example, they might support some syntax like <code class="docutils literal notranslate">--package-config CC=gcc</code>. In case a user provides duplicate string-keys, build frontends SHOULD combine the corresponding string-values into a list of strings. Build frontends MAY also provide arbitrary other mechanisms for users to place entries in this dictionary. For example, <code class="docutils literal notranslate">pip</code> might choose to map a mix of modern and legacy command line arguments like: <div class="highlight-default notranslate"><div class="highlight"><pre>pip install \ --package-config CC=gcc \ --global-option="--some-global-option" \ --build-option="--build-option1" \ --build-option="--build-option2" </pre></div> </div> into a <code class="docutils literal notranslate">config_settings</code> dictionary like: <div class="highlight-default notranslate"><div class="highlight"><pre>{ "CC": "gcc", "--global-option": ["--some-global-option"], "--build-option": ["--build-option1", "--build-option2"], } </pre></div> </div> Of course, it’s up to users to make sure that they pass options which make sense for the particular build backend and package that they are building. The hooks may be called with positional or keyword arguments, so backends implementing them should be careful to make sure that their signatures match both the order and the names of the arguments above. All hooks are run with working directory set to the root of the source tree, and MAY print arbitrary informational text on stdout and stderr. They MUST NOT read from stdin, and the build frontend MAY close stdin before invoking the hooks. The build frontend may capture stdout and/or stderr from the backend. If the backend detects that an output stream is not a terminal/console (e.g. <code class="docutils literal notranslate">not sys.stdout.isatty()</code>), it SHOULD ensure that any output it writes to that stream is UTF-8 encoded. The build frontend MUST NOT fail if captured output is not valid UTF-8, but it MAY not preserve all the information in that case (e.g. it may decode using the replace error handler in Python). If the output stream is a terminal, the build backend is responsible for presenting its output accurately, as for any program running in a terminal. If a hook raises an exception, or causes the process to terminate, then this indicates an error. </section> <section id="build-environment"> <h3><a class="toc-backref" href="#build-environment" role="doc-backlink">Build environment</a></h3> One of the responsibilities of a build frontend is to set up the Python environment in which the build backend will run. We do not require that any particular “virtual environment” mechanism be used; a build frontend might use virtualenv, or venv, or no special mechanism at all. But whatever mechanism is used MUST meet the following criteria: <ul> <li>All requirements specified by the project’s build-requirements must be available for import from Python. In particular:<ul class="simple"> <li>The <code class="docutils literal notranslate">get_requires_for_build_wheel</code> and <code class="docutils literal notranslate">get_requires_for_build_sdist</code> hooks are executed in an environment which contains the bootstrap requirements specified in the <code class="docutils literal notranslate">pyproject.toml</code> file.</li> <li>The <code class="docutils literal notranslate">prepare_metadata_for_build_wheel</code> and <code class="docutils literal notranslate">build_wheel</code> hooks are executed in an environment which contains the bootstrap requirements from <code class="docutils literal notranslate">pyproject.toml</code> and those specified by the <code class="docutils literal notranslate">get_requires_for_build_wheel</code> hook.</li> <li>The <code class="docutils literal notranslate">build_sdist</code> hook is executed in an environment which contains the bootstrap requirements from <code class="docutils literal notranslate">pyproject.toml</code> and those specified by the <code class="docutils literal notranslate">get_requires_for_build_sdist</code> hook.</li> </ul> </li> <li>This must remain true even for new Python subprocesses spawned by the build environment, e.g. code like:<div class="highlight-default notranslate"><div class="highlight"><pre>import sys, subprocess subprocess.check_call([sys.executable, ...]) </pre></div> </div> must spawn a Python process which has access to all the project’s build-requirements. This is necessary e.g. for build backends that want to run legacy <code class="docutils literal notranslate">setup.py</code> scripts in a subprocess. </li> <li>All command-line scripts provided by the build-required packages must be present in the build environment’s PATH. For example, if a project declares a build-requirement on <a class="reference external" href="https://flit.readthedocs.org/en/latest/">flit</a>, then the following must work as a mechanism for running the flit command-line tool:<div class="highlight-default notranslate"><div class="highlight"><pre>import subprocess import shutil subprocess.check_call([shutil.which("flit"), ...]) </pre></div> </div> </li> </ul> A build backend MUST be prepared to function in any environment which meets the above criteria. In particular, it MUST NOT assume that it has access to any packages except those that are present in the stdlib, or that are explicitly declared as build-requirements. Frontends should call each hook in a fresh subprocess, so that backends are free to change process global state (such as environment variables or the working directory). A Python library will be provided which frontends can use to easily call hooks this way. <section id="recommendations-for-build-frontends-non-normative"> <h4><a class="toc-backref" href="#recommendations-for-build-frontends-non-normative" role="doc-backlink">Recommendations for build frontends (non-normative)</a></h4> A build frontend MAY use any mechanism for setting up a build environment that meets the above criteria. For example, simply installing all build-requirements into the global environment would be sufficient to build any compliant package – but this would be sub-optimal for a number of reasons. This section contains non-normative advice to frontend implementors. A build frontend SHOULD, by default, create an isolated environment for each build, containing only the standard library and any explicitly requested build-dependencies. This has two benefits: <ul class="simple"> <li>It allows for a single installation run to build multiple packages that have contradictory build-requirements. E.g. if package1 build-requires pbr==1.8.1, and package2 build-requires pbr==1.7.2, then these cannot both be installed simultaneously into the global environment – which is a problem when the user requests <code class="docutils literal notranslate">pip install package1 package2</code>. Or if the user already has pbr==1.8.1 installed in their global environment, and a package build-requires pbr==1.7.2, then downgrading the user’s version would be rather rude.</li> <li>It acts as a kind of public health measure to maximize the number of packages that actually do declare accurate build-dependencies. We can write all the strongly worded admonitions to package authors we want, but if build frontends don’t enforce isolation by default, then we’ll inevitably end up with lots of packages on PyPI that build fine on the original author’s machine and nowhere else, which is a headache that no-one needs.</li> </ul> However, there will also be situations where build-requirements are problematic in various ways. For example, a package author might accidentally leave off some crucial requirement despite our best efforts; or, a package might declare a build-requirement on <code class="docutils literal notranslate">foo >= 1.0</code> which worked great when 1.0 was the latest version, but now 1.1 is out and it has a showstopper bug; or, the user might decide to build a package against numpy==1.7 – overriding the package’s preferred numpy==1.8 – to guarantee that the resulting build will be compatible at the C ABI level with an older version of numpy (even if this means the resulting build is unsupported upstream). Therefore, build frontends SHOULD provide some mechanism for users to override the above defaults. For example, a build frontend could have a <code class="docutils literal notranslate">--build-with-system-site-packages</code> option that causes the <code class="docutils literal notranslate">--system-site-packages</code> option to be passed to virtualenv-or-equivalent when creating build environments, or a <code class="docutils literal notranslate">--build-requirements-override=my-requirements.txt</code> option that overrides the project’s normal build-requirements. The general principle here is that we want to enforce hygiene on package authors, while still allowing end-users to open up the hood and apply duct tape when necessary. </section> </section> <section id="in-tree-build-backends"> <h3><a class="toc-backref" href="#in-tree-build-backends" role="doc-backlink">In-tree build backends</a></h3> In certain circumstances, projects may wish to include the source code for the build backend directly in the source tree, rather than referencing the backend via the <code class="docutils literal notranslate">requires</code> key. Two specific situations where this would be expected are: <ul class="simple"> <li>Backends themselves, which want to use their own features for building themselves (“self-hosting backends”)</li> <li>Project-specific backends, typically consisting of a custom wrapper around a standard backend, where the wrapper is too project-specific to be worth distributing independently (“in-tree backends”)</li> </ul> Projects can specify that their backend code is hosted in-tree by including the <code class="docutils literal notranslate">backend-path</code> key in <code class="docutils literal notranslate">pyproject.toml</code>. This key contains a list of directories, which the frontend will add to the start of <code class="docutils literal notranslate">sys.path</code> when loading the backend, and running the backend hooks. There are two restrictions on the content of the <code class="docutils literal notranslate">backend-path</code> key: <ul class="simple"> <li>Directories in <code class="docutils literal notranslate">backend-path</code> are interpreted as relative to the project root, and MUST refer to a location within the source tree (after relative paths and symbolic links have been resolved).</li> <li>The backend code MUST be loaded from one of the directories specified in <code class="docutils literal notranslate">backend-path</code> (i.e., it is not permitted to specify <code class="docutils literal notranslate">backend-path</code> and not have in-tree backend code).</li> </ul> The first restriction is to ensure that source trees remain self-contained, and cannot refer to locations outside of the source tree. Frontends SHOULD check this condition (typically by resolving the location to an absolute path and resolving symbolic links, and then checking it against the project root), and fail with an error message if it is violated. The <code class="docutils literal notranslate">backend-path</code> feature is intended to support the implementation of in-tree backends, and not to allow configuration of existing backends. The second restriction above is specifically to ensure that this is how the feature is used. Front ends MAY enforce this check, but are not required to. Doing so would typically involve checking the backend’s <code class="docutils literal notranslate">__file__</code> attribute against the locations in <code class="docutils literal notranslate">backend-path</code>. </section> </section> <section id="source-distributions"> <h2><a class="toc-backref" href="#source-distributions" role="doc-backlink">Source distributions</a></h2> We continue with the legacy sdist format, adding some new restrictions. This format is mostly undefined, but basically comes down to: a file named <code class="docutils literal notranslate">{NAME}-{VERSION}.{EXT}</code>, which unpacks into a buildable source tree called <code class="docutils literal notranslate">{NAME}-{VERSION}/</code>. Traditionally these have always contained <code class="docutils literal notranslate">setup.py</code>-style source trees; we now allow them to also contain <code class="docutils literal notranslate">pyproject.toml</code>-style source trees. Integration frontends require that an sdist named <code class="docutils literal notranslate">{NAME}-{VERSION}.{EXT}</code> will generate a wheel named <code class="docutils literal notranslate">{NAME}-{VERSION}-{COMPAT-INFO}.whl</code>. The new restrictions for sdists built by <a class="pep reference internal" href="../pep-0517/" title="PEP 517 – A build-system independent format for source trees">PEP 517</a> backends are: <ul class="simple"> <li>They will be gzipped tar archives, with the <code class="docutils literal notranslate">.tar.gz</code> extension. Zip archives, or other compression formats for tarballs, are not allowed at present.</li> <li>Tar archives must be created in the modern POSIX.1-2001 pax tar format, which uses UTF-8 for file names.</li> <li>The source tree contained in an sdist is expected to include the <code class="docutils literal notranslate">pyproject.toml</code> file.</li> </ul> </section> <section id="evolutionary-notes"> <h2><a class="toc-backref" href="#evolutionary-notes" role="doc-backlink">Evolutionary notes</a></h2> A goal here is to make it as simple as possible to convert old-style sdists to new-style sdists. (E.g., this is one motivation for supporting dynamic build requirements.) The ideal would be that there would be a single static <code class="docutils literal notranslate">pyproject.toml</code> that could be dropped into any “version 0” VCS checkout to convert it to the new shiny. This is probably not 100% possible, but we can get close, and it’s important to keep track of how close we are… hence this section. A rough plan would be: Create a build system package (<code class="docutils literal notranslate">setuptools_pypackage</code> or whatever) that knows how to speak whatever hook language we come up with, and convert them into calls to <code class="docutils literal notranslate">setup.py</code>. This will probably require some sort of hooking or monkeypatching to setuptools to provide a way to extract the <code class="docutils literal notranslate">setup_requires=</code> argument when needed, and to provide a new version of the sdist command that generates the new-style format. This all seems doable and sufficient for a large proportion of packages (though obviously we’ll want to prototype such a system before we finalize anything here). (Alternatively, these changes could be made to setuptools itself rather than going into a separate package.) But there remain two obstacles that mean we probably won’t be able to automatically upgrade packages to the new format: <ol class="arabic simple"> <li>There currently exist packages which insist on particular packages being available in their environment before setup.py is executed. This means that if we decide to execute build scripts in an isolated virtualenv-like environment, then projects will need to check whether they do this, and if so then when upgrading to the new system they will have to start explicitly declaring these dependencies (either via <code class="docutils literal notranslate">setup_requires=</code> or via static declaration in <code class="docutils literal notranslate">pyproject.toml</code>).</li> <li>There currently exist packages which do not declare consistent metadata (e.g. <code class="docutils literal notranslate">egg_info</code> and <code class="docutils literal notranslate">bdist_wheel</code> might get different <code class="docutils literal notranslate">install_requires=</code>). When upgrading to the new system, projects will have to evaluate whether this applies to them, and if so they will need to stop doing that.</li> </ol> </section> <section id="rejected-options"> <h2><a class="toc-backref" href="#rejected-options" role="doc-backlink">Rejected options</a></h2> <ul class="simple"> <li>We discussed making the wheel and sdist hooks build unpacked directories containing the same contents as their respective archives. In some cases this could avoid the need to pack and unpack an archive, but this seems like premature optimisation. It’s advantageous for tools to work with archives as the canonical interchange formats (especially for wheels, where the archive format is already standardised). Close control of archive creation is important for reproducible builds. And it’s not clear that tasks requiring an unpacked distribution will be more common than those requiring an archive.</li> <li>We considered an extra hook to copy files to a build directory before invoking <code class="docutils literal notranslate">build_wheel</code>. Looking at existing build systems, we found that passing a build directory into <code class="docutils literal notranslate">build_wheel</code> made more sense for many tools than pre-emptively copying files into a build directory.</li> <li>The idea of passing <code class="docutils literal notranslate">build_wheel</code> a build directory was then also deemed an unnecessary complication. Build tools can use a temporary directory or a cache directory to store intermediate files while building. If there is a need, a frontend-controlled cache directory could be added in the future.</li> <li>For <code class="docutils literal notranslate">build_sdist</code> to signal a failure for an expected reason, various options were debated at great length, including raising <code class="docutils literal notranslate">NotImplementedError</code> and returning either <code class="docutils literal notranslate">NotImplemented</code> or <code class="docutils literal notranslate">None</code>. Please do not attempt to reopen this discussion without an extremely good reason, because we are quite tired of it.</li> <li>Allowing the backend to be imported from files in the source tree would be more consistent with the way Python imports often work. However, not allowing this prevents confusing errors from clashing module names. The initial version of this PEP did not provide a means to allow backends to be imported from files within the source tree, but the <code class="docutils literal notranslate">backend-path</code> key was added in the next revision to allow projects to opt into this behaviour if needed.</li> </ul> </section> <section id="summary-of-changes-to-pep-517"> <h2><a class="toc-backref" href="#summary-of-changes-to-pep-517" role="doc-backlink">Summary of changes to PEP 517</a></h2> The following changes were made to this PEP after the initial reference implementation was released in pip 19.0. <ul class="simple"> <li>Cycles in build requirements were explicitly prohibited.</li> <li>Support for in-tree backends and self-hosting of backends was added by the introduction of the <code class="docutils literal notranslate">backend-path</code> key in the <code class="docutils literal notranslate">[build-system]</code> table.</li> <li>Clarified that the <code class="docutils literal notranslate">setuptools.build_meta:__legacy__</code> <a class="pep reference internal" href="../pep-0517/" title="PEP 517 – A build-system independent format for source trees">PEP 517</a> backend is an acceptable alternative to directly invoking <code class="docutils literal notranslate">setup.py</code> for source trees that don’t specify <code class="docutils literal notranslate">build-backend</code> explicitly.</li> </ul> </section> <section id="appendix-a-comparison-to-pep-516"> <h2><a class="toc-backref" href="#appendix-a-comparison-to-pep-516" role="doc-backlink">Appendix A: Comparison to PEP 516</a></h2> <a class="pep reference internal" href="../pep-0516/" title="PEP 516 – Build system abstraction for pip/conda etc">PEP 516</a> is a competing proposal to specify a build system interface, which has now been rejected in favour of this PEP. The primary difference is that our build backend is defined via a Python hook-based interface rather than a command-line based interface. This appendix documents the arguments advanced for this PEP over <a class="pep reference internal" href="../pep-0516/" title="PEP 516 – Build system abstraction for pip/conda etc">PEP 516</a>. We do not expect that specifying Python hooks rather than command line interfaces will, by itself, reduce the complexity of calling into the backend, because build frontends will in any case want to run hooks inside a child – this is important to isolate the build frontend itself from the backend code and to better control the build backends execution environment. So under both proposals, there will need to be some code in <code class="docutils literal notranslate">pip</code> to spawn a subprocess and talk to some kind of command-line/IPC interface, and there will need to be some code in the subprocess that knows how to parse these command line arguments and call the actual build backend implementation. So this diagram applies to all proposals equally: <div class="highlight-default notranslate"><div class="highlight"><pre>+-----------+ +---------------+ +----------------+ | frontend | -spawn-> | child cmdline | -Python-> | backend | | (pip) | | interface | | implementation | +-----------+ +---------------+ +----------------+ </pre></div> </div> The key difference between the two approaches is how these interface boundaries map onto project structure: <div class="highlight-default notranslate"><div class="highlight"><pre>.-= This PEP =-. +-----------+ +---------------+ | +----------------+ | frontend | -spawn-> | child cmdline | -Python-> | backend | | (pip) | | interface | | | implementation | +-----------+ +---------------+ | +----------------+ | |______________________________________| | Owned by pip, updated in lockstep | | | PEP-defined interface boundary Changes here require distutils-sig .-= Alternative =-. +-----------+ | +---------------+ +----------------+ | frontend | -spawn-> | child cmdline | -Python-> | backend | | (pip) | | | interface | | implementation | +-----------+ | +---------------+ +----------------+ | | |____________________________________________| | Owned by build backend, updated in lockstep | PEP-defined interface boundary Changes here require distutils-sig </pre></div> </div> By moving the PEP-defined interface boundary into Python code, we gain three key advantages. First, because there will likely be only a small number of build frontends (<code class="docutils literal notranslate">pip</code>, and… maybe a few others?), while there will likely be a long tail of custom build backends (since these are chosen separately by each package to match their particular build requirements), the actual diagrams probably look more like: <div class="highlight-default notranslate"><div class="highlight"><pre>.-= This PEP =-. +-----------+ +---------------+ +----------------+ | frontend | -spawn-> | child cmdline | -Python+> | backend | | (pip) | | interface | | | implementation | +-----------+ +---------------+ | +----------------+ | | +----------------+ +> | backend | | | implementation | | +----------------+ : : .-= Alternative =-. +-----------+ +---------------+ +----------------+ | frontend | -spawn+> | child cmdline | -Python-> | backend | | (pip) | | | interface | | implementation | +-----------+ | +---------------+ +----------------+ | | +---------------+ +----------------+ +> | child cmdline | -Python-> | backend | | | interface | | implementation | | +---------------+ +----------------+ : : </pre></div> </div> That is, this PEP leads to less total code in the overall ecosystem. And in particular, it reduces the barrier to entry of making a new build system. For example, this is a complete, working build backend: <div class="highlight-default notranslate"><div class="highlight"><pre># mypackage_custom_build_backend.py import os.path import pathlib import shutil import tarfile SDIST_NAME = "mypackage-0.1" SDIST_FILENAME = SDIST_NAME + ".tar.gz" WHEEL_FILENAME = "mypackage-0.1-py2.py3-none-any.whl" ################# # sdist creation ################# def _exclude_hidden_and_special_files(archive_entry): """Tarfile filter to exclude hidden and special files from the archive""" if archive_entry.isfile() or archive_entry.isdir(): if not os.path.basename(archive_entry.name).startswith("."): return archive_entry def _make_sdist(sdist_dir): """Make an sdist and return both the Python object and its filename""" sdist_path = pathlib.Path(sdist_dir) / SDIST_FILENAME sdist = tarfile.open(sdist_path, "w:gz", format=tarfile.PAX_FORMAT) # Tar up the whole directory, minus hidden and special files sdist.add(os.getcwd(), arcname=SDIST_NAME, filter=_exclude_hidden_and_special_files) return sdist, SDIST_FILENAME def build_sdist(sdist_dir, config_settings): """PEP 517 sdist creation hook""" sdist, sdist_filename = _make_sdist(sdist_dir) return sdist_filename ################# # wheel creation ################# def get_requires_for_build_wheel(config_settings): """PEP 517 wheel building dependency definition hook""" # As a simple static requirement, this could also just be # listed in the project's build system dependencies instead return ["wheel"] def build_wheel(wheel_directory, metadata_directory=None, config_settings=None): """PEP 517 wheel creation hook""" from wheel.archive import archive_wheelfile path = os.path.join(wheel_directory, WHEEL_FILENAME) archive_wheelfile(path, "src/") return WHEEL_FILENAME </pre></div> </div> Of course, this is a terrible build backend: it requires the user to have manually set up the wheel metadata in <code class="docutils literal notranslate">src/mypackage-0.1.dist-info/</code>; when the version number changes it must be manually updated in multiple places… but it works, and more features could be added incrementally. Much experience suggests that large successful projects often originate as quick hacks (e.g., Linux – “just a hobby, won’t be big and professional”; <a class="reference external" href="https://en.wikipedia.org/wiki/IPython#Grants_and_awards">IPython/Jupyter</a> – <a class="reference external" href="http://blog.fperez.org/2012/01/ipython-notebook-historical.html">a grad student’s $PYTHONSTARTUP file</a>), so if our goal is to encourage the growth of a vibrant ecosystem of good build tools, it’s important to minimize the barrier to entry. Second, because Python provides a simpler yet richer structure for describing interfaces, we remove unnecessary complexity from the specification – and specifications are the worst place for complexity, because changing specifications requires painful consensus-building across many stakeholders. In the command-line interface approach, we have to come up with ad hoc ways to map multiple different kinds of inputs into a single linear command line (e.g. how do we avoid collisions between user-specified configuration arguments and PEP-defined arguments? how do we specify optional arguments? when working with a Python interface these questions have simple, obvious answers). When spawning and managing subprocesses, there are many fiddly details that must be gotten right, subtle cross-platform differences, and some of the most obvious approaches – e.g., using stdout to return data for the <code class="docutils literal notranslate">build_requires</code> operation – can create unexpected pitfalls (e.g., what happens when computing the build requirements requires spawning some child processes, and these children occasionally print an error message to stdout? obviously a careful build backend author can avoid this problem, but the most obvious way of defining a Python interface removes this possibility entirely, because the hook return value is clearly demarcated). In general, the need to isolate build backends into their own process means that we can’t remove IPC complexity entirely – but by placing both sides of the IPC channel under the control of a single project, we make it much cheaper to fix bugs in the IPC interface than if fixing bugs requires coordinated agreement and coordinated changes across the ecosystem. Third, and most crucially, the Python hook approach gives us much more powerful options for evolving this specification in the future. For concreteness, imagine that next year we add a new <code class="docutils literal notranslate">build_sdist_from_vcs</code> hook, which provides an alternative to the current <code class="docutils literal notranslate">build_sdist</code> hook where the frontend is responsible for passing version control tracking metadata to backends (including indicating when all on disk files are tracked), rather than individual backends having to query that information themselves. In order to manage the transition, we’d want it to be possible for build frontends to transparently use <code class="docutils literal notranslate">build_sdist_from_vcs</code> when available and fall back onto <code class="docutils literal notranslate">build_sdist</code> otherwise; and we’d want it to be possible for build backends to define both methods, for compatibility with both old and new build frontends. Furthermore, our mechanism should also fulfill two more goals: (a) If new versions of e.g. <code class="docutils literal notranslate">pip</code> and <code class="docutils literal notranslate">flit</code> are both updated to support the new interface, then this should be sufficient for it to be used; in particular, it should not be necessary for every project that uses <code class="docutils literal notranslate">flit</code> to update its individual <code class="docutils literal notranslate">pyproject.toml</code> file. (b) We do not want to have to spawn extra processes just to perform this negotiation, because process spawns can easily become a bottleneck when deploying large multi-package stacks on some platforms (Windows). In the interface described here, all of these goals are easy to achieve. Because <code class="docutils literal notranslate">pip</code> controls the code that runs inside the child process, it can easily write it to do something like: <div class="highlight-default notranslate"><div class="highlight"><pre>command, backend, args = parse_command_line_args(...) if command == "build_sdist": if hasattr(backend, "build_sdist_from_vcs"): backend.build_sdist_from_vcs(...) elif hasattr(backend, "build_sdist"): backend.build_sdist(...) else: # error handling </pre></div> </div> In the alternative where the public interface boundary is placed at the subprocess call, this is not possible – either we need to spawn an extra process just to query what interfaces are supported (as was included in an earlier draft of <a class="pep reference internal" href="../pep-0516/" title="PEP 516 – Build system abstraction for pip/conda etc">PEP 516</a>, an alternative to this), or else we give up on autonegotiation entirely (as in the current version of that PEP), meaning that any changes in the interface will require N individual packages to update their <code class="docutils literal notranslate">pyproject.toml</code> files before any change can go live, and that any changes will necessarily be restricted to new releases. One specific consequence of this is that in this PEP, we’re able to make the <code class="docutils literal notranslate">prepare_metadata_for_build_wheel</code> command optional. In our design, this can be readily handled by build frontends, which can put code in their subprocess runner like: <div class="highlight-default notranslate"><div class="highlight"><pre>def dump_wheel_metadata(backend, working_dir): """Dumps wheel metadata to working directory. Returns absolute path to resulting metadata directory """ if hasattr(backend, "prepare_metadata_for_build_wheel"): subdir = backend.prepare_metadata_for_build_wheel(working_dir) else: wheel_fname = backend.build_wheel(working_dir) already_built = os.path.join(working_dir, "ALREADY_BUILT_WHEEL") with open(already_built, "w") as f: f.write(wheel_fname) subdir = unzip_metadata(os.path.join(working_dir, wheel_fname)) return os.path.join(working_dir, subdir) def ensure_wheel_is_built(backend, output_dir, working_dir, metadata_dir): """Ensures built wheel is available in output directory Returns absolute path to resulting wheel file """ already_built = os.path.join(working_dir, "ALREADY_BUILT_WHEEL") if os.path.exists(already_built): with open(already_built, "r") as f: wheel_fname = f.read().strip() working_path = os.path.join(working_dir, wheel_fname) final_path = os.path.join(output_dir, wheel_fname) os.rename(working_path, final_path) os.remove(already_built) else: wheel_fname = backend.build_wheel(output_dir, metadata_dir=metadata_dir) return os.path.join(output_dir, wheel_fname) </pre></div> </div> and thus expose a totally uniform interface to the rest of the frontend, with no extra subprocess calls, no duplicated builds, etc. But obviously this is the kind of code that you only want to write as part of a private, within-project interface (e.g. the given example requires that the working directory be shared between the two calls, but not with any other wheel builds, and that the return value from the metadata helper function will be passed back in to the wheel building one). (And, of course, making the <code class="docutils literal notranslate">metadata</code> command optional is one piece of lowering the barrier to entry for developing new backends, as discussed above.) <section id="other-differences"> <h3><a class="toc-backref" href="#other-differences" role="doc-backlink">Other differences</a></h3> Besides the key command line versus Python hook difference described above, there are a few other differences in this proposal: <ul class="simple"> <li>Metadata command is optional (as described above).</li> <li>We return metadata as a directory, rather than a single METADATA file. This aligns better with the way that in practice wheel metadata is distributed across multiple files (e.g. entry points), and gives us more options in the future. (For example, instead of following the PEP 426 proposal of switching the format of METADATA to JSON, we might decide to keep the existing METADATA the way it is for backcompat, while adding new extensions as JSON “sidecar” files inside the same directory. Or maybe not; the point is it keeps our options more open.)</li> <li>We provide a mechanism for passing information between the metadata step and the wheel building step. I guess everyone probably will agree this is a good idea?</li> <li>We provide more detailed recommendations about the build environment, but these aren’t normative anyway.</li> </ul> </section> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0517.rst">https://github.com/python/peps/blob/main/peps/pep-0517.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0517.rst">2025-02-01 08:59:27 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#terminology-and-goals">Terminology and goals</a></li> <li><a class="reference internal" href="#source-trees">Source trees</a><ul> <li><a class="reference internal" href="#build-requirements">Build requirements</a></li> </ul> </li> <li><a class="reference internal" href="#build-backend-interface">Build backend interface</a><ul> <li><a class="reference internal" href="#mandatory-hooks">Mandatory hooks</a><ul> <li><a class="reference internal" href="#build-wheel">build_wheel</a></li> <li><a class="reference internal" href="#build-sdist">build_sdist</a></li> </ul> </li> <li><a class="reference internal" href="#optional-hooks">Optional hooks</a><ul> <li><a class="reference internal" href="#get-requires-for-build-wheel">get_requires_for_build_wheel</a></li> <li><a class="reference internal" href="#prepare-metadata-for-build-wheel">prepare_metadata_for_build_wheel</a></li> <li><a class="reference internal" href="#get-requires-for-build-sdist">get_requires_for_build_sdist</a></li> </ul> </li> <li><a class="reference internal" href="#config-settings">Config settings</a></li> <li><a class="reference internal" href="#build-environment">Build environment</a><ul> <li><a class="reference internal" href="#recommendations-for-build-frontends-non-normative">Recommendations for build frontends (non-normative)</a></li> </ul> </li> <li><a class="reference internal" href="#in-tree-build-backends">In-tree build backends</a></li> </ul> </li> <li><a class="reference internal" href="#source-distributions">Source distributions</a></li> <li><a class="reference internal" href="#evolutionary-notes">Evolutionary notes</a></li> <li><a class="reference internal" href="#rejected-options">Rejected options</a></li> <li><a class="reference internal" href="#summary-of-changes-to-pep-517">Summary of changes to PEP 517</a></li> <li><a class="reference internal" href="#appendix-a-comparison-to-pep-516">Appendix A: Comparison to PEP 516</a><ul> <li><a class="reference internal" href="#other-differences">Other differences</a></li> </ul> </li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0517.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>