CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 665 – A file format to list Python dependencies for reproducibility of an application | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0665/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 665 – A file format to list Python dependencies for reproducibility of an application | peps.python.org'> <meta property="og:description" content="This PEP specifies a file format to specify the list of Python package installation requirements for an application, and the relation between the specified requirements. The list of requirements is considered exhaustive for the installation target, and ..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0665/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This PEP specifies a file format to specify the list of Python package installation requirements for an application, and the relation between the specified requirements. The list of requirements is considered exhaustive for the installation target, and ..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 665</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> <span class="visually-hidden">Toggle light / dark / auto colour theme</span> </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 665 – A file format to list Python dependencies for reproducibility of an application</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author<span class="colon">:</span></dt> <dd class="field-odd">Brett Cannon <brett at python.org>, Pradyun Gedam <pradyunsg at gmail.com>, Tzu-ping Chung <uranusjr at gmail.com></dd> <dt class="field-even">PEP-Delegate<span class="colon">:</span></dt> <dd class="field-even">Paul Moore <p.f.moore at gmail.com></dd> <dt class="field-odd">Discussions-To<span class="colon">:</span></dt> <dd class="field-odd"><a class="reference external" href="https://discuss.python.org/t/9911">Discourse thread</a></dd> <dt class="field-even">Status<span class="colon">:</span></dt> <dd class="field-even"><abbr title="Formally declined and will not be accepted">Rejected</abbr></dd> <dt class="field-odd">Type<span class="colon">:</span></dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Topic<span class="colon">:</span></dt> <dd class="field-even"><a class="reference external" href="../topic/packaging/">Packaging</a></dd> <dt class="field-odd">Created<span class="colon">:</span></dt> <dd class="field-odd">29-Jul-2021</dd> <dt class="field-even">Post-History<span class="colon">:</span></dt> <dd class="field-even">29-Jul-2021, 03-Nov-2021, 25-Nov-2021</dd> <dt class="field-odd">Superseded-By<span class="colon">:</span></dt> <dd class="field-odd"><a class="reference external" href="../pep-0751/">751</a></dd> <dt class="field-even">Resolution<span class="colon">:</span></dt> <dd class="field-even"><a class="reference external" href="https://discuss.python.org/t/pep-665-take-2-a-file-format-to-list-python-dependencies-for-reproducibility-of-an-application/11736/140">Discourse message</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#terminology">Terminology</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#file-format">File Format</a></li> <li><a class="reference internal" href="#secure-by-design">Secure by Design</a></li> <li><a class="reference internal" href="#cross-platform">Cross-Platform</a></li> <li><a class="reference internal" href="#simple-installer">Simple Installer</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#details">Details</a><ul> <li><a class="reference internal" href="#version"><code class="docutils literal notranslate"><span class="pre">version</span></code></a></li> <li><a class="reference internal" href="#created-at"><code class="docutils literal notranslate"><span class="pre">created-at</span></code></a></li> <li><a class="reference internal" href="#tool"><code class="docutils literal notranslate"><span class="pre">[tool]</span></code></a></li> <li><a class="reference internal" href="#metadata"><code class="docutils literal notranslate"><span class="pre">[metadata]</span></code></a><ul> <li><a class="reference internal" href="#metadata-marker"><code class="docutils literal notranslate"><span class="pre">metadata.marker</span></code></a></li> <li><a class="reference internal" href="#metadata-tag"><code class="docutils literal notranslate"><span class="pre">metadata.tag</span></code></a></li> <li><a class="reference internal" href="#metadata-requires"><code class="docutils literal notranslate"><span class="pre">metadata.requires</span></code></a></li> <li><a class="reference internal" href="#metadata-requires-python"><code class="docutils literal notranslate"><span class="pre">metadata.requires-python</span></code></a></li> </ul> </li> <li><a class="reference internal" href="#package-name-version"><code class="docutils literal notranslate"><span class="pre">[[package._name_._version_]]</span></code></a><ul> <li><a class="reference internal" href="#package-name-version-filename"><code class="docutils literal notranslate"><span class="pre">package._name_._version_.filename</span></code></a></li> <li><a class="reference internal" href="#package-name-version-hashes"><code class="docutils literal notranslate"><span class="pre">[package._name_._version_.hashes]</span></code></a></li> <li><a class="reference internal" href="#package-name-version-url"><code class="docutils literal notranslate"><span class="pre">package._name_._version_.url</span></code></a></li> <li><a class="reference internal" href="#package-name-version-direct"><code class="docutils literal notranslate"><span class="pre">package._name_._version_.direct</span></code></a></li> <li><a class="reference internal" href="#package-name-version-requires-python"><code class="docutils literal notranslate"><span class="pre">package._name_._version_.requires-python</span></code></a></li> <li><a class="reference internal" href="#package-name-version-requires"><code class="docutils literal notranslate"><span class="pre">package._name_._version_.requires</span></code></a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#example">Example</a></li> <li><a class="reference internal" href="#expectations-for-lockers">Expectations for Lockers</a></li> <li><a class="reference internal" href="#expectations-for-installers">Expectations for Installers</a></li> </ul> </li> <li><a class="reference internal" href="#potential-tool-support">(Potential) Tool Support</a></li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a></li> <li><a class="reference internal" href="#transition-plan">Transition Plan</a><ul> <li><a class="reference internal" href="#usability">Usability</a></li> <li><a class="reference internal" href="#interoperability">Interoperability</a></li> <li><a class="reference internal" href="#acceptance">Acceptance</a></li> </ul> </li> <li><a class="reference internal" href="#security-implications">Security Implications</a></li> <li><a class="reference internal" href="#how-to-teach-this">How to Teach This</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a><ul> <li><a class="reference internal" href="#file-formats-other-than-toml">File Formats Other Than TOML</a></li> <li><a class="reference internal" href="#alternative-naming-schemes">Alternative Naming Schemes</a></li> <li><a class="reference internal" href="#supporting-a-single-lock-file">Supporting a Single Lock File</a></li> <li><a class="reference internal" href="#using-a-flat-list-instead-of-a-dependency-graph">Using a Flat List Instead of a Dependency Graph</a></li> <li><a class="reference internal" href="#use-wheel-tags-in-the-file-name">Use Wheel Tags in the File Name</a></li> <li><a class="reference internal" href="#alternative-names-for-requires">Alternative Names for <code class="docutils literal notranslate"><span class="pre">requires</span></code></a></li> <li><a class="reference internal" href="#accepting-pep-650">Accepting PEP 650</a></li> <li><a class="reference internal" href="#specifying-requirements-per-package-instead-of-per-file">Specifying Requirements per Package Instead of per File</a></li> <li><a class="reference internal" href="#specify-where-lockers-gather-input">Specify Where Lockers Gather Input</a></li> <li><a class="reference internal" href="#allowing-source-distributions-and-source-trees-to-be-an-opt-in-supported-file-format">Allowing Source Distributions and Source Trees to be an Opt-In, Supported File Format</a></li> </ul> </li> <li><a class="reference internal" href="#open-issues">Open Issues</a></li> <li><a class="reference internal" href="#acknowledgments">Acknowledgments</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <div class="admonition note"> <p class="admonition-title">Note</p> <p>This PEP was rejected due to lukewarm reception from the community from the lack of source distribution support.</p> </div> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> <p>This PEP specifies a file format to specify the list of Python package installation requirements for an application, and the relation between the specified requirements. The list of requirements is considered exhaustive for the installation target, and thus not requiring any information beyond the platform being installed for, and the file itself. The file format is flexible enough to allow installing the requirements across different platforms, which allows for reproducibility on multiple platforms from the same file.</p> </section> <section id="terminology"> <h2><a class="toc-backref" href="#terminology" role="doc-backlink">Terminology</a></h2> <p>There are several terms whose definition must be agreed upon in order to facilitate a discussion on the topic of this PEP.</p> <p>A <em>package</em> is something you install as a dependency and use via the import system. The packages on PyPI are an example of this.</p> <p>An <em>application</em> or <em>app</em> is an end product that other external code does not directly rely on via the import system (i.e. they are standalone). Desktop applications, command-line tools, etc. are examples of applications.</p> <p>A <em>lock file</em> records the packages that are to be installed for an app. Traditionally, the exact version of the package to be installed is specified by a lock file, but specified packages are not always installed on a given platform (according a filtering logic described in a later section), which enables the lock file to describe reproducibility across multiple platforms. Examples of this are <code class="docutils literal notranslate"><span class="pre">package-lock.json</span></code> from <a class="reference external" href="https://www.npmjs.com/">npm</a>, <code class="docutils literal notranslate"><span class="pre">Poetry.lock</span></code> from <a class="reference external" href="https://pypi.org/project/poetry/">Poetry</a>, etc.</p> <p><em>Locking</em> is the act of taking the input of the packages an app depends on and producing a lock file from that.</p> <p>A <em>locker</em> is a tool which produces a lock file.</p> <p>An <em>installer</em> consumes a lock file to install what the lock file specifies.</p> </section> <section id="motivation"> <h2><a class="toc-backref" href="#motivation" role="doc-backlink">Motivation</a></h2> <p>Applications want reproducible installs for a few reasons (we are not worrying about package development, integration into larger systems that would handle locking dependencies external to the Python application, or other situations where <em>flexible</em> installation requirements are desired over strict, reproducible installations).</p> <p>One, reproducibility eases development. When you and your fellow developers all end up with the same files on a specific platform, you make sure you are all developing towards the same experience for the application. You also want your users to install the same files as you expect to guarantee the experience is the same as you developed for them.</p> <p>Two, you want to be able to reproduce what gets installed across multiple platforms. Thanks to Python’s portability across operating systems, CPUs, etc., it is very easy and often desirable to create applications that are not restricted to a single platform. Thus, you want to be flexible enough to allow for differences in your package dependencies between platforms, while still having consistency and reproducibility on any one specific platform.</p> <p>Three, reproducibility is more secure. When you control exactly what files are installed, you can make sure no malicious actor is attempting to slip nefarious code into your application (i.e. some supply chain attacks). By using a lock file which always leads to reproducible installs, we can avoid certain risks entirely.</p> <p>Four, relying on the <a class="reference external" href="https://packaging.python.org/specifications/binary-distribution-format/">wheel file</a> format provides reproducibility without requiring build tools to support reproducibility themselves. Thanks to wheels being static and not executing code as part of installation, wheels always lead to a reproducible result. Compare this to source distributions (aka sdists) or source trees which only lead to a reproducible install if their build tool supports reproducibility due to inherent code execution. Unfortunately the vast majority of build tools do not support reproducible builds, so this PEP helps alleviate that issue by only supporting wheels as a package format.</p> <p>This PEP proposes a standard for a lock file, as the current solutions don’t meet the outlined goals. Today, the closest we come to a lock file standard is the <a class="reference external" href="https://pip.pypa.io/en/latest/reference/requirements-file-format/">requirements file format</a> from pip. Unfortunately, that format does not lead to inherently reproducible installs (it requires optional features both in the requirements file and the installer itself, to be discussed later).</p> <p>The community itself has also shown a need for lock files based on the fact that multiple tools have independently created their own lock file formats:</p> <ol class="arabic simple"> <li><a class="reference external" href="https://pypi.org/project/pdm/">PDM</a></li> <li><a class="reference external" href="https://pypi.org/project/pip-tools/">pip-tools</a></li> <li><a class="reference external" href="https://pypi.org/project/pipenv/">Pipenv</a></li> <li><a class="reference external" href="https://pypi.org/project/poetry/">Poetry</a></li> <li><a class="reference external" href="https://pypi.org/project/pyflow/">Pyflow</a></li> </ol> <p>Unfortunately, those tools all use differing lock file formats. This means tooling around these tools must be unique. This impacts tooling such as code editors and hosting providers, which want to be as flexible as possible when it comes to accepting a user’s application code, but also have a limit as to how much development resources they can spend to add support for yet another lock file format. A standardized format would allow tools to focus their work on a single target, and make sure that workflow decisions made by developers outside of the lock file format are of no concern to e.g. hosting providers.</p> <p>Other programming language communities have also shown the usefulness of lock files by developing their own solution to this problem. Some of those communities include:</p> <ol class="arabic simple"> <li><a class="reference external" href="https://dart.dev/">Dart</a></li> <li><a class="reference external" href="https://www.npmjs.com/">npm</a>/Node</li> <li>Go</li> <li><a class="reference external" href="https://www.rust-lang.org/">Rust</a></li> </ol> <p>The trend in programming languages in the past decade seems to have been toward providing a lock file solution.</p> </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> <section id="file-format"> <h3><a class="toc-backref" href="#file-format" role="doc-backlink">File Format</a></h3> <p>We wanted the file format to be easy to read as a diff when auditing a change to the lock file. As such, and thanks to <a class="pep reference internal" href="../pep-0518/" title="PEP 518 – Specifying Minimum Build System Requirements for Python Projects">PEP 518</a> and <code class="docutils literal notranslate"><span class="pre">pyproject.toml</span></code>, we decided to go with the <a class="reference external" href="https://toml.io">TOML</a> file format.</p> </section> <section id="secure-by-design"> <h3><a class="toc-backref" href="#secure-by-design" role="doc-backlink">Secure by Design</a></h3> <p>Viewing the <a class="reference external" href="https://pip.pypa.io/en/latest/reference/requirements-file-format/">requirements file format</a> as the closest we have to a lock file standard, there are a few issues with the file format when it comes to security. First is that the file format simply does not require you to specify the exact version of a package. This is why tools like <a class="reference external" href="https://pypi.org/project/pip-tools/">pip-tools</a> exist to help manage that users of requirements files.</p> <p>Second, you must opt into specifying what files are acceptable to be installed by using the <code class="docutils literal notranslate"><span class="pre">--hash</span></code> argument for a specific dependency. This is also optional with pip-tools as it requires specifying the <code class="docutils literal notranslate"><span class="pre">--generate-hashes</span></code> CLI argument. This requires <code class="docutils literal notranslate"><span class="pre">--require-hashes</span></code> for pip to make sure no dependencies lack a hash to check.</p> <p>Third, even when you control what files may be installed, it does not prevent other packages from being installed. If a dependency is not listed in the requirements file, pip will happily go searching for a file to meet that need. You must specify <code class="docutils literal notranslate"><span class="pre">--no-deps</span></code> as an argument to pip to prevent unintended dependency resolution outside of the requirements file.</p> <p>Fourth, the format allows for installing a <a class="reference external" href="https://packaging.python.org/specifications/source-distribution-format/">source distribution file</a> (aka “sdist”). By its very nature, installing an sdist requires executing arbitrary Python code, meaning that there is no control over what files may be installed. Only by specifying <code class="docutils literal notranslate"><span class="pre">--only-binary</span> <span class="pre">:all:</span></code> can you guarantee pip to only use a <a class="reference external" href="https://packaging.python.org/specifications/binary-distribution-format/">wheel file</a> for each package.</p> <p>To recap, in order for a requirements file to be as secure as what is being proposed, a user should always do the following steps:</p> <ol class="arabic simple"> <li>Use pip-tools and its command <code class="docutils literal notranslate"><span class="pre">pip-compile</span> <span class="pre">--generate-hashes</span></code></li> <li>Install the requirements file using <code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">--require-hashes</span> <span class="pre">--no-deps</span> <span class="pre">--only-binary</span> <span class="pre">:all:</span></code></li> </ol> <p>Critically, all of those flags, and both the specificity and exhaustion of what to install that pip-tools provides, are optional for requirements files.</p> <p>As such, the proposal raised in this PEP is secure by design which combats some supply chain attacks. Hashes for files which would be used to install from are <strong>required</strong>. You can <strong>only</strong> install from wheels to unambiguously define what files will be placed in the file system. Installers <strong>must</strong> lead to an deterministic installation from a lock file for a given platform. All of this leads to a reproducible installation which you can deem trustworthy (when you have audited the lock file and what it lists).</p> </section> <section id="cross-platform"> <h3><a class="toc-backref" href="#cross-platform" role="doc-backlink">Cross-Platform</a></h3> <p>Various projects which already have a lock file, like <a class="reference external" href="https://pypi.org/project/pdm/">PDM</a> and <a class="reference external" href="https://pypi.org/project/poetry/">Poetry</a>, provide a lock file which is <em>cross-platform</em>. This allows for a single lock file to work on multiple platforms while still leading to the exact same top-level requirements to be installed everywhere with the installation being consistent/unambiguous on each platform.</p> <p>As to why this is useful, let’s use an example involving <a class="reference external" href="https://pyweek.org/">PyWeek</a> (a week-long game development competition). Assume you are developing on Linux, while someone you choose to partner with is using macOS. Now assume the judges are using Windows. How do you make sure everyone is using the same top-level dependencies, while allowing for any platform-specific requirements (e.g. a package requires a helper package under Windows)?</p> <p>With a cross-platform lock file, you can make sure that the key requirements are met consistently across all platforms. You can then also make sure that all users on the same platform get the same reproducible installation.</p> </section> <section id="simple-installer"> <h3><a class="toc-backref" href="#simple-installer" role="doc-backlink">Simple Installer</a></h3> <p>The separation of concerns between a locker and an installer allows for an installer to have a much simpler operation to perform. As such, it not only allows for installers to be easier to write, but facilitates in making sure installers create unambiguous, reproducible installations correctly.</p> <p>The installer can also expend less computation/energy in creating the installation. This is beneficial not only for faster installs, but also from an energy consumption perspective, as installers are expected to be run more often than lockers.</p> <p>This has led to a design where the locker must do more work upfront to the benefit installers. It also means the complexity of package dependencies is simpler and easier to comprehend in a lock files to avoid ambiguity.</p> </section> </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> <section id="details"> <h3><a class="toc-backref" href="#details" role="doc-backlink">Details</a></h3> <p>Lock files MUST use the <a class="reference external" href="https://toml.io">TOML</a> file format. This not only prevents the need to have another file format in the Python packaging ecosystem thanks to its adoption by <a class="pep reference internal" href="../pep-0518/" title="PEP 518 – Specifying Minimum Build System Requirements for Python Projects">PEP 518</a> for <code class="docutils literal notranslate"><span class="pre">pyproject.toml</span></code>, but also assists in making lock files more human-readable.</p> <p>Lock files MUST end their file names with <code class="docutils literal notranslate"><span class="pre">.pylock.toml</span></code>. The <code class="docutils literal notranslate"><span class="pre">.toml</span></code> part unambiguously distinguishes the format of the file, and helps tools like code editors support the file appropriately. The <code class="docutils literal notranslate"><span class="pre">.pylock</span></code> part distinguishes the file from other TOML files the user has, to make the logic easier for tools to create functionality specific to Python lock files, instead of TOML files in general.</p> <p>The following sections are the top-level keys of the TOML file data format. Any field not listed as <strong>required</strong> is considered optional.</p> <section id="version"> <h4><a class="toc-backref" href="#version" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">version</span></code></a></h4> <p>This field is <strong>required</strong>.</p> <p>The version of the lock file being used. The key MUST be a string consisting of a number that follows the same formatting as the <code class="docutils literal notranslate"><span class="pre">Metadata-Version</span></code> key in the <a class="reference external" href="https://packaging.python.org/specifications/core-metadata/">core metadata spec</a>.</p> <p>The value MUST be set to <code class="docutils literal notranslate"><span class="pre">"1.0"</span></code> until a future PEP allows for a different value. The introduction of a new <em>optional</em> key to the file format SHOULD increase the minor version. The introduction of a new required key or changing the format MUST increase the major version. How to handle other scenarios is left as a per-PEP decision.</p> <p>Installers MUST warn the user if the lock file specifies a version whose major version is supported but whose minor version is unsupported/unrecognized (e.g. the installer supports <code class="docutils literal notranslate"><span class="pre">"1.0"</span></code>, but the lock file specifies <code class="docutils literal notranslate"><span class="pre">"1.1"</span></code>).</p> <p>Installers MUST raise an error if the lock file specifies a major version which is unsupported (e.g. the installer supports <code class="docutils literal notranslate"><span class="pre">"1.9"</span></code> but the lock file specifies <code class="docutils literal notranslate"><span class="pre">"2.0"</span></code>).</p> </section> <section id="created-at"> <h4><a class="toc-backref" href="#created-at" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">created-at</span></code></a></h4> <p>This field is <strong>required</strong>.</p> <p>The timestamp for when the lock file was generated (using TOML’s native timestamp type). It MUST be recorded using the UTC time zone to avoid ambiguity.</p> <p>If the <a class="reference external" href="https://reproducible-builds.org/specs/source-date-epoch/">SOURCE_DATE_EPOCH</a> environment variable is set, it MUST be used as the timestamp by the locker. This facilitates reproducibility of the lock file itself.</p> </section> <section id="tool"> <h4><a class="toc-backref" href="#tool" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">[tool]</span></code></a></h4> <p>Tools may create their own sub-tables under the <code class="docutils literal notranslate"><span class="pre">tool</span></code> table. The rules for this table match those for <code class="docutils literal notranslate"><span class="pre">pyproject.toml</span></code> and its <code class="docutils literal notranslate"><span class="pre">[tool]</span></code> table from the <a class="reference external" href="https://packaging.python.org/specifications/declaring-build-dependencies/">build system declaration spec</a>.</p> </section> <section id="metadata"> <h4><a class="toc-backref" href="#metadata" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">[metadata]</span></code></a></h4> <p>This table is <strong>required</strong>.</p> <p>A table containing data applying to the overall lock file.</p> <section id="metadata-marker"> <h5><a class="toc-backref" href="#metadata-marker" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">metadata.marker</span></code></a></h5> <p>A key storing a string containing an environment marker as specified in the <a class="reference external" href="https://packaging.python.org/specifications/dependency-specifiers/">dependency specifier spec</a>.</p> <p>The locker MAY specify an environment marker which specifies any restrictions the lock file was generated under.</p> <p>If the installer is installing for an environment which does not satisfy the specified environment marker, the installer MUST raise an error as the lock file does not support the target installation environment.</p> </section> <section id="metadata-tag"> <h5><a class="toc-backref" href="#metadata-tag" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">metadata.tag</span></code></a></h5> <p>A key storing a string specifying <a class="reference external" href="https://packaging.python.org/specifications/platform-compatibility-tags/">platform compatibility tags</a> (i.e. wheel tags). The tag MAY be a compressed tag set.</p> <p>If the installer is installing for an environment which does not satisfy the specified tag (set), the installer MUST raise an error as the lock file does not support the targeted installation environment.</p> </section> <section id="metadata-requires"> <h5><a class="toc-backref" href="#metadata-requires" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">metadata.requires</span></code></a></h5> <p>This field is <strong>required</strong>.</p> <p>An array of strings following the <a class="reference external" href="https://packaging.python.org/specifications/dependency-specifiers/">dependency specifier spec</a>. This array represents the top-level package dependencies of the lock file and thus the root of the dependency graph.</p> </section> <section id="metadata-requires-python"> <h5><a class="toc-backref" href="#metadata-requires-python" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">metadata.requires-python</span></code></a></h5> <p>A string specifying the supported version(s) of Python for this lock file. It follows the same format as that specified for the <code class="docutils literal notranslate"><span class="pre">Requires-Python</span></code> field in the <a class="reference external" href="https://packaging.python.org/specifications/core-metadata/">core metadata spec</a>.</p> </section> </section> <section id="package-name-version"> <h4><a class="toc-backref" href="#package-name-version" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">[[package._name_._version_]]</span></code></a></h4> <p>This array is <strong>required</strong>.</p> <p>An array per package and version containing entries for the potential (wheel) files to install (as represented by <code class="docutils literal notranslate"><span class="pre">_name_</span></code> and <code class="docutils literal notranslate"><span class="pre">_version_</span></code>, respectively).</p> <p>Lockers MUST normalize a project’s name according to the <a class="reference external" href="https://packaging.python.org/specifications/simple-repository-api/">simple repository API</a>. If extras are specified as part of the project to install, the extras are to be included in the key name and are to be sorted in lexicographic order.</p> <p>Within the file, the tables for the projects SHOULD be sorted by:</p> <ol class="arabic simple"> <li>Project/key name in lexicographic order</li> <li>Package version, newest/highest to older/lowest according to the <a class="reference external" href="https://packaging.python.org/specifications/version-specifiers/">version specifiers spec</a></li> <li>Optional dependencies (extras) via lexicographic order</li> <li>File name based on the <code class="docutils literal notranslate"><span class="pre">filename</span></code> field (discussed below)</li> </ol> <p>These recommendations are to help minimize diff changes between tool executions.</p> <section id="package-name-version-filename"> <h5><a class="toc-backref" href="#package-name-version-filename" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">package._name_._version_.filename</span></code></a></h5> <p>This field is <strong>required</strong>.</p> <p>A string representing the base name of the file as represented by an entry in the array (i.e. what <code class="docutils literal notranslate"><span class="pre">os.path.basename()</span></code>/<code class="docutils literal notranslate"><span class="pre">pathlib.PurePath.name</span></code> represents). This field is required to simplify installers as the file name is required to resolve wheel tags derived from the file name. It also guarantees that the association of the array entry to the file it is meant for is always clear.</p> </section> <section id="package-name-version-hashes"> <h5><a class="toc-backref" href="#package-name-version-hashes" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">[package._name_._version_.hashes]</span></code></a></h5> <p>This table is <strong>required</strong>.</p> <p>A table with keys specifying a hash algorithm and values as the hash for the file represented by this entry in the <code class="docutils literal notranslate"><span class="pre">package._name_._version_</span></code> table.</p> <p>Lockers SHOULD list hashes in lexicographic order. This is to help minimize diff sizes and the potential to overlook hash value changes.</p> <p>An installer MUST only install a file which matches one of the specified hashes.</p> </section> <section id="package-name-version-url"> <h5><a class="toc-backref" href="#package-name-version-url" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">package._name_._version_.url</span></code></a></h5> <p>A string representing a URL where to get the file.</p> <p>The installer MAY support any schemes it wants for URLs. A URL with no scheme MUST be assumed to be a local file path (both relative paths to the lock file and absolute paths). Installers MUST support, at minimum, HTTPS URLs as well as local file paths.</p> <p>An installer MAY choose to not use the URL to retrieve a file if a file matching the specified hash can be found using alternative means (e.g. on the file system in a cache directory).</p> </section> <section id="package-name-version-direct"> <h5><a class="toc-backref" href="#package-name-version-direct" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">package._name_._version_.direct</span></code></a></h5> <p>A boolean representing whether an installer should consider the project installed “directly” as specified by the <a class="reference external" href="https://packaging.python.org/specifications/direct-url/">direct URL origin of installed distributions spec</a>.</p> <p>If the key is true, then the installer MUST follow the <a class="reference external" href="https://packaging.python.org/specifications/direct-url/">direct URL origin of installed distributions spec</a> for recording the installation as “direct”.</p> </section> <section id="package-name-version-requires-python"> <h5><a class="toc-backref" href="#package-name-version-requires-python" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">package._name_._version_.requires-python</span></code></a></h5> <p>A string specifying the support version(s) of Python for this file. It follows the same format as that specified for the <code class="docutils literal notranslate"><span class="pre">Requires-Python</span></code> field in the <a class="reference external" href="https://packaging.python.org/specifications/core-metadata/">core metadata spec</a>.</p> </section> <section id="package-name-version-requires"> <h5><a class="toc-backref" href="#package-name-version-requires" role="doc-backlink"><code class="docutils literal notranslate"><span class="pre">package._name_._version_.requires</span></code></a></h5> <p>An array of strings following the <a class="reference external" href="https://packaging.python.org/specifications/dependency-specifiers/">dependency specifier spec</a> which represent the dependencies of this file.</p> </section> </section> </section> <section id="example"> <h3><a class="toc-backref" href="#example" role="doc-backlink">Example</a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">version</span> <span class="o">=</span> <span class="s2">"1.0"</span> <span class="n">created</span><span class="o">-</span><span class="n">at</span> <span class="o">=</span> <span class="mi">2021</span><span class="o">-</span><span class="mi">10</span><span class="o">-</span><span class="mi">19</span><span class="n">T22</span><span class="p">:</span><span class="mi">33</span><span class="p">:</span><span class="mf">45.520739</span><span class="o">+</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span> <span class="p">[</span><span class="n">tool</span><span class="p">]</span> <span class="c1"># Tool-specific table.</span> <span class="p">[</span><span class="n">metadata</span><span class="p">]</span> <span class="n">requires</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"mousebender"</span><span class="p">,</span> <span class="s2">"coveragepy[toml]"</span><span class="p">]</span> <span class="n">marker</span> <span class="o">=</span> <span class="s2">"sys_platform == 'linux'"</span> <span class="c1"># As an example for coverage.</span> <span class="n">requires</span><span class="o">-</span><span class="n">python</span> <span class="o">=</span> <span class="s2">">=3.7"</span> <span class="p">[[</span><span class="n">package</span><span class="o">.</span><span class="n">attrs</span><span class="o">.</span><span class="s2">"21.2.0"</span><span class="p">]]</span> <span class="n">filename</span> <span class="o">=</span> <span class="s2">"attrs-21.2.0-py2.py3-none-any.whl"</span> <span class="n">hashes</span><span class="o">.</span><span class="n">sha256</span> <span class="o">=</span> <span class="s2">"149e90d6d8ac20db7a955ad60cf0e6881a3f20d37096140088356da6c716b0b1"</span> <span class="n">url</span> <span class="o">=</span> <span class="s2">"https://files.pythonhosted.org/packages/20/a9/ba6f1cd1a1517ff022b35acd6a7e4246371dfab08b8e42b829b6d07913cc/attrs-21.2.0-py2.py3-none-any.whl"</span> <span class="n">requires</span><span class="o">-</span><span class="n">python</span> <span class="o">=</span> <span class="s2">">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"</span> <span class="p">[[</span><span class="n">package</span><span class="o">.</span><span class="n">attrs</span><span class="o">.</span><span class="s2">"21.2.0"</span><span class="p">]]</span> <span class="c1"># If attrs had another wheel file (e.g. that was platform-specific),</span> <span class="c1"># it could be listed here.</span> <span class="p">[[</span><span class="n">package</span><span class="o">.</span><span class="s2">"coveragepy[toml]"</span><span class="o">.</span><span class="s2">"6.2.0"</span><span class="p">]]</span> <span class="n">filename</span> <span class="o">=</span> <span class="s2">"coverage-6.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl"</span> <span class="n">hashes</span><span class="o">.</span><span class="n">sha256</span> <span class="o">=</span> <span class="s2">"c7912d1526299cb04c88288e148c6c87c0df600eca76efd99d84396cfe00ef1d"</span> <span class="n">url</span> <span class="o">=</span> <span class="s2">"https://files.pythonhosted.org/packages/da/64/468ca923e837285bd0b0a60bd9a287945d6b68e325705b66b368c07518b1/coverage-6.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl"</span> <span class="n">requires</span><span class="o">-</span><span class="n">python</span> <span class="o">=</span> <span class="s2">">=3.6"</span> <span class="n">requires</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"tomli"</span><span class="p">]</span> <span class="p">[[</span><span class="n">package</span><span class="o">.</span><span class="s2">"coveragepy[toml]"</span><span class="o">.</span><span class="s2">"6.2.0"</span><span class="p">]]</span> <span class="n">filename</span> <span class="o">=</span> <span class="s2">"coverage-6.2-cp310-cp310-musllinux_1_1_x86_64.whl "</span> <span class="n">hashes</span><span class="o">.</span><span class="n">sha256</span> <span class="o">=</span> <span class="s2">"276651978c94a8c5672ea60a2656e95a3cce2a3f31e9fb2d5ebd4c215d095840"</span> <span class="n">url</span> <span class="o">=</span> <span class="s2">"https://files.pythonhosted.org/packages/17/d6/a29f2cccacf2315150c31d8685b4842a6e7609279939a478725219794355/coverage-6.2-cp310-cp310-musllinux_1_1_x86_64.whl"</span> <span class="n">requires</span><span class="o">-</span><span class="n">python</span> <span class="o">=</span> <span class="s2">">=3.6"</span> <span class="n">requires</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"tomli"</span><span class="p">]</span> <span class="c1"># More wheel files for `coverage` could be listed for more</span> <span class="c1"># extensive support (i.e. all Linux-based wheels).</span> <span class="p">[[</span><span class="n">package</span><span class="o">.</span><span class="n">mousebender</span><span class="o">.</span><span class="s2">"2.0.0"</span><span class="p">]]</span> <span class="n">filename</span> <span class="o">=</span> <span class="s2">"mousebender-2.0.0-py3-none-any.whl"</span> <span class="n">hashes</span><span class="o">.</span><span class="n">sha256</span> <span class="o">=</span> <span class="s2">"a6f9adfbd17bfb0e6bb5de9a27083e01dfb86ed9c3861e04143d9fd6db373f7c"</span> <span class="n">url</span> <span class="o">=</span> <span class="s2">"https://files.pythonhosted.org/packages/f4/b3/f6fdbff6395e9b77b5619160180489410fb2f42f41272994353e7ecf5bdf/mousebender-2.0.0-py3-none-any.whl"</span> <span class="n">requires</span><span class="o">-</span><span class="n">python</span> <span class="o">=</span> <span class="s2">">=3.6"</span> <span class="n">requires</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"attrs"</span><span class="p">,</span> <span class="s2">"packaging"</span><span class="p">]</span> <span class="p">[[</span><span class="n">package</span><span class="o">.</span><span class="n">packaging</span><span class="o">.</span><span class="s2">"20.9"</span><span class="p">]]</span> <span class="n">filename</span> <span class="o">=</span> <span class="s2">"packaging-20.9-py2.py3-none-any.whl"</span> <span class="n">hashes</span><span class="o">.</span><span class="n">blake</span><span class="o">-</span><span class="mi">256</span> <span class="o">=</span> <span class="s2">"3e897ea760b4daa42653ece2380531c90f64788d979110a2ab51049d92f408af"</span> <span class="n">hashes</span><span class="o">.</span><span class="n">sha256</span> <span class="o">=</span> <span class="s2">"67714da7f7bc052e064859c05c595155bd1ee9f69f76557e21f051443c20947a"</span> <span class="n">url</span> <span class="o">=</span> <span class="s2">"https://files.pythonhosted.org/packages/3e/89/7ea760b4daa42653ece2380531c90f64788d979110a2ab51049d92f408af/packaging-20.9-py2.py3-none-any.whl"</span> <span class="n">requires</span><span class="o">-</span><span class="n">python</span> <span class="o">=</span> <span class="s2">">=3.6"</span> <span class="n">requires</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"pyparsing"</span><span class="p">]</span> <span class="p">[[</span><span class="n">package</span><span class="o">.</span><span class="n">pyparsing</span><span class="o">.</span><span class="s2">"2.4.7"</span><span class="p">]]</span> <span class="n">filename</span> <span class="o">=</span> <span class="s2">"pyparsing-2.4.7-py2.py3-none-any.whl"</span> <span class="n">hashes</span><span class="o">.</span><span class="n">sha256</span> <span class="o">=</span> <span class="s2">"ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b"</span> <span class="n">url</span> <span class="o">=</span> <span class="s2">"https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl"</span> <span class="n">direct</span> <span class="o">=</span> <span class="n">true</span> <span class="c1"># For demonstration purposes.</span> <span class="n">requires</span><span class="o">-</span><span class="n">python</span> <span class="o">=</span> <span class="s2">">=2.6, !=3.0.*, !=3.1.*, !=3.2.*"</span> <span class="p">[[</span><span class="n">package</span><span class="o">.</span><span class="n">tomli</span><span class="o">.</span><span class="s2">"2.0.0"</span><span class="p">]]</span> <span class="n">filename</span> <span class="o">=</span> <span class="s2">"tomli-2.0.0-py3-none-any.whl"</span> <span class="n">hashes</span><span class="o">.</span><span class="n">sha256</span> <span class="o">=</span> <span class="s2">"b5bde28da1fed24b9bd1d4d2b8cba62300bfb4ec9a6187a957e8ddb9434c5224"</span> <span class="n">url</span> <span class="o">=</span> <span class="s2">"https://files.pythonhosted.org/packages/e2/9f/5e1557a57a7282f066351086e78f87289a3446c47b2cb5b8b2f614d8fe99/tomli-2.0.0-py3-none-any.whl"</span> <span class="n">requires</span><span class="o">-</span><span class="n">python</span> <span class="o">=</span> <span class="s2">">=3.7"</span> </pre></div> </div> </section> <section id="expectations-for-lockers"> <h3><a class="toc-backref" href="#expectations-for-lockers" role="doc-backlink">Expectations for Lockers</a></h3> <p>Lockers MUST create lock files for which a topological sort of the packages which qualify for installation on the specified platform results in a graph for which only a single version of any package qualifies for installation and there is at least one compatible file to install for each package. This leads to a lock file for any supported platform where the only decision an installer can make is what the “best-fitting” wheel is to install (which is discussed below).</p> <p>Lockers are expected to utilize <code class="docutils literal notranslate"><span class="pre">metadata.marker</span></code>, <code class="docutils literal notranslate"><span class="pre">metadata.tag</span></code>, and <code class="docutils literal notranslate"><span class="pre">metadata.requires-python</span></code> as appropriate as well as environment markers specified via <code class="docutils literal notranslate"><span class="pre">requires</span></code> and Python version requirements via <code class="docutils literal notranslate"><span class="pre">requires-python</span></code> to enforce this result for installers. Put another way, the information used in the lock file is not expected to be pristine/raw from the locker’s input and instead is to be changed as necessary to the benefit of the locker’s goals.</p> </section> <section id="expectations-for-installers"> <h3><a class="toc-backref" href="#expectations-for-installers" role="doc-backlink">Expectations for Installers</a></h3> <p>The expected algorithm for resolving what to install is:</p> <ol class="arabic simple"> <li>Construct a dependency graph based on the data in the lock file with <code class="docutils literal notranslate"><span class="pre">metadata.requires</span></code> as the starting/root point.</li> <li>Eliminate all files that are unsupported by the specified platform.</li> <li>Eliminate all irrelevant edges between packages based on marker evaluation for <code class="docutils literal notranslate"><span class="pre">requires</span></code>.</li> <li>Raise an error if a package version is still reachable from the root of the dependency graph but lacks any compatible file.</li> <li>Verify that all packages left only have one version to install, raising an error otherwise.</li> <li>Install the best-fitting wheel file for each package which remains.</li> </ol> <p>Installers MUST follow a deterministic algorithm determine what the “best-fitting wheel file” is. A simple solution for this is to rely upon the <a class="reference external" href="https://pypi.org/p/packaging/">packaging project</a> and its <code class="docutils literal notranslate"><span class="pre">packaging.tags</span></code> module to determine wheel file precedence.</p> <p>Installers MUST support installing into an empty environment. Installers MAY support installing into an environment that already contains installed packages (and whatever that would entail to be supported).</p> </section> </section> <section id="potential-tool-support"> <h2><a class="toc-backref" href="#potential-tool-support" role="doc-backlink">(Potential) Tool Support</a></h2> <p>The <a class="reference external" href="https://pip.pypa.io/">pip</a> team has <a class="reference external" href="https://github.com/pypa/pip/issues/10636">said</a> they are interested in supporting this PEP if accepted. The current proposal for pip may even <a class="reference external" href="https://github.com/jazzband/pip-tools/issues/1526#issuecomment-961883367">supplant the need</a> for <a class="reference external" href="https://pypi.org/project/pip-tools/">pip-tools</a>.</p> <p><a class="reference external" href="https://pypi.org/project/pdm/">PDM</a> has also said they would <a class="reference external" href="https://github.com/pdm-project/pdm/issues/718">support the PEP</a> if accepted.</p> <p><a class="reference external" href="https://pypi.org/project/pyflow/">Pyflow</a> has said they <a class="reference external" href="https://github.com/David-OConnor/pyflow/issues/153#issuecomment-962482058">“like the idea”</a> of the PEP.</p> <p><a class="reference external" href="https://pypi.org/project/poetry/">Poetry</a> has said they would <strong>not</strong> support the PEP as-is because <a class="reference external" href="https://github.com/python-poetry/poetry/issues/4710#issuecomment-973946104">“Poetry supports sdists files, directory and VCS dependencies which are not supported”</a>. Recording requirements at the file level, which is on purpose to better reflect what can occur when it comes to dependencies, <a class="reference external" href="https://github.com/python-poetry/poetry/issues/4710#issuecomment-973946104">“is contradictory to the design of Poetry”</a>. This also excludes export support to a this PEP’s lock file as <a class="reference external" href="https://github.com/python-poetry/poetry/issues/4710#issuecomment-974551351">“Poetry exports the information present in the poetry.lock file into another format”</a> and sdists and source trees are included in <code class="docutils literal notranslate"><span class="pre">Poetry.lock</span></code> files. Thus it is not a clean translation from Poetry’s lock file to this PEP’s lock file format.</p> </section> <section id="backwards-compatibility"> <h2><a class="toc-backref" href="#backwards-compatibility" role="doc-backlink">Backwards Compatibility</a></h2> <p>As there is no pre-existing specification regarding lock files, there are no explicit backwards compatibility concerns.</p> <p>As for pre-existing tools that have their own lock file, some updating will be required. Most document the lock file name, but not its contents. For projects which do not commit their lock file to version control, they will need to update the equivalent of their <code class="docutils literal notranslate"><span class="pre">.gitignore</span></code> file. For projects that do commit their lock file to version control, what file(s) get committed will need an update.</p> <p>For projects which do document their lock file format like <a class="reference external" href="https://pypi.org/project/pipenv/">pipenv</a>, they will very likely need a major version release which changes the lock file format.</p> </section> <section id="transition-plan"> <h2><a class="toc-backref" href="#transition-plan" role="doc-backlink">Transition Plan</a></h2> <p>In general, this PEP could be considered successful if:</p> <ol class="arabic simple"> <li>Two pre-existing tools became lockers (e.g. <a class="reference external" href="https://pypi.org/project/pip-tools/">pip-tools</a>, <a class="reference external" href="https://pypi.org/project/pdm/">PDM</a>, <a class="reference external" href="https://pip.pypa.io/">pip</a> via <code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">freeze</span></code>).</li> <li>Pip became an installer.</li> <li>One major, non-Python-specific platform supported the file format (e.g. a cloud provider).</li> </ol> <p>This would show interoperability, usability, and programming community/business acceptance.</p> <p>In terms of a transition plan, there are potentially multiple steps that could lead to this desired outcome. Below is a somewhat idealized plan that would see this PEP being broadly used.</p> <section id="usability"> <h3><a class="toc-backref" href="#usability" role="doc-backlink">Usability</a></h3> <p>First, a <code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">freeze</span></code> equivalent tool could be developed which creates a lock file. While installed packages do not by themselves provide enough information to statically create a lock file, a user could provide local directories and index URLs to construct one. This would then lead to lock files that are stricter than a requirements file by limiting the lock file to the current platform. This would also allow people to see whether their environment would be reproducible.</p> <p>Second, a stand-alone installer should be developed. As the requirements on an installer are much simpler than what pip provides, it should be reasonable to have an installer that is independently developed.</p> <p>Third, a tool to convert a pinned requirements file as emitted by pip-tools could be developed. Much like the <code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">freeze</span></code> equivalent outlined above, some input from the user may be needed. But this tool could act as a transitioning step for anyone who has an appropriate requirements file. This could also act as a test before potentially having pip-tools grow some <code class="docutils literal notranslate"><span class="pre">--lockfile</span></code> flag to use this PEP.</p> <p>All of this could be required before the PEP transitions from conditional acceptance to full acceptance (and give the community a chance to test if this PEP is potentially useful).</p> </section> <section id="interoperability"> <h3><a class="toc-backref" href="#interoperability" role="doc-backlink">Interoperability</a></h3> <p>At this point, the goal would be to increase interoperability between tools.</p> <p>First, pip would become an installer. By having the most widely used installer support the format, people can innovate on the locker side while knowing people will have the tools necessary to actually consume a lock file.</p> <p>Second, pip becomes a locker. Once again, pip’s reach would make the format accessible for the vast majority of Python users very quickly.</p> <p>Third, a project with a pre-existing lock file format supports at least exporting to the lock file format (e.g. PDM or Pyflow). This would show that the format meets the needs of other projects.</p> </section> <section id="acceptance"> <h3><a class="toc-backref" href="#acceptance" role="doc-backlink">Acceptance</a></h3> <p>With the tooling available throughout the community, acceptance would be shown via those not exclusively tied to the Python community supporting the file format based on what they believe their users want.</p> <p>First, tools that operate on requirements files like code editors having equivalent support for lock files.</p> <p>Second, consumers of requirements files like cloud providers would also accept lock files.</p> <p>At this point the PEP would have permeated out far enough to be on par with requirements files in terms of general acceptance and potentially more if projects had dropped their own lock files for this PEP.</p> </section> </section> <section id="security-implications"> <h2><a class="toc-backref" href="#security-implications" role="doc-backlink">Security Implications</a></h2> <p>A lock file should not introduce security issues but instead help solve them. By requiring the recording of hashes for files, a lock file is able to help prevent tampering with code since the hash details were recorded. Relying on only wheel files means what files will be installed can be known ahead of time and is reproducible. A lock file also helps prevent unexpected package updates being installed which may in turn be malicious.</p> </section> <section id="how-to-teach-this"> <h2><a class="toc-backref" href="#how-to-teach-this" role="doc-backlink">How to Teach This</a></h2> <p>Teaching of this PEP will very much be dependent on the lockers and installers being used for day-to-day use. Conceptually, though, users could be taught that a lock file specifies what should be installed for a project to work. The benefits of consistency and security should be emphasized to help users realize why they should care about lock files.</p> </section> <section id="reference-implementation"> <h2><a class="toc-backref" href="#reference-implementation" role="doc-backlink">Reference Implementation</a></h2> <p>A proof-of-concept locker can be found at <a class="reference external" href="https://github.com/frostming/pep665_poc">https://github.com/frostming/pep665_poc</a> . No installer has been implemented yet, but the design of this PEP suggests the locker is the more difficult aspect to implement.</p> </section> <section id="rejected-ideas"> <h2><a class="toc-backref" href="#rejected-ideas" role="doc-backlink">Rejected Ideas</a></h2> <section id="file-formats-other-than-toml"> <h3><a class="toc-backref" href="#file-formats-other-than-toml" role="doc-backlink">File Formats Other Than TOML</a></h3> <p><a class="reference external" href="https://www.json.org/">JSON</a> was briefly considered, but due to:</p> <ol class="arabic simple"> <li>TOML already being used for <code class="docutils literal notranslate"><span class="pre">pyproject.toml</span></code></li> <li>TOML being more human-readable</li> <li>TOML leading to better diffs</li> </ol> <p>the decision was made to go with TOML. There was some concern over Python’s standard library lacking a TOML parser, but most packaging tools already use a TOML parser thanks to <code class="docutils literal notranslate"><span class="pre">pyproject.toml</span></code> so this issue did not seem to be a showstopper. Some have also argued against this concern in the past by the fact that if packaging tools abhor installing dependencies and feel they can’t vendor a package then the packaging ecosystem has much bigger issues to rectify than the need to depend on a third-party TOML parser.</p> </section> <section id="alternative-naming-schemes"> <h3><a class="toc-backref" href="#alternative-naming-schemes" role="doc-backlink">Alternative Naming Schemes</a></h3> <p>Specifying a directory to install file to was considered, but ultimately rejected due to people’s distaste for the idea.</p> <p>It was also suggested to not have a special file name suffix, but it was decided that hurt discoverability by tools too much.</p> </section> <section id="supporting-a-single-lock-file"> <h3><a class="toc-backref" href="#supporting-a-single-lock-file" role="doc-backlink">Supporting a Single Lock File</a></h3> <p>At one point the idea of only supporting single lock file which contained all possible lock information was considered. But it quickly became apparent that trying to devise a data format which could encompass both a lock file format which could support multiple environments as well as strict lock outcomes for reproducible builds would become quite complex and cumbersome.</p> <p>The idea of supporting a directory of lock files as well as a single lock file named <code class="docutils literal notranslate"><span class="pre">pyproject-lock.toml</span></code> was also considered. But any possible simplicity from skipping the directory in the case of a single lock file seemed unnecessary. Trying to define appropriate logic for what should be the <code class="docutils literal notranslate"><span class="pre">pyproject-lock.toml</span></code> file and what should go into <code class="docutils literal notranslate"><span class="pre">pyproject-lock.d</span></code> seemed unnecessarily complicated.</p> </section> <section id="using-a-flat-list-instead-of-a-dependency-graph"> <h3><a class="toc-backref" href="#using-a-flat-list-instead-of-a-dependency-graph" role="doc-backlink">Using a Flat List Instead of a Dependency Graph</a></h3> <p>The first version of this PEP proposed that the lock file have no concept of a dependency graph. Instead, the lock file would list exactly what should be installed for a specific platform such that installers did not have to make any decisions about <em>what</em> to install, only validating that the lock file would work for the target platform.</p> <p>This idea was eventually rejected due to the number of combinations of potential <a class="pep reference internal" href="../pep-0508/" title="PEP 508 – Dependency specification for Python Software Packages">PEP 508</a> environment markers. The decision was made that trying to have lockers generate all possible combinations as individual lock files when a project wants to be cross-platform would be too much.</p> </section> <section id="use-wheel-tags-in-the-file-name"> <h3><a class="toc-backref" href="#use-wheel-tags-in-the-file-name" role="doc-backlink">Use Wheel Tags in the File Name</a></h3> <p>Instead of having the <code class="docutils literal notranslate"><span class="pre">metadata.tag</span></code> field there was a suggestion of encoding the tags into the file name. But due to the addition of the <code class="docutils literal notranslate"><span class="pre">metadata.marker</span></code> field and what to do when no tags were needed, the idea was dropped.</p> </section> <section id="alternative-names-for-requires"> <h3><a class="toc-backref" href="#alternative-names-for-requires" role="doc-backlink">Alternative Names for <code class="docutils literal notranslate"><span class="pre">requires</span></code></a></h3> <p>Some other names for what became <code class="docutils literal notranslate"><span class="pre">requires</span></code> were <code class="docutils literal notranslate"><span class="pre">installs</span></code>, <code class="docutils literal notranslate"><span class="pre">needs</span></code>, and <code class="docutils literal notranslate"><span class="pre">dependencies</span></code>. Initially this PEP chose <code class="docutils literal notranslate"><span class="pre">needs</span></code> after asking a Python beginner which term they preferred. But based on feedback on an earlier draft of this PEP, <code class="docutils literal notranslate"><span class="pre">requires</span></code> was chosen as the term.</p> </section> <section id="accepting-pep-650"> <h3><a class="toc-backref" href="#accepting-pep-650" role="doc-backlink">Accepting PEP 650</a></h3> <p><a class="pep reference internal" href="../pep-0650/" title="PEP 650 – Specifying Installer Requirements for Python Projects">PEP 650</a> was an earlier attempt at trying to tackle this problem by specifying an API for installers instead of standardizing on a lock file format (ala <a class="pep reference internal" href="../pep-0517/" title="PEP 517 – A build-system independent format for source trees">PEP 517</a>). The <a class="reference external" href="https://discuss.python.org/t/pep-650-specifying-installer-requirements-for-python-projects/6657/">initial response</a> to <a class="pep reference internal" href="../pep-0650/" title="PEP 650 – Specifying Installer Requirements for Python Projects">PEP 650</a> could be considered mild/lukewarm. People seemed to be consistently confused over which tools should provide what functionality to implement the PEP. It also potentially incurred more overhead as it would require executing Python APIs to perform any actions involving packaging.</p> <p>This PEP chooses to standardize around an artifact instead of an API (ala <a class="pep reference internal" href="../pep-0621/" title="PEP 621 – Storing project metadata in pyproject.toml">PEP 621</a>). This would allow for more tool integrations as it removes the need to specifically use Python to do things such as create a lock file, update it, or even install packages listed in a lock file. It also allows for easier introspection by forcing dependency graph details to be written in a human-readable format. It also allows for easier sharing of knowledge by standardizing what people need to know more (e.g. tutorials become more portable between tools when it comes to understanding the artifact they produce). It’s also simply the approach other language communities have taken and seem to be happy with.</p> <p>Acceptance of this PEP would mean <a class="pep reference internal" href="../pep-0650/" title="PEP 650 – Specifying Installer Requirements for Python Projects">PEP 650</a> gets rejected.</p> </section> <section id="specifying-requirements-per-package-instead-of-per-file"> <h3><a class="toc-backref" href="#specifying-requirements-per-package-instead-of-per-file" role="doc-backlink">Specifying Requirements per Package Instead of per File</a></h3> <p>An earlier draft of this PEP specified dependencies at the package level instead of per file. While this has traditionally been how packaging systems work, it actually did not reflect accurately how things are specified. As such, this PEP was subsequently updated to reflect the granularity that dependencies can truly be specified at.</p> </section> <section id="specify-where-lockers-gather-input"> <h3><a class="toc-backref" href="#specify-where-lockers-gather-input" role="doc-backlink">Specify Where Lockers Gather Input</a></h3> <p>This PEP does not specify how a locker gets its input. An initial suggestion was to partially reuse <a class="pep reference internal" href="../pep-0621/" title="PEP 621 – Storing project metadata in pyproject.toml">PEP 621</a>, but due to disagreements on how flexible the potential input should be in terms of specifying things such as indexes, etc., it was decided this would best be left to a separate PEP.</p> </section> <section id="allowing-source-distributions-and-source-trees-to-be-an-opt-in-supported-file-format"> <h3><a class="toc-backref" href="#allowing-source-distributions-and-source-trees-to-be-an-opt-in-supported-file-format" role="doc-backlink">Allowing Source Distributions and Source Trees to be an Opt-In, Supported File Format</a></h3> <p>After <a class="reference external" href="https://discuss.python.org/t/supporting-sdists-and-source-trees-in-pep-665/11869/">extensive discussion</a>, it was decided that this PEP would not support source distributions (aka sdists) or source trees as an acceptable format for code. Introducing sdists and source trees to this PEP would immediately undo the reproducibility and security goals due to needing to execute code to build the sdist or source tree. It would also greatly increase the complexity for (at least) installers as the dynamic build nature of sdists and source trees means the installer would need to handle fully resolving whatever requirements the sdists produced dynamically, both from a building and installation perspective.</p> <p>Due to all of this, it was decided it was best to have a separate discussion about what supporting sdists and source trees <strong>after</strong> this PEP is accepted/rejected. As the proposed file format is versioned, introducing sdists and source tree support in a later PEP is doable.</p> <p>It should be noted, though, that this PEP is <strong>not</strong> stop an out-of-band solution from being developed to be used in conjunction with this PEP. Building wheel files from sdists and shipping them with code upon deployment so they can be included in the lock file is one option. Another is to use a requirements file <em>just</em> for sdists and source trees, then relying on a lock file for all wheels.</p> </section> </section> <section id="open-issues"> <h2><a class="toc-backref" href="#open-issues" role="doc-backlink">Open Issues</a></h2> <p>None.</p> </section> <section id="acknowledgments"> <h2><a class="toc-backref" href="#acknowledgments" role="doc-backlink">Acknowledgments</a></h2> <p>Thanks to Frost Ming of <a class="reference external" href="https://pypi.org/project/pdm/">PDM</a> and Sébastien Eustace of <a class="reference external" href="https://pypi.org/project/poetry/">Poetry</a> for providing input around dynamic install-time resolution of <a class="pep reference internal" href="../pep-0508/" title="PEP 508 – Dependency specification for Python Software Packages">PEP 508</a> requirements.</p> <p>Thanks to Kushal Das for making sure reproducible builds stayed a concern for this PEP.</p> <p>Thanks to Andrea McInnes for initially settling the bikeshedding and choosing the paint colour of <code class="docutils literal notranslate"><span class="pre">needs</span></code> (at which point people ralled around the <code class="docutils literal notranslate"><span class="pre">requires</span></code> colour instead).</p> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> <p>This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.</p> </section> </section> <hr class="docutils" /> <p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0665.rst">https://github.com/python/peps/blob/main/peps/pep-0665.rst</a></p> <p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0665.rst">2024-07-26 12:58:25 GMT</a></p> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#terminology">Terminology</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#file-format">File Format</a></li> <li><a class="reference internal" href="#secure-by-design">Secure by Design</a></li> <li><a class="reference internal" href="#cross-platform">Cross-Platform</a></li> <li><a class="reference internal" href="#simple-installer">Simple Installer</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#details">Details</a><ul> <li><a class="reference internal" href="#version"><code class="docutils literal notranslate"><span class="pre">version</span></code></a></li> <li><a class="reference internal" href="#created-at"><code class="docutils literal notranslate"><span class="pre">created-at</span></code></a></li> <li><a class="reference internal" href="#tool"><code class="docutils literal notranslate"><span class="pre">[tool]</span></code></a></li> <li><a class="reference internal" href="#metadata"><code class="docutils literal notranslate"><span class="pre">[metadata]</span></code></a><ul> <li><a class="reference internal" href="#metadata-marker"><code class="docutils literal notranslate"><span class="pre">metadata.marker</span></code></a></li> <li><a class="reference internal" href="#metadata-tag"><code class="docutils literal notranslate"><span class="pre">metadata.tag</span></code></a></li> <li><a class="reference internal" href="#metadata-requires"><code class="docutils literal notranslate"><span class="pre">metadata.requires</span></code></a></li> <li><a class="reference internal" href="#metadata-requires-python"><code class="docutils literal notranslate"><span class="pre">metadata.requires-python</span></code></a></li> </ul> </li> <li><a class="reference internal" href="#package-name-version"><code class="docutils literal notranslate"><span class="pre">[[package._name_._version_]]</span></code></a><ul> <li><a class="reference internal" href="#package-name-version-filename"><code class="docutils literal notranslate"><span class="pre">package._name_._version_.filename</span></code></a></li> <li><a class="reference internal" href="#package-name-version-hashes"><code class="docutils literal notranslate"><span class="pre">[package._name_._version_.hashes]</span></code></a></li> <li><a class="reference internal" href="#package-name-version-url"><code class="docutils literal notranslate"><span class="pre">package._name_._version_.url</span></code></a></li> <li><a class="reference internal" href="#package-name-version-direct"><code class="docutils literal notranslate"><span class="pre">package._name_._version_.direct</span></code></a></li> <li><a class="reference internal" href="#package-name-version-requires-python"><code class="docutils literal notranslate"><span class="pre">package._name_._version_.requires-python</span></code></a></li> <li><a class="reference internal" href="#package-name-version-requires"><code class="docutils literal notranslate"><span class="pre">package._name_._version_.requires</span></code></a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#example">Example</a></li> <li><a class="reference internal" href="#expectations-for-lockers">Expectations for Lockers</a></li> <li><a class="reference internal" href="#expectations-for-installers">Expectations for Installers</a></li> </ul> </li> <li><a class="reference internal" href="#potential-tool-support">(Potential) Tool Support</a></li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a></li> <li><a class="reference internal" href="#transition-plan">Transition Plan</a><ul> <li><a class="reference internal" href="#usability">Usability</a></li> <li><a class="reference internal" href="#interoperability">Interoperability</a></li> <li><a class="reference internal" href="#acceptance">Acceptance</a></li> </ul> </li> <li><a class="reference internal" href="#security-implications">Security Implications</a></li> <li><a class="reference internal" href="#how-to-teach-this">How to Teach This</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a><ul> <li><a class="reference internal" href="#file-formats-other-than-toml">File Formats Other Than TOML</a></li> <li><a class="reference internal" href="#alternative-naming-schemes">Alternative Naming Schemes</a></li> <li><a class="reference internal" href="#supporting-a-single-lock-file">Supporting a Single Lock File</a></li> <li><a class="reference internal" href="#using-a-flat-list-instead-of-a-dependency-graph">Using a Flat List Instead of a Dependency Graph</a></li> <li><a class="reference internal" href="#use-wheel-tags-in-the-file-name">Use Wheel Tags in the File Name</a></li> <li><a class="reference internal" href="#alternative-names-for-requires">Alternative Names for <code class="docutils literal notranslate"><span class="pre">requires</span></code></a></li> <li><a class="reference internal" href="#accepting-pep-650">Accepting PEP 650</a></li> <li><a class="reference internal" href="#specifying-requirements-per-package-instead-of-per-file">Specifying Requirements per Package Instead of per File</a></li> <li><a class="reference internal" href="#specify-where-lockers-gather-input">Specify Where Lockers Gather Input</a></li> <li><a class="reference internal" href="#allowing-source-distributions-and-source-trees-to-be-an-opt-in-supported-file-format">Allowing Source Distributions and Source Trees to be an Opt-In, Supported File Format</a></li> </ul> </li> <li><a class="reference internal" href="#open-issues">Open Issues</a></li> <li><a class="reference internal" href="#acknowledgments">Acknowledgments</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <br> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0665.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>