CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 513 – A Platform Tag for Portable Linux Built Distributions | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0513/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 513 – A Platform Tag for Portable Linux Built Distributions | peps.python.org'> <meta property="og:description" content="This PEP proposes the creation of a new platform tag for Python package built distributions, such as wheels, called manylinux1_{x86_64,i686} with external dependencies limited to a standardized, restricted subset of the Linux kernel and core userspace A..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0513/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This PEP proposes the creation of a new platform tag for Python package built distributions, such as wheels, called manylinux1_{x86_64,i686} with external dependencies limited to a standardized, restricted subset of the Linux kernel and core userspace A..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 513</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 513 – A Platform Tag for Portable Linux Built Distributions</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Robert T. McGibbon <rmcgibbo at gmail.com>, Nathaniel J. Smith <njs at pobox.com></dd> <dt class="field-even">BDFL-Delegate:</dt> <dd class="field-even">Alyssa Coghlan <ncoghlan at gmail.com></dd> <dt class="field-odd">Discussions-To:</dt> <dd class="field-odd"><a class="reference external" href="https://mail.python.org/archives/list/distutils-sig@python.org/">Distutils-SIG list</a></dd> <dt class="field-even">Status:</dt> <dd class="field-even"><abbr title="Replaced by another succeeding PEP">Superseded</abbr></dd> <dt class="field-odd">Type:</dt> <dd class="field-odd"><abbr title="Non-normative PEP containing background, guidelines or other information relevant to the Python ecosystem">Informational</abbr></dd> <dt class="field-even">Topic:</dt> <dd class="field-even"><a class="reference external" href="../topic/packaging/">Packaging</a></dd> <dt class="field-odd">Created:</dt> <dd class="field-odd">19-Jan-2016</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even">19-Jan-2016, 25-Jan-2016, 29-Jan-2016</dd> <dt class="field-odd">Superseded-By:</dt> <dd class="field-odd"><a class="reference external" href="../pep-0600/">600</a></dd> <dt class="field-even">Resolution:</dt> <dd class="field-even"><a class="reference external" href="https://mail.python.org/pipermail/distutils-sig/2016-January/028211.html">Distutils-SIG message</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#key-causes-of-inter-linux-binary-incompatibility">Key Causes of Inter-Linux Binary Incompatibility</a><ul> <li><a class="reference internal" href="#external-shared-libraries">External Shared Libraries</a></li> <li><a class="reference internal" href="#versioning-of-core-shared-libraries">Versioning of Core Shared Libraries</a></li> </ul> </li> <li><a class="reference internal" href="#the-manylinux1-policy">The <code class="docutils literal notranslate">manylinux1</code> policy</a><ul> <li><a class="reference internal" href="#libpythonx-y-so-1">libpythonX.Y.so.1</a></li> <li><a class="reference internal" href="#ucs-2-vs-ucs-4-builds">UCS-2 vs UCS-4 builds</a></li> <li><a class="reference internal" href="#fpectl-builds-vs-no-fpectl-builds">fpectl builds vs. no fpectl builds</a></li> </ul> </li> <li><a class="reference internal" href="#compilation-of-compliant-wheels">Compilation of Compliant Wheels</a><ul> <li><a class="reference internal" href="#docker-image">Docker Image</a></li> <li><a class="reference internal" href="#auditwheel">Auditwheel</a></li> </ul> </li> <li><a class="reference internal" href="#bundled-wheels-on-linux">Bundled Wheels on Linux</a><ul> <li><a class="reference internal" href="#security-implications">Security Implications</a></li> </ul> </li> <li><a class="reference internal" href="#platform-detection-for-installers">Platform Detection for Installers</a></li> <li><a class="reference internal" href="#pypi-support">PyPI Support</a></li> <li><a class="reference internal" href="#rejected-alternatives">Rejected Alternatives</a></li> <li><a class="reference internal" href="#future-updates">Future updates</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> This PEP proposes the creation of a new platform tag for Python package built distributions, such as wheels, called <code class="docutils literal notranslate">manylinux1_{x86_64,i686}</code> with external dependencies limited to a standardized, restricted subset of the Linux kernel and core userspace ABI. It proposes that PyPI support uploading and distributing wheels with this platform tag, and that <code class="docutils literal notranslate">pip</code> support downloading and installing these packages on compatible platforms. </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> Currently, distribution of binary Python extensions for Windows and OS X is straightforward. Developers and packagers build wheels (<a class="pep reference internal" href="../pep-0427/" title="PEP 427 – The Wheel Binary Package Format 1.0">PEP 427</a>, <a class="pep reference internal" href="../pep-0491/" title="PEP 491 – The Wheel Binary Package Format 1.9">PEP 491</a>), which are assigned platform tags such as <code class="docutils literal notranslate">win32</code> or <code class="docutils literal notranslate">macosx_10_6_intel</code>, and upload these wheels to PyPI. Users can download and install these wheels using tools such as <code class="docutils literal notranslate">pip</code>. For Linux, the situation is much more delicate. In general, compiled Python extension modules built on one Linux distribution will not work on other Linux distributions, or even on different machines running the same Linux distribution with different system libraries installed. Build tools using <a class="pep reference internal" href="../pep-0425/" title="PEP 425 – Compatibility Tags for Built Distributions">PEP 425</a> platform tags do not track information about the particular Linux distribution or installed system libraries, and instead assign all wheels the too-vague <code class="docutils literal notranslate">linux_i686</code> or <code class="docutils literal notranslate">linux_x86_64</code> tags. Because of this ambiguity, there is no expectation that <code class="docutils literal notranslate">linux</code>-tagged built distributions compiled on one machine will work properly on another, and for this reason, PyPI has not permitted the uploading of wheels for Linux. It would be ideal if wheel packages could be compiled that would work on any linux system. But, because of the incredible diversity of Linux systems – from PCs to Android to embedded systems with custom libcs – this cannot be guaranteed in general. Instead, we define a standard subset of the kernel+core userspace ABI that, in practice, is compatible enough that packages conforming to this standard will work on many linux systems, including essentially all of the desktop and server distributions in common use. We know this because there are companies who have been distributing such widely-portable pre-compiled Python extension modules for Linux – e.g. Enthought with Canopy <a class="footnote-reference brackets" href="#id16" id="id1">[4]</a> and Continuum Analytics with Anaconda <a class="footnote-reference brackets" href="#id17" id="id2">[5]</a>. Building on the compatibility lessons learned from these companies, we thus define a baseline <code class="docutils literal notranslate">manylinux1</code> platform tag for use by binary Python wheels, and introduce the implementation of preliminary tools to aid in the construction of these <code class="docutils literal notranslate">manylinux1</code> wheels. </section> <section id="key-causes-of-inter-linux-binary-incompatibility"> <h2><a class="toc-backref" href="#key-causes-of-inter-linux-binary-incompatibility" role="doc-backlink">Key Causes of Inter-Linux Binary Incompatibility</a></h2> To properly define a standard that will guarantee that wheel packages meeting this specification will operate on many linux platforms, it is necessary to understand the root causes which often prevent portability of pre-compiled binaries on Linux. The two key causes are dependencies on shared libraries which are not present on users’ systems, and dependencies on particular versions of certain core libraries like <code class="docutils literal notranslate">glibc</code>. <section id="external-shared-libraries"> <h3><a class="toc-backref" href="#external-shared-libraries" role="doc-backlink">External Shared Libraries</a></h3> Most desktop and server linux distributions come with a system package manager (examples include <code class="docutils literal notranslate">APT</code> on Debian-based systems, <code class="docutils literal notranslate">yum</code> on <code class="docutils literal notranslate">RPM</code>-based systems, and <code class="docutils literal notranslate">pacman</code> on Arch linux) that manages, among other responsibilities, the installation of shared libraries installed to system directories such as <code class="docutils literal notranslate">/usr/lib</code>. Most non-trivial Python extensions will depend on one or more of these shared libraries, and thus function properly only on systems where the user has the proper libraries (and the proper versions thereof), either installed using their package manager, or installed manually by setting certain environment variables such as <code class="docutils literal notranslate">LD_LIBRARY_PATH</code> to notify the runtime linker of the location of the depended-upon shared libraries. </section> <section id="versioning-of-core-shared-libraries"> <h3><a class="toc-backref" href="#versioning-of-core-shared-libraries" role="doc-backlink">Versioning of Core Shared Libraries</a></h3> Even if the developers a Python extension module wish to use no external shared libraries, the modules will generally have a dynamic runtime dependency on the GNU C library, <code class="docutils literal notranslate">glibc</code>. While it is possible, statically linking <code class="docutils literal notranslate">glibc</code> is usually a bad idea because certain important C functions like <code class="docutils literal notranslate">dlopen()</code> cannot be called from code that statically links <code class="docutils literal notranslate">glibc</code>. A runtime shared library dependency on a system-provided <code class="docutils literal notranslate">glibc</code> is unavoidable in practice. The maintainers of the GNU C library follow a strict symbol versioning scheme for backward compatibility. This ensures that binaries compiled against an older version of <code class="docutils literal notranslate">glibc</code> can run on systems that have a newer <code class="docutils literal notranslate">glibc</code>. The opposite is generally not true – binaries compiled on newer Linux distributions tend to rely upon versioned functions in <code class="docutils literal notranslate">glibc</code> that are not available on older systems. This generally prevents wheels compiled on the latest Linux distributions from being portable. </section> </section> <section id="the-manylinux1-policy"> <h2><a class="toc-backref" href="#the-manylinux1-policy" role="doc-backlink">The <code class="docutils literal notranslate">manylinux1</code> policy</a></h2> For these reasons, to achieve broad portability, Python wheels <ul class="simple"> <li>should depend only on an extremely limited set of external shared libraries; and</li> <li>should depend only on “old” symbol versions in those external shared libraries; and</li> <li>should depend only on a widely-compatible kernel ABI.</li> </ul> To be eligible for the <code class="docutils literal notranslate">manylinux1</code> platform tag, a Python wheel must therefore both (a) contain binary executables and compiled code that links only to libraries with SONAMEs included in the following list: <div class="highlight-default notranslate"><div class="highlight"><pre>libpanelw.so.5 libncursesw.so.5 libgcc_s.so.1 libstdc++.so.6 libm.so.6 libdl.so.2 librt.so.1 libc.so.6 libnsl.so.1 libutil.so.1 libpthread.so.0 libresolv.so.2 libX11.so.6 libXext.so.6 libXrender.so.1 libICE.so.6 libSM.so.6 libGL.so.1 libgobject-2.0.so.0 libgthread-2.0.so.0 libglib-2.0.so.0 </pre></div> </div> and, (b) work on a stock CentOS 5.11 <a class="footnote-reference brackets" href="#id18" id="id3">[6]</a> system that contains the system package manager’s provided versions of these libraries. <code class="docutils literal notranslate">libcrypt.so.1</code> was retrospectively removed from the whitelist after Fedora 30 was released with <code class="docutils literal notranslate">libcrypt.so.2</code> instead. Because CentOS 5 is only available for x86_64 and i686 architectures, these are the only architectures currently supported by the <code class="docutils literal notranslate">manylinux1</code> policy. On Debian-based systems, these libraries are provided by the packages <div class="highlight-default notranslate"><div class="highlight"><pre>libncurses5 libgcc1 libstdc++6 libc6 libx11-6 libxext6 libxrender1 libice6 libsm6 libgl1-mesa-glx libglib2.0-0 </pre></div> </div> On RPM-based systems, these libraries are provided by the packages <div class="highlight-default notranslate"><div class="highlight"><pre>ncurses libgcc libstdc++ glibc libXext libXrender libICE libSM mesa-libGL glib2 </pre></div> </div> This list was compiled by checking the external shared library dependencies of the Canopy <a class="footnote-reference brackets" href="#id16" id="id4">[4]</a> and Anaconda <a class="footnote-reference brackets" href="#id17" id="id5">[5]</a> distributions, which both include a wide array of the most popular Python modules and have been confirmed in practice to work across a wide swath of Linux systems in the wild. Many of the permitted system libraries listed above use symbol versioning schemes for backward compatibility. The latest symbol versions provided with the CentOS 5.11 versions of these libraries are: <div class="highlight-default notranslate"><div class="highlight"><pre>GLIBC_2.5 CXXABI_3.4.8 GLIBCXX_3.4.9 GCC_4.2.0 </pre></div> </div> Therefore, as a consequence of requirement (b), any wheel that depends on versioned symbols from the above shared libraries may depend only on symbols with the following versions: <div class="highlight-default notranslate"><div class="highlight"><pre>GLIBC <= 2.5 CXXABI <= 3.4.8 GLIBCXX <= 3.4.9 GCC <= 4.2.0 </pre></div> </div> These recommendations are the outcome of the relevant discussions in January 2016 <a class="footnote-reference brackets" href="#id19" id="id6">[7]</a>, <a class="footnote-reference brackets" href="#id20" id="id7">[8]</a>. Note that in our recommendations below, we do not suggest that <code class="docutils literal notranslate">pip</code> or PyPI should attempt to check for and enforce the details of this policy (just as they don’t check for and enforce the details of existing platform tags like <code class="docutils literal notranslate">win32</code>). The text above is provided (a) as advice to package builders, and (b) as a method for allocating blame if a given wheel doesn’t work on some system: if it satisfies the policy above, then this is a bug in the spec or the installation tool; if it does not satisfy the policy above, then it’s a bug in the wheel. One useful consequence of this approach is that it leaves open the possibility of further updates and tweaks as we gain more experience, e.g., we could have a “manylinux 1.1” policy which targets the same systems and uses the same <code class="docutils literal notranslate">manylinux1</code> platform tag (and thus requires no further changes to <code class="docutils literal notranslate">pip</code> or PyPI), but that adjusts the list above to remove libraries that have turned out to be problematic or add libraries that have turned out to be safe. <section id="libpythonx-y-so-1"> <h3><a class="toc-backref" href="#libpythonx-y-so-1" role="doc-backlink">libpythonX.Y.so.1</a></h3> Note that <code class="docutils literal notranslate">libpythonX.Y.so.1</code> is not on the list of libraries that a <code class="docutils literal notranslate">manylinux1</code> extension is allowed to link to. Explicitly linking to <code class="docutils literal notranslate">libpythonX.Y.so.1</code> is unnecessary in almost all cases: the way ELF linking works, extension modules that are loaded into the interpreter automatically get access to all of the interpreter’s symbols, regardless of whether or not the extension itself is explicitly linked against libpython. Furthermore, explicit linking to libpython creates problems in the common configuration where Python is not built with <code class="docutils literal notranslate">--enable-shared</code>. In particular, on Debian and Ubuntu systems, <code class="docutils literal notranslate">apt install pythonX.Y</code> does not even install <code class="docutils literal notranslate">libpythonX.Y.so.1</code>, meaning that any wheel that did depend on <code class="docutils literal notranslate">libpythonX.Y.so.1</code> could fail to import. There is one situation where extensions that are linked in this way can fail to work: if a host program (e.g., <code class="docutils literal notranslate">apache2</code>) uses <code class="docutils literal notranslate">dlopen()</code> to load a module (e.g., <code class="docutils literal notranslate">mod_wsgi</code>) that embeds the CPython interpreter, and the host program does not pass the <code class="docutils literal notranslate">RTLD_GLOBAL</code> flag to <code class="docutils literal notranslate">dlopen()</code>, then the embedded CPython will be unable to load any extension modules that do not themselves link explicitly to <code class="docutils literal notranslate">libpythonX.Y.so.1</code>. Fortunately, <code class="docutils literal notranslate">apache2</code> does set the <code class="docutils literal notranslate">RTLD_GLOBAL</code> flag, as do all the other programs that embed-CPython-via-a-dlopened-plugin that we could locate, so this does not seem to be a serious problem in practice. The incompatibility with Debian/Ubuntu is more of an issue than the theoretical incompatibility with a rather obscure corner case. This is a rather complex and subtle issue that extends beyond the scope of <code class="docutils literal notranslate">manylinux1</code>; for more discussion see: <a class="footnote-reference brackets" href="#id21" id="id8">[9]</a>, <a class="footnote-reference brackets" href="#id22" id="id9">[10]</a>, <a class="footnote-reference brackets" href="#id23" id="id10">[11]</a>. </section> <section id="ucs-2-vs-ucs-4-builds"> <h3><a class="toc-backref" href="#ucs-2-vs-ucs-4-builds" role="doc-backlink">UCS-2 vs UCS-4 builds</a></h3> All versions of CPython 2.x, plus CPython 3.0-3.2 inclusive, can be built in two ABI-incompatible modes: builds using the <code class="docutils literal notranslate">--enable-unicode=ucs2</code> configure flag store Unicode data in UCS-2 (or really UTF-16) format, while builds using the <code class="docutils literal notranslate">--enable-unicode=ucs4</code> configure flag store Unicode data in UCS-4. (CPython 3.3 and greater use a different storage method that always supports UCS-4.) If we want to make sure <code class="docutils literal notranslate">ucs2</code> wheels don’t get installed into <code class="docutils literal notranslate">ucs4</code> CPythons and vice-versa, then something must be done. An earlier version of this PEP included a requirement that <code class="docutils literal notranslate">manylinux1</code> wheels targeting these older CPython versions should always use the <code class="docutils literal notranslate">ucs4</code> ABI. But then, in between the PEP’s initial acceptance and its implementation, <code class="docutils literal notranslate">pip</code> and <code class="docutils literal notranslate">wheel</code> gained first-class support for tracking and checking this aspect of ABI compatibility for the relevant CPython versions, which is a better solution. So we now allow the <code class="docutils literal notranslate">manylinux1</code> platform tags to be used in combination with any ABI tag. However, to maintain compatibility it is crucial to ensure that all <code class="docutils literal notranslate">manylinux1</code> wheels include a non-trivial abi tag. For example, a wheel built against a <code class="docutils literal notranslate">ucs4</code> CPython might have a name like: <div class="highlight-default notranslate"><div class="highlight"><pre>PKG-VERSION-cp27-cp27mu-manylinux1_x86_64.whl ^^^^^^ Good! </pre></div> </div> While a wheel built against the <code class="docutils literal notranslate">ucs2</code> ABI might have a name like: <div class="highlight-default notranslate"><div class="highlight"><pre>PKG-VERSION-cp27-cp27m-manylinux1_x86_64.whl ^^^^^ Okay! </pre></div> </div> But you should never have a wheel with a name like: <div class="highlight-default notranslate"><div class="highlight"><pre>PKG-VERSION-cp27-none-manylinux1_x86_64.whl ^^^^ BAD! Don't do this! </pre></div> </div> This wheel claims to be simultaneously compatible with both ucs2 and ucs4 builds, which is bad. We note for information that the <code class="docutils literal notranslate">ucs4</code> ABI appears to be much more widespread among Linux CPython distributors. </section> <section id="fpectl-builds-vs-no-fpectl-builds"> <h3><a class="toc-backref" href="#fpectl-builds-vs-no-fpectl-builds" role="doc-backlink">fpectl builds vs. no fpectl builds</a></h3> All extant versions of CPython can be built either with or without the <code class="docutils literal notranslate">--with-fpectl</code> flag to <code class="docutils literal notranslate">configure</code>. It turns out that this changes the CPython ABI: extensions that are built against a no-<code class="docutils literal notranslate">fpectl</code> CPython are always compatible with yes-<code class="docutils literal notranslate">fpectl</code> CPython, but the reverse is not necessarily true. (Symptom: errors at import time complaining about <code class="docutils literal notranslate">undefined symbol: PyFPE_jbuf</code>.) See: <a class="footnote-reference brackets" href="#id28" id="id11">[16]</a>. For maximum compatibility, therefore, the CPython used to build manylinux1 wheels must be compiled without the <code class="docutils literal notranslate">--with-fpectl</code> flag, and manylinux1 extensions must not reference the symbol <code class="docutils literal notranslate">PyFPE_jbuf</code>. </section> </section> <section id="compilation-of-compliant-wheels"> <h2><a class="toc-backref" href="#compilation-of-compliant-wheels" role="doc-backlink">Compilation of Compliant Wheels</a></h2> The way glibc, libgcc, and libstdc++ manage their symbol versioning means that in practice, the compiler toolchains that most developers use to do their daily work are incapable of building <code class="docutils literal notranslate">manylinux1</code>-compliant wheels. Therefore, we do not attempt to change the default behavior of <code class="docutils literal notranslate">pip wheel</code> / <code class="docutils literal notranslate">bdist_wheel</code>: they will continue to generate regular <code class="docutils literal notranslate">linux_*</code> platform tags, and developers who wish to use them to generate <code class="docutils literal notranslate">manylinux1</code>-tagged wheels will have to change the tag as a second post-processing step. To support the compilation of wheels meeting the <code class="docutils literal notranslate">manylinux1</code> standard, we provide initial drafts of two tools. <section id="docker-image"> <h3><a class="toc-backref" href="#docker-image" role="doc-backlink">Docker Image</a></h3> The first tool is a Docker image based on CentOS 5.11, which is recommended as an easy to use self-contained build box for compiling <code class="docutils literal notranslate">manylinux1</code> wheels <a class="footnote-reference brackets" href="#id24" id="id12">[12]</a>. Compiling on a more recently-released linux distribution will generally introduce dependencies on too-new versioned symbols. The image comes with a full compiler suite installed (<code class="docutils literal notranslate">gcc</code>, <code class="docutils literal notranslate">g++</code>, and <code class="docutils literal notranslate">gfortran</code> 4.8.2) as well as the latest releases of Python and <code class="docutils literal notranslate">pip</code>. </section> <section id="auditwheel"> <h3><a class="toc-backref" href="#auditwheel" role="doc-backlink">Auditwheel</a></h3> The second tool is a command line executable called <code class="docutils literal notranslate">auditwheel</code> <a class="footnote-reference brackets" href="#id25" id="id13">[13]</a> that may aid in package maintainers in dealing with third-party external dependencies. There are at least three methods for building wheels that use third-party external libraries in a way that meets the above policy. <ol class="arabic simple"> <li>The third-party libraries can be statically linked.</li> <li>The third-party shared libraries can be distributed in separate packages on PyPI which are depended upon by the wheel.</li> <li>The third-party shared libraries can be bundled inside the wheel libraries, linked with a relative path.</li> </ol> All of these are valid option which may be effectively used by different packages and communities. Statically linking generally requires package-specific modifications to the build system, and distributing third-party dependencies on PyPI may require some coordination of the community of users of the package. As an often-automatic alternative to these options, we introduce <code class="docutils literal notranslate">auditwheel</code>. The tool inspects all of the ELF files inside a wheel to check for dependencies on versioned symbols or external shared libraries, and verifies conformance with the <code class="docutils literal notranslate">manylinux1</code> policy. This includes the ability to add the new platform tag to conforming wheels. More importantly, <code class="docutils literal notranslate">auditwheel</code> has the ability to automatically modify wheels that depend on external shared libraries by copying those shared libraries from the system into the wheel itself, and modifying the appropriate <code class="docutils literal notranslate">RPATH</code> entries such that these libraries will be picked up at runtime. This accomplishes a similar result as if the libraries had been statically linked without requiring changes to the build system. Packagers are advised that bundling, like static linking, may implicate copyright concerns. </section> </section> <section id="bundled-wheels-on-linux"> <h2><a class="toc-backref" href="#bundled-wheels-on-linux" role="doc-backlink">Bundled Wheels on Linux</a></h2> While we acknowledge many approaches for dealing with third-party library dependencies within <code class="docutils literal notranslate">manylinux1</code> wheels, we recognize that the <code class="docutils literal notranslate">manylinux1</code> policy encourages bundling external dependencies, a practice which runs counter to the package management policies of many linux distributions’ system package managers <a class="footnote-reference brackets" href="#id26" id="id14">[14]</a>, <a class="footnote-reference brackets" href="#id27" id="id15">[15]</a>. The primary purpose of this is cross-distro compatibility. Furthermore, <code class="docutils literal notranslate">manylinux1</code> wheels on PyPI occupy a different niche than the Python packages available through the system package manager. The decision in this PEP to encourage departure from general Linux distribution unbundling policies is informed by the following concerns: <ol class="arabic simple"> <li>In these days of automated continuous integration and deployment pipelines, publishing new versions and updating dependencies is easier than it was when those policies were defined.</li> <li><code class="docutils literal notranslate">pip</code> users remain free to use the <code class="docutils literal notranslate">"--no-binary"</code> option if they want to force local builds rather than using pre-built wheel files.</li> <li>The popularity of modern container based deployment and “immutable infrastructure” models involve substantial bundling at the application layer anyway.</li> <li>Distribution of bundled wheels through PyPI is currently the norm for Windows and OS X.</li> <li>This PEP doesn’t rule out the idea of offering more targeted binaries for particular Linux distributions in the future.</li> </ol> The model described in this PEP is most ideally suited for cross-platform Python packages, because it means they can reuse much of the work that they’re already doing to make static Windows and OS X wheels. We recognize that it is less optimal for Linux-specific packages that might prefer to interact more closely with Linux’s unique package management functionality and only care about targeting a small set of particular distos. <section id="security-implications"> <h3><a class="toc-backref" href="#security-implications" role="doc-backlink">Security Implications</a></h3> One of the advantages of dependencies on centralized libraries in Linux is that bugfixes and security updates can be deployed system-wide, and applications which depend on these libraries will automatically feel the effects of these patches when the underlying libraries are updated. This can be particularly important for security updates in packages engaged in communication across the network or cryptography. <code class="docutils literal notranslate">manylinux1</code> wheels distributed through PyPI that bundle security-critical libraries like OpenSSL will thus assume responsibility for prompt updates in response disclosed vulnerabilities and patches. This closely parallels the security implications of the distribution of binary wheels on Windows that, because the platform lacks a system package manager, generally bundle their dependencies. In particular, because it lacks a stable ABI, OpenSSL cannot be included in the <code class="docutils literal notranslate">manylinux1</code> profile. </section> </section> <section id="platform-detection-for-installers"> <h2><a class="toc-backref" href="#platform-detection-for-installers" role="doc-backlink">Platform Detection for Installers</a></h2> Above, we defined what it means for a wheel to be <code class="docutils literal notranslate">manylinux1</code>-compatible. Here we discuss what it means for a Python installation to be <code class="docutils literal notranslate">manylinux1</code>-compatible. In particular, this is important for tools like <code class="docutils literal notranslate">pip</code> to know when deciding whether or not they should consider <code class="docutils literal notranslate">manylinux1</code>-tagged wheels for installation. Because the <code class="docutils literal notranslate">manylinux1</code> profile is already known to work for the many thousands of users of popular commercial Python distributions, we suggest that installation tools should error on the side of assuming that a system is compatible, unless there is specific reason to think otherwise. We know of four main sources of potential incompatibility that are likely to arise in practice: <ul class="simple"> <li>Eventually, in the future, there may exist distributions that break compatibility with this profile (e.g., if one of the libraries in the profile changes its ABI in a backwards-incompatible way)</li> <li>A linux distribution that is too old (e.g. RHEL 4)</li> <li>A linux distribution that does not use <code class="docutils literal notranslate">glibc</code> (e.g. Alpine Linux, which is based on musl <code class="docutils literal notranslate">libc</code>, or Android)</li> </ul> To address these we propose a two-pronged approach. To handle potential future incompatibilities, we standardize a mechanism for a Python distributor to signal that a particular Python install definitely is or is not compatible with <code class="docutils literal notranslate">manylinux1</code>: this is done by installing a module named <code class="docutils literal notranslate">_manylinux</code>, and setting its <code class="docutils literal notranslate">manylinux1_compatible</code> attribute. We do not propose adding any such module to the standard library – this is merely a well-known name by which distributors and installation tools can rendezvous. However, if a distributor does add this module, they should add it to the standard library rather than to a <code class="docutils literal notranslate">site-packages/</code> directory, because the standard library is inherited by virtualenvs (which we want), and <code class="docutils literal notranslate">site-packages/</code> in general is not. Then, to handle the last two cases for existing Python distributions, we suggest a simple and reliable method to check for the presence and version of <code class="docutils literal notranslate">glibc</code> (basically using it as a “clock” for the overall age of the distribution). Specifically, the algorithm we propose is: <div class="highlight-default notranslate"><div class="highlight"><pre>def is_manylinux1_compatible(): # Only Linux, and only x86-64 / i686 from distutils.util import get_platform if get_platform() not in ["linux-x86_64", "linux-i686"]: return False # Check for presence of _manylinux module try: import _manylinux return bool(_manylinux.manylinux1_compatible) except (ImportError, AttributeError): # Fall through to heuristic check below pass # Check glibc version. CentOS 5 uses glibc 2.5. return have_compatible_glibc(2, 5) def have_compatible_glibc(major, minimum_minor): import ctypes process_namespace = ctypes.CDLL(None) try: gnu_get_libc_version = process_namespace.gnu_get_libc_version except AttributeError: # Symbol doesn't exist -> therefore, we are not linked to # glibc. return False # Call gnu_get_libc_version, which returns a string like "2.5". gnu_get_libc_version.restype = ctypes.c_char_p version_str = gnu_get_libc_version() # py2 / py3 compatibility: if not isinstance(version_str, str): version_str = version_str.decode("ascii") # Parse string and check against requested version. version = [int(piece) for piece in version_str.split(".")] assert len(version) == 2 if major != version[0]: return False if minimum_minor > version[1]: return False return True </pre></div> </div> Rejected alternatives: We also considered using a configuration file, e.g. <code class="docutils literal notranslate">/etc/python/compatibility.cfg</code>. The problem with this is that a single filesystem might contain many different interpreter environments, each with their own ABI profile – the <code class="docutils literal notranslate">manylinux1</code> compatibility of a system-installed x86_64 CPython might not tell us much about the <code class="docutils literal notranslate">manylinux1</code> compatibility of a user-installed i686 PyPy. Locating this configuration information within the Python environment itself ensures that it remains attached to the correct binary, and dramatically simplifies lookup code. We also considered using a more elaborate structure, like a list of all platform tags that should be considered compatible, together with their preference ordering, for example: <code class="docutils literal notranslate">_binary_compat.compatible = ["manylinux1_x86_64", "centos5_x86_64", "linux_x86_64"]</code>. However, this introduces several complications. For example, we want to be able to distinguish between the state of “doesn’t support <code class="docutils literal notranslate">manylinux1</code>” (or eventually <code class="docutils literal notranslate">manylinux2</code>, etc.) versus “doesn’t specify either way whether it supports <code class="docutils literal notranslate">manylinux1</code>”, which is not entirely obvious in the above representation; and, it’s not at all clear what features are really needed vis a vis preference ordering given that right now the only possible platform tags are <code class="docutils literal notranslate">manylinux1</code> and <code class="docutils literal notranslate">linux</code>. So we’re deferring a more complete solution here for a separate PEP, when / if Linux gets more platform tags. For the library compatibility check, we also considered much more elaborate checks (e.g. checking the kernel version, searching for and checking the versions of all the individual libraries listed in the <code class="docutils literal notranslate">manylinux1</code> profile, etc.), but ultimately decided that this would be more likely to introduce confusing bugs than actually help the user. (For example: different distributions vary in where they actually put these libraries, and if our checking code failed to use the correct path search then it could easily return incorrect answers.) </section> <section id="pypi-support"> <h2><a class="toc-backref" href="#pypi-support" role="doc-backlink">PyPI Support</a></h2> PyPI should permit wheels containing the <code class="docutils literal notranslate">manylinux1</code> platform tag to be uploaded. PyPI should not attempt to formally verify that wheels containing the <code class="docutils literal notranslate">manylinux1</code> platform tag adhere to the <code class="docutils literal notranslate">manylinux1</code> policy described in this document. This verification tasks should be left to other tools, like <code class="docutils literal notranslate">auditwheel</code>, that are developed separately. </section> <section id="rejected-alternatives"> <h2><a class="toc-backref" href="#rejected-alternatives" role="doc-backlink">Rejected Alternatives</a></h2> One alternative would be to provide separate platform tags for each Linux distribution (and each version thereof), e.g. <code class="docutils literal notranslate">RHEL6</code>, <code class="docutils literal notranslate">ubuntu14_10</code>, <code class="docutils literal notranslate">debian_jessie</code>, etc. Nothing in this proposal rules out the possibility of adding such platform tags in the future, or of further extensions to wheel metadata that would allow wheels to declare dependencies on external system-installed packages. However, such extensions would require substantially more work than this proposal, and still might not be appreciated by package developers who would prefer not to have to maintain multiple build environments and build multiple wheels in order to cover all the common Linux distributions. Therefore, we consider such proposals to be out-of-scope for this PEP. </section> <section id="future-updates"> <h2><a class="toc-backref" href="#future-updates" role="doc-backlink">Future updates</a></h2> We anticipate that at some point in the future there will be a <code class="docutils literal notranslate">manylinux2</code> specifying a more modern baseline environment (perhaps based on CentOS 6), and someday a <code class="docutils literal notranslate">manylinux3</code> and so forth, but we defer specifying these until we have more experience with the initial <code class="docutils literal notranslate">manylinux1</code> proposal. </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <aside class="footnote-list brackets"> <aside class="footnote brackets" id="id16" role="doc-footnote"> <dt class="label" id="id16">[4] (<a href='#id1'>1</a>, <a href='#id4'>2</a>) </dt> <dd>Enthought Canopy Python Distribution (<a class="reference external" href="https://store.enthought.com/downloads/">https://store.enthought.com/downloads/</a>)</aside> <aside class="footnote brackets" id="id17" role="doc-footnote"> <dt class="label" id="id17">[5] (<a href='#id2'>1</a>, <a href='#id5'>2</a>) </dt> <dd>Continuum Analytics Anaconda Python Distribution (<a class="reference external" href="https://www.continuum.io/downloads">https://www.continuum.io/downloads</a>)</aside> <aside class="footnote brackets" id="id18" role="doc-footnote"> <dt class="label" id="id18">[<a href="#id3">6</a>]</dt> <dd>CentOS 5.11 Release Notes (<a class="reference external" href="https://wiki.centos.org/Manuals/ReleaseNotes/CentOS5.11">https://wiki.centos.org/Manuals/ReleaseNotes/CentOS5.11</a>)</aside> <aside class="footnote brackets" id="id19" role="doc-footnote"> <dt class="label" id="id19">[<a href="#id6">7</a>]</dt> <dd>manylinux-discuss mailing list discussion (<a class="reference external" href="https://groups.google.com/forum/#!topic/manylinux-discuss/-4l3rrjfr9U">https://groups.google.com/forum/#!topic/manylinux-discuss/-4l3rrjfr9U</a>)</aside> <aside class="footnote brackets" id="id20" role="doc-footnote"> <dt class="label" id="id20">[<a href="#id7">8</a>]</dt> <dd>distutils-sig discussion (<a class="reference external" href="https://mail.python.org/pipermail/distutils-sig/2016-January/027997.html">https://mail.python.org/pipermail/distutils-sig/2016-January/027997.html</a>)</aside> <aside class="footnote brackets" id="id21" role="doc-footnote"> <dt class="label" id="id21">[<a href="#id8">9</a>]</dt> <dd>distutils-sig discussion (<a class="reference external" href="https://mail.python.org/pipermail/distutils-sig/2016-February/028275.html">https://mail.python.org/pipermail/distutils-sig/2016-February/028275.html</a>)</aside> <aside class="footnote brackets" id="id22" role="doc-footnote"> <dt class="label" id="id22">[<a href="#id9">10</a>]</dt> <dd>github issue discussion (<a class="reference external" href="https://github.com/pypa/manylinux/issues/30">https://github.com/pypa/manylinux/issues/30</a>)</aside> <aside class="footnote brackets" id="id23" role="doc-footnote"> <dt class="label" id="id23">[<a href="#id10">11</a>]</dt> <dd>python bug tracker discussion (<a class="reference external" href="https://bugs.python.org/issue21536">https://bugs.python.org/issue21536</a>)</aside> <aside class="footnote brackets" id="id24" role="doc-footnote"> <dt class="label" id="id24">[<a href="#id12">12</a>]</dt> <dd>manylinux1 docker images (Source: <a class="reference external" href="https://github.com/pypa/manylinux">https://github.com/pypa/manylinux</a>; x86-64: <a class="reference external" href="https://quay.io/repository/pypa/manylinux1_x86_64">https://quay.io/repository/pypa/manylinux1_x86_64</a>; x86-32: <a class="reference external" href="https://quay.io/repository/pypa/manylinux1_i686">https://quay.io/repository/pypa/manylinux1_i686</a>)</aside> <aside class="footnote brackets" id="id25" role="doc-footnote"> <dt class="label" id="id25">[<a href="#id13">13</a>]</dt> <dd>auditwheel tool (<a class="reference external" href="https://pypi.python.org/pypi/auditwheel">https://pypi.python.org/pypi/auditwheel</a>)</aside> <aside class="footnote brackets" id="id26" role="doc-footnote"> <dt class="label" id="id26">[<a href="#id14">14</a>]</dt> <dd>Fedora Bundled Software Policy (<a class="reference external" href="https://fedoraproject.org/wiki/Bundled_Software_policy">https://fedoraproject.org/wiki/Bundled_Software_policy</a>)</aside> <aside class="footnote brackets" id="id27" role="doc-footnote"> <dt class="label" id="id27">[<a href="#id15">15</a>]</dt> <dd>Debian Policy Manual – 4.13: Convenience copies of code (<a class="reference external" href="https://www.debian.org/doc/debian-policy/ch-source.html#s-embeddedfiles">https://www.debian.org/doc/debian-policy/ch-source.html#s-embeddedfiles</a>)</aside> <aside class="footnote brackets" id="id28" role="doc-footnote"> <dt class="label" id="id28">[<a href="#id11">16</a>]</dt> <dd>numpy bug report: <a class="reference external" href="https://github.com/numpy/numpy/issues/8415#issuecomment-269095235">https://github.com/numpy/numpy/issues/8415#issuecomment-269095235</a></aside> </aside> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed into the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0513.rst">https://github.com/python/peps/blob/main/peps/pep-0513.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0513.rst">2023-10-11 12:05:51 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#key-causes-of-inter-linux-binary-incompatibility">Key Causes of Inter-Linux Binary Incompatibility</a><ul> <li><a class="reference internal" href="#external-shared-libraries">External Shared Libraries</a></li> <li><a class="reference internal" href="#versioning-of-core-shared-libraries">Versioning of Core Shared Libraries</a></li> </ul> </li> <li><a class="reference internal" href="#the-manylinux1-policy">The <code class="docutils literal notranslate">manylinux1</code> policy</a><ul> <li><a class="reference internal" href="#libpythonx-y-so-1">libpythonX.Y.so.1</a></li> <li><a class="reference internal" href="#ucs-2-vs-ucs-4-builds">UCS-2 vs UCS-4 builds</a></li> <li><a class="reference internal" href="#fpectl-builds-vs-no-fpectl-builds">fpectl builds vs. no fpectl builds</a></li> </ul> </li> <li><a class="reference internal" href="#compilation-of-compliant-wheels">Compilation of Compliant Wheels</a><ul> <li><a class="reference internal" href="#docker-image">Docker Image</a></li> <li><a class="reference internal" href="#auditwheel">Auditwheel</a></li> </ul> </li> <li><a class="reference internal" href="#bundled-wheels-on-linux">Bundled Wheels on Linux</a><ul> <li><a class="reference internal" href="#security-implications">Security Implications</a></li> </ul> </li> <li><a class="reference internal" href="#platform-detection-for-installers">Platform Detection for Installers</a></li> <li><a class="reference internal" href="#pypi-support">PyPI Support</a></li> <li><a class="reference internal" href="#rejected-alternatives">Rejected Alternatives</a></li> <li><a class="reference internal" href="#future-updates">Future updates</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0513.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>