CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 687 – Isolating modules in the standard library | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0687/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 687 – Isolating modules in the standard library | peps.python.org'> <meta property="og:description" content="Extensions in the standard library will be converted to multi-phase initialization (PEP 489) and where possible, all state will be stored on module objects rather than in process-global variables."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0687/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="Extensions in the standard library will be converted to multi-phase initialization (PEP 489) and where possible, all state will be stored on module objects rather than in process-global variables."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 687</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> <span class="visually-hidden">Toggle light / dark / auto colour theme</span> </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 687 – Isolating modules in the standard library</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author<span class="colon">:</span></dt> <dd class="field-odd">Erlend Egeberg Aasland <erlend at python.org>, Petr Viktorin <encukou at gmail.com></dd> <dt class="field-even">Discussions-To<span class="colon">:</span></dt> <dd class="field-even"><a class="reference external" href="https://discuss.python.org/t/14824">Discourse thread</a></dd> <dt class="field-odd">Status<span class="colon">:</span></dt> <dd class="field-odd"><abbr title="Normative proposal accepted for implementation">Accepted</abbr></dd> <dt class="field-even">Type<span class="colon">:</span></dt> <dd class="field-even"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-odd">Requires<span class="colon">:</span></dt> <dd class="field-odd"><a class="reference external" href="../pep-0489/">489</a>, <a class="reference external" href="../pep-0573/">573</a>, <a class="reference external" href="../pep-0630/">630</a></dd> <dt class="field-even">Created<span class="colon">:</span></dt> <dd class="field-even">04-Apr-2022</dd> <dt class="field-odd">Python-Version<span class="colon">:</span></dt> <dd class="field-odd">3.12</dd> <dt class="field-even">Post-History<span class="colon">:</span></dt> <dd class="field-even"><a class="reference external" href="https://discuss.python.org/t/14824" title="Discourse thread">04-Apr-2022</a>, <a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/B3HYQIE4Z5WBJCC3FUZJZHXLM32I4BZA/" title="Python-Dev thread">11-Apr-2022</a></dd> <dt class="field-odd">Resolution<span class="colon">:</span></dt> <dd class="field-odd"><a class="reference external" href="https://discuss.python.org/t/14824/4">Discourse message</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#note-on-backdating">Note on Backdating</a></li> <li><a class="reference internal" href="#motivation-rationale">Motivation & Rationale</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#conversion-to-heap-types">Conversion to heap types</a></li> <li><a class="reference internal" href="#process">Process</a><ul> <li><a class="reference internal" href="#part-1-preparation">Part 1: Preparation</a></li> <li><a class="reference internal" href="#part-2-implementation">Part 2: Implementation</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a></li> <li><a class="reference internal" href="#security-implications">Security Implications</a></li> <li><a class="reference internal" href="#how-to-teach-this">How to Teach This</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> <p>Extensions in the standard library will be converted to multi-phase initialization (<a class="pep reference internal" href="../pep-0489/" title="PEP 489 – Multi-phase extension module initialization">PEP 489</a>) and where possible, all state will be stored on module objects rather than in process-global variables.</p> </section> <section id="note-on-backdating"> <h2><a class="toc-backref" href="#note-on-backdating" role="doc-backlink">Note on Backdating</a></h2> <p>Much of this proposal has already been implemented. We submit this PEP to explain the changes, seek consensus on whether they are good, propose the remaining changes, and set best practices for new modules.</p> </section> <section id="motivation-rationale"> <h2><a class="toc-backref" href="#motivation-rationale" role="doc-backlink">Motivation & Rationale</a></h2> <p>The informational <a class="pep reference internal" href="../pep-0630/" title="PEP 630 – Isolating Extension Modules">PEP 630</a> describes the background, motivation, rationale, implications and implementation notes of the proposed changes as they apply generally to any extension module (not just the standard library).</p> <p>It is an integral part of this proposal. Read it first.</p> <p>This PEP discusses specifics of the standard library.</p> </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> <p>The body of <a class="pep reference internal" href="../pep-0630/" title="PEP 630 – Isolating Extension Modules">PEP 630</a> will be converted to a HOWTO in the Python documentation, and that PEP will be retired (marked Final).</p> <p>All extension modules in the standard library will be converted to multi-phase initialization introduced in <a class="pep reference internal" href="../pep-0489/" title="PEP 489 – Multi-phase extension module initialization">PEP 489</a>.</p> <p>All stdlib extension modules will be <em>isolated</em>. That is:</p> <ul> <li>Types, functions and other objects defined by the module will either be immutable, or not shared with other module instances.</li> <li>State specific to the module will not be shared with other module instances, unless it represents global state.<p>For example, <code class="docutils literal notranslate"><span class="pre">_csv.field_size_limit</span></code> will get/set a module-specific number. On the other hand, functions like <code class="docutils literal notranslate"><span class="pre">readline.get_history_item</span></code> or <code class="docutils literal notranslate"><span class="pre">os.getpid</span></code> will continue to work with state that is process-global (external to the module, and possibly shared across other libraries, including non-Python ones).</p> </li> </ul> <section id="conversion-to-heap-types"> <h3><a class="toc-backref" href="#conversion-to-heap-types" role="doc-backlink">Conversion to heap types</a></h3> <p>Static types that do not need module state access, and have no other reason to be converted, should stay static.</p> <p>Types whose methods need access to their module instance will be converted to heap types following <a class="pep reference internal" href="../pep-0630/" title="PEP 630 – Isolating Extension Modules">PEP 630</a>, with the following considerations:</p> <ul> <li>All standard library types that used to be static types should remain immutable. Heap types must be defined with the <code class="docutils literal notranslate"><span class="pre">Py_TPFLAGS_IMMUTABLE_TYPE</span></code> flag to retain immutability. See <a class="reference external" href="https://bugs.python.org/issue43908">bpo-43908</a>.<p>Tests should ensure <code class="docutils literal notranslate"><span class="pre">TypeError</span></code> is raised when trying to create a new attribute of an immutable type.</p> </li> <li>A static type with <code class="docutils literal notranslate"><span class="pre">tp_new</span> <span class="pre">=</span> <span class="pre">NULL</span></code> does not have a public constructor, but heap types inherit the constructor from the base class. Make sure types that previously were impossible to instantiate retain that feature; use <code class="docutils literal notranslate"><span class="pre">Py_TPFLAGS_DISALLOW_INSTANTIATION</span></code>. Add tests using <code class="docutils literal notranslate"><span class="pre">test.support.check_disallow_instantiation()</span></code>. See <a class="reference external" href="https://bugs.python.org/issue43916">bpo-43916</a>.</li> <li>Converted heap types may unintentionally become serializable (<code class="docutils literal notranslate"><span class="pre">pickle</span></code>-able). Test that calling <code class="docutils literal notranslate"><span class="pre">pickle.dumps</span></code> has the same result before and after conversion, and if the test fails, add a <code class="docutils literal notranslate"><span class="pre">__reduce__</span></code> method that raises <code class="docutils literal notranslate"><span class="pre">TypeError</span></code>. See <a class="reference external" href="https://github.com/python/cpython/pull/21002/files">PR-21002</a> for an example.</li> </ul> <p>These issues will be added to the Devguide to help any future conversions.</p> <p>If another kind of issue is found, the module in question should be unchanged until a solution is found and added to the Devguide, and already converted modules are checked and fixed.</p> </section> <section id="process"> <h3><a class="toc-backref" href="#process" role="doc-backlink">Process</a></h3> <p>The following process should be added to the Devguide, and remain until all modules are converted. Any new findings should be documented there or in the general HOWTO.</p> <section id="part-1-preparation"> <h4><a class="toc-backref" href="#part-1-preparation" role="doc-backlink">Part 1: Preparation</a></h4> <ol class="arabic simple"> <li>Open a discussion, either on the bug tracker or on Discourse. Involve the module maintainer and/or code owner. Explain the reason and rationale for the changes.</li> <li>Identify global state performance bottlenecks. Create a proof-of-concept implementation, and measure the performance impact. <code class="docutils literal notranslate"><span class="pre">pyperf</span></code> is a good tool for benchmarking.</li> <li>Create an implementation plan. For small modules with few types, a single PR may do the job. For larger modules with lots of types, and possibly also external library callbacks, multiple PR’s will be needed.</li> </ol> </section> <section id="part-2-implementation"> <h4><a class="toc-backref" href="#part-2-implementation" role="doc-backlink">Part 2: Implementation</a></h4> <p>Note: this is a suggested implementation plan for a complex module, based on lessons learned with other modules. Feel free to simplify it for smaller modules.</p> <ol class="arabic simple"> <li>Add Argument Clinic where possible; it enables you to easily use the defining class to fetch module state from type methods.</li> <li>Prepare for module state; establish a module state <code class="docutils literal notranslate"><span class="pre">struct</span></code>, add an instance as a static global variable, and create helper stubs for fetching the module state.</li> <li>Add relevant global variables to the module state <code class="docutils literal notranslate"><span class="pre">struct</span></code>, and modify code that accesses the global state to use the module state helpers instead. This step may be broken into several PR’s.</li> <li>Where necessary, convert static types to heap types.</li> <li>Convert the global module state struct to true module state.</li> <li>Implement multi-phase initialisation.</li> </ol> <p>Steps 4 through 6 should preferably land in a single alpha development phase.</p> </section> </section> </section> <section id="backwards-compatibility"> <h2><a class="toc-backref" href="#backwards-compatibility" role="doc-backlink">Backwards Compatibility</a></h2> <p>Extension modules in the standard library will now be loadable more than once. For example, deleting such a module from <code class="docutils literal notranslate"><span class="pre">sys.modules</span></code> and re-importing it will result in a fresh module instance, isolated from any previously loaded instances.</p> <p>This may affect code that expected the previous behavior: globals of extension modules were shallowly copied from the first loaded module.</p> </section> <section id="security-implications"> <h2><a class="toc-backref" href="#security-implications" role="doc-backlink">Security Implications</a></h2> <p>None known.</p> </section> <section id="how-to-teach-this"> <h2><a class="toc-backref" href="#how-to-teach-this" role="doc-backlink">How to Teach This</a></h2> <p>A large part of this proposal is a HOWTO aimed at experienced users, which will be moved to the documentation.</p> <p>Beginners should not be affected.</p> </section> <section id="reference-implementation"> <h2><a class="toc-backref" href="#reference-implementation" role="doc-backlink">Reference Implementation</a></h2> <p>Most of the changes are now in the main branch, as commits for these issues:</p> <ul class="simple"> <li><a class="reference external" href="https://bugs.python.org/issue40077">bpo-40077, Convert static types to heap types: use PyType_FromSpec()</a></li> <li><a class="reference external" href="https://bugs.python.org/issue46417">bpo-46417, Clear static types in Py_Finalize() for embedded Python</a></li> <li><a class="reference external" href="https://bugs.python.org/issue1635741">bpo-1635741, Py_Finalize() doesn’t clear all Python objects at exit</a></li> </ul> <p>As an example, changes and fix-ups done in the <code class="docutils literal notranslate"><span class="pre">_csv</span></code> module are:</p> <ul class="simple"> <li><a class="reference external" href="https://github.com/python/cpython/pull/23224">GH-23224, Remove static state from the _csv module</a></li> <li><a class="reference external" href="https://github.com/python/cpython/pull/26008">GH-26008, Allow subclassing of csv.Error</a></li> <li><a class="reference external" href="https://github.com/python/cpython/pull/26074">GH-26074, Add GC support to _csv heap types</a></li> <li><a class="reference external" href="https://github.com/python/cpython/pull/26351">GH-26351, Make heap types converted during 3.10 alpha immutable</a></li> </ul> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> <p>This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.</p> </section> </section> <hr class="docutils" /> <p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0687.rst">https://github.com/python/peps/blob/main/peps/pep-0687.rst</a></p> <p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0687.rst">2025-02-01 08:55:40 GMT</a></p> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#note-on-backdating">Note on Backdating</a></li> <li><a class="reference internal" href="#motivation-rationale">Motivation & Rationale</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#conversion-to-heap-types">Conversion to heap types</a></li> <li><a class="reference internal" href="#process">Process</a><ul> <li><a class="reference internal" href="#part-1-preparation">Part 1: Preparation</a></li> <li><a class="reference internal" href="#part-2-implementation">Part 2: Implementation</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a></li> <li><a class="reference internal" href="#security-implications">Security Implications</a></li> <li><a class="reference internal" href="#how-to-teach-this">How to Teach This</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <br> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0687.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>