CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 554 – Multiple Interpreters in the Stdlib | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0554/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 554 – Multiple Interpreters in the Stdlib | peps.python.org'> <meta property="og:description" content="CPython has supported multiple interpreters in the same process (AKA “subinterpreters”) since version 1.5 (1997). The feature has been available via the C-API. [c-api] Multiple interpreters operate in relative isolation from one another, which facilit..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0554/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="CPython has supported multiple interpreters in the same process (AKA “subinterpreters”) since version 1.5 (1997). The feature has been available via the C-API. [c-api] Multiple interpreters operate in relative isolation from one another, which facilit..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 554</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 554 – Multiple Interpreters in the Stdlib</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Eric Snow <ericsnowcurrently at gmail.com></dd> <dt class="field-even">Discussions-To:</dt> <dd class="field-even"><a class="reference external" href="https://discuss.python.org/t/pep-554-multiple-interpreters-in-the-stdlib/24855">Discourse thread</a></dd> <dt class="field-odd">Status:</dt> <dd class="field-odd"><abbr title="Replaced by another succeeding PEP">Superseded</abbr></dd> <dt class="field-even">Type:</dt> <dd class="field-even"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-odd">Created:</dt> <dd class="field-odd">05-Sep-2017</dd> <dt class="field-even">Python-Version:</dt> <dd class="field-even">3.13</dd> <dt class="field-odd">Post-History:</dt> <dd class="field-odd"><a class="reference external" href="https://mail.python.org/archives/list/python-ideas@python.org/thread/HQQWEE527HG3ILJVKQTXVSJIQO6NUSIA/" title="Python-Ideas thread">07-Sep-2017</a>, <a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/NBWMA6LVD22XOUYC5ZMPBFWDQOECRP77/" title="Python-Dev thread">08-Sep-2017</a>, <a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/EG4FSFG5E3O22FTIUQOXMQ6X6B5X3DP7/" title="Python-Dev thread">13-Sep-2017</a>, <a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/BCSRGAMCYB3NGXNU42U66J56XNZVMQP2/" title="Python-Dev thread">05-Dec-2017</a>, <a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/X2KPCSRVBD2QD5GP5IMXXZTGZ46OXD3D/" title="Python-Dev thread">04-May-2020</a>, <a class="reference external" href="https://discuss.python.org/t/pep-554-multiple-interpreters-in-the-stdlib/24855/2/" title="Discourse message">14-Mar-2023</a>, <a class="reference external" href="https://discuss.python.org/t/pep-554-multiple-interpreters-in-the-stdlib/24855/26/" title="Discourse message">01-Nov-2023</a></dd> <dt class="field-even">Superseded-By:</dt> <dd class="field-even"><a class="reference external" href="../pep-0734/">734</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#proposal">Proposal</a><ul> <li><a class="reference internal" href="#the-interpreters-module">The “interpreters” Module</a></li> <li><a class="reference internal" href="#api-summary-for-interpreters-module">API summary for interpreters module</a></li> <li><a class="reference internal" href="#concurrent-futures-interpreterpoolexecutor">concurrent.futures.InterpreterPoolExecutor</a></li> <li><a class="reference internal" href="#help-for-extension-module-maintainers">Help for Extension Module Maintainers</a></li> </ul> </li> <li><a class="reference internal" href="#examples">Examples</a><ul> <li><a class="reference internal" href="#run-isolated-code-in-current-os-thread">Run isolated code in current OS thread</a></li> <li><a class="reference internal" href="#run-in-a-different-thread">Run in a different thread</a></li> <li><a class="reference internal" href="#pre-populate-an-interpreter">Pre-populate an interpreter</a></li> <li><a class="reference internal" href="#handling-an-exception">Handling an exception</a></li> <li><a class="reference internal" href="#re-raising-an-exception">Re-raising an exception</a></li> <li><a class="reference internal" href="#interact-with-the-main-namespace">Interact with the __main__ namespace</a></li> <li><a class="reference internal" href="#synchronize-using-an-os-pipe">Synchronize using an OS pipe</a></li> <li><a class="reference internal" href="#sharing-a-file-descriptor">Sharing a file descriptor</a></li> <li><a class="reference internal" href="#passing-objects-via-pickle">Passing objects via pickle</a></li> <li><a class="reference internal" href="#capturing-an-interpreter-s-stdout">Capturing an interpreter’s stdout</a></li> <li><a class="reference internal" href="#running-a-module">Running a module</a></li> <li><a class="reference internal" href="#running-as-script-including-zip-archives-directories">Running as script (including zip archives & directories)</a></li> <li><a class="reference internal" href="#using-a-channel-to-communicate">Using a channel to communicate</a></li> <li><a class="reference internal" href="#sharing-a-memoryview-imagine-map-reduce">Sharing a memoryview (imagine map-reduce)</a></li> </ul> </li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#concerns">Concerns</a></li> </ul> </li> <li><a class="reference internal" href="#about-subinterpreters">About Subinterpreters</a><ul> <li><a class="reference internal" href="#concurrency">Concurrency</a></li> <li><a class="reference internal" href="#shared-data">Shared Data</a></li> <li><a class="reference internal" href="#interpreter-isolation">Interpreter Isolation</a></li> <li><a class="reference internal" href="#existing-usage">Existing Usage</a></li> </ul> </li> <li><a class="reference internal" href="#alternate-python-implementations">Alternate Python Implementations</a></li> <li><a class="reference internal" href="#interpreters-module-api">“interpreters” Module API</a><ul> <li><a class="reference internal" href="#uncaught-exceptions">Uncaught Exceptions</a></li> </ul> </li> <li><a class="reference internal" href="#interpreter-restrictions">Interpreter Restrictions</a></li> <li><a class="reference internal" href="#api-for-communication">API For Communication</a><ul> <li><a class="reference internal" href="#shareable-types">Shareable Types</a><ul> <li><a class="reference internal" href="#communicating-through-os-pipes">Communicating Through OS Pipes</a></li> </ul> </li> <li><a class="reference internal" href="#channels">Channels</a></li> <li><a class="reference internal" href="#caveats-for-shared-objects">Caveats For Shared Objects</a></li> </ul> </li> <li><a class="reference internal" href="#documentation">Documentation</a></li> <li><a class="reference internal" href="#alternative-solutions">Alternative Solutions</a></li> <li><a class="reference internal" href="#open-questions">Open Questions</a></li> <li><a class="reference internal" href="#deferred-functionality">Deferred Functionality</a><ul> <li><a class="reference internal" href="#add-convenience-api">Add convenience API</a></li> <li><a class="reference internal" href="#avoid-possible-confusion-about-interpreters-running-in-the-current-thread">Avoid possible confusion about interpreters running in the current thread</a></li> <li><a class="reference internal" href="#clarify-running-vs-has-threads">Clarify “running” vs. “has threads”</a></li> <li><a class="reference internal" href="#a-dunder-method-for-sharing">A Dunder Method For Sharing</a></li> <li><a class="reference internal" href="#interpreter-call">Interpreter.call()</a></li> <li><a class="reference internal" href="#interpreter-run-in-thread">Interpreter.run_in_thread()</a></li> <li><a class="reference internal" href="#synchronization-primitives">Synchronization Primitives</a></li> <li><a class="reference internal" href="#csp-library">CSP Library</a></li> <li><a class="reference internal" href="#syntactic-support">Syntactic Support</a></li> <li><a class="reference internal" href="#multiprocessing">Multiprocessing</a></li> <li><a class="reference internal" href="#c-extension-opt-in-opt-out">C-extension opt-in/opt-out</a></li> <li><a class="reference internal" href="#poisoning-channels">Poisoning channels</a></li> <li><a class="reference internal" href="#resetting-main">Resetting __main__</a></li> <li><a class="reference internal" href="#resetting-an-interpreter-s-state">Resetting an interpreter’s state</a></li> <li><a class="reference internal" href="#copy-an-existing-interpreter-s-state">Copy an existing interpreter’s state</a></li> <li><a class="reference internal" href="#shareable-file-descriptors-and-sockets">Shareable file descriptors and sockets</a></li> <li><a class="reference internal" href="#integration-with-async">Integration with async</a></li> <li><a class="reference internal" href="#support-for-iteration">Support for iteration</a></li> <li><a class="reference internal" href="#channel-context-managers">Channel context managers</a></li> <li><a class="reference internal" href="#pipes-and-queues">Pipes and Queues</a></li> <li><a class="reference internal" href="#return-a-lock-from-send">Return a lock from send()</a></li> <li><a class="reference internal" href="#support-prioritization-in-channels">Support prioritization in channels</a></li> <li><a class="reference internal" href="#support-inheriting-settings-and-more">Support inheriting settings (and more?)</a></li> <li><a class="reference internal" href="#make-exceptions-shareable">Make exceptions shareable</a></li> <li><a class="reference internal" href="#make-everything-shareable-through-serialization">Make everything shareable through serialization</a></li> <li><a class="reference internal" href="#make-runfailederror-cause-lazy">Make RunFailedError.__cause__ lazy</a></li> <li><a class="reference internal" href="#return-a-value-from-interp-exec">Return a value from <code class="docutils literal notranslate">interp.exec()</code></a></li> <li><a class="reference internal" href="#add-a-shareable-synchronization-primitive">Add a shareable synchronization primitive</a></li> <li><a class="reference internal" href="#propagate-systemexit-and-keyboardinterrupt-differently">Propagate SystemExit and KeyboardInterrupt Differently</a></li> <li><a class="reference internal" href="#add-an-explicit-release-and-close-to-channel-end-classes">Add an explicit release() and close() to channel end classes</a></li> <li><a class="reference internal" href="#add-sendchannel-send-buffer">Add SendChannel.send_buffer()</a></li> <li><a class="reference internal" href="#auto-run-in-a-thread">Auto-run in a thread</a></li> </ul> </li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a><ul> <li><a class="reference internal" href="#explicit-channel-association">Explicit channel association</a></li> <li><a class="reference internal" href="#add-an-api-based-on-pipes">Add an API based on pipes</a></li> <li><a class="reference internal" href="#add-an-api-based-on-queues">Add an API based on queues</a></li> <li><a class="reference internal" href="#enumerate">“enumerate”</a></li> <li><a class="reference internal" href="#alternate-solutions-to-prevent-leaking-exceptions-across-interpreters">Alternate solutions to prevent leaking exceptions across interpreters</a></li> <li><a class="reference internal" href="#always-associate-each-new-interpreter-with-its-own-thread">Always associate each new interpreter with its own thread</a></li> <li><a class="reference internal" href="#only-associate-interpreters-upon-use">Only associate interpreters upon use</a></li> <li><a class="reference internal" href="#allow-multiple-simultaneous-calls-to-interpreter-exec">Allow multiple simultaneous calls to Interpreter.exec()</a></li> <li><a class="reference internal" href="#add-a-reraise-method-to-runfailederror">Add a “reraise” method to RunFailedError</a></li> </ul> </li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <div class="admonition note"> Note This PEP effectively continues in a cleaner form in <a class="pep reference internal" href="../pep-0734/" title="PEP 734 – Multiple Interpreters in the Stdlib">PEP 734</a>. This PEP is kept as-is for the sake of the various sections of background information and deferred/rejected ideas that have been stripped from <a class="pep reference internal" href="../pep-0734/" title="PEP 734 – Multiple Interpreters in the Stdlib">PEP 734</a>. </div> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> CPython has supported multiple interpreters in the same process (AKA “subinterpreters”) since version 1.5 (1997). The feature has been available via the C-API. <a class="reference internal" href="#c-api" id="id1">[c-api]</a> Multiple interpreters operate in <a class="reference internal" href="#interpreter-isolation">relative isolation from one another</a>, which facilitates novel alternative approaches to <a class="reference internal" href="#concurrency">concurrency</a>. This proposal introduces the stdlib <code class="docutils literal notranslate">interpreters</code> module. It exposes the basic functionality of multiple interpreters already provided by the C-API, along with basic support for communicating between interpreters. This module is especially relevant since <a class="pep reference internal" href="../pep-0684/" title="PEP 684 – A Per-Interpreter GIL">PEP 684</a> introduced a per-interpreter GIL in Python 3.12. </section> <section id="proposal"> <h2><a class="toc-backref" href="#proposal" role="doc-backlink">Proposal</a></h2> Summary: <ul class="simple"> <li>add a new stdlib module: “interpreters”</li> <li>add concurrent.futures.InterpreterPoolExecutor</li> <li>help for extension module maintainers</li> </ul> <section id="the-interpreters-module"> <h3><a class="toc-backref" href="#the-interpreters-module" role="doc-backlink">The “interpreters” Module</a></h3> The <code class="docutils literal notranslate">interpreters</code> module will provide a high-level interface to the multiple interpreter functionality, and wrap a new low-level <code class="docutils literal notranslate">_interpreters</code> (in the same way as the <code class="docutils literal notranslate">threading</code> module). See the <a class="reference internal" href="#examples">Examples</a> section for concrete usage and use cases. Along with exposing the existing (in CPython) multiple interpreter support, the module will also support a basic mechanism for passing data between interpreters. That involves setting “shareable” objects in the <code class="docutils literal notranslate">__main__</code> module of a target subinterpreter. Some such objects, like <code class="docutils literal notranslate">os.pipe()</code>, may be used to communicate further. The module will also provide a minimal implementation of “channels” as a demonstration of cross-interpreter communication. Note that objects are not shared between interpreters since they are tied to the interpreter in which they were created. Instead, the objects’ data is passed between interpreters. See the <a class="reference internal" href="#shared-data">Shared Data</a> and <a class="reference internal" href="#api-for-communication">API For Communication</a> sections for more details about sharing/communicating between interpreters. </section> <section id="api-summary-for-interpreters-module"> <h3><a class="toc-backref" href="#api-summary-for-interpreters-module" role="doc-backlink">API summary for interpreters module</a></h3> Here is a summary of the API for the <code class="docutils literal notranslate">interpreters</code> module. For a more in-depth explanation of the proposed classes and functions, see the <a class="reference internal" href="#interpreters-module-api">“interpreters” Module API</a> section below. For creating and using interpreters: <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">signature</th> <th class="head">description</th> </tr> </thead> <tbody> <tr class="row-even"><td><code class="docutils literal notranslate">list_all() -> [Interpreter]</code></td> <td>Get all existing interpreters.</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">get_current() -> Interpreter</code></td> <td>Get the currently running interpreter.</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">get_main() -> Interpreter</code></td> <td>Get the main interpreter.</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">create() -> Interpreter</code></td> <td>Initialize a new (idle) Python interpreter.</td> </tr> </tbody> </table> <div class="line-block"> <div class="line"> </div> </div> <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">signature</th> <th class="head">description</th> </tr> </thead> <tbody> <tr class="row-even"><td><code class="docutils literal notranslate">class Interpreter</code></td> <td>A single interpreter.</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">.id</code></td> <td>The interpreter’s ID (read-only).</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">.is_running() -> bool</code></td> <td>Is the interpreter currently executing code?</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">.close()</code></td> <td>Finalize and destroy the interpreter.</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">.set_main_attrs(**kwargs)</code></td> <td>Bind “shareable” objects in <code class="docutils literal notranslate">__main__</code>.</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">.get_main_attr(name)</code></td> <td>Get a “shareable” object from <code class="docutils literal notranslate">__main__</code>.</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">.exec(src_str, /)</code></td> <td><div class="line-block"> <div class="line">Run the given source code in the interpreter</div> <div class="line">(in the current thread).</div> </div> </td> </tr> </tbody> </table> For communicating between interpreters: <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">signature</th> <th class="head">description</th> </tr> </thead> <tbody> <tr class="row-even"><td><code class="docutils literal notranslate">is_shareable(obj) -> Bool</code></td> <td><div class="line-block"> <div class="line">Can the object’s data be passed</div> <div class="line">between interpreters?</div> </div> </td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">create_channel() -> (RecvChannel, SendChannel)</code></td> <td><div class="line-block"> <div class="line">Create a new channel for passing</div> <div class="line">data between interpreters.</div> </div> </td> </tr> </tbody> </table> </section> <section id="concurrent-futures-interpreterpoolexecutor"> <h3><a class="toc-backref" href="#concurrent-futures-interpreterpoolexecutor" role="doc-backlink">concurrent.futures.InterpreterPoolExecutor</a></h3> An executor will be added that extends <code class="docutils literal notranslate">ThreadPoolExecutor</code> to run per-thread tasks in subinterpreters. Initially, the only supported tasks will be whatever <code class="docutils literal notranslate">Interpreter.exec()</code> takes (e.g. a <code class="docutils literal notranslate">str</code> script). However, we may also support some functions, as well as eventually a separate method for pickling the task and arguments, to reduce friction (at the expense of performance for short-running tasks). </section> <section id="help-for-extension-module-maintainers"> <h3><a class="toc-backref" href="#help-for-extension-module-maintainers" role="doc-backlink">Help for Extension Module Maintainers</a></h3> In practice, an extension that implements multi-phase init (<a class="pep reference internal" href="../pep-0489/" title="PEP 489 – Multi-phase extension module initialization">PEP 489</a>) is considered isolated and thus compatible with multiple interpreters. Otherwise it is “incompatible”. Many extension modules are still incompatible. The maintainers and users of such extension modules will both benefit when they are updated to support multiple interpreters. In the meantime, users may become confused by failures when using multiple interpreters, which could negatively impact extension maintainers. See <a class="reference internal" href="#concerns">Concerns</a> below. To mitigate that impact and accelerate compatibility, we will do the following: <ul class="simple"> <li>be clear that extension modules are not required to support use in multiple interpreters</li> <li>raise <code class="docutils literal notranslate">ImportError</code> when an incompatible module is imported in a subinterpreter</li> <li>provide resources (e.g. docs) to help maintainers reach compatibility</li> <li>reach out to the maintainers of Cython and of the most used extension modules (on PyPI) to get feedback and possibly provide assistance</li> </ul> </section> </section> <section id="examples"> <h2><a class="toc-backref" href="#examples" role="doc-backlink">Examples</a></h2> <section id="run-isolated-code-in-current-os-thread"> <h3><a class="toc-backref" href="#run-isolated-code-in-current-os-thread" role="doc-backlink">Run isolated code in current OS thread</a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre>interp = interpreters.create() print('before') interp.exec('print("during")') print('after') </pre></div> </div> </section> <section id="run-in-a-different-thread"> <h3><a class="toc-backref" href="#run-in-a-different-thread" role="doc-backlink">Run in a different thread</a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre>interp = interpreters.create() def run(): interp.exec('print("during")') t = threading.Thread(target=run) print('before') t.start() t.join() print('after') </pre></div> </div> </section> <section id="pre-populate-an-interpreter"> <h3><a class="toc-backref" href="#pre-populate-an-interpreter" role="doc-backlink">Pre-populate an interpreter</a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre>interp = interpreters.create() interp.exec(tw.dedent(""" import some_lib import an_expensive_module some_lib.set_up() """)) wait_for_request() interp.exec(tw.dedent(""" some_lib.handle_request() """)) </pre></div> </div> </section> <section id="handling-an-exception"> <h3><a class="toc-backref" href="#handling-an-exception" role="doc-backlink">Handling an exception</a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre>interp = interpreters.create() try: interp.exec(tw.dedent(""" raise KeyError """)) except interpreters.RunFailedError as exc: print(f"got the error from the subinterpreter: {exc}") </pre></div> </div> </section> <section id="re-raising-an-exception"> <h3><a class="toc-backref" href="#re-raising-an-exception" role="doc-backlink">Re-raising an exception</a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre>interp = interpreters.create() try: try: interp.exec(tw.dedent(""" raise KeyError """)) except interpreters.RunFailedError as exc: raise exc.__cause__ except KeyError: print("got a KeyError from the subinterpreter") </pre></div> </div> Note that this pattern is a candidate for later improvement. </section> <section id="interact-with-the-main-namespace"> <h3><a class="toc-backref" href="#interact-with-the-main-namespace" role="doc-backlink">Interact with the __main__ namespace</a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre>interp = interpreters.create() interp.set_main_attrs(a=1, b=2) interp.exec(tw.dedent(""" res = do_something(a, b) """)) res = interp.get_main_attr('res') </pre></div> </div> </section> <section id="synchronize-using-an-os-pipe"> <h3><a class="toc-backref" href="#synchronize-using-an-os-pipe" role="doc-backlink">Synchronize using an OS pipe</a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre>interp = interpreters.create() r1, s1 = os.pipe() r2, s2 = os.pipe() def task(): interp.exec(tw.dedent(f""" import os os.read({r1}, 1) print('during B') os.write({s2}, '') """)) t = threading.thread(target=task) t.start() print('before') os.write(s1, '') print('during A') os.read(r2, 1) print('after') t.join() </pre></div> </div> </section> <section id="sharing-a-file-descriptor"> <h3><a class="toc-backref" href="#sharing-a-file-descriptor" role="doc-backlink">Sharing a file descriptor</a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre>interp = interpreters.create() with open('spamspamspam') as infile: interp.set_main_attrs(fd=infile.fileno()) interp.exec(tw.dedent(f""" import os for line in os.fdopen(fd): print(line) """)) </pre></div> </div> </section> <section id="passing-objects-via-pickle"> <h3><a class="toc-backref" href="#passing-objects-via-pickle" role="doc-backlink">Passing objects via pickle</a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre>interp = interpreters.create() r, s = os.pipe() interp.exec(tw.dedent(f""" import os import pickle reader = {r} """)) interp.exec(tw.dedent(""" data = b'' c = os.read(reader, 1) while c != b'\x00': while c != b'\x00': data += c c = os.read(reader, 1) obj = pickle.loads(data) do_something(obj) c = os.read(reader, 1) """)) for obj in input: data = pickle.dumps(obj) os.write(s, data) os.write(s, b'\x00') os.write(s, b'\x00') </pre></div> </div> </section> <section id="capturing-an-interpreter-s-stdout"> <h3><a class="toc-backref" href="#capturing-an-interpreter-s-stdout" role="doc-backlink">Capturing an interpreter’s stdout</a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre>interp = interpreters.create() stdout = io.StringIO() with contextlib.redirect_stdout(stdout): interp.exec(tw.dedent(""" print('spam!') """)) assert(stdout.getvalue() == 'spam!') # alternately: interp.exec(tw.dedent(""" import contextlib, io stdout = io.StringIO() with contextlib.redirect_stdout(stdout): print('spam!') captured = stdout.getvalue() """)) captured = interp.get_main_attr('captured') assert(captured == 'spam!') </pre></div> </div> A pipe (<code class="docutils literal notranslate">os.pipe()</code>) could be used similarly. </section> <section id="running-a-module"> <h3><a class="toc-backref" href="#running-a-module" role="doc-backlink">Running a module</a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre>interp = interpreters.create() main_module = mod_name interp.exec(f'import runpy; runpy.run_module({main_module!r})') </pre></div> </div> </section> <section id="running-as-script-including-zip-archives-directories"> <h3><a class="toc-backref" href="#running-as-script-including-zip-archives-directories" role="doc-backlink">Running as script (including zip archives & directories)</a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre>interp = interpreters.create() main_script = path_name interp.exec(f"import runpy; runpy.run_path({main_script!r})") </pre></div> </div> </section> <section id="using-a-channel-to-communicate"> <h3><a class="toc-backref" href="#using-a-channel-to-communicate" role="doc-backlink">Using a channel to communicate</a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre>tasks_recv, tasks = interpreters.create_channel() results, results_send = interpreters.create_channel() def worker(): interp = interpreters.create() interp.set_main_attrs(tasks=tasks_recv, results=results_send) interp.exec(tw.dedent(""" def handle_request(req): ... def capture_exception(exc): ... while True: try: req = tasks.recv() except Exception: # channel closed break try: res = handle_request(req) except Exception as exc: res = capture_exception(exc) results.send_nowait(res) """)) threads = [threading.Thread(target=worker) for _ in range(20)] for t in threads: t.start() requests = ... for req in requests: tasks.send(req) tasks.close() for t in threads: t.join() </pre></div> </div> </section> <section id="sharing-a-memoryview-imagine-map-reduce"> <h3><a class="toc-backref" href="#sharing-a-memoryview-imagine-map-reduce" role="doc-backlink">Sharing a memoryview (imagine map-reduce)</a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre>data, chunksize = read_large_data_set() buf = memoryview(data) numchunks = (len(buf) + 1) / chunksize results = memoryview(b'\0' * numchunks) tasks_recv, tasks = interpreters.create_channel() def worker(): interp = interpreters.create() interp.set_main_attrs(data=buf, results=results, tasks=tasks_recv) interp.exec(tw.dedent(""" while True: try: req = tasks.recv() except Exception: # channel closed break resindex, start, end = req chunk = data[start: end] res = reduce_chunk(chunk) results[resindex] = res """)) t = threading.Thread(target=worker) t.start() for i in range(numchunks): if not workers_running(): raise ... start = i * chunksize end = start + chunksize if end > len(buf): end = len(buf) tasks.send((start, end, i)) tasks.close() t.join() use_results(results) </pre></div> </div> </section> </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> Running code in multiple interpreters provides a useful level of isolation within the same process. This can be leveraged in a number of ways. Furthermore, subinterpreters provide a well-defined framework in which such isolation may extended. (See <a class="pep reference internal" href="../pep-0684/" title="PEP 684 – A Per-Interpreter GIL">PEP 684</a>.) Alyssa (Nick) Coghlan explained some of the benefits through a comparison with multi-processing <a class="reference internal" href="#benefits" id="id2">[benefits]</a>: <div class="highlight-default notranslate"><div class="highlight"><pre>[I] expect that communicating between subinterpreters is going to end up looking an awful lot like communicating between subprocesses via shared memory. The trade-off between the two models will then be that one still just looks like a single process from the point of view of the outside world, and hence doesn't place any extra demands on the underlying OS beyond those required to run CPython with a single interpreter, while the other gives much stricter isolation (including isolating C globals in extension modules), but also demands much more from the OS when it comes to its IPC capabilities. The security risk profiles of the two approaches will also be quite different, since using subinterpreters won't require deliberately poking holes in the process isolation that operating systems give you by default. </pre></div> </div> CPython has supported multiple interpreters, with increasing levels of support, since version 1.5. While the feature has the potential to be a powerful tool, it has suffered from neglect because the multiple interpreter capabilities are not readily available directly from Python. Exposing the existing functionality in the stdlib will help reverse the situation. This proposal is focused on enabling the fundamental capability of multiple interpreters, isolated from each other, in the same Python process. This is a new area for Python so there is relative uncertainly about the best tools to provide as companions to interpreters. Thus we minimize the functionality we add in the proposal as much as possible. <section id="concerns"> <h3><a class="toc-backref" href="#concerns" role="doc-backlink">Concerns</a></h3> <ul class="simple"> <li>“subinterpreters are not worth the trouble”</li> </ul> Some have argued that subinterpreters do not add sufficient benefit to justify making them an official part of Python. Adding features to the language (or stdlib) has a cost in increasing the size of the language. So an addition must pay for itself. In this case, multiple interpreter support provide a novel concurrency model focused on isolated threads of execution. Furthermore, they provide an opportunity for changes in CPython that will allow simultaneous use of multiple CPU cores (currently prevented by the GIL–see <a class="pep reference internal" href="../pep-0684/" title="PEP 684 – A Per-Interpreter GIL">PEP 684</a>). Alternatives to subinterpreters include threading, async, and multiprocessing. Threading is limited by the GIL and async isn’t the right solution for every problem (nor for every person). Multiprocessing is likewise valuable in some but not all situations. Direct IPC (rather than via the multiprocessing module) provides similar benefits but with the same caveat. Notably, subinterpreters are not intended as a replacement for any of the above. Certainly they overlap in some areas, but the benefits of subinterpreters include isolation and (potentially) performance. In particular, subinterpreters provide a direct route to an alternate concurrency model (e.g. CSP) which has found success elsewhere and will appeal to some Python users. That is the core value that the <code class="docutils literal notranslate">interpreters</code> module will provide. <ul class="simple"> <li>“stdlib support for multiple interpreters adds extra burden on C extension authors”</li> </ul> In the <a class="reference internal" href="#interpreter-isolation">Interpreter Isolation</a> section below we identify ways in which isolation in CPython’s subinterpreters is incomplete. Most notable is extension modules that use C globals to store internal state. (<a class="pep reference internal" href="../pep-3121/" title="PEP 3121 – Extension Module Initialization and Finalization">PEP 3121</a> and <a class="pep reference internal" href="../pep-0489/" title="PEP 489 – Multi-phase extension module initialization">PEP 489</a> provide a solution to that problem, followed by some extra APIs that improve efficiency, e.g. <a class="pep reference internal" href="../pep-0573/" title="PEP 573 – Module State Access from C Extension Methods">PEP 573</a>). Consequently, projects that publish extension modules may face an increased maintenance burden as their users start using subinterpreters, where their modules may break. This situation is limited to modules that use C globals (or use libraries that use C globals) to store internal state. For numpy, the reported-bug rate is one every 6 months. <a class="reference internal" href="#bug-rate" id="id3">[bug-rate]</a> Ultimately this comes down to a question of how often it will be a problem in practice: how many projects would be affected, how often their users will be affected, what the additional maintenance burden will be for projects, and what the overall benefit of subinterpreters is to offset those costs. The position of this PEP is that the actual extra maintenance burden will be small and well below the threshold at which subinterpreters are worth it. <ul class="simple"> <li>“creating a new concurrency API deserves much more thought and experimentation, so the new module shouldn’t go into the stdlib right away, if ever”</li> </ul> Introducing an API for a new concurrency model, like happened with asyncio, is an extremely large project that requires a lot of careful consideration. It is not something that can be done as simply as this PEP proposes and likely deserves significant time on PyPI to mature. (See <a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/message/TUEAZNZHVJGGLL4OFD32OW6JJDKM6FAS/">Nathaniel’s post</a> on python-dev.) However, this PEP does not propose any new concurrency API. At most it exposes minimal tools (e.g. subinterpreters, channels) which may be used to write code that follows patterns associated with (relatively) new-to-Python <a class="reference internal" href="#concurrency">concurrency models</a>. Those tools could also be used as the basis for APIs for such concurrency models. Again, this PEP does not propose any such API. <ul class="simple"> <li>“there is no point to exposing subinterpreters if they still share the GIL”</li> <li>“the effort to make the GIL per-interpreter is disruptive and risky”</li> </ul> A common misconception is that this PEP also includes a promise that interpreters will no longer share the GIL. When that is clarified, the next question is “what is the point?”. This is already answered at length in this PEP. Just to be clear, the value lies in: <div class="highlight-default notranslate"><div class="highlight"><pre>* increase exposure of the existing feature, which helps improve the code health of the entire CPython runtime * expose the (mostly) isolated execution of interpreters * preparation for per-interpreter GIL * encourage experimentation </pre></div> </div> <ul class="simple"> <li>“data sharing can have a negative impact on cache performance in multi-core scenarios”</li> </ul> (See <a class="reference internal" href="#cache-line-ping-pong" id="id4">[cache-line-ping-pong]</a>.) This shouldn’t be a problem for now as we have no immediate plans to actually share data between interpreters, instead focusing on copying. </section> </section> <section id="about-subinterpreters"> <h2><a class="toc-backref" href="#about-subinterpreters" role="doc-backlink">About Subinterpreters</a></h2> <section id="concurrency"> <h3><a class="toc-backref" href="#concurrency" role="doc-backlink">Concurrency</a></h3> Concurrency is a challenging area of software development. Decades of research and practice have led to a wide variety of concurrency models, each with different goals. Most center on correctness and usability. One class of concurrency models focuses on isolated threads of execution that interoperate through some message passing scheme. A notable example is Communicating Sequential Processes <a class="reference internal" href="#csp" id="id5">[CSP]</a> (upon which Go’s concurrency is roughly based). The intended isolation inherent to CPython’s interpreters makes them well-suited to this approach. </section> <section id="shared-data"> <h3><a class="toc-backref" href="#shared-data" role="doc-backlink">Shared Data</a></h3> CPython’s interpreters are inherently isolated (with caveats explained below), in contrast to threads. So the same communicate-via-shared-memory approach doesn’t work. Without an alternative, effective use of concurrency via multiple interpreters is significantly limited. The key challenge here is that sharing objects between interpreters faces complexity due to various constraints on object ownership, visibility, and mutability. At a conceptual level it’s easier to reason about concurrency when objects only exist in one interpreter at a time. At a technical level, CPython’s current memory model limits how Python objects may be shared safely between interpreters; effectively, objects are bound to the interpreter in which they were created. Furthermore, the complexity of object sharing increases as interpreters become more isolated, e.g. after GIL removal (though this is mitigated somewhat for some “immortal” objects (see <a class="pep reference internal" href="../pep-0683/" title="PEP 683 – Immortal Objects, Using a Fixed Refcount">PEP 683</a>). Consequently, the mechanism for sharing needs to be carefully considered. There are a number of valid solutions, several of which may be appropriate to support in Python’s stdlib and C-API. Any such solution is likely to share many characteristics with the others. In the meantime, we propose here a minimal solution (<code class="docutils literal notranslate">Interpreter.set_main_attrs()</code>), which sets some precedent for how objects are shared. More importantly, it facilitates the introduction of more advanced approaches later and allows them to coexist and cooperate. In part to demonstrate that, we will provide a basic implementation of “channels”, as a somewhat more advanced sharing solution. Separate proposals may cover: <ul class="simple"> <li>the addition of a public C-API based on the implementation <code class="docutils literal notranslate">Interpreter.set_main_attrs()</code></li> <li>the addition of other sharing approaches to the “interpreters” module</li> </ul> The fundamental enabling feature for communication is that most objects can be converted to some encoding of underlying raw data, which is safe to be passed between interpreters. For example, an <code class="docutils literal notranslate">int</code> object can be turned into a C <code class="docutils literal notranslate">long</code> value, sent to another interpreter, and turned back into an <code class="docutils literal notranslate">int</code> object there. As another example, <code class="docutils literal notranslate">None</code> may be passed as-is. Regardless, the effort to determine the best way forward here is mostly outside the scope of this PEP. In the meantime, this proposal describes a basic interim solution using pipes (<code class="docutils literal notranslate">os.pipe()</code>), as well as providing a dedicated capability (“channels”). See <a class="reference internal" href="#api-for-communication">API For Communication</a> below. </section> <section id="interpreter-isolation"> <h3><a class="toc-backref" href="#interpreter-isolation" role="doc-backlink">Interpreter Isolation</a></h3> CPython’s interpreters are intended to be strictly isolated from each other. Each interpreter has its own copy of all modules, classes, functions, and variables. The same applies to state in C, including in extension modules. The CPython C-API docs explain more. <a class="reference internal" href="#caveats" id="id6">[caveats]</a> However, there are ways in which interpreters do share some state. First of all, some process-global state remains shared: <ul class="simple"> <li>file descriptors</li> <li>low-level env vars</li> <li>process memory (though allocators are isolated)</li> <li>builtin types (e.g. dict, bytes)</li> <li>singletons (e.g. None)</li> <li>underlying static module data (e.g. functions) for builtin/extension/frozen modules</li> </ul> There are no plans to change this. Second, some isolation is faulty due to bugs or implementations that did not take subinterpreters into account. This includes things like extension modules that rely on C globals. <a class="reference internal" href="#cryptography" id="id7">[cryptography]</a> In these cases bugs should be opened (some are already): <ul class="simple"> <li>readline module hook functions (<a class="reference external" href="http://bugs.python.org/issue4202">http://bugs.python.org/issue4202</a>)</li> <li>memory leaks on re-init (<a class="reference external" href="http://bugs.python.org/issue21387">http://bugs.python.org/issue21387</a>)</li> </ul> Finally, some potential isolation is missing due to the current design of CPython. Improvements are currently going on to address gaps in this area: <ul class="simple"> <li>extensions using the <code class="docutils literal notranslate">PyGILState_*</code> API are somewhat incompatible <a class="reference internal" href="#gilstate" id="id8">[gilstate]</a></li> </ul> </section> <section id="existing-usage"> <h3><a class="toc-backref" href="#existing-usage" role="doc-backlink">Existing Usage</a></h3> Multiple interpreter support has not been a widely used feature. In fact, there have been only a handful of documented cases of widespread usage, including <a class="reference external" href="https://github.com/GrahamDumpleton/mod_wsgi">mod_wsgi</a>, <a class="reference external" href="https://github.com/ceph/ceph/pull/14971">OpenStack Ceph</a>, and <a class="reference external" href="https://github.com/ninia/jep">JEP</a>. On the one hand, these cases provide confidence that existing multiple interpreter support is relatively stable. On the other hand, there isn’t much of a sample size from which to judge the utility of the feature. </section> </section> <section id="alternate-python-implementations"> <h2><a class="toc-backref" href="#alternate-python-implementations" role="doc-backlink">Alternate Python Implementations</a></h2> I’ve solicited feedback from various Python implementors about support for subinterpreters. Each has indicated that they would be able to support multiple interpreters in the same process (if they choose to) without a lot of trouble. Here are the projects I contacted: <ul class="simple"> <li>jython (<a class="reference internal" href="#jython" id="id9">[jython]</a>)</li> <li>ironpython (personal correspondence)</li> <li>pypy (personal correspondence)</li> <li>micropython (personal correspondence)</li> </ul> </section> <section id="interpreters-module-api"> <h2><a class="toc-backref" href="#interpreters-module-api" role="doc-backlink">“interpreters” Module API</a></h2> The module provides the following functions: <div class="highlight-default notranslate"><div class="highlight"><pre>list_all() -> [Interpreter] Return a list of all existing interpreters. get_current() => Interpreter Return the currently running interpreter. get_main() => Interpreter Return the main interpreter. If the Python implementation has no concept of a main interpreter then return None. create() -> Interpreter Initialize a new Python interpreter and return it. It will remain idle until something is run in it and always run in its own thread. is_shareable(obj) -> bool: Return True if the object may be "shared" between interpreters. This does not necessarily mean that the actual objects will be shared. Instead, it means that the objects' underlying data will be shared in a cross-interpreter way, whether via a proxy, a copy, or some other means. </pre></div> </div> The module also provides the following class: <div class="highlight-default notranslate"><div class="highlight"><pre>class Interpreter(id): id -> int: The interpreter's ID. (read-only) is_running() -> bool: Return whether or not the interpreter's "exec()" is currently executing code. Code running in subthreads is ignored. Calling this on the current interpreter will always return True. close(): Finalize and destroy the interpreter. This may not be called on an already running interpreter. Doing so results in a RuntimeError. set_main_attrs(iterable_or_mapping, /): set_main_attrs(**kwargs): Set attributes in the interpreter's __main__ module corresponding to the given name-value pairs. Each value must be a "shareable" object and will be converted to a new object (e.g. copy, proxy) in whatever way that object's type defines. If an attribute with the same name is already set, it will be overwritten. This method is helpful for setting up an interpreter before calling exec(). get_main_attr(name, default=None, /): Return the value of the corresponding attribute of the interpreter's __main__ module. If the attribute isn't set then the default is returned. If it is set, but the value isn't "shareable" then a ValueError is raised. This may be used to introspect the __main__ module, as well as a very basic mechanism for "returning" one or more results from Interpreter.exec(). exec(source_str, /): Run the provided Python source code in the interpreter, in its __main__ module. This may not be called on an already running interpreter. Doing so results in a RuntimeError. An "interp.exec()" call is similar to a builtin exec() call (or to calling a function that returns None). Once "interp.exec()" completes, the code that called "exec()" continues executing (in the original interpreter). Likewise, if there is any uncaught exception then it effectively (see below) propagates into the code where ``interp.exec()`` was called. Like exec() (and threads), but unlike function calls, there is no return value. If any "return" value from the code is needed, send the data out via a pipe (os.pipe()) or channel or other cross-interpreter communication mechanism. The big difference from exec() or functions is that "interp.exec()" executes the code in an entirely different interpreter, with entirely separate state. The interpreters are completely isolated from each other, so the state of the original interpreter (including the code it was executing in the current OS thread) does not affect the state of the target interpreter (the one that will execute the code). Likewise, the target does not affect the original, nor any of its other threads. Instead, the state of the original interpreter (for this thread) is frozen, and the code it's executing code completely blocks. At that point, the target interpreter is given control of the OS thread. Then, when it finishes executing, the original interpreter gets control back and continues executing. So calling "interp.exec()" will effectively cause the current Python thread to completely pause. Sometimes you won't want that pause, in which case you should make the "exec()" call in another thread. To do so, add a function that calls "interp.exec()" and then run that function in a normal "threading.Thread". Note that the interpreter's state is never reset, neither before "interp.exec()" executes the code nor after. Thus the interpreter state is preserved between calls to "interp.exec()". This includes "sys.modules", the "builtins" module, and the internal state of C extension modules. Also note that "interp.exec()" executes in the namespace of the "__main__" module, just like scripts, the REPL, "-m", and "-c". Just as the interpreter's state is not ever reset, the "__main__" module is never reset. You can imagine concatenating the code from each "interp.exec()" call into one long script. This is the same as how the REPL operates. Supported code: source text. </pre></div> </div> In addition to the functionality of <code class="docutils literal notranslate">Interpreter.set_main_attrs()</code>, the module provides a related way to pass data between interpreters: channels. See <a class="reference internal" href="#channels">Channels</a> below. <section id="uncaught-exceptions"> <h3><a class="toc-backref" href="#uncaught-exceptions" role="doc-backlink">Uncaught Exceptions</a></h3> Regarding uncaught exceptions in <code class="docutils literal notranslate">Interpreter.exec()</code>, we noted that they are “effectively” propagated into the code where <code class="docutils literal notranslate">interp.exec()</code> was called. To prevent leaking exceptions (and tracebacks) between interpreters, we create a surrogate of the exception and its traceback (see <a class="reference external" href="https://docs.python.org/3/library/traceback.html#traceback.TracebackException" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">traceback.TracebackException</code></a>), set it to <code class="docutils literal notranslate">__cause__</code> on a new <code class="docutils literal notranslate">interpreters.RunFailedError</code>, and raise that. Directly raising (a proxy of) the exception is problematic since it’s harder to distinguish between an error in the <code class="docutils literal notranslate">interp.exec()</code> call and an uncaught exception from the subinterpreter. </section> </section> <section id="interpreter-restrictions"> <h2><a class="toc-backref" href="#interpreter-restrictions" role="doc-backlink">Interpreter Restrictions</a></h2> Every new interpreter created by <code class="docutils literal notranslate">interpreters.create()</code> now has specific restrictions on any code it runs. This includes the following: <ul class="simple"> <li>importing an extension module fails if it does not implement multi-phase init</li> <li>daemon threads may not be created</li> <li><code class="docutils literal notranslate">os.fork()</code> is not allowed (so no <code class="docutils literal notranslate">multiprocessing</code>)</li> <li><code class="docutils literal notranslate">os.exec*()</code> is not allowed (but “fork+exec”, a la <code class="docutils literal notranslate">subprocess</code> is okay)</li> </ul> Note that interpreters created with the existing C-API do not have these restrictions. The same is true for the “main” interpreter, so existing use of Python will not change. We may choose to later loosen some of the above restrictions or provide a way to enable/disable granular restrictions individually. Regardless, requiring multi-phase init from extension modules will always be a default restriction. </section> <section id="api-for-communication"> <h2><a class="toc-backref" href="#api-for-communication" role="doc-backlink">API For Communication</a></h2> As discussed in <a class="reference internal" href="#shared-data">Shared Data</a> above, multiple interpreter support is less useful without a mechanism for sharing data (communicating) between them. Sharing actual Python objects between interpreters, however, has enough potential problems that we are avoiding support for that in this proposal. Nor, as mentioned earlier, are we adding anything more than a basic mechanism for communication. That mechanism is the <code class="docutils literal notranslate">Interpreter.set_main_attrs()</code> method. It may be used to set up global variables before <code class="docutils literal notranslate">Interpreter.exec()</code> is called. The name-value pairs passed to <code class="docutils literal notranslate">set_main_attrs()</code> are bound as attributes of the interpreter’s <code class="docutils literal notranslate">__main__</code> module. The values must be “shareable”. See <a class="reference internal" href="#shareable-types">Shareable Types</a> below. Additional approaches to communicating and sharing objects are enabled through <code class="docutils literal notranslate">Interpreter.set_main_attrs()</code>. A shareable object could be implemented which works like a queue, but with cross-interpreter safety. In fact, this PEP does include an example of such an approach: channels. <section id="shareable-types"> <h3><a class="toc-backref" href="#shareable-types" role="doc-backlink">Shareable Types</a></h3> An object is “shareable” if its type supports shareable instances. The type must implement a new internal protocol, which is used to convert an object to interpreter-independent data and then converted back to an object on the other side. Also see <a class="reference internal" href="#interpreters-is-shareable">is_shareable()</a> above. A minimal set of simple, immutable builtin types will be supported initially, including: <ul class="simple"> <li><code class="docutils literal notranslate">None</code></li> <li><code class="docutils literal notranslate">bool</code></li> <li><code class="docutils literal notranslate">bytes</code></li> <li><code class="docutils literal notranslate">str</code></li> <li><code class="docutils literal notranslate">int</code></li> <li><code class="docutils literal notranslate">float</code></li> </ul> We will also support a small number of complex types initially: <ul class="simple"> <li><code class="docutils literal notranslate">memoryview</code>, to allow sharing <a class="pep reference internal" href="../pep-3118/" title="PEP 3118 – Revising the buffer protocol">PEP 3118</a> buffers</li> <li><a class="reference internal" href="#channels">channels</a></li> </ul> Further builtin types may be supported later, complex or not. Limiting the initial shareable types is a practical matter, reducing the potential complexity of the initial implementation. There are a number of strategies we may pursue in the future to expand supported objects, once we have more experience with interpreter isolation. In the meantime, a separate proposal will discuss making the internal protocol (and C-API) used by <code class="docutils literal notranslate">Interpreter.set_main_attrs()</code> public. With that protocol, support for other types could be added by extension modules. <section id="communicating-through-os-pipes"> <h4><a class="toc-backref" href="#communicating-through-os-pipes" role="doc-backlink">Communicating Through OS Pipes</a></h4> Even without a dedicated object for communication, users may already use existing tools. For example, one basic approach for sending data between interpreters is to use a pipe (see <code class="docutils literal notranslate">os.pipe()</code>): <ol class="arabic simple"> <li>interpreter A calls <code class="docutils literal notranslate">os.pipe()</code> to get a read/write pair of file descriptors (both <code class="docutils literal notranslate">int</code> objects)</li> <li>interpreter A calls <code class="docutils literal notranslate">interp.set_main_attrs()</code>, binding the read FD (or embeds it using string formatting)</li> <li>interpreter A calls <code class="docutils literal notranslate">interp.exec()</code> on interpreter B</li> <li>interpreter A writes some bytes to the write FD</li> <li>interpreter B reads those bytes</li> </ol> Several of the earlier examples demonstrate this, such as <a class="reference internal" href="#synchronize-using-an-os-pipe">Synchronize using an OS pipe</a>. </section> </section> <section id="channels"> <h3><a class="toc-backref" href="#channels" role="doc-backlink">Channels</a></h3> The <code class="docutils literal notranslate">interpreters</code> module will include a dedicated solution for passing object data between interpreters: channels. They are included in the module in part to provide an easier mechanism than using <code class="docutils literal notranslate">os.pipe()</code> and in part to demonstrate how libraries may take advantage of <code class="docutils literal notranslate">Interpreter.set_main_attrs()</code> and the protocol it uses. A channel is a simplex FIFO. It is a basic, opt-in data sharing mechanism that draws inspiration from pipes, queues, and CSP’s channels. <a class="reference internal" href="#fifo" id="id10">[fifo]</a> The main difference from pipes is that channels can be associated with zero or more interpreters on either end. Like queues, which are also many-to-many, channels are buffered (though they also offer methods with unbuffered semantics). Channels have two operations: send and receive. A key characteristic of those operations is that channels transmit data derived from Python objects rather than the objects themselves. When objects are sent, their data is extracted. When the “object” is received in the other interpreter, the data is converted back into an object owned by that interpreter. To make this work, the mutable shared state will be managed by the Python runtime, not by any of the interpreters. Initially we will support only one type of objects for shared state: the channels provided by <code class="docutils literal notranslate">interpreters.create_channel()</code>. Channels, in turn, will carefully manage passing objects between interpreters. This approach, including keeping the API minimal, helps us avoid further exposing any underlying complexity to Python users. The <code class="docutils literal notranslate">interpreters</code> module provides the following function related to channels: <div class="highlight-default notranslate"><div class="highlight"><pre>create_channel() -> (RecvChannel, SendChannel): Create a new channel and return (recv, send), the RecvChannel and SendChannel corresponding to the ends of the channel. Both ends of the channel are supported "shared" objects (i.e. may be safely shared by different interpreters. Thus they may be set using "Interpreter.set_main_attrs()". </pre></div> </div> The module also provides the following channel-related classes: <div class="highlight-default notranslate"><div class="highlight"><pre>class RecvChannel(id): The receiving end of a channel. An interpreter may use this to receive objects from another interpreter. Any type supported by Interpreter.set_main_attrs() will be supported here, though at first only a few of the simple, immutable builtin types will be supported. id -> int: The channel's unique ID. The "send" end has the same one. recv(*, timeout=None): Return the next object from the channel. If none have been sent then wait until the next send (or until the timeout is hit). At the least, the object will be equivalent to the sent object. That will almost always mean the same type with the same data, though it could also be a compatible proxy. Regardless, it may use a copy of that data or actually share the data. That's up to the object's type. recv_nowait(default=None): Return the next object from the channel. If none have been sent then return the default. Otherwise, this is the same as the "recv()" method. class SendChannel(id): The sending end of a channel. An interpreter may use this to send objects to another interpreter. Any type supported by Interpreter.set_main_attrs() will be supported here, though at first only a few of the simple, immutable builtin types will be supported. id -> int: The channel's unique ID. The "recv" end has the same one. send(obj, *, timeout=None): Send the object (i.e. its data) to the "recv" end of the channel. Wait until the object is received. If the object is not shareable then ValueError is raised. The builtin memoryview is supported, so sending a buffer across involves first wrapping the object in a memoryview and then sending that. send_nowait(obj): Send the object to the "recv" end of the channel. This behaves the same as "send()", except for the waiting part. If no interpreter is currently receiving (waiting on the other end) then queue the object and return False. Otherwise return True. </pre></div> </div> </section> <section id="caveats-for-shared-objects"> <h3><a class="toc-backref" href="#caveats-for-shared-objects" role="doc-backlink">Caveats For Shared Objects</a></h3> Again, Python objects are not shared between interpreters. However, in some cases data those objects wrap is actually shared and not just copied. One example might be <a class="pep reference internal" href="../pep-3118/" title="PEP 3118 – Revising the buffer protocol">PEP 3118</a> buffers. In those cases the object in the original interpreter is kept alive until the shared data in the other interpreter is no longer used. Then object destruction can happen like normal in the original interpreter, along with the previously shared data. </section> </section> <section id="documentation"> <h2><a class="toc-backref" href="#documentation" role="doc-backlink">Documentation</a></h2> The new stdlib docs page for the <code class="docutils literal notranslate">interpreters</code> module will include the following: <ul class="simple"> <li>(at the top) a clear note that support for multiple interpreters is not required from extension modules</li> <li>some explanation about what subinterpreters are</li> <li>brief examples of how to use multiple interpreters (and communicating between them)</li> <li>a summary of the limitations of using multiple interpreters</li> <li>(for extension maintainers) a link to the resources for ensuring multiple interpreters compatibility</li> <li>much of the API information in this PEP</li> </ul> Docs about resources for extension maintainers already exist on the <a class="reference external" href="https://docs.python.org/3/howto/isolating-extensions.html">Isolating Extension Modules</a> howto page. Any extra help will be added there. For example, it may prove helpful to discuss strategies for dealing with linked libraries that keep their own subinterpreter-incompatible global state. Note that the documentation will play a large part in mitigating any negative impact that the new <code class="docutils literal notranslate">interpreters</code> module might have on extension module maintainers. Also, the <code class="docutils literal notranslate">ImportError</code> for incompatible extension modules will be updated to clearly say it is due to missing multiple interpreters compatibility and that extensions are not required to provide it. This will help set user expectations properly. </section> <section id="alternative-solutions"> <h2><a class="toc-backref" href="#alternative-solutions" role="doc-backlink">Alternative Solutions</a></h2> One possible alternative to a new module is to add support for interpreters to <code class="docutils literal notranslate">concurrent.futures</code>. There are several reasons why that wouldn’t work: <ul class="simple"> <li>the obvious place to look for multiple interpreters support is an “interpreters” module, much as with “threading”, etc.</li> <li><code class="docutils literal notranslate">concurrent.futures</code> is all about executing functions but currently we don’t have a good way to run a function from one interpreter in another</li> </ul> Similar reasoning applies for support in the <code class="docutils literal notranslate">multiprocessing</code> module. </section> <section id="open-questions"> <h2><a class="toc-backref" href="#open-questions" role="doc-backlink">Open Questions</a></h2> <ul class="simple"> <li>will is be too confusing that <code class="docutils literal notranslate">interp.exec()</code> runs in the current thread?</li> <li>should we add pickling fallbacks right now for <code class="docutils literal notranslate">interp.exec()</code>, and/or <code class="docutils literal notranslate">Interpreter.set_main_attrs()</code> and <code class="docutils literal notranslate">Interpreter.get_main_attr()</code>?</li> <li>should we support (limited) functions in <code class="docutils literal notranslate">interp.exec()</code> right now?</li> <li>rename <code class="docutils literal notranslate">Interpreter.close()</code> to <code class="docutils literal notranslate">Interpreter.destroy()</code>?</li> <li>drop <code class="docutils literal notranslate">Interpreter.get_main_attr()</code>, since we have channels?</li> <li>should channels be its own PEP?</li> </ul> </section> <section id="deferred-functionality"> <h2><a class="toc-backref" href="#deferred-functionality" role="doc-backlink">Deferred Functionality</a></h2> In the interest of keeping this proposal minimal, the following functionality has been left out for future consideration. Note that this is not a judgement against any of said capability, but rather a deferment. That said, each is arguably valid. <section id="add-convenience-api"> <h3><a class="toc-backref" href="#add-convenience-api" role="doc-backlink">Add convenience API</a></h3> There are a number of things I can imagine would smooth out hypothetical rough edges with the new module: <ul class="simple"> <li>add something like <code class="docutils literal notranslate">Interpreter.run()</code> or <code class="docutils literal notranslate">Interpreter.call()</code> that calls <code class="docutils literal notranslate">interp.exec()</code> and falls back to pickle</li> <li>fall back to pickle in <code class="docutils literal notranslate">Interpreter.set_main_attrs()</code> and <code class="docutils literal notranslate">Interpreter.get_main_attr()</code></li> </ul> These would be easy to do if this proves to be a pain point. </section> <section id="avoid-possible-confusion-about-interpreters-running-in-the-current-thread"> <h3><a class="toc-backref" href="#avoid-possible-confusion-about-interpreters-running-in-the-current-thread" role="doc-backlink">Avoid possible confusion about interpreters running in the current thread</a></h3> One regular point of confusion has been that <code class="docutils literal notranslate">Interpreter.exec()</code> executes in the current OS thread, temporarily blocking the current Python thread. It may be worth doing something to avoid that confusion. Some possible solutions for this hypothetical problem: <ul class="simple"> <li>by default, run in a new thread?</li> <li>add <code class="docutils literal notranslate">Interpreter.exec_in_thread()</code>?</li> <li>add <code class="docutils literal notranslate">Interpreter.exec_in_current_thread()</code>?</li> </ul> In earlier versions of this PEP the method was <code class="docutils literal notranslate">interp.run()</code>. The simple change to <code class="docutils literal notranslate">interp.exec()</code> alone will probably reduce confusion sufficiently, when coupled with educating users via the docs. It it turns out to be a real problem, we can pursue one of the alternatives at that point. </section> <section id="clarify-running-vs-has-threads"> <h3><a class="toc-backref" href="#clarify-running-vs-has-threads" role="doc-backlink">Clarify “running” vs. “has threads”</a></h3> <code class="docutils literal notranslate">Interpreter.is_running()</code> refers specifically to whether or not <code class="docutils literal notranslate">Interpreter.exec()</code> (or similar) is running somewhere. It does not say anything about if the interpreter has any subthreads running. That information might be helpful. Some things we could do: <ul class="simple"> <li>rename <code class="docutils literal notranslate">Interpreter.is_running()</code> to <code class="docutils literal notranslate">Interpreter.is_running_main()</code></li> <li>add <code class="docutils literal notranslate">Interpreter.has_threads()</code>, to complement <code class="docutils literal notranslate">Interpreter.is_running()</code></li> <li>expand to <code class="docutils literal notranslate">Interpreter.is_running(main=True, threads=False)</code></li> </ul> None of these are urgent and any could be done later, if desired. </section> <section id="a-dunder-method-for-sharing"> <h3><a class="toc-backref" href="#a-dunder-method-for-sharing" role="doc-backlink">A Dunder Method For Sharing</a></h3> We could add a special method, like <code class="docutils literal notranslate">__xid__</code> to correspond to <code class="docutils literal notranslate">tp_xid</code>. At the very least, it would allow Python types to convert their instances to some other type that implements <code class="docutils literal notranslate">tp_xid</code>. The problem is that exposing this capability to Python code presents a degree of complixity that hasn’t been explored yet, nor is there a compelling case to investigate that complexity. </section> <section id="interpreter-call"> <h3><a class="toc-backref" href="#interpreter-call" role="doc-backlink">Interpreter.call()</a></h3> It would be convenient to run existing functions in subinterpreters directly. <code class="docutils literal notranslate">Interpreter.exec()</code> could be adjusted to support this or a <code class="docutils literal notranslate">call()</code> method could be added: <div class="highlight-default notranslate"><div class="highlight"><pre>Interpreter.call(f, *args, **kwargs) </pre></div> </div> This suffers from the same problem as sharing objects between interpreters via queues. The minimal solution (running a source string) is sufficient for us to get the feature out where it can be explored. </section> <section id="interpreter-run-in-thread"> <h3><a class="toc-backref" href="#interpreter-run-in-thread" role="doc-backlink">Interpreter.run_in_thread()</a></h3> This method would make a <code class="docutils literal notranslate">interp.exec()</code> call for you in a thread. Doing this using only <code class="docutils literal notranslate">threading.Thread</code> and <code class="docutils literal notranslate">interp.exec()</code> is relatively trivial so we’ve left it out. </section> <section id="synchronization-primitives"> <h3><a class="toc-backref" href="#synchronization-primitives" role="doc-backlink">Synchronization Primitives</a></h3> The <code class="docutils literal notranslate">threading</code> module provides a number of synchronization primitives for coordinating concurrent operations. This is especially necessary due to the shared-state nature of threading. In contrast, interpreters do not share state. Data sharing is restricted to the runtime’s shareable objects capability, which does away with the need for explicit synchronization. If any sort of opt-in shared state support is added to CPython’s interpreters in the future, that same effort can introduce synchronization primitives to meet that need. </section> <section id="csp-library"> <h3><a class="toc-backref" href="#csp-library" role="doc-backlink">CSP Library</a></h3> A <code class="docutils literal notranslate">csp</code> module would not be a large step away from the functionality provided by this PEP. However, adding such a module is outside the minimalist goals of this proposal. </section> <section id="syntactic-support"> <h3><a class="toc-backref" href="#syntactic-support" role="doc-backlink">Syntactic Support</a></h3> The <code class="docutils literal notranslate">Go</code> language provides a concurrency model based on CSP, so it’s similar to the concurrency model that multiple interpreters support. However, <code class="docutils literal notranslate">Go</code> also provides syntactic support, as well as several builtin concurrency primitives, to make concurrency a first-class feature. Conceivably, similar syntactic (and builtin) support could be added to Python using interpreters. However, that is way outside the scope of this PEP! </section> <section id="multiprocessing"> <h3><a class="toc-backref" href="#multiprocessing" role="doc-backlink">Multiprocessing</a></h3> The <code class="docutils literal notranslate">multiprocessing</code> module could support interpreters in the same way it supports threads and processes. In fact, the module’s maintainer, Davin Potts, has indicated this is a reasonable feature request. However, it is outside the narrow scope of this PEP. </section> <section id="c-extension-opt-in-opt-out"> <h3><a class="toc-backref" href="#c-extension-opt-in-opt-out" role="doc-backlink">C-extension opt-in/opt-out</a></h3> By using the <code class="docutils literal notranslate">PyModuleDef_Slot</code> introduced by <a class="pep reference internal" href="../pep-0489/" title="PEP 489 – Multi-phase extension module initialization">PEP 489</a>, we could easily add a mechanism by which C-extension modules could opt out of multiple interpreter support. Then the import machinery, when operating in a subinterpreter, would need to check the module for support. It would raise an ImportError if unsupported. Alternately we could support opting in to multiple interpreters support. However, that would probably exclude many more modules (unnecessarily) than the opt-out approach. Also, note that <a class="pep reference internal" href="../pep-0489/" title="PEP 489 – Multi-phase extension module initialization">PEP 489</a> defined that an extension’s use of the PEP’s machinery implies multiple interpreters support. The scope of adding the ModuleDef slot and fixing up the import machinery is non-trivial, but could be worth it. It all depends on how many extension modules break under subinterpreters. Given that there are relatively few cases we know of through mod_wsgi, we can leave this for later. </section> <section id="poisoning-channels"> <h3><a class="toc-backref" href="#poisoning-channels" role="doc-backlink">Poisoning channels</a></h3> CSP has the concept of poisoning a channel. Once a channel has been poisoned, any <code class="docutils literal notranslate">send()</code> or <code class="docutils literal notranslate">recv()</code> call on it would raise a special exception, effectively ending execution in the interpreter that tried to use the poisoned channel. This could be accomplished by adding a <code class="docutils literal notranslate">poison()</code> method to both ends of the channel. The <code class="docutils literal notranslate">close()</code> method can be used in this way (mostly), but these semantics are relatively specialized and can wait. </section> <section id="resetting-main"> <h3><a class="toc-backref" href="#resetting-main" role="doc-backlink">Resetting __main__</a></h3> As proposed, every call to <code class="docutils literal notranslate">Interpreter.exec()</code> will execute in the namespace of the interpreter’s existing <code class="docutils literal notranslate">__main__</code> module. This means that data persists there between <code class="docutils literal notranslate">interp.exec()</code> calls. Sometimes this isn’t desirable and you want to execute in a fresh <code class="docutils literal notranslate">__main__</code>. Also, you don’t necessarily want to leak objects there that you aren’t using any more. Note that the following won’t work right because it will clear too much (e.g. <code class="docutils literal notranslate">__name__</code> and the other “__dunder__” attributes: <div class="highlight-default notranslate"><div class="highlight"><pre>interp.exec('globals().clear()') </pre></div> </div> Possible solutions include: <ul class="simple"> <li>a <code class="docutils literal notranslate">create()</code> arg to indicate resetting <code class="docutils literal notranslate">__main__</code> after each <code class="docutils literal notranslate">interp.exec()</code> call</li> <li>an <code class="docutils literal notranslate">Interpreter.reset_main</code> flag to support opting in or out after the fact</li> <li>an <code class="docutils literal notranslate">Interpreter.reset_main()</code> method to opt in when desired</li> <li><code class="docutils literal notranslate">importlib.util.reset_globals()</code> <a class="reference internal" href="#reset-globals" id="id11">[reset_globals]</a></li> </ul> Also note that resetting <code class="docutils literal notranslate">__main__</code> does nothing about state stored in other modules. So any solution would have to be clear about the scope of what is being reset. Conceivably we could invent a mechanism by which any (or every) module could be reset, unlike <code class="docutils literal notranslate">reload()</code> which does not clear the module before loading into it. Regardless, since <code class="docutils literal notranslate">__main__</code> is the execution namespace of the interpreter, resetting it has a much more direct correlation to interpreters and their dynamic state than does resetting other modules. So a more generic module reset mechanism may prove unnecessary. This isn’t a critical feature initially. It can wait until later if desirable. </section> <section id="resetting-an-interpreter-s-state"> <h3><a class="toc-backref" href="#resetting-an-interpreter-s-state" role="doc-backlink">Resetting an interpreter’s state</a></h3> It may be nice to re-use an existing subinterpreter instead of spinning up a new one. Since an interpreter has substantially more state than just the <code class="docutils literal notranslate">__main__</code> module, it isn’t so easy to put an interpreter back into a pristine/fresh state. In fact, there may be parts of the state that cannot be reset from Python code. A possible solution is to add an <code class="docutils literal notranslate">Interpreter.reset()</code> method. This would put the interpreter back into the state it was in when newly created. If called on a running interpreter it would fail (hence the main interpreter could never be reset). This would likely be more efficient than creating a new interpreter, though that depends on what optimizations will be made later to interpreter creation. While this would potentially provide functionality that is not otherwise available from Python code, it isn’t a fundamental functionality. So in the spirit of minimalism here, this can wait. Regardless, I doubt it would be controversial to add it post-PEP. </section> <section id="copy-an-existing-interpreter-s-state"> <h3><a class="toc-backref" href="#copy-an-existing-interpreter-s-state" role="doc-backlink">Copy an existing interpreter’s state</a></h3> Relatedly, it may be useful to support creating a new interpreter based on an existing one, e.g. <code class="docutils literal notranslate">Interpreter.copy()</code>. This ties into the idea that a snapshot could be made of an interpreter’s memory, which would make starting up CPython, or creating new interpreters, faster in general. The same mechanism could be used for a hypothetical <code class="docutils literal notranslate">Interpreter.reset()</code>, as described previously. </section> <section id="shareable-file-descriptors-and-sockets"> <h3><a class="toc-backref" href="#shareable-file-descriptors-and-sockets" role="doc-backlink">Shareable file descriptors and sockets</a></h3> Given that file descriptors and sockets are process-global resources, making them shareable is a reasonable idea. They would be a good candidate for the first effort at expanding the supported shareable types. They aren’t strictly necessary for the initial API. </section> <section id="integration-with-async"> <h3><a class="toc-backref" href="#integration-with-async" role="doc-backlink">Integration with async</a></h3> Per Antoine Pitrou <a class="reference internal" href="#async" id="id12">[async]</a>: <div class="highlight-default notranslate"><div class="highlight"><pre>Has any thought been given to how FIFOs could integrate with async code driven by an event loop (e.g. asyncio)? I think the model of executing several asyncio (or Tornado) applications each in their own subinterpreter may prove quite interesting to reconcile multi- core concurrency with ease of programming. That would require the FIFOs to be able to synchronize on something an event loop can wait on (probably a file descriptor?). </pre></div> </div> The basic functionality of multiple interpreters support does not depend on async and can be added later. A possible solution is to provide async implementations of the blocking channel methods (<code class="docutils literal notranslate">recv()</code>, and <code class="docutils literal notranslate">send()</code>). Alternately, “readiness callbacks” could be used to simplify use in async scenarios. This would mean adding an optional <code class="docutils literal notranslate">callback</code> (kw-only) parameter to the <code class="docutils literal notranslate">recv_nowait()</code> and <code class="docutils literal notranslate">send_nowait()</code> channel methods. The callback would be called once the object was sent or received (respectively). (Note that making channels buffered makes readiness callbacks less important.) </section> <section id="support-for-iteration"> <h3><a class="toc-backref" href="#support-for-iteration" role="doc-backlink">Support for iteration</a></h3> Supporting iteration on <code class="docutils literal notranslate">RecvChannel</code> (via <code class="docutils literal notranslate">__iter__()</code> or <code class="docutils literal notranslate">_next__()</code>) may be useful. A trivial implementation would use the <code class="docutils literal notranslate">recv()</code> method, similar to how files do iteration. Since this isn’t a fundamental capability and has a simple analog, adding iteration support can wait until later. </section> <section id="channel-context-managers"> <h3><a class="toc-backref" href="#channel-context-managers" role="doc-backlink">Channel context managers</a></h3> Context manager support on <code class="docutils literal notranslate">RecvChannel</code> and <code class="docutils literal notranslate">SendChannel</code> may be helpful. The implementation would be simple, wrapping a call to <code class="docutils literal notranslate">close()</code> (or maybe <code class="docutils literal notranslate">release()</code>) like files do. As with iteration, this can wait. </section> <section id="pipes-and-queues"> <h3><a class="toc-backref" href="#pipes-and-queues" role="doc-backlink">Pipes and Queues</a></h3> With the proposed object passing mechanism of “os.pipe()”, other similar basic types aren’t strictly required to achieve the minimal useful functionality of multiple interpreters. Such types include pipes (like unbuffered channels, but one-to-one) and queues (like channels, but more generic). See below in <a class="reference internal" href="#rejected-ideas">Rejected Ideas</a> for more information. Even though these types aren’t part of this proposal, they may still be useful in the context of concurrency. Adding them later is entirely reasonable. The could be trivially implemented as wrappers around channels. Alternatively they could be implemented for efficiency at the same low level as channels. </section> <section id="return-a-lock-from-send"> <h3><a class="toc-backref" href="#return-a-lock-from-send" role="doc-backlink">Return a lock from send()</a></h3> When sending an object through a channel, you don’t have a way of knowing when the object gets received on the other end. One way to work around this is to return a locked <code class="docutils literal notranslate">threading.Lock</code> from <code class="docutils literal notranslate">SendChannel.send()</code> that unlocks once the object is received. Alternately, the proposed <code class="docutils literal notranslate">SendChannel.send()</code> (blocking) and <code class="docutils literal notranslate">SendChannel.send_nowait()</code> provide an explicit distinction that is less likely to confuse users. Note that returning a lock would matter for buffered channels (i.e. queues). For unbuffered channels it is a non-issue. </section> <section id="support-prioritization-in-channels"> <h3><a class="toc-backref" href="#support-prioritization-in-channels" role="doc-backlink">Support prioritization in channels</a></h3> A simple example is <code class="docutils literal notranslate">queue.PriorityQueue</code> in the stdlib. </section> <section id="support-inheriting-settings-and-more"> <h3><a class="toc-backref" href="#support-inheriting-settings-and-more" role="doc-backlink">Support inheriting settings (and more?)</a></h3> Folks might find it useful, when creating a new interpreter, to be able to indicate that they would like some things “inherited” by the new interpreter. The mechanism could be a strict copy or it could be copy-on-write. The motivating example is with the warnings module (e.g. copy the filters). The feature isn’t critical, nor would it be widely useful, so it can wait until there’s interest. Notably, both suggested solutions will require significant work, especially when it comes to complex objects and most especially for mutable containers of mutable complex objects. </section> <section id="make-exceptions-shareable"> <h3><a class="toc-backref" href="#make-exceptions-shareable" role="doc-backlink">Make exceptions shareable</a></h3> Exceptions are propagated out of <code class="docutils literal notranslate">run()</code> calls, so it isn’t a big leap to make them shareable. However, as noted elsewhere, it isn’t essential or (particularly common) so we can wait on doing that. </section> <section id="make-everything-shareable-through-serialization"> <h3><a class="toc-backref" href="#make-everything-shareable-through-serialization" role="doc-backlink">Make everything shareable through serialization</a></h3> We could use pickle (or marshal) to serialize everything and thus make them shareable. Doing this is potentially inefficient, but it may be a matter of convenience in the end. We can add it later, but trying to remove it later would be significantly more painful. </section> <section id="make-runfailederror-cause-lazy"> <h3><a class="toc-backref" href="#make-runfailederror-cause-lazy" role="doc-backlink">Make RunFailedError.__cause__ lazy</a></h3> An uncaught exception in a subinterpreter (from <code class="docutils literal notranslate">interp.exec()</code>) is copied to the calling interpreter and set as <code class="docutils literal notranslate">__cause__</code> on a <code class="docutils literal notranslate">RunFailedError</code> which is then raised. That copying part involves some sort of deserialization in the calling interpreter, which can be expensive (e.g. due to imports) yet is not always necessary. So it may be useful to use an <code class="docutils literal notranslate">ExceptionProxy</code> type to wrap the serialized exception and only deserialize it when needed. That could be via <code class="docutils literal notranslate">ExceptionProxy__getattribute__()</code> or perhaps through <code class="docutils literal notranslate">RunFailedError.resolve()</code> (which would raise the deserialized exception and set <code class="docutils literal notranslate">RunFailedError.__cause__</code> to the exception. It may also make sense to have <code class="docutils literal notranslate">RunFailedError.__cause__</code> be a descriptor that does the lazy deserialization (and set <code class="docutils literal notranslate">__cause__</code>) on the <code class="docutils literal notranslate">RunFailedError</code> instance. </section> <section id="return-a-value-from-interp-exec"> <h3><a class="toc-backref" href="#return-a-value-from-interp-exec" role="doc-backlink">Return a value from <code class="docutils literal notranslate">interp.exec()</code></a></h3> Currently <code class="docutils literal notranslate">interp.exec()</code> always returns None. One idea is to return the return value from whatever the subinterpreter ran. However, for now it doesn’t make sense. The only thing folks can run is a string of code (i.e. a script). This is equivalent to <code class="docutils literal notranslate">PyRun_StringFlags()</code>, <code class="docutils literal notranslate">exec()</code>, or a module body. None of those “return” anything. We can revisit this once <code class="docutils literal notranslate">interp.exec()</code> supports functions, etc. </section> <section id="add-a-shareable-synchronization-primitive"> <h3><a class="toc-backref" href="#add-a-shareable-synchronization-primitive" role="doc-backlink">Add a shareable synchronization primitive</a></h3> This would be <code class="docutils literal notranslate">_threading.Lock</code> (or something like it) where interpreters would actually share the underlying mutex. The main concern is that locks and isolated interpreters may not mix well (as learned in Go). We can add this later if it proves desirable without much trouble. </section> <section id="propagate-systemexit-and-keyboardinterrupt-differently"> <h3><a class="toc-backref" href="#propagate-systemexit-and-keyboardinterrupt-differently" role="doc-backlink">Propagate SystemExit and KeyboardInterrupt Differently</a></h3> The exception types that inherit from <code class="docutils literal notranslate">BaseException</code> (aside from <code class="docutils literal notranslate">Exception</code>) are usually treated specially. These types are: <code class="docutils literal notranslate">KeyboardInterrupt</code>, <code class="docutils literal notranslate">SystemExit</code>, and <code class="docutils literal notranslate">GeneratorExit</code>. It may make sense to treat them specially when it comes to propagation from <code class="docutils literal notranslate">interp.exec()</code>. Here are some options: <div class="highlight-default notranslate"><div class="highlight"><pre>* propagate like normal via RunFailedError * do not propagate (handle them somehow in the subinterpreter) * propagate them directly (avoid RunFailedError) * propagate them directly (set RunFailedError as __cause__) </pre></div> </div> We aren’t going to worry about handling them differently. Threads already ignore <code class="docutils literal notranslate">SystemExit</code>, so for now we will follow that pattern. </section> <section id="add-an-explicit-release-and-close-to-channel-end-classes"> <h3><a class="toc-backref" href="#add-an-explicit-release-and-close-to-channel-end-classes" role="doc-backlink">Add an explicit release() and close() to channel end classes</a></h3> It can be convenient to have an explicit way to close a channel against further global use. Likewise it could be useful to have an explicit way to release one of the channel ends relative to the current interpreter. Among other reasons, such a mechanism is useful for communicating overall state between interpreters without the extra boilerplate that passing objects through a channel directly would require. The challenge is getting automatic release/close right without making it hard to understand. This is especially true when dealing with a non-empty channel. We should be able to get by without release/close for now. </section> <section id="add-sendchannel-send-buffer"> <h3><a class="toc-backref" href="#add-sendchannel-send-buffer" role="doc-backlink">Add SendChannel.send_buffer()</a></h3> This method would allow no-copy sending of an object through a channel if it supports the <a class="pep reference internal" href="../pep-3118/" title="PEP 3118 – Revising the buffer protocol">PEP 3118</a> buffer protocol (e.g. memoryview). Support for this is not fundamental to channels and can be added on later without much disruption. </section> <section id="auto-run-in-a-thread"> <h3><a class="toc-backref" href="#auto-run-in-a-thread" role="doc-backlink">Auto-run in a thread</a></h3> The PEP proposes a hard separation between subinterpreters and threads: if you want to run in a thread you must create the thread yourself and call <code class="docutils literal notranslate">interp.exec()</code> in it. However, it might be convenient if <code class="docutils literal notranslate">interp.exec()</code> could do that for you, meaning there would be less boilerplate. Furthermore, we anticipate that users will want to run in a thread much more often than not. So it would make sense to make this the default behavior. We would add a kw-only param “threaded” (default <code class="docutils literal notranslate">True</code>) to <code class="docutils literal notranslate">interp.exec()</code> to allow the run-in-the-current-thread operation. </section> </section> <section id="rejected-ideas"> <h2><a class="toc-backref" href="#rejected-ideas" role="doc-backlink">Rejected Ideas</a></h2> <section id="explicit-channel-association"> <h3><a class="toc-backref" href="#explicit-channel-association" role="doc-backlink">Explicit channel association</a></h3> Interpreters are implicitly associated with channels upon <code class="docutils literal notranslate">recv()</code> and <code class="docutils literal notranslate">send()</code> calls. They are de-associated with <code class="docutils literal notranslate">release()</code> calls. The alternative would be explicit methods. It would be either <code class="docutils literal notranslate">add_channel()</code> and <code class="docutils literal notranslate">remove_channel()</code> methods on <code class="docutils literal notranslate">Interpreter</code> objects or something similar on channel objects. In practice, this level of management shouldn’t be necessary for users. So adding more explicit support would only add clutter to the API. </section> <section id="add-an-api-based-on-pipes"> <h3><a class="toc-backref" href="#add-an-api-based-on-pipes" role="doc-backlink">Add an API based on pipes</a></h3> A pipe would be a simplex FIFO between exactly two interpreters. For most use cases this would be sufficient. It could potentially simplify the implementation as well. However, it isn’t a big step to supporting a many-to-many simplex FIFO via channels. Also, with pipes the API ends up being slightly more complicated, requiring naming the pipes. </section> <section id="add-an-api-based-on-queues"> <h3><a class="toc-backref" href="#add-an-api-based-on-queues" role="doc-backlink">Add an API based on queues</a></h3> Queues and buffered channels are almost the same thing. The main difference is that channels have a stronger relationship with context (i.e. the associated interpreter). The name “Channel” was used instead of “Queue” to avoid confusion with the stdlib <code class="docutils literal notranslate">queue.Queue</code>. </section> <section id="enumerate"> <h3><a class="toc-backref" href="#enumerate" role="doc-backlink">“enumerate”</a></h3> The <code class="docutils literal notranslate">list_all()</code> function provides the list of all interpreters. In the threading module, which partly inspired the proposed API, the function is called <code class="docutils literal notranslate">enumerate()</code>. The name is different here to avoid confusing Python users that are not already familiar with the threading API. For them “enumerate” is rather unclear, whereas “list_all” is clear. </section> <section id="alternate-solutions-to-prevent-leaking-exceptions-across-interpreters"> <h3><a class="toc-backref" href="#alternate-solutions-to-prevent-leaking-exceptions-across-interpreters" role="doc-backlink">Alternate solutions to prevent leaking exceptions across interpreters</a></h3> In function calls, uncaught exceptions propagate to the calling frame. The same approach could be taken with <code class="docutils literal notranslate">interp.exec()</code>. However, this would mean that exception objects would leak across the inter-interpreter boundary. Likewise, the frames in the traceback would potentially leak. While that might not be a problem currently, it would be a problem once interpreters get better isolation relative to memory management (which is necessary to stop sharing the GIL between interpreters). We’ve resolved the semantics of how the exceptions propagate by raising a <code class="docutils literal notranslate">RunFailedError</code> instead, for which <code class="docutils literal notranslate">__cause__</code> wraps a safe proxy for the original exception and traceback. Rejected possible solutions: <ul class="simple"> <li>reproduce the exception and traceback in the original interpreter and raise that.</li> <li>raise a subclass of RunFailedError that proxies the original exception and traceback.</li> <li>raise RuntimeError instead of RunFailedError</li> <li>convert at the boundary (a la <code class="docutils literal notranslate">subprocess.CalledProcessError</code>) (requires a cross-interpreter representation)</li> <li>support customization via <code class="docutils literal notranslate">Interpreter.excepthook</code> (requires a cross-interpreter representation)</li> <li>wrap in a proxy at the boundary (including with support for something like <code class="docutils literal notranslate">err.raise()</code> to propagate the traceback).</li> <li>return the exception (or its proxy) from <code class="docutils literal notranslate">interp.exec()</code> instead of raising it</li> <li>return a result object (like <code class="docutils literal notranslate">subprocess</code> does) <a class="reference internal" href="#result-object" id="id13">[result-object]</a> (unnecessary complexity?)</li> <li>throw the exception away and expect users to deal with unhandled exceptions explicitly in the script they pass to <code class="docutils literal notranslate">interp.exec()</code> (they can pass error info out via channels); with threads you have to do something similar</li> </ul> </section> <section id="always-associate-each-new-interpreter-with-its-own-thread"> <h3><a class="toc-backref" href="#always-associate-each-new-interpreter-with-its-own-thread" role="doc-backlink">Always associate each new interpreter with its own thread</a></h3> As implemented in the C-API, an interpreter is not inherently tied to any thread. Furthermore, it will run in any existing thread, whether created by Python or not. You only have to activate one of its thread states (<code class="docutils literal notranslate">PyThreadState</code>) in the thread first. This means that the same thread may run more than one interpreter (though obviously not at the same time). The proposed module maintains this behavior. Interpreters are not tied to threads. Only calls to <code class="docutils literal notranslate">Interpreter.exec()</code> are. However, one of the key objectives of this PEP is to provide a more human-centric concurrency model. With that in mind, from a conceptual standpoint the module might be easier to understand if each interpreter were associated with its own thread. That would mean <code class="docutils literal notranslate">interpreters.create()</code> would create a new thread and <code class="docutils literal notranslate">Interpreter.exec()</code> would only execute in that thread (and nothing else would). The benefit is that users would not have to wrap <code class="docutils literal notranslate">Interpreter.exec()</code> calls in a new <code class="docutils literal notranslate">threading.Thread</code>. Nor would they be in a position to accidentally pause the current interpreter (in the current thread) while their interpreter executes. The idea is rejected because the benefit is small and the cost is high. The difference from the capability in the C-API would be potentially confusing. The implicit creation of threads is magical. The early creation of threads is potentially wasteful. The inability to run arbitrary interpreters in an existing thread would prevent some valid use cases, frustrating users. Tying interpreters to threads would require extra runtime modifications. It would also make the module’s implementation overly complicated. Finally, it might not even make the module easier to understand. </section> <section id="only-associate-interpreters-upon-use"> <h3><a class="toc-backref" href="#only-associate-interpreters-upon-use" role="doc-backlink">Only associate interpreters upon use</a></h3> Associate interpreters with channel ends only once <code class="docutils literal notranslate">recv()</code>, <code class="docutils literal notranslate">send()</code>, etc. are called. Doing this is potentially confusing and also can lead to unexpected races where a channel is auto-closed before it can be used in the original (creating) interpreter. </section> <section id="allow-multiple-simultaneous-calls-to-interpreter-exec"> <h3><a class="toc-backref" href="#allow-multiple-simultaneous-calls-to-interpreter-exec" role="doc-backlink">Allow multiple simultaneous calls to Interpreter.exec()</a></h3> This would make sense especially if <code class="docutils literal notranslate">Interpreter.exec()</code> were to manage new threads for you (which we’ve rejected). Essentially, each call would run independently, which would be mostly fine from a narrow technical standpoint, since each interpreter can have multiple threads. The problem is that the interpreter has only one <code class="docutils literal notranslate">__main__</code> module and simultaneous <code class="docutils literal notranslate">Interpreter.exec()</code> calls would have to sort out sharing <code class="docutils literal notranslate">__main__</code> or we’d have to invent a new mechanism. Neither would be simple enough to be worth doing. </section> <section id="add-a-reraise-method-to-runfailederror"> <h3><a class="toc-backref" href="#add-a-reraise-method-to-runfailederror" role="doc-backlink">Add a “reraise” method to RunFailedError</a></h3> While having <code class="docutils literal notranslate">__cause__</code> set on <code class="docutils literal notranslate">RunFailedError</code> helps produce a more useful traceback, it’s less helpful when handling the original error. To help facilitate this, we could add <code class="docutils literal notranslate">RunFailedError.reraise()</code>. This method would enable the following pattern: <div class="highlight-default notranslate"><div class="highlight"><pre>try: try: interp.exec(script) except RunFailedError as exc: exc.reraise() except MyException: ... </pre></div> </div> This would be made even simpler if there existed a <code class="docutils literal notranslate">__reraise__</code> protocol. All that said, this is completely unnecessary. Using <code class="docutils literal notranslate">__cause__</code> is good enough: <div class="highlight-default notranslate"><div class="highlight"><pre>try: try: interp.exec(script) except RunFailedError as exc: raise exc.__cause__ except MyException: ... </pre></div> </div> Note that in extreme cases it may require a little extra boilerplate: <div class="highlight-default notranslate"><div class="highlight"><pre>try: try: interp.exec(script) except RunFailedError as exc: if exc.__cause__ is not None: raise exc.__cause__ raise # re-raise except MyException: ... </pre></div> </div> </section> </section> <section id="implementation"> <h2><a class="toc-backref" href="#implementation" role="doc-backlink">Implementation</a></h2> The implementation of the PEP has 4 parts: <ul class="simple"> <li>the high-level module described in this PEP (mostly a light wrapper around a low-level C extension</li> <li>the low-level C extension module</li> <li>additions to the internal C-API needed by the low-level module</li> <li>secondary fixes/changes in the CPython runtime that facilitate the low-level module (among other benefits)</li> </ul> These are at various levels of completion, with more done the lower you go: <ul class="simple"> <li>the high-level module has been, at best, roughly implemented. However, fully implementing it will be almost trivial.</li> <li>the low-level module is mostly complete. The bulk of the implementation was merged into master in December 2018 as the “_xxsubinterpreters” module (for the sake of testing multiple interpreters functionality). Only the exception propagation implementation remains to be finished, which will not require extensive work.</li> <li>all necessary C-API work has been finished</li> <li>all anticipated work in the runtime has been finished</li> </ul> The implementation effort for <a class="pep reference internal" href="../pep-0554/" title="PEP 554 – Multiple Interpreters in the Stdlib">PEP 554</a> is being tracked as part of a larger project aimed at improving multi-core support in CPython. <a class="reference internal" href="#multi-core-project" id="id14">[multi-core-project]</a> </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <div role="list" class="citation-list"> <div class="citation" id="c-api" role="doc-biblioentry"> <dt class="label" id="c-api">[<a href="#id1">c-api</a>]</dt> <dd><a class="reference external" href="https://docs.python.org/3/c-api/init.html#sub-interpreter-support">https://docs.python.org/3/c-api/init.html#sub-interpreter-support</a></div> <div class="citation" id="csp" role="doc-biblioentry"> <dt class="label" id="csp">[<a href="#id5">CSP</a>]</dt> <dd><a class="reference external" href="https://en.wikipedia.org/wiki/Communicating_sequential_processes">https://en.wikipedia.org/wiki/Communicating_sequential_processes</a> <a class="reference external" href="https://github.com/futurecore/python-csp">https://github.com/futurecore/python-csp</a></div> <div class="citation" id="fifo" role="doc-biblioentry"> <dt class="label" id="fifo">[<a href="#id10">fifo</a>]</dt> <dd><a class="reference external" href="https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Pipe">https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Pipe</a> <a class="reference external" href="https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue">https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue</a> <a class="reference external" href="https://docs.python.org/3/library/queue.html#module-queue">https://docs.python.org/3/library/queue.html#module-queue</a> <a class="reference external" href="http://stackless.readthedocs.io/en/2.7-slp/library/stackless/channels.html">http://stackless.readthedocs.io/en/2.7-slp/library/stackless/channels.html</a> <a class="reference external" href="https://golang.org/doc/effective_go.html#sharing">https://golang.org/doc/effective_go.html#sharing</a> <a class="reference external" href="http://www.jtolds.com/writing/2016/03/go-channels-are-bad-and-you-should-feel-bad/">http://www.jtolds.com/writing/2016/03/go-channels-are-bad-and-you-should-feel-bad/</a></div> <div class="citation" id="caveats" role="doc-biblioentry"> <dt class="label" id="caveats">[<a href="#id6">caveats</a>]</dt> <dd><a class="reference external" href="https://docs.python.org/3/c-api/init.html#bugs-and-caveats">https://docs.python.org/3/c-api/init.html#bugs-and-caveats</a></div> <div class="citation" id="cryptography" role="doc-biblioentry"> <dt class="label" id="cryptography">[<a href="#id7">cryptography</a>]</dt> <dd><a class="reference external" href="https://github.com/pyca/cryptography/issues/2299">https://github.com/pyca/cryptography/issues/2299</a></div> <div class="citation" id="gilstate" role="doc-biblioentry"> <dt class="label" id="gilstate">[<a href="#id8">gilstate</a>]</dt> <dd><a class="reference external" href="https://bugs.python.org/issue10915">https://bugs.python.org/issue10915</a> <a class="reference external" href="http://bugs.python.org/issue15751">http://bugs.python.org/issue15751</a></div> <div class="citation" id="bug-rate" role="doc-biblioentry"> <dt class="label" id="bug-rate">[<a href="#id3">bug-rate</a>]</dt> <dd><a class="reference external" href="https://mail.python.org/pipermail/python-ideas/2017-September/047094.html">https://mail.python.org/pipermail/python-ideas/2017-September/047094.html</a></div> <div class="citation" id="benefits" role="doc-biblioentry"> <dt class="label" id="benefits">[<a href="#id2">benefits</a>]</dt> <dd><a class="reference external" href="https://mail.python.org/pipermail/python-ideas/2017-September/047122.html">https://mail.python.org/pipermail/python-ideas/2017-September/047122.html</a></div> <div class="citation" id="reset-globals" role="doc-biblioentry"> <dt class="label" id="reset-globals">[<a href="#id11">reset_globals</a>]</dt> <dd><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2017-September/149545.html">https://mail.python.org/pipermail/python-dev/2017-September/149545.html</a></div> <div class="citation" id="async" role="doc-biblioentry"> <dt class="label" id="async">[<a href="#id12">async</a>]</dt> <dd><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2017-September/149420.html">https://mail.python.org/pipermail/python-dev/2017-September/149420.html</a> <a class="reference external" href="https://mail.python.org/pipermail/python-dev/2017-September/149585.html">https://mail.python.org/pipermail/python-dev/2017-September/149585.html</a></div> <div class="citation" id="result-object" role="doc-biblioentry"> <dt class="label" id="result-object">[<a href="#id13">result-object</a>]</dt> <dd><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2017-September/149562.html">https://mail.python.org/pipermail/python-dev/2017-September/149562.html</a></div> <div class="citation" id="jython" role="doc-biblioentry"> <dt class="label" id="jython">[<a href="#id9">jython</a>]</dt> <dd><a class="reference external" href="https://mail.python.org/pipermail/python-ideas/2017-May/045771.html">https://mail.python.org/pipermail/python-ideas/2017-May/045771.html</a></div> <div class="citation" id="multi-core-project" role="doc-biblioentry"> <dt class="label" id="multi-core-project">[<a href="#id14">multi-core-project</a>]</dt> <dd><a class="reference external" href="https://github.com/ericsnowcurrently/multi-core-python">https://github.com/ericsnowcurrently/multi-core-python</a></div> <div class="citation" id="cache-line-ping-pong" role="doc-biblioentry"> <dt class="label" id="cache-line-ping-pong">[<a href="#id4">cache-line-ping-pong</a>]</dt> <dd><a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/message/3HVRFWHDMWPNR367GXBILZ4JJAUQ2STZ/">https://mail.python.org/archives/list/python-dev@python.org/message/3HVRFWHDMWPNR367GXBILZ4JJAUQ2STZ/</a></div> </div> <ul class="simple"> <li><dl class="simple"> <dt>mp-conn</dt><dd><a class="reference external" href="https://docs.python.org/3/library/multiprocessing.html#connection-objects">https://docs.python.org/3/library/multiprocessing.html#connection-objects</a></dd> </dl> </li> <li><dl class="simple"> <dt>main-thread</dt><dd><a class="reference external" href="https://mail.python.org/pipermail/python-ideas/2017-September/047144.html">https://mail.python.org/pipermail/python-ideas/2017-September/047144.html</a> <a class="reference external" href="https://mail.python.org/pipermail/python-dev/2017-September/149566.html">https://mail.python.org/pipermail/python-dev/2017-September/149566.html</a></dd> </dl> </li> <li><dl class="simple"> <dt>petr-c-ext</dt><dd><a class="reference external" href="https://mail.python.org/pipermail/import-sig/2016-June/001062.html">https://mail.python.org/pipermail/import-sig/2016-June/001062.html</a> <a class="reference external" href="https://mail.python.org/pipermail/python-ideas/2016-April/039748.html">https://mail.python.org/pipermail/python-ideas/2016-April/039748.html</a></dd> </dl> </li> </ul> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0554.rst">https://github.com/python/peps/blob/main/peps/pep-0554.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0554.rst">2025-02-01 08:55:40 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#proposal">Proposal</a><ul> <li><a class="reference internal" href="#the-interpreters-module">The “interpreters” Module</a></li> <li><a class="reference internal" href="#api-summary-for-interpreters-module">API summary for interpreters module</a></li> <li><a class="reference internal" href="#concurrent-futures-interpreterpoolexecutor">concurrent.futures.InterpreterPoolExecutor</a></li> <li><a class="reference internal" href="#help-for-extension-module-maintainers">Help for Extension Module Maintainers</a></li> </ul> </li> <li><a class="reference internal" href="#examples">Examples</a><ul> <li><a class="reference internal" href="#run-isolated-code-in-current-os-thread">Run isolated code in current OS thread</a></li> <li><a class="reference internal" href="#run-in-a-different-thread">Run in a different thread</a></li> <li><a class="reference internal" href="#pre-populate-an-interpreter">Pre-populate an interpreter</a></li> <li><a class="reference internal" href="#handling-an-exception">Handling an exception</a></li> <li><a class="reference internal" href="#re-raising-an-exception">Re-raising an exception</a></li> <li><a class="reference internal" href="#interact-with-the-main-namespace">Interact with the __main__ namespace</a></li> <li><a class="reference internal" href="#synchronize-using-an-os-pipe">Synchronize using an OS pipe</a></li> <li><a class="reference internal" href="#sharing-a-file-descriptor">Sharing a file descriptor</a></li> <li><a class="reference internal" href="#passing-objects-via-pickle">Passing objects via pickle</a></li> <li><a class="reference internal" href="#capturing-an-interpreter-s-stdout">Capturing an interpreter’s stdout</a></li> <li><a class="reference internal" href="#running-a-module">Running a module</a></li> <li><a class="reference internal" href="#running-as-script-including-zip-archives-directories">Running as script (including zip archives & directories)</a></li> <li><a class="reference internal" href="#using-a-channel-to-communicate">Using a channel to communicate</a></li> <li><a class="reference internal" href="#sharing-a-memoryview-imagine-map-reduce">Sharing a memoryview (imagine map-reduce)</a></li> </ul> </li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#concerns">Concerns</a></li> </ul> </li> <li><a class="reference internal" href="#about-subinterpreters">About Subinterpreters</a><ul> <li><a class="reference internal" href="#concurrency">Concurrency</a></li> <li><a class="reference internal" href="#shared-data">Shared Data</a></li> <li><a class="reference internal" href="#interpreter-isolation">Interpreter Isolation</a></li> <li><a class="reference internal" href="#existing-usage">Existing Usage</a></li> </ul> </li> <li><a class="reference internal" href="#alternate-python-implementations">Alternate Python Implementations</a></li> <li><a class="reference internal" href="#interpreters-module-api">“interpreters” Module API</a><ul> <li><a class="reference internal" href="#uncaught-exceptions">Uncaught Exceptions</a></li> </ul> </li> <li><a class="reference internal" href="#interpreter-restrictions">Interpreter Restrictions</a></li> <li><a class="reference internal" href="#api-for-communication">API For Communication</a><ul> <li><a class="reference internal" href="#shareable-types">Shareable Types</a><ul> <li><a class="reference internal" href="#communicating-through-os-pipes">Communicating Through OS Pipes</a></li> </ul> </li> <li><a class="reference internal" href="#channels">Channels</a></li> <li><a class="reference internal" href="#caveats-for-shared-objects">Caveats For Shared Objects</a></li> </ul> </li> <li><a class="reference internal" href="#documentation">Documentation</a></li> <li><a class="reference internal" href="#alternative-solutions">Alternative Solutions</a></li> <li><a class="reference internal" href="#open-questions">Open Questions</a></li> <li><a class="reference internal" href="#deferred-functionality">Deferred Functionality</a><ul> <li><a class="reference internal" href="#add-convenience-api">Add convenience API</a></li> <li><a class="reference internal" href="#avoid-possible-confusion-about-interpreters-running-in-the-current-thread">Avoid possible confusion about interpreters running in the current thread</a></li> <li><a class="reference internal" href="#clarify-running-vs-has-threads">Clarify “running” vs. “has threads”</a></li> <li><a class="reference internal" href="#a-dunder-method-for-sharing">A Dunder Method For Sharing</a></li> <li><a class="reference internal" href="#interpreter-call">Interpreter.call()</a></li> <li><a class="reference internal" href="#interpreter-run-in-thread">Interpreter.run_in_thread()</a></li> <li><a class="reference internal" href="#synchronization-primitives">Synchronization Primitives</a></li> <li><a class="reference internal" href="#csp-library">CSP Library</a></li> <li><a class="reference internal" href="#syntactic-support">Syntactic Support</a></li> <li><a class="reference internal" href="#multiprocessing">Multiprocessing</a></li> <li><a class="reference internal" href="#c-extension-opt-in-opt-out">C-extension opt-in/opt-out</a></li> <li><a class="reference internal" href="#poisoning-channels">Poisoning channels</a></li> <li><a class="reference internal" href="#resetting-main">Resetting __main__</a></li> <li><a class="reference internal" href="#resetting-an-interpreter-s-state">Resetting an interpreter’s state</a></li> <li><a class="reference internal" href="#copy-an-existing-interpreter-s-state">Copy an existing interpreter’s state</a></li> <li><a class="reference internal" href="#shareable-file-descriptors-and-sockets">Shareable file descriptors and sockets</a></li> <li><a class="reference internal" href="#integration-with-async">Integration with async</a></li> <li><a class="reference internal" href="#support-for-iteration">Support for iteration</a></li> <li><a class="reference internal" href="#channel-context-managers">Channel context managers</a></li> <li><a class="reference internal" href="#pipes-and-queues">Pipes and Queues</a></li> <li><a class="reference internal" href="#return-a-lock-from-send">Return a lock from send()</a></li> <li><a class="reference internal" href="#support-prioritization-in-channels">Support prioritization in channels</a></li> <li><a class="reference internal" href="#support-inheriting-settings-and-more">Support inheriting settings (and more?)</a></li> <li><a class="reference internal" href="#make-exceptions-shareable">Make exceptions shareable</a></li> <li><a class="reference internal" href="#make-everything-shareable-through-serialization">Make everything shareable through serialization</a></li> <li><a class="reference internal" href="#make-runfailederror-cause-lazy">Make RunFailedError.__cause__ lazy</a></li> <li><a class="reference internal" href="#return-a-value-from-interp-exec">Return a value from <code class="docutils literal notranslate">interp.exec()</code></a></li> <li><a class="reference internal" href="#add-a-shareable-synchronization-primitive">Add a shareable synchronization primitive</a></li> <li><a class="reference internal" href="#propagate-systemexit-and-keyboardinterrupt-differently">Propagate SystemExit and KeyboardInterrupt Differently</a></li> <li><a class="reference internal" href="#add-an-explicit-release-and-close-to-channel-end-classes">Add an explicit release() and close() to channel end classes</a></li> <li><a class="reference internal" href="#add-sendchannel-send-buffer">Add SendChannel.send_buffer()</a></li> <li><a class="reference internal" href="#auto-run-in-a-thread">Auto-run in a thread</a></li> </ul> </li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a><ul> <li><a class="reference internal" href="#explicit-channel-association">Explicit channel association</a></li> <li><a class="reference internal" href="#add-an-api-based-on-pipes">Add an API based on pipes</a></li> <li><a class="reference internal" href="#add-an-api-based-on-queues">Add an API based on queues</a></li> <li><a class="reference internal" href="#enumerate">“enumerate”</a></li> <li><a class="reference internal" href="#alternate-solutions-to-prevent-leaking-exceptions-across-interpreters">Alternate solutions to prevent leaking exceptions across interpreters</a></li> <li><a class="reference internal" href="#always-associate-each-new-interpreter-with-its-own-thread">Always associate each new interpreter with its own thread</a></li> <li><a class="reference internal" href="#only-associate-interpreters-upon-use">Only associate interpreters upon use</a></li> <li><a class="reference internal" href="#allow-multiple-simultaneous-calls-to-interpreter-exec">Allow multiple simultaneous calls to Interpreter.exec()</a></li> <li><a class="reference internal" href="#add-a-reraise-method-to-runfailederror">Add a “reraise” method to RunFailedError</a></li> </ul> </li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0554.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>