CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 651 – Robust Stack Overflow Handling | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0651/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 651 – Robust Stack Overflow Handling | peps.python.org'> <meta property="og:description" content="This PEP proposes that Python should treat machine stack overflow differently from runaway recursion."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0651/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This PEP proposes that Python should treat machine stack overflow differently from runaway recursion."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 651</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> <span class="visually-hidden">Toggle light / dark / auto colour theme</span> </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 651 – Robust Stack Overflow Handling</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author<span class="colon">:</span></dt> <dd class="field-odd">Mark Shannon <mark at hotpy.org></dd> <dt class="field-even">Status<span class="colon">:</span></dt> <dd class="field-even"><abbr title="Formally declined and will not be accepted">Rejected</abbr></dd> <dt class="field-odd">Type<span class="colon">:</span></dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created<span class="colon">:</span></dt> <dd class="field-even">18-Jan-2021</dd> <dt class="field-odd">Post-History<span class="colon">:</span></dt> <dd class="field-odd">19-Jan-2021</dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#rejection-notice">Rejection Notice</a></li> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#stackoverflow-exception">StackOverflow exception</a></li> <li><a class="reference internal" href="#recursionoverflow-exception">RecursionOverflow exception</a></li> <li><a class="reference internal" href="#decoupling-the-python-stack-from-the-c-stack">Decoupling the Python stack from the C stack</a></li> <li><a class="reference internal" href="#other-implementations">Other Implementations</a></li> <li><a class="reference internal" href="#c-api">C-API</a><ul> <li><a class="reference internal" href="#py-checkstackdepth">Py_CheckStackDepth()</a></li> <li><a class="reference internal" href="#py-enterrecursivecall">Py_EnterRecursiveCall()</a></li> <li><a class="reference internal" href="#pyleaverecursivecall">PyLeaveRecursiveCall()</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a></li> <li><a class="reference internal" href="#security-implications">Security Implications</a></li> <li><a class="reference internal" href="#performance-impact">Performance Impact</a></li> <li><a class="reference internal" href="#implementation">Implementation</a><ul> <li><a class="reference internal" href="#monitoring-c-stack-consumption">Monitoring C stack consumption</a></li> <li><a class="reference internal" href="#making-python-to-python-calls-without-consuming-the-c-stack">Making Python-to-Python calls without consuming the C stack</a></li> </ul> </li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a></li> <li><a class="reference internal" href="#open-issues">Open Issues</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="rejection-notice"> <h2><a class="toc-backref" href="#rejection-notice" role="doc-backlink">Rejection Notice</a></h2> <p>This PEP has been <a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/75BFSBM5AJWXOF5OSPLMJQSTP3TDOKRP/">rejected by the Python Steering Council</a>.</p> </section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> <p>This PEP proposes that Python should treat machine stack overflow differently from runaway recursion.</p> <p>This would allow programs to set the maximum recursion depth to fit their needs and provide additional safety guarantees.</p> <p>If this PEP is accepted, then the following program will run safely to completion:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">sys</span><span class="o">.</span><span class="n">setrecursionlimit</span><span class="p">(</span><span class="mi">1_000_000</span><span class="p">)</span> <span class="k">def</span><span class="w"> </span><span class="nf">f</span><span class="p">(</span><span class="n">n</span><span class="p">):</span> <span class="k">if</span> <span class="n">n</span><span class="p">:</span> <span class="n">f</span><span class="p">(</span><span class="n">n</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="n">f</span><span class="p">(</span><span class="mi">500_000</span><span class="p">)</span> </pre></div> </div> <p>and the following program will raise a <code class="docutils literal notranslate"><span class="pre">StackOverflow</span></code>, without causing a VM crash:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">sys</span><span class="o">.</span><span class="n">setrecursionlimit</span><span class="p">(</span><span class="mi">1_000_000</span><span class="p">)</span> <span class="k">class</span><span class="w"> </span><span class="nc">X</span><span class="p">:</span> <span class="k">def</span><span class="w"> </span><span class="fm">__add__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span> <span class="o">+</span> <span class="n">other</span> <span class="n">X</span><span class="p">()</span> <span class="o">+</span> <span class="mi">1</span> </pre></div> </div> </section> <section id="motivation"> <h2><a class="toc-backref" href="#motivation" role="doc-backlink">Motivation</a></h2> <p>CPython uses a single recursion depth counter to prevent both runaway recursion and C stack overflow. However, runaway recursion and machine stack overflow are two different things. Allowing machine stack overflow is a potential security vulnerability, but limiting recursion depth can prevent the use of some algorithms in Python.</p> <p>Currently, if a program needs to deeply recurse it must manage the maximum recursion depth allowed, hopefully managing to set it in the region between the minimum needed to run correctly and the maximum that is safe to avoid a memory protection error.</p> <p>By separating the checks for C stack overflow from checks for recursion depth, pure Python programs can run safely, using whatever level of recursion they require.</p> </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> <p>CPython currently relies on a single limit to guard against potentially dangerous stack overflow in the virtual machine and to guard against run away recursion in the Python program.</p> <p>This is a consequence of the implementation which couples the C and Python call stacks. By breaking this coupling, we can improve both the usability of CPython and its safety.</p> <p>The recursion limit exists to protect against runaway recursion, the integrity of the virtual machine should not depend on it. Similarly, recursion should not be limited by implementation details.</p> </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> <p>Two new exception classes will be added, <code class="docutils literal notranslate"><span class="pre">StackOverflow</span></code> and <code class="docutils literal notranslate"><span class="pre">RecursionOverflow</span></code>, both of which will be sub-classes of <code class="docutils literal notranslate"><span class="pre">RecursionError</span></code></p> <section id="stackoverflow-exception"> <h3><a class="toc-backref" href="#stackoverflow-exception" role="doc-backlink">StackOverflow exception</a></h3> <p>A <code class="docutils literal notranslate"><span class="pre">StackOverflow</span></code> exception will be raised whenever the interpreter or builtin module code determines that the C stack is at or nearing a limit of safety. <code class="docutils literal notranslate"><span class="pre">StackOverflow</span></code> is a sub-class of <code class="docutils literal notranslate"><span class="pre">RecursionError</span></code>, so any code that handles <code class="docutils literal notranslate"><span class="pre">RecursionError</span></code> will handle <code class="docutils literal notranslate"><span class="pre">StackOverflow</span></code></p> </section> <section id="recursionoverflow-exception"> <h3><a class="toc-backref" href="#recursionoverflow-exception" role="doc-backlink">RecursionOverflow exception</a></h3> <p>A <code class="docutils literal notranslate"><span class="pre">RecursionOverflow</span></code> exception will be raised when a call to a Python function causes the recursion limit to be exceeded. This is a slight change from current behavior which raises a <code class="docutils literal notranslate"><span class="pre">RecursionError</span></code>. <code class="docutils literal notranslate"><span class="pre">RecursionOverflow</span></code> is a sub-class of <code class="docutils literal notranslate"><span class="pre">RecursionError</span></code>, so any code that handles <code class="docutils literal notranslate"><span class="pre">RecursionError</span></code> will continue to work as before.</p> </section> <section id="decoupling-the-python-stack-from-the-c-stack"> <h3><a class="toc-backref" href="#decoupling-the-python-stack-from-the-c-stack" role="doc-backlink">Decoupling the Python stack from the C stack</a></h3> <p>In order to provide the above guarantees and ensure that any program that worked previously continues to do so, the Python and C stack will need to be separated. That is, calls to Python functions from Python functions, should not consume space on the C stack. Calls to and from builtin functions will continue to consume space on the C stack.</p> <p>The size of the C stack will be implementation defined, and may vary from machine to machine. It may even differ between threads. However, there is an expectation that any code that could run with the recursion limit set to the previous default value, will continue to run.</p> <p>Many operations in Python perform some sort of call at the C level. Most of these will continue to consume C stack, and will result in a <code class="docutils literal notranslate"><span class="pre">StackOverflow</span></code> exception if uncontrolled recursion occurs.</p> </section> <section id="other-implementations"> <h3><a class="toc-backref" href="#other-implementations" role="doc-backlink">Other Implementations</a></h3> <p>Other implementations are required to fail safely regardless of what value the recursion limit is set to.</p> <p>If the implementation couples the Python stack to the underlying VM or hardware stack, then it should raise a <code class="docutils literal notranslate"><span class="pre">RecursionOverflow</span></code> exception when the recursion limit is exceeded, but the underlying stack does not overflow. If the underlying stack overflows, or is near to overflow, then a <code class="docutils literal notranslate"><span class="pre">StackOverflow</span></code> exception should be raised.</p> </section> <section id="c-api"> <h3><a class="toc-backref" href="#c-api" role="doc-backlink">C-API</a></h3> <p>A new function, <code class="docutils literal notranslate"><span class="pre">Py_CheckStackDepth()</span></code> will be added, and the behavior of <code class="docutils literal notranslate"><span class="pre">Py_EnterRecursiveCall()</span></code> will be modified slightly.</p> <section id="py-checkstackdepth"> <h4><a class="toc-backref" href="#py-checkstackdepth" role="doc-backlink">Py_CheckStackDepth()</a></h4> <p><code class="docutils literal notranslate"><span class="pre">int</span> <span class="pre">Py_CheckStackDepth(const</span> <span class="pre">char</span> <span class="pre">*where)</span></code> will return 0 if there is no immediate danger of C stack overflow. It will return -1 and set an exception, if the C stack is near to overflowing. The <code class="docutils literal notranslate"><span class="pre">where</span></code> parameter is used in the exception message, in the same fashion as the <code class="docutils literal notranslate"><span class="pre">where</span></code> parameter of <code class="docutils literal notranslate"><span class="pre">Py_EnterRecursiveCall()</span></code>.</p> </section> <section id="py-enterrecursivecall"> <h4><a class="toc-backref" href="#py-enterrecursivecall" role="doc-backlink">Py_EnterRecursiveCall()</a></h4> <p><code class="docutils literal notranslate"><span class="pre">Py_EnterRecursiveCall()</span></code> will be modified to call <code class="docutils literal notranslate"><span class="pre">Py_CheckStackDepth()</span></code> before performing its current function.</p> </section> <section id="pyleaverecursivecall"> <h4><a class="toc-backref" href="#pyleaverecursivecall" role="doc-backlink">PyLeaveRecursiveCall()</a></h4> <p><code class="docutils literal notranslate"><span class="pre">Py_LeaveRecursiveCall()</span></code> will remain unchanged.</p> </section> </section> </section> <section id="backwards-compatibility"> <h2><a class="toc-backref" href="#backwards-compatibility" role="doc-backlink">Backwards Compatibility</a></h2> <p>This feature is fully backwards compatible at the Python level. Some low-level tools, such as machine-code debuggers, will need to be modified. For example, the gdb scripts for Python will need to be aware that there may be more than one Python frame per C frame.</p> <p>C code that uses the <code class="docutils literal notranslate"><span class="pre">Py_EnterRecursiveCall()</span></code>, <code class="docutils literal notranslate"><span class="pre">PyLeaveRecursiveCall()</span></code> pair of functions will continue to work correctly. In addition, <code class="docutils literal notranslate"><span class="pre">Py_EnterRecursiveCall()</span></code> may raise a <code class="docutils literal notranslate"><span class="pre">StackOverflow</span></code> exception.</p> <p>New code should use the <code class="docutils literal notranslate"><span class="pre">Py_CheckStackDepth()</span></code> function, unless the code wants to count as a Python function call with regard to the recursion limit.</p> <p>We recommend that “python-like” code, such as Cython-generated functions, use <code class="docutils literal notranslate"><span class="pre">Py_EnterRecursiveCall()</span></code>, but other code use <code class="docutils literal notranslate"><span class="pre">Py_CheckStackDepth()</span></code>.</p> </section> <section id="security-implications"> <h2><a class="toc-backref" href="#security-implications" role="doc-backlink">Security Implications</a></h2> <p>It will no longer be possible to crash the CPython virtual machine through recursion.</p> </section> <section id="performance-impact"> <h2><a class="toc-backref" href="#performance-impact" role="doc-backlink">Performance Impact</a></h2> <p>It is unlikely that the performance impact will be significant.</p> <p>The additional logic required will probably have a very small negative impact on performance. The improved locality of reference from reduced C stack use should have some small positive impact.</p> <p>It is hard to predict whether the overall effect will be positive or negative, but it is quite likely that the net effect will be too small to be measured.</p> </section> <section id="implementation"> <h2><a class="toc-backref" href="#implementation" role="doc-backlink">Implementation</a></h2> <section id="monitoring-c-stack-consumption"> <h3><a class="toc-backref" href="#monitoring-c-stack-consumption" role="doc-backlink">Monitoring C stack consumption</a></h3> <p>Gauging whether a C stack overflow is imminent is difficult. So we need to be conservative. We need to determine a safe bounds for the stack, which is not something possible in portable C code.</p> <p>For major platforms, the platform specific API will be used to provide an accurate stack bounds. However, for minor platforms some amount of guessing may be required. While this might sound bad, it is no worse than the current situation, where we guess that the size of the C stack is at least 1000 times the stack space required for the chain of calls from <code class="docutils literal notranslate"><span class="pre">_PyEval_EvalFrameDefault</span></code> to <code class="docutils literal notranslate"><span class="pre">_PyEval_EvalFrameDefault</span></code>.</p> <p>This means that in some cases the amount of recursion possible may be reduced. In general, however, the amount of recursion possible should be increased, as many calls will use no C stack.</p> <p>Our general approach to determining a limit for the C stack is to get an address within the current C frame, as early as possible in the call chain. The limit can then be guessed by adding some constant to that.</p> </section> <section id="making-python-to-python-calls-without-consuming-the-c-stack"> <h3><a class="toc-backref" href="#making-python-to-python-calls-without-consuming-the-c-stack" role="doc-backlink">Making Python-to-Python calls without consuming the C stack</a></h3> <p>Calls in the interpreter are handled by the <code class="docutils literal notranslate"><span class="pre">CALL_FUNCTION</span></code>, <code class="docutils literal notranslate"><span class="pre">CALL_FUNCTION_KW</span></code>, <code class="docutils literal notranslate"><span class="pre">CALL_FUNCTION_EX</span></code> and <code class="docutils literal notranslate"><span class="pre">CALL_METHOD</span></code> instructions. The code for those instructions will be modified so that when a Python function or method is called, instead of making a call in C, the interpreter will setup the callee’s frame and continue interpretation as normal.</p> <p>The <code class="docutils literal notranslate"><span class="pre">RETURN_VALUE</span></code> instruction will perform the reverse operation, except when the current frame is the entry frame of the interpreter when it will return as normal.</p> </section> </section> <section id="rejected-ideas"> <h2><a class="toc-backref" href="#rejected-ideas" role="doc-backlink">Rejected Ideas</a></h2> <p>None, as yet.</p> </section> <section id="open-issues"> <h2><a class="toc-backref" href="#open-issues" role="doc-backlink">Open Issues</a></h2> <p>None, as yet.</p> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> <p>This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.</p> </section> </section> <hr class="docutils" /> <p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0651.rst">https://github.com/python/peps/blob/main/peps/pep-0651.rst</a></p> <p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0651.rst">2025-02-01 08:55:40 GMT</a></p> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#rejection-notice">Rejection Notice</a></li> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#stackoverflow-exception">StackOverflow exception</a></li> <li><a class="reference internal" href="#recursionoverflow-exception">RecursionOverflow exception</a></li> <li><a class="reference internal" href="#decoupling-the-python-stack-from-the-c-stack">Decoupling the Python stack from the C stack</a></li> <li><a class="reference internal" href="#other-implementations">Other Implementations</a></li> <li><a class="reference internal" href="#c-api">C-API</a><ul> <li><a class="reference internal" href="#py-checkstackdepth">Py_CheckStackDepth()</a></li> <li><a class="reference internal" href="#py-enterrecursivecall">Py_EnterRecursiveCall()</a></li> <li><a class="reference internal" href="#pyleaverecursivecall">PyLeaveRecursiveCall()</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a></li> <li><a class="reference internal" href="#security-implications">Security Implications</a></li> <li><a class="reference internal" href="#performance-impact">Performance Impact</a></li> <li><a class="reference internal" href="#implementation">Implementation</a><ul> <li><a class="reference internal" href="#monitoring-c-stack-consumption">Monitoring C stack consumption</a></li> <li><a class="reference internal" href="#making-python-to-python-calls-without-consuming-the-c-stack">Making Python-to-Python calls without consuming the C stack</a></li> </ul> </li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a></li> <li><a class="reference internal" href="#open-issues">Open Issues</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <br> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0651.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>