Issue 812369: module shutdown procedure based on GC

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <title> Issue 812369: module shutdown procedure based on GC - Python tracker </title> <link rel="shortcut icon" href="@@file/favicon.ico" /> <link rel="stylesheet" type="text/css" href="@@file/main.css" /> <link rel="stylesheet" type="text/css" href="@@file/style.css" /> <link rel="search" type="application/opensearchdescription+xml" href="@@file/osd.xml" title="Python bug tracker search" /> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <script nonce="b03251f7b59557863160abf105c99451a45b1c27677a38053c435762743fdb04" type="text/javascript"> submitted = false; function submit_once() { if (submitted) { alert("Your request is being processed.\nPlease be patient."); return false; } submitted = true; return true; } function help_window(helpurl, width, height) { HelpWin = window.open('https://bugs.python.org/' + helpurl, 'RoundupHelpWindow', 'scrollbars=yes,resizable=yes,toolbar=no,height='+height+',width='+width); HelpWin.focus () } </script> <script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.6.2/jquery.min.js"></script> <script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.15/jquery-ui.js"></script> <script type="text/javascript" src="@@file/issue.item.js"></script> <link rel="stylesheet" type="text/css" href="https://ajax.googleapis.com/ajax/libs/jqueryui/1.8/themes/smoothness/jquery-ui.css" /> </head> <body>  <h1 id="logoheader"> <a accesskey="1" href="." id="logolink"> <img src="@@file/python-logo.gif" alt="homepage" border="0" id="logo" /></a> </h1> <div id="utility-menu">  <div id="searchbox"> <form name="searchform" method="get" action="issue" id="searchform"> <div id="search"> <input type="hidden" name="@columns" value="id,github,activity,title,creator,assignee,status,type" /> <input type="hidden" name="@sort" value="-activity" /> <input type="hidden" name="@filter" value="status" /> <input type="hidden" name="@action" value="searchid" /> <input type="hidden" name="ignore" value="file:content" /> <input class="input-text" id="search-text" name="@search_text" size="10" /> <input type="submit" id="submit" value="search" name="submit" class="input-button" /> <input type="radio" name="status" id="status_notresolved" value="-1,1,3" /> <label for="status_notresolved">open</label> <input type="radio" name="status" checked="checked" id="status_all" value="-1,1,2,3" /> <label for="status_all">all</label> </div> </form> </div> </div> <div id="left-hand-navigation">  <div id="menu"> <ul class="level-one"> <li><a href="https://www.python.org/" title="Go to the Python homepage">Python Home</a></li> <li><a href="https://www.python.org/about/" title="About The Python Language">About</a></li> <li><a href="https://www.python.org/blogs/" title="">News</a></li> <li><a href="https://www.python.org/doc/" title="">Documentation</a></li> <li><a href="https://www.python.org/downloads/" title="">Downloads</a></li> <li><a href="https://www.python.org/community/" title="">Community</a></li> <li><a href="https://www.python.org/psf/" title="Python Software Foundation">Foundation</a></li> <li><a href="https://devguide.python.org/" title="Python Developer's Guide">Developer's Guide</a></li> <li class="selected"><a href="." class="selected" title="Python Issue Tracker">Issue Tracker</a> <ul class="level-two"> <li> <strong>Issues</strong> <ul class="level-three"> <li><a href="issue?@template=search&status=1">Search</a></li> <li><a href="issue?@action=random">Random Issue</a></li> <li> <form method="post" action="issue812369"> <input type="submit" class="form-small" value="Show issue:" /> <input class="form-small" size="4" type="text" name="@number" /> <input type="hidden" name="@type" value="issue" /> <input type="hidden" name="@action" value="show" /> </form> </li> </ul> </li> <li> <strong>Summaries</strong> <ul class="level-three"> <li> <a href="issue?status=1&@sort=-activity&@columns=id%2Cgithub%2Cactivity%2Ctitle%2Ccreator%2Cstatus&@dispname=Issues%20with%20patch&@startwith=0&@group=priority&keywords=2&@action=search&@filter=&@pagesize=50">Issues with patch</a> </li> <li> <a href="issue?status=1&@sort=-activity&@columns=id%2Cgithub%2Cactivity%2Ctitle%2Ccreator%2Cstatus&@dispname=Easy%20issues&@startwith=0&@group=priority&keywords=6&@action=search&@filter=&@pagesize=50">Easy issues</a> </li> <li> <a href="issue?@template=stats">Stats</a> </li> </ul> </li> <li> <strong>User</strong> <form method="post" action="issue812369"> <ul class="level-three"> <li> Login<br /> <input size="10" name="openid_identifier" style="" /><br /> <input size="10" type="password" name="__login_password" /><br /> <input type="hidden" name="@action" value="Login" /> <input type="checkbox" name="remember" id="remember" /> <label for="remember">Remember me?</label><br /> <input class="form-small" type="submit" value="Login" /><br /> <input type="hidden" name="__came_from" value="https://bugs.python.org/issue812369?"> <input type="hidden" name="@sort" value=""/> <input type="hidden" name="@group" value=""/> <input type="hidden" name="@pagesize" value="50"/> <input type="hidden" name="@startwith" value="0"/> </li> <li> </li> <li><a href="user?@template=forgotten">Lost your login?</a></li> </ul> </form> </li> <li> <strong>Administration</strong> <ul class="level-three"> <li> <a href="user?@sort=username">User List</a></li> <li> <a href="user?iscommitter=1&@action=search&@sort=username&@pagesize=300">Committer List</a></li> </ul> </li> <li> <strong>Help</strong> <ul class="level-three"> <li><a href="http://docs.python.org/devguide/triaging.html"> Tracker Documentation</a></li> <li><a href="http://wiki.python.org/moin/TrackerDevelopment"> Tracker Development</a></li> <li><a href="https://github.com/python/psf-infra-meta/issues"> Report Tracker Problem</a></li> </ul> </li> </ul> </li> </ul> </div>  </div>  <div id="content-body"> <div id="body-main"> <div id="content"> <div id="breadcrumb"> Issue812369 </div> <div id="migration-notice"> <div id="migration-images"> <img width="32" src="@@file/python-logo-small.png" /> ➜ <a href="https://github.com/python/cpython/issues"><img width="32" src="@@file/gh-icon.png" /></a> </div> <p>This issue tracker <b>has been migrated to <a href="https://github.com/python/cpython/issues">GitHub</a></b>, and is currently <b>read-only</b>.<br /> For more information, <a title="GitHub FAQs" href="https://devguide.python.org/gh-faq/"> see the GitHub FAQs in the Python's Developer Guide.</a></p> </div> <div> <form method="post" name="itemSynopsis" onsubmit="return submit_once()" enctype="multipart/form-data" action="issue812369"> <div id="gh-issue-link"> <a href="https://github.com/python/cpython/issues/39300"> <img width="32" src="@@file/gh-icon.png" /> <p> <span>This issue has been migrated to GitHub:</span> https://github.com/python/cpython/issues/39300 </p> </a> </div> <fieldset><legend>classification</legend> <table class="form"> <tr> <th class="required"><a href="http://docs.python.org/devguide/triaging.html#title" target="_blank">Title</a>:</th> <td colspan="3"> <span>module shutdown procedure based on GC</span> <input type="hidden" name="title" value="module shutdown procedure based on GC"> </td> </tr> <tr> <th class="required"><a href="http://docs.python.org/devguide/triaging.html#type" target="_blank">Type</a>:</th> <td>behavior</td> <th><a href="http://docs.python.org/devguide/triaging.html#stage" target="_blank">Stage</a>:</th> <td>patch review</td> </tr> <tr> <th><a href="http://docs.python.org/devguide/triaging.html#components" target="_blank">Components</a>:</th> <td>Interpreter Core</td> <th><a href="http://docs.python.org/devguide/triaging.html#versions" target="_blank">Versions</a>:</th> <td>Python 3.2</td> </tr> </table> </fieldset> <fieldset><legend>process</legend> <table class="form"> <tr> <th><a href="http://docs.python.org/devguide/triaging.html#status" target="_blank">Status</a>:</th> <td>closed</td> <th><a href="http://docs.python.org/devguide/triaging.html#resolution" target="_blank">Resolution</a>:</th> <td>duplicate</td> </tr> <tr> <th> <a href="http://docs.python.org/devguide/triaging.html#dependencies" target="_blank">Dependencies</a>: </th> <td> </td> <th><a href="http://docs.python.org/devguide/triaging.html#superseder" target="_blank">Superseder</a>:</th> <td> Stop purging modules which are garbage collected before shutdown <span>  <br>View: <a href="issue18214" class="closed" title="Stop purging modules which are garbage collected before shutdown">18214</a> </span> </td> </tr> <tr> <th> <a href="http://docs.python.org/devguide/triaging.html#assigned-to" target="_blank">Assigned To</a>: </th> <td> </td> <th> <a href="http://docs.python.org/devguide/triaging.html#nosy-list" target="_blank">Nosy List</a>: </th> <td> BreamoreBoy, Stefan.Friesel, andrea.corbellini, arigo, asvetlov, belopolsky, brett.cannon, cburroughs, christian.heimes, eric.snow, flox, glchapman, gregory.p.smith, loewis, nascheme, ncoghlan, pitrou </td> </tr> <tr> <th> <a href="http://docs.python.org/devguide/triaging.html#priority" target="_blank">Priority</a>: </th> <td>normal</td> <th> <a href="http://docs.python.org/devguide/triaging.html#keywords" target="_blank">Keywords</a>: </th> <td>patch</td> </tr> </table> </fieldset> </form> <p>Created on <strong>2003-09-25 10:49</strong> by <strong>arigo</strong>, last changed <strong>2022-04-10 16:11</strong> by <strong>admin</strong>. This issue is now <strong style="color:#00F; background-color:inherit;">closed</strong>.</p> <table class="files"> <tr><th colspan="5" class="header">Files</th></tr> <tr> <th>File name</th> <th>Uploaded</th> <th>Description</th> <th>Edit</th> </tr> <tr> <td> <a href="file5605/x.txt">x.txt</a> </td> <td> <span>arigo</span>, <span>2003-09-25 10:49</span> </td> <td>patch</td> <td> </td> </tr> <tr> <td> <a href="file13091/0001-update-GC-shutdown-patch.patch">0001-update-GC-shutdown-patch.patch</a> </td> <td> <span>nascheme</span>, <span>2009-02-15 01:00</span> </td> <td></td> <td> </td> </tr> </table> <table class="messages"> <tr><th colspan="4" class="header">Messages (24)</th></tr> <tr> <th> <a href="#msg44689" id="msg44689">msg44689</a> - <a href="msg44689">(view)</a></th> <th>Author: Armin Rigo (arigo) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2003-09-25 10:49</th> </tr> <tr> <td colspan="4" class="content"> <pre>This patches changes PyImport_Cleanup() in an attempt to make the module shutdown order more predictable in the presence of modules that import each other. Before it explicitely clears the globals of the modules, it relies on the GC to try to do it more cleanly. <a href="http://mail.python.org/pipermail/python-dev/2003-September/038238.html">http://mail.python.org/pipermail/python-dev/2003-September/038238.html</a> To prevent objects with __del__ methods from keeping whole modules alive I actually replace each module with a weak reference to it in sys.modules. This allows me to find modules that remain alive after a call to PyGC_Collect(), and then go back to the old technique of clearing their globals. Note that weak references to dead cycles containing objects with finalizers have a strange property in Python: if you use the weak reference again you can break the cycles, but the objects with finalizers still won't be collected because they are in gc.garbage. As this is exactly what occurs above, I clear the gc.garbage list explicitely before the final PyGC_Collect() call. I'm not sure exactly what this might do; could it release older objects that were never put in a module but that at some time were put in gc.garbage and whose cycles were later broken? If so, is it a good thing to release them now? (Would it make sense to clear gc.garbage and call gc.collect again from time to time to check if the objects are still in a cycle?) This patch does not change the behavior of module objects clearing their globals dictionary as soon as they are deallocated. This could be work investigating. This patch puts weak references (actually proxies) in sys.modules but only at shutdown time. Moure thoughts could be given towards allowing weak references during normal program execution as well. To do so we must ensure that 'import' statements return the real module, not the weak proxy object, which is more difficult than it first seems in the presence of packages. And finally -- this patch has not really been tested, apart from the fact that it passes the test suite. </pre> </td> </tr> <tr> <th> <a href="#msg44690" id="msg44690">msg44690</a> - <a href="msg44690">(view)</a></th> <th>Author: Martin v. Löwis (loewis) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2003-09-27 17:31</th> </tr> <tr> <td colspan="4" class="content"> <pre>Logged In: YES user_id=21627 I think clearing gc.garbage at shutdown time is a good idea; gc.garbage is mostly a debugging aid, and should be empty in production code. If it still contains objects at shutdown time, it is IMO reasonable to give them a chance to become collected, in case somebody broke their cycles.</pre> </td> </tr> <tr> <th> <a href="#msg44691" id="msg44691">msg44691</a> - <a href="msg44691">(view)</a></th> <th>Author: Greg Chapman (glchapman)</th> <th>Date: 2004-03-11 13:56</th> </tr> <tr> <td colspan="4" class="content"> <pre>Logged In: YES user_id=86307 Not sure if this is a good idea, but I wonder if the extensions dictionary should be cleared (ie _PyImport_Fini called) sooner, possibly even before PyImport_Cleanup. This would allow garbage collection during PyImport_Cleanup to catch anything a C module might have created which is in a cycle with its module (through a bad design on my part, I recently discovered I had created just such a cycle). I suppose _PyImport_Fini is called when it is called because some code during PyImport_Cleanup might import a dynamic module, which would then create a new extensions dictionary if _PyImport_Fini had already been run. Perhaps a flag could be added so that _PyImport_FixupExtension would not try to add a module's dict to extensions if Python is currently shutting down. </pre> </td> </tr> <tr> <th> <a href="#msg44692" id="msg44692">msg44692</a> - <a href="msg44692">(view)</a></th> <th>Author: Brett Cannon (brett.cannon) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2004-05-18 18:40</th> </tr> <tr> <td colspan="4" class="content"> <pre>Logged In: YES user_id=357491 Just so you know, Armin, the patch did not apply cleanly; the comment for pythonrun.c did not apply. I also need to add an extern declaration in import.c for _PyGC_garbage for the code to compile (OS X 10.3, gcc 3.3).</pre> </td> </tr> <tr> <th> <a href="#msg59257" id="msg59257">msg59257</a> - <a href="msg59257">(view)</a></th> <th>Author: Christian Heimes (christian.heimes) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2008-01-04 19:57</th> </tr> <tr> <td colspan="4" class="content"> <pre>Consider this patch for 2.6 and discuss it at the bug day.</pre> </td> </tr> <tr> <th> <a href="#msg79671" id="msg79671">msg79671</a> - <a href="msg79671">(view)</a></th> <th>Author: Antoine Pitrou (pitrou) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2009-01-12 12:57</th> </tr> <tr> <td colspan="4" class="content"> <pre>Looking at the patch, is there any reason it doesn't get rid of the current _PyModule_Clear() implementation to replace it by a call to PyDict_Clear() followed by PyGC_Collect()? (the call to PyGC_Collect could be disabled while finalizing, because there's no use calling it as many times as there are modules to be disbanded) The major annoyance with the current scheme is that, at interpreter shutdown, some globals you want to rely on in your destructors suddenly become None. About what to do of gc.garbage at shutdown, there was another proposal in <a class="closed" title="[closed] Print warning at shutdown if gc.garbage not empty" href="issue477863">#477863</a>.</pre> </td> </tr> <tr> <th> <a href="#msg82136" id="msg82136">msg82136</a> - <a href="msg82136">(view)</a></th> <th>Author: Neil Schemenauer (nascheme) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2009-02-15 01:00</th> </tr> <tr> <td colspan="4" class="content"> <pre>This sounds like a nice idea. The current cleanup procedure in pythonrun.c is pretty lame since it can play havoc with __del__ methods (e.g. if they run after globals have been cleared). I've updated the patch to work with the current SVN head. Probably this should get tested with big applications based on Zope, Django, etc.</pre> </td> </tr> <tr> <th> <a href="#msg84775" id="msg84775">msg84775</a> - <a href="msg84775">(view)</a></th> <th>Author: Antoine Pitrou (pitrou) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2009-03-31 13:18</th> </tr> <tr> <td colspan="4" class="content"> <pre>Retargetting, and I hope someone can take a look at the patch and give it the green light :-)</pre> </td> </tr> <tr> <th> <a href="#msg94027" id="msg94027">msg94027</a> - <a href="msg94027">(view)</a></th> <th>Author: Alexander Belopolsky (belopolsky) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2009-10-14 20:40</th> </tr> <tr> <td colspan="4" class="content"> <pre>Does this patch fix <a class="closed" title="[closed] New-style classes fail to cleanup attributes" href="issue1545463">issue1545463</a> by any chance? I am away from a development box ATM and cannot test the patch myself.</pre> </td> </tr> <tr> <th> <a href="#msg94033" id="msg94033">msg94033</a> - <a href="msg94033">(view)</a></th> <th>Author: Neil Schemenauer (nascheme) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2009-10-14 21:20</th> </tr> <tr> <td colspan="4" class="content"> <pre>It should fix <a class="closed" title="[closed] New-style classes fail to cleanup attributes" href="issue1545463">issue1545463</a> and running a quick test seems to show that it does.</pre> </td> </tr> <tr> <th> <a href="#msg110337" id="msg110337">msg110337</a> - <a href="msg110337">(view)</a></th> <th>Author: Mark Lawrence (BreamoreBoy) <span title="Contributor form received">*</span></th> <th>Date: 2010-07-14 22:42</th> </tr> <tr> <td colspan="4" class="content"> <pre><a class="closed" title="[closed] New-style classes fail to cleanup attributes" href="issue1545463">issue1545463</a> has been closed as "won't fix", so wouldn't implementing this patch kill two birds with one stone?</pre> </td> </tr> <tr> <th> <a href="#msg114285" id="msg114285">msg114285</a> - <a href="msg114285">(view)</a></th> <th>Author: Mark Lawrence (BreamoreBoy) <span title="Contributor form received">*</span></th> <th>Date: 2010-08-18 22:25</th> </tr> <tr> <td colspan="4" class="content"> <pre><a class="closed" title="[closed] New-style classes fail to cleanup attributes" href="issue1545463">#1545463</a> has been reopened with comments about being used as a stop gap. Possibly review and implementation of the patch here would be a better option, sorry it's over my head.</pre> </td> </tr> <tr> <th> <a href="#msg118245" id="msg118245">msg118245</a> - <a href="msg118245">(view)</a></th> <th>Author: Gregory P. Smith (gregory.p.smith) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2010-10-09 04:50</th> </tr> <tr> <td colspan="4" class="content"> <pre>0001-update-GC-shutdown-patch.patch looks sane to me at first glance. any other opinions?</pre> </td> </tr> <tr> <th> <a href="#msg118259" id="msg118259">msg118259</a> - <a href="msg118259">(view)</a></th> <th>Author: Antoine Pitrou (pitrou) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2010-10-09 10:01</th> </tr> <tr> <td colspan="4" class="content"> <pre>The patch is obviously against 2.x (there are some PyString_Check's on module names, for example). It should be regenerated against 3.x. Also, it would be nice if a test could be devised to check that the shutdown procedure works as expected (I'm not sure how such a test would look like).</pre> </td> </tr> <tr> <th> <a href="#msg153530" id="msg153530">msg153530</a> - <a href="msg153530">(view)</a></th> <th>Author: Alyssa Coghlan (ncoghlan) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2012-02-17 03:47</th> </tr> <tr> <td colspan="4" class="content"> <pre>In <a class="closed" title="[closed] behavior of test.support.import_fresh_module" href="issue14035">#14035</a>, Florent pointed out the current behaviour potentially causes problems for some uses of import_fresh_modules() in the test suite (with globals sometimes being set to None if there's no indepenent reference to the module). GC based module cleanup would avoid that problem automatically.</pre> </td> </tr> <tr> <th> <a href="#msg153576" id="msg153576">msg153576</a> - <a href="msg153576">(view)</a></th> <th>Author: Armin Rigo (arigo) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2012-02-17 17:19</th> </tr> <tr> <td colspan="4" class="content"> <pre>Fwiw, the behavior in PyPy is: don't do anything particular at shut-down, just shut down and quit the process. No hacking at module globals to replace them with None, but also no guaranteeing that any __del__ method is ever called. We didn't get a particular bug report about this so far, so it seems to work. (This is just a report about PyPy's situation; I understand that the situation in CPython is a bit more delicate if CPython is embedded in a larger process.)</pre> </td> </tr> <tr> <th> <a href="#msg153603" id="msg153603">msg153603</a> - <a href="msg153603">(view)</a></th> <th>Author: Alyssa Coghlan (ncoghlan) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2012-02-17 21:52</th> </tr> <tr> <td colspan="4" class="content"> <pre>Also, since this issue was last updated, Antoine devised a scheme to test some of the embedding functionality (mainly to test subinterpreters, IIRC). Perhaps that could be harnessed to check GC-based shutdown is working correctly (it might even do it already, without any changes to the test suite).</pre> </td> </tr> <tr> <th> <a href="#msg154094" id="msg154094">msg154094</a> - <a href="msg154094">(view)</a></th> <th>Author: Martin v. Löwis (loewis) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2012-02-23 22:13</th> </tr> <tr> <td colspan="4" class="content"> <pre>> (This is just a report about PyPy's situation; I understand that the > situation in CPython is a bit more delicate if CPython is embedded in > a larger process.) I think that would indeed be unacceptable for Python - there is a long-standing expectation that we free all memory that we allocated, as well as release any other resources that we hold. There are also expectations wrt. running atexit code. So there clearly must be a shutdown procedure.</pre> </td> </tr> <tr> <th> <a href="#msg154122" id="msg154122">msg154122</a> - <a href="msg154122">(view)</a></th> <th>Author: Armin Rigo (arigo) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2012-02-24 08:31</th> </tr> <tr> <td colspan="4" class="content"> <pre>Obviously we run atexit code too. There is no point in having atexit if it's not guaranteed to run in a normal shutdown.</pre> </td> </tr> <tr> <th> <a href="#msg172099" id="msg172099">msg172099</a> - <a href="msg172099">(view)</a></th> <th>Author: Stefan Friesel (Stefan.Friesel)</th> <th>Date: 2012-10-05 16:57</th> </tr> <tr> <td colspan="4" class="content"> <pre>What is the status of this? Does the patch need more reviewing?</pre> </td> </tr> <tr> <th> <a href="#msg172106" id="msg172106">msg172106</a> - <a href="msg172106">(view)</a></th> <th>Author: Martin v. Löwis (loewis) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2012-10-05 18:23</th> </tr> <tr> <td colspan="4" class="content"> <pre>At the moment, it's like that the status of the patch needs to be reestablished. Does it apply? Does it work? Does the test suite still pass?</pre> </td> </tr> <tr> <th> <a href="#msg172165" id="msg172165">msg172165</a> - <a href="msg172165">(view)</a></th> <th>Author: Neil Schemenauer (nascheme) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2012-10-06 03:18</th> </tr> <tr> <td colspan="4" class="content"> <pre>It's been quite a long time since I played with this patch so my memory might be a bit fuzzy. As I recall, it sounds good in theory but in practice it doesn't really work. One of the core problems is that many extension modules keep references to Python objects in global or static variables. These references keep pretty much everything alive and prevent GC cleanup of modules. So, a necessary condition to this working is to get rid of those references and use the new module struct facility introduced by Martin. That would be a huge amount of work but I think should be the long term goal.</pre> </td> </tr> <tr> <th> <a href="#msg179098" id="msg179098">msg179098</a> - <a href="msg179098">(view)</a></th> <th>Author: Alyssa Coghlan (ncoghlan) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2013-01-05 01:33</th> </tr> <tr> <td colspan="4" class="content"> <pre>In addition to the problem Neil noted with references in extension modules keeping module objects themselves alive, Antoine recently noted that the other major challenge is the reference cycles between module global dictionaries and their contents. As soon as a module global has both a __del__ method and a reference back to the module globals, the entire cycle becomes uncollectable. I suspect one of the reasons PyPy can cope without the explicit reference breaking step is that their GC is better able to cope with __del__ methods than ours. I wonder if a useful interim step might be to make the current explicit reference breaking hack a bit smarter by looking at the reference counts. (Note: some aspects of this idea could be made simpler if modules supported weak references) 1. Call importlib.invalidate_caches() 2. Delete the first module from sys.modules that has a reference count of exactly one 3. Repeat 2 until sys.modules is empty or every remaining module has a reference count greater than 1 (meaning another module has a reference to it one way or another) 4. Pick the module in sys.modules with the lowest number of references to it, delete it from sys.modules and delete the reference from the module object to its dictionary 5. Repeat 4 until sys.modules is empty Throughout the process, keep an eye on gc.garbage - if we see a module dict show up there, hit it with the "set all globals to None" hammer. (The new callback functionality in 3.3 makes that easier - for example, you could put a sentinel object in the globals of the module being cleared and watching for a dict containing that sentinel object showing up in 'uncollectable' during the stop phase)</pre> </td> </tr> <tr> <th> <a href="#msg193946" id="msg193946">msg193946</a> - <a href="msg193946">(view)</a></th> <th>Author: Antoine Pitrou (pitrou) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2013-07-30 18:34</th> </tr> <tr> <td colspan="4" class="content"> <pre>Superceded by patch in <a class="closed" title="[closed] Stop purging modules which are garbage collected before shutdown" href="issue18214">issue 18214</a>.</pre> </td> </tr> </table> <table class="history table table-condensed table-striped"><tr><th colspan="4" class="header"> History </th></tr><tr> <th>Date</th> <th>User</th> <th>Action</th> <th>Args</th> </tr> <tr><td>2022-04-10 16:11:22</td><td>admin</td><td>set</td><td>github: 39300</td></tr> <tr><td>2013-08-21 06:13:40</td><td>pitrou</td><td>set</td><td>superseder: <a ref="nofollow" href="issue812369">module shutdown procedure based on GC</a> -> <a rel="nofollow" href="issue18214">Stop purging modules which are garbage collected before shutdown</a></td></tr> <tr><td>2013-08-21 06:13:40</td><td>pitrou</td><td>unlink</td><td><a rel="nofollow" href="issue812369">issue812369 superseder</a></td></tr> <tr><td>2013-07-30 18:34:23</td><td>pitrou</td><td>set</td><td>status: open -> closed<br />superseder: <a ref="nofollow" href="issue812369">module shutdown procedure based on GC</a><br />resolution: duplicate<br />messages: + <a rel="nofollow" href="msg193946">msg193946</a><br /></td></tr> <tr><td>2013-07-30 18:34:23</td><td>pitrou</td><td>link</td><td><a rel="nofollow" href="issue812369">issue812369 superseder</a></td></tr> <tr><td>2013-01-05 01:33:08</td><td>ncoghlan</td><td>set</td><td>messages: + <a rel="nofollow" href="msg179098">msg179098</a></td></tr> <tr><td>2012-12-24 16:31:36</td><td>asvetlov</td><td>set</td><td>nosy: + <a rel="nofollow" href="user10303">asvetlov</a><br /></td></tr> <tr><td>2012-11-13 02:45:18</td><td>eric.snow</td><td>set</td><td>nosy: + <a rel="nofollow" href="user14000">eric.snow</a><br /></td></tr> <tr><td>2012-11-10 22:38:57</td><td>gregory.p.smith</td><td>set</td><td>assignee: <a ref="nofollow" href="user252">gregory.p.smith</a> -> <a rel="nofollow" href="userNone"></a></td></tr> <tr><td>2012-10-06 03:18:32</td><td>nascheme</td><td>set</td><td>messages: + <a rel="nofollow" href="msg172165">msg172165</a></td></tr> <tr><td>2012-10-05 18:23:20</td><td>loewis</td><td>set</td><td>messages: + <a rel="nofollow" href="msg172106">msg172106</a></td></tr> <tr><td>2012-10-05 16:57:07</td><td>Stefan.Friesel</td><td>set</td><td>nosy: + <a rel="nofollow" href="user16654">Stefan.Friesel</a><br />messages: + <a rel="nofollow" href="msg172099">msg172099</a><br /></td></tr> <tr><td>2012-02-24 08:31:39</td><td>arigo</td><td>set</td><td>messages: + <a rel="nofollow" href="msg154122">msg154122</a></td></tr> <tr><td>2012-02-23 22:13:02</td><td>loewis</td><td>set</td><td>messages: + <a rel="nofollow" href="msg154094">msg154094</a></td></tr> <tr><td>2012-02-17 21:52:35</td><td>ncoghlan</td><td>set</td><td>messages: + <a rel="nofollow" href="msg153603">msg153603</a></td></tr> <tr><td>2012-02-17 17:19:14</td><td>arigo</td><td>set</td><td>messages: + <a rel="nofollow" href="msg153576">msg153576</a></td></tr> <tr><td>2012-02-17 07:45:29</td><td>flox</td><td>set</td><td>nosy: + <a rel="nofollow" href="user11184">flox</a><br /></td></tr> <tr><td>2012-02-17 03:47:59</td><td>ncoghlan</td><td>set</td><td>nosy: + <a rel="nofollow" href="user1309">ncoghlan</a><br />messages: + <a rel="nofollow" href="msg153530">msg153530</a><br /></td></tr> <tr><td>2012-02-17 03:45:22</td><td>ncoghlan</td><td>link</td><td><a rel="nofollow" href="issue14035">issue14035 superseder</a></td></tr> <tr><td>2010-10-09 10:01:44</td><td>pitrou</td><td>set</td><td>messages: + <a rel="nofollow" href="msg118259">msg118259</a><br />versions: - Python 3.1, Python 2.7</td></tr> <tr><td>2010-10-09 04:50:20</td><td>gregory.p.smith</td><td>set</td><td>messages: + <a rel="nofollow" href="msg118245">msg118245</a></td></tr> <tr><td>2010-08-18 22:25:33</td><td>BreamoreBoy</td><td>set</td><td>messages: + <a rel="nofollow" href="msg114285">msg114285</a><br />versions: + Python 3.2</td></tr> <tr><td>2010-07-29 16:52:57</td><td>andrea.corbellini</td><td>set</td><td>nosy: + <a rel="nofollow" href="user9931">andrea.corbellini</a><br /></td></tr> <tr><td>2010-07-14 22:42:22</td><td>BreamoreBoy</td><td>set</td><td>nosy: + <a rel="nofollow" href="user12545">BreamoreBoy</a><br />messages: + <a rel="nofollow" href="msg110337">msg110337</a><br /></td></tr> <tr><td>2010-05-04 19:19:56</td><td>cburroughs</td><td>set</td><td>nosy: + <a rel="nofollow" href="user12135">cburroughs</a><br /></td></tr> <tr><td>2009-10-14 21:20:40</td><td>nascheme</td><td>set</td><td>messages: + <a rel="nofollow" href="msg94033">msg94033</a></td></tr> <tr><td>2009-10-14 20:44:16</td><td>gregory.p.smith</td><td>set</td><td>assignee: <a ref="nofollow" href="user252">gregory.p.smith</a><br /><br />nosy: + <a rel="nofollow" href="user252">gregory.p.smith</a></td></tr> <tr><td>2009-10-14 20:40:52</td><td>belopolsky</td><td>set</td><td>nosy: + <a rel="nofollow" href="user1427">belopolsky</a><br />messages: + <a rel="nofollow" href="msg94027">msg94027</a><br /></td></tr> <tr><td>2009-03-31 13:18:10</td><td>pitrou</td><td>set</td><td>type: behavior<br />stage: patch review<br />messages: + <a rel="nofollow" href="msg84775">msg84775</a><br />versions: + Python 3.1, Python 2.7, - Python 2.6, Python 3.0</td></tr> <tr><td>2009-03-31 13:16:00</td><td>pitrou</td><td>link</td><td><a rel="nofollow" href="issue1717900">issue1717900 superseder</a></td></tr> <tr><td>2009-02-15 01:00:47</td><td>nascheme</td><td>set</td><td>files: + <a rel="nofollow" href="file13091">0001-update-GC-shutdown-patch.patch</a><br />nosy: + <a rel="nofollow" href="user83">nascheme</a><br />messages: + <a rel="nofollow" href="msg82136">msg82136</a></td></tr> <tr><td>2009-01-12 12:57:46</td><td>pitrou</td><td>set</td><td>nosy: + <a rel="nofollow" href="user2040">pitrou</a><br />messages: + <a rel="nofollow" href="msg79671">msg79671</a></td></tr> <tr><td>2008-01-04 19:57:20</td><td>christian.heimes</td><td>set</td><td>nosy: + <a rel="nofollow" href="user3108">christian.heimes</a><br />messages: + <a rel="nofollow" href="msg59257">msg59257</a><br />versions: + Python 2.6, Python 3.0, - Python 2.4</td></tr> <tr><td>2003-09-25 10:49:56</td><td>arigo</td><td>create</td><td></td></tr> </table> </div> </div>  <div id="footer"> <div id="credits"> Supported by <a href="https://python.org/psf-landing/" title="The Python Software Foundation">The Python Software Foundation</a>, <br> Powered by <a href="http://roundup.sourceforge.net" title="Powered by the Roundup Issue Tracker">Roundup</a> </div>  Copyright © 1990-2022, <a href="http://python.org/psf">Python Software Foundation</a><br /> <a href="http://python.org/about/legal">Legal Statements</a> </div>  </div>  </div>  </body> </html>

CINXE.COM

Issue 812369: module shutdown procedure based on GC - Python tracker