Issue 210674: memory bloat in binary file upload (PR#381)

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <title> Issue 210674: memory bloat in binary file upload (PR#381) - Python tracker </title> <link rel="shortcut icon" href="@@file/favicon.ico" /> <link rel="stylesheet" type="text/css" href="@@file/main.css" /> <link rel="stylesheet" type="text/css" href="@@file/style.css" /> <link rel="search" type="application/opensearchdescription+xml" href="@@file/osd.xml" title="Python bug tracker search" /> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <script nonce="539175191a5394a58f5aec679944e5a0b14bbd0127db7939a0512bbeb9c7d96a" type="text/javascript"> submitted = false; function submit_once() { if (submitted) { alert("Your request is being processed.\nPlease be patient."); return false; } submitted = true; return true; } function help_window(helpurl, width, height) { HelpWin = window.open('https://bugs.python.org/' + helpurl, 'RoundupHelpWindow', 'scrollbars=yes,resizable=yes,toolbar=no,height='+height+',width='+width); HelpWin.focus () } </script> <script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.6.2/jquery.min.js"></script> <script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.15/jquery-ui.js"></script> <script type="text/javascript" src="@@file/issue.item.js"></script> <link rel="stylesheet" type="text/css" href="https://ajax.googleapis.com/ajax/libs/jqueryui/1.8/themes/smoothness/jquery-ui.css" /> </head> <body>  <h1 id="logoheader"> <a accesskey="1" href="." id="logolink"> <img src="@@file/python-logo.gif" alt="homepage" border="0" id="logo" /></a> </h1> <div id="utility-menu">  <div id="searchbox"> <form name="searchform" method="get" action="issue" id="searchform"> <div id="search"> <input type="hidden" name="@columns" value="id,github,activity,title,creator,assignee,status,type" /> <input type="hidden" name="@sort" value="-activity" /> <input type="hidden" name="@filter" value="status" /> <input type="hidden" name="@action" value="searchid" /> <input type="hidden" name="ignore" value="file:content" /> <input class="input-text" id="search-text" name="@search_text" size="10" /> <input type="submit" id="submit" value="search" name="submit" class="input-button" /> <input type="radio" name="status" id="status_notresolved" value="-1,1,3" /> <label for="status_notresolved">open</label> <input type="radio" name="status" checked="checked" id="status_all" value="-1,1,2,3" /> <label for="status_all">all</label> </div> </form> </div> </div> <div id="left-hand-navigation">  <div id="menu"> <ul class="level-one"> <li><a href="https://www.python.org/" title="Go to the Python homepage">Python Home</a></li> <li><a href="https://www.python.org/about/" title="About The Python Language">About</a></li> <li><a href="https://www.python.org/blogs/" title="">News</a></li> <li><a href="https://www.python.org/doc/" title="">Documentation</a></li> <li><a href="https://www.python.org/downloads/" title="">Downloads</a></li> <li><a href="https://www.python.org/community/" title="">Community</a></li> <li><a href="https://www.python.org/psf/" title="Python Software Foundation">Foundation</a></li> <li><a href="https://devguide.python.org/" title="Python Developer's Guide">Developer's Guide</a></li> <li class="selected"><a href="." class="selected" title="Python Issue Tracker">Issue Tracker</a> <ul class="level-two"> <li> <strong>Issues</strong> <ul class="level-three"> <li><a href="issue?@template=search&status=1">Search</a></li> <li><a href="issue?@action=random">Random Issue</a></li> <li> <form method="post" action="issue210674"> <input type="submit" class="form-small" value="Show issue:" /> <input class="form-small" size="4" type="text" name="@number" /> <input type="hidden" name="@type" value="issue" /> <input type="hidden" name="@action" value="show" /> </form> </li> </ul> </li> <li> <strong>Summaries</strong> <ul class="level-three"> <li> <a href="issue?status=1&@sort=-activity&@columns=id%2Cgithub%2Cactivity%2Ctitle%2Ccreator%2Cstatus&@dispname=Issues%20with%20patch&@startwith=0&@group=priority&keywords=2&@action=search&@filter=&@pagesize=50">Issues with patch</a> </li> <li> <a href="issue?status=1&@sort=-activity&@columns=id%2Cgithub%2Cactivity%2Ctitle%2Ccreator%2Cstatus&@dispname=Easy%20issues&@startwith=0&@group=priority&keywords=6&@action=search&@filter=&@pagesize=50">Easy issues</a> </li> <li> <a href="issue?@template=stats">Stats</a> </li> </ul> </li> <li> <strong>User</strong> <form method="post" action="issue210674"> <ul class="level-three"> <li> Login<br /> <input size="10" name="openid_identifier" style="" /><br /> <input size="10" type="password" name="__login_password" /><br /> <input type="hidden" name="@action" value="Login" /> <input type="checkbox" name="remember" id="remember" /> <label for="remember">Remember me?</label><br /> <input class="form-small" type="submit" value="Login" /><br /> <input type="hidden" name="__came_from" value="https://bugs.python.org/issue210674?"> <input type="hidden" name="@sort" value=""/> <input type="hidden" name="@group" value=""/> <input type="hidden" name="@pagesize" value="50"/> <input type="hidden" name="@startwith" value="0"/> </li> <li> </li> <li><a href="user?@template=forgotten">Lost your login?</a></li> </ul> </form> </li> <li> <strong>Administration</strong> <ul class="level-three"> <li> <a href="user?@sort=username">User List</a></li> <li> <a href="user?iscommitter=1&@action=search&@sort=username&@pagesize=300">Committer List</a></li> </ul> </li> <li> <strong>Help</strong> <ul class="level-three"> <li><a href="http://docs.python.org/devguide/triaging.html"> Tracker Documentation</a></li> <li><a href="http://wiki.python.org/moin/TrackerDevelopment"> Tracker Development</a></li> <li><a href="https://github.com/python/psf-infra-meta/issues"> Report Tracker Problem</a></li> </ul> </li> </ul> </li> </ul> </div>  </div>  <div id="content-body"> <div id="body-main"> <div id="content"> <div id="breadcrumb"> Issue210674 </div> <div id="migration-notice"> <div id="migration-images"> <img width="32" src="@@file/python-logo-small.png" /> ➜ <a href="https://github.com/python/cpython/issues"><img width="32" src="@@file/gh-icon.png" /></a> </div> <p>This issue tracker <b>has been migrated to <a href="https://github.com/python/cpython/issues">GitHub</a></b>, and is currently <b>read-only</b>.<br /> For more information, <a title="GitHub FAQs" href="https://devguide.python.org/gh-faq/"> see the GitHub FAQs in the Python's Developer Guide.</a></p> </div> <div> <form method="post" name="itemSynopsis" onsubmit="return submit_once()" enctype="multipart/form-data" action="issue210674"> <div id="gh-issue-link"> <a href="https://github.com/python/cpython/issues/32744"> <img width="32" src="@@file/gh-icon.png" /> <p> <span>This issue has been migrated to GitHub:</span> https://github.com/python/cpython/issues/32744 </p> </a> </div> <fieldset><legend>classification</legend> <table class="form"> <tr> <th class="required"><a href="http://docs.python.org/devguide/triaging.html#title" target="_blank">Title</a>:</th> <td colspan="3"> <span>memory bloat in binary file upload (PR#381)</span> <input type="hidden" name="title" value="memory bloat in binary file upload (PR#381)"> </td> </tr> <tr> <th class="required"><a href="http://docs.python.org/devguide/triaging.html#type" target="_blank">Type</a>:</th> <td>enhancement</td> <th><a href="http://docs.python.org/devguide/triaging.html#stage" target="_blank">Stage</a>:</th> <td></td> </tr> <tr> <th><a href="http://docs.python.org/devguide/triaging.html#components" target="_blank">Components</a>:</th> <td>Library (Lib)</td> <th><a href="http://docs.python.org/devguide/triaging.html#versions" target="_blank">Versions</a>:</th> <td></td> </tr> </table> </fieldset> <fieldset><legend>process</legend> <table class="form"> <tr> <th><a href="http://docs.python.org/devguide/triaging.html#status" target="_blank">Status</a>:</th> <td>closed</td> <th><a href="http://docs.python.org/devguide/triaging.html#resolution" target="_blank">Resolution</a>:</th> <td>fixed</td> </tr> <tr> <th> <a href="http://docs.python.org/devguide/triaging.html#dependencies" target="_blank">Dependencies</a>: </th> <td> </td> <th><a href="http://docs.python.org/devguide/triaging.html#superseder" target="_blank">Superseder</a>:</th> <td> </td> </tr> <tr> <th> <a href="http://docs.python.org/devguide/triaging.html#assigned-to" target="_blank">Assigned To</a>: </th> <td> barry </td> <th> <a href="http://docs.python.org/devguide/triaging.html#nosy-list" target="_blank">Nosy List</a>: </th> <td> barry, naris </td> </tr> <tr> <th> <a href="http://docs.python.org/devguide/triaging.html#priority" target="_blank">Priority</a>: </th> <td>normal</td> <th> <a href="http://docs.python.org/devguide/triaging.html#keywords" target="_blank">Keywords</a>: </th> <td></td> </tr> </table> </fieldset> </form> <p>Created on <strong>2000-07-31 21:13</strong> by <strong>anonymous</strong>, last changed <strong>2022-04-10 16:02</strong> by <strong>admin</strong>. This issue is now <strong style="color:#00F; background-color:inherit;">closed</strong>.</p> <table class="messages"> <tr><th colspan="4" class="header">Messages (5)</th></tr> <tr> <th> <a href="#msg357" id="msg357">msg357</a> - <a href="msg357">(view)</a></th> <th>Author: Nobody/Anonymous (nobody)</th> <th>Date: 2000-07-31 21:13</th> </tr> <tr> <td colspan="4" class="content"> <pre>Jitterbug-Id: 381 Submitted-By: <a href="mailto:naris@ensim.com">naris@ensim.com</a> Date: Mon, 3 Jul 2000 21:29:25 -0400 (EDT) Version: 1.5.2 OS: RedHat 6.1 read_lines_to_outerboundary chews up memory. there's a while (1) loop that does a readline(), which is probably not good, as you could have a binary file that doesn't have an endline til the end, and thus readline would read the entire file into memory. i did the following: dd'd a 100MB /dev/zero file and used zope to upload it. it chewed up a large amount of memory. there seems to be bug reports about text files and windows systems (and needing to use python -u), but this problem is unrelated. i am using a unix browser -> unix web server. ==================================================================== Audit trail: Tue Jul 11 08:24:22 2000 guido moved from incoming to open</pre> </td> </tr> <tr> <th> <a href="#msg358" id="msg358">msg358</a> - <a href="msg358">(view)</a></th> <th>Author: Naris Siamwalla (naris)</th> <th>Date: 2000-08-24 19:38</th> </tr> <tr> <td colspan="4" class="content"> <pre>there's a bug i submitted to zope that deals with a second problem with read_lines_to_outerboundary, which will also cause a memory leak. the read_lines_to_outerboundary has an unnecessary line self.lines.append(line). removing will fix it this second problem.</pre> </td> </tr> <tr> <th> <a href="#msg359" id="msg359">msg359</a> - <a href="msg359">(view)</a></th> <th>Author: Barry A. Warsaw (barry) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2000-09-25 22:06</th> </tr> <tr> <td colspan="4" class="content"> <pre>Agreed that readlines() isn't the best thing to use in the face of large binary uploads. I've added this feature request to <a href="https://www.python.org/dev/peps/pep-0042/">PEP 42</a> (and am closing this bug report). Removing self.lines might be more problematic, since although it's not documented as part of the public interface for FieldStorage, it is possible that existing code is using it, and that code would break.</pre> </td> </tr> <tr> <th> <a href="#msg360" id="msg360">msg360</a> - <a href="msg360">(view)</a></th> <th>Author: Naris Siamwalla (naris)</th> <th>Date: 2000-09-27 10:38</th> </tr> <tr> <td colspan="4" class="content"> <pre>i feel the self.lines.append() is is a pretty serious problem...have you got any second opinions on how to proceed? if its not part of the public interface for FieldStorage, then wouldn't it be somewhat safe to remove it (you might anger some people, but they could always stick to using the old cgi.py)? my argument is that if they changed the socket connect/bind functions to reject connect(host,port) in favor of connect((host,port)), then we should certainly be able to remove this self.lines.append() :-) what do you think? </pre> </td> </tr> <tr> <th> <a href="#msg361" id="msg361">msg361</a> - <a href="msg361">(view)</a></th> <th>Author: Barry A. Warsaw (barry) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2000-11-06 18:39</th> </tr> <tr> <td colspan="4" class="content"> <pre>We've decided to remove the self.lines attribute for Python 2.1. This part of the bug report is identical to 119806. No one can remember why self.lines was there in the first place, so we'll remove it and see who complains.</pre> </td> </tr> </table> <table class="history table table-condensed table-striped"><tr><th colspan="4" class="header"> History </th></tr><tr> <th>Date</th> <th>User</th> <th>Action</th> <th>Args</th> </tr> <tr><td>2022-04-10 16:02:11</td><td>admin</td><td>set</td><td>github: 32744</td></tr> <tr><td>2000-07-31 21:13:00</td><td>anonymous</td><td>create</td><td></td></tr> </table> </div> </div>  <div id="footer"> <div id="credits"> Supported by <a href="https://python.org/psf-landing/" title="The Python Software Foundation">The Python Software Foundation</a>, <br> Powered by <a href="http://roundup.sourceforge.net" title="Powered by the Roundup Issue Tracker">Roundup</a> </div>  Copyright © 1990-2022, <a href="http://python.org/psf">Python Software Foundation</a><br /> <a href="http://python.org/about/legal">Legal Statements</a> </div>  </div>  </div>  </body> </html>

CINXE.COM

Issue 210674: memory bloat in binary file upload (PR#381) - Python tracker