Issue 12215: TextIOWrapper: issues with interlaced read-write

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <title> Issue 12215: TextIOWrapper: issues with interlaced read-write - Python tracker </title> <link rel="shortcut icon" href="@@file/favicon.ico" /> <link rel="stylesheet" type="text/css" href="@@file/main.css" /> <link rel="stylesheet" type="text/css" href="@@file/style.css" /> <link rel="search" type="application/opensearchdescription+xml" href="@@file/osd.xml" title="Python bug tracker search" /> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <script nonce="ce858510e51906293a17faef688562b1dd4eba6d40f967eaed41f5b6d7fb5590" type="text/javascript"> submitted = false; function submit_once() { if (submitted) { alert("Your request is being processed.\nPlease be patient."); return false; } submitted = true; return true; } function help_window(helpurl, width, height) { HelpWin = window.open('https://bugs.python.org/' + helpurl, 'RoundupHelpWindow', 'scrollbars=yes,resizable=yes,toolbar=no,height='+height+',width='+width); HelpWin.focus () } </script> <script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.6.2/jquery.min.js"></script> <script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.15/jquery-ui.js"></script> <script type="text/javascript" src="@@file/issue.item.js"></script> <link rel="stylesheet" type="text/css" href="https://ajax.googleapis.com/ajax/libs/jqueryui/1.8/themes/smoothness/jquery-ui.css" /> </head> <body>  <h1 id="logoheader"> <a accesskey="1" href="." id="logolink"> <img src="@@file/python-logo.gif" alt="homepage" border="0" id="logo" /></a> </h1> <div id="utility-menu">  <div id="searchbox"> <form name="searchform" method="get" action="issue" id="searchform"> <div id="search"> <input type="hidden" name="@columns" value="id,github,activity,title,creator,assignee,status,type" /> <input type="hidden" name="@sort" value="-activity" /> <input type="hidden" name="@filter" value="status" /> <input type="hidden" name="@action" value="searchid" /> <input type="hidden" name="ignore" value="file:content" /> <input class="input-text" id="search-text" name="@search_text" size="10" /> <input type="submit" id="submit" value="search" name="submit" class="input-button" /> <input type="radio" name="status" id="status_notresolved" value="-1,1,3" /> <label for="status_notresolved">open</label> <input type="radio" name="status" checked="checked" id="status_all" value="-1,1,2,3" /> <label for="status_all">all</label> </div> </form> </div> </div> <div id="left-hand-navigation">  <div id="menu"> <ul class="level-one"> <li><a href="https://www.python.org/" title="Go to the Python homepage">Python Home</a></li> <li><a href="https://www.python.org/about/" title="About The Python Language">About</a></li> <li><a href="https://www.python.org/blogs/" title="">News</a></li> <li><a href="https://www.python.org/doc/" title="">Documentation</a></li> <li><a href="https://www.python.org/downloads/" title="">Downloads</a></li> <li><a href="https://www.python.org/community/" title="">Community</a></li> <li><a href="https://www.python.org/psf/" title="Python Software Foundation">Foundation</a></li> <li><a href="https://devguide.python.org/" title="Python Developer's Guide">Developer's Guide</a></li> <li class="selected"><a href="." class="selected" title="Python Issue Tracker">Issue Tracker</a> <ul class="level-two"> <li> <strong>Issues</strong> <ul class="level-three"> <li><a href="issue?@template=search&status=1">Search</a></li> <li><a href="issue?@action=random">Random Issue</a></li> <li> <form method="post" action="issue12215"> <input type="submit" class="form-small" value="Show issue:" /> <input class="form-small" size="4" type="text" name="@number" /> <input type="hidden" name="@type" value="issue" /> <input type="hidden" name="@action" value="show" /> </form> </li> </ul> </li> <li> <strong>Summaries</strong> <ul class="level-three"> <li> <a href="issue?status=1&@sort=-activity&@columns=id%2Cgithub%2Cactivity%2Ctitle%2Ccreator%2Cstatus&@dispname=Issues%20with%20patch&@startwith=0&@group=priority&keywords=2&@action=search&@filter=&@pagesize=50">Issues with patch</a> </li> <li> <a href="issue?status=1&@sort=-activity&@columns=id%2Cgithub%2Cactivity%2Ctitle%2Ccreator%2Cstatus&@dispname=Easy%20issues&@startwith=0&@group=priority&keywords=6&@action=search&@filter=&@pagesize=50">Easy issues</a> </li> <li> <a href="issue?@template=stats">Stats</a> </li> </ul> </li> <li> <strong>User</strong> <form method="post" action="issue12215"> <ul class="level-three"> <li> Login<br /> <input size="10" name="openid_identifier" style="" /><br /> <input size="10" type="password" name="__login_password" /><br /> <input type="hidden" name="@action" value="Login" /> <input type="checkbox" name="remember" id="remember" /> <label for="remember">Remember me?</label><br /> <input class="form-small" type="submit" value="Login" /><br /> <input type="hidden" name="__came_from" value="https://bugs.python.org/issue12215?"> <input type="hidden" name="@sort" value=""/> <input type="hidden" name="@group" value=""/> <input type="hidden" name="@pagesize" value="50"/> <input type="hidden" name="@startwith" value="0"/> </li> <li> </li> <li><a href="user?@template=forgotten">Lost your login?</a></li> </ul> </form> </li> <li> <strong>Administration</strong> <ul class="level-three"> <li> <a href="user?@sort=username">User List</a></li> <li> <a href="user?iscommitter=1&@action=search&@sort=username&@pagesize=300">Committer List</a></li> </ul> </li> <li> <strong>Help</strong> <ul class="level-three"> <li><a href="http://docs.python.org/devguide/triaging.html"> Tracker Documentation</a></li> <li><a href="http://wiki.python.org/moin/TrackerDevelopment"> Tracker Development</a></li> <li><a href="https://github.com/python/psf-infra-meta/issues"> Report Tracker Problem</a></li> </ul> </li> </ul> </li> </ul> </div>  </div>  <div id="content-body"> <div id="body-main"> <div id="content"> <div id="breadcrumb"> Issue12215 </div> <div id="migration-notice"> <div id="migration-images"> <img width="32" src="@@file/python-logo-small.png" /> ➜ <a href="https://github.com/python/cpython/issues"><img width="32" src="@@file/gh-icon.png" /></a> </div> <p>This issue tracker <b>has been migrated to <a href="https://github.com/python/cpython/issues">GitHub</a></b>, and is currently <b>read-only</b>.<br /> For more information, <a title="GitHub FAQs" href="https://devguide.python.org/gh-faq/"> see the GitHub FAQs in the Python's Developer Guide.</a></p> </div> <div> <form method="post" name="itemSynopsis" onsubmit="return submit_once()" enctype="multipart/form-data" action="issue12215"> <div id="gh-issue-link"> <a href="https://github.com/python/cpython/issues/56424"> <img width="32" src="@@file/gh-icon.png" /> <p> <span>This issue has been migrated to GitHub:</span> https://github.com/python/cpython/issues/56424 </p> </a> </div> <fieldset><legend>classification</legend> <table class="form"> <tr> <th class="required"><a href="http://docs.python.org/devguide/triaging.html#title" target="_blank">Title</a>:</th> <td colspan="3"> <span>TextIOWrapper: issues with interlaced read-write</span> <input type="hidden" name="title" value="TextIOWrapper: issues with interlaced read-write"> </td> </tr> <tr> <th class="required"><a href="http://docs.python.org/devguide/triaging.html#type" target="_blank">Type</a>:</th> <td></td> <th><a href="http://docs.python.org/devguide/triaging.html#stage" target="_blank">Stage</a>:</th> <td>resolved</td> </tr> <tr> <th><a href="http://docs.python.org/devguide/triaging.html#components" target="_blank">Components</a>:</th> <td>IO</td> <th><a href="http://docs.python.org/devguide/triaging.html#versions" target="_blank">Versions</a>:</th> <td>Python 3.4, Python 3.5, Python 2.7</td> </tr> </table> </fieldset> <fieldset><legend>process</legend> <table class="form"> <tr> <th><a href="http://docs.python.org/devguide/triaging.html#status" target="_blank">Status</a>:</th> <td>closed</td> <th><a href="http://docs.python.org/devguide/triaging.html#resolution" target="_blank">Resolution</a>:</th> <td>wont fix</td> </tr> <tr> <th> <a href="http://docs.python.org/devguide/triaging.html#dependencies" target="_blank">Dependencies</a>: </th> <td> </td> <th><a href="http://docs.python.org/devguide/triaging.html#superseder" target="_blank">Superseder</a>:</th> <td> </td> </tr> <tr> <th> <a href="http://docs.python.org/devguide/triaging.html#assigned-to" target="_blank">Assigned To</a>: </th> <td> </td> <th> <a href="http://docs.python.org/devguide/triaging.html#nosy-list" target="_blank">Nosy List</a>: </th> <td> Arfrever, martin.panter, pitrou, terry.reedy, vstinner </td> </tr> <tr> <th> <a href="http://docs.python.org/devguide/triaging.html#priority" target="_blank">Priority</a>: </th> <td>normal</td> <th> <a href="http://docs.python.org/devguide/triaging.html#keywords" target="_blank">Keywords</a>: </th> <td>patch</td> </tr> </table> </fieldset> </form> <p>Created on <strong>2011-05-30 12:38</strong> by <strong>vstinner</strong>, last changed <strong>2022-04-11 14:57</strong> by <strong>admin</strong>. This issue is now <strong style="color:#00F; background-color:inherit;">closed</strong>.</p> <table class="files"> <tr><th colspan="5" class="header">Files</th></tr> <tr> <th>File name</th> <th>Uploaded</th> <th>Description</th> <th>Edit</th> </tr> <tr> <td> <a href="file22192/textiowrapper_interlaced_read_write.patch">textiowrapper_interlaced_read_write.patch</a> </td> <td> <span>vstinner</span>, <span>2011-05-30 13:07</span> </td> <td></td> <td> <a href="/review/12215/#ps2741">review</a> </td> </tr> </table> <table class="messages"> <tr><th colspan="4" class="header">Messages (9)</th></tr> <tr> <th> <a href="#msg137260" id="msg137260">msg137260</a> - <a href="msg137260">(view)</a></th> <th>Author: STINNER Victor (vstinner) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2011-05-30 12:38</th> </tr> <tr> <td colspan="4" class="content"> <pre>The following code fails on an assertion error (Python exception for _pyio, C assertion for io): ------------------ with io.BytesIO(b'abcd') as raw: with _pyio.TextIOWrapper(raw, encoding='ascii') as f: f.read(1) f.write('2') f.tell() ------------------ I found this assertion while testing interlaced read-write on TextIOWrapper: ------------------ with io.BytesIO(b'abcd') as raw: with _pyio.TextIOWrapper(raw, encoding='ascii') as f: f.write("1") # read() must call writer.flush() assertEqual(f.read(1), 'b') # write() must rewind the raw stream f.write('2') assertEqual(f.read(), 'd') f.flush() assertEqual(raw.getvalue(), b'1b2d') with io.BytesIO(b'abc') as raw: with _pyio.TextIOWrapper(raw, encoding='ascii') as f: self.assertEqual(f.read(1), b'a') # write() must undo reader readahead f.write(b"2") assertEqual(f.read(1), b'c') f.flush() assertEqual(raw.getvalue(), b'a2c') ------------------ These tests fails on "read, write, read" path: write() breaks TextIOWrapper internal state if a read() occured just before. Note that _pyio.TextIOWrapper.write() contains the following comment... # XXX What if we were just reading? See also the issue <a class="closed" title="[closed] BufferedRandom: issues with interlaced read-write" href="issue12213">#12213</a> for BufferedRandom and BufferedRWPair.</pre> </td> </tr> <tr> <th> <a href="#msg137262" id="msg137262">msg137262</a> - <a href="msg137262">(view)</a></th> <th>Author: STINNER Victor (vstinner) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2011-05-30 13:07</th> </tr> <tr> <td colspan="4" class="content"> <pre>textiowrapper_interlaced_read_write.patch: TextIOWrapper.write() calls self.seek(self.tell()) if it has a decoder or if snapshot is not None. I suppose that we can do better, but at least it does fix this issue. "read(); write(); write()" only calls self.seek(self.tell()) once, at the first write. So I don't think that it changes anything with performances. In which case snapshot can be not different than None, whereas decoder is None?</pre> </td> </tr> <tr> <th> <a href="#msg137596" id="msg137596">msg137596</a> - <a href="msg137596">(view)</a></th> <th>Author: Terry J. Reedy (terry.reedy) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2011-06-03 22:00</th> </tr> <tr> <td colspan="4" class="content"> <pre>For c stdio files, intermixed reads and writes require a file positioning operation. This is a nuisance and source of program bugs. I do not see any such limitation documented for our io module. So for both reasons, it will be nice to not have the limitation in the code. If I understand, the essence of the patch is to do the file positioning automatically internally when needed.</pre> </td> </tr> <tr> <th> <a href="#msg137598" id="msg137598">msg137598</a> - <a href="msg137598">(view)</a></th> <th>Author: STINNER Victor (vstinner) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2011-06-03 22:02</th> </tr> <tr> <td colspan="4" class="content"> <pre>> If I understand, the essence of the patch is to do > the file positioning automatically internally when needed. My patch is just a proposition to fix the issue. I wrote "I suppose that we can do better": self.seek(self.tell()) is more a workaround than a real fix. I don't understand why it does fix this bug :-)</pre> </td> </tr> <tr> <th> <a href="#msg137616" id="msg137616">msg137616</a> - <a href="msg137616">(view)</a></th> <th>Author: Terry J. Reedy (terry.reedy) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2011-06-04 00:53</th> </tr> <tr> <td colspan="4" class="content"> <pre>Perhaps the stdio requirement was based on an underlying OS (*nix?) requirement, which io has to fulfill even if it does not use stdio. Stdio was, I presume, optimized for speed. In the relatively rare case of mixed read/write, it *should* put the burden on the programmer. Python is a bit different.</pre> </td> </tr> <tr> <th> <a href="#msg221439" id="msg221439">msg221439</a> - <a href="msg221439">(view)</a></th> <th>Author: Mark Lawrence (BreamoreBoy) <span title="Contributor form received">*</span></th> <th>Date: 2014-06-24 08:54</th> </tr> <tr> <td colspan="4" class="content"> <pre>Does anybody want to follow up on this? <a class="closed" title="[closed] BufferedRandom: issues with interlaced read-write" href="issue12213">#12213</a> was closed as fixed, <a class="closed" title="[closed] codec.StreamReaderWriter: issues with interlaced read-write" href="issue12513">#12513</a> is still open.</pre> </td> </tr> <tr> <th> <a href="#msg258636" id="msg258636">msg258636</a> - <a href="msg258636">(view)</a></th> <th>Author: Martin Panter (martin.panter) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2016-01-20 02:53</th> </tr> <tr> <td colspan="4" class="content"> <pre>Consider codecs that maintain an internal buffer (UTF-7) or other state (ISO-2022). When you call TextIOWrapper.read() and then tell(), the I think the returned number is supposed to hold the _decoder_ state, so you can seek back and read again. But I don’t think the number holds any _encoder_ state, so seek() cannot set up the encoder properly for these more awkward codecs. I don’t think it is practical to fix this problem using the incremental codec API. You would need to construct the encoder’s state from the decoder. There are a couple of bugs marked as duplicates of this. What are the real-world use cases?</pre> </td> </tr> <tr> <th> <a href="#msg297126" id="msg297126">msg297126</a> - <a href="msg297126">(view)</a></th> <th>Author: STINNER Victor (vstinner) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2017-06-28 01:38</th> </tr> <tr> <td colspan="4" class="content"> <pre>Given that nobody complains the last 9 years (since Python 3.0 was released), I'm not sure that it's worth it to fix this corner case. If you consider that I'm wrong, please reopen the issue ;-)</pre> </td> </tr> <tr> <th> <a href="#msg330367" id="msg330367">msg330367</a> - <a href="msg330367">(view)</a></th> <th>Author: Martin Panter (martin.panter) <span title="Contributor form received">*</span> <img src="@@file/committer.png" title="Python committer" alt="(Python committer)" /></th> <th>Date: 2018-11-24 00:31</th> </tr> <tr> <td colspan="4" class="content"> <pre>For the record, the more recent bug I mentioned was a complaint from 2015 (one and a half years before Victor’s comment). Even if it is not worth supporting writing after reading, the problem could be documented.</pre> </td> </tr> </table> <table class="history table table-condensed table-striped"><tr><th colspan="4" class="header"> History </th></tr><tr> <th>Date</th> <th>User</th> <th>Action</th> <th>Args</th> </tr> <tr><td>2022-04-11 14:57:17</td><td>admin</td><td>set</td><td>github: 56424</td></tr> <tr><td>2019-11-17 23:20:53</td><td>martin.panter</td><td>link</td><td><a rel="nofollow" href="issue38710">issue38710 superseder</a></td></tr> <tr><td>2018-11-24 00:31:23</td><td>martin.panter</td><td>set</td><td>resolution: out of date -> wont fix<br />messages: + <a rel="nofollow" href="msg330367">msg330367</a></td></tr> <tr><td>2017-06-28 01:38:47</td><td>vstinner</td><td>set</td><td>status: open -> closed<br />resolution: out of date<br />messages: + <a rel="nofollow" href="msg297126">msg297126</a><br /><br />stage: resolved</td></tr> <tr><td>2016-01-20 11:25:50</td><td>BreamoreBoy</td><td>set</td><td>nosy: - <a rel="nofollow" href="user12545">BreamoreBoy</a><br /></td></tr> <tr><td>2016-01-20 02:53:42</td><td>martin.panter</td><td>set</td><td>nosy: + <a rel="nofollow" href="user14751">martin.panter</a><br />messages: + <a rel="nofollow" href="msg258636">msg258636</a><br /></td></tr> <tr><td>2015-12-20 09:56:50</td><td>SilentGhost</td><td>link</td><td><a rel="nofollow" href="issue25915">issue25915 superseder</a></td></tr> <tr><td>2015-03-02 15:45:52</td><td>r.david.murray</td><td>link</td><td><a rel="nofollow" href="issue23562">issue23562 superseder</a></td></tr> <tr><td>2014-06-24 14:02:12</td><td>terry.reedy</td><td>set</td><td>versions: + Python 3.4, Python 3.5, - Python 3.2, Python 3.3</td></tr> <tr><td>2014-06-24 08:54:14</td><td>BreamoreBoy</td><td>set</td><td>nosy: + <a rel="nofollow" href="user12545">BreamoreBoy</a><br />messages: + <a rel="nofollow" href="msg221439">msg221439</a><br /></td></tr> <tr><td>2011-07-01 16:51:00</td><td>Arfrever</td><td>set</td><td>nosy: + <a rel="nofollow" href="user9138">Arfrever</a><br /></td></tr> <tr><td>2011-06-04 00:53:06</td><td>terry.reedy</td><td>set</td><td>messages: + <a rel="nofollow" href="msg137616">msg137616</a></td></tr> <tr><td>2011-06-03 22:02:40</td><td>vstinner</td><td>set</td><td>messages: + <a rel="nofollow" href="msg137598">msg137598</a></td></tr> <tr><td>2011-06-03 22:00:05</td><td>terry.reedy</td><td>set</td><td>nosy: + <a rel="nofollow" href="user155">terry.reedy</a><br />messages: + <a rel="nofollow" href="msg137596">msg137596</a><br /></td></tr> <tr><td>2011-05-30 13:08:20</td><td>vstinner</td><td>set</td><td>versions: + Python 2.7</td></tr> <tr><td>2011-05-30 13:08:13</td><td>vstinner</td><td>set</td><td>versions: + Python 3.2</td></tr> <tr><td>2011-05-30 13:07:55</td><td>vstinner</td><td>set</td><td>files: + <a rel="nofollow" href="file22192">textiowrapper_interlaced_read_write.patch</a><br />keywords: + <a rel="nofollow" href="keyword2">patch</a><br />messages: + <a rel="nofollow" href="msg137262">msg137262</a><br /></td></tr> <tr><td>2011-05-30 12:38:03</td><td>vstinner</td><td>create</td><td></td></tr> </table> </div> </div>  <div id="footer"> <div id="credits"> Supported by <a href="https://python.org/psf-landing/" title="The Python Software Foundation">The Python Software Foundation</a>, <br> Powered by <a href="http://roundup.sourceforge.net" title="Powered by the Roundup Issue Tracker">Roundup</a> </div>  Copyright © 1990-2022, <a href="http://python.org/psf">Python Software Foundation</a><br /> <a href="http://python.org/about/legal">Legal Statements</a> </div>  </div>  </div>  </body> </html>

CINXE.COM

Issue 12215: TextIOWrapper: issues with interlaced read-write - Python tracker