CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 400 – Deprecate codecs.StreamReader and codecs.StreamWriter | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0400/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 400 – Deprecate codecs.StreamReader and codecs.StreamWriter | peps.python.org'> <meta property="og:description" content="io.TextIOWrapper and codecs.StreamReaderWriter offer the same API 1. TextIOWrapper has more features and is faster than StreamReaderWriter. Duplicate code means that bugs should be fixed twice and that we may have subtle differences between the two impl..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0400/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="io.TextIOWrapper and codecs.StreamReaderWriter offer the same API 1. TextIOWrapper has more features and is faster than StreamReaderWriter. Duplicate code means that bugs should be fixed twice and that we may have subtle differences between the two impl..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 400</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> <span class="visually-hidden">Toggle light / dark / auto colour theme</span> </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 400 – Deprecate codecs.StreamReader and codecs.StreamWriter</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author<span class="colon">:</span></dt> <dd class="field-odd">Victor Stinner <vstinner at python.org></dd> <dt class="field-even">Status<span class="colon">:</span></dt> <dd class="field-even"><abbr title="Inactive draft that may be taken up again at a later time">Deferred</abbr></dd> <dt class="field-odd">Type<span class="colon">:</span></dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created<span class="colon">:</span></dt> <dd class="field-even">28-May-2011</dd> <dt class="field-odd">Python-Version<span class="colon">:</span></dt> <dd class="field-odd">3.3</dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#pep-deferral">PEP Deferral</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#streamreader-and-streamwriter-issues">StreamReader and StreamWriter issues</a></li> <li><a class="reference internal" href="#textiowrapper-features">TextIOWrapper features</a></li> <li><a class="reference internal" href="#textiowrapper-issues">TextIOWrapper issues</a></li> <li><a class="reference internal" href="#possible-improvements-of-streamreader-and-streamwriter">Possible improvements of StreamReader and StreamWriter</a></li> <li><a class="reference internal" href="#usage-of-streamreader-and-streamwriter">Usage of StreamReader and StreamWriter</a></li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a><ul> <li><a class="reference internal" href="#keep-the-public-api-codecs-open">Keep the public API, codecs.open</a></li> <li><a class="reference internal" href="#deprecate-streamreader-and-streamwriter">Deprecate StreamReader and StreamWriter</a></li> </ul> </li> <li><a class="reference internal" href="#alternative-approach">Alternative Approach</a></li> <li><a class="reference internal" href="#appendix-a-issues-with-stateful-codecs">Appendix A: Issues with stateful codecs</a><ul> <li><a class="reference internal" href="#stateful-codecs">Stateful codecs</a></li> <li><a class="reference internal" href="#read-and-seek-0">Read and seek(0)</a></li> <li><a class="reference internal" href="#seek-n">seek(n)</a></li> <li><a class="reference internal" href="#append-mode">Append mode</a></li> </ul> </li> <li><a class="reference internal" href="#links">Links</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> <li><a class="reference internal" href="#footnotes">Footnotes</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> <p>io.TextIOWrapper and codecs.StreamReaderWriter offer the same API <a class="footnote-reference brackets" href="#f1" id="id1">[1]</a>. TextIOWrapper has more features and is faster than StreamReaderWriter. Duplicate code means that bugs should be fixed twice and that we may have subtle differences between the two implementations.</p> <p>The codecs module was introduced in Python 2.0 (see the <a class="pep reference internal" href="../pep-0100/" title="PEP 100 – Python Unicode Integration">PEP 100</a>). The io module was introduced in Python 2.6 and 3.0 (see the <a class="pep reference internal" href="../pep-3116/" title="PEP 3116 – New I/O">PEP 3116</a>), and reimplemented in C in Python 2.7 and 3.1.</p> </section> <section id="pep-deferral"> <h2><a class="toc-backref" href="#pep-deferral" role="doc-backlink">PEP Deferral</a></h2> <p>Further exploration of the concepts covered in this PEP has been deferred for lack of a current champion interested in promoting the goals of the PEP and collecting and incorporating feedback, and with sufficient available time to do so effectively.</p> </section> <section id="motivation"> <h2><a class="toc-backref" href="#motivation" role="doc-backlink">Motivation</a></h2> <p>When the Python I/O model was updated for 3.0, the concept of a “stream-with-known-encoding” was introduced in the form of io.TextIOWrapper. As this class is critical to the performance of text-based I/O in Python 3, this module has an optimised C version which is used by CPython by default. Many corner cases in handling buffering, stateful codecs and universal newlines have been dealt with since the release of Python 3.0.</p> <p>This new interface overlaps heavily with the legacy codecs.StreamReader, codecs.StreamWriter and codecs.StreamReaderWriter interfaces that were part of the original codec interface design in <a class="pep reference internal" href="../pep-0100/" title="PEP 100 – Python Unicode Integration">PEP 100</a>. These interfaces are organised around the principle of an encoding with an associated stream (i.e. the reverse of arrangement in the io module), so the original <a class="pep reference internal" href="../pep-0100/" title="PEP 100 – Python Unicode Integration">PEP 100</a> design required that codec writers provide appropriate StreamReader and StreamWriter implementations in addition to the core codec encode() and decode() methods. This places a heavy burden on codec authors providing these specialised implementations to correctly handle many of the corner cases (see <a class="reference internal" href="#pep-400-appendix-a">Appendix A</a>) that have now been dealt with by io.TextIOWrapper. While deeper integration between the codec and the stream allows for additional optimisations in theory, these optimisations have in practice either not been carried out and else the associated code duplication means that the corner cases that have been fixed in io.TextIOWrapper are still not handled correctly in the various StreamReader and StreamWriter implementations.</p> <p>Accordingly, this PEP proposes that:</p> <ul class="simple"> <li>codecs.open() be updated to delegate to the builtin open() in Python 3.3;</li> <li>the legacy codecs.Stream* interfaces, including the streamreader and streamwriter attributes of codecs.CodecInfo be deprecated in Python 3.3.</li> </ul> </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> <section id="streamreader-and-streamwriter-issues"> <h3><a class="toc-backref" href="#streamreader-and-streamwriter-issues" role="doc-backlink">StreamReader and StreamWriter issues</a></h3> <ul class="simple"> <li>StreamReader is unable to translate newlines.</li> <li>StreamWriter doesn’t support “line buffering” (flush if the input text contains a newline).</li> <li>StreamReader classes of the CJK encodings (e.g. GB18030) only supports UNIX newlines (’\n’).</li> <li>StreamReader and StreamWriter are stateful codecs but don’t expose functions to control their state (getstate() or setstate()). Each codec has to handle corner cases, see <a class="reference internal" href="#pep-400-appendix-a">Appendix A</a>.</li> <li>StreamReader and StreamWriter are very similar to IncrementalReader and IncrementalEncoder, some code is duplicated for stateful codecs (e.g. UTF-16).</li> <li>Each codec has to reimplement its own StreamReader and StreamWriter class, even if it’s trivial (just call the encoder/decoder).</li> <li>codecs.open(filename, “r”) creates an io.TextIOWrapper object.</li> <li>No codec implements an optimized method in StreamReader or StreamWriter based on the specificities of the codec.</li> </ul> <p>Issues in the bug tracker:</p> <ul class="simple"> <li><a class="reference external" href="http://bugs.python.org/issue5445">Issue #5445</a> (2009-03-08): codecs.StreamWriter.writelines problem when passed generator</li> <li><a class="reference external" href="http://bugs.python.org/issue7262">Issue #7262:</a> (2009-11-04): codecs.open() + eol (windows)</li> <li><a class="reference external" href="http://bugs.python.org/issue8260">Issue #8260</a> (2010-03-29): When I use codecs.open(…) and f.readline() follow up by f.read() return bad result</li> <li><a class="reference external" href="http://bugs.python.org/issue8630">Issue #8630</a> (2010-05-05): Keepends param in codec readline(s)</li> <li><a class="reference external" href="http://bugs.python.org/issue10344">Issue #10344</a> (2010-11-06): codecs.readline doesn’t care buffering</li> <li><a class="reference external" href="http://bugs.python.org/issue11461">Issue #11461</a> (2011-03-10): Reading UTF-16 with codecs.readline() breaks on surrogate pairs</li> <li><a class="reference external" href="http://bugs.python.org/issue12446">Issue #12446</a> (2011-06-30): StreamReader Readlines behavior odd</li> <li><a class="reference external" href="http://bugs.python.org/issue12508">Issue #12508</a> (2011-07-06): Codecs Anomaly</li> <li><a class="reference external" href="http://bugs.python.org/issue12512">Issue #12512</a> (2011-07-07): codecs: StreamWriter issues with stateful codecs after a seek or with append mode</li> <li><a class="reference external" href="http://bugs.python.org/issue12513">Issue #12513</a> (2011-07-07): codec.StreamReaderWriter: issues with interlaced read-write</li> </ul> </section> <section id="textiowrapper-features"> <h3><a class="toc-backref" href="#textiowrapper-features" role="doc-backlink">TextIOWrapper features</a></h3> <ul class="simple"> <li>TextIOWrapper supports any kind of newline, including translating newlines (to UNIX newlines), to read and write.</li> <li>TextIOWrapper reuses codecs incremental encoders and decoders (no duplication of code).</li> <li>The io module (TextIOWrapper) is faster than the codecs module (StreamReader). It is implemented in C, whereas codecs is implemented in Python.</li> <li>TextIOWrapper has a readahead algorithm which speeds up small reads: read character by character or line by line (io is 10x through 25x faster than codecs on these operations).</li> <li>TextIOWrapper has a write buffer.</li> <li>TextIOWrapper.tell() is optimized.</li> <li>TextIOWrapper supports random access (read+write) using a single class which permit to optimize interlaced read-write (but no such optimization is implemented).</li> </ul> </section> <section id="textiowrapper-issues"> <h3><a class="toc-backref" href="#textiowrapper-issues" role="doc-backlink">TextIOWrapper issues</a></h3> <ul class="simple"> <li><a class="reference external" href="http://bugs.python.org/issue12215">Issue #12215</a> (2011-05-30): TextIOWrapper: issues with interlaced read-write</li> </ul> </section> <section id="possible-improvements-of-streamreader-and-streamwriter"> <h3><a class="toc-backref" href="#possible-improvements-of-streamreader-and-streamwriter" role="doc-backlink">Possible improvements of StreamReader and StreamWriter</a></h3> <p>By adding codec state read/write functions to the StreamReader and StreamWriter classes, it will become possible to fix issues with stateful codecs in a base class instead of in each stateful StreamReader and StreamWriter classes.</p> <p>It would be possible to change StreamReader and StreamWriter to make them use IncrementalDecoder and IncrementalEncoder.</p> <p>A codec can implement variants which are optimized for the specific encoding or intercept certain stream methods to add functionality or improve the encoding/decoding performance. TextIOWrapper cannot implement such optimization, but TextIOWrapper uses incremental encoders and decoders and uses read and write buffers, so the overhead of incomplete inputs is low or nul.</p> <p>A lot more could be done for other variable length encoding codecs, e.g. UTF-8, since these often have problems near the end of a read due to missing bytes. The UTF-32-BE/LE codecs could simply multiply the character position by 4 to get the byte position.</p> </section> <section id="usage-of-streamreader-and-streamwriter"> <h3><a class="toc-backref" href="#usage-of-streamreader-and-streamwriter" role="doc-backlink">Usage of StreamReader and StreamWriter</a></h3> <p>These classes are rarely used directly, but indirectly using codecs.open(). They are not used in Python 3 standard library (except in the codecs module).</p> <p>Some projects implement their own codec with StreamReader and StreamWriter, but don’t use these classes.</p> </section> </section> <section id="backwards-compatibility"> <h2><a class="toc-backref" href="#backwards-compatibility" role="doc-backlink">Backwards Compatibility</a></h2> <section id="keep-the-public-api-codecs-open"> <h3><a class="toc-backref" href="#keep-the-public-api-codecs-open" role="doc-backlink">Keep the public API, codecs.open</a></h3> <p>codecs.open() can be replaced by the builtin open() function. open() has a similar API but has also more options. Both functions return file-like objects (same API).</p> <p>codecs.open() was the only way to open a text file in Unicode mode until Python 2.6. Many Python 2 programs uses this function. Removing codecs.open() implies more work to port programs from Python 2 to Python 3, especially projects using the same code base for the two Python versions (without using 2to3 program).</p> <p>codecs.open() is kept for backward compatibility with Python 2.</p> </section> <section id="deprecate-streamreader-and-streamwriter"> <h3><a class="toc-backref" href="#deprecate-streamreader-and-streamwriter" role="doc-backlink">Deprecate StreamReader and StreamWriter</a></h3> <p>Instantiating StreamReader or StreamWriter must emit a DeprecationWarning in Python 3.3. Defining a subclass doesn’t emit a DeprecationWarning.</p> <p>codecs.open() will be changed to reuse the builtin open() function (TextIOWrapper) to read-write text files.</p> </section> </section> <section id="alternative-approach"> <span id="pep-400-appendix-a"></span><h2><a class="toc-backref" href="#alternative-approach" role="doc-backlink">Alternative Approach</a></h2> <p>An alternative to the deprecation of the codecs.Stream* classes is to rename codecs.open() to codecs.open_stream(), and to create a new codecs.open() function reusing open() and so io.TextIOWrapper.</p> </section> <section id="appendix-a-issues-with-stateful-codecs"> <h2><a class="toc-backref" href="#appendix-a-issues-with-stateful-codecs" role="doc-backlink">Appendix A: Issues with stateful codecs</a></h2> <p>It is difficult to use correctly a stateful codec with a stream. Some cases are supported by the codecs module, while io has no more known bug related to stateful codecs. The main difference between the codecs and the io module is that bugs have to be fixed in StreamReader and/or StreamWriter classes of each codec for the codecs module, whereas bugs can be fixed only once in io.TextIOWrapper. Here are some examples of issues with stateful codecs.</p> <section id="stateful-codecs"> <h3><a class="toc-backref" href="#stateful-codecs" role="doc-backlink">Stateful codecs</a></h3> <p>Python supports the following stateful codecs:</p> <ul class="simple"> <li>cp932</li> <li>cp949</li> <li>cp950</li> <li>euc_jis_2004</li> <li>euc_jisx2003</li> <li>euc_jp</li> <li>euc_kr</li> <li>gb18030</li> <li>gbk</li> <li>hz</li> <li>iso2022_jp</li> <li>iso2022_jp_1</li> <li>iso2022_jp_2</li> <li>iso2022_jp_2004</li> <li>iso2022_jp_3</li> <li>iso2022_jp_ext</li> <li>iso2022_kr</li> <li>shift_jis</li> <li>shift_jis_2004</li> <li>shift_jisx0213</li> <li>utf_8_sig</li> <li>utf_16</li> <li>utf_32</li> </ul> </section> <section id="read-and-seek-0"> <h3><a class="toc-backref" href="#read-and-seek-0" role="doc-backlink">Read and seek(0)</a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-16'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'abc'</span><span class="p">)</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'def'</span><span class="p">)</span> <span class="n">f</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="k">assert</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'abcdef'</span> <span class="n">f</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="k">assert</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'abcdef'</span> </pre></div> </div> <p>The io and codecs modules support this usecase correctly.</p> </section> <section id="seek-n"> <h3><a class="toc-backref" href="#seek-n" role="doc-backlink">seek(n)</a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-16'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'abc'</span><span class="p">)</span> <span class="n">pos</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">tell</span><span class="p">()</span> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-16'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="n">pos</span><span class="p">)</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'def'</span><span class="p">)</span> <span class="n">f</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'###'</span><span class="p">)</span> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-16'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="k">assert</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'###def'</span> </pre></div> </div> <p>The io module supports this usecase, whereas codecs fails because it writes a new BOM on the second write (<a class="reference external" href="http://bugs.python.org/issue12512">issue #12512</a>).</p> </section> <section id="append-mode"> <h3><a class="toc-backref" href="#append-mode" role="doc-backlink">Append mode</a></h3> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-16'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'abc'</span><span class="p">)</span> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">'a'</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-16'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'def'</span><span class="p">)</span> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-16'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="k">assert</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'abcdef'</span> </pre></div> </div> <p>The io module supports this usecase, whereas codecs fails because it writes a new BOM on the second write (<a class="reference external" href="http://bugs.python.org/issue12512">issue #12512</a>).</p> </section> </section> <section id="links"> <h2><a class="toc-backref" href="#links" role="doc-backlink">Links</a></h2> <ul class="simple"> <li><a class="pep reference internal" href="../pep-0100/" title="PEP 100 – Python Unicode Integration">PEP 100: Python Unicode Integration</a></li> <li><a class="pep reference internal" href="../pep-3116/" title="PEP 3116 – New I/O">PEP 3116: New I/O</a></li> <li><a class="reference external" href="http://bugs.python.org/issue8796">Issue #8796: Deprecate codecs.open()</a></li> <li><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2011-May/111591.html">[python-dev] Deprecate codecs.open() and StreamWriter/StreamReader</a></li> </ul> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> <p>This document has been placed in the public domain.</p> </section> <section id="footnotes"> <h2><a class="toc-backref" href="#footnotes" role="doc-backlink">Footnotes</a></h2> <aside class="footnote-list brackets"> <aside class="footnote brackets" id="f1" role="doc-footnote"> <dt class="label" id="f1">[<a href="#id1">1</a>]</dt> <dd>StreamReaderWriter has two more attributes than TextIOWrapper, reader and writer.</aside> </aside> </section> </section> <hr class="docutils" /> <p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0400.rst">https://github.com/python/peps/blob/main/peps/pep-0400.rst</a></p> <p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0400.rst">2025-02-01 08:59:27 GMT</a></p> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#pep-deferral">PEP Deferral</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#streamreader-and-streamwriter-issues">StreamReader and StreamWriter issues</a></li> <li><a class="reference internal" href="#textiowrapper-features">TextIOWrapper features</a></li> <li><a class="reference internal" href="#textiowrapper-issues">TextIOWrapper issues</a></li> <li><a class="reference internal" href="#possible-improvements-of-streamreader-and-streamwriter">Possible improvements of StreamReader and StreamWriter</a></li> <li><a class="reference internal" href="#usage-of-streamreader-and-streamwriter">Usage of StreamReader and StreamWriter</a></li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a><ul> <li><a class="reference internal" href="#keep-the-public-api-codecs-open">Keep the public API, codecs.open</a></li> <li><a class="reference internal" href="#deprecate-streamreader-and-streamwriter">Deprecate StreamReader and StreamWriter</a></li> </ul> </li> <li><a class="reference internal" href="#alternative-approach">Alternative Approach</a></li> <li><a class="reference internal" href="#appendix-a-issues-with-stateful-codecs">Appendix A: Issues with stateful codecs</a><ul> <li><a class="reference internal" href="#stateful-codecs">Stateful codecs</a></li> <li><a class="reference internal" href="#read-and-seek-0">Read and seek(0)</a></li> <li><a class="reference internal" href="#seek-n">seek(n)</a></li> <li><a class="reference internal" href="#append-mode">Append mode</a></li> </ul> </li> <li><a class="reference internal" href="#links">Links</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> <li><a class="reference internal" href="#footnotes">Footnotes</a></li> </ul> <br> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0400.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>