CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 528 – Change Windows console encoding to UTF-8 | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0528/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 528 – Change Windows console encoding to UTF-8 | peps.python.org'> <meta property="og:description" content="Historically, Python uses the ANSI APIs for interacting with the Windows operating system, often via C Runtime functions. However, these have been long discouraged in favor of the UTF-16 APIs. Within the operating system, all text is represented as UTF-..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0528/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="Historically, Python uses the ANSI APIs for interacting with the Windows operating system, often via C Runtime functions. However, these have been long discouraged in favor of the UTF-16 APIs. Within the operating system, all text is represented as UTF-..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 528</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 528 – Change Windows console encoding to UTF-8</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Steve Dower <steve.dower at python.org></dd> <dt class="field-even">Status:</dt> <dd class="field-even"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-odd">Type:</dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created:</dt> <dd class="field-even">27-Aug-2016</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">3.6</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even">01-Sep-2016, 04-Sep-2016</dd> <dt class="field-odd">Resolution:</dt> <dd class="field-odd"><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2016-September/146278.html">Python-Dev message</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#specific-changes">Specific Changes</a><ul> <li><a class="reference internal" href="#add-io-windowsconsoleio">Add _io.WindowsConsoleIO</a></li> <li><a class="reference internal" href="#add-pyos-windowsconsolereadline">Add _PyOS_WindowsConsoleReadline</a></li> <li><a class="reference internal" href="#add-legacy-mode">Add legacy mode</a></li> </ul> </li> <li><a class="reference internal" href="#alternative-approaches">Alternative Approaches</a></li> <li><a class="reference internal" href="#code-that-may-break">Code that may break</a><ul> <li><a class="reference internal" href="#assuming-stdin-stdout-encoding">Assuming stdin/stdout encoding</a></li> <li><a class="reference internal" href="#incorrectly-using-the-raw-object">Incorrectly using the raw object</a></li> <li><a class="reference internal" href="#using-the-raw-object-with-small-buffers">Using the raw object with small buffers</a></li> </ul> </li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> Historically, Python uses the ANSI APIs for interacting with the Windows operating system, often via C Runtime functions. However, these have been long discouraged in favor of the UTF-16 APIs. Within the operating system, all text is represented as UTF-16, and the ANSI APIs perform encoding and decoding using the active code page. This PEP proposes changing the default standard stream implementation on Windows to use the Unicode APIs. This will allow users to print and input the full range of Unicode characters at the default Windows console. This also requires a subtle change to how the tokenizer parses text from readline hooks. </section> <section id="specific-changes"> <h2><a class="toc-backref" href="#specific-changes" role="doc-backlink">Specific Changes</a></h2> <section id="add-io-windowsconsoleio"> <h3><a class="toc-backref" href="#add-io-windowsconsoleio" role="doc-backlink">Add _io.WindowsConsoleIO</a></h3> Currently an instance of <code class="docutils literal notranslate">_io.FileIO</code> is used to wrap the file descriptors representing standard input, output and error. We add a new class (implemented in C) <code class="docutils literal notranslate">_io.WindowsConsoleIO</code> that acts as a raw IO object using the Windows console functions, specifically, <code class="docutils literal notranslate">ReadConsoleW</code> and <code class="docutils literal notranslate">WriteConsoleW</code>. This class will be used when the legacy-mode flag is not in effect, when opening a standard stream by file descriptor and the stream is a console buffer rather than a redirected file. Otherwise, <code class="docutils literal notranslate">_io.FileIO</code> will be used as it is today. This is a raw (bytes) IO class that requires text to be passed encoded with utf-8, which will be decoded to utf-16-le and passed to the Windows APIs. Similarly, bytes read from the class will be provided by the operating system as utf-16-le and converted into utf-8 when returned to Python. The use of an ASCII compatible encoding is required to maintain compatibility with code that bypasses the <code class="docutils literal notranslate">TextIOWrapper</code> and directly writes ASCII bytes to the standard streams (for example, <a class="reference external" href="https://github.com/twisted/twisted/blob/trunk/src/twisted/test/process_stdinreader.py">Twisted’s process_stdinreader.py</a>). Code that assumes a particular encoding for the standard streams other than ASCII will likely break. </section> <section id="add-pyos-windowsconsolereadline"> <h3><a class="toc-backref" href="#add-pyos-windowsconsolereadline" role="doc-backlink">Add _PyOS_WindowsConsoleReadline</a></h3> To allow Unicode entry at the interactive prompt, a new readline hook is required. The existing <code class="docutils literal notranslate">PyOS_StdioReadline</code> function will delegate to the new <code class="docutils literal notranslate">_PyOS_WindowsConsoleReadline</code> function when reading from a file descriptor that is a console buffer and the legacy-mode flag is not in effect (the logic should be identical to above). Since the readline interface is required to return an 8-bit encoded string with no embedded nulls, the <code class="docutils literal notranslate">_PyOS_WindowsConsoleReadline</code> function transcodes from utf-16-le as read from the operating system into utf-8. The function <code class="docutils literal notranslate">PyRun_InteractiveOneObject</code> which currently obtains the encoding from <code class="docutils literal notranslate">sys.stdin</code> will select utf-8 unless the legacy-mode flag is in effect. This may require readline hooks to change their encodings to utf-8, or to require legacy-mode for correct behaviour. </section> <section id="add-legacy-mode"> <h3><a class="toc-backref" href="#add-legacy-mode" role="doc-backlink">Add legacy mode</a></h3> Launching Python with the environment variable <code class="docutils literal notranslate">PYTHONLEGACYWINDOWSSTDIO</code> set will enable the legacy-mode flag, which completely restores the previous behaviour. </section> </section> <section id="alternative-approaches"> <h2><a class="toc-backref" href="#alternative-approaches" role="doc-backlink">Alternative Approaches</a></h2> The <a class="reference external" href="https://pypi.org/project/win_unicode_console/">win_unicode_console package</a> is a pure-Python alternative to changing the default behaviour of the console. It implements essentially the same modifications as described here using pure Python code. </section> <section id="code-that-may-break"> <h2><a class="toc-backref" href="#code-that-may-break" role="doc-backlink">Code that may break</a></h2> The following code patterns may break or see different behaviour as a result of this change. All of these code samples require explicitly choosing to use a raw file object in place of a more convenient wrapper that would prevent any visible change. <section id="assuming-stdin-stdout-encoding"> <h3><a class="toc-backref" href="#assuming-stdin-stdout-encoding" role="doc-backlink">Assuming stdin/stdout encoding</a></h3> Code that assumes that the encoding required by <code class="docutils literal notranslate">sys.stdin.buffer</code> or <code class="docutils literal notranslate">sys.stdout.buffer</code> is <code class="docutils literal notranslate">'mbcs'</code> or a more specific encoding may currently be working by chance, but could encounter issues under this change. For example: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> sys.stdout.buffer.write(text.encode('mbcs')) >>> r = sys.stdin.buffer.read(16).decode('cp437') </pre></div> </div> To correct this code, the encoding specified on the <code class="docutils literal notranslate">TextIOWrapper</code> should be used, either implicitly or explicitly: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> # Fix 1: Use wrapper correctly >>> sys.stdout.write(text) >>> r = sys.stdin.read(16) >>> # Fix 2: Use encoding explicitly >>> sys.stdout.buffer.write(text.encode(sys.stdout.encoding)) >>> r = sys.stdin.buffer.read(16).decode(sys.stdin.encoding) </pre></div> </div> </section> <section id="incorrectly-using-the-raw-object"> <h3><a class="toc-backref" href="#incorrectly-using-the-raw-object" role="doc-backlink">Incorrectly using the raw object</a></h3> Code that uses the raw IO object and does not correctly handle partial reads and writes may be affected. This is particularly important for reads, where the number of characters read will never exceed one-fourth of the number of bytes allowed, as there is no feasible way to prevent input from encoding as much longer utf-8 strings: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> raw_stdin = sys.stdin.buffer.raw >>> data = raw_stdin.read(15) abcdefghijklm b'abc' # data contains at most 3 characters, and never more than 12 bytes # error, as "defghijklm\r\n" is passed to the interactive prompt </pre></div> </div> To correct this code, the buffered reader/writer should be used, or the caller should continue reading until its buffer is full: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> # Fix 1: Use the buffered reader/writer >>> stdin = sys.stdin.buffer >>> data = stdin.read(15) abcedfghijklm b'abcdefghijklm\r\n' >>> # Fix 2: Loop until enough bytes have been read >>> raw_stdin = sys.stdin.buffer.raw >>> b = b'' >>> while len(b) < 15: ... b += raw_stdin.read(15) abcedfghijklm b'abcdefghijklm\r\n' </pre></div> </div> </section> <section id="using-the-raw-object-with-small-buffers"> <h3><a class="toc-backref" href="#using-the-raw-object-with-small-buffers" role="doc-backlink">Using the raw object with small buffers</a></h3> Code that uses the raw IO object and attempts to read less than four characters will now receive an error. Because it’s possible that any single character may require up to four bytes when represented in utf-8, requests must fail: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> raw_stdin = sys.stdin.buffer.raw >>> data = raw_stdin.read(3) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: must read at least 4 bytes </pre></div> </div> The only workaround is to pass a larger buffer: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> # Fix: Request at least four bytes >>> raw_stdin = sys.stdin.buffer.raw >>> data = raw_stdin.read(4) a b'a' >>> >>> </pre></div> </div> (The extra <code class="docutils literal notranslate">>>></code> is due to the newline remaining in the input buffer and is expected in this situation.) </section> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0528.rst">https://github.com/python/peps/blob/main/peps/pep-0528.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0528.rst">2025-02-01 08:59:27 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#specific-changes">Specific Changes</a><ul> <li><a class="reference internal" href="#add-io-windowsconsoleio">Add _io.WindowsConsoleIO</a></li> <li><a class="reference internal" href="#add-pyos-windowsconsolereadline">Add _PyOS_WindowsConsoleReadline</a></li> <li><a class="reference internal" href="#add-legacy-mode">Add legacy mode</a></li> </ul> </li> <li><a class="reference internal" href="#alternative-approaches">Alternative Approaches</a></li> <li><a class="reference internal" href="#code-that-may-break">Code that may break</a><ul> <li><a class="reference internal" href="#assuming-stdin-stdout-encoding">Assuming stdin/stdout encoding</a></li> <li><a class="reference internal" href="#incorrectly-using-the-raw-object">Incorrectly using the raw object</a></li> <li><a class="reference internal" href="#using-the-raw-object-with-small-buffers">Using the raw object with small buffers</a></li> </ul> </li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0528.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>