CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 529 – Change Windows filesystem encoding to UTF-8 | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0529/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 529 – Change Windows filesystem encoding to UTF-8 | peps.python.org'> <meta property="og:description" content="Historically, Python uses the ANSI APIs for interacting with the Windows operating system, often via C Runtime functions. However, these have been long discouraged in favor of the UTF-16 APIs. Within the operating system, all text is represented as UTF-..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0529/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="Historically, Python uses the ANSI APIs for interacting with the Windows operating system, often via C Runtime functions. However, these have been long discouraged in favor of the UTF-16 APIs. Within the operating system, all text is represented as UTF-..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 529</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 529 – Change Windows filesystem encoding to UTF-8</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Steve Dower <steve.dower at python.org></dd> <dt class="field-even">Status:</dt> <dd class="field-even"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-odd">Type:</dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created:</dt> <dd class="field-even">27-Aug-2016</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">3.6</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even">01-Sep-2016, 04-Sep-2016</dd> <dt class="field-odd">Resolution:</dt> <dd class="field-odd"><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2016-September/146277.html">Python-Dev message</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#background">Background</a></li> <li><a class="reference internal" href="#proposal">Proposal</a></li> <li><a class="reference internal" href="#specific-changes">Specific Changes</a><ul> <li><a class="reference internal" href="#update-sys-getfilesystemencoding">Update sys.getfilesystemencoding</a></li> <li><a class="reference internal" href="#add-sys-getfilesystemencodeerrors">Add sys.getfilesystemencodeerrors</a></li> <li><a class="reference internal" href="#update-path-converter">Update path_converter</a></li> <li><a class="reference internal" href="#remove-unused-ansi-code">Remove unused ANSI code</a></li> <li><a class="reference internal" href="#add-legacy-mode">Add legacy mode</a></li> <li><a class="reference internal" href="#undeprecate-bytes-paths-on-windows">Undeprecate bytes paths on Windows</a></li> <li><a class="reference internal" href="#beta-experiment">Beta experiment</a></li> <li><a class="reference internal" href="#affected-modules">Affected Modules</a></li> </ul> </li> <li><a class="reference internal" href="#rejected-alternatives">Rejected Alternatives</a><ul> <li><a class="reference internal" href="#use-strict-mbcs-decoding">Use strict mbcs decoding</a></li> <li><a class="reference internal" href="#make-bytes-paths-an-error-on-windows">Make bytes paths an error on Windows</a></li> <li><a class="reference internal" href="#make-bytes-paths-an-error-on-all-platforms">Make bytes paths an error on all platforms</a></li> </ul> </li> <li><a class="reference internal" href="#code-that-may-break">Code that may break</a><ul> <li><a class="reference internal" href="#not-managing-encodings-across-boundaries">Not managing encodings across boundaries</a></li> <li><a class="reference internal" href="#explicitly-using-mbcs">Explicitly using ‘mbcs’</a></li> </ul> </li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> Historically, Python uses the ANSI APIs for interacting with the Windows operating system, often via C Runtime functions. However, these have been long discouraged in favor of the UTF-16 APIs. Within the operating system, all text is represented as UTF-16, and the ANSI APIs perform encoding and decoding using the active code page. See <a class="reference external" href="https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247.aspx">Naming Files, Paths, and Namespaces</a> for more details. This PEP proposes changing the default filesystem encoding on Windows to utf-8, and changing all filesystem functions to use the Unicode APIs for filesystem paths. This will not affect code that uses strings to represent paths, however those that use bytes for paths will now be able to correctly round-trip all valid paths in Windows filesystems. Currently, the conversions between Unicode (in the OS) and bytes (in Python) were lossy and would fail to round-trip characters outside of the user’s active code page. Notably, this does not impact the encoding of the contents of files. These will continue to default to <code class="docutils literal notranslate">locale.getpreferredencoding()</code> (for text files) or plain bytes (for binary files). This only affects the encoding used when users pass a bytes object to Python where it is then passed to the operating system as a path name. </section> <section id="background"> <h2><a class="toc-backref" href="#background" role="doc-backlink">Background</a></h2> File system paths are almost universally represented as text with an encoding determined by the file system. In Python, we expose these paths via a number of interfaces, such as the <code class="docutils literal notranslate">os</code> and <code class="docutils literal notranslate">io</code> modules. Paths may be passed either direction across these interfaces, that is, from the filesystem to the application (for example, <code class="docutils literal notranslate">os.listdir()</code>), or from the application to the filesystem (for example, <code class="docutils literal notranslate">os.unlink()</code>). When paths are passed between the filesystem and the application, they are either passed through as a bytes blob or converted to/from str using <code class="docutils literal notranslate">os.fsencode()</code> and <code class="docutils literal notranslate">os.fsdecode()</code> or explicit encoding using <code class="docutils literal notranslate">sys.getfilesystemencoding()</code>. The result of encoding a string with <code class="docutils literal notranslate">sys.getfilesystemencoding()</code> is a blob of bytes in the native format for the default file system. On Windows, the native format for the filesystem is utf-16-le. The recommended platform APIs for accessing the filesystem all accept and return text encoded in this format. However, prior to Windows NT (and possibly further back), the native format was a configurable machine option and a separate set of APIs existed to accept this format. The option (the “active code page”) and these APIs (the “*A functions”) still exist in recent versions of Windows for backwards compatibility, though new functionality often only has a utf-16-le API (the “*W functions”). In Python, str is recommended because it can correctly round-trip all characters used in paths (on POSIX with surrogateescape handling; on Windows because str maps to the native representation). On Windows bytes cannot round-trip all characters used in paths, as Python internally uses the *A functions and hence the encoding is “whatever the active code page is”. Since the active code page cannot represent all Unicode characters, the conversion of a path into bytes can lose information without warning or any available indication. As a demonstration of this: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> open('test\uAB00.txt', 'wb').close() >>> import glob >>> glob.glob('test*') ['test\uab00.txt'] >>> glob.glob(b'test*') [b'test?.txt'] </pre></div> </div> The Unicode character in the second call to glob has been replaced by a ‘?’, which means passing the path back into the filesystem will result in a <code class="docutils literal notranslate">FileNotFoundError</code>. The same results may be observed with <code class="docutils literal notranslate">os.listdir()</code> or any function that matches the return type to the parameter type. While one user-accessible fix is to use str everywhere, POSIX systems generally do not suffer from data loss when using bytes exclusively as the bytes are the canonical representation. Even if the encoding is “incorrect” by some standard, the file system will still map the bytes back to the file. Making use of this avoids the cost of decoding and reencoding, such that (theoretically, and only on POSIX), code such as this may be faster because of the use of <code class="docutils literal notranslate">b'.'</code> compared to using <code class="docutils literal notranslate">'.'</code>: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> for f in os.listdir(b'.'): ... os.stat(f) ... </pre></div> </div> As a result, POSIX-focused library authors prefer to use bytes to represent paths. For some authors it is also a convenience, as their code may receive bytes already known to be encoded correctly, while others are attempting to simplify porting their code from Python 2. However, the correctness assumptions do not carry over to Windows where Unicode is the canonical representation, and errors may result. This potential data loss is why the use of bytes paths on Windows was deprecated in Python 3.3 - all of the above code snippets produce deprecation warnings on Windows. </section> <section id="proposal"> <h2><a class="toc-backref" href="#proposal" role="doc-backlink">Proposal</a></h2> Currently the default filesystem encoding is ‘mbcs’, which is a meta-encoder that uses the active code page. However, when bytes are passed to the filesystem they go through the *A APIs and the operating system handles encoding. In this case, paths are always encoded using the equivalent of ‘mbcs:replace’ with no opportunity for Python to override or change this. This proposal would remove all use of the *A APIs and only ever call the *W APIs. When Windows returns paths to Python as <code class="docutils literal notranslate">str</code>, they will be decoded from utf-16-le and returned as text (in whatever the minimal representation is). When Python code requests paths as <code class="docutils literal notranslate">bytes</code>, the paths will be transcoded from utf-16-le into utf-8 using surrogatepass (Windows does not validate surrogate pairs, so it is possible to have invalid surrogates in filenames). Equally, when paths are provided as <code class="docutils literal notranslate">bytes</code>, they are transcoded from utf-8 into utf-16-le and passed to the *W APIs. The use of utf-8 will not be configurable, except for the provision of a “legacy mode” flag to revert to the previous behaviour. The <code class="docutils literal notranslate">surrogateescape</code> error mode does not apply here, as the concern is not about retaining nonsensical bytes. Any path returned from the operating system will be valid Unicode, while invalid paths created by the user should raise a decoding error (currently these would raise <code class="docutils literal notranslate">OSError</code> or a subclass). The choice of utf-8 bytes (as opposed to utf-16-le bytes) is to ensure the ability to round-trip path names and allow basic manipulation (for example, using the <code class="docutils literal notranslate">os.path</code> module) when assuming an ASCII-compatible encoding. Using utf-16-le as the encoding is more pure, but will cause more issues than are resolved. This change would also undeprecate the use of bytes paths on Windows. No change to the semantics of using bytes as a path is required - as before, they must be encoded with the encoding specified by <code class="docutils literal notranslate">sys.getfilesystemencoding()</code>. </section> <section id="specific-changes"> <h2><a class="toc-backref" href="#specific-changes" role="doc-backlink">Specific Changes</a></h2> <section id="update-sys-getfilesystemencoding"> <h3><a class="toc-backref" href="#update-sys-getfilesystemencoding" role="doc-backlink">Update sys.getfilesystemencoding</a></h3> Remove the default value for <code class="docutils literal notranslate">Py_FileSystemDefaultEncoding</code> and set it in <code class="docutils literal notranslate">initfsencoding()</code> to utf-8, or if the legacy-mode switch is enabled to mbcs. Update the implementations of <code class="docutils literal notranslate">PyUnicode_DecodeFSDefaultAndSize()</code> and <code class="docutils literal notranslate">PyUnicode_EncodeFSDefault()</code> to use the utf-8 codec, or if the legacy-mode switch is enabled the existing mbcs codec. </section> <section id="add-sys-getfilesystemencodeerrors"> <h3><a class="toc-backref" href="#add-sys-getfilesystemencodeerrors" role="doc-backlink">Add sys.getfilesystemencodeerrors</a></h3> As the error mode may now change between <code class="docutils literal notranslate">surrogatepass</code> and <code class="docutils literal notranslate">replace</code>, Python code that manually performs encoding also needs access to the current error mode. This includes the implementation of <code class="docutils literal notranslate">os.fsencode()</code> and <code class="docutils literal notranslate">os.fsdecode()</code>, which currently assume an error mode based on the codec. Add a public <code class="docutils literal notranslate">Py_FileSystemDefaultEncodeErrors</code>, similar to the existing <code class="docutils literal notranslate">Py_FileSystemDefaultEncoding</code>. The default value on Windows will be <code class="docutils literal notranslate">surrogatepass</code> or in legacy mode, <code class="docutils literal notranslate">replace</code>. The default value on all other platforms will be <code class="docutils literal notranslate">surrogateescape</code>. Add a public <code class="docutils literal notranslate">sys.getfilesystemencodeerrors()</code> function that returns the current error mode. Update the implementations of <code class="docutils literal notranslate">PyUnicode_DecodeFSDefaultAndSize()</code> and <code class="docutils literal notranslate">PyUnicode_EncodeFSDefault()</code> to use the variable for error mode rather than constant strings. Update the implementations of <code class="docutils literal notranslate">os.fsencode()</code> and <code class="docutils literal notranslate">os.fsdecode()</code> to use <code class="docutils literal notranslate">sys.getfilesystemencodeerrors()</code> instead of assuming the mode. </section> <section id="update-path-converter"> <h3><a class="toc-backref" href="#update-path-converter" role="doc-backlink">Update path_converter</a></h3> Update the path converter to always decode bytes or buffer objects into text using <code class="docutils literal notranslate">PyUnicode_DecodeFSDefaultAndSize()</code>. Change the <code class="docutils literal notranslate">narrow</code> field from a <code class="docutils literal notranslate">char*</code> string into a flag that indicates whether the original object was bytes. This is required for functions that need to return paths using the same type as was originally provided. </section> <section id="remove-unused-ansi-code"> <h3><a class="toc-backref" href="#remove-unused-ansi-code" role="doc-backlink">Remove unused ANSI code</a></h3> Remove all code paths using the <code class="docutils literal notranslate">narrow</code> field, as these will no longer be reachable by any caller. These are only used within <code class="docutils literal notranslate">posixmodule.c</code>. Other uses of paths should have use of bytes paths replaced with decoding and use of the *W APIs. </section> <section id="add-legacy-mode"> <h3><a class="toc-backref" href="#add-legacy-mode" role="doc-backlink">Add legacy mode</a></h3> Add a legacy mode flag, enabled by the environment variable <code class="docutils literal notranslate">PYTHONLEGACYWINDOWSFSENCODING</code> or by a function call to <code class="docutils literal notranslate">sys._enablelegacywindowsfsencoding()</code>. The function call can only be used to enable the flag and should be used by programs as close to initialization as possible. Legacy mode cannot be disabled while Python is running. When this flag is set, the default filesystem encoding is set to mbcs rather than utf-8, and the error mode is set to <code class="docutils literal notranslate">replace</code> rather than <code class="docutils literal notranslate">surrogatepass</code>. Paths will continue to decode to wide characters and only *W APIs will be called, however, the bytes passed in and received from Python will be encoded the same as prior to this change. </section> <section id="undeprecate-bytes-paths-on-windows"> <h3><a class="toc-backref" href="#undeprecate-bytes-paths-on-windows" role="doc-backlink">Undeprecate bytes paths on Windows</a></h3> Using bytes as paths on Windows is currently deprecated. We would announce that this is no longer the case, and that paths when encoded as bytes should use whatever is returned from <code class="docutils literal notranslate">sys.getfilesystemencoding()</code> rather than the user’s active code page. </section> <section id="beta-experiment"> <h3><a class="toc-backref" href="#beta-experiment" role="doc-backlink">Beta experiment</a></h3> To assist with determining the impact of this change, we propose applying it to 3.6.0b1 provisionally with the intent being to make a final decision before 3.6.0b4. During the experiment period, decoding and encoding exception messages will be expanded to include a link to an active online discussion and encourage reporting of problems. If it is decided to revert the functionality for 3.6.0b4, the implementation change would be to permanently enable the legacy mode flag, change the environment variable to <code class="docutils literal notranslate">PYTHONWINDOWSUTF8FSENCODING</code> and function to <code class="docutils literal notranslate">sys._enablewindowsutf8fsencoding()</code> to allow enabling the functionality on a case-by-case basis, as opposed to disabling it. It is expected that if we cannot feasibly make the change for 3.6 due to compatibility concerns, it will not be possible to make the change at any later time in Python 3.x. </section> <section id="affected-modules"> <h3><a class="toc-backref" href="#affected-modules" role="doc-backlink">Affected Modules</a></h3> This PEP implicitly includes all modules within the Python that either pass path names to the operating system, or otherwise use <code class="docutils literal notranslate">sys.getfilesystemencoding()</code>. As of 3.6.0a4, the following modules require modification: <ul class="simple"> <li><code class="docutils literal notranslate">os</code></li> <li><code class="docutils literal notranslate">_overlapped</code></li> <li><code class="docutils literal notranslate">_socket</code></li> <li><code class="docutils literal notranslate">subprocess</code></li> <li><code class="docutils literal notranslate">zipimport</code></li> </ul> The following modules use <code class="docutils literal notranslate">sys.getfilesystemencoding()</code> but do not need modification: <ul class="simple"> <li><code class="docutils literal notranslate">gc</code> (already assumes bytes are utf-8)</li> <li><code class="docutils literal notranslate">grp</code> (not compiled for Windows)</li> <li><code class="docutils literal notranslate">http.server</code> (correctly includes codec name with transmitted data)</li> <li><code class="docutils literal notranslate">idlelib.editor</code> (should not be needed; has fallback handling)</li> <li><code class="docutils literal notranslate">nis</code> (not compiled for Windows)</li> <li><code class="docutils literal notranslate">pwd</code> (not compiled for Windows)</li> <li><code class="docutils literal notranslate">spwd</code> (not compiled for Windows)</li> <li><code class="docutils literal notranslate">_ssl</code> (only used for ASCII constants)</li> <li><code class="docutils literal notranslate">tarfile</code> (code unused on Windows)</li> <li><code class="docutils literal notranslate">_tkinter</code> (already assumes bytes are utf-8)</li> <li><code class="docutils literal notranslate">wsgiref</code> (assumed as the default encoding for unknown environments)</li> <li><code class="docutils literal notranslate">zipapp</code> (code unused on Windows)</li> </ul> The following native code uses one of the encoding or decoding functions, but do not require any modification: <ul class="simple"> <li><code class="docutils literal notranslate">Parser/parsetok.c</code> (docs already specify <code class="docutils literal notranslate">sys.getfilesystemencoding()</code>)</li> <li><code class="docutils literal notranslate">Python/ast.c</code> (docs already specify <code class="docutils literal notranslate">sys.getfilesystemencoding()</code>)</li> <li><code class="docutils literal notranslate">Python/compile.c</code> (undocumented, but Python filesystem encoding implied)</li> <li><code class="docutils literal notranslate">Python/errors.c</code> (docs already specify <code class="docutils literal notranslate">os.fsdecode()</code>)</li> <li><code class="docutils literal notranslate">Python/fileutils.c</code> (code unused on Windows)</li> <li><code class="docutils literal notranslate">Python/future.c</code> (undocumented, but Python filesystem encoding implied)</li> <li><code class="docutils literal notranslate">Python/import.c</code> (docs already specify utf-8)</li> <li><code class="docutils literal notranslate">Python/importdl.c</code> (code unused on Windows)</li> <li><code class="docutils literal notranslate">Python/pythonrun.c</code> (docs already specify <code class="docutils literal notranslate">sys.getfilesystemencoding()</code>)</li> <li><code class="docutils literal notranslate">Python/symtable.c</code> (undocumented, but Python filesystem encoding implied)</li> <li><code class="docutils literal notranslate">Python/thread.c</code> (code unused on Windows)</li> <li><code class="docutils literal notranslate">Python/traceback.c</code> (encodes correctly for comparing strings)</li> <li><code class="docutils literal notranslate">Python/_warnings.c</code> (docs already specify <code class="docutils literal notranslate">os.fsdecode()</code>)</li> </ul> </section> </section> <section id="rejected-alternatives"> <h2><a class="toc-backref" href="#rejected-alternatives" role="doc-backlink">Rejected Alternatives</a></h2> <section id="use-strict-mbcs-decoding"> <h3><a class="toc-backref" href="#use-strict-mbcs-decoding" role="doc-backlink">Use strict mbcs decoding</a></h3> This is essentially the same as the proposed change, but instead of changing <code class="docutils literal notranslate">sys.getfilesystemencoding()</code> to utf-8 it is changed to mbcs (which dynamically maps to the active code page). This approach allows the use of new functionality that is only available as *W APIs and also detection of encoding/decoding errors. For example, rather than silently replacing Unicode characters with ‘?’, it would be possible to warn or fail the operation. Compared to the proposed fix, this could enable some new functionality but does not fix any of the problems described initially. New runtime errors may cause some problems to be more obvious and lead to fixes, provided library maintainers are interested in supporting Windows and adding a separate code path to treat filesystem paths as strings. Making the encoding mbcs without strict errors is equivalent to the legacy-mode switch being enabled by default. This is a possible course of action if there is significant breakage of actual code and a need to extend the deprecation period, but still a desire to have the simplifications to the CPython source. </section> <section id="make-bytes-paths-an-error-on-windows"> <h3><a class="toc-backref" href="#make-bytes-paths-an-error-on-windows" role="doc-backlink">Make bytes paths an error on Windows</a></h3> By preventing the use of bytes paths on Windows completely we prevent users from hitting encoding issues. However, the motivation for this PEP is to increase the likelihood that code written on POSIX will also work correctly on Windows. This alternative would move the other direction and make such code completely incompatible. As this does not benefit users in any way, we reject it. </section> <section id="make-bytes-paths-an-error-on-all-platforms"> <h3><a class="toc-backref" href="#make-bytes-paths-an-error-on-all-platforms" role="doc-backlink">Make bytes paths an error on all platforms</a></h3> By deprecating and then disable the use of bytes paths on all platforms we prevent users from hitting encoding issues regardless of where the code was originally written. This would require a full deprecation cycle, as there are currently no warnings on platforms other than Windows. This is likely to be seen as a hostile action against Python developers in general, and as such is rejected at this time. </section> </section> <section id="code-that-may-break"> <h2><a class="toc-backref" href="#code-that-may-break" role="doc-backlink">Code that may break</a></h2> The following code patterns may break or see different behaviour as a result of this change. Each of these examples would have been fragile in code intended for cross-platform use. The suggested fixes demonstrate the most compatible way to handle path encoding issues across all platforms and across multiple Python versions. Note that all of these examples produce deprecation warnings on Python 3.3 and later. <section id="not-managing-encodings-across-boundaries"> <h3><a class="toc-backref" href="#not-managing-encodings-across-boundaries" role="doc-backlink">Not managing encodings across boundaries</a></h3> Code that does not manage encodings when crossing protocol boundaries may currently be working by chance, but could encounter issues when either encoding changes. Note that the source of <code class="docutils literal notranslate">filename</code> may be any function that returns a bytes object, as illustrated in a second example below: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> filename = open('filename_in_mbcs.txt', 'rb').read() >>> text = open(filename, 'r').read() </pre></div> </div> To correct this code, the encoding of the bytes in <code class="docutils literal notranslate">filename</code> should be specified, either when reading from the file or before using the value: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> # Fix 1: Open file as text (default encoding) >>> filename = open('filename_in_mbcs.txt', 'r').read() >>> text = open(filename, 'r').read() >>> # Fix 2: Open file as text (explicit encoding) >>> filename = open('filename_in_mbcs.txt', 'r', encoding='mbcs').read() >>> text = open(filename, 'r').read() >>> # Fix 3: Explicitly decode the path >>> filename = open('filename_in_mbcs.txt', 'rb').read() >>> text = open(filename.decode('mbcs'), 'r').read() </pre></div> </div> Where the creator of <code class="docutils literal notranslate">filename</code> is separated from the user of <code class="docutils literal notranslate">filename</code>, the encoding is important information to include: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> some_object.filename = r'C:\Users\Steve\Documents\my_file.txt'.encode('mbcs') >>> filename = some_object.filename >>> type(filename) <class 'bytes'> >>> text = open(filename, 'r').read() </pre></div> </div> To fix this code for best compatibility across operating systems and Python versions, the filename should be exposed as str: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> # Fix 1: Expose as str >>> some_object.filename = r'C:\Users\Steve\Documents\my_file.txt' >>> filename = some_object.filename >>> type(filename) <class 'str'> >>> text = open(filename, 'r').read() </pre></div> </div> Alternatively, the encoding used for the path needs to be made available to the user. Specifying <code class="docutils literal notranslate">os.fsencode()</code> (or <code class="docutils literal notranslate">sys.getfilesystemencoding()</code>) is an acceptable choice, or a new attribute could be added with the exact encoding: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> # Fix 2: Use fsencode >>> some_object.filename = os.fsencode(r'C:\Users\Steve\Documents\my_file.txt') >>> filename = some_object.filename >>> type(filename) <class 'bytes'> >>> text = open(filename, 'r').read() >>> # Fix 3: Expose as explicit encoding >>> some_object.filename = r'C:\Users\Steve\Documents\my_file.txt'.encode('cp437') >>> some_object.filename_encoding = 'cp437' >>> filename = some_object.filename >>> type(filename) <class 'bytes'> >>> filename = filename.decode(some_object.filename_encoding) >>> type(filename) <class 'str'> >>> text = open(filename, 'r').read() </pre></div> </div> </section> <section id="explicitly-using-mbcs"> <h3><a class="toc-backref" href="#explicitly-using-mbcs" role="doc-backlink">Explicitly using ‘mbcs’</a></h3> Code that explicitly encodes text using ‘mbcs’ before passing to file system APIs is now passing incorrectly encoded bytes. Note that the source of <code class="docutils literal notranslate">filename</code> in this example is not relevant, provided that it is a str: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> filename = open('files.txt', 'r').readline().rstrip() >>> text = open(filename.encode('mbcs'), 'r') </pre></div> </div> To correct this code, the string should be passed without explicit encoding, or should use <code class="docutils literal notranslate">os.fsencode()</code>: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> # Fix 1: Do not encode the string >>> filename = open('files.txt', 'r').readline().rstrip() >>> text = open(filename, 'r') >>> # Fix 2: Use correct encoding >>> filename = open('files.txt', 'r').readline().rstrip() >>> text = open(os.fsencode(filename), 'r') </pre></div> </div> </section> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0529.rst">https://github.com/python/peps/blob/main/peps/pep-0529.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0529.rst">2025-02-01 08:59:27 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#background">Background</a></li> <li><a class="reference internal" href="#proposal">Proposal</a></li> <li><a class="reference internal" href="#specific-changes">Specific Changes</a><ul> <li><a class="reference internal" href="#update-sys-getfilesystemencoding">Update sys.getfilesystemencoding</a></li> <li><a class="reference internal" href="#add-sys-getfilesystemencodeerrors">Add sys.getfilesystemencodeerrors</a></li> <li><a class="reference internal" href="#update-path-converter">Update path_converter</a></li> <li><a class="reference internal" href="#remove-unused-ansi-code">Remove unused ANSI code</a></li> <li><a class="reference internal" href="#add-legacy-mode">Add legacy mode</a></li> <li><a class="reference internal" href="#undeprecate-bytes-paths-on-windows">Undeprecate bytes paths on Windows</a></li> <li><a class="reference internal" href="#beta-experiment">Beta experiment</a></li> <li><a class="reference internal" href="#affected-modules">Affected Modules</a></li> </ul> </li> <li><a class="reference internal" href="#rejected-alternatives">Rejected Alternatives</a><ul> <li><a class="reference internal" href="#use-strict-mbcs-decoding">Use strict mbcs decoding</a></li> <li><a class="reference internal" href="#make-bytes-paths-an-error-on-windows">Make bytes paths an error on Windows</a></li> <li><a class="reference internal" href="#make-bytes-paths-an-error-on-all-platforms">Make bytes paths an error on all platforms</a></li> </ul> </li> <li><a class="reference internal" href="#code-that-may-break">Code that may break</a><ul> <li><a class="reference internal" href="#not-managing-encodings-across-boundaries">Not managing encodings across boundaries</a></li> <li><a class="reference internal" href="#explicitly-using-mbcs">Explicitly using ‘mbcs’</a></li> </ul> </li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0529.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>