CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 100 – Python Unicode Integration | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0100/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 100 – Python Unicode Integration | peps.python.org'> <meta property="og:description" content="The idea of this proposal is to add native Unicode 3.0 support to Python in a way that makes use of Unicode strings as simple as possible without introducing too many pitfalls along the way."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0100/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="The idea of this proposal is to add native Unicode 3.0 support to Python in a way that makes use of Unicode strings as simple as possible without introducing too many pitfalls along the way."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 100</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 100 – Python Unicode Integration</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Marc-André Lemburg <mal at lemburg.com></dd> <dt class="field-even">Status:</dt> <dd class="field-even"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-odd">Type:</dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created:</dt> <dd class="field-even">10-Mar-2000</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">2.0</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even"></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#historical-note">Historical Note</a></li> <li><a class="reference internal" href="#introduction">Introduction</a></li> <li><a class="reference internal" href="#conventions">Conventions</a></li> <li><a class="reference internal" href="#general-remarks">General Remarks</a></li> <li><a class="reference internal" href="#unicode-default-encoding">Unicode Default Encoding</a></li> <li><a class="reference internal" href="#unicode-constructors">Unicode Constructors</a></li> <li><a class="reference internal" href="#unicode-type-object">Unicode Type Object</a></li> <li><a class="reference internal" href="#unicode-output">Unicode Output</a></li> <li><a class="reference internal" href="#unicode-ordinals">Unicode Ordinals</a></li> <li><a class="reference internal" href="#comparison-hash-value">Comparison & Hash Value</a></li> <li><a class="reference internal" href="#coercion">Coercion</a></li> <li><a class="reference internal" href="#exceptions">Exceptions</a></li> <li><a class="reference internal" href="#codecs-coder-decoders-lookup">Codecs (Coder/Decoders) Lookup</a></li> <li><a class="reference internal" href="#standard-codecs">Standard Codecs</a></li> <li><a class="reference internal" href="#codecs-interface-definition">Codecs Interface Definition</a></li> <li><a class="reference internal" href="#whitespace">Whitespace</a></li> <li><a class="reference internal" href="#case-conversion">Case Conversion</a></li> <li><a class="reference internal" href="#line-breaks">Line Breaks</a></li> <li><a class="reference internal" href="#unicode-character-properties">Unicode Character Properties</a></li> <li><a class="reference internal" href="#private-code-point-areas">Private Code Point Areas</a></li> <li><a class="reference internal" href="#internal-format">Internal Format</a></li> <li><a class="reference internal" href="#buffer-interface">Buffer Interface</a></li> <li><a class="reference internal" href="#pickle-marshalling">Pickle/Marshalling</a></li> <li><a class="reference internal" href="#regular-expressions">Regular Expressions</a></li> <li><a class="reference internal" href="#formatting-markers">Formatting Markers</a></li> <li><a class="reference internal" href="#internal-argument-parsing">Internal Argument Parsing</a></li> <li><a class="reference internal" href="#file-stream-output">File/Stream Output</a></li> <li><a class="reference internal" href="#file-stream-input">File/Stream Input</a></li> <li><a class="reference internal" href="#unicode-methods-attributes">Unicode Methods & Attributes</a></li> <li><a class="reference internal" href="#code-base">Code Base</a></li> <li><a class="reference internal" href="#test-cases">Test Cases</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#history-of-this-proposal">History of this Proposal</a><ul> <li><a class="reference internal" href="#id1">1.7</a></li> <li><a class="reference internal" href="#id2">1.6</a></li> <li><a class="reference internal" href="#id3">1.5</a></li> <li><a class="reference internal" href="#id4">1.4</a></li> <li><a class="reference internal" href="#id5">1.3</a></li> <li><a class="reference internal" href="#id6">1.2</a></li> <li><a class="reference internal" href="#id7">1.1</a></li> <li><a class="reference internal" href="#id8">1.0</a></li> <li><a class="reference internal" href="#id9">0.9</a></li> <li><a class="reference internal" href="#id10">0.8</a></li> <li><a class="reference internal" href="#id11">0.7</a></li> <li><a class="reference internal" href="#id12">0.6</a></li> <li><a class="reference internal" href="#id13">0.5</a></li> <li><a class="reference internal" href="#id14">0.4</a></li> <li><a class="reference internal" href="#id15">0.3</a></li> <li><a class="reference internal" href="#id16">0.2</a></li> <li><a class="reference internal" href="#id17">0.1</a></li> </ul> </li> </ul> </details></section> <section id="historical-note"> <h2><a class="toc-backref" href="#historical-note" role="doc-backlink">Historical Note</a></h2> This document was first written by Marc-Andre in the pre-PEP days, and was originally distributed as Misc/unicode.txt in Python distributions up to and included Python 2.1. The last revision of the proposal in that location was labeled version 1.7 (CVS revision 3.10). Because the document clearly serves the purpose of an informational PEP in the post-PEP era, it has been moved here and reformatted to comply with PEP guidelines. Future revisions will be made to this document, while Misc/unicode.txt will contain a pointer to this PEP. -Barry Warsaw, PEP editor </section> <section id="introduction"> <h2><a class="toc-backref" href="#introduction" role="doc-backlink">Introduction</a></h2> The idea of this proposal is to add native Unicode 3.0 support to Python in a way that makes use of Unicode strings as simple as possible without introducing too many pitfalls along the way. Since this goal is not easy to achieve – strings being one of the most fundamental objects in Python – we expect this proposal to undergo some significant refinements. Note that the current version of this proposal is still a bit unsorted due to the many different aspects of the Unicode-Python integration. The latest version of this document is always available at: <a class="reference external" href="http://starship.python.net/~lemburg/unicode-proposal.txt">http://starship.python.net/~lemburg/unicode-proposal.txt</a> Older versions are available as: <a class="reference external" href="http://starship.python.net/~lemburg/unicode-proposal-X.X.txt">http://starship.python.net/~lemburg/unicode-proposal-X.X.txt</a> [ed. note: new revisions should be made to this PEP document, while the historical record previous to version 1.7 should be retrieved from MAL’s url, or Misc/unicode.txt] </section> <section id="conventions"> <h2><a class="toc-backref" href="#conventions" role="doc-backlink">Conventions</a></h2> <ul class="simple"> <li>In examples we use u = Unicode object and s = Python string</li> <li>‘XXX’ markings indicate points of discussion (PODs)</li> </ul> </section> <section id="general-remarks"> <h2><a class="toc-backref" href="#general-remarks" role="doc-backlink">General Remarks</a></h2> <ul class="simple"> <li>Unicode encoding names should be lower case on output and case-insensitive on input (they will be converted to lower case by all APIs taking an encoding name as input).</li> <li>Encoding names should follow the name conventions as used by the Unicode Consortium: spaces are converted to hyphens, e.g. ‘utf 16’ is written as ‘utf-16’.</li> <li>Codec modules should use the same names, but with hyphens converted to underscores, e.g. <code class="docutils literal notranslate">utf_8</code>, <code class="docutils literal notranslate">utf_16</code>, <code class="docutils literal notranslate">iso_8859_1</code>.</li> </ul> </section> <section id="unicode-default-encoding"> <h2><a class="toc-backref" href="#unicode-default-encoding" role="doc-backlink">Unicode Default Encoding</a></h2> The Unicode implementation has to make some assumption about the encoding of 8-bit strings passed to it for coercion and about the encoding to as default for conversion of Unicode to strings when no specific encoding is given. This encoding is called <default encoding> throughout this text. For this, the implementation maintains a global which can be set in the site.py Python startup script. Subsequent changes are not possible. The <default encoding> can be set and queried using the two sys module APIs: <dl> <dt><code class="docutils literal notranslate">sys.setdefaultencoding(encoding)</code></dt><dd>Sets the <default encoding> used by the Unicode implementation. encoding has to be an encoding which is supported by the Python installation, otherwise, a LookupError is raised.Note: This API is only available in site.py! It is removed from the sys module by site.py after usage. </dd> <dt><code class="docutils literal notranslate">sys.getdefaultencoding()</code></dt><dd>Returns the current <default encoding>.</dd> </dl> If not otherwise defined or set, the <default encoding> defaults to ‘ascii’. This encoding is also the startup default of Python (and in effect before site.py is executed). Note that the default site.py startup module contains disabled optional code which can set the <default encoding> according to the encoding defined by the current locale. The locale module is used to extract the encoding from the locale default settings defined by the OS environment (see locale.py). If the encoding cannot be determined, is unknown or unsupported, the code defaults to setting the <default encoding> to ‘ascii’. To enable this code, edit the site.py file or place the appropriate code into the sitecustomize.py module of your Python installation. </section> <section id="unicode-constructors"> <h2><a class="toc-backref" href="#unicode-constructors" role="doc-backlink">Unicode Constructors</a></h2> Python should provide a built-in constructor for Unicode strings which is available through <code class="docutils literal notranslate">__builtins__</code>: <div class="highlight-default notranslate"><div class="highlight"><pre>u = unicode(encoded_string[,encoding=<default encoding>][,errors="strict"]) u = u'<unicode-escape encoded Python string>' u = ur'<raw-unicode-escape encoded Python string>' </pre></div> </div> With the ‘unicode-escape’ encoding being defined as: <ul class="simple"> <li>all non-escape characters represent themselves as Unicode ordinal (e.g. ‘a’ -> U+0061).</li> <li>all existing defined Python escape sequences are interpreted as Unicode ordinals; note that <code class="docutils literal notranslate">\xXXXX</code> can represent all Unicode ordinals, and <code class="docutils literal notranslate">\OOO</code> (octal) can represent Unicode ordinals up to U+01FF.</li> <li>a new escape sequence, <code class="docutils literal notranslate">\uXXXX</code>, represents U+XXXX; it is a syntax error to have fewer than 4 digits after <code class="docutils literal notranslate">\u</code>.</li> </ul> For an explanation of possible values for errors see the Codec section below. Examples: <div class="highlight-default notranslate"><div class="highlight"><pre>u'abc' -> U+0061 U+0062 U+0063 u'\u1234' -> U+1234 u'abc\u1234\n' -> U+0061 U+0062 U+0063 U+1234 U+005c </pre></div> </div> The ‘raw-unicode-escape’ encoding is defined as follows: <ul class="simple"> <li><code class="docutils literal notranslate">\uXXXX</code> sequence represent the U+XXXX Unicode character if and only if the number of leading backslashes is odd</li> <li>all other characters represent themselves as Unicode ordinal (e.g. ‘b’ -> U+0062)</li> </ul> Note that you should provide some hint to the encoding you used to write your programs as pragma line in one the first few comment lines of the source file (e.g. ‘# source file encoding: latin-1’). If you only use 7-bit ASCII then everything is fine and no such notice is needed, but if you include Latin-1 characters not defined in ASCII, it may well be worthwhile including a hint since people in other countries will want to be able to read your source strings too. </section> <section id="unicode-type-object"> <h2><a class="toc-backref" href="#unicode-type-object" role="doc-backlink">Unicode Type Object</a></h2> Unicode objects should have the type UnicodeType with type name ‘unicode’, made available through the standard types module. </section> <section id="unicode-output"> <h2><a class="toc-backref" href="#unicode-output" role="doc-backlink">Unicode Output</a></h2> Unicode objects have a method .encode([encoding=<default encoding>]) which returns a Python string encoding the Unicode string using the given scheme (see Codecs). <div class="highlight-default notranslate"><div class="highlight"><pre>print u := print u.encode() # using the <default encoding> str(u) := u.encode() # using the <default encoding> repr(u) := "u%s" % repr(u.encode('unicode-escape')) </pre></div> </div> Also see Internal Argument Parsing and Buffer Interface for details on how other APIs written in C will treat Unicode objects. </section> <section id="unicode-ordinals"> <h2><a class="toc-backref" href="#unicode-ordinals" role="doc-backlink">Unicode Ordinals</a></h2> Since Unicode 3.0 has a 32-bit ordinal character set, the implementation should provide 32-bit aware ordinal conversion APIs: <div class="highlight-default notranslate"><div class="highlight"><pre>ord(u[:1]) (this is the standard ord() extended to work with Unicode objects) --> Unicode ordinal number (32-bit) unichr(i) --> Unicode object for character i (provided it is 32-bit); ValueError otherwise </pre></div> </div> Both APIs should go into <code class="docutils literal notranslate">__builtins__</code> just like their string counterparts <code class="docutils literal notranslate">ord()</code> and <code class="docutils literal notranslate">chr()</code>. Note that Unicode provides space for private encodings. Usage of these can cause different output representations on different machines. This problem is not a Python or Unicode problem, but a machine setup and maintenance one. </section> <section id="comparison-hash-value"> <h2><a class="toc-backref" href="#comparison-hash-value" role="doc-backlink">Comparison & Hash Value</a></h2> Unicode objects should compare equal to other objects after these other objects have been coerced to Unicode. For strings this means that they are interpreted as Unicode string using the <default encoding>. Unicode objects should return the same hash value as their ASCII equivalent strings. Unicode strings holding non-ASCII values are not guaranteed to return the same hash values as the default encoded equivalent string representation. When compared using <code class="docutils literal notranslate">cmp()</code> (or <code class="docutils literal notranslate">PyObject_Compare()</code>) the implementation should mask <code class="docutils literal notranslate">TypeErrors</code> raised during the conversion to remain in synch with the string behavior. All other errors such as <code class="docutils literal notranslate">ValueErrors</code> raised during coercion of strings to Unicode should not be masked and passed through to the user. In containment tests (‘a’ in u’abc’ and u’a’ in ‘abc’) both sides should be coerced to Unicode before applying the test. Errors occurring during coercion (e.g. None in u’abc’) should not be masked. </section> <section id="coercion"> <h2><a class="toc-backref" href="#coercion" role="doc-backlink">Coercion</a></h2> Using Python strings and Unicode objects to form new objects should always coerce to the more precise format, i.e. Unicode objects. <div class="highlight-default notranslate"><div class="highlight"><pre>u + s := u + unicode(s) s + u := unicode(s) + u </pre></div> </div> All string methods should delegate the call to an equivalent Unicode object method call by converting all involved strings to Unicode and then applying the arguments to the Unicode method of the same name, e.g. <div class="highlight-default notranslate"><div class="highlight"><pre>string.join((s,u),sep) := (s + sep) + u sep.join((s,u)) := (s + sep) + u </pre></div> </div> For a discussion of %-formatting w/r to Unicode objects, see Formatting Markers. </section> <section id="exceptions"> <h2><a class="toc-backref" href="#exceptions" role="doc-backlink">Exceptions</a></h2> <code class="docutils literal notranslate">UnicodeError</code> is defined in the exceptions module as a subclass of <code class="docutils literal notranslate">ValueError</code>. It is available at the C level via <code class="docutils literal notranslate">PyExc_UnicodeError</code>. All exceptions related to Unicode encoding/decoding should be subclasses of <code class="docutils literal notranslate">UnicodeError</code>. </section> <section id="codecs-coder-decoders-lookup"> <h2><a class="toc-backref" href="#codecs-coder-decoders-lookup" role="doc-backlink">Codecs (Coder/Decoders) Lookup</a></h2> A Codec (see Codec Interface Definition) search registry should be implemented by a module “codecs”: <div class="highlight-default notranslate"><div class="highlight"><pre>codecs.register(search_function) </pre></div> </div> Search functions are expected to take one argument, the encoding name in all lower case letters and with hyphens and spaces converted to underscores, and return a tuple of functions (encoder, decoder, stream_reader, stream_writer) taking the following arguments: <dl> <dt>encoder and decoder</dt><dd>These must be functions or methods which have the same interface as the <code class="docutils literal notranslate">.encode</code>/<code class="docutils literal notranslate">.decode</code> methods of Codec instances (see Codec Interface). The functions/methods are expected to work in a stateless mode.</dd> <dt>stream_reader and stream_writer</dt><dd>These need to be factory functions with the following interface:<div class="highlight-default notranslate"><div class="highlight"><pre>factory(stream,errors='strict') </pre></div> </div> The factory functions must return objects providing the interfaces defined by <code class="docutils literal notranslate">StreamWriter</code>/<code class="docutils literal notranslate">StreamReader</code> resp. (see Codec Interface). Stream codecs can maintain state. Possible values for errors are defined in the Codec section below. </dd> </dl> In case a search function cannot find a given encoding, it should return None. Aliasing support for encodings is left to the search functions to implement. The codecs module will maintain an encoding cache for performance reasons. Encodings are first looked up in the cache. If not found, the list of registered search functions is scanned. If no codecs tuple is found, a LookupError is raised. Otherwise, the codecs tuple is stored in the cache and returned to the caller. To query the Codec instance the following API should be used: <div class="highlight-default notranslate"><div class="highlight"><pre>codecs.lookup(encoding) </pre></div> </div> This will either return the found codecs tuple or raise a <code class="docutils literal notranslate">LookupError</code>. </section> <section id="standard-codecs"> <h2><a class="toc-backref" href="#standard-codecs" role="doc-backlink">Standard Codecs</a></h2> Standard codecs should live inside an encodings/ package directory in the Standard Python Code Library. The <code class="docutils literal notranslate">__init__.py</code> file of that directory should include a Codec Lookup compatible search function implementing a lazy module based codec lookup. Python should provide a few standard codecs for the most relevant encodings, e.g. <div class="highlight-default notranslate"><div class="highlight"><pre>'utf-8': 8-bit variable length encoding 'utf-16': 16-bit variable length encoding (little/big endian) 'utf-16-le': utf-16 but explicitly little endian 'utf-16-be': utf-16 but explicitly big endian 'ascii': 7-bit ASCII codepage 'iso-8859-1': ISO 8859-1 (Latin 1) codepage 'unicode-escape': See Unicode Constructors for a definition 'raw-unicode-escape': See Unicode Constructors for a definition 'native': Dump of the Internal Format used by Python </pre></div> </div> Common aliases should also be provided per default, e.g. ‘latin-1’ for ‘iso-8859-1’. Note: ‘utf-16’ should be implemented by using and requiring byte order marks (BOM) for file input/output. All other encodings such as the CJK ones to support Asian scripts should be implemented in separate packages which do not get included in the core Python distribution and are not a part of this proposal. </section> <section id="codecs-interface-definition"> <h2><a class="toc-backref" href="#codecs-interface-definition" role="doc-backlink">Codecs Interface Definition</a></h2> The following base class should be defined in the module “codecs”. They provide not only templates for use by encoding module implementors, but also define the interface which is expected by the Unicode implementation. Note that the Codec Interface defined here is well suitable for a larger range of applications. The Unicode implementation expects Unicode objects on input for <code class="docutils literal notranslate">.encode()</code> and <code class="docutils literal notranslate">.write()</code> and character buffer compatible objects on input for <code class="docutils literal notranslate">.decode()</code>. Output of <code class="docutils literal notranslate">.encode()</code> and <code class="docutils literal notranslate">.read()</code> should be a Python string and <code class="docutils literal notranslate">.decode()</code> must return an Unicode object. First, we have the stateless encoders/decoders. These do not work in chunks as the stream codecs (see below) do, because all components are expected to be available in memory. <div class="highlight-default notranslate"><div class="highlight"><pre>class Codec: """Defines the interface for stateless encoders/decoders. The .encode()/.decode() methods may implement different error handling schemes by providing the errors argument. These string values are defined: 'strict' - raise an error (or a subclass) 'ignore' - ignore the character and continue with the next 'replace' - replace with a suitable replacement character; Python will use the official U+FFFD REPLACEMENT CHARACTER for the builtin Unicode codecs. """ def encode(self,input,errors='strict'): """Encodes the object input and returns a tuple (output object, length consumed). errors defines the error handling to apply. It defaults to 'strict' handling. The method may not store state in the Codec instance. Use StreamCodec for codecs which have to keep state in order to make encoding/decoding efficient. """ def decode(self,input,errors='strict'): """Decodes the object input and returns a tuple (output object, length consumed). input must be an object which provides the bf_getreadbuf buffer slot. Python strings, buffer objects and memory mapped files are examples of objects providing this slot. errors defines the error handling to apply. It defaults to 'strict' handling. The method may not store state in the Codec instance. Use StreamCodec for codecs which have to keep state in order to make encoding/decoding efficient. """ </pre></div> </div> <code class="docutils literal notranslate">StreamWriter</code> and <code class="docutils literal notranslate">StreamReader</code> define the interface for stateful encoders/decoders which work on streams. These allow processing of the data in chunks to efficiently use memory. If you have large strings in memory, you may want to wrap them with <code class="docutils literal notranslate">cStringIO</code> objects and then use these codecs on them to be able to do chunk processing as well, e.g. to provide progress information to the user. <div class="highlight-default notranslate"><div class="highlight"><pre>class StreamWriter(Codec): def __init__(self,stream,errors='strict'): """Creates a StreamWriter instance. stream must be a file-like object open for writing (binary) data. The StreamWriter may implement different error handling schemes by providing the errors keyword argument. These parameters are defined: 'strict' - raise a ValueError (or a subclass) 'ignore' - ignore the character and continue with the next 'replace'- replace with a suitable replacement character """ self.stream = stream self.errors = errors def write(self,object): """Writes the object's contents encoded to self.stream. """ data, consumed = self.encode(object,self.errors) self.stream.write(data) def writelines(self, list): """Writes the concatenated list of strings to the stream using .write(). """ self.write(''.join(list)) def reset(self): """Flushes and resets the codec buffers used for keeping state. Calling this method should ensure that the data on the output is put into a clean state, that allows appending of new fresh data without having to rescan the whole stream to recover state. """ pass def __getattr__(self,name, getattr=getattr): """Inherit all other methods from the underlying stream. """ return getattr(self.stream,name) class StreamReader(Codec): def __init__(self,stream,errors='strict'): """Creates a StreamReader instance. stream must be a file-like object open for reading (binary) data. The StreamReader may implement different error handling schemes by providing the errors keyword argument. These parameters are defined: 'strict' - raise a ValueError (or a subclass) 'ignore' - ignore the character and continue with the next 'replace'- replace with a suitable replacement character; """ self.stream = stream self.errors = errors def read(self,size=-1): """Decodes data from the stream self.stream and returns the resulting object. size indicates the approximate maximum number of bytes to read from the stream for decoding purposes. The decoder can modify this setting as appropriate. The default value -1 indicates to read and decode as much as possible. size is intended to prevent having to decode huge files in one step. The method should use a greedy read strategy meaning that it should read as much data as is allowed within the definition of the encoding and the given size, e.g. if optional encoding endings or state markers are available on the stream, these should be read too. """ # Unsliced reading: if size < 0: return self.decode(self.stream.read())[0] # Sliced reading: read = self.stream.read decode = self.decode data = read(size) i = 0 while 1: try: object, decodedbytes = decode(data) except ValueError,why: # This method is slow but should work under pretty # much all conditions; at most 10 tries are made i = i + 1 newdata = read(1) if not newdata or i > 10: raise data = data + newdata else: return object def readline(self, size=None): """Read one line from the input stream and return the decoded data. Note: Unlike the .readlines() method, this method inherits the line breaking knowledge from the underlying stream's .readline() method -- there is currently no support for line breaking using the codec decoder due to lack of line buffering. Subclasses should however, if possible, try to implement this method using their own knowledge of line breaking. size, if given, is passed as size argument to the stream's .readline() method. """ if size is None: line = self.stream.readline() else: line = self.stream.readline(size) return self.decode(line)[0] def readlines(self, sizehint=0): """Read all lines available on the input stream and return them as list of lines. Line breaks are implemented using the codec's decoder method and are included in the list entries. sizehint, if given, is passed as size argument to the stream's .read() method. """ if sizehint is None: data = self.stream.read() else: data = self.stream.read(sizehint) return self.decode(data)[0].splitlines(1) def reset(self): """Resets the codec buffers used for keeping state. Note that no stream repositioning should take place. This method is primarily intended to be able to recover from decoding errors. """ pass def __getattr__(self,name, getattr=getattr): """ Inherit all other methods from the underlying stream. """ return getattr(self.stream,name) </pre></div> </div> Stream codec implementors are free to combine the <code class="docutils literal notranslate">StreamWriter</code> and <code class="docutils literal notranslate">StreamReader</code> interfaces into one class. Even combining all these with the Codec class should be possible. Implementors are free to add additional methods to enhance the codec functionality or provide extra state information needed for them to work. The internal codec implementation will only use the above interfaces, though. It is not required by the Unicode implementation to use these base classes, only the interfaces must match; this allows writing Codecs as extension types. As guideline, large mapping tables should be implemented using static C data in separate (shared) extension modules. That way multiple processes can share the same data. A tool to auto-convert Unicode mapping files to mapping modules should be provided to simplify support for additional mappings (see References). </section> <section id="whitespace"> <h2><a class="toc-backref" href="#whitespace" role="doc-backlink">Whitespace</a></h2> The <code class="docutils literal notranslate">.split()</code> method will have to know about what is considered whitespace in Unicode. </section> <section id="case-conversion"> <h2><a class="toc-backref" href="#case-conversion" role="doc-backlink">Case Conversion</a></h2> Case conversion is rather complicated with Unicode data, since there are many different conditions to respect. See <blockquote> <div><a class="reference external" href="http://www.unicode.org/unicode/reports/tr13/">http://www.unicode.org/unicode/reports/tr13/</a></div></blockquote> for some guidelines on implementing case conversion. For Python, we should only implement the 1-1 conversions included in Unicode. Locale dependent and other special case conversions (see the Unicode standard file SpecialCasing.txt) should be left to user land routines and not go into the core interpreter. The methods <code class="docutils literal notranslate">.capitalize()</code> and <code class="docutils literal notranslate">.iscapitalized()</code> should follow the case mapping algorithm defined in the above technical report as closely as possible. </section> <section id="line-breaks"> <h2><a class="toc-backref" href="#line-breaks" role="doc-backlink">Line Breaks</a></h2> Line breaking should be done for all Unicode characters having the B property as well as the combinations CRLF, CR, LF (interpreted in that order) and other special line separators defined by the standard. The Unicode type should provide a <code class="docutils literal notranslate">.splitlines()</code> method which returns a list of lines according to the above specification. See Unicode Methods. </section> <section id="unicode-character-properties"> <h2><a class="toc-backref" href="#unicode-character-properties" role="doc-backlink">Unicode Character Properties</a></h2> A separate module “unicodedata” should provide a compact interface to all Unicode character properties defined in the standard’s UnicodeData.txt file. Among other things, these properties provide ways to recognize numbers, digits, spaces, whitespace, etc. Since this module will have to provide access to all Unicode characters, it will eventually have to contain the data from UnicodeData.txt which takes up around 600kB. For this reason, the data should be stored in static C data. This enables compilation as shared module which the underlying OS can shared between processes (unlike normal Python code modules). There should be a standard Python interface for accessing this information so that other implementors can plug in their own possibly enhanced versions, e.g. ones that do decompressing of the data on-the-fly. </section> <section id="private-code-point-areas"> <h2><a class="toc-backref" href="#private-code-point-areas" role="doc-backlink">Private Code Point Areas</a></h2> Support for these is left to user land Codecs and not explicitly integrated into the core. Note that due to the Internal Format being implemented, only the area between <code class="docutils literal notranslate">\uE000</code> and <code class="docutils literal notranslate">\uF8FF</code> is usable for private encodings. </section> <section id="internal-format"> <h2><a class="toc-backref" href="#internal-format" role="doc-backlink">Internal Format</a></h2> The internal format for Unicode objects should use a Python specific fixed format <PythonUnicode> implemented as ‘unsigned short’ (or another unsigned numeric type having 16 bits). Byte order is platform dependent. This format will hold UTF-16 encodings of the corresponding Unicode ordinals. The Python Unicode implementation will address these values as if they were UCS-2 values. UCS-2 and UTF-16 are the same for all currently defined Unicode character points. UTF-16 without surrogates provides access to about 64k characters and covers all characters in the Basic Multilingual Plane (BMP) of Unicode. It is the Codec’s responsibility to ensure that the data they pass to the Unicode object constructor respects this assumption. The constructor does not check the data for Unicode compliance or use of surrogates. Future implementations can extend the 32 bit restriction to the full set of all UTF-16 addressable characters (around 1M characters). The Unicode API should provide interface routines from <PythonUnicode> to the compiler’s wchar_t which can be 16 or 32 bit depending on the compiler/libc/platform being used. Unicode objects should have a pointer to a cached Python string object <defenc> holding the object’s value using the <default encoding>. This is needed for performance and internal parsing (see Internal Argument Parsing) reasons. The buffer is filled when the first conversion request to the <default encoding> is issued on the object. Interning is not needed (for now), since Python identifiers are defined as being ASCII only. <code class="docutils literal notranslate">codecs.BOM</code> should return the byte order mark (BOM) for the format used internally. The codecs module should provide the following additional constants for convenience and reference (<code class="docutils literal notranslate">codecs.BOM</code> will either be <code class="docutils literal notranslate">BOM_BE</code> or <code class="docutils literal notranslate">BOM_LE</code> depending on the platform): <div class="highlight-default notranslate"><div class="highlight"><pre>BOM_BE: '\376\377' (corresponds to Unicode U+0000FEFF in UTF-16 on big endian platforms == ZERO WIDTH NO-BREAK SPACE) BOM_LE: '\377\376' (corresponds to Unicode U+0000FFFE in UTF-16 on little endian platforms == defined as being an illegal Unicode character) BOM4_BE: '\000\000\376\377' (corresponds to Unicode U+0000FEFF in UCS-4) BOM4_LE: '\377\376\000\000' (corresponds to Unicode U+0000FFFE in UCS-4) </pre></div> </div> Note that Unicode sees big endian byte order as being “correct”. The swapped order is taken to be an indicator for a “wrong” format, hence the illegal character definition. The configure script should provide aid in deciding whether Python can use the native <code class="docutils literal notranslate">wchar_t</code> type or not (it has to be a 16-bit unsigned type). </section> <section id="buffer-interface"> <h2><a class="toc-backref" href="#buffer-interface" role="doc-backlink">Buffer Interface</a></h2> Implement the buffer interface using the <defenc> Python string object as basis for <code class="docutils literal notranslate">bf_getcharbuf</code> and the internal buffer for <code class="docutils literal notranslate">bf_getreadbuf</code>. If <code class="docutils literal notranslate">bf_getcharbuf</code> is requested and the <defenc> object does not yet exist, it is created first. Note that as special case, the parser marker “s#” will not return raw Unicode UTF-16 data (which the <code class="docutils literal notranslate">bf_getreadbuf</code> returns), but instead tries to encode the Unicode object using the default encoding and then returns a pointer to the resulting string object (or raises an exception in case the conversion fails). This was done in order to prevent accidentally writing binary data to an output stream which the other end might not recognize. This has the advantage of being able to write to output streams (which typically use this interface) without additional specification of the encoding to use. If you need to access the read buffer interface of Unicode objects, use the <code class="docutils literal notranslate">PyObject_AsReadBuffer()</code> interface. The internal format can also be accessed using the ‘unicode-internal’ codec, e.g. via <code class="docutils literal notranslate">u.encode('unicode-internal')</code>. </section> <section id="pickle-marshalling"> <h2><a class="toc-backref" href="#pickle-marshalling" role="doc-backlink">Pickle/Marshalling</a></h2> Should have native Unicode object support. The objects should be encoded using platform independent encodings. Marshal should use UTF-8 and Pickle should either choose Raw-Unicode-Escape (in text mode) or UTF-8 (in binary mode) as encoding. Using UTF-8 instead of UTF-16 has the advantage of eliminating the need to store a BOM mark. </section> <section id="regular-expressions"> <h2><a class="toc-backref" href="#regular-expressions" role="doc-backlink">Regular Expressions</a></h2> Secret Labs AB is working on a Unicode-aware regular expression machinery. It works on plain 8-bit, UCS-2, and (optionally) UCS-4 internal character buffers. Also see <blockquote> <div><a class="reference external" href="http://www.unicode.org/unicode/reports/tr18/">http://www.unicode.org/unicode/reports/tr18/</a></div></blockquote> for some remarks on how to treat Unicode REs. </section> <section id="formatting-markers"> <h2><a class="toc-backref" href="#formatting-markers" role="doc-backlink">Formatting Markers</a></h2> Format markers are used in Python format strings. If Python strings are used as format strings, the following interpretations should be in effect: <div class="highlight-default notranslate"><div class="highlight"><pre>'%s': For Unicode objects this will cause coercion of the whole format string to Unicode. Note that you should use a Unicode format string to start with for performance reasons. </pre></div> </div> In case the format string is an Unicode object, all parameters are coerced to Unicode first and then put together and formatted according to the format string. Numbers are first converted to strings and then to Unicode. <div class="highlight-default notranslate"><div class="highlight"><pre>'%s': Python strings are interpreted as Unicode string using the <default encoding>. Unicode objects are taken as is. </pre></div> </div> All other string formatters should work accordingly. Example: <div class="highlight-default notranslate"><div class="highlight"><pre>u"%s %s" % (u"abc", "abc") == u"abc abc" </pre></div> </div> </section> <section id="internal-argument-parsing"> <h2><a class="toc-backref" href="#internal-argument-parsing" role="doc-backlink">Internal Argument Parsing</a></h2> These markers are used by the <code class="docutils literal notranslate">PyArg_ParseTuple()</code> APIs: <dl> <dt>“U”</dt><dd>Check for Unicode object and return a pointer to it</dd> <dt>“s”</dt><dd>For Unicode objects: return a pointer to the object’s <defenc> buffer (which uses the <default encoding>).</dd> <dt>“s#”</dt><dd>Access to the default encoded version of the Unicode object (see Buffer Interface); note that the length relates to the length of the default encoded string rather than the Unicode object length.</dd> <dt>“t#”</dt><dd>Same as “s#”.</dd> <dt>“es”</dt><dd>Takes two parameters: encoding (<code class="docutils literal notranslate">const char *</code>) and buffer (<code class="docutils literal notranslate">char **</code>).The input object is first coerced to Unicode in the usual way and then encoded into a string using the given encoding. On output, a buffer of the needed size is allocated and returned through <code class="docutils literal notranslate">*buffer</code> as NULL-terminated string. The encoded may not contain embedded NULL characters. The caller is responsible for calling <code class="docutils literal notranslate">PyMem_Free()</code> to free the allocated <code class="docutils literal notranslate">*buffer</code> after usage. </dd> <dt>“es#”</dt><dd>Takes three parameters: encoding (<code class="docutils literal notranslate">const char *</code>), buffer (<code class="docutils literal notranslate">char **</code>) and buffer_len (<code class="docutils literal notranslate">int *</code>).The input object is first coerced to Unicode in the usual way and then encoded into a string using the given encoding. If <code class="docutils literal notranslate">*buffer</code> is non-NULL, <code class="docutils literal notranslate">*buffer_len</code> must be set to <code class="docutils literal notranslate">sizeof(buffer)</code> on input. Output is then copied to <code class="docutils literal notranslate">*buffer</code>. If <code class="docutils literal notranslate">*buffer</code> is NULL, a buffer of the needed size is allocated and output copied into it. <code class="docutils literal notranslate">*buffer</code> is then updated to point to the allocated memory area. The caller is responsible for calling <code class="docutils literal notranslate">PyMem_Free()</code> to free the allocated <code class="docutils literal notranslate">*buffer</code> after usage. In both cases <code class="docutils literal notranslate">*buffer_len</code> is updated to the number of characters written (excluding the trailing NULL-byte). The output buffer is assured to be NULL-terminated. </dd> </dl> Examples: Using “es#” with auto-allocation: <div class="highlight-default notranslate"><div class="highlight"><pre>static PyObject * test_parser(PyObject *self, PyObject *args) { PyObject *str; const char *encoding = "latin-1"; char *buffer = NULL; int buffer_len = 0; if (!PyArg_ParseTuple(args, "es#:test_parser", encoding, &buffer, &buffer_len)) return NULL; if (!buffer) { PyErr_SetString(PyExc_SystemError, "buffer is NULL"); return NULL; } str = PyString_FromStringAndSize(buffer, buffer_len); PyMem_Free(buffer); return str; } </pre></div> </div> Using “es” with auto-allocation returning a NULL-terminated string: <div class="highlight-default notranslate"><div class="highlight"><pre>static PyObject * test_parser(PyObject *self, PyObject *args) { PyObject *str; const char *encoding = "latin-1"; char *buffer = NULL; if (!PyArg_ParseTuple(args, "es:test_parser", encoding, &buffer)) return NULL; if (!buffer) { PyErr_SetString(PyExc_SystemError, "buffer is NULL"); return NULL; } str = PyString_FromString(buffer); PyMem_Free(buffer); return str; } </pre></div> </div> Using “es#” with a pre-allocated buffer: <div class="highlight-default notranslate"><div class="highlight"><pre>static PyObject * test_parser(PyObject *self, PyObject *args) { PyObject *str; const char *encoding = "latin-1"; char _buffer[10]; char *buffer = _buffer; int buffer_len = sizeof(_buffer); if (!PyArg_ParseTuple(args, "es#:test_parser", encoding, &buffer, &buffer_len)) return NULL; if (!buffer) { PyErr_SetString(PyExc_SystemError, "buffer is NULL"); return NULL; } str = PyString_FromStringAndSize(buffer, buffer_len); return str; } </pre></div> </div> </section> <section id="file-stream-output"> <h2><a class="toc-backref" href="#file-stream-output" role="doc-backlink">File/Stream Output</a></h2> Since file.write(object) and most other stream writers use the “s#” or “t#” argument parsing marker for querying the data to write, the default encoded string version of the Unicode object will be written to the streams (see Buffer Interface). For explicit handling of files using Unicode, the standard stream codecs as available through the codecs module should be used. The codecs module should provide a short-cut open(filename,mode,encoding) available which also assures that mode contains the ‘b’ character when needed. </section> <section id="file-stream-input"> <h2><a class="toc-backref" href="#file-stream-input" role="doc-backlink">File/Stream Input</a></h2> Only the user knows what encoding the input data uses, so no special magic is applied. The user will have to explicitly convert the string data to Unicode objects as needed or use the file wrappers defined in the codecs module (see File/Stream Output). </section> <section id="unicode-methods-attributes"> <h2><a class="toc-backref" href="#unicode-methods-attributes" role="doc-backlink">Unicode Methods & Attributes</a></h2> All Python string methods, plus: <div class="highlight-default notranslate"><div class="highlight"><pre>.encode([encoding=<default encoding>][,errors="strict"]) --> see Unicode Output .splitlines([include_breaks=0]) --> breaks the Unicode string into a list of (Unicode) lines; returns the lines with line breaks included, if include_breaks is true. See Line Breaks for a specification of how line breaking is done. </pre></div> </div> </section> <section id="code-base"> <h2><a class="toc-backref" href="#code-base" role="doc-backlink">Code Base</a></h2> We should use Fredrik Lundh’s Unicode object implementation as basis. It already implements most of the string methods needed and provides a well written code base which we can build upon. The object sharing implemented in Fredrik’s implementation should be dropped. </section> <section id="test-cases"> <h2><a class="toc-backref" href="#test-cases" role="doc-backlink">Test Cases</a></h2> Test cases should follow those in Lib/test/test_string.py and include additional checks for the Codec Registry and the Standard Codecs. </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <ul class="simple"> <li>Unicode Consortium: <a class="reference external" href="http://www.unicode.org/">http://www.unicode.org/</a></li> <li>Unicode FAQ: <a class="reference external" href="http://www.unicode.org/unicode/faq/">http://www.unicode.org/unicode/faq/</a></li> <li>Unicode 3.0: <a class="reference external" href="http://www.unicode.org/unicode/standard/versions/Unicode3.0.html">http://www.unicode.org/unicode/standard/versions/Unicode3.0.html</a></li> <li>Unicode-TechReports: <a class="reference external" href="http://www.unicode.org/unicode/reports/techreports.html">http://www.unicode.org/unicode/reports/techreports.html</a></li> <li>Unicode-Mappings: <a class="reference external" href="ftp://ftp.unicode.org/Public/MAPPINGS/">ftp://ftp.unicode.org/Public/MAPPINGS/</a></li> <li>Introduction to Unicode (a little outdated by still nice to read): <a class="reference external" href="http://www.nada.kth.se/i18n/ucs/unicode-iso10646-oview.html">http://www.nada.kth.se/i18n/ucs/unicode-iso10646-oview.html</a></li> <li>For comparison: Introducing Unicode to ECMAScript (aka JavaScript) – <a class="reference external" href="http://www-4.ibm.com/software/developer/library/internationalization-support.html">http://www-4.ibm.com/software/developer/library/internationalization-support.html</a></li> <li>IANA Character Set Names: <a class="reference external" href="ftp://ftp.isi.edu/in-notes/iana/assignments/character-sets">ftp://ftp.isi.edu/in-notes/iana/assignments/character-sets</a></li> <li>Discussion of UTF-8 and Unicode support for POSIX and Linux: <a class="reference external" href="http://www.cl.cam.ac.uk/~mgk25/unicode.html">http://www.cl.cam.ac.uk/~mgk25/unicode.html</a></li> <li>Encodings:<ul> <li>Overview: <a class="reference external" href="http://czyborra.com/utf/">http://czyborra.com/utf/</a></li> <li>UCS-2: <a class="reference external" href="http://www.uazone.org/multiling/unicode/ucs2.html">http://www.uazone.org/multiling/unicode/ucs2.html</a></li> <li>UTF-7: Defined in <a class="rfc reference external" href="https://datatracker.ietf.org/doc/html/rfc2152.html">RFC 2152</a></li> <li>UTF-8: Defined in <a class="rfc reference external" href="https://datatracker.ietf.org/doc/html/rfc2279.html">RFC 2279</a></li> <li>UTF-16: <a class="reference external" href="http://www.uazone.org/multiling/unicode/wg2n1035.html">http://www.uazone.org/multiling/unicode/wg2n1035.html</a></li> </ul> </li> </ul> </section> <section id="history-of-this-proposal"> <h2><a class="toc-backref" href="#history-of-this-proposal" role="doc-backlink">History of this Proposal</a></h2> [ed. note: revisions prior to 1.7 are available in the CVS history of Misc/unicode.txt from the standard Python distribution. All subsequent history is available via the CVS revisions on this file.] <section id="id1"> <h3><a class="toc-backref" href="#id1" role="doc-backlink">1.7</a></h3> <ul class="simple"> <li>Added note about the changed behaviour of “s#”.</li> </ul> </section> <section id="id2"> <h3><a class="toc-backref" href="#id2" role="doc-backlink">1.6</a></h3> <ul class="simple"> <li>Changed <defencstr> to <defenc> since this is the name used in the implementation.</li> <li>Added notes about the usage of <defenc> in the buffer protocol implementation.</li> </ul> </section> <section id="id3"> <h3><a class="toc-backref" href="#id3" role="doc-backlink">1.5</a></h3> <ul class="simple"> <li>Added notes about setting the <default encoding>.</li> <li>Fixed some typos (thanks to Andrew Kuchling).</li> <li>Changed <defencstr> to <utf8str>.</li> </ul> </section> <section id="id4"> <h3><a class="toc-backref" href="#id4" role="doc-backlink">1.4</a></h3> <ul class="simple"> <li>Added note about mixed type comparisons and contains tests.</li> <li>Changed treating of Unicode objects in format strings (if used with <code class="docutils literal notranslate">'%s' % u</code> they will now cause the format string to be coerced to Unicode, thus producing a Unicode object on return).</li> <li>Added link to IANA charset names (thanks to Lars Marius Garshol).</li> <li>Added new codec methods <code class="docutils literal notranslate">.readline()</code>, <code class="docutils literal notranslate">.readlines()</code> and <code class="docutils literal notranslate">.writelines()</code>.</li> </ul> </section> <section id="id5"> <h3><a class="toc-backref" href="#id5" role="doc-backlink">1.3</a></h3> <ul class="simple"> <li>Added new “es” and “es#” parser markers</li> </ul> </section> <section id="id6"> <h3><a class="toc-backref" href="#id6" role="doc-backlink">1.2</a></h3> <ul class="simple"> <li>Removed POD about <code class="docutils literal notranslate">codecs.open()</code></li> </ul> </section> <section id="id7"> <h3><a class="toc-backref" href="#id7" role="doc-backlink">1.1</a></h3> <ul class="simple"> <li>Added note about comparisons and hash values.</li> <li>Added note about case mapping algorithms.</li> <li>Changed stream codecs <code class="docutils literal notranslate">.read()</code> and <code class="docutils literal notranslate">.write()</code> method to match the standard file-like object methods (bytes consumed information is no longer returned by the methods)</li> </ul> </section> <section id="id8"> <h3><a class="toc-backref" href="#id8" role="doc-backlink">1.0</a></h3> <ul class="simple"> <li>changed encode Codec method to be symmetric to the decode method (they both return (object, data consumed) now and thus become interchangeable);</li> <li>removed <code class="docutils literal notranslate">__init__</code> method of Codec class (the methods are stateless) and moved the errors argument down to the methods;</li> <li>made the Codec design more generic w/r to type of input and output objects;</li> <li>changed <code class="docutils literal notranslate">StreamWriter.flush</code> to <code class="docutils literal notranslate">StreamWriter.reset</code> in order to avoid overriding the stream’s <code class="docutils literal notranslate">.flush()</code> method;</li> <li>renamed <code class="docutils literal notranslate">.breaklines()</code> to <code class="docutils literal notranslate">.splitlines()</code>;</li> <li>renamed the module unicodec to codecs;</li> <li>modified the File I/O section to refer to the stream codecs.</li> </ul> </section> <section id="id9"> <h3><a class="toc-backref" href="#id9" role="doc-backlink">0.9</a></h3> <ul class="simple"> <li>changed errors keyword argument definition;</li> <li>added ‘replace’ error handling;</li> <li>changed the codec APIs to accept buffer like objects on input;</li> <li>some minor typo fixes;</li> <li>added Whitespace section and included references for Unicode characters that have the whitespace and the line break characteristic;</li> <li>added note that search functions can expect lower-case encoding names;</li> <li>dropped slicing and offsets in the codec APIs</li> </ul> </section> <section id="id10"> <h3><a class="toc-backref" href="#id10" role="doc-backlink">0.8</a></h3> <ul class="simple"> <li>added encodings package and raw unicode escape encoding;</li> <li>untabified the proposal;</li> <li>added notes on Unicode format strings;</li> <li>added <code class="docutils literal notranslate">.breaklines()</code> method</li> </ul> </section> <section id="id11"> <h3><a class="toc-backref" href="#id11" role="doc-backlink">0.7</a></h3> <ul class="simple"> <li>added a whole new set of codec APIs;</li> <li>added a different encoder lookup scheme;</li> <li>fixed some names</li> </ul> </section> <section id="id12"> <h3><a class="toc-backref" href="#id12" role="doc-backlink">0.6</a></h3> <ul class="simple"> <li>changed “s#” to “t#”;</li> <li>changed <defencbuf> to <defencstr> holding a real Python string object;</li> <li>changed Buffer Interface to delegate requests to <defencstr>’s buffer interface;</li> <li>removed the explicit reference to the unicodec.codecs dictionary (the module can implement this in way fit for the purpose);</li> <li>removed the settable default encoding;</li> <li>move <code class="docutils literal notranslate">UnicodeError</code> from unicodec to exceptions;</li> <li>“s#” not returns the internal data;</li> <li>passed the UCS-2/UTF-16 checking from the Unicode constructor to the Codecs</li> </ul> </section> <section id="id13"> <h3><a class="toc-backref" href="#id13" role="doc-backlink">0.5</a></h3> <ul class="simple"> <li>moved <code class="docutils literal notranslate">sys.bom</code> to <code class="docutils literal notranslate">unicodec.BOM</code>;</li> <li>added sections on case mapping,</li> <li>private use encodings and Unicode character properties</li> </ul> </section> <section id="id14"> <h3><a class="toc-backref" href="#id14" role="doc-backlink">0.4</a></h3> <ul class="simple"> <li>added Codec interface, notes on %-formatting,</li> <li>changed some encoding details,</li> <li>added comments on stream wrappers,</li> <li>fixed some discussion points (most important: Internal Format),</li> <li>clarified the ‘unicode-escape’ encoding, added encoding references</li> </ul> </section> <section id="id15"> <h3><a class="toc-backref" href="#id15" role="doc-backlink">0.3</a></h3> <ul class="simple"> <li>added references, comments on codec modules, the internal format, bf_getcharbuffer and the RE engine;</li> <li>added ‘unicode-escape’ encoding proposed by Tim Peters and fixed repr(u) accordingly</li> </ul> </section> <section id="id16"> <h3><a class="toc-backref" href="#id16" role="doc-backlink">0.2</a></h3> <ul class="simple"> <li>integrated Guido’s suggestions, added stream codecs and file wrapping</li> </ul> </section> <section id="id17"> <h3><a class="toc-backref" href="#id17" role="doc-backlink">0.1</a></h3> <ul class="simple"> <li>first version</li> </ul> </section> </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0100.rst">https://github.com/python/peps/blob/main/peps/pep-0100.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0100.rst">2025-02-01 08:55:40 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#historical-note">Historical Note</a></li> <li><a class="reference internal" href="#introduction">Introduction</a></li> <li><a class="reference internal" href="#conventions">Conventions</a></li> <li><a class="reference internal" href="#general-remarks">General Remarks</a></li> <li><a class="reference internal" href="#unicode-default-encoding">Unicode Default Encoding</a></li> <li><a class="reference internal" href="#unicode-constructors">Unicode Constructors</a></li> <li><a class="reference internal" href="#unicode-type-object">Unicode Type Object</a></li> <li><a class="reference internal" href="#unicode-output">Unicode Output</a></li> <li><a class="reference internal" href="#unicode-ordinals">Unicode Ordinals</a></li> <li><a class="reference internal" href="#comparison-hash-value">Comparison & Hash Value</a></li> <li><a class="reference internal" href="#coercion">Coercion</a></li> <li><a class="reference internal" href="#exceptions">Exceptions</a></li> <li><a class="reference internal" href="#codecs-coder-decoders-lookup">Codecs (Coder/Decoders) Lookup</a></li> <li><a class="reference internal" href="#standard-codecs">Standard Codecs</a></li> <li><a class="reference internal" href="#codecs-interface-definition">Codecs Interface Definition</a></li> <li><a class="reference internal" href="#whitespace">Whitespace</a></li> <li><a class="reference internal" href="#case-conversion">Case Conversion</a></li> <li><a class="reference internal" href="#line-breaks">Line Breaks</a></li> <li><a class="reference internal" href="#unicode-character-properties">Unicode Character Properties</a></li> <li><a class="reference internal" href="#private-code-point-areas">Private Code Point Areas</a></li> <li><a class="reference internal" href="#internal-format">Internal Format</a></li> <li><a class="reference internal" href="#buffer-interface">Buffer Interface</a></li> <li><a class="reference internal" href="#pickle-marshalling">Pickle/Marshalling</a></li> <li><a class="reference internal" href="#regular-expressions">Regular Expressions</a></li> <li><a class="reference internal" href="#formatting-markers">Formatting Markers</a></li> <li><a class="reference internal" href="#internal-argument-parsing">Internal Argument Parsing</a></li> <li><a class="reference internal" href="#file-stream-output">File/Stream Output</a></li> <li><a class="reference internal" href="#file-stream-input">File/Stream Input</a></li> <li><a class="reference internal" href="#unicode-methods-attributes">Unicode Methods & Attributes</a></li> <li><a class="reference internal" href="#code-base">Code Base</a></li> <li><a class="reference internal" href="#test-cases">Test Cases</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#history-of-this-proposal">History of this Proposal</a><ul> <li><a class="reference internal" href="#id1">1.7</a></li> <li><a class="reference internal" href="#id2">1.6</a></li> <li><a class="reference internal" href="#id3">1.5</a></li> <li><a class="reference internal" href="#id4">1.4</a></li> <li><a class="reference internal" href="#id5">1.3</a></li> <li><a class="reference internal" href="#id6">1.2</a></li> <li><a class="reference internal" href="#id7">1.1</a></li> <li><a class="reference internal" href="#id8">1.0</a></li> <li><a class="reference internal" href="#id9">0.9</a></li> <li><a class="reference internal" href="#id10">0.8</a></li> <li><a class="reference internal" href="#id11">0.7</a></li> <li><a class="reference internal" href="#id12">0.6</a></li> <li><a class="reference internal" href="#id13">0.5</a></li> <li><a class="reference internal" href="#id14">0.4</a></li> <li><a class="reference internal" href="#id15">0.3</a></li> <li><a class="reference internal" href="#id16">0.2</a></li> <li><a class="reference internal" href="#id17">0.1</a></li> </ul> </li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0100.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>