CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 461 – Adding % formatting to bytes and bytearray | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0461/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 461 – Adding % formatting to bytes and bytearray | peps.python.org'> <meta property="og:description" content="This PEP proposes adding % formatting operations similar to Python 2’s str type to bytes and bytearray 1 2."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0461/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This PEP proposes adding % formatting operations similar to Python 2’s str type to bytes and bytearray 1 2."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 461</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 461 – Adding % formatting to bytes and bytearray</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Ethan Furman <ethan at stoneleaf.us></dd> <dt class="field-even">Status:</dt> <dd class="field-even"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-odd">Type:</dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created:</dt> <dd class="field-even">13-Jan-2014</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">3.5</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even">14-Jan-2014, 15-Jan-2014, 17-Jan-2014, 22-Feb-2014, 25-Mar-2014, 27-Mar-2014</dd> <dt class="field-odd">Resolution:</dt> <dd class="field-odd"><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2014-March/133621.html">Python-Dev message</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#proposed-semantics-for-bytes-and-bytearray-formatting">Proposed semantics for <code class="docutils literal notranslate">bytes</code> and <code class="docutils literal notranslate">bytearray</code> formatting</a><ul> <li><a class="reference internal" href="#interpolation">%-interpolation</a></li> </ul> </li> <li><a class="reference internal" href="#compatibility-with-python-2">Compatibility with Python 2</a></li> <li><a class="reference internal" href="#proposed-variations">Proposed variations</a></li> <li><a class="reference internal" href="#objections">Objections</a></li> <li><a class="reference internal" href="#footnotes">Footnotes</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> This PEP proposes adding % formatting operations similar to Python 2’s <code class="docutils literal notranslate">str</code> type to <code class="docutils literal notranslate">bytes</code> and <code class="docutils literal notranslate">bytearray</code> <a class="footnote-reference brackets" href="#id9" id="id1">[1]</a> <a class="footnote-reference brackets" href="#id10" id="id2">[2]</a>. </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> While interpolation is usually thought of as a string operation, there are cases where interpolation on <code class="docutils literal notranslate">bytes</code> or <code class="docutils literal notranslate">bytearrays</code> make sense, and the work needed to make up for this missing functionality detracts from the overall readability of the code. </section> <section id="motivation"> <h2><a class="toc-backref" href="#motivation" role="doc-backlink">Motivation</a></h2> With Python 3 and the split between <code class="docutils literal notranslate">str</code> and <code class="docutils literal notranslate">bytes</code>, one small but important area of programming became slightly more difficult, and much more painful – wire format protocols <a class="footnote-reference brackets" href="#id11" id="id3">[3]</a>. This area of programming is characterized by a mixture of binary data and ASCII compatible segments of text (aka ASCII-encoded text). Bringing back a restricted %-interpolation for <code class="docutils literal notranslate">bytes</code> and <code class="docutils literal notranslate">bytearray</code> will aid both in writing new wire format code, and in porting Python 2 wire format code. Common use-cases include <code class="docutils literal notranslate">dbf</code> and <code class="docutils literal notranslate">pdf</code> file formats, <code class="docutils literal notranslate">email</code> formats, and <code class="docutils literal notranslate">FTP</code> and <code class="docutils literal notranslate">HTTP</code> communications, among many others. </section> <section id="proposed-semantics-for-bytes-and-bytearray-formatting"> <h2><a class="toc-backref" href="#proposed-semantics-for-bytes-and-bytearray-formatting" role="doc-backlink">Proposed semantics for <code class="docutils literal notranslate">bytes</code> and <code class="docutils literal notranslate">bytearray</code> formatting</a></h2> <section id="interpolation"> <h3><a class="toc-backref" href="#interpolation" role="doc-backlink">%-interpolation</a></h3> All the numeric formatting codes (<code class="docutils literal notranslate">d</code>, <code class="docutils literal notranslate">i</code>, <code class="docutils literal notranslate">o</code>, <code class="docutils literal notranslate">u</code>, <code class="docutils literal notranslate">x</code>, <code class="docutils literal notranslate">X</code>, <code class="docutils literal notranslate">e</code>, <code class="docutils literal notranslate">E</code>, <code class="docutils literal notranslate">f</code>, <code class="docutils literal notranslate">F</code>, <code class="docutils literal notranslate">g</code>, <code class="docutils literal notranslate">G</code>, and any that are subsequently added to Python 3) will be supported, and will work as they do for str, including the padding, justification and other related modifiers (currently <code class="docutils literal notranslate">#</code>, <code class="docutils literal notranslate">0</code>, <code class="docutils literal notranslate">-</code>, space, and <code class="docutils literal notranslate">+</code> (plus any added to Python 3)). The only non-numeric codes allowed are <code class="docutils literal notranslate">c</code>, <code class="docutils literal notranslate">b</code>, <code class="docutils literal notranslate">a</code>, and <code class="docutils literal notranslate">s</code> (which is a synonym for b). For the numeric codes, the only difference between <code class="docutils literal notranslate">str</code> and <code class="docutils literal notranslate">bytes</code> (or <code class="docutils literal notranslate">bytearray</code>) interpolation is that the results from these codes will be ASCII-encoded text, not unicode. In other words, for any numeric formatting code <code class="docutils literal notranslate">%x</code>: <div class="highlight-default notranslate"><div class="highlight"><pre>b"%x" % val </pre></div> </div> is equivalent to: <div class="highlight-default notranslate"><div class="highlight"><pre>("%x" % val).encode("ascii") </pre></div> </div> Examples: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> b'%4x' % 10 b' a' >>> b'%#4x' % 10 ' 0xa' >>> b'%04X' % 10 '000A' </pre></div> </div> <code class="docutils literal notranslate">%c</code> will insert a single byte, either from an <code class="docutils literal notranslate">int</code> in range(256), or from a <code class="docutils literal notranslate">bytes</code> argument of length 1, not from a <code class="docutils literal notranslate">str</code>. Examples: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> b'%c' % 48 b'0' >>> b'%c' % b'a' b'a' </pre></div> </div> <code class="docutils literal notranslate">%b</code> will insert a series of bytes. These bytes are collected in one of two ways: <ul class="simple"> <li>input type supports <code class="docutils literal notranslate">Py_buffer</code> <a class="footnote-reference brackets" href="#id12" id="id4">[4]</a>? use it to collect the necessary bytes</li> <li>input type is something else? use its <code class="docutils literal notranslate">__bytes__</code> method <a class="footnote-reference brackets" href="#id13" id="id5">[5]</a> ; if there isn’t one, raise a <code class="docutils literal notranslate">TypeError</code></li> </ul> In particular, <code class="docutils literal notranslate">%b</code> will not accept numbers nor <code class="docutils literal notranslate">str</code>. <code class="docutils literal notranslate">str</code> is rejected as the string to bytes conversion requires an encoding, and we are refusing to guess; numbers are rejected because: <ul class="simple"> <li>what makes a number is fuzzy (float? Decimal? Fraction? some user type?)</li> <li>allowing numbers would lead to ambiguity between numbers and textual representations of numbers (3.14 vs ‘3.14’)</li> <li>given the nature of wire formats, explicit is definitely better than implicit</li> </ul> <code class="docutils literal notranslate">%s</code> is included as a synonym for <code class="docutils literal notranslate">%b</code> for the sole purpose of making 2/3 code bases easier to maintain. Python 3 only code should use <code class="docutils literal notranslate">%b</code>. Examples: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> b'%b' % b'abc' b'abc' >>> b'%b' % 'some string'.encode('utf8') b'some string' >>> b'%b' % 3.14 Traceback (most recent call last): ... TypeError: b'%b' does not accept 'float' >>> b'%b' % 'hello world!' Traceback (most recent call last): ... TypeError: b'%b' does not accept 'str' </pre></div> </div> <code class="docutils literal notranslate">%a</code> will give the equivalent of <code class="docutils literal notranslate">repr(some_obj).encode('ascii', 'backslashreplace')</code> on the interpolated value. Use cases include developing a new protocol and writing landmarks into the stream; debugging data going into an existing protocol to see if the problem is the protocol itself or bad data; a fall-back for a serialization format; or any situation where defining <code class="docutils literal notranslate">__bytes__</code> would not be appropriate but a readable/informative representation is needed <a class="footnote-reference brackets" href="#id14" id="id6">[6]</a>. <code class="docutils literal notranslate">%r</code> is included as a synonym for <code class="docutils literal notranslate">%a</code> for the sole purpose of making 2/3 code bases easier to maintain. Python 3 only code use <code class="docutils literal notranslate">%a</code> <a class="footnote-reference brackets" href="#id15" id="id7">[7]</a>. Examples: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> b'%a' % 3.14 b'3.14' >>> b'%a' % b'abc' b"b'abc'" >>> b'%a' % 'def' b"'def'" </pre></div> </div> </section> </section> <section id="compatibility-with-python-2"> <h2><a class="toc-backref" href="#compatibility-with-python-2" role="doc-backlink">Compatibility with Python 2</a></h2> As noted above, <code class="docutils literal notranslate">%s</code> and <code class="docutils literal notranslate">%r</code> are being included solely to help ease migration from, and/or have a single code base with, Python 2. This is important as there are modules both in the wild and behind closed doors that currently use the Python 2 <code class="docutils literal notranslate">str</code> type as a <code class="docutils literal notranslate">bytes</code> container, and hence are using <code class="docutils literal notranslate">%s</code> as a bytes interpolator. However, <code class="docutils literal notranslate">%b</code> and <code class="docutils literal notranslate">%a</code> should be used in new, Python 3 only code, so <code class="docutils literal notranslate">%s</code> and <code class="docutils literal notranslate">%r</code> will immediately be deprecated, but not removed from the 3.x series <a class="footnote-reference brackets" href="#id15" id="id8">[7]</a>. </section> <section id="proposed-variations"> <h2><a class="toc-backref" href="#proposed-variations" role="doc-backlink">Proposed variations</a></h2> It has been proposed to automatically use <code class="docutils literal notranslate">.encode('ascii','strict')</code> for <code class="docutils literal notranslate">str</code> arguments to <code class="docutils literal notranslate">%b</code>. <ul class="simple"> <li>Rejected as this would lead to intermittent failures. Better to have the operation always fail so the trouble-spot can be correctly fixed.</li> </ul> It has been proposed to have <code class="docutils literal notranslate">%b</code> return the ascii-encoded repr when the value is a <code class="docutils literal notranslate">str</code> (b’%b’ % ‘abc’ –> b“‘abc’”). <ul class="simple"> <li>Rejected as this would lead to hard to debug failures far from the problem site. Better to have the operation always fail so the trouble-spot can be easily fixed.</li> </ul> Originally this PEP also proposed adding format-style formatting, but it was decided that format and its related machinery were all strictly text (aka <code class="docutils literal notranslate">str</code>) based, and it was dropped. Various new special methods were proposed, such as <code class="docutils literal notranslate">__ascii__</code>, <code class="docutils literal notranslate">__format_bytes__</code>, etc.; such methods are not needed at this time, but can be visited again later if real-world use shows deficiencies with this solution. A competing PEP, <a class="pep reference internal" href="../pep-0460/" title="PEP 460 – Add binary interpolation and formatting">PEP 460 Add binary interpolation and formatting</a>, also exists. </section> <section id="objections"> <h2><a class="toc-backref" href="#objections" role="doc-backlink">Objections</a></h2> The objections raised against this PEP were mainly variations on two themes: <ul class="simple"> <li>the <code class="docutils literal notranslate">bytes</code> and <code class="docutils literal notranslate">bytearray</code> types are for pure binary data, with no assumptions about encodings</li> <li>offering %-interpolation that assumes an ASCII encoding will be an attractive nuisance and lead us back to the problems of the Python 2 <code class="docutils literal notranslate">str</code>/<code class="docutils literal notranslate">unicode</code> text model</li> </ul> As was seen during the discussion, <code class="docutils literal notranslate">bytes</code> and <code class="docutils literal notranslate">bytearray</code> are also used for mixed binary data and ASCII-compatible segments: file formats such as <code class="docutils literal notranslate">dbf</code> and <code class="docutils literal notranslate">pdf</code>, network protocols such as <code class="docutils literal notranslate">ftp</code> and <code class="docutils literal notranslate">email</code>, etc. <code class="docutils literal notranslate">bytes</code> and <code class="docutils literal notranslate">bytearray</code> already have several methods which assume an ASCII compatible encoding. <code class="docutils literal notranslate">upper()</code>, <code class="docutils literal notranslate">isalpha()</code>, and <code class="docutils literal notranslate">expandtabs()</code> to name just a few. %-interpolation, with its very restricted mini-language, will not be any more of a nuisance than the already existing methods. Some have objected to allowing the full range of numeric formatting codes with the claim that decimal alone would be sufficient. However, at least two formats (dbf and pdf) make use of non-decimal numbers. </section> <section id="footnotes"> <h2><a class="toc-backref" href="#footnotes" role="doc-backlink">Footnotes</a></h2> <aside class="footnote-list brackets"> <aside class="footnote brackets" id="id9" role="doc-footnote"> <dt class="label" id="id9">[<a href="#id1">1</a>]</dt> <dd><a class="reference external" href="http://docs.python.org/2/library/stdtypes.html#string-formatting">http://docs.python.org/2/library/stdtypes.html#string-formatting</a></aside> <aside class="footnote brackets" id="id10" role="doc-footnote"> <dt class="label" id="id10">[<a href="#id2">2</a>]</dt> <dd>neither string.Template, format, nor str.format are under consideration</aside> <aside class="footnote brackets" id="id11" role="doc-footnote"> <dt class="label" id="id11">[<a href="#id3">3</a>]</dt> <dd><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2014-January/131518.html">https://mail.python.org/pipermail/python-dev/2014-January/131518.html</a></aside> <aside class="footnote brackets" id="id12" role="doc-footnote"> <dt class="label" id="id12">[<a href="#id4">4</a>]</dt> <dd><a class="reference external" href="http://docs.python.org/3/c-api/buffer.html">http://docs.python.org/3/c-api/buffer.html</a> examples: <code class="docutils literal notranslate">memoryview</code>, <code class="docutils literal notranslate">array.array</code>, <code class="docutils literal notranslate">bytearray</code>, <code class="docutils literal notranslate">bytes</code></aside> <aside class="footnote brackets" id="id13" role="doc-footnote"> <dt class="label" id="id13">[<a href="#id5">5</a>]</dt> <dd><a class="reference external" href="http://docs.python.org/3/reference/datamodel.html#object.__bytes__">http://docs.python.org/3/reference/datamodel.html#object.__bytes__</a></aside> <aside class="footnote brackets" id="id14" role="doc-footnote"> <dt class="label" id="id14">[<a href="#id6">6</a>]</dt> <dd><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2014-February/132750.html">https://mail.python.org/pipermail/python-dev/2014-February/132750.html</a></aside> <aside class="footnote brackets" id="id15" role="doc-footnote"> <dt class="label" id="id15">[7] (<a href='#id7'>1</a>, <a href='#id8'>2</a>) </dt> <dd><a class="reference external" href="http://bugs.python.org/issue23467">http://bugs.python.org/issue23467</a> – originally <code class="docutils literal notranslate">%r</code> was not allowed, but was added for consistency during the 3.5 alpha stage.</aside> </aside> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0461.rst">https://github.com/python/peps/blob/main/peps/pep-0461.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0461.rst">2023-09-09 17:39:29 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#proposed-semantics-for-bytes-and-bytearray-formatting">Proposed semantics for <code class="docutils literal notranslate">bytes</code> and <code class="docutils literal notranslate">bytearray</code> formatting</a><ul> <li><a class="reference internal" href="#interpolation">%-interpolation</a></li> </ul> </li> <li><a class="reference internal" href="#compatibility-with-python-2">Compatibility with Python 2</a></li> <li><a class="reference internal" href="#proposed-variations">Proposed variations</a></li> <li><a class="reference internal" href="#objections">Objections</a></li> <li><a class="reference internal" href="#footnotes">Footnotes</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0461.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>