CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 782 – Add PyBytesWriter C API | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0782/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 782 – Add PyBytesWriter C API | peps.python.org'> <meta property="og:description" content="Add a new PyBytesWriter C API to create bytes objects."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0782/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="Add a new PyBytesWriter C API to create bytes objects."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 782</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 782 – Add PyBytesWriter C API</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Victor Stinner <vstinner at python.org></dd> <dt class="field-even">Discussions-To:</dt> <dd class="field-even"><a class="reference external" href="https://discuss.python.org/t/86617">Discourse thread</a></dd> <dt class="field-odd">Status:</dt> <dd class="field-odd"><abbr title="Proposal under active discussion and revision">Draft</abbr></dd> <dt class="field-even">Type:</dt> <dd class="field-even"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-odd">Created:</dt> <dd class="field-odd">27-Mar-2025</dd> <dt class="field-even">Python-Version:</dt> <dd class="field-even">3.14</dd> <dt class="field-odd">Post-History:</dt> <dd class="field-odd"><a class="reference external" href="https://discuss.python.org/t/81182" title="Discourse thread">18-Feb-2025</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#disallow-creation-of-incomplete-inconsistent-objects">Disallow creation of incomplete/inconsistent objects</a></li> <li><a class="reference internal" href="#inefficient-allocation-strategy">Inefficient allocation strategy</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#api">API</a><ul> <li><a class="reference internal" href="#create-finish-discard">Create, Finish, Discard</a></li> <li><a class="reference internal" href="#high-level-api">High-level API</a></li> <li><a class="reference internal" href="#getters">Getters</a></li> <li><a class="reference internal" href="#low-level-api">Low-level API</a></li> </ul> </li> <li><a class="reference internal" href="#overallocation">Overallocation</a></li> <li><a class="reference internal" href="#thread-safety">Thread safety</a></li> <li><a class="reference internal" href="#soft-deprecations">Soft deprecations</a></li> </ul> </li> <li><a class="reference internal" href="#examples">Examples</a><ul> <li><a class="reference internal" href="#id1">High-level API</a></li> <li><a class="reference internal" href="#create-the-bytes-string-abc">Create the bytes string “abc”</a></li> <li><a class="reference internal" href="#growandupdatepointer-example"><code class="docutils literal notranslate">GrowAndUpdatePointer()</code> example</a></li> <li><a class="reference internal" href="#update-pybytes-fromstringandsize-code">Update <code class="docutils literal notranslate">PyBytes_FromStringAndSize()</code> code</a></li> <li><a class="reference internal" href="#update-pybytes-resize-code">Update <code class="docutils literal notranslate">_PyBytes_Resize()</code> code</a></li> </ul> </li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a></li> <li><a class="reference internal" href="#prior-discussions">Prior Discussions</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> Add a new <code class="docutils literal notranslate">PyBytesWriter</code> C API to create <code class="docutils literal notranslate">bytes</code> objects. Soft deprecate <code class="docutils literal notranslate">PyBytes_FromStringAndSize(NULL, size)</code> and <code class="docutils literal notranslate">_PyBytes_Resize()</code> APIs. These APIs treat an immutable <code class="docutils literal notranslate">bytes</code> object as a mutable object. They remain available and maintained, don’t emit deprecation warning, but are no longer recommended when writing new code. </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> <section id="disallow-creation-of-incomplete-inconsistent-objects"> <h3><a class="toc-backref" href="#disallow-creation-of-incomplete-inconsistent-objects" role="doc-backlink">Disallow creation of incomplete/inconsistent objects</a></h3> Creating a Python <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#bytes" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">bytes</code></a> object using <code class="docutils literal notranslate">PyBytes_FromStringAndSize(NULL, size)</code> and <code class="docutils literal notranslate">_PyBytes_Resize()</code> treats an immutable <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#bytes" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">bytes</code></a> object as mutable. It goes against the principle that <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#bytes" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">bytes</code></a> objects are immutable. It also creates an incomplete or “invalid” object since bytes are not initialized. In Python, a <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#bytes" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">bytes</code></a> object should always have its bytes fully initialized. <ul class="simple"> <li><a class="reference external" href="https://github.com/capi-workgroup/api-evolution/issues/36">Avoid creating incomplete/invalid objects api-evolution#36</a></li> <li><a class="reference external" href="https://github.com/capi-workgroup/api-evolution/issues/20">Disallow mutating immutable objects api-evolution#20</a></li> <li><a class="reference external" href="https://github.com/capi-workgroup/problems/issues/56">Disallow creation of incomplete/inconsistent objects problems#56</a></li> </ul> </section> <section id="inefficient-allocation-strategy"> <h3><a class="toc-backref" href="#inefficient-allocation-strategy" role="doc-backlink">Inefficient allocation strategy</a></h3> When creating a bytes string and the output size is unknown, one strategy is to allocate a short buffer and extend it (to the exact size) each time a larger write is needed. This strategy is inefficient because it requires to enlarge the buffer multiple timess. It’s more efficient to overallocate the buffer the first time that a larger write is needed. It reduces the number of expensive <code class="docutils literal notranslate">realloc()</code> operations which can imply a memory copy. </section> </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> <section id="api"> <h3><a class="toc-backref" href="#api" role="doc-backlink">API</a></h3> <dl class="c type"> <dt class="sig sig-object c" id="c.PyBytesWriter"> type PyBytesWriter </dt> <dd>A Python <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#bytes" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">bytes</code></a> writer instance created by <a class="reference internal" href="#c.PyBytesWriter_Create" title="PyBytesWriter_Create"><code class="xref c c-func docutils literal notranslate">PyBytesWriter_Create()</code></a>.The instance must be destroyed by <a class="reference internal" href="#c.PyBytesWriter_Finish" title="PyBytesWriter_Finish"><code class="xref c c-func docutils literal notranslate">PyBytesWriter_Finish()</code></a> or <a class="reference internal" href="#c.PyBytesWriter_Discard" title="PyBytesWriter_Discard"><code class="xref c c-func docutils literal notranslate">PyBytesWriter_Discard()</code></a>. </dd></dl> <section id="create-finish-discard"> <h4><a class="toc-backref" href="#create-finish-discard" role="doc-backlink">Create, Finish, Discard</a></h4> <dl class="c function"> <dt class="sig sig-object c" id="c.PyBytesWriter_Create"> <a class="reference internal" href="#c.PyBytesWriter" title="PyBytesWriter">PyBytesWriter</a> *PyBytesWriter_Create(<a class="reference external" href="https://docs.python.org/3/c-api/intro.html#c.Py_ssize_t" title="(in Python v3.13)">Py_ssize_t</a> size) </dt> <dd>Create a <a class="reference internal" href="#c.PyBytesWriter" title="PyBytesWriter"><code class="xref c c-type docutils literal notranslate">PyBytesWriter</code></a> to write size bytes.If size is greater than zero, allocate size bytes, and set the writer size to size. The caller is responsible to write size bytes using <a class="reference internal" href="#c.PyBytesWriter_GetData" title="PyBytesWriter_GetData"><code class="xref c c-func docutils literal notranslate">PyBytesWriter_GetData()</code></a>. On error, set an exception and return NULL. size must be positive or zero. </dd></dl> <dl class="c function"> <dt class="sig sig-object c" id="c.PyBytesWriter_Finish"> <a class="reference external" href="https://docs.python.org/3/c-api/structures.html#c.PyObject" title="(in Python v3.13)">PyObject</a> *PyBytesWriter_Finish(<a class="reference internal" href="#c.PyBytesWriter" title="PyBytesWriter">PyBytesWriter</a> *writer) </dt> <dd>Finish a <a class="reference internal" href="#c.PyBytesWriter" title="PyBytesWriter"><code class="xref c c-type docutils literal notranslate">PyBytesWriter</code></a> created by <a class="reference internal" href="#c.PyBytesWriter_Create" title="PyBytesWriter_Create"><code class="xref c c-func docutils literal notranslate">PyBytesWriter_Create()</code></a>.On success, return a Python <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#bytes" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">bytes</code></a> object. On error, set an exception and return <code class="docutils literal notranslate">NULL</code>. The writer instance is invalid after the call in any case. </dd></dl> <dl class="c function"> <dt class="sig sig-object c" id="c.PyBytesWriter_FinishWithSize"> <a class="reference external" href="https://docs.python.org/3/c-api/structures.html#c.PyObject" title="(in Python v3.13)">PyObject</a> *PyBytesWriter_FinishWithSize(<a class="reference internal" href="#c.PyBytesWriter" title="PyBytesWriter">PyBytesWriter</a> *writer, <a class="reference external" href="https://docs.python.org/3/c-api/intro.html#c.Py_ssize_t" title="(in Python v3.13)">Py_ssize_t</a> size) </dt> <dd>Similar to <a class="reference internal" href="#c.PyBytesWriter_Finish" title="PyBytesWriter_Finish"><code class="xref c c-func docutils literal notranslate">PyBytesWriter_Finish()</code></a>, but resize the writer to size bytes before creating the <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#bytes" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">bytes</code></a> object.</dd></dl> <dl class="c function"> <dt class="sig sig-object c" id="c.PyBytesWriter_FinishWithPointer"> <a class="reference external" href="https://docs.python.org/3/c-api/structures.html#c.PyObject" title="(in Python v3.13)">PyObject</a> *PyBytesWriter_FinishWithPointer(<a class="reference internal" href="#c.PyBytesWriter" title="PyBytesWriter">PyBytesWriter</a> *writer, void *buf) </dt> <dd>Similar to <a class="reference internal" href="#c.PyBytesWriter_Finish" title="PyBytesWriter_Finish"><code class="xref c c-func docutils literal notranslate">PyBytesWriter_Finish()</code></a>, but resize the writer using buf pointer before creating the <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#bytes" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">bytes</code></a> object.Set an exception and return <code class="docutils literal notranslate">NULL</code> if buf pointer is outside the internal buffer bounds. Function pseudo-code: <div class="highlight-c notranslate"><div class="highlight"><pre>Py_ssize_t size = (char*)buf - (char*)PyBytesWriter_GetData(writer); return PyBytesWriter_FinishWithSize(writer, size); </pre></div> </div> </dd></dl> <dl class="c function"> <dt class="sig sig-object c" id="c.PyBytesWriter_Discard"> void PyBytesWriter_Discard(<a class="reference internal" href="#c.PyBytesWriter" title="PyBytesWriter">PyBytesWriter</a> *writer) </dt> <dd>Discard a <a class="reference internal" href="#c.PyBytesWriter" title="PyBytesWriter"><code class="xref c c-type docutils literal notranslate">PyBytesWriter</code></a> created by <a class="reference internal" href="#c.PyBytesWriter_Create" title="PyBytesWriter_Create"><code class="xref c c-func docutils literal notranslate">PyBytesWriter_Create()</code></a>.Do nothing if writer is <code class="docutils literal notranslate">NULL</code>. The writer instance is invalid after the call. </dd></dl> </section> <section id="high-level-api"> <h4><a class="toc-backref" href="#high-level-api" role="doc-backlink">High-level API</a></h4> <dl class="c function"> <dt class="sig sig-object c" id="c.PyBytesWriter_WriteBytes"> int PyBytesWriter_WriteBytes(<a class="reference internal" href="#c.PyBytesWriter" title="PyBytesWriter">PyBytesWriter</a> *writer, const void *bytes, <a class="reference external" href="https://docs.python.org/3/c-api/intro.html#c.Py_ssize_t" title="(in Python v3.13)">Py_ssize_t</a> size) </dt> <dd>Grow the writer internal buffer by size bytes, write size bytes of bytes at the writer end, and add size to the writer size.If size is equal to <code class="docutils literal notranslate">-1</code>, call <code class="docutils literal notranslate">strlen(bytes)</code> to get the string length. On success, return <code class="docutils literal notranslate">0</code>. On error, set an exception and return <code class="docutils literal notranslate">-1</code>. </dd></dl> <dl class="c function"> <dt class="sig sig-object c" id="c.PyBytesWriter_Format"> int PyBytesWriter_Format(<a class="reference internal" href="#c.PyBytesWriter" title="PyBytesWriter">PyBytesWriter</a> *writer, const char *format, ...) </dt> <dd>Similar to <code class="docutils literal notranslate">PyBytes_FromFormat()</code>, but write the output directly at the writer end. Grow the writer internal buffer on demand. Then add the written size to the writer size.On success, return <code class="docutils literal notranslate">0</code>. On error, set an exception and return <code class="docutils literal notranslate">-1</code>. </dd></dl> </section> <section id="getters"> <h4><a class="toc-backref" href="#getters" role="doc-backlink">Getters</a></h4> <dl class="c function"> <dt class="sig sig-object c" id="c.PyBytesWriter_GetSize"> <a class="reference external" href="https://docs.python.org/3/c-api/intro.html#c.Py_ssize_t" title="(in Python v3.13)">Py_ssize_t</a> PyBytesWriter_GetSize(<a class="reference internal" href="#c.PyBytesWriter" title="PyBytesWriter">PyBytesWriter</a> *writer) </dt> <dd>Get the writer size.</dd></dl> <dl class="c function"> <dt class="sig sig-object c" id="c.PyBytesWriter_GetData"> void *PyBytesWriter_GetData(<a class="reference internal" href="#c.PyBytesWriter" title="PyBytesWriter">PyBytesWriter</a> *writer) </dt> <dd>Get the writer data: start of the internal buffer.The pointer is valid until <a class="reference internal" href="#c.PyBytesWriter_Finish" title="PyBytesWriter_Finish"><code class="xref c c-func docutils literal notranslate">PyBytesWriter_Finish()</code></a> or <a class="reference internal" href="#c.PyBytesWriter_Discard" title="PyBytesWriter_Discard"><code class="xref c c-func docutils literal notranslate">PyBytesWriter_Discard()</code></a> is called on writer. </dd></dl> </section> <section id="low-level-api"> <h4><a class="toc-backref" href="#low-level-api" role="doc-backlink">Low-level API</a></h4> <dl class="c function"> <dt class="sig sig-object c" id="c.PyBytesWriter_Resize"> int PyBytesWriter_Resize(<a class="reference internal" href="#c.PyBytesWriter" title="PyBytesWriter">PyBytesWriter</a> *writer, <a class="reference external" href="https://docs.python.org/3/c-api/intro.html#c.Py_ssize_t" title="(in Python v3.13)">Py_ssize_t</a> size) </dt> <dd>Resize the writer to size bytes. It can be used to enlarge or to shrink the writer.Newly allocated bytes are left uninitialized. On success, return <code class="docutils literal notranslate">0</code>. On error, set an exception and return <code class="docutils literal notranslate">-1</code>. size must be positive or zero. </dd></dl> <dl class="c function"> <dt class="sig sig-object c" id="c.PyBytesWriter_Grow"> int PyBytesWriter_Grow(<a class="reference internal" href="#c.PyBytesWriter" title="PyBytesWriter">PyBytesWriter</a> *writer, <a class="reference external" href="https://docs.python.org/3/c-api/intro.html#c.Py_ssize_t" title="(in Python v3.13)">Py_ssize_t</a> grow) </dt> <dd>Resize the writer by adding grow bytes to the current writer size.Newly allocated bytes are left uninitialized. On success, return <code class="docutils literal notranslate">0</code>. On error, set an exception and return <code class="docutils literal notranslate">-1</code>. size can be negative to shrink the writer. </dd></dl> <dl class="c function"> <dt class="sig sig-object c" id="c.PyBytesWriter_GrowAndUpdatePointer"> void *PyBytesWriter_GrowAndUpdatePointer(<a class="reference internal" href="#c.PyBytesWriter" title="PyBytesWriter">PyBytesWriter</a> *writer, <a class="reference external" href="https://docs.python.org/3/c-api/intro.html#c.Py_ssize_t" title="(in Python v3.13)">Py_ssize_t</a> size, void *buf) </dt> <dd>Similar to <a class="reference internal" href="#c.PyBytesWriter_Grow" title="PyBytesWriter_Grow"><code class="xref c c-func docutils literal notranslate">PyBytesWriter_Grow()</code></a>, but update also the buf pointer.The buf pointer is moved if the internal buffer is moved in memory. The buf relative position within the internal buffer is left unchanged. On error, set an exception and return <code class="docutils literal notranslate">NULL</code>. buf must not be <code class="docutils literal notranslate">NULL</code>. Function pseudo-code: <div class="highlight-c notranslate"><div class="highlight"><pre>Py_ssize_t pos = (char*)buf - (char*)PyBytesWriter_GetData(writer); if (PyBytesWriter_Grow(writer, size) < 0) { return NULL; } return (char*)PyBytesWriter_GetData(writer) + pos; </pre></div> </div> </dd></dl> </section> </section> <section id="overallocation"> <h3><a class="toc-backref" href="#overallocation" role="doc-backlink">Overallocation</a></h3> <a class="reference internal" href="#c.PyBytesWriter_Resize" title="PyBytesWriter_Resize"><code class="xref c c-func docutils literal notranslate">PyBytesWriter_Resize()</code></a> and <a class="reference internal" href="#c.PyBytesWriter_Grow" title="PyBytesWriter_Grow"><code class="xref c c-func docutils literal notranslate">PyBytesWriter_Grow()</code></a> overallocate the internal buffer to reduce the number of <code class="docutils literal notranslate">realloc()</code> calls and so reduce memory copies. <a class="reference internal" href="#c.PyBytesWriter_Finish" title="PyBytesWriter_Finish"><code class="xref c c-func docutils literal notranslate">PyBytesWriter_Finish()</code></a> trims overallocations: it shrinks the internal buffer to the exact size when creating the final <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#bytes" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">bytes</code></a> object. </section> <section id="thread-safety"> <h3><a class="toc-backref" href="#thread-safety" role="doc-backlink">Thread safety</a></h3> The API is not thread safe: a writer should only be used by a single thread at the same time. </section> <section id="soft-deprecations"> <h3><a class="toc-backref" href="#soft-deprecations" role="doc-backlink">Soft deprecations</a></h3> Soft deprecate <code class="docutils literal notranslate">PyBytes_FromStringAndSize(NULL, size)</code> and <code class="docutils literal notranslate">_PyBytes_Resize()</code> APIs. These APIs treat an immutable <code class="docutils literal notranslate">bytes</code> object as a mutable object. They remain available and maintained, don’t emit deprecation warning, but are no longer recommended when writing new code. <code class="docutils literal notranslate">PyBytes_FromStringAndSize(str, size)</code> is not soft deprecated. Only calls with <code class="docutils literal notranslate">NULL</code> str are soft deprecated. </section> </section> <section id="examples"> <h2><a class="toc-backref" href="#examples" role="doc-backlink">Examples</a></h2> <section id="id1"> <h3><a class="toc-backref" href="#id1" role="doc-backlink">High-level API</a></h3> Create the bytes string <code class="docutils literal notranslate">b"Hello World!"</code>: <div class="highlight-c notranslate"><div class="highlight"><pre>PyObject* hello_world(void) { PyBytesWriter *writer = PyBytesWriter_Create(0); if (writer == NULL) { goto error; } if (PyBytesWriter_WriteBytes(writer, "Hello", -1) < 0) { goto error; } if (PyBytesWriter_Format(writer, " %s!", "World") < 0) { goto error; } return PyBytesWriter_Finish(writer); error: PyBytesWriter_Discard(writer); return NULL; } </pre></div> </div> </section> <section id="create-the-bytes-string-abc"> <h3><a class="toc-backref" href="#create-the-bytes-string-abc" role="doc-backlink">Create the bytes string “abc”</a></h3> Example creating the bytes string <code class="docutils literal notranslate">b"abc"</code>, with a fixed size of 3 bytes: <div class="highlight-c notranslate"><div class="highlight"><pre>PyObject* create_abc(void) { PyBytesWriter *writer = PyBytesWriter_Create(3); if (writer == NULL) { return NULL; } char *str = PyBytesWriter_GetData(writer); memcpy(str, "abc", 3); return PyBytesWriter_Finish(writer); } </pre></div> </div> </section> <section id="growandupdatepointer-example"> <h3><a class="toc-backref" href="#growandupdatepointer-example" role="doc-backlink"><code class="docutils literal notranslate">GrowAndUpdatePointer()</code> example</a></h3> Example using a pointer to write bytes and to track the written size. Create the bytes string <code class="docutils literal notranslate">b"Hello World"</code>: <div class="highlight-c notranslate"><div class="highlight"><pre>PyObject* grow_example(void) { // Allocate 10 bytes PyBytesWriter *writer = PyBytesWriter_Create(10); if (writer == NULL) { return NULL; } // Write some bytes char *buf = PyBytesWriter_GetData(writer); memcpy(buf, "Hello ", strlen("Hello ")); buf += strlen("Hello "); // Allocate 10 more bytes buf = PyBytesWriter_GrowAndUpdatePointer(writer, 10, buf); if (buf == NULL) { PyBytesWriter_Discard(writer); return NULL; } // Write more bytes memcpy(buf, "World", strlen("World")); buf += strlen("World"); // Truncate the string at 'buf' position // and create a bytes object return PyBytesWriter_FinishWithPointer(writer, buf); } </pre></div> </div> </section> <section id="update-pybytes-fromstringandsize-code"> <h3><a class="toc-backref" href="#update-pybytes-fromstringandsize-code" role="doc-backlink">Update <code class="docutils literal notranslate">PyBytes_FromStringAndSize()</code> code</a></h3> Example of code using the soft deprecated <code class="docutils literal notranslate">PyBytes_FromStringAndSize(NULL, size)</code> API: <div class="highlight-c notranslate"><div class="highlight"><pre>PyObject *result = PyBytes_FromStringAndSize(NULL, num_bytes); if (result == NULL) { return NULL; } if (copy_bytes(PyBytes_AS_STRING(result), start, num_bytes) < 0) { Py_CLEAR(result); } return result; </pre></div> </div> It can now be updated to: <div class="highlight-c notranslate"><div class="highlight"><pre>PyBytesWriter *writer = PyBytesWriter_Create(num_bytes); if (writer == NULL) { return NULL; } if (copy_bytes(PyBytesWriter_GetData(writer), start, num_bytes) < 0) { PyBytesWriter_Discard(writer); return NULL; } return PyBytesWriter_Finish(writer); </pre></div> </div> </section> <section id="update-pybytes-resize-code"> <h3><a class="toc-backref" href="#update-pybytes-resize-code" role="doc-backlink">Update <code class="docutils literal notranslate">_PyBytes_Resize()</code> code</a></h3> Example of code using the soft deprecated <code class="docutils literal notranslate">_PyBytes_Resize()</code> API: <div class="highlight-c notranslate"><div class="highlight"><pre>PyObject *v = PyBytes_FromStringAndSize(NULL, size); if (v == NULL) { return NULL; } char *p = PyBytes_AS_STRING(v); // ... fill bytes into 'p' ... if (_PyBytes_Resize(&v, (p - PyBytes_AS_STRING(v)))) { return NULL; } return v; </pre></div> </div> It can now be updated to: <div class="highlight-c notranslate"><div class="highlight"><pre>PyBytesWriter *writer = PyBytesWriter_Create(size); if (writer == NULL) { return NULL; } char *p = PyBytesWriter_GetData(writer); // ... fill bytes into 'p' ... return PyBytesWriter_FinishWithPointer(writer, p); </pre></div> </div> </section> </section> <section id="reference-implementation"> <h2><a class="toc-backref" href="#reference-implementation" role="doc-backlink">Reference Implementation</a></h2> <a class="reference external" href="https://github.com/python/cpython/pull/131681">Pull request gh-131681</a>. Notes on the CPython reference implementation which are not part of the Specification: <ul class="simple"> <li>The implementation allocates internally a <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#bytes" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">bytes</code></a> object, so <a class="reference internal" href="#c.PyBytesWriter_Finish" title="PyBytesWriter_Finish"><code class="xref c c-func docutils literal notranslate">PyBytesWriter_Finish()</code></a> just returns the object without having to copy memory.</li> <li>For strings up to 256 bytes, a small internal raw buffer of bytes is used. It avoids having to resize a <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#bytes" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">bytes</code></a> object which is inefficient. At the end, <a class="reference internal" href="#c.PyBytesWriter_Finish" title="PyBytesWriter_Finish"><code class="xref c c-func docutils literal notranslate">PyBytesWriter_Finish()</code></a> creates the <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#bytes" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">bytes</code></a> object from this small buffer.</li> <li>A free list is used to reduce the cost of allocating a <a class="reference internal" href="#c.PyBytesWriter" title="PyBytesWriter"><code class="xref c c-type docutils literal notranslate">PyBytesWriter</code></a> on the heap memory.</li> </ul> </section> <section id="backwards-compatibility"> <h2><a class="toc-backref" href="#backwards-compatibility" role="doc-backlink">Backwards Compatibility</a></h2> There is no impact on the backward compatibility, only new APIs are added. <code class="docutils literal notranslate">PyBytes_FromStringAndSize(NULL, size)</code> and <code class="docutils literal notranslate">_PyBytes_Resize()</code> APIs are soft deprecated. No new warnings is emitted when these functions are used and they are not planned for removal. </section> <section id="prior-discussions"> <h2><a class="toc-backref" href="#prior-discussions" role="doc-backlink">Prior Discussions</a></h2> <ul class="simple"> <li>March 2025: Third public API attempt, using size rather than pointers:<ul> <li><a class="reference external" href="https://discuss.python.org/t/81182/56">Discussion</a></li> <li><a class="reference external" href="https://github.com/python/cpython/pull/131681">Pull request gh-131681</a></li> </ul> </li> <li>February 2025: Second public API attempt:<ul> <li><a class="reference external" href="https://github.com/python/cpython/issues/129813">Issue gh-129813</a> and <a class="reference external" href="https://github.com/python/cpython/pull/129814">pull request gh-129814</a></li> </ul> </li> <li>July 2024: First public API attempt:<ul> <li>C API Working Group decision: <a class="reference external" href="https://github.com/capi-workgroup/decisions/issues/39">Add PyBytes_Writer() API</a> (August 2024)</li> <li><a class="reference external" href="https://github.com/python/cpython/pull/121726">Pull request gh-121726</a>: first public API attempt (July 2024)</li> </ul> </li> <li>March 2016: <a class="reference external" href="https://vstinner.github.io/pybyteswriter.html">Fast _PyAccu, _PyUnicodeWriter and _PyBytesWriter APIs to produce strings in CPython</a>: Article on the original private <code class="docutils literal notranslate">_PyBytesWriter</code> C API.</li> </ul> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0782.rst">https://github.com/python/peps/blob/main/peps/pep-0782.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0782.rst">2025-03-31 18:34:24 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#disallow-creation-of-incomplete-inconsistent-objects">Disallow creation of incomplete/inconsistent objects</a></li> <li><a class="reference internal" href="#inefficient-allocation-strategy">Inefficient allocation strategy</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#api">API</a><ul> <li><a class="reference internal" href="#create-finish-discard">Create, Finish, Discard</a></li> <li><a class="reference internal" href="#high-level-api">High-level API</a></li> <li><a class="reference internal" href="#getters">Getters</a></li> <li><a class="reference internal" href="#low-level-api">Low-level API</a></li> </ul> </li> <li><a class="reference internal" href="#overallocation">Overallocation</a></li> <li><a class="reference internal" href="#thread-safety">Thread safety</a></li> <li><a class="reference internal" href="#soft-deprecations">Soft deprecations</a></li> </ul> </li> <li><a class="reference internal" href="#examples">Examples</a><ul> <li><a class="reference internal" href="#id1">High-level API</a></li> <li><a class="reference internal" href="#create-the-bytes-string-abc">Create the bytes string “abc”</a></li> <li><a class="reference internal" href="#growandupdatepointer-example"><code class="docutils literal notranslate">GrowAndUpdatePointer()</code> example</a></li> <li><a class="reference internal" href="#update-pybytes-fromstringandsize-code">Update <code class="docutils literal notranslate">PyBytes_FromStringAndSize()</code> code</a></li> <li><a class="reference internal" href="#update-pybytes-resize-code">Update <code class="docutils literal notranslate">_PyBytes_Resize()</code> code</a></li> </ul> </li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a></li> <li><a class="reference internal" href="#prior-discussions">Prior Discussions</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0782.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>