CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 574 – Pickle protocol 5 with out-of-band data | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0574/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 574 – Pickle protocol 5 with out-of-band data | peps.python.org'> <meta property="og:description" content="This PEP proposes to standardize a new pickle protocol version, and accompanying APIs to take full advantage of it:"> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0574/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This PEP proposes to standardize a new pickle protocol version, and accompanying APIs to take full advantage of it:"> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 574</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 574 – Pickle protocol 5 with out-of-band data</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Antoine Pitrou <solipsis at pitrou.net></dd> <dt class="field-even">BDFL-Delegate:</dt> <dd class="field-even">Alyssa Coghlan</dd> <dt class="field-odd">Status:</dt> <dd class="field-odd"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-even">Type:</dt> <dd class="field-even"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-odd">Created:</dt> <dd class="field-odd">23-Mar-2018</dd> <dt class="field-even">Python-Version:</dt> <dd class="field-even">3.8</dd> <dt class="field-odd">Post-History:</dt> <dd class="field-odd">28-Mar-2018, 30-Apr-2019</dd> <dt class="field-even">Resolution:</dt> <dd class="field-even"><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2019-May/157284.html">Python-Dev message</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#example">Example</a></li> <li><a class="reference internal" href="#producer-api">Producer API</a><ul> <li><a class="reference internal" href="#picklebuffer-objects">PickleBuffer objects</a></li> <li><a class="reference internal" href="#buffer-requirements">Buffer requirements</a></li> </ul> </li> <li><a class="reference internal" href="#consumer-api">Consumer API</a></li> <li><a class="reference internal" href="#protocol-changes">Protocol changes</a></li> <li><a class="reference internal" href="#side-effects">Side effects</a><ul> <li><a class="reference internal" href="#improved-in-band-performance">Improved in-band performance</a></li> </ul> </li> <li><a class="reference internal" href="#caveats">Caveats</a><ul> <li><a class="reference internal" href="#mutability">Mutability</a></li> <li><a class="reference internal" href="#data-sharing">Data sharing</a></li> </ul> </li> <li><a class="reference internal" href="#rejected-alternatives">Rejected alternatives</a><ul> <li><a class="reference internal" href="#using-the-existing-persistent-load-interface">Using the existing persistent load interface</a></li> <li><a class="reference internal" href="#passing-a-sequence-of-buffers-in-buffer-callback">Passing a sequence of buffers in <code class="docutils literal notranslate">buffer_callback</code></a></li> <li><a class="reference internal" href="#allow-serializing-a-picklebuffer-in-protocol-4-and-earlier">Allow serializing a <code class="docutils literal notranslate">PickleBuffer</code> in protocol 4 and earlier</a></li> </ul> </li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#related-work">Related work</a></li> <li><a class="reference internal" href="#acknowledgements">Acknowledgements</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> This PEP proposes to standardize a new pickle protocol version, and accompanying APIs to take full advantage of it: <ol class="arabic simple"> <li>A new pickle protocol version (5) to cover the extra metadata needed for out-of-band data buffers.</li> <li>A new <code class="docutils literal notranslate">PickleBuffer</code> type for <code class="docutils literal notranslate">__reduce_ex__</code> implementations to return out-of-band data buffers.</li> <li>A new <code class="docutils literal notranslate">buffer_callback</code> parameter when pickling, to handle out-of-band data buffers.</li> <li>A new <code class="docutils literal notranslate">buffers</code> parameter when unpickling to provide out-of-band data buffers.</li> </ol> The PEP guarantees unchanged behaviour for anyone not using the new APIs. </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> The pickle protocol was originally designed in 1995 for on-disk persistency of arbitrary Python objects. The performance of a 1995-era storage medium probably made it irrelevant to focus on performance metrics such as use of RAM bandwidth when copying temporary data before writing it to disk. Nowadays the pickle protocol sees a growing use in applications where most of the data isn’t ever persisted to disk (or, when it is, it uses a portable format instead of Python-specific). Instead, pickle is being used to transmit data and commands from one process to another, either on the same machine or on multiple machines. Those applications will sometimes deal with very large data (such as Numpy arrays or Pandas dataframes) that need to be transferred around. For those applications, pickle is currently wasteful as it imposes spurious memory copies of the data being serialized. As a matter of fact, the standard <code class="docutils literal notranslate">multiprocessing</code> module uses pickle for serialization, and therefore also suffers from this problem when sending large data to another process. Third-party Python libraries, such as Dask <a class="footnote-reference brackets" href="#dask" id="id1">[1]</a>, PyArrow <a class="footnote-reference brackets" href="#pyarrow" id="id2">[4]</a> and IPyParallel <a class="footnote-reference brackets" href="#ipyparallel" id="id3">[3]</a>, have started implementing alternative serialization schemes with the explicit goal of avoiding copies on large data. Implementing a new serialization scheme is difficult and often leads to reduced generality (since many Python objects support pickle but not the new serialization scheme). Falling back on pickle for unsupported types is an option, but then you get back the spurious memory copies you wanted to avoid in the first place. For example, <code class="docutils literal notranslate">dask</code> is able to avoid memory copies for Numpy arrays and built-in containers thereof (such as lists or dicts containing Numpy arrays), but if a large Numpy array is an attribute of a user-defined object, <code class="docutils literal notranslate">dask</code> will serialize the user-defined object as a pickle stream, leading to memory copies. The common theme of these third-party serialization efforts is to generate a stream of object metadata (which contains pickle-like information about the objects being serialized) and a separate stream of zero-copy buffer objects for the payloads of large objects. Note that, in this scheme, small objects such as ints, etc. can be dumped together with the metadata stream. Refinements can include opportunistic compression of large data depending on its type and layout, like <code class="docutils literal notranslate">dask</code> does. This PEP aims to make <code class="docutils literal notranslate">pickle</code> usable in a way where large data is handled as a separate stream of zero-copy buffers, letting the application handle those buffers optimally. </section> <section id="example"> <h2><a class="toc-backref" href="#example" role="doc-backlink">Example</a></h2> To keep the example simple and avoid requiring knowledge of third-party libraries, we will focus here on a bytearray object (but the issue is conceptually the same with more sophisticated objects such as Numpy arrays). Like most objects, the bytearray object isn’t immediately understood by the pickle module and must therefore specify its decomposition scheme. Here is how a bytearray object currently decomposes for pickling: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> b.__reduce_ex__(4) (<class 'bytearray'>, (b'abc',), None) </pre></div> </div> This is because the <code class="docutils literal notranslate">bytearray.__reduce_ex__</code> implementation reads morally as follows: <div class="highlight-default notranslate"><div class="highlight"><pre>class bytearray: def __reduce_ex__(self, protocol): if protocol == 4: return type(self), bytes(self), None # Legacy code for earlier protocols omitted </pre></div> </div> In turn it produces the following pickle code: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> pickletools.dis(pickletools.optimize(pickle.dumps(b, protocol=4))) 0: \x80 PROTO 4 2: \x95 FRAME 30 11: \x8c SHORT_BINUNICODE 'builtins' 21: \x8c SHORT_BINUNICODE 'bytearray' 32: \x93 STACK_GLOBAL 33: C SHORT_BINBYTES b'abc' 38: \x85 TUPLE1 39: R REDUCE 40: . STOP </pre></div> </div> (the call to <code class="docutils literal notranslate">pickletools.optimize</code> above is only meant to make the pickle stream more readable by removing the MEMOIZE opcodes) We can notice several things about the bytearray’s payload (the sequence of bytes <code class="docutils literal notranslate">b'abc'</code>): <ul class="simple"> <li><code class="docutils literal notranslate">bytearray.__reduce_ex__</code> produces a first copy by instantiating a new bytes object from the bytearray’s data.</li> <li><code class="docutils literal notranslate">pickle.dumps</code> produces a second copy when inserting the contents of that bytes object into the pickle stream, after the SHORT_BINBYTES opcode.</li> <li>Furthermore, when deserializing the pickle stream, a temporary bytes object is created when the SHORT_BINBYTES opcode is encountered (inducing a data copy).</li> </ul> What we really want is something like the following: <ul class="simple"> <li><code class="docutils literal notranslate">bytearray.__reduce_ex__</code> produces a view of the bytearray’s data.</li> <li><code class="docutils literal notranslate">pickle.dumps</code> doesn’t try to copy that data into the pickle stream but instead passes the buffer view to its caller (which can decide on the most efficient handling of that buffer).</li> <li>When deserializing, <code class="docutils literal notranslate">pickle.loads</code> takes the pickle stream and the buffer view separately, and passes the buffer view directly to the bytearray constructor.</li> </ul> We see that several conditions are required for the above to work: <ul class="simple"> <li><code class="docutils literal notranslate">__reduce__</code> or <code class="docutils literal notranslate">__reduce_ex__</code> must be able to return something that indicates a serializable no-copy buffer view.</li> <li>The pickle protocol must be able to represent references to such buffer views, instructing the unpickler that it may have to get the actual buffer out of band.</li> <li>The <code class="docutils literal notranslate">pickle.Pickler</code> API must provide its caller with a way to receive such buffer views while serializing.</li> <li>The <code class="docutils literal notranslate">pickle.Unpickler</code> API must similarly allow its caller to provide the buffer views required for deserialization.</li> <li>For compatibility, the pickle protocol must also be able to contain direct serializations of such buffer views, such that current uses of the <code class="docutils literal notranslate">pickle</code> API don’t have to be modified if they are not concerned with memory copies.</li> </ul> </section> <section id="producer-api"> <h2><a class="toc-backref" href="#producer-api" role="doc-backlink">Producer API</a></h2> We are introducing a new type <code class="docutils literal notranslate">pickle.PickleBuffer</code> which can be instantiated from any buffer-supporting object, and is specifically meant to be returned from <code class="docutils literal notranslate">__reduce__</code> implementations: <div class="highlight-default notranslate"><div class="highlight"><pre>class bytearray: def __reduce_ex__(self, protocol): if protocol >= 5: return type(self), (PickleBuffer(self),), None # Legacy code for earlier protocols omitted </pre></div> </div> <code class="docutils literal notranslate">PickleBuffer</code> is a simple wrapper that doesn’t have all the memoryview semantics and functionality, but is specifically recognized by the <code class="docutils literal notranslate">pickle</code> module if protocol 5 or higher is enabled. It is an error to try to serialize a <code class="docutils literal notranslate">PickleBuffer</code> with pickle protocol version 4 or earlier. Only the raw data of the <code class="docutils literal notranslate">PickleBuffer</code> will be considered by the <code class="docutils literal notranslate">pickle</code> module. Any type-specific metadata (such as shapes or datatype) must be returned separately by the type’s <code class="docutils literal notranslate">__reduce__</code> implementation, as is already the case. <section id="picklebuffer-objects"> <h3><a class="toc-backref" href="#picklebuffer-objects" role="doc-backlink">PickleBuffer objects</a></h3> The <code class="docutils literal notranslate">PickleBuffer</code> class supports a very simple Python API. Its constructor takes a single <a class="pep reference internal" href="../pep-3118/" title="PEP 3118 – Revising the buffer protocol">PEP 3118</a>-compatible object. <code class="docutils literal notranslate">PickleBuffer</code> objects themselves support the buffer protocol, so consumers can call <code class="docutils literal notranslate">memoryview(...)</code> on them to get additional information about the underlying buffer (such as the original type, shape, etc.). In addition, <code class="docutils literal notranslate">PickleBuffer</code> objects have the following methods: <code class="docutils literal notranslate">raw()</code> <blockquote> <div>Return a memoryview of the raw memory bytes underlying the PickleBuffer, erasing any shape, strides and format information. This is required to handle Fortran-contiguous buffers correctly in the pure Python pickle implementation.</div></blockquote> <code class="docutils literal notranslate">release()</code> <blockquote> <div>Release the PickleBuffer’s underlying buffer, making it unusable.</div></blockquote> On the C side, a simple API will be provided to create and inspect PickleBuffer objects: <code class="docutils literal notranslate">PyObject *PyPickleBuffer_FromObject(PyObject *obj)</code> <blockquote> <div>Create a <code class="docutils literal notranslate">PickleBuffer</code> object holding a view over the <a class="pep reference internal" href="../pep-3118/" title="PEP 3118 – Revising the buffer protocol">PEP 3118</a>-compatible obj.</div></blockquote> <code class="docutils literal notranslate">PyPickleBuffer_Check(PyObject *obj)</code> <blockquote> <div>Return whether obj is a <code class="docutils literal notranslate">PickleBuffer</code> instance.</div></blockquote> <code class="docutils literal notranslate">const Py_buffer *PyPickleBuffer_GetBuffer(PyObject *picklebuf)</code> <blockquote> <div>Return a pointer to the internal <code class="docutils literal notranslate">Py_buffer</code> owned by the <code class="docutils literal notranslate">PickleBuffer</code> instance. An exception is raised if the buffer is released.</div></blockquote> <code class="docutils literal notranslate">int PyPickleBuffer_Release(PyObject *picklebuf)</code> <blockquote> <div>Release the <code class="docutils literal notranslate">PickleBuffer</code> instance’s underlying buffer.</div></blockquote> </section> <section id="buffer-requirements"> <h3><a class="toc-backref" href="#buffer-requirements" role="doc-backlink">Buffer requirements</a></h3> <code class="docutils literal notranslate">PickleBuffer</code> can wrap any kind of buffer, including non-contiguous buffers. However, it is required that <code class="docutils literal notranslate">__reduce__</code> only returns a contiguous <code class="docutils literal notranslate">PickleBuffer</code> (contiguity here is meant in the <a class="pep reference internal" href="../pep-3118/" title="PEP 3118 – Revising the buffer protocol">PEP 3118</a> sense: either C-ordered or Fortran-ordered). Non-contiguous buffers will raise an error when pickled. This restriction is primarily an ease-of-implementation issue for the <code class="docutils literal notranslate">pickle</code> module but also other consumers of out-of-band buffers. The simplest solution on the provider side is to return a contiguous copy of a non-contiguous buffer; a sophisticated provider, though, may decide instead to return a sequence of contiguous sub-buffers. </section> </section> <section id="consumer-api"> <h2><a class="toc-backref" href="#consumer-api" role="doc-backlink">Consumer API</a></h2> <code class="docutils literal notranslate">pickle.Pickler.__init__</code> and <code class="docutils literal notranslate">pickle.dumps</code> are augmented with an additional <code class="docutils literal notranslate">buffer_callback</code> parameter: <div class="highlight-default notranslate"><div class="highlight"><pre>class Pickler: def __init__(self, file, protocol=None, ..., buffer_callback=None): """ If *buffer_callback* is None (the default), buffer views are serialized into *file* as part of the pickle stream. If *buffer_callback* is not None, then it can be called any number of times with a buffer view. If the callback returns a false value (such as None), the given buffer is out-of-band; otherwise the buffer is serialized in-band, i.e. inside the pickle stream. The callback should arrange to store or transmit out-of-band buffers without changing their order. It is an error if *buffer_callback* is not None and *protocol* is None or smaller than 5. """ def pickle.dumps(obj, protocol=None, *, ..., buffer_callback=None): """ See above for *buffer_callback*. """ </pre></div> </div> <code class="docutils literal notranslate">pickle.Unpickler.__init__</code> and <code class="docutils literal notranslate">pickle.loads</code> are augmented with an additional <code class="docutils literal notranslate">buffers</code> parameter: <div class="highlight-default notranslate"><div class="highlight"><pre>class Unpickler: def __init__(file, *, ..., buffers=None): """ If *buffers* is not None, it should be an iterable of buffer-enabled objects that is consumed each time the pickle stream references an out-of-band buffer view. Such buffers have been given in order to the *buffer_callback* of a Pickler object. If *buffers* is None (the default), then the buffers are taken from the pickle stream, assuming they are serialized there. It is an error for *buffers* to be None if the pickle stream was produced with a non-None *buffer_callback*. """ def pickle.loads(data, *, ..., buffers=None): """ See above for *buffers*. """ </pre></div> </div> </section> <section id="protocol-changes"> <h2><a class="toc-backref" href="#protocol-changes" role="doc-backlink">Protocol changes</a></h2> Three new opcodes are introduced: <ul class="simple"> <li><code class="docutils literal notranslate">BYTEARRAY8</code> creates a bytearray from the data following it in the pickle stream and pushes it on the stack (just like <code class="docutils literal notranslate">BINBYTES8</code> does for bytes objects);</li> <li><code class="docutils literal notranslate">NEXT_BUFFER</code> fetches a buffer from the <code class="docutils literal notranslate">buffers</code> iterable and pushes it on the stack.</li> <li><code class="docutils literal notranslate">READONLY_BUFFER</code> makes a readonly view of the top of the stack.</li> </ul> When pickling encounters a <code class="docutils literal notranslate">PickleBuffer</code>, that buffer can be considered in-band or out-of-band depending on the following conditions: <ul class="simple"> <li>if no <code class="docutils literal notranslate">buffer_callback</code> is given, the buffer is in-band;</li> <li>if a <code class="docutils literal notranslate">buffer_callback</code> is given, it is called with the buffer. If the callback returns a true value, the buffer is in-band; if the callback returns a false value, the buffer is out-of-band.</li> </ul> An in-band buffer is serialized as follows: <ul class="simple"> <li>If the buffer is writable, it is serialized into the pickle stream as if it were a <code class="docutils literal notranslate">bytearray</code> object.</li> <li>If the buffer is readonly, it is serialized into the pickle stream as if it were a <code class="docutils literal notranslate">bytes</code> object.</li> </ul> An out-of-band buffer is serialized as follows: <ul class="simple"> <li>If the buffer is writable, a <code class="docutils literal notranslate">NEXT_BUFFER</code> opcode is appended to the pickle stream.</li> <li>If the buffer is readonly, a <code class="docutils literal notranslate">NEXT_BUFFER</code> opcode is appended to the pickle stream, followed by a <code class="docutils literal notranslate">READONLY_BUFFER</code> opcode.</li> </ul> The distinction between readonly and writable buffers is motivated below (see “Mutability”). </section> <section id="side-effects"> <h2><a class="toc-backref" href="#side-effects" role="doc-backlink">Side effects</a></h2> <section id="improved-in-band-performance"> <h3><a class="toc-backref" href="#improved-in-band-performance" role="doc-backlink">Improved in-band performance</a></h3> Even in-band pickling can be improved by returning a <code class="docutils literal notranslate">PickleBuffer</code> instance from <code class="docutils literal notranslate">__reduce_ex__</code>, as one copy is avoided on the serialization path <a class="footnote-reference brackets" href="#arrow-pickle5-benchmark" id="id4">[10]</a> <a class="footnote-reference brackets" href="#numpy-pickle5-benchmark" id="id5">[12]</a>. </section> </section> <section id="caveats"> <h2><a class="toc-backref" href="#caveats" role="doc-backlink">Caveats</a></h2> <section id="mutability"> <h3><a class="toc-backref" href="#mutability" role="doc-backlink">Mutability</a></h3> <a class="pep reference internal" href="../pep-3118/" title="PEP 3118 – Revising the buffer protocol">PEP 3118</a> buffers can be readonly or writable. Some objects, such as Numpy arrays, need to be backed by a mutable buffer for full operation. Pickle consumers that use the <code class="docutils literal notranslate">buffer_callback</code> and <code class="docutils literal notranslate">buffers</code> arguments will have to be careful to recreate mutable buffers. When doing I/O, this implies using buffer-passing API variants such as <code class="docutils literal notranslate">readinto</code> (which are also often preferable for performance). </section> <section id="data-sharing"> <h3><a class="toc-backref" href="#data-sharing" role="doc-backlink">Data sharing</a></h3> If you pickle and then unpickle an object in the same process, passing out-of-band buffer views, then the unpickled object may be backed by the same buffer as the original pickled object. For example, it might be reasonable to implement reduction of a Numpy array as follows (crucial metadata such as shapes is omitted for simplicity): <div class="highlight-default notranslate"><div class="highlight"><pre>class ndarray: def __reduce_ex__(self, protocol): if protocol == 5: return numpy.frombuffer, (PickleBuffer(self), self.dtype) # Legacy code for earlier protocols omitted </pre></div> </div> Then simply passing the PickleBuffer around from <code class="docutils literal notranslate">dumps</code> to <code class="docutils literal notranslate">loads</code> will produce a new Numpy array sharing the same underlying memory as the original Numpy object (and, incidentally, keeping it alive): <div class="highlight-default notranslate"><div class="highlight"><pre>>>> import numpy as np >>> a = np.zeros(10) >>> a[0] 0.0 >>> buffers = [] >>> data = pickle.dumps(a, protocol=5, buffer_callback=buffers.append) >>> b = pickle.loads(data, buffers=buffers) >>> b[0] = 42 >>> a[0] 42.0 </pre></div> </div> This won’t happen with the traditional <code class="docutils literal notranslate">pickle</code> API (i.e. without passing <code class="docutils literal notranslate">buffers</code> and <code class="docutils literal notranslate">buffer_callback</code> parameters), because then the buffer view is serialized inside the pickle stream with a copy. </section> </section> <section id="rejected-alternatives"> <h2><a class="toc-backref" href="#rejected-alternatives" role="doc-backlink">Rejected alternatives</a></h2> <section id="using-the-existing-persistent-load-interface"> <h3><a class="toc-backref" href="#using-the-existing-persistent-load-interface" role="doc-backlink">Using the existing persistent load interface</a></h3> The <code class="docutils literal notranslate">pickle</code> persistence interface is a way of storing references to designated objects in the pickle stream while handling their actual serialization out of band. For example, one might consider the following for zero-copy serialization of bytearrays: <div class="highlight-default notranslate"><div class="highlight"><pre>class MyPickle(pickle.Pickler): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.buffers = [] def persistent_id(self, obj): if type(obj) is not bytearray: return None else: index = len(self.buffers) self.buffers.append(obj) return ('bytearray', index) class MyUnpickle(pickle.Unpickler): def __init__(self, *args, buffers, **kwargs): super().__init__(*args, **kwargs) self.buffers = buffers def persistent_load(self, pid): type_tag, index = pid if type_tag == 'bytearray': return self.buffers[index] else: assert 0 # unexpected type </pre></div> </div> This mechanism has two drawbacks: <ul> <li>Each <code class="docutils literal notranslate">pickle</code> consumer must reimplement <code class="docutils literal notranslate">Pickler</code> and <code class="docutils literal notranslate">Unpickler</code> subclasses, with custom code for each type of interest. Essentially, N pickle consumers end up each implementing custom code for M producers. This is difficult (especially for sophisticated types such as Numpy arrays) and poorly scalable.</li> <li>Each object encountered by the pickle module (even simple built-in objects such as ints and strings) triggers a call to the user’s <code class="docutils literal notranslate">persistent_id()</code> method, leading to a possible performance drop compared to nominal.(the Python 2 <code class="docutils literal notranslate">cPickle</code> module supported an undocumented <code class="docutils literal notranslate">inst_persistent_id()</code> hook that was only called on non-built-in types; it was added in 1997 in order to alleviate the performance issue of calling <code class="docutils literal notranslate">persistent_id</code>, presumably at ZODB’s request) </li> </ul> </section> <section id="passing-a-sequence-of-buffers-in-buffer-callback"> <h3><a class="toc-backref" href="#passing-a-sequence-of-buffers-in-buffer-callback" role="doc-backlink">Passing a sequence of buffers in <code class="docutils literal notranslate">buffer_callback</code></a></h3> By passing a sequence of buffers, rather than a single buffer, we would potentially save on function call overhead in case a large number of buffers are produced during serialization. This would need additional support in the Pickler to save buffers before calling the callback. However, it would also prevent the buffer callback from returning a boolean to indicate whether a buffer is to be serialized in-band or out-of-band. We consider that having a large number of buffers to serialize is an unlikely case, and decided to pass a single buffer to the buffer callback. </section> <section id="allow-serializing-a-picklebuffer-in-protocol-4-and-earlier"> <h3><a class="toc-backref" href="#allow-serializing-a-picklebuffer-in-protocol-4-and-earlier" role="doc-backlink">Allow serializing a <code class="docutils literal notranslate">PickleBuffer</code> in protocol 4 and earlier</a></h3> If we were to allow serializing a <code class="docutils literal notranslate">PickleBuffer</code> in protocols 4 and earlier, it would actually make a supplementary memory copy when the buffer is mutable. Indeed, a mutable <code class="docutils literal notranslate">PickleBuffer</code> would serialize as a bytearray object in those protocols (that is a first copy), and serializing the bytearray object would call <code class="docutils literal notranslate">bytearray.__reduce_ex__</code> which returns a bytes object (that is a second copy). To prevent <code class="docutils literal notranslate">__reduce__</code> implementors from introducing involuntary performance regressions, we decided to reject <code class="docutils literal notranslate">PickleBuffer</code> when the protocol is smaller than 5. This forces implementors to switch to <code class="docutils literal notranslate">__reduce_ex__</code> and implement protocol-dependent serialization, taking advantage of the best path for each protocol (or at least treat protocol 5 and upwards separately from protocols 4 and downwards). </section> </section> <section id="implementation"> <h2><a class="toc-backref" href="#implementation" role="doc-backlink">Implementation</a></h2> The PEP was initially implemented in the author’s GitHub fork <a class="footnote-reference brackets" href="#pickle5-git" id="id6">[6]</a>. It was later merged into Python 3.8 <a class="footnote-reference brackets" href="#pickle5-pr" id="id7">[7]</a>. A backport for Python 3.6 and 3.7 is downloadable from PyPI <a class="footnote-reference brackets" href="#pickle5-pypi" id="id8">[8]</a>. Support for pickle protocol 5 and out-of-band buffers was added to Numpy <a class="footnote-reference brackets" href="#numpy-pickle5-pr" id="id9">[11]</a>. Support for pickle protocol 5 and out-of-band buffers was added to the Apache Arrow Python bindings <a class="footnote-reference brackets" href="#arrow-pickle5-pr" id="id10">[9]</a>. </section> <section id="related-work"> <h2><a class="toc-backref" href="#related-work" role="doc-backlink">Related work</a></h2> Dask.distributed implements a custom zero-copy serialization with fallback to pickle <a class="footnote-reference brackets" href="#dask-serialization" id="id11">[2]</a>. PyArrow implements zero-copy component-based serialization for a few selected types <a class="footnote-reference brackets" href="#pyarrow-serialization" id="id12">[5]</a>. <a class="pep reference internal" href="../pep-0554/" title="PEP 554 – Multiple Interpreters in the Stdlib">PEP 554</a> proposes hosting multiple interpreters in a single process, with provisions for transferring buffers between interpreters as a communication scheme. </section> <section id="acknowledgements"> <h2><a class="toc-backref" href="#acknowledgements" role="doc-backlink">Acknowledgements</a></h2> Thanks to the following people for early feedback: Alyssa Coghlan, Olivier Grisel, Stefan Krah, MinRK, Matt Rocklin, Eric Snow. Thanks to Pierre Glaser and Olivier Grisel for experimenting with the implementation. </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <aside class="footnote-list brackets"> <aside class="footnote brackets" id="dask" role="doc-footnote"> <dt class="label" id="dask">[<a href="#id1">1</a>]</dt> <dd>Dask.distributed – A lightweight library for distributed computing in Python <a class="reference external" href="https://distributed.readthedocs.io/">https://distributed.readthedocs.io/</a></aside> <aside class="footnote brackets" id="dask-serialization" role="doc-footnote"> <dt class="label" id="dask-serialization">[<a href="#id11">2</a>]</dt> <dd>Dask.distributed custom serialization <a class="reference external" href="https://distributed.readthedocs.io/en/latest/serialization.html">https://distributed.readthedocs.io/en/latest/serialization.html</a></aside> <aside class="footnote brackets" id="ipyparallel" role="doc-footnote"> <dt class="label" id="ipyparallel">[<a href="#id3">3</a>]</dt> <dd>IPyParallel – Using IPython for parallel computing <a class="reference external" href="https://ipyparallel.readthedocs.io/">https://ipyparallel.readthedocs.io/</a></aside> <aside class="footnote brackets" id="pyarrow" role="doc-footnote"> <dt class="label" id="pyarrow">[<a href="#id2">4</a>]</dt> <dd>PyArrow – A cross-language development platform for in-memory data <a class="reference external" href="https://arrow.apache.org/docs/python/">https://arrow.apache.org/docs/python/</a></aside> <aside class="footnote brackets" id="pyarrow-serialization" role="doc-footnote"> <dt class="label" id="pyarrow-serialization">[<a href="#id12">5</a>]</dt> <dd>PyArrow IPC and component-based serialization <a class="reference external" href="https://arrow.apache.org/docs/python/ipc.html#component-based-serialization">https://arrow.apache.org/docs/python/ipc.html#component-based-serialization</a></aside> <aside class="footnote brackets" id="pickle5-git" role="doc-footnote"> <dt class="label" id="pickle5-git">[<a href="#id6">6</a>]</dt> <dd><code class="docutils literal notranslate">pickle5</code> branch on GitHub <a class="reference external" href="https://github.com/pitrou/cpython/tree/pickle5">https://github.com/pitrou/cpython/tree/pickle5</a></aside> <aside class="footnote brackets" id="pickle5-pr" role="doc-footnote"> <dt class="label" id="pickle5-pr">[<a href="#id7">7</a>]</dt> <dd>PEP 574 Pull Request on GitHub <a class="reference external" href="https://github.com/python/cpython/pull/7076">https://github.com/python/cpython/pull/7076</a></aside> <aside class="footnote brackets" id="pickle5-pypi" role="doc-footnote"> <dt class="label" id="pickle5-pypi">[<a href="#id8">8</a>]</dt> <dd><code class="docutils literal notranslate">pickle5</code> project on PyPI <a class="reference external" href="https://pypi.org/project/pickle5/">https://pypi.org/project/pickle5/</a></aside> <aside class="footnote brackets" id="arrow-pickle5-pr" role="doc-footnote"> <dt class="label" id="arrow-pickle5-pr">[<a href="#id10">9</a>]</dt> <dd>Pull request: Experimental zero-copy pickling in Apache Arrow <a class="reference external" href="https://github.com/apache/arrow/pull/2161">https://github.com/apache/arrow/pull/2161</a></aside> <aside class="footnote brackets" id="arrow-pickle5-benchmark" role="doc-footnote"> <dt class="label" id="arrow-pickle5-benchmark">[<a href="#id4">10</a>]</dt> <dd>Benchmark zero-copy pickling in Apache Arrow <a class="reference external" href="https://github.com/apache/arrow/pull/2161#issuecomment-407859213">https://github.com/apache/arrow/pull/2161#issuecomment-407859213</a></aside> <aside class="footnote brackets" id="numpy-pickle5-pr" role="doc-footnote"> <dt class="label" id="numpy-pickle5-pr">[<a href="#id9">11</a>]</dt> <dd>Pull request: Support pickle protocol 5 in Numpy <a class="reference external" href="https://github.com/numpy/numpy/pull/12011">https://github.com/numpy/numpy/pull/12011</a></aside> <aside class="footnote brackets" id="numpy-pickle5-benchmark" role="doc-footnote"> <dt class="label" id="numpy-pickle5-benchmark">[<a href="#id5">12</a>]</dt> <dd>Benchmark pickling Numpy arrays with different pickle protocols <a class="reference external" href="https://github.com/numpy/numpy/issues/11161#issuecomment-424035962">https://github.com/numpy/numpy/issues/11161#issuecomment-424035962</a></aside> </aside> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed into the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0574.rst">https://github.com/python/peps/blob/main/peps/pep-0574.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0574.rst">2023-10-11 12:05:51 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#example">Example</a></li> <li><a class="reference internal" href="#producer-api">Producer API</a><ul> <li><a class="reference internal" href="#picklebuffer-objects">PickleBuffer objects</a></li> <li><a class="reference internal" href="#buffer-requirements">Buffer requirements</a></li> </ul> </li> <li><a class="reference internal" href="#consumer-api">Consumer API</a></li> <li><a class="reference internal" href="#protocol-changes">Protocol changes</a></li> <li><a class="reference internal" href="#side-effects">Side effects</a><ul> <li><a class="reference internal" href="#improved-in-band-performance">Improved in-band performance</a></li> </ul> </li> <li><a class="reference internal" href="#caveats">Caveats</a><ul> <li><a class="reference internal" href="#mutability">Mutability</a></li> <li><a class="reference internal" href="#data-sharing">Data sharing</a></li> </ul> </li> <li><a class="reference internal" href="#rejected-alternatives">Rejected alternatives</a><ul> <li><a class="reference internal" href="#using-the-existing-persistent-load-interface">Using the existing persistent load interface</a></li> <li><a class="reference internal" href="#passing-a-sequence-of-buffers-in-buffer-callback">Passing a sequence of buffers in <code class="docutils literal notranslate">buffer_callback</code></a></li> <li><a class="reference internal" href="#allow-serializing-a-picklebuffer-in-protocol-4-and-earlier">Allow serializing a <code class="docutils literal notranslate">PickleBuffer</code> in protocol 4 and earlier</a></li> </ul> </li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#related-work">Related work</a></li> <li><a class="reference internal" href="#acknowledgements">Acknowledgements</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0574.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>