CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 3118 – Revising the buffer protocol | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-3118/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 3118 – Revising the buffer protocol | peps.python.org'> <meta property="og:description" content="This PEP proposes re-designing the buffer interface (PyBufferProcs function pointers) to improve the way Python allows memory sharing in Python 3.0"> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-3118/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This PEP proposes re-designing the buffer interface (PyBufferProcs function pointers) to improve the way Python allows memory sharing in Python 3.0"> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 3118</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 3118 – Revising the buffer protocol</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Travis Oliphant <oliphant at ee.byu.edu>, Carl Banks <pythondev at aerojockey.com></dd> <dt class="field-even">Status:</dt> <dd class="field-even"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-odd">Type:</dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created:</dt> <dd class="field-even">28-Aug-2006</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">3.0</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even"></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#proposal-overview">Proposal Overview</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#access-flags">Access flags</a></li> <li><a class="reference internal" href="#the-py-buffer-struct">The Py_buffer struct</a></li> <li><a class="reference internal" href="#releasing-the-buffer">Releasing the buffer</a></li> </ul> </li> <li><a class="reference internal" href="#new-c-api-calls-are-proposed">New C-API calls are proposed</a></li> <li><a class="reference internal" href="#additions-to-the-struct-string-syntax">Additions to the struct string-syntax</a></li> <li><a class="reference internal" href="#examples-of-data-format-descriptions">Examples of Data-Format Descriptions</a></li> <li><a class="reference internal" href="#code-to-be-affected">Code to be affected</a></li> <li><a class="reference internal" href="#issues-and-details">Issues and Details</a></li> <li><a class="reference internal" href="#code">Code</a></li> <li><a class="reference internal" href="#examples">Examples</a><ul> <li><a class="reference internal" href="#ex-1">Ex. 1</a></li> <li><a class="reference internal" href="#ex-2">Ex. 2</a></li> <li><a class="reference internal" href="#ex-3">Ex. 3</a></li> <li><a class="reference internal" href="#ex-4">Ex. 4</a></li> </ul> </li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> This PEP proposes re-designing the buffer interface (<code class="docutils literal notranslate">PyBufferProcs</code> function pointers) to improve the way Python allows memory sharing in Python 3.0 In particular, it is proposed that the character buffer portion of the API be eliminated and the multiple-segment portion be re-designed in conjunction with allowing for strided memory to be shared. In addition, the new buffer interface will allow the sharing of any multi-dimensional nature of the memory and what data-format the memory contains. This interface will allow any extension module to either create objects that share memory or create algorithms that use and manipulate raw memory from arbitrary objects that export the interface. </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> The Python 2.X buffer protocol allows different Python types to exchange a pointer to a sequence of internal buffers. This functionality is extremely useful for sharing large segments of memory between different high-level objects, but it is too limited and has issues: <ol class="arabic"> <li>There is the little used “sequence-of-segments” option (bf_getsegcount) that is not well motivated.</li> <li>There is the apparently redundant character-buffer option (bf_getcharbuffer)</li> <li>There is no way for a consumer to tell the buffer-API-exporting object it is “finished” with its view of the memory and therefore no way for the exporting object to be sure that it is safe to reallocate the pointer to the memory that it owns (for example, the array object reallocating its memory after sharing it with the buffer object which held the original pointer led to the infamous buffer-object problem).</li> <li>Memory is just a pointer with a length. There is no way to describe what is “in” the memory (float, int, C-structure, etc.)</li> <li>There is no shape information provided for the memory. But, several array-like Python types could make use of a standard way to describe the shape-interpretation of the memory (wxPython, GTK, pyQT, CVXOPT, PyVox, Audio and Video Libraries, ctypes, NumPy, data-base interfaces, etc.)</li> <li>There is no way to share discontiguous memory (except through the sequence of segments notion).There are two widely used libraries that use the concept of discontiguous memory: PIL and NumPy. Their view of discontiguous arrays is different, though. The proposed buffer interface allows sharing of either memory model. Exporters will typically use only one approach and consumers may choose to support discontiguous arrays of each type however they choose. NumPy uses the notion of constant striding in each dimension as its basic concept of an array. With this concept, a simple sub-region of a larger array can be described without copying the data. Thus, stride information is the additional information that must be shared. The PIL uses a more opaque memory representation. Sometimes an image is contained in a contiguous segment of memory, but sometimes it is contained in an array of pointers to the contiguous segments (usually lines) of the image. The PIL is where the idea of multiple buffer segments in the original buffer interface came from. NumPy’s strided memory model is used more often in computational libraries and because it is so simple it makes sense to support memory sharing using this model. The PIL memory model is sometimes used in C-code where a 2-d array can then be accessed using double pointer indirection: e.g. <code class="docutils literal notranslate">image[i][j]</code>. The buffer interface should allow the object to export either of these memory models. Consumers are free to either require contiguous memory or write code to handle one or both of these memory models. </li> </ol> </section> <section id="proposal-overview"> <h2><a class="toc-backref" href="#proposal-overview" role="doc-backlink">Proposal Overview</a></h2> <ul class="simple"> <li>Eliminate the char-buffer and multiple-segment sections of the buffer-protocol.</li> <li>Unify the read/write versions of getting the buffer.</li> <li>Add a new function to the interface that should be called when the consumer object is “done” with the memory area.</li> <li>Add a new variable to allow the interface to describe what is in memory (unifying what is currently done now in struct and array)</li> <li>Add a new variable to allow the protocol to share shape information</li> <li>Add a new variable for sharing stride information</li> <li>Add a new mechanism for sharing arrays that must be accessed using pointer indirection.</li> <li>Fix all objects in the core and the standard library to conform to the new interface</li> <li>Extend the struct module to handle more format specifiers</li> <li>Extend the buffer object into a new memory object which places a Python veneer around the buffer interface.</li> <li>Add a few functions to make it easy to copy contiguous data in and out of object supporting the buffer interface.</li> </ul> </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> While the new specification allows for complicated memory sharing, simple contiguous buffers of bytes can still be obtained from an object. In fact, the new protocol allows a standard mechanism for doing this even if the original object is not represented as a contiguous chunk of memory. The easiest way to obtain a simple contiguous chunk of memory is to use the provided C-API to obtain a chunk of memory. Change the <code class="docutils literal notranslate">PyBufferProcs</code> structure to <div class="highlight-default notranslate"><div class="highlight"><pre>typedef struct { getbufferproc bf_getbuffer; releasebufferproc bf_releasebuffer; } PyBufferProcs; </pre></div> </div> Both of these routines are optional for a type object <div class="highlight-default notranslate"><div class="highlight"><pre>typedef int (*getbufferproc)(PyObject *obj, PyBuffer *view, int flags) </pre></div> </div> This function returns <code class="docutils literal notranslate">0</code> on success and <code class="docutils literal notranslate">-1</code> on failure (and raises an error). The first variable is the “exporting” object. The second argument is the address to a bufferinfo structure. Both arguments must never be NULL. The third argument indicates what kind of buffer the consumer is prepared to deal with and therefore what kind of buffer the exporter is allowed to return. The new buffer interface allows for much more complicated memory sharing possibilities. Some consumers may not be able to handle all the complexity but may want to see if the exporter will let them take a simpler view to its memory. In addition, some exporters may not be able to share memory in every possible way and may need to raise errors to signal to some consumers that something is just not possible. These errors should be <code class="docutils literal notranslate">PyErr_BufferError</code> unless there is another error that is actually causing the problem. The exporter can use flags information to simplify how much of the PyBuffer structure is filled in with non-default values and/or raise an error if the object can’t support a simpler view of its memory. The exporter should always fill in all elements of the buffer structure (with defaults or NULLs if nothing else is requested). The PyBuffer_FillInfo function can be used for simple cases. <section id="access-flags"> <h3><a class="toc-backref" href="#access-flags" role="doc-backlink">Access flags</a></h3> Some flags are useful for requesting a specific kind of memory segment, while others indicate to the exporter what kind of information the consumer can deal with. If certain information is not asked for by the consumer, but the exporter cannot share its memory without that information, then a <code class="docutils literal notranslate">PyErr_BufferError</code> should be raised. <code class="docutils literal notranslate">PyBUF_SIMPLE</code> <blockquote> <div>This is the default flag state (0). The returned buffer may or may not have writable memory. The format will be assumed to be unsigned bytes. This is a “stand-alone” flag constant. It never needs to be |’d to the others. The exporter will raise an error if it cannot provide such a contiguous buffer of bytes.</div></blockquote> <code class="docutils literal notranslate">PyBUF_WRITABLE</code> <blockquote> <div>The returned buffer must be writable. If it is not writable, then raise an error.</div></blockquote> <code class="docutils literal notranslate">PyBUF_FORMAT</code> <blockquote> <div>The returned buffer must have true format information if this flag is provided. This would be used when the consumer is going to be checking for what ‘kind’ of data is actually stored. An exporter should always be able to provide this information if requested. If format is not explicitly requested then the format must be returned as <code class="docutils literal notranslate">NULL</code> (which means “B”, or unsigned bytes)</div></blockquote> <code class="docutils literal notranslate">PyBUF_ND</code> <blockquote> <div>The returned buffer must provide shape information. The memory will be assumed C-style contiguous (last dimension varies the fastest). The exporter may raise an error if it cannot provide this kind of contiguous buffer. If this is not given then shape will be NULL.</div></blockquote> <code class="docutils literal notranslate">PyBUF_STRIDES</code> (implies <code class="docutils literal notranslate">PyBUF_ND</code>) <blockquote> <div>The returned buffer must provide strides information (i.e. the strides cannot be NULL). This would be used when the consumer can handle strided, discontiguous arrays. Handling strides automatically assumes you can handle shape. The exporter may raise an error if cannot provide a strided-only representation of the data (i.e. without the suboffsets).</div></blockquote> <div class="line-block"> <div class="line"><code class="docutils literal notranslate">PyBUF_C_CONTIGUOUS</code></div> <div class="line"><code class="docutils literal notranslate">PyBUF_F_CONTIGUOUS</code></div> <div class="line"><code class="docutils literal notranslate">PyBUF_ANY_CONTIGUOUS</code></div> </div> <blockquote> <div>These flags indicate that the returned buffer must be respectively, C-contiguous (last dimension varies the fastest), Fortran contiguous (first dimension varies the fastest) or either one. All of these flags imply PyBUF_STRIDES and guarantee that the strides buffer info structure will be filled in correctly.</div></blockquote> <code class="docutils literal notranslate">PyBUF_INDIRECT</code> (implies <code class="docutils literal notranslate">PyBUF_STRIDES</code>) <blockquote> <div>The returned buffer must have suboffsets information (which can be NULL if no suboffsets are needed). This would be used when the consumer can handle indirect array referencing implied by these suboffsets.</div></blockquote> Specialized combinations of flags for specific kinds of memory_sharing. <blockquote> <div>Multi-dimensional (but contiguous)<blockquote> <div><div class="line-block"> <div class="line"><code class="docutils literal notranslate">PyBUF_CONTIG</code> (<code class="docutils literal notranslate">PyBUF_ND | PyBUF_WRITABLE</code>)</div> <div class="line"><code class="docutils literal notranslate">PyBUF_CONTIG_RO</code> (<code class="docutils literal notranslate">PyBUF_ND</code>)</div> </div> </div></blockquote> Multi-dimensional using strides but aligned <blockquote> <div><div class="line-block"> <div class="line"><code class="docutils literal notranslate">PyBUF_STRIDED</code> (<code class="docutils literal notranslate">PyBUF_STRIDES | PyBUF_WRITABLE</code>)</div> <div class="line"><code class="docutils literal notranslate">PyBUF_STRIDED_RO</code> (<code class="docutils literal notranslate">PyBUF_STRIDES</code>)</div> </div> </div></blockquote> Multi-dimensional using strides and not necessarily aligned <blockquote> <div><div class="line-block"> <div class="line"><code class="docutils literal notranslate">PyBUF_RECORDS</code> (<code class="docutils literal notranslate">PyBUF_STRIDES | PyBUF_WRITABLE | PyBUF_FORMAT</code>)</div> <div class="line"><code class="docutils literal notranslate">PyBUF_RECORDS_RO</code> (<code class="docutils literal notranslate">PyBUF_STRIDES | PyBUF_FORMAT</code>)</div> </div> </div></blockquote> Multi-dimensional using sub-offsets <blockquote> <div><div class="line-block"> <div class="line"><code class="docutils literal notranslate">PyBUF_FULL</code> (<code class="docutils literal notranslate">PyBUF_INDIRECT | PyBUF_WRITABLE | PyBUF_FORMAT</code>)</div> <div class="line"><code class="docutils literal notranslate">PyBUF_FULL_RO</code> (<code class="docutils literal notranslate">PyBUF_INDIRECT | PyBUF_FORMAT</code>)</div> </div> </div></blockquote> </div></blockquote> Thus, the consumer simply wanting a contiguous chunk of bytes from the object would use <code class="docutils literal notranslate">PyBUF_SIMPLE</code>, while a consumer that understands how to make use of the most complicated cases could use <code class="docutils literal notranslate">PyBUF_FULL</code>. The format information is only guaranteed to be non-NULL if <code class="docutils literal notranslate">PyBUF_FORMAT</code> is in the flag argument, otherwise it is expected the consumer will assume unsigned bytes. There is a C-API that simple exporting objects can use to fill-in the buffer info structure correctly according to the provided flags if a contiguous chunk of “unsigned bytes” is all that can be exported. </section> <section id="the-py-buffer-struct"> <h3><a class="toc-backref" href="#the-py-buffer-struct" role="doc-backlink">The Py_buffer struct</a></h3> The bufferinfo structure is: <div class="highlight-default notranslate"><div class="highlight"><pre>struct bufferinfo { void *buf; Py_ssize_t len; int readonly; const char *format; int ndim; Py_ssize_t *shape; Py_ssize_t *strides; Py_ssize_t *suboffsets; Py_ssize_t itemsize; void *internal; } Py_buffer; </pre></div> </div> Before calling the bf_getbuffer function, the bufferinfo structure can be filled with whatever, but the <code class="docutils literal notranslate">buf</code> field must be NULL when requesting a new buffer. Upon return from bf_getbuffer, the bufferinfo structure is filled in with relevant information about the buffer. This same bufferinfo structure must be passed to bf_releasebuffer (if available) when the consumer is done with the memory. The caller is responsible for keeping a reference to obj until releasebuffer is called (i.e. the call to bf_getbuffer does not alter the reference count of obj). The members of the bufferinfo structure are: <dl> <dt><code class="docutils literal notranslate">buf</code></dt><dd>a pointer to the start of the memory for the object</dd> <dt><code class="docutils literal notranslate">len</code></dt><dd>the total bytes of memory the object uses. This should be the same as the product of the shape array multiplied by the number of bytes per item of memory.</dd> <dt><code class="docutils literal notranslate">readonly</code></dt><dd>an integer variable to hold whether or not the memory is readonly. 1 means the memory is readonly, zero means the memory is writable.</dd> <dt><code class="docutils literal notranslate">format</code></dt><dd>a NULL-terminated format-string (following the struct-style syntax including extensions) indicating what is in each element of memory. The number of elements is len / itemsize, where itemsize is the number of bytes implied by the format. This can be NULL which implies standard unsigned bytes (“B”).</dd> <dt><code class="docutils literal notranslate">ndim</code></dt><dd>a variable storing the number of dimensions the memory represents. Must be >=0. A value of 0 means that shape and strides and suboffsets must be <code class="docutils literal notranslate">NULL</code> (i.e. the memory represents a scalar).</dd> <dt><code class="docutils literal notranslate">shape</code></dt><dd>an array of <code class="docutils literal notranslate">Py_ssize_t</code> of length <code class="docutils literal notranslate">ndims</code> indicating the shape of the memory as an N-D array. Note that <code class="docutils literal notranslate">((*shape)[0] * ... * (*shape)[ndims-1])*itemsize = len</code>. If ndims is 0 (indicating a scalar), then this must be <code class="docutils literal notranslate">NULL</code>.</dd> <dt><code class="docutils literal notranslate">strides</code></dt><dd>address of a <code class="docutils literal notranslate">Py_ssize_t*</code> variable that will be filled with a pointer to an array of <code class="docutils literal notranslate">Py_ssize_t</code> of length <code class="docutils literal notranslate">ndims</code> (or <code class="docutils literal notranslate">NULL</code> if <code class="docutils literal notranslate">ndims</code> is 0). indicating the number of bytes to skip to get to the next element in each dimension. If this is not requested by the caller (<code class="docutils literal notranslate">PyBUF_STRIDES</code> is not set), then this should be set to NULL which indicates a C-style contiguous array or a PyExc_BufferError raised if this is not possible.</dd> <dt><code class="docutils literal notranslate">suboffsets</code></dt><dd>address of a <code class="docutils literal notranslate">Py_ssize_t *</code> variable that will be filled with a pointer to an array of <code class="docutils literal notranslate">Py_ssize_t</code> of length <code class="docutils literal notranslate">*ndims</code>. If these suboffset numbers are >=0, then the value stored along the indicated dimension is a pointer and the suboffset value dictates how many bytes to add to the pointer after de-referencing. A suboffset value that it negative indicates that no de-referencing should occur (striding in a contiguous memory block). If all suboffsets are negative (i.e. no de-referencing is needed, then this must be NULL (the default value). If this is not requested by the caller (PyBUF_INDIRECT is not set), then this should be set to NULL or an PyExc_BufferError raised if this is not possible.For clarity, here is a function that returns a pointer to the element in an N-D array pointed to by an N-dimensional index when there are both non-NULL strides and suboffsets: <div class="highlight-default notranslate"><div class="highlight"><pre>void *get_item_pointer(int ndim, void *buf, Py_ssize_t *strides, Py_ssize_t *suboffsets, Py_ssize_t *indices) { char *pointer = (char*)buf; int i; for (i = 0; i < ndim; i++) { pointer += strides[i] * indices[i]; if (suboffsets[i] >=0 ) { pointer = *((char**)pointer) + suboffsets[i]; } } return (void*)pointer; } </pre></div> </div> Notice the suboffset is added “after” the dereferencing occurs. Thus slicing in the ith dimension would add to the suboffsets in the (i-1)st dimension. Slicing in the first dimension would change the location of the starting pointer directly (i.e. buf would be modified). </dd> <dt><code class="docutils literal notranslate">itemsize</code></dt><dd>This is a storage for the itemsize (in bytes) of each element of the shared memory. It is technically un-necessary as it can be obtained using <code class="docutils literal notranslate">PyBuffer_SizeFromFormat</code>, however an exporter may know this information without parsing the format string and it is necessary to know the itemsize for proper interpretation of striding. Therefore, storing it is more convenient and faster.</dd> <dt><code class="docutils literal notranslate">internal</code></dt><dd>This is for use internally by the exporting object. For example, this might be re-cast as an integer by the exporter and used to store flags about whether or not the shape, strides, and suboffsets arrays must be freed when the buffer is released. The consumer should never alter this value.</dd> </dl> The exporter is responsible for making sure that any memory pointed to by buf, format, shape, strides, and suboffsets is valid until releasebuffer is called. If the exporter wants to be able to change an object’s shape, strides, and/or suboffsets before releasebuffer is called then it should allocate those arrays when getbuffer is called (pointing to them in the buffer-info structure provided) and free them when releasebuffer is called. </section> <section id="releasing-the-buffer"> <h3><a class="toc-backref" href="#releasing-the-buffer" role="doc-backlink">Releasing the buffer</a></h3> The same bufferinfo struct should be used in the release-buffer interface call. The caller is responsible for the memory of the Py_buffer structure itself. <div class="highlight-default notranslate"><div class="highlight"><pre>typedef void (*releasebufferproc)(PyObject *obj, Py_buffer *view) </pre></div> </div> Callers of getbufferproc must make sure that this function is called when memory previously acquired from the object is no longer needed. The exporter of the interface must make sure that any memory pointed to in the bufferinfo structure remains valid until releasebuffer is called. If the bf_releasebuffer function is not provided (i.e. it is NULL), then it does not ever need to be called. Exporters will need to define a bf_releasebuffer function if they can re-allocate their memory, strides, shape, suboffsets, or format variables which they might share through the struct bufferinfo. Several mechanisms could be used to keep track of how many getbuffer calls have been made and shared. Either a single variable could be used to keep track of how many “views” have been exported, or a linked-list of bufferinfo structures filled in could be maintained in each object. All that is specifically required by the exporter, however, is to ensure that any memory shared through the bufferinfo structure remains valid until releasebuffer is called on the bufferinfo structure exporting that memory. </section> </section> <section id="new-c-api-calls-are-proposed"> <h2><a class="toc-backref" href="#new-c-api-calls-are-proposed" role="doc-backlink">New C-API calls are proposed</a></h2> <div class="highlight-default notranslate"><div class="highlight"><pre>int PyObject_CheckBuffer(PyObject *obj) </pre></div> </div> Return 1 if the getbuffer function is available otherwise 0. <div class="highlight-default notranslate"><div class="highlight"><pre>int PyObject_GetBuffer(PyObject *obj, Py_buffer *view, int flags) </pre></div> </div> This is a C-API version of the getbuffer function call. It checks to make sure object has the required function pointer and issues the call. Returns -1 and raises an error on failure and returns 0 on success. <div class="highlight-default notranslate"><div class="highlight"><pre>void PyBuffer_Release(PyObject *obj, Py_buffer *view) </pre></div> </div> This is a C-API version of the releasebuffer function call. It checks to make sure the object has the required function pointer and issues the call. This function always succeeds even if there is no releasebuffer function for the object. <div class="highlight-default notranslate"><div class="highlight"><pre>PyObject *PyObject_GetMemoryView(PyObject *obj) </pre></div> </div> Return a memory-view object from an object that defines the buffer interface. A memory-view object is an extended buffer object that could replace the buffer object (but doesn’t have to as that could be kept as a simple 1-d memory-view object). Its C-structure is <div class="highlight-default notranslate"><div class="highlight"><pre>typedef struct { PyObject_HEAD PyObject *base; Py_buffer view; } PyMemoryViewObject; </pre></div> </div> This is functionally similar to the current buffer object except a reference to base is kept and the memory view is not re-grabbed. Thus, this memory view object holds on to the memory of base until it is deleted. This memory-view object will support multi-dimensional slicing and be the first object provided with Python to do so. Slices of the memory-view object are other memory-view objects with the same base but with a different view of the base object. When an “element” from the memory-view is returned it is always a bytes object whose format should be interpreted by the format attribute of the memoryview object. The struct module can be used to “decode” the bytes in Python if desired. Or the contents can be passed to a NumPy array or other object consuming the buffer protocol. The Python name will be <code class="docutils literal notranslate">__builtin__.memoryview</code> Methods: <div class="line-block"> <div class="line"><code class="docutils literal notranslate">__getitem__</code> (will support multi-dimensional slicing)</div> <div class="line"><code class="docutils literal notranslate">__setitem__</code> (will support multi-dimensional slicing)</div> <div class="line"><code class="docutils literal notranslate">tobytes</code> (obtain a new bytes-object of a copy of the memory).</div> <div class="line"><code class="docutils literal notranslate">tolist</code> (obtain a “nested” list of the memory. Everything is interpreted into standard Python objects as the struct module unpack would do – in fact it uses struct.unpack to accomplish it).</div> </div> Attributes (taken from the memory of the base object): <ul class="simple"> <li><code class="docutils literal notranslate">format</code></li> <li><code class="docutils literal notranslate">itemsize</code></li> <li><code class="docutils literal notranslate">shape</code></li> <li><code class="docutils literal notranslate">strides</code></li> <li><code class="docutils literal notranslate">suboffsets</code></li> <li><code class="docutils literal notranslate">readonly</code></li> <li><code class="docutils literal notranslate">ndim</code></li> </ul> <div class="highlight-default notranslate"><div class="highlight"><pre>Py_ssize_t PyBuffer_SizeFromFormat(const char *) </pre></div> </div> Return the implied itemsize of the data-format area from a struct-style description. <div class="highlight-default notranslate"><div class="highlight"><pre>PyObject * PyMemoryView_GetContiguous(PyObject *obj, int buffertype, char fortran) </pre></div> </div> Return a memoryview object to a contiguous chunk of memory represented by obj. If a copy must be made (because the memory pointed to by obj is not contiguous), then a new bytes object will be created and become the base object for the returned memory view object. The buffertype argument can be PyBUF_READ, PyBUF_WRITE, PyBUF_UPDATEIFCOPY to determine whether the returned buffer should be readable, writable, or set to update the original buffer if a copy must be made. If buffertype is PyBUF_WRITE and the buffer is not contiguous an error will be raised. In this circumstance, the user can use PyBUF_UPDATEIFCOPY to ensure that a writable temporary contiguous buffer is returned. The contents of this contiguous buffer will be copied back into the original object after the memoryview object is deleted as long as the original object is writable. If this is not allowed by the original object, then a BufferError is raised. If the object is multi-dimensional, then if fortran is ‘F’, the first dimension of the underlying array will vary the fastest in the buffer. If fortran is ‘C’, then the last dimension will vary the fastest (C-style contiguous). If fortran is ‘A’, then it does not matter and you will get whatever the object decides is more efficient. If a copy is made, then the memory must be freed by calling <code class="docutils literal notranslate">PyMem_Free</code>. You receive a new reference to the memoryview object. <div class="highlight-default notranslate"><div class="highlight"><pre>int PyObject_CopyToObject(PyObject *obj, void *buf, Py_ssize_t len, char fortran) </pre></div> </div> Copy <code class="docutils literal notranslate">len</code> bytes of data pointed to by the contiguous chunk of memory pointed to by <code class="docutils literal notranslate">buf</code> into the buffer exported by obj. Return 0 on success and return -1 and raise an error on failure. If the object does not have a writable buffer, then an error is raised. If fortran is ‘F’, then if the object is multi-dimensional, then the data will be copied into the array in Fortran-style (first dimension varies the fastest). If fortran is ‘C’, then the data will be copied into the array in C-style (last dimension varies the fastest). If fortran is ‘A’, then it does not matter and the copy will be made in whatever way is more efficient. <div class="highlight-default notranslate"><div class="highlight"><pre>int PyObject_CopyData(PyObject *dest, PyObject *src) </pre></div> </div> These last three C-API calls allow a standard way of getting data in and out of Python objects into contiguous memory areas no matter how it is actually stored. These calls use the extended buffer interface to perform their work. <div class="highlight-default notranslate"><div class="highlight"><pre>int PyBuffer_IsContiguous(Py_buffer *view, char fortran) </pre></div> </div> Return 1 if the memory defined by the view object is C-style (fortran = ‘C’) or Fortran-style (fortran = ‘F’) contiguous or either one (fortran = ‘A’). Return 0 otherwise. <div class="highlight-default notranslate"><div class="highlight"><pre>void PyBuffer_FillContiguousStrides(int ndim, Py_ssize_t *shape, Py_ssize_t *strides, Py_ssize_t itemsize, char fortran) </pre></div> </div> Fill the strides array with byte-strides of a contiguous (C-style if fortran is ‘C’ or Fortran-style if fortran is ‘F’ array of the given shape with the given number of bytes per element. <div class="highlight-default notranslate"><div class="highlight"><pre>int PyBuffer_FillInfo(Py_buffer *view, void *buf, Py_ssize_t len, int readonly, int infoflags) </pre></div> </div> Fills in a buffer-info structure correctly for an exporter that can only share a contiguous chunk of memory of “unsigned bytes” of the given length. Returns 0 on success and -1 (with raising an error) on error. <div class="highlight-default notranslate"><div class="highlight"><pre>PyExc_BufferError </pre></div> </div> A new error object for returning buffer errors which arise because an exporter cannot provide the kind of buffer that a consumer expects. This will also be raised when a consumer requests a buffer from an object that does not provide the protocol. </section> <section id="additions-to-the-struct-string-syntax"> <h2><a class="toc-backref" href="#additions-to-the-struct-string-syntax" role="doc-backlink">Additions to the struct string-syntax</a></h2> The struct string-syntax is missing some characters to fully implement data-format descriptions already available elsewhere (in ctypes and NumPy for example). The Python 2.5 specification is at <a class="reference external" href="http://docs.python.org/library/struct.html">http://docs.python.org/library/struct.html</a>. Here are the proposed additions: <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Character</th> <th class="head">Description</th> </tr> </thead> <tbody> <tr class="row-even"><td>‘t’</td> <td>bit (number before states how many bits)</td> </tr> <tr class="row-odd"><td>‘?’</td> <td>platform _Bool type</td> </tr> <tr class="row-even"><td>‘g’</td> <td>long double</td> </tr> <tr class="row-odd"><td>‘c’</td> <td>ucs-1 (latin-1) encoding</td> </tr> <tr class="row-even"><td>‘u’</td> <td>ucs-2</td> </tr> <tr class="row-odd"><td>‘w’</td> <td>ucs-4</td> </tr> <tr class="row-even"><td>‘O’</td> <td>pointer to Python Object</td> </tr> <tr class="row-odd"><td>‘Z’</td> <td>complex (whatever the next specifier is)</td> </tr> <tr class="row-even"><td>‘&’</td> <td>specific pointer (prefix before another character)</td> </tr> <tr class="row-odd"><td>‘T{}’</td> <td>structure (detailed layout inside {})</td> </tr> <tr class="row-even"><td>‘(k1,k2,…,kn)’</td> <td>multi-dimensional array of whatever follows</td> </tr> <tr class="row-odd"><td>‘:name:’</td> <td>optional name of the preceding element</td> </tr> <tr class="row-even"><td>‘X{}’</td> <td><dl class="simple"> <dt>pointer to a function (optional function</dt><dd>signature inside {} with any return value preceded by -> and placed at the end)</dd> </dl> </td> </tr> </tbody> </table> The struct module will be changed to understand these as well and return appropriate Python objects on unpacking. Unpacking a long-double will return a decimal object or a ctypes long-double. Unpacking ‘u’ or ‘w’ will return Python unicode. Unpacking a multi-dimensional array will return a list (of lists if >1d). Unpacking a pointer will return a ctypes pointer object. Unpacking a function pointer will return a ctypes call-object (perhaps). Unpacking a bit will return a Python Bool. White-space in the struct-string syntax will be ignored if it isn’t already. Unpacking a named-object will return some kind of named-tuple-like object that acts like a tuple but whose entries can also be accessed by name. Unpacking a nested structure will return a nested tuple. Endian-specification (‘!’, ‘@’,’=’,’>’,’<’, ‘^’) is also allowed inside the string so that it can change if needed. The previously-specified endian string is in force until changed. The default endian is ‘@’ which means native data-types and alignment. If un-aligned, native data-types are requested, then the endian specification is ‘^’. According to the struct-module, a number can precede a character code to specify how many of that type there are. The <code class="docutils literal notranslate">(k1,k2,...,kn)</code> extension also allows specifying if the data is supposed to be viewed as a (C-style contiguous, last-dimension varies the fastest) multi-dimensional array of a particular format. Functions should be added to ctypes to create a ctypes object from a struct description, and add long-double, and ucs-2 to ctypes. </section> <section id="examples-of-data-format-descriptions"> <h2><a class="toc-backref" href="#examples-of-data-format-descriptions" role="doc-backlink">Examples of Data-Format Descriptions</a></h2> Here are some examples of C-structures and how they would be represented using the struct-style syntax. <named> is the constructor for a named-tuple (not-specified yet). <dl> <dt>float</dt><dd><code class="docutils literal notranslate">'d'</code> <–> Python float</dd> <dt>complex double</dt><dd><code class="docutils literal notranslate">'Zd'</code> <–> Python complex</dd> <dt>RGB Pixel data</dt><dd><code class="docutils literal notranslate">'BBB'</code> <–> (int, int, int) <code class="docutils literal notranslate">'B:r: B:g: B:b:'</code> <–> <named>((int, int, int), (‘r’,’g’,’b’))</dd> <dt>Mixed endian (weird but possible)</dt><dd><code class="docutils literal notranslate">'>i:big: <i:little:'</code> <–> <named>((int, int), (‘big’, ‘little’))</dd> <dt>Nested structure</dt><dd><div class="highlight-default notranslate"><div class="highlight"><pre>struct { int ival; struct { unsigned short sval; unsigned char bval; unsigned char cval; } sub; } """i:ival: T{ H:sval: B:bval: B:cval: }:sub: """ </pre></div> </div> </dd> <dt>Nested array</dt><dd><div class="highlight-default notranslate"><div class="highlight"><pre>struct { int ival; double data[16*4]; } """i:ival: (16,4)d:data: """ </pre></div> </div> </dd> </dl> Note, that in the last example, the C-structure compared against is intentionally a 1-d array and not a 2-d array data[16][4]. The reason for this is to avoid the confusions between static multi-dimensional arrays in C (which are laid out contiguously) and dynamic multi-dimensional arrays which use the same syntax to access elements, data[0][1], but whose memory is not necessarily contiguous. The struct-syntax always uses contiguous memory and the multi-dimensional character is information about the memory to be communicated by the exporter. In other words, the struct-syntax description does not have to match the C-syntax exactly as long as it describes the same memory layout. The fact that a C-compiler would think of the memory as a 1-d array of doubles is irrelevant to the fact that the exporter wanted to communicate to the consumer that this field of the memory should be thought of as a 2-d array where a new dimension is considered after every 4 elements. </section> <section id="code-to-be-affected"> <h2><a class="toc-backref" href="#code-to-be-affected" role="doc-backlink">Code to be affected</a></h2> All objects and modules in Python that export or consume the old buffer interface will be modified. Here is a partial list. <ul class="simple"> <li>buffer object</li> <li>bytes object</li> <li>string object</li> <li>unicode object</li> <li>array module</li> <li>struct module</li> <li>mmap module</li> <li>ctypes module</li> </ul> Anything else using the buffer API. </section> <section id="issues-and-details"> <h2><a class="toc-backref" href="#issues-and-details" role="doc-backlink">Issues and Details</a></h2> It is intended that this PEP will be back-ported to Python 2.6 by adding the C-API and the two functions to the existing buffer protocol. Previous versions of this PEP proposed a read/write locking scheme, but it was later perceived as a) too complicated for common simple use cases that do not require any locking and b) too simple for use cases that required concurrent read/write access to a buffer with changing, short-living locks. It is therefore left to users to implement their own specific locking scheme around buffer objects if they require consistent views across concurrent read/write access. A future PEP may be proposed which includes a separate locking API after some experience with these user-schemes is obtained The sharing of strided memory and suboffsets is new and can be seen as a modification of the multiple-segment interface. It is motivated by NumPy and the PIL. NumPy objects should be able to share their strided memory with code that understands how to manage strided memory because strided memory is very common when interfacing with compute libraries. Also, with this approach it should be possible to write generic code that works with both kinds of memory without copying. Memory management of the format string, the shape array, the strides array, and the suboffsets array in the bufferinfo structure is always the responsibility of the exporting object. The consumer should not set these pointers to any other memory or try to free them. Several ideas were discussed and rejected: <blockquote> <div>Having a “releaser” object whose release-buffer was called. This was deemed unacceptable because it caused the protocol to be asymmetric (you called release on something different than you “got” the buffer from). It also complicated the protocol without providing a real benefit.Passing all the struct variables separately into the function. This had the advantage that it allowed one to set NULL to variables that were not of interest, but it also made the function call more difficult. The flags variable allows the same ability of consumers to be “simple” in how they call the protocol. </div></blockquote> </section> <section id="code"> <h2><a class="toc-backref" href="#code" role="doc-backlink">Code</a></h2> The authors of the PEP promise to contribute and maintain the code for this proposal but will welcome any help. </section> <section id="examples"> <h2><a class="toc-backref" href="#examples" role="doc-backlink">Examples</a></h2> <section id="ex-1"> <h3><a class="toc-backref" href="#ex-1" role="doc-backlink">Ex. 1</a></h3> This example shows how an image object that uses contiguous lines might expose its buffer: <div class="highlight-default notranslate"><div class="highlight"><pre>struct rgba { unsigned char r, g, b, a; }; struct ImageObject { PyObject_HEAD; ... struct rgba** lines; Py_ssize_t height; Py_ssize_t width; Py_ssize_t shape_array[2]; Py_ssize_t stride_array[2]; Py_ssize_t view_count; }; </pre></div> </div> “lines” points to malloced 1-D array of <code class="docutils literal notranslate">(struct rgba*)</code>. Each pointer in THAT block points to a separately malloced array of <code class="docutils literal notranslate">(struct rgba)</code>. In order to access, say, the red value of the pixel at x=30, y=50, you’d use “lines[50][30].r”. So what does ImageObject’s getbuffer do? Leaving error checking out: <div class="highlight-default notranslate"><div class="highlight"><pre>int Image_getbuffer(PyObject *self, Py_buffer *view, int flags) { static Py_ssize_t suboffsets[2] = { 0, -1}; view->buf = self->lines; view->len = self->height*self->width; view->readonly = 0; view->ndims = 2; self->shape_array[0] = height; self->shape_array[1] = width; view->shape = &self->shape_array; self->stride_array[0] = sizeof(struct rgba*); self->stride_array[1] = sizeof(struct rgba); view->strides = &self->stride_array; view->suboffsets = suboffsets; self->view_count ++; return 0; } int Image_releasebuffer(PyObject *self, Py_buffer *view) { self->view_count--; return 0; } </pre></div> </div> </section> <section id="ex-2"> <h3><a class="toc-backref" href="#ex-2" role="doc-backlink">Ex. 2</a></h3> This example shows how an object that wants to expose a contiguous chunk of memory (which will never be re-allocated while the object is alive) would do that. <div class="highlight-default notranslate"><div class="highlight"><pre>int myobject_getbuffer(PyObject *self, Py_buffer *view, int flags) { void *buf; Py_ssize_t len; int readonly=0; buf = /* Point to buffer */ len = /* Set to size of buffer */ readonly = /* Set to 1 if readonly */ return PyObject_FillBufferInfo(view, buf, len, readonly, flags); } /* No releasebuffer is necessary because the memory will never be re-allocated */ </pre></div> </div> </section> <section id="ex-3"> <h3><a class="toc-backref" href="#ex-3" role="doc-backlink">Ex. 3</a></h3> A consumer that wants to only get a simple contiguous chunk of bytes from a Python object, obj would do the following: <div class="highlight-default notranslate"><div class="highlight"><pre>Py_buffer view; int ret; if (PyObject_GetBuffer(obj, &view, Py_BUF_SIMPLE) < 0) { /* error return */ } /* Now, view.buf is the pointer to memory view.len is the length view.readonly is whether or not the memory is read-only. */ /* After using the information and you don't need it anymore */ if (PyBuffer_Release(obj, &view) < 0) { /* error return */ } </pre></div> </div> </section> <section id="ex-4"> <h3><a class="toc-backref" href="#ex-4" role="doc-backlink">Ex. 4</a></h3> A consumer that wants to be able to use any object’s memory but is writing an algorithm that only handle contiguous memory could do the following: <div class="highlight-default notranslate"><div class="highlight"><pre>void *buf; Py_ssize_t len; char *format; int copy; copy = PyObject_GetContiguous(obj, &buf, &len, &format, 0, 'A'); if (copy < 0) { /* error return */ } /* process memory pointed to by buffer if format is correct */ /* Optional: if, after processing, we want to copy data from buffer back into the object we could do */ if (PyObject_CopyToObject(obj, buf, len, 'A') < 0) { /* error return */ } /* Make sure that if a copy was made, the memory is freed */ if (copy == 1) PyMem_Free(buf); </pre></div> </div> </section> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This PEP is placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-3118.rst">https://github.com/python/peps/blob/main/peps/pep-3118.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-3118.rst">2023-09-09 17:39:29 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#proposal-overview">Proposal Overview</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#access-flags">Access flags</a></li> <li><a class="reference internal" href="#the-py-buffer-struct">The Py_buffer struct</a></li> <li><a class="reference internal" href="#releasing-the-buffer">Releasing the buffer</a></li> </ul> </li> <li><a class="reference internal" href="#new-c-api-calls-are-proposed">New C-API calls are proposed</a></li> <li><a class="reference internal" href="#additions-to-the-struct-string-syntax">Additions to the struct string-syntax</a></li> <li><a class="reference internal" href="#examples-of-data-format-descriptions">Examples of Data-Format Descriptions</a></li> <li><a class="reference internal" href="#code-to-be-affected">Code to be affected</a></li> <li><a class="reference internal" href="#issues-and-details">Issues and Details</a></li> <li><a class="reference internal" href="#code">Code</a></li> <li><a class="reference internal" href="#examples">Examples</a><ul> <li><a class="reference internal" href="#ex-1">Ex. 1</a></li> <li><a class="reference internal" href="#ex-2">Ex. 2</a></li> <li><a class="reference internal" href="#ex-3">Ex. 3</a></li> <li><a class="reference internal" href="#ex-4">Ex. 4</a></li> </ul> </li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-3118.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>