CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 688 – Making the buffer protocol accessible in Python | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0688/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 688 – Making the buffer protocol accessible in Python | peps.python.org'> <meta property="og:description" content="This PEP proposes a Python-level API for the buffer protocol, which is currently accessible only to C code. This allows type checkers to evaluate whether objects implement the protocol."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0688/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This PEP proposes a Python-level API for the buffer protocol, which is currently accessible only to C code. This allows type checkers to evaluate whether objects implement the protocol."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 688</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 688 – Making the buffer protocol accessible in Python</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Jelle Zijlstra <jelle.zijlstra at gmail.com></dd> <dt class="field-even">Discussions-To:</dt> <dd class="field-even"><a class="reference external" href="https://discuss.python.org/t/19756">Discourse thread</a></dd> <dt class="field-odd">Status:</dt> <dd class="field-odd"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-even">Type:</dt> <dd class="field-even"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-odd">Topic:</dt> <dd class="field-odd"><a class="reference external" href="../topic/typing/">Typing</a></dd> <dt class="field-even">Created:</dt> <dd class="field-even">23-Apr-2022</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">3.12</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even"><a class="reference external" href="https://mail.python.org/archives/list/typing-sig@python.org/thread/CX7GPSIYQEL23RXMYL66GAKGP4RLUD7P/" title="Typing-SIG thread">23-Apr-2022</a>, <a class="reference external" href="https://discuss.python.org/t/15265" title="Discourse thread">25-Apr-2022</a>, <a class="reference external" href="https://discuss.python.org/t/19756" title="Discourse thread">06-Oct-2022</a>, <a class="reference external" href="https://mail.python.org/archives/list/typing-sig@python.org/thread/XH5ZK2MSZIQLL62PYZ6I5532SQKKVCBL/" title="Typing-SIG thread">26-Oct-2022</a></dd> <dt class="field-odd">Resolution:</dt> <dd class="field-odd"><a class="reference external" href="https://discuss.python.org/t/pep-688-making-the-buffer-protocol-accessible-in-python/15265/35">07-Mar-2023</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#current-options">Current options</a></li> <li><a class="reference internal" href="#kinds-of-buffers">Kinds of buffers</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#python-level-buffer-protocol">Python-level buffer protocol</a></li> <li><a class="reference internal" href="#inspect-bufferflags"><code class="docutils literal notranslate">inspect.BufferFlags</code></a></li> <li><a class="reference internal" href="#collections-abc-buffer"><code class="docutils literal notranslate">collections.abc.Buffer</code></a></li> <li><a class="reference internal" href="#example">Example</a></li> <li><a class="reference internal" href="#equivalent-for-older-python-versions">Equivalent for older Python versions</a></li> <li><a class="reference internal" href="#no-special-meaning-for-bytes">No special meaning for <code class="docutils literal notranslate">bytes</code></a></li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a><ul> <li><a class="reference internal" href="#buffer-and-release-buffer-attributes"><code class="docutils literal notranslate">__buffer__</code> and <code class="docutils literal notranslate">__release_buffer__</code> attributes</a></li> <li><a class="reference internal" href="#removal-of-the-bytes-special-case">Removal of the <code class="docutils literal notranslate">bytes</code> special case</a></li> </ul> </li> <li><a class="reference internal" href="#how-to-teach-this">How to Teach This</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a><ul> <li><a class="reference internal" href="#types-buffer"><code class="docutils literal notranslate">types.Buffer</code></a></li> <li><a class="reference internal" href="#keep-bytearray-compatible-with-bytes">Keep <code class="docutils literal notranslate">bytearray</code> compatible with <code class="docutils literal notranslate">bytes</code></a></li> <li><a class="reference internal" href="#distinguish-between-mutable-and-immutable-buffers">Distinguish between mutable and immutable buffers</a></li> </ul> </li> <li><a class="reference internal" href="#acknowledgments">Acknowledgments</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <div class="pep-banner canonical-doc sticky-banner admonition important"> Important This PEP is a historical document. The up-to-date, canonical documentation can now be found at <a class="reference external" href="https://docs.python.org/3/reference/datamodel.html#python-buffer-protocol" title="(in Python v3.13)">Emulating buffer types</a>. × See <a class="pep reference internal" href="../pep-0001/" title="PEP 1 – PEP Purpose and Guidelines">PEP 1</a> for how to propose changes. </div> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> This PEP proposes a Python-level API for the buffer protocol, which is currently accessible only to C code. This allows type checkers to evaluate whether objects implement the protocol. </section> <section id="motivation"> <h2><a class="toc-backref" href="#motivation" role="doc-backlink">Motivation</a></h2> The CPython C API provides a versatile mechanism for accessing the underlying memory of an object—the <a class="reference external" href="https://docs.python.org/3/c-api/buffer.html">buffer protocol</a> introduced in <a class="pep reference internal" href="../pep-3118/" title="PEP 3118 – Revising the buffer protocol">PEP 3118</a>. Functions that accept binary data are usually written to handle any object implementing the buffer protocol. For example, at the time of writing, there are around 130 functions in CPython using the Argument Clinic <code class="docutils literal notranslate">Py_buffer</code> type, which accepts the buffer protocol. Currently, there is no way for Python code to inspect whether an object supports the buffer protocol. Moreover, the static type system does not provide a type annotation to represent the protocol. This is a <a class="reference external" href="https://github.com/python/typing/issues/593">common problem</a> when writing type annotations for code that accepts generic buffers. Similarly, it is impossible for a class written in Python to support the buffer protocol. A buffer class in Python would give users the ability to easily wrap a C buffer object, or to test the behavior of an API that consumes the buffer protocol. Granted, this is not a particularly common need. However, there has been a <a class="reference external" href="https://github.com/python/cpython/issues/58006">CPython feature request</a> for supporting buffer classes written in Python that has been open since 2012. </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> <section id="current-options"> <h3><a class="toc-backref" href="#current-options" role="doc-backlink">Current options</a></h3> There are two known workarounds for annotating buffer types in the type system, but neither is adequate. First, the <a class="reference external" href="https://github.com/python/typeshed/blob/2a0fc1b582ef84f7a82c0beb39fa617de2539d3d/stdlib/_typeshed/__init__.pyi#L194">current workaround</a> for buffer types in typeshed is a type alias that lists well-known buffer types in the standard library, such as <code class="docutils literal notranslate">bytes</code>, <code class="docutils literal notranslate">bytearray</code>, <code class="docutils literal notranslate">memoryview</code>, and <code class="docutils literal notranslate">array.array</code>. This approach works for the standard library, but it does not extend to third-party buffer types. Second, the <a class="reference external" href="https://docs.python.org/3.10/library/typing.html#typing.ByteString">documentation</a> for <code class="docutils literal notranslate">typing.ByteString</code> currently states: <blockquote> <div>This type represents the types <code class="docutils literal notranslate">bytes</code>, <code class="docutils literal notranslate">bytearray</code>, and <code class="docutils literal notranslate">memoryview</code> of byte sequences.As a shorthand for this type, <code class="docutils literal notranslate">bytes</code> can be used to annotate arguments of any of the types mentioned above. </div></blockquote> Although this sentence has been in the documentation <a class="reference external" href="https://github.com/python/cpython/commit/2a19d956ab92fc9084a105cc11292cb0438b322f">since 2015</a>, the use of <code class="docutils literal notranslate">bytes</code> to include these other types is not specified in any of the typing PEPs. Furthermore, this mechanism has a number of problems. It does not include all possible buffer types, and it makes the <code class="docutils literal notranslate">bytes</code> type ambiguous in type annotations. After all, there are many operations that are valid on <code class="docutils literal notranslate">bytes</code> objects, but not on <code class="docutils literal notranslate">memoryview</code> objects, and it is perfectly possible for a function to accept <code class="docutils literal notranslate">bytes</code> but not <code class="docutils literal notranslate">memoryview</code> objects. A mypy user <a class="reference external" href="https://github.com/python/mypy/issues/12643#issuecomment-1105914159">reports</a> that this shortcut has caused significant problems for the <code class="docutils literal notranslate">psycopg</code> project. </section> <section id="kinds-of-buffers"> <h3><a class="toc-backref" href="#kinds-of-buffers" role="doc-backlink">Kinds of buffers</a></h3> The C buffer protocol supports <a class="reference external" href="https://docs.python.org/3.10/c-api/buffer.html#buffer-request-types">many options</a>, affecting strides, contiguity, and support for writing to the buffer. Some of these options would be useful in the type system. For example, typeshed currently provides separate type aliases for writable and read-only buffers. However, in the C buffer protocol, most of these options cannot be queried directly on the type object. The only way to figure out whether an object supports a particular flag is to actually ask for the buffer. For some types, such as <code class="docutils literal notranslate">memoryview</code>, the supported flags depend on the instance. As a result, it would be difficult to represent support for these flags in the type system. </section> </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> <section id="python-level-buffer-protocol"> <h3><a class="toc-backref" href="#python-level-buffer-protocol" role="doc-backlink">Python-level buffer protocol</a></h3> We propose to add two Python-level special methods, <code class="docutils literal notranslate">__buffer__</code> and <code class="docutils literal notranslate">__release_buffer__</code>. Python classes that implement these methods are usable as buffers from C code. Conversely, classes implemented in C that support the buffer protocol acquire synthesized methods accessible from Python code. The <code class="docutils literal notranslate">__buffer__</code> method is called to create a buffer from a Python object, for example by the <code class="docutils literal notranslate">memoryview()</code> constructor. It corresponds to the <code class="docutils literal notranslate">bf_getbuffer</code> C slot. The Python signature for this method is <code class="docutils literal notranslate">def __buffer__(self, flags: int, /) -> memoryview: ...</code>. The method must return a <code class="docutils literal notranslate">memoryview</code> object. If the <code class="docutils literal notranslate">bf_getbuffer</code> slot is invoked on a Python class with a <code class="docutils literal notranslate">__buffer__</code> method, the interpreter extracts the underlying <code class="docutils literal notranslate">Py_buffer</code> from the <code class="docutils literal notranslate">memoryview</code> returned by the method and returns it to the C caller. Similarly, if Python code calls the <code class="docutils literal notranslate">__buffer__</code> method on an instance of a C class that implements <code class="docutils literal notranslate">bf_getbuffer</code>, the returned buffer is wrapped in a <code class="docutils literal notranslate">memoryview</code> for consumption by Python code. The <code class="docutils literal notranslate">__release_buffer__</code> method should be called when a caller no longer needs the buffer returned by <code class="docutils literal notranslate">__buffer__</code>. It corresponds to the <code class="docutils literal notranslate">bf_releasebuffer</code> C slot. This is an optional part of the buffer protocol. The Python signature for this method is <code class="docutils literal notranslate">def __release_buffer__(self, buffer: memoryview, /) -> None: ...</code>. The buffer to be released is wrapped in a <code class="docutils literal notranslate">memoryview</code>. When this method is invoked through CPython’s buffer API (for example, through calling <code class="docutils literal notranslate">memoryview.release</code> on a <code class="docutils literal notranslate">memoryview</code> returned by <code class="docutils literal notranslate">__buffer__</code>), the passed <code class="docutils literal notranslate">memoryview</code> is the same object as was returned by <code class="docutils literal notranslate">__buffer__</code>. It is also possible to call <code class="docutils literal notranslate">__release_buffer__</code> on a C class that implements <code class="docutils literal notranslate">bf_releasebuffer</code>. If <code class="docutils literal notranslate">__release_buffer__</code> exists on an object, Python code that calls <code class="docutils literal notranslate">__buffer__</code> directly on the object must call <code class="docutils literal notranslate">__release_buffer__</code> on the same object when it is done with the buffer. Otherwise, resources used by the object may not be reclaimed. Similarly, it is a programming error to call <code class="docutils literal notranslate">__release_buffer__</code> without a previous call to <code class="docutils literal notranslate">__buffer__</code>, or to call it multiple times for a single call to <code class="docutils literal notranslate">__buffer__</code>. For objects that implement the C buffer protocol, calls to <code class="docutils literal notranslate">__release_buffer__</code> where the argument is not a <code class="docutils literal notranslate">memoryview</code> wrapping the same object will raise an exception. After a valid call to <code class="docutils literal notranslate">__release_buffer__</code>, the <code class="docutils literal notranslate">memoryview</code> is invalidated (as if its <code class="docutils literal notranslate">release()</code> method had been called), and any subsequent calls to <code class="docutils literal notranslate">__release_buffer__</code> with the same <code class="docutils literal notranslate">memoryview</code> will raise an exception. The interpreter will ensure that misuse of the Python API will not break invariants at the C level – for example, it will not cause memory safety violations. </section> <section id="inspect-bufferflags"> <h3><a class="toc-backref" href="#inspect-bufferflags" role="doc-backlink"><code class="docutils literal notranslate">inspect.BufferFlags</code></a></h3> To help implementations of <code class="docutils literal notranslate">__buffer__</code>, we add <code class="docutils literal notranslate">inspect.BufferFlags</code>, a subclass of <code class="docutils literal notranslate">enum.IntFlag</code>. This enum contains all flags defined in the C buffer protocol. For example, <code class="docutils literal notranslate">inspect.BufferFlags.SIMPLE</code> has the same value as the <code class="docutils literal notranslate">PyBUF_SIMPLE</code> constant. </section> <section id="collections-abc-buffer"> <h3><a class="toc-backref" href="#collections-abc-buffer" role="doc-backlink"><code class="docutils literal notranslate">collections.abc.Buffer</code></a></h3> We add a new abstract base classes, <code class="docutils literal notranslate">collections.abc.Buffer</code>, which requires the <code class="docutils literal notranslate">__buffer__</code> method. This class is intended primarily for use in type annotations: <div class="highlight-python notranslate"><div class="highlight"><pre>def need_buffer(b: Buffer) -> memoryview: return memoryview(b) need_buffer(b"xy") # ok need_buffer("xy") # rejected by static type checkers </pre></div> </div> It can also be used in <code class="docutils literal notranslate">isinstance</code> and <code class="docutils literal notranslate">issubclass</code> checks: <div class="highlight-pycon notranslate"><div class="highlight"><pre>>>> from collections.abc import Buffer >>> isinstance(b"xy", Buffer) True >>> issubclass(bytes, Buffer) True >>> issubclass(memoryview, Buffer) True >>> isinstance("xy", Buffer) False >>> issubclass(str, Buffer) False </pre></div> </div> In the typeshed stub files, the class should be defined as a <code class="docutils literal notranslate">Protocol</code>, following the precedent of other simple ABCs in <code class="docutils literal notranslate">collections.abc</code> such as <code class="docutils literal notranslate">collections.abc.Iterable</code> or <code class="docutils literal notranslate">collections.abc.Sized</code>. </section> <section id="example"> <h3><a class="toc-backref" href="#example" role="doc-backlink">Example</a></h3> The following is an example of a Python class that implements the buffer protocol: <div class="highlight-python notranslate"><div class="highlight"><pre>import contextlib import inspect class MyBuffer: def __init__(self, data: bytes): self.data = bytearray(data) self.view = None def __buffer__(self, flags: int) -> memoryview: if flags != inspect.BufferFlags.FULL_RO: raise TypeError("Only BufferFlags.FULL_RO supported") if self.view is not None: raise RuntimeError("Buffer already held") self.view = memoryview(self.data) return self.view def __release_buffer__(self, view: memoryview) -> None: assert self.view is view # guaranteed to be true self.view.release() self.view = None def extend(self, b: bytes) -> None: if self.view is not None: raise RuntimeError("Cannot extend held buffer") self.data.extend(b) buffer = MyBuffer(b"capybara") with memoryview(buffer) as view: view[0] = ord("C") with contextlib.suppress(RuntimeError): buffer.extend(b"!") # raises RuntimeError buffer.extend(b"!") # ok, buffer is no longer held with memoryview(buffer) as view: assert view.tobytes() == b"Capybara!" </pre></div> </div> </section> <section id="equivalent-for-older-python-versions"> <h3><a class="toc-backref" href="#equivalent-for-older-python-versions" role="doc-backlink">Equivalent for older Python versions</a></h3> New typing features are usually backported to older Python versions in the <a class="reference external" href="https://pypi.org/project/typing-extensions/">typing_extensions</a> package. Because the buffer protocol is currently accessible only in C, this PEP cannot be fully implemented in a pure-Python package like <code class="docutils literal notranslate">typing_extensions</code>. As a temporary workaround, an abstract base class <code class="docutils literal notranslate">typing_extensions.Buffer</code> will be provided for Python versions that do not have <code class="docutils literal notranslate">collections.abc.Buffer</code> available. After this PEP is implemented, inheriting from <code class="docutils literal notranslate">collections.abc.Buffer</code> will not be necessary to indicate that an object supports the buffer protocol. However, in older Python versions, it will be necessary to explicitly inherit from <code class="docutils literal notranslate">typing_extensions.Buffer</code> to indicate to type checkers that a class supports the buffer protocol, since objects supporting the buffer protocol will not have a <code class="docutils literal notranslate">__buffer__</code> method. It is expected that this will happen primarily in stub files, because buffer classes are necessarily implemented in C code, which cannot have types defined inline. For runtime uses, the <code class="docutils literal notranslate">ABC.register</code> API can be used to register buffer classes with <code class="docutils literal notranslate">typing_extensions.Buffer</code>. </section> <section id="no-special-meaning-for-bytes"> <h3><a class="toc-backref" href="#no-special-meaning-for-bytes" role="doc-backlink">No special meaning for <code class="docutils literal notranslate">bytes</code></a></h3> The special case stating that <code class="docutils literal notranslate">bytes</code> may be used as a shorthand for other <code class="docutils literal notranslate">ByteString</code> types will be removed from the <code class="docutils literal notranslate">typing</code> documentation. With <code class="docutils literal notranslate">collections.abc.Buffer</code> available as an alternative, there will be no good reason to allow <code class="docutils literal notranslate">bytes</code> as a shorthand. Type checkers currently implementing this behavior should deprecate and eventually remove it. </section> </section> <section id="backwards-compatibility"> <h2><a class="toc-backref" href="#backwards-compatibility" role="doc-backlink">Backwards Compatibility</a></h2> <section id="buffer-and-release-buffer-attributes"> <h3><a class="toc-backref" href="#buffer-and-release-buffer-attributes" role="doc-backlink"><code class="docutils literal notranslate">__buffer__</code> and <code class="docutils literal notranslate">__release_buffer__</code> attributes</a></h3> As the runtime changes in this PEP only add new functionality, there are few backwards compatibility concerns. However, code that uses a <code class="docutils literal notranslate">__buffer__</code> or <code class="docutils literal notranslate">__release_buffer__</code> attribute for other purposes may be affected. While all dunders are technically reserved for the language, it is still good practice to ensure that a new dunder does not interfere with too much existing code, especially widely used packages. A survey of publicly accessible code found: <ul class="simple"> <li>PyPy <a class="reference external" href="https://doc.pypy.org/en/latest/__pypy__-module.html#generally-available-functionality">supports</a> a <code class="docutils literal notranslate">__buffer__</code> method with compatible semantics to those proposed in this PEP. A PyPy core developer <a class="reference external" href="https://discuss.python.org/t/pep-688-making-the-buffer-protocol-accessible-in-python/15265/34">expressed his support</a> for this PEP.</li> <li>pyzmq <a class="reference external" href="https://github.com/zeromq/pyzmq/blob/fe18dc55516ef50d168fc02f8550a67ff5b5633d/zmq/backend/cffi/message.py#L190">implements</a> a PyPy-compatible <code class="docutils literal notranslate">__buffer__</code> method.</li> <li>mpi4py <a class="reference external" href="https://github.com/mpi4py/mpi4py/blob/453b87d0da37c5914b91afb511b188556dff2a9c/src/mpi4py/typing.py#L66">defines</a> a <code class="docutils literal notranslate">SupportsBuffer</code> protocol that would be equivalent to this PEP’s <code class="docutils literal notranslate">collections.abc.Buffer</code>.</li> <li>NumPy used to have an undocumented behavior where it would access a <code class="docutils literal notranslate">__buffer__</code> attribute (not method) to get an object’s buffer. This was <a class="reference external" href="https://github.com/numpy/numpy/pull/13049">removed</a> in 2019 for NumPy 1.17. The behavior would have last worked in NumPy 1.16, which only supported Python 3.7 and older. Python 3.7 will have reached its end of life by the time this PEP is expected to be implemented.</li> </ul> Thus, this PEP’s use of the <code class="docutils literal notranslate">__buffer__</code> method will improve interoperability with PyPy and not interfere with the current versions of any major Python packages. No publicly accessible code uses the name <code class="docutils literal notranslate">__release_buffer__</code>. </section> <section id="removal-of-the-bytes-special-case"> <h3><a class="toc-backref" href="#removal-of-the-bytes-special-case" role="doc-backlink">Removal of the <code class="docutils literal notranslate">bytes</code> special case</a></h3> Separately, the recommendation to remove the special behavior for <code class="docutils literal notranslate">bytes</code> in type checkers does have a backwards compatibility impact on their users. An <a class="reference external" href="https://github.com/python/mypy/pull/12661">experiment</a> with mypy shows that several major open source projects that use it for type checking will see new errors if the <code class="docutils literal notranslate">bytes</code> promotion is removed. Many of these errors can be fixed by improving the stubs in typeshed, as has already been done for the <a class="reference external" href="https://github.com/python/typeshed/pull/7631">builtins</a>, <a class="reference external" href="https://github.com/python/typeshed/pull/7677">binascii</a>, <a class="reference external" href="https://github.com/python/typeshed/pull/7678">pickle</a>, and <a class="reference external" href="https://github.com/python/typeshed/pull/7679">re</a> modules. A <a class="reference external" href="https://github.com/python/typeshed/issues/9006">review</a> of all usage of <code class="docutils literal notranslate">bytes</code> types in typeshed is in progress. Overall, the change improves type safety and makes the type system more consistent, so we believe the migration cost is worth it. </section> </section> <section id="how-to-teach-this"> <h2><a class="toc-backref" href="#how-to-teach-this" role="doc-backlink">How to Teach This</a></h2> We will add notes pointing to <code class="docutils literal notranslate">collections.abc.Buffer</code> in appropriate places in the documentation, such as <a class="reference external" href="https://typing.readthedocs.io/en/latest/">typing.readthedocs.io</a> and the <a class="reference external" href="https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html">mypy cheat sheet</a>. Type checkers may provide additional pointers in their error messages. For example, when they encounter a buffer object being passed to a function that is annotated to only accept <code class="docutils literal notranslate">bytes</code>, the error message could include a note suggesting the use of <code class="docutils literal notranslate">collections.abc.Buffer</code> instead. </section> <section id="reference-implementation"> <h2><a class="toc-backref" href="#reference-implementation" role="doc-backlink">Reference Implementation</a></h2> An implementation of this PEP is <a class="reference external" href="https://github.com/python/cpython/compare/main...JelleZijlstra:pep688v2?expand=1">available</a> in the author’s fork. </section> <section id="rejected-ideas"> <h2><a class="toc-backref" href="#rejected-ideas" role="doc-backlink">Rejected Ideas</a></h2> <section id="types-buffer"> <h3><a class="toc-backref" href="#types-buffer" role="doc-backlink"><code class="docutils literal notranslate">types.Buffer</code></a></h3> An earlier version of this PEP proposed adding a new <code class="docutils literal notranslate">types.Buffer</code> type with an <code class="docutils literal notranslate">__instancecheck__</code> implemented in C so that <code class="docutils literal notranslate">isinstance()</code> checks can be used to check whether a type implements the buffer protocol. This avoids the complexity of exposing the full buffer protocol to Python code, while still allowing the type system to check for the buffer protocol. However, that approach does not compose well with the rest of the type system, because <code class="docutils literal notranslate">types.Buffer</code> would be a nominal type, not a structural one. For example, there would be no way to represent “an object that supports both the buffer protocol and <code class="docutils literal notranslate">__len__</code>”. With the current proposal, <code class="docutils literal notranslate">__buffer__</code> is like any other special method, so a <code class="docutils literal notranslate">Protocol</code> can be defined combining it with another method. More generally, no other part of Python works like the proposed <code class="docutils literal notranslate">types.Buffer</code>. The current proposal is more consistent with the rest of the language, where C-level slots usually have corresponding Python-level special methods. </section> <section id="keep-bytearray-compatible-with-bytes"> <h3><a class="toc-backref" href="#keep-bytearray-compatible-with-bytes" role="doc-backlink">Keep <code class="docutils literal notranslate">bytearray</code> compatible with <code class="docutils literal notranslate">bytes</code></a></h3> It has been suggested to remove the special case where <code class="docutils literal notranslate">memoryview</code> is always compatible with <code class="docutils literal notranslate">bytes</code>, but keep it for <code class="docutils literal notranslate">bytearray</code>, because the two types have very similar interfaces. However, several standard library functions (e.g., <code class="docutils literal notranslate">re.compile</code>, <code class="docutils literal notranslate">socket.getaddrinfo</code>, and most functions accepting path-like arguments) accept <code class="docutils literal notranslate">bytes</code> but not <code class="docutils literal notranslate">bytearray</code>. In most codebases, <code class="docutils literal notranslate">bytearray</code> is also not a very common type. We prefer to have users spell out accepted types explicitly (or use <code class="docutils literal notranslate">Protocol</code> from <a class="pep reference internal" href="../pep-0544/" title="PEP 544 – Protocols: Structural subtyping (static duck typing)">PEP 544</a> if only a specific set of methods is required). This aspect of the proposal was <a class="reference external" href="https://mail.python.org/archives/list/typing-sig@python.org/thread/XH5ZK2MSZIQLL62PYZ6I5532SQKKVCBL/">specifically discussed</a> on the typing-sig mailing list, without any strong disagreement from the typing community. </section> <section id="distinguish-between-mutable-and-immutable-buffers"> <h3><a class="toc-backref" href="#distinguish-between-mutable-and-immutable-buffers" role="doc-backlink">Distinguish between mutable and immutable buffers</a></h3> The most frequently used distinction within buffer types is whether or not the buffer is mutable. Some functions accept only mutable buffers (e.g., <code class="docutils literal notranslate">bytearray</code>, some <code class="docutils literal notranslate">memoryview</code> objects), others accept all buffers. An earlier version of this PEP proposed using the presence of the <code class="docutils literal notranslate">bf_releasebuffer</code> slot to determine whether a buffer type is mutable. This rule holds for most standard library buffer types, but the relationship between mutability and the presence of this slot is not absolute. For example, <code class="docutils literal notranslate">numpy</code> arrays are mutable but do not have this slot. The current buffer protocol does not provide any way to reliably determine whether a buffer type represents a mutable or immutable buffer. Therefore, this PEP does not add type system support for this distinction. The question can be revisited in the future if the buffer protocol is enhanced to provide static introspection support. A <a class="reference external" href="https://discuss.python.org/t/introspection-and-mutable-xor-shared-semantics-for-pybuffer/20314">sketch</a> for such a mechanism exists. </section> </section> <section id="acknowledgments"> <h2><a class="toc-backref" href="#acknowledgments" role="doc-backlink">Acknowledgments</a></h2> Many people have provided useful feedback on drafts of this PEP. Petr Viktorin has been particularly helpful in improving my understanding of the subtleties of the buffer protocol. </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0688.rst">https://github.com/python/peps/blob/main/peps/pep-0688.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0688.rst">2024-10-17 12:49:39 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#rationale">Rationale</a><ul> <li><a class="reference internal" href="#current-options">Current options</a></li> <li><a class="reference internal" href="#kinds-of-buffers">Kinds of buffers</a></li> </ul> </li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#python-level-buffer-protocol">Python-level buffer protocol</a></li> <li><a class="reference internal" href="#inspect-bufferflags"><code class="docutils literal notranslate">inspect.BufferFlags</code></a></li> <li><a class="reference internal" href="#collections-abc-buffer"><code class="docutils literal notranslate">collections.abc.Buffer</code></a></li> <li><a class="reference internal" href="#example">Example</a></li> <li><a class="reference internal" href="#equivalent-for-older-python-versions">Equivalent for older Python versions</a></li> <li><a class="reference internal" href="#no-special-meaning-for-bytes">No special meaning for <code class="docutils literal notranslate">bytes</code></a></li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a><ul> <li><a class="reference internal" href="#buffer-and-release-buffer-attributes"><code class="docutils literal notranslate">__buffer__</code> and <code class="docutils literal notranslate">__release_buffer__</code> attributes</a></li> <li><a class="reference internal" href="#removal-of-the-bytes-special-case">Removal of the <code class="docutils literal notranslate">bytes</code> special case</a></li> </ul> </li> <li><a class="reference internal" href="#how-to-teach-this">How to Teach This</a></li> <li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a><ul> <li><a class="reference internal" href="#types-buffer"><code class="docutils literal notranslate">types.Buffer</code></a></li> <li><a class="reference internal" href="#keep-bytearray-compatible-with-bytes">Keep <code class="docutils literal notranslate">bytearray</code> compatible with <code class="docutils literal notranslate">bytes</code></a></li> <li><a class="reference internal" href="#distinguish-between-mutable-and-immutable-buffers">Distinguish between mutable and immutable buffers</a></li> </ul> </li> <li><a class="reference internal" href="#acknowledgments">Acknowledgments</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0688.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>