CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 757 – C API to import-export Python integers | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0757/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 757 – C API to import-export Python integers | peps.python.org'> <meta property="og:description" content="Add a new C API to import and export Python integers, int objects: especially PyLongWriter_Create() and PyLong_Export() functions."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0757/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="Add a new C API to import and export Python integers, int objects: especially PyLongWriter_Create() and PyLong_Export() functions."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 757</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 757 – C API to import-export Python integers</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Sergey B Kirpichev <skirpichev at gmail.com>, Victor Stinner <vstinner at python.org></dd> <dt class="field-even">Discussions-To:</dt> <dd class="field-even"><a class="reference external" href="https://discuss.python.org/t/63895">Discourse thread</a></dd> <dt class="field-odd">Status:</dt> <dd class="field-odd"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-even">Type:</dt> <dd class="field-even"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-odd">Created:</dt> <dd class="field-odd">13-Sep-2024</dd> <dt class="field-even">Python-Version:</dt> <dd class="field-even">3.14</dd> <dt class="field-odd">Post-History:</dt> <dd class="field-odd"><a class="reference external" href="https://discuss.python.org/t/63895" title="Discourse thread">14-Sep-2024</a></dd> <dt class="field-even">Resolution:</dt> <dd class="field-even"><a class="reference external" href="https://discuss.python.org/t/63895/79">08-Dec-2024</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#layout-api">Layout API</a></li> <li><a class="reference internal" href="#id1">Export API</a></li> <li><a class="reference internal" href="#import-api">Import API</a></li> </ul> </li> <li><a class="reference internal" href="#optimize-import-for-small-integers">Optimize import for small integers</a></li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#benchmarks">Benchmarks</a><ul> <li><a class="reference internal" href="#export-pylong-export-with-gmpy2">Export: <a class="reference internal" href="#c.PyLong_Export" title="PyLong_Export"><code class="xref c c-func docutils literal notranslate">PyLong_Export()</code></a> with gmpy2</a></li> <li><a class="reference internal" href="#import-pylongwriter-create-with-gmpy2">Import: <a class="reference internal" href="#c.PyLongWriter_Create" title="PyLongWriter_Create"><code class="xref c c-func docutils literal notranslate">PyLongWriter_Create()</code></a> with gmpy2</a></li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a></li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a><ul> <li><a class="reference internal" href="#support-arbitrary-layout">Support arbitrary layout</a></li> <li><a class="reference internal" href="#don-t-add-pylong-getnativelayout-function">Don’t add <a class="reference internal" href="#c.PyLong_GetNativeLayout" title="PyLong_GetNativeLayout"><code class="xref c c-func docutils literal notranslate">PyLong_GetNativeLayout()</code></a> function</a></li> <li><a class="reference internal" href="#provide-mpz-import-export-like-api-instead">Provide mpz_import/export-like API instead</a></li> <li><a class="reference internal" href="#drop-value-field-from-the-export-api">Drop <a class="reference internal" href="#c.PyLongExport.value" title="PyLongExport.value"><code class="xref c c-member docutils literal notranslate">value</code></a> field from the export API</a></li> </ul> </li> <li><a class="reference internal" href="#discussions">Discussions</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <div class="pep-banner canonical-doc sticky-banner admonition important"> Important This PEP is a historical document. The up-to-date, canonical documentation can now be found at the <a class="reference external" href="https://docs.python.org/dev/c-api/long.html#export-api">Export API</a> and the <a class="reference external" href="https://docs.python.org/dev/c-api/long.html#pylongwriter-api">PyLongWriter API</a>. × See <a class="pep reference internal" href="../pep-0001/" title="PEP 1 – PEP Purpose and Guidelines">PEP 1</a> for how to propose changes. </div> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> Add a new C API to import and export Python integers, <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">int</code></a> objects: especially <a class="reference internal" href="#c.PyLongWriter_Create" title="PyLongWriter_Create"><code class="xref c c-func docutils literal notranslate">PyLongWriter_Create()</code></a> and <a class="reference internal" href="#c.PyLong_Export" title="PyLong_Export"><code class="xref c c-func docutils literal notranslate">PyLong_Export()</code></a> functions. </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> Projects such as <a class="reference external" href="https://github.com/aleaxit/gmpy">gmpy2</a>, <a class="reference external" href="https://www.sagemath.org/">SAGE</a> and <a class="reference external" href="https://github.com/flintlib/python-flint">Python-FLINT</a> access directly Python “internals” (the <a class="reference external" href="https://docs.python.org/3/c-api/long.html#c.PyLongObject" title="(in Python v3.13)"><code class="xref c c-type docutils literal notranslate">PyLongObject</code></a> structure) or use an inefficient temporary format (hex strings for Python-FLINT) to import and export Python <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">int</code></a> objects. The Python <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">int</code></a> implementation changed in Python 3.12 to add a tag and “compact values”. In the 3.13 alpha 1 release, the private undocumented <code class="xref c c-func docutils literal notranslate">_PyLong_New()</code> function had been removed, but it is being used by these projects to import Python integers. The private function has been restored in 3.13 alpha 2. A public efficient abstraction is needed to interface Python with these projects without exposing implementation details. It would allow Python to change its internals without breaking these projects. For example, implementation for gmpy2 was changed recently for CPython 3.9 and for CPython 3.12. </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> <section id="layout-api"> <h3><a class="toc-backref" href="#layout-api" role="doc-backlink">Layout API</a></h3> Data needed by <a class="reference external" href="https://gmplib.org/">GMP</a>-like <a class="reference external" href="https://gmplib.org/manual/Integer-Import-and-Export#index-mpz_005fimport">import</a>-<a class="reference external" href="https://gmplib.org/manual/Integer-Import-and-Export#index-mpz_005fexport">export</a> functions. <dl class="c struct"> <dt class="sig sig-object c" id="c.PyLongLayout"> struct PyLongLayout </dt> <dd>Layout of an array of “digits” (“limbs” in the GMP terminology), used to represent absolute value for arbitrary precision integers.Use <a class="reference internal" href="#c.PyLong_GetNativeLayout" title="PyLong_GetNativeLayout"><code class="xref c c-func docutils literal notranslate">PyLong_GetNativeLayout()</code></a> to get the native layout of Python <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">int</code></a> objects, used internally for integers with “big enough” absolute value. See also <a class="reference external" href="https://docs.python.org/3/library/sys.html#sys.int_info" title="(in Python v3.13)"><code class="xref py py-data docutils literal notranslate">sys.int_info</code></a> which exposes similar information to Python. <dl class="c member"> <dt class="sig sig-object c" id="c.PyLongLayout.bits_per_digit"> uint8_t bits_per_digit </dt> <dd>Bits per digit. For example, a 15 bit digit means that bits 0-14 contain meaningful information.</dd></dl> <dl class="c member"> <dt class="sig sig-object c" id="c.PyLongLayout.digit_size"> uint8_t digit_size </dt> <dd>Digit size in bytes. For example, a 15 bit digit will require at least 2 bytes.</dd></dl> <dl class="c member"> <dt class="sig sig-object c" id="c.PyLongLayout.digits_order"> int8_t digits_order </dt> <dd>Digits order:<ul class="simple"> <li><code class="docutils literal notranslate">1</code> for most significant digit first</li> <li><code class="docutils literal notranslate">-1</code> for least significant digit first</li> </ul> </dd></dl> <dl class="c member"> <dt class="sig sig-object c" id="c.PyLongLayout.digit_endianness"> int8_t digit_endianness </dt> <dd>Digit endianness:<ul class="simple"> <li><code class="docutils literal notranslate">1</code> for most significant byte first (big endian)</li> <li><code class="docutils literal notranslate">-1</code> for least significant byte first (little endian)</li> </ul> </dd></dl> </dd></dl> <dl class="c function"> <dt class="sig sig-object c" id="c.PyLong_GetNativeLayout"> const <a class="reference internal" href="#c.PyLongLayout" title="PyLongLayout">PyLongLayout</a> *PyLong_GetNativeLayout(void) </dt> <dd>Get the native layout of Python <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">int</code></a> objects.See the <a class="reference internal" href="#c.PyLongLayout" title="PyLongLayout"><code class="xref c c-struct docutils literal notranslate">PyLongLayout</code></a> structure. The function must not be called before Python initialization nor after Python finalization. The returned layout is valid until Python is finalized. The layout is the same for all Python sub-interpreters and so it can be cached. </dd></dl> </section> <section id="id1"> <h3><a class="toc-backref" href="#id1" role="doc-backlink">Export API</a></h3> <dl class="c struct"> <dt class="sig sig-object c" id="c.PyLongExport"> struct PyLongExport </dt> <dd>Export of a Python <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">int</code></a> object.There are two cases: <ul class="simple"> <li>If <a class="reference internal" href="#c.PyLongExport.digits" title="digits"><code class="xref c c-member docutils literal notranslate">digits</code></a> is <code class="docutils literal notranslate">NULL</code>, only use the <a class="reference internal" href="#c.PyLongExport.value" title="value"><code class="xref c c-member docutils literal notranslate">value</code></a> member.</li> <li>If <a class="reference internal" href="#c.PyLongExport.digits" title="digits"><code class="xref c c-member docutils literal notranslate">digits</code></a> is not <code class="docutils literal notranslate">NULL</code>, use <a class="reference internal" href="#c.PyLongExport.negative" title="negative"><code class="xref c c-member docutils literal notranslate">negative</code></a>, <a class="reference internal" href="#c.PyLongExport.ndigits" title="ndigits"><code class="xref c c-member docutils literal notranslate">ndigits</code></a> and <a class="reference internal" href="#c.PyLongExport.digits" title="digits"><code class="xref c c-member docutils literal notranslate">digits</code></a> members.</li> </ul> <dl class="c member"> <dt class="sig sig-object c" id="c.PyLongExport.value"> int64_t value </dt> <dd>The native integer value of the exported <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">int</code></a> object. Only valid if <a class="reference internal" href="#c.PyLongExport.digits" title="digits"><code class="xref c c-member docutils literal notranslate">digits</code></a> is <code class="docutils literal notranslate">NULL</code>.</dd></dl> <dl class="c member"> <dt class="sig sig-object c" id="c.PyLongExport.negative"> uint8_t negative </dt> <dd>1 if the number is negative, 0 otherwise. Only valid if <a class="reference internal" href="#c.PyLongExport.digits" title="digits"><code class="xref c c-member docutils literal notranslate">digits</code></a> is not <code class="docutils literal notranslate">NULL</code>.</dd></dl> <dl class="c member"> <dt class="sig sig-object c" id="c.PyLongExport.ndigits"> <a class="reference external" href="https://docs.python.org/3/c-api/intro.html#c.Py_ssize_t" title="(in Python v3.13)">Py_ssize_t</a> ndigits </dt> <dd>Number of digits in <a class="reference internal" href="#c.PyLongExport.digits" title="digits"><code class="xref c c-member docutils literal notranslate">digits</code></a> array. Only valid if <a class="reference internal" href="#c.PyLongExport.digits" title="digits"><code class="xref c c-member docutils literal notranslate">digits</code></a> is not <code class="docutils literal notranslate">NULL</code>.</dd></dl> <dl class="c member"> <dt class="sig sig-object c" id="c.PyLongExport.digits"> const void *digits </dt> <dd>Read-only array of unsigned digits. Can be <code class="docutils literal notranslate">NULL</code>.</dd></dl> </dd></dl> If <a class="reference internal" href="#c.PyLongExport.digits" title="PyLongExport.digits"><code class="xref c c-member docutils literal notranslate">PyLongExport.digits</code></a> is not <code class="docutils literal notranslate">NULL</code>, a private field of the <a class="reference internal" href="#c.PyLongExport" title="PyLongExport"><code class="xref c c-struct docutils literal notranslate">PyLongExport</code></a> structure stores a strong reference to the Python <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">int</code></a> object to make sure that that structure remains valid until <a class="reference internal" href="#c.PyLong_FreeExport" title="PyLong_FreeExport"><code class="xref c c-func docutils literal notranslate">PyLong_FreeExport()</code></a> is called. <dl class="c function"> <dt class="sig sig-object c" id="c.PyLong_Export"> int PyLong_Export(<a class="reference external" href="https://docs.python.org/3/c-api/structures.html#c.PyObject" title="(in Python v3.13)">PyObject</a> *obj, <a class="reference internal" href="#c.PyLongExport" title="PyLongExport">PyLongExport</a> *export_long) </dt> <dd>Export a Python <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">int</code></a> object.export_long must point to a <a class="reference internal" href="#c.PyLongExport" title="PyLongExport"><code class="xref c c-struct docutils literal notranslate">PyLongExport</code></a> structure allocated by the caller. It must not be <code class="docutils literal notranslate">NULL</code>. On success, fill in *export_long and return 0. On error, set an exception and return -1. <a class="reference internal" href="#c.PyLong_FreeExport" title="PyLong_FreeExport"><code class="xref c c-func docutils literal notranslate">PyLong_FreeExport()</code></a> must be called when the export is no longer needed. CPython implementation detail: This function always succeeds if obj is a Python <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">int</code></a> object or a subclass. </dd></dl> On CPython 3.14, no memory copy is needed in <a class="reference internal" href="#c.PyLong_Export" title="PyLong_Export"><code class="xref c c-func docutils literal notranslate">PyLong_Export()</code></a>, it’s just a thin wrapper to expose Python <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">int</code></a> internal digits array. <dl class="c function"> <dt class="sig sig-object c" id="c.PyLong_FreeExport"> void PyLong_FreeExport(<a class="reference internal" href="#c.PyLongExport" title="PyLongExport">PyLongExport</a> *export_long) </dt> <dd>Release the export export_long created by <a class="reference internal" href="#c.PyLong_Export" title="PyLong_Export"><code class="xref c c-func docutils literal notranslate">PyLong_Export()</code></a>.CPython implementation detail: Calling <a class="reference internal" href="#c.PyLong_FreeExport" title="PyLong_FreeExport"><code class="xref c c-func docutils literal notranslate">PyLong_FreeExport()</code></a> is optional if export_long->digits is <code class="docutils literal notranslate">NULL</code>. </dd></dl> </section> <section id="import-api"> <h3><a class="toc-backref" href="#import-api" role="doc-backlink">Import API</a></h3> The <a class="reference internal" href="#c.PyLongWriter" title="PyLongWriter"><code class="xref c c-type docutils literal notranslate">PyLongWriter</code></a> API can be used to import an integer. <dl class="c struct"> <dt class="sig sig-object c" id="c.PyLongWriter"> struct PyLongWriter </dt> <dd>A Python <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">int</code></a> writer instance.The instance must be destroyed by <a class="reference internal" href="#c.PyLongWriter_Finish" title="PyLongWriter_Finish"><code class="xref c c-func docutils literal notranslate">PyLongWriter_Finish()</code></a> or <a class="reference internal" href="#c.PyLongWriter_Discard" title="PyLongWriter_Discard"><code class="xref c c-func docutils literal notranslate">PyLongWriter_Discard()</code></a>. </dd></dl> <dl class="c function"> <dt class="sig sig-object c" id="c.PyLongWriter_Create"> <a class="reference internal" href="#c.PyLongWriter" title="PyLongWriter">PyLongWriter</a> *PyLongWriter_Create(int negative, <a class="reference external" href="https://docs.python.org/3/c-api/intro.html#c.Py_ssize_t" title="(in Python v3.13)">Py_ssize_t</a> ndigits, void **digits) </dt> <dd>Create a <a class="reference internal" href="#c.PyLongWriter" title="PyLongWriter"><code class="xref c c-type docutils literal notranslate">PyLongWriter</code></a>.On success, allocate *digits and return a writer. On error, set an exception and return <code class="docutils literal notranslate">NULL</code>. negative is <code class="docutils literal notranslate">1</code> if the number is negative, or <code class="docutils literal notranslate">0</code> otherwise. ndigits is the number of digits in the digits array. It must be greater than 0. digits must not be NULL. After a successful call to this function, the caller should fill in the array of digits digits and then call <a class="reference internal" href="#c.PyLongWriter_Finish" title="PyLongWriter_Finish"><code class="xref c c-func docutils literal notranslate">PyLongWriter_Finish()</code></a> to get a Python <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">int</code></a>. The layout of digits is described by <a class="reference internal" href="#c.PyLong_GetNativeLayout" title="PyLong_GetNativeLayout"><code class="xref c c-func docutils literal notranslate">PyLong_GetNativeLayout()</code></a>. Digits must be in the range [<code class="docutils literal notranslate">0</code>; <code class="docutils literal notranslate">(1 << bits_per_digit) - 1</code>] (where the <a class="reference internal" href="#c.PyLongLayout.bits_per_digit" title="PyLongLayout.bits_per_digit"><code class="xref c c-struct docutils literal notranslate">bits_per_digit</code></a> is the number of bits per digit). Any unused most significant digits must be set to <code class="docutils literal notranslate">0</code>. Alternately, call <a class="reference internal" href="#c.PyLongWriter_Discard" title="PyLongWriter_Discard"><code class="xref c c-func docutils literal notranslate">PyLongWriter_Discard()</code></a> to destroy the writer instance without creating an <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">int</code></a> object. </dd></dl> On CPython 3.14, the <a class="reference internal" href="#c.PyLongWriter_Create" title="PyLongWriter_Create"><code class="xref c c-func docutils literal notranslate">PyLongWriter_Create()</code></a> implementation is a thin wrapper to the private <code class="xref c c-func docutils literal notranslate">_PyLong_New()</code> function. <dl class="c function"> <dt class="sig sig-object c" id="c.PyLongWriter_Finish"> <a class="reference external" href="https://docs.python.org/3/c-api/structures.html#c.PyObject" title="(in Python v3.13)">PyObject</a> *PyLongWriter_Finish(<a class="reference internal" href="#c.PyLongWriter" title="PyLongWriter">PyLongWriter</a> *writer) </dt> <dd>Finish a <a class="reference internal" href="#c.PyLongWriter" title="PyLongWriter"><code class="xref c c-type docutils literal notranslate">PyLongWriter</code></a> created by <a class="reference internal" href="#c.PyLongWriter_Create" title="PyLongWriter_Create"><code class="xref c c-func docutils literal notranslate">PyLongWriter_Create()</code></a>.On success, return a Python <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">int</code></a> object. On error, set an exception and return <code class="docutils literal notranslate">NULL</code>. The function takes care of normalizing the digits and converts the object to a compact integer if needed. The writer instance and the digits array are invalid after the call. </dd></dl> <dl class="c function"> <dt class="sig sig-object c" id="c.PyLongWriter_Discard"> void PyLongWriter_Discard(<a class="reference internal" href="#c.PyLongWriter" title="PyLongWriter">PyLongWriter</a> *writer) </dt> <dd>Discard a <a class="reference internal" href="#c.PyLongWriter" title="PyLongWriter"><code class="xref c c-type docutils literal notranslate">PyLongWriter</code></a> created by <a class="reference internal" href="#c.PyLongWriter_Create" title="PyLongWriter_Create"><code class="xref c c-func docutils literal notranslate">PyLongWriter_Create()</code></a>.writer must not be <code class="docutils literal notranslate">NULL</code>. The writer instance and the digits array are invalid after the call. </dd></dl> </section> </section> <section id="optimize-import-for-small-integers"> <h2><a class="toc-backref" href="#optimize-import-for-small-integers" role="doc-backlink">Optimize import for small integers</a></h2> Proposed import API is efficient for large integers. Compared to accessing directly Python internals, the proposed import API can have a significant performance overhead on small integers. For small integers of a few digits (for example, 1 or 2 digits), existing APIs can be used: <ul class="simple"> <li><a class="reference external" href="https://docs.python.org/3.14/c-api/long.html#c.PyLong_FromUInt64" title="(in Python v3.14)"><code class="xref c c-func docutils literal notranslate">PyLong_FromUInt64()</code></a>;</li> <li><a class="reference external" href="https://docs.python.org/3/c-api/long.html#c.PyLong_FromLong" title="(in Python v3.13)"><code class="xref c c-func docutils literal notranslate">PyLong_FromLong()</code></a>;</li> <li><a class="reference external" href="https://docs.python.org/3/c-api/long.html#c.PyLong_FromNativeBytes" title="(in Python v3.13)"><code class="xref c c-func docutils literal notranslate">PyLong_FromNativeBytes()</code></a>.</li> </ul> </section> <section id="implementation"> <h2><a class="toc-backref" href="#implementation" role="doc-backlink">Implementation</a></h2> <ul class="simple"> <li>CPython:<ul> <li><a class="reference external" href="https://github.com/python/cpython/pull/121339">https://github.com/python/cpython/pull/121339</a></li> <li><a class="reference external" href="https://github.com/vstinner/cpython/pull/5">https://github.com/vstinner/cpython/pull/5</a></li> </ul> </li> <li>gmpy:<ul> <li><a class="reference external" href="https://github.com/aleaxit/gmpy/pull/495">https://github.com/aleaxit/gmpy/pull/495</a></li> </ul> </li> </ul> </section> <section id="benchmarks"> <h2><a class="toc-backref" href="#benchmarks" role="doc-backlink">Benchmarks</a></h2> Code: <div class="highlight-c notranslate"><div class="highlight"><pre>/* Query parameters of Python’s internal representation of integers. */ const PyLongLayout *layout = PyLong_GetNativeLayout(); size_t int_digit_size = layout->digit_size; int int_digits_order = layout->digits_order; size_t int_bits_per_digit = layout->bits_per_digit; size_t int_nails = int_digit_size*8 - int_bits_per_digit; int int_endianness = layout->digit_endianness; </pre></div> </div> <section id="export-pylong-export-with-gmpy2"> <h3><a class="toc-backref" href="#export-pylong-export-with-gmpy2" role="doc-backlink">Export: <a class="reference internal" href="#c.PyLong_Export" title="PyLong_Export"><code class="xref c c-func docutils literal notranslate">PyLong_Export()</code></a> with gmpy2</a></h3> Code: <div class="highlight-c notranslate"><div class="highlight"><pre>static int mpz_set_PyLong(mpz_t z, PyObject *obj) { static PyLongExport long_export; if (PyLong_Export(obj, &long_export) < 0) { return -1; } if (long_export.digits) { mpz_import(z, long_export.ndigits, int_digits_order, int_digit_size, int_endianness, int_nails, long_export.digits); if (long_export.negative) { mpz_neg(z, z); } PyLong_FreeExport(&long_export); } else { const int64_t value = long_export.value; if (LONG_MIN <= value && value <= LONG_MAX) { mpz_set_si(z, value); } else { mpz_import(z, 1, -1, sizeof(int64_t), 0, 0, &value); if (value < 0) { mpz_t tmp; mpz_init(tmp); mpz_ui_pow_ui(tmp, 2, 64); mpz_sub(z, z, tmp); mpz_clear(tmp); } } } return 0; } </pre></div> </div> Reference code: <a class="reference external" href="https://github.com/aleaxit/gmpy/blob/9177648c23f5c507e46b81c1eb7d527c79c96f00/src/gmpy2_convert_gmp.c#L42-L69">mpz_set_PyLong() in the gmpy2 master for commit 9177648</a>. Benchmark: <div class="highlight-py notranslate"><div class="highlight"><pre>import pyperf from gmpy2 import mpz runner = pyperf.Runner() runner.bench_func('1<<7', mpz, 1 << 7) runner.bench_func('1<<38', mpz, 1 << 38) runner.bench_func('1<<300', mpz, 1 << 300) runner.bench_func('1<<3000', mpz, 1 << 3000) </pre></div> </div> Results on Linux Fedora 40 with CPU isolation, Python built in release mode: <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Benchmark</th> <th class="head">ref</th> <th class="head">pep757</th> </tr> </thead> <tbody> <tr class="row-even"><td>1<<7</td> <td>91.3 ns</td> <td>89.9 ns: 1.02x faster</td> </tr> <tr class="row-odd"><td>1<<38</td> <td>120 ns</td> <td>94.9 ns: 1.27x faster</td> </tr> <tr class="row-even"><td>1<<300</td> <td>196 ns</td> <td>203 ns: 1.04x slower</td> </tr> <tr class="row-odd"><td>1<<3000</td> <td>939 ns</td> <td>945 ns: 1.01x slower</td> </tr> <tr class="row-even"><td>Geometric mean</td> <td>(ref)</td> <td>1.05x faster</td> </tr> </tbody> </table> </section> <section id="import-pylongwriter-create-with-gmpy2"> <h3><a class="toc-backref" href="#import-pylongwriter-create-with-gmpy2" role="doc-backlink">Import: <a class="reference internal" href="#c.PyLongWriter_Create" title="PyLongWriter_Create"><code class="xref c c-func docutils literal notranslate">PyLongWriter_Create()</code></a> with gmpy2</a></h3> Code: <div class="highlight-c notranslate"><div class="highlight"><pre>static PyObject * GMPy_PyLong_From_MPZ(MPZ_Object *obj, CTXT_Object *context) { if (mpz_fits_slong_p(obj->z)) { return PyLong_FromLong(mpz_get_si(obj->z)); } size_t size = (mpz_sizeinbase(obj->z, 2) + int_bits_per_digit - 1) / int_bits_per_digit; void *digits; PyLongWriter *writer = PyLongWriter_Create(mpz_sgn(obj->z) < 0, size, &digits); if (writer == NULL) { return NULL; } mpz_export(digits, NULL, int_digits_order, int_digit_size, int_endianness, int_nails, obj->z); return PyLongWriter_Finish(writer); } </pre></div> </div> Reference code: <a class="reference external" href="https://github.com/aleaxit/gmpy/blob/9177648c23f5c507e46b81c1eb7d527c79c96f00/src/gmpy2_convert_gmp.c#L128-L156">GMPy_PyLong_From_MPZ() in the gmpy2 master for commit 9177648</a>. Benchmark: <div class="highlight-py notranslate"><div class="highlight"><pre>import pyperf from gmpy2 import mpz runner = pyperf.Runner() runner.bench_func('1<<7', int, mpz(1 << 7)) runner.bench_func('1<<38', int, mpz(1 << 38)) runner.bench_func('1<<300', int, mpz(1 << 300)) runner.bench_func('1<<3000', int, mpz(1 << 3000)) </pre></div> </div> Results on Linux Fedora 40 with CPU isolation, Python built in release mode: <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Benchmark</th> <th class="head">ref</th> <th class="head">pep757</th> </tr> </thead> <tbody> <tr class="row-even"><td>1<<7</td> <td>56.7 ns</td> <td>56.2 ns: 1.01x faster</td> </tr> <tr class="row-odd"><td>1<<300</td> <td>191 ns</td> <td>213 ns: 1.12x slower</td> </tr> <tr class="row-even"><td>Geometric mean</td> <td>(ref)</td> <td>1.03x slower</td> </tr> </tbody> </table> Benchmark hidden because not significant (2): 1<<38, 1<<3000. </section> </section> <section id="backwards-compatibility"> <h2><a class="toc-backref" href="#backwards-compatibility" role="doc-backlink">Backwards Compatibility</a></h2> There is no impact on the backward compatibility, only new APIs are added. </section> <section id="rejected-ideas"> <h2><a class="toc-backref" href="#rejected-ideas" role="doc-backlink">Rejected Ideas</a></h2> <section id="support-arbitrary-layout"> <h3><a class="toc-backref" href="#support-arbitrary-layout" role="doc-backlink">Support arbitrary layout</a></h3> It would be convenient to support arbitrary layout to import-export Python integers. For example, it was proposed to add a layout parameter to <a class="reference internal" href="#c.PyLongWriter_Create" title="PyLongWriter_Create"><code class="xref c c-func docutils literal notranslate">PyLongWriter_Create()</code></a> and a layout member to the <a class="reference internal" href="#c.PyLongExport" title="PyLongExport"><code class="xref c c-struct docutils literal notranslate">PyLongExport</code></a> structure. The problem is that it’s more complex to implement and not really needed. What’s strictly needed is only an API to import-export using the Python “native” layout. If later there are use cases for arbitrary layouts, new APIs can be added. </section> <section id="don-t-add-pylong-getnativelayout-function"> <h3><a class="toc-backref" href="#don-t-add-pylong-getnativelayout-function" role="doc-backlink">Don’t add <a class="reference internal" href="#c.PyLong_GetNativeLayout" title="PyLong_GetNativeLayout"><code class="xref c c-func docutils literal notranslate">PyLong_GetNativeLayout()</code></a> function</a></h3> Currently, most required information for <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">int</code></a> import/export is already available via <a class="reference external" href="https://docs.python.org/3/c-api/long.html#c.PyLong_GetInfo" title="(in Python v3.13)"><code class="xref c c-func docutils literal notranslate">PyLong_GetInfo()</code></a> (and <a class="reference external" href="https://docs.python.org/3/library/sys.html#sys.int_info" title="(in Python v3.13)"><code class="xref py py-data docutils literal notranslate">sys.int_info</code></a>). We also can add more (like order of digits), this interface doesn’t poses any constraints on future evolution of the <a class="reference external" href="https://docs.python.org/3/c-api/long.html#c.PyLongObject" title="(in Python v3.13)"><code class="xref c c-type docutils literal notranslate">PyLongObject</code></a>. The problem is that the <a class="reference external" href="https://docs.python.org/3/c-api/long.html#c.PyLong_GetInfo" title="(in Python v3.13)"><code class="xref c c-func docutils literal notranslate">PyLong_GetInfo()</code></a> returns a Python object, <a class="reference external" href="https://docs.python.org/3/glossary.html#term-named-tuple" title="(in Python v3.13)">named tuple</a>, not a convenient C structure and that might distract people from using it in favor e.g. of current semi-private macros like <code class="xref c c-macro docutils literal notranslate">PyLong_SHIFT</code> and <code class="xref c c-macro docutils literal notranslate">PyLong_BASE</code>. </section> <section id="provide-mpz-import-export-like-api-instead"> <h3><a class="toc-backref" href="#provide-mpz-import-export-like-api-instead" role="doc-backlink">Provide mpz_import/export-like API instead</a></h3> The other approach to import/export data from <a class="reference external" href="https://docs.python.org/3/library/functions.html#int" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">int</code></a> objects might be following: expect, that C extensions provide contiguous buffers that CPython then exports (or imports) the absolute value of an integer. API example: <div class="highlight-c notranslate"><div class="highlight"><pre>struct PyLongLayout { uint8_t bits_per_digit; uint8_t digit_size; int8_t digits_order; }; size_t PyLong_GetDigitsNeeded(PyLongObject *obj, PyLongLayout layout); int PyLong_Export(PyLongObject *obj, PyLongLayout layout, void *buffer); PyLongObject *PyLong_Import(PyLongLayout layout, void *buffer); </pre></div> </div> This might work for the GMP, as it has <code class="xref c c-func docutils literal notranslate">mpz_limbs_read()</code> and <code class="xref c c-func docutils literal notranslate">mpz_limbs_write()</code> functions, that can provide required access to internals of <code class="xref c c-struct docutils literal notranslate">mpz_t</code>. Other libraries may require using temporary buffers and then mpz_import/export-like functions on their side. The major drawback of this approach is that it’s much more complex on the CPython side (i.e. actual conversion between different layouts). For example, implementation of the <a class="reference external" href="https://docs.python.org/3/c-api/long.html#c.PyLong_FromNativeBytes" title="(in Python v3.13)"><code class="xref c c-func docutils literal notranslate">PyLong_FromNativeBytes()</code></a> and the <a class="reference external" href="https://docs.python.org/3/c-api/long.html#c.PyLong_AsNativeBytes" title="(in Python v3.13)"><code class="xref c c-func docutils literal notranslate">PyLong_AsNativeBytes()</code></a> (together provided restricted version of the required API) in the CPython took ~500 LOC (c.f. ~100 LOC in the current implementation). </section> <section id="drop-value-field-from-the-export-api"> <h3><a class="toc-backref" href="#drop-value-field-from-the-export-api" role="doc-backlink">Drop <a class="reference internal" href="#c.PyLongExport.value" title="PyLongExport.value"><code class="xref c c-member docutils literal notranslate">value</code></a> field from the export API</a></h3> With this suggestion, only one export type will exist (array of “digits”). If such view is not available for a given integer, it will be either emulated by export functions or the <a class="reference internal" href="#c.PyLong_Export" title="PyLong_Export"><code class="xref c c-func docutils literal notranslate">PyLong_Export()</code></a> will return an error. In both cases, it’s assumed that users will use other C-API functions to get “small enough” integers (i.e., that fits to some machine integer types), like the <a class="reference external" href="https://docs.python.org/3/c-api/long.html#c.PyLong_AsLongAndOverflow" title="(in Python v3.13)"><code class="xref c c-func docutils literal notranslate">PyLong_AsLongAndOverflow()</code></a>. The <a class="reference internal" href="#c.PyLong_Export" title="PyLong_Export"><code class="xref c c-func docutils literal notranslate">PyLong_Export()</code></a> will be inefficient (or just fail) in this case. An example: <div class="highlight-c notranslate"><div class="highlight"><pre>static int mpz_set_PyLong(mpz_t z, PyObject *obj) { int overflow; #if SIZEOF_LONG == 8 long value = PyLong_AsLongAndOverflow(obj, &overflow); #else /* Windows has 32-bit long, so use 64-bit long long instead */ long long value = PyLong_AsLongLongAndOverflow(obj, &overflow); #endif Py_BUILD_ASSERT(sizeof(value) == sizeof(int64_t)); if (!overflow) { if (LONG_MIN <= value && value <= LONG_MAX) { mpz_set_si(z, (long)value); } else { mpz_import(z, 1, -1, sizeof(int64_t), 0, 0, &value); if (value < 0) { mpz_t tmp; mpz_init(tmp); mpz_ui_pow_ui(tmp, 2, 64); mpz_sub(z, z, tmp); mpz_clear(tmp); } } } else { static PyLongExport long_export; if (PyLong_Export(obj, &long_export) < 0) { return -1; } mpz_import(z, long_export.ndigits, int_digits_order, int_digit_size, int_endianness, int_nails, long_export.digits); if (long_export.negative) { mpz_neg(z, z); } PyLong_FreeExport(&long_export); } return 0; } </pre></div> </div> This might look as a simplification from the API designer point of view, but will be less convenient for end users. They will have to follow Python development, benchmark different variants for exporting small integers (is that obvious why above case was chosen instead of <a class="reference external" href="https://docs.python.org/3.14/c-api/long.html#c.PyLong_AsInt64" title="(in Python v3.14)"><code class="xref c c-func docutils literal notranslate">PyLong_AsInt64()</code></a>?), maybe support different code paths for various CPython versions or across different Python implementations. </section> </section> <section id="discussions"> <h2><a class="toc-backref" href="#discussions" role="doc-backlink">Discussions</a></h2> <ul class="simple"> <li>Discourse: <a class="reference external" href="https://discuss.python.org/t/63895">PEP 757 – C API to import-export Python integers</a></li> <li><a class="reference external" href="https://github.com/capi-workgroup/decisions/issues/35">C API Working Group decision issue #35</a></li> <li><a class="reference external" href="https://github.com/python/cpython/pull/121339">Pull request #121339</a></li> <li><a class="reference external" href="https://github.com/python/cpython/issues/102471">Issue #102471</a>: The C-API for Python to C integer conversion is, to be frank, a mess.</li> <li><a class="reference external" href="https://github.com/capi-workgroup/decisions/issues/31">Add public function PyLong_GetDigits()</a></li> <li><a class="reference external" href="https://github.com/python/cpython/issues/111415">Consider restoring _PyLong_New() function as public</a></li> <li><a class="reference external" href="https://github.com/python/cpython/pull/108604">Pull request gh-106320</a>: Remove private _PyLong_New() function.</li> </ul> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0757.rst">https://github.com/python/peps/blob/main/peps/pep-0757.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0757.rst">2024-12-16 07:23:59 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#layout-api">Layout API</a></li> <li><a class="reference internal" href="#id1">Export API</a></li> <li><a class="reference internal" href="#import-api">Import API</a></li> </ul> </li> <li><a class="reference internal" href="#optimize-import-for-small-integers">Optimize import for small integers</a></li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#benchmarks">Benchmarks</a><ul> <li><a class="reference internal" href="#export-pylong-export-with-gmpy2">Export: <code class="xref c c-func docutils literal notranslate">PyLong_Export()</code> with gmpy2</a></li> <li><a class="reference internal" href="#import-pylongwriter-create-with-gmpy2">Import: <code class="xref c c-func docutils literal notranslate">PyLongWriter_Create()</code> with gmpy2</a></li> </ul> </li> <li><a class="reference internal" href="#backwards-compatibility">Backwards Compatibility</a></li> <li><a class="reference internal" href="#rejected-ideas">Rejected Ideas</a><ul> <li><a class="reference internal" href="#support-arbitrary-layout">Support arbitrary layout</a></li> <li><a class="reference internal" href="#don-t-add-pylong-getnativelayout-function">Don’t add <code class="xref c c-func docutils literal notranslate">PyLong_GetNativeLayout()</code> function</a></li> <li><a class="reference internal" href="#provide-mpz-import-export-like-api-instead">Provide mpz_import/export-like API instead</a></li> <li><a class="reference internal" href="#drop-value-field-from-the-export-api">Drop <code class="xref c c-member docutils literal notranslate">value</code> field from the export API</a></li> </ul> </li> <li><a class="reference internal" href="#discussions">Discussions</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0757.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>