Ideas for forward-compatible and fast extension libraries in Python 3.12 - Core Development

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <title>Ideas for forward-compatible and fast extension libraries in Python 3.12 - Core Development - Discussions on Python.org</title> <meta name="description" content="Dear CPython team, I am currently in the process of making the nanobind C++ ↔ Python binding library ready for an eventual switch to Py_LIMITED_API. nanobind is essentially a faster minified rewrite of pybind11, hence t&hellip;"> <meta name="generator" content="Discourse 3.5.0.beta3-dev - https://github.com/discourse/discourse version 601fecde064cc11ee2704f06c2d5083f9876961d"> <link rel="icon" type="image/png" href="https://us1.discourse-cdn.com/flex002/uploads/python1/optimized/1X/9997f0605d56c4bfecd63594f52f42cdafd6b06a_2_32x32.png"> <link rel="apple-touch-icon" type="image/png" href="https://us1.discourse-cdn.com/flex002/uploads/python1/optimized/1X/4c06143de7870c35963b818b15b395092a434991_2_180x180.png"> <meta name="theme-color" media="(prefers-color-scheme: light)" content="#fff"> <meta name="theme-color" media="(prefers-color-scheme: dark)" content="#111111"> <meta name="color-scheme" content="light dark"> <meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0, viewport-fit=cover"> <link rel="canonical" href="https://discuss.python.org/t/ideas-for-forward-compatible-and-fast-extension-libraries-in-python-3-12/15993" /> <link rel="search" type="application/opensearchdescription+xml" href="https://discuss.python.org/opensearch.xml" title="Discussions on Python.org Search"> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/color_definitions_base__2_eb0d5a741b57128d23682ca696b208cf73a29eac.css?__ws=discuss.python.org" media="(prefers-color-scheme: light)" rel="stylesheet" class="light-scheme"/><link href="https://sea2.discourse-cdn.com/flex002/stylesheets/color_definitions_dark_1_2_ffed44bdc0eac3c3a8411e2f3d376c0b759dfc7c.css?__ws=discuss.python.org" media="(prefers-color-scheme: dark)" rel="stylesheet" class="dark-scheme" data-scheme-id="1"/> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/desktop_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="desktop" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/automation_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="automation" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/checklist_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="checklist" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-ai_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-ai" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-akismet_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-akismet" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-chat-integration_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-chat-integration" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-data-explorer_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-data-explorer" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-details_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-details" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-lazy-videos_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-lazy-videos" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-local-dates_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-local-dates" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-math_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-math" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-narrative-bot_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-narrative-bot" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-policy_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-policy" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-presence_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-presence" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-solved_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-solved" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-templates_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-templates" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-topic-voting_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-topic-voting" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-user-notes_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-user-notes" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/footnote_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="footnote" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/hosted-site_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="hosted-site" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/poll_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="poll" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/spoiler-alert_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="spoiler-alert" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-ai_desktop_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-ai_desktop" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/discourse-topic-voting_desktop_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="discourse-topic-voting_desktop" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/poll_desktop_bd4b4d0d14c768b45ff80e502c1742b15583bde2.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="poll_desktop" /> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/desktop_theme_4_a35caff525d253104d0dd192d1f6b7ef18411fca.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="desktop_theme" data-theme-id="4" data-theme-name="unformatted code detector"/> <link href="https://sea2.discourse-cdn.com/flex002/stylesheets/desktop_theme_2_4fe55dd06ca87507c4cdc0b57e78c3e576b4c52c.css?__ws=discuss.python.org" media="all" rel="stylesheet" data-target="desktop_theme" data-theme-id="2" data-theme-name="light"/> <link rel="alternate nofollow" type="application/rss+xml" title="RSS feed of 'Ideas for forward-compatible and fast extension libraries in Python 3.12'" href="https://discuss.python.org/t/ideas-for-forward-compatible-and-fast-extension-libraries-in-python-3-12/15993.rss" /> <meta property="og:site_name" content="Discussions on Python.org" /> <meta property="og:type" content="website" /> <meta name="twitter:card" content="summary" /> <meta name="twitter:image" content="https://us1.discourse-cdn.com/flex002/uploads/python1/original/1X/f93ff97c4f381b5e8add5a0c163b4ded29f20ed7.png" /> <meta property="og:image" content="https://us1.discourse-cdn.com/flex002/uploads/python1/original/1X/f93ff97c4f381b5e8add5a0c163b4ded29f20ed7.png" /> <meta property="og:url" content="https://discuss.python.org/t/ideas-for-forward-compatible-and-fast-extension-libraries-in-python-3-12/15993" /> <meta name="twitter:url" content="https://discuss.python.org/t/ideas-for-forward-compatible-and-fast-extension-libraries-in-python-3-12/15993" /> <meta property="og:title" content="Ideas for forward-compatible and fast extension libraries in Python 3.12" /> <meta name="twitter:title" content="Ideas for forward-compatible and fast extension libraries in Python 3.12" /> <meta property="og:description" content="Dear CPython team, I am currently in the process of making the nanobind C++ ↔ Python binding library ready for an eventual switch to Py_LIMITED_API. nanobind is essentially a faster minified rewrite of pybind11, hence this can also be interpreted as a proof-of-concept for eventually making similar changes in pybind11. This is to … be resilient to changes in internal data structures that are likely going to be pursued by the faster CPython team. dramatically simplify the wheel distributio..." /> <meta name="twitter:description" content="Dear CPython team, I am currently in the process of making the nanobind C++ ↔ Python binding library ready for an eventual switch to Py_LIMITED_API. nanobind is essentially a faster minified rewrite of pybind11, hence this can also be interpreted as a proof-of-concept for eventually making similar changes in pybind11. This is to … be resilient to changes in internal data structures that are likely going to be pursued by the faster CPython team. dramatically simplify the wheel distributio..." /> <meta property="og:article:section" content="Core Development" /> <meta property="og:article:section:color" content="25AAE2" /> <meta name="twitter:label1" value="Reading time" /> <meta name="twitter:data1" value="12 mins 🕑" /> <meta name="twitter:label2" value="Likes" /> <meta name="twitter:data2" value="40 ❤" /> <meta property="article:published_time" content="2022-05-25T11:55:02+00:00" /> <meta property="og:ignore_canonical" content="true" /> <link rel="next" href="/t/ideas-for-forward-compatible-and-fast-extension-libraries-in-python-3-12/15993?page=2"> </head> <body class="crawler browser-update"> <header> <a href="/"> Discussions on Python.org </a> </header> <div id="main-outlet" class="wrap" role="main"> <div id="topic-title"> <h1> <a href="/t/ideas-for-forward-compatible-and-fast-extension-libraries-in-python-3-12/15993">Ideas for forward-compatible and fast extension libraries in Python 3.12</a> </h1> <div class="topic-category" itemscope itemtype="http://schema.org/BreadcrumbList"> <span itemprop="itemListElement" itemscope itemtype="http://schema.org/ListItem"> <a href="/c/core-dev/23" class="badge-wrapper bullet" itemprop="item"> <span class='badge-category-bg' style='background-color: #25AAE2'></span> <span class='badge-category clear-badge'> <span class='category-name' itemprop='name'>Core Development</span> </span> </a> <meta itemprop="position" content="1" /> </span> </div> </div> <div itemscope itemtype='http://schema.org/DiscussionForumPosting'> <meta itemprop='headline' content='Ideas for forward-compatible and fast extension libraries in Python 3.12'> <link itemprop='url' href='https://discuss.python.org/t/ideas-for-forward-compatible-and-fast-extension-libraries-in-python-3-12/15993'> <meta itemprop='datePublished' content='2022-05-25T11:55:02Z'> <meta itemprop='articleSection' content='Core Development'> <meta itemprop='keywords' content=''> <div itemprop='publisher' itemscope itemtype="http://schema.org/Organization"> <meta itemprop='name' content='Python Software Foundation'> <div itemprop='logo' itemscope itemtype="http://schema.org/ImageObject"> <meta itemprop='url' content='https://us1.discourse-cdn.com/flex002/uploads/python1/original/1X/c7591c98caf3b31d4d9c6f322f41ed9d80a50800.png'> </div> </div> <div id='post_1' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/wjakob'><span itemprop='name'>wjakob</span></a> (Wenzel Jakob) </span> <link itemprop="mainEntityOfPage" href="https://discuss.python.org/t/ideas-for-forward-compatible-and-fast-extension-libraries-in-python-3-12/15993"> <span class="crawler-post-infos"> <time datetime='2022-05-25T11:55:02Z' class='post-time'> May 25, 2022, 11:55am </time> <meta itemprop='dateModified' content='2022-05-25T13:39:21Z'> <span itemprop='position'>1</span> </span> </div> <div class='post' itemprop='text'> <p>Dear CPython team,</p> <p>I am currently in the process of making the <a href="https://github.com/wjakob/nanobind" rel="noopener nofollow ugc">nanobind</a> C++ ↔ Python binding library ready for an eventual switch to <code>Py_LIMITED_API</code>. nanobind is essentially a faster minified rewrite of pybind11, hence this can also be interpreted as a proof-of-concept for eventually making similar changes in pybind11.</p> <p>This is to …</p> <ol> <li> <p>be resilient to changes in internal data structures that are likely going to be pursued by the faster CPython team.</p> </li> <li> <p>dramatically simplify the wheel distribution situation. The effect of this won’t be felt for some years due to the latency between merging something into CPython <code>main</code> and this feature actually being present on the user’s side. Still, the eventual possibility of a shipping a single wheel per platform would be magnificent and makes this a worthy pursuit.</p> </li> </ol> <p>Of course, switching to a limited API would be a shame if this means that the extension library runs slowly. I would like to start a discussion about changes in CPython that would make it possible to have a forward-compatible ABI <em>and</em> retain performance.</p> <p>Hence, here is a “pie in the sky” wish list for Python 3.12, in order of perceived importance, with some commentary.</p> <ol> <li> <p>A way to create types with a custom metaclass using the <code>PyType_FromSpec</code> - style interface. A PR for a proposed new function <code>PyType_FromMetaclass</code> is here: <a href="https://github.com/python/cpython/pull/93012" class="inline-onebox" rel="noopener nofollow ugc">gh-60074: add new stable API function PyType_FromMetaclass by wjakob · Pull Request #93012 · python/cpython · GitHub</a></p> <p><strong>Why needed</strong>: this allows the binding tool to dynamically create a type without messing with opaque <code>PyTypeObject</code> internals. The ability to specify a metaclass is important because the binding tool may, e.g., need to intercept type-related operations like subclassing to update its internal data structures.</p> </li> <li> <p>A way for this function to extend the <code>__basicsize__</code> of types (including <code>PyTypeObject</code> itself). Petr Viktorin had a proposal that would legitimize this and provide a safe API for obtaining a pointer to the added storage region: <a href="https://mail.python.org/archives/list/capi-sig@python.org/thread/SIP3VP7JU4OBWP62KBOYGOYCVIOTXEFH/" class="inline-onebox" rel="noopener nofollow ugc">Mailman 3 C-API for extending opaque types - capi-sig - python.org</a></p> <p><strong>Why needed</strong>: this will allow the binding tool to store its type data structures directly in the Python type object, which significantly reduces pointer chasing and improves performance. Without this, any type-related binding code will require indirections through a hash-table (to map <code>PyTypeObject *</code> to an internal data structure), which is just not a good idea (imagine if every call to <code>Py_TYPE(..)</code> performed a dictionary lookup…)</p> </li> <li> <p>A way for custom callables to receive PEP 590 vector calls. <code>PyType_FromSpec</code> allows creating types with a <code>__vectorcalloffset__</code> – nice! Is this usage allowed as part of <code>Py_LIMITED_API</code>? The documentation is not super-clear on this. There are certainly some obvious omissions… For example, a type providing <code>__vectorcalloffset__</code> should also specify a <code>Py_tp_call</code> slot with the compatibility dispatcher <code>PyVectorcall_Call</code>, but this function is not part of the limited API. Similarly, decoding a vector call requires some constants/inline functions that aren’t included: <code>PyVectorcall_NARGS</code>, potentially also <code>PY_VECTORCALL_ARGUMENTS_OFFSET</code>.</p> <p><strong>Why needed</strong>: The most performance-critical function in a binding tool like nanobind or pybind11 is the function that receives a Python function call and then figures out what to do with it (it has to decode positional/keyword arguments, identify the right overload, perform implicit type conversions, handle exceptions, etc.). Binding frameworks normally implement custom callable objects that co-locate information needed to dispatch the function calls, which is why the traditional <code>PyMethodDef</code>-style interface is insufficient.</p> <p>This code is very “tuple/dictionary-heavy” when implemented using the old <code>tp_call</code>-style interface, and that mixes badly with <code>Py_LIMITED_API</code> especially because the tuple memory layout is opaque. The vector call API is a huge improvement here because it get rid of most tuples <em>and</em> also the dictionaries and lots of CPython API calls related to them. Being able to efficiently receive vector calls is (IMO) much more important to issuing vector calls, but I have a wish list entry about the latter as well <img src="https://emoji.discourse-cdn.com/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> – see point number 6.</p> </li> <li> <p>Add a limited API function that provides access to a <code>PyObject**</code> representation of a sequence type, but in a future-proof way. Concrete example code:</p> <pre><code class="lang-cpp">PyObject *const *PySequence_Items(PyObject *seq, PyObject **owner_out, Py_ssize_t *size_out) { if (PyTuple_Check(seq)) { *size_out = PyTuple_GET_SIZE(seq); *owner_out = Py_NewRef(seq); return ((PyTupleObject *) seq)->ob_item; } else if (PyList_Check(seq)) { *size_out = PyList_GET_SIZE(seq); *owner_out = Py_NewRef(seq); return ((PyListObject *) seq)->ob_item; } else { PyObject *temp = PySequence_List(seq); if (!temp) return NULL; *owner_out = temp; *size_out = PyList_GET_SIZE(temp); return ((PyListObject *) temp)->ob_item; } } </code></pre> <p><strong>Why needed</strong>: type conversion become more expensive when lists/tuples are opaque, especially when working with very large lists/vectors. The idea of this change is to provide a read-only interface akin to <code>PySequence_Fast_ITEMS</code> which provides direct access to a <code>PyObject **</code>-style representation that the binding code can quickly iterate over.</p> <p><code>PySequence_Fast_ITEMS</code> is not part of the limited API because it relies on implementation details (for example, the fact that lists/tuples actually have a contiguous <code>PyObject **</code>-based representation). The proposed function also uses that in this specific implementation to be fast, but it could also be implemented in numerous different ways differently. The important thing is that it returns an <strong>owner</strong> object of an unspecified type (via the <code>owner_out</code> parameter). Its only purpose is to hold a strong reference that the callee must continue to hold until it is done accessing the return value. For example, the function could <code>malloc</code> an array of <code>PyObject *</code> pointers and then set owner to a <code>PyCapsule</code>, whose destructor will <code>free</code> the memory region. Note that it is illegal to write to the returned array of pointers, since this may not update the original sequence.</p> <p>This feature is related to a discussion by Victor Stinner here: <a href="https://mail.python.org/archives/list/python-dev@python.org/thread/632CV42376SWVYAZTHG4ROOV2HRHOVZ7/" class="inline-onebox" rel="noopener nofollow ugc">Mailman 3 (PEP 620) C API for efficient loop iterating on a sequence of PyObject** or other C types - Python-Dev - python.org</a></p> </li> <li> <p>In addition to the functions in point 3, I propose adding <code>PyObject_Vectorcall</code> and <code>PyObject_VectorcallMethod</code> to the limited API.</p> <p><strong>Why needed</strong>: For binding code that implements callbacks (e.g. a C++ GUI library like Qt that wants to dispatch a button push to a Python handler), it is also useful to be able to <em>issue</em> vector calls. This will significantly cut down tuple/dictionary construction which, again, is costly when operations like <code>PyTuple_SET_ITEM</code> cannot be inlined and dictionaries potentially have to be created to pass keywords. This is (IMO) less important than the receiving end mentioned in point 3, but making both of those changes would be nicely symmetric.</p> </li> <li> <p>Clarify the status of keyword arguments in PEP 590 vector calls. It would be helpful if the spec would explicitly say that keyword arguments names must always be interned strings.</p> <p><strong>Why needed</strong>: to resolve keyword arguments, the binding library would need to perform many calls to the slow <code>PyUnicode_Compare</code> function. Python internally often uses a pattern where it first tries a pointer equality check and then falls back to <code>_PyUnicode_EQ</code> (not part of the limited API). It would IMO be much easier to simply establish a clear rule that all extension libraries must abide by.</p> </li> </ol> <p>Anyways, this was a long thread — thank you if you read all the way to the end. It would be wonderful if some of these features could be included in Python 3.12 so that it could provide a foundation for extension modules that are both forward-compatible <em>and</em> fast.</p> <p>Some of these ideas might be controversial, and the purpose of this post was to stimulate some discussion. I don’t think that all of these features are critical, but especially the first 2-3 offer a big bang for the buck.</p> <p>Thanks,<br> Wenzel</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="11" /> <span class='post-likes'>11 Likes</span> </div> <div class='crawler-linkback-list' itemscope itemtype='http://schema.org/ItemList'> <div itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <a itemprop='url' href="https://discuss.python.org/t/pep-697-limited-c-api-for-extending-opaque-types/19743">PEP 697 – Limited C API for Extending Opaque Types</a> <meta itemprop='position' content='5'> </div> </div> </div> <div id='post_2' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/encukou'><span itemprop='name'>encukou</span></a> (Petr Viktorin) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2022-05-25T14:14:38Z' class='post-time'> May 25, 2022, 2:14pm </time> <meta itemprop='dateModified' content='2022-05-25T14:14:38Z'> <span itemprop='position'>2</span> </span> </div> <div class='post' itemprop='text'> <ol> <li>Let’s do that!</li> <li>I’m in favor, obviously</li> <li>Yes, it seems vectorcall is indeed ready for getting a promise of decades of support.<br> I Perhaps we can even make <code>PyType-FromSpec</code> set <code>tp_call</code> to <code>PyVectorcall_Call</code> by default.</li> <li> </li> </ol> <ul> <li>Like <code>PySequence_Fast_ITEMS</code>, there are issues with a list being mutated while its underlying array is in use. The behavior is predictable in CPython, but might be different in other implementations. <ul> <li>Is this actually needed for lists?</li> </ul> </li> <li>Is this still needed with vectorcall?</li> </ul> <ol start="5"> <li>Sounds reasonable.</li> <li>I don’t think we can tighten the spec now, since vectorcall is used in the wild.<br> Can we expose something closer to <code>_PyUnicode_EQ</code> instead?</li> </ol> <p>cc <a class="mention" href="/u/ctismer">@ctismer</a> — would this help PySide?</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="3" /> <span class='post-likes'>3 Likes</span> </div> </div> <div id='post_3' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/wjakob'><span itemprop='name'>wjakob</span></a> (Wenzel Jakob) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2022-05-25T14:40:26Z' class='post-time'> May 25, 2022, 2:40pm </time> <meta itemprop='dateModified' content='2022-05-25T14:47:11Z'> <span itemprop='position'>3</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote group-committers" data-username="encukou" data-post="2" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/encukou/48/2461_2.png" class="avatar"> Petr Viktorin:</div> <blockquote> <p>Perhaps we can even make <code>PyType-FromSpec</code> set <code>tp_call</code> to <code>PyVectorcall_Call</code> by default.</p> </blockquote> </aside> <p>That would make sense.</p> <aside class="quote group-committers" data-username="encukou" data-post="2" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/encukou/48/2461_2.png" class="avatar"> Petr Viktorin:</div> <blockquote> <ul> <li>Like <code>PySequence_Fast_ITEMS</code> , there are issues with a list being mutated while its underlying array is in use. The behavior is predictable in CPython, but might be different in other implementations. <ul> <li>Is this actually needed for lists?</li> </ul> </li> <li>Is this still needed with vectorcall?</li> </ul> </blockquote> </aside> <p>This is relevant even when the vector call feature is implemented. For example, consider taking a Python <code>List[float]</code> and using it to call a C++ function taking a <code>std::vector<float></code> – the suggested interface would remove one layer of indirection to access the list elements during this conversion. I don’t see how lists/tuples would make a fundamental difference. If anything, it’s more important for lists because they tend to be larger.</p> <p>This kind of function would only be used temporarily for read-only access. Perhaps the contract of the function could be set up so that it’s really only valid in a purely read-only context: no guarantees are made if the original list is mutated, and the user of this function can also not update the returned pointer array in any way.</p> <aside class="quote group-committers" data-username="encukou" data-post="2" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/encukou/48/2461_2.png" class="avatar"> Petr Viktorin:</div> <blockquote> <ol start="5"> <li>I don’t think we can tighten the spec now, since vectorcall is used in the wild.<br> Can we expose something closer to <code>_PyUnicode_EQ</code> instead?</li> </ol> </blockquote> </aside> <p>It should be possible to ensure in CPython itself, and there probably aren’t many extensions that are using vectorcalls yet. Elevating vector calls to a stable ABI without addressing this point would IMO be an important missed optimization opportunity.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_4' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/encukou'><span itemprop='name'>encukou</span></a> (Petr Viktorin) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2022-05-25T15:18:19Z' class='post-time'> May 25, 2022, 3:18pm </time> <meta itemprop='dateModified' content='2022-05-25T15:39:13Z'> <span itemprop='position'>4</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote no-group" data-username="wjakob" data-post="3" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/wjakob/48/7265_2.png" class="avatar"> Wenzel Jakob:</div> <blockquote> <p>or example, consider taking a Python <code>List[float]</code> and using it to call a C++ function taking a <code>std::vector<float></code> – the suggested interface would remove one layer of indirection to access the list elements during this conversion.</p> </blockquote> </aside> <p>To call a user-defined function, you already need to copy the list and to incref each element (since the function can mutate the list and thus drop the references you’re borrowing). Right?<br> Would it work to have a function to copy the pointers to a pre-allocated <code>PyObject **</code> buffer, increfing each? And a function for mass decref?</p> <aside class="quote no-group" data-username="wjakob" data-post="3" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/wjakob/48/7265_2.png" class="avatar"> Wenzel Jakob:</div> <blockquote> <p>there probably aren’t many extensions that are using vectorcalls yet</p> </blockquote> </aside> <p>But if there are <em>any</em>, they’ll start failing in mysterious ways. I really don’t think we can do that change.<br> We can <em>encourage</em> interned strings, but the checks/fallbacks needs to stay.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_5' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/ctismer'><span itemprop='name'>ctismer</span></a> (Christian Tismer-Sperling) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2022-05-25T15:26:43Z' class='post-time'> May 25, 2022, 3:26pm </time> <meta itemprop='dateModified' content='2022-05-25T15:26:43Z'> <span itemprop='position'>5</span> </span> </div> <div class='post' itemprop='text'> <p>Hi Petr,</p> <p>I have been struggling quite much until I could tweak the CPython implementation<br> to play well with PySide and the Limited API, of course. In CPython, it was possible to do<br> some nasty tricks like changing the meta type after type creation.</p> <p>For porting PySide to PyPy, a heavy undertaking in fact, I had to do much more drastic things:</p> <ul> <li> <p>write a special typefactory.cpp which creates types in a PyPy compatible way: The base function<br> for type creation is</p> <p>LIBSHIBOKEN_API PyTypeObject *SbkType_FromSpec_BMDWB(PyType_Spec *spec,<br> PyObject *bases,<br> PyTypeObject *meta,<br> int dictoffset,<br> int weaklistoffset,<br> PyBufferProcs *bufferprocs);<br> and of course a few convenience versions of that with more speaking name</p> </li> <li> <p>instead of using opaque type extensions, I had to replace that by lookup dicts.<br> A bit slower, but works nicely.</p> </li> </ul> <p>The SbkType_FromSpec_BMDWB function had a lot of CPython code replication, because<br> I needed a version that does not call PyType_Ready too early. Instead, all preparations<br> must be done, and <em>then</em> PyType_Ready must be called.</p> <p>Summary:</p> <ul> <li>Vectorcall would be nice to have. It will probably give some speed.</li> <li>A function like the above SbkType_FromSpec_BMDWB function would be <em>very</em> helpful,<br> because we could get rid of those ugly patches.</li> <li>A general type extension framework would be quite helpful, but not crucial.</li> </ul> <p>And vectorcall is not interesting for PyPy at all. The acceleration of PySide code is great.</p> <p>Cheers – Chris</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_6' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/guido'><span itemprop='name'>guido</span></a> (Guido van Rossum) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2022-05-25T15:33:06Z' class='post-time'> May 25, 2022, 3:33pm </time> <meta itemprop='dateModified' content='2022-05-25T15:33:06Z'> <span itemprop='position'>6</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote no-group" data-username="wjakob" data-post="3" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/wjakob/48/7265_2.png" class="avatar"> Wenzel Jakob:</div> <blockquote> <p>This is relevant even when the vector call feature is implemented. For example, consider taking a Python <code>List[float]</code> and using it to call a C++ function taking a <code>std::vector<float></code> – the suggested interface would remove one layer of indirection to access the list elements during this conversion. I don’t see how lists/tuples would make a fundamental difference. If anything, it’s more important for lists because they tend to be larger.</p> </blockquote> </aside> <p>The problem is that we cannot guarantee that the list isn’t mutated during the time that you are accessing the items this way (even an innocent <code>DECREF</code> call could run Python code that could manipulate the list, causing you to read garbage).</p> <p>Since tuples are immutable they don’t have this problem, and the temporary copy you make for other types is also fine (since you are the only code with a reference to it). I think this would be okay if you took out the special case for lists. Would that work? Or do you expect to routinely work with lists that are so long that copying them would be prohibitive?</p> <p>If that’s the case I’m not sure what to suggest. Making the caller promise on the Zen of Python that they won’t make any Python API calls, not even one little DECREF, while they access that array of object pointers is asking for trouble in my opinion – because <em>usually</em>, you can get away with it just fine (it doesn’t crash every time you violate the rule), so people get lax.</p> <p>I could imagine that list objects would grow a new flag bit indicating that they are temporarily in read-only mode. All list-mutating operations should raise an error if this bit is set. Your API would then have to have an explicit “end access” operation, rather than just a DECREF of the owner object.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_7' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/encukou'><span itemprop='name'>encukou</span></a> (Petr Viktorin) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2022-05-25T15:38:30Z' class='post-time'> May 25, 2022, 3:38pm </time> <meta itemprop='dateModified' content='2022-05-25T15:38:30Z'> <span itemprop='position'>7</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote no-group" data-username="ctismer" data-post="5" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/ctismer/48/2705_2.png" class="avatar"> Christian Tismer-Sperling:</div> <blockquote> <p>int dictoffset,<br> int weaklistoffset,</p> </blockquote> </aside> <p>These can be set <a href="https://docs.python.org/3.11/c-api/structures.html#pymemberdef-offsets">using PyMemberDef</a>.</p> <aside class="quote no-group" data-username="ctismer" data-post="5" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/ctismer/48/2705_2.png" class="avatar"> Christian Tismer-Sperling:</div> <blockquote> <p>PyBufferProcs *bufferprocs);</p> </blockquote> </aside> <p>Those are coming in 3.11!</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_8' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/wjakob'><span itemprop='name'>wjakob</span></a> (Wenzel Jakob) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2022-05-25T15:39:52Z' class='post-time'> May 25, 2022, 3:39pm </time> <meta itemprop='dateModified' content='2022-05-25T15:39:52Z'> <span itemprop='position'>8</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote group-committers" data-username="guido" data-post="6" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/guido/48/21_2.png" class="avatar"> Guido van Rossum:</div> <blockquote> <p>Since tuples are immutable they don’t have this problem, and the temporary copy you make for other types is also fine (since you are the only code with a reference to it). I think this would be okay if you took out the special case for lists. Would that work? Or do you expect to routinely work with lists that are so long that copying them would be prohibitive?</p> </blockquote> </aside> <p>It sounds like a copy is unavoidable in some cases. I do like the idea about tuples – that would at least give a copy-free option for one important case.</p> <p>If the input type is not a tuple, it could be converted into one temporarily using <code>PySequence_Tuple</code> that is returned as the owner of the <code>PyObject**</code> pointer. Performing that copy + mass incref within CPython should be more efficient than getting out <code>PyObject *</code> pointers individually using the limited API.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_9' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/encukou'><span itemprop='name'>encukou</span></a> (Petr Viktorin) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2022-05-25T15:53:36Z' class='post-time'> May 25, 2022, 3:53pm </time> <meta itemprop='dateModified' content='2022-05-25T15:53:36Z'> <span itemprop='position'>9</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote no-group" data-username="wjakob" data-post="8" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/wjakob/48/7265_2.png" class="avatar"> Wenzel Jakob:</div> <blockquote> <p>I do like the idea about tuples – that would at least give a copy-free option for one important case.</p> </blockquote> </aside> <p>This is still making assumptions about memory layout: consider an implementation with tagged pointers and specialized small-int-only tuples which don’t store a <code>PyObject*</code> array at all.</p> <p>My question might have been buried:</p> <aside class="quote group-committers" data-username="encukou" data-post="4" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/encukou/48/2461_2.png" class="avatar"> Petr Viktorin:</div> <blockquote> <p>Would it work to have a function to copy the pointers to a pre-allocated <code>PyObject **</code> buffer, increfing each? And a function for mass decref?</p> </blockquote> </aside> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_10' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/guido'><span itemprop='name'>guido</span></a> (Guido van Rossum) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2022-05-25T15:55:05Z' class='post-time'> May 25, 2022, 3:55pm </time> <meta itemprop='dateModified' content='2022-05-25T15:55:05Z'> <span itemprop='position'>10</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote no-group" data-username="wjakob" data-post="8" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/wjakob/48/7265_2.png" class="avatar"> Wenzel Jakob:</div> <blockquote> <p>If the input type is not a tuple, it could be converted into one temporarily using <code>PySequence_Tuple</code> that is returned as the owner of the <code>PyObject**</code> pointer. Performing that copy + mass incref within CPython should be more efficient than getting out <code>PyObject *</code> pointers individually using the limited API.</p> </blockquote> </aside> <p>Yes, that sounds like a good idea. So now perhaps all we need is an API that takes a tuple and returns a pointer to its array of items, as part of the stable ABI. The user can call <code>PySequence_Tuple</code> prior to making that call if desired.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_11' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/wjakob'><span itemprop='name'>wjakob</span></a> (Wenzel Jakob) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2022-05-25T18:45:09Z' class='post-time'> May 25, 2022, 6:45pm </time> <meta itemprop='dateModified' content='2022-05-25T18:51:01Z'> <span itemprop='position'>11</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote group-committers" data-username="guido" data-post="10" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/guido/48/21_2.png" class="avatar"> Guido van Rossum:</div> <blockquote> <p>Yes, that sounds like a good idea. So now perhaps all we need is an API that takes a tuple and returns a pointer to its array of items, as part of the stable ABI. The user can call <code>PySequence_Tuple</code> prior to making that call if desired.</p> </blockquote> </aside> <p>As <a class="mention" href="/u/encukou">@encukou</a>’s mentioned, a hypothetical Python implementation might not represent tuples internally as <code>PyObject**</code> (tagged pointers, etc.), and so a hypothetical function</p> <p><code>PyObject ** PyTuple_Data(PyObject *)</code></p> <p>specific to tuples might not be sufficiently future-proof solution.</p> <aside class="quote group-committers" data-username="encukou" data-post="4" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/encukou/48/2461_2.png" class="avatar"> Petr Viktorin:</div> <blockquote> <p>Would it work to have a function to copy the pointers to a pre-allocated <code>PyObject **</code> buffer, increfing each? And a function for mass decref?</p> </blockquote> </aside> <p>Yes, that is not a bad solution, but I think we can do better. If the underlying implementation has tuples stored as contiguous <code>PyObject*</code> pointers, then this copy is superfluous, and it would be nice to avoid it. In a more exotic implementation, the function could still <code>malloc</code> a dedicated output buffer, mass-<code>Py_INCREF</code> pointers, etc.</p> <p>So my suggestion would be a slightly reduced version of my previous proposal that would be compatible with both of these requirements:</p> <pre data-code-wrap="cpp"><code class="lang-cpp"> /** PySequence_Items() Take a sequence 'seq' and return a pointer to an `PyObject *` array representing its contents. The returned object array is immutable: it may neither be changed by the caller, nor will it reflect future changes performed to 'seq'. The 'owner_out' parameter is used to return a Python object owning returned memory region. It may or may not equal 'seq', and no assumptions should be made about its type. Its only purpose is lifetime management -- in particular, the caller should `Py_DECREF(..)` this object once it no longer needs access to the object array returned by this function. The 'size_out' parameter returns the size of the returned sequence. */ PyObject *const *PySequence_Items(PyObject *seq, PyObject **owner_out, Py_ssize_t *size_out) { if (PyTuple_Check(seq)) { *size_out = PyTuple_GET_SIZE(seq); *owner_out = Py_NewRef(seq); return ((PyTupleObject *) seq)->ob_item; } else { PyObject *temp = PySequence_Tuple(seq); if (!temp) { *owner_out = NULL; *size_out = -1; return NULL; } *owner_out = temp; *size_out = PyTuple_GET_SIZE(temp); return ((PyTupleObject *) temp)->ob_item; } } </code></pre> <p>A more exotic python implementation simply would not have the special case for tuples. It would always allocate new memory for a pointer array, mass-incref, and then return a Python capsule or similar via <code>owner_out</code>, which performs a mass-decref and releases the memory upon expiration of the capsule.</p> <p>The caller would use this function as follows</p> <pre data-code-wrap="cpp"><code class="lang-cpp">PyObject *seq = ...; PyObject *owner; Py_ssize_t size; PyObject **items = Py_Sequence_Items(seq, &owner, &size); if (!items) { .. error handling .. } for (Py_ssize_t i = 0; i < size; ++i) { .. something involving items[i] .. } Py_DECREF(owner); </code></pre> <p>What do you think?</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_12' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/encukou'><span itemprop='name'>encukou</span></a> (Petr Viktorin) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2022-05-26T10:16:12Z' class='post-time'> May 26, 2022, 10:16am </time> <meta itemprop='dateModified' content='2022-05-26T10:16:12Z'> <span itemprop='position'>12</span> </span> </div> <div class='post' itemprop='text'> <p>One possible downside I see is that copying a Python list to a <code>std::vector<PyObject*></code> would involve <em>two</em> copies – one for the temporary tuple, and another to the vector. (For those not familiar with <code>std::vector</code>: it’s a resizable array much like a CPython list).<br> I don’t know if that’s a use case worth optimizing for.<br> OTOH, I also don’t know if zero-copy for tuples is worth it, since vectorcall might replace most uses of large tuples.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> <div class='crawler-linkback-list' itemscope itemtype='http://schema.org/ItemList'> <div itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <a itemprop='url' href="https://discuss.python.org/t/lets-get-rid-of-the-stable-abi-but-keep-the-limited-api/18458/19">Let's get rid of the stable ABI, but keep the limited API</a> <meta itemprop='position' content='1'> </div> </div> </div> <div id='post_13' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/markshannon'><span itemprop='name'>markshannon</span></a> (Mark Shannon) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2022-06-16T17:26:30Z' class='post-time'> June 16, 2022, 5:26pm </time> <meta itemprop='dateModified' content='2022-06-16T17:26:30Z'> <span itemprop='position'>13</span> </span> </div> <div class='post' itemprop='text'> <p>In reply to the numbered points:</p> <p>1,2 as Petr said.</p> <ol start="3"> <li> <p>Just use the vectorcall protocol, already <img src="https://emoji.discourse-cdn.com/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br> All the macros and function pointer typedefs should be part of the limited API; they are specified in PEP 590 as final.<br> Feel free to copy and paste them if necessary, they aren’t going to change.</p> </li> <li> <p>Not going to happen, sorry. Apart from not being safe, it is likely to be an obstacle to future improvements.</p> </li> <li> <p>See 3. The protocol works both ways. If we add the functions to the limited API (which I have no problem with), then you’ll only be able to use them in 3.12. The underlying protocol is good for 3.8 onward.</p> </li> <li> <p>I don’t think that we should force the strings to be interned, as that is a breaking change.<br> I’d be happy documenting that performance is likely to be better if they are interned, though.</p> </li> </ol> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_14' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/encukou'><span itemprop='name'>encukou</span></a> (Petr Viktorin) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2022-06-17T11:37:56Z' class='post-time'> June 17, 2022, 11:37am </time> <meta itemprop='dateModified' content='2022-06-17T11:37:56Z'> <span itemprop='position'>14</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote group-committers" data-username="markshannon" data-post="13" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/markshannon/48/72_2.png" class="avatar"> Mark Shannon:</div> <blockquote> <p>All the macros and function pointer typedefs should be part of the limited API; they are specified in PEP 590 as final.</p> </blockquote> </aside> <p>PEP 590 <a href="https://peps.python.org/pep-0590/#stable-abi">says</a> they’re not part of the stable ABI (and so, the limited API).<br> If we add them we need to make sure calls stay consistent when <code>__call__</code> on a class from Python code. The discussion is in <a href="https://github.com/python/cpython/issues/93274">gh-93274</a>.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_15' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/steve-s'><span itemprop='name'>steve-s</span></a> (Stepan Sindelar) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2022-07-15T12:58:28Z' class='post-time'> July 15, 2022, 12:58pm </time> <meta itemprop='dateModified' content='2022-07-15T13:00:30Z'> <span itemprop='position'>15</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote group-committers" data-username="markshannon" data-post="13" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/markshannon/48/72_2.png" class="avatar"> Mark Shannon:</div> <blockquote> <p>Not going to happen, sorry. Apart from not being safe, it is likely to be an obstacle to future improvements.</p> </blockquote> </aside> <p>Could you expand on this? I think that the point that doing an actual API call to get each element one by one is going to cause performance issues may be valid. Is this comment about that idea in general, or about the details?</p> <p>I would like to better understand your concerns, because HPy is considering providing similar APIs.</p> <p>Note that the API is not meant to force the implementation to use some specific storage strategy, the array of <code>PyObject*</code> is just a way to exchange the data between the caller and the Python engine.</p> <p>I think that providing a buffered access would be better to avoid materialization of some optimized compact representation of otherwise large list/tuple – the idea is to provide something like <code>PySequence_Items(PyObject *seq, PyObject **buffer, Py_SSize_t start, Py_SSize_t size)</code> and the user would iterate the sequence in two loops, outer loop fetching a buffer of some smallish size like 256 elements, inner loop would iterate over that. In such way, the inner loop would be free of the API calls to fetch elements, i.e., can be better optimized by the C compiler. This is API that R provides to iterate over its vectors. The issue is, however, that what useful thing can one do with opaque <code>PyObject*</code> other than do some other API calls. <a class="mention" href="/u/wjakob">@wjakob</a> do you have any use-case in mind? In R the vectors are typed, so you’re getting out, e.g., a buffer of plain C integers.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_16' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/encukou'><span itemprop='name'>encukou</span></a> (Petr Viktorin) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2022-08-08T13:33:13Z' class='post-time'> August 8, 2022, 1:33pm </time> <meta itemprop='dateModified' content='2022-08-08T13:33:13Z'> <span itemprop='position'>16</span> </span> </div> <div class='post' itemprop='text'> <ol> <li><span class="chcklst-box checked fa fa-check-square-o fa-fw"></span> <code>PyType_FromMetaclass</code> is in 3.12</li> <li><span class="chcklst-box fa fa-square-o fa-fw"></span> <strong>Extending opaque types</strong> is next up on my TODO list</li> <li><span class="chcklst-box checked fa fa-check-square-o fa-fw"></span> <strong>Receiving vectorcall</strong> <a href="https://github.com/python/cpython/issues/93274">is now in 3.12</a></li> <li><span class="chcklst-box fa fa-square-o fa-fw"></span> <strong>sequences as arrays</strong> is currently low on my TODO list, but I’m still leaning toward adding API where you pass in a pre-allocated buffer, like <ul> <li><code>PySequence_IntoArray(PyObject* seq, PyObject** array, size)</code></li> <li><code>Py_DecrefArray(PyObject** array, size)</code></li> </ul> </li> <li><span class="chcklst-box fa fa-square-o fa-fw"></span> <strong>Calling vectorcall</strong> – I’ll get to it eventually, but if anyone wants to push it further go ahead.</li> <li><span class="chcklst-box fa fa-square-o fa-fw"></span> <strong>interned strings</strong> are not required, but using them is likely to speed things up in- and outside CPython core. Adding docs & helpers would be nice (maybe expose an <code>eq</code> function optimized for interned strings?)</li> </ol> <hr> <aside class="quote no-group" data-username="steve-s" data-post="15" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/steve-s/48/8086_2.png" class="avatar"> Stepan Sindelar:</div> <blockquote> <p>This is API that R provides to iterate over its vectors.</p> </blockquote> </aside> <p>Python sequences are un-specialized <code>PyObject*</code> arrays. I think R’s API would be more suitable for NumPy arrays, or Python <a href="https://docs.python.org/3/library/array.html">array</a> if we wanted it to be more heavy-duty (which we don’t, IMO), or other specialized types.<br> IOW, specialized list-of-int sounds like a reasonable optimization, but exposing it in API doesn’t sound worthwhile.</p> <p>If avoiding materialization is a concern, you ideally want an iterator. Or, for <em>N</em>-D arrays, NumPy again.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="2" /> <span class='post-likes'>2 Likes</span> </div> </div> <div id='post_17' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/ronaldoussoren'><span itemprop='name'>ronaldoussoren</span></a> (Ronald Oussoren) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2022-08-09T09:42:22Z' class='post-time'> August 9, 2022, 9:42am </time> <meta itemprop='dateModified' content='2022-08-09T09:42:22Z'> <span itemprop='position'>17</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote group-committers" data-username="encukou" data-post="16" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/encukou/48/2461_2.png" class="avatar"> Petr Viktorin:</div> <blockquote> <p><strong>sequences as arrays</strong> is currently low on my TODO list, but I’m still leaning toward adding API where you pass in a pre-allocated buffer, like</p> <ul> <li><code>PySequence_IntoArray(PyObject* seq, PyObject** array, size)</code></li> <li><code>Py_DecrefArray(PyObject** array, size)</code></li> </ul> </blockquote> </aside> <p>Objective-C/Cocoa has an <code>NSFastEnumeration</code> protocol that provides a middle ground between the current protocol and your proposal: the caller provides a buffer and an iteration state and calls that API in a loop to avoid having to copy the entire contents into a temporary buffer. Basically <code>PyDict_Next</code>, but with more than one item at a time.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_18' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/encukou'><span itemprop='name'>encukou</span></a> (Petr Viktorin) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2022-08-09T09:49:57Z' class='post-time'> August 9, 2022, 9:49am </time> <meta itemprop='dateModified' content='2022-08-09T09:49:57Z'> <span itemprop='position'>18</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote group-committers" data-username="ronaldoussoren" data-post="17" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/ronaldoussoren/48/61_2.png" class="avatar"> Ronald Oussoren:</div> <blockquote> <p>Objective-C/Cocoa has an <code>NSFastEnumeration</code> protocol that provides a middle ground between the current protocol and your proposal: the caller provides a buffer and an iteration state and calls that API in a loop to avoid having to copy the entire contents into a temporary buffer. Basically <code>PyDict_Next</code>, but with more than one item at a time.</p> </blockquote> </aside> <p>Out of curiosity, what happens there if the sequence changes while you’re iterating?</p> <p>But the bigger question is <em>why</em>. What’s the use case for external API that avoids <em>full</em> materialization of a generic sequence?<br> For the Limited API specifically, it doesn’t sound like a good fit.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_19' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/ronaldoussoren'><span itemprop='name'>ronaldoussoren</span></a> (Ronald Oussoren) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2022-08-09T11:38:57Z' class='post-time'> August 9, 2022, 11:38am </time> <meta itemprop='dateModified' content='2022-08-09T11:38:57Z'> <span itemprop='position'>19</span> </span> </div> <div class='post' itemprop='text'> <p>To be honest I don’t know what happens when the sequence changes during iteration. The Cocoa protocol does not have a way to signal errors, and in Cocoa exceptions are generally not used other than to signal programming errors.</p> <p>A use case for avoiding full materialisation is that this might use a lot of memory. E.g. when iterating over all elements in a sequence of a couple of million entries. In the grand scheme of things that’s still not a lot of memory, but using a smaller buffer might allow using a stack-based buffer and hence avoid dynamic allocations.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_20' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" rel='nofollow' href='https://discuss.python.org/u/steve.dower'><span itemprop='name'>steve.dower</span></a> (Steve Dower) </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2022-08-09T12:57:07Z' class='post-time'> August 9, 2022, 12:57pm </time> <meta itemprop='dateModified' content='2022-08-09T12:57:07Z'> <span itemprop='position'>20</span> </span> </div> <div class='post' itemprop='text'> <aside class="quote group-committers" data-username="ronaldoussoren" data-post="17" data-topic="15993"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.python.org/ronaldoussoren/48/61_2.png" class="avatar"> Ronald Oussoren:</div> <blockquote> <p>Objective-C/Cocoa has an <code>NSFastEnumeration</code> protocol that provides a middle ground between the current protocol and your proposal: the caller provides a buffer and an iteration state and calls that API in a loop to avoid having to copy the entire contents into a temporary buffer. Basically <code>PyDict_Next</code>, but with more than one item at a time.</p> </blockquote> </aside> <p>Windows also uses this pattern fairly frequently (<a href="https://docs.microsoft.com/en-us/windows/win32/api/objidl/nf-objidl-ienumstring-next">example</a>), mostly because many implementations might have long roundtrip times (e.g. there’s a “find files” one somewhere that might involve network calls), so it lets the caller decide how much latency they care about.</p> <p>A “PyIter_GetNextFew” API would probably fit the bill, and should certainly be useful. If we wanted to specialise by taking a (single) ParseArgs-style format character to extract raw values as well, that could be nicely efficient for a lot of common cases.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> </div> <div role='navigation' itemscope itemtype='http://schema.org/SiteNavigationElement' class="topic-body crawler-post"> <span itemprop='name'><b><a rel="next" itemprop="url" href="/t/ideas-for-forward-compatible-and-fast-extension-libraries-in-python-3-12/15993?page=2">next page →</a></b></span> </div> </div> <footer class="container wrap"> <nav class='crawler-nav'> <ul> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/' itemprop="url">Home </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/categories' itemprop="url">Categories </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/guidelines' itemprop="url">Guidelines </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/tos' itemprop="url">Terms of Service </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/privacy' itemprop="url">Privacy Policy </a> </span> </li> </ul> </nav> <p class='powered-by-link'>Powered by <a href="https://www.discourse.org">Discourse</a>, best viewed with JavaScript enabled</p> </footer> <div class="buorg"><div>Unfortunately, <a href="https://www.discourse.org/faq/#browser">your browser is unsupported</a>. Please <a href="https://browsehappy.com">switch to a supported browser</a> to view rich content, log in and reply.</div></div> </body> </html>

CINXE.COM

Ideas for forward-compatible and fast extension libraries in Python 3.12 - Core Development - Discussions on Python.org