CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 267 – Optimized Access to Module Namespaces | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0267/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 267 – Optimized Access to Module Namespaces | peps.python.org'> <meta property="og:description" content="This PEP proposes a new implementation of global module namespaces and the builtin namespace that speeds name resolution. The implementation would use an array of object pointers for most operations in these namespaces. The compiler would assign indic..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0267/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This PEP proposes a new implementation of global module namespaces and the builtin namespace that speeds name resolution. The implementation would use an array of object pointers for most operations in these namespaces. The compiler would assign indic..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 267</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 267 – Optimized Access to Module Namespaces</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Jeremy Hylton <jeremy at alum.mit.edu></dd> <dt class="field-even">Status:</dt> <dd class="field-even"><abbr title="Inactive draft that may be taken up again at a later time">Deferred</abbr></dd> <dt class="field-odd">Type:</dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created:</dt> <dd class="field-even">23-May-2001</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">2.2</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even"></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#deferral">Deferral</a></li> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#introduction">Introduction</a></li> <li><a class="reference internal" href="#dlict-design">DLict design</a></li> <li><a class="reference internal" href="#compiler-issues">Compiler issues</a></li> <li><a class="reference internal" href="#runtime-model">Runtime model</a></li> <li><a class="reference internal" href="#backwards-compatibility">Backwards compatibility</a></li> <li><a class="reference internal" href="#related-peps">Related PEPs</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="deferral"> <h2><a class="toc-backref" href="#deferral" role="doc-backlink">Deferral</a></h2> While this PEP is a nice idea, no-one has yet emerged to do the work of hashing out the differences between this PEP, <a class="pep reference internal" href="../pep-0266/" title="PEP 266 – Optimizing Global Variable/Attribute Access">PEP 266</a> and <a class="pep reference internal" href="../pep-0280/" title="PEP 280 – Optimizing access to globals">PEP 280</a>. Hence, it is being deferred. </section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> This PEP proposes a new implementation of global module namespaces and the builtin namespace that speeds name resolution. The implementation would use an array of object pointers for most operations in these namespaces. The compiler would assign indices for global variables and module attributes at compile time. The current implementation represents these namespaces as dictionaries. A global name incurs a dictionary lookup each time it is used; a builtin name incurs two dictionary lookups, a failed lookup in the global namespace and a second lookup in the builtin namespace. This implementation should speed Python code that uses module-level functions and variables. It should also eliminate awkward coding styles that have evolved to speed access to these names. The implementation is complicated because the global and builtin namespaces can be modified dynamically in ways that are impossible for the compiler to detect. (Example: A module’s namespace is modified by a script after the module is imported.) As a result, the implementation must maintain several auxiliary data structures to preserve these dynamic features. </section> <section id="introduction"> <h2><a class="toc-backref" href="#introduction" role="doc-backlink">Introduction</a></h2> This PEP proposes a new implementation of attribute access for module objects that optimizes access to module variables known at compile time. The module will store these variables in an array and provide an interface to lookup attributes using array offsets. For globals, builtins, and attributes of imported modules, the compiler will generate code that uses the array offsets for fast access. [describe the key parts of the design: dlict, compiler support, stupid name trick workarounds, optimization of other module’s globals] The implementation will preserve existing semantics for module namespaces, including the ability to modify module namespaces at runtime in ways that affect the visibility of builtin names. </section> <section id="dlict-design"> <h2><a class="toc-backref" href="#dlict-design" role="doc-backlink">DLict design</a></h2> The namespaces are implemented using a data structure that has sometimes gone under the name <code class="docutils literal notranslate">dlict</code>. It is a dictionary that has numbered slots for some dictionary entries. The type must be implemented in C to achieve acceptable performance. The new type-class unification work should make this fairly easy. The <code class="docutils literal notranslate">DLict</code> will presumably be a subclass of dictionary with an alternate storage module for some keys. A Python implementation is included here to illustrate the basic design: <div class="highlight-default notranslate"><div class="highlight"><pre>"""A dictionary-list hybrid""" import types class DLict: def __init__(self, names): assert isinstance(names, types.DictType) self.names = {} self.list = [None] * size self.empty = [1] * size self.dict = {} self.size = 0 def __getitem__(self, name): i = self.names.get(name) if i is None: return self.dict[name] if self.empty[i] is not None: raise KeyError, name return self.list[i] def __setitem__(self, name, val): i = self.names.get(name) if i is None: self.dict[name] = val else: self.empty[i] = None self.list[i] = val self.size += 1 def __delitem__(self, name): i = self.names.get(name) if i is None: del self.dict[name] else: if self.empty[i] is not None: raise KeyError, name self.empty[i] = 1 self.list[i] = None self.size -= 1 def keys(self): if self.dict: return self.names.keys() + self.dict.keys() else: return self.names.keys() def values(self): if self.dict: return self.names.values() + self.dict.values() else: return self.names.values() def items(self): if self.dict: return self.names.items() else: return self.names.items() + self.dict.items() def __len__(self): return self.size + len(self.dict) def __cmp__(self, dlict): c = cmp(self.names, dlict.names) if c != 0: return c c = cmp(self.size, dlict.size) if c != 0: return c for i in range(len(self.names)): c = cmp(self.empty[i], dlict.empty[i]) if c != 0: return c if self.empty[i] is None: c = cmp(self.list[i], dlict.empty[i]) if c != 0: return c return cmp(self.dict, dlict.dict) def clear(self): self.dict.clear() for i in range(len(self.names)): if self.empty[i] is None: self.empty[i] = 1 self.list[i] = None def update(self): pass def load(self, index): """dlict-special method to support indexed access""" if self.empty[index] is None: return self.list[index] else: raise KeyError, index # XXX might want reverse mapping def store(self, index, val): """dlict-special method to support indexed access""" self.empty[index] = None self.list[index] = val def delete(self, index): """dlict-special method to support indexed access""" self.empty[index] = 1 self.list[index] = None </pre></div> </div> </section> <section id="compiler-issues"> <h2><a class="toc-backref" href="#compiler-issues" role="doc-backlink">Compiler issues</a></h2> The compiler currently collects the names of all global variables in a module. These are names bound at the module level or bound in a class or function body that declares them to be global. The compiler would assign indices for each global name and add the names and indices of the globals to the module’s code object. Each code object would then be bound irrevocably to the module it was defined in. (Not sure if there are some subtle problems with this.) For attributes of imported modules, the module will store an indirection record. Internally, the module will store a pointer to the defining module and the offset of the attribute in the defining module’s global variable array. The offset would be initialized the first time the name is looked up. </section> <section id="runtime-model"> <h2><a class="toc-backref" href="#runtime-model" role="doc-backlink">Runtime model</a></h2> The PythonVM will be extended with new opcodes to access globals and module attributes via a module-level array. A function object would need to point to the module that defined it in order to provide access to the module-level global array. For module attributes stored in the <code class="docutils literal notranslate">dlict</code> (call them static attributes), the get/delattr implementation would need to track access to these attributes using the old by-name interface. If a static attribute is updated dynamically, e.g.: <div class="highlight-default notranslate"><div class="highlight"><pre>mod.__dict__["foo"] = 2 </pre></div> </div> The implementation would need to update the array slot instead of the backup dict. </section> <section id="backwards-compatibility"> <h2><a class="toc-backref" href="#backwards-compatibility" role="doc-backlink">Backwards compatibility</a></h2> The <code class="docutils literal notranslate">dlict</code> will need to maintain meta-information about whether a slot is currently used or not. It will also need to maintain a pointer to the builtin namespace. When a name is not currently used in the global namespace, the lookup will have to fail over to the builtin namespace. In the reverse case, each module may need a special accessor function for the builtin namespace that checks to see if a global shadowing the builtin has been added dynamically. This check would only occur if there was a dynamic change to the module’s <code class="docutils literal notranslate">dlict</code>, i.e. when a name is bound that wasn’t discovered at compile-time. These mechanisms would have little if any cost for the common case whether a module’s global namespace is not modified in strange ways at runtime. They would add overhead for modules that did unusual things with global names, but this is an uncommon practice and probably one worth discouraging. It may be desirable to disable dynamic additions to the global namespace in some future version of Python. If so, the new implementation could provide warnings. </section> <section id="related-peps"> <h2><a class="toc-backref" href="#related-peps" role="doc-backlink">Related PEPs</a></h2> <a class="pep reference internal" href="../pep-0266/" title="PEP 266 – Optimizing Global Variable/Attribute Access">PEP 266</a>, Optimizing Global Variable/Attribute Access, proposes a different mechanism for optimizing access to global variables as well as attributes of objects. The mechanism uses two new opcodes <code class="docutils literal notranslate">TRACK_OBJECT</code> and <code class="docutils literal notranslate">UNTRACK_OBJECT</code> to create a slot in the local variables array that aliases the global or object attribute. If the object being aliases is rebound, the rebind operation is responsible for updating the aliases. The objecting tracking approach applies to a wider range of objects than just module. It may also have a higher runtime cost, because each function that uses a global or object attribute must execute extra opcodes to register its interest in an object and unregister on exit; the cost of registration is unclear, but presumably involves a dynamically resizable data structure to hold a list of callbacks. The implementation proposed here avoids the need for registration, because it does not create aliases. Instead it allows functions that reference a global variable or module attribute to retain a pointer to the location where the original binding is stored. A second advantage is that the initial lookup is performed once per module rather than once per function call. </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0267.rst">https://github.com/python/peps/blob/main/peps/pep-0267.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0267.rst">2023-09-09 17:39:29 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#deferral">Deferral</a></li> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#introduction">Introduction</a></li> <li><a class="reference internal" href="#dlict-design">DLict design</a></li> <li><a class="reference internal" href="#compiler-issues">Compiler issues</a></li> <li><a class="reference internal" href="#runtime-model">Runtime model</a></li> <li><a class="reference internal" href="#backwards-compatibility">Backwards compatibility</a></li> <li><a class="reference internal" href="#related-peps">Related PEPs</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0267.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>