CINXE.COM
PEP 253 – Subtyping Built-in Types | peps.python.org
<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 253 – Subtyping Built-in Types | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0253/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 253 – Subtyping Built-in Types | peps.python.org'> <meta property="og:description" content="This PEP proposes additions to the type object API that will allow the creation of subtypes of built-in types, in C and in Python."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0253/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This PEP proposes additions to the type object API that will allow the creation of subtypes of built-in types, in C and in Python."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 253</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> <span class="visually-hidden">Toggle light / dark / auto colour theme</span> </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 253 – Subtyping Built-in Types</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author<span class="colon">:</span></dt> <dd class="field-odd">Guido van Rossum <guido at python.org></dd> <dt class="field-even">Status<span class="colon">:</span></dt> <dd class="field-even"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-odd">Type<span class="colon">:</span></dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created<span class="colon">:</span></dt> <dd class="field-even">14-May-2001</dd> <dt class="field-odd">Python-Version<span class="colon">:</span></dt> <dd class="field-odd">2.2</dd> <dt class="field-even">Post-History<span class="colon">:</span></dt> <dd class="field-even"><p></p></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#introduction">Introduction</a></li> <li><a class="reference internal" href="#about-metatypes">About metatypes</a></li> <li><a class="reference internal" href="#making-a-type-a-factory-for-its-instances">Making a type a factory for its instances</a></li> <li><a class="reference internal" href="#preparing-a-type-for-subtyping">Preparing a type for subtyping</a></li> <li><a class="reference internal" href="#creating-a-subtype-of-a-built-in-type-in-c">Creating a subtype of a built-in type in C</a></li> <li><a class="reference internal" href="#subtyping-in-python">Subtyping in Python</a></li> <li><a class="reference internal" href="#multiple-inheritance">Multiple inheritance</a></li> <li><a class="reference internal" href="#mro-method-resolution-order-the-lookup-rule">MRO: Method resolution order (the lookup rule)</a></li> <li><a class="reference internal" href="#xxx-to-be-done">XXX To be done</a><ul> <li><a class="reference internal" href="#open-issues">open issues</a></li> </ul> </li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> <p>This PEP proposes additions to the type object API that will allow the creation of subtypes of built-in types, in C and in Python.</p> <p>[Editor’s note: the ideas described in this PEP have been incorporated into Python. The PEP no longer accurately describes the implementation.]</p> </section> <section id="introduction"> <h2><a class="toc-backref" href="#introduction" role="doc-backlink">Introduction</a></h2> <p>Traditionally, types in Python have been created statically, by declaring a global variable of type PyTypeObject and initializing it with a static initializer. The slots in the type object describe all aspects of a Python type that are relevant to the Python interpreter. A few slots contain dimensional information (like the basic allocation size of instances), others contain various flags, but most slots are pointers to functions to implement various kinds of behaviors. A NULL pointer means that the type does not implement the specific behavior; in that case the system may provide a default behavior or raise an exception when the behavior is invoked for an instance of the type. Some collections of function pointers that are usually defined together are obtained indirectly via a pointer to an additional structure containing more function pointers.</p> <p>While the details of initializing a PyTypeObject structure haven’t been documented as such, they are easily gleaned from the examples in the source code, and I am assuming that the reader is sufficiently familiar with the traditional way of creating new Python types in C.</p> <p>This PEP will introduce the following features:</p> <ul class="simple"> <li>a type can be a factory function for its instances</li> <li>types can be subtyped in C</li> <li>types can be subtyped in Python with the class statement</li> <li>multiple inheritance from types is supported (insofar as practical – you still can’t multiply inherit from list and dictionary)</li> <li>the standard coercion functions (int, tuple, str etc.) will be redefined to be the corresponding type objects, which serve as their own factory functions</li> <li>a class statement can contain a <code class="docutils literal notranslate"><span class="pre">__metaclass__</span></code> declaration, specifying the metaclass to be used to create the new class</li> <li>a class statement can contain a <code class="docutils literal notranslate"><span class="pre">__slots__</span></code> declaration, specifying the specific names of the instance variables supported</li> </ul> <p>This PEP builds on <a class="pep reference internal" href="../pep-0252/" title="PEP 252 – Making Types Look More Like Classes">PEP 252</a>, which adds standard introspection to types; for example, when a particular type object initializes the <code class="docutils literal notranslate"><span class="pre">tp_hash</span></code> slot, that type object has a <code class="docutils literal notranslate"><span class="pre">__hash__</span></code> method when introspected. <a class="pep reference internal" href="../pep-0252/" title="PEP 252 – Making Types Look More Like Classes">PEP 252</a> also adds a dictionary to type objects which contains all methods. At the Python level, this dictionary is read-only for built-in types; at the C level, it is accessible directly (but it should not be modified except as part of initialization).</p> <p>For binary compatibility, a flag bit in the tp_flags slot indicates the existence of the various new slots in the type object introduced below. Types that don’t have the <code class="docutils literal notranslate"><span class="pre">Py_TPFLAGS_HAVE_CLASS</span></code> bit set in their <code class="docutils literal notranslate"><span class="pre">tp_flags</span></code> slot are assumed to have NULL values for all the subtyping slots. (Warning: the current implementation prototype is not yet consistent in its checking of this flag bit. This should be fixed before the final release.)</p> <p>In current Python, a distinction is made between types and classes. This PEP together with <a class="pep reference internal" href="../pep-0254/" title="PEP 254 – Making Classes Look More Like Types">PEP 254</a> will remove that distinction. However, for backwards compatibility the distinction will probably remain for years to come, and without <a class="pep reference internal" href="../pep-0254/" title="PEP 254 – Making Classes Look More Like Types">PEP 254</a>, the distinction is still large: types ultimately have a built-in type as a base class, while classes ultimately derive from a user-defined class. Therefore, in the rest of this PEP, I will use the word type whenever I can – including base type or supertype, derived type or subtype, and metatype. However, sometimes the terminology necessarily blends, for example an object’s type is given by its <code class="docutils literal notranslate"><span class="pre">__class__</span></code> attribute, and subtyping in Python is spelled with a class statement. If further distinction is necessary, user-defined classes can be referred to as “classic” classes.</p> </section> <section id="about-metatypes"> <h2><a class="toc-backref" href="#about-metatypes" role="doc-backlink">About metatypes</a></h2> <p>Inevitably the discussion comes to metatypes (or metaclasses). Metatypes are nothing new in Python: Python has always been able to talk about the type of a type:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">a</span> <span class="o">=</span> <span class="mi">0</span> <span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="go"><type 'int'></span> <span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="nb">type</span><span class="p">(</span><span class="n">a</span><span class="p">))</span> <span class="go"><type 'type'></span> <span class="gp">>>> </span><span class="nb">type</span><span class="p">(</span><span class="nb">type</span><span class="p">(</span><span class="nb">type</span><span class="p">(</span><span class="n">a</span><span class="p">)))</span> <span class="go"><type 'type'></span> <span class="gp">>>></span> </pre></div> </div> <p>In this example, <code class="docutils literal notranslate"><span class="pre">type(a)</span></code> is a “regular” type, and <code class="docutils literal notranslate"><span class="pre">type(type(a))</span></code> is a metatype. While as distributed all types have the same metatype (<code class="docutils literal notranslate"><span class="pre">PyType_Type</span></code>, which is also its own metatype), this is not a requirement, and in fact a useful and relevant 3rd party extension (ExtensionClasses by Jim Fulton) creates an additional metatype. The type of classic classes, known as <code class="docutils literal notranslate"><span class="pre">types.ClassType</span></code>, can also be considered a distinct metatype.</p> <p>A feature closely connected to metatypes is the “Don Beaudry hook”, which says that if a metatype is callable, its instances (which are regular types) can be subclassed (really subtyped) using a Python class statement. I will use this rule to support subtyping of built-in types, and in fact it greatly simplifies the logic of class creation to always simply call the metatype. When no base class is specified, a default metatype is called – the default metatype is the “ClassType” object, so the class statement will behave as before in the normal case. (This default can be changed per module by setting the global variable <code class="docutils literal notranslate"><span class="pre">__metaclass__</span></code>.)</p> <p>Python uses the concept of metatypes or metaclasses in a different way than Smalltalk. In Smalltalk-80, there is a hierarchy of metaclasses that mirrors the hierarchy of regular classes, metaclasses map 1-1 to classes (except for some funny business at the root of the hierarchy), and each class statement creates both a regular class and its metaclass, putting class methods in the metaclass and instance methods in the regular class.</p> <p>Nice though this may be in the context of Smalltalk, it’s not compatible with the traditional use of metatypes in Python, and I prefer to continue in the Python way. This means that Python metatypes are typically written in C, and may be shared between many regular types. (It will be possible to subtype metatypes in Python, so it won’t be absolutely necessary to write C to use metatypes; but the power of Python metatypes will be limited. For example, Python code will never be allowed to allocate raw memory and initialize it at will.)</p> <p>Metatypes determine various <strong>policies</strong> for types, such as what happens when a type is called, how dynamic types are (whether a type’s <code class="docutils literal notranslate"><span class="pre">__dict__</span></code> can be modified after it is created), what the method resolution order is, how instance attributes are looked up, and so on.</p> <p>I’ll argue that left-to-right depth-first is not the best solution when you want to get the most use from multiple inheritance.</p> <p>I’ll argue that with multiple inheritance, the metatype of the subtype must be a descendant of the metatypes of all base types.</p> <p>I’ll come back to metatypes later.</p> </section> <section id="making-a-type-a-factory-for-its-instances"> <h2><a class="toc-backref" href="#making-a-type-a-factory-for-its-instances" role="doc-backlink">Making a type a factory for its instances</a></h2> <p>Traditionally, for each type there is at least one C factory function that creates instances of the type (<code class="docutils literal notranslate"><span class="pre">PyTuple_New()</span></code>, <code class="docutils literal notranslate"><span class="pre">PyInt_FromLong()</span></code> and so on). These factory functions take care of both allocating memory for the object and initializing that memory. As of Python 2.0, they also have to interface with the garbage collection subsystem, if the type chooses to participate in garbage collection (which is optional, but strongly recommended for so-called “container” types: types that may contain references to other objects, and hence may participate in reference cycles).</p> <p>In this proposal, type objects can be factory functions for their instances, making the types directly callable from Python. This mimics the way classes are instantiated. The C APIs for creating instances of various built-in types will remain valid and in some cases more efficient. Not all types will become their own factory functions.</p> <p>The type object has a new slot, tp_new, which can act as a factory for instances of the type. Types are now callable, because the tp_call slot is set in <code class="docutils literal notranslate"><span class="pre">PyType_Type</span></code> (the metatype); the function looks for the tp_new slot of the type that is being called.</p> <p>Explanation: the <code class="docutils literal notranslate"><span class="pre">tp_call</span></code> slot of a regular type object (such as <code class="docutils literal notranslate"><span class="pre">PyInt_Type</span></code> or <code class="docutils literal notranslate"><span class="pre">PyList_Type</span></code>) defines what happens when <strong>instances</strong> of that type are called; in particular, the <code class="docutils literal notranslate"><span class="pre">tp_call</span></code> slot in the function type, <code class="docutils literal notranslate"><span class="pre">PyFunction_Type</span></code>, is the key to making functions callable. As another example, <code class="docutils literal notranslate"><span class="pre">PyInt_Type.tp_call</span></code> is <code class="docutils literal notranslate"><span class="pre">NULL</span></code>, because integers are not callable. The new paradigm makes <strong>type objects</strong> callable. Since type objects are instances of their metatype (<code class="docutils literal notranslate"><span class="pre">PyType_Type</span></code>), the metatype’s <code class="docutils literal notranslate"><span class="pre">tp_call</span></code> slot (<code class="docutils literal notranslate"><span class="pre">PyType_Type.tp_call</span></code>) points to a function that is invoked when any type object is called. Now, since each type has to do something different to create an instance of itself, <code class="docutils literal notranslate"><span class="pre">PyType_Type.tp_call</span></code> immediately defers to the <code class="docutils literal notranslate"><span class="pre">tp_new</span></code> slot of the type that is being called. <code class="docutils literal notranslate"><span class="pre">PyType_Type</span></code> itself is also callable: its <code class="docutils literal notranslate"><span class="pre">tp_new</span></code> slot creates a new type. This is used by the class statement (formalizing the Don Beaudry hook, see above). And what makes <code class="docutils literal notranslate"><span class="pre">PyType_Type</span></code> callable? The <code class="docutils literal notranslate"><span class="pre">tp_call</span></code> slot of <strong>its</strong> metatype – but since it is its own metatype, that is its own <code class="docutils literal notranslate"><span class="pre">tp_call</span></code> slot!</p> <p>If the type’s <code class="docutils literal notranslate"><span class="pre">tp_new</span></code> slot is NULL, an exception is raised. Otherwise, the tp_new slot is called. The signature for the <code class="docutils literal notranslate"><span class="pre">tp_new</span></code> slot is</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">PyObject</span> <span class="o">*</span><span class="n">tp_new</span><span class="p">(</span><span class="n">PyTypeObject</span> <span class="o">*</span><span class="nb">type</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">kwds</span><span class="p">)</span> </pre></div> </div> <p>where ‘type’ is the type whose <code class="docutils literal notranslate"><span class="pre">tp_new</span></code> slot is called, and ‘args’ and ‘kwds’ are the sequential and keyword arguments to the call, passed unchanged from tp_call. (The ‘type’ argument is used in combination with inheritance, see below.)</p> <p>There are no constraints on the object type that is returned, although by convention it should be an instance of the given type. It is not necessary that a new object is returned; a reference to an existing object is fine too. The return value should always be a new reference, owned by the caller.</p> <p>Once the <code class="docutils literal notranslate"><span class="pre">tp_new</span></code> slot has returned an object, further initialization is attempted by calling the <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code> slot of the resulting object’s type, if not NULL. This has the following signature:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="nb">int</span> <span class="n">tp_init</span><span class="p">(</span><span class="n">PyObject</span> <span class="o">*</span><span class="bp">self</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="n">PyObject</span> <span class="o">*</span><span class="n">kwds</span><span class="p">)</span> </pre></div> </div> <p>It corresponds more closely to the <code class="docutils literal notranslate"><span class="pre">__init__()</span></code> method of classic classes, and in fact is mapped to that by the slot/special-method correspondence rules. The difference in responsibilities between the <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> slot and the <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code> slot lies in the invariants they ensure. The <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> slot should ensure only the most essential invariants, without which the C code that implements the objects would break. The <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code> slot should be used for overridable user-specific initializations. Take for example the dictionary type. The implementation has an internal pointer to a hash table which should never be NULL. This invariant is taken care of by the <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> slot for dictionaries. The dictionary <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code> slot, on the other hand, could be used to give the dictionary an initial set of keys and values based on the arguments passed in.</p> <p>Note that for immutable object types, the initialization cannot be done by the <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code> slot: this would provide the Python user with a way to change the initialization. Therefore, immutable objects typically have an empty <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code> implementation and do all their initialization in their <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> slot.</p> <p>You may wonder why the <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> slot shouldn’t call the <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code> slot itself. The reason is that in certain circumstances (like support for persistent objects), it is important to be able to create an object of a particular type without initializing it any further than necessary. This may conveniently be done by calling the <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> slot without calling <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code>. It is also possible that <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code> is not called, or called more than once – its operation should be robust even in these anomalous cases.</p> <p>For some objects, <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> may return an existing object. For example, the factory function for integers caches the integers -1 through 99. This is permissible only when the type argument to <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> is the type that defined the <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> function (in the example, if <code class="docutils literal notranslate"><span class="pre">type</span> <span class="pre">==</span> <span class="pre">&PyInt_Type</span></code>), and when the <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code> slot for this type does nothing. If the type argument differs, the <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> call is initiated by a derived type’s <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> to create the object and initialize the base type portion of the object; in this case <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> should always return a new object (or raise an exception).</p> <p>Both <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> and <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code> should receive exactly the same ‘args’ and ‘kwds’ arguments, and both should check that the arguments are acceptable, because they may be called independently.</p> <p>There’s a third slot related to object creation: <code class="docutils literal notranslate"><span class="pre">tp_alloc()</span></code>. Its responsibility is to allocate the memory for the object, initialize the reference count (<code class="docutils literal notranslate"><span class="pre">ob_refcnt</span></code>) and the type pointer (<code class="docutils literal notranslate"><span class="pre">ob_type</span></code>), and initialize the rest of the object to all zeros. It should also register the object with the garbage collection subsystem if the type supports garbage collection. This slot exists so that derived types can override the memory allocation policy (like which heap is being used) separately from the initialization code. The signature is:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">PyObject</span> <span class="o">*</span><span class="n">tp_alloc</span><span class="p">(</span><span class="n">PyTypeObject</span> <span class="o">*</span><span class="nb">type</span><span class="p">,</span> <span class="nb">int</span> <span class="n">nitems</span><span class="p">)</span> </pre></div> </div> <p>The type argument is the type of the new object. The nitems argument is normally zero, except for objects with a variable allocation size (basically strings, tuples, and longs). The allocation size is given by the following expression:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="nb">type</span><span class="o">-></span><span class="n">tp_basicsize</span> <span class="o">+</span> <span class="n">nitems</span> <span class="o">*</span> <span class="nb">type</span><span class="o">-></span><span class="n">tp_itemsize</span> </pre></div> </div> <p>The <code class="docutils literal notranslate"><span class="pre">tp_alloc</span></code> slot is only used for subclassable types. The <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> function of the base class must call the <code class="docutils literal notranslate"><span class="pre">tp_alloc()</span></code> slot of the type passed in as its first argument. It is the <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> function’s responsibility to calculate the number of items. The <code class="docutils literal notranslate"><span class="pre">tp_alloc()</span></code> slot will set the ob_size member of the new object if the <code class="docutils literal notranslate"><span class="pre">type->tp_itemsize</span></code> member is nonzero.</p> <p>(Note: in certain debugging compilation modes, the type structure used to have members named <code class="docutils literal notranslate"><span class="pre">tp_alloc</span></code> and a <code class="docutils literal notranslate"><span class="pre">tp_free</span></code> slot already, counters for the number of allocations and deallocations. These are renamed to <code class="docutils literal notranslate"><span class="pre">tp_allocs</span></code> and <code class="docutils literal notranslate"><span class="pre">tp_deallocs</span></code>.)</p> <p>Standard implementations for <code class="docutils literal notranslate"><span class="pre">tp_alloc()</span></code> and <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> are available. <code class="docutils literal notranslate"><span class="pre">PyType_GenericAlloc()</span></code> allocates an object from the standard heap and initializes it properly. It uses the above formula to determine the amount of memory to allocate, and takes care of GC registration. The only reason not to use this implementation would be to allocate objects from a different heap (as is done by some very small frequently used objects like ints and tuples). <code class="docutils literal notranslate"><span class="pre">PyType_GenericNew()</span></code> adds very little: it just calls the type’s <code class="docutils literal notranslate"><span class="pre">tp_alloc()</span></code> slot with zero for nitems. But for mutable types that do all their initialization in their <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code> slot, this may be just the ticket.</p> </section> <section id="preparing-a-type-for-subtyping"> <h2><a class="toc-backref" href="#preparing-a-type-for-subtyping" role="doc-backlink">Preparing a type for subtyping</a></h2> <p>The idea behind subtyping is very similar to that of single inheritance in C++. A base type is described by a structure declaration (similar to the C++ class declaration) plus a type object (similar to the C++ vtable). A derived type can extend the structure (but must leave the names, order and type of the members of the base structure unchanged) and can override certain slots in the type object, leaving others the same. (Unlike C++ vtables, all Python type objects have the same memory layout.)</p> <p>The base type must do the following:</p> <ul class="simple"> <li>Add the flag value <code class="docutils literal notranslate"><span class="pre">Py_TPFLAGS_BASETYPE</span></code> to <code class="docutils literal notranslate"><span class="pre">tp_flags</span></code>.</li> <li>Declare and use <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code>, <code class="docutils literal notranslate"><span class="pre">tp_alloc()</span></code> and optional <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code> slots.</li> <li>Declare and use <code class="docutils literal notranslate"><span class="pre">tp_dealloc()</span></code> and <code class="docutils literal notranslate"><span class="pre">tp_free()</span></code>.</li> <li>Export its object structure declaration.</li> <li>Export a subtyping-aware type-checking macro.</li> </ul> <p>The requirements and signatures for <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code>, <code class="docutils literal notranslate"><span class="pre">tp_alloc()</span></code> and <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code> have already been discussed above: <code class="docutils literal notranslate"><span class="pre">tp_alloc()</span></code> should allocate the memory and initialize it to mostly zeros; <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> should call the <code class="docutils literal notranslate"><span class="pre">tp_alloc()</span></code> slot and then proceed to do the minimally required initialization; <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code> should be used for more extensive initialization of mutable objects.</p> <p>It should come as no surprise that there are similar conventions at the end of an object’s lifetime. The slots involved are <code class="docutils literal notranslate"><span class="pre">tp_dealloc()</span></code> (familiar to all who have ever implemented a Python extension type) and <code class="docutils literal notranslate"><span class="pre">tp_free()</span></code>, the new kid on the block. (The names aren’t quite symmetric; <code class="docutils literal notranslate"><span class="pre">tp_free()</span></code> corresponds to <code class="docutils literal notranslate"><span class="pre">tp_alloc()</span></code>, which is fine, but <code class="docutils literal notranslate"><span class="pre">tp_dealloc()</span></code> corresponds to <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code>. Maybe the tp_dealloc slot should be renamed?)</p> <p>The <code class="docutils literal notranslate"><span class="pre">tp_free()</span></code> slot should be used to free the memory and unregister the object with the garbage collection subsystem, and can be overridden by a derived class; <code class="docutils literal notranslate"><span class="pre">tp_dealloc()</span></code> should deinitialize the object (usually by calling <code class="docutils literal notranslate"><span class="pre">Py_XDECREF()</span></code> for various sub-objects) and then call <code class="docutils literal notranslate"><span class="pre">tp_free()</span></code> to deallocate the memory. The signature for <code class="docutils literal notranslate"><span class="pre">tp_dealloc()</span></code> is the same as it always was:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">void</span> <span class="n">tp_dealloc</span><span class="p">(</span><span class="n">PyObject</span> <span class="o">*</span><span class="nb">object</span><span class="p">)</span> </pre></div> </div> <p>The signature for tp_free() is the same:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">void</span> <span class="n">tp_free</span><span class="p">(</span><span class="n">PyObject</span> <span class="o">*</span><span class="nb">object</span><span class="p">)</span> </pre></div> </div> <p>(In a previous version of this PEP, there was also a role reserved for the <code class="docutils literal notranslate"><span class="pre">tp_clear()</span></code> slot. This turned out to be a bad idea.)</p> <p>To be usefully subtyped in C, a type must export the structure declaration for its instances through a header file, as it is needed to derive a subtype. The type object for the base type must also be exported.</p> <p>If the base type has a type-checking macro (like <code class="docutils literal notranslate"><span class="pre">PyDict_Check()</span></code>), this macro should be made to recognize subtypes. This can be done by using the new <code class="docutils literal notranslate"><span class="pre">PyObject_TypeCheck(object,</span> <span class="pre">type)</span></code> macro, which calls a function that follows the base class links.</p> <p>The <code class="docutils literal notranslate"><span class="pre">PyObject_TypeCheck()</span></code> macro contains a slight optimization: it first compares <code class="docutils literal notranslate"><span class="pre">object->ob_type</span></code> directly to the type argument, and if this is a match, bypasses the function call. This should make it fast enough for most situations.</p> <p>Note that this change in the type-checking macro means that C functions that require an instance of the base type may be invoked with instances of the derived type. Before enabling subtyping of a particular type, its code should be checked to make sure that this won’t break anything. It has proved useful in the prototype to add another type-checking macro for the built-in Python object types, to check for exact type match too (for example, <code class="docutils literal notranslate"><span class="pre">PyDict_Check(x)</span></code> is true if x is an instance of dictionary or of a dictionary subclass, while <code class="docutils literal notranslate"><span class="pre">PyDict_CheckExact(x)</span></code> is true only if x is a dictionary).</p> </section> <section id="creating-a-subtype-of-a-built-in-type-in-c"> <h2><a class="toc-backref" href="#creating-a-subtype-of-a-built-in-type-in-c" role="doc-backlink">Creating a subtype of a built-in type in C</a></h2> <p>The simplest form of subtyping is subtyping in C. It is the simplest form because we can require the C code to be aware of some of the problems, and it’s acceptable for C code that doesn’t follow the rules to dump core. For added simplicity, it is limited to single inheritance.</p> <p>Let’s assume we’re deriving from a mutable base type whose tp_itemsize is zero. The subtype code is not GC-aware, although it may inherit GC-awareness from the base type (this is automatic). The base type’s allocation uses the standard heap.</p> <p>The derived type begins by declaring a type structure which contains the base type’s structure. For example, here’s the type structure for a subtype of the built-in list type:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">typedef</span> <span class="n">struct</span> <span class="p">{</span> <span class="n">PyListObject</span> <span class="nb">list</span><span class="p">;</span> <span class="nb">int</span> <span class="n">state</span><span class="p">;</span> <span class="p">}</span> <span class="n">spamlistobject</span><span class="p">;</span> </pre></div> </div> <p>Note that the base type structure member (here <code class="docutils literal notranslate"><span class="pre">PyListObject</span></code>) must be the first member of the structure; any following members are additions. Also note that the base type is not referenced via a pointer; the actual contents of its structure must be included! (The goal is for the memory layout of the beginning of the subtype instance to be the same as that of the base type instance.)</p> <p>Next, the derived type must declare a type object and initialize it. Most of the slots in the type object may be initialized to zero, which is a signal that the base type slot must be copied into it. Some slots that must be initialized properly:</p> <ul class="simple"> <li>The object header must be filled in as usual; the type should be <code class="docutils literal notranslate"><span class="pre">&PyType_Type</span></code>.</li> <li>The tp_basicsize slot must be set to the size of the subtype instance struct (in the above example: <code class="docutils literal notranslate"><span class="pre">sizeof(spamlistobject)</span></code>).</li> <li>The tp_base slot must be set to the address of the base type’s type object.</li> <li>If the derived slot defines any pointer members, the <code class="docutils literal notranslate"><span class="pre">tp_dealloc</span></code> slot function requires special attention, see below; otherwise, it can be set to zero, to inherit the base type’s deallocation function.</li> <li>The <code class="docutils literal notranslate"><span class="pre">tp_flags</span></code> slot must be set to the usual <code class="docutils literal notranslate"><span class="pre">Py_TPFLAGS_DEFAULT</span></code> value.</li> <li>The <code class="docutils literal notranslate"><span class="pre">tp_name</span></code> slot must be set; it is recommended to set <code class="docutils literal notranslate"><span class="pre">tp_doc</span></code> as well (these are not inherited).</li> </ul> <p>If the subtype defines no additional structure members (it only defines new behavior, no new data), the <code class="docutils literal notranslate"><span class="pre">tp_basicsize</span></code> and the <code class="docutils literal notranslate"><span class="pre">tp_dealloc</span></code> slots may be left set to zero.</p> <p>The subtype’s <code class="docutils literal notranslate"><span class="pre">tp_dealloc</span></code> slot deserves special attention. If the derived type defines no additional pointer members that need to be DECREF’ed or freed when the object is deallocated, it can be set to zero. Otherwise, the subtype’s <code class="docutils literal notranslate"><span class="pre">tp_dealloc()</span></code> function must call <code class="docutils literal notranslate"><span class="pre">Py_XDECREF()</span></code> for any <code class="docutils literal notranslate"><span class="pre">PyObject</span> <span class="pre">*</span></code> members and the correct memory freeing function for any other pointers it owns, and then call the base class’s <code class="docutils literal notranslate"><span class="pre">tp_dealloc()</span></code> slot. This call has to be made via the base type’s type structure, for example, when deriving from the standard list type:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">PyList_Type</span><span class="o">.</span><span class="n">tp_dealloc</span><span class="p">(</span><span class="bp">self</span><span class="p">);</span> </pre></div> </div> <p>If the subtype wants to use a different allocation heap than the base type, the subtype must override both the <code class="docutils literal notranslate"><span class="pre">tp_alloc()</span></code> and the <code class="docutils literal notranslate"><span class="pre">tp_free()</span></code> slots. These will be called by the base class’s <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> and <code class="docutils literal notranslate"><span class="pre">tp_dealloc()</span></code> slots, respectively.</p> <p>To complete the initialization of the type, <code class="docutils literal notranslate"><span class="pre">PyType_InitDict()</span></code> must be called. This replaces slots initialized to zero in the subtype with the value of the corresponding base type slots. (It also fills in <code class="docutils literal notranslate"><span class="pre">tp_dict</span></code>, the type’s dictionary, and does various other initializations necessary for type objects.)</p> <p>A subtype is not usable until <code class="docutils literal notranslate"><span class="pre">PyType_InitDict()</span></code> is called for it; this is best done during module initialization, assuming the subtype belongs to a module. An alternative for subtypes added to the Python core (which don’t live in a particular module) would be to initialize the subtype in their constructor function. It is allowed to call <code class="docutils literal notranslate"><span class="pre">PyType_InitDict()</span></code> more than once; the second and further calls have no effect. To avoid unnecessary calls, a test for <code class="docutils literal notranslate"><span class="pre">tp_dict==NULL</span></code> can be made.</p> <p>(During initialization of the Python interpreter, some types are actually used before they are initialized. As long as the slots that are actually needed are initialized, especially <code class="docutils literal notranslate"><span class="pre">tp_dealloc</span></code>, this works, but it is fragile and not recommended as a general practice.)</p> <p>To create a subtype instance, the subtype’s <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> slot is called. This should first call the base type’s <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> slot and then initialize the subtype’s additional data members. To further initialize the instance, the <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code> slot is typically called. Note that the <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> slot should <strong>not</strong> call the <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code> slot; this is up to <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code>’s caller (typically a factory function). There are circumstances where it is appropriate not to call <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code>.</p> <p>If a subtype defines a <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code> slot, the <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code> slot should normally first call the base type’s <code class="docutils literal notranslate"><span class="pre">tp_init()</span></code> slot.</p> <p>(XXX There should be a paragraph or two about argument passing here.)</p> </section> <section id="subtyping-in-python"> <h2><a class="toc-backref" href="#subtyping-in-python" role="doc-backlink">Subtyping in Python</a></h2> <p>The next step is to allow subtyping of selected built-in types through a class statement in Python. Limiting ourselves to single inheritance for now, here is what happens for a simple class statement:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">C</span><span class="p">(</span><span class="n">B</span><span class="p">):</span> <span class="n">var1</span> <span class="o">=</span> <span class="mi">1</span> <span class="k">def</span> <span class="nf">method1</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">pass</span> <span class="c1"># etc.</span> </pre></div> </div> <p>The body of the class statement is executed in a fresh environment (basically, a new dictionary used as local namespace), and then C is created. The following explains how C is created.</p> <p>Assume B is a type object. Since type objects are objects, and every object has a type, B has a type. Since B is itself a type, we also call its type its metatype. B’s metatype is accessible via <code class="docutils literal notranslate"><span class="pre">type(B)</span></code> or <code class="docutils literal notranslate"><span class="pre">B.__class__</span></code> (the latter notation is new for types; it is introduced in <a class="pep reference internal" href="../pep-0252/" title="PEP 252 – Making Types Look More Like Classes">PEP 252</a>). Let’s say this metatype is M (for Metatype). The class statement will create a new type, C. Since C will be a type object just like B, we view the creation of C as an instantiation of the metatype, M. The information that needs to be provided for the creation of a subclass is:</p> <ul class="simple"> <li>its name (in this example the string “C”);</li> <li>its bases (a singleton tuple containing B);</li> <li>the results of executing the class body, in the form of a dictionary (for example <code class="docutils literal notranslate"><span class="pre">{"var1":</span> <span class="pre">1,</span> <span class="pre">"method1":</span> <span class="pre"><functionmethod1</span> <span class="pre">at</span> <span class="pre">...>,</span> <span class="pre">...}</span></code>).</li> </ul> <p>The class statement will result in the following call:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">C</span> <span class="o">=</span> <span class="n">M</span><span class="p">(</span><span class="s2">"C"</span><span class="p">,</span> <span class="p">(</span><span class="n">B</span><span class="p">,),</span> <span class="nb">dict</span><span class="p">)</span> </pre></div> </div> <p>where dict is the dictionary resulting from execution of the class body. In other words, the metatype (M) is called.</p> <p>Note that even though the example has only one base, we still pass in a (singleton) sequence of bases; this makes the interface uniform with the multiple-inheritance case.</p> <p>In current Python, this is called the “Don Beaudry hook” after its inventor; it is an exceptional case that is only invoked when a base class is not a regular class. For a regular base class (or when no base class is specified), current Python calls <code class="docutils literal notranslate"><span class="pre">PyClass_New()</span></code>, the C level factory function for classes, directly.</p> <p>Under the new system this is changed so that Python <strong>always</strong> determines a metatype and calls it as given above. When one or more bases are given, the type of the first base is used as the metatype; when no base is given, a default metatype is chosen. By setting the default metatype to <code class="docutils literal notranslate"><span class="pre">PyClass_Type</span></code>, the metatype of “classic” classes, the classic behavior of the class statement is retained. This default can be changed per module by setting the global variable <code class="docutils literal notranslate"><span class="pre">__metaclass__</span></code>.</p> <p>There are two further refinements here. First, a useful feature is to be able to specify a metatype directly. If the class suite defines a variable <code class="docutils literal notranslate"><span class="pre">__metaclass__</span></code>, that is the metatype to call. (Note that setting <code class="docutils literal notranslate"><span class="pre">__metaclass__</span></code> at the module level only affects class statements without a base class and without an explicit <code class="docutils literal notranslate"><span class="pre">__metaclass__</span></code> declaration; but setting <code class="docutils literal notranslate"><span class="pre">__metaclass__</span></code> in a class suite overrides the default metatype unconditionally.)</p> <p>Second, with multiple bases, not all bases need to have the same metatype. This is called a metaclass conflict <a class="footnote-reference brackets" href="#id3" id="id1">[1]</a>. Some metaclass conflicts can be resolved by searching through the set of bases for a metatype that derives from all other given metatypes. If such a metatype cannot be found, an exception is raised and the class statement fails.</p> <p>This conflict resolution can be implemented by the metatype constructors: the class statement just calls the metatype of the first base (or that specified by the <code class="docutils literal notranslate"><span class="pre">__metaclass__</span></code> variable), and this metatype’s constructor looks for the most derived metatype. If that is itself, it proceeds; otherwise, it calls that metatype’s constructor. (Ultimate flexibility: another metatype might choose to require that all bases have the same metatype, or that there’s only one base class, or whatever.)</p> <p>(In <a class="footnote-reference brackets" href="#id3" id="id2">[1]</a>, a new metaclass is automatically derived that is a subclass of all given metaclasses. But since it is questionable in Python how conflicting method definitions of the various metaclasses should be merged, I don’t think this is feasible. Should the need arise, the user can derive such a metaclass manually and specify it using the <code class="docutils literal notranslate"><span class="pre">__metaclass__</span></code> variable. It is also possible to have a new metaclass that does this.)</p> <p>Note that calling M requires that M itself has a type: the meta-metatype. And the meta-metatype has a type, the meta-meta-metatype. And so on. This is normally cut short at some level by making a metatype be its own metatype. This is indeed what happens in Python: the <code class="docutils literal notranslate"><span class="pre">ob_type</span></code> reference in <code class="docutils literal notranslate"><span class="pre">PyType_Type</span></code> is set to <code class="docutils literal notranslate"><span class="pre">&PyType_Type</span></code>. In the absence of third party metatypes, <code class="docutils literal notranslate"><span class="pre">PyType_Type</span></code> is the only metatype in the Python interpreter.</p> <p>(In a previous version of this PEP, there was one additional meta-level, and there was a meta-metatype called “turtle”. This turned out to be unnecessary.)</p> <p>In any case, the work for creating C is done by M’s <code class="docutils literal notranslate"><span class="pre">tp_new()</span></code> slot. It allocates space for an “extended” type structure, containing: the type object; the auxiliary structures (as_sequence etc.); the string object containing the type name (to ensure that this object isn’t deallocated while the type object is still referencing it); and some auxiliary storage (to be described later). It initializes this storage to zeros except for a few crucial slots (for example, tp_name is set to point to the type name) and then sets the tp_base slot to point to B. Then <code class="docutils literal notranslate"><span class="pre">PyType_InitDict()</span></code> is called to inherit B’s slots. Finally, C’s <code class="docutils literal notranslate"><span class="pre">tp_dict</span></code> slot is updated with the contents of the namespace dictionary (the third argument to the call to M).</p> </section> <section id="multiple-inheritance"> <h2><a class="toc-backref" href="#multiple-inheritance" role="doc-backlink">Multiple inheritance</a></h2> <p>The Python class statement supports multiple inheritance, and we will also support multiple inheritance involving built-in types.</p> <p>However, there are some restrictions. The C runtime architecture doesn’t make it feasible to have a meaningful subtype of two different built-in types except in a few degenerate cases. Changing the C runtime to support fully general multiple inheritance would be too much of an upheaval of the code base.</p> <p>The main problem with multiple inheritance from different built-in types stems from the fact that the C implementation of built-in types accesses structure members directly; the C compiler generates an offset relative to the object pointer and that’s that. For example, the list and dictionary type structures each declare a number of different but overlapping structure members. A C function accessing an object expecting a list won’t work when passed a dictionary, and vice versa, and there’s not much we could do about this without rewriting all code that accesses lists and dictionaries. This would be too much work, so we won’t do this.</p> <p>The problem with multiple inheritance is caused by conflicting structure member allocations. Classes defined in Python normally don’t store their instance variables in structure members: they are stored in an instance dictionary. This is the key to a partial solution. Suppose we have the following two classes:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">A</span><span class="p">(</span><span class="n">dictionary</span><span class="p">):</span> <span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">pass</span> <span class="k">class</span> <span class="nc">B</span><span class="p">(</span><span class="n">dictionary</span><span class="p">):</span> <span class="k">def</span> <span class="nf">bar</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">pass</span> <span class="k">class</span> <span class="nc">C</span><span class="p">(</span><span class="n">A</span><span class="p">,</span> <span class="n">B</span><span class="p">):</span> <span class="k">pass</span> </pre></div> </div> <p>(Here, ‘dictionary’ is the type of built-in dictionary objects, a.k.a. <code class="docutils literal notranslate"><span class="pre">type({})</span></code> or <code class="docutils literal notranslate"><span class="pre">{}.__class__</span></code> or <code class="docutils literal notranslate"><span class="pre">types.DictType</span></code>.) If we look at the structure layout, we find that an A instance has the layout of a dictionary followed by the <code class="docutils literal notranslate"><span class="pre">__dict__</span></code> pointer, and a B instance has the same layout; since there are no structure member layout conflicts, this is okay.</p> <p>Here’s another example:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">X</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span> <span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">pass</span> <span class="k">class</span> <span class="nc">Y</span><span class="p">(</span><span class="n">dictionary</span><span class="p">):</span> <span class="k">def</span> <span class="nf">bar</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="k">pass</span> <span class="k">class</span> <span class="nc">Z</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">Y</span><span class="p">):</span> <span class="k">pass</span> </pre></div> </div> <p>(Here, ‘object’ is the base for all built-in types; its structure layout only contains the <code class="docutils literal notranslate"><span class="pre">ob_refcnt</span></code> and <code class="docutils literal notranslate"><span class="pre">ob_type</span></code> members.) This example is more complicated, because the <code class="docutils literal notranslate"><span class="pre">__dict__</span></code> pointer for X instances has a different offset than that for Y instances. Where is the <code class="docutils literal notranslate"><span class="pre">__dict__</span></code> pointer for Z instances? The answer is that the offset for the <code class="docutils literal notranslate"><span class="pre">__dict__</span></code> pointer is not hardcoded, it is stored in the type object.</p> <p>Suppose on a particular machine an ‘object’ structure is 8 bytes long, and a ‘dictionary’ struct is 60 bytes, and an object pointer is 4 bytes. Then an X structure is 12 bytes (an object structure followed by a <code class="docutils literal notranslate"><span class="pre">__dict__</span></code> pointer), and a Y structure is 64 bytes (a dictionary structure followed by a <code class="docutils literal notranslate"><span class="pre">__dict__</span></code> pointer). The Z structure has the same layout as the Y structure in this example. Each type object (X, Y and Z) has a “__dict__ offset” which is used to find the <code class="docutils literal notranslate"><span class="pre">__dict__</span></code> pointer. Thus, the recipe for looking up an instance variable is:</p> <ol class="arabic simple"> <li>get the type of the instance</li> <li>get the <code class="docutils literal notranslate"><span class="pre">__dict__</span></code> offset from the type object</li> <li>add the <code class="docutils literal notranslate"><span class="pre">__dict__</span></code> offset to the instance pointer</li> <li>look in the resulting address to find a dictionary reference</li> <li>look up the instance variable name in that dictionary</li> </ol> <p>Of course, this recipe can only be implemented in C, and I have left out some details. But this allows us to use multiple inheritance patterns similar to the ones we can use with classic classes.</p> <p>XXX I should write up the complete algorithm here to determine base class compatibility, but I can’t be bothered right now. Look at <code class="docutils literal notranslate"><span class="pre">best_base()</span></code> in typeobject.c in the implementation mentioned below.</p> </section> <section id="mro-method-resolution-order-the-lookup-rule"> <h2><a class="toc-backref" href="#mro-method-resolution-order-the-lookup-rule" role="doc-backlink">MRO: Method resolution order (the lookup rule)</a></h2> <p>With multiple inheritance comes the question of method resolution order: the order in which a class or type and its bases are searched looking for a method of a given name.</p> <p>In classic Python, the rule is given by the following recursive function, also known as the left-to-right depth-first rule:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">classic_lookup</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span> <span class="k">if</span> <span class="bp">cls</span><span class="o">.</span><span class="vm">__dict__</span><span class="o">.</span><span class="n">has_key</span><span class="p">(</span><span class="n">name</span><span class="p">):</span> <span class="k">return</span> <span class="bp">cls</span><span class="o">.</span><span class="vm">__dict__</span><span class="p">[</span><span class="n">name</span><span class="p">]</span> <span class="k">for</span> <span class="n">base</span> <span class="ow">in</span> <span class="bp">cls</span><span class="o">.</span><span class="vm">__bases__</span><span class="p">:</span> <span class="k">try</span><span class="p">:</span> <span class="k">return</span> <span class="n">classic_lookup</span><span class="p">(</span><span class="n">base</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span> <span class="k">except</span> <span class="ne">AttributeError</span><span class="p">:</span> <span class="k">pass</span> <span class="k">raise</span> <span class="ne">AttributeError</span><span class="p">,</span> <span class="n">name</span> </pre></div> </div> <p>The problem with this becomes apparent when we consider a “diamond diagram”:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="k">class</span> <span class="nc">A</span><span class="p">:</span> <span class="o">^</span> <span class="o">^</span> <span class="k">def</span> <span class="nf">save</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="o">...</span> <span class="o">/</span> \ <span class="o">/</span> \ <span class="o">/</span> \ <span class="o">/</span> \ <span class="k">class</span> <span class="nc">B</span> <span class="k">class</span> <span class="nc">C</span><span class="p">:</span> <span class="o">^</span> <span class="o">^</span> <span class="k">def</span> <span class="nf">save</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> <span class="o">...</span> \ <span class="o">/</span> \ <span class="o">/</span> \ <span class="o">/</span> \ <span class="o">/</span> <span class="k">class</span> <span class="nc">D</span> </pre></div> </div> <p>Arrows point from a subtype to its base <code class="docutils literal notranslate"><span class="pre">type(s)</span></code>. This particular diagram means B and C derive from A, and D derives from B and C (and hence also, indirectly, from A).</p> <p>Assume that C overrides the method <code class="docutils literal notranslate"><span class="pre">save()</span></code>, which is defined in the base A. (<code class="docutils literal notranslate"><span class="pre">C.save()</span></code> probably calls <code class="docutils literal notranslate"><span class="pre">A.save()</span></code> and then saves some of its own state.) B and D don’t override <code class="docutils literal notranslate"><span class="pre">save()</span></code>. When we invoke <code class="docutils literal notranslate"><span class="pre">save()</span></code> on a D instance, which method is called? According to the classic lookup rule, <code class="docutils literal notranslate"><span class="pre">A.save()</span></code> is called, ignoring <code class="docutils literal notranslate"><span class="pre">C.save()</span></code>!</p> <p>This is not good. It probably breaks C (its state doesn’t get saved), defeating the whole purpose of inheriting from C in the first place.</p> <p>Why was this not a problem in classic Python? Diamond diagrams are rarely found in classic Python class hierarchies. Most class hierarchies use single inheritance, and multiple inheritance is usually confined to mix-in classes. In fact, the problem shown here is probably the reason why multiple inheritance is unpopular in classic Python.</p> <p>Why will this be a problem in the new system? The ‘object’ type at the top of the type hierarchy defines a number of methods that can usefully be extended by subtypes, for example <code class="docutils literal notranslate"><span class="pre">__getattr__()</span></code>.</p> <p>(Aside: in classic Python, the <code class="docutils literal notranslate"><span class="pre">__getattr__()</span></code> method is not really the implementation for the get-attribute operation; it is a hook that only gets invoked when an attribute cannot be found by normal means. This has often been cited as a shortcoming – some class designs have a legitimate need for a <code class="docutils literal notranslate"><span class="pre">__getattr__()</span></code> method that gets called for <strong>all</strong> attribute references. But then of course this method has to be able to invoke the default implementation directly. The most natural way is to make the default implementation available as <code class="docutils literal notranslate"><span class="pre">object.__getattr__(self,</span> <span class="pre">name)</span></code>.)</p> <p>Thus, a classic class hierarchy like this:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">B</span> <span class="k">class</span> <span class="nc">C</span><span class="p">:</span> <span class="o">^</span> <span class="o">^</span> <span class="k">def</span> <span class="fm">__getattr__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span> <span class="o">...</span> \ <span class="o">/</span> \ <span class="o">/</span> \ <span class="o">/</span> \ <span class="o">/</span> <span class="k">class</span> <span class="nc">D</span> </pre></div> </div> <p>will change into a diamond diagram under the new system:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span> <span class="nb">object</span><span class="p">:</span> <span class="o">^</span> <span class="o">^</span> <span class="fm">__getattr__</span><span class="p">()</span> <span class="o">/</span> \ <span class="o">/</span> \ <span class="o">/</span> \ <span class="o">/</span> \ <span class="k">class</span> <span class="nc">B</span> <span class="k">class</span> <span class="nc">C</span><span class="p">:</span> <span class="o">^</span> <span class="o">^</span> <span class="k">def</span> <span class="fm">__getattr__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span> <span class="o">...</span> \ <span class="o">/</span> \ <span class="o">/</span> \ <span class="o">/</span> \ <span class="o">/</span> <span class="k">class</span> <span class="nc">D</span> </pre></div> </div> <p>and while in the original diagram <code class="docutils literal notranslate"><span class="pre">C.__getattr__()</span></code> is invoked, under the new system with the classic lookup rule, <code class="docutils literal notranslate"><span class="pre">object.__getattr__()</span></code> would be invoked!</p> <p>Fortunately, there’s a lookup rule that’s better. It’s a bit difficult to explain, but it does the right thing in the diamond diagram, and it is the same as the classic lookup rule when there are no diamonds in the inheritance graph (when it is a tree).</p> <p>The new lookup rule constructs a list of all classes in the inheritance diagram in the order in which they will be searched. This construction is done at class definition time to save time. To explain the new lookup rule, let’s first consider what such a list would look like for the classic lookup rule. Note that in the presence of diamonds the classic lookup visits some classes multiple times. For example, in the ABCD diamond diagram above, the classic lookup rule visits the classes in this order:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">D</span><span class="p">,</span> <span class="n">B</span><span class="p">,</span> <span class="n">A</span><span class="p">,</span> <span class="n">C</span><span class="p">,</span> <span class="n">A</span> </pre></div> </div> <p>Note how A occurs twice in the list. The second occurrence is redundant, since anything that could be found there would already have been found when searching the first occurrence.</p> <p>We use this observation to explain our new lookup rule. Using the classic lookup rule, construct the list of classes that would be searched, including duplicates. Now for each class that occurs in the list multiple times, remove all occurrences except for the last. The resulting list contains each ancestor class exactly once (including the most derived class, D in the example).</p> <p>Searching for methods in this order will do the right thing for the diamond diagram. Because of the way the list is constructed, it does not change the search order in situations where no diamond is involved.</p> <p>Isn’t this backwards incompatible? Won’t it break existing code? It would, if we changed the method resolution order for all classes. However, in Python 2.2, the new lookup rule will only be applied to types derived from built-in types, which is a new feature. Class statements without a base class create “classic classes”, and so do class statements whose base classes are themselves classic classes. For classic classes the classic lookup rule will be used. (To experiment with the new lookup rule for classic classes, you will be able to specify a different metaclass explicitly.) We’ll also provide a tool that analyzes a class hierarchy looking for methods that would be affected by a change in method resolution order.</p> <p>XXX Another way to explain the motivation for the new MRO, due to Damian Conway: you never use the method defined in a base class if it is defined in a derived class that you haven’t explored yet (using the old search order).</p> </section> <section id="xxx-to-be-done"> <h2><a class="toc-backref" href="#xxx-to-be-done" role="doc-backlink">XXX To be done</a></h2> <p>Additional topics to be discussed in this PEP:</p> <ul class="simple"> <li>backwards compatibility issues!!!</li> <li>class methods and static methods</li> <li>cooperative methods and <code class="docutils literal notranslate"><span class="pre">super()</span></code></li> <li>mapping between type object slots (tp_foo) and special methods (<code class="docutils literal notranslate"><span class="pre">__foo__</span></code>) (actually, this may belong in <a class="pep reference internal" href="../pep-0252/" title="PEP 252 – Making Types Look More Like Classes">PEP 252</a>)</li> <li>built-in names for built-in types (object, int, str, list etc.)</li> <li><code class="docutils literal notranslate"><span class="pre">__dict__</span></code> and <code class="docutils literal notranslate"><span class="pre">__dictoffset__</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">__slots__</span></code></li> <li>the <code class="docutils literal notranslate"><span class="pre">HEAPTYPE</span></code> flag bit</li> <li>GC support</li> <li>API docs for all the new functions</li> <li>how to use <code class="docutils literal notranslate"><span class="pre">__new__</span></code></li> <li>writing metaclasses (using <code class="docutils literal notranslate"><span class="pre">mro()</span></code> etc.)</li> <li>high level user overview</li> </ul> <section id="open-issues"> <h3><a class="toc-backref" href="#open-issues" role="doc-backlink">open issues</a></h3> <ul class="simple"> <li>do we need <code class="docutils literal notranslate"><span class="pre">__del__</span></code>?</li> <li>assignment to <code class="docutils literal notranslate"><span class="pre">__dict__</span></code>, <code class="docutils literal notranslate"><span class="pre">__bases__</span></code></li> <li>inconsistent naming (e.g. tp_dealloc/tp_new/tp_init/tp_alloc/tp_free)</li> <li>add builtin alias ‘dict’ for ‘dictionary’?</li> <li>when subclasses of dict/list etc. are passed to system functions, the <code class="docutils literal notranslate"><span class="pre">__getitem__</span></code> overrides (etc.) aren’t always used</li> </ul> </section> </section> <section id="implementation"> <h2><a class="toc-backref" href="#implementation" role="doc-backlink">Implementation</a></h2> <p>A prototype implementation of this PEP (and for <a class="pep reference internal" href="../pep-0252/" title="PEP 252 – Making Types Look More Like Classes">PEP 252</a>) is available from CVS, and in the series of Python 2.2 alpha and beta releases. For some examples of the features described here, see the file Lib/test/test_descr.py and the extension module Modules/xxsubtype.c.</p> </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <aside class="footnote-list brackets"> <aside class="footnote brackets" id="id3" role="doc-footnote"> <dt class="label" id="id3">[1]<em> (<a href='#id1'>1</a>, <a href='#id2'>2</a>) </em></dt> <dd>“Putting Metaclasses to Work”, by Ira R. Forman and Scott H. Danforth, Addison-Wesley 1999. (<a class="reference external" href="http://www.aw.com/product/0,2627,0201433052,00.html">http://www.aw.com/product/0,2627,0201433052,00.html</a>)</aside> </aside> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> <p>This document has been placed in the public domain.</p> </section> </section> <hr class="docutils" /> <p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0253.rst">https://github.com/python/peps/blob/main/peps/pep-0253.rst</a></p> <p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0253.rst">2023-09-09 17:39:29 GMT</a></p> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#introduction">Introduction</a></li> <li><a class="reference internal" href="#about-metatypes">About metatypes</a></li> <li><a class="reference internal" href="#making-a-type-a-factory-for-its-instances">Making a type a factory for its instances</a></li> <li><a class="reference internal" href="#preparing-a-type-for-subtyping">Preparing a type for subtyping</a></li> <li><a class="reference internal" href="#creating-a-subtype-of-a-built-in-type-in-c">Creating a subtype of a built-in type in C</a></li> <li><a class="reference internal" href="#subtyping-in-python">Subtyping in Python</a></li> <li><a class="reference internal" href="#multiple-inheritance">Multiple inheritance</a></li> <li><a class="reference internal" href="#mro-method-resolution-order-the-lookup-rule">MRO: Method resolution order (the lookup rule)</a></li> <li><a class="reference internal" href="#xxx-to-be-done">XXX To be done</a><ul> <li><a class="reference internal" href="#open-issues">open issues</a></li> </ul> </li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <br> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0253.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>