CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 437 – A DSL for specifying signatures, annotations and argument converters | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0437/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 437 – A DSL for specifying signatures, annotations and argument converters | peps.python.org'> <meta property="og:description" content="The Python C-API currently has no mechanism for specifying and auto-generating function signatures, annotations or custom argument converters."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0437/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="The Python C-API currently has no mechanism for specifying and auto-generating function signatures, annotations or custom argument converters."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 437</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 437 – A DSL for specifying signatures, annotations and argument converters</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Stefan Krah <skrah at bytereef.org></dd> <dt class="field-even">Status:</dt> <dd class="field-even"><abbr title="Formally declined and will not be accepted">Rejected</abbr></dd> <dt class="field-odd">Type:</dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created:</dt> <dd class="field-even">11-Mar-2013</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">3.4</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even"></dd> <dt class="field-odd">Resolution:</dt> <dd class="field-odd"><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2013-May/126117.html">Python-Dev message</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rejection-notice">Rejection Notice</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#scope">Scope</a></li> <li><a class="reference internal" href="#dsl-overview">DSL overview</a><ul> <li><a class="reference internal" href="#type-safety-and-annotations">Type safety and annotations</a></li> <li><a class="reference internal" href="#include-converters-h">Include/converters.h</a></li> <li><a class="reference internal" href="#function-specifications">Function specifications</a><ul> <li><a class="reference internal" href="#keyword-arguments">Keyword arguments</a><ul> <li><a class="reference internal" href="#define-block">Define block</a></li> <li><a class="reference internal" href="#declaration">Declaration</a></li> <li><a class="reference internal" href="#c-declarations">C-declarations</a></li> <li><a class="reference internal" href="#cleanup">Cleanup</a></li> <li><a class="reference internal" href="#output">Output</a></li> </ul> </li> <li><a class="reference internal" href="#positional-only-arguments">Positional-only arguments</a></li> <li><a class="reference internal" href="#left-and-right-optional-arguments">Left and right optional arguments</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#flexibility-in-formatting">Flexibility in formatting</a></li> <li><a class="reference internal" href="#benefits-of-a-compact-notation">Benefits of a compact notation</a></li> <li><a class="reference internal" href="#easy-validation-of-the-definition">Easy validation of the definition</a></li> <li><a class="reference internal" href="#reference-implementation">Reference implementation</a></li> <li><a class="reference internal" href="#grammar">Grammar</a></li> <li><a class="reference internal" href="#comparison-with-pep-436">Comparison with PEP 436</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> The Python C-API currently has no mechanism for specifying and auto-generating function signatures, annotations or custom argument converters. There are several possible approaches to the problem. Cython uses cdef definitions in .pyx files to generate the required information. However, CPython’s C-API functions often require additional initialization and cleanup snippets that would be hard to specify in a cdef. <a class="pep reference internal" href="../pep-0436/" title="PEP 436 – The Argument Clinic DSL">PEP 436</a> proposes a domain specific language (DSL) enclosed in C comments that largely resembles a per-parameter configuration file. A preprocessor reads the comment and emits an argument parsing function, docstrings and a header for the function that utilizes the results of the parsing step. The latter function is subsequently referred to as the implementation function. </section> <section id="rejection-notice"> <h2><a class="toc-backref" href="#rejection-notice" role="doc-backlink">Rejection Notice</a></h2> This PEP was rejected by Guido van Rossum at PyCon US 2013. However, several of the specific issues raised by this PEP were taken into account when designing the <a class="reference external" href="http://hg.python.org/peps/rev/a2fa10b2424b">second iteration of the PEP 436 DSL</a>. </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> Opinions differ regarding the suitability of the <a class="pep reference internal" href="../pep-0436/" title="PEP 436 – The Argument Clinic DSL">PEP 436</a> DSL in the context of a C file. This PEP proposes an alternative DSL. The specific issues with <a class="pep reference internal" href="../pep-0436/" title="PEP 436 – The Argument Clinic DSL">PEP 436</a> that spurred the counter proposal will be explained in the final section of this PEP. </section> <section id="scope"> <h2><a class="toc-backref" href="#scope" role="doc-backlink">Scope</a></h2> The PEP focuses exclusively on the DSL. Topics like the output locations of docstrings or the generated code are outside the scope of this PEP. It is however vital that the DSL is suitable for generating custom argument parsers, a feature that is already implemented in Cython. Therefore, one of the goals of this PEP is to keep the DSL close to existing solutions, thus facilitating a possible inclusion of the relevant parts of Cython into the CPython source tree. </section> <section id="dsl-overview"> <h2><a class="toc-backref" href="#dsl-overview" role="doc-backlink">DSL overview</a></h2> <section id="type-safety-and-annotations"> <h3><a class="toc-backref" href="#type-safety-and-annotations" role="doc-backlink">Type safety and annotations</a></h3> A conversion from a Python to a C value is fully defined by the type of the converter function. The PyArg_Parse* family of functions accepts custom converters in addition to the well-known default converters “i”, “f”, etc. This PEP views the default converters as abstract functions, regardless of how they are actually implemented. </section> <section id="include-converters-h"> <h3><a class="toc-backref" href="#include-converters-h" role="doc-backlink">Include/converters.h</a></h3> Converter functions must be forward-declared. All converter functions shall be entered into the file Include/converters.h. The file is read by the preprocessor prior to translating .c files. This is an excerpt: <div class="highlight-default notranslate"><div class="highlight"><pre>/*[converter] ##### Default converters ##### "s": str -> const char *res; "s*": [str, bytes, bytearray, rw_buffer] -> Py_buffer &res; [...] "es#": str -> (const char *res_encoding, char **res, Py_ssize_t *res_length); [...] ##### Custom converters ##### path_converter: [str, bytes, int] -> path_t &res; OS_STAT_DIR_FD_CONVERTER: [int, None] -> int res; [converter_end]*/ </pre></div> </div> Converters are specified by their name, Python input type(s) and C output type(s). Default converters must have quoted names, custom converters must have regular names. A Python type is given by its name. If a function accepts multiple Python types, the set is written in list form. Since the default converters may have multiple implicit return values, the C output type(s) are written according to the following convention: The main return value must be named res. This is a placeholder for the actual variable name given later in the DSL. Additional implicit return values must be prefixed by res_. By default the variables are passed by value to the implementation function. If the address should be passed instead, res must be prefixed with an ampersand. Additional declarations may be placed into .c files. Duplicate declarations are allowed as long as the function types are identical. It is encouraged to declare custom converter types a second time right above the converter function definition. The preprocessor will then catch any mismatch between the declarations. In order to keep the converter complexity manageable, PY_SSIZE_T_CLEAN will be deprecated and Py_ssize_t will be assumed for all length arguments. TBD: Make a list of fantasy types like rw_buffer. </section> <section id="function-specifications"> <h3><a class="toc-backref" href="#function-specifications" role="doc-backlink">Function specifications</a></h3> <section id="keyword-arguments"> <h4><a class="toc-backref" href="#keyword-arguments" role="doc-backlink">Keyword arguments</a></h4> This example contains the definition of os.stat. The individual sections will be explained in detail. Grammatically, the whole define block consists of a function specification and an output section. The function specification in turn consists of a declaration section, an optional C-declaration section and an optional cleanup code section. Sections within the function specification are separated in yacc style by ‘%%’: <div class="highlight-default notranslate"><div class="highlight"><pre>/*[define posix_stat] def os.stat(path: path_converter, *, dir_fd: OS_STAT_DIR_FD_CONVERTER = None, follow_symlinks: "p" = True) -> os.stat_result: pass %% path_t path = PATH_T_INITIALIZE("stat", 0, 1); int dir_fd = DEFAULT_DIR_FD; int follow_symlinks = 1; %% path_cleanup(&path); [define_end]*/ <literal C output> /*[define_output_end]*/ </pre></div> </div> <section id="define-block"> <h5><a class="toc-backref" href="#define-block" role="doc-backlink">Define block</a></h5> The function specification block starts with a <code class="docutils literal notranslate">/*[define</code> token, followed by an optional C function name, followed by a right bracket. If the C function name is not given, it is generated from the declaration name. In the example, omitting the name posix_stat would result in a C function name of os_stat. </section> <section id="declaration"> <h5><a class="toc-backref" href="#declaration" role="doc-backlink">Declaration</a></h5> The required declaration is (almost) a valid Python function definition. The ‘def’ keyword and the function body are redundant, but the author of this PEP finds the definition more readable if they are present. The function name may be a path instead of a plain identifier. Each argument is annotated with the name of the converter function that will be applied to it. Default values are given in the usual Python manner and may be any valid Python expression. The return value may be any Python expression. Usually it will be the name of an object, but alternative return values could be specified in list form. </section> <section id="c-declarations"> <h5><a class="toc-backref" href="#c-declarations" role="doc-backlink">C-declarations</a></h5> This optional section contains C variable declarations. Since the converter functions have been declared beforehand, the preprocessor can type-check the declarations. </section> <section id="cleanup"> <h5><a class="toc-backref" href="#cleanup" role="doc-backlink">Cleanup</a></h5> The optional cleanup section contains literal C code that will be inserted unmodified after the implementation function. </section> <section id="output"> <h5><a class="toc-backref" href="#output" role="doc-backlink">Output</a></h5> The output section contains the code emitted by the preprocessor. </section> </section> <section id="positional-only-arguments"> <h4><a class="toc-backref" href="#positional-only-arguments" role="doc-backlink">Positional-only arguments</a></h4> Functions that do not take keyword arguments are indicated by the presence of the slash special parameter: <div class="highlight-default notranslate"><div class="highlight"><pre>/*[define stat_float_times] def os.stat_float_times(/, newval: "i") -> os.stat_result: pass %% int newval = -1; [define_end]*/ </pre></div> </div> The preprocessor translates this definition to a PyArg_ParseTuple() call. All arguments to the right of the slash are optional arguments. </section> <section id="left-and-right-optional-arguments"> <h4><a class="toc-backref" href="#left-and-right-optional-arguments" role="doc-backlink">Left and right optional arguments</a></h4> Some legacy functions contain optional arguments groups both to the left and right of a central parameter. It is debatable whether a new tool should support such functions. For completeness’ sake, this is the proposed syntax: <div class="highlight-default notranslate"><div class="highlight"><pre>/*[define] def curses.window.addch(y: "i", x: "i", ch: "O", attr: "l") -> None: pass where groups = [[ch], [ch, attr], [y, x, ch], [y, x, ch, attr]] [define_end]*/ </pre></div> </div> Here ch is the central parameter, attr can optionally be added on the right, and the group [y, x] can optionally be added on the left. Essentially the rule is that all ordered combinations of the central parameter and the optional groups must be possible such that no two combinations have the same length. This is concisely expressed by putting the central parameter first in the list and subsequently adding the optional arguments groups to the left and right. </section> </section> </section> <section id="flexibility-in-formatting"> <h2><a class="toc-backref" href="#flexibility-in-formatting" role="doc-backlink">Flexibility in formatting</a></h2> If the above os.stat example is considered too compact, it can easily be formatted this way: <div class="highlight-default notranslate"><div class="highlight"><pre>/*[define posix_stat] def os.stat(path: path_converter, *, dir_fd: OS_STAT_DIR_FD_CONVERTER = None, follow_symlinks: "p" = True) -> os.stat_result: pass %% path_t path = PATH_T_INITIALIZE("stat", 0, 1); int dir_fd = DEFAULT_DIR_FD; int follow_symlinks = 1; %% path_cleanup(&path); [define_end]*/ <literal C output> /*[define_output_end]*/ </pre></div> </div> </section> <section id="benefits-of-a-compact-notation"> <h2><a class="toc-backref" href="#benefits-of-a-compact-notation" role="doc-backlink">Benefits of a compact notation</a></h2> The advantages of a concise notation are especially obvious when a large number of parameters is involved. The argument parsing part of <code class="docutils literal notranslate">_posixsubprocess.fork_exec</code> is fully specified by this definition: <div class="highlight-default notranslate"><div class="highlight"><pre>/*[define subprocess_fork_exec] def _posixsubprocess.fork_exec( process_args: "O", executable_list: "O", close_fds: "p", py_fds_to_keep: "O", cwd_obj: "O", env_list: "O", p2cread: "i", p2cwrite: "i", c2pread: "i", c2pwrite: "i", errread: "i", errwrite: "i", errpipe_read: "i", errpipe_write: "i", restore_signals: "i", call_setsid: "i", preexec_fn: "i", /) -> int: pass [define_end]*/ </pre></div> </div> Note that the preprocess tool currently emits a redundant C-declaration section for this example, so the output is longer than necessary. </section> <section id="easy-validation-of-the-definition"> <h2><a class="toc-backref" href="#easy-validation-of-the-definition" role="doc-backlink">Easy validation of the definition</a></h2> How can an inexperienced user validate a definition like os.stat? Simply by changing os.stat to os_stat, defining missing converters and pasting the definition into the Python interactive interpreter! In fact, a converters.py module could be auto-generated from converters.h. </section> <section id="reference-implementation"> <h2><a class="toc-backref" href="#reference-implementation" role="doc-backlink">Reference implementation</a></h2> A reference implementation is available at <a class="reference external" href="http://bugs.python.org/issue16612">issue 16612</a>. Since this PEP was written under time constraints and the author is unfamiliar with the PLY toolchain, the software is written in Standard ML and utilizes the ml-yacc/ml-lex toolchain. The grammar is conflict-free and available in ml-yacc readable BNF form. Two tools are available: <ul class="simple"> <li>printsemant reads a converter header and a .c file and dumps the semantically checked parse tree to stdout.</li> <li>preprocess reads a converter header and a .c file and dumps the preprocessed .c file to stdout.</li> </ul> Known deficiencies: <ul class="simple"> <li>The Python ‘test’ expression is not semantically checked. The syntax however is checked since it is part of the grammar.</li> <li>The lexer does not handle triple quoted strings.</li> <li>C declarations are parsed in a primitive way. The final implementation should utilize ‘declarator’ and ‘init-declarator’ from the C grammar.</li> <li>The preprocess tool does not emit code for the left-and-right optional arguments case. The printsemant tool can deal with this case.</li> <li>Since the preprocess tool generates the output from the parse tree, the original indentation of the define block is lost.</li> </ul> </section> <section id="grammar"> <h2><a class="toc-backref" href="#grammar" role="doc-backlink">Grammar</a></h2> <blockquote> <div>TBD: The grammar exists in ml-yacc readable form, but should probably be included here in EBNF notation.</div></blockquote> </section> <section id="comparison-with-pep-436"> <h2><a class="toc-backref" href="#comparison-with-pep-436" role="doc-backlink">Comparison with PEP 436</a></h2> The author of this PEP has the following concerns about the DSL proposed in <a class="pep reference internal" href="../pep-0436/" title="PEP 436 – The Argument Clinic DSL">PEP 436</a>: <ul> <li>The whitespace sensitive configuration file like syntax looks out of place in a C file.</li> <li>The structure of the function definition gets lost in the per-parameter specifications. Keywords like positional-only, required and keyword-only are scattered across too many different places.By contrast, in the alternative DSL the structure of the function definition can be understood at a single glance. </li> <li>The <a class="pep reference internal" href="../pep-0436/" title="PEP 436 – The Argument Clinic DSL">PEP 436</a> DSL has 14 documented flags and at least one undocumented (allow_fd) flag. Figuring out which of the 2**15 possible combinations are valid places an unnecessary burden on the user.Experience with the <a class="pep reference internal" href="../pep-3118/" title="PEP 3118 – Revising the buffer protocol">PEP 3118</a> buffer flags has shown that sorting out (and exhaustively testing!) valid combinations is an extremely tedious task. The <a class="pep reference internal" href="../pep-3118/" title="PEP 3118 – Revising the buffer protocol">PEP 3118</a> flags are still not well understood by many people. By contrast, the alternative DSL has a central file Include/converters.h that can be quickly searched for the desired converter. Many of the converters are already known, perhaps even memorized by people (due to frequent use). </li> <li>The <a class="pep reference internal" href="../pep-0436/" title="PEP 436 – The Argument Clinic DSL">PEP 436</a> DSL allows too much freedom. Types can apparently be omitted, the preprocessor accepts (and ignores) unknown keywords, sometimes adding white space after a docstring results in an assertion error.The alternative DSL on the other hand allows no such freedoms. Omitting converter or return value annotations is plainly a syntax error. The LALR(1) grammar is unambiguous and specified for the complete translation unit. </li> </ul> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document is licensed under the <a class="reference external" href="http://www.opencontent.org/openpub/">Open Publication License</a>. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0437.rst">https://github.com/python/peps/blob/main/peps/pep-0437.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0437.rst">2025-02-01 08:59:27 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rejection-notice">Rejection Notice</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#scope">Scope</a></li> <li><a class="reference internal" href="#dsl-overview">DSL overview</a><ul> <li><a class="reference internal" href="#type-safety-and-annotations">Type safety and annotations</a></li> <li><a class="reference internal" href="#include-converters-h">Include/converters.h</a></li> <li><a class="reference internal" href="#function-specifications">Function specifications</a><ul> <li><a class="reference internal" href="#keyword-arguments">Keyword arguments</a><ul> <li><a class="reference internal" href="#define-block">Define block</a></li> <li><a class="reference internal" href="#declaration">Declaration</a></li> <li><a class="reference internal" href="#c-declarations">C-declarations</a></li> <li><a class="reference internal" href="#cleanup">Cleanup</a></li> <li><a class="reference internal" href="#output">Output</a></li> </ul> </li> <li><a class="reference internal" href="#positional-only-arguments">Positional-only arguments</a></li> <li><a class="reference internal" href="#left-and-right-optional-arguments">Left and right optional arguments</a></li> </ul> </li> </ul> </li> <li><a class="reference internal" href="#flexibility-in-formatting">Flexibility in formatting</a></li> <li><a class="reference internal" href="#benefits-of-a-compact-notation">Benefits of a compact notation</a></li> <li><a class="reference internal" href="#easy-validation-of-the-definition">Easy validation of the definition</a></li> <li><a class="reference internal" href="#reference-implementation">Reference implementation</a></li> <li><a class="reference internal" href="#grammar">Grammar</a></li> <li><a class="reference internal" href="#comparison-with-pep-436">Comparison with PEP 436</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0437.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>