CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 501 – General purpose template literal strings | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0501/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 501 – General purpose template literal strings | peps.python.org'> <meta property="og:description" content="Though easy and elegant to use, Python f-strings can be vulnerable to injection attacks when used to construct shell commands, SQL queries, HTML snippets and similar (for example, os.system(f"echo {message_from_user}")). This PEP introduces template lit..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0501/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="Though easy and elegant to use, Python f-strings can be vulnerable to injection attacks when used to construct shell commands, SQL queries, HTML snippets and similar (for example, os.system(f"echo {message_from_user}")). This PEP introduces template lit..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 501</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 501 – General purpose template literal strings</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Alyssa Coghlan <ncoghlan at gmail.com>, Nick Humrich <nick at humrich.us></dd> <dt class="field-even">Discussions-To:</dt> <dd class="field-even"><a class="reference external" href="https://discuss.python.org/t/pep-501-reopen-general-purpose-string-template-literals/24625">Discourse thread</a></dd> <dt class="field-odd">Status:</dt> <dd class="field-odd"><abbr title="Removed from consideration by sponsor or authors">Withdrawn</abbr></dd> <dt class="field-even">Type:</dt> <dd class="field-even"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-odd">Requires:</dt> <dd class="field-odd"><a class="reference external" href="../pep-0701/">701</a></dd> <dt class="field-even">Created:</dt> <dd class="field-even">08-Aug-2015</dd> <dt class="field-odd">Python-Version:</dt> <dd class="field-odd">3.12</dd> <dt class="field-even">Post-History:</dt> <dd class="field-even"><a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/EAZ3P2M3CDDIQFR764NF6FXQHWXYMKJF/" title="Python-Dev thread">08-Aug-2015</a>, <a class="reference external" href="https://mail.python.org/archives/list/python-dev@python.org/thread/ILVRPS6DTFZ7IHL5HONDBB6INVXTFOZ2/" title="Python-Dev thread">05-Sep-2015</a>, <a class="reference external" href="https://discuss.python.org/t/pep-501-reopen-general-purpose-string-template-literals/24625" title="Discourse thread">09-Mar-2023</a></dd> <dt class="field-odd">Superseded-By:</dt> <dd class="field-odd"><a class="reference external" href="../pep-0750/">750</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#pep-withdrawal">PEP Withdrawal</a></li> <li><a class="reference internal" href="#relationship-with-other-peps">Relationship with other PEPs</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#proposal">Proposal</a><ul> <li><a class="reference internal" href="#dedicated-template-literal-syntax">Dedicated template literal syntax</a></li> <li><a class="reference internal" href="#lazy-field-evaluation-conversion-specifier">Lazy field evaluation conversion specifier</a></li> <li><a class="reference internal" href="#custom-conversion-specifiers">Custom conversion specifiers</a></li> <li><a class="reference internal" href="#template-renderer-for-posix-shell-commands">Template renderer for POSIX shell commands</a></li> </ul> </li> <li><a class="reference internal" href="#background">Background</a><ul> <li><a class="reference internal" href="#summary-of-differences-from-f-strings">Summary of differences from f-strings</a></li> <li><a class="reference internal" href="#summary-of-differences-from-tagged-strings">Summary of differences from tagged strings</a></li> </ul> </li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#rendering-templates">Rendering templates</a></li> <li><a class="reference internal" href="#format-specifiers">Format specifiers</a></li> <li><a class="reference internal" href="#conversion-specifiers">Conversion specifiers</a></li> <li><a class="reference internal" href="#new-field-conversion-api-in-the-operator-module">New field conversion API in the <a class="reference external" href="https://docs.python.org/3/library/operator.html#module-operator" title="(in Python v3.13)"><code class="xref py py-mod docutils literal notranslate">operator</code></a> module</a></li> <li><a class="reference internal" href="#conversion-specifier-parameter-added-to-format">Conversion specifier parameter added to <a class="reference external" href="https://docs.python.org/3/library/functions.html#format" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">format()</code></a></a></li> <li><a class="reference internal" href="#structural-typing-and-duck-typing">Structural typing and duck typing</a></li> <li><a class="reference internal" href="#writing-custom-renderers">Writing custom renderers</a></li> <li><a class="reference internal" href="#expression-evaluation">Expression evaluation</a></li> <li><a class="reference internal" href="#handling-code-injection-attacks">Handling code injection attacks</a></li> <li><a class="reference internal" href="#error-handling">Error handling</a></li> <li><a class="reference internal" href="#renderer-for-shell-escaping-added-to-shlex">Renderer for shell escaping added to <a class="reference external" href="https://docs.python.org/3/library/shlex.html#module-shlex" title="(in Python v3.13)"><code class="xref py py-mod docutils literal notranslate">shlex</code></a></a></li> <li><a class="reference internal" href="#changes-to-subprocess-module">Changes to subprocess module</a></li> </ul> </li> <li><a class="reference internal" href="#how-to-teach-this">How to Teach This</a></li> <li><a class="reference internal" href="#discussion">Discussion</a><ul> <li><a class="reference internal" href="#support-for-binary-interpolation">Support for binary interpolation</a></li> <li><a class="reference internal" href="#interoperability-with-str-only-interfaces">Interoperability with str-only interfaces</a></li> <li><a class="reference internal" href="#preserving-the-raw-template-string">Preserving the raw template string</a></li> <li><a class="reference internal" href="#creating-a-rich-object-rather-than-a-global-name-lookup">Creating a rich object rather than a global name lookup</a></li> <li><a class="reference internal" href="#building-atop-f-strings-rather-than-replacing-them">Building atop f-strings rather than replacing them</a></li> <li><a class="reference internal" href="#defining-repetition-and-concatenation-semantics">Defining repetition and concatenation semantics</a></li> <li><a class="reference internal" href="#new-conversion-specifier-for-lazy-field-evaluation">New conversion specifier for lazy field evaluation</a></li> <li><a class="reference internal" href="#allowing-arbitrary-conversion-specifiers-in-custom-renderers">Allowing arbitrary conversion specifiers in custom renderers</a></li> <li><a class="reference internal" href="#only-reserving-a-single-new-string-prefix">Only reserving a single new string prefix</a></li> <li><a class="reference internal" href="#deferring-consideration-of-more-concise-delayed-evaluation-syntax">Deferring consideration of more concise delayed evaluation syntax</a></li> <li><a class="reference internal" href="#deferring-consideration-of-possible-logging-integration">Deferring consideration of possible logging integration</a></li> <li><a class="reference internal" href="#deferring-consideration-of-possible-use-in-i18n-use-cases">Deferring consideration of possible use in i18n use cases</a></li> <li><a class="reference internal" href="#deferring-escaped-rendering-support-for-non-posix-shells">Deferring escaped rendering support for non-POSIX shells</a></li> </ul> </li> <li><a class="reference internal" href="#acknowledgements">Acknowledgements</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <div class="pep-banner sticky-banner deprecated superseded admonition important"> Important This PEP has been superseded by <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a>. × </div> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> Though easy and elegant to use, Python <a class="reference external" href="https://docs.python.org/3/glossary.html#term-f-string" title="(in Python v3.13)">f-string</a>s can be vulnerable to injection attacks when used to construct shell commands, SQL queries, HTML snippets and similar (for example, <code class="docutils literal notranslate">os.system(f"echo {message_from_user}")</code>). This PEP introduces template literal strings (or “t-strings”), which have syntax and semantics that are similar to f-strings, but with rendering deferred until <a class="reference external" href="https://docs.python.org/3/library/functions.html#format" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">format()</code></a> or another template rendering function is called on them. This will allow standard library calls, helper functions and third party tools to safety and intelligently perform appropriate escaping and other string processing on inputs while retaining the usability and convenience of f-strings. </section> <section id="pep-withdrawal"> <h2><a class="toc-backref" href="#pep-withdrawal" role="doc-backlink">PEP Withdrawal</a></h2> When <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a> was first published as a “tagged strings” proposal (allowing for arbitrary string prefixes), this PEP was kept open to continue championing the simpler “template literal” approach that used a single dedicated string prefix to produce instances of a new “interpolation template” type. The <a class="reference external" href="https://github.com/python/peps/pull/4062">October 2024 updates</a> to <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a> agreed that template strings were a better fit for Python than the broader tagged strings concept. All of the other concerns the authors of this PEP had with <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a> were also either addressed in those updates, or else left in a state where they could reasonably be addressed in a future change proposal. Due to the clear improvements in the updated <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a> proposal, this PEP has been withdrawn in favour of <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a>. <div class="admonition important"> Important The remainder of this PEP still reflects the state of the tagged strings proposal in August 2024. It has not been updated to reflect the October 2024 changes to <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a>, since the PEP withdrawal makes doing so redundant. </div> </section> <section id="relationship-with-other-peps"> <h2><a class="toc-backref" href="#relationship-with-other-peps" role="doc-backlink">Relationship with other PEPs</a></h2> This PEP is inpired by and builds on top of the f-string syntax first implemented in <a class="pep reference internal" href="../pep-0498/" title="PEP 498 – Literal String Interpolation">PEP 498</a> and formalised in <a class="pep reference internal" href="../pep-0701/" title="PEP 701 – Syntactic formalization of f-strings">PEP 701</a>. This PEP complements the literal string typing support added to Python’s formal type system in <a class="pep reference internal" href="../pep-0675/" title="PEP 675 – Arbitrary Literal String Type">PEP 675</a> by introducing a safe way to do dynamic interpolation of runtime values into security sensitive strings. This PEP competes with some aspects of the tagged string proposal in <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a> (most notably in whether template rendering is expressed as <code class="docutils literal notranslate">render(t"template literal")</code> or as <code class="docutils literal notranslate">render"template literal"</code>), but also shares many common features (after <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a> was published, this PEP was updated with <a class="reference external" href="https://github.com/python/peps/issues/3904">several new changes</a> inspired by the tagged strings proposal). This PEP does NOT propose an alternative to <a class="pep reference internal" href="../pep-0292/" title="PEP 292 – Simpler String Substitutions">PEP 292</a> for user interface internationalization use cases (but does note the potential for future syntactic enhancements aimed at that use case that would benefit from the compiler-supported value interpolation machinery that this PEP and <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a> introduce). </section> <section id="motivation"> <h2><a class="toc-backref" href="#motivation" role="doc-backlink">Motivation</a></h2> <a class="pep reference internal" href="../pep-0498/" title="PEP 498 – Literal String Interpolation">PEP 498</a> added new syntactic support for string interpolation that is transparent to the compiler, allowing name references from the interpolation operation full access to containing namespaces (as with any other expression), rather than being limited to explicit name references. These are referred to in the PEP (and elsewhere) as “f-strings” (a mnemonic for “formatted strings”). Since acceptance of <a class="pep reference internal" href="../pep-0498/" title="PEP 498 – Literal String Interpolation">PEP 498</a>, f-strings have become well-established and very popular. f-strings became even more useful and flexible with the formalised grammar in <a class="pep reference internal" href="../pep-0701/" title="PEP 701 – Syntactic formalization of f-strings">PEP 701</a>. While f-strings are great, eager rendering has its limitations. For example, the eagerness of f-strings has made code like the following unfortunately plausible: <div class="highlight-python notranslate"><div class="highlight"><pre>os.system(f"echo {message_from_user}") </pre></div> </div> This kind of code is superficially elegant, but poses a significant problem if the interpolated value <code class="docutils literal notranslate">message_from_user</code> is in fact provided by an untrusted user: it’s an opening for a form of code injection attack, where the supplied user data has not been properly escaped before being passed to the <code class="docutils literal notranslate">os.system</code> call. While the <code class="docutils literal notranslate">LiteralString</code> type annotation introduced in <a class="pep reference internal" href="../pep-0675/" title="PEP 675 – Arbitrary Literal String Type">PEP 675</a> means that typecheckers are able to report a type error for this kind of unsafe function usage, those errors don’t help make it easier to write code that uses safer alternatives (such as <a class="reference external" href="https://docs.python.org/3/library/subprocess.html#subprocess.run" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">subprocess.run()</code></a>). To address that problem (and a number of other concerns), this PEP proposes the complementary introduction of “t-strings” (a mnemonic for “template literal strings”), where <code class="docutils literal notranslate">format(t"Message with {data}")</code> would produce the same result as <code class="docutils literal notranslate">f"Message with {data}"</code>, but the template literal instance can instead be passed to other template rendering functions which process the contents of the template differently. </section> <section id="proposal"> <h2><a class="toc-backref" href="#proposal" role="doc-backlink">Proposal</a></h2> <section id="dedicated-template-literal-syntax"> <h3><a class="toc-backref" href="#dedicated-template-literal-syntax" role="doc-backlink">Dedicated template literal syntax</a></h3> This PEP proposes a new string prefix that declares the string to be a template literal rather than an ordinary string: <div class="highlight-python notranslate"><div class="highlight"><pre>template = t"Substitute {names:>{field_width}} and {expressions()!r} at runtime" </pre></div> </div> This would be effectively interpreted as: <div class="highlight-python notranslate"><div class="highlight"><pre>template = TemplateLiteral( r"Substitute {names:>{field_width}} and {expressions()} at runtime", TemplateLiteralText(r"Substitute "), TemplateLiteralField("names", names, f">{field_width}", ""), TemplateLiteralText(r" and "), TemplateLiteralField("expressions()", expressions(), f"", "r"), ) </pre></div> </div> (Note: this is an illustrative example implementation. The exact compile time construction syntax of <code class="docutils literal notranslate">types.TemplateLiteral</code> is considered an implementation detail not specified by the PEP. In particular, the compiler may bypass the default constructor’s runtime logic that detects consecutive text segments and merges them into a single text segment, as well as checking the runtime types of all supplied arguments). The <code class="docutils literal notranslate">__format__</code> method on <code class="docutils literal notranslate">types.TemplateLiteral</code> would then implement the following <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str.format" title="(in Python v3.13)"><code class="xref py py-meth docutils literal notranslate">str.format()</code></a> inspired semantics: <div class="highlight-python-console notranslate"><div class="highlight"><pre>>>> import datetime >>> name = 'Jane' >>> age = 50 >>> anniversary = datetime.date(1991, 10, 12) >>> format(t'My name is {name}, my age next year is {age+1}, my anniversary is {anniversary:%A, %B %d, %Y}.') 'My name is Jane, my age next year is 51, my anniversary is Saturday, October 12, 1991.' >>> format(t'She said her name is {name!r}.') "She said her name is 'Jane'." </pre></div> </div> The syntax of template literals would be based on <a class="pep reference internal" href="../pep-0701/" title="PEP 701 – Syntactic formalization of f-strings">PEP 701</a>, and largely use the same syntax for the string portion of the template. Aside from using a different prefix, the one other syntactic change is in the definition and handling of conversion specifiers, both to allow <code class="docutils literal notranslate">!()</code> as a standard conversion specifier to request evaluation of a field at rendering time, and to allow custom renderers to also define custom conversion specifiers. This PEP does not propose to remove or deprecate any of the existing string formatting mechanisms, as those will remain valuable when formatting strings that are not present directly in the source code of the application. </section> <section id="lazy-field-evaluation-conversion-specifier"> <h3><a class="toc-backref" href="#lazy-field-evaluation-conversion-specifier" role="doc-backlink">Lazy field evaluation conversion specifier</a></h3> In addition to the existing support for the <code class="docutils literal notranslate">a</code>, <code class="docutils literal notranslate">r</code>, and <code class="docutils literal notranslate">s</code> conversion specifiers, <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str.format" title="(in Python v3.13)"><code class="xref py py-meth docutils literal notranslate">str.format()</code></a>, <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str.format_map" title="(in Python v3.13)"><code class="xref py py-meth docutils literal notranslate">str.format_map()</code></a>, and <a class="reference external" href="https://docs.python.org/3/library/string.html#string.Formatter" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">string.Formatter</code></a> will be updated to accept <code class="docutils literal notranslate">()</code> as a conversion specifier that means “call the interpolated value”. To support application of the standard conversion specifiers in custom template rendering functions, a new <code class="xref py py-func docutils literal notranslate">operator.convert_field()</code> function will be added. The signature and behaviour of the <a class="reference external" href="https://docs.python.org/3/library/functions.html#format" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">format()</code></a> builtin will also be updated to accept a conversion specifier as a third optional parameter. If a non-empty conversion specifier is given, the value will be converted with <code class="xref py py-func docutils literal notranslate">operator.convert_field()</code> before looking up the <code class="docutils literal notranslate">__format__</code> method. </section> <section id="custom-conversion-specifiers"> <h3><a class="toc-backref" href="#custom-conversion-specifiers" role="doc-backlink">Custom conversion specifiers</a></h3> To allow additional field-specific directives to be passed to custom rendering functions in a way that still allows formatting of the template with the default renderer, the conversion specifier field will be allowed to contain a second <code class="docutils literal notranslate">!</code> character. <code class="xref py py-func docutils literal notranslate">operator.convert_field()</code> and <a class="reference external" href="https://docs.python.org/3/library/functions.html#format" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">format()</code></a> (and hence the default <code class="docutils literal notranslate">TemplateLiteral.render</code> template rendering method), will ignore that character and any subsequent text in the conversion specifier field. <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str.format" title="(in Python v3.13)"><code class="xref py py-meth docutils literal notranslate">str.format()</code></a>, <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str.format_map" title="(in Python v3.13)"><code class="xref py py-meth docutils literal notranslate">str.format_map()</code></a>, and <a class="reference external" href="https://docs.python.org/3/library/string.html#string.Formatter" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">string.Formatter</code></a> will also be updated to accept (and ignore) custom conversion specifiers. </section> <section id="template-renderer-for-posix-shell-commands"> <h3><a class="toc-backref" href="#template-renderer-for-posix-shell-commands" role="doc-backlink">Template renderer for POSIX shell commands</a></h3> As both a practical demonstration of the benefits of delayed rendering support, and as a valuable feature in its own right, a new <code class="docutils literal notranslate">sh</code> template renderer will be added to the <a class="reference external" href="https://docs.python.org/3/library/shlex.html#module-shlex" title="(in Python v3.13)"><code class="xref py py-mod docutils literal notranslate">shlex</code></a> module. This renderer will produce strings where all interpolated fields are escaped with <a class="reference external" href="https://docs.python.org/3/library/shlex.html#shlex.quote" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">shlex.quote()</code></a>. The <a class="reference external" href="https://docs.python.org/3/library/subprocess.html#subprocess.Popen" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">subprocess.Popen</code></a> API (and higher level APIs that depend on it, such as <a class="reference external" href="https://docs.python.org/3/library/subprocess.html#subprocess.run" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">subprocess.run()</code></a>) will be updated to accept interpolation templates and handle them in accordance with the new <code class="docutils literal notranslate">shlex.sh</code> renderer. </section> </section> <section id="background"> <h2><a class="toc-backref" href="#background" role="doc-backlink">Background</a></h2> This PEP was initially proposed as a competitor to <a class="pep reference internal" href="../pep-0498/" title="PEP 498 – Literal String Interpolation">PEP 498</a>. After it became clear that the eager rendering proposal had sustantially more immediate support, it then spent several years in a deferred state, pending further experience with <a class="pep reference internal" href="../pep-0498/" title="PEP 498 – Literal String Interpolation">PEP 498</a>’s simpler approach of only supporting eager rendering without the additional complexity of also supporting deferred rendering. Since then, f-strings have become very popular and <a class="pep reference internal" href="../pep-0701/" title="PEP 701 – Syntactic formalization of f-strings">PEP 701</a> was introduced to tidy up some rough edges and limitations in their syntax and semantics. The template literal proposal was updated in 2023 to reflect current knowledge of f-strings, and improvements from <a class="pep reference internal" href="../pep-0701/" title="PEP 701 – Syntactic formalization of f-strings">PEP 701</a>. In 2024, <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a> was published, proposing a general purpose mechanism for custom tagged string prefixes, rather than the narrower template literal proposal in this PEP. This PEP was again updated, both to incorporate new ideas inspired by the tagged strings proposal, and to describe the perceived benefits of the narrower template literal syntax proposal in this PEP over the more general tagged string proposal. <section id="summary-of-differences-from-f-strings"> <h3><a class="toc-backref" href="#summary-of-differences-from-f-strings" role="doc-backlink">Summary of differences from f-strings</a></h3> The key differences between f-strings and t-strings are: <ul class="simple"> <li>the <code class="docutils literal notranslate">t</code> (template literal) prefix indicates delayed rendering, but otherwise largely uses the same syntax and semantics as formatted strings</li> <li>template literals are available at runtime as a new kind of object (<code class="docutils literal notranslate">types.TemplateLiteral</code>)</li> <li>the default rendering used by formatted strings is invoked on a template literal object by calling <code class="docutils literal notranslate">format(template)</code> rather than being done implicitly in the compiled code</li> <li>unlike f-strings (where conversion specifiers are handled directly in the compiler), t-string conversion specifiers are handled at rendering time by the rendering function</li> <li>the new <code class="docutils literal notranslate">!()</code> conversion specifier indicates that the field expression is a callable that should be called when using the default <a class="reference external" href="https://docs.python.org/3/library/functions.html#format" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">format()</code></a> rendering function. This specifier is specifically not being added to f-strings (since it is pointless there).</li> <li>a second <code class="docutils literal notranslate">!</code> is allowed in t-string conversion specifiers (with any subsequent text being ignored) as a way to allow custom template rendering functions to accept custom conversion specifiers without breaking the default <code class="xref py py-func docutils literal notranslate">TemplateLiteral.render()</code> rendering method. This feature is specifically not being added to f-strings (since it is pointless there).</li> <li>while f-string <code class="docutils literal notranslate">f"Message {here}"</code> would be semantically equivalent to <code class="docutils literal notranslate">format(t"Message {here}")</code>, f-strings will continue to be supported directly in the compiler and hence avoid the runtime overhead of actually using the delayed rendering machinery that is needed for t-strings</li> </ul> </section> <section id="summary-of-differences-from-tagged-strings"> <h3><a class="toc-backref" href="#summary-of-differences-from-tagged-strings" role="doc-backlink">Summary of differences from tagged strings</a></h3> When tagged strings were <a class="reference external" href="https://discuss.python.org/t/pep-750-tag-strings-for-writing-domain-specific-languages/60408">first proposed</a>, there were several notable differences from the proposal in PEP 501 beyond the surface syntax difference between whether rendering function invocations are written as <code class="docutils literal notranslate">render(t"template literal")</code> or as <code class="docutils literal notranslate">render"template literal"</code>. Over the course of the initial PEP 750 discussion, many of those differences were eliminated, either by PEP 501 adopting that aspect of PEP 750’s proposal (such as lazily applying conversion specifiers), or by PEP 750 changing to retain some aspect of PEP 501’s proposal (such as defining a dedicated type to hold template segments rather than representing them as simple sequences). The main remaining significant difference is that this PEP argues that adding only the t-string prefix is a sufficient enhancement to give all the desired benefits described in PEP 750. The expansion to a generalised “tagged string” syntax isn’t necessary, and causes additional problems that can be avoided. The two PEPs also differ in their proposed approaches to handling lazy evaluation of template fields. While there are other differences between the two proposals, those differences are more cosmetic than substantive. In particular: <ul class="simple"> <li>this PEP proposes different names for the structural typing protocols</li> <li>this PEP proposes specific names for the concrete implementation types</li> <li>this PEP proposes exact details for the proposed APIs of the concrete implementation types (including concatenation and repetition support, which are not part of the structural typing protocols)</li> <li>this PEP proposes changes to the existing <a class="reference external" href="https://docs.python.org/3/library/functions.html#format" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">format()</code></a> builtin to make it usable directly as a template field renderer</li> </ul> The two PEPs also differ in how they make their case for delayed rendering support. This PEP focuses more on the concrete implementation concept of using template literals to allow the “interpolation” and “rendering” steps in f-string processing to be separated in time, and then taking advantage of that to reduce the potential code injection risks associated with misuse of f-strings. PEP 750 focuses more on the way that native templating support allows behaviours that are difficult or impossible to achieve via existing string based templating methods. As with the cosmetic differences noted above, this is more a difference in style than a difference in substance. </section> </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> f-strings (<a class="pep reference internal" href="../pep-0498/" title="PEP 498 – Literal String Interpolation">PEP 498</a>) made interpolating values into strings with full access to Python’s lexical namespace semantics simpler, but it does so at the cost of creating a situation where interpolating values into sensitive targets like SQL queries, shell commands and HTML templates will enjoy a much cleaner syntax when handled without regard for code injection attacks than when they are handled correctly. This PEP proposes to provide the option of delaying the actual rendering of a template literal to a formatted string to its <code class="docutils literal notranslate">__format__</code> method, allowing the use of other template renderers by passing the template around as a first class object. While very different in the technical details, the <code class="docutils literal notranslate">types.TemplateLiteral</code> interface proposed in this PEP is conceptually quite similar to the <code class="docutils literal notranslate">FormattableString</code> type underlying the <a class="reference external" href="https://msdn.microsoft.com/en-us/library/dn961160.aspx">native interpolation</a> support introduced in C# 6.0, as well as the <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals">JavaScript template literals</a> introduced in ES6. While not the original motivation for developing the proposal, many of the benefits for defining domain specific languages described in <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a> also apply to this PEP (including the potential for per-DSL semantic highlighting in code editors based on the type specifications of declared template variables and rendering function parameters). </section> <section id="specification"> <h2><a class="toc-backref" href="#specification" role="doc-backlink">Specification</a></h2> This PEP proposes a new <code class="docutils literal notranslate">t</code> string prefix that results in the creation of an instance of a new type, <code class="docutils literal notranslate">types.TemplateLiteral</code>. Template literals are Unicode strings (bytes literals are not permitted), and string literal concatenation operates as normal, with the entire combined literal forming the template literal. The template string is parsed into literals, expressions, format specifiers, and conversion specifiers as described for f-strings in <a class="pep reference internal" href="../pep-0498/" title="PEP 498 – Literal String Interpolation">PEP 498</a> and <a class="pep reference internal" href="../pep-0701/" title="PEP 701 – Syntactic formalization of f-strings">PEP 701</a>. The syntax for conversion specifiers is relaxed such that arbitrary strings are accepted (excluding those containing <code class="docutils literal notranslate">{</code>, <code class="docutils literal notranslate">}</code> or <code class="docutils literal notranslate">:</code>) rather than being restricted to valid Python identifiers. However, rather than being rendered directly into a formatted string, these components are instead organised into instances of new types with the following behaviour: <div class="highlight-python notranslate"><div class="highlight"><pre>class TemplateLiteralText(str): # This is a renamed and extended version of the DecodedConcrete type in PEP 750 # Real type would be implemented in C, this is an API compatible Python equivalent _raw: str def __new__(cls, raw: str): decoded = raw.encode("utf-8").decode("unicode-escape") if decoded == raw: decoded = raw text = super().__new__(cls, decoded) text._raw = raw return text @staticmethod def merge(text_segments:Sequence[TemplateLiteralText]) -> TemplateLiteralText: if len(text_segments) == 1: return text_segments[0] return TemplateLiteralText("".join(t._raw for t in text_segments)) @property def raw(self) -> str: return self._raw def __repr__(self) -> str: return f"{type(self).__name__}(r{self._raw!r})" def __add__(self, other:Any) -> TemplateLiteralText|NotImplemented: if isinstance(other, TemplateLiteralText): return TemplateLiteralText(self._raw + other._raw) return NotImplemented def __mul__(self, other:Any) -> TemplateLiteralText|NotImplemented: try: factor = operator.index(other) except TypeError: return NotImplemented return TemplateLiteralText(self._raw * factor) __rmul__ = __mul__ class TemplateLiteralField(NamedTuple): # This is mostly a renamed version of the InterpolationConcrete type in PEP 750 # However: # - value is eagerly evaluated (values were all originally lazy in PEP 750) # - conversion specifiers are allowed to be arbitrary strings # - order of fields is adjusted so the text form is the first field and the # remaining parameters match the updated signature of the `*format` builtin # Real type would be implemented in C, this is an API compatible Python equivalent expr: str value: Any format_spec: str | None = None conversion_spec: str | None = None def __repr__(self) -> str: return (f"{type(self).__name__}({self.expr}, {self.value!r}, " f"{self.format_spec!r}, {self.conversion_spec!r})") def __str__(self) -> str: return format(self.value, self.format_spec, self.conversion_spec) def __format__(self, format_override) -> str: if format_override: format_spec = format_override else: format_spec = self.format_spec return format(self.value, format_spec, self.conversion_spec) class TemplateLiteral: # This type corresponds to the TemplateConcrete type in PEP 750 # Real type would be implemented in C, this is an API compatible Python equivalent _raw_template: str _segments = tuple[TemplateLiteralText|TemplateLiteralField] def __new__(cls, raw_template:str, *segments:TemplateLiteralText|TemplateLiteralField): self = super().__new__(cls) self._raw_template = raw_template # Check if there are any adjacent text segments that need merging # or any empty text segments that need discarding type_err = "Template literal segments must be template literal text or field instances" text_expected = True needs_merge = False for segment in segments: match segment: case TemplateLiteralText(): if not text_expected or not segment: needs_merge = True break text_expected = False case TemplateLiteralField(): text_expected = True case _: raise TypeError(type_err) if not needs_merge: # Match loop above will have checked all segments self._segments = segments return self # Merge consecutive runs of text fields and drop any empty text fields merged_segments:list[TemplateLiteralText|TemplateLiteralField] = [] pending_merge:list[TemplateLiteralText] = [] for segment in segments: match segment: case TemplateLiteralText() as text_segment: if text_segment: pending_merge.append(text_segment) case TemplateLiteralField(): if pending_merge: merged_segments.append(TemplateLiteralText.merge(pending_merge)) pending_merge.clear() merged_segments.append(segment) case _: # First loop above may not check all segments when a merge is needed raise TypeError(type_err) if pending_merge: merged_segments.append(TemplateLiteralText.merge(pending_merge)) pending_merge.clear() self._segments = tuple(merged_segments) return self @property def raw_template(self) -> str: return self._raw_template @property def segments(self) -> tuple[TemplateLiteralText|TemplateLiteralField]: return self._segments def __len__(self) -> int: return len(self._segments) def __iter__(self) -> Iterable[TemplateLiteralText|TemplateLiteralField]: return iter(self._segments) # Note: template literals do NOT define any relative ordering def __eq__(self, other): if not isinstance(other, TemplateLiteral): return NotImplemented return ( self._raw_template == other._raw_template and self._segments == other._segments and self.field_values == other.field_values and self.format_specifiers == other.format_specifiers ) def __repr__(self) -> str: return (f"{type(self).__name__}(r{self._raw!r}, " f"{', '.join(map(repr, self._segments))})") def __format__(self, format_specifier) -> str: # When formatted, render to a string, and then use string formatting return format(self.render(), format_specifier) def render(self, *, render_template=''.join, render_text=str, render_field=format): ... # See definition of the template rendering semantics below def __add__(self, other) -> TemplateLiteral|NotImplemented: if isinstance(other, TemplateLiteral): combined_raw_text = self._raw + other._raw combined_segments = self._segments + other._segments return TemplateLiteral(combined_raw_text, *combined_segments) if isinstance(other, str): # Treat the given string as a new raw text segment combined_raw_text = self._raw + other combined_segments = self._segments + (TemplateLiteralText(other),) return TemplateLiteral(combined_raw_text, *combined_segments) return NotImplemented def __radd__(self, other) -> TemplateLiteral|NotImplemented: if isinstance(other, str): # Treat the given string as a new raw text segment. This effectively # has precedence over string concatenation in CPython due to # https://github.com/python/cpython/issues/55686 combined_raw_text = other + self._raw combined_segments = (TemplateLiteralText(other),) + self._segments return TemplateLiteral(combined_raw_text, *combined_segments) return NotImplemented def __mul__(self, other) -> TemplateLiteral|NotImplemented: try: factor = operator.index(other) except TypeError: return NotImplemented if not self or factor == 1: return self if factor < 1: return TemplateLiteral("") repeated_text = self._raw_template * factor repeated_segments = self._segments * factor return TemplateLiteral(repeated_text, *repeated_segments) __rmul__ = __mul__ </pre></div> </div> (Note: this is an illustrative example implementation, the exact compile time construction method and internal data management details of <code class="docutils literal notranslate">types.TemplateLiteral</code> are considered an implementation detail not specified by the PEP. However, the expected post-construction behaviour of the public APIs on <code class="docutils literal notranslate">types.TemplateLiteral</code> instances is specified by the above code, as is the constructor signature for building template instances at runtime) The result of a template literal expression is an instance of this type, rather than an already rendered string. Rendering only takes place when the instance’s <code class="docutils literal notranslate">render</code> method is called (either directly, or indirectly via <code class="docutils literal notranslate">__format__</code>). The compiler will pass the following details to the template literal for later use: <ul class="simple"> <li>a string containing the raw template as written in the source code</li> <li>a sequence of template segments, with each segment being either:<ul> <li>a literal text segment (a regular Python string that also provides access to its raw form)</li> <li>a parsed template interpolation field, specifying the text of the interpolated expression (as a regular string), its evaluated result, the format specifier text (with any substitution fields eagerly evaluated as an f-string), and the conversion specifier text (as a regular string)</li> </ul> </li> </ul> The raw template is just the template literal as a string. By default, it is used to provide a human-readable representation for the template literal, but template renderers may also use it for other purposes (e.g. as a cache lookup key). The parsed template structure is taken from <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a> and consists of a sequence of template segments corresponding to the text segments and interpolation fields in the template string. This approach is designed to allow compilers to fully process each segment of the template in order, before finally emitting code to pass all of the template segments to the template literal constructor. For example, assuming the following runtime values: <div class="highlight-python notranslate"><div class="highlight"><pre>names = ["Alice", "Bob", "Carol", "Eve"] field_width = 10 def expressions(): return 42 </pre></div> </div> The template from the proposal section would be represented at runtime as: <div class="highlight-python notranslate"><div class="highlight"><pre>TemplateLiteral( r"Substitute {names:>{field_width}} and {expressions()!r} at runtime", TemplateLiteralText(r"Substitute "), TemplateLiteralField("names", ["Alice", "Bob", "Carol", "Eve"], ">10", ""), TemplateLiteralText(r" and "), TemplateLiteralField("expressions()", 42, "", "r"), ) </pre></div> </div> <section id="rendering-templates"> <h3><a class="toc-backref" href="#rendering-templates" role="doc-backlink">Rendering templates</a></h3> The <code class="docutils literal notranslate">TemplateLiteral.render</code> implementation defines the rendering process in terms of the following renderers: <ul class="simple"> <li>an overall <code class="docutils literal notranslate">render_template</code> operation that defines how the sequence of rendered text and field segments are composed into a fully rendered result. The default template renderer is string concatenation using <code class="docutils literal notranslate">''.join</code>.</li> <li>a per text segment <code class="docutils literal notranslate">render_text</code> operation that receives the individual literal text segments within the template. The default text renderer is the builtin <code class="docutils literal notranslate">str</code> constructor.</li> <li>a per field segment <code class="docutils literal notranslate">render_field</code> operation that receives the field value, format specifier, and conversion specifier for substitution fields within the template. The default field renderer is the <a class="reference external" href="https://docs.python.org/3/library/functions.html#format" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">format()</code></a> builtin.</li> </ul> Given the parsed template representation above, the semantics of template rendering would then be equivalent to the following: <div class="highlight-python notranslate"><div class="highlight"><pre>def render(self, *, render_template=''.join, render_text=str, render_field=format): rendered_segments = [] for segment in self._segments: match segment: case TemplateLiteralText() as text_segment: rendered_segments.append(render_text(text_segment)) case TemplateLiteralField() as field_segment: rendered_segments.append(render_field(*field_segment[1:])) return render_template(rendered_segments) </pre></div> </div> </section> <section id="format-specifiers"> <h3><a class="toc-backref" href="#format-specifiers" role="doc-backlink">Format specifiers</a></h3> The syntax and processing of field specifiers in t-strings is defined to be the same as it is for f-strings. This includes allowing field specifiers to themselves contain f-string substitution fields. The raw text of the field specifiers (without processing any substitution fields) is retained as part of the full raw template string. The parsed field specifiers receive the field specifier string with those substitutions already resolved. The <code class="docutils literal notranslate">:</code> prefix is also omitted. Aside from separating them out from the substitution expression during parsing, format specifiers are otherwise treated as opaque strings by the interpolation template parser - assigning semantics to those (or, alternatively, prohibiting their use) is handled at rendering time by the field renderer. </section> <section id="conversion-specifiers"> <h3><a class="toc-backref" href="#conversion-specifiers" role="doc-backlink">Conversion specifiers</a></h3> In addition to the existing support for <code class="docutils literal notranslate">a</code>, <code class="docutils literal notranslate">r</code>, and <code class="docutils literal notranslate">s</code> conversion specifiers, <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str.format" title="(in Python v3.13)"><code class="xref py py-meth docutils literal notranslate">str.format()</code></a> and <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str.format_map" title="(in Python v3.13)"><code class="xref py py-meth docutils literal notranslate">str.format_map()</code></a> will be updated to accept <code class="docutils literal notranslate">()</code> as a conversion specifier that means “call the interpolated value”. Where <a class="pep reference internal" href="../pep-0701/" title="PEP 701 – Syntactic formalization of f-strings">PEP 701</a> restricts conversion specifiers to <code class="docutils literal notranslate">NAME</code> tokens, this PEP will instead allow <code class="docutils literal notranslate">FSTRING_MIDDLE</code> tokens (such that only <code class="docutils literal notranslate">{</code>, <code class="docutils literal notranslate">}</code> and <code class="docutils literal notranslate">:</code> are disallowed). This change is made primarily to support lazy field rendering with the <code class="docutils literal notranslate">!()</code> conversion specifier, but also allows custom rendering functions more flexibility when defining their own conversion specifiers in preference to those defined for the default <a class="reference external" href="https://docs.python.org/3/library/functions.html#format" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">format()</code></a> field renderer. Conversion specifiers are still handled as plain strings, and do NOT support the use of substitution fields. The parsed conversion specifiers receive the conversion specifier string with the <code class="docutils literal notranslate">!</code> prefix omitted. To allow custom template renderers to define their own custom conversion specifiers without causing the default renderer to fail, conversion specifiers will be permitted to contain a custom suffix prefixed with a second <code class="docutils literal notranslate">!</code> character. That is, <code class="docutils literal notranslate">!!<custom></code>, <code class="docutils literal notranslate">!a!<custom></code>, <code class="docutils literal notranslate">!r!<custom></code>, <code class="docutils literal notranslate">!s!<custom></code>, and <code class="docutils literal notranslate">!()!<custom></code> would all be valid conversion specifiers in a template literal. As described above, the default rendering supports the original <code class="docutils literal notranslate">!a</code>, <code class="docutils literal notranslate">!r</code> and <code class="docutils literal notranslate">!s</code> conversion specifiers defined in <a class="pep reference internal" href="../pep-3101/" title="PEP 3101 – Advanced String Formatting">PEP 3101</a>, together with the new <code class="docutils literal notranslate">!()</code> lazy field evaluation conversion specifier defined in this PEP. The default rendering ignores any custom conversion specifier suffixes. The full mapping between the standard conversion specifiers and the special methods called on the interpolated value when the field is rendered: <ul class="simple"> <li>No conversion (empty string): <code class="docutils literal notranslate">__format__</code> (with format specifier as parameter)</li> <li><code class="docutils literal notranslate">a</code>: <code class="docutils literal notranslate">__repr__</code> (as per the <a class="reference external" href="https://docs.python.org/3/library/functions.html#ascii" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">ascii()</code></a> builtin)</li> <li><code class="docutils literal notranslate">r</code>: <code class="docutils literal notranslate">__repr__</code> (as per the <a class="reference external" href="https://docs.python.org/3/library/functions.html#repr" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">repr()</code></a> builtin)</li> <li><code class="docutils literal notranslate">s</code>: <code class="docutils literal notranslate">__str__</code> (as per the <code class="docutils literal notranslate">str</code> builtin)</li> <li><code class="docutils literal notranslate">()</code>: <code class="docutils literal notranslate">__call__</code> (with no parameters)</li> </ul> When a conversion occurs, <code class="docutils literal notranslate">__format__</code> (with the format specifier) is called on the result of the conversion rather than being called on the original object. The changes to <a class="reference external" href="https://docs.python.org/3/library/functions.html#format" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">format()</code></a> and the addition of <code class="xref py py-func docutils literal notranslate">operator.convert_field()</code> make it straightforward for custom renderers to also support the standard conversion specifiers. f-strings themselves will NOT support the new <code class="docutils literal notranslate">!()</code> conversion specifier (as it is redundant when value interpolation and value rendering always occur at the same time). They also will NOT support the use of custom conversion specifiers (since the rendering function is known at compile time and doesn’t make use of the custom specifiers). </section> <section id="new-field-conversion-api-in-the-operator-module"> <h3><a class="toc-backref" href="#new-field-conversion-api-in-the-operator-module" role="doc-backlink">New field conversion API in the <a class="reference external" href="https://docs.python.org/3/library/operator.html#module-operator" title="(in Python v3.13)"><code class="xref py py-mod docutils literal notranslate">operator</code></a> module</a></h3> To support application of the standard conversion specifiers in custom template rendering functions, a new <code class="xref py py-func docutils literal notranslate">operator.convert_field()</code> function will be added: <div class="highlight-python notranslate"><div class="highlight"><pre>def convert_field(value, conversion_spec=''): """Apply the given string formatting conversion specifier to the given value""" std_spec, sep, custom_spec = conversion_spec.partition("!") match std_spec: case '': return value case 'a': return ascii(value) case 'r': return repr(value) case 's': return str(value) case '()': return value() if not sep: err = f"Invalid conversion specifier {std_spec!r}" else: err = f"Invalid conversion specifier {std_spec!r} in {conversion_spec!r}" raise ValueError(f"{err}: expected '', 'a', 'r', 's' or '()') </pre></div> </div> </section> <section id="conversion-specifier-parameter-added-to-format"> <h3><a class="toc-backref" href="#conversion-specifier-parameter-added-to-format" role="doc-backlink">Conversion specifier parameter added to <a class="reference external" href="https://docs.python.org/3/library/functions.html#format" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">format()</code></a></a></h3> The signature and behaviour of the <a class="reference external" href="https://docs.python.org/3/library/functions.html#format" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">format()</code></a> builtin will be updated: <div class="highlight-python notranslate"><div class="highlight"><pre>def format(value, format_spec='', conversion_spec=''): if conversion_spec: value_to_format = operator.convert_field(value) else: value_to_format = value return type(value_to_format).__format__(value, format_spec) </pre></div> </div> If a non-empty conversion specifier is given, the value will be converted with <code class="xref py py-func docutils literal notranslate">operator.convert_field()</code> before looking up the <code class="docutils literal notranslate">__format__</code> method. The signature of the <code class="docutils literal notranslate">__format__</code> special method does NOT change (only format specifiers are handled by the object being formatted). </section> <section id="structural-typing-and-duck-typing"> <h3><a class="toc-backref" href="#structural-typing-and-duck-typing" role="doc-backlink">Structural typing and duck typing</a></h3> To allow custom renderers to accept alternative interpolation template implementations (rather than being tightly coupled to the native template literal types), the following structural protocols will be added to the <code class="docutils literal notranslate">typing</code> module: <div class="highlight-python notranslate"><div class="highlight"><pre>@runtime_checkable class TemplateText(Protocol): # Renamed version of PEP 750's Decoded protocol def __str__(self) -> str: ... raw: str @runtime_checkable class TemplateField(Protocol): # Renamed and modified version of PEP 750's Interpolation protocol def __len__(self): ... def __getitem__(self, index: int): ... def __str__(self) -> str: ... expr: str value: Any format_spec: str | None = None conversion_spec: str | None = None @runtime_checkable class InterpolationTemplate(Protocol): # Corresponds to PEP 750's Template protocol def __iter__(self) -> Iterable[TemplateText|TemplateField]: ... raw_template: str </pre></div> </div> Note that the structural protocol APIs are substantially narrower than the full implementation APIs defined for <code class="docutils literal notranslate">TemplateLiteralText</code>, <code class="docutils literal notranslate">TemplateLiteralField</code>, and <code class="docutils literal notranslate">TemplateLiteral</code>. Code that wants to accept interpolation templates and define specific handling for them without introducing a dependency on the <code class="docutils literal notranslate">typing</code> module, or restricting the code to handling the concrete template literal types, should instead perform an attribute existence check on <code class="docutils literal notranslate">raw_template</code>. </section> <section id="writing-custom-renderers"> <h3><a class="toc-backref" href="#writing-custom-renderers" role="doc-backlink">Writing custom renderers</a></h3> Writing a custom renderer doesn’t require any special syntax. Instead, custom renderers are ordinary callables that process an interpolation template directly either by calling the <code class="docutils literal notranslate">render()</code> method with alternate <code class="docutils literal notranslate">render_template</code>, <code class="docutils literal notranslate">render_text</code>, and/or <code class="docutils literal notranslate">render_field</code> implementations, or by accessing the template’s data attributes directly. For example, the following function would render a template using objects’ <code class="docutils literal notranslate">repr</code> implementations rather than their native formatting support: <div class="highlight-python notranslate"><div class="highlight"><pre>def repr_format(template): def render_field(value, format_spec, conversion_spec): converted_value = operator.convert_field(value, conversion_spec) return format(repr(converted_value), format_spec) return template.render(render_field=render_field) </pre></div> </div> The customer renderer shown respects the conversion specifiers in the original template, but it is also possible to ignore them and render the interpolated values directly: <div class="highlight-python notranslate"><div class="highlight"><pre>def input_repr_format(template): def render_field(value, format_spec, __): return format(repr(value), format_spec) return template.render(render_field=render_field) </pre></div> </div> When writing custom renderers, note that the return type of the overall rendering operation is determined by the return type of the passed in <code class="docutils literal notranslate">render_template</code> callable. While this will still be a string for formatting related use cases, producing non-string objects is permitted. For example, a custom SQL template renderer could involve an <code class="docutils literal notranslate">sqlalchemy.sql.text</code> call that produces an <a class="reference external" href="http://docs.sqlalchemy.org/en/rel_1_0/core/tutorial.html#using-textual-sql">SQL Alchemy query object</a>. A subprocess invocation related template renderer could produce a string sequence suitable for passing to <code class="docutils literal notranslate">subprocess.run</code>, or it could even call <code class="docutils literal notranslate">subprocess.run</code> directly, and return the result. Non-strings may also be returned from <code class="docutils literal notranslate">render_text</code> and <code class="docutils literal notranslate">render_field</code>, as long as they are paired with a <code class="docutils literal notranslate">render_template</code> implementation that expects that behaviour. Custom renderers using the pattern matching style described in <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a> are also supported: <div class="highlight-python notranslate"><div class="highlight"><pre># Use the structural typing protocols rather than the concrete implementation types from typing import InterpolationTemplate, TemplateText, TemplateField def greet(template: InterpolationTemplate) -> str: """Render an interpolation template using structural pattern matching.""" result = [] for segment in template: match segment: match segment: case TemplateText() as text_segment: result.append(text_segment) case TemplateField() as field_segment: result.append(str(field_segment).upper()) return f"{''.join(result)}!" </pre></div> </div> </section> <section id="expression-evaluation"> <h3><a class="toc-backref" href="#expression-evaluation" role="doc-backlink">Expression evaluation</a></h3> As with f-strings, the subexpressions that are extracted from the interpolation template are evaluated in the context where the template literal appears. This means the expression has full access to local, nonlocal and global variables. Any valid Python expression can be used inside <code class="docutils literal notranslate">{}</code>, including function and method calls. Because the substitution expressions are evaluated where the string appears in the source code, there are no additional security concerns related to the contents of the expression itself, as you could have also just written the same expression and used runtime field parsing: <div class="highlight-python-console notranslate"><div class="highlight"><pre>>>> bar=10 >>> def foo(data): ... return data + 20 ... >>> str(t'input={bar}, output={foo(bar)}') 'input=10, output=30' </pre></div> </div> Is essentially equivalent to: <div class="highlight-python-console notranslate"><div class="highlight"><pre>>>> 'input={}, output={}'.format(bar, foo(bar)) 'input=10, output=30' </pre></div> </div> </section> <section id="handling-code-injection-attacks"> <h3><a class="toc-backref" href="#handling-code-injection-attacks" role="doc-backlink">Handling code injection attacks</a></h3> The <a class="pep reference internal" href="../pep-0498/" title="PEP 498 – Literal String Interpolation">PEP 498</a> formatted string syntax makes it potentially attractive to write code like the following: <div class="highlight-python notranslate"><div class="highlight"><pre>runquery(f"SELECT {column} FROM {table};") runcommand(f"cat {filename}") return_response(f"<html><body>{response.body}</body></html>") </pre></div> </div> These all represent potential vectors for code injection attacks, if any of the variables being interpolated happen to come from an untrusted source. The specific proposal in this PEP is designed to make it straightforward to write use case specific renderers that take care of quoting interpolated values appropriately for the relevant security context: <div class="highlight-python notranslate"><div class="highlight"><pre>runquery(sql(t"SELECT {column} FROM {table} WHERE column={value};")) runcommand(sh(t"cat {filename}")) return_response(html(t"<html><body>{response.body}</body></html>")) </pre></div> </div> This PEP does not cover adding all such renderers to the standard library immediately (though one for shell escaping is proposed), but rather proposes to ensure that they can be readily provided by third party libraries, and potentially incorporated into the standard library at a later date. Over time, it is expected that APIs processing potentially dangerous string inputs may be updated to accept interpolation templates natively, allowing problematic code examples to be fixed simply by replacing the <code class="docutils literal notranslate">f</code> string prefix with a <code class="docutils literal notranslate">t</code>: <div class="highlight-python notranslate"><div class="highlight"><pre>runquery(t"SELECT {column} FROM {table};") runcommand(t"cat {filename}") return_response(t"<html><body>{response.body}</body></html>") </pre></div> </div> It is proposed that a renderer is included in the <a class="reference external" href="https://docs.python.org/3/library/shlex.html#module-shlex" title="(in Python v3.13)"><code class="xref py py-mod docutils literal notranslate">shlex</code></a> module, aiming to offer a more POSIX shell style experience for accessing external programs, without the significant risks posed by running <code class="docutils literal notranslate">os.system</code> or enabling the system shell when using the <code class="docutils literal notranslate">subprocess</code> module APIs. This renderer will provide an interface for running external programs inspired by that offered by the <a class="reference external" href="https://docs.julialang.org/en/v1/manual/running-external-programs/">Julia programming language</a>, only with the backtick based <code class="docutils literal notranslate">\`cat $filename\`</code> syntax replaced by <code class="docutils literal notranslate">t"cat {filename}"</code> style template literals. See more in the <a class="reference internal" href="#pep-501-shlex-module">Renderer for shell escaping added to shlex</a> section. </section> <section id="error-handling"> <h3><a class="toc-backref" href="#error-handling" role="doc-backlink">Error handling</a></h3> Either compile time or run time errors can occur when processing interpolation expressions. Compile time errors are limited to those errors that can be detected when parsing a template string into its component tuples. These errors all raise SyntaxError. Unmatched braces: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> t'x={x' File "<stdin>", line 1 t'x={x' ^ SyntaxError: missing '}' in template literal expression </pre></div> </div> Invalid expressions: <div class="highlight-default notranslate"><div class="highlight"><pre>>>> t'x={!x}' File "<fstring>", line 1 !x ^ SyntaxError: invalid syntax </pre></div> </div> Run time errors occur when evaluating the expressions inside a template string before creating the template literal object. See <a class="pep reference internal" href="../pep-0498/" title="PEP 498 – Literal String Interpolation">PEP 498</a> for some examples. Different renderers may also impose additional runtime constraints on acceptable interpolated expressions and other formatting details, which will be reported as runtime exceptions. </section> <section id="renderer-for-shell-escaping-added-to-shlex"> <h3><a class="toc-backref" href="#renderer-for-shell-escaping-added-to-shlex" role="doc-backlink">Renderer for shell escaping added to <a class="reference external" href="https://docs.python.org/3/library/shlex.html#module-shlex" title="(in Python v3.13)"><code class="xref py py-mod docutils literal notranslate">shlex</code></a></a></h3> As a reference implementation, a renderer for safe POSIX shell escaping can be added to the <a class="reference external" href="https://docs.python.org/3/library/shlex.html#module-shlex" title="(in Python v3.13)"><code class="xref py py-mod docutils literal notranslate">shlex</code></a> module. This renderer would be called <code class="docutils literal notranslate">sh</code> and would be equivalent to calling <code class="docutils literal notranslate">shlex.quote</code> on each field value in the template literal. Thus: <div class="highlight-python notranslate"><div class="highlight"><pre>os.system(shlex.sh(t'cat {myfile}')) </pre></div> </div> would have the same behavior as: <div class="highlight-python notranslate"><div class="highlight"><pre>os.system('cat ' + shlex.quote(myfile))) </pre></div> </div> The implementation would be: <div class="highlight-python notranslate"><div class="highlight"><pre>def sh(template: TemplateLiteral): def render_field(value, format_spec, conversion_spec) field_text = format(value, format_spec, conversion_spec) return quote(field_text) return template.render(render_field=render_field) </pre></div> </div> The addition of <code class="docutils literal notranslate">shlex.sh</code> will NOT change the existing admonishments in the <a class="reference external" href="https://docs.python.org/3/library/subprocess.html#module-subprocess" title="(in Python v3.13)"><code class="xref py py-mod docutils literal notranslate">subprocess</code></a> documentation that passing <code class="docutils literal notranslate">shell=True</code> is best avoided, nor the reference from the <a class="reference external" href="https://docs.python.org/3/library/os.html#os.system" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">os.system()</code></a> documentation the higher level <code class="docutils literal notranslate">subprocess</code> APIs. </section> <section id="changes-to-subprocess-module"> <h3><a class="toc-backref" href="#changes-to-subprocess-module" role="doc-backlink">Changes to subprocess module</a></h3> With the additional renderer in the shlex module, and the addition of template literals, the <a class="reference external" href="https://docs.python.org/3/library/subprocess.html#module-subprocess" title="(in Python v3.13)"><code class="xref py py-mod docutils literal notranslate">subprocess</code></a> module can be changed to handle accepting template literals as an additional input type to <code class="docutils literal notranslate">Popen</code>, as it already accepts a sequence, or a string, with different behavior for each. With the addition of template literals, <a class="reference external" href="https://docs.python.org/3/library/subprocess.html#subprocess.Popen" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">subprocess.Popen</code></a> (and in return, all its higher level functions such as <a class="reference external" href="https://docs.python.org/3/library/subprocess.html#subprocess.run" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">subprocess.run()</code></a>) could accept strings in a safe way (at least on <a class="reference internal" href="#pep-501-defer-non-posix-shells">POSIX systems</a>). For example: <div class="highlight-python notranslate"><div class="highlight"><pre>subprocess.run(t'cat {myfile}', shell=True) </pre></div> </div> would automatically use the <code class="docutils literal notranslate">shlex.sh</code> renderer provided in this PEP. Therefore, using <code class="docutils literal notranslate">shlex</code> inside a <code class="docutils literal notranslate">subprocess.run</code> call like so: <div class="highlight-python notranslate"><div class="highlight"><pre>subprocess.run(shlex.sh(t'cat {myfile}'), shell=True) </pre></div> </div> would be redundant, as <code class="docutils literal notranslate">run</code> would automatically render any template literals through <code class="docutils literal notranslate">shlex.sh</code> Alternatively, when <code class="docutils literal notranslate">subprocess.Popen</code> is run without <code class="docutils literal notranslate">shell=True</code>, it could still provide subprocess with a more ergonomic syntax. For example: <div class="highlight-python notranslate"><div class="highlight"><pre>subprocess.run(t'cat {myfile} --flag {value}') </pre></div> </div> would be equivalent to: <div class="highlight-python notranslate"><div class="highlight"><pre>subprocess.run(['cat', myfile, '--flag', value]) </pre></div> </div> or, more accurately: <div class="highlight-python notranslate"><div class="highlight"><pre>subprocess.run(shlex.split(f'cat {shlex.quote(myfile)} --flag {shlex.quote(value)}')) </pre></div> </div> It would do this by first using the <code class="docutils literal notranslate">shlex.sh</code> renderer, as above, then using <code class="docutils literal notranslate">shlex.split</code> on the result. The implementation inside <code class="docutils literal notranslate">subprocess.Popen._execute_child</code> would look like: <div class="highlight-python notranslate"><div class="highlight"><pre>if hasattr(args, "raw_template"): import shlex if shell: args = [shlex.sh(args)] else: args = shlex.split(shlex.sh(args)) </pre></div> </div> </section> </section> <section id="how-to-teach-this"> <h2><a class="toc-backref" href="#how-to-teach-this" role="doc-backlink">How to Teach This</a></h2> This PEP intentionally includes two standard renderers that will always be available in teaching environments: the <a class="reference external" href="https://docs.python.org/3/library/functions.html#format" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">format()</code></a> builtin and the new <code class="docutils literal notranslate">shlex.sh</code> POSIX shell renderer. Together, these two renderers can be used to build an initial understanding of delayed rendering on top of a student’s initial introduction to string formatting with f-strings. This initial understanding would have the goal of allowing students to use template literals effectively, in combination with pre-existing template rendering functions. For example, <code class="docutils literal notranslate">f"{'some text'}"</code>, <code class="docutils literal notranslate">f"{value}"</code>, <code class="docutils literal notranslate">f"{value!r}"</code>, , <code class="docutils literal notranslate">f"{callable()}"</code> could all be introduced. Those same operations could then be rewritten as <code class="docutils literal notranslate">format(t"{'some text'}")</code>, <code class="docutils literal notranslate">format(t"{value}")</code>, <code class="docutils literal notranslate">format(t"{value!r}")</code>, , <code class="docutils literal notranslate">format(t"{callable()}")</code> to illustrate the relationship between the eager rendering form and the delayed rendering form. The difference between “template definition time” (or “interpolation time” ) and “template rendering time” can then be investigated further by storing the template literals as local variables and looking at their representations separately from the results of the <code class="docutils literal notranslate">format</code> calls. At this point, the <code class="docutils literal notranslate">t"{callable!()}"</code> syntax can be introduced to distinguish between field expressions that are called at template definition time and those that are called at template rendering time. Finally, the differences between the results of <code class="docutils literal notranslate">f"{'some text'}"</code>, <code class="docutils literal notranslate">format(t"{'some text'}")</code>, and <code class="docutils literal notranslate">shlex.sh(t"{'some text'}")</code> could be explored to illustrate the potential for differences between the default rendering function and custom rendering functions. Actually defining your own custom template rendering functions would then be a separate more advanced topic (similar to the way students are routinely taught to use decorators and context managers well before they learn how to write their own custom ones). <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a> includes further ideas for teaching aspects of the delayed rendering topic. </section> <section id="discussion"> <h2><a class="toc-backref" href="#discussion" role="doc-backlink">Discussion</a></h2> Refer to <a class="pep reference internal" href="../pep-0498/" title="PEP 498 – Literal String Interpolation">PEP 498</a> for previous discussion, as several of the points there also apply to this PEP. <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a>’s design discussions are also highly relevant, as that PEP inspired several aspects of the current design. <section id="support-for-binary-interpolation"> <h3><a class="toc-backref" href="#support-for-binary-interpolation" role="doc-backlink">Support for binary interpolation</a></h3> As f-strings don’t handle byte strings, neither will t-strings. </section> <section id="interoperability-with-str-only-interfaces"> <h3><a class="toc-backref" href="#interoperability-with-str-only-interfaces" role="doc-backlink">Interoperability with str-only interfaces</a></h3> For interoperability with interfaces that only accept strings, interpolation templates can still be prerendered with <a class="reference external" href="https://docs.python.org/3/library/functions.html#format" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">format()</code></a>, rather than delegating the rendering to the called function. This reflects the key difference from <a class="pep reference internal" href="../pep-0498/" title="PEP 498 – Literal String Interpolation">PEP 498</a>, which always eagerly applies the default rendering, without any way to delegate the choice of renderer to another section of the code. </section> <section id="preserving-the-raw-template-string"> <h3><a class="toc-backref" href="#preserving-the-raw-template-string" role="doc-backlink">Preserving the raw template string</a></h3> Earlier versions of this PEP failed to make the raw template string available on the template literal. Retaining it makes it possible to provide a more attractive template representation, as well as providing the ability to precisely reconstruct the original string, including both the expression text and the details of any eagerly rendered substitution fields in format specifiers. </section> <section id="creating-a-rich-object-rather-than-a-global-name-lookup"> <h3><a class="toc-backref" href="#creating-a-rich-object-rather-than-a-global-name-lookup" role="doc-backlink">Creating a rich object rather than a global name lookup</a></h3> Earlier versions of this PEP used an <code class="docutils literal notranslate">__interpolate__</code> builtin, rather than creating a new kind of object for later consumption by interpolation functions. Creating a rich descriptive object with a useful default renderer made it much easier to support customisation of the semantics of interpolation. </section> <section id="building-atop-f-strings-rather-than-replacing-them"> <h3><a class="toc-backref" href="#building-atop-f-strings-rather-than-replacing-them" role="doc-backlink">Building atop f-strings rather than replacing them</a></h3> Earlier versions of this PEP attempted to serve as a complete substitute for <a class="pep reference internal" href="../pep-0498/" title="PEP 498 – Literal String Interpolation">PEP 498</a> (f-strings) . With the acceptance of that PEP and the more recent <a class="pep reference internal" href="../pep-0701/" title="PEP 701 – Syntactic formalization of f-strings">PEP 701</a>, this PEP can instead build a more flexible delayed rendering capability on top of the existing f-string eager rendering. Assuming the presence of f-strings as a supporting capability simplified a number of aspects of the proposal in this PEP (such as how to handle substitution fields in format specifiers). </section> <section id="defining-repetition-and-concatenation-semantics"> <h3><a class="toc-backref" href="#defining-repetition-and-concatenation-semantics" role="doc-backlink">Defining repetition and concatenation semantics</a></h3> This PEP explicitly defines repetition and concatenation semantics for <code class="docutils literal notranslate">TemplateLiteral</code> and <code class="docutils literal notranslate">TemplateLiteralText</code>. While not strictly necessary, defining these is expected to make the types easier to work with in code that historically only supported regular strings. </section> <section id="new-conversion-specifier-for-lazy-field-evaluation"> <h3><a class="toc-backref" href="#new-conversion-specifier-for-lazy-field-evaluation" role="doc-backlink">New conversion specifier for lazy field evaluation</a></h3> The initially published version of <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a> defaulted to lazy evaluation for all interpolation fields. While it was subsequently updated to default to eager evaluation (as happens for f-strings and this PEP), the discussions around the topic prompted the idea of providing a way to indicate to rendering functions that the interpolated field value should be called at rendering time rather than being used without modification. Since PEP 750 also deferred the processing of conversion specifiers until evaluation time, the suggestion was put forward that invoking <code class="docutils literal notranslate">__call__</code> without arguments could be seen as similar to the existing conversion specifiers that invoke <code class="docutils literal notranslate">__repr__</code> (<code class="docutils literal notranslate">!a</code>, <code class="docutils literal notranslate">!r</code>) or <code class="docutils literal notranslate">__str__</code> (<code class="docutils literal notranslate">!s</code>). Accordingly, this PEP was updated to also make conversion specifier processing the responsibility of rendering functions, and to introduce <code class="docutils literal notranslate">!()</code> as a new conversion specifier for lazy evaluation. Adding <code class="xref py py-func docutils literal notranslate">operator.convert_field()</code> and updating the <a class="reference external" href="https://docs.python.org/3/library/functions.html#format" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">format()</code></a> builtin was than a matter of providing appropriate support to rendering function implementations that wanted to accept the default conversion specifiers. </section> <section id="allowing-arbitrary-conversion-specifiers-in-custom-renderers"> <h3><a class="toc-backref" href="#allowing-arbitrary-conversion-specifiers-in-custom-renderers" role="doc-backlink">Allowing arbitrary conversion specifiers in custom renderers</a></h3> Accepting <code class="docutils literal notranslate">!()</code> as a new conversion specifier necessarily requires updating the syntax that the parser accepts for conversion specifiers (they are currently restricted to identifiers). This then raised the question of whether t-string compilation should enforce the additional restriction that f-string compilation imposes: that the conversion specifier be exactly one of <code class="docutils literal notranslate">!a</code>, <code class="docutils literal notranslate">!r</code>, or <code class="docutils literal notranslate">!s</code>. With t-strings already being updated to allow <code class="docutils literal notranslate">!()</code> when compiled, it made sense to treat conversion specifiers as relating to rendering function similar to the way that format specifiers related to the formatting of individual objects: aside from some characters that are excluded for parsing reasons, they are otherwise free text fields with the meaning decided by the consuming function or object. This reduces the temptation to introduce renderer specific metaformatting into the template’s format specifiers (since any renderer specific information can be placed in the conversion specifier instead). </section> <section id="only-reserving-a-single-new-string-prefix"> <h3><a class="toc-backref" href="#only-reserving-a-single-new-string-prefix" role="doc-backlink">Only reserving a single new string prefix</a></h3> The primary difference between this PEP and <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a> is that the latter aims to enable the use of arbitrary string prefixes, rather than requiring the creation of template literal instances that are then passed to other APIs. For example, PEP 750 would allow the <code class="docutils literal notranslate">sh</code> render described in this PEP to be used as <code class="docutils literal notranslate">sh"cat {somefile}"</code> rather than requiring the template literal to be created explicitly and then passed to a regular function call (as in <code class="docutils literal notranslate">sh(t"cat {somefile}")</code>). The main reason the PEP authors prefer the second spelling is because it makes it clearer to a reader what is going on: a template literal instance is being created, and then passed to a callable that knows how to do something useful with interpolation template instances. A <a class="reference external" href="https://discuss.python.org/t/pep-750-tag-strings-for-writing-domain-specific-languages/60408/176">draft proposal</a> from one of the <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a> authors also suggests that static typecheckers will be able to infer the use of particular domain specific languages just as readily from the form that uses an explicit function call as they would be able to infer it from a directly tagged string. With the tagged string syntax at least arguably reducing clarity for human readers without increasing the overall expressiveness of the construct, it seems reasonable to start with the smallest viable proposal (a single new string prefix), and then revisit the potential value of generalising to arbitrary prefixes in the future. As a lesser, but still genuine, consideration, only using a single new string prefix for this use case leaves open the possibility of defining alternate prefixes in the future that still produce <code class="docutils literal notranslate">TemplateLiteral</code> objects, but use a different syntax within the string to define the interpolation fields (see the <a class="reference internal" href="#pep-501-defer-i18n">i18n discussion</a> below). </section> <section id="deferring-consideration-of-more-concise-delayed-evaluation-syntax"> <h3><a class="toc-backref" href="#deferring-consideration-of-more-concise-delayed-evaluation-syntax" role="doc-backlink">Deferring consideration of more concise delayed evaluation syntax</a></h3> During the discussions of delayed evaluation, <code class="docutils literal notranslate">{-> expr}</code> was <a class="reference external" href="https://discuss.python.org/t/pep-750-tag-strings-for-writing-domain-specific-languages/60408/112">suggested</a> as potential syntactic sugar for the already supported <code class="docutils literal notranslate">lambda</code> based syntax: <code class="docutils literal notranslate">{(lambda: expr)}</code> (the parentheses are required in the existing syntax to avoid misinterpretation of the <code class="docutils literal notranslate">:</code> character as indicating the start of the format specifier). While adding such a spelling would complement the rendering time function call syntax proposed in this PEP (that is, writing <code class="docutils literal notranslate">{-> expr!()}</code> to evaluate arbitrary expressions at rendering time), it is a topic that the PEP authors consider to be better left to a future PEP if this PEP or <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a> is accepted. </section> <section id="deferring-consideration-of-possible-logging-integration"> <h3><a class="toc-backref" href="#deferring-consideration-of-possible-logging-integration" role="doc-backlink">Deferring consideration of possible logging integration</a></h3> One of the challenges with the logging module has been that we have previously been unable to devise a reasonable migration strategy away from the use of printf-style formatting. While the logging module does allow formatters to specify the use of <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str.format" title="(in Python v3.13)"><code class="xref py py-meth docutils literal notranslate">str.format()</code></a> or <a class="reference external" href="https://docs.python.org/3/library/string.html#string.Template" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">string.Template</code></a> style substitution, it can be awkward to ensure that messages written that way are only ever processed by log record formatters that are expecting that syntax. The runtime parsing and interpolation overhead for logging messages also poses a problem for extensive logging of runtime events for monitoring purposes. While beyond the scope of this initial PEP, template literal support could potentially be added to the logging module’s event reporting APIs, permitting relevant details to be captured using forms like: <div class="highlight-python notranslate"><div class="highlight"><pre>logging.debug(t"Event: {event}; Details: {data}") logging.critical(t"Error: {error}; Details: {data}") </pre></div> </div> Rather than the historical mod-formatting style: <div class="highlight-python notranslate"><div class="highlight"><pre>logging.debug("Event: %s; Details: %s", event, data) logging.critical("Error: %s; Details: %s", event, data) </pre></div> </div> As the template literal is passed in as an ordinary argument, other keyword arguments would also remain available: <div class="highlight-python notranslate"><div class="highlight"><pre>logging.critical(t"Error: {error}; Details: {data}", exc_info=True) </pre></div> </div> The approach to standardising lazy field evaluation described in this PEP is primarily based on the anticipated needs of this hypothetical integration into the logging module: <div class="highlight-python notranslate"><div class="highlight"><pre>logging.debug(t"Eager evaluation of {expensive_call()}") logging.debug(t"Lazy evaluation of {expensive_call!()}") logging.debug(t"Eager evaluation of {expensive_call_with_args(x, y, z)}") logging.debug(t"Lazy evaluation of {(lambda: expensive_call_with_args(x, y, z))!()}") </pre></div> </div> It’s an open question whether the definition of logging formatters would be updated to support template strings, but if they were, the most likely way of defining fields which should be <a class="reference external" href="https://docs.python.org/3/library/logging.html#logrecord-attributes" title="(in Python v3.13)">looked up on the log record</a> instead of being interpreted eagerly is simply to escape them so they’re available as part of the literal text: <div class="highlight-python notranslate"><div class="highlight"><pre>proc_id = get_process_id() formatter = logging.Formatter(t"{{asctime}}:{proc_id}:{{name}}:{{levelname}}{{message}}") </pre></div> </div> </section> <section id="deferring-consideration-of-possible-use-in-i18n-use-cases"> <h3><a class="toc-backref" href="#deferring-consideration-of-possible-use-in-i18n-use-cases" role="doc-backlink">Deferring consideration of possible use in i18n use cases</a></h3> The initial motivating use case for this PEP was providing a cleaner syntax for i18n (internationalization) translation, as that requires access to the original unmodified template. As such, it focused on compatibility with the substitution syntax used in Python’s <a class="reference external" href="https://docs.python.org/3/library/string.html#string.Template" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">string.Template</code></a> formatting and Mozilla’s l20n project. However, subsequent discussion revealed there are significant additional considerations to be taken into account in the i18n use case, which don’t impact the simpler cases of handling interpolation into security sensitive contexts (like HTML, system shells, and database queries), or producing application debugging messages in the preferred language of the development team (rather than the native language of end users). Due to that realisation, the PEP was switched to use the <a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#str.format" title="(in Python v3.13)"><code class="xref py py-meth docutils literal notranslate">str.format()</code></a> substitution syntax originally defined in <a class="pep reference internal" href="../pep-3101/" title="PEP 3101 – Advanced String Formatting">PEP 3101</a> and subsequently used as the basis for <a class="pep reference internal" href="../pep-0498/" title="PEP 498 – Literal String Interpolation">PEP 498</a>. While it would theoretically be possible to update <a class="reference external" href="https://docs.python.org/3/library/string.html#string.Template" title="(in Python v3.13)"><code class="xref py py-class docutils literal notranslate">string.Template</code></a> to support the creation of instances from native template literals, and to implement the structural <code class="docutils literal notranslate">typing.Template</code> protocol, the PEP authors have not identified any practical benefit in doing so. However, one significant benefit of the “only one string prefix” approach used in this PEP is that while it generalises the existing f-string interpolation syntax to support delayed rendering through t-strings, it doesn’t imply that that should be the only compiler supported interpolation syntax that Python should ever offer. Most notably, it leaves the door open to an alternate “t$-string” syntax that would allow <code class="docutils literal notranslate">TemplateLiteral</code> instances to be created using a <a class="pep reference internal" href="../pep-0292/" title="PEP 292 – Simpler String Substitutions">PEP 292</a> based interpolation syntax rather than a <a class="pep reference internal" href="../pep-3101/" title="PEP 3101 – Advanced String Formatting">PEP 3101</a> based syntax: <blockquote> <div>template = t$”Substitute $words and ${other_values} at runtime”</div></blockquote> The only runtime distinction between templates created that way and templates created from regular t-strings would be in the contents of their <code class="docutils literal notranslate">raw_template</code> attributes. </section> <section id="deferring-escaped-rendering-support-for-non-posix-shells"> <h3><a class="toc-backref" href="#deferring-escaped-rendering-support-for-non-posix-shells" role="doc-backlink">Deferring escaped rendering support for non-POSIX shells</a></h3> <a class="reference external" href="https://docs.python.org/3/library/shlex.html#shlex.quote" title="(in Python v3.13)"><code class="xref py py-func docutils literal notranslate">shlex.quote()</code></a> works by classifying the regex character set <code class="docutils literal notranslate">[\w@%+=:,./-]</code> to be safe, deeming all other characters to be unsafe, and hence requiring quoting of the string containing them. The quoting mechanism used is then specific to the way that string quoting works in POSIX shells, so it cannot be trusted when running a shell that doesn’t follow POSIX shell string quoting rules. For example, running <code class="docutils literal notranslate">subprocess.run(f'echo {shlex.quote(sys.argv[1])}', shell=True)</code> is safe when using a shell that follows POSIX quoting rules: <div class="highlight-default notranslate"><div class="highlight"><pre>$ cat > run_quoted.py import sys, shlex, subprocess subprocess.run(f"echo {shlex.quote(sys.argv[1])}", shell=True) $ python3 run_quoted.py pwd pwd $ python3 run_quoted.py '; pwd' ; pwd $ python3 run_quoted.py "'pwd'" 'pwd' </pre></div> </div> but remains unsafe when running a shell from Python invokes <code class="docutils literal notranslate">cmd.exe</code> (or Powershell): <div class="highlight-default notranslate"><div class="highlight"><pre>S:\> echo import sys, shlex, subprocess > run_quoted.py S:\> echo subprocess.run(f"echo {shlex.quote(sys.argv[1])}", shell=True) >> run_quoted.py S:\> type run_quoted.py import sys, shlex, subprocess subprocess.run(f"echo {shlex.quote(sys.argv[1])}", shell=True) S:\> python3 run_quoted.py "echo OK" 'echo OK' S:\> python3 run_quoted.py "'& echo Oh no!" ''"'"' Oh no!' </pre></div> </div> Resolving this standard library limitation is beyond the scope of this PEP. </section> </section> <section id="acknowledgements"> <h2><a class="toc-backref" href="#acknowledgements" role="doc-backlink">Acknowledgements</a></h2> <ul class="simple"> <li>Eric V. Smith for creating <a class="pep reference internal" href="../pep-0498/" title="PEP 498 – Literal String Interpolation">PEP 498</a> and demonstrating the feasibility of arbitrary expression substitution in string interpolation</li> <li>The authors of <a class="pep reference internal" href="../pep-0750/" title="PEP 750 – Template Strings">PEP 750</a> for the substantial design improvements that tagged strings inspired for this PEP, their general advocacy for the value of language level delayed template rendering support, and their efforts to ensure that any native interpolation template support lays a strong foundation for future efforts in providing robust syntax highlighting and static type checking support for domain specific languages</li> <li>Barry Warsaw, Armin Ronacher, and Mike Miller for their contributions to exploring the feasibility of using this model of delayed rendering in i18n use cases (even though the ultimate conclusion was that it was a poor fit, at least for current approaches to i18n in Python)</li> </ul> </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <ul class="simple"> <li><a class="reference external" href="https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting">%-formatting</a></li> <li><a class="reference external" href="https://docs.python.org/3/library/string.html#formatstrings">str.format</a></li> <li><a class="reference external" href="https://docs.python.org/3/library/string.html#template-strings">string.Template documentation</a></li> <li><a class="pep reference internal" href="../pep-0215/" title="PEP 215 – String Interpolation">PEP 215</a>: String Interpolation</li> <li><a class="pep reference internal" href="../pep-0292/" title="PEP 292 – Simpler String Substitutions">PEP 292</a>: Simpler String Substitutions</li> <li><a class="pep reference internal" href="../pep-3101/" title="PEP 3101 – Advanced String Formatting">PEP 3101</a>: Advanced String Formatting</li> <li><a class="pep reference internal" href="../pep-0498/" title="PEP 498 – Literal String Interpolation">PEP 498</a>: Literal string formatting</li> <li><a class="pep reference internal" href="../pep-0675/" title="PEP 675 – Arbitrary Literal String Type">PEP 675</a>: Arbitrary Literal String Type</li> <li><a class="pep reference internal" href="../pep-0701/" title="PEP 701 – Syntactic formalization of f-strings">PEP 701</a>: Syntactic formalization of f-strings</li> <li><a class="reference external" href="https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/tokens/interpolated">FormattableString and C# native string interpolation</a></li> <li><a class="reference external" href="https://docs.microsoft.com/en-us/dotnet/api/system.iformattable">IFormattable interface in C# (see remarks for globalization notes)</a></li> <li><a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals">TemplateLiterals in Javascript</a></li> <li><a class="reference external" href="https://docs.julialang.org/en/v1/manual/running-external-programs/">Running external commands in Julia</a></li> </ul> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0501.rst">https://github.com/python/peps/blob/main/peps/pep-0501.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0501.rst">2024-10-19 14:00:43 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#pep-withdrawal">PEP Withdrawal</a></li> <li><a class="reference internal" href="#relationship-with-other-peps">Relationship with other PEPs</a></li> <li><a class="reference internal" href="#motivation">Motivation</a></li> <li><a class="reference internal" href="#proposal">Proposal</a><ul> <li><a class="reference internal" href="#dedicated-template-literal-syntax">Dedicated template literal syntax</a></li> <li><a class="reference internal" href="#lazy-field-evaluation-conversion-specifier">Lazy field evaluation conversion specifier</a></li> <li><a class="reference internal" href="#custom-conversion-specifiers">Custom conversion specifiers</a></li> <li><a class="reference internal" href="#template-renderer-for-posix-shell-commands">Template renderer for POSIX shell commands</a></li> </ul> </li> <li><a class="reference internal" href="#background">Background</a><ul> <li><a class="reference internal" href="#summary-of-differences-from-f-strings">Summary of differences from f-strings</a></li> <li><a class="reference internal" href="#summary-of-differences-from-tagged-strings">Summary of differences from tagged strings</a></li> </ul> </li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#specification">Specification</a><ul> <li><a class="reference internal" href="#rendering-templates">Rendering templates</a></li> <li><a class="reference internal" href="#format-specifiers">Format specifiers</a></li> <li><a class="reference internal" href="#conversion-specifiers">Conversion specifiers</a></li> <li><a class="reference internal" href="#new-field-conversion-api-in-the-operator-module">New field conversion API in the <code class="xref py py-mod docutils literal notranslate">operator</code> module</a></li> <li><a class="reference internal" href="#conversion-specifier-parameter-added-to-format">Conversion specifier parameter added to <code class="xref py py-func docutils literal notranslate">format()</code></a></li> <li><a class="reference internal" href="#structural-typing-and-duck-typing">Structural typing and duck typing</a></li> <li><a class="reference internal" href="#writing-custom-renderers">Writing custom renderers</a></li> <li><a class="reference internal" href="#expression-evaluation">Expression evaluation</a></li> <li><a class="reference internal" href="#handling-code-injection-attacks">Handling code injection attacks</a></li> <li><a class="reference internal" href="#error-handling">Error handling</a></li> <li><a class="reference internal" href="#renderer-for-shell-escaping-added-to-shlex">Renderer for shell escaping added to <code class="xref py py-mod docutils literal notranslate">shlex</code></a></li> <li><a class="reference internal" href="#changes-to-subprocess-module">Changes to subprocess module</a></li> </ul> </li> <li><a class="reference internal" href="#how-to-teach-this">How to Teach This</a></li> <li><a class="reference internal" href="#discussion">Discussion</a><ul> <li><a class="reference internal" href="#support-for-binary-interpolation">Support for binary interpolation</a></li> <li><a class="reference internal" href="#interoperability-with-str-only-interfaces">Interoperability with str-only interfaces</a></li> <li><a class="reference internal" href="#preserving-the-raw-template-string">Preserving the raw template string</a></li> <li><a class="reference internal" href="#creating-a-rich-object-rather-than-a-global-name-lookup">Creating a rich object rather than a global name lookup</a></li> <li><a class="reference internal" href="#building-atop-f-strings-rather-than-replacing-them">Building atop f-strings rather than replacing them</a></li> <li><a class="reference internal" href="#defining-repetition-and-concatenation-semantics">Defining repetition and concatenation semantics</a></li> <li><a class="reference internal" href="#new-conversion-specifier-for-lazy-field-evaluation">New conversion specifier for lazy field evaluation</a></li> <li><a class="reference internal" href="#allowing-arbitrary-conversion-specifiers-in-custom-renderers">Allowing arbitrary conversion specifiers in custom renderers</a></li> <li><a class="reference internal" href="#only-reserving-a-single-new-string-prefix">Only reserving a single new string prefix</a></li> <li><a class="reference internal" href="#deferring-consideration-of-more-concise-delayed-evaluation-syntax">Deferring consideration of more concise delayed evaluation syntax</a></li> <li><a class="reference internal" href="#deferring-consideration-of-possible-logging-integration">Deferring consideration of possible logging integration</a></li> <li><a class="reference internal" href="#deferring-consideration-of-possible-use-in-i18n-use-cases">Deferring consideration of possible use in i18n use cases</a></li> <li><a class="reference internal" href="#deferring-escaped-rendering-support-for-non-posix-shells">Deferring escaped rendering support for non-POSIX shells</a></li> </ul> </li> <li><a class="reference internal" href="#acknowledgements">Acknowledgements</a></li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0501.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>