CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 444 – Python Web3 Interface | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0444/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 444 – Python Web3 Interface | peps.python.org'> <meta property="og:description" content="This document specifies a proposed second-generation standard interface between web servers and Python web applications or frameworks."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0444/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This document specifies a proposed second-generation standard interface between web servers and Python web applications or frameworks."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 444</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 444 – Python Web3 Interface</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Chris McDonough <chrism at plope.com>, Armin Ronacher <armin.ronacher at active-4.com></dd> <dt class="field-even">Discussions-To:</dt> <dd class="field-even"><a class="reference external" href="https://mail.python.org/mailman/listinfo/web-sig">Web-SIG list</a></dd> <dt class="field-odd">Status:</dt> <dd class="field-odd"><abbr title="Inactive draft that may be taken up again at a later time">Deferred</abbr></dd> <dt class="field-even">Type:</dt> <dd class="field-even"><abbr title="Non-normative PEP containing background, guidelines or other information relevant to the Python ecosystem">Informational</abbr></dd> <dt class="field-odd">Created:</dt> <dd class="field-odd">19-Jul-2010</dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#pep-deferral">PEP Deferral</a></li> <li><a class="reference internal" href="#rationale-and-goals">Rationale and Goals</a></li> <li><a class="reference internal" href="#differences-from-wsgi">Differences from WSGI</a></li> <li><a class="reference internal" href="#specification-overview">Specification Overview</a><ul> <li><a class="reference internal" href="#the-application-framework-side">The Application/Framework Side</a></li> <li><a class="reference internal" href="#the-server-gateway-side">The Server/Gateway Side</a></li> <li><a class="reference internal" href="#middleware-components-that-play-both-sides">Middleware: Components that Play Both Sides</a></li> </ul> </li> <li><a class="reference internal" href="#specification-details">Specification Details</a><ul> <li><a class="reference internal" href="#environ-variables"><code class="docutils literal notranslate">environ</code> Variables</a><ul> <li><a class="reference internal" href="#input-stream">Input Stream</a></li> <li><a class="reference internal" href="#error-stream">Error Stream</a></li> </ul> </li> <li><a class="reference internal" href="#values-returned-by-a-web3-application">Values Returned by A Web3 Application</a></li> <li><a class="reference internal" href="#dealing-with-compatibility-across-python-versions">Dealing with Compatibility Across Python Versions</a></li> <li><a class="reference internal" href="#buffering-and-streaming">Buffering and Streaming</a></li> <li><a class="reference internal" href="#unicode-issues">Unicode Issues</a></li> <li><a class="reference internal" href="#http-1-1-expect-continue">HTTP 1.1 Expect/Continue</a></li> <li><a class="reference internal" href="#other-http-features">Other HTTP Features</a></li> <li><a class="reference internal" href="#thread-support">Thread Support</a></li> </ul> </li> <li><a class="reference internal" href="#implementation-application-notes">Implementation/Application Notes</a><ul> <li><a class="reference internal" href="#server-extension-apis">Server Extension APIs</a></li> <li><a class="reference internal" href="#application-configuration">Application Configuration</a></li> <li><a class="reference internal" href="#url-reconstruction">URL Reconstruction</a></li> </ul> </li> <li><a class="reference internal" href="#open-questions">Open Questions</a></li> <li><a class="reference internal" href="#points-of-contention">Points of Contention</a><ul> <li><a class="reference internal" href="#wsgi-1-0-compatibility">WSGI 1.0 Compatibility</a></li> <li><a class="reference internal" href="#environ-and-response-values-as-bytes">Environ and Response Values as Bytes</a></li> <li><a class="reference internal" href="#applications-should-be-allowed-to-read-web3-input-past-content-length">Applications Should be Allowed to Read <code class="docutils literal notranslate">web3.input</code> Past <code class="docutils literal notranslate">CONTENT_LENGTH</code></a></li> <li><a class="reference internal" href="#web3-input-unknown-length"><code class="docutils literal notranslate">web3.input</code> Unknown Length</a></li> <li><a class="reference internal" href="#read-of-web3-input-should-support-no-size-calling-convention"><code class="docutils literal notranslate">read()</code> of <code class="docutils literal notranslate">web3.input</code> Should Support No-Size Calling Convention</a><ul> <li><a class="reference internal" href="#input-filters-should-set-environ-content-length-to-1">Input Filters should set environ <code class="docutils literal notranslate">CONTENT_LENGTH</code> to -1</a></li> </ul> </li> <li><a class="reference internal" href="#headers-as-literal-list-of-two-tuples"><code class="docutils literal notranslate">headers</code> as Literal List of Two-Tuples</a></li> <li><a class="reference internal" href="#removed-requirement-that-middleware-not-block">Removed Requirement that Middleware Not Block</a></li> <li><a class="reference internal" href="#web3-script-name-and-web3-path-info"><code class="docutils literal notranslate">web3.script_name</code> and <code class="docutils literal notranslate">web3.path_info</code></a></li> <li><a class="reference internal" href="#long-response-headers">Long Response Headers</a></li> <li><a class="reference internal" href="#request-trailers-and-chunked-transfer-encoding">Request Trailers and Chunked Transfer Encoding</a></li> </ul> </li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> This document specifies a proposed second-generation standard interface between web servers and Python web applications or frameworks. </section> <section id="pep-deferral"> <h2><a class="toc-backref" href="#pep-deferral" role="doc-backlink">PEP Deferral</a></h2> Further exploration of the concepts covered in this PEP has been deferred for lack of a current champion interested in promoting the goals of the PEP and collecting and incorporating feedback, and with sufficient available time to do so effectively. Note that since this PEP was first created, <a class="pep reference internal" href="../pep-3333/" title="PEP 3333 – Python Web Server Gateway Interface v1.0.1">PEP 3333</a> was created as a more incremental update that permitted use of WSGI on Python 3.2+. However, an alternative specification that furthers the Python 3 goals of a cleaner separation of binary and text data may still be valuable. </section> <section id="rationale-and-goals"> <h2><a class="toc-backref" href="#rationale-and-goals" role="doc-backlink">Rationale and Goals</a></h2> This protocol and specification is influenced heavily by the Web Services Gateway Interface (WSGI) 1.0 standard described in <a class="pep reference internal" href="../pep-0333/" title="PEP 333 – Python Web Server Gateway Interface v1.0">PEP 333</a>. The high-level rationale for having any standard that allows Python-based web servers and applications to interoperate is outlined in <a class="pep reference internal" href="../pep-0333/" title="PEP 333 – Python Web Server Gateway Interface v1.0">PEP 333</a>. This document essentially uses <a class="pep reference internal" href="../pep-0333/" title="PEP 333 – Python Web Server Gateway Interface v1.0">PEP 333</a> as a template, and changes its wording in various places for the purpose of forming a different standard. Python currently boasts a wide variety of web application frameworks which use the WSGI 1.0 protocol. However, due to changes in the language, the WSGI 1.0 protocol is not compatible with Python 3. This specification describes a standardized WSGI-like protocol that lets Python 2.6, 2.7 and 3.1+ applications communicate with web servers. Web3 is clearly a WSGI derivative; it only uses a different name than “WSGI” in order to indicate that it is not in any way backwards compatible. Applications and servers which are written to this specification are meant to work properly under Python 2.6.X, Python 2.7.X and Python 3.1+. Neither an application nor a server that implements the Web3 specification can be easily written which will work under Python 2 versions earlier than 2.6 nor Python 3 versions earlier than 3.1. <div class="admonition note"> Note Whatever Python 3 version fixed <a class="reference external" href="http://bugs.python.org/issue4006">http://bugs.python.org/issue4006</a> so <code class="docutils literal notranslate">os.environ['foo']</code> returns surrogates (ala <a class="pep reference internal" href="../pep-0383/" title="PEP 383 – Non-decodable Bytes in System Character Interfaces">PEP 383</a>) when the value of ‘foo’ cannot be decoded using the current locale instead of failing with a KeyError is the true minimum Python 3 version. In particular, however, Python 3.0 is not supported. </div> <div class="admonition note"> Note Python 2.6 is the first Python version that supported an alias for <code class="docutils literal notranslate">bytes</code> and the <code class="docutils literal notranslate">b"foo"</code> literal syntax. This is why it is the minimum version supported by Web3. </div> Explicability and documentability are the main technical drivers for the decisions made within the standard. </section> <section id="differences-from-wsgi"> <h2><a class="toc-backref" href="#differences-from-wsgi" role="doc-backlink">Differences from WSGI</a></h2> <ul class="simple"> <li>All protocol-specific environment names are prefixed with <code class="docutils literal notranslate">web3.</code> rather than <code class="docutils literal notranslate">wsgi.</code>, eg. <code class="docutils literal notranslate">web3.input</code> rather than <code class="docutils literal notranslate">wsgi.input</code>.</li> <li>All values present as environment dictionary values are explicitly bytes instances instead of native strings. (Environment keys however are native strings, always <code class="docutils literal notranslate">str</code> regardless of platform).</li> <li>All values returned by an application must be bytes instances, including status code, header names and values, and the body.</li> <li>Wherever WSGI 1.0 referred to an <code class="docutils literal notranslate">app_iter</code>, this specification refers to a <code class="docutils literal notranslate">body</code>.</li> <li>No <code class="docutils literal notranslate">start_response()</code> callback (and therefore no <code class="docutils literal notranslate">write()</code> callable nor <code class="docutils literal notranslate">exc_info</code> data).</li> <li>The <code class="docutils literal notranslate">readline()</code> function of <code class="docutils literal notranslate">web3.input</code> must support a size hint parameter.</li> <li>The <code class="docutils literal notranslate">read()</code> function of <code class="docutils literal notranslate">web3.input</code> must be length delimited. A call without a size argument must not read more than the content length header specifies. In case a content length header is absent the stream must not return anything on read. It must never request more data than specified from the client.</li> <li>No requirement for middleware to yield an empty string if it needs more information from an application to produce output (e.g. no “Middleware Handling of Block Boundaries”).</li> <li>Filelike objects passed to a “file_wrapper” must have an <code class="docutils literal notranslate">__iter__</code> which returns bytes (never text).</li> <li><code class="docutils literal notranslate">wsgi.file_wrapper</code> is not supported.</li> <li><code class="docutils literal notranslate">QUERY_STRING</code>, <code class="docutils literal notranslate">SCRIPT_NAME</code>, <code class="docutils literal notranslate">PATH_INFO</code> values required to be placed in environ by server (each as the empty bytes instance if no associated value is received in the HTTP request).</li> <li><code class="docutils literal notranslate">web3.path_info</code> and <code class="docutils literal notranslate">web3.script_name</code> should be put into the Web3 environment, if possible, by the origin Web3 server. When available, each is the original, plain 7-bit ASCII, URL-encoded variant of its CGI equivalent derived directly from the request URI (with %2F segment markers and other meta-characters intact). If the server cannot provide one (or both) of these values, it must omit the value(s) it cannot provide from the environment.</li> <li>This requirement was removed: “middleware components must not block iteration waiting for multiple values from an application iterable. If the middleware needs to accumulate more data from the application before it can produce any output, it must yield an empty string.”</li> <li><code class="docutils literal notranslate">SERVER_PORT</code> must be a bytes instance (not an integer).</li> <li>The server must not inject an additional <code class="docutils literal notranslate">Content-Length</code> header by guessing the length from the response iterable. This must be set by the application itself in all situations.</li> <li>If the origin server advertises that it has the <code class="docutils literal notranslate">web3.async</code> capability, a Web3 application callable used by the server is permitted to return a callable that accepts no arguments. When it does so, this callable is to be called periodically by the origin server until it returns a non-<code class="docutils literal notranslate">None</code> response, which must be a normal Web3 response tuple.</li> </ul> </section> <section id="specification-overview"> <h2><a class="toc-backref" href="#specification-overview" role="doc-backlink">Specification Overview</a></h2> The Web3 interface has two sides: the “server” or “gateway” side, and the “application” or “framework” side. The server side invokes a callable object that is provided by the application side. The specifics of how that object is provided are up to the server or gateway. It is assumed that some servers or gateways will require an application’s deployer to write a short script to create an instance of the server or gateway, and supply it with the application object. Other servers and gateways may use configuration files or other mechanisms to specify where an application object should be imported from, or otherwise obtained. In addition to “pure” servers/gateways and applications/frameworks, it is also possible to create “middleware” components that implement both sides of this specification. Such components act as an application to their containing server, and as a server to a contained application, and can be used to provide extended APIs, content transformation, navigation, and other useful functions. Throughout this specification, we will use the term “application callable” to mean “a function, a method, or an instance with a <code class="docutils literal notranslate">__call__</code> method”. It is up to the server, gateway, or application implementing the application callable to choose the appropriate implementation technique for their needs. Conversely, a server, gateway, or application that is invoking a callable must not have any dependency on what kind of callable was provided to it. Application callables are only to be called, not introspected upon. <section id="the-application-framework-side"> <h3><a class="toc-backref" href="#the-application-framework-side" role="doc-backlink">The Application/Framework Side</a></h3> The application object is simply a callable object that accepts one argument. The term “object” should not be misconstrued as requiring an actual object instance: a function, method, or instance with a <code class="docutils literal notranslate">__call__</code> method are all acceptable for use as an application object. Application objects must be able to be invoked more than once, as virtually all servers/gateways (other than CGI) will make such repeated requests. If this cannot be guaranteed by the implementation of the actual application, it has to be wrapped in a function that creates a new instance on each call. <div class="admonition note"> Note Although we refer to it as an “application” object, this should not be construed to mean that application developers will use Web3 as a web programming API. It is assumed that application developers will continue to use existing, high-level framework services to develop their applications. Web3 is a tool for framework and server developers, and is not intended to directly support application developers.) </div> An example of an application which is a function (<code class="docutils literal notranslate">simple_app</code>): <div class="highlight-default notranslate"><div class="highlight"><pre>def simple_app(environ): """Simplest possible application object""" status = b'200 OK' headers = [(b'Content-type', b'text/plain')] body = [b'Hello world!\n'] return body, status, headers </pre></div> </div> An example of an application which is an instance (<code class="docutils literal notranslate">simple_app</code>): <div class="highlight-default notranslate"><div class="highlight"><pre>class AppClass(object): """Produce the same output, but using an instance. An instance of this class must be instantiated before it is passed to the server. """ def __call__(self, environ): status = b'200 OK' headers = [(b'Content-type', b'text/plain')] body = [b'Hello world!\n'] return body, status, headers simple_app = AppClass() </pre></div> </div> Alternately, an application callable may return a callable instead of the tuple if the server supports asynchronous execution. See information concerning <code class="docutils literal notranslate">web3.async</code> for more information. </section> <section id="the-server-gateway-side"> <h3><a class="toc-backref" href="#the-server-gateway-side" role="doc-backlink">The Server/Gateway Side</a></h3> The server or gateway invokes the application callable once for each request it receives from an HTTP client, that is directed at the application. To illustrate, here is a simple CGI gateway, implemented as a function taking an application object. Note that this simple example has limited error handling, because by default an uncaught exception will be dumped to <code class="docutils literal notranslate">sys.stderr</code> and logged by the web server. <div class="highlight-default notranslate"><div class="highlight"><pre>import locale import os import sys encoding = locale.getpreferredencoding() stdout = sys.stdout if hasattr(sys.stdout, 'buffer'): # Python 3 compatibility; we need to be able to push bytes out stdout = sys.stdout.buffer def get_environ(): d = {} for k, v in os.environ.items(): # Python 3 compatibility if not isinstance(v, bytes): # We must explicitly encode the string to bytes under # Python 3.1+ v = v.encode(encoding, 'surrogateescape') d[k] = v return d def run_with_cgi(application): environ = get_environ() environ['web3.input'] = sys.stdin environ['web3.errors'] = sys.stderr environ['web3.version'] = (1, 0) environ['web3.multithread'] = False environ['web3.multiprocess'] = True environ['web3.run_once'] = True environ['web3.async'] = False if environ.get('HTTPS', b'off') in (b'on', b'1'): environ['web3.url_scheme'] = b'https' else: environ['web3.url_scheme'] = b'http' rv = application(environ) if hasattr(rv, '__call__'): raise TypeError('This webserver does not support asynchronous ' 'responses.') body, status, headers = rv CLRF = b'\r\n' try: stdout.write(b'Status: ' + status + CRLF) for header_name, header_val in headers: stdout.write(header_name + b': ' + header_val + CRLF) stdout.write(CRLF) for chunk in body: stdout.write(chunk) stdout.flush() finally: if hasattr(body, 'close'): body.close() </pre></div> </div> </section> <section id="middleware-components-that-play-both-sides"> <h3><a class="toc-backref" href="#middleware-components-that-play-both-sides" role="doc-backlink">Middleware: Components that Play Both Sides</a></h3> A single object may play the role of a server with respect to some application(s), while also acting as an application with respect to some server(s). Such “middleware” components can perform such functions as: <ul class="simple"> <li>Routing a request to different application objects based on the target URL, after rewriting the <code class="docutils literal notranslate">environ</code> accordingly.</li> <li>Allowing multiple applications or frameworks to run side by side in the same process.</li> <li>Load balancing and remote processing, by forwarding requests and responses over a network.</li> <li>Perform content postprocessing, such as applying XSL stylesheets.</li> </ul> The presence of middleware in general is transparent to both the “server/gateway” and the “application/framework” sides of the interface, and should require no special support. A user who desires to incorporate middleware into an application simply provides the middleware component to the server, as if it were an application, and configures the middleware component to invoke the application, as if the middleware component were a server. Of course, the “application” that the middleware wraps may in fact be another middleware component wrapping another application, and so on, creating what is referred to as a “middleware stack”. A middleware must support asynchronous execution if possible or fall back to disabling itself. Here a middleware that changes the <code class="docutils literal notranslate">HTTP_HOST</code> key if an <code class="docutils literal notranslate">X-Host</code> header exists and adds a comment to all html responses: <div class="highlight-default notranslate"><div class="highlight"><pre>import time def apply_filter(app, environ, filter_func): """Helper function that passes the return value from an application to a filter function when the results are ready. """ app_response = app(environ) # synchronous response, filter now if not hasattr(app_response, '__call__'): return filter_func(*app_response) # asynchronous response. filter when results are ready def polling_function(): rv = app_response() if rv is not None: return filter_func(*rv) return polling_function def proxy_and_timing_support(app): def new_application(environ): def filter_func(body, status, headers): now = time.time() for key, value in headers: if key.lower() == b'content-type' and \ value.split(b';')[0] == b'text/html': # assumes ascii compatible encoding in body, # but the middleware should actually parse the # content type header and figure out the # encoding when doing that. body += ('' % (now - then)).encode('ascii') break return body, status, headers then = time.time() host = environ.get('HTTP_X_HOST') if host is not None: environ['HTTP_HOST'] = host # use the apply_filter function that applies a given filter # function for both async and sync responses. return apply_filter(app, environ, filter_func) return new_application app = proxy_and_timing_support(app) </pre></div> </div> </section> </section> <section id="specification-details"> <h2><a class="toc-backref" href="#specification-details" role="doc-backlink">Specification Details</a></h2> The application callable must accept one positional argument. For the sake of illustration, we have named it <code class="docutils literal notranslate">environ</code>, but it is not required to have this name. A server or gateway must invoke the application object using a positional (not keyword) argument. (E.g. by calling <code class="docutils literal notranslate">body, status, headers = application(environ)</code> as shown above.) The <code class="docutils literal notranslate">environ</code> parameter is a dictionary object, containing CGI-style environment variables. This object must be a builtin Python dictionary (not a subclass, <code class="docutils literal notranslate">UserDict</code> or other dictionary emulation), and the application is allowed to modify the dictionary in any way it desires. The dictionary must also include certain Web3-required variables (described in a later section), and may also include server-specific extension variables, named according to a convention that will be described below. When called by the server, the application object must return a tuple yielding three elements: <code class="docutils literal notranslate">status</code>, <code class="docutils literal notranslate">headers</code> and <code class="docutils literal notranslate">body</code>, or, if supported by an async server, an argumentless callable which either returns <code class="docutils literal notranslate">None</code> or a tuple of those three elements. The <code class="docutils literal notranslate">status</code> element is a status in bytes of the form <code class="docutils literal notranslate">b'999 Message here'</code>. <code class="docutils literal notranslate">headers</code> is a Python list of <code class="docutils literal notranslate">(header_name, header_value)</code> pairs describing the HTTP response header. The <code class="docutils literal notranslate">headers</code> structure must be a literal Python list; it must yield two-tuples. Both <code class="docutils literal notranslate">header_name</code> and <code class="docutils literal notranslate">header_value</code> must be bytes values. The <code class="docutils literal notranslate">body</code> is an iterable yielding zero or more bytes instances. This can be accomplished in a variety of ways, such as by returning a list containing bytes instances as <code class="docutils literal notranslate">body</code>, or by returning a generator function as <code class="docutils literal notranslate">body</code> that yields bytes instances, or by the <code class="docutils literal notranslate">body</code> being an instance of a class which is iterable. Regardless of how it is accomplished, the application object must always return a <code class="docutils literal notranslate">body</code> iterable yielding zero or more bytes instances. The server or gateway must transmit the yielded bytes to the client in an unbuffered fashion, completing the transmission of each set of bytes before requesting another one. (In other words, applications should perform their own buffering. See the <a class="reference internal" href="#buffering-and-streaming">Buffering and Streaming</a> section below for more on how application output must be handled.) The server or gateway should treat the yielded bytes as binary byte sequences: in particular, it should ensure that line endings are not altered. The application is responsible for ensuring that the string(s) to be written are in a format suitable for the client. (The server or gateway may apply HTTP transfer encodings, or perform other transformations for the purpose of implementing HTTP features such as byte-range transmission. See <a class="reference internal" href="#other-http-features">Other HTTP Features</a>, below, for more details.) If the <code class="docutils literal notranslate">body</code> iterable returned by the application has a <code class="docutils literal notranslate">close()</code> method, the server or gateway must call that method upon completion of the current request, whether the request was completed normally, or terminated early due to an error. This is to support resource release by the application amd is intended to complement PEP 325’s generator support, and other common iterables with <code class="docutils literal notranslate">close()</code> methods. Finally, servers and gateways must not directly use any other attributes of the <code class="docutils literal notranslate">body</code> iterable returned by the application. <section id="environ-variables"> <h3><a class="toc-backref" href="#environ-variables" role="doc-backlink"><code class="docutils literal notranslate">environ</code> Variables</a></h3> The <code class="docutils literal notranslate">environ</code> dictionary is required to contain various CGI environment variables, as defined by the Common Gateway Interface specification <a class="footnote-reference brackets" href="#id9" id="id1">[2]</a>. The following CGI variables must be present. Each key is a native string. Each value is a bytes instance. <div class="admonition note"> Note In Python 3.1+, a “native string” is a <code class="docutils literal notranslate">str</code> type decoded using the <code class="docutils literal notranslate">surrogateescape</code> error handler, as done by <code class="docutils literal notranslate">os.environ.__getitem__</code>. In Python 2.6 and 2.7, a “native string” is a <code class="docutils literal notranslate">str</code> types representing a set of bytes. </div> <dl class="simple"> <dt><code class="docutils literal notranslate">REQUEST_METHOD</code></dt><dd>The HTTP request method, such as <code class="docutils literal notranslate">"GET"</code> or <code class="docutils literal notranslate">"POST"</code>.</dd> <dt><code class="docutils literal notranslate">SCRIPT_NAME</code></dt><dd>The initial portion of the request URL’s “path” that corresponds to the application object, so that the application knows its virtual “location”. This may be the empty bytes instance if the application corresponds to the “root” of the server. SCRIPT_NAME will be a bytes instance representing a sequence of URL-encoded segments separated by the slash character (<code class="docutils literal notranslate">/</code>). It is assumed that <code class="docutils literal notranslate">%2F</code> characters will be decoded into literal slash characters within <code class="docutils literal notranslate">PATH_INFO</code>, as per CGI.</dd> <dt><code class="docutils literal notranslate">PATH_INFO</code></dt><dd>The remainder of the request URL’s “path”, designating the virtual “location” of the request’s target within the application. This may be a bytes instance if the request URL targets the application root and does not have a trailing slash. PATH_INFO will be a bytes instance representing a sequence of URL-encoded segments separated by the slash character (<code class="docutils literal notranslate">/</code>). It is assumed that <code class="docutils literal notranslate">%2F</code> characters will be decoded into literal slash characters within <code class="docutils literal notranslate">PATH_INFO</code>, as per CGI.</dd> <dt><code class="docutils literal notranslate">QUERY_STRING</code></dt><dd>The portion of the request URL (in bytes) that follows the <code class="docutils literal notranslate">"?"</code>, if any, or the empty bytes instance.</dd> <dt><code class="docutils literal notranslate">SERVER_NAME</code>, <code class="docutils literal notranslate">SERVER_PORT</code></dt><dd>When combined with <code class="docutils literal notranslate">SCRIPT_NAME</code> and <code class="docutils literal notranslate">PATH_INFO</code> (or their raw equivalents), these variables can be used to complete the URL. Note, however, that <code class="docutils literal notranslate">HTTP_HOST</code>, if present, should be used in preference to <code class="docutils literal notranslate">SERVER_NAME</code> for reconstructing the request URL. See the <a class="reference internal" href="#url-reconstruction">URL Reconstruction</a> section below for more detail. <code class="docutils literal notranslate">SERVER_PORT</code> should be a bytes instance, not an integer.</dd> <dt><code class="docutils literal notranslate">SERVER_PROTOCOL</code></dt><dd>The version of the protocol the client used to send the request. Typically this will be something like <code class="docutils literal notranslate">"HTTP/1.0"</code> or <code class="docutils literal notranslate">"HTTP/1.1"</code> and may be used by the application to determine how to treat any HTTP request headers. (This variable should probably be called <code class="docutils literal notranslate">REQUEST_PROTOCOL</code>, since it denotes the protocol used in the request, and is not necessarily the protocol that will be used in the server’s response. However, for compatibility with CGI we have to keep the existing name.)</dd> </dl> The following CGI values may present be in the Web3 environment. Each key is a native string. Each value is a bytes instances. <dl class="simple"> <dt><code class="docutils literal notranslate">CONTENT_TYPE</code></dt><dd>The contents of any <code class="docutils literal notranslate">Content-Type</code> fields in the HTTP request.</dd> <dt><code class="docutils literal notranslate">CONTENT_LENGTH</code></dt><dd>The contents of any <code class="docutils literal notranslate">Content-Length</code> fields in the HTTP request.</dd> <dt><code class="docutils literal notranslate">HTTP_</code> Variables</dt><dd>Variables corresponding to the client-supplied HTTP request headers (i.e., variables whose names begin with <code class="docutils literal notranslate">"HTTP_"</code>). The presence or absence of these variables should correspond with the presence or absence of the appropriate HTTP header in the request.</dd> </dl> A server or gateway should attempt to provide as many other CGI variables as are applicable, each with a string for its key and a bytes instance for its value. In addition, if SSL is in use, the server or gateway should also provide as many of the Apache SSL environment variables <a class="footnote-reference brackets" href="#id11" id="id2">[4]</a> as are applicable, such as <code class="docutils literal notranslate">HTTPS=on</code> and <code class="docutils literal notranslate">SSL_PROTOCOL</code>. Note, however, that an application that uses any CGI variables other than the ones listed above are necessarily non-portable to web servers that do not support the relevant extensions. (For example, web servers that do not publish files will not be able to provide a meaningful <code class="docutils literal notranslate">DOCUMENT_ROOT</code> or <code class="docutils literal notranslate">PATH_TRANSLATED</code>.) A Web3-compliant server or gateway should document what variables it provides, along with their definitions as appropriate. Applications should check for the presence of any variables they require, and have a fallback plan in the event such a variable is absent. Note that CGI variable values must be bytes instances, if they are present at all. It is a violation of this specification for a CGI variable’s value to be of any type other than <code class="docutils literal notranslate">bytes</code>. On Python 2, this means they will be of type <code class="docutils literal notranslate">str</code>. On Python 3, this means they will be of type <code class="docutils literal notranslate">bytes</code>. They keys of all CGI and non-CGI variables in the environ, however, must be “native strings” (on both Python 2 and Python 3, they will be of type <code class="docutils literal notranslate">str</code>). In addition to the CGI-defined variables, the <code class="docutils literal notranslate">environ</code> dictionary may also contain arbitrary operating-system “environment variables”, and must contain the following Web3-defined variables. <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Variable</th> <th class="head">Value</th> </tr> </thead> <tbody> <tr class="row-even"><td><code class="docutils literal notranslate">web3.version</code></td> <td>The tuple <code class="docutils literal notranslate">(1, 0)</code>, representing Web3 version 1.0.</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">web3.url_scheme</code></td> <td>A bytes value representing the “scheme” portion of the URL at which the application is being invoked. Normally, this will have the value <code class="docutils literal notranslate">b"http"</code> or <code class="docutils literal notranslate">b"https"</code>, as appropriate.</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">web3.input</code></td> <td>An input stream (file-like object) from which bytes constituting the HTTP request body can be read. (The server or gateway may perform reads on-demand as requested by the application, or it may pre- read the client’s request body and buffer it in-memory or on disk, or use any other technique for providing such an input stream, according to its preference.)</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">web3.errors</code></td> <td>An output stream (file-like object) to which error output text can be written, for the purpose of recording program or other errors in a standardized and possibly centralized location. This should be a “text mode” stream; i.e., applications should use <code class="docutils literal notranslate">"\n"</code> as a line ending, and assume that it will be converted to the correct line ending by the server/gateway. Applications may not send bytes to the ‘write’ method of this stream; they may only send text.For many servers, <code class="docutils literal notranslate">web3.errors</code> will be the server’s main error log. Alternatively, this may be <code class="docutils literal notranslate">sys.stderr</code>, or a log file of some sort. The server’s documentation should include an explanation of how to configure this or where to find the recorded output. A server or gateway may supply different error streams to different applications, if this is desired. </td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">web3.multithread</code></td> <td>This value should evaluate true if the application object may be simultaneously invoked by another thread in the same process, and should evaluate false otherwise.</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">web3.multiprocess</code></td> <td>This value should evaluate true if an equivalent application object may be simultaneously invoked by another process, and should evaluate false otherwise.</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">web3.run_once</code></td> <td>This value should evaluate true if the server or gateway expects (but does not guarantee!) that the application will only be invoked this one time during the life of its containing process. Normally, this will only be true for a gateway based on CGI (or something similar).</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">web3.script_name</code></td> <td>The non-URL-decoded <code class="docutils literal notranslate">SCRIPT_NAME</code> value. Through a historical inequity, by virtue of the CGI specification, <code class="docutils literal notranslate">SCRIPT_NAME</code> is present within the environment as an already URL-decoded string. This is the original URL-encoded value derived from the request URI. If the server cannot provide this value, it must omit it from the environ.</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">web3.path_info</code></td> <td>The non-URL-decoded <code class="docutils literal notranslate">PATH_INFO</code> value. Through a historical inequity, by virtue of the CGI specification, <code class="docutils literal notranslate">PATH_INFO</code> is present within the environment as an already URL-decoded string. This is the original URL-encoded value derived from the request URI. If the server cannot provide this value, it must omit it from the environ.</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">web3.async</code></td> <td>This is <code class="docutils literal notranslate">True</code> if the webserver supports async invocation. In that case an application is allowed to return a callable instead of a tuple with the response. The exact semantics are not specified by this specification.</td> </tr> </tbody> </table> Finally, the <code class="docutils literal notranslate">environ</code> dictionary may also contain server-defined variables. These variables should have names which are native strings, composed of only lower-case letters, numbers, dots, and underscores, and should be prefixed with a name that is unique to the defining server or gateway. For example, <code class="docutils literal notranslate">mod_web3</code> might define variables with names like <code class="docutils literal notranslate">mod_web3.some_variable</code>. <section id="input-stream"> <h4><a class="toc-backref" href="#input-stream" role="doc-backlink">Input Stream</a></h4> The input stream (<code class="docutils literal notranslate">web3.input</code>) provided by the server must support the following methods: <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Method</th> <th class="head">Notes</th> </tr> </thead> <tbody> <tr class="row-even"><td><code class="docutils literal notranslate">read(size)</code></td> <td>1,4</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">readline([size])</code></td> <td>1,2,4</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">readlines([size])</code></td> <td>1,3,4</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">__iter__()</code></td> <td>4</td> </tr> </tbody> </table> The semantics of each method are as documented in the Python Library Reference, except for these notes as listed in the table above: <ol class="arabic simple"> <li>The server is not required to read past the client’s specified <code class="docutils literal notranslate">Content-Length</code>, and is allowed to simulate an end-of-file condition if the application attempts to read past that point. The application should not attempt to read more data than is specified by the <code class="docutils literal notranslate">CONTENT_LENGTH</code> variable.</li> <li>The implementation must support the optional <code class="docutils literal notranslate">size</code> argument to <code class="docutils literal notranslate">readline()</code>.</li> <li>The application is free to not supply a <code class="docutils literal notranslate">size</code> argument to <code class="docutils literal notranslate">readlines()</code>, and the server or gateway is free to ignore the value of any supplied <code class="docutils literal notranslate">size</code> argument.</li> <li>The <code class="docutils literal notranslate">read</code>, <code class="docutils literal notranslate">readline</code> and <code class="docutils literal notranslate">__iter__</code> methods must return a bytes instance. The <code class="docutils literal notranslate">readlines</code> method must return a sequence which contains instances of bytes.</li> </ol> The methods listed in the table above must be supported by all servers conforming to this specification. Applications conforming to this specification must not use any other methods or attributes of the <code class="docutils literal notranslate">input</code> object. In particular, applications must not attempt to close this stream, even if it possesses a <code class="docutils literal notranslate">close()</code> method. The input stream should silently ignore attempts to read more than the content length of the request. If no content length is specified the stream must be a dummy stream that does not return anything. </section> <section id="error-stream"> <h4><a class="toc-backref" href="#error-stream" role="doc-backlink">Error Stream</a></h4> The error stream (<code class="docutils literal notranslate">web3.errors</code>) provided by the server must support the following methods: <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Method</th> <th class="head">Stream</th> <th class="head">Notes</th> </tr> </thead> <tbody> <tr class="row-even"><td><code class="docutils literal notranslate">flush()</code></td> <td><code class="docutils literal notranslate">errors</code></td> <td>1</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">write(str)</code></td> <td><code class="docutils literal notranslate">errors</code></td> <td>2</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">writelines(seq)</code></td> <td><code class="docutils literal notranslate">errors</code></td> <td>2</td> </tr> </tbody> </table> The semantics of each method are as documented in the Python Library Reference, except for these notes as listed in the table above: <ol class="arabic simple"> <li>Since the <code class="docutils literal notranslate">errors</code> stream may not be rewound, servers and gateways are free to forward write operations immediately, without buffering. In this case, the <code class="docutils literal notranslate">flush()</code> method may be a no-op. Portable applications, however, cannot assume that output is unbuffered or that <code class="docutils literal notranslate">flush()</code> is a no-op. They must call <code class="docutils literal notranslate">flush()</code> if they need to ensure that output has in fact been written. (For example, to minimize intermingling of data from multiple processes writing to the same error log.)</li> <li>The <code class="docutils literal notranslate">write()</code> method must accept a string argument, but needn’t necessarily accept a bytes argument. The <code class="docutils literal notranslate">writelines()</code> method must accept a sequence argument that consists entirely of strings, but needn’t necessarily accept any bytes instance as a member of the sequence.</li> </ol> The methods listed in the table above must be supported by all servers conforming to this specification. Applications conforming to this specification must not use any other methods or attributes of the <code class="docutils literal notranslate">errors</code> object. In particular, applications must not attempt to close this stream, even if it possesses a <code class="docutils literal notranslate">close()</code> method. </section> </section> <section id="values-returned-by-a-web3-application"> <h3><a class="toc-backref" href="#values-returned-by-a-web3-application" role="doc-backlink">Values Returned by A Web3 Application</a></h3> Web3 applications return a tuple in the form (<code class="docutils literal notranslate">status</code>, <code class="docutils literal notranslate">headers</code>, <code class="docutils literal notranslate">body</code>). If the server supports asynchronous applications (<code class="docutils literal notranslate">web3.async</code>), the response may be a callable object (which accepts no arguments). The <code class="docutils literal notranslate">status</code> value is assumed by a gateway or server to be an HTTP “status” bytes instance like <code class="docutils literal notranslate">b'200 OK'</code> or <code class="docutils literal notranslate">b'404 Not Found'</code>. That is, it is a string consisting of a Status-Code and a Reason-Phrase, in that order and separated by a single space, with no surrounding whitespace or other characters. (See <a class="rfc reference external" href="https://datatracker.ietf.org/doc/html/rfc2616.html">RFC 2616</a>, Section 6.1.1 for more information.) The string must not contain control characters, and must not be terminated with a carriage return, linefeed, or combination thereof. The <code class="docutils literal notranslate">headers</code> value is assumed by a gateway or server to be a literal Python list of <code class="docutils literal notranslate">(header_name, header_value)</code> tuples. Each <code class="docutils literal notranslate">header_name</code> must be a bytes instance representing a valid HTTP header field-name (as defined by <a class="rfc reference external" href="https://datatracker.ietf.org/doc/html/rfc2616.html">RFC 2616</a>, Section 4.2), without a trailing colon or other punctuation. Each <code class="docutils literal notranslate">header_value</code> must be a bytes instance and must not include any control characters, including carriage returns or linefeeds, either embedded or at the end. (These requirements are to minimize the complexity of any parsing that must be performed by servers, gateways, and intermediate response processors that need to inspect or modify response headers.) In general, the server or gateway is responsible for ensuring that correct headers are sent to the client: if the application omits a header required by HTTP (or other relevant specifications that are in effect), the server or gateway must add it. For example, the HTTP <code class="docutils literal notranslate">Date:</code> and <code class="docutils literal notranslate">Server:</code> headers would normally be supplied by the server or gateway. The gateway must however not override values with the same name if they are emitted by the application. (A reminder for server/gateway authors: HTTP header names are case-insensitive, so be sure to take that into consideration when examining application-supplied headers!) Applications and middleware are forbidden from using HTTP/1.1 “hop-by-hop” features or headers, any equivalent features in HTTP/1.0, or any headers that would affect the persistence of the client’s connection to the web server. These features are the exclusive province of the actual web server, and a server or gateway should consider it a fatal error for an application to attempt sending them, and raise an error if they are supplied as return values from an application in the <code class="docutils literal notranslate">headers</code> structure. (For more specifics on “hop-by-hop” features and headers, please see the <a class="reference internal" href="#other-http-features">Other HTTP Features</a> section below.) </section> <section id="dealing-with-compatibility-across-python-versions"> <h3><a class="toc-backref" href="#dealing-with-compatibility-across-python-versions" role="doc-backlink">Dealing with Compatibility Across Python Versions</a></h3> Creating Web3 code that runs under both Python 2.6/2.7 and Python 3.1+ requires some care on the part of the developer. In general, the Web3 specification assumes a certain level of equivalence between the Python 2 <code class="docutils literal notranslate">str</code> type and the Python 3 <code class="docutils literal notranslate">bytes</code> type. For example, under Python 2, the values present in the Web3 <code class="docutils literal notranslate">environ</code> will be instances of the <code class="docutils literal notranslate">str</code> type; in Python 3, these will be instances of the <code class="docutils literal notranslate">bytes</code> type. The Python 3 <code class="docutils literal notranslate">bytes</code> type does not possess all the methods of the Python 2 <code class="docutils literal notranslate">str</code> type, and some methods which it does possess behave differently than the Python 2 <code class="docutils literal notranslate">str</code> type. Effectively, to ensure that Web3 middleware and applications work across Python versions, developers must do these things: <ol class="arabic simple"> <li>Do not assume comparison equivalence between text values and bytes values. If you do so, your code may work under Python 2, but it will not work properly under Python 3. For example, don’t write <code class="docutils literal notranslate">somebytes == 'abc'</code>. This will sometimes be true on Python 2 but it will never be true on Python 3, because a sequence of bytes never compares equal to a string under Python 3. Instead, always compare a bytes value with a bytes value, e.g. “somebytes == b’abc’”. Code which does this is compatible with and works the same in Python 2.6, 2.7, and 3.1. The <code class="docutils literal notranslate">b</code> in front of <code class="docutils literal notranslate">'abc'</code> signals to Python 3 that the value is a literal bytes instance; under Python 2 it’s a forward compatibility placebo.</li> <li>Don’t use the <code class="docutils literal notranslate">__contains__</code> method (directly or indirectly) of items that are meant to be byteslike without ensuring that its argument is also a bytes instance. If you do so, your code may work under Python 2, but it will not work properly under Python 3. For example, <code class="docutils literal notranslate">'abc' in somebytes'</code> will raise a <code class="docutils literal notranslate">TypeError</code> under Python 3, but it will return <code class="docutils literal notranslate">True</code> under Python 2.6 and 2.7. However, <code class="docutils literal notranslate">b'abc' in somebytes</code> will work the same on both versions. In Python 3.2, this restriction may be partially removed, as it’s rumored that bytes types may obtain a <code class="docutils literal notranslate">__mod__</code> implementation.</li> <li><code class="docutils literal notranslate">__getitem__</code> should not be used.</li> <li>Don’t try to use the <code class="docutils literal notranslate">format</code> method or the <code class="docutils literal notranslate">__mod__</code> method of instances of bytes (directly or indirectly). In Python 2, the <code class="docutils literal notranslate">str</code> type which we treat equivalently to Python 3’s <code class="docutils literal notranslate">bytes</code> supports these method but actual Python 3’s <code class="docutils literal notranslate">bytes</code> instances don’t support these methods. If you use these methods, your code will work under Python 2, but not under Python 3.</li> <li>Do not try to concatenate a bytes value with a string value. This may work under Python 2, but it will not work under Python 3. For example, doing <code class="docutils literal notranslate">'abc' + somebytes</code> will work under Python 2, but it will result in a <code class="docutils literal notranslate">TypeError</code> under Python 3. Instead, always make sure you’re concatenating two items of the same type, e.g. <code class="docutils literal notranslate">b'abc' + somebytes</code>.</li> </ol> Web3 expects byte values in other places, such as in all the values returned by an application. In short, to ensure compatibility of Web3 application code between Python 2 and Python 3, in Python 2, treat CGI and server variable values in the environment as if they had the Python 3 <code class="docutils literal notranslate">bytes</code> API even though they actually have a more capable API. Likewise for all stringlike values returned by a Web3 application. </section> <section id="buffering-and-streaming"> <h3><a class="toc-backref" href="#buffering-and-streaming" role="doc-backlink">Buffering and Streaming</a></h3> Generally speaking, applications will achieve the best throughput by buffering their (modestly-sized) output and sending it all at once. This is a common approach in existing frameworks: the output is buffered in a StringIO or similar object, then transmitted all at once, along with the response headers. The corresponding approach in Web3 is for the application to simply return a single-element <code class="docutils literal notranslate">body</code> iterable (such as a list) containing the response body as a single string. This is the recommended approach for the vast majority of application functions, that render HTML pages whose text easily fits in memory. For large files, however, or for specialized uses of HTTP streaming (such as multipart “server push”), an application may need to provide output in smaller blocks (e.g. to avoid loading a large file into memory). It’s also sometimes the case that part of a response may be time-consuming to produce, but it would be useful to send ahead the portion of the response that precedes it. In these cases, applications will usually return a <code class="docutils literal notranslate">body</code> iterator (often a generator-iterator) that produces the output in a block-by-block fashion. These blocks may be broken to coincide with multipart boundaries (for “server push”), or just before time-consuming tasks (such as reading another block of an on-disk file). Web3 servers, gateways, and middleware must not delay the transmission of any block; they must either fully transmit the block to the client, or guarantee that they will continue transmission even while the application is producing its next block. A server/gateway or middleware may provide this guarantee in one of three ways: <ol class="arabic simple"> <li>Send the entire block to the operating system (and request that any O/S buffers be flushed) before returning control to the application, OR</li> <li>Use a different thread to ensure that the block continues to be transmitted while the application produces the next block.</li> <li>(Middleware only) send the entire block to its parent gateway/server.</li> </ol> By providing this guarantee, Web3 allows applications to ensure that transmission will not become stalled at an arbitrary point in their output data. This is critical for proper functioning of e.g. multipart “server push” streaming, where data between multipart boundaries should be transmitted in full to the client. </section> <section id="unicode-issues"> <h3><a class="toc-backref" href="#unicode-issues" role="doc-backlink">Unicode Issues</a></h3> HTTP does not directly support Unicode, and neither does this interface. All encoding/decoding must be handled by the application; all values passed to or from the server must be of the Python 3 type <code class="docutils literal notranslate">bytes</code> or instances of the Python 2 type <code class="docutils literal notranslate">str</code>, not Python 2 <code class="docutils literal notranslate">unicode</code> or Python 3 <code class="docutils literal notranslate">str</code> objects. All “bytes instances” referred to in this specification must: <ul class="simple"> <li>On Python 2, be of type <code class="docutils literal notranslate">str</code>.</li> <li>On Python 3, be of type <code class="docutils literal notranslate">bytes</code>.</li> </ul> All “bytes instances” must not : <ul class="simple"> <li>On Python 2, be of type <code class="docutils literal notranslate">unicode</code>.</li> <li>On Python 3, be of type <code class="docutils literal notranslate">str</code>.</li> </ul> The result of using a textlike object where a byteslike object is required is undefined. Values returned from a Web3 app as a status or as response headers must follow <a class="rfc reference external" href="https://datatracker.ietf.org/doc/html/rfc2616.html">RFC 2616</a> with respect to encoding. That is, the bytes returned must contain a character stream of ISO-8859-1 characters, or the character stream should use <a class="rfc reference external" href="https://datatracker.ietf.org/doc/html/rfc2047.html">RFC 2047</a> MIME encoding. On Python platforms which do not have a native bytes-like type (e.g. IronPython, etc.), but instead which generally use textlike strings to represent bytes data, the definition of “bytes instance” can be changed: their “bytes instances” must be native strings that contain only code points representable in ISO-8859-1 encoding (<code class="docutils literal notranslate">\u0000</code> through <code class="docutils literal notranslate">\u00FF</code>, inclusive). It is a fatal error for an application on such a platform to supply strings containing any other Unicode character or code point. Similarly, servers and gateways on those platforms must not supply strings to an application containing any other Unicode characters. </section> <section id="http-1-1-expect-continue"> <h3><a class="toc-backref" href="#http-1-1-expect-continue" role="doc-backlink">HTTP 1.1 Expect/Continue</a></h3> Servers and gateways that implement HTTP 1.1 must provide transparent support for HTTP 1.1’s “expect/continue” mechanism. This may be done in any of several ways: <ol class="arabic simple"> <li>Respond to requests containing an <code class="docutils literal notranslate">Expect: 100-continue</code> request with an immediate “100 Continue” response, and proceed normally.</li> <li>Proceed with the request normally, but provide the application with a <code class="docutils literal notranslate">web3.input</code> stream that will send the “100 Continue” response if/when the application first attempts to read from the input stream. The read request must then remain blocked until the client responds.</li> <li>Wait until the client decides that the server does not support expect/continue, and sends the request body on its own. (This is suboptimal, and is not recommended.)</li> </ol> Note that these behavior restrictions do not apply for HTTP 1.0 requests, or for requests that are not directed to an application object. For more information on HTTP 1.1 Expect/Continue, see <a class="rfc reference external" href="https://datatracker.ietf.org/doc/html/rfc2616.html">RFC 2616</a>, sections 8.2.3 and 10.1.1. </section> <section id="other-http-features"> <h3><a class="toc-backref" href="#other-http-features" role="doc-backlink">Other HTTP Features</a></h3> In general, servers and gateways should “play dumb” and allow the application complete control over its output. They should only make changes that do not alter the effective semantics of the application’s response. It is always possible for the application developer to add middleware components to supply additional features, so server/gateway developers should be conservative in their implementation. In a sense, a server should consider itself to be like an HTTP “gateway server”, with the application being an HTTP “origin server”. (See <a class="rfc reference external" href="https://datatracker.ietf.org/doc/html/rfc2616.html">RFC 2616</a>, section 1.3, for the definition of these terms.) However, because Web3 servers and applications do not communicate via HTTP, what <a class="rfc reference external" href="https://datatracker.ietf.org/doc/html/rfc2616.html">RFC 2616</a> calls “hop-by-hop” headers do not apply to Web3 internal communications. Web3 applications must not generate any <a class="rfc reference external" href="https://datatracker.ietf.org/doc/html/rfc2616.html#section-13.5.1">“hop-by-hop” headers</a>, attempt to use HTTP features that would require them to generate such headers, or rely on the content of any incoming “hop-by-hop” headers in the <code class="docutils literal notranslate">environ</code> dictionary. Web3 servers must handle any supported inbound “hop-by-hop” headers on their own, such as by decoding any inbound <code class="docutils literal notranslate">Transfer-Encoding</code>, including chunked encoding if applicable. Applying these principles to a variety of HTTP features, it should be clear that a server may handle cache validation via the <code class="docutils literal notranslate">If-None-Match</code> and <code class="docutils literal notranslate">If-Modified-Since</code> request headers and the <code class="docutils literal notranslate">Last-Modified</code> and <code class="docutils literal notranslate">ETag</code> response headers. However, it is not required to do this, and the application should perform its own cache validation if it wants to support that feature, since the server/gateway is not required to do such validation. Similarly, a server may re-encode or transport-encode an application’s response, but the application should use a suitable content encoding on its own, and must not apply a transport encoding. A server may transmit byte ranges of the application’s response if requested by the client, and the application doesn’t natively support byte ranges. Again, however, the application should perform this function on its own if desired. Note that these restrictions on applications do not necessarily mean that every application must reimplement every HTTP feature; many HTTP features can be partially or fully implemented by middleware components, thus freeing both server and application authors from implementing the same features over and over again. </section> <section id="thread-support"> <h3><a class="toc-backref" href="#thread-support" role="doc-backlink">Thread Support</a></h3> Thread support, or lack thereof, is also server-dependent. Servers that can run multiple requests in parallel, should also provide the option of running an application in a single-threaded fashion, so that applications or frameworks that are not thread-safe may still be used with that server. </section> </section> <section id="implementation-application-notes"> <h2><a class="toc-backref" href="#implementation-application-notes" role="doc-backlink">Implementation/Application Notes</a></h2> <section id="server-extension-apis"> <h3><a class="toc-backref" href="#server-extension-apis" role="doc-backlink">Server Extension APIs</a></h3> Some server authors may wish to expose more advanced APIs, that application or framework authors can use for specialized purposes. For example, a gateway based on <code class="docutils literal notranslate">mod_python</code> might wish to expose part of the Apache API as a Web3 extension. In the simplest case, this requires nothing more than defining an <code class="docutils literal notranslate">environ</code> variable, such as <code class="docutils literal notranslate">mod_python.some_api</code>. But, in many cases, the possible presence of middleware can make this difficult. For example, an API that offers access to the same HTTP headers that are found in <code class="docutils literal notranslate">environ</code> variables, might return different data if <code class="docutils literal notranslate">environ</code> has been modified by middleware. In general, any extension API that duplicates, supplants, or bypasses some portion of Web3 functionality runs the risk of being incompatible with middleware components. Server/gateway developers should not assume that nobody will use middleware, because some framework developers specifically organize their frameworks to function almost entirely as middleware of various kinds. So, to provide maximum compatibility, servers and gateways that provide extension APIs that replace some Web3 functionality, must design those APIs so that they are invoked using the portion of the API that they replace. For example, an extension API to access HTTP request headers must require the application to pass in its current <code class="docutils literal notranslate">environ</code>, so that the server/gateway may verify that HTTP headers accessible via the API have not been altered by middleware. If the extension API cannot guarantee that it will always agree with <code class="docutils literal notranslate">environ</code> about the contents of HTTP headers, it must refuse service to the application, e.g. by raising an error, returning <code class="docutils literal notranslate">None</code> instead of a header collection, or whatever is appropriate to the API. These guidelines also apply to middleware that adds information such as parsed cookies, form variables, sessions, and the like to <code class="docutils literal notranslate">environ</code>. Specifically, such middleware should provide these features as functions which operate on <code class="docutils literal notranslate">environ</code>, rather than simply stuffing values into <code class="docutils literal notranslate">environ</code>. This helps ensure that information is calculated from <code class="docutils literal notranslate">environ</code> after any middleware has done any URL rewrites or other <code class="docutils literal notranslate">environ</code> modifications. It is very important that these “safe extension” rules be followed by both server/gateway and middleware developers, in order to avoid a future in which middleware developers are forced to delete any and all extension APIs from <code class="docutils literal notranslate">environ</code> to ensure that their mediation isn’t being bypassed by applications using those extensions! </section> <section id="application-configuration"> <h3><a class="toc-backref" href="#application-configuration" role="doc-backlink">Application Configuration</a></h3> This specification does not define how a server selects or obtains an application to invoke. These and other configuration options are highly server-specific matters. It is expected that server/gateway authors will document how to configure the server to execute a particular application object, and with what options (such as threading options). Framework authors, on the other hand, should document how to create an application object that wraps their framework’s functionality. The user, who has chosen both the server and the application framework, must connect the two together. However, since both the framework and the server have a common interface, this should be merely a mechanical matter, rather than a significant engineering effort for each new server/framework pair. Finally, some applications, frameworks, and middleware may wish to use the <code class="docutils literal notranslate">environ</code> dictionary to receive simple string configuration options. Servers and gateways should support this by allowing an application’s deployer to specify name-value pairs to be placed in <code class="docutils literal notranslate">environ</code>. In the simplest case, this support can consist merely of copying all operating system-supplied environment variables from <code class="docutils literal notranslate">os.environ</code> into the <code class="docutils literal notranslate">environ</code> dictionary, since the deployer in principle can configure these externally to the server, or in the CGI case they may be able to be set via the server’s configuration files. Applications should try to keep such required variables to a minimum, since not all servers will support easy configuration of them. Of course, even in the worst case, persons deploying an application can create a script to supply the necessary configuration values: <div class="highlight-default notranslate"><div class="highlight"><pre>from the_app import application def new_app(environ): environ['the_app.configval1'] = b'something' return application(environ) </pre></div> </div> But, most existing applications and frameworks will probably only need a single configuration value from <code class="docutils literal notranslate">environ</code>, to indicate the location of their application or framework-specific configuration file(s). (Of course, applications should cache such configuration, to avoid having to re-read it upon each invocation.) </section> <section id="url-reconstruction"> <h3><a class="toc-backref" href="#url-reconstruction" role="doc-backlink">URL Reconstruction</a></h3> If an application wishes to reconstruct a request’s complete URL (as a bytes object), it may do so using the following algorithm: <div class="highlight-default notranslate"><div class="highlight"><pre>host = environ.get('HTTP_HOST') scheme = environ['web3.url_scheme'] port = environ['SERVER_PORT'] query = environ['QUERY_STRING'] url = scheme + b'://' if host: url += host else: url += environ['SERVER_NAME'] if scheme == b'https': if port != b'443': url += b':' + port else: if port != b'80': url += b':' + port if 'web3.script_name' in url: url += url_quote(environ['web3.script_name']) else: url += environ['SCRIPT_NAME'] if 'web3.path_info' in environ: url += url_quote(environ['web3.path_info']) else: url += environ['PATH_INFO'] if query: url += b'?' + query </pre></div> </div> Note that such a reconstructed URL may not be precisely the same URI as requested by the client. Server rewrite rules, for example, may have modified the client’s originally requested URL to place it in a canonical form. </section> </section> <section id="open-questions"> <h2><a class="toc-backref" href="#open-questions" role="doc-backlink">Open Questions</a></h2> <ul class="simple"> <li><code class="docutils literal notranslate">file_wrapper</code> replacement. Currently nothing is specified here but it’s clear that the old system of in-band signalling is broken if it does not provide a way to figure out as a middleware in the process if the response is a file wrapper.</li> </ul> </section> <section id="points-of-contention"> <h2><a class="toc-backref" href="#points-of-contention" role="doc-backlink">Points of Contention</a></h2> Outlined below are potential points of contention regarding this specification. <section id="wsgi-1-0-compatibility"> <h3><a class="toc-backref" href="#wsgi-1-0-compatibility" role="doc-backlink">WSGI 1.0 Compatibility</a></h3> Components written using the WSGI 1.0 specification will not transparently interoperate with components written using this specification. That’s because the goals of this proposal and the goals of WSGI 1.0 are not directly aligned. WSGI 1.0 is obliged to provide specification-level backwards compatibility with versions of Python between 2.2 and 2.7. This specification, however, ditches Python 2.5 and lower compatibility in order to provide compatibility between relatively recent versions of Python 2 (2.6 and 2.7) as well as relatively recent versions of Python 3 (3.1). It is currently impossible to write components which work reliably under both Python 2 and Python 3 using the WSGI 1.0 specification, because the specification implicitly posits that CGI and server variable values in the environ and values returned via <code class="docutils literal notranslate">start_response</code> represent a sequence of bytes that can be addressed using the Python 2 string API. It posits such a thing because that sort of data type was the sensible way to represent bytes in all Python 2 versions, and WSGI 1.0 was conceived before Python 3 existed. Python 3’s <code class="docutils literal notranslate">str</code> type supports the full API provided by the Python 2 <code class="docutils literal notranslate">str</code> type, but Python 3’s <code class="docutils literal notranslate">str</code> type does not represent a sequence of bytes, it instead represents text. Therefore, using it to represent environ values also requires that the environ byte sequence be decoded to text via some encoding. We cannot decode these bytes to text (at least in any way where the decoding has any meaning other than as a tunnelling mechanism) without widening the scope of WSGI to include server and gateway knowledge of decoding policies and mechanics. WSGI 1.0 never concerned itself with encoding and decoding. It made statements about allowable transport values, and suggested that various values might be best decoded as one encoding or another, but it never required a server to perform any decoding before Python 3 does not have a stringlike type that can be used instead to represent bytes: it has a <code class="docutils literal notranslate">bytes</code> type. A bytes type operates quite a bit like a Python 2 <code class="docutils literal notranslate">str</code> in Python 3.1+, but it lacks behavior equivalent to <code class="docutils literal notranslate">str.__mod__</code> and its iteration protocol, and containment, sequence treatment, and equivalence comparisons are different. In either case, there is no type in Python 3 that behaves just like the Python 2 <code class="docutils literal notranslate">str</code> type, and a way to create such a type doesn’t exist because there is no such thing as a “String ABC” which would allow a suitable type to be built. Due to this design incompatibility, existing WSGI 1.0 servers, middleware, and applications will not work under Python 3, even after they are run through <code class="docutils literal notranslate">2to3</code>. Existing Web-SIG discussions about updating the WSGI specification so that it is possible to write a WSGI application that runs in both Python 2 and Python 3 tend to revolve around creating a specification-level equivalence between the Python 2 <code class="docutils literal notranslate">str</code> type (which represents a sequence of bytes) and the Python 3 <code class="docutils literal notranslate">str</code> type (which represents text). Such an equivalence becomes strained in various areas, given the different roles of these types. An arguably more straightforward equivalence exists between the Python 3 <code class="docutils literal notranslate">bytes</code> type API and a subset of the Python 2 <code class="docutils literal notranslate">str</code> type API. This specification exploits this subset equivalence. In the meantime, aside from any Python 2 vs. Python 3 compatibility issue, as various discussions on Web-SIG have pointed out, the WSGI 1.0 specification is too general, providing support (via <code class="docutils literal notranslate">.write</code>) for asynchronous applications at the expense of implementation complexity. This specification uses the fundamental incompatibility between WSGI 1.0 and Python 3 as a natural divergence point to create a specification with reduced complexity by changing specialized support for asynchronous applications. To provide backwards compatibility for older WSGI 1.0 applications, so that they may run on a Web3 stack, it is presumed that Web3 middleware will be created which can be used “in front” of existing WSGI 1.0 applications, allowing those existing WSGI 1.0 applications to run under a Web3 stack. This middleware will require, when under Python 3, an equivalence to be drawn between Python 3 <code class="docutils literal notranslate">str</code> types and the bytes values represented by the HTTP request and all the attendant encoding-guessing (or configuration) it implies. <div class="admonition note"> Note Such middleware might in the future, instead of drawing an equivalence between Python 3 <code class="docutils literal notranslate">str</code> and HTTP byte values, make use of a yet-to-be-created “ebytes” type (aka “bytes-with-benefits”), particularly if a String ABC proposal is accepted into the Python core and implemented. </div> Conversely, it is presumed that WSGI 1.0 middleware will be created which will allow a Web3 application to run behind a WSGI 1.0 stack on the Python 2 platform. </section> <section id="environ-and-response-values-as-bytes"> <h3><a class="toc-backref" href="#environ-and-response-values-as-bytes" role="doc-backlink">Environ and Response Values as Bytes</a></h3> Casual middleware and application writers may consider the use of bytes as environment values and response values inconvenient. In particular, they won’t be able to use common string formatting functions such as <code class="docutils literal notranslate">('%s' % bytes_val)</code> or <code class="docutils literal notranslate">bytes_val.format('123')</code> because bytes don’t have the same API as strings on platforms such as Python 3 where the two types differ. Likewise, on such platforms, stdlib HTTP-related API support for using bytes interchangeably with text can be spotty. In places where bytes are inconvenient or incompatible with library APIs, middleware and application writers will have to decode such bytes to text explicitly. This is particularly inconvenient for middleware writers: to work with environment values as strings, they’ll have to decode them from an implied encoding and if they need to mutate an environ value, they’ll then need to encode the value into a byte stream before placing it into the environ. While the use of bytes by the specification as environ values might be inconvenient for casual developers, it provides several benefits. Using bytes types to represent HTTP and server values to an application most closely matches reality because HTTP is fundamentally a bytes-oriented protocol. If the environ values are mandated to be strings, each server will need to use heuristics to guess about the encoding of various values provided by the HTTP environment. Using all strings might increase casual middleware writer convenience, but will also lead to ambiguity and confusion when a value cannot be decoded to a meaningful non-surrogate string. Use of bytes as environ values avoids any potential for the need for the specification to mandate that a participating server be informed of encoding configuration parameters. If environ values are treated as strings, and so must be decoded from bytes, configuration parameters may eventually become necessary as policy clues from the application deployer. Such a policy would be used to guess an appropriate decoding strategy in various circumstances, effectively placing the burden for enforcing a particular application encoding policy upon the server. If the server must serve more than one application, such configuration would quickly become complex. Many policies would also be impossible to express declaratively. In reality, HTTP is a complicated and legacy-fraught protocol which requires a complex set of heuristics to make sense of. It would be nice if we could allow this protocol to protect us from this complexity, but we cannot do so reliably while still providing to application writers a level of control commensurate with reality. Python applications must often deal with data embedded in the environment which not only must be parsed by legacy heuristics, but does not conform even to any existing HTTP specification. While these eventualities are unpleasant, they crop up with regularity, making it impossible and undesirable to hide them from application developers, as application developers are the only people who are able to decide upon an appropriate action when an HTTP specification violation is detected. Some have argued for mixed use of bytes and string values as environ values. This proposal avoids that strategy. Sole use of bytes as environ values makes it possible to fit this specification entirely in one’s head; you won’t need to guess about which values are strings and which are bytes. This protocol would also fit in a developer’s head if all environ values were strings, but this specification doesn’t use that strategy. This will likely be the point of greatest contention regarding the use of bytes. In defense of bytes: developers often prefer protocols with consistent contracts, even if the contracts themselves are suboptimal. If we hide encoding issues from a developer until a value that contains surrogates causes problems after it has already reached beyond the I/O boundary of their application, they will need to do a lot more work to fix assumptions made by their application than if we were to just present the problem much earlier in terms of “here’s some bytes, you decode them”. This is also a counter-argument to the “bytes are inconvenient” assumption: while presenting bytes to an application developer may be inconvenient for a casual application developer who doesn’t care about edge cases, they are extremely convenient for the application developer who needs to deal with complex, dirty eventualities, because use of bytes allows him the appropriate level of control with a clear separation of responsibility. If the protocol uses bytes, it is presumed that libraries will be created to make working with bytes-only in the environ and within return values more pleasant; for example, analogues of the WSGI 1.0 libraries named “WebOb” and “Werkzeug”. Such libraries will fill the gap between convenience and control, allowing the spec to remain simple and regular while still allowing casual authors a convenient way to create Web3 middleware and application components. This seems to be a reasonable alternative to baking encoding policy into the protocol, because many such libraries can be created independently from the protocol, and application developers can choose the one that provides them the appropriate levels of control and convenience for a particular job. Here are some alternatives to using all bytes: <ul class="simple"> <li>Have the server decode all values representing CGI and server environ values into strings using the <code class="docutils literal notranslate">latin-1</code> encoding, which is lossless. Smuggle any undecodable bytes within the resulting string.</li> <li>Encode all CGI and server environ values to strings using the <code class="docutils literal notranslate">utf-8</code> encoding with the <code class="docutils literal notranslate">surrogateescape</code> error handler. This does not work under any existing Python 2.</li> <li>Encode some values into bytes and other values into strings, as decided by their typical usages.</li> </ul> </section> <section id="applications-should-be-allowed-to-read-web3-input-past-content-length"> <h3><a class="toc-backref" href="#applications-should-be-allowed-to-read-web3-input-past-content-length" role="doc-backlink">Applications Should be Allowed to Read <code class="docutils literal notranslate">web3.input</code> Past <code class="docutils literal notranslate">CONTENT_LENGTH</code></a></h3> At <a class="footnote-reference brackets" href="#id12" id="id3">[5]</a>, Graham Dumpleton makes the assertion that <code class="docutils literal notranslate">wsgi.input</code> should be required to return the empty string as a signifier of out-of-data, and that applications should be allowed to read past the number of bytes specified in <code class="docutils literal notranslate">CONTENT_LENGTH</code>, depending only upon the empty string as an EOF marker. WSGI relies on an application “being well behaved and once all data specified by <code class="docutils literal notranslate">CONTENT_LENGTH</code> is read, that it processes the data and returns any response. That same socket connection could then be used for a subsequent request.” Graham would like WSGI adapters to be required to wrap raw socket connections: “this wrapper object will need to count how much data has been read, and when the amount of data reaches that as defined by <code class="docutils literal notranslate">CONTENT_LENGTH</code>, any subsequent reads should return an empty string instead.” This may be useful to support chunked encoding and input filters. </section> <section id="web3-input-unknown-length"> <h3><a class="toc-backref" href="#web3-input-unknown-length" role="doc-backlink"><code class="docutils literal notranslate">web3.input</code> Unknown Length</a></h3> There’s no documented way to indicate that there is content in <code class="docutils literal notranslate">environ['web3.input']</code>, but the content length is unknown. </section> <section id="read-of-web3-input-should-support-no-size-calling-convention"> <h3><a class="toc-backref" href="#read-of-web3-input-should-support-no-size-calling-convention" role="doc-backlink"><code class="docutils literal notranslate">read()</code> of <code class="docutils literal notranslate">web3.input</code> Should Support No-Size Calling Convention</a></h3> At <a class="footnote-reference brackets" href="#id12" id="id4">[5]</a>, Graham Dumpleton makes the assertion that the <code class="docutils literal notranslate">read()</code> method of <code class="docutils literal notranslate">wsgi.input</code> should be callable without arguments, and that the result should be “all available request content”. Needs discussion. Comment Armin: I changed the spec to require that from an implementation. I had too much pain with that in the past already. Open for discussions though. <section id="input-filters-should-set-environ-content-length-to-1"> <h4><a class="toc-backref" href="#input-filters-should-set-environ-content-length-to-1" role="doc-backlink">Input Filters should set environ <code class="docutils literal notranslate">CONTENT_LENGTH</code> to -1</a></h4> At <a class="footnote-reference brackets" href="#id12" id="id5">[5]</a>, Graham Dumpleton suggests that an input filter might set <code class="docutils literal notranslate">environ['CONTENT_LENGTH']</code> to -1 to indicate that it mutated the input. </section> </section> <section id="headers-as-literal-list-of-two-tuples"> <h3><a class="toc-backref" href="#headers-as-literal-list-of-two-tuples" role="doc-backlink"><code class="docutils literal notranslate">headers</code> as Literal List of Two-Tuples</a></h3> Why do we make applications return a <code class="docutils literal notranslate">headers</code> structure that is a literal list of two-tuples? I think the iterability of <code class="docutils literal notranslate">headers</code> needs to be maintained while it moves up the stack, but I don’t think we need to be able to mutate it in place at all times. Could we loosen that requirement? Comment Armin: Strong yes </section> <section id="removed-requirement-that-middleware-not-block"> <h3><a class="toc-backref" href="#removed-requirement-that-middleware-not-block" role="doc-backlink">Removed Requirement that Middleware Not Block</a></h3> This requirement was removed: “middleware components must not block iteration waiting for multiple values from an application iterable. If the middleware needs to accumulate more data from the application before it can produce any output, it must yield an empty string.” This requirement existed to support asynchronous applications and servers (see <a class="pep reference internal" href="../pep-0333/" title="PEP 333 – Python Web Server Gateway Interface v1.0">PEP 333</a>’s “Middleware Handling of Block Boundaries”). Asynchronous applications are now serviced explicitly by <code class="docutils literal notranslate">web3.async</code> capable protocol (a Web3 application callable may itself return a callable). </section> <section id="web3-script-name-and-web3-path-info"> <h3><a class="toc-backref" href="#web3-script-name-and-web3-path-info" role="doc-backlink"><code class="docutils literal notranslate">web3.script_name</code> and <code class="docutils literal notranslate">web3.path_info</code></a></h3> These values are required to be placed into the environment by an origin server under this specification. Unlike <code class="docutils literal notranslate">SCRIPT_NAME</code> and <code class="docutils literal notranslate">PATH_INFO</code>, these must be the original URL-encoded variants derived from the request URI. We probably need to figure out how these should be computed originally, and what their values should be if the server performs URL rewriting. </section> <section id="long-response-headers"> <h3><a class="toc-backref" href="#long-response-headers" role="doc-backlink">Long Response Headers</a></h3> Bob Brewer notes on Web-SIG <a class="footnote-reference brackets" href="#id13" id="id6">[6]</a>: <blockquote> <div>Each header_value must not include any control characters, including carriage returns or linefeeds, either embedded or at the end. (These requirements are to minimize the complexity of any parsing that must be performed by servers, gateways, and intermediate response processors that need to inspect or modify response headers.) (<a class="pep reference internal" href="../pep-0333/" title="PEP 333 – Python Web Server Gateway Interface v1.0">PEP 333</a>)</div></blockquote> That’s understandable, but HTTP headers are defined as (mostly) *TEXT, and “words of *TEXT MAY contain characters from character sets other than ISO-8859-1 only when encoded according to the rules of <a class="rfc reference external" href="https://datatracker.ietf.org/doc/html/rfc2047.html">RFC 2047</a>.” <a class="footnote-reference brackets" href="#id9" id="id7">[2]</a> And <a class="rfc reference external" href="https://datatracker.ietf.org/doc/html/rfc2047.html">RFC 2047</a> specifies that “an ‘encoded-word’ may not be more than 75 characters long… If it is desirable to encode more text than will fit in an ‘encoded-word’ of 75 characters, multiple ‘encoded-word’s (separated by CRLF SPACE) may be used.” <a class="footnote-reference brackets" href="#id10" id="id8">[3]</a> This satisfies HTTP header folding rules, as well: “Header fields can be extended over multiple lines by preceding each extra line with at least one SP or HT.” (<a class="pep reference internal" href="../pep-0333/" title="PEP 333 – Python Web Server Gateway Interface v1.0">PEP 333</a>) So in my reading of HTTP, some code somewhere should introduce newlines in longish, encoded response header values. I see three options: <ol class="arabic simple"> <li>Keep things as they are and disallow response header values if they contain words over 75 chars that are outside the ISO-8859-1 character set.</li> <li>Allow newline characters in WSGI response headers.</li> <li>Require/strongly suggest WSGI servers to do the encoding and folding before sending the value over HTTP.</li> </ol> </section> <section id="request-trailers-and-chunked-transfer-encoding"> <h3><a class="toc-backref" href="#request-trailers-and-chunked-transfer-encoding" role="doc-backlink">Request Trailers and Chunked Transfer Encoding</a></h3> When using chunked transfer encoding on request content, the RFCs allow there to be request trailers. These are like request headers but come after the final null data chunk. These trailers are only available when the chunked data stream is finite length and when it has all been read in. Neither WSGI nor Web3 currently supports them. </section> </section> <section id="references"> <h2><a class="toc-backref" href="#references" role="doc-backlink">References</a></h2> <aside class="footnote-list brackets"> <aside class="footnote brackets" id="id9" role="doc-footnote"> <dt class="label" id="id9">[2] (<a href='#id1'>1</a>, <a href='#id7'>2</a>) </dt> <dd>The Common Gateway Interface Specification, v 1.1, 3rd Draft (<a class="reference external" href="https://datatracker.ietf.org/doc/html/draft-coar-cgi-v11-03">https://datatracker.ietf.org/doc/html/draft-coar-cgi-v11-03</a>)</aside> <aside class="footnote brackets" id="id10" role="doc-footnote"> <dt class="label" id="id10">[<a href="#id8">3</a>]</dt> <dd>“Chunked Transfer Coding” – HTTP/1.1, <a class="rfc reference external" href="https://datatracker.ietf.org/doc/html/rfc2616.html#section-3.6.1">RFC 2616#section-3.6.1</a></aside> <aside class="footnote brackets" id="id11" role="doc-footnote"> <dt class="label" id="id11">[<a href="#id2">4</a>]</dt> <dd>mod_ssl Reference, “Environment Variables” (<a class="reference external" href="http://www.modssl.org/docs/2.8/ssl_reference.html#ToC25">http://www.modssl.org/docs/2.8/ssl_reference.html#ToC25</a>)</aside> <aside class="footnote brackets" id="id12" role="doc-footnote"> <dt class="label" id="id12">[5] (<a href='#id3'>1</a>, <a href='#id4'>2</a>, <a href='#id5'>3</a>) </dt> <dd>Details on WSGI 1.0 amendments/clarifications. (<a class="reference external" href="http://blog.dscpl.com.au/2009/10/details-on-wsgi-10-amendmentsclarificat.html">http://blog.dscpl.com.au/2009/10/details-on-wsgi-10-amendmentsclarificat.html</a>)</aside> <aside class="footnote brackets" id="id13" role="doc-footnote"> <dt class="label" id="id13">[<a href="#id6">6</a>]</dt> <dd>[Web-SIG] WSGI and long response header values <a class="reference external" href="https://mail.python.org/pipermail/web-sig/2006-September/002244.html">https://mail.python.org/pipermail/web-sig/2006-September/002244.html</a></aside> </aside> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0444.rst">https://github.com/python/peps/blob/main/peps/pep-0444.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0444.rst">2025-02-01 08:59:27 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#pep-deferral">PEP Deferral</a></li> <li><a class="reference internal" href="#rationale-and-goals">Rationale and Goals</a></li> <li><a class="reference internal" href="#differences-from-wsgi">Differences from WSGI</a></li> <li><a class="reference internal" href="#specification-overview">Specification Overview</a><ul> <li><a class="reference internal" href="#the-application-framework-side">The Application/Framework Side</a></li> <li><a class="reference internal" href="#the-server-gateway-side">The Server/Gateway Side</a></li> <li><a class="reference internal" href="#middleware-components-that-play-both-sides">Middleware: Components that Play Both Sides</a></li> </ul> </li> <li><a class="reference internal" href="#specification-details">Specification Details</a><ul> <li><a class="reference internal" href="#environ-variables"><code class="docutils literal notranslate">environ</code> Variables</a><ul> <li><a class="reference internal" href="#input-stream">Input Stream</a></li> <li><a class="reference internal" href="#error-stream">Error Stream</a></li> </ul> </li> <li><a class="reference internal" href="#values-returned-by-a-web3-application">Values Returned by A Web3 Application</a></li> <li><a class="reference internal" href="#dealing-with-compatibility-across-python-versions">Dealing with Compatibility Across Python Versions</a></li> <li><a class="reference internal" href="#buffering-and-streaming">Buffering and Streaming</a></li> <li><a class="reference internal" href="#unicode-issues">Unicode Issues</a></li> <li><a class="reference internal" href="#http-1-1-expect-continue">HTTP 1.1 Expect/Continue</a></li> <li><a class="reference internal" href="#other-http-features">Other HTTP Features</a></li> <li><a class="reference internal" href="#thread-support">Thread Support</a></li> </ul> </li> <li><a class="reference internal" href="#implementation-application-notes">Implementation/Application Notes</a><ul> <li><a class="reference internal" href="#server-extension-apis">Server Extension APIs</a></li> <li><a class="reference internal" href="#application-configuration">Application Configuration</a></li> <li><a class="reference internal" href="#url-reconstruction">URL Reconstruction</a></li> </ul> </li> <li><a class="reference internal" href="#open-questions">Open Questions</a></li> <li><a class="reference internal" href="#points-of-contention">Points of Contention</a><ul> <li><a class="reference internal" href="#wsgi-1-0-compatibility">WSGI 1.0 Compatibility</a></li> <li><a class="reference internal" href="#environ-and-response-values-as-bytes">Environ and Response Values as Bytes</a></li> <li><a class="reference internal" href="#applications-should-be-allowed-to-read-web3-input-past-content-length">Applications Should be Allowed to Read <code class="docutils literal notranslate">web3.input</code> Past <code class="docutils literal notranslate">CONTENT_LENGTH</code></a></li> <li><a class="reference internal" href="#web3-input-unknown-length"><code class="docutils literal notranslate">web3.input</code> Unknown Length</a></li> <li><a class="reference internal" href="#read-of-web3-input-should-support-no-size-calling-convention"><code class="docutils literal notranslate">read()</code> of <code class="docutils literal notranslate">web3.input</code> Should Support No-Size Calling Convention</a><ul> <li><a class="reference internal" href="#input-filters-should-set-environ-content-length-to-1">Input Filters should set environ <code class="docutils literal notranslate">CONTENT_LENGTH</code> to -1</a></li> </ul> </li> <li><a class="reference internal" href="#headers-as-literal-list-of-two-tuples"><code class="docutils literal notranslate">headers</code> as Literal List of Two-Tuples</a></li> <li><a class="reference internal" href="#removed-requirement-that-middleware-not-block">Removed Requirement that Middleware Not Block</a></li> <li><a class="reference internal" href="#web3-script-name-and-web3-path-info"><code class="docutils literal notranslate">web3.script_name</code> and <code class="docutils literal notranslate">web3.path_info</code></a></li> <li><a class="reference internal" href="#long-response-headers">Long Response Headers</a></li> <li><a class="reference internal" href="#request-trailers-and-chunked-transfer-encoding">Request Trailers and Chunked Transfer Encoding</a></li> </ul> </li> <li><a class="reference internal" href="#references">References</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0444.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>