CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 476 – Enabling certificate verification by default for stdlib http clients | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0476/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 476 – Enabling certificate verification by default for stdlib http clients | peps.python.org'> <meta property="og:description" content="Currently when a standard library http client (the urllib, urllib2, http, and httplib modules) encounters an https:// URL it will wrap the network HTTP traffic in a TLS stream, as is necessary to communicate with such a server. However, during the TLS h..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0476/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="Currently when a standard library http client (the urllib, urllib2, http, and httplib modules) encounters an https:// URL it will wrap the network HTTP traffic in a TLS stream, as is necessary to communicate with such a server. However, during the TLS h..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 476</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> <span class="visually-hidden">Toggle light / dark / auto colour theme</span> </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 476 – Enabling certificate verification by default for stdlib http clients</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author<span class="colon">:</span></dt> <dd class="field-odd">Alex Gaynor <alex.gaynor at gmail.com></dd> <dt class="field-even">Status<span class="colon">:</span></dt> <dd class="field-even"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-odd">Type<span class="colon">:</span></dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created<span class="colon">:</span></dt> <dd class="field-even">28-Aug-2014</dd> <dt class="field-odd">Python-Version<span class="colon">:</span></dt> <dd class="field-odd">2.7.9, 3.4.3, 3.5</dd> <dt class="field-even">Resolution<span class="colon">:</span></dt> <dd class="field-even"><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2014-October/136676.html">Python-Dev message</a></dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#technical-details">Technical Details</a><ul> <li><a class="reference internal" href="#trust-database">Trust database</a></li> <li><a class="reference internal" href="#backwards-compatibility">Backwards compatibility</a></li> <li><a class="reference internal" href="#opting-out">Opting out</a></li> </ul> </li> <li><a class="reference internal" href="#other-protocols">Other protocols</a></li> <li><a class="reference internal" href="#python-versions">Python Versions</a></li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> <p>Currently when a standard library http client (the <code class="docutils literal notranslate"><span class="pre">urllib</span></code>, <code class="docutils literal notranslate"><span class="pre">urllib2</span></code>, <code class="docutils literal notranslate"><span class="pre">http</span></code>, and <code class="docutils literal notranslate"><span class="pre">httplib</span></code> modules) encounters an <code class="docutils literal notranslate"><span class="pre">https://</span></code> URL it will wrap the network HTTP traffic in a TLS stream, as is necessary to communicate with such a server. However, during the TLS handshake it will not actually check that the server has an X509 certificate is signed by a CA in any trust root, nor will it verify that the Common Name (or Subject Alternate Name) on the presented certificate matches the requested host.</p> <p>The failure to do these checks means that anyone with a privileged network position is able to trivially execute a man in the middle attack against a Python application using either of these HTTP clients, and change traffic at will.</p> <p>This PEP proposes to enable verification of X509 certificate signatures, as well as hostname verification for Python’s HTTP clients by default, subject to opt-out on a per-call basis. This change would be applied to Python 2.7, Python 3.4, and Python 3.5.</p> </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> <p>The “S” in “HTTPS” stands for secure. When Python’s users type “HTTPS” they are expecting a secure connection, and Python should adhere to a reasonable standard of care in delivering this. Currently we are failing at this, and in doing so, APIs which appear simple are misleading users.</p> <p>When asked, many Python users state that they were not aware that Python failed to perform these validations, and are shocked.</p> <p>The popularity of <code class="docutils literal notranslate"><span class="pre">requests</span></code> (which enables these checks by default) demonstrates that these checks are not overly burdensome in any way, and the fact that it is widely recommended as a major security improvement over the standard library clients demonstrates that many expect a higher standard for “security by default” from their tools.</p> <p>The failure of various applications to note Python’s negligence in this matter is a source of <em>regular</em> CVE assignment <a class="footnote-reference brackets" href="#id12" id="id1">[1]</a> <a class="footnote-reference brackets" href="#id13" id="id2">[2]</a> <a class="footnote-reference brackets" href="#id14" id="id3">[3]</a> <a class="footnote-reference brackets" href="#id15" id="id4">[4]</a> <a class="footnote-reference brackets" href="#id16" id="id5">[5]</a> <a class="footnote-reference brackets" href="#id17" id="id6">[6]</a> <a class="footnote-reference brackets" href="#id18" id="id7">[7]</a> <a class="footnote-reference brackets" href="#id19" id="id8">[8]</a> <a class="footnote-reference brackets" href="#id20" id="id9">[9]</a> <a class="footnote-reference brackets" href="#id21" id="id10">[10]</a> <a class="footnote-reference brackets" href="#id22" id="id11">[11]</a>.</p> <aside class="footnote-list brackets"> <aside class="footnote brackets" id="id12" role="doc-footnote"> <dt class="label" id="id12">[<a href="#id1">1</a>]</dt> <dd><a class="reference external" href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4340">https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4340</a></aside> <aside class="footnote brackets" id="id13" role="doc-footnote"> <dt class="label" id="id13">[<a href="#id2">2</a>]</dt> <dd><a class="reference external" href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-3533">https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-3533</a></aside> <aside class="footnote brackets" id="id14" role="doc-footnote"> <dt class="label" id="id14">[<a href="#id3">3</a>]</dt> <dd><a class="reference external" href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-5822">https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-5822</a></aside> <aside class="footnote brackets" id="id15" role="doc-footnote"> <dt class="label" id="id15">[<a href="#id4">4</a>]</dt> <dd><a class="reference external" href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-5825">https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-5825</a></aside> <aside class="footnote brackets" id="id16" role="doc-footnote"> <dt class="label" id="id16">[<a href="#id5">5</a>]</dt> <dd><a class="reference external" href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-1909">https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-1909</a></aside> <aside class="footnote brackets" id="id17" role="doc-footnote"> <dt class="label" id="id17">[<a href="#id6">6</a>]</dt> <dd><a class="reference external" href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2037">https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2037</a></aside> <aside class="footnote brackets" id="id18" role="doc-footnote"> <dt class="label" id="id18">[<a href="#id7">7</a>]</dt> <dd><a class="reference external" href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2073">https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2073</a></aside> <aside class="footnote brackets" id="id19" role="doc-footnote"> <dt class="label" id="id19">[<a href="#id8">8</a>]</dt> <dd><a class="reference external" href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2191">https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2191</a></aside> <aside class="footnote brackets" id="id20" role="doc-footnote"> <dt class="label" id="id20">[<a href="#id9">9</a>]</dt> <dd><a class="reference external" href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-4111">https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-4111</a></aside> <aside class="footnote brackets" id="id21" role="doc-footnote"> <dt class="label" id="id21">[<a href="#id10">10</a>]</dt> <dd><a class="reference external" href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-6396">https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-6396</a></aside> <aside class="footnote brackets" id="id22" role="doc-footnote"> <dt class="label" id="id22">[<a href="#id11">11</a>]</dt> <dd><a class="reference external" href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-6444">https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-6444</a></aside> </aside> </section> <section id="technical-details"> <h2><a class="toc-backref" href="#technical-details" role="doc-backlink">Technical Details</a></h2> <p>Python would use the system provided certificate database on all platforms. Failure to locate such a database would be an error, and users would need to explicitly specify a location to fix it.</p> <p>This will be achieved by adding a new <code class="docutils literal notranslate"><span class="pre">ssl._create_default_https_context</span></code> function, which is the same as <code class="docutils literal notranslate"><span class="pre">ssl.create_default_context</span></code>.</p> <p><code class="docutils literal notranslate"><span class="pre">http.client</span></code> can then replace its usage of <code class="docutils literal notranslate"><span class="pre">ssl._create_stdlib_context</span></code> with the <code class="docutils literal notranslate"><span class="pre">ssl._create_default_https_context</span></code>.</p> <p>Additionally <code class="docutils literal notranslate"><span class="pre">ssl._create_stdlib_context</span></code> is renamed <code class="docutils literal notranslate"><span class="pre">ssl._create_unverified_context</span></code> (an alias is kept around for backwards compatibility reasons).</p> <section id="trust-database"> <h3><a class="toc-backref" href="#trust-database" role="doc-backlink">Trust database</a></h3> <p>This PEP proposes using the system-provided certificate database. Previous discussions have suggested bundling Mozilla’s certificate database and using that by default. This was decided against for several reasons:</p> <ul class="simple"> <li>Using the platform trust database imposes a lower maintenance burden on the Python developers – shipping our own trust database would require doing a release every time a certificate was revoked.</li> <li>Linux vendors, and other downstreams, would unbundle the Mozilla certificates, resulting in a more fragmented set of behaviors.</li> <li>Using the platform stores makes it easier to handle situations such as corporate internal CAs.</li> </ul> <p>OpenSSL also has a pair of environment variables, <code class="docutils literal notranslate"><span class="pre">SSL_CERT_DIR</span></code> and <code class="docutils literal notranslate"><span class="pre">SSL_CERT_FILE</span></code> which can be used to point Python at a different certificate database.</p> </section> <section id="backwards-compatibility"> <h3><a class="toc-backref" href="#backwards-compatibility" role="doc-backlink">Backwards compatibility</a></h3> <p>This change will have the appearance of causing some HTTPS connections to “break”, because they will now raise an Exception during handshake.</p> <p>This is misleading however, in fact these connections are presently failing silently, an HTTPS URL indicates an expectation of confidentiality and authentication. The fact that Python does not actually verify that the user’s request has been made is a bug, further: “Errors should never pass silently.”</p> <p>Nevertheless, users who have a need to access servers with self-signed or incorrect certificates would be able to do so by providing a context with custom trust roots or which disables validation (documentation should strongly recommend the former where possible). Users will also be able to add necessary certificates to system trust stores in order to trust them globally.</p> <p>Twisted’s 14.0 release made this same change, and it has been met with almost no opposition.</p> </section> <section id="opting-out"> <h3><a class="toc-backref" href="#opting-out" role="doc-backlink">Opting out</a></h3> <p>For users who wish to opt out of certificate verification on a single connection, they can achieve this by providing the <code class="docutils literal notranslate"><span class="pre">context</span></code> argument to <code class="docutils literal notranslate"><span class="pre">urllib.urlopen</span></code>:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">ssl</span> <span class="c1"># This restores the same behavior as before.</span> <span class="n">context</span> <span class="o">=</span> <span class="n">ssl</span><span class="o">.</span><span class="n">_create_unverified_context</span><span class="p">()</span> <span class="n">urllib</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="s2">"https://no-valid-cert"</span><span class="p">,</span> <span class="n">context</span><span class="o">=</span><span class="n">context</span><span class="p">)</span> </pre></div> </div> <p>It is also possible, <strong>though highly discouraged</strong>, to globally disable verification by monkeypatching the <code class="docutils literal notranslate"><span class="pre">ssl</span></code> module in versions of Python that implement this PEP:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">ssl</span> <span class="k">try</span><span class="p">:</span> <span class="n">_create_unverified_https_context</span> <span class="o">=</span> <span class="n">ssl</span><span class="o">.</span><span class="n">_create_unverified_context</span> <span class="k">except</span> <span class="ne">AttributeError</span><span class="p">:</span> <span class="c1"># Legacy Python that doesn't verify HTTPS certificates by default</span> <span class="k">pass</span> <span class="k">else</span><span class="p">:</span> <span class="c1"># Handle target environment that doesn't support HTTPS verification</span> <span class="n">ssl</span><span class="o">.</span><span class="n">_create_default_https_context</span> <span class="o">=</span> <span class="n">_create_unverified_https_context</span> </pre></div> </div> <p>This guidance is aimed primarily at system administrators that wish to adopt newer versions of Python that implement this PEP in legacy environments that do not yet support certificate verification on HTTPS connections. For example, an administrator may opt out by adding the monkeypatch above to <code class="docutils literal notranslate"><span class="pre">sitecustomize.py</span></code> in their Standard Operating Environment for Python. Applications and libraries SHOULD NOT be making this change process wide (except perhaps in response to a system administrator controlled configuration setting).</p> <p>Particularly security sensitive applications should always provide an explicit application defined SSL context rather than relying on the default behaviour of the underlying Python implementation.</p> </section> </section> <section id="other-protocols"> <h2><a class="toc-backref" href="#other-protocols" role="doc-backlink">Other protocols</a></h2> <p>This PEP only proposes requiring this level of validation for HTTP clients, not for other protocols such as SMTP.</p> <p>This is because while a high percentage of HTTPS servers have correct certificates, as a result of the validation performed by browsers, for other protocols self-signed or otherwise incorrect certificates are far more common. Note that for SMTP at least, this appears to be changing and should be reviewed for a potential similar PEP in the future:</p> <ul class="simple"> <li><a class="reference external" href="https://www.facebook.com/notes/protect-the-graph/the-current-state-of-smtp-starttls-deployment/1453015901605223">https://www.facebook.com/notes/protect-the-graph/the-current-state-of-smtp-starttls-deployment/1453015901605223</a></li> <li><a class="reference external" href="https://www.facebook.com/notes/protect-the-graph/massive-growth-in-smtp-starttls-deployment/1491049534468526">https://www.facebook.com/notes/protect-the-graph/massive-growth-in-smtp-starttls-deployment/1491049534468526</a></li> </ul> </section> <section id="python-versions"> <h2><a class="toc-backref" href="#python-versions" role="doc-backlink">Python Versions</a></h2> <p>This PEP describes changes that will occur on both the 3.4.x, 3.5 and 2.7.X branches. For 2.7.X this will require backporting the <code class="docutils literal notranslate"><span class="pre">context</span></code> (<code class="docutils literal notranslate"><span class="pre">SSLContext</span></code>) argument to <code class="docutils literal notranslate"><span class="pre">httplib</span></code>, in addition to the features already backported in <a class="pep reference internal" href="../pep-0466/" title="PEP 466 – Network Security Enhancements for Python 2.7.x">PEP 466</a>.</p> </section> <section id="implementation"> <h2><a class="toc-backref" href="#implementation" role="doc-backlink">Implementation</a></h2> <ul class="simple"> <li><strong>LANDED</strong>: <a class="reference external" href="http://bugs.python.org/issue22366">Issue 22366</a> adds the <code class="docutils literal notranslate"><span class="pre">context</span></code> argument to <code class="docutils literal notranslate"><span class="pre">urlib.request.urlopen</span></code>.</li> <li><a class="reference external" href="http://bugs.python.org/issue22417">Issue 22417</a> implements the substance of this PEP.</li> </ul> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> <p>This document has been placed into the public domain.</p> </section> </section> <hr class="docutils" /> <p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0476.rst">https://github.com/python/peps/blob/main/peps/pep-0476.rst</a></p> <p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0476.rst">2025-02-01 08:59:27 GMT</a></p> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#technical-details">Technical Details</a><ul> <li><a class="reference internal" href="#trust-database">Trust database</a></li> <li><a class="reference internal" href="#backwards-compatibility">Backwards compatibility</a></li> <li><a class="reference internal" href="#opting-out">Opting out</a></li> </ul> </li> <li><a class="reference internal" href="#other-protocols">Other protocols</a></li> <li><a class="reference internal" href="#python-versions">Python Versions</a></li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <br> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0476.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>