CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 524 – Make os.urandom() blocking on Linux | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0524/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 524 – Make os.urandom() blocking on Linux | peps.python.org'> <meta property="og:description" content="Modify os.urandom() to block on Linux 3.17 and newer until the OS urandom is initialized to increase the security."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0524/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="Modify os.urandom() to block on Linux 3.17 and newer until the OS urandom is initialized to increase the security."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 524</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> <span class="visually-hidden">Toggle light / dark / auto colour theme</span> </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 524 – Make os.urandom() blocking on Linux</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author<span class="colon">:</span></dt> <dd class="field-odd">Victor Stinner <vstinner at python.org></dd> <dt class="field-even">Status<span class="colon">:</span></dt> <dd class="field-even"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-odd">Type<span class="colon">:</span></dt> <dd class="field-odd"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-even">Created<span class="colon">:</span></dt> <dd class="field-even">20-Jun-2016</dd> <dt class="field-odd">Python-Version<span class="colon">:</span></dt> <dd class="field-odd">3.6</dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#the-bug">The bug</a><ul> <li><a class="reference internal" href="#original-bug">Original bug</a></li> <li><a class="reference internal" href="#status-in-python-3-5-2">Status in Python 3.5.2</a></li> </ul> </li> <li><a class="reference internal" href="#use-cases">Use Cases</a><ul> <li><a class="reference internal" href="#use-case-1-init-script">Use Case 1: init script</a><ul> <li><a class="reference internal" href="#use-case-1-1-no-secret-needed">Use case 1.1: No secret needed</a></li> <li><a class="reference internal" href="#use-case-1-2-secure-secret-required">Use case 1.2: Secure secret required</a></li> </ul> </li> <li><a class="reference internal" href="#use-case-2-web-server">Use Case 2: Web server</a></li> </ul> </li> <li><a class="reference internal" href="#fix-system-urandom">Fix system urandom</a><ul> <li><a class="reference internal" href="#load-entropy-from-disk-at-boot">Load entropy from disk at boot</a></li> <li><a class="reference internal" href="#virtual-machines">Virtual machines</a></li> <li><a class="reference internal" href="#embedded-devices">Embedded devices</a></li> </ul> </li> <li><a class="reference internal" href="#denial-of-service-when-reading-random">Denial-of-service when reading random</a><ul> <li><a class="reference internal" href="#don-t-use-dev-random-but-dev-urandom">Don’t use /dev/random but /dev/urandom</a></li> <li><a class="reference internal" href="#getrandom-size-0-can-block-forever-on-linux">getrandom(size, 0) can block forever on Linux</a></li> </ul> </li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#changes">Changes</a><ul> <li><a class="reference internal" href="#make-os-urandom-blocking-on-linux">Make os.urandom() blocking on Linux</a></li> <li><a class="reference internal" href="#add-a-new-os-getrandom-function">Add a new os.getrandom() function</a></li> </ul> </li> <li><a class="reference internal" href="#examples-using-os-getrandom">Examples using os.getrandom()</a><ul> <li><a class="reference internal" href="#best-effort-rng">Best-effort RNG</a></li> <li><a class="reference internal" href="#wait-for-system-rng">wait_for_system_rng()</a></li> <li><a class="reference internal" href="#create-a-best-effort-rng">Create a best-effort RNG</a></li> </ul> </li> <li><a class="reference internal" href="#alternative">Alternative</a><ul> <li><a class="reference internal" href="#leave-os-urandom-unchanged-add-os-getrandom">Leave os.urandom() unchanged, add os.getrandom()</a></li> <li><a class="reference internal" href="#raise-blockingioerror-in-os-urandom">Raise BlockingIOError in os.urandom()</a><ul> <li><a class="reference internal" href="#proposition">Proposition</a></li> <li><a class="reference internal" href="#criticism">Criticism</a></li> </ul> </li> <li><a class="reference internal" href="#add-an-optional-block-parameter-to-os-urandom">Add an optional block parameter to os.urandom()</a></li> </ul> </li> <li><a class="reference internal" href="#acceptance">Acceptance</a></li> <li><a class="reference internal" href="#annexes">Annexes</a><ul> <li><a class="reference internal" href="#operating-system-random-functions">Operating system random functions</a></li> <li><a class="reference internal" href="#why-using-os-urandom">Why using os.urandom()?</a></li> </ul> </li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> <p>Modify <code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code> to block on Linux 3.17 and newer until the OS urandom is initialized to increase the security.</p> <p>Add also a new <code class="docutils literal notranslate"><span class="pre">os.getrandom()</span></code> function (for Linux and Solaris) to be able to choose how to handle when <code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code> is going to block on Linux.</p> </section> <section id="the-bug"> <h2><a class="toc-backref" href="#the-bug" role="doc-backlink">The bug</a></h2> <section id="original-bug"> <h3><a class="toc-backref" href="#original-bug" role="doc-backlink">Original bug</a></h3> <p>Python 3.5.0 was enhanced to use the new <code class="docutils literal notranslate"><span class="pre">getrandom()</span></code> syscall introduced in Linux 3.17 and Solaris 11.3. The problem is that users started to complain that Python 3.5 blocks at startup on Linux in virtual machines and embedded devices: see issues <a class="reference external" href="http://bugs.python.org/issue25420">#25420</a> and <a class="reference external" href="http://bugs.python.org/issue26839">#26839</a>.</p> <p>On Linux, <code class="docutils literal notranslate"><span class="pre">getrandom(0)</span></code> blocks until the kernel initialized urandom with 128 bits of entropy. The issue #25420 describes a Linux build platform blocking at <code class="docutils literal notranslate"><span class="pre">import</span> <span class="pre">random</span></code>. The issue #26839 describes a short Python script used to compute a MD5 hash, systemd-cron, script called very early in the init process. The system initialization blocks on this script which blocks on <code class="docutils literal notranslate"><span class="pre">getrandom(0)</span></code> to initialize Python.</p> <p>The Python initialization requires random bytes to implement a counter-measure against the hash denial-of-service (hash DoS), see:</p> <ul class="simple"> <li><a class="reference external" href="http://bugs.python.org/issue13703">Issue #13703: Hash collision security issue</a></li> <li><a class="pep reference internal" href="../pep-0456/" title="PEP 456 – Secure and interchangeable hash algorithm">PEP 456: Secure and interchangeable hash algorithm</a></li> </ul> <p>Importing the <code class="docutils literal notranslate"><span class="pre">random</span></code> module creates an instance of <code class="docutils literal notranslate"><span class="pre">random.Random</span></code>: <code class="docutils literal notranslate"><span class="pre">random._inst</span></code>. On Python 3.5, random.Random constructor reads 2500 bytes from <code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code> to seed a Mersenne Twister RNG (random number generator).</p> <p>Other platforms may be affected by this bug, but in practice, only Linux systems use Python scripts to initialize the system.</p> </section> <section id="status-in-python-3-5-2"> <h3><a class="toc-backref" href="#status-in-python-3-5-2" role="doc-backlink">Status in Python 3.5.2</a></h3> <p>Python 3.5.2 behaves like Python 2.7 and Python 3.4. If the system urandom is not initialized, the startup does not block, but <code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code> can return low-quality entropy (even it is not easily guessable).</p> </section> </section> <section id="use-cases"> <h2><a class="toc-backref" href="#use-cases" role="doc-backlink">Use Cases</a></h2> <p>The following use cases are used to help to choose the right compromise between security and practicability.</p> <section id="use-case-1-init-script"> <h3><a class="toc-backref" href="#use-case-1-init-script" role="doc-backlink">Use Case 1: init script</a></h3> <p>Use a Python 3 script to initialize the system, like systemd-cron. If the script blocks, the system initialize is stuck too. The issue #26839 is a good example of this use case.</p> <section id="use-case-1-1-no-secret-needed"> <h4><a class="toc-backref" href="#use-case-1-1-no-secret-needed" role="doc-backlink">Use case 1.1: No secret needed</a></h4> <p>If the init script doesn’t have to generate any secure secret, this use case is already handled correctly in Python 3.5.2: Python startup doesn’t block on system urandom anymore.</p> </section> <section id="use-case-1-2-secure-secret-required"> <h4><a class="toc-backref" href="#use-case-1-2-secure-secret-required" role="doc-backlink">Use case 1.2: Secure secret required</a></h4> <p>If the init script has to generate a secure secret, there is no safe solution.</p> <p>Falling back to weak entropy is not acceptable, it would reduce the security of the program.</p> <p>Python cannot produce itself secure entropy, it can only wait until system urandom is initialized. But in this use case, the whole system initialization is blocked by this script, so the system fails to boot.</p> <p>The real answer is that the system initialization must not be blocked by such script. It is ok to start the script very early at system initialization, but the script may blocked a few seconds until it is able to generate the secret.</p> <p>Reminder: in some cases, the initialization of the system urandom never occurs and so programs waiting for system urandom blocks forever.</p> </section> </section> <section id="use-case-2-web-server"> <h3><a class="toc-backref" href="#use-case-2-web-server" role="doc-backlink">Use Case 2: Web server</a></h3> <p>Run a Python 3 web server serving web pages using HTTP and HTTPS protocols. The server is started as soon as possible.</p> <p>The first target of the hash DoS attack was web server: it’s important that the hash secret cannot be easily guessed by an attacker.</p> <p>If serving a web page needs a secret to create a cookie, create an encryption key, …, the secret must be created with good entropy: again, it must be hard to guess the secret.</p> <p>A web server requires security. If a choice must be made between security and running the server with weak entropy, security is more important. If there is no good entropy: the server must block or fail with an error.</p> <p>The question is if it makes sense to start a web server on a host before system urandom is initialized.</p> <p>The issues #25420 and #26839 are restricted to the Python startup, not to generate a secret before the system urandom is initialized.</p> </section> </section> <section id="fix-system-urandom"> <h2><a class="toc-backref" href="#fix-system-urandom" role="doc-backlink">Fix system urandom</a></h2> <section id="load-entropy-from-disk-at-boot"> <h3><a class="toc-backref" href="#load-entropy-from-disk-at-boot" role="doc-backlink">Load entropy from disk at boot</a></h3> <p>Collecting entropy can take up to several minutes. To accelerate the system initialization, operating systems store entropy on disk at shutdown, and then reload entropy from disk at the boot.</p> <p>If a system collects enough entropy at least once, the system urandom will be initialized quickly, as soon as the entropy is reloaded from disk.</p> </section> <section id="virtual-machines"> <h3><a class="toc-backref" href="#virtual-machines" role="doc-backlink">Virtual machines</a></h3> <p>Virtual machines don’t have a direct access to the hardware and so have less sources of entropy than bare metal. A solution is to add a <a class="reference external" href="https://fedoraproject.org/wiki/Features/Virtio_RNG">virtio-rng device</a> to pass entropy from the host to the virtual machine.</p> </section> <section id="embedded-devices"> <h3><a class="toc-backref" href="#embedded-devices" role="doc-backlink">Embedded devices</a></h3> <p>A solution for embedded devices is to plug an hardware RNG.</p> <p>For example, Raspberry Pi have an hardware RNG but it’s not used by default. See: <a class="reference external" href="http://fios.sector16.net/hardware-rng-on-raspberry-pi/">Hardware RNG on Raspberry Pi</a>.</p> </section> </section> <section id="denial-of-service-when-reading-random"> <h2><a class="toc-backref" href="#denial-of-service-when-reading-random" role="doc-backlink">Denial-of-service when reading random</a></h2> <section id="don-t-use-dev-random-but-dev-urandom"> <h3><a class="toc-backref" href="#don-t-use-dev-random-but-dev-urandom" role="doc-backlink">Don’t use /dev/random but /dev/urandom</a></h3> <p>The <code class="docutils literal notranslate"><span class="pre">/dev/random</span></code> device should only used for very specific use cases. Reading from <code class="docutils literal notranslate"><span class="pre">/dev/random</span></code> on Linux is likely to block. Users don’t like when an application blocks longer than 5 seconds to generate a secret. It is only expected for specific cases like generating explicitly an encryption key.</p> <p>When the system has no available entropy, choosing between blocking until entropy is available or falling back on lower quality entropy is a matter of compromise between security and practicability. The choice depends on the use case.</p> <p>On Linux, <code class="docutils literal notranslate"><span class="pre">/dev/urandom</span></code> is secure, it should be used instead of <code class="docutils literal notranslate"><span class="pre">/dev/random</span></code>. See <a class="reference external" href="http://www.2uo.de/myths-about-urandom/">Myths about /dev/urandom</a> by Thomas Hühn: “Fact: /dev/urandom is the preferred source of cryptographic randomness on UNIX-like systems”</p> </section> <section id="getrandom-size-0-can-block-forever-on-linux"> <h3><a class="toc-backref" href="#getrandom-size-0-can-block-forever-on-linux" role="doc-backlink">getrandom(size, 0) can block forever on Linux</a></h3> <p>The origin of the Python issue #26839 is the <a class="reference external" href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=822431">Debian bug report #822431</a>: in fact, <code class="docutils literal notranslate"><span class="pre">getrandom(size,</span> <span class="pre">0)</span></code> blocks forever on the virtual machine. The system succeeded to boot because systemd killed the blocked process after 90 seconds.</p> <p>Solutions like <a class="reference internal" href="#load-entropy-from-disk-at-boot">Load entropy from disk at boot</a> reduces the risk of this bug.</p> </section> </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> <p>On Linux, reading the <code class="docutils literal notranslate"><span class="pre">/dev/urandom</span></code> can return “weak” entropy before urandom is fully initialized, before the kernel collected 128 bits of entropy. Linux 3.17 adds a new <code class="docutils literal notranslate"><span class="pre">getrandom()</span></code> syscall which allows to block until urandom is initialized.</p> <p>On Python 3.5.2, os.urandom() uses the <code class="docutils literal notranslate"><span class="pre">getrandom(size,</span> <span class="pre">GRND_NONBLOCK)</span></code>, but falls back on reading the non-blocking <code class="docutils literal notranslate"><span class="pre">/dev/urandom</span></code> if <code class="docutils literal notranslate"><span class="pre">getrandom(size,</span> <span class="pre">GRND_NONBLOCK)</span></code> fails with <code class="docutils literal notranslate"><span class="pre">EAGAIN</span></code>.</p> <p>Security experts promotes <code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code> to generate cryptographic keys because it is implemented with a <a class="reference external" href="https://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator">Cryptographically secure pseudo-random number generator (CSPRNG)</a>. By the way, <code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code> is preferred over <code class="docutils literal notranslate"><span class="pre">ssl.RAND_bytes()</span></code> for different reasons.</p> <p>This PEP proposes to modify os.urandom() to use <code class="docutils literal notranslate"><span class="pre">getrandom()</span></code> in blocking mode to not return weak entropy, but also ensure that Python will not block at startup.</p> </section> <section id="changes"> <h2><a class="toc-backref" href="#changes" role="doc-backlink">Changes</a></h2> <section id="make-os-urandom-blocking-on-linux"> <h3><a class="toc-backref" href="#make-os-urandom-blocking-on-linux" role="doc-backlink">Make os.urandom() blocking on Linux</a></h3> <p>All changes described in this section are specific to the Linux platform.</p> <p>Changes:</p> <ul class="simple"> <li>Modify os.urandom() to block until system urandom is initialized: <code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code> (C function <code class="docutils literal notranslate"><span class="pre">_PyOS_URandom()</span></code>) is modified to always call <code class="docutils literal notranslate"><span class="pre">getrandom(size,</span> <span class="pre">0)</span></code> (blocking mode) on Linux and Solaris.</li> <li>Add a new private <code class="docutils literal notranslate"><span class="pre">_PyOS_URandom_Nonblocking()</span></code> function: try to call <code class="docutils literal notranslate"><span class="pre">getrandom(size,</span> <span class="pre">GRND_NONBLOCK)</span></code> on Linux and Solaris, but falls back on reading <code class="docutils literal notranslate"><span class="pre">/dev/urandom</span></code> if it fails with <code class="docutils literal notranslate"><span class="pre">EAGAIN</span></code>.</li> <li>Initialize hash secret from non-blocking system urandom: <code class="docutils literal notranslate"><span class="pre">_PyRandom_Init()</span></code> is modified to call <code class="docutils literal notranslate"><span class="pre">_PyOS_URandom_Nonblocking()</span></code>.</li> <li><code class="docutils literal notranslate"><span class="pre">random.Random</span></code> constructor now uses non-blocking system urandom: it is modified to use internally the new <code class="docutils literal notranslate"><span class="pre">_PyOS_URandom_Nonblocking()</span></code> function to seed the RNG.</li> </ul> </section> <section id="add-a-new-os-getrandom-function"> <h3><a class="toc-backref" href="#add-a-new-os-getrandom-function" role="doc-backlink">Add a new os.getrandom() function</a></h3> <p>A new <code class="docutils literal notranslate"><span class="pre">os.getrandom(size,</span> <span class="pre">flags=0)</span></code> function is added: use <code class="docutils literal notranslate"><span class="pre">getrandom()</span></code> syscall on Linux and <code class="docutils literal notranslate"><span class="pre">getrandom()</span></code> C function on Solaris.</p> <p>The function comes with 2 new flags:</p> <ul class="simple"> <li><code class="docutils literal notranslate"><span class="pre">os.GRND_RANDOM</span></code>: read bytes from <code class="docutils literal notranslate"><span class="pre">/dev/random</span></code> rather than reading <code class="docutils literal notranslate"><span class="pre">/dev/urandom</span></code></li> <li><code class="docutils literal notranslate"><span class="pre">os.GRND_NONBLOCK</span></code>: raise a BlockingIOError if <code class="docutils literal notranslate"><span class="pre">os.getrandom()</span></code> would block</li> </ul> <p>The <code class="docutils literal notranslate"><span class="pre">os.getrandom()</span></code> is a thin wrapper on the <code class="docutils literal notranslate"><span class="pre">getrandom()</span></code> syscall/C function and so inherit of its behaviour. For example, on Linux, it can return less bytes than requested if the syscall is interrupted by a signal.</p> </section> </section> <section id="examples-using-os-getrandom"> <h2><a class="toc-backref" href="#examples-using-os-getrandom" role="doc-backlink">Examples using os.getrandom()</a></h2> <section id="best-effort-rng"> <h3><a class="toc-backref" href="#best-effort-rng" role="doc-backlink">Best-effort RNG</a></h3> <p>Example of a portable non-blocking RNG function: try to get random bytes from the OS urandom, or fallback on the random module.</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">best_effort_rng</span><span class="p">(</span><span class="n">size</span><span class="p">):</span> <span class="c1"># getrandom() is only available on Linux and Solaris</span> <span class="k">if</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">os</span><span class="p">,</span> <span class="s1">'getrandom'</span><span class="p">):</span> <span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">urandom</span><span class="p">(</span><span class="n">size</span><span class="p">)</span> <span class="n">result</span> <span class="o">=</span> <span class="nb">bytearray</span><span class="p">()</span> <span class="k">try</span><span class="p">:</span> <span class="c1"># need a loop because getrandom() can return less bytes than</span> <span class="c1"># requested for different reasons</span> <span class="k">while</span> <span class="n">size</span><span class="p">:</span> <span class="n">data</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getrandom</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">GRND_NONBLOCK</span><span class="p">)</span> <span class="n">result</span> <span class="o">+=</span> <span class="n">data</span> <span class="n">size</span> <span class="o">-=</span> <span class="nb">len</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="k">except</span> <span class="ne">BlockingIOError</span><span class="p">:</span> <span class="c1"># OS urandom is not initialized yet:</span> <span class="c1"># fallback on the Python random module</span> <span class="n">data</span> <span class="o">=</span> <span class="nb">bytes</span><span class="p">(</span><span class="n">random</span><span class="o">.</span><span class="n">randrange</span><span class="p">(</span><span class="mi">256</span><span class="p">)</span> <span class="k">for</span> <span class="n">byte</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">size</span><span class="p">))</span> <span class="n">result</span> <span class="o">+=</span> <span class="n">data</span> <span class="k">return</span> <span class="nb">bytes</span><span class="p">(</span><span class="n">result</span><span class="p">)</span> </pre></div> </div> <p>This function <em>can</em> block in theory on a platform where <code class="docutils literal notranslate"><span class="pre">os.getrandom()</span></code> is not available but <code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code> can block.</p> </section> <section id="wait-for-system-rng"> <h3><a class="toc-backref" href="#wait-for-system-rng" role="doc-backlink">wait_for_system_rng()</a></h3> <p>Example of function waiting <em>timeout</em> seconds until the OS urandom is initialized on Linux or Solaris:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">wait_for_system_rng</span><span class="p">(</span><span class="n">timeout</span><span class="p">,</span> <span class="n">interval</span><span class="o">=</span><span class="mf">1.0</span><span class="p">):</span> <span class="k">if</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">os</span><span class="p">,</span> <span class="s1">'getrandom'</span><span class="p">):</span> <span class="k">return</span> <span class="n">deadline</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">monotonic</span><span class="p">()</span> <span class="o">+</span> <span class="n">timeout</span> <span class="k">while</span> <span class="kc">True</span><span class="p">:</span> <span class="k">try</span><span class="p">:</span> <span class="n">os</span><span class="o">.</span><span class="n">getrandom</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">GRND_NONBLOCK</span><span class="p">)</span> <span class="k">except</span> <span class="ne">BlockingIOError</span><span class="p">:</span> <span class="k">pass</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="k">if</span> <span class="n">time</span><span class="o">.</span><span class="n">monotonic</span><span class="p">()</span> <span class="o">></span> <span class="n">deadline</span><span class="p">:</span> <span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span><span class="s1">'OS urandom not initialized after </span><span class="si">%s</span><span class="s1"> seconds'</span> <span class="o">%</span> <span class="n">timeout</span><span class="p">)</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">interval</span><span class="p">)</span> </pre></div> </div> <p>This function is <em>not</em> portable. For example, <code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code> can block on FreeBSD in theory, at the early stage of the system initialization.</p> </section> <section id="create-a-best-effort-rng"> <h3><a class="toc-backref" href="#create-a-best-effort-rng" role="doc-backlink">Create a best-effort RNG</a></h3> <p>Simpler example to create a non-blocking RNG on Linux: choose between <code class="docutils literal notranslate"><span class="pre">Random.SystemRandom</span></code> and <code class="docutils literal notranslate"><span class="pre">Random.Random</span></code> depending if <code class="docutils literal notranslate"><span class="pre">getrandom(size)</span></code> would block.</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">create_nonblocking_random</span><span class="p">():</span> <span class="k">if</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">os</span><span class="p">,</span> <span class="s1">'getrandom'</span><span class="p">):</span> <span class="k">return</span> <span class="n">random</span><span class="o">.</span><span class="n">Random</span><span class="p">()</span> <span class="k">try</span><span class="p">:</span> <span class="n">os</span><span class="o">.</span><span class="n">getrandom</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">GRND_NONBLOCK</span><span class="p">)</span> <span class="k">except</span> <span class="ne">BlockingIOError</span><span class="p">:</span> <span class="k">return</span> <span class="n">random</span><span class="o">.</span><span class="n">Random</span><span class="p">()</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="n">random</span><span class="o">.</span><span class="n">SystemRandom</span><span class="p">()</span> </pre></div> </div> <p>This function is <em>not</em> portable. For example, <code class="docutils literal notranslate"><span class="pre">random.SystemRandom</span></code> can block on FreeBSD in theory, at the early stage of the system initialization.</p> </section> </section> <section id="alternative"> <h2><a class="toc-backref" href="#alternative" role="doc-backlink">Alternative</a></h2> <section id="leave-os-urandom-unchanged-add-os-getrandom"> <h3><a class="toc-backref" href="#leave-os-urandom-unchanged-add-os-getrandom" role="doc-backlink">Leave os.urandom() unchanged, add os.getrandom()</a></h3> <p>os.urandom() remains unchanged: never block, but it can return weak entropy if system urandom is not initialized yet.</p> <p>Only add the new <code class="docutils literal notranslate"><span class="pre">os.getrandom()</span></code> function (wrapper to the <code class="docutils literal notranslate"><span class="pre">getrandom()</span></code> syscall/C function).</p> <p>The <code class="docutils literal notranslate"><span class="pre">secrets.token_bytes()</span></code> function should be used to write portable code.</p> <p>The problem with this change is that it expects that users understand well security and know well each platforms. Python has the tradition of hiding “implementation details”. For example, <code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code> is not a thin wrapper to the <code class="docutils literal notranslate"><span class="pre">/dev/urandom</span></code> device: it uses <code class="docutils literal notranslate"><span class="pre">CryptGenRandom()</span></code> on Windows, it uses <code class="docutils literal notranslate"><span class="pre">getentropy()</span></code> on OpenBSD, it tries <code class="docutils literal notranslate"><span class="pre">getrandom()</span></code> on Linux and Solaris or falls back on reading <code class="docutils literal notranslate"><span class="pre">/dev/urandom</span></code>. Python already uses the best available system RNG depending on the platform.</p> <p>This PEP does not change the API:</p> <ul class="simple"> <li><code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code>, <code class="docutils literal notranslate"><span class="pre">random.SystemRandom</span></code> and <code class="docutils literal notranslate"><span class="pre">secrets</span></code> for security</li> <li><code class="docutils literal notranslate"><span class="pre">random</span></code> module (except <code class="docutils literal notranslate"><span class="pre">random.SystemRandom</span></code>) for all other usages</li> </ul> </section> <section id="raise-blockingioerror-in-os-urandom"> <h3><a class="toc-backref" href="#raise-blockingioerror-in-os-urandom" role="doc-backlink">Raise BlockingIOError in os.urandom()</a></h3> <section id="proposition"> <h4><a class="toc-backref" href="#proposition" role="doc-backlink">Proposition</a></h4> <p><a class="pep reference internal" href="../pep-0522/" title="PEP 522 – Allow BlockingIOError in security sensitive APIs">PEP 522: Allow BlockingIOError in security sensitive APIs on Linux</a>.</p> <p>Python should not decide for the developer how to handle <a class="reference internal" href="#the-bug">The bug</a>: raising immediately a <code class="docutils literal notranslate"><span class="pre">BlockingIOError</span></code> if <code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code> is going to block allows developers to choose how to handle this case:</p> <ul class="simple"> <li>catch the exception and falls back to a non-secure entropy source: read <code class="docutils literal notranslate"><span class="pre">/dev/urandom</span></code> on Linux, use the Python <code class="docutils literal notranslate"><span class="pre">random</span></code> module (which is not secure at all), use time, use process identifier, etc.</li> <li>don’t catch the error, the whole program fails with this fatal exception</li> </ul> <p>More generally, the exception helps to notify when sometimes goes wrong. The application can emit a warning when it starts to wait for <code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code>.</p> </section> <section id="criticism"> <h4><a class="toc-backref" href="#criticism" role="doc-backlink">Criticism</a></h4> <p>For the use case 2 (web server), falling back on non-secure entropy is not acceptable. The application must handle <code class="docutils literal notranslate"><span class="pre">BlockingIOError</span></code>: poll <code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code> until it completes. Example:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">secret</span><span class="p">(</span><span class="n">n</span><span class="o">=</span><span class="mi">16</span><span class="p">):</span> <span class="k">try</span><span class="p">:</span> <span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">urandom</span><span class="p">(</span><span class="n">n</span><span class="p">)</span> <span class="k">except</span> <span class="ne">BlockingIOError</span><span class="p">:</span> <span class="k">pass</span> <span class="nb">print</span><span class="p">(</span><span class="s2">"Wait for system urandom initialization: move your "</span> <span class="s2">"mouse, use your keyboard, use your disk, ..."</span><span class="p">)</span> <span class="k">while</span> <span class="mi">1</span><span class="p">:</span> <span class="c1"># Avoid busy-loop: sleep 1 ms</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.001</span><span class="p">)</span> <span class="k">try</span><span class="p">:</span> <span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">urandom</span><span class="p">(</span><span class="n">n</span><span class="p">)</span> <span class="k">except</span> <span class="ne">BlockingIOError</span><span class="p">:</span> <span class="k">pass</span> </pre></div> </div> <p>For correctness, all applications which must generate a secure secret must be modified to handle <code class="docutils literal notranslate"><span class="pre">BlockingIOError</span></code> even if <a class="reference internal" href="#the-bug">The bug</a> is unlikely.</p> <p>The case of applications using <code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code> but don’t really require security is not well defined. Maybe these applications should not use <code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code> at the first place, but always the non-blocking <code class="docutils literal notranslate"><span class="pre">random</span></code> module. If <code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code> is used for security, we are back to the use case 2 described above: <a class="reference internal" href="#use-case-2-web-server">Use Case 2: Web server</a>. If a developer doesn’t want to drop <code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code>, the code should be modified. Example:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">almost_secret</span><span class="p">(</span><span class="n">n</span><span class="o">=</span><span class="mi">16</span><span class="p">):</span> <span class="k">try</span><span class="p">:</span> <span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">urandom</span><span class="p">(</span><span class="n">n</span><span class="p">)</span> <span class="k">except</span> <span class="ne">BlockingIOError</span><span class="p">:</span> <span class="k">return</span> <span class="nb">bytes</span><span class="p">(</span><span class="n">random</span><span class="o">.</span><span class="n">randrange</span><span class="p">(</span><span class="mi">256</span><span class="p">)</span> <span class="k">for</span> <span class="n">byte</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n</span><span class="p">))</span> </pre></div> </div> <p>The question is if <a class="reference internal" href="#the-bug">The bug</a> is common enough to require that so many applications have to be modified.</p> <p>Another simpler choice is to refuse to start before the system urandom is initialized:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">secret</span><span class="p">(</span><span class="n">n</span><span class="o">=</span><span class="mi">16</span><span class="p">):</span> <span class="k">try</span><span class="p">:</span> <span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">urandom</span><span class="p">(</span><span class="n">n</span><span class="p">)</span> <span class="k">except</span> <span class="ne">BlockingIOError</span><span class="p">:</span> <span class="nb">print</span><span class="p">(</span><span class="s2">"Fatal error: the system urandom is not initialized"</span><span class="p">)</span> <span class="nb">print</span><span class="p">(</span><span class="s2">"Wait a bit, and rerun the program later."</span><span class="p">)</span> <span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> </pre></div> </div> <p>Compared to Python 2.7, Python 3.4 and Python 3.5.2 where os.urandom() never blocks nor raise an exception on Linux, such behaviour change can be seen as a major regression.</p> </section> </section> <section id="add-an-optional-block-parameter-to-os-urandom"> <h3><a class="toc-backref" href="#add-an-optional-block-parameter-to-os-urandom" role="doc-backlink">Add an optional block parameter to os.urandom()</a></h3> <p>See the <a class="reference external" href="http://bugs.python.org/issue27250">issue #27250: Add os.urandom_block()</a>.</p> <p>Add an optional block parameter to os.urandom(). The default value may be <code class="docutils literal notranslate"><span class="pre">True</span></code> (block by default) or <code class="docutils literal notranslate"><span class="pre">False</span></code> (non-blocking).</p> <p>The first technical issue is to implement <code class="docutils literal notranslate"><span class="pre">os.urandom(block=False)</span></code> on all platforms. Only Linux 3.17 (and newer) and Solaris 11.3 (and newer) have a well defined non-blocking API (<code class="docutils literal notranslate"><span class="pre">getrandom(size,</span> <span class="pre">GRND_NONBLOCK)</span></code>).</p> <p>As <a class="reference internal" href="#raise-blockingioerror-in-os-urandom">Raise BlockingIOError in os.urandom()</a>, it doesn’t seem worth it to make the API more complex for a theoretical (or at least very rare) use case.</p> <p>As <a class="reference internal" href="#leave-os-urandom-unchanged-add-os-getrandom">Leave os.urandom() unchanged, add os.getrandom()</a>, the problem is that it makes the API more complex and so more error-prone.</p> </section> </section> <section id="acceptance"> <h2><a class="toc-backref" href="#acceptance" role="doc-backlink">Acceptance</a></h2> <p>The PEP was <a class="reference external" href="https://mail.python.org/pipermail/security-sig/2016-August/000101.html">accepted on 2016-08-08 by Guido van Rossum</a>.</p> </section> <section id="annexes"> <h2><a class="toc-backref" href="#annexes" role="doc-backlink">Annexes</a></h2> <section id="operating-system-random-functions"> <h3><a class="toc-backref" href="#operating-system-random-functions" role="doc-backlink">Operating system random functions</a></h3> <p><code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code> uses the following functions:</p> <ul class="simple"> <li><a class="reference external" href="http://man.openbsd.org/OpenBSD-current/man2/getentropy.2">OpenBSD: getentropy()</a> (OpenBSD 5.6)</li> <li><a class="reference external" href="http://man7.org/linux/man-pages/man2/getrandom.2.html">Linux: getrandom()</a> (Linux 3.17) – see also <a class="reference external" href="https://lwn.net/Articles/606141/">A system call for random numbers: getrandom()</a></li> <li>Solaris: <a class="reference external" href="https://docs.oracle.com/cd/E53394_01/html/E54765/getentropy-2.html#scrolltoc">getentropy()</a>, <a class="reference external" href="https://docs.oracle.com/cd/E53394_01/html/E54765/getrandom-2.html">getrandom()</a> (both need Solaris 11.3)</li> <li>UNIX, BSD: /dev/urandom, /dev/random</li> <li>Windows: <a class="reference external" href="https://msdn.microsoft.com/en-us/library/windows/desktop/aa379942%28v=vs.85%29.aspx">CryptGenRandom()</a> (Windows XP)</li> </ul> <p>On Linux, commands to get the status of <code class="docutils literal notranslate"><span class="pre">/dev/random</span></code> (results are number of bytes):</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ cat /proc/sys/kernel/random/entropy_avail 2850 $ cat /proc/sys/kernel/random/poolsize 4096 </pre></div> </div> </section> <section id="why-using-os-urandom"> <h3><a class="toc-backref" href="#why-using-os-urandom" role="doc-backlink">Why using os.urandom()?</a></h3> <p>Since <code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code> is implemented in the kernel, it doesn’t have issues of user-space RNG. For example, it is much harder to get its state. It is usually built on a CSPRNG, so even if its state is “stolen”, it is hard to compute previously generated numbers. The kernel has a good knowledge of entropy sources and feed regularly the entropy pool.</p> <p>That’s also why <code class="docutils literal notranslate"><span class="pre">os.urandom()</span></code> is preferred over <code class="docutils literal notranslate"><span class="pre">ssl.RAND_bytes()</span></code>.</p> </section> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> <p>This document has been placed in the public domain.</p> </section> </section> <hr class="docutils" /> <p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0524.rst">https://github.com/python/peps/blob/main/peps/pep-0524.rst</a></p> <p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0524.rst">2025-02-01 08:59:27 GMT</a></p> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#the-bug">The bug</a><ul> <li><a class="reference internal" href="#original-bug">Original bug</a></li> <li><a class="reference internal" href="#status-in-python-3-5-2">Status in Python 3.5.2</a></li> </ul> </li> <li><a class="reference internal" href="#use-cases">Use Cases</a><ul> <li><a class="reference internal" href="#use-case-1-init-script">Use Case 1: init script</a><ul> <li><a class="reference internal" href="#use-case-1-1-no-secret-needed">Use case 1.1: No secret needed</a></li> <li><a class="reference internal" href="#use-case-1-2-secure-secret-required">Use case 1.2: Secure secret required</a></li> </ul> </li> <li><a class="reference internal" href="#use-case-2-web-server">Use Case 2: Web server</a></li> </ul> </li> <li><a class="reference internal" href="#fix-system-urandom">Fix system urandom</a><ul> <li><a class="reference internal" href="#load-entropy-from-disk-at-boot">Load entropy from disk at boot</a></li> <li><a class="reference internal" href="#virtual-machines">Virtual machines</a></li> <li><a class="reference internal" href="#embedded-devices">Embedded devices</a></li> </ul> </li> <li><a class="reference internal" href="#denial-of-service-when-reading-random">Denial-of-service when reading random</a><ul> <li><a class="reference internal" href="#don-t-use-dev-random-but-dev-urandom">Don’t use /dev/random but /dev/urandom</a></li> <li><a class="reference internal" href="#getrandom-size-0-can-block-forever-on-linux">getrandom(size, 0) can block forever on Linux</a></li> </ul> </li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#changes">Changes</a><ul> <li><a class="reference internal" href="#make-os-urandom-blocking-on-linux">Make os.urandom() blocking on Linux</a></li> <li><a class="reference internal" href="#add-a-new-os-getrandom-function">Add a new os.getrandom() function</a></li> </ul> </li> <li><a class="reference internal" href="#examples-using-os-getrandom">Examples using os.getrandom()</a><ul> <li><a class="reference internal" href="#best-effort-rng">Best-effort RNG</a></li> <li><a class="reference internal" href="#wait-for-system-rng">wait_for_system_rng()</a></li> <li><a class="reference internal" href="#create-a-best-effort-rng">Create a best-effort RNG</a></li> </ul> </li> <li><a class="reference internal" href="#alternative">Alternative</a><ul> <li><a class="reference internal" href="#leave-os-urandom-unchanged-add-os-getrandom">Leave os.urandom() unchanged, add os.getrandom()</a></li> <li><a class="reference internal" href="#raise-blockingioerror-in-os-urandom">Raise BlockingIOError in os.urandom()</a><ul> <li><a class="reference internal" href="#proposition">Proposition</a></li> <li><a class="reference internal" href="#criticism">Criticism</a></li> </ul> </li> <li><a class="reference internal" href="#add-an-optional-block-parameter-to-os-urandom">Add an optional block parameter to os.urandom()</a></li> </ul> </li> <li><a class="reference internal" href="#acceptance">Acceptance</a></li> <li><a class="reference internal" href="#annexes">Annexes</a><ul> <li><a class="reference internal" href="#operating-system-random-functions">Operating system random functions</a></li> <li><a class="reference internal" href="#why-using-os-urandom">Why using os.urandom()?</a></li> </ul> </li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <br> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0524.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>