CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="color-scheme" content="light dark"> <title>PEP 471 – os.scandir() function – a better and faster directory iterator | peps.python.org</title> <link rel="shortcut icon" href="../_static/py.png"> <link rel="canonical" href="https://peps.python.org/pep-0471/"> <link rel="stylesheet" href="../_static/style.css" type="text/css"> <link rel="stylesheet" href="../_static/mq.css" type="text/css"> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light"> <link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark"> <link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss"> <meta property="og:title" content='PEP 471 – os.scandir() function – a better and faster directory iterator | peps.python.org'> <meta property="og:description" content="This PEP proposes including a new directory iteration function, os.scandir(), in the standard library. This new function adds useful functionality and increases the speed of os.walk() by 2-20 times (depending on the platform and file system) by avoiding..."> <meta property="og:type" content="website"> <meta property="og:url" content="https://peps.python.org/pep-0471/"> <meta property="og:site_name" content="Python Enhancement Proposals (PEPs)"> <meta property="og:image" content="https://peps.python.org/_static/og-image.png"> <meta property="og:image:alt" content="Python PEPs"> <meta property="og:image:width" content="200"> <meta property="og:image:height" content="200"> <meta name="description" content="This PEP proposes including a new directory iteration function, os.scandir(), in the standard library. This new function adds useful functionality and increases the speed of os.walk() by 2-20 times (depending on the platform and file system) by avoiding..."> <meta name="theme-color" content="#3776ab"> </head> <body> <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"> <symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all"> <title>Following system colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="9"></circle> <path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path> </svg> </symbol> <symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all"> <title>Selected dark colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <path stroke="none" d="M0 0h24v24H0z" fill="none"></path> <path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path> </svg> </symbol> <symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all"> <title>Selected light colour scheme</title> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"> <circle cx="12" cy="12" r="5"></circle> <line x1="12" y1="1" x2="12" y2="3"></line> <line x1="12" y1="21" x2="12" y2="23"></line> <line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line> <line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line> <line x1="1" y1="12" x2="3" y2="12"></line> <line x1="21" y1="12" x2="23" y2="12"></line> <line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line> <line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line> </svg> </symbol> </svg> <script> document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto" </script> <section id="pep-page-section"> <header> <h1>Python Enhancement Proposals</h1> <ul class="breadcrumbs"> <li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> » </li> <li><a href="../pep-0000/">PEP Index</a> » </li> <li>PEP 471</li> </ul> <button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())"> <svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg> <svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg> Toggle light / dark / auto colour theme </button> </header> <article> <section id="pep-content"> <h1 class="page-title">PEP 471 – os.scandir() function – a better and faster directory iterator</h1> <dl class="rfc2822 field-list simple"> <dt class="field-odd">Author:</dt> <dd class="field-odd">Ben Hoyt <benhoyt at gmail.com></dd> <dt class="field-even">BDFL-Delegate:</dt> <dd class="field-even">Victor Stinner <vstinner at python.org></dd> <dt class="field-odd">Status:</dt> <dd class="field-odd"><abbr title="Accepted and implementation complete, or no longer active">Final</abbr></dd> <dt class="field-even">Type:</dt> <dd class="field-even"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd> <dt class="field-odd">Created:</dt> <dd class="field-odd">30-May-2014</dd> <dt class="field-even">Python-Version:</dt> <dd class="field-even">3.5</dd> <dt class="field-odd">Post-History:</dt> <dd class="field-odd">27-Jun-2014, 08-Jul-2014, 14-Jul-2014</dd> </dl> <hr class="docutils" /> <section id="contents"> <details><summary>Table of Contents</summary><ul class="simple"> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#specifics-of-proposal">Specifics of proposal</a><ul> <li><a class="reference internal" href="#os-scandir">os.scandir()</a></li> <li><a class="reference internal" href="#os-walk">os.walk()</a></li> </ul> </li> <li><a class="reference internal" href="#examples">Examples</a><ul> <li><a class="reference internal" href="#notes-on-caching">Notes on caching</a></li> <li><a class="reference internal" href="#notes-on-exception-handling">Notes on exception handling</a></li> </ul> </li> <li><a class="reference internal" href="#support">Support</a></li> <li><a class="reference internal" href="#use-in-the-wild">Use in the wild</a></li> <li><a class="reference internal" href="#rejected-ideas">Rejected ideas</a><ul> <li><a class="reference internal" href="#naming">Naming</a></li> <li><a class="reference internal" href="#wildcard-support">Wildcard support</a></li> <li><a class="reference internal" href="#methods-not-following-symlinks-by-default">Methods not following symlinks by default</a></li> <li><a class="reference internal" href="#direntry-attributes-being-properties">DirEntry attributes being properties</a></li> <li><a class="reference internal" href="#direntry-fields-being-static-attribute-only-objects">DirEntry fields being “static” attribute-only objects</a></li> <li><a class="reference internal" href="#direntry-fields-being-static-with-an-ensure-lstat-option">DirEntry fields being static with an ensure_lstat option</a></li> <li><a class="reference internal" href="#return-values-being-name-stat-result-two-tuples">Return values being (name, stat_result) two-tuples</a></li> <li><a class="reference internal" href="#return-values-being-overloaded-stat-result-objects">Return values being overloaded stat_result objects</a></li> <li><a class="reference internal" href="#return-values-being-pathlib-path-objects">Return values being pathlib.Path objects</a></li> </ul> </li> <li><a class="reference internal" href="#possible-improvements">Possible improvements</a></li> <li><a class="reference internal" href="#previous-discussion">Previous discussion</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> </details></section> <section id="abstract"> <h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2> This PEP proposes including a new directory iteration function, <code class="docutils literal notranslate">os.scandir()</code>, in the standard library. This new function adds useful functionality and increases the speed of <code class="docutils literal notranslate">os.walk()</code> by 2-20 times (depending on the platform and file system) by avoiding calls to <code class="docutils literal notranslate">os.stat()</code> in most cases. </section> <section id="rationale"> <h2><a class="toc-backref" href="#rationale" role="doc-backlink">Rationale</a></h2> Python’s built-in <code class="docutils literal notranslate">os.walk()</code> is significantly slower than it needs to be, because – in addition to calling <code class="docutils literal notranslate">os.listdir()</code> on each directory – it executes the <code class="docutils literal notranslate">stat()</code> system call or <code class="docutils literal notranslate">GetFileAttributes()</code> on each file to determine whether the entry is a directory or not. But the underlying system calls – <code class="docutils literal notranslate">FindFirstFile</code> / <code class="docutils literal notranslate">FindNextFile</code> on Windows and <code class="docutils literal notranslate">readdir</code> on POSIX systems – already tell you whether the files returned are directories or not, so no further system calls are needed. Further, the Windows system calls return all the information for a <code class="docutils literal notranslate">stat_result</code> object on the directory entry, such as file size and last modification time. In short, you can reduce the number of system calls required for a tree function like <code class="docutils literal notranslate">os.walk()</code> from approximately 2N to N, where N is the total number of files and directories in the tree. (And because directory trees are usually wider than they are deep, it’s often much better than this.) In practice, removing all those extra system calls makes <code class="docutils literal notranslate">os.walk()</code> about 8-9 times as fast on Windows, and about 2-3 times as fast on POSIX systems. So we’re not talking about micro-optimizations. See more <a class="reference external" href="https://github.com/benhoyt/scandir#benchmarks">benchmarks here</a>. Somewhat relatedly, many people (see Python <a class="reference external" href="http://bugs.python.org/issue11406">Issue 11406</a>) are also keen on a version of <code class="docutils literal notranslate">os.listdir()</code> that yields filenames as it iterates instead of returning them as one big list. This improves memory efficiency for iterating very large directories. So, as well as providing a <code class="docutils literal notranslate">scandir()</code> iterator function for calling directly, Python’s existing <code class="docutils literal notranslate">os.walk()</code> function can be sped up a huge amount. </section> <section id="implementation"> <h2><a class="toc-backref" href="#implementation" role="doc-backlink">Implementation</a></h2> The implementation of this proposal was written by Ben Hoyt (initial version) and Tim Golden (who helped a lot with the C extension module). It lives on GitHub at <a class="reference external" href="https://github.com/benhoyt/scandir">benhoyt/scandir</a>. (The implementation may lag behind the updates to this PEP a little.) Note that this module has been used and tested (see “Use in the wild” section in this PEP), so it’s more than a proof-of-concept. However, it is marked as beta software and is not extensively battle-tested. It will need some cleanup and more thorough testing before going into the standard library, as well as integration into <code class="docutils literal notranslate">posixmodule.c</code>. </section> <section id="specifics-of-proposal"> <h2><a class="toc-backref" href="#specifics-of-proposal" role="doc-backlink">Specifics of proposal</a></h2> <section id="os-scandir"> <h3><a class="toc-backref" href="#os-scandir" role="doc-backlink">os.scandir()</a></h3> Specifically, this PEP proposes adding a single function to the <code class="docutils literal notranslate">os</code> module in the standard library, <code class="docutils literal notranslate">scandir</code>, that takes a single, optional string as its argument: <div class="highlight-default notranslate"><div class="highlight"><pre>scandir(path='.') -> generator of DirEntry objects </pre></div> </div> Like <code class="docutils literal notranslate">listdir</code>, <code class="docutils literal notranslate">scandir</code> calls the operating system’s directory iteration system calls to get the names of the files in the given <code class="docutils literal notranslate">path</code>, but it’s different from <code class="docutils literal notranslate">listdir</code> in two ways: <ul class="simple"> <li>Instead of returning bare filename strings, it returns lightweight <code class="docutils literal notranslate">DirEntry</code> objects that hold the filename string and provide simple methods that allow access to the additional data the operating system may have returned.</li> <li>It returns a generator instead of a list, so that <code class="docutils literal notranslate">scandir</code> acts as a true iterator instead of returning the full list immediately.</li> </ul> <code class="docutils literal notranslate">scandir()</code> yields a <code class="docutils literal notranslate">DirEntry</code> object for each file and sub-directory in <code class="docutils literal notranslate">path</code>. Just like <code class="docutils literal notranslate">listdir</code>, the <code class="docutils literal notranslate">'.'</code> and <code class="docutils literal notranslate">'..'</code> pseudo-directories are skipped, and the entries are yielded in system-dependent order. Each <code class="docutils literal notranslate">DirEntry</code> object has the following attributes and methods: <ul class="simple"> <li><code class="docutils literal notranslate">name</code>: the entry’s filename, relative to the scandir <code class="docutils literal notranslate">path</code> argument (corresponds to the return values of <code class="docutils literal notranslate">os.listdir</code>)</li> <li><code class="docutils literal notranslate">path</code>: the entry’s full path name (not necessarily an absolute path) – the equivalent of <code class="docutils literal notranslate">os.path.join(scandir_path, entry.name)</code></li> <li><code class="docutils literal notranslate">inode()</code>: return the inode number of the entry. The result is cached on the <code class="docutils literal notranslate">DirEntry</code> object, use <code class="docutils literal notranslate">os.stat(entry.path, follow_symlinks=False).st_ino</code> to fetch up-to-date information. On Unix, no system call is required.</li> <li><code class="docutils literal notranslate">is_dir(*, follow_symlinks=True)</code>: similar to <code class="docutils literal notranslate">pathlib.Path.is_dir()</code>, but the return value is cached on the <code class="docutils literal notranslate">DirEntry</code> object; doesn’t require a system call in most cases; don’t follow symbolic links if <code class="docutils literal notranslate">follow_symlinks</code> is False</li> <li><code class="docutils literal notranslate">is_file(*, follow_symlinks=True)</code>: similar to <code class="docutils literal notranslate">pathlib.Path.is_file()</code>, but the return value is cached on the <code class="docutils literal notranslate">DirEntry</code> object; doesn’t require a system call in most cases; don’t follow symbolic links if <code class="docutils literal notranslate">follow_symlinks</code> is False</li> <li><code class="docutils literal notranslate">is_symlink()</code>: similar to <code class="docutils literal notranslate">pathlib.Path.is_symlink()</code>, but the return value is cached on the <code class="docutils literal notranslate">DirEntry</code> object; doesn’t require a system call in most cases</li> <li><code class="docutils literal notranslate">stat(*, follow_symlinks=True)</code>: like <code class="docutils literal notranslate">os.stat()</code>, but the return value is cached on the <code class="docutils literal notranslate">DirEntry</code> object; does not require a system call on Windows (except for symlinks); don’t follow symbolic links (like <code class="docutils literal notranslate">os.lstat()</code>) if <code class="docutils literal notranslate">follow_symlinks</code> is False</li> </ul> All methods may perform system calls in some cases and therefore possibly raise <code class="docutils literal notranslate">OSError</code> – see the “Notes on exception handling” section for more details. The <code class="docutils literal notranslate">DirEntry</code> attribute and method names were chosen to be the same as those in the new <code class="docutils literal notranslate">pathlib</code> module where possible, for consistency. The only difference in functionality is that the <code class="docutils literal notranslate">DirEntry</code> methods cache their values on the entry object after the first call. Like the other functions in the <code class="docutils literal notranslate">os</code> module, <code class="docutils literal notranslate">scandir()</code> accepts either a bytes or str object for the <code class="docutils literal notranslate">path</code> parameter, and returns the <code class="docutils literal notranslate">DirEntry.name</code> and <code class="docutils literal notranslate">DirEntry.path</code> attributes with the same type as <code class="docutils literal notranslate">path</code>. However, it is strongly recommended to use the str type, as this ensures cross-platform support for Unicode filenames. (On Windows, bytes filenames have been deprecated since Python 3.3). </section> <section id="os-walk"> <h3><a class="toc-backref" href="#os-walk" role="doc-backlink">os.walk()</a></h3> As part of this proposal, <code class="docutils literal notranslate">os.walk()</code> will also be modified to use <code class="docutils literal notranslate">scandir()</code> rather than <code class="docutils literal notranslate">listdir()</code> and <code class="docutils literal notranslate">os.path.isdir()</code>. This will increase the speed of <code class="docutils literal notranslate">os.walk()</code> very significantly (as mentioned above, by 2-20 times, depending on the system). </section> </section> <section id="examples"> <h2><a class="toc-backref" href="#examples" role="doc-backlink">Examples</a></h2> First, a very simple example of <code class="docutils literal notranslate">scandir()</code> showing use of the <code class="docutils literal notranslate">DirEntry.name</code> attribute and the <code class="docutils literal notranslate">DirEntry.is_dir()</code> method: <div class="highlight-default notranslate"><div class="highlight"><pre>def subdirs(path): """Yield directory names not starting with '.' under given path.""" for entry in os.scandir(path): if not entry.name.startswith('.') and entry.is_dir(): yield entry.name </pre></div> </div> This <code class="docutils literal notranslate">subdirs()</code> function will be significantly faster with scandir than <code class="docutils literal notranslate">os.listdir()</code> and <code class="docutils literal notranslate">os.path.isdir()</code> on both Windows and POSIX systems, especially on medium-sized or large directories. Or, for getting the total size of files in a directory tree, showing use of the <code class="docutils literal notranslate">DirEntry.stat()</code> method and <code class="docutils literal notranslate">DirEntry.path</code> attribute: <div class="highlight-default notranslate"><div class="highlight"><pre>def get_tree_size(path): """Return total size of files in given path and subdirs.""" total = 0 for entry in os.scandir(path): if entry.is_dir(follow_symlinks=False): total += get_tree_size(entry.path) else: total += entry.stat(follow_symlinks=False).st_size return total </pre></div> </div> This also shows the use of the <code class="docutils literal notranslate">follow_symlinks</code> parameter to <code class="docutils literal notranslate">is_dir()</code> – in a recursive function like this, we probably don’t want to follow links. (To properly follow links in a recursive function like this we’d want special handling for the case where following a symlink leads to a recursive loop.) Note that <code class="docutils literal notranslate">get_tree_size()</code> will get a huge speed boost on Windows, because no extra stat call are needed, but on POSIX systems the size information is not returned by the directory iteration functions, so this function won’t gain anything there. <section id="notes-on-caching"> <h3><a class="toc-backref" href="#notes-on-caching" role="doc-backlink">Notes on caching</a></h3> The <code class="docutils literal notranslate">DirEntry</code> objects are relatively dumb – the <code class="docutils literal notranslate">name</code> and <code class="docutils literal notranslate">path</code> attributes are obviously always cached, and the <code class="docutils literal notranslate">is_X</code> and <code class="docutils literal notranslate">stat</code> methods cache their values (immediately on Windows via <code class="docutils literal notranslate">FindNextFile</code>, and on first use on POSIX systems via a <code class="docutils literal notranslate">stat</code> system call) and never refetch from the system. For this reason, <code class="docutils literal notranslate">DirEntry</code> objects are intended to be used and thrown away after iteration, not stored in long-lived data structured and the methods called again and again. If developers want “refresh” behaviour (for example, for watching a file’s size change), they can simply use <code class="docutils literal notranslate">pathlib.Path</code> objects, or call the regular <code class="docutils literal notranslate">os.stat()</code> or <code class="docutils literal notranslate">os.path.getsize()</code> functions which get fresh data from the operating system every call. </section> <section id="notes-on-exception-handling"> <h3><a class="toc-backref" href="#notes-on-exception-handling" role="doc-backlink">Notes on exception handling</a></h3> <code class="docutils literal notranslate">DirEntry.is_X()</code> and <code class="docutils literal notranslate">DirEntry.stat()</code> are explicitly methods rather than attributes or properties, to make it clear that they may not be cheap operations (although they often are), and they may do a system call. As a result, these methods may raise <code class="docutils literal notranslate">OSError</code>. For example, <code class="docutils literal notranslate">DirEntry.stat()</code> will always make a system call on POSIX-based systems, and the <code class="docutils literal notranslate">DirEntry.is_X()</code> methods will make a <code class="docutils literal notranslate">stat()</code> system call on such systems if <code class="docutils literal notranslate">readdir()</code> does not support <code class="docutils literal notranslate">d_type</code> or returns a <code class="docutils literal notranslate">d_type</code> with a value of <code class="docutils literal notranslate">DT_UNKNOWN</code>, which can occur under certain conditions or on certain file systems. Often this does not matter – for example, <code class="docutils literal notranslate">os.walk()</code> as defined in the standard library only catches errors around the <code class="docutils literal notranslate">listdir()</code> calls. Also, because the exception-raising behaviour of the <code class="docutils literal notranslate">DirEntry.is_X</code> methods matches that of <code class="docutils literal notranslate">pathlib</code> – which only raises <code class="docutils literal notranslate">OSError</code> in the case of permissions or other fatal errors, but returns False if the path doesn’t exist or is a broken symlink – it’s often not necessary to catch errors around the <code class="docutils literal notranslate">is_X()</code> calls. However, when a user requires fine-grained error handling, it may be desirable to catch <code class="docutils literal notranslate">OSError</code> around all method calls and handle as appropriate. For example, below is a version of the <code class="docutils literal notranslate">get_tree_size()</code> example shown above, but with fine-grained error handling added: <div class="highlight-default notranslate"><div class="highlight"><pre>def get_tree_size(path): """Return total size of files in path and subdirs. If is_dir() or stat() fails, print an error message to stderr and assume zero size (for example, file has been deleted). """ total = 0 for entry in os.scandir(path): try: is_dir = entry.is_dir(follow_symlinks=False) except OSError as error: print('Error calling is_dir():', error, file=sys.stderr) continue if is_dir: total += get_tree_size(entry.path) else: try: total += entry.stat(follow_symlinks=False).st_size except OSError as error: print('Error calling stat():', error, file=sys.stderr) return total </pre></div> </div> </section> </section> <section id="support"> <h2><a class="toc-backref" href="#support" role="doc-backlink">Support</a></h2> The scandir module on GitHub has been forked and used quite a bit (see “Use in the wild” in this PEP), but there’s also been a fair bit of direct support for a scandir-like function from core developers and others on the python-dev and python-ideas mailing lists. A sampling: <ul class="simple"> <li>python-dev: a good number of +1’s and very few negatives for scandir and <a class="pep reference internal" href="../pep-0471/" title="PEP 471 – os.scandir() function – a better and faster directory iterator">PEP 471</a> on <a class="reference external" href="https://mail.python.org/pipermail/python-dev/2014-June/135217.html">this June 2014 python-dev thread</a></li> <li>Alyssa Coghlan, a core Python developer: “I’ve had the local Red Hat release engineering team express their displeasure at having to stat every file in a network mounted directory tree for info that is present in the dirent structure, so a definite +1 to os.scandir from me, so long as it makes that info available.” [<a class="reference external" href="http://bugs.python.org/issue11406">source1</a>]</li> <li>Tim Golden, a core Python developer, supports scandir enough to have spent time refactoring and significantly improving scandir’s C extension module. [<a class="reference external" href="https://github.com/tjguk/scandir">source2</a>]</li> <li>Christian Heimes, a core Python developer: “+1 for something like yielddir()” [<a class="reference external" href="https://mail.python.org/pipermail/python-ideas/2012-November/017772.html">source3</a>] and “Indeed! I’d like to see the feature in 3.4 so I can remove my own hack from our code base.” [<a class="reference external" href="http://bugs.python.org/issue11406">source4</a>]</li> <li>Gregory P. Smith, a core Python developer: “As 3.4beta1 happens tonight, this isn’t going to make 3.4 so i’m bumping this to 3.5. I really like the proposed design outlined above.” [<a class="reference external" href="http://bugs.python.org/issue11406">source5</a>]</li> <li>Guido van Rossum on the possibility of adding scandir to Python 3.5 (as it was too late for 3.4): “The ship has likewise sailed for adding scandir() (whether to os or pathlib). By all means experiment and get it ready for consideration for 3.5, but I don’t want to add it to 3.4.” [<a class="reference external" href="https://mail.python.org/pipermail/python-dev/2013-November/130583.html">source6</a>]</li> </ul> Support for this PEP itself (meta-support?) was given by Alyssa (Nick) Coghlan on python-dev: “A PEP reviewing all this for 3.5 and proposing a specific os.scandir API would be a good thing.” [<a class="reference external" href="https://mail.python.org/pipermail/python-dev/2013-November/130588.html">source7</a>] </section> <section id="use-in-the-wild"> <h2><a class="toc-backref" href="#use-in-the-wild" role="doc-backlink">Use in the wild</a></h2> To date, the <code class="docutils literal notranslate">scandir</code> implementation is definitely useful, but has been clearly marked “beta”, so it’s uncertain how much use of it there is in the wild. Ben Hoyt has had several reports from people using it. For example: <ul class="simple"> <li>Chris F: “I am processing some pretty large directories and was half expecting to have to modify getdents. So thanks for saving me the effort.” [via personal email]</li> <li>bschollnick: “I wanted to let you know about this, since I am using Scandir as a building block for this code. Here’s a good example of scandir making a radical performance improvement over os.listdir.” [<a class="reference external" href="https://github.com/benhoyt/scandir/issues/19">source8</a>]</li> <li>Avram L: “I’m testing our scandir for a project I’m working on. Seems pretty solid, so first thing, just want to say nice work!” [via personal email]</li> <li>Matt Z: “I used scandir to dump the contents of a network dir in under 15 seconds. 13 root dirs, 60,000 files in the structure. This will replace some old VBA code embedded in a spreadsheet that was taking 15-20 minutes to do the exact same thing.” [via personal email]</li> </ul> Others have <a class="reference external" href="https://github.com/benhoyt/scandir/issues/12">requested a PyPI package</a> for it, which has been created. See <a class="reference external" href="https://pypi.python.org/pypi/scandir">PyPI package</a>. GitHub stats don’t mean too much, but scandir does have several watchers, issues, forks, etc. Here’s the run-down as of the stats as of July 7, 2014: <ul class="simple"> <li>Watchers: 17</li> <li>Stars: 57</li> <li>Forks: 20</li> <li>Issues: 4 open, 26 closed</li> </ul> Also, because this PEP will increase the speed of <code class="docutils literal notranslate">os.walk()</code> significantly, there are thousands of developers and scripts, and a lot of production code, that would benefit from it. For example, on GitHub, there are almost as many uses of <code class="docutils literal notranslate">os.walk</code> (194,000) as there are of <code class="docutils literal notranslate">os.mkdir</code> (230,000). </section> <section id="rejected-ideas"> <h2><a class="toc-backref" href="#rejected-ideas" role="doc-backlink">Rejected ideas</a></h2> <section id="naming"> <h3><a class="toc-backref" href="#naming" role="doc-backlink">Naming</a></h3> The only other real contender for this function’s name was <code class="docutils literal notranslate">iterdir()</code>. However, <code class="docutils literal notranslate">iterX()</code> functions in Python (mostly found in Python 2) tend to be simple iterator equivalents of their non-iterator counterparts. For example, <code class="docutils literal notranslate">dict.iterkeys()</code> is just an iterator version of <code class="docutils literal notranslate">dict.keys()</code>, but the objects returned are identical. In <code class="docutils literal notranslate">scandir()</code>’s case, however, the return values are quite different objects (<code class="docutils literal notranslate">DirEntry</code> objects vs filename strings), so this should probably be reflected by a difference in name – hence <code class="docutils literal notranslate">scandir()</code>. See some <a class="reference external" href="https://mail.python.org/pipermail/python-dev/2014-June/135228.html">relevant discussion on python-dev</a>. </section> <section id="wildcard-support"> <h3><a class="toc-backref" href="#wildcard-support" role="doc-backlink">Wildcard support</a></h3> <code class="docutils literal notranslate">FindFirstFile</code>/<code class="docutils literal notranslate">FindNextFile</code> on Windows support passing a “wildcard” like <code class="docutils literal notranslate">*.jpg</code>, so at first folks (this PEP’s author included) felt it would be a good idea to include a <code class="docutils literal notranslate">windows_wildcard</code> keyword argument to the <code class="docutils literal notranslate">scandir</code> function so users could pass this in. However, on further thought and discussion it was decided that this would be bad idea, unless it could be made cross-platform (a <code class="docutils literal notranslate">pattern</code> keyword argument or similar). This seems easy enough at first – just use the OS wildcard support on Windows, and something like <code class="docutils literal notranslate">fnmatch</code> or <code class="docutils literal notranslate">re</code> afterwards on POSIX-based systems. Unfortunately the exact Windows wildcard matching rules aren’t really documented anywhere by Microsoft, and they’re quite quirky (see this <a class="reference external" href="http://blogs.msdn.com/b/oldnewthing/archive/2007/12/17/6785519.aspx">blog post</a>), meaning it’s very problematic to emulate using <code class="docutils literal notranslate">fnmatch</code> or regexes. So the consensus was that Windows wildcard support was a bad idea. It would be possible to add at a later date if there’s a cross-platform way to achieve it, but not for the initial version. Read more on the <a class="reference external" href="https://mail.python.org/pipermail/python-ideas/2012-November/017770.html">this Nov 2012 python-ideas thread</a> and this <a class="reference external" href="https://mail.python.org/pipermail/python-dev/2014-June/135217.html">June 2014 python-dev thread on PEP 471</a>. </section> <section id="methods-not-following-symlinks-by-default"> <h3><a class="toc-backref" href="#methods-not-following-symlinks-by-default" role="doc-backlink">Methods not following symlinks by default</a></h3> There was much debate on python-dev (see messages in <a class="reference external" href="https://mail.python.org/pipermail/python-dev/2014-July/135485.html">this thread</a>) over whether the <code class="docutils literal notranslate">DirEntry</code> methods should follow symbolic links or not (when the <code class="docutils literal notranslate">is_X()</code> methods had no <code class="docutils literal notranslate">follow_symlinks</code> parameter). Initially they did not (see previous versions of this PEP and the scandir.py module), but Victor Stinner made a pretty compelling case on python-dev that following symlinks by default is a better idea, because: <ul class="simple"> <li>following links is usually what you want (in 92% of cases in the standard library, functions using <code class="docutils literal notranslate">os.listdir()</code> and <code class="docutils literal notranslate">os.path.isdir()</code> do follow symlinks)</li> <li>that’s the precedent set by the similar functions <code class="docutils literal notranslate">os.path.isdir()</code> and <code class="docutils literal notranslate">pathlib.Path.is_dir()</code>, so to do otherwise would be confusing</li> <li>with the non-link-following approach, if you wanted to follow links you’d have to say something like <code class="docutils literal notranslate">if (entry.is_symlink() and os.path.isdir(entry.path)) or entry.is_dir()</code>, which is clumsy</li> </ul> As a case in point that shows the non-symlink-following version is error prone, this PEP’s author had a bug caused by getting this exact test wrong in his initial implementation of <code class="docutils literal notranslate">scandir.walk()</code> in scandir.py (see <a class="reference external" href="https://github.com/benhoyt/scandir/issues/4">Issue #4 here</a>). In the end there was not total agreement that the methods should follow symlinks, but there was basic consensus among the most involved participants, and this PEP’s author believes that the above case is strong enough to warrant following symlinks by default. In addition, it’s straightforward to call the relevant methods with <code class="docutils literal notranslate">follow_symlinks=False</code> if the other behaviour is desired. </section> <section id="direntry-attributes-being-properties"> <h3><a class="toc-backref" href="#direntry-attributes-being-properties" role="doc-backlink">DirEntry attributes being properties</a></h3> In some ways it would be nicer for the <code class="docutils literal notranslate">DirEntry</code> <code class="docutils literal notranslate">is_X()</code> and <code class="docutils literal notranslate">stat()</code> to be properties instead of methods, to indicate they’re very cheap or free. However, this isn’t quite the case, as <code class="docutils literal notranslate">stat()</code> will require an OS call on POSIX-based systems but not on Windows. Even <code class="docutils literal notranslate">is_dir()</code> and friends may perform an OS call on POSIX-based systems if the <code class="docutils literal notranslate">dirent.d_type</code> value is <code class="docutils literal notranslate">DT_UNKNOWN</code> (on certain file systems). Also, people would expect the attribute access <code class="docutils literal notranslate">entry.is_dir</code> to only ever raise <code class="docutils literal notranslate">AttributeError</code>, not <code class="docutils literal notranslate">OSError</code> in the case it makes a system call under the covers. Calling code would have to have a <code class="docutils literal notranslate">try</code>/<code class="docutils literal notranslate">except</code> around what looks like a simple attribute access, and so it’s much better to make them methods. See <a class="reference external" href="https://mail.python.org/pipermail/python-dev/2013-May/126184.html">this May 2013 python-dev thread</a> where this PEP author makes this case and there’s agreement from a core developers. </section> <section id="direntry-fields-being-static-attribute-only-objects"> <h3><a class="toc-backref" href="#direntry-fields-being-static-attribute-only-objects" role="doc-backlink">DirEntry fields being “static” attribute-only objects</a></h3> In <a class="reference external" href="https://mail.python.org/pipermail/python-dev/2014-July/135303.html">this July 2014 python-dev message</a>, Paul Moore suggested a solution that was a “thin wrapper round the OS feature”, where the <code class="docutils literal notranslate">DirEntry</code> object had only static attributes: <code class="docutils literal notranslate">name</code>, <code class="docutils literal notranslate">path</code>, and <code class="docutils literal notranslate">is_X</code>, with the <code class="docutils literal notranslate">st_X</code> attributes only present on Windows. The idea was to use this simpler, lower-level function as a building block for higher-level functions. At first there was general agreement that simplifying in this way was a good thing. However, there were two problems with this approach. First, the assumption is the <code class="docutils literal notranslate">is_dir</code> and similar attributes are always present on POSIX, which isn’t the case (if <code class="docutils literal notranslate">d_type</code> is not present or is <code class="docutils literal notranslate">DT_UNKNOWN</code>). Second, it’s a much harder-to-use API in practice, as even the <code class="docutils literal notranslate">is_dir</code> attributes aren’t always present on POSIX, and would need to be tested with <code class="docutils literal notranslate">hasattr()</code> and then <code class="docutils literal notranslate">os.stat()</code> called if they weren’t present. See <a class="reference external" href="https://mail.python.org/pipermail/python-dev/2014-July/135312.html">this July 2014 python-dev response</a> from this PEP’s author detailing why this option is a non-ideal solution, and the subsequent reply from Paul Moore voicing agreement. </section> <section id="direntry-fields-being-static-with-an-ensure-lstat-option"> <h3><a class="toc-backref" href="#direntry-fields-being-static-with-an-ensure-lstat-option" role="doc-backlink">DirEntry fields being static with an ensure_lstat option</a></h3> Another seemingly simpler and attractive option was suggested by Alyssa Coghlan in this <a class="reference external" href="https://mail.python.org/pipermail/python-dev/2014-June/135261.html">June 2014 python-dev message</a>: make <code class="docutils literal notranslate">DirEntry.is_X</code> and <code class="docutils literal notranslate">DirEntry.lstat_result</code> properties, and populate <code class="docutils literal notranslate">DirEntry.lstat_result</code> at iteration time, but only if the new argument <code class="docutils literal notranslate">ensure_lstat=True</code> was specified on the <code class="docutils literal notranslate">scandir()</code> call. This does have the advantage over the above in that you can easily get the stat result from <code class="docutils literal notranslate">scandir()</code> if you need it. However, it has the serious disadvantage that fine-grained error handling is messy, because <code class="docutils literal notranslate">stat()</code> will be called (and hence potentially raise <code class="docutils literal notranslate">OSError</code>) during iteration, leading to a rather ugly, hand-made iteration loop: <div class="highlight-default notranslate"><div class="highlight"><pre>it = os.scandir(path) while True: try: entry = next(it) except OSError as error: handle_error(path, error) except StopIteration: break </pre></div> </div> Or it means that <code class="docutils literal notranslate">scandir()</code> would have to accept an <code class="docutils literal notranslate">onerror</code> argument – a function to call when <code class="docutils literal notranslate">stat()</code> errors occur during iteration. This seems to this PEP’s author neither as direct nor as Pythonic as <code class="docutils literal notranslate">try</code>/<code class="docutils literal notranslate">except</code> around a <code class="docutils literal notranslate">DirEntry.stat()</code> call. Another drawback is that <code class="docutils literal notranslate">os.scandir()</code> is written to make code faster. Always calling <code class="docutils literal notranslate">os.lstat()</code> on POSIX would not bring any speedup. In most cases, you don’t need the full <code class="docutils literal notranslate">stat_result</code> object – the <code class="docutils literal notranslate">is_X()</code> methods are enough and this information is already known. See <a class="reference external" href="https://mail.python.org/pipermail/python-dev/2014-July/135312.html">Ben Hoyt’s July 2014 reply</a> to the discussion summarizing this and detailing why he thinks the original <a class="pep reference internal" href="../pep-0471/" title="PEP 471 – os.scandir() function – a better and faster directory iterator">PEP 471</a> proposal is “the right one” after all. </section> <section id="return-values-being-name-stat-result-two-tuples"> <h3><a class="toc-backref" href="#return-values-being-name-stat-result-two-tuples" role="doc-backlink">Return values being (name, stat_result) two-tuples</a></h3> Initially this PEP’s author proposed this concept as a function called <code class="docutils literal notranslate">iterdir_stat()</code> which yielded two-tuples of (name, stat_result). This does have the advantage that there are no new types introduced. However, the <code class="docutils literal notranslate">stat_result</code> is only partially filled on POSIX-based systems (most fields set to <code class="docutils literal notranslate">None</code> and other quirks), so they’re not really <code class="docutils literal notranslate">stat_result</code> objects at all, and this would have to be thoroughly documented as different from <code class="docutils literal notranslate">os.stat()</code>. Also, Python has good support for proper objects with attributes and methods, which makes for a saner and simpler API than two-tuples. It also makes the <code class="docutils literal notranslate">DirEntry</code> objects more extensible and future-proof as operating systems add functionality and we want to include this in <code class="docutils literal notranslate">DirEntry</code>. See also some previous discussion: <ul class="simple"> <li><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2013-May/126148.html">May 2013 python-dev thread</a> where Alyssa Coghlan makes the original case for a <code class="docutils literal notranslate">DirEntry</code>-style object.</li> <li><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2014-June/135244.html">June 2014 python-dev thread</a> where Alyssa Coghlan makes (another) good case against the two-tuple approach.</li> </ul> </section> <section id="return-values-being-overloaded-stat-result-objects"> <h3><a class="toc-backref" href="#return-values-being-overloaded-stat-result-objects" role="doc-backlink">Return values being overloaded stat_result objects</a></h3> Another alternative discussed was making the return values to be overloaded <code class="docutils literal notranslate">stat_result</code> objects with <code class="docutils literal notranslate">name</code> and <code class="docutils literal notranslate">path</code> attributes. However, apart from this being a strange (and strained!) kind of overloading, this has the same problems mentioned above – most of the <code class="docutils literal notranslate">stat_result</code> information is not fetched by <code class="docutils literal notranslate">readdir()</code> on POSIX systems, only (part of) the <code class="docutils literal notranslate">st_mode</code> value. </section> <section id="return-values-being-pathlib-path-objects"> <h3><a class="toc-backref" href="#return-values-being-pathlib-path-objects" role="doc-backlink">Return values being pathlib.Path objects</a></h3> With Antoine Pitrou’s new standard library <code class="docutils literal notranslate">pathlib</code> module, it at first seems like a great idea for <code class="docutils literal notranslate">scandir()</code> to return instances of <code class="docutils literal notranslate">pathlib.Path</code>. However, <code class="docutils literal notranslate">pathlib.Path</code>’s <code class="docutils literal notranslate">is_X()</code> and <code class="docutils literal notranslate">stat()</code> functions are explicitly not cached, whereas <code class="docutils literal notranslate">scandir</code> has to cache them by design, because it’s (often) returning values from the original directory iteration system call. And if the <code class="docutils literal notranslate">pathlib.Path</code> instances returned by <code class="docutils literal notranslate">scandir</code> cached stat values, but the ordinary <code class="docutils literal notranslate">pathlib.Path</code> objects explicitly don’t, that would be more than a little confusing. Guido van Rossum explicitly rejected <code class="docutils literal notranslate">pathlib.Path</code> caching stat in the context of scandir <a class="reference external" href="https://mail.python.org/pipermail/python-dev/2013-November/130583.html">here</a>, making <code class="docutils literal notranslate">pathlib.Path</code> objects a bad choice for scandir return values. </section> </section> <section id="possible-improvements"> <h2><a class="toc-backref" href="#possible-improvements" role="doc-backlink">Possible improvements</a></h2> There are many possible improvements one could make to scandir, but here is a short list of some this PEP’s author has in mind: <ul class="simple"> <li>scandir could potentially be further sped up by calling <code class="docutils literal notranslate">readdir</code> / <code class="docutils literal notranslate">FindNextFile</code> say 50 times per <code class="docutils literal notranslate">Py_BEGIN_ALLOW_THREADS</code> block so that it stays in the C extension module for longer, and may be somewhat faster as a result. This approach hasn’t been tested, but was suggested by on Issue 11406 by Antoine Pitrou. [<a class="reference external" href="http://bugs.python.org/msg130125">source9</a>]</li> <li>scandir could use a free list to avoid the cost of memory allocation for each iteration – a short free list of 10 or maybe even 1 may help. Suggested by Victor Stinner on a <a class="reference external" href="https://mail.python.org/pipermail/python-dev/2014-June/135232.html">python-dev thread on June 27</a>.</li> </ul> </section> <section id="previous-discussion"> <h2><a class="toc-backref" href="#previous-discussion" role="doc-backlink">Previous discussion</a></h2> <ul class="simple"> <li><a class="reference external" href="https://mail.python.org/pipermail/python-ideas/2012-November/017770.html">Original November 2012 thread Ben Hoyt started on python-ideas</a> about speeding up <code class="docutils literal notranslate">os.walk()</code></li> <li>Python <a class="reference external" href="http://bugs.python.org/issue11406">Issue 11406</a>, which includes the original proposal for a scandir-like function</li> <li><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2013-May/126119.html">Further May 2013 thread Ben Hoyt started on python-dev</a> that refined the <code class="docutils literal notranslate">scandir()</code> API, including Alyssa Coghlan’s suggestion of scandir yielding <code class="docutils literal notranslate">DirEntry</code>-like objects</li> <li><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2013-November/130572.html">November 2013 thread Ben Hoyt started on python-dev</a> to discuss the interaction between scandir and the new <code class="docutils literal notranslate">pathlib</code> module</li> <li><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2014-June/135215.html">June 2014 thread Ben Hoyt started on python-dev</a> to discuss the first version of this PEP, with extensive discussion about the API</li> <li><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2014-July/135377.html">First July 2014 thread Ben Hoyt started on python-dev</a> to discuss his updates to <a class="pep reference internal" href="../pep-0471/" title="PEP 471 – os.scandir() function – a better and faster directory iterator">PEP 471</a></li> <li><a class="reference external" href="https://mail.python.org/pipermail/python-dev/2014-July/135485.html">Second July 2014 thread Ben Hoyt started on python-dev</a> to discuss the remaining decisions needed to finalize <a class="pep reference internal" href="../pep-0471/" title="PEP 471 – os.scandir() function – a better and faster directory iterator">PEP 471</a>, specifically whether the <code class="docutils literal notranslate">DirEntry</code> methods should follow symlinks by default</li> <li><a class="reference external" href="http://stackoverflow.com/questions/2485719/very-quickly-getting-total-size-of-folder">Question on StackOverflow</a> about why <code class="docutils literal notranslate">os.walk()</code> is slow and pointers on how to fix it (this inspired the author of this PEP early on)</li> <li><a class="reference external" href="https://github.com/benhoyt/betterwalk">BetterWalk</a>, this PEP’s author’s previous attempt at this, on which the scandir code is based</li> </ul> </section> <section id="copyright"> <h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2> This document has been placed in the public domain. </section> </section> <hr class="docutils" /> Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0471.rst">https://github.com/python/peps/blob/main/peps/pep-0471.rst</a> Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0471.rst">2023-10-11 12:05:51 GMT</a> </article> <nav id="pep-sidebar"> <h2>Contents</h2> <ul> <li><a class="reference internal" href="#abstract">Abstract</a></li> <li><a class="reference internal" href="#rationale">Rationale</a></li> <li><a class="reference internal" href="#implementation">Implementation</a></li> <li><a class="reference internal" href="#specifics-of-proposal">Specifics of proposal</a><ul> <li><a class="reference internal" href="#os-scandir">os.scandir()</a></li> <li><a class="reference internal" href="#os-walk">os.walk()</a></li> </ul> </li> <li><a class="reference internal" href="#examples">Examples</a><ul> <li><a class="reference internal" href="#notes-on-caching">Notes on caching</a></li> <li><a class="reference internal" href="#notes-on-exception-handling">Notes on exception handling</a></li> </ul> </li> <li><a class="reference internal" href="#support">Support</a></li> <li><a class="reference internal" href="#use-in-the-wild">Use in the wild</a></li> <li><a class="reference internal" href="#rejected-ideas">Rejected ideas</a><ul> <li><a class="reference internal" href="#naming">Naming</a></li> <li><a class="reference internal" href="#wildcard-support">Wildcard support</a></li> <li><a class="reference internal" href="#methods-not-following-symlinks-by-default">Methods not following symlinks by default</a></li> <li><a class="reference internal" href="#direntry-attributes-being-properties">DirEntry attributes being properties</a></li> <li><a class="reference internal" href="#direntry-fields-being-static-attribute-only-objects">DirEntry fields being “static” attribute-only objects</a></li> <li><a class="reference internal" href="#direntry-fields-being-static-with-an-ensure-lstat-option">DirEntry fields being static with an ensure_lstat option</a></li> <li><a class="reference internal" href="#return-values-being-name-stat-result-two-tuples">Return values being (name, stat_result) two-tuples</a></li> <li><a class="reference internal" href="#return-values-being-overloaded-stat-result-objects">Return values being overloaded stat_result objects</a></li> <li><a class="reference internal" href="#return-values-being-pathlib-path-objects">Return values being pathlib.Path objects</a></li> </ul> </li> <li><a class="reference internal" href="#possible-improvements">Possible improvements</a></li> <li><a class="reference internal" href="#previous-discussion">Previous discussion</a></li> <li><a class="reference internal" href="#copyright">Copyright</a></li> </ul> <a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0471.rst">Page Source (GitHub)</a> </nav> </section> <script src="../_static/colour_scheme.js"></script> <script src="../_static/wrap_tables.js"></script> <script src="../_static/sticky_banner.js"></script> </body> </html>