Apex (A PyTorch Extension) — Apex 0.1.0 documentation

<!DOCTYPE html>   <html class="no-js" lang="en" >  <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <title>Apex (A PyTorch Extension) — Apex 0.1.0 documentation</title> <script type="text/javascript" src="_static/js/modernizr.min.js"></script> <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script> <script type="text/javascript" src="_static/jquery.js"></script> <script type="text/javascript" src="_static/underscore.js"></script> <script type="text/javascript" src="_static/doctools.js"></script> <script type="text/javascript" src="_static/language_data.js"></script> <script async="async" type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/latest.js?config=TeX-AMS-MML_HTMLorMML"></script> <script type="text/javascript" src="_static/js/theme.js"></script> <link rel="stylesheet" href="_static/css/theme.css" type="text/css" /> <link rel="stylesheet" href="_static/pygments.css" type="text/css" /> <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Lato" type="text/css" /> <link rel="stylesheet" href="_static/css/pytorch_theme.css" type="text/css" /> <link rel="index" title="Index" href="genindex.html" /> <link rel="search" title="Search" href="search.html" /> <link rel="next" title="apex.amp" href="amp.html" /> </head> <body class="wy-body-for-nav"> <div class="wy-grid-for-nav"> <nav data-toggle="wy-nav-shift" class="wy-nav-side"> <div class="wy-side-scroll"> <div class="wy-side-nav-search" > <a href="#" class="icon icon-home"> Apex </a> <div class="version"> 0.1 </div> <div role="search"> <form id="rtd-search-form" class="wy-form" action="search.html" method="get"> <input type="text" name="q" placeholder="Search docs" /> <input type="hidden" name="check_keywords" value="yes" /> <input type="hidden" name="area" value="default" /> </form> </div> <style> /* Sidebar header (and topbar for mobile) */ .wy-side-nav-search, .wy-nav-top { background: #76b900; } .wy-side-nav-search a:link, .wy-nav-top a:link { color: #fff; } .wy-side-nav-search a:visited, .wy-nav-top a:visited { color: #fff; } .wy-side-nav-search a:hover, .wy-nav-top a:hover { color: #fff; } .wy-menu-vertical a:link, .wy-menu-vertical a:visited { color: #d9d9d9 } .wy-menu-vertical a:active { background-color: #76b900 } .wy-side-nav-search>div.version { color: rgba(0, 0, 0, 0.3) } </style> </div> <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> <p class="caption"><span class="caption-text">AMP: Automatic Mixed Precision</span></p> <ul> <li class="toctree-l1"><a class="reference internal" href="amp.html">apex.amp</a><ul> <li class="toctree-l2"><a class="reference internal" href="amp.html#opt-levels-and-properties"><code class="docutils literal notranslate"><span class="pre">opt_level</span></code>s and Properties</a><ul> <li class="toctree-l3"><a class="reference internal" href="amp.html#properties">Properties</a></li> <li class="toctree-l3"><a class="reference internal" href="amp.html#opt-levels"><code class="docutils literal notranslate"><span class="pre">opt_level</span></code>s</a><ul> <li class="toctree-l4"><a class="reference internal" href="amp.html#o0-fp32-training"><code class="docutils literal notranslate"><span class="pre">O0</span></code>: FP32 training</a></li> <li class="toctree-l4"><a class="reference internal" href="amp.html#o1-mixed-precision-recommended-for-typical-use"><code class="docutils literal notranslate"><span class="pre">O1</span></code>: Mixed Precision (recommended for typical use)</a></li> <li class="toctree-l4"><a class="reference internal" href="amp.html#o2-almost-fp16-mixed-precision"><code class="docutils literal notranslate"><span class="pre">O2</span></code>: “Almost FP16” Mixed Precision</a></li> <li class="toctree-l4"><a class="reference internal" href="amp.html#o3-fp16-training"><code class="docutils literal notranslate"><span class="pre">O3</span></code>: FP16 training</a></li> </ul> </li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="amp.html#module-apex.amp">Unified API</a></li> <li class="toctree-l2"><a class="reference internal" href="amp.html#checkpointing">Checkpointing</a></li> <li class="toctree-l2"><a class="reference internal" href="amp.html#advanced-use-cases">Advanced use cases</a><ul> <li class="toctree-l3"><a class="reference internal" href="advanced.html">Advanced Amp Usage</a><ul> <li class="toctree-l4"><a class="reference internal" href="advanced.html#gans">GANs</a></li> <li class="toctree-l4"><a class="reference internal" href="advanced.html#gradient-clipping">Gradient clipping</a></li> <li class="toctree-l4"><a class="reference internal" href="advanced.html#custom-user-defined-autograd-functions">Custom/user-defined autograd functions</a></li> <li class="toctree-l4"><a class="reference internal" href="advanced.html#forcing-particular-layers-functions-to-a-desired-type">Forcing particular layers/functions to a desired type</a></li> <li class="toctree-l4"><a class="reference internal" href="advanced.html#multiple-models-optimizers-losses">Multiple models/optimizers/losses</a></li> <li class="toctree-l4"><a class="reference internal" href="advanced.html#gradient-accumulation-across-iterations">Gradient accumulation across iterations</a></li> <li class="toctree-l4"><a class="reference internal" href="advanced.html#custom-data-batch-types">Custom data batch types</a></li> </ul> </li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="amp.html#transition-guide-for-old-api-users">Transition guide for old API users</a><ul> <li class="toctree-l3"><a class="reference internal" href="amp.html#for-users-of-the-old-amp-api">For users of the old “Amp” API</a></li> <li class="toctree-l3"><a class="reference internal" href="amp.html#for-users-of-the-old-fp16-optimizer">For users of the old FP16_Optimizer</a></li> </ul> </li> </ul> </li> </ul> <p class="caption"><span class="caption-text">Distributed Training</span></p> <ul> <li class="toctree-l1"><a class="reference internal" href="parallel.html">apex.parallel</a><ul> <li class="toctree-l2"><a class="reference internal" href="parallel.html#utility-functions">Utility functions</a></li> </ul> </li> </ul> <p class="caption"><span class="caption-text">Fused Optimizers</span></p> <ul> <li class="toctree-l1"><a class="reference internal" href="optimizers.html">apex.optimizers</a></li> </ul> <p class="caption"><span class="caption-text">Fused Layer Norm</span></p> <ul> <li class="toctree-l1"><a class="reference internal" href="layernorm.html">apex.normalization.fused_layer_norm</a></li> </ul> </div> </div> </nav> <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> <nav class="wy-nav-top" aria-label="top navigation"> <i data-toggle="wy-nav-top" class="fa fa-bars"></i> <a href="#">Apex</a> </nav> <div class="wy-nav-content"> <div class="rst-content"> <div role="navigation" aria-label="breadcrumbs navigation"> <ul class="wy-breadcrumbs"> <li><a href="#">Docs</a> »</li> <li>Apex (A PyTorch Extension)</li> <li class="wy-breadcrumbs-aside">  <a href="https://github.com/nvidia/apex" class="fa fa-github"> Edit on GitHub</a> </li> </ul> <hr/> </div> <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> <div itemprop="articleBody"> <div class="section" id="apex-a-pytorch-extension"> <h1>Apex (A PyTorch Extension)<a class="headerlink" href="#apex-a-pytorch-extension" title="Permalink to this headline">¶</a></h1> <p>This site contains the API documentation for Apex (<a class="reference external" href="https://github.com/nvidia/apex">https://github.com/nvidia/apex</a>), a Pytorch extension with NVIDIA-maintained utilities to streamline mixed precision and distributed training. Some of the code here will be included in upstream Pytorch eventually. The intention of Apex is to make up-to-date utilities available to users as quickly as possible.</p> <p>Installation instructions can be found here: <a class="reference external" href="https://github.com/NVIDIA/apex#quick-start">https://github.com/NVIDIA/apex#quick-start</a>.</p> <p>Some other useful material, including GTC 2019 and Pytorch DevCon 2019 Slides, can be found here: <a class="reference external" href="https://github.com/mcarilli/mixed_precision_references">https://github.com/mcarilli/mixed_precision_references</a>.</p> <div class="toctree-wrapper compound"> <p class="caption"><span class="caption-text">AMP: Automatic Mixed Precision</span></p> <ul> <li class="toctree-l1"><a class="reference internal" href="amp.html">apex.amp</a></li> </ul> </div> <div class="toctree-wrapper compound"> <p class="caption"><span class="caption-text">Distributed Training</span></p> <ul> <li class="toctree-l1"><a class="reference internal" href="parallel.html">apex.parallel</a></li> </ul> </div> <div class="toctree-wrapper compound"> <p class="caption"><span class="caption-text">Fused Optimizers</span></p> <ul> <li class="toctree-l1"><a class="reference internal" href="optimizers.html">apex.optimizers</a></li> </ul> </div> <div class="toctree-wrapper compound"> <p class="caption"><span class="caption-text">Fused Layer Norm</span></p> <ul> <li class="toctree-l1"><a class="reference internal" href="layernorm.html">apex.normalization.fused_layer_norm</a></li> </ul> </div> </div> <div class="section" id="indices-and-tables"> <h1>Indices and tables<a class="headerlink" href="#indices-and-tables" title="Permalink to this headline">¶</a></h1> <ul class="simple"> <li><p><a class="reference internal" href="genindex.html"><span class="std std-ref">Index</span></a></p></li> <li><p><a class="reference internal" href="py-modindex.html"><span class="std std-ref">Module Index</span></a></p></li> </ul> </div> </div> </div> <footer> <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> <a href="amp.html" class="btn btn-neutral float-right" title="apex.amp" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a> </div> <hr/> <div role="contentinfo"> <p> © Copyright 2018 </p> </div> Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. </footer> </div> </div> </section> </div> <script type="text/javascript"> jQuery(function () { SphinxRtdTheme.Navigation.enable(true); }); </script> <style> a:link, a:visited { color: #76b900; } a:hover { color: #8c0; } .rst-content dl:not(.docutils) dt { background: rgba(118, 185, 0, 0.1); color: rgba(59,93,0,1); border-top: solid 3px rgba(59,93,0,1); } </style> </body> </html>

CINXE.COM

Apex (A PyTorch Extension) — Apex 0.1.0 documentation