TensorRT Documentation — NVIDIA TensorRT Documentation

<!DOCTYPE html> <html lang="en" data-content_root="./" > <head> <meta charset="utf-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" /> <title>TensorRT Documentation — NVIDIA TensorRT Documentation</title> <script data-cfasync="false"> document.documentElement.dataset.mode = localStorage.getItem("mode") || ""; document.documentElement.dataset.theme = localStorage.getItem("theme") || ""; </script>  <link href="_static/styles/theme.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" /> <link href="_static/styles/bootstrap.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" /> <link href="_static/styles/pydata-sphinx-theme.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" /> <link href="_static/vendor/fontawesome/6.5.2/css/all.min.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" /> <link rel="preload" as="font" type="font/woff2" crossorigin href="_static/vendor/fontawesome/6.5.2/webfonts/fa-solid-900.woff2" /> <link rel="preload" as="font" type="font/woff2" crossorigin href="_static/vendor/fontawesome/6.5.2/webfonts/fa-brands-400.woff2" /> <link rel="preload" as="font" type="font/woff2" crossorigin href="_static/vendor/fontawesome/6.5.2/webfonts/fa-regular-400.woff2" /> <link rel="stylesheet" type="text/css" href="_static/pygments.css?v=a746c00c" /> <link rel="stylesheet" type="text/css" href="_static/styles/nvidia-sphinx-theme.css?v=c5edffae" /> <link rel="stylesheet" type="text/css" href="_static/custom.css?v=7abaf8bc" /> <link rel="stylesheet" type="text/css" href="_static/copybutton.css?v=76b2166b" /> <link rel="stylesheet" type="text/css" href="_static/sphinx-design.min.css?v=95c83b7e" />  <link rel="preload" as="script" href="_static/scripts/bootstrap.js?digest=dfe6caa3a7d634c4db9b" /> <link rel="preload" as="script" href="_static/scripts/pydata-sphinx-theme.js?digest=dfe6caa3a7d634c4db9b" /> <script src="_static/vendor/fontawesome/6.5.2/js/all.min.js?digest=dfe6caa3a7d634c4db9b"></script> <script src="_static/documentation_options.js?v=b8c444a7"></script> <script src="_static/doctools.js?v=9a2dae69"></script> <script src="_static/sphinx_highlight.js?v=dc90522c"></script> <script src="_static/external-links.js?v=e8b80c26"></script> <script src="_static/clipboard.min.js?v=a7894cd8"></script> <script src="_static/copybutton.js?v=65e89d2a"></script> <script src="_static/design-tabs.js?v=f930bc37"></script> <script async="async" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script> <script>DOCUMENTATION_OPTIONS.pagename = 'index';</script> <script> DOCUMENTATION_OPTIONS.theme_version = '0.15.4'; DOCUMENTATION_OPTIONS.theme_switcher_json_url = '../versions1.json'; DOCUMENTATION_OPTIONS.theme_switcher_version_match = '10.8.0'; DOCUMENTATION_OPTIONS.show_version_warning_banner = true; </script> <script src="_static/version-patch.js?v=c24f8c5d"></script> <link rel="icon" href="_static/favicon.png"/> <link rel="index" title="Index" href="genindex.html" /> <link rel="search" title="Search" href="search.html" /> <link rel="next" title="Release Notes" href="getting-started/release-notes.html" /> <meta name="viewport" content="width=device-width, initial-scale=1"/> <meta name="docsearch:language" content="en"/> <meta name="docbuild:last-update" content="Feb 04, 2025"/> <script src="https://assets.adobedtm.com/5d4962a43b79/c1061d2c5e7b/launch-191c2462b890.min.js" ></script> </head> <body data-bs-spy="scroll" data-bs-target=".bd-toc-nav" data-offset="180" data-bs-root-margin="0px 0px -60%" data-default-mode=""> <div id="pst-skip-link" class="skip-link d-print-none"><a href="#main-content">Skip to main content</a></div> <div id="pst-scroll-pixel-helper"></div> <button type="button" class="btn rounded-pill" id="pst-back-to-top"> <i class="fa-solid fa-arrow-up"></i>Back to top</button> <input type="checkbox" class="sidebar-toggle" id="pst-primary-sidebar-checkbox"/> <label class="overlay overlay-primary" for="pst-primary-sidebar-checkbox"></label> <input type="checkbox" class="sidebar-toggle" id="pst-secondary-sidebar-checkbox"/> <label class="overlay overlay-secondary" for="pst-secondary-sidebar-checkbox"></label> <div class="search-button__wrapper"> <div class="search-button__overlay"></div> <div class="search-button__search-container"> <form class="bd-search d-flex align-items-center" action="search.html" method="get"> <i class="fa-solid fa-magnifying-glass"></i> <input type="search" class="form-control" name="q" id="search-input" placeholder="Search the docs ..." aria-label="Search the docs ..." autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/> <span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd>K</kbd></span> </form></div> </div> <div class="pst-async-banner-revealer d-none"> <aside id="bd-header-version-warning" class="d-none d-print-none" aria-label="Version warning"></aside> </div> <header class="bd-header navbar navbar-expand-lg bd-navbar d-print-none"> <div class="bd-header__inner bd-page-width"> <button class="pst-navbar-icon sidebar-toggle primary-toggle" aria-label="Site navigation"> <span class="fa-solid fa-bars"></span> </button> <div class="col-lg-3 navbar-header-items__start"> <div class="navbar-item"> <a class="navbar-brand logo" href="#"> <img src="_static/nvidia-logo-horiz-rgb-blk-for-screen.svg" class="logo__image only-light" alt="NVIDIA TensorRT Documentation - Home"/> <script>document.write(`<img src="_static/nvidia-logo-horiz-rgb-wht-for-screen.svg" class="logo__image only-dark" alt="NVIDIA TensorRT Documentation - Home"/>`);</script> <p class="title logo__title">NVIDIA TensorRT Documentation</p> </a></div> </div> <div class="col-lg-9 navbar-header-items"> <div class="me-auto navbar-header-items__center"> <div class="navbar-item"> <div> <ul class="bd-navbar-elements navbar-nav"> <li class="nav-item pst-header-nav-item"> <a class="nav-link nav-external" href="https://docs.nvidia.com/tensorrt/">Documentation Home</a> </li> </ul> </div> </div> </div> <div class="navbar-header-items__end"> <div class="navbar-item navbar-persistent--container"> <script> document.write(` <button class="btn search-button-field search-button__button" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip"> <i class="fa-solid fa-magnifying-glass"></i> <span class="search-button__default-text">Search</span> <span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd class="kbd-shortcut__modifier">K</kbd></span> </button> `); </script> </div> <div class="navbar-item"> <script> document.write(` <button class="btn btn-sm nav-link pst-navbar-icon theme-switch-button" title="light/dark" aria-label="light/dark" data-bs-placement="bottom" data-bs-toggle="tooltip"> <i class="theme-switch fa-solid fa-sun fa-lg" data-mode="light"></i> <i class="theme-switch fa-solid fa-moon fa-lg" data-mode="dark"></i> <i class="theme-switch fa-solid fa-circle-half-stroke fa-lg" data-mode="auto"></i> </button> `); </script></div> </div> </div> <div class="navbar-persistent--mobile"> <script> document.write(` <button class="btn search-button-field search-button__button" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip"> <i class="fa-solid fa-magnifying-glass"></i> <span class="search-button__default-text">Search</span> <span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd class="kbd-shortcut__modifier">K</kbd></span> </button> `); </script> </div> </div> </header> <div class="bd-container"> <div class="bd-container__inner bd-page-width"> <div class="bd-sidebar-primary bd-sidebar"> <a class="navbar-brand logo" href="#"> <img src="_static/nvidia-logo-horiz-rgb-blk-for-screen.svg" class="logo__image only-light" alt="NVIDIA TensorRT Documentation - Home"/> <script>document.write(`<img src="_static/nvidia-logo-horiz-rgb-wht-for-screen.svg" class="logo__image only-dark" alt="NVIDIA TensorRT Documentation - Home"/>`);</script> <p class="title logo__title">NVIDIA TensorRT Documentation</p> </a> <div class="sidebar-header-items sidebar-primary__section"> <div class="sidebar-header-items__center"> <div class="navbar-item"> <div> <ul class="bd-navbar-elements navbar-nav"> <li class="nav-item pst-header-nav-item"> <a class="nav-link nav-external" href="https://docs.nvidia.com/tensorrt/">Documentation Home</a> </li> </ul> </div> </div> </div> <div class="sidebar-header-items__end"> <div class="navbar-item"> <script> document.write(` <button class="btn btn-sm nav-link pst-navbar-icon theme-switch-button" title="light/dark" aria-label="light/dark" data-bs-placement="bottom" data-bs-toggle="tooltip"> <i class="theme-switch fa-solid fa-sun fa-lg" data-mode="light"></i> <i class="theme-switch fa-solid fa-moon fa-lg" data-mode="dark"></i> <i class="theme-switch fa-solid fa-circle-half-stroke fa-lg" data-mode="auto"></i> </button> `); </script></div> </div> </div> <div class="sidebar-primary-items__start sidebar-primary__section"> <div class="sidebar-primary-item"> <nav class="bd-docs-nav bd-links" aria-label="Table of Contents"> <p class="bd-links__title" role="heading" aria-level="1">Table of Contents</p> <div class="bd-toc-item navbar-nav"><p aria-level="2" class="caption" role="heading"><span class="caption-text">Getting Started</span></p> <ul class="nav bd-sidenav"> <li class="toctree-l1"><a class="reference internal" href="getting-started/release-notes.html">Release Notes</a></li> <li class="toctree-l1"><a class="reference internal" href="getting-started/quick-start-guide.html">Quick Start Guide</a></li> <li class="toctree-l1"><a class="reference internal" href="getting-started/support-matrix.html">Support Matrix</a></li> </ul> <p aria-level="2" class="caption" role="heading"><span class="caption-text">Installing TensorRT</span></p> <ul class="nav bd-sidenav"> <li class="toctree-l1"><a class="reference internal" href="installing-tensorrt/overview.html">Overview</a></li> <li class="toctree-l1"><a class="reference internal" href="installing-tensorrt/prerequisites.html">Prerequisites</a></li> <li class="toctree-l1"><a class="reference internal" href="installing-tensorrt/installing.html">Installing TensorRT</a></li> <li class="toctree-l1"><a class="reference internal" href="installing-tensorrt/upgrading.html">Upgrading TensorRT</a></li> <li class="toctree-l1"><a class="reference internal" href="installing-tensorrt/uninstalling.html">Uninstalling TensorRT</a></li> </ul> <p aria-level="2" class="caption" role="heading"><span class="caption-text">Architecture</span></p> <ul class="nav bd-sidenav"> <li class="toctree-l1"><a class="reference internal" href="architecture/architecture-overview.html">Overview</a></li> <li class="toctree-l1"><a class="reference internal" href="architecture/capabilities.html">TensorRT鈥檚 Capabilities</a></li> <li class="toctree-l1"><a class="reference internal" href="architecture/how-trt-works.html">How TensorRT Works</a></li> </ul> <p aria-level="2" class="caption" role="heading"><span class="caption-text">Inference Library</span></p> <ul class="nav bd-sidenav"> <li class="toctree-l1"><a class="reference internal" href="inference-library/c-api-docs.html">C++ API Documentation</a></li> <li class="toctree-l1"><a class="reference internal" href="inference-library/python-api-docs.html">Python API Documentation</a></li> <li class="toctree-l1"><a class="reference internal" href="inference-library/sample-support-guide.html">Sample Support Guide</a></li> <li class="toctree-l1"><a class="reference internal" href="inference-library/advanced.html">Advanced Topics</a></li> <li class="toctree-l1"><a class="reference internal" href="inference-library/work-quantized-types.html">Working with Quantized Types</a></li> <li class="toctree-l1"><a class="reference internal" href="inference-library/accuracy-considerations.html">Accuracy Considerations</a></li> <li class="toctree-l1"><a class="reference internal" href="inference-library/work-dynamic-shapes.html">Working with Dynamic Shapes</a></li> <li class="toctree-l1"><a class="reference internal" href="inference-library/extending-custom-layers.html">Extending TensorRT with Custom Layers</a></li> <li class="toctree-l1"><a class="reference internal" href="inference-library/work-with-loops.html">Working with Loops</a></li> <li class="toctree-l1"><a class="reference internal" href="inference-library/work-with-conditionals.html">Working with Conditionals</a></li> <li class="toctree-l1"><a class="reference internal" href="inference-library/work-with-dla.html">Working with DLA</a></li> </ul> <p aria-level="2" class="caption" role="heading"><span class="caption-text">Performance</span></p> <ul class="nav bd-sidenav"> <li class="toctree-l1"><a class="reference internal" href="performance/best-practices.html">Best Practices</a></li> </ul> <p aria-level="2" class="caption" role="heading"><span class="caption-text">API</span></p> <ul class="nav bd-sidenav"> <li class="toctree-l1"><a class="reference internal" href="api/c-api.html">C++ API</a></li> <li class="toctree-l1"><a class="reference internal" href="api/python-api.html">Python API</a></li> <li class="toctree-l1"><a class="reference internal" href="api/migration-guide.html">API Migration Guide</a></li> <li class="toctree-l1"><a class="reference internal" href="api/onnx-graphsurgeon-api.html">ONNX GraphSurgeon API</a></li> <li class="toctree-l1"><a class="reference internal" href="api/polygraphy-api.html">Polygraphy API</a></li> </ul> <p aria-level="2" class="caption" role="heading"><span class="caption-text">Reference</span></p> <ul class="nav bd-sidenav"> <li class="toctree-l1"><a class="reference internal" href="reference/troubleshooting.html">Troubleshooting</a></li> <li class="toctree-l1"><a class="reference internal" href="reference/data-format-desc.html">Data Format Descriptions</a></li> <li class="toctree-l1"><a class="reference internal" href="reference/command-line-programs.html">Command-Line Programs</a></li> <li class="toctree-l1"><a class="reference internal" href="reference/operators.html">Operators Documentation</a></li> <li class="toctree-l1"><a class="reference internal" href="reference/additional-resources.html">Additional Resources</a></li> <li class="toctree-l1"><a class="reference internal" href="reference/archives.html">Documentation Archvies</a></li> <li class="toctree-l1"><a class="reference internal" href="reference/sla.html">LICENSE AGREEMENT</a></li> <li class="toctree-l1"><a class="reference internal" href="reference/glossary.html">Glossary</a></li> </ul> </div> </nav></div> </div> <div class="sidebar-primary-items__end sidebar-primary__section"> <div class="sidebar-primary-item"> <script> document.write(` <div class="version-switcher__container dropdown"> <button id="pst-version-switcher-button-2" type="button" class="version-switcher__button btn btn-sm dropdown-toggle" data-bs-toggle="dropdown" aria-haspopup="listbox" aria-controls="pst-version-switcher-list-2" aria-label="Version switcher list" > Choose version  <span class="caret"></span> </button> <div id="pst-version-switcher-list-2" class="version-switcher__menu dropdown-menu list-group-flush py-0" role="listbox" aria-labelledby="pst-version-switcher-button-2">  </div> </div> `); </script></div> </div> <div id="rtd-footer-container"></div> </div> <main id="main-content" class="bd-main" role="main"> <div class="bd-content"> <div class="bd-article-container"> <div class="bd-header-article d-print-none"></div> <div id="searchbox"></div> <article class="bd-article"> <section id="tensorrt-documentation"> <h1>TensorRT Documentation<a class="headerlink" href="#tensorrt-documentation" title="Link to this heading">#</a></h1> <p>NVIDIA TensorRT is an SDK that facilitates high-performance machine learning inference. It complements training frameworks such as TensorFlow, PyTorch, and MXNet. It focuses on running an already-trained network quickly and efficiently on NVIDIA hardware.</p> <div class="admonition attention"> <p class="admonition-title">Attention</p> <p>Ensure you refer to the <a class="reference internal" href="getting-started/release-notes.html#release-notes"><span class="std std-ref">Release Notes</span></a>, which describes the newest features, software enhancements and improvements, and known issues for the TensorRT release product package.</p> </div> <ul> <li><p>The <a class="reference internal" href="getting-started/quick-start-guide.html#quick-start-guide"><span class="std std-ref">Quick Start Guide</span></a> is a starting point for developers who want to try out the TensorRT SDK; specifically, it demonstrates how to quickly construct an application to run inference on a TensorRT engine.</p></li> <li><p>The <a class="reference internal" href="getting-started/support-matrix.html#support-matrix"><span class="std std-ref">Support Matrix</span></a> provides an overview of the supported platforms, features, and hardware capabilities of the TensorRT APIs, parsers, and layers.</p></li> <li><p>The <a class="reference internal" href="installing-tensorrt/overview.html#install-guide"><span class="std std-ref">Installing TensorRT</span></a> section provides the installation requirements, a list of what is included in the TensorRT package, and step-by-step instructions for installing TensorRT.</p></li> <li><p>The <a class="reference internal" href="architecture/architecture-overview.html#architecture-overview"><span class="std std-ref">Architecture</span></a> section demonstrates how to use the C++ and Python APIs to implement the most common deep learning layers. It shows how to take an existing model built with a deep learning framework and build a TensorRT engine using the provided parsers.</p></li> <li><p>The <em>Inference Library</em> section demonstrates how to use the C++ and Python APIs for implementing the most common deep learning layers. It shows how you can take an existing model built with a deep learning framework and build a TensorRT engine using the provided parsers.</p> <blockquote> <div><ul class="simple"> <li><p>The <a class="reference internal" href="inference-library/sample-support-guide.html#sample-support-guide"><span class="std std-ref">Sample Support Guide</span></a> provides an overview of all the supported TensorRT samples on GitHub and the product package. The TensorRT samples specifically help in recommenders, machine comprehension, character recognition, image classification, and object detection.</p></li> </ul> </div></blockquote> </li> <li><p>The <a class="reference internal" href="performance/best-practices.html#best-practices"><span class="std std-ref">Performance</span></a> section introduces how to use <code class="docutils literal notranslate"><span class="pre">trtexec</span></code>, a command-line tool designed for TensorRT performance benchmarking, to get the inference performance measurements of your deep learning models.</p></li> <li><p>The <em>API</em> section enables developers in C++ and Python based development environments and those looking to experiment with TensorRT to easily parse models (for example, from ONNX) and generate and run PLAN files.</p> <blockquote> <div><ul class="simple"> <li><p>The <a class="reference internal" href="api/migration-guide.html#migration-guide"><span class="std std-ref">API Migration Guide</span></a> highlights the TensorRT API modifications. If you are unfamiliar with these changes, refer to our sample code for clarification.</p></li> </ul> </div></blockquote> </li> <li><p>The <em>Reference</em> section will help answer commonly asked questions regarding typical use cases as well as provide additiona resources for assistance.</p></li> </ul> <div class="toctree-wrapper compound" id="getting-started"> </div> <div class="toctree-wrapper compound" id="installing-tensorrt"> </div> <div class="toctree-wrapper compound" id="architecture"> </div> <div class="toctree-wrapper compound" id="inference-library"> </div> <div class="toctree-wrapper compound" id="performance"> </div> <div class="toctree-wrapper compound" id="api"> </div> <div class="toctree-wrapper compound" id="reference"> </div> </section> </article> <footer class="prev-next-footer d-print-none"> <div class="prev-next-area"> <a class="right-next" href="getting-started/release-notes.html" title="next page"> <div class="prev-next-info"> <p class="prev-next-subtitle">next</p> <p class="prev-next-title">Release Notes</p> </div> <i class="fa-solid fa-angle-right"></i> </a> </div> </footer> </div> </div> <footer class="bd-footer-content"> </footer> </main> </div> </div>  <script src="_static/scripts/bootstrap.js?digest=dfe6caa3a7d634c4db9b"></script> <script src="_static/scripts/pydata-sphinx-theme.js?digest=dfe6caa3a7d634c4db9b"></script> <footer class="bd-footer"> <div class="bd-footer__inner bd-page-width"> <div class="footer-items__start"> <div class="footer-item"> <a class="footer-brand logo" href="https://www.nvidia.com"> <img src="_static/nvidia-logo-horiz-rgb-1c-blk-for-screen.svg" class="logo__image only-light" alt="NVIDIA"/> <img src="_static/nvidia-logo-horiz-rgb-1c-wht-for-screen.svg" class="logo__image only-dark" alt="NVIDIA"/> </a></div> <div class="footer-item"> <div class="footer-links"> <a class="external" href="https://www.nvidia.com/en-us/about-nvidia/privacy-policy/">Privacy Policy</a> | <a class="external" href="https://www.nvidia.com/en-us/about-nvidia/privacy-center/">Manage My Privacy</a> | <a class="external" href="https://www.nvidia.com/en-us/preferences/start/">Do Not Sell or Share My Data</a> | <a class="external" href="https://www.nvidia.com/en-us/about-nvidia/terms-of-service/">Terms of Service</a> | <a class="external" href="https://www.nvidia.com/en-us/about-nvidia/accessibility/">Accessibility</a> | <a class="external" href="https://www.nvidia.com/en-us/about-nvidia/company-policies/">Corporate Policies</a> | <a class="external" href="https://www.nvidia.com/en-us/product-security/">Product Security</a> | <a class="external" href="https://www.nvidia.com/en-us/contact/">Contact</a> </div> </div> <div class="footer-item"> <p class="copyright"> Copyright 漏 2021-2025, NVIDIA Corporation. <br/> </p> </div> <div class="footer-item"><p class="last-updated"> Last updated on Feb 04, 2025. <br/> </p></div> <div class="footer-item"> <div class="extra_footer"> <script type="text/javascript">if (typeof _satellite !== "undefined") {_satellite.pageBottom();}</script> </div></div> </div> </div> </footer> </body> </html>

CINXE.COM

TensorRT Documentation — NVIDIA TensorRT Documentation