Running LLMs with TensorRT-LLM on Nvidia Jetson AGX Orin Dev Kit - Jetson Projects

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <title>Running LLMs with TensorRT-LLM on Nvidia Jetson AGX Orin Dev Kit - Jetson Projects - NVIDIA Developer Forums</title> <meta name="description" content="Greetings to all, Below is the link to my latest post on deploying LLMs using the TensorRT-LLM on the Nvidia Jetson AGX Orin Developer Kit. Thanks!"> <meta name="generator" content="Discourse 3.4.0.beta3-dev - https://github.com/discourse/discourse version d71016522e8d9bb21c20312388271f8f0dd53069"> <link rel="icon" type="image/png" href="https://global.discourse-cdn.com/nvidia/optimized/2X/0/0372ccc95874f71d7fbff64bbbff6f8c69dd850b_2_32x32.png"> <link rel="apple-touch-icon" type="image/png" href="https://global.discourse-cdn.com/nvidia/optimized/2X/8/819b2855e1f1f3249e77dc713405cf77d1eda57c_2_180x180.png"> <meta name="theme-color" media="all" content="#000000"> <meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0, user-scalable=yes, viewport-fit=cover"> <link rel="canonical" href="https://forums.developer.nvidia.com/t/running-llms-with-tensorrt-llm-on-nvidia-jetson-agx-orin-dev-kit/314488" /> <link rel="search" type="application/opensearchdescription+xml" href="https://forums.developer.nvidia.com/opensearch.xml" title="NVIDIA Developer Forums Search"> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/color_definitions_nvidia_4_13_b0e6a9bf93713cf1d1478e25b62f18833c3cf820.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" class="light-scheme"/> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/desktop_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="desktop" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/automation_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="automation" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/checklist_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="checklist" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-ai_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-ai" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-akismet_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-akismet" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-antivirus_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-antivirus" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-assign_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-assign" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-cakeday_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-cakeday" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-calendar_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-calendar" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-chat-integration_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-chat-integration" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-data-explorer_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-data-explorer" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-details_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-details" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-docs_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-docs" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-jira_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-jira" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-lazy-videos_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-lazy-videos" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-local-dates_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-local-dates" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-policy_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-policy" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-presence_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-presence" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-solved_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-solved" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-templates_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-templates" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-topic-voting_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-topic-voting" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-translator_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-translator" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-user-notes_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-user-notes" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-yearly-review_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-yearly-review" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/footnote_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="footnote" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/hosted-site_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="hosted-site" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/poll_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="poll" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/spoiler-alert_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="spoiler-alert" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-ai_desktop_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-ai_desktop" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-calendar_desktop_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-calendar_desktop" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/discourse-topic-voting_desktop_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="discourse-topic-voting_desktop" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/poll_desktop_a0eee66b08799ab07414e8b689b5f4dada707e91.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="poll_desktop" /> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/desktop_theme_11_d44f674dbebc6d04cb93320be0a0aa5b72e0d2d7.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="desktop_theme" data-theme-id="11" data-theme-name="custom header links"/> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/desktop_theme_18_9a79ebc15ad3c4f807e85fe309e8959ee11d4f96.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="desktop_theme" data-theme-id="18" data-theme-name="topic thumbnails"/> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/desktop_theme_19_fb9ac007701ff5ca6e7c095e97fa1b654e7756b3.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="desktop_theme" data-theme-id="19" data-theme-name="versatile banner"/> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/desktop_theme_13_0af64655fecb6a9ed407080d18cfdc208ddbe8ec.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="desktop_theme" data-theme-id="13" data-theme-name="discourse-nvidia-theme"/> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/desktop_theme_17_fa34540ca6989abe2302b4ea9d21a1ad1c16219e.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="desktop_theme" data-theme-id="17" data-theme-name="fix bulk actions button focus"/> <link href="https://sea2.discourse-cdn.com/nvidia/stylesheets/desktop_theme_20_dcf048c51982d668d078dac82d8ac6c5d2966e55.css?__ws=forums.developer.nvidia.com" media="all" rel="stylesheet" data-target="desktop_theme" data-theme-id="20" data-theme-name="versatile banner adjustments"/> <script src="//assets.adobedtm.com/5d4962a43b79/76a76b0bb8ea/launch-7d434965ec64.min.js" async="" nonce="Q4x9V041jEYdNWdx1L8ilVvQA"></script> <meta name="google-site-verification" content="If3jxgzZoS1XODkXDOo83AD2VzBqttpfA2TfyU7YQlk"> <script defer="" src="https://sea2.discourse-cdn.com/nvidia/theme-javascripts/dbd2b5fc7317c86b8d3bb000f99224d82134a5d8.js?__ws=forums.developer.nvidia.com" data-theme-id="13" nonce="Q4x9V041jEYdNWdx1L8ilVvQA"></script> <script defer="" src="https://sea2.discourse-cdn.com/nvidia/theme-javascripts/dc40147ee1dbb3eacb2870c9212fe522658a046b.js?__ws=forums.developer.nvidia.com" data-theme-id="13" nonce="Q4x9V041jEYdNWdx1L8ilVvQA"></script>  <script src="https://cdn.cookielaw.org/scripttemplates/otSDKStub.js" data-document-language="true" type="text/javascript" charset="UTF-8" data-domain-script="3e2b62ff-7ae7-4ac5-87c8-d5949ecafff5" nonce="Q4x9V041jEYdNWdx1L8ilVvQA"></script>  <script type="text/javascript" src="https://images.nvidia.com/aem-dam/Solutions/ot-js/ot-custom.js" nonce="Q4x9V041jEYdNWdx1L8ilVvQA"></script><script defer="" src="https://sea2.discourse-cdn.com/nvidia/theme-javascripts/098438cd37b784b1cdbf6a91223ae055e9a0d312.js?__ws=forums.developer.nvidia.com" data-theme-id="22" nonce="Q4x9V041jEYdNWdx1L8ilVvQA"></script> <link rel="alternate nofollow" type="application/rss+xml" title="RSS feed of 'Running LLMs with TensorRT-LLM on Nvidia Jetson AGX Orin Dev Kit'" href="https://forums.developer.nvidia.com/t/running-llms-with-tensorrt-llm-on-nvidia-jetson-agx-orin-dev-kit/314488.rss" /> <meta property="og:site_name" content="NVIDIA Developer Forums" /> <meta property="og:type" content="website" /> <meta name="twitter:card" content="summary" /> <meta name="twitter:image" content="https://global.discourse-cdn.com/nvidia/original/4X/8/c/c/8cc6064cbc9e07cec9906ad9302aadc245d1ecb8.jpeg" /> <meta property="og:image" content="https://global.discourse-cdn.com/nvidia/original/4X/8/c/c/8cc6064cbc9e07cec9906ad9302aadc245d1ecb8.jpeg" /> <meta property="og:url" content="https://forums.developer.nvidia.com/t/running-llms-with-tensorrt-llm-on-nvidia-jetson-agx-orin-dev-kit/314488" /> <meta name="twitter:url" content="https://forums.developer.nvidia.com/t/running-llms-with-tensorrt-llm-on-nvidia-jetson-agx-orin-dev-kit/314488" /> <meta property="og:title" content="Running LLMs with TensorRT-LLM on Nvidia Jetson AGX Orin Dev Kit" /> <meta name="twitter:title" content="Running LLMs with TensorRT-LLM on Nvidia Jetson AGX Orin Dev Kit" /> <meta property="og:description" content="Greetings to all, Below is the link to my latest post on deploying LLMs using the TensorRT-LLM on the Nvidia Jetson AGX Orin Developer Kit. Thanks!" /> <meta name="twitter:description" content="Greetings to all, Below is the link to my latest post on deploying LLMs using the TensorRT-LLM on the Nvidia Jetson AGX Orin Developer Kit. Thanks!" /> <meta property="og:article:section" content="Autonomous Machines" /> <meta property="og:article:section:color" content="76B900" /> <meta property="og:article:section" content="Jetson & Embedded Systems" /> <meta property="og:article:section:color" content="76B900" /> <meta property="og:article:section" content="Jetson Projects" /> <meta property="og:article:section:color" content="76B900" /> <meta property="og:article:tag" content="generative_ai" /> <meta property="og:article:tag" content="jetson" /> <meta property="article:published_time" content="2024-11-24T07:33:12+00:00" /> <meta property="og:ignore_canonical" content="true" /> <script type="application/ld+json">{"@context":"http://schema.org","@type":"QAPage","name":"Running LLMs with TensorRT-LLM on Nvidia Jetson AGX Orin Dev Kit","mainEntity":{"@type":"Question","name":"Running LLMs with TensorRT-LLM on Nvidia Jetson AGX Orin Dev Kit","text":"Greetings to all,\n\nBelow is the link to my latest post on deploying LLMs using the TensorRT-LLM on the Nvidia Jetson AGX Orin Developer Kit.\n\n[image]\n\n<a href=\"https://www.hackster.io/shahizat/running-llms-with-tensorrt-llm-on-nvidia-jetson-agx-orin-34372f\" target=\"_blank\" rel=\"noopener nofollow ugc\">Running LLMs with TensorRT-LLM on Nvidia Jetson AGX Orin<\/a>\n\nIn this tutorial, I will deploy Large Language Models (LLMs) using the TensorRT-L…","upvoteCount":0,"answerCount":0,"datePublished":"2024-11-24T07:33:12.203Z","author":{"@type":"Person","name":"shahizat","url":"https://forums.developer.nvidia.com/u/shahizat"}}}</script> </head> <body class="crawler browser-update"> <header> <a href="/"> NVIDIA Developer Forums </a> </header> <div id="main-outlet" class="wrap" role="main"> <div id="topic-title"> <h1> <a href="/t/running-llms-with-tensorrt-llm-on-nvidia-jetson-agx-orin-dev-kit/314488">Running LLMs with TensorRT-LLM on Nvidia Jetson AGX Orin Dev Kit</a> </h1> <div class="topic-category" itemscope itemtype="http://schema.org/BreadcrumbList"> <span itemprop="itemListElement" itemscope itemtype="http://schema.org/ListItem"> <a href="/c/agx-autonomous-machines/jetson-embedded-systems/jetson-projects/78" class="badge-wrapper bullet" itemprop="item"> <span class='badge-category-bg' style='background-color: #76B900'></span> <span class='badge-category clear-badge'> <span class='category-name' itemprop='name'>Autonomous Machines</span> </span> </a> <meta itemprop="position" content="1" /> </span> <span itemprop="itemListElement" itemscope itemtype="http://schema.org/ListItem"> <a href="/c/agx-autonomous-machines/jetson-embedded-systems/jetson-projects/78" class="badge-wrapper bullet" itemprop="item"> <span class='badge-category-bg' style='background-color: #76B900'></span> <span class='badge-category clear-badge'> <span class='category-name' itemprop='name'>Jetson & Embedded Systems</span> </span> </a> <meta itemprop="position" content="2" /> </span> <span itemprop="itemListElement" itemscope itemtype="http://schema.org/ListItem"> <a href="/c/agx-autonomous-machines/jetson-embedded-systems/jetson-projects/78" class="badge-wrapper bullet" itemprop="item"> <span class='badge-category-bg' style='background-color: #76B900'></span> <span class='badge-category clear-badge'> <span class='category-name' itemprop='name'>Jetson Projects</span> </span> </a> <meta itemprop="position" content="3" /> </span> </div> <div class="topic-category"> <div class='discourse-tags list-tags'> <a href='https://forums.developer.nvidia.com/tag/generative_ai' class='discourse-tag' rel="tag">generative_ai</a>, <a href='https://forums.developer.nvidia.com/tag/jetson' class='discourse-tag' rel="tag">jetson</a> </div> </div> </div> <div itemscope itemtype='http://schema.org/DiscussionForumPosting'> <meta itemprop='headline' content='Running LLMs with TensorRT-LLM on Nvidia Jetson AGX Orin Dev Kit'> <link itemprop='url' href='https://forums.developer.nvidia.com/t/running-llms-with-tensorrt-llm-on-nvidia-jetson-agx-orin-dev-kit/314488'> <meta itemprop='datePublished' content='2024-11-24T07:33:12Z'> <meta itemprop='articleSection' content='Jetson Projects'> <meta itemprop='keywords' content='generative_ai, jetson'> <div itemprop='publisher' itemscope itemtype="http://schema.org/Organization"> <meta itemprop='name' content='NVIDIA'> <div itemprop='logo' itemscope itemtype="http://schema.org/ImageObject"> <meta itemprop='url' content='https://global.discourse-cdn.com/nvidia/original/3X/a/1/a1ef6e0c1fbd3fad5bf82538b78dfaa9c5fa1a61.png'> </div> </div> <div id='post_1' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://forums.developer.nvidia.com/u/shahizat'><span itemprop='name'>shahizat</span></a> </span> <link itemprop="mainEntityOfPage" href="https://forums.developer.nvidia.com/t/running-llms-with-tensorrt-llm-on-nvidia-jetson-agx-orin-dev-kit/314488"> <link itemprop="image" href="https://global.discourse-cdn.com/nvidia/original/4X/8/c/c/8cc6064cbc9e07cec9906ad9302aadc245d1ecb8.jpeg"> <span class="crawler-post-infos"> <time datetime='2024-11-24T07:33:12Z' class='post-time'> November 24, 2024, 7:33am </time> <meta itemprop='dateModified' content='2024-11-24T07:33:12Z'> <span itemprop='position'>1</span> </span> </div> <div class='post' itemprop='text'> <p>Greetings to all,</p> <p>Below is the link to my latest post on deploying LLMs using the TensorRT-LLM on the Nvidia Jetson AGX Orin Developer Kit.</p> <aside class="onebox allowlistedgeneric" data-onebox-src="https://www.hackster.io/shahizat/running-llms-with-tensorrt-llm-on-nvidia-jetson-agx-orin-34372f"> <header class="source"> <img class="site-icon" src="https://global.discourse-cdn.com/nvidia/original/2X/9/9e566708de48e9d40949216fd3bfa178ae8a0f28.png" data-dominant-color="43BDF9" width="16" height="16"> <a href="https://www.hackster.io/shahizat/running-llms-with-tensorrt-llm-on-nvidia-jetson-agx-orin-34372f" target="_blank" rel="noopener nofollow ugc">Hackster.io</a> </header> <article class="onebox-body"> <img width="600" height="450" class="thumbnail" src="https://global.discourse-cdn.com/nvidia/original/4X/8/c/c/8cc6064cbc9e07cec9906ad9302aadc245d1ecb8.jpeg" data-dominant-color="121318"> <h3><a href="https://www.hackster.io/shahizat/running-llms-with-tensorrt-llm-on-nvidia-jetson-agx-orin-34372f" target="_blank" rel="noopener nofollow ugc">Running LLMs with TensorRT-LLM on Nvidia Jetson AGX Orin</a></h3> <p>In this tutorial, I will deploy Large Language Models (LLMs) using the TensorRT-LLM on the Nvidia Jetson AGX Orin Developer Kit. By Nurgaliyev Shakhizat.</p> </article> <div class="onebox-metadata"> </div> <div style="clear: both"></div> </aside> <p>Thanks!</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> </div> <div id="related-topics" class="more-topics__list " role="complementary" aria-labelledby="related-topics-title"> <h3 id="related-topics-title" class="more-topics__list-title"> Related topics </h3> <div class="topic-list-container" itemscope itemtype='http://schema.org/ItemList'> <meta itemprop='itemListOrder' content='http://schema.org/ItemListOrderDescending'> <table class='topic-list'> <thead> <tr> <th>Topic</th> <th></th> <th class="replies">Replies</th> <th class="views">Views</th> <th>Activity</th> </tr> </thead> <tbody> <tr class="topic-list-item" id="topic-list-item-313227"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='1'> <span class="link-top-line"> <a itemprop='url' href='https://forums.developer.nvidia.com/t/tensorrt-llm-for-jetson/313227' class='title raw-link raw-topic-link'>TensorRT-LLM for Jetson</a> </span> <div class="link-bottom-line"> <a href='/c/agx-autonomous-machines/jetson-embedded-systems/jetson-agx-orin/486' class='badge-wrapper bullet'> <span class='badge-category-bg' style='background-color: #76B900'></span> <span class='badge-category clear-badge'> <span class='category-name'>Jetson AGX Orin</span> </span> </a> <div class="discourse-tags"> <a href='https://forums.developer.nvidia.com/tag/generative_ai' class='discourse-tag'>generative_ai</a> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>2</span> </td> <td class="views"> <span class='views' title='views'>119</span> </td> <td> November 21, 2024 </td> </tr> <tr class="topic-list-item" id="topic-list-item-312008"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='2'> <span class="link-top-line"> <a itemprop='url' href='https://forums.developer.nvidia.com/t/triton-inference-server-vllm-backend-on-the-nvidia-jetson-agx-orin-64gb-developer-kit/312008' class='title raw-link raw-topic-link'>Triton Inference Server + vLLM Backend on the NVIDIA Jetson AGX Orin 64GB Developer Kit</a> </span> <div class="link-bottom-line"> <a href='/c/agx-autonomous-machines/jetson-embedded-systems/jetson-projects/78' class='badge-wrapper bullet'> <span class='badge-category-bg' style='background-color: #76B900'></span> <span class='badge-category clear-badge'> <span class='category-name'>Jetson Projects</span> </span> </a> <div class="discourse-tags"> <a href='https://forums.developer.nvidia.com/tag/generative_ai' class='discourse-tag'>generative_ai</a> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>0</span> </td> <td class="views"> <span class='views' title='views'>49</span> </td> <td> November 3, 2024 </td> </tr> <tr class="topic-list-item" id="topic-list-item-309120"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='3'> <span class="link-top-line"> <a itemprop='url' href='https://forums.developer.nvidia.com/t/can-tensorrt-llm-be-used-on-jetson-orin-nx-with-jetpack-6-1/309120' class='title raw-link raw-topic-link'>Can TensorRT-LLM be used on Jetson Orin NX with JetPack 6.1?</a> </span> <div class="link-bottom-line"> <a href='/c/agx-autonomous-machines/jetson-embedded-systems/jetson-orin-nx/487' class='badge-wrapper bullet'> <span class='badge-category-bg' style='background-color: #76B900'></span> <span class='badge-category clear-badge'> <span class='category-name'>Jetson Orin NX</span> </span> </a> <div class="discourse-tags"> <a href='https://forums.developer.nvidia.com/tag/tensorrt' class='discourse-tag'>tensorrt</a> ,  <a href='https://forums.developer.nvidia.com/tag/generative_ai' class='discourse-tag'>generative_ai</a> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>5</span> </td> <td class="views"> <span class='views' title='views'>116</span> </td> <td> November 21, 2024 </td> </tr> <tr class="topic-list-item" id="topic-list-item-309200"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='4'> <span class="link-top-line"> <a itemprop='url' href='https://forums.developer.nvidia.com/t/deploy-llms-with-microk8s-on-the-nvidia-jetson-agx-orin-dev-kit/309200' class='title raw-link raw-topic-link'>Deploy LLMs with microk8s on the NVIDIA Jetson AGX Orin Dev Kit</a> </span> <div class="link-bottom-line"> <a href='/c/agx-autonomous-machines/jetson-embedded-systems/jetson-projects/78' class='badge-wrapper bullet'> <span class='badge-category-bg' style='background-color: #76B900'></span> <span class='badge-category clear-badge'> <span class='category-name'>Jetson Projects</span> </span> </a> <div class="discourse-tags"> <a href='https://forums.developer.nvidia.com/tag/kubernetes' class='discourse-tag'>kubernetes</a> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>0</span> </td> <td class="views"> <span class='views' title='views'>50</span> </td> <td> October 9, 2024 </td> </tr> <tr class="topic-list-item" id="topic-list-item-265923"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='5'> <span class="link-top-line"> <a itemprop='url' href='https://forums.developer.nvidia.com/t/tensorrt-for-large-language-models/265923' class='title raw-link raw-topic-link'>TensorRT for Large Language Models</a> </span> <div class="link-bottom-line"> <a href='/c/agx-autonomous-machines/jetson-embedded-systems/jetson-agx-orin/486' class='badge-wrapper bullet'> <span class='badge-category-bg' style='background-color: #76B900'></span> <span class='badge-category clear-badge'> <span class='category-name'>Jetson AGX Orin</span> </span> </a> <div class="discourse-tags"> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>2</span> </td> <td class="views"> <span class='views' title='views'>577</span> </td> <td> September 11, 2023 </td> </tr> <tr class="topic-list-item" id="topic-list-item-313228"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='6'> <span class="link-top-line"> <a itemprop='url' href='https://forums.developer.nvidia.com/t/tensorrt-llm-for-jetson/313228' class='title raw-link raw-topic-link'>TensorRT-LLM for Jetson</a> </span> <div class="link-bottom-line"> <a href='/c/agx-autonomous-machines/jetson-embedded-systems/announcements/71' class='badge-wrapper bullet'> <span class='badge-category-bg' style='background-color: #76B900'></span> <span class='badge-category clear-badge'> <span class='category-name'>Announcements</span> </span> </a> <div class="discourse-tags"> <a href='https://forums.developer.nvidia.com/tag/generative_ai' class='discourse-tag'>generative_ai</a> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>0</span> </td> <td class="views"> <span class='views' title='views'>40</span> </td> <td> November 13, 2024 </td> </tr> <tr class="topic-list-item" id="topic-list-item-296059"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='7'> <span class="link-top-line"> <a itemprop='url' href='https://forums.developer.nvidia.com/t/can-i-use-tensorrt-llm-in-jetson-agx-orin/296059' class='title raw-link raw-topic-link'>Can I use TensorRT-LLM in Jetson AGX orin?</a> </span> <div class="link-bottom-line"> <a href='/c/agx-autonomous-machines/jetson-embedded-systems/jetson-agx-orin/486' class='badge-wrapper bullet'> <span class='badge-category-bg' style='background-color: #76B900'></span> <span class='badge-category clear-badge'> <span class='category-name'>Jetson AGX Orin</span> </span> </a> <div class="discourse-tags"> <a href='https://forums.developer.nvidia.com/tag/nvbugs' class='discourse-tag'>nvbugs</a> ,  <a href='https://forums.developer.nvidia.com/tag/generative_ai' class='discourse-tag'>generative_ai</a> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>3</span> </td> <td class="views"> <span class='views' title='views'>505</span> </td> <td> July 15, 2024 </td> </tr> <tr class="topic-list-item" id="topic-list-item-301488"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='8'> <span class="link-top-line"> <a itemprop='url' href='https://forums.developer.nvidia.com/t/does-tensorrt-llm-supports-on-nvidia-jetson-agx-orin-edge-device/301488' class='title raw-link raw-topic-link'>Does TensorRT-LLM Supports on NVIDIA Jetson AGX Orin Edge Device?</a> </span> <div class="link-bottom-line"> <a href='/c/agx-autonomous-machines/jetson-embedded-systems/jetson-agx-orin/486' class='badge-wrapper bullet'> <span class='badge-category-bg' style='background-color: #76B900'></span> <span class='badge-category clear-badge'> <span class='category-name'>Jetson AGX Orin</span> </span> </a> <div class="discourse-tags"> <a href='https://forums.developer.nvidia.com/tag/generative_ai' class='discourse-tag'>generative_ai</a> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>2</span> </td> <td class="views"> <span class='views' title='views'>115</span> </td> <td> July 29, 2024 </td> </tr> <tr class="topic-list-item" id="topic-list-item-303202"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='9'> <span class="link-top-line"> <a itemprop='url' href='https://forums.developer.nvidia.com/t/nvidia-nim-tensorrt-turboxl/303202' class='title raw-link raw-topic-link'>NVIDIA NIM - TensorRT TurboXL</a> </span> <div class="link-bottom-line"> <a href='/c/agx-autonomous-machines/jetson-embedded-systems/jetson-agx-orin/486' class='badge-wrapper bullet'> <span class='badge-category-bg' style='background-color: #76B900'></span> <span class='badge-category clear-badge'> <span class='category-name'>Jetson AGX Orin</span> </span> </a> <div class="discourse-tags"> <a href='https://forums.developer.nvidia.com/tag/generative_ai' class='discourse-tag'>generative_ai</a> ,  <a href='https://forums.developer.nvidia.com/tag/nim' class='discourse-tag'>nim</a> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>2</span> </td> <td class="views"> <span class='views' title='views'>79</span> </td> <td> September 16, 2024 </td> </tr> <tr class="topic-list-item" id="topic-list-item-301413"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='10'> <span class="link-top-line"> <a itemprop='url' href='https://forums.developer.nvidia.com/t/nvidia-jetson-orin-nano-tensorrt-llm/301413' class='title raw-link raw-topic-link'>Nvidia Jetson Orin Nano tensorrt llm</a> </span> <div class="link-bottom-line"> <a href='/c/agx-autonomous-machines/jetson-embedded-systems/jetson-orin-nano/632' class='badge-wrapper bullet'> <span class='badge-category-bg' style='background-color: #76B900'></span> <span class='badge-category clear-badge'> <span class='category-name'>Jetson Orin Nano</span> </span> </a> <div class="discourse-tags"> <a href='https://forums.developer.nvidia.com/tag/tensorrt' class='discourse-tag'>tensorrt</a> ,  <a href='https://forums.developer.nvidia.com/tag/generative_ai' class='discourse-tag'>generative_ai</a> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>6</span> </td> <td class="views"> <span class='views' title='views'>106</span> </td> <td> August 5, 2024 </td> </tr> </tbody> </table> </div> </div> </div> <footer class="container wrap"> <nav class='crawler-nav'> <ul> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/' itemprop="url">Home </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/categories' itemprop="url">Categories </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/guidelines' itemprop="url">Guidelines </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='https://www.nvidia.com/en-us/about-nvidia/legal-info/' itemprop="url">Terms of Service </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='https://www.nvidia.com/en-us/about-nvidia/privacy-policy/' itemprop="url">Privacy Policy </a> </span> </li> </ul> </nav> <p class='powered-by-link'>Powered by <a href="https://www.discourse.org">Discourse</a>, best viewed with JavaScript enabled</p> </footer> <script defer="" src="https://sea2.discourse-cdn.com/nvidia/theme-javascripts/d8ea190aad487b8feae316757854a563d15c4c27.js?__ws=forums.developer.nvidia.com" data-theme-id="13" nonce="Q4x9V041jEYdNWdx1L8ilVvQA"></script> <footer> <div class="footer-links"> <div class="container"> <div class="row"> <div class="col-xs-12 col-sm-12 col-md-3 col-lg-3"> <div class="col-xs-12 col-sm-12 col-md-12 col-lg-12"> <div class="padding-md-footer"> <div class="logo-footer"></div> </div> </div> <div class="col-xs-12 col-sm-12 col-md-9 col-lg-9 padding-section-footer"></div> </div> <div class="col-xs-12 col-sm-12 col-md-9 col-lg-9"></div> </div> </div> </div> <div class="footer-boilerplate"> <div class="container"> <div class="boilerplate"> <div class="col-xs-12 col-sm-12 col-lg-9 padding-sm-bottom"> Copyright 漏 2024 NVIDIA Corporation <ul class="legal_links"> <li class="first leaf"> <a href="https://www.nvidia.com/en-us/about-nvidia/legal-info/" title="">Legal Information</a> </li> <li class="leaf"> <a href="https://developer.nvidia.com/legal/terms" title="">Terms of Use</a> </li> <li class="leaf"> <a href="https://www.nvidia.com/en-us/about-nvidia/privacy-policy/" title="">Privacy Policy</a> </li> <li class="leaf"> <a href="https://developer.nvidia.com/contact" title="">Contact</a> </li> <li class="last leaf"> <a href="https://www.nvidia.com/en-us/about-nvidia/cookie-policy/" title="NVIDIA websites use cookies to deliver and improve the website experience. See our cookie policy for further details on how we use cookies and how to change your cookie settings.">Cookie Policy</a> </li> </ul> </div> </div> </div> </div> </footer> <div class="buorg"><div>Unfortunately, <a href="https://www.discourse.org/faq/#browser">your browser is unsupported</a>. Please <a href="https://browsehappy.com">switch to a supported browser</a> to view rich content, log in and reply.</div></div> </body> </html>

CINXE.COM

Running LLMs with TensorRT-LLM on Nvidia Jetson AGX Orin Dev Kit - Jetson Projects - NVIDIA Developer Forums