DOCA GPUNetIO - NVIDIA Docs

<!DOCTYPE html> <html class="TopicPage"xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en" dir="ltr"> <head> <meta charset="UTF-8"> <meta property="og:url" content="https://docs.nvidia.com/doca/sdk/DOCA+GPUNetIO/index.html"> <meta property="og:site_name" content="NVIDIA Docs"> <meta property="og:type" content="website"> <meta name="twitter:card" content="summary_large_image"> <meta name="title" content="DOCA GPUNetIO"> <meta name="version" content="DOCA SDK 2.9.0"> <meta name="nv-pub-date" content="Tue Nov 19 13:34:52 EST 2024"> <meta name="docs_products" content="BlueField DPUs / SuperNICs & DOCA"> <meta name="docs_version" content="DOCA SDK 2.9.0"> <meta name="docs_books" content="DOCA Documentation v2.9.0"> <meta name="subcollection" content="Technical Documentation"> <meta property="mimetype" content="webpage"> <style>:root { } </style> <style>:root { --button-border-radius:0px; }</style> <style>:root { --font-1: "NVIDIA Sans"; --font-2: "Times"; } body { --font-1: "NVIDIA Sans"; --font-2: "Merriweather"; } </style> <script src="https://cdn.cookielaw.org/scripttemplates/otSDKStub.js" data-document-language="true" type="text/javascript" charset="UTF-8" data-domain-script="3e2b62ff-7ae7-4ac5-87c8-d5949ecafff5" ></script> <script type="text/javascript"> function OptanonWrapper() { var event = new Event('bannerLoaded'); window.dispatchEvent(event); } </script>  <script type="text/javascript" src="https://images.nvidia.com/aem-dam/Solutions/ot-js/ot-custom.js"></script><script src="https://assets.adobedtm.com/5d4962a43b79/c1061d2c5e7b/launch-191c2462b890.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/3.2.2/es5/tex-mml-svg.min.js"></script> <script src="https://js.hcaptcha.com/1/api.js"></script> <script src="https://tbyb.rivaspeech.com/tbyb.js" defer></script> <script src="https://t.contentsquare.net/uxa/c6af8848c2687.js"></script> <script src="https://unpkg.com/mermaid@9.2.2/dist/mermaid.min.js"></script> <script>mermaid.initialize({startOnLoad:true})</script> <link rel="preload" as="style" href="https://docscontent.nvidia.com/resource/00000183-6725-d7fd-a5bb-6727dc560000/styles/default/All.min.8d017581b5425d9458367c048c94a4a8.gz.css" /> <link data-cssvarsponyfill="true" class="Webpack-css" rel="stylesheet" href="https://docscontent.nvidia.com/resource/00000183-6725-d7fd-a5bb-6727dc560000/styles/default/All.min.8d017581b5425d9458367c048c94a4a8.gz.css"> <style>.Page-content { margin-top: 0; } .Page-search input { padding-left: 32px!important; } .Page-search button { line-height: 24px; }</style> <style>#onetrust-banner-sdk #onetrust-policy-text p#nv-gpc-text, #onetrust-banner-sdk #onetrust-policy-text p#nv-non-gpc-text { padding: 0 !important; } #onetrust-consent-sdk #ot-pc-title, #onetrust-consent-sdk #ot-pc-lst h3 { font-size: 16px !important; } #onetrust-consent-sdk .ot-always-active { font-size: 14px !important; } #onetrust-pc-sdk #ot-pc-lst .ot-host-item .ot-acc-hdr .ot-host-hdr .ot-host-expand { font-size: 12px !important; }One Trust Inline P Styling</style> <meta name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=5"><title>DOCA GPUNetIO - NVIDIA Docs</title><link rel="canonical" href="https://docs.nvidia.com/doca/sdk/DOCA+GPUNetIO/index.html"><meta name="brightspot.contentId" content="00000193-4502-d86e-a5f3-fd0f98450000"><link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png"><link rel="icon" type="image/png" href="/favicon-32x32.png"><link rel="icon" type="image/png" href="/favicon-16x16.png"><script type="application/ld+json">{"@context":"http://schema.org","@type":"WebPage","url":"https://docs.nvidia.com/doca/sdk/DOCA+GPUNetIO/index.html","publisher":{"@type":"Organization","name":"NVIDIA Docs","logo":{"@type":"ImageObject","url":"https://docscontent.nvidia.com/bf/6f/f2d5da4743aebb3dff0e6a6129ec/nvidia-docshub-logo-2.svg"}},"name":"DOCA GPUNetIO - NVIDIA Docs"}</script>  <script defer src="https://docscontent.nvidia.com/resource/00000183-6725-d7fd-a5bb-6727dc560000/webcomponents-loader/webcomponents-loader.2938a610ca02c611209b1a5ba2884385.gz.js"></script> <script> /** This allows us to load the IE polyfills via feature detection so that they do not load needlessly in the browsers that do not need them. It also ensures they are loaded non async so that they load before the rest of our JS. */ if (!window.CSS || !window.CSS.supports || !window.CSS.supports('--fake-var', 0)) { var script = document.createElement('script'); script.setAttribute('src', "https://docscontent.nvidia.com/resource/00000183-6725-d7fd-a5bb-6727dc560000/util/IEPolyfills.3af8d32fc874177c8b40f77d0ae336b5.gz.js"); script.setAttribute('type', 'text/javascript'); script.async = false; script.defer = true; document.head.appendChild(script); } </script> <script defer src="https://docscontent.nvidia.com/resource/00000183-6725-d7fd-a5bb-6727dc560000/styles/default/All.min.09629531d7098fb938880c46edb9dedf.gz.js"></script> <script> var link = document.createElement('link'); link.setAttribute('href', 'https://fonts.googleapis.com/css?family=Roboto+Mono&family=Roboto:300,400,700'); var relList = link.relList; if (relList && relList.supports('preload')) { link.setAttribute('as', 'style'); link.setAttribute('rel', 'preload'); link.setAttribute('onload', 'this.rel="stylesheet"'); link.setAttribute('crossorigin', 'anonymous'); } else { link.setAttribute('rel', 'stylesheet'); } document.head.appendChild(link); </script> <script defer src="https://ajax.googleapis.com/ajax/libs/jquery/1.12.4/jquery.min.js" integrity="sha384-nvAa0+6Qg9clwYCGGPpDQLVpLNn0fRaROjHqs13t4Ggj3Ez50XnGQqc/r8MhnRDZ" crossorigin="anonymous"></script> </head> <body class="Page-body">  <svg xmlns="http://www.w3.org/2000/svg" style="display:none" id="iconsMap"> <symbol id="back-to-top-icon" fill="#000000" height="20" width="15" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 490 490" xml:space="preserve"> <g id="SVGRepo_bgCarrier" stroke-width="0"></g> <g id="SVGRepo_tracerCarrier" stroke-linecap="round" stroke-linejoin="round"></g> <g id="SVGRepo_iconCarrier"> <polygon points="245,33.299 0,456.701 490,456.701"></polygon> </g> </symbol> <symbol id="mono-icon-facebook" viewBox="0 0 10 19"> <path fill-rule="evenodd" d="M2.707 18.25V10.2H0V7h2.707V4.469c0-1.336.375-2.373 1.125-3.112C4.582.62 5.578.25 6.82.25c1.008 0 1.828.047 2.461.14v2.848H7.594c-.633 0-1.067.14-1.301.422-.188.235-.281.61-.281 1.125V7H9l-.422 3.2H6.012v8.05H2.707z"></path> </symbol> <symbol id="action-icon-facebook" viewBox="0 0 7 12" xmlns="http://www.w3.org/2000/svg"> <g stroke="none" stroke-width="1" fill="none" fill-rule="evenodd"> <g transform="translate(-112.000000, -395.000000)" fill="#FFFFFF"> <g transform="translate(100.000000, 386.000000)"> <g transform="translate(12.000000, 9.000000)"> <path d="M1.777125,12 L1.777125,6.48659606 L0,6.48659606 L0,4.05412254 L1.777125,4.05412254 L1.777125,3.10626869 C1.777125,1.47184919 2.99976923,0 4.50308654,0 L6.46153846,0 L6.46153846,2.43247352 L4.50308654,2.43247352 C4.28864423,2.43247352 4.03846154,2.69376172 4.03846154,3.08498454 L4.03846154,4.05412254 L6.46153846,4.05412254 L6.46153846,6.48659606 L4.03846154,6.48659606 L4.03846154,12" /> </g> </g> </g> </g> </symbol> <symbol id="mono-icon-instagram" viewBox="0 0 17 17"> <g> <path fill-rule="evenodd" d="M8.281 4.207c.727 0 1.4.182 2.022.545a4.055 4.055 0 0 1 1.476 1.477c.364.62.545 1.294.545 2.021 0 .727-.181 1.4-.545 2.021a4.055 4.055 0 0 1-1.476 1.477 3.934 3.934 0 0 1-2.022.545c-.726 0-1.4-.182-2.021-.545a4.055 4.055 0 0 1-1.477-1.477 3.934 3.934 0 0 1-.545-2.021c0-.727.182-1.4.545-2.021A4.055 4.055 0 0 1 6.26 4.752a3.934 3.934 0 0 1 2.021-.545zm0 6.68a2.54 2.54 0 0 0 1.864-.774 2.54 2.54 0 0 0 .773-1.863 2.54 2.54 0 0 0-.773-1.863 2.54 2.54 0 0 0-1.864-.774 2.54 2.54 0 0 0-1.863.774 2.54 2.54 0 0 0-.773 1.863c0 .727.257 1.348.773 1.863a2.54 2.54 0 0 0 1.863.774zM13.45 4.03c-.023.258-.123.48-.299.668a.856.856 0 0 1-.65.281.913.913 0 0 1-.668-.28.913.913 0 0 1-.281-.669c0-.258.094-.48.281-.668a.913.913 0 0 1 .668-.28c.258 0 .48.093.668.28.187.188.281.41.281.668zm2.672.95c.023.656.035 1.746.035 3.269 0 1.523-.017 2.62-.053 3.287-.035.668-.134 1.248-.298 1.74a4.098 4.098 0 0 1-.967 1.53 4.098 4.098 0 0 1-1.53.966c-.492.164-1.072.264-1.74.3-.668.034-1.763.052-3.287.052-1.523 0-2.619-.018-3.287-.053-.668-.035-1.248-.146-1.74-.334a3.747 3.747 0 0 1-1.53-.931 4.098 4.098 0 0 1-.966-1.53c-.164-.492-.264-1.072-.299-1.74C.424 10.87.406 9.773.406 8.25S.424 5.63.46 4.963c.035-.668.135-1.248.299-1.74.21-.586.533-1.096.967-1.53A4.098 4.098 0 0 1 3.254.727c.492-.164 1.072-.264 1.74-.3C5.662.394 6.758.376 8.281.376c1.524 0 2.62.018 3.287.053.668.035 1.248.135 1.74.299a4.098 4.098 0 0 1 2.496 2.496c.165.492.27 1.078.317 1.757zm-1.687 7.91c.14-.399.234-1.032.28-1.899.024-.515.036-1.242.036-2.18V7.689c0-.961-.012-1.688-.035-2.18-.047-.89-.14-1.524-.281-1.899a2.537 2.537 0 0 0-1.512-1.511c-.375-.14-1.008-.235-1.899-.282a51.292 51.292 0 0 0-2.18-.035H7.72c-.938 0-1.664.012-2.18.035-.867.047-1.5.141-1.898.282a2.537 2.537 0 0 0-1.512 1.511c-.14.375-.234 1.008-.281 1.899a51.292 51.292 0 0 0-.036 2.18v1.125c0 .937.012 1.664.036 2.18.047.866.14 1.5.28 1.898.306.726.81 1.23 1.513 1.511.398.141 1.03.235 1.898.282.516.023 1.242.035 2.18.035h1.125c.96 0 1.687-.012 2.18-.035.89-.047 1.523-.141 1.898-.282.726-.304 1.23-.808 1.512-1.511z"> </path> </g> </symbol> <symbol id="mono-icon-mailto" viewBox="0 0 512 512"> <g> <path d="M67,148.7c11,5.8,163.8,89.1,169.5,92.1c5.7,3,11.5,4.4,20.5,4.4c9,0,14.8-1.4,20.5-4.4c5.7-3,158.5-86.3,169.5-92.1 c4.1-2.1,11-5.9,12.5-10.2c2.6-7.6-0.2-10.5-11.3-10.5H257H65.8c-11.1,0-13.9,3-11.3,10.5C56,142.9,62.9,146.6,67,148.7z"> </path> <path d="M455.7,153.2c-8.2,4.2-81.8,56.6-130.5,88.1l82.2,92.5c2,2,2.9,4.4,1.8,5.6c-1.2,1.1-3.8,0.5-5.9-1.4l-98.6-83.2 c-14.9,9.6-25.4,16.2-27.2,17.2c-7.7,3.9-13.1,4.4-20.5,4.4c-7.4,0-12.8-0.5-20.5-4.4c-1.9-1-12.3-7.6-27.2-17.2l-98.6,83.2 c-2,2-4.7,2.6-5.9,1.4c-1.2-1.1-0.3-3.6,1.7-5.6l82.1-92.5c-48.7-31.5-123.1-83.9-131.3-88.1c-8.8-4.5-9.3,0.8-9.3,4.9 c0,4.1,0,205,0,205c0,9.3,13.7,20.9,23.5,20.9H257h185.5c9.8,0,21.5-11.7,21.5-20.9c0,0,0-201,0-205 C464,153.9,464.6,148.7,455.7,153.2z"></path> </g> </symbol> <svg id="action-icon-mailto" viewBox="0 0 20 14" xmlns="http://www.w3.org/2000/svg"> <g stroke="none" stroke-width="1" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"> <g transform="translate(-245.000000, -2697.000000)" stroke="#000000"> <g transform="translate(100.000000, 1138.000000)"> <g transform="translate(0.000000, 132.000000)"> <g transform="translate(0.000000, 1419.000000)"> <g transform="translate(140.000000, 0.000000)"> <g transform="translate(6.000000, 9.000000)"> <path d="M17.5909091,10.6363636 C17.5909091,11.3138182 17.0410909,11.8636364 16.3636364,11.8636364 L1.63636364,11.8636364 C0.958909091,11.8636364 0.409090909,11.3138182 0.409090909,10.6363636 L0.409090909,1.63636364 C0.409090909,0.958090909 0.958909091,0.409090909 1.63636364,0.409090909 L16.3636364,0.409090909 C17.0410909,0.409090909 17.5909091,0.958090909 17.5909091,1.63636364 L17.5909091,10.6363636 L17.5909091,10.6363636 Z" /> <polyline points="17.1818182 0.818181818 9 7.36363636 0.818181818 0.818181818" /> </g> </g> </g> </g> </g> </g> </g> </svg> <symbol id="mono-icon-print" viewBox="0 0 12 12"> <g fill-rule="evenodd"> <path fill-rule="nonzero" d="M9 10V7H3v3H1a1 1 0 0 1-1-1V4a1 1 0 0 1 1-1h10a1 1 0 0 1 1 1v3.132A2.868 2.868 0 0 1 9.132 10H9zm.5-4.5a1 1 0 1 0 0-2 1 1 0 0 0 0 2zM3 0h6v2H3z"> </path> <path d="M4 8h4v4H4z"></path> </g> </symbol> <symbol id="action-icon-print" viewBox="0 0 20 18" xmlns="http://www.w3.org/2000/svg"> <g stroke="none" stroke-width="1" fill="none" fill-rule="evenodd" stroke-linecap="round" stroke-linejoin="round"> <g transform="translate(-280.000000, -2695.000000)" stroke="#000000" stroke-width="0.9"> <g transform="translate(100.000000, 1138.000000)"> <g transform="translate(0.000000, 132.000000)"> <g transform="translate(0.000000, 1419.000000)"> <g transform="translate(140.000000, 0.000000)"> <g transform="translate(35.000000, 0.000000)"> <g transform="translate(6.000000, 7.000000)"> <path d="M4.125,12.375 L1.875,12.375 C1.05,12.375 0.375,11.7 0.375,10.875 L0.375,6.375 C0.375,5.55 1.05,4.875 1.875,4.875 L16.125,4.875 C16.95075,4.875 17.625,5.55 17.625,6.375 L17.625,10.875 C17.625,11.7 16.95075,12.375 16.125,12.375 L13.875,12.375" /> <g transform="translate(4.125000, 0.375000)"> <polyline id="Stroke-2384" points="0 3 0 0 7.5 0 9.75 2.25 9.75 3" /> <polygon id="Stroke-2385" points="0 15.75 9.75 15.75 9.75 9.75 0 9.75" /> </g> <path d="M3.375,7.125 C3.375,7.539 3.03975,7.875 2.625,7.875 C2.211,7.875 1.875,7.539 1.875,7.125 C1.875,6.71025 2.211,6.375 2.625,6.375 C3.03975,6.375 3.375,6.71025 3.375,7.125 L3.375,7.125 Z" /> <line x1="5.625" y1="11.625" x2="12.375" y2="11.625" id="Stroke-2387" /> <line x1="5.625" y1="13.125" x2="12.375" y2="13.125" id="Stroke-2388" /> <line x1="5.625" y1="14.625" x2="12.375" y2="14.625" id="Stroke-2389" /> <polyline points="11.625 0.375 11.625 2.625 13.875 2.625" /> </g> </g> </g> </g> </g> </g> </g> </g> </symbol> <symbol id="mono-icon-copylink" viewBox="0 0 12 12"> <g fill-rule="evenodd"> <path d="M10.199 2.378c.222.205.4.548.465.897.062.332.016.614-.132.774L8.627 6.106c-.187.203-.512.232-.75-.014a.498.498 0 0 0-.706.028.499.499 0 0 0 .026.706 1.509 1.509 0 0 0 2.165-.04l1.903-2.06c.37-.398.506-.98.382-1.636-.105-.557-.392-1.097-.77-1.445L9.968.8C9.591.452 9.03.208 8.467.145 7.803.072 7.233.252 6.864.653L4.958 2.709a1.509 1.509 0 0 0 .126 2.161.5.5 0 1 0 .68-.734c-.264-.218-.26-.545-.071-.747L7.597 1.33c.147-.16.425-.228.76-.19.353.038.71.188.931.394l.91.843.001.001zM1.8 9.623c-.222-.205-.4-.549-.465-.897-.062-.332-.016-.614.132-.774l1.905-2.057c.187-.203.512-.232.75.014a.498.498 0 0 0 .706-.028.499.499 0 0 0-.026-.706 1.508 1.508 0 0 0-2.165.04L.734 7.275c-.37.399-.506.98-.382 1.637.105.557.392 1.097.77 1.445l.91.843c.376.35.937.594 1.5.656.664.073 1.234-.106 1.603-.507L7.04 9.291a1.508 1.508 0 0 0-.126-2.16.5.5 0 0 0-.68.734c.264.218.26.545.071.747l-1.904 2.057c-.147.16-.425.228-.76.191-.353-.038-.71-.188-.931-.394l-.91-.843z"> </path> <path d="M8.208 3.614a.5.5 0 0 0-.707.028L3.764 7.677a.5.5 0 0 0 .734.68L8.235 4.32a.5.5 0 0 0-.027-.707"> </path> </g> </symbol> <symbol id="mono-icon-linkedin" viewBox="0 0 14 14" xmlns="http://www.w3.org/2000/svg"> <path d="M9.245 7.318c-.704 0-1.273.57-1.273 1.273v4.454H4.79s.038-7.636 0-8.272h3.182v.945s.985-.919 2.507-.919c1.884 0 3.22 1.364 3.22 4.012v4.234h-3.182V8.591c0-.703-.57-1.273-1.272-1.273zM1.92 3.5h-.018C.94 3.5.318 2.748.318 1.921.318 1.075.96.386 1.94.386s1.583.712 1.603 1.557c0 .828-.622 1.557-1.622 1.557zm1.597 9.545H.336V4.773h3.181v8.272z" fill-rule="evenodd" /> </symbol> <symbol id="action-icon-linkedin" viewBox="0 0 14 14" xmlns="http://www.w3.org/2000/svg"> <g stroke="none" stroke-width="1" fill="none" fill-rule="evenodd"> <g transform="translate(-214.000000, -394.000000)" fill="#FFFFFF"> <g transform="translate(100.000000, 386.000000)"> <g transform="translate(114.318373, 8.386273)"> <path d="M8.92627273,6.93190909 C8.22309091,6.93190909 7.65354545,7.50209091 7.65354545,8.20463636 L7.65354545,12.6591818 L4.47172727,12.6591818 C4.47172727,12.6591818 4.50927273,5.02281818 4.47172727,4.38645455 L7.65354545,4.38645455 L7.65354545,5.33145455 C7.65354545,5.33145455 8.63863636,4.41318182 10.1601818,4.41318182 C12.0444545,4.41318182 13.3808182,5.77690909 13.3808182,8.42481818 L13.3808182,12.6591818 L10.199,12.6591818 L10.199,8.20463636 C10.199,7.50209091 9.62945455,6.93190909 8.92627273,6.93190909 L8.92627273,6.93190909 Z M1.60172727,3.11372727 L1.58327273,3.11372727 C0.621727273,3.11372727 0,2.36154545 0,1.53490909 C0,0.689181818 0.640818182,1.77635684e-15 1.62081818,1.77635684e-15 C2.60145455,1.77635684e-15 3.20409091,0.711454545 3.22318182,1.55718182 C3.22318182,2.38445455 2.60145455,3.11372727 1.60172727,3.11372727 L1.60172727,3.11372727 Z M3.199,12.6591818 L0.0171818182,12.6591818 L0.0171818182,4.38645455 L3.199,4.38645455 L3.199,12.6591818 Z" /> </g> </g> </g> </g> </symbol> <symbol id="download" xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24"> <g fill="none" fill-rule="evenodd"> <g> <g> <g> <path d="M0 0H24V24H0z" transform="translate(-802.000000, -1914.000000) translate(0.000000, 1756.000000) translate(802.000000, 158.000000)" /> <path fill="#000" fill-rule="nonzero" d="M12 2C6.49 2 2 6.49 2 12s4.49 10 10 10 10-4.49 10-10S17.51 2 12 2zm-1 8V6h2v4h3l-4 4-4-4h3zm6 7H7v-2h10v2z" transform="translate(-802.000000, -1914.000000) translate(0.000000, 1756.000000) translate(802.000000, 158.000000)" /> </g> </g> </g> </g> </symbol> <symbol id="mono-icon-pinterest" viewBox="0 0 512 512"> <g> <path d="M256,32C132.3,32,32,132.3,32,256c0,91.7,55.2,170.5,134.1,205.2c-0.6-15.6-0.1-34.4,3.9-51.4 c4.3-18.2,28.8-122.1,28.8-122.1s-7.2-14.3-7.2-35.4c0-33.2,19.2-58,43.2-58c20.4,0,30.2,15.3,30.2,33.6 c0,20.5-13.1,51.1-19.8,79.5c-5.6,23.8,11.9,43.1,35.4,43.1c42.4,0,71-54.5,71-119.1c0-49.1-33.1-85.8-93.2-85.8 c-67.9,0-110.3,50.7-110.3,107.3c0,19.5,5.8,33.3,14.8,43.9c4.1,4.9,4.7,6.9,3.2,12.5c-1.1,4.1-3.5,14-4.6,18 c-1.5,5.7-6.1,7.7-11.2,5.6c-31.3-12.8-45.9-47-45.9-85.6c0-63.6,53.7-139.9,160.1-139.9c85.5,0,141.8,61.9,141.8,128.3 c0,87.9-48.9,153.5-120.9,153.5c-24.2,0-46.9-13.1-54.7-27.9c0,0-13,51.6-15.8,61.6c-4.7,17.3-14,34.5-22.5,48 c20.1,5.9,41.4,9.2,63.5,9.2c123.7,0,224-100.3,224-224C480,132.3,379.7,32,256,32z"></path> </g> </symbol> <symbol id="action-icon-pinterest" viewBox="0 0 30 30" xmlns="http://www.w3.org/2000/svg"> <g stroke="none" stroke-width="1" fill="none" fill-rule="evenodd"> <g transform="translate(-135.000000, -526.000000)"> <g transform="translate(100.000000, 526.000000)"> <g transform="translate(35.000000, 0.000000)"> <circle fill="#BD081C" cx="15" cy="15" r="15" /> <path d="M10.125,13.6640625 C10.125,13.1015597 10.2226553,12.5716171 10.4179687,12.0742188 C10.6132822,11.5768204 10.8828108,11.143231 11.2265625,10.7734375 C11.5703142,10.403644 11.9661436,10.0833347 12.4140625,9.8125 C12.8619814,9.54166531 13.3437474,9.33854234 13.859375,9.203125 C14.3750026,9.06770766 14.901039,9 15.4375,9 C16.2604208,9 17.0260381,9.17317535 17.734375,9.51953125 C18.4427119,9.86588715 19.018227,10.3697884 19.4609375,11.03125 C19.903648,11.6927116 20.125,12.4401 20.125,13.2734375 C20.125,13.77344 20.0755213,14.2630184 19.9765625,14.7421875 C19.8776037,15.2213566 19.7213552,15.6822895 19.5078125,16.125 C19.2942698,16.5677105 19.0338557,16.9570296 18.7265625,17.2929688 C18.4192693,17.6289079 18.0416689,17.8971344 17.59375,18.0976562 C17.1458311,18.2981781 16.6536485,18.3984375 16.1171875,18.3984375 C15.7630191,18.3984375 15.4114601,18.315105 15.0625,18.1484375 C14.7135399,17.98177 14.4635424,17.7526056 14.3125,17.4609375 C14.2604164,17.6640635 14.1875005,17.9570293 14.09375,18.3398438 C13.9999995,18.7226582 13.9388022,18.9700515 13.9101562,19.0820312 C13.8815103,19.194011 13.8281254,19.378905 13.75,19.6367188 C13.6718746,19.8945325 13.604167,20.0794265 13.546875,20.1914062 C13.489583,20.303386 13.4062505,20.4661448 13.296875,20.6796875 C13.1874995,20.8932302 13.067709,21.0950511 12.9375,21.2851562 C12.807291,21.4752614 12.6458343,21.7005195 12.453125,21.9609375 L12.34375,22 L12.2734375,21.921875 C12.1953121,21.1041626 12.15625,20.6145841 12.15625,20.453125 C12.15625,19.9739559 12.212239,19.4362009 12.3242187,18.8398438 C12.4361985,18.2434866 12.6093738,17.4947962 12.84375,16.59375 C13.0781262,15.6927038 13.2135415,15.1640633 13.25,15.0078125 C13.0833325,14.6692691 13,14.2291694 13,13.6875 C13,13.2552062 13.1354153,12.8489602 13.40625,12.46875 C13.6770847,12.0885398 14.0208312,11.8984375 14.4375,11.8984375 C14.7552099,11.8984375 15.0026033,12.0039052 15.1796875,12.2148438 C15.3567717,12.4257823 15.4453125,12.6927067 15.4453125,13.015625 C15.4453125,13.3593767 15.3307303,13.8567676 15.1015625,14.5078125 C14.8723947,15.1588574 14.7578125,15.6458317 14.7578125,15.96875 C14.7578125,16.2968766 14.8749988,16.5690093 15.109375,16.7851562 C15.3437512,17.0013032 15.6276025,17.109375 15.9609375,17.109375 C16.2473973,17.109375 16.5130196,17.0442715 16.7578125,16.9140625 C17.0026054,16.7838535 17.2070304,16.606772 17.3710937,16.3828125 C17.5351571,16.158853 17.6809889,15.9114597 17.8085937,15.640625 C17.9361986,15.3697903 18.0351559,15.0820328 18.1054687,14.7773438 C18.1757816,14.4726547 18.2278644,14.1835951 18.2617187,13.9101562 C18.2955731,13.6367174 18.3125,13.3776054 18.3125,13.1328125 C18.3125,12.2317663 18.0273466,11.5299504 17.4570312,11.0273438 C16.8867159,10.5247371 16.1432337,10.2734375 15.2265625,10.2734375 C14.1848906,10.2734375 13.3151077,10.6106737 12.6171875,11.2851562 C11.9192673,11.9596388 11.5703125,12.815099 11.5703125,13.8515625 C11.5703125,14.0807303 11.6028643,14.3020823 11.6679687,14.515625 C11.7330732,14.7291677 11.803385,14.8984369 11.8789062,15.0234375 C11.9544275,15.1484381 12.0247393,15.2669265 12.0898437,15.3789062 C12.1549482,15.490886 12.1875,15.5703123 12.1875,15.6171875 C12.1875,15.7630216 12.1484379,15.9531238 12.0703125,16.1875 C11.9921871,16.4218762 11.8958339,16.5390625 11.78125,16.5390625 C11.7708333,16.5390625 11.7265629,16.5312501 11.6484375,16.515625 C11.3828112,16.4374996 11.1471364,16.2916677 10.9414062,16.078125 C10.7356761,15.8645823 10.5768235,15.618491 10.4648437,15.3398438 C10.352864,15.0611965 10.2682295,14.7799493 10.2109375,14.4960938 C10.1536455,14.2122382 10.125,13.9348972 10.125,13.6640625 Z" fill="#FFFFFF" /> </g> </g> </g> </g> </symbol> <symbol id="icon-fullscreen" xmlns="http://www.w3.org/2000/svg" fill="white" viewBox="0 0 1024 1024"> <path d="M290 236.4l43.9-43.9a8.01 8.01 0 0 0-4.7-13.6L169 160c-5.1-.6-9.5 3.7-8.9 8.9L179 329.1c.8 6.6 8.9 9.4 13.6 4.7l43.7-43.7L370 423.7c3.1 3.1 8.2 3.1 11.3 0l42.4-42.3c3.1-3.1 3.1-8.2 0-11.3L290 236.4zm352.7 187.3c3.1 3.1 8.2 3.1 11.3 0l133.7-133.6 43.7 43.7a8.01 8.01 0 0 0 13.6-4.7L863.9 169c.6-5.1-3.7-9.5-8.9-8.9L694.8 179c-6.6.8-9.4 8.9-4.7 13.6l43.9 43.9L600.3 370a8.03 8.03 0 0 0 0 11.3l42.4 42.4zM845 694.9c-.8-6.6-8.9-9.4-13.6-4.7l-43.7 43.7L654 600.3a8.03 8.03 0 0 0-11.3 0l-42.4 42.3a8.03 8.03 0 0 0 0 11.3L734 787.6l-43.9 43.9a8.01 8.01 0 0 0 4.7 13.6L855 864c5.1.6 9.5-3.7 8.9-8.9L845 694.9zm-463.7-94.6a8.03 8.03 0 0 0-11.3 0L236.3 733.9l-43.7-43.7a8.01 8.01 0 0 0-13.6 4.7L160.1 855c-.6 5.1 3.7 9.5 8.9 8.9L329.2 845c6.6-.8 9.4-8.9 4.7-13.6L290 787.6 423.7 654c3.1-3.1 3.1-8.2 0-11.3l-42.4-42.4z" /> </symbol> <symbol id="mono-icon-tumblr" viewBox="0 0 512 512"> <g> <path d="M321.2,396.3c-11.8,0-22.4-2.8-31.5-8.3c-6.9-4.1-11.5-9.6-14-16.4c-2.6-6.9-3.6-22.3-3.6-46.4V224h96v-64h-96V48h-61.9 c-2.7,21.5-7.5,44.7-14.5,58.6c-7,13.9-14,25.8-25.6,35.7c-11.6,9.9-25.6,17.9-41.9,23.3V224h48v140.4c0,19,2,33.5,5.9,43.5 c4,10,11.1,19.5,21.4,28.4c10.3,8.9,22.8,15.7,37.3,20.5c14.6,4.8,31.4,7.2,50.4,7.2c16.7,0,30.3-1.7,44.7-5.1 c14.4-3.4,30.5-9.3,48.2-17.6v-65.6C363.2,389.4,342.3,396.3,321.2,396.3z"></path> </g> </symbol> <symbol id="mono-icon-twitter" viewBox="0 0 19 16"> <g> <path fill-rule="evenodd" d="M16.48 4.594c.024.094.036.246.036.457 0 1.687-.41 3.316-1.23 4.887-.845 1.664-2.028 2.988-3.552 3.972-1.664 1.102-3.574 1.652-5.73 1.652-2.063 0-3.95-.55-5.66-1.652.258.024.55.035.879.035 1.71 0 3.246-.527 4.605-1.582-.82 0-1.54-.24-2.162-.72a3.671 3.671 0 0 1-1.283-1.811c.234.023.457.035.668.035.328 0 .656-.035.984-.105a3.72 3.72 0 0 1-2.11-1.301 3.54 3.54 0 0 1-.843-2.32v-.07a3.475 3.475 0 0 0 1.652.492A4.095 4.095 0 0 1 1.54 5.227a3.546 3.546 0 0 1-.457-1.776c0-.644.176-1.26.527-1.845a10.18 10.18 0 0 0 3.358 2.742 10.219 10.219 0 0 0 4.236 1.125 5.124 5.124 0 0 1-.07-.844c0-.656.164-1.272.492-1.846a3.613 3.613 0 0 1 1.336-1.353 3.565 3.565 0 0 1 1.828-.492c.54 0 1.037.105 1.494.316.457.21.861.492 1.213.844a7.51 7.51 0 0 0 2.356-.88c-.282.868-.82 1.548-1.618 2.04a7.93 7.93 0 0 0 2.11-.598 8.114 8.114 0 0 1-1.864 1.934z"> </path> </g> </symbol> <symbol id="action-icon-twitter" viewBox="0 0 30 30" xmlns="http://www.w3.org/2000/svg"> <g stroke="none" stroke-width="1" fill="none" fill-rule="evenodd"> <g transform="translate(-170.000000, -526.000000)"> <g transform="translate(100.000000, 526.000000)"> <g transform="translate(70.000000, 0.000000)"> <circle fill="#1DA1F2" cx="15" cy="15" r="15" /> <g transform="translate(9.000000, 10.000000)" fill="#FFFFFF"> <path d="M10.5859761,1.45966124 C11.0943048,1.17128539 11.4846922,0.714562198 11.6684736,0.170424684 C11.1928015,0.43748706 10.6659497,0.631385646 10.1051245,0.735917217 C9.65605179,0.28301632 9.01621859,0 8.30813124,0 C6.94840338,0 5.84610995,1.04340456 5.84610995,2.33027296 C5.84610995,2.51295392 5.86788105,2.69077393 5.90984308,2.86132325 C3.86375508,2.76414545 2.04968726,1.83640698 0.835421925,0.42656027 C0.623504883,0.770733376 0.502095907,1.17103611 0.502095907,1.59809494 C0.502095907,2.40655275 0.936727752,3.119827 1.59732251,3.53774555 C1.19376715,3.52565545 0.814133655,3.42079151 0.482212224,3.2462537 C0.482036651,3.25597563 0.481992758,3.26573911 0.481992758,3.27554414 C0.481992758,4.40461806 1.33062658,5.34648241 2.45688577,5.56057271 C2.25027982,5.61379407 2.03278832,5.64225358 1.80827389,5.64225358 C1.64959947,5.64225358 1.49540217,5.62767068 1.34506749,5.60045757 C1.65837814,6.52628488 2.56758477,7.20000665 3.64490289,7.21886886 C2.80228245,7.84385631 1.74076594,8.21636429 0.587292878,8.21636429 C0.388587732,8.21636429 0.192603972,8.2053544 0,8.1838332 C1.08956436,8.84500785 2.38362778,9.23076923 3.7739493,9.23076923 C8.30238121,9.23076923 10.7786678,5.67993646 10.7786678,2.60049288 C10.7786678,2.49945124 10.7762537,2.3989497 10.7715571,2.29902981 C11.2525403,1.97047828 11.6699221,1.56003814 12,1.09272053 C11.5584769,1.27806049 11.0840338,1.40336542 10.5859761,1.45966124 Z" /> </g> </g> </g> </g> </g> </symbol> <symbol id="mono-icon-youtube" viewBox="0 0 512 512"> <g> <path fill-rule="evenodd" d="M508.6,148.8c0-45-33.1-81.2-74-81.2C379.2,65,322.7,64,265,64c-3,0-6,0-9,0s-6,0-9,0c-57.6,0-114.2,1-169.6,3.6 c-40.8,0-73.9,36.4-73.9,81.4C1,184.6-0.1,220.2,0,255.8C-0.1,291.4,1,327,3.4,362.7c0,45,33.1,81.5,73.9,81.5 c58.2,2.7,117.9,3.9,178.6,3.8c60.8,0.2,120.3-1,178.6-3.8c40.9,0,74-36.5,74-81.5c2.4-35.7,3.5-71.3,3.4-107 C512.1,220.1,511,184.5,508.6,148.8z M207,353.9V157.4l145,98.2L207,353.9z"></path> </g> </symbol> <symbol id="icon-magnify" viewBox="0 0 30 30" xmlns="http://www.w3.org/2000/svg"> <path d="M21.876 18.281l-.314.548L30 27.343 27.343 30l-8.437-8.516-.546.392c-2.083 1.3-4.245 1.951-6.486 1.951-3.28 0-6.08-1.171-8.398-3.515C1.16 17.968 0 15.156 0 11.874c0-3.28 1.159-6.08 3.476-8.398C5.794 1.16 8.595 0 11.876 0c3.282 0 6.079 1.159 8.398 3.476 2.318 2.318 3.477 5.119 3.477 8.4a11.786 11.786 0 0 1-1.875 6.405zm-3.36-13.046c-1.823-1.824-4.037-2.736-6.642-2.736-2.604 0-4.818.912-6.639 2.736-1.824 1.822-2.736 4.035-2.736 6.64 0 2.606.912 4.818 2.736 6.64 1.822 1.823 4.035 2.734 6.64 2.734 2.606 0 4.818-.91 6.64-2.733 1.876-1.875 2.813-4.088 2.813-6.642 0-2.55-.937-4.764-2.812-6.639z" fill-rule="evenodd" /> </symbol> <symbol id="burger-menu" viewBox="0 0 14 10"> <g> <path fill-rule="evenodd" d="M0 5.5v-1h14v1H0zM0 1V0h14v1H0zm0 9V9h14v1H0z"></path> </g> </symbol> <symbol id="close-x" viewBox="0 0 14 14" xmlns="http://www.w3.org/2000/svg"> <g> <path fill-rule="nonzero" d="M6.336 7L0 .664.664 0 7 6.336 13.336 0 14 .664 7.664 7 14 13.336l-.664.664L7 7.664.664 14 0 13.336 6.336 7z"> </path> </g> </symbol> <symbol id="share-more-arrow" viewBox="0 0 512 512" style="enable-background:new 0 0 512 512;"> <g> <g> <path d="M512,241.7L273.643,3.343v156.152c-71.41,3.744-138.015,33.337-188.958,84.28C30.075,298.384,0,370.991,0,448.222v60.436 l29.069-52.985c45.354-82.671,132.173-134.027,226.573-134.027c5.986,0,12.004,0.212,18.001,0.632v157.779L512,241.7z M255.642,290.666c-84.543,0-163.661,36.792-217.939,98.885c26.634-114.177,129.256-199.483,251.429-199.483h15.489V78.131 l163.568,163.568L304.621,405.267V294.531l-13.585-1.683C279.347,291.401,267.439,290.666,255.642,290.666z"> </path> </g> </g> </symbol> <symbol id="chevron" viewBox="0 0 100 100"> <g> <path d="M22.4566257,37.2056786 L-21.4456527,71.9511488 C-22.9248661,72.9681457 -24.9073712,72.5311671 -25.8758148,70.9765924 L-26.9788683,69.2027424 C-27.9450684,67.6481676 -27.5292733,65.5646602 -26.0500598,64.5484493 L20.154796,28.2208967 C21.5532435,27.2597011 23.3600078,27.2597011 24.759951,28.2208967 L71.0500598,64.4659264 C72.5292733,65.4829232 72.9450684,67.5672166 71.9788683,69.1217913 L70.8750669,70.8956413 C69.9073712,72.4502161 67.9241183,72.8848368 66.4449048,71.8694118 L22.4566257,37.2056786 Z" id="Transparent-Chevron" transform="translate(22.500000, 50.000000) rotate(90.000000) translate(-22.500000, -50.000000) "></path> </g> </symbol> <symbol id="chevron-down" viewBox="0 0 10 6" xmlns="http://www.w3.org/2000/svg"> <path d="M5.108 4.088L.965.158a.587.587 0 0 0-.8 0 .518.518 0 0 0 0 .758L5.13 5.625 9.845.905a.517.517 0 0 0-.021-.758.588.588 0 0 0-.8.02l-3.916 3.92z" fill-rule="evenodd" /> </symbol> <symbol id="chevron-up" viewBox="0 0 284.929 284.929" xmlns="http://www.w3.org/2000/svg"> <path d="M282.082 195.285L149.028 62.24c-1.901-1.903-4.088-2.856-6.562-2.856s-4.665.953-6.567 2.856L2.856 195.285C.95 197.191 0 199.378 0 201.853c0 2.474.953 4.664 2.856 6.566l14.272 14.271c1.903 1.903 4.093 2.854 6.567 2.854s4.664-.951 6.567-2.854l112.204-112.202 112.208 112.209c1.902 1.903 4.093 2.848 6.563 2.848 2.478 0 4.668-.951 6.57-2.848l14.274-14.277c1.902-1.902 2.847-4.093 2.847-6.566.001-2.476-.944-4.666-2.846-6.569z" /> </symbol> <symbol id="icon-arrow-down" viewBox="0 0 10 6" xmlns="http://www.w3.org/2000/svg"> <path d="M5.108 4.088L.965.158a.587.587 0 0 0-.8 0 .518.518 0 0 0 0 .758L5.13 5.625 9.845.905a.517.517 0 0 0-.021-.758.588.588 0 0 0-.8.02l-3.916 3.92z" fill="inherit" fill-rule="evenodd" /> </symbol> <symbol id="play-arrow" viewBox="0 0 30 30"> <g fill-rule="evenodd"> <path d="M9 9l12 6-12 6z"></path> </g> </symbol> <symbol id="play-icon" viewBox="0 0 50 50" xmlns="http://www.w3.org/2000/svg"> <g stroke="none" stroke-width="1" fill="none" fill-rule="evenodd"> <g transform="translate(-341.000000, -11574.000000)" fill-rule="nonzero"> <g transform="translate(320.000000, 11422.000000)"> <g transform="translate(21.000000, 152.000000)"> <circle id="Oval" fill="#FFFFFF" cx="25" cy="25" r="24" /> <path d="M25,0 C11.2,0 0,11.2 0,25 C0,38.8 11.2,50 25,50 C38.8,50 50,38.8 50,25 C50,11.2 38.8,0 25,0 Z" fill="#EE0120" /> <polygon fill="#FFFFFF" points="18 36.25 18 13.75 33 25" /> </g> </g> </g> </g> </symbol> <symbol id="grid" viewBox="0 0 32 32"> <g> <path d="M6.4,5.7 C6.4,6.166669 6.166669,6.4 5.7,6.4 L0.7,6.4 C0.233331,6.4 0,6.166669 0,5.7 L0,0.7 C0,0.233331 0.233331,0 0.7,0 L5.7,0 C6.166669,0 6.4,0.233331 6.4,0.7 L6.4,5.7 Z M19.2,5.7 C19.2,6.166669 18.966669,6.4 18.5,6.4 L13.5,6.4 C13.033331,6.4 12.8,6.166669 12.8,5.7 L12.8,0.7 C12.8,0.233331 13.033331,0 13.5,0 L18.5,0 C18.966669,0 19.2,0.233331 19.2,0.7 L19.2,5.7 Z M32,5.7 C32,6.166669 31.766669,6.4 31.3,6.4 L26.3,6.4 C25.833331,6.4 25.6,6.166669 25.6,5.7 L25.6,0.7 C25.6,0.233331 25.833331,0 26.3,0 L31.3,0 C31.766669,0 32,0.233331 32,0.7 L32,5.7 Z M6.4,18.5 C6.4,18.966669 6.166669,19.2 5.7,19.2 L0.7,19.2 C0.233331,19.2 0,18.966669 0,18.5 L0,13.5 C0,13.033331 0.233331,12.8 0.7,12.8 L5.7,12.8 C6.166669,12.8 6.4,13.033331 6.4,13.5 L6.4,18.5 Z M19.2,18.5 C19.2,18.966669 18.966669,19.2 18.5,19.2 L13.5,19.2 C13.033331,19.2 12.8,18.966669 12.8,18.5 L12.8,13.5 C12.8,13.033331 13.033331,12.8 13.5,12.8 L18.5,12.8 C18.966669,12.8 19.2,13.033331 19.2,13.5 L19.2,18.5 Z M32,18.5 C32,18.966669 31.766669,19.2 31.3,19.2 L26.3,19.2 C25.833331,19.2 25.6,18.966669 25.6,18.5 L25.6,13.5 C25.6,13.033331 25.833331,12.8 26.3,12.8 L31.3,12.8 C31.766669,12.8 32,13.033331 32,13.5 L32,18.5 Z M6.4,31.3 C6.4,31.766669 6.166669,32 5.7,32 L0.7,32 C0.233331,32 0,31.766669 0,31.3 L0,26.3 C0,25.833331 0.233331,25.6 0.7,25.6 L5.7,25.6 C6.166669,25.6 6.4,25.833331 6.4,26.3 L6.4,31.3 Z M19.2,31.3 C19.2,31.766669 18.966669,32 18.5,32 L13.5,32 C13.033331,32 12.8,31.766669 12.8,31.3 L12.8,26.3 C12.8,25.833331 13.033331,25.6 13.5,25.6 L18.5,25.6 C18.966669,25.6 19.2,25.833331 19.2,26.3 L19.2,31.3 Z M32,31.3 C32,31.766669 31.766669,32 31.3,32 L26.3,32 C25.833331,32 25.6,31.766669 25.6,31.3 L25.6,26.3 C25.6,25.833331 25.833331,25.6 26.3,25.6 L31.3,25.6 C31.766669,25.6 32,25.833331 32,26.3 L32,31.3 Z"></path> </g> </symbol> <symbol id="filter" viewBox="0 0 16 12"> <path d="M0 2V0h16v2H0zm2.4 5V5h11.2v2H2.4zm2.4 5v-2h6.4v2H4.8z"></path> </symbol> <symbol id="icon-filter" viewBox="0 0 512 512"> <path fill="currentColor" d="M487.976 0H24.028C2.71 0-8.047 25.866 7.058 40.971L192 225.941V432c0 7.831 3.821 15.17 10.237 19.662l80 55.98C298.02 518.69 320 507.493 320 487.98V225.941l184.947-184.97C520.021 25.896 509.338 0 487.976 0z"> </path> </symbol> <symbol id="icon-radio-on" viewBox="0 0 24 24"> <path fill="currentColor" fill-rule="nonzero" d="M12 7c-2.76 0-5 2.24-5 5s2.24 5 5 5 5-2.24 5-5-2.24-5-5-5zm0-5C6.48 2 2 6.48 2 12s4.48 10 10 10 10-4.48 10-10S17.52 2 12 2zm0 18c-4.42 0-8-3.58-8-8s3.58-8 8-8 8 3.58 8 8-3.58 8-8 8z" /> </symbol> <symbol id="icon-radio-off" viewBox="0 0 24 24"> <path fill="currentColor" fill-rule="nonzero" d="M12 2C6.48 2 2 6.48 2 12s4.48 10 10 10 10-4.48 10-10S17.52 2 12 2zm0 18c-4.42 0-8-3.58-8-8s3.58-8 8-8 8 3.58 8 8-3.58 8-8 8z" /> </symbol> <symbol id="icon-pause" viewBox="0 0 13 16"> <rect width="4.636" height="16" rx="2.318" /> <rect width="4.636" height="16" x="7.727" rx="2.318" /> </symbol> <symbol id="icon-plus" viewBox="0 0 14 14" height="14px" width="14px"> <path d="M13, 6H8V1A1, 1, 0, 0, 0, 6, 1V6H1A1, 1, 0, 0, 0, 1, 8H6v5a1, 1, 0, 0, 0, 2, 0V8h5a1, 1, 0, 0, 0, 0-2Z" /> </symbol> <symbol id="icon-minus" viewBox="0 0 14 14" height="14px" width="14px"> <path d="M13, 8H1A1, 1, 0, 0, 1, 1, 6H13a1, 1, 0, 0, 1, 0, 2Z" /> </symbol> <symbol id="amazon-alexa" fill="none" viewBox="0 0 14 14"> <path clip-rule="evenodd" d="m52.4998 105c-28.9946 0-52.4998-23.5055-52.4998-52.5003 0-26.6214 19.8151-48.61142 45.5003-52.03433v10.57883c0 2.9832-1.8814 5.6636-4.7091 6.6127-14.8197 4.9787-25.4216 19.1433-25.0311 35.7287.4754 20.1911 16.923 36.0663 37.1187 35.8628 20.122-.2035 36.3714-16.5785 36.3714-36.7487 0-.4597-.0113-.9172-.0283-1.3721-.004-.102-.0072-.2048-.0113-.307-.0194-.4228-.0452-.8446-.0784-1.2639-.0097-.1335-.0234-.2666-.0355-.3998-.0259-.2928-.0558-.5837-.0889-.8739-.0372-.3297-.08-.6576-.1261-.9838-.0202-.1492-.0404-.2988-.063-.4467-4.4995-29.6389-43.1025-46.296896-43.3144-46.388077 2.2893-.304462 4.6239-.464423 6.9955-.464423 28.9951 0 52.5002 23.5048 52.5002 52.4997 0 28.9948-23.5051 52.5003-52.5002 52.5003z" fill="#5fcaf4" fill-rule="evenodd" transform="matrix(.133333 0 0 -.133333 0 14)" /> </symbol> <symbol id="apple" viewBox="0 0 511.86 511.86"> <rect fill="#822cbe" width="511.86" height="511.86" rx="113.87" /> <path fill="#ffffff" d="M240.44,448.26c-15.27-5.46-18.54-12.88-24.81-56.38-7.28-50.55-8.87-81.84-4.65-91.75,5.59-13.14,20.78-20.6,42-20.68,21.07-.09,36.39,7.44,42,20.68,4.24,9.89,2.65,41.2-4.64,91.75-4.95,35.32-7.67,44.25-14.5,50.25-9.38,8.31-22.69,10.61-35.32,6.14Zm-65.22-51.87c-52.73-25.94-86.51-69.82-98.79-128.2-3.07-15.07-3.59-51-.69-64.84,7.68-37.11,22.36-66.13,46.75-91.79,35.15-37.06,80.37-56.65,130.7-56.65,49.82,0,94.86,19.23,129.16,55.18,26.1,27.13,40.77,55.84,48.28,93.67,2.56,12.59,2.56,46.92.17,61.08a181.4,181.4,0,0,1-69.1,113c-13,9.79-44.7,26.89-49.82,26.89-1.88,0-2.05-1.94-1.19-9.81,1.53-12.63,3.07-15.25,10.23-18.26,11.44-4.77,30.89-18.63,42.83-30.61a163,163,0,0,0,42.82-75.41c4.44-17.57,3.93-56.64-1-74.73C390,138.36,343.1,93.66,285.61,81.72c-16.73-3.42-47.1-3.42-64,0-58.18,11.94-106.29,58.86-121,117.89-3.92,16-3.92,55.11,0,71.15,9.73,39.07,35,74.9,68.08,96.23a147.45,147.45,0,0,0,17.58,10.07c7.16,3.07,8.7,5.63,10.06,18.25.85,7.68.68,9.9-1.19,9.9-1.2,0-9.9-3.75-19.11-8.19ZM175.9,327c-17.75-14.16-33.44-39.28-39.93-63.91-3.92-14.88-3.92-43.17.17-58,10.75-40.06,40.27-71.12,81.22-85.71,14-4.94,45-6,62.27-2.25C339,130.33,381.15,189.79,373,248.77c-3.24,23.77-11.43,43.29-25.93,61.42-7.17,9.16-24.57,24.54-27.64,24.54-.51,0-1-5.8-1-12.87V309l8.88-10.58c33.44-40.06,31.05-96-5.46-132.74-14.16-14.29-30.54-22.69-51.7-26.56-13.65-2.53-16.55-2.53-30.88-.17-21.76,3.55-38.61,12-53.58,26.78C148.94,202,146.55,258.29,180,298.38L188.8,309v13c0,7.17-.57,13-1.27,13s-5.63-3.41-10.92-7.68Zm59.2-69.66c-15.18-7.07-23.37-20.39-23.54-37.76,0-15.61,8.7-29.23,23.71-37.2,9.56-5,26.45-5,36,0a46.34,46.34,0,0,1,22.18,26.85c9.9,33.65-25.76,63.13-58,48.07Z" transform="translate(-0.14 -0.14)" /> <circle fill="#ffffff" cx="255.74" cy="221.24" r="42.55" /> <path fill="#ffffff" d="M260,280.42a83.61,83.61,0,0,1,14.25,2.9,51.55,51.55,0,0,1,14.26,7.13c3.76,2.73,6.49,5.56,8.19,9s2.56,7.54,3.07,14.25c.34,6.72.34,16-1.19,33.44s-4.61,42.79-7,59.08-4.09,23.55-6.31,28.84a25.82,25.82,0,0,1-18.94,16.55,41,41,0,0,1-9.89.85,42.41,42.41,0,0,1-9.9-.85c-3.92-.86-9-2.39-12.8-5.46-3.92-3.07-6.48-7.51-8.53-13.31s-3.58-12.8-5.63-26.79-4.78-34.8-6.48-50.67-2.39-26.79-2.53-34.13.17-11,1-14.5a28.93,28.93,0,0,1,4.09-9.38,31.14,31.14,0,0,1,6.83-7,27.87,27.87,0,0,1,8-4.3A60.8,60.8,0,0,1,241.35,283c4.43-.85,9.55-1.7,12.18-2.05a13.34,13.34,0,0,1,5.56,0Z" transform="translate(-0.14 -0.14)" /> </symbol> <symbol id="google" fill="none" viewBox="0 0 14 14"> <path d="m0 0v2.12148.84795 2.12148h1.75v-2.12148-.84795-2.12148z" fill="#fab908" transform="translate(6.125 4.45453)" /> <path d="m.875 1.75c.48325 0 .875-.39175.875-.875 0-.483249-.39175-.875-.875-.875-.483249 0-.875.391751-.875.875 0 .48325.391751.875.875.875z" fill="#0066d9" transform="translate(0 5.64764)" /> <path d="m.875 1.75c.48325 0 .875-.39175.875-.875 0-.483249-.39175-.875-.875-.875-.483249 0-.875.391751-.875.875 0 .48325.391751.875.875.875z" fill="#0066d9" transform="translate(0 6.60236)" /> <path d="m1.75 0h-1.75v.954546h1.75z" fill="#0066d9" transform="translate(0 6.52264)" /> <path d="m.875 1.75c.48325 0 .875-.39175.875-.875 0-.483249-.39175-.875-.875-.875-.483249 0-.875.391751-.875.875 0 .48325.391751.875.875.875z" fill="#4285f4" transform="translate(12.25 6.60236)" /> <path d="m.875 1.75c.48325 0 .875-.39175.875-.875 0-.483249-.39175-.875-.875-.875-.483249 0-.875.391751-.875.875 0 .48325.391751.875.875.875z" fill="#4285f4" transform="translate(12.25 5.64764)" /> <path d="m1.75 0h-1.75v.954546h1.75z" fill="#4285f4" transform="matrix(-1 0 0 -1 14 7.47736)" /> <path d="m.875 1.75c.48325 0 .875-.39175.875-.875 0-.483249-.39175-.875-.875-.875-.483249 0-.875.391751-.875.875 0 .48325.391751.875.875.875z" fill="#ea4335" transform="translate(3.02274 8.51141)" /> <path d="m.875 1.75c.48325 0 .875-.39175.875-.875 0-.483249-.39175-.875-.875-.875-.483249 0-.875.391751-.875.875 0 .48325.391751.875.875.875z" fill="#ea4335" transform="translate(3.02274 9.46594)" /> <path d="m1.75 0h-1.75v.954546h1.75z" fill="#ea4335" transform="translate(3.02274 9.38641)" /> <path d="m.875 1.75c.48325 0 .875-.39175.875-.875 0-.483249-.39175-.875-.875-.875-.483249 0-.875.391751-.875.875 0 .48325.391751.875.875.875z" fill="#ea4335" transform="translate(3.02274 2.78406)" /> <path d="m.875 1.75c.48325 0 .875-.39175.875-.875 0-.483249-.39175-.875-.875-.875-.483249 0-.875.391751-.875.875 0 .48325.391751.875.875.875z" fill="#ea4335" transform="translate(3.02274 5.88641)" /> <path d="m1.75 0h-1.75v3.12216h1.75z" fill="#ea4335" transform="translate(3.02274 3.65906)" /> <path d="m.875 1.75c.48325 0 .875-.39175.875-.875 0-.483249-.39175-.875-.875-.875-.483249 0-.875.391751-.875.875 0 .48325.391751.875.875.875z" fill="#34a853" transform="translate(9.22726 3.73859)" /> <path d="m.875 1.75c.48325 0 .875-.39175.875-.875 0-.483249-.39175-.875-.875-.875-.483249 0-.875.391751-.875.875 0 .48325.391751.875.875.875z" fill="#34a853" transform="translate(9.22726 2.78406)" /> <path d="m1.75 0h-1.75v.954546h1.75z" fill="#34a853" transform="matrix(-1 0 0 -1 10.9773 4.61359)" /> <g fill="#fab908"> <path d="m.875 1.75c.48325 0 .875-.39175.875-.875 0-.483249-.39175-.875-.875-.875-.483249 0-.875.391751-.875.875 0 .48325.391751.875.875.875z" transform="translate(6.125 .954529)" /> <path d="m.875 1.75c.48325 0 .875-.39175.875-.875 0-.483249-.39175-.875-.875-.875-.483249 0-.875.391751-.875.875 0 .48325.391751.875.875.875z" transform="translate(6.125)" /> <path d="m1.75 0h-1.75v.954546h1.75z" transform="matrix(-1 0 0 -1 7.875 1.82953)" /> <path d="m.875 1.75c.48325 0 .875-.39175.875-.875 0-.483249-.39175-.875-.875-.875-.483249 0-.875.391751-.875.875 0 .48325.391751.875.875.875z" transform="translate(6.125 12.25)" /> <path d="m.875 1.75c.48325 0 .875-.39175.875-.875 0-.483249-.39175-.875-.875-.875-.483249 0-.875.391751-.875.875 0 .48325.391751.875.875.875z" transform="translate(6.125 11.2955)" /> <path d="m1.75 0h-1.75v.954546h1.75z" transform="matrix(-1 0 0 -1 7.875 13.125)" /> </g> <path d="m.875 1.75c.48325 0 .875-.39175.875-.875 0-.483249-.39175-.875-.875-.875-.483249 0-.875.391751-.875.875 0 .48325.391751.875.875.875z" fill="#34a853" transform="translate(9.22726 9.46594)" /> <path d="m.875 1.75c.48325 0 .875-.39175.875-.875 0-.483249-.39175-.875-.875-.875-.483249 0-.875.391751-.875.875 0 .48325.391751.875.875.875z" fill="#34a853" transform="translate(9.22726 6.36359)" /> <path d="m1.75 0h-1.75v3.10227h1.75z" fill="#34a853" transform="matrix(-1 0 0 -1 10.9773 10.3409)" /> <path d="m.875 1.75c.48325 0 .875-.39175.875-.875 0-.483249-.39175-.875-.875-.875-.483249 0-.875.391751-.875.875 0 .48325.391751.875.875.875z" fill="#fab908" transform="translate(6.125 3.57953)" /> <path d="m.875 1.75c.48325 0 .875-.39175.875-.875 0-.483249-.39175-.875-.875-.875-.483249 0-.875.391751-.875.875 0 .48325.391751.875.875.875z" fill="#fab908" transform="translate(6.125 8.67047)" /> </symbol> <symbol id="npr-one" viewBox="0 0 15 16"> <g clip-rule="evenodd" fill-rule="evenodd"> <path d="m-.00000001 3.99903 3.11681001 1.9192c.6747-1.24356 1.93544-2.08032 3.37935-2.08032v-3.83791c-2.77571 0-5.1993 1.60843-6.49616001 3.99903z" fill="#d61900" transform="translate(1.00378)" /> <path d="m3.78823 5.28599c-.12325-.40495-.18984-.83705-.18984-1.28579 0-.75804.19011-1.4688.52217-2.081-1.53085-.942623-2.25153-1.386398-3.11681-1.91919993-.638301 1.17671993-1.00375 2.54287993-1.00375 4.00019993 0 1.45723.365449 2.82349 1.00375 4.0002l1.97257-1.21466z" fill="#f30" transform="translate(0 4.00128)" /> <path d="m-.00000001 1.21467c1.29686001 2.3906 3.72045001 3.99903 6.49616001 3.99903v-3.83801c-1.15017 0-2.18402-.530854-2.89814-1.37539816h-1.62563l.00018-.00029136z" fill="#83aeed" transform="translate(1.00378 10.7867)" /> <path d="m6.49625 1.9192-3.1168-1.91919949c-.67471 1.24355949-1.93535 2.08022949-3.37944981 2.08022949v3.838c2.77589981 0 5.19938981-1.60843 6.49624981-3.99903z" fill="#3266cc" transform="translate(7.50012 10.0768)" /> <path d="m.522174 4.0002c0 .75815-.190114 1.46881-.52217447 2.08101l3.11681047 1.91919c.63839-1.17671 1.00375-2.54297 1.00375-4.0002 0-1.45732-.36536-2.82348-1.00375-4.00019993-.86528.53280193-1.58596.97657693-3.11681047 1.91919993.33206047.6122.52217447 1.32296.52217447 2.081z" fill="#3a3a3a" transform="translate(10.8795 4.00128)" /> <path d="m.00000019 3.83791c1.44409981 0 2.70473981.83676 3.37944981 2.08032l3.1168-1.9192c-1.29686-2.3906-3.72044-3.99903-6.49624981-3.99903z" fill="#191919" transform="translate(7.50012)" /> </g> </symbol> <symbol id="overcast" viewBox="0 0 365 365" fill="none"> <g xmlns="http://www.w3.org/2000/svg" id="surface1"> <path style=" stroke:none;fill-rule:nonzero;fill:rgb(96.099854%,49.398804%,12.5%);fill-opacity:1;" d="M 90.179688 274.820313 C 66.550781 251.195313 51.9375 218.554688 51.9375 182.5 C 51.9375 146.445313 66.550781 113.804688 90.179688 90.179688 L 101.789063 101.789063 C 81.136719 122.445313 68.359375 150.980469 68.359375 182.5 C 68.359375 214.019531 81.136719 242.554688 101.789063 263.210938 Z M 125.890625 239.109375 C 111.402344 224.621094 102.441406 204.609375 102.441406 182.5 C 102.441406 160.390625 111.402344 140.378906 125.890625 125.890625 L 137.503906 137.503906 C 125.988281 149.019531 118.863281 164.925781 118.863281 182.5 C 118.863281 200.074219 125.988281 215.980469 137.503906 227.496094 Z M 227.496094 227.496094 C 239.011719 215.980469 246.136719 200.074219 246.136719 182.5 C 246.136719 164.925781 239.011719 149.019531 227.496094 137.503906 L 239.109375 125.890625 C 253.597656 140.378906 262.558594 160.390625 262.558594 182.5 C 262.558594 204.609375 253.597656 224.621094 239.109375 239.109375 Z M 274.820313 274.820313 L 263.210938 263.210938 C 283.863281 242.554688 296.640625 214.019531 296.640625 182.5 C 296.640625 150.980469 283.863281 122.445313 263.210938 101.789063 L 274.820313 90.179688 C 298.449219 113.804688 313.0625 146.445313 313.0625 182.5 C 313.0625 218.554688 298.449219 251.195313 274.820313 274.820313 Z M 182.5 16.640625 C 90.898438 16.640625 16.640625 90.898438 16.640625 182.5 C 16.640625 255.765625 64.148438 317.933594 130.039063 339.886719 L 172.632813 212.101563 C 160.238281 207.972656 151.296875 196.28125 151.296875 182.5 C 151.296875 165.265625 165.265625 151.296875 182.5 151.296875 C 199.734375 151.296875 213.703125 165.265625 213.703125 182.5 C 213.703125 196.28125 204.761719 207.972656 192.367188 212.101563 L 234.960938 339.886719 C 300.851563 317.933594 348.359375 255.765625 348.359375 182.5 C 348.359375 90.898438 274.101563 16.640625 182.5 16.640625 Z M 202.902344 347.109375 C 195.519531 341.375 188.371094 335.058594 181.539063 328.164063 C 174.933594 334.996094 168.046875 341.269531 160.945313 346.96875 C 168 347.882813 175.195313 348.359375 182.5 348.359375 C 189.40625 348.359375 196.214844 347.933594 202.902344 347.109375 Z M 191.792969 316.824219 C 200.890625 327.066406 210.65625 336.113281 220.847656 343.898438 C 221.527344 343.734375 222.203125 343.570313 222.878906 343.402344 L 205.261719 299.355469 C 200.972656 305.472656 196.472656 311.300781 191.792969 316.824219 Z M 182.257813 305.34375 C 187.945313 298.007813 193.300781 290.164063 198.257813 281.84375 L 182.5 242.453125 L 166.550781 282.324219 C 171.425781 290.464844 176.679688 298.148438 182.257813 305.34375 Z M 143.78125 343.804688 C 153.617188 336.28125 163.054688 327.59375 171.875 317.78125 C 167.367188 312.621094 163.03125 307.179688 158.894531 301.46875 L 142.121094 343.402344 C 142.671875 343.539063 143.226563 343.671875 143.78125 343.804688 Z M 182.5 364.796875 C 81.820313 364.796875 0.207031 283.179688 0.207031 182.5 C 0.207031 81.820313 81.820313 0.203125 182.5 0.203125 C 283.179688 0.203125 364.796875 81.820313 364.796875 182.5 C 364.796875 283.179688 283.179688 364.796875 182.5 364.796875 " /> <path style=" stroke:none;fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;" d="M 90.179688 274.820313 L 90.324219 274.675781 C 66.734375 251.085938 52.140625 218.496094 52.140625 182.5 C 52.140625 146.503906 66.730469 113.914063 90.324219 90.324219 L 90.179688 90.179688 L 90.03125 90.324219 L 101.644531 101.9375 L 101.789063 101.789063 L 101.644531 101.644531 C 80.953125 122.339844 68.15625 150.925781 68.15625 182.5 C 68.15625 214.074219 80.953125 242.660156 101.644531 263.355469 L 101.789063 263.210938 L 101.644531 263.0625 L 90.03125 274.675781 L 90.179688 274.820313 L 90.324219 274.675781 L 90.179688 274.820313 L 90.324219 274.96875 L 102.082031 263.210938 L 101.9375 263.0625 C 81.316406 242.445313 68.566406 213.964844 68.566406 182.5 C 68.566406 151.035156 81.316406 122.554688 101.9375 101.9375 L 102.082031 101.789063 L 90.179688 89.886719 L 90.03125 90.03125 C 66.367188 113.695313 51.730469 146.390625 51.730469 182.5 C 51.730469 218.609375 66.367188 251.304688 90.03125 274.96875 L 90.179688 275.113281 L 90.324219 274.96875 Z M 125.890625 239.109375 L 126.035156 238.964844 C 111.582031 224.515625 102.644531 204.550781 102.644531 182.5 C 102.644531 160.449219 111.582031 140.484375 126.035156 126.035156 L 125.890625 125.890625 L 125.742188 126.035156 L 137.355469 137.648438 L 137.503906 137.503906 L 137.355469 137.355469 C 125.804688 148.910156 118.660156 164.871094 118.660156 182.5 C 118.660156 200.128906 125.804688 216.089844 137.355469 227.644531 L 137.503906 227.496094 L 137.355469 227.351563 L 125.742188 238.964844 L 125.890625 239.109375 L 126.035156 238.964844 L 125.890625 239.109375 L 126.035156 239.257813 L 137.792969 227.496094 L 137.648438 227.351563 C 126.167969 215.871094 119.070313 200.015625 119.070313 182.5 C 119.070313 164.984375 126.167969 149.128906 137.648438 137.648438 L 137.792969 137.503906 L 125.890625 125.597656 L 125.742188 125.742188 C 111.21875 140.269531 102.234375 160.335938 102.234375 182.5 C 102.234375 204.664063 111.21875 224.730469 125.742188 239.257813 L 125.890625 239.402344 L 126.035156 239.257813 Z M 227.496094 227.496094 L 227.644531 227.644531 C 239.195313 216.089844 246.339844 200.128906 246.339844 182.5 C 246.339844 164.871094 239.195313 148.910156 227.644531 137.355469 L 227.496094 137.503906 L 227.644531 137.648438 L 239.257813 126.035156 L 239.109375 125.890625 L 238.964844 126.035156 C 253.417969 140.484375 262.355469 160.449219 262.355469 182.5 C 262.355469 204.550781 253.417969 224.515625 238.964844 238.964844 L 239.109375 239.109375 L 239.257813 238.964844 L 227.644531 227.351563 L 227.496094 227.496094 L 227.644531 227.644531 L 227.496094 227.496094 L 227.351563 227.644531 L 239.109375 239.402344 L 239.257813 239.257813 C 253.78125 224.730469 262.765625 204.664063 262.765625 182.5 C 262.765625 160.335938 253.78125 140.269531 239.257813 125.742188 L 239.109375 125.597656 L 227.207031 137.503906 L 227.351563 137.648438 C 238.832031 149.128906 245.929688 164.984375 245.929688 182.5 C 245.929688 200.015625 238.832031 215.871094 227.351563 227.351563 L 227.207031 227.496094 L 227.351563 227.644531 Z M 274.820313 274.820313 L 274.96875 274.675781 L 263.355469 263.0625 L 263.210938 263.210938 L 263.355469 263.355469 C 284.046875 242.660156 296.84375 214.074219 296.84375 182.5 C 296.84375 150.925781 284.046875 122.339844 263.355469 101.644531 L 263.210938 101.789063 L 263.355469 101.9375 L 274.96875 90.324219 L 274.820313 90.179688 L 274.675781 90.324219 C 298.265625 113.914063 312.855469 146.503906 312.855469 182.5 C 312.855469 218.496094 298.265625 251.085938 274.675781 274.675781 L 274.820313 274.820313 L 274.96875 274.675781 L 274.820313 274.820313 L 274.96875 274.96875 C 298.632813 251.304688 313.269531 218.609375 313.269531 182.5 C 313.269531 146.390625 298.632813 113.695313 274.96875 90.03125 L 274.820313 89.886719 L 262.917969 101.789063 L 263.0625 101.9375 C 283.683594 122.554688 296.433594 151.035156 296.433594 182.5 C 296.433594 213.964844 283.683594 242.445313 263.0625 263.0625 L 262.917969 263.210938 L 274.820313 275.113281 L 274.96875 274.96875 Z M 182.5 16.640625 L 182.5 16.433594 C 90.785156 16.433594 16.433594 90.785156 16.433594 182.5 C 16.433594 255.859375 64.003906 318.097656 129.972656 340.082031 L 130.167969 340.144531 L 172.894531 211.972656 L 172.699219 211.90625 C 160.382813 207.804688 151.503906 196.191406 151.503906 182.5 C 151.503906 173.941406 154.972656 166.191406 160.582031 160.582031 C 166.191406 154.972656 173.9375 151.503906 182.5 151.503906 C 191.058594 151.503906 198.808594 154.972656 204.417969 160.582031 C 210.027344 166.191406 213.496094 173.9375 213.496094 182.5 C 213.496094 196.191406 204.617188 207.804688 192.300781 211.90625 L 192.105469 211.972656 L 234.832031 340.144531 L 235.027344 340.082031 C 301 318.097656 348.566406 255.859375 348.566406 182.5 C 348.566406 90.785156 274.214844 16.433594 182.5 16.433594 L 182.5 16.84375 C 228.246094 16.84375 269.65625 35.386719 299.636719 65.363281 C 329.613281 95.34375 348.15625 136.753906 348.15625 182.5 C 348.15625 255.675781 300.703125 317.765625 234.898438 339.691406 L 234.960938 339.886719 L 235.15625 339.820313 L 192.5625 212.035156 L 192.367188 212.101563 L 192.433594 212.296875 C 204.910156 208.140625 213.90625 196.375 213.910156 182.5 C 213.90625 165.152344 199.847656 151.09375 182.5 151.089844 C 165.152344 151.09375 151.09375 165.152344 151.089844 182.5 C 151.09375 196.375 160.089844 208.140625 172.566406 212.296875 L 172.632813 212.101563 L 172.4375 212.035156 L 129.84375 339.820313 L 130.039063 339.886719 L 130.101563 339.691406 C 64.296875 317.765625 16.84375 255.675781 16.84375 182.5 C 16.84375 136.753906 35.386719 95.34375 65.363281 65.363281 C 95.34375 35.386719 136.753906 16.84375 182.5 16.84375 Z M 202.902344 347.109375 L 203.027344 346.949219 C 195.652344 341.21875 188.511719 334.90625 181.683594 328.019531 L 181.535156 327.867188 L 181.390625 328.019531 C 174.792969 334.847656 167.910156 341.113281 160.816406 346.808594 L 160.441406 347.109375 L 160.917969 347.171875 C 167.984375 348.089844 175.1875 348.566406 182.5 348.566406 C 189.414063 348.566406 196.230469 348.136719 202.925781 347.316406 L 203.417969 347.253906 L 203.027344 346.949219 L 202.902344 347.109375 L 202.875 346.90625 C 196.199219 347.726563 189.398438 348.15625 182.5 348.15625 C 175.203125 348.15625 168.019531 347.679688 160.972656 346.765625 L 160.945313 346.96875 L 161.074219 347.128906 C 168.179688 341.421875 175.078125 335.144531 181.6875 328.304688 L 181.539063 328.164063 L 181.394531 328.308594 C 188.230469 335.210938 195.386719 341.53125 202.773438 347.273438 L 202.902344 347.109375 L 202.875 346.90625 Z M 191.792969 316.824219 L 191.636719 316.964844 C 200.742188 327.210938 210.519531 336.269531 220.722656 344.058594 L 220.800781 344.117188 L 220.894531 344.097656 C 221.574219 343.9375 222.253906 343.769531 222.929688 343.601563 L 223.15625 343.542969 L 205.3125 298.929688 L 205.09375 299.238281 C 200.808594 305.347656 196.3125 311.171875 191.636719 316.695313 L 191.519531 316.832031 L 191.636719 316.964844 L 191.792969 316.824219 L 191.949219 316.960938 C 196.632813 311.429688 201.136719 305.59375 205.429688 299.472656 L 205.261719 299.355469 L 205.070313 299.433594 L 222.6875 343.476563 L 222.878906 343.402344 L 222.828125 343.203125 C 222.152344 343.371094 221.476563 343.535156 220.800781 343.695313 L 220.847656 343.898438 L 220.972656 343.734375 C 210.789063 335.957031 201.035156 326.917969 191.945313 316.691406 L 191.792969 316.824219 L 191.949219 316.960938 Z M 182.257813 305.34375 L 182.421875 305.46875 C 188.113281 298.128906 193.472656 290.273438 198.433594 281.945313 L 198.484375 281.859375 L 182.5 241.898438 L 166.320313 282.34375 L 166.375 282.429688 C 171.253906 290.578125 176.511719 298.269531 182.097656 305.46875 L 182.257813 305.679688 L 182.421875 305.46875 L 182.257813 305.34375 L 182.421875 305.21875 C 176.847656 298.03125 171.597656 290.351563 166.726563 282.21875 L 166.550781 282.324219 L 166.742188 282.402344 L 182.5 243.003906 L 198.066406 281.917969 L 198.257813 281.84375 L 198.078125 281.738281 C 193.128906 290.050781 187.777344 297.890625 182.097656 305.21875 L 182.257813 305.34375 L 182.421875 305.21875 Z M 143.78125 343.804688 L 143.90625 343.96875 C 153.75 336.4375 163.199219 327.738281 172.027344 317.917969 L 172.152344 317.785156 L 172.03125 317.648438 C 167.527344 312.492188 163.195313 307.054688 159.058594 301.351563 L 158.839844 301.046875 L 141.84375 343.542969 L 142.070313 343.601563 C 142.625 343.738281 143.179688 343.871094 143.734375 344.007813 L 143.828125 344.027344 L 143.90625 343.96875 L 143.78125 343.804688 L 143.828125 343.605469 C 143.273438 343.472656 142.722656 343.339844 142.171875 343.203125 L 142.121094 343.402344 L 142.3125 343.476563 L 159.085938 301.546875 L 158.894531 301.46875 L 158.726563 301.589844 C 162.871094 307.304688 167.207031 312.753906 171.722656 317.917969 L 171.875 317.78125 L 171.722656 317.644531 C 162.910156 327.445313 153.480469 336.128906 143.65625 343.644531 L 143.78125 343.804688 L 143.828125 343.605469 Z M 182.5 364.796875 L 182.5 364.589844 C 132.21875 364.589844 86.695313 344.210938 53.742188 311.257813 C 20.792969 278.304688 0.410156 232.78125 0.410156 182.5 C 0.410156 132.21875 20.792969 86.695313 53.742188 53.742188 C 86.695313 20.789063 132.21875 0.410156 182.5 0.410156 C 232.78125 0.410156 278.304688 20.789063 311.257813 53.742188 C 344.210938 86.695313 364.589844 132.21875 364.589844 182.5 C 364.589844 232.78125 344.210938 278.304688 311.257813 311.257813 C 278.304688 344.210938 232.78125 364.589844 182.5 364.589844 L 182.5 365 C 283.292969 365 365 283.292969 365 182.5 C 365 81.707031 283.292969 0 182.5 0 C 81.707031 0 0 81.707031 0 182.5 C 0 283.292969 81.707031 365 182.5 365 L 182.5 364.796875 " /> </g> </symbol> <symbol id="pandora" viewBox="0 0 180 200"> <g id="bg"> <rect fill="#FFFFFF" width="180" height="200" /> </g> <g id="icons"> <g> <path fill="#019FEE" d="M160.7,74.2c-0.9-8.8-3.4-17.1-8.8-24.4c-4.1-5.6-9.4-9.8-15.6-13c-6-3-12.3-5-19-5.9l0,0 c-3.9-0.6-7.8-0.9-11.8-0.9H43.8c-0.4,0-0.7,0.3-0.7,0.7l0,0c0,46.1,0,92.2,0,138.3v0.1c0,0.4,0.3,0.7,0.7,0.7l0,0 c12.3,0,24.7,0,37,0c3.3,0,6-2.7,6.1-5.9c0-0.3,0-0.6,0-0.7c0-8.6,0-17.1,0-25.5v-0.3c0-0.4,0.3-0.7,0.7-0.7c0,0,0,0,0.1,0 c3.1,0,6.4,0,9.6,0c2.4,0,4.9,0,7.3-0.3c7.8-0.6,15.7-2,23.1-5c2.3-0.9,4.6-2,6.9-3.1c9-4.9,15.8-11.7,20.6-20.8 c2.3-4.6,3.9-9.4,4.9-14.6c0.6-2.7,0.9-5.4,1-8.3c0-0.1,0-0.1,0-0.3s0-0.1,0-0.3v-7c0,0,0,0,0-0.1C160.9,76,160.8,75.1,160.7,74.2 z" /> </g> </g> </symbol> <symbol id="pocket-casts" viewBox="0 0 39 36" style="enable-background:new 0 0 39 36;"> <g> <path fill="#F44336" d="M18,36c9.9,0,18-8,18-18S28,0,18,0S0,8,0,18S8,36,18,36L18,36L18,36z" /> <path fill="#FFFFFF" d="M17.8,27.1c-4.8-0.2-8.7-4.1-8.7-9c0-5,4.1-9,9-9s8.9,3.9,9,8.7c0,0.2,0.1,0.3,0.3,0.3H30 c0.2,0,0.3-0.1,0.3-0.3C30.1,11.2,24.7,5.9,18,5.9c-6.8,0-12.3,5.5-12.3,12.3c0,6.7,5.3,12.1,11.9,12.3c0.2,0,0.3-0.1,0.3-0.3v-2.6 C18.1,27.2,17.9,27.1,17.8,27.1L17.8,27.1L17.8,27.1z M18.1,11c-3.9,0-7.1,3.2-7.1,7.1c0,3.8,3,6.9,6.8,7.1c0.2,0,0.3-0.1,0.3-0.3 v-2.1c0-0.2-0.1-0.3-0.3-0.3c-2.3-0.2-4.1-2.1-4.1-4.4c0-2.4,2-4.4,4.4-4.4c2.3,0,4.2,1.8,4.4,4.1c0,0.2,0.2,0.3,0.3,0.3h2.1 c0.2,0,0.3-0.2,0.3-0.3C25,14,21.9,11,18.1,11L18.1,11L18.1,11z" /> </g> </symbol> <symbol id="rss" viewBox="0 0 12 12"> <path d="m1.59148 8.7795c-.878186.0032-1.58780186.71645-1.58566708 1.5938.00213478.8773.71521308 1.5871 1.59340708 1.5861.87819-.0011 1.58954-.7126 1.58955-1.59-.0032-.879-.71739-1.58991-1.59729-1.5899zm-1.59147997-4.7118v2.32108c1.48808997.00154 2.91461997.59358 3.96575997 1.64589 1.05115 1.05231 1.64082 2.47873 1.63927 3.96533h2.32333c.00464-2.10318-.82881-4.12172-2.31634-5.60999-1.48753-1.48826-3.5068-2.32386-5.61201997-2.32231zm0-4.06189726v2.32108726c5.33125997.00952 9.65431997 4.31803 9.67666997 9.64411h2.32333c-.0096-6.61422-5.37938-11.97100707-11.99999997-11.9710001z" fill="#257BBC" /> </symbol> <symbol id="spotify" viewBox="0 0 170.1 170.1" style="enable-background:new 0 0 170.1 170.1;" xml:space="preserve"> <g> <g> <g> <path fill="#1ED760" d="M85,1.3C38.8,1.3,1.3,38.8,1.3,85c0,46.3,37.5,83.7,83.7,83.7c46.3,0,83.7-37.5,83.7-83.7 C168.8,38.8,131.3,1.3,85,1.3z M123.4,122.1c-1.5,2.5-4.7,3.2-7.2,1.7c-19.7-12-44.4-14.7-73.6-8.1c-2.8,0.6-5.6-1.1-6.2-3.9 c-0.6-2.8,1.1-5.6,3.9-6.2c31.9-7.3,59.3-4.2,81.3,9.3C124.2,116.4,124.9,119.6,123.4,122.1z M133.7,99.3c-1.9,3.1-5.9,4-9,2.2 c-22.5-13.8-56.8-17.8-83.4-9.8c-3.5,1-7.1-0.9-8.1-4.3c-1-3.5,0.9-7.1,4.4-8.1c30.4-9.2,68.2-4.8,94.1,11.1 C134.6,92.2,135.6,96.2,133.7,99.3z M134.6,75.5c-27-16-71.5-17.5-97.3-9.7c-4.1,1.3-8.5-1.1-9.8-5.2c-1.3-4.1,1.1-8.5,5.2-9.8 c29.6-9,78.8-7.2,109.8,11.2c3.7,2.2,4.9,7,2.7,10.7C143.1,76.5,138.3,77.7,134.6,75.5z" /> </g> </g> </g> </symbol> <symbol id="stitcher" viewBox="0 0 122.88 51.61" style="enable-background:new 0 0 122.88 51.61"> <g> <polygon fill="#3590C5" points="0,10.14 21.5,10.14 21.5,49.15 0,49.15 0,10.14" /> <polygon fill="#C53827" points="101.38,7.99 122.88,7.99 122.88,43.93 101.38,43.93 101.38,7.99" /> <polygon fill="#EC8B24" points="76.19,0 97.69,0 97.69,46.7 76.19,46.7 76.19,0" /> <polygon fill="#FBC618" points="51,6.76 72.5,6.76 72.5,51.61 51,51.61 51,6.76" /> <polygon fill="#612268" points="25.5,3.07 47,3.07 47,46.39 25.5,46.39 25.5,3.07" /> </g> </symbol> <symbol id="tuneIn" viewBox="56 56 200 88" fill="none"> <g id="Logos"> <g id="Logo__x2013__Horizontal_1A_12_"> <g id="Logo_2_"> <path fill="#1C203C" d="M95,96.8v4c0,0.4-0.4,0.8-0.8,0.8h-5c-0.2,0-0.4,0.2-0.4,0.4v21.2c0,0.4-0.4,0.8-0.8,0.8h-4 c-0.4,0-0.8-0.4-0.8-0.8V102c0-0.2-0.2-0.4-0.4-0.4h-5c-0.4,0-0.8-0.4-0.8-0.8v-4c0-0.4,0.4-0.8,0.8-0.8h16.4 C94.6,96,95,96.4,95,96.8z M117,96h-4c-0.4,0-0.8,0.4-0.8,0.8v18.8c0,1.8-1.4,3.2-3.9,3.2c-2.4,0-3.8-1.4-3.8-3.2V96.8 c0-0.4-0.4-0.8-0.8-0.8h-4c-0.4,0-0.8,0.4-0.8,0.8v18.8c0,4.8,4,8.8,9.4,8.8c5.5,0,9.5-4,9.5-8.8V96.8 C117.8,96.4,117.4,96,117,96z M139.6,96h-4c-0.4,0-0.8,0.4-0.8,0.8v15c0,0.2-0.3,0.3-0.4,0.1l-5.5-15.2c-0.1-0.4-0.4-0.7-1-0.7 h-4.8c-0.4,0-0.8,0.4-0.8,0.8v26.4c0,0.4,0.4,0.8,0.8,0.8h4c0.4,0,0.8-0.4,0.8-0.8v-13.8c0-0.2,0.3-0.3,0.4-0.1l5.4,14 c0.1,0.3,0.4,0.7,1.1,0.7h4.7c0.4,0,0.8-0.4,0.8-0.8V96.8C140.4,96.4,140,96,139.6,96z M160.3,118.4h-9.5c-0.2,0-0.4-0.2-0.4-0.4 v-4.8c0-0.2,0.2-0.4,0.4-0.4h9c0.4,0,0.8-0.4,0.8-0.8v-4c0-0.4-0.4-0.8-0.8-0.8h-9c-0.2,0-0.4-0.2-0.4-0.4V102 c0-0.2,0.2-0.4,0.4-0.4h9.3c0.4,0,0.8-0.4,0.8-0.8v-4c0-0.4-0.4-0.8-0.8-0.8h-14.5c-0.4,0-0.8,0.4-0.8,0.8v26.4 c0,0.4,0.4,0.8,0.8,0.8h14.7c0.4,0,0.8-0.4,0.8-0.8v-4C161.1,118.8,160.7,118.4,160.3,118.4z M256,59v62c0,1.7-1.3,3-3,3h-69v17 c0,1.7-1.3,3-3,3H59c-1.7,0-3-1.3-3-3V79c0-1.7,1.3-3,3-3h119V59c0-1.7,1.3-3,3-3h72C254.7,56,256,57.3,256,59z M178,82H62.5 c-0.3,0-0.5,0.2-0.5,0.5v55c0,0.3,0.2,0.5,0.5,0.5h115c0.3,0,0.5-0.2,0.5-0.5V82z M213.8,99.4c0-0.4-0.4-0.8-0.8-0.8h-3.4 c-0.2,0-0.4-0.2-0.4-0.4V81.8c0-0.2,0.2-0.4,0.4-0.4h3c0.4,0,0.8-0.4,0.8-0.8v-3.8c0-0.4-0.4-0.8-0.8-0.8h-12.4 c-0.4,0-0.8,0.4-0.8,0.8v3.8c0,0.4,0.4,0.8,0.8,0.8h3c0.2,0,0.4,0.2,0.4,0.4v16.4c0,0.2-0.2,0.4-0.4,0.4h-3.4 c-0.4,0-0.8,0.4-0.8,0.8v3.8c0,0.4,0.4,0.8,0.8,0.8H213c0.4,0,0.8-0.4,0.8-0.8V99.4z M236.2,76.8c0-0.4-0.4-0.8-0.8-0.8h-4 c-0.4,0-0.8,0.4-0.8,0.8v15c0,0.2-0.3,0.3-0.4,0.1l-5.5-15.2c-0.1-0.4-0.4-0.7-1-0.7H219c-0.4,0-0.8,0.4-0.8,0.8v26.4 c0,0.4,0.4,0.8,0.8,0.8h4c0.4,0,0.8-0.4,0.8-0.8V89.4c0-0.2,0.3-0.3,0.4-0.1l5.4,14c0.1,0.3,0.4,0.7,1.1,0.7h4.7 c0.4,0,0.8-0.4,0.8-0.8V76.8z" /> </g> </g> </g> </symbol> <symbol id="check" viewBox="0 0 22 16" xmlns="http://www.w3.org/2000/svg"> <path d="M1 7.5 7.5 14l13-13" stroke-width="2" fill="none" fill-rule="evenodd" /> </symbol> <symbol id="calloutbox-icon-check" viewBox="0 0 22 16"> <path d="M1 7.5 7.5 14l13-13" stroke="#000" stroke-width="2" fill="none" fill-rule="evenodd" /> </symbol> <symbol id="calloutbox-icon-alert" viewBox="0 0 24 21"> <path d="M12 8.923v5.385m0 1.538v1.539M12 2l10 17.69H2L12 2z" stroke="#000" stroke-width="1.5" fill="none" fill-rule="evenodd" /> </symbol> <symbol id="calloutbox-icon-info" viewBox="0 0 22 22"> <path d="M11 6.385V4.846m0 12.308v-9.23M21 11c0 5.523-4.477 10-10 10S1 16.523 1 11 5.477 1 11 1s10 4.477 10 10z" stroke="#000" stroke-width="1.5" fill="none" fill-rule="evenodd" /> </symbol> <symbol id="icon-arrow-right" viewBox="0 0 15 11"> <g stroke="#74B900" stroke-width="2" fill="none" fill-rule="evenodd"> <path d="M13.232 5.504H.696M8.893 9.504l4.34-4-4.34-4" /> </g> </symbol> <symbol id="caret-up" viewBox="0 0 25 25"> <path d="m15.792 16.01 1.468-1.52-5.223-5.11-5.323 5.11 1.485 1.528 3.843-3.73z" fill="#000" fill-rule="nonzero" /> </symbol> <symbol id="caret-right" viewBox="0 0 25 25"> <path d="m15.792 16.01 1.468-1.52-5.223-5.11-5.323 5.11 1.485 1.528 3.843-3.73z" fill-rule="nonzero" transform="translate(25, 0) rotate(90)" /> </symbol> </svg> <bsp-header class="Page-header"> <div class="Page-header-hamburger-menu"> <div class="Page-header-hamburger-menu-wrapper"> <div class="Page-header-hamburger-menu-content"> <bsp-search class="Page-search" > <form action="https://docs.nvidia.com/search/index.html" method="GET"> <input type="text" name="term" placeholder="Search all documentation"> <input type="hidden" name="facet.subcollection[]" value="Technical Documentation"> <input type="hidden" name="facet.mimetype[]" value="webpage"> <button type="submit" aria-label="Submit Search"><span class="sr-only">Submit Search</span><svg height="12px" width="12px" viewBox="0 0 12 12"><use xlink:href="#icon-magnify"></use></svg></button></form></bsp-search> <nav class="Navigation"> <ul class="Navigation-items"> <li class="Navigation-items-item"><div class="NavigationItem"> <div class="NavigationItem-text"> <a href="https://developer.nvidia.com/" target="_blank" data-cms-ai="0">NVIDIA Developer</a> </div> </div> </li> <li class="Navigation-items-item"><div class="NavigationItem"> <div class="NavigationItem-text"> <a href="https://developer.nvidia.com/blog/" target="_blank" data-cms-ai="0">Blog</a> </div> </div> </li> <li class="Navigation-items-item"><div class="NavigationItem"> <div class="NavigationItem-text"> <a href="https://forums.developer.nvidia.com/" target="_blank" data-cms-ai="0">Forums</a> </div> </div> </li> <li class="Navigation-items-item"><div class="NavigationItem"> <div class="NavigationItem-text"> <a class="Button" data-theme="secondary" data-size="small" style="--button-border-radius: 0px" href="https://docs.nvidia.com/login" rel="nofollow noopener" data-cms-ai="0">Join</a> </div> </div> </li> </ul> </nav> </div> </div> </div> <div class="Page-header-bar"> <div class="Page-header-bar-logo"> <a aria-label="home page" href="/" data-cms-ai="0" > <img class="PageLogo-image" src="https://docscontent.nvidia.com/bf/6f/f2d5da4743aebb3dff0e6a6129ec/nvidia-docshub-logo-2.svg" alt="NVIDIA Documentation Home" width="280" > </a> <bsp-search class="Page-search" > <form action="https://docs.nvidia.com/search/index.html" method="GET"> <input type="text" name="term" placeholder="Search all documentation"> <input type="hidden" name="facet.subcollection[]" value="Technical Documentation"> <input type="hidden" name="facet.mimetype[]" value="webpage"> <button type="submit" aria-label="Submit Search"><span class="sr-only">Submit Search</span><svg height="12px" width="12px" viewBox="0 0 12 12"><use xlink:href="#icon-magnify"></use></svg></button></form></bsp-search> </div> <div class="Page-header-navigation"> <nav class="Navigation"> <ul class="Navigation-items"> <li class="Navigation-items-item"><div class="NavigationItem"> <div class="NavigationItem-text"> <a href="https://developer.nvidia.com/" target="_blank" data-cms-ai="0">NVIDIA Developer</a> </div> </div> </li> <li class="Navigation-items-item"><div class="NavigationItem"> <div class="NavigationItem-text"> <a href="https://developer.nvidia.com/blog/" target="_blank" data-cms-ai="0">Blog</a> </div> </div> </li> <li class="Navigation-items-item"><div class="NavigationItem"> <div class="NavigationItem-text"> <a href="https://forums.developer.nvidia.com/" target="_blank" data-cms-ai="0">Forums</a> </div> </div> </li> <li class="Navigation-items-item"><div class="NavigationItem"> <div class="NavigationItem-text"> <a class="Button" data-theme="secondary" data-size="small" style="--button-border-radius: 0px" href="https://docs.nvidia.com/login" rel="nofollow noopener" data-cms-ai="0">Join</a> </div> </div> </li> </ul> </nav> </div> <div class="Page-header-end"><button class="Page-header-menu-trigger" aria-expanded="false"><svg class="burger-menu"><use xlink:href="#burger-menu"></use></svg><svg class="close-x"><use xlink:href="#close-x"></use></svg><span class="label">Menu</span></button></div> </div> </bsp-header> <bsp-backtotop> <a href="#" class="Page-BackToTop" data-cms-ai="0"> <svg> <use href="#back-to-top-icon"></use> </svg> </a> </bsp-backtotop> <div class="Page-content"> <div class="PageHeading"> <div class="PageHeading-content-wrapper"> <div class="PageHeading-content"> <div class="PageHeading-title">DOCA Documentation v2.9.0</div> <bsp-search class="Page-search" data-search-header> <form action="https://docs.nvidia.com/search/index.html" method="GET" id="searchForm" onsubmit="InbookSearch(event)"> <input type="text" name="term" placeholder="Search DOCA Documentation v2.9.0"> <input type="hidden" name="facet.docs_book[]" value="DOCA Documentation v2.9.0"> <input type="hidden" name="facet.docs_version[]" value="DOCA SDK 2.9.0"> <input type="hidden" name="facet.subcollection[]" value="Technical Documentation"> <input type="hidden" name="facet.mimetype[]" value="webpage"> <button type="submit" aria-label="Submit Search"><span class="sr-only">Submit Search</span><svg height="12px" width="12px" viewBox="0 0 12 12"><use xlink:href="#icon-magnify"></use></svg></button><div id="nvidia-in-book-search-page-widget"></div></form><script> document.addEventListener("DOMContentLoaded", () => { if (!document.querySelector('script[src="https://images.nvidia.com/search-in-book/nvidia-in-book-search-page-widget.js"]')) { var script = document.createElement('script'); script.src = 'https://images.nvidia.com/search-in-book/nvidia-in-book-search-page-widget.js'; document.head.appendChild(script); } }) </script> <script> window.addEventListener("load", () => { NvidiaInBookSearchPageWidget.mount({ apiUrl: 'https://api-prod.nvidia.com/search/graphql', site: 'https://docs.nvidia.com', language: 'en-us', path: '/en-us/search/', searchParam: '', widgetPlacementId: '', queryParam: 'q', facets: 'mimetype, subcollection', facetValues: [], inBookSearch: true, modalPosition: {} }); }); function InbookSearch(event) { event.preventDefault(); NvidiaInBookSearchPageWidget.unmount(); NvidiaInBookSearchPageWidget.mount({ apiUrl: 'https://api-prod.nvidia.com/search/graphql', site: 'https://docs.nvidia.com', language: 'en-us', path: '/en-us/search/', searchParam: '', widgetPlacementId: '', queryParam: 'q', facets: 'mimetype, subcollection', facetValues: [], inBookSearch: true, modalPosition: {} }); } </script> </bsp-search> </div> </div> </div> <div class="Page-twoColumn" data-aside-left> <main class="Page-main" data-module > <bsp-search class="Page-search" mobile-only> <form action="https://docs.nvidia.com/search/index.html" method="GET" id="searchFormMob" onsubmit="InbookSearchMob(event)"> <input type="text" name="term" placeholder="Search DOCA Documentation v2.9.0"> <input type="hidden" name="facet.docs_book[]" value="DOCA Documentation v2.9.0"> <input type="hidden" name="facet.docs_version[]" value="DOCA SDK 2.9.0"> <input type="hidden" name="facet.subcollection[]" value="Technical Documentation"> <input type="hidden" name="facet.mimetype[]" value="webpage"> <button type="submit" aria-label="Submit Search"><span class="sr-only">Submit Search</span><svg height="12px" width="12px" viewBox="0 0 12 12"><use xlink:href="#icon-magnify"></use></svg></button><div id="nvidia-in-book-search-page-widget"></div></form><script> document.addEventListener("DOMContentLoaded", () => { if (!document.querySelector('script[src="https://images.nvidia.com/search-in-book/nvidia-in-book-search-page-widget.js"]')) { var script = document.createElement('script'); script.src = 'https://images.nvidia.com/search-in-book/nvidia-in-book-search-page-widget.js'; document.head.appendChild(script); } }) </script> <script> window.addEventListener("load", () => { NvidiaInBookSearchPageWidget.mount({ apiUrl: 'https://api-prod.nvidia.com/search/graphql', site: 'https://docs.nvidia.com', language: 'en-us', path: '/en-us/search/', searchParam: '', widgetPlacementId: '', queryParam: 'q', facets: 'mimetype, subcollection', facetValues: [], inBookSearch: true, modalPosition: {} }); }); function InbookSearchMob(event) { event.preventDefault(); NvidiaInBookSearchPageWidget.unmount(); NvidiaInBookSearchPageWidget.mount({ apiUrl: 'https://api-prod.nvidia.com/search/graphql', site: 'https://docs.nvidia.com', language: 'en-us', path: '/en-us/search/', searchParam: '', widgetPlacementId: '', queryParam: 'q', facets: 'mimetype, subcollection', facetValues: [], inBookSearch: true, modalPosition: {} }); } </script></bsp-search> <div class="Page-breadcrumbs"><a class="Link" href="https://docs.nvidia.com/" data-cms-ai="0" >NVIDIA Docs Hub</a> <svg class="chevron"><use xlink:href="#chevron"></use></svg> <a class="Link" href="https://docs.nvidia.com/networking/index.html" data-cms-ai="0" >NVIDIA Networking</a> <svg class="chevron"><use xlink:href="#chevron"></use></svg> <a class="Link" href="https://docs.nvidia.com/networking/dpu-doca/index.html" data-cms-ai="0" >BlueField DPUs / SuperNICs & DOCA</a> <svg class="chevron"><use xlink:href="#chevron"></use></svg> <a class="Link" href="https://docs.nvidia.com/doca/sdk/index.html" data-cms-ai="0" >DOCA Documentation v2.9.0</a> <svg class="chevron"><use xlink:href="#chevron"></use></svg> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+GPUNetIO/index.html" data-cms-ai="0" >DOCA GPUNetIO</a><hr class="breadcrumb-divider"/> </div> <div class="TopicPage-actions"> <span class="TopicPage-version">DOCA SDK 2.9.0<span class="TopicPage-separator"></span></span> <a class="Link" href="https://docs.nvidia.com/doca/sdk/doca-gpunetio.pdf" data-cms-ai="0" >Download PDF</a> </div> <div class="Page-toc"> <h4 class="Page-toc-heading">On This Page</h4> <ul> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Introduction" data-cms-ai="0" >Introduction</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ChangesFromPreviousReleases" data-cms-ai="0" >Changes From Previous Releases</a> <ul> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Changesin2.9.0" data-cms-ai="0" >Changes in 2.9.0</a> <ul> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ChangedFunctions" data-cms-ai="0" >Changed Functions</a></li> </ul></li> </ul></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-SystemConfiguration" data-cms-ai="0" >System Configuration</a> <ul> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ApplicationonHostCPU" data-cms-ai="0" >Application on Host CPU</a> <ul> <li><a class="Link" href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1PcHRpb24xOkNvbm5lY3RYQWRhcHRlcmluRXRoZXJuZXRNb2Rl" data-cms-ai="0" >Option 1: ConnectX Adapter in Ethernet Mode</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Option2" data-cms-ai="0" >Option 2: DPU Converged Accelerator in NIC mode</a></li> </ul></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ApplicationonDPUConvergedArmCPU" data-cms-ai="0" >Application on BlueField Converged Arm CPU</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-PCIeConfiguration" data-cms-ai="0" >PCIe Configuration</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Hugepages" data-cms-ai="0" >Hugepages</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-GPUConfiguration" data-cms-ai="0" >GPU Configuration</a> <ul> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-GDRCopy" data-cms-ai="0" >GDRCopy</a></li> <li><a class="Link" href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1HUFVNZW1vcnlNYXBwaW5nKG52aWRpYS1wZWVybWVtdnMuZG1hYnVmKQ" data-cms-ai="0" >GPU Memory Mapping (nvidia-peermem vs. dmabuf)</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-GPUBAR1Size" data-cms-ai="0" >GPU BAR1 Size</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-BlueField-3SpecificConfiguration" data-cms-ai="0" >BlueField-3 Specific Configuration</a></li> </ul></li> </ul></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Architecture" data-cms-ai="0" >Architecture</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-API" data-cms-ai="0" >API</a> <ul> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-CPUFunctions" data-cms-ai="0" >CPU Functions</a> <ul> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_mem_type" data-cms-ai="0" >doca_gpu_mem_type</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_create" data-cms-ai="0" >doca_gpu_create</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_mem_alloc" data-cms-ai="0" >doca_gpu_mem_alloc</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_create" data-cms-ai="0" >doca_gpu_semaphore_create</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_set_memory_type" data-cms-ai="0" >doca_gpu_semaphore_set_memory_type</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_set_items_num" data-cms-ai="0" >doca_gpu_semaphore_set_items_num</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_set_custom_info" data-cms-ai="0" >doca_gpu_semaphore_set_custom_info</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_get_status" data-cms-ai="0" >doca_gpu_semaphore_get_status</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_get_custom_info_addr" data-cms-ai="0" >doca_gpu_semaphore_get_custom_info_addr</a></li> </ul></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-DOCAPE" data-cms-ai="0" >DOCA PE</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-StrongModevs.WeakMode" data-cms-ai="0" >Strong Mode vs. Weak Mode</a></li> <li><a class="Link" href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1HUFVGdW5jdGlvbnPigJNFdGhlcm5ldA" data-cms-ai="0" >GPU Functions – Ethernet</a> <ul> <li><a class="Link" href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfZXRoX3J4cV9yZWNlaXZlXyo" data-cms-ai="0" >doca_gpu_dev_eth_rxq_receive_*</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_send_flags" data-cms-ai="0" >doca_gpu_send_flags</a></li> <li><a class="Link" href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfZXRoX3R4cV9zZW5kXyo" data-cms-ai="0" >doca_gpu_dev_eth_txq_send_*</a></li> <li><a class="Link" href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfZXRoX3R4cV93YWl0Xyo" data-cms-ai="0" >doca_gpu_dev_eth_txq_wait_*</a></li> <li><a class="Link" href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfZXRoX3R4cV9jb21taXRfKg" data-cms-ai="0" >doca_gpu_dev_eth_txq_commit_*</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_dev_eth_txq_push" data-cms-ai="0" >doca_gpu_dev_eth_txq_push</a></li> </ul></li> <li><a class="Link" href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1HUFVGdW5jdGlvbnMtUkRNQUdQVUZ1bmN0aW9uc-KAk1JETUE" data-cms-ai="0" >GPU Functions – RDMA</a> <ul> <li><a class="Link" href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfcmRtYV93cml0ZV8q" data-cms-ai="0" >doca_gpu_dev_rdma_write_*</a></li> <li><a class="Link" href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfcmRtYV9yZWFkXyo" data-cms-ai="0" >doca_gpu_dev_rdma_read_*</a></li> <li><a class="Link" href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfcmRtYV9zZW5kXyo" data-cms-ai="0" >doca_gpu_dev_rdma_send_*</a></li> <li><a class="Link" href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfcmRtYV9jb21taXRfKg" data-cms-ai="0" >doca_gpu_dev_rdma_commit_*</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_dev_rdma_wait_all" data-cms-ai="0" >doca_gpu_dev_rdma_wait_all</a></li> <li><a class="Link" href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfcmRtYV9yZWN2Xyo" data-cms-ai="0" >doca_gpu_dev_rdma_recv_*</a></li> <li><a class="Link" href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfcmRtYV9yZWN2X2NvbW1pdF8q" data-cms-ai="0" >doca_gpu_dev_rdma_recv_commit_*</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_dev_rdma_recv_wait_all" data-cms-ai="0" >doca_gpu_dev_rdma_recv_wait_all</a></li> </ul></li> <li><a class="Link" href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1HUFVGdW5jdGlvbnMtRE1BR1BVRnVuY3Rpb25z4oCTRE1B" data-cms-ai="0" >GPU Functions – DMA</a> <ul> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_dev_dma_memcpy" data-cms-ai="0" >doca_gpu_dev_dma_memcpy</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_dev_dma_commit" data-cms-ai="0" >doca_gpu_dev_dma_commit</a></li> </ul></li> </ul></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-BuildingBlocks" data-cms-ai="0" >Building Blocks</a> <ul> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-InitializeGPUandNIC" data-cms-ai="0" >Initialize GPU and NIC</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Semaphore" data-cms-ai="0" >Semaphore</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-EthernetQueuewithGPUDataPath" data-cms-ai="0" >Ethernet Queue with GPU Data Path</a> <ul> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ReceiveQueue" data-cms-ai="0" >Receive Queue</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-SendQueue" data-cms-ai="0" >Send Queue</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ReceiveandProcess" data-cms-ai="0" >Receive and Process</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ProduceandSend" data-cms-ai="0" >Produce and Send</a></li> </ul></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-RDMAQueuewithGPUDataPath" data-cms-ai="0" >RDMA Queue with GPU Data Path</a> <ul> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-CUDAKernelforRDMAWrite" data-cms-ai="0" >CUDA Kernel for RDMA Write</a></li> </ul></li> </ul></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-GPUNetIOSamplesGPUNetIOSamples" data-cms-ai="0" >GPUNetIO Samples</a> <ul> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-EthernetSendWaitTime" data-cms-ai="0" >Ethernet Send Wait Time</a> <ul> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-SynchronizingClocks" data-cms-ai="0" >Synchronizing Clocks</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-RunningtheSample" data-cms-ai="0" >Running the Sample</a></li> </ul></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-EthernetSimpleReceive" data-cms-ai="0" >Ethernet Simple Receive</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-RDMAClientServer" data-cms-ai="0" >RDMA Client Server</a></li> <li><a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-GPUDMACopy" data-cms-ai="0" >GPU DMA Copy</a></li> </ul></li> </ul> </div> <div class="Page-articleBody"> <bsp-book-content> <main class="RichTextArticleBody RichTextBody cookies"> <h1><a href="#" data-cms-ai="0">DOCA GPUNetIO</a></h1> <div class="StepModule" data-module><span class="StepModule-anchor" id="6761412e-0289-426a-9858-6a61bc54aa7f"></span> <div class="StepModule-body RichTextBody"> <p>This document provides an overview and configuration instructions for DOCA GPUNetIO API.</p> </div> </div> <div class="StepModule" data-module><span class="StepModule-anchor" id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Introduction"></span> <div class="StepModuleHeader"><h2 class="StepModuleHeader-title"><a class="StepModuleHeader-anchorLink" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Introduction" data-cms-ai="0">Introduction</a></h2> <div class="StepModuleHeader-divider"></div></div> <div class="StepModule-body RichTextBody"> <p>Real-time GPU processing of network packets is a technique useful for application domains involving signal processing, network security, information gathering, input reconstruction, and more. These applications involve the CPU in the critical path (CPU-centric approach) to coordinate the network card (NIC) for receiving packets in the GPU memory (GPUDirect RDMA) and notifying a packet-processing CUDA kernel waiting on the GPU for a new set of packets. In lower-power platforms, the CPU can easily become the bottleneck, masking GPU value. The aim is to maximize the zero-packet-loss throughput at the the lowest latency possible.</p> <p>A CPU-centric approach may not be scalable when increasing the number of clients connected to the application as the time between two receive operations on the same queue (client) would increase with the number of queues. The new DOCA GPUNetIO library allows developers to orchestrate these kinds of applications while optimizing performance, combining GPUDirect RDMA for data-path acceleration, GDRCopy library to give the CPU direct access to GPU memory, and GPUDirect async kernel-initiated network (GDAKIN) communications to allow a CUDA kernel to directly control the NIC.</p> <p>CPU-centric approach:</p> <p></p><p></p><div class="Enhancement" data-align-inline has-width style="--image-width: 644px"> <div class="Enhancement-item"> <figure class="Figure"> <picture> <source type="image/webp" width="1440" height="622" srcset="https://docscontent.nvidia.com/dims4/default/bea52ef/2147483647/strip/true/crop/1866x806+0+0/resize/1440x622!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-3-17_17-16-6-version-1-modificationdate-1725534953590-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/7ed16e5/2147483647/strip/true/crop/1866x806+0+0/resize/2880x1244!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-3-17_17-16-6-version-1-modificationdate-1725534953590-api-v2.png 2x"> <source type="image/png" width="1440" height="622" srcset="https://docscontent.nvidia.com/dims4/default/2d1b00f/2147483647/strip/true/crop/1866x806+0+0/resize/1440x622!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-3-17_17-16-6-version-1-modificationdate-1725534953590-api-v2.png"> <img class="Image" alt="image2023-3-17_17-16-6-version-1-modificationdate-1725534953590-api-v2.png" srcset="https://docscontent.nvidia.com/dims4/default/2d1b00f/2147483647/strip/true/crop/1866x806+0+0/resize/1440x622!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-3-17_17-16-6-version-1-modificationdate-1725534953590-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/c58bb6f/2147483647/strip/true/crop/1866x806+0+0/resize/2880x1244!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-3-17_17-16-6-version-1-modificationdate-1725534953590-api-v2.png 2x" width="1440" height="622" src="https://docscontent.nvidia.com/dims4/default/2d1b00f/2147483647/strip/true/crop/1866x806+0+0/resize/1440x622!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-3-17_17-16-6-version-1-modificationdate-1725534953590-api-v2.png" loading="lazy"> </picture> </figure> </div> </div><p> </p><p>GPU-centric approach:</p> <p></p><p></p><div class="Enhancement" data-align-inline has-height style="--image-height: 250px"> <div class="Enhancement-item"> <figure class="Figure"> <picture> <source type="image/webp" width="932" height="355" srcset="https://docscontent.nvidia.com/dims4/default/04e66dc/2147483647/strip/true/crop/932x355+0+0/resize/932x355!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-19_11-47-9-version-1-modificationdate-1725534957220-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/a1295a1/2147483647/strip/true/crop/932x355+0+0/resize/1864x710!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-19_11-47-9-version-1-modificationdate-1725534957220-api-v2.png 2x"> <source type="image/png" width="932" height="355" srcset="https://docscontent.nvidia.com/dims4/default/ff0f672/2147483647/strip/true/crop/932x355+0+0/resize/932x355!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-19_11-47-9-version-1-modificationdate-1725534957220-api-v2.png"> <img class="Image" alt="image2023-4-19_11-47-9-version-1-modificationdate-1725534957220-api-v2.png" srcset="https://docscontent.nvidia.com/dims4/default/ff0f672/2147483647/strip/true/crop/932x355+0+0/resize/932x355!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-19_11-47-9-version-1-modificationdate-1725534957220-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/f4619ed/2147483647/strip/true/crop/932x355+0+0/resize/1864x710!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-19_11-47-9-version-1-modificationdate-1725534957220-api-v2.png 2x" width="932" height="355" src="https://docscontent.nvidia.com/dims4/default/ff0f672/2147483647/strip/true/crop/932x355+0+0/resize/932x355!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-19_11-47-9-version-1-modificationdate-1725534957220-api-v2.png" loading="lazy"> </picture> </figure> </div> </div><p> </p><p></p><p></p> <p> DOCA GPUNetIO enables GPU-centric solutions that remove the CPU from the critical path by providing the following features: </p> <ul class=" "><li class=" "><p> GPUDirect async kernel-initiated technology – a GPU CUDA kernel can directly control other hardware components like the network card or NVIDIA® BlueField®'s DMA engine </p> <ul class=" "><li class=" "><p>GDAKIN communications – a GPU CUDA kernel can control network communications to send or receive data </p><p></p><p></p> <ul class=" "><li class=" "><p> GPU can control Ethernet communications </p> </li><li class=" "><p> GPU can control RDMA communications (InfiniBand or RoCE are supported) </p> </li><li class=" "><p> CPU intervention is not needed in the application critical path </p> </li></ul></li><li class=" "><p> DMA engine – a GPU CUDA kernel can trigger a memory copy using BlueField's DMA engine </p> </li></ul></li><li class=" "><p> <a class="Link" href="https://docs.nvidia.com/cuda/gpudirect-rdma/index.html" class="external-link" data-cms-ai="0" >GPUDirect RDMA</a> – use a contiguous GPU memory to send or receive RDMA data or Ethernet packets without CPU memory staging copies</p><p></p> <p></p> </li><li class=" "><p> Semaphores – provide a standardized low-latency message passing protocol between two CUDA kernels or a CUDA kernel and a CPU thread </p> </li><li class=" "><p> Smart memory allocation – allocate aligned GPU memory buffers, possibly exposing them to direct CPU access </p> <ul class=" "><li class=" "><p> Combination of CUDA, <a class="Link" href="https://github.com/DPDK/dpdk/blob/main/lib/gpudev/rte_gpudev.h" class="external-link" data-cms-ai="0" >DPDK gpudev</a> library and <a class="Link" href="https://github.com/NVIDIA/gdrcopy" class="external-link" data-cms-ai="0" >GDRCopy</a> library already embedded in the DPDK released with DOCA </p> </li></ul></li><li class=" "><p> Accurate send scheduling – schedule Ethernet packets' send in the future according to a user-provided timestamp </p> </li></ul><p><a class="Link" href="https://developer.nvidia.com/aerial-sdk" class="external-link" data-cms-ai="0" >Aerial 5G SDK</a>, <a class="Link" href="https://developer.nvidia.com/morpheus-cybersecurity" class="external-link" data-cms-ai="0" >Morpheus</a>, and <a class="Link" href="https://github.com/nvidia-holoscan/holohub/tree/main/operators/advanced_network" class="external-link" data-cms-ai="0" >Holoscan Advanced Network Operator</a> are examples of NVIDIA applications actively using DOCA GPUNetIO.</p> <p>For a deep dive into the technology and motivations, please refer to the NVIDIA blog posts <a class="Link" href="https://developer.nvidia.com/blog/inline-gpu-packet-processing-with-nvidia-doca-gpunetio/" class="external-link" data-cms-ai="0" >Inline GPU Packet Processing with NVIDIA DOCA GPUNetIO</a> and <a class="Link" href="https://developer.nvidia.com/blog/unlocking-gpu-accelerated-rdma-with-nvidia-doca-gpunetio/" class="external-link" data-cms-ai="0" >Unlocking GPU-Accelerated RDMA with NVIDIA DOCA GPUNetIO</a>. Another NVIDIA blog post <a class="Link" href="https://developer.nvidia.com/blog/realizing-the-power-of-real-time-network-processing-with-nvidia-doca-gpunetio/" class="external-link" data-cms-ai="0" >Realizing the Power of Real-time Network Processing with NVIDIA DOCA GPUNetIO</a> has been published to provide more use-case examples where DOCA GPUNetIO has been useful to improve the execution.</p> <div class="CallOutBox" data-option-Warning> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-info"></use></svg> </div><div class="CallOutBox-title">Warning</div> <div class="CallOutBox-body"> <p>RDMA on DOCA GPUNetIO is currently supported at alpha level.</p> </div> </div> <p> </p> </div> </div> <div class="StepModule" data-module><span class="StepModule-anchor" id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ChangesFromPreviousReleases"></span> <div class="StepModuleHeader"><h2 class="StepModuleHeader-title"><a class="StepModuleHeader-anchorLink" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ChangesFromPreviousReleases" data-cms-ai="0">Changes From Previous Releases</a></h2> <div class="StepModuleHeader-divider"></div></div> <div class="StepModule-body RichTextBody"> <h2 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Changesin2.9.0"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Changesin2.9.0" data-cms-ai="0">Changes in 2.9.0</a> </h2> <p> The following section details the <code class="InlineCodeBlock"><span>doca_gpunetio</span></code> library updates in version 2.9.0. </p> <ul class=" "><li class=" "><p>GPUNetIO RDMA functions now support the RDMA CM mode</p> <ul class=" "><li class=" "><p>Multiple connections can be assigned to the same RDMA object to send/write/read</p> </li><li class=" "><p>The receive side is a shared queue among all the connections associated to the GPU RDMA object</p> </li><li class=" "><p>Each RDMA function related to send/write/read takes the connection index as input. By default, it is 0 (i.e., only 1 RDMA connection per RDMA object).</p> </li></ul></li><li class=" "><p> <code class="InlineCodeBlock"><span>doca_gpu_dev_rdma_wait_all</span></code> function now returns the number of commits correctly executed, not the number of operations per commit</p> </li><li class=" "><p>Sample RDMA Client Server has been extended to optionally support the RDMA CM mode with two connections associated to the same RDMA GPU object</p> </li></ul> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ChangedFunctions"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ChangedFunctions" data-cms-ai="0">Changed Functions</a> </h3> <ul class=" "><li class=" "><p> <code class="InlineCodeBlock"><span>__device__ doca_error_t doca_gpu_dev_rdma_write_weak(struct doca_gpu_dev_rdma *rdma,</span></code><code class="InlineCodeBlock"><span>uint32_t connection_index,</span></code><code class="InlineCodeBlock"><span>struct doca_gpu_buf *remote_buf,</span></code><code class="InlineCodeBlock"><span>uint64_t remote_offset,</span></code><code class="InlineCodeBlock"><span>struct doca_gpu_buf *local_buf,</span></code><code class="InlineCodeBlock"><span>uint64_t local_offset,</span></code><code class="InlineCodeBlock"><span>size_t length,</span></code><code class="InlineCodeBlock"><span>uint32_t imm,</span></code><code class="InlineCodeBlock"><span>const enum doca_gpu_dev_rdma_write_flags flags,</span></code><code class="InlineCodeBlock"><span>uint32_t position);</span></code> </p> </li><li class=" "><p> <code class="InlineCodeBlock"><span>__device__ doca_error_t doca_gpu_dev_rdma_write_strong(struct doca_gpu_dev_rdma *rdma,</span></code><code class="InlineCodeBlock"><span>uint32_t connection_index,</span></code><code class="InlineCodeBlock"><span>struct doca_gpu_buf *remote_buf,</span></code><code class="InlineCodeBlock"><span>uint64_t remote_offset,</span></code><code class="InlineCodeBlock"><span>struct doca_gpu_buf *local_buf,</span></code><code class="InlineCodeBlock"><span>uint64_t local_offset,</span></code><code class="InlineCodeBlock"><span>size_t length,</span></code><code class="InlineCodeBlock"><span>uint32_t imm,</span></code><code class="InlineCodeBlock"><span>const enum doca_gpu_dev_rdma_write_flags flags);</span></code> </p> </li><li class=" "><p> <code class="InlineCodeBlock"><span>__device__ doca_error_t doca_gpu_dev_rdma_read_weak(struct doca_gpu_dev_rdma *rdma,</span></code><code class="InlineCodeBlock"><span>uint32_t connection_index,</span></code><code class="InlineCodeBlock"><span>struct doca_gpu_buf *remote_buf,</span></code><code class="InlineCodeBlock"><span>uint64_t remote_offset,</span></code><code class="InlineCodeBlock"><span>struct doca_gpu_buf *local_buf,</span></code><code class="InlineCodeBlock"><span>uint64_t local_offset,</span></code><code class="InlineCodeBlock"><span>size_t length, const uint32_t flags_bitmask,</span></code><code class="InlineCodeBlock"><span>uint32_t position);</span></code> </p> </li><li class=" "><p> <code class="InlineCodeBlock"><span>__device__ doca_error_t doca_gpu_dev_rdma_read_strong(</span></code><code class="InlineCodeBlock"><span>struct doca_gpu_dev_rdma *rdma, uint32_t connection_index, struct doca_gpu_buf *remote_buf, uint64_t remote_offset, struct doca_gpu_buf *local_buf, uint64_t local_offset, size_t length, const uint32_t flags_bitmask);</span></code> </p> </li><li class=" "><p> <code class="InlineCodeBlock"><span>__device__ doca_error_t doca_gpu_dev_rdma_send_weak(struct doca_gpu_dev_rdma *rdma,</span></code><code class="InlineCodeBlock"><span>uint32_t connection_index,</span></code><code class="InlineCodeBlock"><span>struct doca_gpu_buf *send_buf,</span></code><code class="InlineCodeBlock"><span>uint64_t send_offset,</span></code><code class="InlineCodeBlock"><span>size_t src_length,</span></code><code class="InlineCodeBlock"><span>uint32_t imm,</span></code><code class="InlineCodeBlock"><span>const enum doca_gpu_dev_rdma_send_flags flags,</span></code><code class="InlineCodeBlock"><span>uint32_t position);</span></code> </p> </li><li class=" "><p> <code class="InlineCodeBlock"><span>__device__ doca_error_t doca_gpu_dev_rdma_send_strong(struct doca_gpu_dev_rdma *rdma,</span></code><code class="InlineCodeBlock"><span>uint32_t connection_index,</span></code><code class="InlineCodeBlock"><span>struct doca_gpu_buf *send_buf,</span></code><code class="InlineCodeBlock"><span>uint64_t send_offset,</span></code><code class="InlineCodeBlock"><span>size_t src_length,</span></code><code class="InlineCodeBlock"><span>uint32_t imm,</span></code><code class="InlineCodeBlock"><span>const enum doca_gpu_dev_rdma_send_flags flags);</span></code> </p> </li><li class=" "><p> <code class="InlineCodeBlock"><span>__device__ doca_error_t doca_gpu_dev_rdma_commit_weak(struct doca_gpu_dev_rdma *rdma,</span></code><code class="InlineCodeBlock"><span>uint32_t connection_index,</span></code><code class="InlineCodeBlock"><span>uint32_t num_ops);</span></code> </p> </li><li class=" "><p> <code class="InlineCodeBlock"><span>__device__ doca_error_t doca_gpu_dev_rdma_commit_strong(struct doca_gpu_dev_rdma *rdma,</span></code><code class="InlineCodeBlock"><span>uint32_t connection_index</span></code><code class="InlineCodeBlock"><span>);</span></code> </p> </li><li class=" "><p> <code class="InlineCodeBlock"><span>__device__ doca_error_t doca_gpu_dev_rdma_get_info(struct doca_gpu_dev_rdma *rdma,</span></code><code class="InlineCodeBlock"><span>uint32_t connection_index</span></code><code class="InlineCodeBlock"><span>,</span></code><code class="InlineCodeBlock"><span>uint32_t *curr_position, uint32_t *mask_max_position);</span></code> </p> </li></ul><p> </p> </div> </div> <div class="StepModule" data-module><span class="StepModule-anchor" id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-SystemConfiguration"></span> <div class="StepModuleHeader"><h2 class="StepModuleHeader-title"><a class="StepModuleHeader-anchorLink" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-SystemConfiguration" data-cms-ai="0">System Configuration</a></h2> <div class="StepModuleHeader-divider"></div></div> <div class="StepModule-body RichTextBody"> <p class="p">DOCA GPUNetIO requires a properly configured environment which depends on whether the application should run on the x86 host or DPU Arm cores. The following subsections describe the required configuration in both scenarios, assuming DOCA, CUDA Toolkit and NVIDIA driver are installed on the system (x86 host or BlueField Arm) where the DOCA GPUNetIO is built and executed.</p> <p class="p">DOCA GPUNetIO is available for all DOCA for host and BFB packages downloadable <a class="Link" href="https://developer.nvidia.com/doca-downloads" class="external-link" data-cms-ai="0" >here</a>.</p> <p>Assuming the DOCA package has been downloaded and the prerequisites listed below have been satisfied, to install DOCA GPUNetIO components, run:</p> <ul class=" "><li class=" "><p class="auto-cursor-target">For Ubuntu/Debian:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>apt install doca-all doca-sdk-gpunetio libdoca-sdk-gpunetio-dev</p></code> </pre> </div> </bsp-code-block> <p> </p> </li><li class="auto-cursor-target "><p class="auto-cursor-target">For RHEL:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>yum install doca-all doca-sdk-gpunetio doca-sdk-gpunetio-devel</p></code> </pre> </div> </bsp-code-block> <p> </p> </li></ul><p class="p">Internal hardware topology of the system should be <a class="Link" href="https://docs.nvidia.com/cuda/gpudirect-rdma/index.html" class="external-link" data-cms-ai="0" >GPUDirect-RDMA</a>-friendly to <a class="Link" href="https://developer.nvidia.com/blog/benchmarking-gpudirect-rdma-on-modern-server-platforms/" class="external-link" data-cms-ai="0" >maximize the internal throughput</a> between the GPU and the NIC.</p> <p class="p">As DOCA GPUNetIO is present in both DOCA-for-Host and DOCA BFB (for BlueField Arm), a GPUNetIO application can be executed either on the host CPU or on the BlueField's Arm cores. The following subsections provide a description of both scenarios.</p> <div class="CallOutBox" data-option-Note> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-check"></use></svg> </div><div class="CallOutBox-title">Note</div> <div class="CallOutBox-body"> <p>DOCA GPUNetIO has been tested on bare-metal and in docker but never in a virtualized environment. Using KVM is discouraged for now.</p> </div> </div> <p> </p> <h2 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ApplicationonHostCPU"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ApplicationonHostCPU" data-cms-ai="0">Application on Host CPU</a> </h2> <p class="p">Assuming the DOCA GPUNetIO application is running on the host x86 CPU cores, it is highly recommended to have a dedicated PCIe connection between the GPU and the NIC. This topology can be realized in two ways:</p> <ul class=" "><li class="p "><p>Adding an additional PCIe switch to one of the PCIe root complex slots and attaching to this switch a GPU and a NVIDIA® ConnectX® adapter</p> </li><li class="p "><p>Connecting an NVIDIA® Converged Accelerator DPU to the PCIe root complex and setting it to NIC mode (i.e., exposing the GPU and NIC devices to the host)</p> </li></ul><p></p><p></p><div class="Enhancement" data-align-inline has-height style="--image-height: 400px"> <div class="Enhancement-item"> <figure class="Figure"> <picture> <source type="image/webp" width="1440" height="663" srcset="https://docscontent.nvidia.com/dims4/default/ded782b/2147483647/strip/true/crop/3382x1557+0+0/resize/1440x663!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_13-13-37-version-1-modificationdate-1725534950217-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/52b6721/2147483647/strip/true/crop/3382x1557+0+0/resize/2880x1326!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_13-13-37-version-1-modificationdate-1725534950217-api-v2.png 2x"> <source type="image/png" width="1440" height="663" srcset="https://docscontent.nvidia.com/dims4/default/cd89ba9/2147483647/strip/true/crop/3382x1557+0+0/resize/1440x663!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_13-13-37-version-1-modificationdate-1725534950217-api-v2.png"> <img class="Image" alt="image-2024-6-26_13-13-37-version-1-modificationdate-1725534950217-api-v2.png" srcset="https://docscontent.nvidia.com/dims4/default/cd89ba9/2147483647/strip/true/crop/3382x1557+0+0/resize/1440x663!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_13-13-37-version-1-modificationdate-1725534950217-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/ddc60e3/2147483647/strip/true/crop/3382x1557+0+0/resize/2880x1326!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_13-13-37-version-1-modificationdate-1725534950217-api-v2.png 2x" width="1440" height="663" src="https://docscontent.nvidia.com/dims4/default/cd89ba9/2147483647/strip/true/crop/3382x1557+0+0/resize/1440x663!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_13-13-37-version-1-modificationdate-1725534950217-api-v2.png" loading="lazy"> </picture> </figure> </div> </div><p> </p><p>You may check the topology of your system using <code class="InlineCodeBlock"><span>lspci -tvvv</span></code> or <code class="InlineCodeBlock"><span>nvidia-smi topo -m</span></code>.</p> <h3 id="src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1PcHRpb24xOkNvbm5lY3RYQWRhcHRlcmluRXRoZXJuZXRNb2Rl"> <a href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1PcHRpb24xOkNvbm5lY3RYQWRhcHRlcmluRXRoZXJuZXRNb2Rl" data-cms-ai="0">Option 1: ConnectX Adapter in Ethernet Mode</a> </h3> <div class="CallOutBox" data-option-Note> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-check"></use></svg> </div><div class="CallOutBox-title">Note</div> <div class="CallOutBox-body"> <p>NVIDIA® ConnectX® firmware must be 22.36.1010 or later. It is highly recommended to only use NVIDIA adapter from ConnectX-6 Dx and later.</p> </div> </div> <p> </p> <p>DOCA GPUNetIO allows a CUDA kernel to control the NIC when working with Ethernet protocol. For this reason, the ConnectX must be set to Ethernet mode.</p> <p>To do that, follow these steps:</p> <ol class=" "><li class=" "><p class="auto-cursor-target">Start MST, check the status, and copy the MST device name:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="comments"># Start MST</code> mst start mst status -<code class="functions">v</code>   MST modules: ------------ MST PCI module is not loaded MST PCI configuration module loaded PCI devices: ------------ DEVICE_TYPE MST PCI RDMA NET NUMA ConnectX6DX(rev:0) /dev/mst/mt4125_pciconf0.1 b5:00.1 mlx5_1 net-ens6f1 0 ConnectX6DX(rev:0) /dev/mst/mt4125_pciconf0 b5:00.0 mlx5_0 net-ens6f0 0</p></code> </pre> </div> </bsp-code-block> <p> </p> </li><li class=" "><p class="auto-cursor-target">Configure the NIC to Ethernet mode and enable Accurate Send Scheduling (if required on the send side):</p> <div class="CallOutBox" data-option-Info> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-info"></use></svg> </div><div class="CallOutBox-title">Info</div> <div class="CallOutBox-body"> <p>The following example assumes that the adapter is dual-port. If single port, only P1 options apply.</p> </div> </div> <p> </p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>mlxconfig -d <mst_device> s KEEP_ETH_LINK_UP_P1=1 KEEP_ETH_LINK_UP_P2=1 KEEP_IB_LINK_UP_P1=0 KEEP_IB_LINK_UP_P2=0 mlxconfig -d <mst_device> --<code class="functions">yes</code> <code class="functions">set</code> ACCURATE_TX_SCHEDULER=1 REAL_TIME_CLOCK_ENABLE=1</p></code> </pre> </div> </bsp-code-block> <p> </p> </li><li class=" "><p class="auto-cursor-target">Perform cold reboot to apply the configuration changes:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>ipmitool power cycle</p></code> </pre> </div> </bsp-code-block> <p> </p> </li></ol> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Option2"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Option2" data-cms-ai="0"><span id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Option2" class="confluence-anchor-link"></span>Option 2: DPU Converged Accelerator in NIC mode</a> </h3> <p>To expose and use the GPU and the NIC on the converged accelerator DPU to an application running on the Host x86, configure the DPU to operate in NIC mode.</p> <p>To do that, follow these steps:</p> <div class="CallOutBox" data-option-Info> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-info"></use></svg> </div><div class="CallOutBox-title">Info</div> <div class="CallOutBox-body"> <p>Valid for both NVIDIA® BlueField®-2 and NVIDIA® BlueField®-3 converged accelerator DPUs.</p> </div> </div> <p> </p> <ol class=" "><li class=" "><p class="auto-cursor-target">Start MST, check the status, and copy the MST device name:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="comments"># Enable MST</code> <code class="functions">sudo</code> mst start <code class="functions">sudo</code> mst status   MST devices: ------------ /dev/mst/mt41686_pciconf0 - PCI configuration cycles access. domain:bus:dev.fn=0000:b8:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1 Chip revision is: 01</p></code> </pre> </div> </bsp-code-block> <p> </p> </li><li class=" "><p class="auto-cursor-target">Expose the GPU on the converged accelerator DPU to the host.</p> <ul class=" "><li class=" "><p class="auto-cursor-target">For BlueField-2, the <code class="InlineCodeBlock"><span>PCI_DOWNSTREAM_PORT_OWNER</span></code> offset must be set to 4:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="functions">sudo</code> mlxconfig -d <mst_device> --<code class="functions">yes</code> s PCI_DOWNSTREAM_PORT_OWNER[4]=0x0</p></code> </pre> </div> </bsp-code-block> <p> </p> </li><li class=" "><p class="auto-cursor-target">For BlueField-3, the <code class="InlineCodeBlock"><span>PCI_DOWNSTREAM_PORT_OWNER</span></code> offset must be set to 8:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="functions">sudo</code> mlxconfig -d <mst_device> --<code class="functions">yes</code> s PCI_DOWNSTREAM_PORT_OWNER[8]=0x0</p></code> </pre> </div> </bsp-code-block> <p> </p> </li></ul></li><li class=" "><p class="auto-cursor-target">Set BlueField to Ethernet mode, enable Accurate Send Scheduling (if required on the send side), and set it to NIC mode:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="functions">sudo</code> mlxconfig -d <mst_device> --<code class="functions">yes</code> <code class="functions">set</code> LINK_TYPE_P1=2 LINK_TYPE_P2=2 INTERNAL_CPU_MODEL=1 INTERNAL_CPU_PAGE_SUPPLIER=1 INTERNAL_CPU_ESWITCH_MANAGER=1 INTERNAL_CPU_IB_VPORT0=1 INTERNAL_CPU_OFFLOAD_ENGINE=DISABLED <code class="functions">sudo</code> mlxconfig -d <mst_device> --<code class="functions">yes</code> <code class="functions">set</code> ACCURATE_TX_SCHEDULER=1 REAL_TIME_CLOCK_ENABLE=1</p></code> </pre> </div> </bsp-code-block> <p> </p> </li><li class=" "><p class="auto-cursor-target">Perform cold reboot to apply the configuration changes:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>ipmitool power cycle</p></code> </pre> </div> </bsp-code-block> <p> </p> </li><li class=" "><p class="auto-cursor-target">Verify configuration:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="functions">sudo</code> mlxconfig -d <mst_device> q LINK_TYPE_P1 LINK_TYPE_P2 INTERNAL_CPU_MODEL INTERNAL_CPU_PAGE_SUPPLIER INTERNAL_CPU_ESWITCH_MANAGER INTERNAL_CPU_IB_VPORT0 INTERNAL_CPU_OFFLOAD_ENGINE ACCURATE_TX_SCHEDULER REAL_TIME_CLOCK_ENABLE LINK_TYPE_P1 ETH(2) LINK_TYPE_P2 ETH(2) INTERNAL_CPU_MODEL EMBEDDED_CPU(1) INTERNAL_CPU_PAGE_SUPPLIER EXT_HOST_PF(1) INTERNAL_CPU_ESWITCH_MANAGER EXT_HOST_PF(1) INTERNAL_CPU_IB_VPORT0 EXT_HOST_PF(1) INTERNAL_CPU_OFFLOAD_ENGINE DISABLED(1) ACCURATE_TX_SCHEDULER True(1) REAL_TIME_CLOCK_ENABLE True(1)</p></code> </pre> </div> </bsp-code-block> <p> </p> </li></ol><p> </p> <h2 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ApplicationonDPUConvergedArmCPU"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ApplicationonDPUConvergedArmCPU" data-cms-ai="0"><span id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ApplicationonDPUConvergedArmCPU" class="confluence-anchor-link"></span>Application on BlueField Converged Arm CPU</a> </h2> <p>In this scenario, the DOCA GPUNetIO is running on the CPU Arm cores of the BlueField using the GPU and NIC on the same BlueField.</p> <p></p><p></p><div class="Enhancement" data-align-inline has-height style="--image-height: 400px"> <div class="Enhancement-item"> <figure class="Figure"> <picture> <source type="image/webp" width="1440" height="820" srcset="https://docscontent.nvidia.com/dims4/default/0517a1f/2147483647/strip/true/crop/2744x1563+0+0/resize/1440x820!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_13-14-11-version-1-modificationdate-1725534950000-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/af66981/2147483647/strip/true/crop/2744x1563+0+0/resize/2880x1640!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_13-14-11-version-1-modificationdate-1725534950000-api-v2.png 2x"> <source type="image/png" width="1440" height="820" srcset="https://docscontent.nvidia.com/dims4/default/0e2f243/2147483647/strip/true/crop/2744x1563+0+0/resize/1440x820!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_13-14-11-version-1-modificationdate-1725534950000-api-v2.png"> <img class="Image" alt="image-2024-6-26_13-14-11-version-1-modificationdate-1725534950000-api-v2.png" srcset="https://docscontent.nvidia.com/dims4/default/0e2f243/2147483647/strip/true/crop/2744x1563+0+0/resize/1440x820!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_13-14-11-version-1-modificationdate-1725534950000-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/f3ddcca/2147483647/strip/true/crop/2744x1563+0+0/resize/2880x1640!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_13-14-11-version-1-modificationdate-1725534950000-api-v2.png 2x" width="1440" height="820" src="https://docscontent.nvidia.com/dims4/default/0e2f243/2147483647/strip/true/crop/2744x1563+0+0/resize/1440x820!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_13-14-11-version-1-modificationdate-1725534950000-api-v2.png" loading="lazy"> </picture> </figure> </div> </div><p> </p><p>The converged accelerator DPU must be set to CPU mode after flashing the right BFB image (refer to <a class="Link" href="/doca/sdk/NVIDIA+DOCA+Installation+Guide+for+Linux" data-cms-ai="0" >NVIDIA DOCA Installation Guide for Linux</a> for details). From the x86 host, configure the DPU as detailed in the following steps:</p> <div class="CallOutBox" data-option-Info> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-info"></use></svg> </div><div class="CallOutBox-title">Info</div> <div class="CallOutBox-body"> <p>Valid for both BlueField-2 and BlueField-3 converged accelerator DPUs.</p> </div> </div> <p> </p> <ol class=" "><li class=" "><p class="auto-cursor-target">Start MST, check the status, and copy the MST device name:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="comments"># Enable MST</code> <code class="functions">sudo</code> mst start <code class="functions">sudo</code> mst status   MST devices: ------------ /dev/mst/mt41686_pciconf0 - PCI configuration cycles access. domain:bus:dev.fn=0000:b8:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1 Chip revision is: 01</p></code> </pre> </div> </bsp-code-block> <p> </p> </li><li class=" "><p class="auto-cursor-target">Set the DPU as the GPU owner.</p> <ol class=" "><li class=" "><p class="auto-cursor-target">For BlueField-2 the <code class="InlineCodeBlock"><span>PCI_DOWNSTREAM_PORT_OWNER</span></code> offset must be set to 4:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="functions">sudo</code> mlxconfig -d <mst_device> --<code class="functions">yes</code> s PCI_DOWNSTREAM_PORT_OWNER[4]=0xF</p></code> </pre> </div> </bsp-code-block> <p> </p> </li><li class=" "><p class="auto-cursor-target">For BlueField-3 the <code class="InlineCodeBlock"><span>PCI_DOWNSTREAM_PORT_OWNER</span></code> offset must be set to 8:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="functions">sudo</code> mlxconfig -d <mst_device> --<code class="functions">yes</code> s PCI_DOWNSTREAM_PORT_OWNER[8]=0xF</p></code> </pre> </div> </bsp-code-block> <p> </p> </li></ol></li><li class=" "><p class="auto-cursor-target">Set BlueField to Ethernet mode and enable Accurate Send Scheduling (if required on the send side):</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="functions">sudo</code> mlxconfig -d <mst_device> --<code class="functions">yes</code> <code class="functions">set</code> LINK_TYPE_P1=2 LINK_TYPE_P2=2 INTERNAL_CPU_MODEL=1 INTERNAL_CPU_PAGE_SUPPLIER=0 INTERNAL_CPU_ESWITCH_MANAGER=0 INTERNAL_CPU_IB_VPORT0=0 INTERNAL_CPU_OFFLOAD_ENGINE=ENABLED <code class="functions">sudo</code> mlxconfig -d <mst_device> --<code class="functions">yes</code> <code class="functions">set</code> ACCURATE_TX_SCHEDULER=1 REAL_TIME_CLOCK_ENABLE=1</p></code> </pre> </div> </bsp-code-block> <p> </p> </li><li class=" "><p class="auto-cursor-target">Perform cold reboot to apply the configuration changes:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>ipmitool power cycle</p></code> </pre> </div> </bsp-code-block> <p> </p> </li><li class=" "><p class="auto-cursor-target">Verify configuration:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>mlxconfig -d <mst_device> q LINK_TYPE_P1 LINK_TYPE_P2 INTERNAL_CPU_MODEL INTERNAL_CPU_PAGE_SUPPLIER INTERNAL_CPU_ESWITCH_MANAGER INTERNAL_CPU_IB_VPORT0 INTERNAL_CPU_OFFLOAD_ENGINE ACCURATE_TX_SCHEDULER REAL_TIME_CLOCK_ENABLE ... Configurations: Next Boot LINK_TYPE_P1 ETH(2) LINK_TYPE_P2 ETH(2) INTERNAL_CPU_MODEL EMBEDDED_CPU(1) INTERNAL_CPU_PAGE_SUPPLIER ECPF(0) INTERNAL_CPU_ESWITCH_MANAGER ECPF(0) INTERNAL_CPU_IB_VPORT0 ECPF(0) INTERNAL_CPU_OFFLOAD_ENGINE ENABLED(0) ACCURATE_TX_SCHEDULER True(1) REAL_TIME_CLOCK_ENABLE True(1)</p></code> </pre> </div> </bsp-code-block> <p> </p> </li></ol><p>At this point, it should be possible to SSH into BlueField to access the OS installed on it. Before installing DOCA GPUNetIO as previously described, CUDA Toolkit (and NVIDIA driver) must be installed.</p><p> </p> <h2 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-PCIeConfiguration"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-PCIeConfiguration" data-cms-ai="0">PCIe Configuration</a> </h2> <p>On some x86 systems, the Access Control Services (ACS) must be disabled to ensure direct communication between the NIC and GPU, whether they reside on the same converged accelerator DPU or on different PCIe slots in the system. The recommended solution is to disable ACS control via BIOS (e.g., <a class="Link" href="https://www.supermicro.com/support/faqs/faq.cfm?faq=22226" target="_blank" rel="noopener" class="external-link" data-cms-ai="0" >Supermicro</a> or <a class="Link" href="https://techlibrary.hpe.com/docs/iss/proliant-gen10-uefi/GUID-34A85970-B963-4518-8655-DA66170D52F2.html" class="external-link" data-cms-ai="0" >HPE</a>). Alternatively, it is also possible to disable it via command line, but it may not be as effective as the BIOS option. Assuming system topology <a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Option2" data-cms-ai="0" >Option 2</a>, with a converged accelerator DPU as follows:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>$ lspci -tvvv...+-[0000:b0]-+-00.0 Intel Corporation Device 09a2 | +-00.1 Intel Corporation Device 09a4 | +-00.2 Intel Corporation Device 09a3 | +-00.4 Intel Corporation Device 0998 | \-02.0-[b1-b6]----00.0-[b2-b6]--+-00.0-[b3]--+-00.0 Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller | | +-00.1 Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller | | \-00.2 Mellanox Technologies MT42822 BlueField-2 SoC Management Interface | \-01.0-[b4-b6]----00.0-[b5-b6]----08.0-[b6]----00.0 NVIDIA Corporation Device 20b8</p></code> </pre> </div> </bsp-code-block> <p> </p> <p>The PCIe switch address to consider is <code class="InlineCodeBlock"><span>b2:00.0</span></code> (entry point of the DPU). ACSCtl must have all negative values:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">PCIe set</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>setpci -s b2:00.0 ECAP_ACS+6.w=0:fc</p></code> </pre> </div> </bsp-code-block> <p> </p> <p class="auto-cursor-target">To verify that the setting has been applied correctly:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">PCIe check</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>$ <code class="functions">sudo</code> lspci -s b2:00.0 -vvvv | <code class="functions">grep</code> -i ACSCtl ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-</p></code> </pre> </div> </bsp-code-block> <p> </p> <p>Please refer to <a class="Link" href="https://www.supermicro.com/support/faqs/faq.cfm?faq=20732" target="_blank" rel="noopener" class="external-link" data-cms-ai="0" >this page</a> and <a class="Link" href="https://forums.developer.nvidia.com/t/multi-gpu-peer-to-peer-access-failing-on-tesla-k80/39748/15" class="external-link" data-cms-ai="0" >this page</a> for more information.</p> <p>If the application still does not report any received packets, try to disable IOMMU. On some systems, it can be done from the BIOS looking for the the <code class="InlineCodeBlock"><span>VT-d</span></code> or <code class="InlineCodeBlock"><span>IOMMU</span></code> from the NorthBridge configuration and change that setting to <code class="InlineCodeBlock"><span>Disable</span></code> and save it. The system may also require adding <code class="InlineCodeBlock"><span>intel_iommu=off</span></code> or <code class="InlineCodeBlock"><span>amd_iommu=off</span></code> to the kernel options. That can be done through the grub command line as follows:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">IOMMU</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>$ <code class="functions">sudo</code> vim /etc/default/grub <code class="comments"># GRUB_CMDLINE_LINUX_DEFAULT="iommu=off intel_iommu=off <more options>"</code> $ <code class="functions">sudo</code> update-grub $ <code class="functions">sudo</code> reboot</p></code> </pre> </div> </bsp-code-block> <p> <br> </p> <h2 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Hugepages"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Hugepages" data-cms-ai="0">Hugepages</a> </h2> <p>DOCA GPUNetIO has some dependency on the DPDK <code class="InlineCodeBlock"><span>rte_eal_init</span></code> initialization function (this dependency will be resolved in future releases). <a class="Link" href="https://doc.dpdk.org/guides/linux_gsg/sys_reqs.html#use-of-hugepages-in-the-linux-environment" class="external-link" data-cms-ai="0" >DPDK requires hugepages</a>, which can be enabled permanently via grub command line or temporary using the following command line:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">IOMMU</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="functions">echo</code> 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages</p></code> </pre> </div> </bsp-code-block> <p> <br> </p> <h2 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-GPUConfiguration"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-GPUConfiguration" data-cms-ai="0">GPU Configuration</a> </h2> <p><a class="Link" href="https://developer.nvidia.com/cuda-downloads" class="external-link" data-cms-ai="0" >CUDA Toolkit</a> 12.1 or newer must be installed on the host. It is also recommended to enable persistence mode to decrease initial application latency <code class="InlineCodeBlock"><span>nvidia-smi -pm 1</span></code>.</p> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-GDRCopy"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-GDRCopy" data-cms-ai="0">GDRCopy</a> </h3> <p>To allow the CPU to access the GPU memory directly without the need for CUDA API, DPDK and DOCA require the <a class="Link" href="https://github.com/NVIDIA/gdrcopy" class="external-link" data-cms-ai="0" >GDRCopy</a> kernel module to be installed on the system:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">GDRCopy configuration</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="comments"># Install GDRCopy</code> <code class="functions">sudo</code> apt <code class="functions">install</code> -y check kmod git clone https://github.com/NVIDIA/gdrcopy.git /opt/mellanox/gdrcopy <code class="functions">cd</code> /opt/mellanox/gdrcopy <code class="functions">make</code> <code class="comments"># Run gdrdrv kernel module</code> ./insmod.sh   <code class="comments"># Double check nvidia-peermem and gdrdrv module are running</code> $ lsmod | <code class="functions">egrep</code> gdrdrv gdrdrv 24576 0 nvidia 55726080 4 nvidia_uvm,nvidia_peermem,gdrdrv,nvidia_modeset   <code class="comments"># Export library path</code> <code class="functions">export</code> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/opt/mellanox/gdrcopy/src   <code class="comments"># Ensure CUDA library path is in the env var</code> <code class="functions">export</code> PATH=<code class="string">"/usr/local/cuda/bin:${PATH}"</code> <code class="functions">export</code> LD_LIBRARY_PATH=<code class="string">"/usr/local/cuda/lib:/usr/local/cuda/lib64:${LD_LIBRARY_PATH}"</code> <code class="functions">export</code> CPATH=<code class="string">"$(echo /usr/local/cuda/targets/{x86_64,sbsa}-linux/include | sed 's/ /:/'):${CPATH}"</code></p></code> </pre> </div> </bsp-code-block> <p> <br> </p> <h3 id="src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1HUFVNZW1vcnlNYXBwaW5nKG52aWRpYS1wZWVybWVtdnMuZG1hYnVmKQ"> <a href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1HUFVNZW1vcnlNYXBwaW5nKG52aWRpYS1wZWVybWVtdnMuZG1hYnVmKQ" data-cms-ai="0">GPU Memory Mapping (nvidia-peermem vs. dmabuf)</a> </h3> <p>To allow the NIC to send and receive packets using GPU memory, it is required to launch the NVIDIA kernel module <code class="InlineCodeBlock"><span>nvidia-peermem</span></code>. It is shipped by default with the CUDA Toolkit installation.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">Launch nvidia-peermem</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="functions">sudo</code> modprobe nvidia-peermem</p></code> </pre> </div> </bsp-code-block> <p> </p> <p>Mapping buffers through the <code class="InlineCodeBlock"><span>nvidia-peermem</span></code> module is the legacy mapping mode.</p> <p>Alternatively, DOCA offers the ability to map GPU memory through the <code class="InlineCodeBlock"><span>dmabuf</span></code> providing a set high-level function. Prerequisites are DOCA installed on a system with:</p> <ul class=" "><li class=" "><p>Linux Kernel ≥ 6.2</p> </li><li class=" "><p>libibverbs ≥ 1.14.44</p> </li><li class=" "><p>CUDA Toolkit 12.5 or older – installed with the <code class="InlineCodeBlock"><span>-m=kernel-open</span></code> flag (which implies the NVIDIA driver in open-source mode)</p> </li><li class=" "><p>CUDA Toolkit 12.6 or newer – open kernel mode is enabled by default</p> </li></ul> <div class="CallOutBox" data-option-Note> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-check"></use></svg> </div><div class="CallOutBox-title">Note</div> <div class="CallOutBox-body"> <p>Installing DOCA on kernel 6.2 to enable the <code class="InlineCodeBlock"><span>dmabuf</span></code> is experimental.</p> </div> </div> <p> </p> <p>An example can be found in the DOCA GPU Packet Processing application:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">GPU Configuration</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>/* Get from CUDA the dmabuf <code class="functions">file</code>-descriptor <code class="keyword">for</code> the GPU memory buffer */ result = doca_gpu_dmabuf_fd(gpu_dev, gpu_buffer_addr, gpu_buffer_size, &(dmabuf_fd)); <code class="keyword">if</code> (result != DOCA_SUCCESS) { /* If it fails, create a DOCA mmap <code class="keyword">for</code> the GPU memory buffer with the nvidia-peermem legacy method */ doca_mmap_set_memrange(gpu_buffer_mmap, gpu_buffer_addr, gpu_buffer_size); } <code class="keyword">else</code> { /* If it succeeds, create a DOCA mmap <code class="keyword">for</code> the GPU memory buffer using the dmabuf method */ doca_mmap_set_dmabuf_memrange(gpu_buffer_mmap, dmabuf_fd, gpu_buffer_addr, 0, gpu_buffer_size); }</p></code> </pre> </div> </bsp-code-block> <p> </p> <p>If the function <code class="InlineCodeBlock"><span>doca_gpu_dmabuf_fd</span></code> fails, it probably means the NVIDIA driver is not installed with the open-source mode.</p> <p>Later, when calling the <code class="InlineCodeBlock"><span>doca_mmap_start</span></code>, the DOCA library tries to map the GPU memory buffer using the <code class="InlineCodeBlock"><span>dmabuf</span></code> file descriptor. If it fails (something incorrectly set on the Linux system), it fallbacks trying to map the GPU buffer with the legacy mode (<code class="InlineCodeBlock"><span>nvidia-peermem</span></code> ). If it fails, an informative error is returned.</p><p> </p> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-GPUBAR1Size"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-GPUBAR1Size" data-cms-ai="0">GPU BAR1 Size</a> </h3> <p>Every time a GPU buffer is mapped to the NIC (e.g., buffers associated with send or receive queues), a portion of the GPU BAR1 mapping space is used. Therefore, it is important to check that the BAR1 mapping is large enough to hold all the bytes the DOCA GPUNetIO application is trying to map. To verify the BAR1 mapping space of a GPU you can use <code class="InlineCodeBlock"><span>nvidia-smi</span></code>:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">BAR1 mapping</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>$ nvidia-smi -q   ==============NVSMI LOG============== ..... Attached GPUs : 1 GPU 00000000:CA:00.0 Product Name : NVIDIA A100 80GB PCIe Product Architecture : Ampere Persistence Mode : Enabled ..... BAR1 Memory Usage Total : 131072 MiB Used : 1 MiB Free : 131071 MiB</p></code> </pre> </div> </bsp-code-block> <p> </p> <p>By default, some GPUs (e.g. RTX models) may have a very small BAR1 size:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">BAR1 mapping</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>$ nvidia-smi -q | <code class="functions">grep</code> -i bar -A 3 BAR1 Memory Usage Total : 256 MiB Used : 6 MiB Free : 250 MiB</p></code> </pre> </div> </bsp-code-block> <p> </p> <p>If the BAR1 size is not enough, DOCA GPUNetIO applications may exit with errors because DOCA mmap fails to map the GPU memory buffers to the NIC (e.g., <code class="InlineCodeBlock"><span>Failed to start mmap DOCA Driver call failure</span></code>). To overcome this issue, the GPU BAR1 must be increased from the BIOS. The system should have "Resizable BAR" option enabled. For further information, refer to <a class="Link" href="https://forums.developer.nvidia.com/t/doca-gpu-packet-processing-failed-to-start-mmap-doca-driver-call-failure/267827" class="external-link" data-cms-ai="0" >this</a> NVIDIA forum post.</p><p> </p> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-BlueField-3SpecificConfiguration"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-BlueField-3SpecificConfiguration" data-cms-ai="0">BlueField-3 Specific Configuration</a> </h3> <p>To run a DOCA GPUNetIO application on the BlueField-3 Arm cores in a converged card (section "<a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ApplicationonDPUConvergedArmCPU" data-cms-ai="0" >Application on DPU Converged Arm CPU</a>"), it is mandatory to set an NVIDIA driver option at the end of the driver configuration file:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">Set NVIDIA driver option</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="functions">cat</code> <<EOF | <code class="functions">sudo</code> <code class="functions">tee</code> /etc/modprobe.d/nvidia.conf options nvidia NVreg_RegistryDwords=<code class="string">"RmDmaAdjustPeerMmioBF3=1;"</code> EOF</p></code> </pre> </div> </bsp-code-block> <p> </p> <p>To make sure the option has been detected by the NVIDIA driver, run:</p> <p> </p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">Check NVIDIA driver option</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>$ <code class="functions">grep</code> RegistryDwords /proc/driver/nvidia/params RegistryDwords: <code class="string">"RmDmaAdjustPeerMmioBF3=1;"</code> RegistryDwordsPerDevice: <code class="string">""</code></p></code> </pre> </div> </bsp-code-block> <p> </p> </div> </div> <div class="StepModule" data-module><span class="StepModule-anchor" id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Architecture"></span> <div class="StepModuleHeader"><h2 class="StepModuleHeader-title"><a class="StepModuleHeader-anchorLink" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Architecture" data-cms-ai="0">Architecture</a></h2> <div class="StepModuleHeader-divider"></div></div> <div class="StepModule-body RichTextBody"> <p>A GPU packet processing network application can be split into two fundamental phases:</p> <ul class=" "><li class=" "><p>Setup on the CPU (devices configuration, memory allocation, launch of CUDA kernels, etc.)</p> </li><li class=" "><p>Main data path where GPU and NIC interact to exercise their functions</p> </li></ul><p>DOCA GPUNetIO provides different building blocks, some of them in combination with the <a class="Link" href="/doca/sdk/DOCA+Ethernet" data-cms-ai="0" >DOCA Ethernet</a> or <a class="Link" href="https://confluence.nvidia.com/display/docadev/DOCA+RDMA" data-cms-ai="0" >DOCA RDMA</a> library, to create a full pipeline running entirely on the GPU.</p> <p>During the setup phase on the CPU, applications must:</p> <ol class=" "><li class=" "><p>Prepare all the objects on the CPU.</p> </li><li class=" "><p>Export a GPU handler for them.</p> </li><li class=" "><p>Launch a CUDA kernel passing the object's GPU handler to work with the object during the data path.</p> </li></ol><p>For this reason, DOCA GPUNetIO is composed of two libraries:</p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>libdoca_gpunetio</span></code> with functions invoked by CPU to prepare the GPU, allocate memory and objects</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>libdoca_gpunetio_device</span></code> with functions invoked by GPU within CUDA kernels during the data path</p> </li></ul><p></p><p></p><p></p> <div class="CallOutBox" data-option-Note> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-check"></use></svg> </div><div class="CallOutBox-title">Note</div> <div class="CallOutBox-body"> <p>The pkgconfig file for the DOCA GPUNetIO shared library is <code class="InlineCodeBlock"><span>doca-gpunetio.pc</span></code>. However, there is no pkgconfig file for the DOCA GPUNetIO CUDA device's static library <code class="InlineCodeBlock"><span>/opt/mellanox/doca/lib/x86_64-linux-gnu/libdoca_gpunetio_device.a</span></code>, so it must be explicitly linked to the CUDA application if DOCA GPUNetIO CUDA device functions are required.</p> </div> </div> <p> </p> <p>The following diagram presents the typical flow:</p> <p></p><p></p><div class="Enhancement" data-align-inline has-height style="--image-height: 400px"> <div class="Enhancement-item"> <figure class="Figure"> <picture> <source type="image/webp" width="1440" height="1606" srcset="https://docscontent.nvidia.com/dims4/default/2fee2df/2147483647/strip/true/crop/2076x2316+0+0/resize/1440x1606!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-1-12_12-24-42-version-1-modificationdate-1725534952300-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/7598eb2/2147483647/strip/true/crop/2076x2316+0+0/resize/2880x3212!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-1-12_12-24-42-version-1-modificationdate-1725534952300-api-v2.png 2x"> <source type="image/png" width="1440" height="1606" srcset="https://docscontent.nvidia.com/dims4/default/a020685/2147483647/strip/true/crop/2076x2316+0+0/resize/1440x1606!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-1-12_12-24-42-version-1-modificationdate-1725534952300-api-v2.png"> <img class="Image" alt="image-2024-1-12_12-24-42-version-1-modificationdate-1725534952300-api-v2.png" srcset="https://docscontent.nvidia.com/dims4/default/a020685/2147483647/strip/true/crop/2076x2316+0+0/resize/1440x1606!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-1-12_12-24-42-version-1-modificationdate-1725534952300-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/970c0db/2147483647/strip/true/crop/2076x2316+0+0/resize/2880x3212!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-1-12_12-24-42-version-1-modificationdate-1725534952300-api-v2.png 2x" width="1440" height="1606" src="https://docscontent.nvidia.com/dims4/default/a020685/2147483647/strip/true/crop/2076x2316+0+0/resize/1440x1606!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-1-12_12-24-42-version-1-modificationdate-1725534952300-api-v2.png" loading="lazy"> </picture> </figure> </div> </div><p> </p><p>Refer to the <a class="Link" href="/doca/sdk/NVIDIA+DOCA+GPU+Packet+Processing+Application+Guide" data-cms-ai="0" >NVIDIA DOCA GPU Packet Processing Application Guide</a> for an example of using DOCA GPUNetIO to send and receive Ethernet packets.</p> </div> </div> <div class="StepModule" data-module><span class="StepModule-anchor" id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-API"></span> <div class="StepModuleHeader"><h2 class="StepModuleHeader-title"><a class="StepModuleHeader-anchorLink" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-API" data-cms-ai="0">API</a></h2> <div class="StepModuleHeader-divider"></div></div> <div class="StepModule-body RichTextBody"> <p class="p">This section details the specific structures and operations related to the main DOCA GPUNetIO API on CPU and GPU. GPUNetIO headers are:</p> <ul class=" "><li class="p "><p><code class="InlineCodeBlock"><span>doca_gpunetio.h</span></code> – CPU functions</p> </li><li class="p "><p><code class="InlineCodeBlock"><span>doca_gpunetio_dev_buf.cuh</span></code> – GPU functions to manage a DOCA buffer array</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>doca_gpunetio_dev_eth_rxq.cuh</span></code> – GPU functions to manage a DOCA Ethernet receive queue</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>doca_gpunetio_dev_eth_txq.cuh</span></code> – GPU functions to manage a DOCA Ethernet send queue</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>doca_gpunetio_dev_sem.cuh</span></code> – GPU functions to manage a DOCA GPUNetIO semaphore</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>doca_gpunetio_dev_rdma.cuh</span></code> – GPU functions to manage a DOCA RDMA queue</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>doca_gpunetio_dev_dma.cuh</span></code> – GPU functions to manage a DOCA DMA queue</p> </li></ul><p>This section lists the main functions of DOCA GPUNetIO. To better understand their usage, refer to section "<a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-BuildingBlocks" data-cms-ai="0" >Building Blocks</a>" which includes several code examples.</p> <div class="CallOutBox" data-option-Tip> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-check"></use></svg> </div><div class="CallOutBox-title">Tip</div> <div class="CallOutBox-body"> <p> To better understand structures, objects, and functions related to Ethernet send and receive, please refer to the <a class="Link" href="/doca/sdk/DOCA+Ethernet" data-cms-ai="0" >DOCA Ethernet</a>. </p> </div> </div> <p> </p> <div class="CallOutBox" data-option-Tip> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-check"></use></svg> </div><div class="CallOutBox-title">Tip</div> <div class="CallOutBox-body"> <p> To better understand structures, objects, and functions related to RDMA operations, please refer to the <a class="Link" href="/doca/sdk/DOCA+RDMA" data-cms-ai="0" >DOCA RDMA</a>. </p> </div> </div> <p> </p> <div class="CallOutBox" data-option-Tip> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-check"></use></svg> </div><div class="CallOutBox-title">Tip</div> <div class="CallOutBox-body"> <p> To better understand structures, objects, and functions related to DMA operations, please refer to the <a class="Link" href="/doca/sdk/DOCA+DMA" data-cms-ai="0" >DOCA DMA</a>. </p> </div> </div> <p> </p> <div class="CallOutBox" data-option-Tip> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-check"></use></svg> </div><div class="CallOutBox-title">Tip</div> <div class="CallOutBox-body"> <p> To better understand DOCA core objects like <code class="InlineCodeBlock"><span>doca_mmap</span></code> or <code class="InlineCodeBlock"><span>doca_buf_array</span></code>, please refer to the <a class="Link" href="/doca/sdk/DOCA+Core" data-cms-ai="0" >DOCA Core</a>.</p><p></p> <p></p> </div> </div> <p> </p> <p>All DOCA Core and Ethernet object used with GPUNetIO have a GPU export function to obtain a GPU handler for that object. The following are a few examples:</p> <ul class=" "><li class=" "><p class="auto-cursor-target"> <code class="InlineCodeBlock"><span>doca_buf_array</span></code> is exported as <code class="InlineCodeBlock"><span>doca_gpu_buf_arr</span></code> :</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">DOCA buf array</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="keyword">struct</code> doca_mmap *mmap; <code class="keyword">struct</code> doca_buf_arr *buf_arr_cpu; <code class="keyword">struct</code> doca_gpu_buf_arr *buf_arr_gpu;   doca_mmap_create(&(mmap)); <code class="comments">/* Populate and start mmap */</code> doca_buf_arr_create(mmap, &buf_arr_cpu); <code class="comments">/* Populate and start buf arr attributes. Set datapath on GPU */</code> <code class="comments">/* Export the buf array CPU handler to a buf array GPU handler */</code> doca_buf_arr_get_gpu_handle(buf_arr_cpu, &(buf_arr_gpu)); <code class="comments">/* To use the GPU handler, pass it as parameter of the CUDA kernel */</code> cuda_kernel<<<...>>>(buf_arr_gpu, ...);</p></code> </pre> </div> </bsp-code-block> <p> </p> </li><li class=" "><p class="auto-cursor-target"> <code class="InlineCodeBlock"><span>doca_eth_rxq</span></code> is exported as <code class="InlineCodeBlock"><span>doca_gpu_eth_rxq</span></code> :</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">DOCA buf array</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="keyword">struct</code> doca_mmap *mmap; <code class="keyword">struct</code> doca_eth_rxq *eth_rxq_cpu; <code class="keyword">struct</code> doca_gpu_eth_rxq *eth_rxq_gpu; <code class="keyword">struct</code> doca_dev *ddev;   <code class="comments">/* Create DOCA network device ddev */</code> <code class="comments">/* Create the DOCA Ethernet receive queue */</code> doca_eth_rxq_create(ddev, MAX_NUM_PACKETS, MAX_PACKET_SIZE, &eth_rxq_cpu,); <code class="comments">/* Populate and start Ethernet receive queue attributes. Set datapath on GPU */</code> <code class="comments">/* Export the Ethernet receive queue CPU handler to a Ethernet receive queue GPU handler */</code> doca_eth_rxq_get_gpu_handle(eth_rxq_cpu, &(eth_rxq_gpu)); <code class="comments">/* To use the GPU handler, pass it as parameter of the CUDA kernel */</code> cuda_kernel<<<...>>>(eth_rxq_gpu, ...);</p></code> </pre> </div> </bsp-code-block> <p> </p> </li></ul> <h2 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-CPUFunctions"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-CPUFunctions" data-cms-ai="0">CPU Functions</a> </h2> <p>In this section there is the list of DOCA GPUNetIO functions that can be used on the CPU only.</p> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_mem_type"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_mem_type" data-cms-ai="0">doca_gpu_mem_type</a> </h3> <p>This enum lists all the possible memory types that can be allocated with GPUNetIO.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="keyword">enum</code> doca_gpu_mem_type { DOCA_GPU_MEM_TYPE_GPU = 0, DOCA_GPU_MEM_TYPE_GPU_CPU = 1, DOCA_GPU_MEM_TYPE_CPU_GPU = 2, };</p></code> </pre> </div> </bsp-code-block> <p> </p> <div class="CallOutBox" data-option-Note> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-check"></use></svg> </div><div class="CallOutBox-title">Note</div> <div class="CallOutBox-body"> <p>With regards to the syntax, the text string after the <code class="InlineCodeBlock"><span>DOCA_GPU_MEM_TYPE_</span></code> prefix signifies <code class="InlineCodeBlock"><span><where-memory-resides>_<who-has-access></who-has-access></where-memory-resides></span></code>.</p> </div> </div> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>DOCA_GPU_MEM_TYPE_GPU</span></code> – memory resides on the GPU and is accessible from the GPU only</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>DOCA_GPU_MEM_TYPE_GPU_CPU</span></code> – memory resides on the GPU and is accessible also by the CPU</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>DOCA_GPU_MEM_TYPE_CPU_GPU</span></code> – memory resides on the CPU and is accessible also by the GPU</p> </li></ul><p>Typical usage of the <code class="InlineCodeBlock"><span>DOCA_GPU_MEM_TYPE_GPU_CPU</span></code> memory type is to send a notification from the CPU to the GPU (e.g., a CUDA kernel periodically checking to see if the exit condition set by the CPU is met).</p><p> </p> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_create"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_create" data-cms-ai="0">doca_gpu_create</a> </h3> <p>This is the first function a GPUNetIO application must invoke to create an handler on a GPU device. The function initializes a pointer to a structure in memory with type <code class="InlineCodeBlock"><span>struct doca_gpu *</span></code>.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>doca_error_t doca_gpu_create(<code class="keyword">const</code> <code class="color1">char</code> *gpu_bus_id, <code class="keyword">struct</code> doca_gpu **gpu_dev);</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>gpu_bus_id</span></code> – <code class="InlineCodeBlock"><span><pcie-bus>:<device>.<function></function></device></pcie-bus></span></code> of the GPU device you want to use in your application</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>gpu_dev [out]</span></code> – GPUNetIO handler to that GPU device</p> </li></ul><p>To get the PCIe address, users can use the commands <code class="InlineCodeBlock"><span>lspci</span></code> or <code class="InlineCodeBlock"><span>nvidia-smi</span></code>.</p><p> </p> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_mem_alloc"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_mem_alloc" data-cms-ai="0">doca_gpu_mem_alloc</a> </h3> <p>This CPU function allocates different flavors of memory.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>doca_error_t doca_gpu_mem_alloc(<code class="keyword">struct</code> doca_gpu *gpu_dev, <code class="color1">size_t</code> size, <code class="color1">size_t</code> alignment, <code class="keyword">enum</code> doca_gpu_mem_type mtype, <code class="keyword">void</code> **memptr_gpu, <code class="keyword">void</code> **memptr_cpu)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>gpu_dev</span></code> – GPUNetIO device handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>size</span></code> – Size, in bytes, of the memory area to allocate</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>alignment</span></code> – Memory address alignment to use. If 0, default one will be used</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>mtype</span></code> – Type of memory to allocate</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>memptr_gpu [out]</span></code> – GPU pointer to use to modify that memory from the GPU if memory is allocated on or is visible by the GPU</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>memptr_cpu</span></code><code class="InlineCodeBlock"><span>[out]</span></code> – CPU pointer to use to modify that memory from the CPU if memory is allocated on or is visible by the CPU. Can be NULL if memory is GPU-only</p> </li></ul><p></p><p></p><p></p> <div class="CallOutBox" data-option-Warning> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-info"></use></svg> </div><div class="CallOutBox-title">Warning</div> <div class="CallOutBox-body"> <p>Make sure to use the right pointer on the right device! I f an application tries to access the memory using the <code class="InlineCodeBlock"><span>memptr_gpu</span></code> address from the CPU, a segmentation fault will result. </p> </div> </div> <p> <br> </p> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_create"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_create" data-cms-ai="0"><span id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_create" class="confluence-anchor-link"></span>doca_gpu_semaphore_create</a> </h3> <p>Creates a new instance of a DOCA GPUNetIO semaphore. A semaphore is composed by a list of items each having, by default, a status flag, number of packets, and the index of a <code class="InlineCodeBlock"><span>doca_gpu_buf</span></code> in a <code class="InlineCodeBlock"><span>doca_gpu_buf_arr</span></code>.</p> <p>For example, a GPUNetIO semaphore can be used in applications where a CUDA kernel is responsible for receiving packets in a <code class="InlineCodeBlock"><span>doca_gpu_buf_arr</span></code> array associated with an Ethernet receive queue object, <code class="InlineCodeBlock"><span>doca_gpu_eth_rxq</span></code> (see section "<a class="Link" href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfZXRoX3J4cV9yZWNlaXZlXyo" data-cms-ai="0" >doca_gpu_dev_eth_rxq_receive_*</a>"), and dispatching packet info to a second CUDA kernel which processes them.</p> <p>Another way to use a GPUNetIO semaphore is to exchange data across different entities like two CUDA kernels or a CUDA kernel and a CPU thread. The reason for this scenario may be that the CUDA kernel needs to provide the outcome of the packet processing to the CPU which would in turn compile a statistics report. Therefore, it is possible to associate a custom application-defined structure to each item in the semaphore. This way, the semaphore can be used as a message passing object.</p> <p>Both situations are illustrated in the "<a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ReceiveandProcess" data-cms-ai="0" >Receive and Process</a>" section.</p> <p></p><p></p><div class="Enhancement" data-align-inline has-width style="--image-width: 450px"> <div class="Enhancement-item"> <figure class="Figure"> <picture> <source type="image/webp" width="877" height="917" srcset="https://docscontent.nvidia.com/dims4/default/1a0e6e5/2147483647/strip/true/crop/877x917+0+0/resize/877x917!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-18_12-6-25-version-1-modificationdate-1725534957027-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/dcee3d8/2147483647/strip/true/crop/877x917+0+0/resize/1754x1834!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-18_12-6-25-version-1-modificationdate-1725534957027-api-v2.png 2x"> <source type="image/png" width="877" height="917" srcset="https://docscontent.nvidia.com/dims4/default/941e456/2147483647/strip/true/crop/877x917+0+0/resize/877x917!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-18_12-6-25-version-1-modificationdate-1725534957027-api-v2.png"> <img class="Image" alt="image2023-4-18_12-6-25-version-1-modificationdate-1725534957027-api-v2.png" srcset="https://docscontent.nvidia.com/dims4/default/941e456/2147483647/strip/true/crop/877x917+0+0/resize/877x917!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-18_12-6-25-version-1-modificationdate-1725534957027-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/5123957/2147483647/strip/true/crop/877x917+0+0/resize/1754x1834!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-18_12-6-25-version-1-modificationdate-1725534957027-api-v2.png 2x" width="877" height="917" src="https://docscontent.nvidia.com/dims4/default/941e456/2147483647/strip/true/crop/877x917+0+0/resize/877x917!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-18_12-6-25-version-1-modificationdate-1725534957027-api-v2.png" loading="lazy"> </picture> </figure> </div> </div><p> </p><p>Entities communicating through a semaphore must adopt a poll/update mechanism according to the following logic:</p> <ul class=" "><li class=" "><p>Update:</p> <ol class=" "><li class=" "><p>Populate the next item of the semaphore (packets' info and/or custom application-defined info).</p> </li><li class=" "><p>Set status flag to READY.</p> </li></ol></li><li class=" "><p>Poll:</p> <ol class=" "><li class=" "><p>Wait for the next item to have a status flag equal to <code class="InlineCodeBlock"><span>READY</span></code>.</p> </li><li class=" "><p>Read and process info.</p> </li><li class=" "><p>Set status flag to <code class="InlineCodeBlock"><span>DONE</span></code>.</p> </li></ol></li></ul><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>doca_error_t doca_gpu_semaphore_create(<code class="keyword">struct</code> doca_gpu *gpu_dev, <code class="keyword">struct</code> doca_gpu_semaphore **semaphore)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>gpu_dev</span></code> – GPUNetIO handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>semaphore [out]</span></code> – GPUNetIO semaphore handler associated to the GPU device</p> </li></ul> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_set_memory_type"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_set_memory_type" data-cms-ai="0">doca_gpu_semaphore_set_memory_type</a> </h3> <p>This function defines the type of memory for the semaphore allocation.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>doca_error_t doca_gpu_semaphore_set_memory_type(<code class="keyword">struct</code> doca_gpu_semaphore *semaphore, <code class="keyword">enum</code> doca_gpu_mem_type mtype)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>semaphore</span></code> – GPUNetIO semaphore handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>mtype</span></code> – Type of memory to allocate the custom info structure</p> <ul class=" "><li class=" "><p>If the application must share packet info only across CUDA kernels, then <code class="InlineCodeBlock"><span>DOCA_GPU_MEM_GPU</span></code> is the suggested memory type.</p> </li><li class=" "><p>If the application must share info from a CUDA kernel to a CPU (e.g., to report statistics or output of the pipeline computation), then <code class="InlineCodeBlock"><span>DOCA_GPU_MEM_CPU_GPU</span></code> is the suggested memory type</p> </li></ul></li></ul> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_set_items_num"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_set_items_num" data-cms-ai="0">doca_gpu_semaphore_set_items_num</a> </h3> <p>This function defines the number of items in a semaphore.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>doca_error_t doca_gpu_semaphore_set_items_num(<code class="keyword">struct</code> doca_gpu_semaphore *semaphore, uint32_t num_items)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>semaphore</span></code> – GPUNetIO semaphore handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>num_items</span></code> – Number of items to allocate</p> </li></ul> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_set_custom_info"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_set_custom_info" data-cms-ai="0">doca_gpu_semaphore_set_custom_info</a> </h3> <p>This function associates an application-specific structure to semaphore items as explained under "<a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_create" data-cms-ai="0" >doca_gpu_semaphore_create</a>".</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>doca_error_t doca_gpu_semaphore_set_custom_info(<code class="keyword">struct</code> doca_gpu_semaphore *semaphore, uint32_t nbytes, <code class="keyword">enum</code> doca_gpu_mem_type mtype)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>semaphore</span></code> – GPUNetIO semaphore handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>nbytes</span></code> – Size of the custom info structure to associate</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>mtype</span></code> – Type of memory to allocate the custom info structure</p> <ul class=" "><li class=" "><p>If the application must share packet info only across CUDA kernels, then <code class="InlineCodeBlock"><span>DOCA_GPU_MEM_GPU</span></code> is the suggested memory type</p> </li><li class=" "><p>If the application must share info from a CUDA kernel to a CPU (e.g., to report statistics or output of the pipeline computation), then <code class="InlineCodeBlock"><span>DOCA_GPU_MEM_CPU_GPU</span></code> is the suggested memory type</p> </li></ul></li></ul> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_get_status"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_get_status" data-cms-ai="0">doca_gpu_semaphore_get_status</a> </h3> <p>From the CPU, query the status of a semaphore item. If the semaphore is allocated with <code class="InlineCodeBlock"><span>DOCA_GPU_MEM_GPU</span></code>, this function results in a segmentation fault.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>doca_error_t doca_gpu_semaphore_get_status(<code class="keyword">struct</code> doca_gpu_semaphore *semaphore_cpu, uint32_t idx, <code class="keyword">enum</code> doca_gpu_semaphore_status *status)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>semaphore_cpu</span></code> – GPUNetIO semaphore CPU handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>idx</span></code> – Semaphore item index</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>status [out]</span></code> – Output semaphore status</p> </li></ul> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_get_custom_info_addr"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_semaphore_get_custom_info_addr" data-cms-ai="0">doca_gpu_semaphore_get_custom_info_addr</a> </h3> <p>From the CPU, retrieve the address of the custom info structure associated to a semaphore item. If the semaphore or the custom info is allocated with <code class="InlineCodeBlock"><span>DOCA_GPU_MEM_GPU</span></code> this function results in a segmentation fault.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>doca_error_t doca_gpu_semaphore_get_custom_info_addr(<code class="keyword">struct</code> doca_gpu_semaphore *semaphore_cpu, uint32_t idx, <code class="keyword">void</code> **custom_info)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>semaphore_cpu</span></code> – GPUNetIO semaphore CPU handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>idx</span></code> – Semaphore item index</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>custom_info [out]</span></code> – Output semaphore custom info address</p> </li></ul><p> </p> <h2 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-DOCAPE"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-DOCAPE" data-cms-ai="0">DOCA PE</a> </h2> <p>A DOCA Ethernet Txq context, exported for GPUNetIO usage, can be tracked via DOCA PE on the CPU side to check if there are errors when sending packets or to retrieve notification info after sending a packet with any of the <code class="InlineCodeBlock"><span>doca_gpu_dev_eth_txq_*_enqueue_*</span></code> functions on the GPU. An example can be found in the DOCA GPU packet processing application with ICMP traffic.</p><p></p> <p></p><p> </p> <h2 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-StrongModevs.WeakMode"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-StrongModevs.WeakMode" data-cms-ai="0">Strong Mode vs. Weak Mode</a> </h2> <p>Some Ethernet and RDMA GPU functions present two modes of operation: Weak and strong.</p> <ul class=" "><li class=" "><p>In weak mode, the application calculates the next available position in the queue. With the help of functions like <code class="InlineCodeBlock"><span>doca_gpu_eth_txq_get_info</span></code>, <code class="InlineCodeBlock"><span>doca_gpu_rdma_get_info</span></code>, or <code class="InlineCodeBlock"><span>doca_gpu_dev_rdma_recv_get_info</span></code> it is possible to know the next available position in the queue and the mask of the number of total entries in the queue (so the incremental descriptor index can be wrapped). In this mode, the developer must specify a queue descriptor number for where to enqueue the packet, ensuring that no descriptor in the queue is left empty. It's a bit more complex to manage but it should result in better performance and developer can emphasize GPU memory coalescing enqueuing sequential operations using sequential memory locations.</p> </li><li class=" "><p class="auto-cursor-target">In strong mode, the GPU function enqueues the Ethernet/RDMA operation in the next available position in the queue. It is simpler to manage as developer does not have to worry about operation's position, but it may introduce an extra latency to atomically guarantee the access of multiple threads to the same queue. Moreover, it does not guarantee that sequential operations refer to sequential memory locations.</p> <div class="CallOutBox" data-option-Note> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-check"></use></svg> </div><div class="CallOutBox-title">Note</div> <div class="CallOutBox-body"> <p>All strong mode functions work at the CUDA block level. That is, it is not possible to access the same Eth/RDMA queue at the same time from two different CUDA blocks.</p> </div> </div> <p> </p> </li></ul><p>In sections "<a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ProduceandSend" data-cms-ai="0" >Produce and Send</a>" and "<a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-CUDAKernelforRDMAWrite" data-cms-ai="0" >CUDA Kernel for RDMA Write</a>", there are a few examples about how to use the weak mode API.</p><p> </p> <h2 id="src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1HUFVGdW5jdGlvbnPigJNFdGhlcm5ldA"> <a href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1HUFVGdW5jdGlvbnPigJNFdGhlcm5ldA" data-cms-ai="0">GPU Functions – Ethernet</a> </h2> <p>This section provides a list of DOCA GPUNetIO functions that can be used for Ethernet network operations on the GPU only within a CUDA kernel.</p> <h3 id="src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfZXRoX3J4cV9yZWNlaXZlXyo"> <a href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfZXRoX3J4cV9yZWNlaXZlXyo" data-cms-ai="0"><span id="src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfZXRoX3J4cV9yZWNlaXZlXyo" class="confluence-anchor-link"></span>doca_gpu_dev_eth_rxq_receive_*</a> </h3> <p>To acquire packets in a CUDA kernel, DOCA GPUNetIO offers different flavors of the receive function for different scopes: per CUDA block, per CUDA warp, and per CUDA thread.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_eth_rxq_receive_block(<code class="keyword">struct</code> doca_gpu_eth_rxq *eth_rxq, uint32_t max_rx_pkts, uint64_t timeout_ns, uint32_t *num_rx_pkts, uint64_t *doca_gpu_buf_idx) __device__ doca_error_t doca_gpu_dev_eth_rxq_receive_warp(<code class="keyword">struct</code> doca_gpu_eth_rxq *eth_rxq, uint32_t max_rx_pkts, uint64_t timeout_ns, uint32_t *num_rx_pkts, uint64_t *doca_gpu_buf_idx) __device__ doca_error_t doca_gpu_dev_eth_rxq_receive_thread(<code class="keyword">struct</code> doca_gpu_eth_rxq *eth_rxq, uint32_t max_rx_pkts, uint64_t timeout_ns, uint32_t *num_rx_pkts, uint64_t *doca_gpu_buf_idx)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>eth_rxq</span></code> – Ethernet receive queue GPU handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>max_rx_pkts</span></code> – Maximum number of packets to receive. It ensures the number of packets returned by the function is lower or equal to this number.</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>timeout_ns</span></code> – Nanoseconds to wait for packets before returning</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>num_rx_pkts [out]</span></code> – Effective number of received packets. With CUDA block or warp scopes, this variable should be visible in memory by all the other threads (shared or global memory).</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>doca_gpu_buf_idx [out]</span></code> – DOCA buffer index of the first packet received in this function. With CUDA block or warp scopes, this variable should be visible in memory by all the other threads (shared or global memory).</p> </li></ul> <div class="CallOutBox" data-option-Note> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-check"></use></svg> </div><div class="CallOutBox-title">Note</div> <div class="CallOutBox-body"> <p>If both <code class="InlineCodeBlock"><span>max_rx_pkts</span></code> and <code class="InlineCodeBlock"><span>timeout_ns</span></code> are 0, the function never returns.</p> </div> </div> <p> </p> <p>CUDA threads in the same scope (thread, warp, or block) must invoke the function on the same receive queue. The output parameters <code class="InlineCodeBlock"><span>num_rx_pkts</span></code> and <code class="InlineCodeBlock"><span>doca_gpu_buf_idx</span></code> must be visible by all threads in the scope (e.g., CUDA shared memory for warp and block).</p> <p>Each packet received by this function goes to the <code class="InlineCodeBlock"><span>doca_gpu_buf_arr</span></code> internally created and associated with the Ethernet queues (see section "<a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-BuildingBlocks" data-cms-ai="0" >Building Blocks</a>").</p> <p>The function exits when <code class="InlineCodeBlock"><span>timeout_ns</span></code> is reached or when the maximum number of packets is received.</p> <div class="CallOutBox" data-option-Note> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-check"></use></svg> </div><div class="CallOutBox-title">Note</div> <div class="CallOutBox-body"> <p>For CUDA block scope, the block invoking the receive function must have at least 32 CUDA threads (i.e., one warp).</p> </div> </div> <p> </p> <p>The output parameters indicate how many packets have been received (<code class="InlineCodeBlock"><span>num_rx_pkts</span></code>) and the index of the first received packet in the <code class="InlineCodeBlock"><span>doca_gpu_buf_arr</span></code> internally associated with the Ethernet receive queue. Packets are stored consecutively in the <code class="InlineCodeBlock"><span>doca_gpu_buf_arr</span></code> so if the function returns <code class="InlineCodeBlock"><span>num_rx_pkts=N</span></code> and <code class="InlineCodeBlock"><span>doca_gpu_buf_idx=X</span></code>, this means that all the <code class="InlineCodeBlock"><span>doca_gpu_buf</span></code> in the <code class="InlineCodeBlock"><span>doca_gpu_buf_arr</span></code> within the range <code class="InlineCodeBlock"><span>[X, .. ,X + (N-1)]</span></code> have been filled with packets.</p> <p></p><p></p><div class="Enhancement" data-align-inline has-height style="--image-height: 150px"> <div class="Enhancement-item"> <figure class="Figure"> <picture> <source type="image/webp" width="1440" height="283" srcset="https://docscontent.nvidia.com/dims4/default/f0807bc/2147483647/strip/true/crop/3116x613+0+0/resize/1440x283!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_16-38-42-version-1-modificationdate-1725534949637-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/1cf3bac/2147483647/strip/true/crop/3116x613+0+0/resize/2880x566!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_16-38-42-version-1-modificationdate-1725534949637-api-v2.png 2x"> <source type="image/png" width="1440" height="283" srcset="https://docscontent.nvidia.com/dims4/default/eda1879/2147483647/strip/true/crop/3116x613+0+0/resize/1440x283!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_16-38-42-version-1-modificationdate-1725534949637-api-v2.png"> <img class="Image" alt="image-2024-6-26_16-38-42-version-1-modificationdate-1725534949637-api-v2.png" srcset="https://docscontent.nvidia.com/dims4/default/eda1879/2147483647/strip/true/crop/3116x613+0+0/resize/1440x283!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_16-38-42-version-1-modificationdate-1725534949637-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/abe12e4/2147483647/strip/true/crop/3116x613+0+0/resize/2880x566!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_16-38-42-version-1-modificationdate-1725534949637-api-v2.png 2x" width="1440" height="283" src="https://docscontent.nvidia.com/dims4/default/eda1879/2147483647/strip/true/crop/3116x613+0+0/resize/1440x283!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_16-38-42-version-1-modificationdate-1725534949637-api-v2.png" loading="lazy"> </picture> </figure> </div> </div><p> </p><p>The DOCA buffer array is treated in a circular fashion so that once the last DOCA buffer is filled by a packet, the queue circles back to the first DOCA buffer. There is no need for the application to lock or free <code class="InlineCodeBlock"><span>doca_gpu_buf_arr</span></code> buffers.</p> <div class="CallOutBox" data-option-Note> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-check"></use></svg> </div><div class="CallOutBox-title">Note</div> <div class="CallOutBox-body"> <p>It is the application's responsibility to consume packets before they are overwritten when circling back, properly dimensioning the DOCA buffer array size and scaling across multiple receive queues.</p> </div> </div> <p> <br> </p> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_send_flags"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_send_flags" data-cms-ai="0">doca_gpu_send_flags</a> </h3> <p>This enum lists all the possible flags for the txq functions. The usage of those flags makes sense if a DOCA PE has been attached to the DOCA Ethernet Txq context with GPU data path and a CPU thread, in a loop, keeps invoking <code class="InlineCodeBlock"><span>doca_pe_progress</span></code>.</p> <div class="CallOutBox" data-option-Warning> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-info"></use></svg> </div><div class="CallOutBox-title">Warning</div> <div class="CallOutBox-body"> <p>If no DOCA PE has been attached to the DOCA Ethernet Txq context, it's mandatory to use the <code class="InlineCodeBlock"><span>DOCA_GPU_SEND_FLAG_NONE</span></code> flag.</p> </div> </div> <p> </p> <p></p><p></p><p></p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="keyword">enum</code> doca_gpu_mem_type { DOCA_GPU_SEND_FLAG_NONE = 0, DOCA_GPU_SEND_FLAG_NOTIFY = 1 << 0, };</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>DOCA_GPU_SEND_FLAG_NONE</span></code> (default) – send is executed and no notification info is returned. If an error occurs, an event is generated. This error can be detected from the CPU side using DOCA PE.</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>DOCA_GPU_SEND_FLAG_NOTIFY</span></code> – once the send (or wait) is executed, return a notification with packet info. This notification can be detected from the CPU side using DOCA PE.</p> </li></ul> <h3 id="src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfZXRoX3R4cV9zZW5kXyo"> <a href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfZXRoX3R4cV9zZW5kXyo" data-cms-ai="0">doca_gpu_dev_eth_txq_send_*</a> </h3> <p>To send packets from a CUDA kernel, DOCA GPUNetIO offers a strong and weak modes for enqueuing a packet in the Ethernet TXQ. For both modes, the scope is the single CUDA thread each populating and enqueuing a different <code class="InlineCodeBlock"><span>doca_gpu_buf</span></code> from a <code class="InlineCodeBlock"><span>doca_gpu_buf_arr</span></code> in the send queue.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_eth_txq_get_info(<code class="keyword">struct</code> doca_gpu_eth_txq *eth_txq, uint32_t *curr_position, uint32_t *mask_max_position)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>eth_txq</span></code> – Ethernet send queue GPU handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>curr_position</span></code> – Next available position in the queue</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>mask_max_position</span></code> – Mask of the total number of positions in the queue</p> </li></ul><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_eth_txq_send_enqueue_strong(<code class="keyword">struct</code> doca_gpu_eth_txq *eth_txq, <code class="keyword">const</code> <code class="keyword">struct</code> doca_gpu_buf *buf_ptr, <code class="keyword">const</code> uint32_t nbytes, <code class="keyword">const</code> uint32_t flags_bitmask)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>eth_txq</span></code> – Ethernet send queue GPU handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>buf_ptr</span></code> – DOCA buffer from a DOCA GPU buffer array to be sent</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>nbytes</span></code> – Number of bytes to be sent in the packet</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>flags_bitmask</span></code> – One of the flags in the <code class="InlineCodeBlock"><span>doca_gpu_send_flags</span></code> enum</p> </li></ul><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_eth_txq_send_enqueue_weak(<code class="keyword">const</code> <code class="keyword">struct</code> doca_gpu_eth_txq *eth_txq, <code class="keyword">const</code> <code class="keyword">struct</code> doca_gpu_buf *buf_ptr, <code class="keyword">const</code> uint32_t nbytes, <code class="keyword">const</code> uint32_t ndescr, <code class="keyword">const</code> uint32_t flags_bitmask)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>eth_txq</span></code> – Ethernet send queue GPU handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>buf_ptr</span></code> – DOCA buffer from a DOCA GPU buffer array to be sent</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>nbytes</span></code> – Number of bytes to be sent in the packet</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>ndescr</span></code> – Position in the queue to place the packet. Range: 0 - <code class="InlineCodeBlock"><span>mask_max_position</span></code>.</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>flags_bitmask</span></code> – One of the flags in the <code class="InlineCodeBlock"><span>doca_gpu_send_flags</span></code> enum</p> </li></ul> <h3 id="src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfZXRoX3R4cV93YWl0Xyo"> <a href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfZXRoX3R4cV93YWl0Xyo" data-cms-ai="0">doca_gpu_dev_eth_txq_wait_*</a> </h3> <p>To enable Accurate Send Scheduling, the "wait on time" barrier (based on timestamp) must be set in the send queue before enqueuing more packets. Like <code class="InlineCodeBlock"><span>doca_gpu_dev_eth_txq_send_*</span></code>, <code class="InlineCodeBlock"><span>doca_gpu_dev_eth_txq_wait_*</span></code> also has a strong and weak mode.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_eth_txq_wait_time_enqueue_strong(struct doca_gpu_eth_txq *eth_txq, const uint64_t wait_on_time_value, const uint32_t flags_bitmask)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>eth_txq</span></code> – Ethernet send queue GPU handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>wait_on_time_value</span></code> – Timestamp to specify when packets must be sent after this barrier</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>flags_bitmask</span></code> – One of the flags in the <code class="InlineCodeBlock"><span>doca_gpu_send_flags</span></code> enum</p> </li></ul><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_eth_txq_wait_time_enqueue_weak(<code class="keyword">struct</code> doca_gpu_eth_txq *eth_txq, <code class="keyword">const</code> uint64_t wait_on_time_value, <code class="keyword">const</code> uint32_t ndescr, <code class="keyword">const</code> uint32_t flags_bitmask)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>eth_txq</span></code> – Ethernet send queue GPU handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>wait_on_time_value</span></code> – Timestamp to specify when packets must be sent after this barrier</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>ndescr</span></code> – Position in the queue to place the packet. Range: 0 - <code class="InlineCodeBlock"><span>mask_max_position</span></code>.</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>flags_bitmask</span></code> – One of the flags in the <code class="InlineCodeBlock"><span>doca_gpu_send_flags</span></code> enum</p> </li></ul><p>Please refer to section "<a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-GPUNetIOSamples" data-cms-ai="0" >GPUNetIO Samples</a>" to understand how to enable and use Accurate Send Scheduling.</p><p> </p> <h3 id="src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfZXRoX3R4cV9jb21taXRfKg"> <a href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfZXRoX3R4cV9jb21taXRfKg" data-cms-ai="0">doca_gpu_dev_eth_txq_commit_*</a> </h3> <p>After enqueuing all the packets to be sent and time barriers, a commit function must be invoked on the txq queue. The right commit function must be used according to the type of enqueue mode (i.e., strong or weak) used in <code class="InlineCodeBlock"><span>doca_gpu_dev_eth_txq_send_*</span></code> and <code class="InlineCodeBlock"><span>doca_gpu_dev_eth_txq_wait_*</span></code>.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_eth_txq_commit_strong(<code class="keyword">struct</code> doca_gpu_eth_txq *eth_txq)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>eth_txq</span></code> – Ethernet send queue GPU handler</p> </li></ul><p></p><p></p><p></p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_eth_txq_commit_weak(<code class="keyword">struct</code> doca_gpu_eth_txq *eth_txq, <code class="keyword">const</code> uint32_t descr_num)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>eth_txq</span></code> – Ethernet send queue GPU handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>descr_num</span></code> – Number of queue items enqueued thus far</p> </li></ul><p>Only one CUDA thread in the scope (CUDA block or CUDA warp) can invoke this function on the send queue after several enqueue operations. Typical flow is as follows:</p> <ol class=" "><li class=" "><p>All threads in the scope enqueue packets in the send queue.</p> </li><li class=" "><p>Synchronization point.</p> </li><li class=" "><p>Only one thread in the scope performs the send queue commit.</p> </li></ol> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_dev_eth_txq_push"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_dev_eth_txq_push" data-cms-ai="0">doca_gpu_dev_eth_txq_push</a> </h3> <p>After committing, the items in the send queue must be actually pushed to the network card.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_eth_txq_push(<code class="keyword">struct</code> doca_gpu_eth_txq *eth_txq)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>eth_txq</span></code> – Ethernet send queue GPU handler</p> </li></ul><p>Only one CUDA thread in the scope (CUDA block or CUDA warp) can invoke this function on the send queue after several enqueue or commit operations. Typical flow is as follows:</p> <ol class=" "><li class=" "><p>All threads in the scope enqueue packets in the send queue.</p> </li><li class=" "><p>Synchronization point.</p> </li><li class=" "><p>Only one thread in the scope does the send queue commit.</p> </li><li class=" "><p>Only one thread in the scope does the send queue push.</p> </li></ol><p>Section "<a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ProduceandSend" data-cms-ai="0" >Produce and Send</a>" provides an example where the scope is a block (e.g., each CUDA block operates on a different Ethernet send queue).</p> <p> </p> <h2 id="src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1HUFVGdW5jdGlvbnMtUkRNQUdQVUZ1bmN0aW9uc-KAk1JETUE"> <a href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1HUFVGdW5jdGlvbnMtUkRNQUdQVUZ1bmN0aW9uc-KAk1JETUE" data-cms-ai="0">GPU Functions – RDMA</a> </h2> <p>This section provides a list of DOCA GPUNetIO functions that can be used on the GPU only within a CUDA kernel to execute RDMA operations. These functions offer a strong and a weak mode.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t __device__ doca_error_t doca_gpu_dev_rdma_get_info(<code class="keyword">struct</code> doca_gpu_dev_rdma *rdma, uint32_t connection_index, uint32_t *curr_position, uint32_t *mask_max_position)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>rdma</span></code> – RDMA queue GPU handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>connection_index</span></code> – In case of RDMA CM, the connection index must be specified. By default, it is 0.</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>curr_position</span></code> – Next available position in the queue</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>mask_max_position</span></code> – Mask of the total number of positions in the queue</p> </li></ul><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t __device__ doca_error_t doca_gpu_dev_rdma_recv_get_info(<code class="keyword">struct</code> doca_gpu_dev_rdma_r *rdma_r, uint32_t *curr_position, uint32_t *mask_max_position)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>rdma_r</span></code> – RDMA receive queue GPU handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>curr_position</span></code> – Next available position in the queue</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>mask_max_position</span></code> – Mask of the total number of positions in the queue</p> </li></ul> <h3 id="src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfcmRtYV93cml0ZV8q"> <a href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfcmRtYV93cml0ZV8q" data-cms-ai="0">doca_gpu_dev_rdma_write_*</a> </h3> <p>To RDMA write data onto a remote memory location from a CUDA kernel, DOCA GPUNetIO offers strong and weak modes for enqueuing operations on the RDMA queue. For both modes, the scope is the single CUDA thread.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_rdma_write_strong(<code class="keyword">struct</code> doca_gpu_dev_rdma *rdma, uint32_t connection_index, <code class="keyword">struct</code> doca_gpu_buf *remote_buf, uint64_t remote_offset, <code class="keyword">struct</code> doca_gpu_buf *local_buf, uint64_t local_offset, <code class="color1">size_t</code> length, uint32_t imm, <code class="keyword">const</code> <code class="keyword">enum</code> doca_gpu_dev_rdma_write_flags flags)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>rdma</span></code> – RDMA queue GPU handler </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>connection_index</span></code> – In case of RDMA CM, the connection index must be specified. By default, it is 0.</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>remote_buf</span></code> – Remote DOCA buffer from a DOCA GPU buffer array to write data to</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>remote_offset</span></code> – Offset, in bytes, to write data to in the remote buffer </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>local_buf</span></code> – Local DOCA buffer from a DOCA GPU buffer array from which to fetch data to write </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>local_offset</span></code> – Offset, in bytes, to fetch data from in the local buffer </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>length</span></code> – Number of bytes to write</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>imm</span></code> – Immediate value <code class="InlineCodeBlock"><span>uint32_t</span></code></p> </li><li class=" "><p><code class="InlineCodeBlock"><span>flags</span></code> – One of the flags in the <code class="InlineCodeBlock"><span>doca_gpu_dev_rdma_write_flags</span></code> enum</p> </li></ul><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_rdma_write_weak(<code class="keyword">struct</code> doca_gpu_dev_rdma *rdma, uint32_t connection_index, <code class="keyword">struct</code> doca_gpu_buf *remote_buf, uint64_t remote_offset, <code class="keyword">struct</code> doca_gpu_buf *local_buf, uint64_t local_offset, <code class="color1">size_t</code> length, uint32_t imm, <code class="keyword">const</code> <code class="keyword">enum</code> doca_gpu_dev_rdma_write_flags flags, uint32_t position);</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>rdma</span></code> – RDMA queue GPU handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>connection_index</span></code> – In case of RDMA CM, the connection index must be specified. By default, it is 0.</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>remote_buf</span></code> – Remote DOCA buffer from a DOCA GPU buffer array to write data to</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>remote_offset</span></code> – Offset, in bytes, to write data to in the remote buffer </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>local_buf</span></code> – Local DOCA buffer from a DOCA GPU buffer array where to fetch data to write </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>local_offset</span></code> – Offset, in bytes, to fetch data in the local buffer </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>length</span></code> – Number of bytes to write</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>imm</span></code> – Immediate value <code class="InlineCodeBlock"><span>uint32_t</span></code></p> </li><li class=" "><p><code class="InlineCodeBlock"><span>flags</span></code> – One of the flags in the <code class="InlineCodeBlock"><span>doca_gpu_dev_rdma_write_flags</span></code> enum</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>position</span></code> – Position in the queue to place the RDMA operation. Range: 0 - <code class="InlineCodeBlock"><span>mask_max_position</span></code>.</p> </li></ul> <h3 id="src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfcmRtYV9yZWFkXyo"> <a href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfcmRtYV9yZWFkXyo" data-cms-ai="0">doca_gpu_dev_rdma_read_*</a> </h3> <p>To RDMA read data onto a remote memory location from a CUDA kernel, DOCA GPUNetIO offers strong and weak modes to enqueue operations on the RDMA queue. For both modes, the scope is the single CUDA thread.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_rdma_read_strong(<code class="keyword">struct</code> doca_gpu_dev_rdma *rdma, uint32_t connection_index, <code class="keyword">struct</code> doca_gpu_buf *remote_buf, uint64_t remote_offset, <code class="keyword">struct</code> doca_gpu_buf *local_buf, uint64_t local_offset, <code class="color1">size_t</code> length, <code class="keyword">const</code> uint32_t flags_bitmask)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>rdma</span></code> – RDMA queue GPU handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>connection_index</span></code> – In case of RDMA CM, the connection index must be specified. By default, it is 0.</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>remote_buf</span></code> – Remote DOCA buffer from a DOCA GPU buffer array where to read data</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>remote_offset</span></code> – Offset in bytes to read data to in the remote buffer </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>local_buf</span></code> – Local DOCA buffer from a DOCA GPU buffer array where to store remote data </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>local_offset</span></code> – Offset in bytes to store data in the local buffer </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>length</span></code> – Number of bytes to be read</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>flags_bitmask</span></code> – Must be 0; reserved for future use</p> </li></ul><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_rdma_read_weak(<code class="keyword">struct</code> doca_gpu_dev_rdma *rdma, uint32_t connection_index, <code class="keyword">struct</code> doca_gpu_buf *remote_buf, uint64_t remote_offset, <code class="keyword">struct</code> doca_gpu_buf *local_buf, uint64_t local_offset, <code class="color1">size_t</code> length, <code class="keyword">const</code> uint32_t flags_bitmask, uint32_t position);</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>rdma</span></code> – RDMA queue GPU handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>connection_index</span></code> – In case of RDMA CM, the connection index must be specified. By default, it is 0.</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>remote_buf</span></code> – Remote DOCA buffer from a DOCA GPU buffer array where to read data</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>remote_offset</span></code> – Offset in bytes to read data to in the remote buffer </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>local_buf</span></code> – Local DOCA buffer from a DOCA GPU buffer array where to store remote data </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>local_offset</span></code> – Offset in bytes to store data in the local buffer </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>length</span></code> – Number of bytes to be read</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>flags_bitmask</span></code> – Must be 0; reserved for future use</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>position</span></code> – Position in the queue to place the RDMA operation. Range: 0 - <code class="InlineCodeBlock"><span>mask_max_position</span></code>.</p> </li></ul> <h3 id="src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfcmRtYV9zZW5kXyo"> <a href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfcmRtYV9zZW5kXyo" data-cms-ai="0">doca_gpu_dev_rdma_send_*</a> </h3> <p>To RDMA send data from a CUDA kernel, DOCA GPUNetIO offers strong and weak modes for enqueuing operations on the RDMA queue. For both modes, the scope is the single CUDA thread.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_rdma_send_strong(<code class="keyword">struct</code> doca_gpu_dev_rdma *rdma, uint32_t connection_index, <code class="keyword">struct</code> doca_gpu_buf *local_buf, uint64_t local_offset, <code class="color1">size_t</code> length, uint32_t imm, <code class="keyword">const</code> <code class="keyword">enum</code> doca_gpu_dev_rdma_write_flags flags)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>rdma</span></code> – RDMA queue GPU handler </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>connection_index</span></code> – In case of RDMA CM, the connection index must be specified. By default, it is 0.</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>local_buf</span></code> – Local DOCA buffer from a DOCA GPU buffer array from which to fetch data to send </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>local_offset</span></code> – Offset in bytes to fetch data in the local buffer </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>length</span></code> – Number of bytes to send</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>imm</span></code> – Immediate value <code class="InlineCodeBlock"><span>uint32_t</span></code></p> </li><li class=" "><p><code class="InlineCodeBlock"><span>flags</span></code> – One of the flags in the <code class="InlineCodeBlock"><span>doca_gpu_dev_rdma_write_flags</span></code> enum</p> </li></ul><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_rdma_send_weak(<code class="keyword">struct</code> doca_gpu_dev_rdma *rdma, uint32_t connection_index, <code class="keyword">struct</code> doca_gpu_buf *local_buf, uint64_t local_offset, <code class="color1">size_t</code> length, uint32_t imm, <code class="keyword">const</code> <code class="keyword">enum</code> doca_gpu_dev_rdma_write_flags flags, uint32_t position);</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>rdma</span></code> – RDMA queue GPU handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>connection_index</span></code> – In case of RDMA CM, the connection index must be specified. By default, it is 0.</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>local_buf</span></code> – Local DOCA buffer from a DOCA GPU buffer array from which to fetch data to send </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>local_offset</span></code> – Offset in bytes to fetch data in the local buffer </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>length</span></code> – Number of bytes to send</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>imm</span></code> – Immediate value <code class="InlineCodeBlock"><span>uint32_t</span></code></p> </li><li class=" "><p><code class="InlineCodeBlock"><span>flags</span></code> – One of the flags in the <code class="InlineCodeBlock"><span>doca_gpu_dev_rdma_write_flags</span></code> enum</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>position</span></code> – Position in the queue to place the RDMA operation. Range: 0 - <code class="InlineCodeBlock"><span>mask_max_position</span></code>.</p> </li></ul> <h3 id="src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfcmRtYV9jb21taXRfKg"> <a href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfcmRtYV9jb21taXRfKg" data-cms-ai="0">doca_gpu_dev_rdma_commit_*</a> </h3> <p>Once all RDMA write, send or read requests have been enqueue in the RDMA queue, a synchronization point must be reached to consolidate and execute those requests. Only 1 CUDA thread can invoke this function at a time.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_rdma_commit_strong(<code class="keyword">struct</code> doca_gpu_dev_rdma *rdma, uint32_t connection_index)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>rdma</span></code> – RDMA queue GPU handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>connection_index</span></code> – In case of RDMA CM, the connection index must be specified. By default, it is 0.</p> </li></ul><p></p><p></p><p></p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_rdma_commit_weak(<code class="keyword">struct</code> doca_gpu_dev_rdma *rdma, uint32_t connection_index, uint32_t num_ops)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>rdma</span></code> – RDMA queue GPU handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>connection_index</span></code> – In case of RDMA CM, the connection index must be specified. By default, it is 0.</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>num_ops</span></code> – Number of RDMA requests enqueued since the last commit</p> </li></ul> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_dev_rdma_wait_all"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_dev_rdma_wait_all" data-cms-ai="0">doca_gpu_dev_rdma_wait_all</a> </h3> <p>After a commit, RDMA requests are executed by the network card as applications move forward doing other operations. If the application needs to verify all RDMA operations have been done by the network card, this "wait all" function can be used to wait for all previous posted operations. Only 1 CUDA thread can invoke this function at a time.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_rdma_wait_all(<code class="keyword">struct</code> doca_gpu_dev_rdma *rdma, uint32_t *num_commits)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>rdma</span></code> – RDMA queue GPU handler</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>num_commits</span></code> – Output parameter; the number of commit operations completed</p><p></p> <p></p> </li></ul> <div class="CallOutBox" data-option-Info> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-info"></use></svg> </div><div class="CallOutBox-title">Info</div> <div class="CallOutBox-body"> <p>This function is optional, and it can be used to ensure all the RDMA Send/Write/Read operations have actually been executed before moving forward with the application.</p> </div> </div> <p> <br> </p> <h3 id="src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfcmRtYV9yZWN2Xyo"> <a href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfcmRtYV9yZWN2Xyo" data-cms-ai="0">doca_gpu_dev_rdma_recv_*</a> </h3> <p>To receive data from an RDMA send, send with immediate, or write with immediate, the destination peer should post a receive operation. DOCA GPUNetIO RDMA receive operations must be done with a <code class="InlineCodeBlock"><span>doca_gpu_dev_rdma_r</span></code> handler. This handler can be obtained with the function <code class="InlineCodeBlock"><span>doca_gpu_dev_rdma_get_recv</span></code>.</p> <div class="CallOutBox" data-option-Note> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-check"></use></svg> </div><div class="CallOutBox-title">Note</div> <div class="CallOutBox-body"> <p>All receive operations must use this object.</p> </div> </div> <p> </p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_rdma_get_recv(<code class="keyword">struct</code> doca_gpu_dev_rdma *rdma, <code class="keyword">struct</code> doca_gpu_dev_rdma_r **rdma_r)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>rdma</span></code> – RDMA queue GPU handler </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>rdma_r</span></code> – RDMA receive queue GPU handler </p> </li></ul><p>Even for the receive side, in this case, DOCA GPUNetIO offers strong and weak modes for enqueuing operations on the RDMA queue. For both modes, the scope is the single CUDA thread.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_rdma_recv_strong(<code class="keyword">struct</code> doca_gpu_dev_rdma_r *rdma_r, <code class="keyword">struct</code> doca_gpu_buf *recv_buf, <code class="color1">size_t</code> recv_length, uint64_t recv_offset, <code class="keyword">const</code> uint32_t flags_bitmask)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>rdma_r</span></code> – RDMA receive queue GPU handler </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>recv_buf</span></code> – Local DOCA buffer from a DOCA GPU buffer array from which to fetch data to send </p> </li><li class=" "><p> <code class="InlineCodeBlock"><span>recv_length</span></code> – Number of bytes to send</p><p></p> <p></p> </li><li class=" "><p><code class="InlineCodeBlock"><span>recv_offset</span></code> – Offset in bytes to fetch data in the local buffer </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>flags_bitmask</span></code> – Must be 0; reserved for future use</p> </li></ul><p></p><p></p><p></p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_rdma_recv_weak(<code class="keyword">struct</code> doca_gpu_dev_rdma_r *rdma_r, <code class="keyword">struct</code> doca_gpu_buf *recv_buf, <code class="color1">size_t</code> recv_length, uint64_t recv_offset, <code class="keyword">const</code> uint32_t flags_bitmask, uint32_t position);</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>rdma_r</span></code> – RDMA receive queue GPU handler </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>recv_buf</span></code> – Local DOCA buffer from a DOCA GPU buffer array from which to fetch data to send </p> </li><li class=" "><p> <code class="InlineCodeBlock"><span>recv_length</span></code> – Number of bytes to send</p><p></p> <p></p> </li><li class=" "><p><code class="InlineCodeBlock"><span>recv_offset</span></code> – Offset in bytes to fetch data in the local buffer </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>flags_bitmask</span></code> - Must be 0; reserved for future use</p> </li><li class=" "><p><code class="InlineCodeBlock"><span>position</span></code> – Position in the queue to place the RDMA operation. Range: 0 - <code class="InlineCodeBlock"><span>mask_max_position</span></code>.</p> </li></ul> <h3 id="src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfcmRtYV9yZWN2X2NvbW1pdF8q"> <a href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1kb2NhX2dwdV9kZXZfcmRtYV9yZWN2X2NvbW1pdF8q" data-cms-ai="0">doca_gpu_dev_rdma_recv_commit_*</a> </h3> <p>After posting several RDMA receive operations, a commit function must be invoked to activate the receive in the queue. Only 1 CUDA thread can invoke this function at a time.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_rdma_recv_commit_strong(<code class="keyword">struct</code> doca_gpu_dev_rdma_r *rdma_r)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>rdma_r</span></code> – RDMA receive queue GPU handler </p> </li></ul><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_rdma_recv_commit_weak(<code class="keyword">struct</code> doca_gpu_dev_rdma_r *rdma_r, uint32_t num_ops)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>rdma_r</span></code> – RDMA receive queue GPU handler </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>num_ops</span></code> – Number of RDMA receive requests enqueued since the last commit</p> </li></ul> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_dev_rdma_recv_wait_all"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_dev_rdma_recv_wait_all" data-cms-ai="0">doca_gpu_dev_rdma_recv_wait_all</a> </h3> <p>This function waits for the completion of all previously posted RDMA receive operation. Only 1 CUDA thread can invoke this function at a time. It works in blocking or non-blocking mode.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p> <code class="keyword">enum</code> doca_gpu_dev_rdma_recv_wait_flags { DOCA_GPU_RDMA_RECV_WAIT_FLAG_NB = 0, <code class="comments">/**< Non-Blocking mode: the wait receive function doca_gpu_dev_rdma_recv_wait</code> <code class="comments"> * checks if the receive operation happened (data has been received)</code> <code class="comments"> * and exit from the function. If nothing has been received,</code> <code class="comments"> * the function doesn't block the execution.</code> <code class="comments"> */</code> DOCA_GPU_RDMA_RECV_WAIT_FLAG_B = 1, <code class="comments">/**< Blocking mode: the wait receive function doca_gpu_dev_rdma_recv_wait</code> <code class="comments"> * blocks the execution waiting for the receive operations to be executed.</code> <code class="comments"> */</code> };</p></code> </pre> </div> </bsp-code-block> <p> </p> <p>Function:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_rdma_recv_wait_all(<code class="keyword">struct</code> doca_gpu_dev_rdma_r *rdma_r, <code class="keyword">const</code> <code class="keyword">enum</code> doca_gpu_dev_rdma_recv_wait_flags flags, uint32_t *num_ops, uint32_t *imm_val)</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>rdma_r</span></code> – RDMA receive queue GPU handler </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>flags</span></code> – receive flags</p><p></p> <p></p> </li><li class=" "><p><code class="InlineCodeBlock"><span>num_ops</span></code> – Output parameter. Function reports number of completed operations. </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>imm_val</span></code> – Output parameter. Application-provided buffer where the function can store received immediate values, if any (or 0xFFFFFFFF if no immediate value is received). If <code class="InlineCodeBlock"><span>nullptr</span></code>, the function ignores this parameter. </p> </li></ul><p> </p> <h2 id="src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1HUFVGdW5jdGlvbnMtRE1BR1BVRnVuY3Rpb25z4oCTRE1B"> <a href="#src-3137267888_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjkuMExUUy1HUFVGdW5jdGlvbnMtRE1BR1BVRnVuY3Rpb25z4oCTRE1B" data-cms-ai="0">GPU Functions – DMA</a> </h2> <p>This section provides a list of DOCA GPUNetIO functions that can be used on the GPU only within a CUDA kernel to execute DMA operations.</p> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_dev_dma_memcpy"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_dev_dma_memcpy" data-cms-ai="0">doca_gpu_dev_dma_memcpy</a> </h3> <p>This function allows a CUDA kernel to trigger a DMA memory copy operation through the DMA GPU engine. There is no strong/weak mode here, the DMA is assuming the strong behavior by default.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_dma_memcpy(<code class="keyword">struct</code> doca_gpu_dma *dma, <code class="keyword">struct</code> doca_gpu_buf *src_buf, uint64_t src_offset, <code class="keyword">struct</code> doca_gpu_buf *dst_buf, uint64_t dst_offset, <code class="color1">size_t</code> length);</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>dma</span></code> – DMA queue GPU handler </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>src_buf</span></code> – memcpy source buffer </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>src_offset</span></code> – fetch data starting from this source buffer offset </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>dst_buf</span></code> – memcpy destination buffer </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>dst_offset</span></code> – copy data starting from this destination buffer offset </p> </li><li class=" "><p><code class="InlineCodeBlock"><span>lenght</span></code> – number of bytes to copy</p> </li></ul> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_dev_dma_commit"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-doca_gpu_dev_dma_commit" data-cms-ai="0">doca_gpu_dev_dma_commit</a> </h3> <p>After posting several DMA memory copies, a commit function must be invoked to execute the operations enqueued in the DMA queue. Only 1 CUDA thread can invoke this function at a time.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations > <div class="CodeBlock-wrapper"> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__device__ doca_error_t doca_gpu_dev_dma_commit(<code class="keyword">struct</code> doca_gpu_dma *dma);</p></code> </pre> </div> </bsp-code-block> <p> </p> <ul class=" "><li class=" "><p><code class="InlineCodeBlock"><span>dma</span></code> – DMA queue GPU handler </p> </li></ul><p> </p> </div> </div> <div class="StepModule" data-module><span class="StepModule-anchor" id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-BuildingBlocks"></span> <div class="StepModuleHeader"><h2 class="StepModuleHeader-title"><a class="StepModuleHeader-anchorLink" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-BuildingBlocks" data-cms-ai="0">Building Blocks</a></h2> <div class="StepModuleHeader-divider"></div></div> <div class="StepModule-body RichTextBody"> <p>This section explains general concepts behind the fundamental building blocks to use when creating a DOCA GPUNetIO application.</p> <h2 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-InitializeGPUandNIC"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-InitializeGPUandNIC" data-cms-ai="0">Initialize GPU and NIC</a> </h2> <p>When DOCA GPUNetIO is used in combination with the NIC to send or receive Ethernet traffic, the following must be performed to properly set up the application and devices:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">GPUNetIO setup</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>uint16_t dpdk_port_id; <code class="keyword">struct</code> doca_dev *ddev; <code class="keyword">struct</code> doca_gpu *gdev; <code class="color1">char</code> *eal_param[3] = {<code class="string">""</code>, <code class="string">"-a"</code>, <code class="string">"00:00.0"</code>};   <code class="comments">/* Initialize DPDK with empty device. DOCA device will hot-plug the network card later. */</code> rte_eal_init(3, eal_param); <code class="comments">/* Create DOCA device on a specific network card */</code> doca_dpdk_port_probe(&ddev); get_dpdk_port_id_doca_dev(&ddev, &dpdk_port_id); <code class="comments">/* Create GPUNetIO handler on a specific GPU */</code> doca_gpu_create(gpu_pcie_address, &gdev);</p></code> </pre> </div> </bsp-code-block> <p> </p> <p>The application would may have to enable different items depending on the task at hand.</p><p> </p> <h2 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Semaphore"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-Semaphore" data-cms-ai="0">Semaphore</a> </h2> <p>If the DOCA application must dispatch some packets' info across CUDA kernels or from the CUDA kernel and some CPU thread, a semaphore must be created.</p> <p>A semaphore is a list of items, allocated either on the GPU or CPU (depending on the use case) visible by both the GPU and CPU. This object can be used to discipline communication across items in the GPU pipeline between CUDA kernels or a CUDA kernel and a CPU thread.</p> <p>By default, each semaphore item can hold info about its status (<code class="InlineCodeBlock"><span>FREE</span></code>, <code class="InlineCodeBlock"><span>READY</span></code>, <code class="InlineCodeBlock"><span>HOLD</span></code>, <code class="InlineCodeBlock"><span>DONE</span></code>, <code class="InlineCodeBlock"><span>ERROR</span></code>), the number of received packets, and an index of a <code class="InlineCodeBlock"><span>doca_gpu_buf</span></code> in a <code class="InlineCodeBlock"><span>doca_gpu_buf_arr</span></code>.</p> <p>If the semaphore must be used to exchange data with the CPU, a preferred memory layout would be <code class="InlineCodeBlock"><span>DOCA_GPU_MEM_CPU_GPU</span></code>. Whereas, if the semaphore is only needed across CUDA kernels, <code class="InlineCodeBlock"><span>DOCA_GPU_MEM_GPU</span></code> is the best memory layout to use.</p> <p>As an optional feature, if the application must pass more application-specific info through the semaphore items, it is possible to attach a custom structure to each item of the semaphore.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">Semaphore</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="preprocessor">#define SEMAPHORE_ITEMS 1024</code>   <code class="comments">/* Application defined custom structure to pass info through semaphore items */</code> <code class="keyword">struct</code> custom_info { <code class="color1">int</code> a; uint64_t b; };   <code class="comments">/* Semaphore to share info from the GPU to the CPU */</code> <code class="keyword">struct</code> doca_gpu_semaphore *sem_to_cpu; <code class="keyword">struct</code> doca_gpu_semaphore_gpu *sem_to_cpu_gpu;   doca_gpu_semaphore_create(gdev, &sem_to_cpu); doca_gpu_semaphore_set_memory_type(sem_to_cpu, DOCA_GPU_MEM_CPU_GPU); doca_gpu_semaphore_set_items_num(sem_to_cpu, SEMAPHORE_ITEMS); <code class="comments">/* This is optional */</code> doca_gpu_semaphore_set_custom_info(sem_to_cpu, <code class="keyword">sizeof</code>(<code class="keyword">struct</code> custom_info), DOCA_GPU_MEM_CPU_GPU); doca_gpu_semaphore_start(sem_to_cpu); doca_gpu_semaphore_get_gpu_handle(sem_to_cpu, &sem_to_cpu_gpu);   <code class="comments">/* Semaphore to share info across GPU CUDA kernels with no CPU involvment */</code> <code class="keyword">struct</code> doca_gpu_semaphore *sem_to_gpu; <code class="keyword">struct</code> doca_gpu_semaphore_gpu *sem_to_gpu_gpu;   doca_gpu_semaphore_create(gdev, &sem_to_gpu); doca_gpu_semaphore_set_memory_type(sem_to_gpu, DOCA_GPU_MEM_GPU); doca_gpu_semaphore_set_items_num(sem_to_gpu, SEMAPHORE_ITEMS); <code class="comments">/* This is optional */</code> doca_gpu_semaphore_set_custom_info(sem_to_gpu, <code class="keyword">sizeof</code>(<code class="keyword">struct</code> custom_info), DOCA_GPU_MEM_GPU); doca_gpu_semaphore_start(sem_to_gpu); doca_gpu_semaphore_get_gpu_handle(sem_to_gpu, &sem_to_gpu_gpu);</p></code> </pre> </div> </bsp-code-block> <p> <br> </p> <h2 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-EthernetQueuewithGPUDataPath"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-EthernetQueuewithGPUDataPath" data-cms-ai="0">Ethernet Queue with GPU Data Path</a> </h2> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ReceiveQueue"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ReceiveQueue" data-cms-ai="0">Receive Queue</a> </h3> <p>If the DOCA application must receive Ethernet packets, receive queues must be created. The receive queue works in a circular way: At creation time, each receive queue is associated with a DOCA buffer array allocated on the GPU by the application. Each DOCA buffer of the buffer array has a maximum fixed size.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">GPUNetIO receive</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="comments">/* Start DPDK device */</code> rte_eth_dev_start(dpdk_port_id); <code class="comments">/* Initialise DOCA Flow */</code> <code class="keyword">struct</code> doca_flow_port_cfg port_cfg; port_cfg.port_id = port_id; doca_flow_init(port_cfg); doca_flow_port_start();   <code class="keyword">struct</code> doca_dev *ddev; <code class="keyword">struct</code> doca_eth_rxq *eth_rxq_cpu; <code class="keyword">struct</code> doca_gpu_eth_rxq *eth_rxq_gpu; <code class="keyword">struct</code> doca_mmap *mmap; <code class="keyword">void</code> *gpu_buffer;   <code class="comments">/* Create DOCA Ethernet receive queues */</code> doca_eth_rxq_create(ddev, MAX_PACKETS_NUM, MAX_PACKETS_SIZE, &eth_rxq_cpu);   <code class="comments">/* Set Ethernet receive queue properties */</code> <code class="comments">/* ... */</code>   <code class="comments">/* Create DOCA mmap in GPU memory to be used for the DOCA buffer array associated to this Ethernet queue */</code> doca_mmap_create(&mmap); <code class="comments">/* Set DOCA mmap properties */</code> doca_gpu_mem_alloc(gdev, buffer_size, alignment, DOCA_GPU_MEM_GPU, (<code class="keyword">void</code> **)&gpu_buffer, NULL); doca_mmap_start(mmap); doca_eth_rxq_set_pkt_buffer(eth_rxq_cpu, mmap, 0, buffer_size); <code class="comments">/* This DOCA Ethernet Rxq object will be managed by the GPU */</code> doca_ctx_set_datapath_on_gpu(); <code class="comments">/* Start the Ethernet queue object */</code> <code class="comments">/* Export GPU handle for the receive queue */</code> doca_eth_rxq_get_gpu_handle(eth_rxq_cpu, &eth_rxq_gpu);</p></code> </pre> </div> </bsp-code-block> <p> </p> <p>It is mandatory to associate DOCA Flow pipe(s) to the receive queues. Otherwise, the application cannot receive any packet.</p><p> </p> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-SendQueue"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-SendQueue" data-cms-ai="0">Send Queue</a> </h3> <p>If the DOCA application must send Ethernet packets, send queues must be created in combination with <code class="InlineCodeBlock"><span>doca_gpu_buf_arr</span></code> to prepare and send packets from GPU memory.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">GPUNetIO receive</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="keyword">struct</code> doca_dev *ddev; <code class="keyword">struct</code> doca_eth_txq *eth_txq_cpu; <code class="keyword">struct</code> doca_gpu_eth_txq *eth_txq_gpu;   <code class="comments">/* Create DOCA Ethernet send queues */</code> doca_eth_txq_create(ddev, QUEUE_DEPTH, &eth_txq_cpu); <code class="comments">/* Set properties to send queues */</code>   <code class="comments">/* This DOCA Ethernet Rxq object will be managed by the GPU */</code> doca_ctx_set_datapath_on_gpu(); <code class="comments">/* Start the Ethernet queue object */</code> <code class="comments">/* Export GPU handle for the send queue */</code> doca_eth_txq_get_gpu_handle(eth_txq_cpu, &eth_txq_gpu);   <code class="comments">/* Create DOCA mmap to define memory layout and type for the DOCA buf array */</code> <code class="keyword">struct</code> doca_mmap *mmap; doca_mmap_create(&mmap); <code class="comments">/* Set DOCA mmap properties */</code>   <code class="comments">/* Create DOCA buf arr and export it to GPU */</code> <code class="keyword">struct</code> doca_buf_arr *buf_arr; <code class="keyword">struct</code> doca_gpu_buf_arr *buf_arr_gpu; doca_buf_arr_create(mmap, &buf_arr); <code class="comments">/* Set DOCA buf array properties */</code> ... <code class="comments">/* Export GPU handle for the buf arr */</code> doca_buf_arr_get_gpu_handle(buf_arr, &buf_arr_gpu);</p></code> </pre> </div> </bsp-code-block> <p> <br> </p> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ReceiveandProcess"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ReceiveandProcess" data-cms-ai="0"><span id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ReceiveandProcess" class="confluence-anchor-link"></span>Receive and Process</a> </h3> <p>At this point, the application has created and initialized all the objects required by the GPU to exercise the data path to send or receive packets with GPUNetIO.</p> <p>In this example, the application must receive packets from different queues with a receiver CUDA kernel and dispatch packet info to a second CUDA kernel responsible for packet processing.</p> <p>The CPU launches the CUDA kernels and waits on the semaphore for output:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">CPU code</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="preprocessor">#define CUDA_THREADS 512</code> <code class="preprocessor">#define CUDA_BLOCKS 1</code> <code class="color1">int</code> semaphore_index = 0; <code class="keyword">enum</code> doca_gpu_semaphore_status status; <code class="keyword">struct</code> custom_info *gpu_info;   <code class="comments">/* On the CPU */</code> cuda_kernel_receive_dispatch<<<CUDA_THREADS, CUDA_BLOCKS, ..., stream_0>>>(eth_rxq_gpu, sem_to_gpu_gpu) cuda_kernel_process<<<CUDA_THREADS, CUDA_BLOCKS, ..., stream_1>>>(eth_rxq_gpu, sem_to_cpu_gpu, sem_to_gpu_gpu)   <code class="keyword">while</code>(<code class="comments">/* condition */</code>) { doca_gpu_semaphore_get_status(sem_to_cpu, semaphore_index, &status); <code class="keyword">if</code> (status == DOCA_GPU_SEMAPHORE_STATUS_READY) { doca_gpu_semaphore_get_custom_info_addr(sem_to_cpu, semaphore_index, (<code class="keyword">void</code> **)&(gpu_info)); report_info(gpu_info); doca_gpu_semaphore_set_status(sem_to_cpu, semaphore_index, DOCA_GPU_SEMAPHORE_STATUS_FREE); semaphore_index = (semaphore_index+1) % SEMAPHORE_ITEMS; } }</p></code> </pre> </div> </bsp-code-block> <p> </p> <p>On the GPU, the two CUDA kernels are running on different streams:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">GPU code</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>cuda_kernel_receive_dispatch(eth_rxq_gpu, sem_to_gpu_gpu) { __shared__ uint32_t rx_pkt_num; __shared__ uint64_t rx_buf_idx; <code class="color1">int</code> semaphore_index = 0;   <code class="keyword">while</code> (<code class="comments">/* exit condition */</code>) { doca_gpu_dev_eth_rxq_receive_block(eth_rxq_gpu, MAX_NUM_RECEIVE_PACKETS, TIMEOUT_RECEIVE_NS, &rx_pkt_num, &rx_buf_idx); <code class="keyword">if</code> (threadIdx.x == 0 && rx_pkt_num > 0) { doca_gpu_dev_sem_set_packet_info(sem_to_gpu_gpu, semaphore_index, DOCA_GPU_SEMAPHORE_STATUS_READY, rx_pkt_num, rx_buf_idx); semaphore_index = (semaphore_index+1) % SEMAPHORE_ITEMS; } } }   cuda_kernel_process(eth_rxq_gpu, sem_to_cpu_gpu, sem_to_gpu_gpu) { __shared__ uint32_t rx_pkt_num; __shared__ uint64_t rx_buf_idx; <code class="color1">int</code> semaphore_index = 0; <code class="color1">int</code> thread_buf_idx = 0; <code class="keyword">struct</code> doca_gpu_buf *buf_ptr; <code class="color1">uintptr_t</code> buf_addr; <code class="keyword">struct</code> custom_info *gpu_info;   <code class="keyword">while</code> (<code class="comments">/* exit condition */</code>) { <code class="keyword">if</code> (threadIdx.x == 0) { <code class="keyword">do</code> { result = doca_gpu_dev_sem_get_packet_info_status(sem_to_gpu_gpu, semaphore_index, DOCA_GPU_SEMAPHORE_STATUS_READY, &rx_pkt_num, &rx_buf_idx); } <code class="keyword">while</code>(result != DOCA_ERROR_NOT_FOUND <code class="comments">/* && other exit condition */</code>); } __syncthreads();   thread_buf_idx = threadIdx.x; <code class="keyword">while</code> (thread_buf_idx < rx_pkt_num) { <code class="comments">/* Get DOCA GPU buffer from the GPU buffer in the receive queue */</code> doca_gpu_dev_eth_rxq_get_buf(eth_rxq_gpu, rx_buf_idx + thread_buf_idx, &buf_ptr); <code class="comments">/* Get DOCA GPU buffer memory address */</code> doca_gpu_dev_buf_get_addr(buf_ptr, &buf_addr); <code class="comments">/*</code> <code class="comments"> * Atomic here is has the entire CUDA block accesses the same semaphore to CPU.</code> <code class="comments"> * Smarter implementation can be done at warp level, with multiple semaphores, etc.. to avoid this atomic</code> <code class="comments"> */</code> <code class="color1">int</code> semaphore_index_tmp = atomicAdd_block(&semaphore_index, 1); semaphore_index_tmp = semaphore_index_tmp % SEMAPHORE_ITEMS; doca_gpu_dev_sem_get_custom_info_addr(sem_to_cpu_gpu, semaphore_index_tmp, (<code class="keyword">void</code> **)&gpu_info); populate_custom_info(buf_addr, gpu_info); doca_gpu_dev_sem_set_status(sem_to_cpu_gpu, semaphore_index_tmp, DOCA_GPU_SEMAPHORE_STATUS_READY); } __syncthreads();   <code class="keyword">if</code> (threadIdx.x == 0) { doca_gpu_dev_sem_set_status(sem_to_gpu_gpu, semaphore_index, DOCA_GPU_SEMAPHORE_STATUS_READY); } } }</p></code> </pre> </div> </bsp-code-block> <p> </p> <p>This code can be represented with the following diagram when multiple queues and/or semaphores are used:</p> <p></p><p></p><div class="Enhancement" data-align-inline has-width style="--image-width: 1100px"> <div class="Enhancement-item"> <figure class="Figure"> <picture> <source type="image/webp" width="1440" height="493" srcset="https://docscontent.nvidia.com/dims4/default/8ef9976/2147483647/strip/true/crop/2728x934+0+0/resize/1440x493!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-3_18-18-20-version-1-modificationdate-1725534954347-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/69bd978/2147483647/strip/true/crop/2728x934+0+0/resize/2880x986!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-3_18-18-20-version-1-modificationdate-1725534954347-api-v2.png 2x"> <source type="image/png" width="1440" height="493" srcset="https://docscontent.nvidia.com/dims4/default/b1c7f5c/2147483647/strip/true/crop/2728x934+0+0/resize/1440x493!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-3_18-18-20-version-1-modificationdate-1725534954347-api-v2.png"> <img class="Image" alt="image2023-4-3_18-18-20-version-1-modificationdate-1725534954347-api-v2.png" srcset="https://docscontent.nvidia.com/dims4/default/b1c7f5c/2147483647/strip/true/crop/2728x934+0+0/resize/1440x493!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-3_18-18-20-version-1-modificationdate-1725534954347-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/cb19b22/2147483647/strip/true/crop/2728x934+0+0/resize/2880x986!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-3_18-18-20-version-1-modificationdate-1725534954347-api-v2.png 2x" width="1440" height="493" src="https://docscontent.nvidia.com/dims4/default/b1c7f5c/2147483647/strip/true/crop/2728x934+0+0/resize/1440x493!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-3_18-18-20-version-1-modificationdate-1725534954347-api-v2.png" loading="lazy"> </picture> </figure> </div> </div><p> </p><p>Please note that receiving and dispatching packets to another CUDA kernel is not required. A simpler scenario can have a single CUDA kernel receiving and processing packets:</p> <p></p><p></p><div class="Enhancement" data-align-inline has-width style="--image-width: 794px"> <div class="Enhancement-item"> <figure class="Figure"> <picture> <source type="image/webp" width="1440" height="615" srcset="https://docscontent.nvidia.com/dims4/default/75e4357/2147483647/strip/true/crop/2005x856+0+0/resize/1440x615!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-4_12-13-32-version-1-modificationdate-1725534954533-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/5871dcb/2147483647/strip/true/crop/2005x856+0+0/resize/2880x1230!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-4_12-13-32-version-1-modificationdate-1725534954533-api-v2.png 2x"> <source type="image/png" width="1440" height="615" srcset="https://docscontent.nvidia.com/dims4/default/6643abb/2147483647/strip/true/crop/2005x856+0+0/resize/1440x615!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-4_12-13-32-version-1-modificationdate-1725534954533-api-v2.png"> <img class="Image" alt="image2023-4-4_12-13-32-version-1-modificationdate-1725534954533-api-v2.png" srcset="https://docscontent.nvidia.com/dims4/default/6643abb/2147483647/strip/true/crop/2005x856+0+0/resize/1440x615!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-4_12-13-32-version-1-modificationdate-1725534954533-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/4ee0fa3/2147483647/strip/true/crop/2005x856+0+0/resize/2880x1230!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-4_12-13-32-version-1-modificationdate-1725534954533-api-v2.png 2x" width="1440" height="615" src="https://docscontent.nvidia.com/dims4/default/6643abb/2147483647/strip/true/crop/2005x856+0+0/resize/1440x615!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage2023-4-4_12-13-32-version-1-modificationdate-1725534954533-api-v2.png" loading="lazy"> </picture> </figure> </div> </div><p> </p><p>The drawback of this approach is that the time between two receives depends on the time taken by the CUDA kernel to process received packets.</p> <p>The type of pipeline that must be built heavily depends on the specific use case.</p><p> </p> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ProduceandSend"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ProduceandSend" data-cms-ai="0"><span id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-ProduceandSend" class="confluence-anchor-link"></span>Produce and Send</a> </h3> <p>In this example, the GPU produces some data, stores it into packets and then sends them over the network. The CPU launches the CUDA kernels and continues doing other work:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">CPU code</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="preprocessor">#define CUDA_THREADS 512</code> <code class="preprocessor">#define CUDA_BLOCKS 1</code> <code class="color1">int</code> semaphore_index = 0; <code class="keyword">enum</code> doca_gpu_semaphore_status status; <code class="keyword">struct</code> custom_info *gpu_info;   <code class="comments">/* On the CPU */</code> cuda_kernel_produce_send<<<CUDA_THREADS, CUDA_BLOCKS, ..., stream_0>>>(eth_txq_gpu, buf_arr_gpu)   <code class="comments">/* do other stuff */</code></p></code> </pre> </div> </bsp-code-block> <p> </p> <p>On the GPU, the CUDA kernel fills the packets with meaningful data and sends them. In the following example, the scope is CUDA block so each block uses a different DOCA Ethernet send queue:</p> <p> </p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">GPU code</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>cuda_kernel_produce_send(eth_txq_gpu, buf_arr_gpu) { uint64_t doca_gpu_buf_idx = threadIdx.x; <code class="keyword">struct</code> doca_gpu_buf *buf; <code class="color1">uintptr_t</code> buf_addr; uint32_t packet_len; uint32_t curr_position; uint32_t mask_max_position; uint32_t num_pkts_per_send = blockDim.x;   <code class="comments">/* Get last occupied position in the Tx queue */</code> doca_gpu_dev_eth_txq_get_info(eth_txq_gpu, &curr_position, &mask_max_position); __syncthreads();   <code class="keyword">while</code> (<code class="comments">/* exit condition */</code>) {   <code class="comments">/* Each CUDA thread retrieves doca_gpu_buf from doca_gpu_buf_arr */</code> doca_gpu_dev_buf_get_buf(buf_arr_gpu, doca_gpu_buf_idx, &buf); <code class="comments">/* Get memory address of the packet in the doca_gpu_buf */</code> doca_gpu_dev_buf_get_addr(buf, &buf_addr);   <code class="comments">/* Application produces data and crafts the packet in the doca_gpu_buf */</code> populate_packet(buf_addr, &packet_len);   <code class="comments">/* Enqueue packet in the send queue with weak mode: each thread posts the packet in a different and sequential position of the queue */</code> doca_gpu_dev_eth_txq_send_enqueue_weak(eth_txq_gpu, buf, packet_len, ((curr_position + doca_gpu_buf_idx) & mask_max_position), DOCA_GPU_SEND_FLAG_NONE);   <code class="comments">/* Synchronization point */</code> __synchthreads();   <code class="comments">/* Only one CUDA thread in the block must commit and push the send queue */</code> <code class="keyword">if</code> (threadIdx.x == 0) { doca_gpu_dev_eth_txq_commit_weak(eth_txq_gpu, num_pkts_per_send); doca_gpu_dev_eth_txq_push(eth_txq_gpu); } <code class="comments">/* Synchronization point */</code> __synchthreads(); <code class="comments">/* Assume all threads in the block pushed a packet in the send queue */</code> doca_gpu_buf_idx += blockDim.x; } }</p></code> </pre> </div> </bsp-code-block> <p> </p> <h2 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-RDMAQueuewithGPUDataPath"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-RDMAQueuewithGPUDataPath" data-cms-ai="0">RDMA Queue with GPU Data Path</a> </h2> <p>To execute RDMA operations from a GPU CUDA kernel, in the setup phase, the application must first create a DOCA RDMA queue, export the RDMA as context, and then set the datapath of the context on the GPU (as shown in the following code snippet).</p> <p>The following is a pseudo-code to serve as a guide. Please refer to real function signatures in header files (<code class="InlineCodeBlock"><span>*.h</span></code>) and documentation for a complete overview of the functions.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">GPU code</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="keyword">struct</code> doca_dev *doca_device; <code class="comments">/* DOCA device */</code> <code class="keyword">struct</code> doca_gpu *gpudev; <code class="comments">/* DOCA GPU device */</code> <code class="keyword">struct</code> doca_rdma *rdma; <code class="comments">/* DOCA RDMA instance */</code> <code class="keyword">struct</code> doca_gpu_dev_rdma *gpu_rdma; <code class="comments">/* DOCA RDMA instance GPU handler */</code> <code class="keyword">struct</code> doca_ctx *rdma_ctx;   <code class="comments">// Initialize IBDev RDMA device</code> open_doca_device_with_ibdev_name(&doca_device) <code class="comments">// Initialize DPDK (hugepages not needed)</code> <code class="color1">char</code> *eal_param[4] = {<code class="string">""</code>, <code class="string">"-a"</code>, <code class="string">"00:00.0"</code>, <code class="string">"--in-memory"</code>}; rte_eal_init(4, eal_param);   <code class="comments">// Initialize the GPU device</code> doca_gpu_create(&gpudev); <code class="comments">// Create the RDMA queue object with the DOCA device</code> doca_rdma_create(doca_device, &(rdma)); <code class="comments">// Export the RDMA queue object context</code> rdma_ctx = doca_rdma_as_ctx(rdma)   <code class="comments">// Set RDMA queue attributes</code>   <code class="comments">// Set GPU data path for the RDMA object</code> doca_ctx_set_datapath_on_gpu(ctx, gpudev) doca_ctx_start(rdma_ctx);</p></code> </pre> </div> </bsp-code-block> <p> </p> <p>At this point, the application has an RDMA queue usable from a GPU CUDA kernel. The next step would be to establish a connection using some OOB (out-of-band) mechanism (e.g., Linux sockets) to exchange RDMA queue info so each peer can connect to the other's queues.</p> <p>To exchange data, users must create DOCA GPU buffer arrays to send or receive data. If the application also requires read or write, then the GPU memory associated with the buffer arrays must be exported and exchanged with the remote peers using the OOB mechanism.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">GPU code</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="comments">/* Create DOCA mmap to define memory layout and type for the DOCA buf array */</code> <code class="keyword">struct</code> doca_mmap *mmap; doca_mmap_create(&mmap); <code class="comments">/* Set DOCA mmap properties */</code> doca_mmap_start(mmap); <code class="comments">/* Export mmap info to share with remote peer */</code> doca_mmap_export_rdma(mmap, ...);   <code class="comments">/* Exchange export info with remote peer */</code>   <code class="comments">/* Create DOCA buf arr and export it to GPU */</code> <code class="keyword">struct</code> doca_buf_arr *buf_arr; <code class="keyword">struct</code> doca_gpu_buf_arr *buf_arr_gpu; doca_buf_arr_create(mmap, &buf_arr); <code class="comments">/* Set DOCA buf array properties */</code> ... <code class="comments">/* Export GPU handle for the buf arr */</code> doca_buf_arr_get_gpu_handle(buf_arr, &buf_arr_gpu);</p></code> </pre> </div> </bsp-code-block> <p> </p> <p>Please refer to the "<a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-RDMAClientServer" data-cms-ai="0" >RDMA Client Server</a>" sample as a basic layout to implement all the steps described in this section.</p> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-CUDAKernelforRDMAWrite"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-CUDAKernelforRDMAWrite" data-cms-ai="0"><span id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-CUDAKernelforRDMAWrite" class="confluence-anchor-link"></span>CUDA Kernel for RDMA Write</a> </h3> <p>Assuming the RDMA queues and buffer arrays are correctly created and exchanged across peers, the application can launch a CUDA kernel to remotely write data. As typically applications use <code class="InlineCodeBlock"><span>strong</span></code> mode, the following code snippet shows how to use <code class="InlineCodeBlock"><span>weak</span></code> mode to post multiple writes from different CUDA threads in the same CUDA block.</p> <p> </p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">GPU code</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>__global__ <code class="keyword">void</code> rdma_write_bw(<code class="keyword">struct</code> doca_gpu_dev_rdma *rdma_gpu, <code class="keyword">struct</code> doca_gpu_buf_arr *local_buf_arr, <code class="keyword">struct</code> doca_gpu_buf_arr *remote_buf_arr) { <code class="keyword">struct</code> doca_gpu_buf *remote_buf; <code class="keyword">struct</code> doca_gpu_buf *local_buf; <code class="keyword">struct</code> doca_gpu_dev_rdma *rdma_gpu; <code class="keyword">struct</code> doca_gpu_buf_arr *server_local_buf_arr; <code class="keyword">struct</code> doca_gpu_buf_arr *server_remote_buf_arr; uint32_t curr_position; uint32_t mask_max_position; uint32_t num_ops;   doca_gpu_dev_buf_get_buf(server_local_buf_arr, threadIdx.x, &local_buf); doca_gpu_dev_buf_get_buf(server_remote_buf_arr, threadIdx.x, &remote_buf); <code class="comments">/* Get RDMA queue current available position and mask of the max position number */</code> doca_gpu_dev_rdma_get_info(rdma_gpu, &curr_position, &mask_max_position);   doca_gpu_dev_rdma_write_weak(rdma_gpu, <code class="comments">/* Write into this remote buffer at offset 0 */</code> remote_buf, 0, <code class="comments">/* Fetch data from this local buffer at offset 0 */</code> local_buf, 0, <code class="comments">/* Number of bytes to write */</code> msg_size, <code class="comments">/* Don't use immediate */</code> 0, DOCA_GPU_RDMA_WRITE_FLAG_NONE, <code class="comments">/* Position in the RDMA queue to post the write */</code> (curr_position + threadIdx.x) & mask_max_position);   <code class="comments">/* Wait all CUDA threads to post their RDMA Write */</code> __syncthreads();   <code class="keyword">if</code> (threadIdx.x == 0) { <code class="comments">/* Only 1 CUDA thread can push the write op just posted */</code> doca_gpu_dev_rdma_commit_weak(rdma_gpu, blockDim.x); doca_gpu_dev_rdma_wait_all(rdma_gpu, &num_ops); } __syncthreads(); }</p></code> </pre> </div> </bsp-code-block> <p> </p> <div class="CallOutBox" data-option-Info> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-info"></use></svg> </div><div class="CallOutBox-title">Info</div> <div class="CallOutBox-body"> <p>The code in the "<a class="Link" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-RDMAClientServer" data-cms-ai="0" >RDMA Client Server</a>" sample shows how to use write and send with immediate flag set.</p> </div> </div> <p> </p> </div> </div> <div class="StepModule" data-module><span class="StepModule-anchor" id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-GPUNetIOSamplesGPUNetIOSamples"></span> <div class="StepModuleHeader"><h2 class="StepModuleHeader-title"><a class="StepModuleHeader-anchorLink" href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-GPUNetIOSamplesGPUNetIOSamples" data-cms-ai="0">GPUNetIO Samples</a></h2> <div class="StepModuleHeader-divider"></div></div> <div class="StepModule-body RichTextBody"> <p> This section contains two samples that show how to enable simple GPUNetIO features. Be sure to correctly set the following environment variables:</p><p></p> <p></p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">Build the sample</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="functions">export</code> PATH=${PATH}:/usr/local/cuda/bin <code class="functions">export</code> CPATH=<code class="string">"$(echo /usr/local/cuda/targets/{x86_64,sbsa}-linux/include | sed 's/ /:/'):${CPATH}"</code> <code class="functions">export</code> PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:/usr/lib/pkgconfig:/opt/mellanox/grpc/lib/{x86_64,aarch64}-linux-gnu/pkgconfig:/opt/mellanox/dpdk/lib/{x86_64,aarch64}-linux-gnu/pkgconfig:/opt/mellanox/doca/lib/{x86_64,aarch64}-linux-gnu/pkgconfigexport LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/cuda/lib64:/opt/mellanox/gdrcopy/src:/opt/mellanox/dpdk/lib/{x86_64,aarch64}-linux-gnu:/opt/mellanox/doca/lib/{x86_64,aarch64}-linux-gnu</p></code> </pre> </div> </bsp-code-block> <p> </p> <div class="CallOutBox" data-option-Info> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-info"></use></svg> </div><div class="CallOutBox-title">Info</div> <div class="CallOutBox-body"> <p> All the DOCA samples described in this section are governed under the BSD-3 software license agreement. </p> </div> </div> <p> </p> <div class="CallOutBox" data-option-Note> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-check"></use></svg> </div><div class="CallOutBox-title">Note</div> <div class="CallOutBox-body"> <p> Please ensure the arch of your GPU is included in the <code class="InlineCodeBlock"><span>meson.build</span></code> file before building the samples (e.g., <code class="InlineCodeBlock"><span>sm_80</span></code> for Ampere, <code class="InlineCodeBlock"><span>sm_89</span></code> for L40, <code class="InlineCodeBlock"><span>sm_90</span></code> for H100, etc). </p> </div> </div> <p> </p> <h2 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-EthernetSendWaitTime"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-EthernetSendWaitTime" data-cms-ai="0">Ethernet Send Wait Time</a> </h2> <p> The sample shows how to enable Accurate Send Scheduling (or wait-on-time) in the context of a GPUNetIO application. Accurate Send Scheduling is the ability of an NVIDIA NIC to send packets in the future according to application-provided timestamps. </p> <div class="CallOutBox" data-option-Note> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-check"></use></svg> </div><div class="CallOutBox-title">Note</div> <div class="CallOutBox-body"> <p> This feature is supported on ConnectX-6 Dx and later .</p> </div> </div> <p> </p> <div class="CallOutBox" data-option-Info> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-info"></use></svg> </div><div class="CallOutBox-title">Info</div> <div class="CallOutBox-body"> <p> This <a class="Link" href="https://developer.nvidia.com/blog/inline-gpu-packet-processing-with-nvidia-doca-gpunetio/#:~:text=time%20(Figure%2017).-,Figure%2017,-.%20NVIDIA%20Aerial%205G" class="external-link" data-cms-ai="0" >NVIDIA blog post</a> offers an example for how this feature has been used in 5G networks. </p> </div> </div> <p> </p> <p> This DOCA GPUNetIO sample provides a simple application to send packets with Accurate Send Scheduling from the GPU . </p> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-SynchronizingClocks"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-SynchronizingClocks" data-cms-ai="0">Synchronizing Clocks</a> </h3> <p> Before starting the sample, it is important to properly synchronize the CPU clock with the NIC clock. This way, timestamps provided by the system clock are synchronized with the time in the NIC. </p> <p> For this purpose, at least the <code class="InlineCodeBlock"><span>phc2sys</span></code> service must be used. To install it on an Ubuntu system: </p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">phc2sys</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="functions">sudo</code> apt <code class="functions">install</code> linuxptp</p></code> </pre> </div> </bsp-code-block> <p> </p> <p> To start the <code class="InlineCodeBlock"><span>phc2sys</span></code> service properly, a config file must be created in <code class="InlineCodeBlock"><span>/lib/systemd/system/phc2sys.service</span></code>:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">phc2sys</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>[Unit] Description=Synchronize system clock or PTP hardware clock (PHC) Documentation=<code class="functions">man</code>:phc2sys   [Service] Restart=always RestartSec=5s Type=simple ExecStart=/bin/sh -c <code class="string">"taskset -c 15 /usr/sbin/phc2sys -s /dev/ptp$(ethtool -T ens6f0 | grep PTP | awk '{print $4}') -c CLOCK_REALTIME -n 24 -O 0 -R 256 -u 256"</code>   [Install] WantedBy=multi-user.target</p></code> </pre> </div> </bsp-code-block> <p> </p> <p> Now <code class="InlineCodeBlock"><span>phc2sys</span></code> service can be started: </p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">phc2sys</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="functions">sudo</code> systemctl stop systemd-timesyncd <code class="functions">sudo</code> systemctl disable systemd-timesyncd <code class="functions">sudo</code> systemctl daemon-reload <code class="functions">sudo</code> systemctl start phc2sys.service</p></code> </pre> </div> </bsp-code-block> <p> </p> <p> To check the status of <code class="InlineCodeBlock"><span>phc2sys</span></code>: </p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">phc2sys</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>$ <code class="functions">sudo</code> systemctl status phc2sys.service   ● phc2sys.service - Synchronize system clock or PTP hardware clock (PHC) Loaded: loaded (/lib/systemd/system/phc2sys.service; disabled; vendor preset: enabled) Active: active (running) since Mon 2023-04-03 10:59:13 UTC; 2 days ago Docs: <code class="functions">man</code>:phc2sys Main PID: 337824 (sh) Tasks: 2 (limit: 303788) Memory: 560.0K CPU: 52min 8.199s CGroup: /system.slice/phc2sys.service ├─337824 /bin/sh -c "taskset -c 15 /usr/sbin/phc2sys -s /dev/ptp\$(<code class="functions">ethtool</code> -T enp23s0f1np1 | <code class="functions">grep</code> PTP | <code class="functions">awk</code> <code class="string">'{print \$4}'</code>) -c CLOCK_REALTIME -n 24 -O 0 -R > └─337829 /usr/sbin/phc2sys -s /dev/ptp3 -c CLOCK_REALTIME -n 24 -O 0 -R 256 -u 256   Apr 05 16:35:52 doca-vr-045 phc2sys[337829]: [457395.040] CLOCK_REALTIME rms 8 max 18 freq +110532 +/- 27 delay 770 +/- 3 Apr 05 16:35:53 doca-vr-045 phc2sys[337829]: [457396.071] CLOCK_REALTIME rms 8 max 20 freq +110513 +/- 30 delay 769 +/- 3 Apr 05 16:35:54 doca-vr-045 phc2sys[337829]: [457397.102] CLOCK_REALTIME rms 8 max 18 freq +110527 +/- 30 delay 769 +/- 3 Apr 05 16:35:55 doca-vr-045 phc2sys[337829]: [457398.130] CLOCK_REALTIME rms 8 max 18 freq +110517 +/- 31 delay 769 +/- 3 Apr 05 16:35:56 doca-vr-045 phc2sys[337829]: [457399.159] CLOCK_REALTIME rms 8 max 19 freq +110523 +/- 32 delay 770 +/- 3 Apr 05 16:35:57 doca-vr-045 phc2sys[337829]: [457400.191] CLOCK_REALTIME rms 8 max 20 freq +110528 +/- 33 delay 770 +/- 3 Apr 05 16:35:58 doca-vr-045 phc2sys[337829]: [457401.221] CLOCK_REALTIME rms 8 max 19 freq +110512 +/- 38 delay 770 +/- 3 Apr 05 16:35:59 doca-vr-045 phc2sys[337829]: [457402.253] CLOCK_REALTIME rms 9 max 20 freq +110538 +/- 47 delay 770 +/- 4 Apr 05 16:36:00 doca-vr-045 phc2sys[337829]: [457403.281] CLOCK_REALTIME rms 8 max 21 freq +110517 +/- 38 delay 769 +/- 3 Apr 05 16:36:01 doca-vr-045 phc2sys[337829]: [457404.311] CLOCK_REALTIME rms 8 max 17 freq +110526 +/- 26 delay 769 +/- 3 ...</p></code> </pre> </div> </bsp-code-block> <p> </p> <p> At this point, the system and NIC clocks are synchronized so timestamps provided by the CPU are correctly interpreted by the NIC. </p> <div class="CallOutBox" data-option-Warning> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-info"></use></svg> </div><div class="CallOutBox-title">Warning</div> <div class="CallOutBox-body"> <p> The timestamps you get may not reflect the real time and day. To get that, you must properly set the <code class="InlineCodeBlock"><span>ptp4l</span></code> service with an external grand master on the system. Doing that is out of the scope of this sample. </p> </div> </div> <p> <br> </p> <h3 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-RunningtheSample"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-RunningtheSample" data-cms-ai="0">Running the Sample</a> </h3> <p>The sample is shipped with the source files that must be built:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">phc2sys</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="comments"># Ensure DOCA and DPDK are in the pkgconfig environment variable</code> <code class="functions">cd</code> /opt/mellanox/doca/samples/doca_gpunetio/gpunetio_send_wait_time meson build ninja -C build</p></code> </pre> </div> </bsp-code-block> <p> </p> <p> The sample sends 8 bursts of 32 raw Ethernet packets or 1kB to a dummy Ethernet address, <code class="InlineCodeBlock"><span>10:11:12:13:14:15</span></code>, in a timed way. Program the NIC to send every <code class="InlineCodeBlock"><span>t</span></code> nanoseconds (command line option <code class="InlineCodeBlock"><span>-t</span></code>). </p> <p> The following example programs a system with GPU PCIe address <code class="InlineCodeBlock"><span>ca:00.0</span></code> and NIC PCIe address <code class="InlineCodeBlock"><span>17:00.0</span></code> to send 32 packets every 5 milliseconds:</p><p></p> <p></p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">Run</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="comments"># Ensure DOCA and DPDK are in the LD_LIBRARY_PATH environment variable</code> $ <code class="functions">sudo</code> ./build/doca_gpunetio_send_wait_time -n 17:00.0 -g ca:00.0 -t 5000000[09:22:54:165778][1316878][DOCA][INF][gpunetio_send_wait_time_main.c:195][main] Starting the sample [09:22:54:438260][1316878][DOCA][INF][gpunetio_send_wait_time_main.c:224][main] Sample configuration: GPU ca:00.0 NIC 17:00.0 Timeout 5000000ns EAL: Detected CPU lcores: 128 ... EAL: Probe PCI driver: mlx5_pci (15b3:a2d6) device: 0000:17:00.0 (socket 0) [09:22:54:819996][1316878][DOCA][INF][gpunetio_send_wait_time_sample.c:607][gpunetio_send_wait_time] Wait on <code class="functions">time</code> supported mode: DPDK EAL: Probe PCI driver: gpu_cuda (10de:20b5) device: 0000:ca:00.0 (socket 1) [09:22:54:830212][1316878][DOCA][INF][gpunetio_send_wait_time_sample.c:252][create_tx_buf] Mapping send queue buffer (0x0x7f48e32a0000 size 262144B) with legacy nvidia-peermem mode [09:22:54:832462][1316878][DOCA][INF][gpunetio_send_wait_time_sample.c:657][gpunetio_send_wait_time] Launching CUDA kernel to send packets [09:22:54:842945][1316878][DOCA][INF][gpunetio_send_wait_time_sample.c:664][gpunetio_send_wait_time] Waiting 10 sec <code class="keyword">for</code> 256 packets to be sent [09:23:04:883309][1316878][DOCA][INF][gpunetio_send_wait_time_sample.c:684][gpunetio_send_wait_time] Sample finished successfully [09:23:04:883339][1316878][DOCA][INF][gpunetio_send_wait_time_main.c:239][main] Sample finished successfully</p></code> </pre> </div> </bsp-code-block> <p> </p> <p> To verify that packets are actually sent at the right time, use a packet sniffer on the other side (e.g., <code class="InlineCodeBlock"><span>tcpdump</span></code>):</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">phc2sys</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>$ <code class="functions">sudo</code> tcpdump -i enp23s0f1np1 -A -s 64   17:12:23.480318 IP5 (invalid) Sent from DOCA GPUNetIO........................... .... 17:12:23.480368 IP5 (invalid) Sent from DOCA GPUNetIO........................... <code class="comments"># end of first burst of 32 packets, bump to +5ms</code> 17:12:23.485321 IP5 (invalid) Sent from DOCA GPUNetIO........................... ... 17:12:23.485369 IP5 (invalid) Sent from DOCA GPUNetIO........................... <code class="comments"># end of second burst of 32 packets, bump to +5ms</code> 17:12:23.490278 IP5 (invalid) Sent from DOCA GPUNetIO........................... ...</p></code> </pre> </div> </bsp-code-block> <p> </p> <p class="auto-cursor-target">The output should show a jump of approximately 5 milliseconds every 32 packets.</p> <p> </p> <div class="CallOutBox" data-option-Note> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-check"></use></svg> </div><div class="CallOutBox-title">Note</div> <div class="CallOutBox-body"> <p><code class="InlineCodeBlock"><span>tcpdump</span></code> may increase latency in sniffing packets and reporting the receive timestamp, so the difference between bursts of 32 packets reported may be less than expected, especially with small interval times like 500 microseconds (<code class="InlineCodeBlock"><span>-t 500000</span></code>).</p> </div> </div> <p> </p> <h2 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-EthernetSimpleReceive"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-EthernetSimpleReceive" data-cms-ai="0">Ethernet Simple Receive</a> </h2> <p class="auto-cursor-target">This simple application shows the fundamental steps to build a DOCA GPUNetIO receiver application with one queue for UDP packets and one CUDA kernel receiving those packets from the GPU, printing packet info to the console.</p> <div class="CallOutBox" data-option-Warning> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-info"></use></svg> </div><div class="CallOutBox-title">Warning</div> <div class="CallOutBox-body"> <p>Invoking a <code class="InlineCodeBlock"><span>printf</span></code> from a CUDA kernel is not good practice for release software and should be used only to print debug information as it slows down the overall execution of the CUDA kernel.</p> </div> </div> <p> </p> <p class="auto-cursor-target">To build and run the application:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">Build the sample</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="comments"># Ensure DOCA and DPDK are in the pkgconfig environment variable</code> <code class="functions">cd</code> /opt/mellanox/doca/samples/doca_gpunetio/gpunetio_simple_receive meson build ninja -C build</p></code> </pre> </div> </bsp-code-block> <p> </p> <p class="auto-cursor-target">To test the application, this guide assumes the usual setup with two machines: one with the DOCA receiver application and the second one acting as packet generator. As UDP packet generator, this example considers the <code class="InlineCodeBlock"><span>nping</span></code> application that can be easily installed easily on any Linux machine.</p> <p class="auto-cursor-target">The command to send 10 UDP packets via <code class="InlineCodeBlock"><span>nping</span></code> on the packet generator machine is:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">nping generator</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>$ nping --udp -c 10 -p 2090 192.168.1.1 --data-length 1024 --delay 500ms   Starting Nping 0.7.80 ( https://nmap.org/nping ) at 2023-11-20 11:05 UTC SENT (0.0018s) UDP packet with 1024 bytes to 192.168.1.1:2090 SENT (0.5018s) UDP packet with 1024 bytes to 192.168.1.1:2090 SENT (1.0025s) UDP packet with 1024 bytes to 192.168.1.1:2090 SENT (1.5025s) UDP packet with 1024 bytes to 192.168.1.1:2090 SENT (2.0032s) UDP packet with 1024 bytes to 192.168.1.1:2090 SENT (2.5033s) UDP packet with 1024 bytes to 192.168.1.1:2090 SENT (3.0040s) UDP packet with 1024 bytes to 192.168.1.1:2090 SENT (3.5040s) UDP packet with 1024 bytes to 192.168.1.1:2090 SENT (4.0047s) UDP packet with 1024 bytes to 192.168.1.1:2090 SENT (4.5048s) UDP packet with 1024 bytes to 192.168.1.1:2090   Max rtt: N/A | Min rtt: N/A | Avg rtt: N/A UDP packets sent: 10 | Rcvd: 0 | Lost: 10 (100.00%) Nping <code class="keyword">done</code>: 1 IP address pinged <code class="keyword">in</code> 5.50 seconds</p></code> </pre> </div> </bsp-code-block> <p> </p> <p class="auto-cursor-target">Assuming the DOCA Simple Receive sample is waiting on the other machine at IP address <code class="InlineCodeBlock"><span>192.168.1.1</span></code>.</p> <p class="auto-cursor-target">The DOCA Simple Receive sample is launched on a system with NIC at <code class="InlineCodeBlock"><span>17:00.1</span></code> PCIe address and GPU at <code class="InlineCodeBlock"><span>ca:00.0</span></code> PCIe address:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">DOCA Simple Receive</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="comments"># Ensure DOCA and DPDK are in the LD_LIBRARY_PATH environment variable</code> $ <code class="functions">sudo</code> ./build/doca_gpunetio_simple_receive -n 17:00.1 -g ca:00.0 [11:00:30:397080][2328673][DOCA][INF][gpunetio_simple_receive_main.c:159][main] Starting the sample [11:00:30:652622][2328673][DOCA][INF][gpunetio_simple_receive_main.c:189][main] Sample configuration: GPU ca:00.0 NIC 17:00.1   EAL: Detected CPU lcores: 128 EAL: Detected NUMA nodes: 2 EAL: Detected shared linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode <code class="string">'PA'</code> EAL: VFIO support initialized TELEMETRY: No legacy callbacks, legacy socket not created EAL: Probe PCI driver: mlx5_pci (15b3:a2d6) device: 0000:17:00.1 (socket 0) [11:00:31:036760][2328673][DOCA][WRN][engine_model.c:72][adapt_queue_depth] adapting queue depth to 128. [11:00:31:928926][2328673][DOCA][WRN][engine_port.c:321][port_driver_process_properties] detected representor used <code class="keyword">in</code> VNF mode (driver port <code class="functions">id</code> 0) EAL: Probe PCI driver: gpu_cuda (10de:20b5) device: 0000:ca:00.0 (socket 1) [11:00:31:977261][2328673][DOCA][INF][gpunetio_simple_receive_sample.c:425][create_rxq] Creating Sample Eth Rxq   [11:00:31:977841][2328673][DOCA][INF][gpunetio_simple_receive_sample.c:466][create_rxq] Mapping receive queue buffer (0x0x7f86cc000000 size 33554432B) with nvidia-peermem mode [11:00:32:043182][2328673][DOCA][INF][gpunetio_simple_receive_sample.c:610][gpunetio_simple_receive] Launching CUDA kernel to receive packets [11:00:32:055193][2328673][DOCA][INF][gpunetio_simple_receive_sample.c:614][gpunetio_simple_receive] Waiting <code class="keyword">for</code> termination Thread 0 received UDP packet with Eth src 10:70:fd:fa:77:f5 - Eth dst 10:70:fd:fa:77:e9 Thread 0 received UDP packet with Eth src 10:70:fd:fa:77:f5 - Eth dst 10:70:fd:fa:77:e9 Thread 0 received UDP packet with Eth src 10:70:fd:fa:77:f5 - Eth dst 10:70:fd:fa:77:e9 Thread 0 received UDP packet with Eth src 10:70:fd:fa:77:f5 - Eth dst 10:70:fd:fa:77:e9 Thread 0 received UDP packet with Eth src 10:70:fd:fa:77:f5 - Eth dst 10:70:fd:fa:77:e9 Thread 0 received UDP packet with Eth src 10:70:fd:fa:77:f5 - Eth dst 10:70:fd:fa:77:e9 Thread 0 received UDP packet with Eth src 10:70:fd:fa:77:f5 - Eth dst 10:70:fd:fa:77:e9 Thread 0 received UDP packet with Eth src 10:70:fd:fa:77:f5 - Eth dst 10:70:fd:fa:77:e9 Thread 0 received UDP packet with Eth src 10:70:fd:fa:77:f5 - Eth dst 10:70:fd:fa:77:e9 Thread 0 received UDP packet with Eth src 10:70:fd:fa:77:f5 - Eth dst 10:70:fd:fa:77:e9   <code class="comments"># Type Ctrl+C to kill the sample</code>   [11:01:44:265141][2328673][DOCA][INF][gpunetio_simple_receive_sample.c:45][signal_handler] Signal 2 received, preparing to <code class="functions">exit</code>! [11:01:44:265189][2328673][DOCA][INF][gpunetio_simple_receive_sample.c:620][gpunetio_simple_receive] Exiting from sample [11:01:44:265533][2328673][DOCA][INF][gpunetio_simple_receive_sample.c:362][destroy_rxq] Destroying Rxq [11:01:44:307829][2328673][DOCA][INF][gpunetio_simple_receive_sample.c:631][gpunetio_simple_receive] Sample finished successfully [11:01:44:307861][2328673][DOCA][INF][gpunetio_simple_receive_main.c:204][main] Sample finished successfully</p></code> </pre> </div> </bsp-code-block> <p> <br> </p> <h2 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-RDMAClientServer"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-RDMAClientServer" data-cms-ai="0"><span id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-RDMAClientServer" class="confluence-anchor-link"></span>RDMA Client Server</a> </h2> <p>This sample exhibits how to use the GPUNetIO RDMA API to receive and send/write with immediate using a single RDMA queue.</p> <p>The server has a GPU buffer array A composed by <code class="InlineCodeBlock"><span>GPU_BUF_NUM doca_gpu_buf</span></code> elements, each 1kB in size. The client has two GPU buffer arrays, B and C, each composed by <code class="InlineCodeBlock"><span>GPU_BUF_NUM doca_gpu_buf</span></code> elements, each 512B in size.</p> <p>The goal is for the client to fill a single server buffer of 1kB with two GPU buffers of 512B as illustrated in the following figure:</p> <p></p><p></p><div class="Enhancement" data-align-inline has-height style="--image-height: 250px"> <div class="Enhancement-item"> <figure class="Figure"> <picture> <source type="image/webp" width="1440" height="348" srcset="https://docscontent.nvidia.com/dims4/default/b8354b5/2147483647/strip/true/crop/4882x1180+0+0/resize/1440x348!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-4-17_12-29-48-version-1-modificationdate-1725534957893-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/2059522/2147483647/strip/true/crop/4882x1180+0+0/resize/2880x696!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-4-17_12-29-48-version-1-modificationdate-1725534957893-api-v2.png 2x"> <source type="image/png" width="1440" height="348" srcset="https://docscontent.nvidia.com/dims4/default/8c4bbe7/2147483647/strip/true/crop/4882x1180+0+0/resize/1440x348!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-4-17_12-29-48-version-1-modificationdate-1725534957893-api-v2.png"> <img class="Image" alt="image-2024-4-17_12-29-48-version-1-modificationdate-1725534957893-api-v2.png" srcset="https://docscontent.nvidia.com/dims4/default/8c4bbe7/2147483647/strip/true/crop/4882x1180+0+0/resize/1440x348!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-4-17_12-29-48-version-1-modificationdate-1725534957893-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/874e2ad/2147483647/strip/true/crop/4882x1180+0+0/resize/2880x696!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-4-17_12-29-48-version-1-modificationdate-1725534957893-api-v2.png 2x" width="1440" height="348" src="https://docscontent.nvidia.com/dims4/default/8c4bbe7/2147483647/strip/true/crop/4882x1180+0+0/resize/1440x348!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-4-17_12-29-48-version-1-modificationdate-1725534957893-api-v2.png" loading="lazy"> </picture> </figure> </div> </div><p> </p><p>To show how to use RDMA write and send, even buffers are sent from the client with write immediate, while odd buffers are sent with send immediate. In both cases, the server must pre-post the RDMA receive operations.</p> <p>For each buffer, the CUDA kernel code repeats the handshake:</p> <p></p><p></p><div class="Enhancement" data-align-inline has-height style="--image-height: 250px"> <div class="Enhancement-item"> <figure class="Figure"> <picture> <source type="image/webp" width="1440" height="695" srcset="https://docscontent.nvidia.com/dims4/default/225b584/2147483647/strip/true/crop/1910x922+0+0/resize/1440x695!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_16-43-24-version-1-modificationdate-1725534950450-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/426e083/2147483647/strip/true/crop/1910x922+0+0/resize/2880x1390!/format/webp/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_16-43-24-version-1-modificationdate-1725534950450-api-v2.png 2x"> <source type="image/png" width="1440" height="695" srcset="https://docscontent.nvidia.com/dims4/default/cc7f719/2147483647/strip/true/crop/1910x922+0+0/resize/1440x695!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_16-43-24-version-1-modificationdate-1725534950450-api-v2.png"> <img class="Image" alt="image-2024-6-26_16-43-24-version-1-modificationdate-1725534950450-api-v2.png" srcset="https://docscontent.nvidia.com/dims4/default/cc7f719/2147483647/strip/true/crop/1910x922+0+0/resize/1440x695!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_16-43-24-version-1-modificationdate-1725534950450-api-v2.png 1x,https://docscontent.nvidia.com/dims4/default/7054c79/2147483647/strip/true/crop/1910x922+0+0/resize/2880x1390!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_16-43-24-version-1-modificationdate-1725534950450-api-v2.png 2x" width="1440" height="695" src="https://docscontent.nvidia.com/dims4/default/cc7f719/2147483647/strip/true/crop/1910x922+0+0/resize/1440x695!/quality/90/?url=https%3A%2F%2Fk3-prod-nvidia-docs.s3.us-west-2.amazonaws.com%2Fbrightspot%2Fconfluence%2F00000193-44fd-ddc9-adbf-e6ffd7010000%2Fdoca%2Fsdk%2Fimages%2Fdownload%2Fattachments%2F3137267888%2Fimage-2024-6-26_16-43-24-version-1-modificationdate-1725534950450-api-v2.png" loading="lazy"> </picture> </figure> </div> </div><p> </p><p>Once all buffers are filled, the server double checks that all values are valid. The server output should be as follows:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">DOCA RDMA Server side</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="comments"># Ensure DOCA and DPDK are in the LD_LIBRARY_PATH environment variable</code> $ <code class="functions">cd</code> /opt/mellanox/doca/samples/doca_gpunetio/gpunetio_rdma_client_server_write $ ./build/doca_gpunetio_rdma_client_server_write -gpu 17:00.0 -d mlx5_0   [14:11:43:000930][1173110][DOCA][INF][gpunetio_rdma_client_server_write_main.c:250][main] Starting the sample ... [14:11:43:686610][1173110][DOCA][INF][rdma_common.c:91][oob_connection_server_setup] Listening <code class="keyword">for</code> incoming connections [14:11:45:681523][1173110][DOCA][INF][rdma_common.c:105][oob_connection_server_setup] Client connected at IP: 192.168.2.28 and port: 46274 ... [14:11:45:771807][1173110][DOCA][INF][gpunetio_rdma_client_server_write_sample.c:644][rdma_write_server] Before launching CUDA kernel, buffer array A is: [14:11:45:771822][1173110][DOCA][INF][gpunetio_rdma_client_server_write_sample.c:646][rdma_write_server] Buffer 0 -> offset 0: 1111 | offset 128: 1111 [14:11:45:771837][1173110][DOCA][INF][gpunetio_rdma_client_server_write_sample.c:646][rdma_write_server] Buffer 1 -> offset 0: 1111 | offset 128: 1111 [14:11:45:771851][1173110][DOCA][INF][gpunetio_rdma_client_server_write_sample.c:646][rdma_write_server] Buffer 2 -> offset 0: 1111 | offset 128: 1111 [14:11:45:771864][1173110][DOCA][INF][gpunetio_rdma_client_server_write_sample.c:646][rdma_write_server] Buffer 3 -> offset 0: 1111 | offset 128: 1111 RDMA Recv 2 ops completed with immediate values 0 and 1! RDMA Recv 2 ops completed with immediate values 1 and 2! RDMA Recv 2 ops completed with immediate values 2 and 3! RDMA Recv 2 ops completed with immediate values 3 and 4! [14:11:45:781561][1173110][DOCA][INF][gpunetio_rdma_client_server_write_sample.c:671][rdma_write_server] After launching CUDA kernel, buffer array A is: [14:11:45:781574][1173110][DOCA][INF][gpunetio_rdma_client_server_write_sample.c:673][rdma_write_server] Buffer 0 -> offset 0: 2222 | offset 128: 3333 [14:11:45:781583][1173110][DOCA][INF][gpunetio_rdma_client_server_write_sample.c:673][rdma_write_server] Buffer 1 -> offset 0: 2222 | offset 128: 3333 [14:11:45:781593][1173110][DOCA][INF][gpunetio_rdma_client_server_write_sample.c:673][rdma_write_server] Buffer 2 -> offset 0: 2222 | offset 128: 3333 [14:11:45:781602][1173110][DOCA][INF][gpunetio_rdma_client_server_write_sample.c:673][rdma_write_server] Buffer 3 -> offset 0: 2222 | offset 128: 3333 [14:11:45:781640][1173110][DOCA][INF][gpunetio_rdma_client_server_write_main.c:294][main] Sample finished successfully</p></code> </pre> </div> </bsp-code-block> <p> </p> <p>On the other side, assuming the server is at IP address <code class="InlineCodeBlock"><span>192.168.2.28</span></code>, the client output should be as follows:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">DOCA RDMA Client side</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="comments"># Ensure DOCA and DPDK are in the LD_LIBRARY_PATH environment variable</code>   $ <code class="functions">cd</code> /opt/mellanox/doca/samples/doca_gpunetio/gpunetio_rdma_client_server_write $ ./build/doca_gpunetio_rdma_client_server_write -gpu 17:00.0 -d mlx5_0 -c 192.168.2.28   [16:08:22:335744][160913][DOCA][INF][gpunetio_rdma_client_server_write_main.c:197][main] Starting the sample ... [16:08:25:753316][160913][DOCA][INF][rdma_common.c:147][oob_connection_client_setup] Connected with server successfully ...... Client waiting on flag 7f6596735000 <code class="keyword">for</code> server to post RDMA Recvs Thread 0 post rdma write imm 0 Thread 1 post rdma write imm 0 Client waiting on flag 7f6596735001 <code class="keyword">for</code> server to post RDMA Recvs Thread 0 post rdma send imm 1 Thread 1 post rdma send imm 1 Client waiting on flag 7f6596735002 <code class="keyword">for</code> server to post RDMA Recvs Thread 0 post rdma write imm 2 Thread 1 post rdma write imm 2 Client waiting on flag 7f6596735003 <code class="keyword">for</code> server to post RDMA Recvs Thread 0 post rdma send imm 3 Thread 1 post rdma send imm 3 [16:08:25:853454][160913][DOCA][INF][gpunetio_rdma_client_server_write_main.c:241][main] Sample finished successfully </p></code> </pre> </div> </bsp-code-block> <p> </p> <div class="CallOutBox" data-option-Note> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-check"></use></svg> </div><div class="CallOutBox-title">Note</div> <div class="CallOutBox-body"> <p>With RDMA, the network device must be specified by name (e.g., <code class="InlineCodeBlock"><span>mlx5_0</span></code> ) instead of the PCIe address (as is the case for Ethernet).</p> </div> </div> <p> </p> <p class="auto-cursor-target">It is also possible to enable the RDMA CM mode, establishing two connections with the same RDMA GPU handler. An example on the client side:</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">DOCA RDMA Client side with CM</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p><code class="comments"># Ensure DOCA and DPDK are in the LD_LIBRARY_PATH environment variable</code>   $ <code class="functions">cd</code> /opt/mellanox/doca/samples/doca_gpunetio/gpunetio_rdma_client_server_write $ ./build/samples/doca_gpunetio_rdma_client_server_write -d mlx5_0 -gpu 17:00.0 -gid 3 -c 10.137.189.28 -cm --server-addr-<code class="functions">type</code> ipv4 --server-addr 192.168.2.28   [11:30:34:489781][3853018][DOCA][INF][gpunetio_rdma_client_server_write_main.c:461][main] Starting the sample ... [11:30:35:038828][3853018][DOCA][INF][gpunetio_rdma_client_server_write_sample.c:950][rdma_write_client] Client is waiting <code class="keyword">for</code> a connection establishment [11:30:35:082039][3853018][DOCA][INF][gpunetio_rdma_client_server_write_sample.c:963][rdma_write_client] Client - Connection 1 is established ... [11:30:35:095282][3853018][DOCA][INF][gpunetio_rdma_client_server_write_sample.c:1006][rdma_write_client] Establishing connection 2.. [11:30:35:097521][3853018][DOCA][INF][gpunetio_rdma_client_server_write_sample.c:1016][rdma_write_client] Client is waiting <code class="keyword">for</code> a connection establishment [11:30:35:102718][3853018][DOCA][INF][gpunetio_rdma_client_server_write_sample.c:1029][rdma_write_client] Client - Connection 2 is established [11:30:35:102783][3853018][DOCA][INF][gpunetio_rdma_client_server_write_sample.c:1046][rdma_write_client] Client, terminate kernels Client waiting on flag 7f16067b5000 <code class="keyword">for</code> server to post RDMA Recvs Thread 0 post rdma write imm 0 Thread 1 post rdma write imm 1 Client waiting on flag 7f16067b5001 <code class="keyword">for</code> server to post RDMA Recvs Thread 0 post rdma send imm 1 Thread 1 post rdma send imm 2 Client waiting on flag 7f16067b5002 <code class="keyword">for</code> server to post RDMA Recvs Thread 0 post rdma write imm 2 Thread 1 post rdma write imm 3 Client waiting on flag 7f16067b5003 <code class="keyword">for</code> server to post RDMA Recvs Thread 0 post rdma send imm 3 Thread 1 post rdma send imm 4 Client posted and completed 4 RDMA commits on connection 0. Waiting on the <code class="functions">exit</code> flag. Client waiting on flag 7f16067b5000 <code class="keyword">for</code> server to post RDMA Recvs Thread 0 post rdma write imm 0 Thread 1 post rdma write imm 1 Client waiting on flag 7f16067b5001 <code class="keyword">for</code> server to post RDMA Recvs Thread 0 post rdma send imm 1 Thread 1 post rdma send imm 2 Client waiting on flag 7f16067b5002 <code class="keyword">for</code> server to post RDMA Recvs Thread 0 post rdma write imm 2 Thread 1 post rdma write imm 3 Client waiting on flag 7f16067b5003 <code class="keyword">for</code> server to post RDMA Recvs Thread 0 post rdma send imm 3 Thread 1 post rdma send imm 4 Client posted and completed 4 RDMA commits on connection 1. Waiting on the <code class="functions">exit</code> flag. [11:30:35:122448][3853018][DOCA][INF][gpunetio_rdma_client_server_write_main.c:512][main] Sample finished successfully</p></code> </pre> </div> </bsp-code-block> <p> </p> <p class="auto-cursor-target">In case of RDMA CM, the command option <code class="InlineCodeBlock"><span>-cm</span></code> must be specified on the server side.</p> <div class="CallOutBox" data-option-Warning> <div class="CallOutBox-icon"><svg><use xlink:href="#calloutbox-icon-info"></use></svg> </div><div class="CallOutBox-title">Warning</div> <div class="CallOutBox-body"> <p>Printing from a CUDA kernel is not recommended for performance. It may make sense for debugging purposes and for simple samples like this one.</p> </div> </div> <p> <br> </p> <h2 id="src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-GPUDMACopy"> <a href="#src-3137267888_id-.DOCAGPUNetIOv2.9.0LTS-GPUDMACopy" data-cms-ai="0">GPU DMA Copy</a> </h2> <p>This sample exhibits how to use the DOCA DMA and DOCA GPUNetIO libraries to DMA copy a memory buffer from the CPU to the GPU (with DOCA DMA CPU functions) and from the GPU to the CPU (with DOCA GPUNetIO DMA device functions) from a CUDA kernel. This sample requires a DPU as it uses the DMA engine on it.</p><bsp-code-block class="CodeBlock" data-module="standard" data-language="text/plain" data-noAnnotations data-iscollapsible > <div class="CodeBlock-wrapper"> <div class="CodeBlock-header"> <h4 class="CodeBlock-title">DOCA RDMA Client side</h4> <div class="CodeBlock-collapseButton"> <svg class="CodeBlock-chevron"> <use xlink:href="#chevron-down"></use> </svg> <button class="CodeBlock-collapse">Collapse Source</button> </div> </div> <div class="CodeBlock-copy"> <div class="CodeBlock-copy-message"> <div class="CodeBlock-copy-message-prompt">Copy</div> <div class="CodeBlock-copy-message-done">Copied!</div> </div> <div class="CodeBlock-copy-button"> <div class="CodeBlock-copy-icon"> <svg class="done"><use xlink:href="#check"></use></svg> </div> </div> </div> <pre id="play" class="language- " plain-text data-line=""> <div class="CodeBlock-lineNumbers"></div> <code><p>$ <code class="functions">cd</code> /opt/mellanox/doca/samples/doca_gpunetio/gpunetio_dma_memcpy   <code class="comments"># Build the sample and then execute</code>   $ ./build/doca_gpunetio_dma_memcpy -g 17:00.0 -n ca:00.0 [15:44:04:189462][862197][DOCA][INF][gpunetio_dma_memcpy_main.c:164][main] Starting the sample EAL: Detected CPU lcores: 64 EAL: Detected NUMA nodes: 2 EAL: Detected shared linkage of DPDK EAL: Selected IOVA mode <code class="string">'VA'</code> EAL: No <code class="functions">free</code> 2048 kB hugepages reported on node 0 EAL: No <code class="functions">free</code> 2048 kB hugepages reported on node 1 EAL: VFIO support initialized TELEMETRY: No legacy callbacks, legacy socket not created EAL: Probe PCI driver: gpu_cuda (10de:2331) device: 0000:17:00.0 (socket 0) [15:44:04:857251][862197][DOCA][INF][gpunetio_dma_memcpy_sample.c:211][init_sample_mem_objs] The CPU <code class="functions">source</code> buffer value to be copied to GPU memory: This is a sample piece of text from CPU [15:44:04:857359][862197][DOCA][WRN][doca_mmap.cpp:1743][doca_mmap_set_memrange] Mmap 0x55aec6206140: Memory range isn't cache-line aligned - addr=0x55aec52ceb10. For best performance align address to 64B [15:44:04:858839][862197][DOCA][INF][gpunetio_dma_memcpy_sample.c:158][init_sample_mem_objs] The GPU <code class="functions">source</code> buffer value to be copied to CPU memory: This is a sample piece of text from GPU [15:44:04:921702][862197][DOCA][INF][gpunetio_dma_memcpy_sample.c:570][submit_dma_memcpy_task] Success, DMA memcpy job <code class="keyword">done</code> successfully CUDA KERNEL INFO: The GPU destination buffer value after the memcpy: This is a sample piece of text from CPU CPU received message from GPU: This is a sample piece of text from GPU [15:44:04:930087][862197][DOCA][INF][gpunetio_dma_memcpy_sample.c:364][gpu_dma_cleanup] Cleanup DMA ctx with GPU data path [15:44:04:932658][862197][DOCA][INF][gpunetio_dma_memcpy_sample.c:404][gpu_dma_cleanup] Cleanup DMA ctx with CPU data path [15:44:04:954156][862197][DOCA][INF][gpunetio_dma_memcpy_main.c:197][main] Sample finished successfully</p></code> </pre> </div> </bsp-code-block> <p> <br> </p> </div> </div> </main> </bsp-book-content> </div> <bsp-copyright class='Page-copyright'> <span class="Page-copyright-text">© Copyright 2024, NVIDIA.</span> <span class="Page-copyright-update">Last updated on Nov 19, 2024.</span> </bsp-copyright> </main> <aside class="Page-aside"> <bsp-book class="Book"> <div class="Book-container"> <div class="Book-header"> <div class="Book-title">Topics</div> <div class="Book-toggle"> <div class="Book-toggle-icon"> <div></div> <div></div> <div></div> </div> </div> </div> <ul class="Book-items"> <li class="Book-items-item"><bsp-chapter class="Chapter" > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/index.html" data-cms-ai="0">DOCA Documentation v2.9.0</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li class="Book-items-item"><bsp-chapter class="Chapter" data-has-chapters > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+SDK+v2-9-0/index.html" data-cms-ai="0">DOCA SDK v2.9.0</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Overview/index.html" data-cms-ai="0">NVIDIA DOCA Overview</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Release+Notes/index.html" data-cms-ai="0">NVIDIA DOCA Release Notes</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/General+Support/index.html" data-cms-ai="0">General Support</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Changes+and+New+Features/index.html" data-cms-ai="0">Changes and New Features</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Bug+Fixes+in+This+Version/index.html" data-cms-ai="0">Bug Fixes in This Version</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Known+Issues/index.html" data-cms-ai="0">Known Issues</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/BlueField+and+DOCA+User+Types/index.html" data-cms-ai="0">BlueField and DOCA User Types</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+EULA/index.html" data-cms-ai="0">NVIDIA DOCA EULA</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li class="Book-items-item"><bsp-chapter class="Chapter" data-has-chapters > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Quick+Start+for+BlueField+Developers/index.html" data-cms-ai="0">Quick Start for BlueField Developers</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Developer+Quick+Start+Guide/index.html" data-cms-ai="0">NVIDIA DOCA Developer Quick Start Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li class="Book-items-item"><bsp-chapter class="Chapter" data-has-chapters > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Installation+and+Setup/index.html" data-cms-ai="0">Installation and Setup</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-has-chapters data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Profiles/index.html" data-cms-ai="0">NVIDIA DOCA Profiles</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+MLNX_OFED+to+DOCA-OFED+Transition+Guide/index.html" data-cms-ai="0">NVIDIA MLNX_OFED to DOCA-OFED Transition Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Installation+Guide+for+Linux/index.html" data-cms-ai="0">NVIDIA DOCA Installation Guide for Linux</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Developer+Guide/index.html" data-cms-ai="0">NVIDIA DOCA Developer Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li class="Book-items-item"><bsp-chapter class="Chapter" data-has-chapters > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Programming+Guide/index.html" data-cms-ai="0">DOCA Programming Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-has-chapters data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Programming+Overview/index.html" data-cms-ai="0">DOCA Programming Overview</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Hardware+Overview/index.html" data-cms-ai="0">Hardware Overview</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+SDK+Architecture/index.html" data-cms-ai="0">DOCA SDK Architecture</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Backward+Compatibility+Policy/index.html" data-cms-ai="0">DOCA Backward Compatibility Policy</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Development+Best+Practices/index.html" data-cms-ai="0">DOCA Development Best Practices</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Capability+Checking/index.html" data-cms-ai="0">Capability Checking</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Debuggability/index.html" data-cms-ai="0">Debuggability</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Libraries/index.html" data-cms-ai="0">DOCA Libraries</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-has-chapters data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Common/index.html" data-cms-ai="0">DOCA Common</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-has-chapters data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Core/index.html" data-cms-ai="0">DOCA Core</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Sync+Event/index.html" data-cms-ai="0">Sync Event</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Mmap+Advise/index.html" data-cms-ai="0">Mmap Advise</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Log/index.html" data-cms-ai="0">DOCA Log</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Flow/index.html" data-cms-ai="0">DOCA Flow</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Flow+Connection+Tracking/index.html" data-cms-ai="0">DOCA Flow Connection Tracking</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Flow+Tune+Server/index.html" data-cms-ai="0">DOCA Flow Tune Server</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DPA+Subsystem/index.html" data-cms-ai="0">DPA Subsystem</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DPA+Development/index.html" data-cms-ai="0">DPA Development</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+DPA/index.html" data-cms-ai="0">DOCA DPA</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+PCC/index.html" data-cms-ai="0">DOCA PCC</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+DMA/index.html" data-cms-ai="0">DOCA DMA</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Comch/index.html" data-cms-ai="0">DOCA Comch</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+UROM/index.html" data-cms-ai="0">DOCA UROM</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+RDMA/index.html" data-cms-ai="0">DOCA RDMA</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Ethernet/index.html" data-cms-ai="0">DOCA Ethernet</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+GPUNetIO/index.html" data-cms-ai="0">DOCA GPUNetIO</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+App+Shield/index.html" data-cms-ai="0">DOCA App Shield</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Compress/index.html" data-cms-ai="0">DOCA Compress</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+SHA/index.html" data-cms-ai="0">DOCA SHA</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Erasure+Coding/index.html" data-cms-ai="0">DOCA Erasure Coding</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+AES-GCM/index.html" data-cms-ai="0">DOCA AES-GCM</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Rivermax/index.html" data-cms-ai="0">DOCA Rivermax</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Telemetry+Exporter/index.html" data-cms-ai="0">DOCA Telemetry Exporter</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Telemetry/index.html" data-cms-ai="0">DOCA Telemetry</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Device+Emulation/index.html" data-cms-ai="0">DOCA Device Emulation</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-has-chapters data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+DevEmu+PCI/index.html" data-cms-ai="0">DOCA DevEmu PCI</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+DevEmu+PCI+Generic/index.html" data-cms-ai="0">DOCA DevEmu PCI Generic</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+DevEmu+Virtio/index.html" data-cms-ai="0">DOCA DevEmu Virtio</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+DevEmu+Virtio-FS/index.html" data-cms-ai="0">DOCA DevEmu Virtio-FS</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> </ul> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Utils/index.html" data-cms-ai="0">DOCA Utils</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Arg+Parser/index.html" data-cms-ai="0">DOCA Arg Parser</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Drivers/index.html" data-cms-ai="0">DOCA Drivers</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+UCX/index.html" data-cms-ai="0">DOCA UCX</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/MLX+Drivers+(MLNX_OFED)/index.html" data-cms-ai="0">MLX Drivers (MLNX_OFED)</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-has-chapters data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/InfiniBand+Network/index.html" data-cms-ai="0">InfiniBand Network</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/InfiniBand+Interface/index.html" data-cms-ai="0">InfiniBand Interface</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+SM/index.html" data-cms-ai="0">NVIDIA SM</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/QoS+-+Quality+of+Service/index.html" data-cms-ai="0">QoS - Quality of Service</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/IP+over+InfiniBand+(IPoIB)/index.html" data-cms-ai="0">IP over InfiniBand (IPoIB)</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Advanced+Transport/index.html" data-cms-ai="0">Advanced Transport</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Optimized+Memory+Access/index.html" data-cms-ai="0">Optimized Memory Access</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+PeerDirect/index.html" data-cms-ai="0">NVIDIA PeerDirect</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/CPU+Overhead+Distribution/index.html" data-cms-ai="0">CPU Overhead Distribution</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Out-of-Order+(OOO)+Data+Placement/index.html" data-cms-ai="0">Out-of-Order (OOO) Data Placement</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/IB+Router/index.html" data-cms-ai="0">IB Router</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/MAD+Congestion+Control/index.html" data-cms-ai="0">MAD Congestion Control</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DPU/index.html" data-cms-ai="0">DPU</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/QoS+Configuration/index.html" data-cms-ai="0">QoS Configuration</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Shared+RQ+Mode/index.html" data-cms-ai="0">Shared RQ Mode</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Storage+Protocols/index.html" data-cms-ai="0">Storage Protocols</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/SRP+-+SCSI+RDMA+Protocol/index.html" data-cms-ai="0">SRP - SCSI RDMA Protocol</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/iSCSI+Extensions+for+RDMA+(iSER)/index.html" data-cms-ai="0">iSCSI Extensions for RDMA (iSER)</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Lustre/index.html" data-cms-ai="0">Lustre</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVME-oF+-+NVM+Express+over+Fabrics/index.html" data-cms-ai="0">NVME-oF - NVM Express over Fabrics</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Virtualization/index.html" data-cms-ai="0">Virtualization</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-has-chapters data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Single+Root+IO+Virtualization+(SR-IOV)/index.html" data-cms-ai="0">Single Root IO Virtualization (SR-IOV)</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=5 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/SR-IOV+Live+Migration/index.html" data-cms-ai="0">SR-IOV Live Migration</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Enabling+Paravirtualization/index.html" data-cms-ai="0">Enabling Paravirtualization</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/VXLAN+Hardware+Stateless+Offloads/index.html" data-cms-ai="0">VXLAN Hardware Stateless Offloads</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Q-in-Q+Encapsulation+per+VF+in+Linux+(VST)/index.html" data-cms-ai="0">Q-in-Q Encapsulation per VF in Linux (VST)</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/802-1Q+Double-Tagging/index.html" data-cms-ai="0">802.1Q Double-Tagging</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Scalable+Functions/index.html" data-cms-ai="0">Scalable Functions</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Resiliency/index.html" data-cms-ai="0">Resiliency</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Reset+Flow/index.html" data-cms-ai="0">Reset Flow</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Docker+Containers/index.html" data-cms-ai="0">Docker Containers</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Docker+Using+SR-IOV/index.html" data-cms-ai="0">Docker Using SR-IOV</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Kubernetes+Using+SR-IOV/index.html" data-cms-ai="0">Kubernetes Using SR-IOV</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=4 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Kubernetes+with+Shared+HCA/index.html" data-cms-ai="0">Kubernetes with Shared HCA</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/HPC-X/index.html" data-cms-ai="0">HPC-X</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Fast+Driver+Unload/index.html" data-cms-ai="0">Fast Driver Unload</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> </ul> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li class="Book-items-item"><bsp-chapter class="Chapter" data-has-chapters > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Reference+Applications/index.html" data-cms-ai="0">DOCA Reference Applications</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+App+Shield+Agent+Application+Guide/index.html" data-cms-ai="0">NVIDIA DOCA App Shield Agent Application Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+DMA+Copy+Application+Guide/index.html" data-cms-ai="0">NVIDIA DOCA DMA Copy Application Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+DPA+All-to-all+Application+Guide/index.html" data-cms-ai="0">NVIDIA DOCA DPA All-to-all Application Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+East-West+Overlay+Encryption+Application/index.html" data-cms-ai="0">NVIDIA DOCA East-West Overlay Encryption Application</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Eth+L2+Forwarding+Application+Guide/index.html" data-cms-ai="0">NVIDIA DOCA Eth L2 Forwarding Application Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+File+Compression+Application+Guide/index.html" data-cms-ai="0">NVIDIA DOCA File Compression Application Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+File+Integrity+Application+Guide/index.html" data-cms-ai="0">NVIDIA DOCA File Integrity Application Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+GPU+Packet+Processing+Application+Guide/index.html" data-cms-ai="0">NVIDIA DOCA GPU Packet Processing Application Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+IPsec+Security+Gateway+Application+Guide/index.html" data-cms-ai="0">NVIDIA DOCA IPsec Security Gateway Application Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+NVMe+Emulation+App+Guide/index.html" data-cms-ai="0">NVIDIA DOCA NVMe Emulation App Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+PCC+Application+Guide/index.html" data-cms-ai="0">NVIDIA DOCA PCC Application Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+PSP+Gateway+Application+Guide/index.html" data-cms-ai="0">NVIDIA DOCA PSP Gateway Application Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Secure+Channel+Application+Guide/index.html" data-cms-ai="0">NVIDIA DOCA Secure Channel Application Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Simple+Forward+VNF+Application+Guide/index.html" data-cms-ai="0">NVIDIA DOCA Simple Forward VNF Application Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Storage+Zero+Copy/index.html" data-cms-ai="0">NVIDIA DOCA Storage Zero Copy</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Storage+Zero+Copy+Comch+to+RDMA+Application+Guide/index.html" data-cms-ai="0">NVIDIA DOCA Storage Zero Copy Comch to RDMA Application Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Storage+Zero+Copy+Initiator+Comch+Application+Guide/index.html" data-cms-ai="0">NVIDIA DOCA Storage Zero Copy Initiator Comch Application Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Storage+Zero+Copy+Target+RDMA+Application+Guide/index.html" data-cms-ai="0">NVIDIA DOCA Storage Zero Copy Target RDMA Application Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Switch+Application+Guide/index.html" data-cms-ai="0">NVIDIA DOCA Switch Application Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+UPF+Application+Guide/index.html" data-cms-ai="0">NVIDIA DOCA UPF Application Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+UROM+RDMO+Application+Guide/index.html" data-cms-ai="0">NVIDIA DOCA UROM RDMO Application Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+YARA+Inspection+Application+Guide/index.html" data-cms-ai="0">NVIDIA DOCA YARA Inspection Application Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li class="Book-items-item"><bsp-chapter class="Chapter" data-has-chapters > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Tools/index.html" data-cms-ai="0">DOCA Tools</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-has-chapters data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Bench/index.html" data-cms-ai="0">NVIDIA DOCA Bench</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Bench+Sample+Invocations/index.html" data-cms-ai="0">DOCA Bench Sample Invocations</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Capabilities+Print+Tool/index.html" data-cms-ai="0">NVIDIA DOCA Capabilities Print Tool</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Comm+Channel+Admin+Tool/index.html" data-cms-ai="0">NVIDIA DOCA Comm Channel Admin Tool</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DPA+Tools/index.html" data-cms-ai="0">NVIDIA DPA Tools</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+DPACC+Compiler/index.html" data-cms-ai="0">NVIDIA DOCA DPACC Compiler</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+DPA+Execution+Unit+Management+Tool/index.html" data-cms-ai="0">NVIDIA DOCA DPA Execution Unit Management Tool</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+DPA+GDB+Server+Tool/index.html" data-cms-ai="0">NVIDIA DOCA DPA GDB Server Tool</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+DPA+PS+Tool/index.html" data-cms-ai="0">NVIDIA DOCA DPA PS Tool</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+DPA+Statistics+Tool/index.html" data-cms-ai="0">NVIDIA DOCA DPA Statistics Tool</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Flow+Tune+Tool/index.html" data-cms-ai="0">NVIDIA DOCA Flow Tune Tool</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+PCC+Counter+Tool/index.html" data-cms-ai="0">NVIDIA DOCA PCC Counter Tool</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Socket+Relay/index.html" data-cms-ai="0">NVIDIA DOCA Socket Relay</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li class="Book-items-item"><bsp-chapter class="Chapter" data-has-chapters > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Services/index.html" data-cms-ai="0">DOCA Services</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+BlueField+Container+Deployment+Guide/index.html" data-cms-ai="0">NVIDIA BlueField Container Deployment Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+BlueMan+Service+Guide/index.html" data-cms-ai="0">NVIDIA DOCA BlueMan Service Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Firefly+Service+Guide/index.html" data-cms-ai="0">NVIDIA DOCA Firefly Service Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Flow+Inspector+Service+Guide/index.html" data-cms-ai="0">NVIDIA DOCA Flow Inspector Service Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+HBN+Service+Guide/index.html" data-cms-ai="0">NVIDIA DOCA HBN Service Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/HBN+Service+Release+Notes/index.html" data-cms-ai="0">HBN Service Release Notes</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/HBN+Service+Deployment/index.html" data-cms-ai="0">HBN Service Deployment</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/HBN+Service+Configuration/index.html" data-cms-ai="0">HBN Service Configuration</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/HBN+Service+Troubleshooting/index.html" data-cms-ai="0">HBN Service Troubleshooting</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Management+Service+Guide/index.html" data-cms-ai="0">NVIDIA DOCA Management Service Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+SNAP+Services/index.html" data-cms-ai="0">NVIDIA DOCA SNAP Services</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-has-chapters data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+SNAP+Virtio-fs+Service+Guide/index.html" data-cms-ai="0">NVIDIA DOCA SNAP Virtio-fs Service Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/SNAP+Virtio-fs+Release+Notes/index.html" data-cms-ai="0">SNAP Virtio-fs Release Notes</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+SNAP-4+Service+Guide/index.html" data-cms-ai="0">NVIDIA DOCA SNAP-4 Service Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/SNAP-4+Service+Release+Notes/index.html" data-cms-ai="0">SNAP-4 Service Release Notes</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+SNAP-3+User+Guide/index.html" data-cms-ai="0">NVIDIA DOCA SNAP-3 User Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/SNAP-3+Release+Notes/index.html" data-cms-ai="0">SNAP-3 Release Notes</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Telemetry+Service+Guide/index.html" data-cms-ai="0">NVIDIA DOCA Telemetry Service Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+UROM+Service+Guide/index.html" data-cms-ai="0">NVIDIA DOCA UROM Service Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Virtio-net+Service+Guide/index.html" data-cms-ai="0">NVIDIA DOCA Virtio-net Service Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Virtio-net+Service+Guide+Release+Notes/index.html" data-cms-ai="0">Virtio-net Service Guide Release Notes</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+OpenvSwitch+Acceleration+-+OVS+in+DOCA/index.html" data-cms-ai="0">NVIDIA OpenvSwitch Acceleration - OVS in DOCA</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/OVS-Kernel+Hardware+Acceleration/index.html" data-cms-ai="0">OVS-Kernel Hardware Acceleration</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/OVS-DPDK+Hardware+Acceleration/index.html" data-cms-ai="0">OVS-DPDK Hardware Acceleration</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/OVS-DOCA+Hardware+Acceleration/index.html" data-cms-ai="0">OVS-DOCA Hardware Acceleration</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/OVS+Metrics/index.html" data-cms-ai="0">OVS Metrics</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/OVS+Inside+BlueField/index.html" data-cms-ai="0">OVS Inside BlueField</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Virtual+Switch+on+BlueField/index.html" data-cms-ai="0">Virtual Switch on BlueField</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li class="Book-items-item"><bsp-chapter class="Chapter" data-has-chapters > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/API+References/index.html" data-cms-ai="0">API References</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Driver+APIs/index.html" data-cms-ai="0">NVIDIA DOCA Driver APIs</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Library+APIs/index.html" data-cms-ai="0">NVIDIA DOCA Library APIs</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li class="Book-items-item"><bsp-chapter class="Chapter" data-has-chapters > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Miscellaneous+(Runtime)/index.html" data-cms-ai="0">Miscellaneous (Runtime)</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Glossary/index.html" data-cms-ai="0">NVIDIA DOCA Glossary</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Crypto+Acceleration/index.html" data-cms-ai="0">NVIDIA DOCA Crypto Acceleration</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Services+Fluent+Logger/index.html" data-cms-ai="0">NVIDIA DOCA Services Fluent Logger</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+DPU+CLI/index.html" data-cms-ai="0">NVIDIA DOCA DPU CLI</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/fTPM+over+OP-TEE/index.html" data-cms-ai="0">fTPM over OP-TEE</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Emulated+Devices/index.html" data-cms-ai="0">NVIDIA DOCA Emulated Devices</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+BlueField+Modes+of+Operation/index.html" data-cms-ai="0">NVIDIA BlueField Modes of Operation</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DOCA+Switching/index.html" data-cms-ai="0">DOCA Switching</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DPU+Kernel+Representors+Model/index.html" data-cms-ai="0">DPU Kernel Representors Model</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/VirtIO+Acceleration+through+Hardware+vDPA/index.html" data-cms-ai="0">Virtio Acceleration through Hardware vDPA</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Bridge+Offload/index.html" data-cms-ai="0">Bridge Offload</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Link+Aggregation/index.html" data-cms-ai="0">Link Aggregation</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Controlling+Host+PF+and+VF+Parameters/index.html" data-cms-ai="0">Controlling Host PF and VF Parameters</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=3 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Configuring+Uplink+MTU/index.html" data-cms-ai="0">Configuring Uplink MTU</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/DPDK+on+BlueField/index.html" data-cms-ai="0">DPDK on BlueField</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+with+OpenSSL/index.html" data-cms-ai="0">NVIDIA DOCA with OpenSSL</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+BlueField+DPU+Scalable+Function+User+Guide/index.html" data-cms-ai="0">NVIDIA BlueField DPU Scalable Function User Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+TLS+Offload+Guide/index.html" data-cms-ai="0">NVIDIA TLS Offload Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-has-chapters data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Troubleshooting+Guide/index.html" data-cms-ai="0">NVIDIA DOCA Troubleshooting Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=2 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+BlueField+Reset+and+Reboot+Procedures/index.html" data-cms-ai="0">NVIDIA BlueField Reset and Reboot Procedures</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/BlueField+SR-IOV/index.html" data-cms-ai="0">BlueField SR-IOV</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Virtual+Functions+User+Guide/index.html" data-cms-ai="0">NVIDIA DOCA Virtual Functions User Guide</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> <li class="Book-items-item"><bsp-chapter class="Chapter" data-has-chapters > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/Archives/index.html" data-cms-ai="0">Archives</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> <ul class="Chapter-chapters"> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+LTS+Versions/index.html" data-cms-ai="0">NVIDIA DOCA LTS Versions</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> <li><bsp-chapter class="Chapter" data-level=1 > <div class="Chapter-header"> <div class="Chapter-title"> <a class="Link" href="https://docs.nvidia.com/doca/sdk/NVIDIA+DOCA+Documentation+Archives/index.html" data-cms-ai="0">NVIDIA DOCA Documentation Archives</a> </div> <div class="Chapter-icon"> <svg class="Chapter-iconCollapsed"> <use xlink:href="#icon-plus"></use> </svg> <svg class="Chapter-iconExpanded"> <use xlink:href="#icon-minus"></use> </svg> </div> <div class="Chapter-icon-sub"> <svg class="Chapter-iconExpanded sub"> <use xlink:href="#chevron-up"></use> </svg> <svg class="Chapter-iconCollapsed sub"> <use xlink:href="#chevron-down"></use> </svg> </div> </div> </bsp-chapter> </li> </ul> </bsp-chapter> </li> </ul> <span class="Book-resizer"></span> </div> </bsp-book> </aside> </div> </div> <footer class="Page-footer-wrapper"> <div class="Page-footer"> <div class="Page-footer-container"> <div class="Page-footer-columns"> <div class="Page-footer-column"></div> <div class="Page-footer-column"> <div class="Page-footer-navigation" data-no-content> <nav class="FooterNavigation"> <ul class="FooterNavigation-items" data-column-count="3"> <li class="FooterNavigation-items-item"> <div class="FooterNavigationItem"> <div class="FooterNavigationItem-text"> <span>Corporate Info</span> </div> <ul class="FooterNavigationItem-items"> <li class="FooterNavigationItem-items-item"><a class="NavigationLink" href="https://www.nvidia.com/en-us/" target="_blank" data-cms-ai="0"><span class="NavigationLink-text">NVIDIA.com Home</span></a> </li> <li class="FooterNavigationItem-items-item"><a class="NavigationLink" href="https://www.nvidia.com/en-us/about-nvidia/" target="_blank" data-cms-ai="0"><span class="NavigationLink-text">About NVIDIA</span></a> </li> </ul> </div> </li> <li class="FooterNavigation-items-item"> <div class="FooterNavigationItem"> <div class="FooterNavigationItem-text"> <span>‎NVIDIA Developer</span> </div> <ul class="FooterNavigationItem-items"> <li class="FooterNavigationItem-items-item"><a class="NavigationLink" href="https://developer.nvidia.com/" target="_blank" data-cms-ai="0"><span class="NavigationLink-text">Developer Home</span></a> </li> <li class="FooterNavigationItem-items-item"><a class="NavigationLink" href="https://blogs.nvidia.com/" target="_blank" data-cms-ai="0"><span class="NavigationLink-text">Blog</span></a> </li> </ul> </div> </li> <li class="FooterNavigation-items-item"> <div class="FooterNavigationItem"> <div class="FooterNavigationItem-text"> <span>Resources</span> </div> <ul class="FooterNavigationItem-items"> <li class="FooterNavigationItem-items-item"><a class="NavigationLink" href="https://www.nvidia.com/en-us/contact/" target="_blank" data-cms-ai="0"><span class="NavigationLink-text">Contact Us</span></a> </li> <li class="FooterNavigationItem-items-item"><a class="NavigationLink" href="https://developer.nvidia.com/developer-program" target="_blank" data-cms-ai="0"><span class="NavigationLink-text">Developer Program</span></a> </li> </ul> </div> </li> </ul> </nav> </div> </div> </div> </div> </div> <div class="Page-footer-bottom"> <div class="Page-footer-disclaimer"><p><a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/privacy-policy/" target="_blank" rel="noopener" data-cms-ai="0" >Privacy Policy </a>| <a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/privacy-center/" target="_blank" rel="noopener" data-cms-ai="0" >Manage My Privacy</a> | <a class="Link" href="https://www.nvidia.com/en-us/preferences/start/" target="_blank" rel="noopener" data-cms-ai="0" >Do Not Sell or Share My Data</a> | <a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/terms-of-service/" target="_blank" rel="noopener" data-cms-ai="0" >Terms of Service</a> | <a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/accessibility/" target="_blank" rel="noopener" data-cms-ai="0" >Accessibility</a> | <a class="Link" href="https://www.nvidia.com/en-us/about-nvidia/company-policies/" target="_blank" rel="noopener" data-cms-ai="0" >Corporate Policies</a> | <a class="Link" href="https://www.nvidia.com/en-us/product-security/" target="_blank" rel="noopener" data-cms-ai="0" >Product Security</a> | <a class="Link" href="https://www.nvidia.com/en-us/contact/" target="_blank" rel="noopener" data-cms-ai="0" >Contact</a></p><p>Copyright © 2024 NVIDIA Corporation</p></div> </div> </footer> <script>_satellite.pageBottom();</script> <script></script><div id="chatbot"></div></script> <script src="https://d4j85rjepgcta.cloudfront.net/nvidiadocs/chatbot.js"></script> <script> const tables = document.querySelectorAll('.Page-twoColumn table'); const className = 'table-wrapper' tables.forEach(tbl => { const parent = tbl.parentNode; if(!parent.classList.contains(className)){ const wrapper = document.createElement('div'); wrapper.classList.add(className) parent.insertBefore( wrapper, tbl ); wrapper.appendChild(tbl); } }) </script> <bsp-modal class="Modal"> <div class="Modal-wrapper"> <div class="Modal-close"><button>Close</button></div> <div class="Modal-content"> content here </div> </div> </bsp-modal> <span data-bsp-pv="75918a10-b479-3fb1-bb84-6a57df4b0241"></span><span data-bsp-pv="00000193-4502-d86e-a5f3-fd0f98450000"></span><script data-bsp-site="00000183-66b2-dbb3-afcb-6ebe6f520000" data-bsp-contentid="00000193-4502-d86e-a5f3-fd0f98450000"></script><script data-bsp-site="00000183-66b2-dbb3-afcb-6ebe6f520000" data-bsp-contentid="00000193-4502-d86e-a5f3-fd0f98450000" src="https://docscontent.nvidia.com/resource/00000183-6725-d7fd-a5bb-6727dc560000/_resource/analytics/bsp-analytics.min.3d492319d8b084de04ab3a208c32f0b5.gz.js"></script></body> </html>

CINXE.COM

DOCA GPUNetIO - NVIDIA Docs