CINXE.COM

<!DOCTYPE html> <html class="writer-html5" lang="en" data-content_root="../"> <head> <meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> <title>Spock Quick-Start Guide — OLCF User Documentation</title> <link rel="stylesheet" type="text/css" href="../_static/pygments.css?v=b86133f3" /> <link rel="stylesheet" type="text/css" href="../_static/css/theme.css?v=e59714d7" /> <link rel="stylesheet" type="text/css" href="../_static/copybutton.css?v=76b2166b" /> <link rel="stylesheet" type="text/css" href="../_static/sphinx-design.min.css?v=95c83b7e" /> <link rel="stylesheet" type="text/css" href="../_static/css/theme_overrides.css?v=b668e930" /> <link rel="shortcut icon" href="../_static/favicon.ico"/> <link rel="canonical" href="https://docs.olcf.ornl.govsystems/spock_quick_start_guide.html"/> <script src="../_static/jquery.js?v=5d32c60e"></script> <script src="../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script> <script src="../_static/documentation_options.js?v=5929fcd5"></script> <script src="../_static/doctools.js?v=9a2dae69"></script> <script src="../_static/sphinx_highlight.js?v=dc90522c"></script> <script src="../_static/clipboard.min.js?v=a7894cd8"></script> <script src="../_static/copybutton.js?v=99362dbf"></script> <script src="../_static/design-tabs.js?v=f930bc37"></script> <script src="../_static/js/custom.js?v=1617c1c8"></script> <script src="../_static/js/theme.js"></script> <link rel="index" title="Index" href="../genindex.html" /> <link rel="search" title="Search" href="../search.html" /> <link rel="next" title="Crusher Quick-Start Guide" href="crusher_quick_start_guide.html" /> <link rel="prev" title="Odo" href="odo_user_guide.html" /> </head> <body class="wy-body-for-nav"> <div class="wy-grid-for-nav"> <nav data-toggle="wy-nav-shift" class="wy-nav-side"> <div class="wy-side-scroll"> <div class="wy-side-nav-search" style="background: #efefef" > <a href="../index.html"> <img src="../_static/olcf_logo.png" class="logo" alt="Logo"/> </a> <div role="search"> <form id="rtd-search-form" class="wy-form" action="../search.html" method="get"> <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" /> <input type="hidden" name="check_keywords" value="yes" /> <input type="hidden" name="area" value="default" /> </form> </div> </div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu"> <ul> <li class="toctree-l1"><a class="reference internal" href="../support/index.html">Contact & Support</a><ul> <li class="toctree-l2"><a class="reference internal" href="../support/index.html#olcf-user-assistance-center">OLCF User Assistance Center</a></li> <li class="toctree-l2"><a class="reference internal" href="../support/index.html#authentication-support">Authentication Support</a></li> <li class="toctree-l2"><a class="reference internal" href="../support/index.html#olcf-office-hours">OLCF Office Hours</a></li> <li class="toctree-l2"><a class="reference internal" href="../support/index.html#communication-to-users">Communication to Users</a></li> </ul> </li> </ul> <ul> <li class="toctree-l1"><a class="reference internal" href="../accounts/index.html">Accounts and Projects</a><ul> <li class="toctree-l2"><a class="reference internal" href="../accounts/accounts_and_projects.html">Request a New Allocation</a><ul> <li class="toctree-l3"><a class="reference internal" href="../accounts/accounts_and_projects.html#what-are-the-differences-between-project-types">What are the differences between project types?</a></li> <li class="toctree-l3"><a class="reference internal" href="../accounts/accounts_and_projects.html#what-happens-after-a-project-request-is-approved">What happens after a project request is approved?</a></li> <li class="toctree-l3"><a class="reference internal" href="../accounts/accounts_and_projects.html#guidance-on-frontier-allocation-requests">Guidance on Frontier Allocation Requests</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../accounts/accounts_and_projects.html#applying-for-a-user-account">Applying for a user account</a></li> <li class="toctree-l2"><a class="reference internal" href="../accounts/accounts_and_projects.html#checking-the-status-of-your-application">Checking the status of your application</a></li> <li class="toctree-l2"><a class="reference internal" href="../accounts/accounts_and_projects.html#get-access-to-additional-projects">Get access to additional projects</a></li> <li class="toctree-l2"><a class="reference internal" href="../accounts/frequently_asked_questions.html">Frequently Asked Questions</a><ul> <li class="toctree-l3"><a class="reference internal" href="../accounts/frequently_asked_questions.html#how-do-i-apply-for-an-account">How do I apply for an account?</a></li> <li class="toctree-l3"><a class="reference internal" href="../accounts/frequently_asked_questions.html#what-is-the-status-of-my-application">What is the status of my application?</a></li> <li class="toctree-l3"><a class="reference internal" href="../accounts/frequently_asked_questions.html#how-should-i-acknowledge-the-olcf-in-my-publications-and-presentations">How should I acknowledge the OLCF in my publications and presentations?</a></li> <li class="toctree-l3"><a class="reference internal" href="../accounts/frequently_asked_questions.html#what-is-a-subproject">What is a subproject?</a></li> <li class="toctree-l3"><a class="reference internal" href="../accounts/frequently_asked_questions.html#i-no-longer-need-my-account-who-should-i-inform-and-what-should-i-do-with-my-olcf-issued-rsa-securid-token">I no longer need my account. Who should I inform and what should I do with my OLCF issued RSA SecurID token?</a></li> <li class="toctree-l3"><a class="reference internal" href="../accounts/frequently_asked_questions.html#my-securid-token-is-broken-expired-what-should-i-do">My SecurID token is broken/expired. What should I do?</a></li> <li class="toctree-l3"><a class="reference internal" href="../accounts/frequently_asked_questions.html#getting-help">Getting Help</a></li> <li class="toctree-l3"><a class="reference internal" href="../accounts/frequently_asked_questions.html#additional-resources">Additional Resources</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../accounts/documents_and_forms.html">Documents and Forms</a><ul> <li class="toctree-l3"><a class="reference internal" href="../accounts/documents_and_forms.html#forms-for-requesting-a-project-allocation">Forms for Requesting a Project Allocation</a></li> <li class="toctree-l3"><a class="reference internal" href="../accounts/documents_and_forms.html#forms-for-requesting-an-account">Forms for Requesting an Account</a></li> <li class="toctree-l3"><a class="reference internal" href="../accounts/documents_and_forms.html#forms-to-request-changes-to-computers-jobs-or-accounts">Forms to Request Changes to Computers, Jobs, or Accounts</a></li> <li class="toctree-l3"><a class="reference internal" href="../accounts/documents_and_forms.html#report-templates">Report Templates</a></li> <li class="toctree-l3"><a class="reference internal" href="../accounts/documents_and_forms.html#miscellaneous-forms">Miscellaneous Forms</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../accounts/olcf_policy_guide.html">OLCF Policy Guides</a><ul> <li class="toctree-l3"><a class="reference internal" href="../accounts/olcf_policy_guide.html#olcf-acknowledgement">OLCF Acknowledgement</a></li> <li class="toctree-l3"><a class="reference internal" href="../accounts/olcf_policy_guide.html#software-requests">Software Requests</a></li> <li class="toctree-l3"><a class="reference internal" href="../accounts/olcf_policy_guide.html#special-requests-and-policy-exemptions">Special Requests and Policy Exemptions</a></li> <li class="toctree-l3"><a class="reference internal" href="../accounts/olcf_policy_guide.html#computing-policy">Computing Policy</a><ul> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#computer-use">Computer Use</a></li> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#data-use">Data Use</a></li> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#software-use">Software Use</a></li> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#user-accountability">User Accountability</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../accounts/olcf_policy_guide.html#data-management-policy">Data Management Policy</a><ul> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#introduction">Introduction</a></li> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#data-retention-purge-quotas">Data Retention, Purge, & Quotas</a></li> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#data-prohibitions-safeguards">Data Prohibitions & Safeguards</a></li> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#software">Software</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../accounts/olcf_policy_guide.html#security-policy">Security Policy</a><ul> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#scope">Scope</a></li> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#personal-use">Personal Use</a></li> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#accessing-olcf-computational-resources">Accessing OLCF Computational Resources</a></li> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#data-management">Data Management</a></li> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#sensitive-data">Sensitive Data</a></li> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#data-transfer">Data Transfer</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../accounts/olcf_policy_guide.html#incite-allocation-under-utilization-policy">INCITE Allocation Under-utilization Policy</a></li> <li class="toctree-l3"><a class="reference internal" href="../accounts/olcf_policy_guide.html#project-reporting-policy">Project Reporting Policy</a></li> <li class="toctree-l3"><a class="reference internal" href="../accounts/olcf_policy_guide.html#non-proprietary-institutional-user-agreement-policy">Non-proprietary Institutional User Agreement Policy</a><ul> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#access">Access</a></li> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#rules-and-regulations">Rules and Regulations</a></li> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#safety-and-health">Safety and Health</a></li> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#intent-to-publish">Intent to Publish</a></li> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#export-control">Export Control</a></li> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#intellectual-property">Intellectual Property</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../accounts/olcf_policy_guide.html#hipaa-itar-project-rules-of-behavior-policy">HIPAA/ITAR Project Rules of Behavior Policy</a></li> <li class="toctree-l3"><a class="reference internal" href="../accounts/olcf_policy_guide.html#user-managed-software-ums-policy">User-Managed Software (UMS) Policy</a><ul> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#purpose">Purpose</a></li> <li class="toctree-l4"><a class="reference internal" href="../accounts/olcf_policy_guide.html#policies">Policies</a></li> </ul> </li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../accounts/glossary.html">Glossary</a></li> <li class="toctree-l2"><a class="reference internal" href="../accounts/index.html#additional-resources">Additional Resources</a></li> </ul> </li> </ul> <ul> <li class="toctree-l1"><a class="reference internal" href="../connecting/index.html">Connecting</a><ul> <li class="toctree-l2"><a class="reference internal" href="../connecting/index.html#connecting-for-the-first-time">Connecting for the first time</a></li> <li class="toctree-l2"><a class="reference internal" href="../connecting/index.html#activating-a-new-securid-fob">Activating a new SecurID fob</a></li> <li class="toctree-l2"><a class="reference internal" href="../connecting/index.html#pins-passcodes-and-tokencodes">PINs, Passcodes, and Tokencodes</a></li> <li class="toctree-l2"><a class="reference internal" href="../connecting/index.html#x11-forwarding">X11 Forwarding</a></li> <li class="toctree-l2"><a class="reference internal" href="../connecting/index.html#id2">Systems Available to All Projects</a></li> <li class="toctree-l2"><a class="reference internal" href="../connecting/index.html#olcf-system-hostnames">OLCF System Hostnames</a></li> <li class="toctree-l2"><a class="reference internal" href="../connecting/index.html#starting-a-tmux-session">Starting a Tmux Session</a></li> <li class="toctree-l2"><a class="reference internal" href="../connecting/index.html#checking-system-availability">Checking System Availability</a></li> </ul> </li> </ul> <ul class="current"> <li class="toctree-l1 current"><a class="reference internal" href="index.html">Systems</a><ul class="current"> <li class="toctree-l2"><a class="reference internal" href="2024_olcf_system_changes.html">2024 Notable System Changes</a><ul> <li class="toctree-l3"><a class="reference internal" href="2024_olcf_system_changes.html#hpss-decommission-and-kronos-availability">HPSS Decommission and Kronos Availability</a><ul> <li class="toctree-l4"><a class="reference internal" href="2024_olcf_system_changes.html#late-july-2024-kronos-available">Late July 2024 - Kronos available</a></li> <li class="toctree-l4"><a class="reference internal" href="2024_olcf_system_changes.html#august-30-2024-hpss-becomes-read-only">August 30, 2024 - HPSS becomes read-only</a></li> <li class="toctree-l4"><a class="reference internal" href="2024_olcf_system_changes.html#january-31-2025-hpss-decommissioned">January 31, 2025 - HPSS decommissioned</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="2024_olcf_system_changes.html#summit-and-alpine2-decommissions">Summit and Alpine2 Decommissions</a><ul> <li class="toctree-l4"><a class="reference internal" href="2024_olcf_system_changes.html#november-15-2024-summit-decommissioned">November 15, 2024 - Summit decommissioned</a></li> <li class="toctree-l4"><a class="reference internal" href="2024_olcf_system_changes.html#november-19-2024-alpine2-read-only">November 19, 2024 - Alpine2 read-only</a></li> <li class="toctree-l4"><a class="reference internal" href="2024_olcf_system_changes.html#january-31-2025-alpine2-decommissioned">January 31, 2025 - Alpine2 decommissioned</a></li> </ul> </li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="frontier_user_guide.html">Frontier User Guide</a><ul> <li class="toctree-l3"><a class="reference internal" href="frontier_user_guide.html#system-overview">System Overview</a><ul> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#frontier-compute-nodes">Frontier Compute Nodes</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#node-types">Node Types</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#system-interconnect">System Interconnect</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#file-systems">File Systems</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#operating-system">Operating System</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#gpus">GPUs</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="frontier_user_guide.html#connecting">Connecting</a></li> <li class="toctree-l3"><a class="reference internal" href="frontier_user_guide.html#data-and-storage">Data and Storage</a><ul> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#transition-from-alpine-to-orion">Transition from Alpine to Orion</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#lfs-setstripe-wrapper">LFS setstripe wrapper</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#nfs-filesystem">NFS Filesystem</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#lustre-filesystem">Lustre Filesystem</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#kronos-archival-storage">Kronos Archival Storage</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#nvme">NVMe</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#nvme-usage">NVMe Usage</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="frontier_user_guide.html#using-globus-to-move-data-to-and-from-orion">Using Globus to Move Data to and from Orion</a></li> <li class="toctree-l3"><a class="reference internal" href="frontier_user_guide.html#amd-gpus">AMD GPUs</a><ul> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#amd-vs-nvidia-terminology">AMD vs NVIDIA Terminology</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#blocks-workgroups-threads-work-items-grids-wavefronts">Blocks (workgroups), Threads (work items), Grids, Wavefronts</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#the-compute-unit">The Compute Unit</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#hip">HIP</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#things-to-remember-when-programming-for-amd-gpus">Things To Remember When Programming for AMD GPUs</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="frontier_user_guide.html#programming-environment">Programming Environment</a><ul> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#environment-modules-lmod">Environment Modules (Lmod)</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#compilers">Compilers</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#mpi">MPI</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="frontier_user_guide.html#compiling">Compiling</a><ul> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#id4">Compilers</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#id7">MPI</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#openmp">OpenMP</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#openmp-gpu-offload">OpenMP GPU Offload</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#openacc">OpenACC</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#id9">HIP</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#hip-openmp-cpu-threading">HIP + OpenMP CPU Threading</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#sycl">SYCL</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="frontier_user_guide.html#running-jobs">Running Jobs</a><ul> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#login-vs-compute-nodes">Login vs Compute Nodes</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#simplified-node-layout">Simplified Node Layout</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#slurm">Slurm</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#batch-scripts">Batch Scripts</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#interactive-jobs">Interactive Jobs</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#common-slurm-options">Common Slurm Options</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#slurm-environment-variables">Slurm Environment Variables</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#job-states">Job States</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#job-reason-codes">Job Reason Codes</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#scheduling-policy">Scheduling Policy</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#job-dependencies">Job Dependencies</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#monitoring-and-modifying-batch-jobs">Monitoring and Modifying Batch Jobs</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#srun">Srun</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#process-and-thread-mapping-examples">Process and Thread Mapping Examples</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#ensemble-jobs">Ensemble Jobs</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#tips-for-launching-at-scale">Tips for Launching at Scale</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="frontier_user_guide.html#software">Software</a></li> <li class="toctree-l3"><a class="reference internal" href="frontier_user_guide.html#debugging">Debugging</a><ul> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#linaro-ddt">Linaro DDT</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#gdb">GDB</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#valgrind4hpc">Valgrind4hpc</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#omnitrace">Omnitrace</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="frontier_user_guide.html#profiling-applications">Profiling Applications</a><ul> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#getting-started-with-the-hpe-performance-analysis-tools-pat">Getting Started with the HPE Performance Analysis Tools (PAT)</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#getting-started-with-hpctoolkit">Getting Started with HPCToolkit</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#getting-started-with-the-rocm-profiler">Getting Started with the ROCm Profiler</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#roofline-profiling-with-the-rocm-profiler">Roofline Profiling with the ROCm Profiler</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#omniperf">Omniperf</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="frontier_user_guide.html#tips-and-tricks">Tips and Tricks</a><ul> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#using-reduced-precision-fp16-and-bf16-datatypes">Using reduced precision (FP16 and BF16 datatypes)</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#enabling-gpu-page-migration">Enabling GPU Page Migration</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#floating-point-fp-atomic-operations-and-coarse-fine-grained-memory-allocations">Floating-Point (FP) Atomic Operations and Coarse/Fine Grained Memory Allocations</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#performance-considerations-for-lds-fp-atomicadd">Performance considerations for LDS FP atomicAdd()</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#library-considerations-with-atomic-operations">Library considerations with atomic operations</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="frontier_user_guide.html#system-updates">System Updates</a><ul> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#id17">2025-02-18</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#id18">2025-01-14</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#id19">2024-11-12</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#id20">2024-09-03</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#id21">2024-08-20</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#id22">2024-07-16</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#id23">2024-04-17</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#id24">2024-03-19</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#id25">2024-01-23</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#id26">2023-12-05</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#id27">2023-10-03</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#id28">2023-09-19</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#id29">2023-07-18</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#id30">2023-05-09</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="frontier_user_guide.html#known-issues">Known Issues</a><ul> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#open-issues">Open Issues</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#open-issues-w-workaround">Open Issues w/Workaround</a></li> <li class="toctree-l4"><a class="reference internal" href="frontier_user_guide.html#resolved-issues">Resolved Issues</a></li> </ul> </li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="citadel_user_guide.html">Citadel User Guide</a><ul> <li class="toctree-l3"><a class="reference internal" href="citadel_user_guide.html#what-is-citadel">What is Citadel</a><ul> <li class="toctree-l4"><a class="reference internal" href="citadel_user_guide.html#citadel-spi-documentation">Citadel (SPI) Documentation</a></li> </ul> </li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="andes_user_guide.html">Andes User Guide</a><ul> <li class="toctree-l3"><a class="reference internal" href="andes_user_guide.html#system-overview">System Overview</a><ul> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#compute-nodes">Compute Nodes</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#login-nodes">Login Nodes</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#file-systems">File Systems</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#lfs-setstripe-wrapper">LFS setstripe Wrapper</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="andes_user_guide.html#shell-and-programming-environments">Shell and Programming Environments</a><ul> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#default-shell">Default shell</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#environment-management-with-lmod">Environment Management with lmod</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#installed-software">Installed Software</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="andes_user_guide.html#compiling">Compiling</a><ul> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#available-compilers">Available Compilers</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#changing-compilers">Changing Compilers</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#compiler-wrappers">Compiler Wrappers</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#compiling-threaded-codes">Compiling Threaded Codes</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="andes_user_guide.html#running-jobs">Running Jobs</a><ul> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#login-vs-compute-nodes-on-commodity-clusters">Login vs Compute Nodes on Commodity Clusters</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#slurm">Slurm</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#interactive-batch-jobs-on-commodity-clusters">Interactive Batch Jobs on Commodity Clusters</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#common-batch-options-to-slurm">Common Batch Options to Slurm</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#batch-environment-variables">Batch Environment Variables</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#modifying-batch-jobs">Modifying Batch Jobs</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#monitoring-batch-jobs">Monitoring Batch Jobs</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#job-execution">Job Execution</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#batch-queues-on-andes">Batch Queues on Andes</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#job-accounting-on-andes">Job Accounting on Andes</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="andes_user_guide.html#andes-debugging">Debugging</a><ul> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#linaro-ddt">Linaro DDT</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#gdb">GDB</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#valgrind">Valgrind</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="andes_user_guide.html#visualization-tools">Visualization tools</a><ul> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#paraview">ParaView</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#visit">VisIt</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#remote-visualization-using-vnc-non-gpu">Remote Visualization using VNC (non-GPU)</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#remote-visualization-using-vnc-gpu-nodes">Remote Visualization using VNC (GPU nodes)</a></li> <li class="toctree-l4"><a class="reference internal" href="andes_user_guide.html#remote-visualization-using-nice-dcv-gpu-nodes-only">Remote Visualization using Nice DCV (GPU nodes only)</a></li> </ul> </li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="home_user_guide.html">Home</a><ul> <li class="toctree-l3"><a class="reference internal" href="home_user_guide.html#system-overview">System Overview</a></li> <li class="toctree-l3"><a class="reference internal" href="home_user_guide.html#access-connecting">Access & Connecting</a></li> <li class="toctree-l3"><a class="reference internal" href="home_user_guide.html#usage">Usage</a><ul> <li class="toctree-l4"><a class="reference internal" href="home_user_guide.html#acceptable-tasks">Acceptable Tasks</a></li> <li class="toctree-l4"><a class="reference internal" href="home_user_guide.html#unacceptable-tasks">Unacceptable Tasks</a></li> </ul> </li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="dtn_user_guide.html">Data Transfer Nodes (DTNs)</a><ul> <li class="toctree-l3"><a class="reference internal" href="dtn_user_guide.html#system-overview">System Overview</a></li> <li class="toctree-l3"><a class="reference internal" href="dtn_user_guide.html#interactive-access">Interactive Access</a></li> <li class="toctree-l3"><a class="reference internal" href="dtn_user_guide.html#access-from-globus-online">Access From Globus Online</a></li> <li class="toctree-l3"><a class="reference internal" href="dtn_user_guide.html#batch-queue-slurm">Batch Queue (Slurm)</a><ul> <li class="toctree-l4"><a class="reference internal" href="dtn_user_guide.html#queue-policy">Queue Policy</a></li> <li class="toctree-l4"><a class="reference internal" href="dtn_user_guide.html#submitting-jobs-to-frontier">Submitting jobs to Frontier</a></li> </ul> </li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="hpss_user_guide.html">High Performance Storage System</a><ul> <li class="toctree-l3"><a class="reference internal" href="hpss_user_guide.html#system-overview">System Overview</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="odo_user_guide.html">Odo</a><ul> <li class="toctree-l3"><a class="reference internal" href="odo_user_guide.html#system-overview">System Overview</a><ul> <li class="toctree-l4"><a class="reference internal" href="odo_user_guide.html#file-systems">File Systems</a></li> <li class="toctree-l4"><a class="reference internal" href="odo_user_guide.html#obtaining-access-to-odo">Obtaining Access to Odo</a></li> <li class="toctree-l4"><a class="reference internal" href="odo_user_guide.html#logging-in-to-odo">Logging In to Odo</a></li> </ul> </li> </ul> </li> <li class="toctree-l2 current"><a class="current reference internal" href="#">Spock Quick-Start Guide</a><ul> <li class="toctree-l3"><a class="reference internal" href="#system-overview">System Overview</a><ul> <li class="toctree-l4"><a class="reference internal" href="#spock-compute-nodes">Spock Compute Nodes</a></li> <li class="toctree-l4"><a class="reference internal" href="#system-interconnect">System Interconnect</a></li> <li class="toctree-l4"><a class="reference internal" href="#file-systems">File Systems</a></li> <li class="toctree-l4"><a class="reference internal" href="#gpus">GPUs</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="#connecting">Connecting</a></li> <li class="toctree-l3"><a class="reference internal" href="#data-and-storage">Data and Storage</a><ul> <li class="toctree-l4"><a class="reference internal" href="#nfs">NFS</a></li> <li class="toctree-l4"><a class="reference internal" href="#gpfs">GPFS</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="#programming-environment">Programming Environment</a><ul> <li class="toctree-l4"><a class="reference internal" href="#environment-modules-lmod">Environment Modules (Lmod)</a></li> <li class="toctree-l4"><a class="reference internal" href="#compilers">Compilers</a></li> <li class="toctree-l4"><a class="reference internal" href="#mpi">MPI</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="#compiling">Compiling</a><ul> <li class="toctree-l4"><a class="reference internal" href="#id3">MPI</a></li> <li class="toctree-l4"><a class="reference internal" href="#openmp">OpenMP</a></li> <li class="toctree-l4"><a class="reference internal" href="#openmp-gpu-offload">OpenMP GPU Offload</a></li> <li class="toctree-l4"><a class="reference internal" href="#hip">HIP</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="#running-jobs">Running Jobs</a><ul> <li class="toctree-l4"><a class="reference internal" href="#slurm-workload-manager">Slurm Workload Manager</a></li> <li class="toctree-l4"><a class="reference internal" href="#slurm-compute-node-partitions">Slurm Compute Node Partitions</a></li> <li class="toctree-l4"><a class="reference internal" href="#process-and-thread-mapping">Process and Thread Mapping</a></li> <li class="toctree-l4"><a class="reference internal" href="#nvme-usage">NVMe Usage</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="#getting-help">Getting Help</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="crusher_quick_start_guide.html">Crusher Quick-Start Guide</a><ul> <li class="toctree-l3"><a class="reference internal" href="crusher_quick_start_guide.html#system-overview">System Overview</a><ul> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#crusher-compute-nodes">Crusher Compute Nodes</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#system-interconnect">System Interconnect</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#file-systems">File Systems</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#gpus">GPUs</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="crusher_quick_start_guide.html#connecting">Connecting</a></li> <li class="toctree-l3"><a class="reference internal" href="crusher_quick_start_guide.html#data-and-storage">Data and Storage</a><ul> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#nfs-filesystem">NFS Filesystem</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#lustre-filesystem">Lustre Filesystem</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="crusher_quick_start_guide.html#programming-environment">Programming Environment</a><ul> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#environment-modules-lmod">Environment Modules (Lmod)</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#compilers">Compilers</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#mpi">MPI</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="crusher_quick_start_guide.html#compiling">Compiling</a><ul> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#id3">MPI</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#openmp">OpenMP</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#openmp-gpu-offload">OpenMP GPU Offload</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#hip">HIP</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#hip-openmp-cpu-threading">HIP + OpenMP CPU Threading</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#sycl">SYCL</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="crusher_quick_start_guide.html#running-jobs">Running Jobs</a><ul> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#slurm-workload-manager">Slurm Workload Manager</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#slurm-compute-node-partitions">Slurm Compute Node Partitions</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#process-and-thread-mapping">Process and Thread Mapping</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#nvme-usage">NVMe Usage</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#tips-for-launching-at-scale">Tips for Launching at Scale</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="crusher_quick_start_guide.html#profiling-applications">Profiling Applications</a><ul> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#getting-started-with-the-hpe-performance-analysis-tools-pat">Getting Started with the HPE Performance Analysis Tools (PAT)</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#getting-started-with-hpctoolkit">Getting Started with HPCToolkit</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#getting-started-with-the-rocm-profiler">Getting Started with the ROCm Profiler</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#roofline-profiling-with-the-rocm-profiler">Roofline Profiling with the ROCm Profiler</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#omnitrace">Omnitrace</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#omniperf">Omniperf</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="crusher_quick_start_guide.html#notable-differences-between-summit-and-crusher">Notable Differences between Summit and Crusher</a><ul> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#using-reduced-precision-fp16-and-bf16-datatypes">Using reduced precision (FP16 and BF16 datatypes)</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#enabling-gpu-page-migration">Enabling GPU Page Migration</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#floating-point-fp-atomic-operations-and-coarse-fine-grained-memory-allocations">Floating-Point (FP) Atomic Operations and Coarse/Fine Grained Memory Allocations</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#performance-considerations-for-lds-fp-atomicadd">Performance considerations for LDS FP atomicAdd()</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="crusher_quick_start_guide.html#system-updates">System Updates</a><ul> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#id6">2024-03-19</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#id7">2024-01-23</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#id8">2023-12-05</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#id9">2023-10-03</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#id10">2023-09-19</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#id11">2023-07-18</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#id12">2023-04-05</a></li> <li class="toctree-l4"><a class="reference internal" href="crusher_quick_start_guide.html#id13">2022-12-29</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="crusher_quick_start_guide.html#getting-help">Getting Help</a></li> </ul> </li> </ul> </li> </ul> <ul> <li class="toctree-l1"><a class="reference internal" href="../services_and_applications/index.html">Services and Applications</a><ul> <li class="toctree-l2"><a class="reference internal" href="../services_and_applications/slate/index.html">Slate</a><ul> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/slate/overview.html">Overview</a><ul> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/overview.html#what-is-slate">What is Slate?</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/overview.html#what-is-kubernetes">What is Kubernetes?</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/overview.html#what-is-openshift">What is OpenShift?</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/slate/getting_started.html">Getting Started</a><ul> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/getting_started.html#requesting-a-slate-project-allocation">Requesting A Slate Project Allocation</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/getting_started.html#logging-in">Logging in</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/getting_started.html#slate-namespaces">Slate Namespaces</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/getting_started.html#install-the-oc-tool">Install the OC tool</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/getting_started.html#test-login-with-oc-tool">Test login with OC Tool</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/slate/guided_tutorial.html">Guided Tutorial</a><ul> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/guided_tutorial.html#creating-your-project">Creating your project</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/guided_tutorial.html#guided-web-gui-tutorial">Guided Web GUI Tutorial</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/slate/guided_tutorial_cli.html">Guided Tutorial: CLI</a><ul> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/guided_tutorial_cli.html#adding-a-pod-to-your-project">Adding a Pod to your Project</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/slate/image_building.html">Image Building</a><ul> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/image_building.html#build-types">Build Types</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/image_building.html#examples">Examples</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/image_building.html#logging-into-the-registry-externally">Logging into the Registry Externally</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/slate/workloads/index.html">Workloads</a><ul> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/workloads/pod.html">Pods</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/workloads/deployment.html">Deployments</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/slate/networking/index.html">Networking</a><ul> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/networking/services.html">Services</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/networking/nodeport.html">NodePorts</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/networking/route.html">Routes</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/networking/networkpolicy.html">Network Policies</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/networking/port_forwarding.html">Quick Access from Outside Slate</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/slate/storage.html">Persistent Storage</a><ul> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/storage.html#creating-a-persistent-volume-claim">Creating A Persistent Volume Claim</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/storage.html#adding-pvc-to-pod">Adding PVC To Pod</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/storage.html#backups">Backups</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/slate/workflows/index.html">Workflows</a><ul> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/workflows/overview.html">Workflows Overview</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/workflows/openshift_gitops.html">OpenShift GitOps</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/workflows/openshift_pipelines.html">OpenShift Pipelines</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/slate/use_cases/index.html">Application Deployment Examples</a><ul> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/use_cases/simple_website.html">Build and Deploy Simple Website</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/use_cases/mongodb_service.html">Deploy MongoDB</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/use_cases/nginx_hello_world.html">Deploy NGINX with Hello World</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/use_cases/gitlab_runner.html">GitLab Runners</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/use_cases/helm_example.html">Deploy Packages with Helm</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/use_cases/helm_prerequisite.html">Helm Prerequisites</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/use_cases/minio.html">MinIO Object Store (On an NCCS Filesystem)</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/slate/access_olcf_resources/index.html">Access OLCF Resources From Containers</a><ul> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/access_olcf_resources/job_submit.html">Batch Job Submission</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/access_olcf_resources/mount_fs.html">Mount OLCF Filesystems</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/slate/other_resources.html">Schedule Other Slate Resources</a><ul> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/other_resources.html#gpus">GPUs</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/slate/olcf_provided_applications/index.html">OLCF-Provided Applications on Slate</a><ul class="simple"> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/slate/troubleshooting/index.html">Troubleshooting</a><ul> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/troubleshooting/fix-writable-directories.html">Fix Container Image Permissions</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/troubleshooting/debugging.html">Debugging</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/slate/examples.html">YAML Object Quick Reference</a><ul> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/examples.html#cronjobs">CronJobs</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/examples.html#deployments-and-stateful-sets">Deployments and Stateful Sets</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/examples.html#pods">Pods</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/examples.html#roles-and-rolebindings">Roles and Rolebindings</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/examples.html#routes-services-and-nodeports">Routes, Services and Nodeports</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/slate/examples.html#persistent-volume-claims">Persistent Volume Claims</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/slate/glossary.html">Glossary</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../services_and_applications/myolcf/index.html">myOLCF</a><ul> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/myolcf/overview.html">Overview</a><ul> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/myolcf/overview.html#what-is-myolcf">What is myOLCF?</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/myolcf/overview.html#what-can-it-do">What can it do?</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/myolcf/overview.html#can-i-suggest-a-feature">Can I suggest a feature?</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/myolcf/authenticating.html">Authenticating</a><ul> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/myolcf/authenticating.html#olcf-moderate-accounts">OLCF Moderate Accounts</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/myolcf/authenticating.html#olcf-open-accounts">OLCF Open Accounts</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/myolcf/project_pages/project_pages.html">Project Pages</a><ul> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/myolcf/project_pages/project_pages.html#project-context">Project Context</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/myolcf/project_pages/project_pages.html#switching-project-contexts">Switching Project Contexts</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/myolcf/project_pages/project_pages.html#available-pages">Available Pages</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/myolcf/account_pages/account_pages.html">Account Pages</a><ul> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/myolcf/account_pages/account_pages.html#account-context">Account Context</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/myolcf/account_pages/account_pages.html#available-pages">Available Pages</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/myolcf/account_pages/processing_membership_requests.html">Processing Project Membership Requests</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../services_and_applications/jupyter/index.html">Jupyter</a><ul> <li class="toctree-l3"><a class="reference internal" href="../services_and_applications/jupyter/overview.html">Overview</a><ul> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/jupyter/overview.html#jupyter-at-olcf">Jupyter at OLCF</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/jupyter/overview.html#access">Access</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/jupyter/overview.html#cpu-vs-gpu-jupyterlab-available-resources">CPU vs. GPU JupyterLab (Available Resources)</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/jupyter/overview.html#working-within-lustre-and-nfs-launching-a-notebook">Working within Lustre and NFS (Launching a Notebook)</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/jupyter/overview.html#conda-environments-and-custom-notebooks">Conda Environments and Custom Notebooks</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/jupyter/overview.html#manually-stopping-your-jupyterlab-session">Manually Stopping Your JupyterLab Session</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/jupyter/overview.html#things-to-be-aware-of">Things to Be Aware Of</a></li> <li class="toctree-l4"><a class="reference internal" href="../services_and_applications/jupyter/overview.html#example-jupyter-notebooks">Example Jupyter Notebooks</a></li> </ul> </li> </ul> </li> </ul> </li> </ul> <ul> <li class="toctree-l1"><a class="reference internal" href="../data/index.html">Data Storage and Transfers</a><ul> <li class="toctree-l2"><a class="reference internal" href="../data/index.html#summary-of-storage-areas">Summary of Storage Areas</a><ul> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#notes-on-user-centric-data-storage">Notes on User-Centric Data Storage</a><ul> <li class="toctree-l4"><a class="reference internal" href="../data/index.html#user-home-directories-nfs">User Home Directories (NFS)</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#notes-on-project-centric-data-storage">Notes on Project-Centric Data Storage</a><ul> <li class="toctree-l4"><a class="reference internal" href="../data/index.html#project-home-directories-nfs">Project Home Directories (NFS)</a></li> <li class="toctree-l4"><a class="reference internal" href="../data/index.html#project-work-areas">Project Work Areas</a></li> <li class="toctree-l4"><a class="reference internal" href="../data/index.html#project-archive-directories">Project Archive Directories</a></li> </ul> </li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../data/index.html#data-policies">Data Policies</a><ul> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#information">Information</a></li> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#special-requests">Special Requests</a></li> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#data-retention">Data Retention</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../data/index.html#orion-lustre-hpe-clusterstor-filesystem">Orion Lustre HPE ClusterStor Filesystem</a><ul> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#orion-performance-tiers-and-file-striping-policy">Orion Performance Tiers and File Striping Policy</a></li> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#i-o-patterns-that-benefit-from-file-striping">I/O Patterns that Benefit from File Striping</a></li> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#lfs-setstripe-wrapper">LFS setstripe wrapper</a></li> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#lustre-file-locking-tips">Lustre File Locking Tips</a></li> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#darshan-runtime-and-i-o-profiling">Darshan-runtime and I/O Profiling</a></li> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#purge">Purge</a></li> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#major-difference-between-lustre-hpe-clusterstor-and-ibm-spectrum-scale">Major difference between Lustre HPE ClusterStor and IBM Spectrum Scale</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../data/index.html#hpss-data-archival-system">HPSS Data Archival System</a></li> <li class="toctree-l2"><a class="reference internal" href="../data/index.html#kronos-nearline-archival-storage-system">Kronos Nearline Archival Storage System</a><ul> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#access-data-transfer">Access / Data Transfer</a></li> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#directory-structure">Directory Structure</a></li> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#project-quotas">Project Quotas</a></li> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#kronos-and-hpss-comparison">Kronos and HPSS Comparison</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../data/index.html#transferring-data">Transferring Data</a><ul> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#data-transferring-data-globus">Globus</a><ul> <li class="toctree-l4"><a class="reference internal" href="../data/index.html#using-globus-to-move-data-between-collections">Using Globus to Move Data Between Collections</a></li> <li class="toctree-l4"><a class="reference internal" href="../data/index.html#using-globus-from-your-local-workstation">Using Globus From Your Local Workstation</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#hsi">HSI</a><ul> <li class="toctree-l4"><a class="reference internal" href="../data/index.html#additional-hsi-documentation">Additional HSI Documentation</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#htar">HTAR</a><ul> <li class="toctree-l4"><a class="reference internal" href="../data/index.html#htar-limitations">HTAR Limitations</a></li> <li class="toctree-l4"><a class="reference internal" href="../data/index.html#additional-htar-documentation">Additional HTAR Documentation</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../data/index.html#command-line-terminal-tools">Command-Line/Terminal Tools</a></li> </ul> </li> </ul> </li> </ul> <ul> <li class="toctree-l1"><a class="reference internal" href="../software/index.html">Software</a><ul> <li class="toctree-l2"><a class="reference internal" href="../software/software-news.html">Software News</a><ul> <li class="toctree-l3"><a class="reference internal" href="../software/software-news.html#frontier-system-software-update-february-18-2025">Frontier: System Software Update (February 18, 2025)</a></li> <li class="toctree-l3"><a class="reference internal" href="../software/software-news.html#frontier-updated-modules-for-cpe-23-12-october-16-2024">Frontier: Updated Modules for cpe/23.12 (October 16 2024)</a></li> <li class="toctree-l3"><a class="reference internal" href="../software/software-news.html#frontier-core-module-october-15-2024">Frontier: Core Module (October 15, 2024)</a></li> <li class="toctree-l3"><a class="reference internal" href="../software/software-news.html#frontier-system-software-update-july-16-2024">Frontier: System Software Update (July 16, 2024)</a></li> <li class="toctree-l3"><a class="reference internal" href="../software/software-news.html#frontier-user-environment-changes-july-9-2024">Frontier: User Environment Changes (July 9, 2024)</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../software/analytics/index.html">ML/DL & Data Analytics</a><ul> <li class="toctree-l3"><a class="reference internal" href="../software/analytics/apache-spark.html">Apache Spark</a><ul> <li class="toctree-l4"><a class="reference internal" href="../software/analytics/apache-spark.html#overview">Overview</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/analytics/apache-spark.html#getting-started">Getting Started</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../software/analytics/pytorch_frontier.html">PyTorch on Frontier</a><ul> <li class="toctree-l4"><a class="reference internal" href="../software/analytics/pytorch_frontier.html#table-of-contents">Table of Contents:</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/analytics/pytorch_frontier.html#installing-pytorch">Installing PyTorch</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/analytics/pytorch_frontier.html#best-practices">Best Practices</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/analytics/pytorch_frontier.html#pytorch-geometric">PyTorch Geometric</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/analytics/pytorch_frontier.html#troubleshooting">Troubleshooting</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/analytics/pytorch_frontier.html#additional-resources">Additional Resources</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../software/analytics/jax.html">JAX</a><ul> <li class="toctree-l4"><a class="reference internal" href="../software/analytics/jax.html#setting-up-the-environment">Setting up the environment</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/analytics/jax.html#installing-jax">Installing JAX</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/analytics/jax.html#testing-jax">Testing JAX</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/analytics/jax.html#additional-resources">Additional Resources</a></li> </ul> </li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../software/python/index.html">Python on OLCF Systems</a><ul> <li class="toctree-l3"><a class="reference internal" href="../software/python/index.html#overview">Overview</a></li> <li class="toctree-l3"><a class="reference internal" href="../software/python/index.html#olcf-python-guides">OLCF Python Guides</a><ul> <li class="toctree-l4"><a class="reference internal" href="../software/python/conda_basics.html">Conda Basics</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/python/parallel_h5py.html">Installing mpi4py and h5py</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/python/cupy.html">Installing CuPy</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/python/sbcast_conda.html">Sbcast Conda Environments</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/python/jupyter_envs.html">Jupyter Visibility</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/analytics/pytorch_frontier.html">PyTorch on Frontier</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/analytics/jax.html">JAX</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../software/python/index.html#module-usage">Module Usage</a><ul> <li class="toctree-l4"><a class="reference internal" href="../software/python/index.html#base-environment">Base Environment</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/python/index.html#custom-environments">Custom Environments</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../software/python/index.html#how-to-run">How to Run</a><ul> <li class="toctree-l4"><a class="reference internal" href="../software/python/index.html#frontier-andes">Frontier / Andes</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../software/python/index.html#best-practices">Best Practices</a></li> <li class="toctree-l3"><a class="reference internal" href="../software/python/index.html#additional-resources">Additional Resources</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../software/profiling/index.html">Profiling Tools</a><ul> <li class="toctree-l3"><a class="reference internal" href="../software/profiling/Scorep.html">Score-P</a><ul> <li class="toctree-l4"><a class="reference internal" href="../software/profiling/Scorep.html#overview">Overview</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/profiling/Scorep.html#usage">Usage</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/profiling/Scorep.html#instrumentation">Instrumentation</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/profiling/Scorep.html#measurement">Measurement</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/profiling/Scorep.html#profiling">Profiling</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/profiling/Scorep.html#tracing">Tracing</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/profiling/Scorep.html#manual-instrumentation">Manual Instrumentation</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/profiling/Scorep.html#score-p-demo-video">Score-P Demo Video</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../software/profiling/TAU.html">Tuning and Analysis Utilities (TAU)</a><ul> <li class="toctree-l4"><a class="reference internal" href="../software/profiling/TAU.html#enabling-tau-at-olcf">Enabling TAU at OLCF</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/profiling/TAU.html#profile-and-trace-using-tau-exec-exe">Profile and trace using “tau_exec exe”</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/profiling/TAU.html#automatic-source-instrumentation-using-compiler-wrappers">Automatic source instrumentation using compiler wrappers</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/profiling/TAU.html#selective-instrumentation">Selective Instrumentation</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/profiling/TAU.html#manual-source-instrumentation">Manual source instrumentation</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/profiling/TAU.html#run-time-environment-variables">Run-Time Environment Variables</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/profiling/TAU.html#compile-time-environment-variables">Compile-Time Environment Variables</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/profiling/TAU.html#references">References</a></li> </ul> </li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../software/UMS/index.html">User-Managed Software</a><ul> <li class="toctree-l3"><a class="reference internal" href="../software/UMS/index.html#introduction">Introduction</a></li> <li class="toctree-l3"><a class="reference internal" href="../software/UMS/index.html#currently-available-user-managed-software">Currently Available User-Managed Software</a></li> <li class="toctree-l3"><a class="reference internal" href="../software/UMS/index.html#usage">Usage</a></li> <li class="toctree-l3"><a class="reference internal" href="../software/UMS/index.html#policies">Policies</a></li> <li class="toctree-l3"><a class="reference internal" href="../software/UMS/index.html#writing-ums-modulefiles">Writing UMS Modulefiles</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../software/workflows/index.html">Workflows</a><ul> <li class="toctree-l3"><a class="reference internal" href="../software/workflows/index.html#running-workflows-on-olcf-resources">Running Workflows on OLCF Resources</a></li> <li class="toctree-l3"><a class="reference internal" href="../software/workflows/index.html#workflow-systems">Workflow Systems</a><ul> <li class="toctree-l4"><a class="reference internal" href="../software/workflows/entk.html">Ensemble Toolkit (EnTK)</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/workflows/parsl.html">Parsl</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/workflows/swift_t.html">Swift/T</a></li> </ul> </li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../software/spack_environments.html">Spack Environments</a><ul> <li class="toctree-l3"><a class="reference internal" href="../software/spack_environments.html#purpose">Purpose</a></li> <li class="toctree-l3"><a class="reference internal" href="../software/spack_environments.html#getting-started">Getting Started</a></li> <li class="toctree-l3"><a class="reference internal" href="../software/spack_environments.html#add-dependencies-to-the-environment">Add Dependencies to the environment</a><ul> <li class="toctree-l4"><a class="reference internal" href="../software/spack_environments.html#adding-olcf-modulefiles-as-external-packages">Adding OLCF Modulefiles as External Packages</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/spack_environments.html#adding-user-defined-dependencies-to-the-environment">Adding User-Defined Dependencies to the environment</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../software/spack_environments.html#installing-the-environment">Installing the Environment</a></li> <li class="toctree-l3"><a class="reference internal" href="../software/spack_environments.html#more-details">More Details</a></li> <li class="toctree-l3"><a class="reference internal" href="../software/spack_environments.html#references">References</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../software/viz_tools/index.html">Visualization Tools</a><ul> <li class="toctree-l3"><a class="reference internal" href="../software/viz_tools/visit.html">VisIt</a><ul> <li class="toctree-l4"><a class="reference internal" href="../software/viz_tools/visit.html#overview">Overview</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/viz_tools/visit.html#installing-and-setting-up-visit">Installing and Setting Up Visit</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/viz_tools/visit.html#remote-gui-usage">Remote GUI Usage</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/viz_tools/visit.html#command-line-example">Command Line Example</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/viz_tools/visit.html#troubleshooting">Troubleshooting</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/viz_tools/visit.html#additional-resources">Additional Resources</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../software/viz_tools/paraview.html">ParaView</a><ul> <li class="toctree-l4"><a class="reference internal" href="../software/viz_tools/paraview.html#overview">Overview</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/viz_tools/paraview.html#installing-and-setting-up-paraview">Installing and Setting Up ParaView</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/viz_tools/paraview.html#remote-gui-usage">Remote GUI Usage</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/viz_tools/paraview.html#command-line-example">Command Line Example</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/viz_tools/paraview.html#troubleshooting">Troubleshooting</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/viz_tools/paraview.html#additional-resources">Additional Resources</a></li> </ul> </li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../software/containers_on_frontier.html">Containers on Frontier</a><ul> <li class="toctree-l3"><a class="reference internal" href="../software/containers_on_frontier.html#examples-for-building-and-running-containers">Examples for Building and Running Containers</a><ul> <li class="toctree-l4"><a class="reference internal" href="../software/containers_on_frontier.html#building-and-running-a-container-image-from-a-base-linux-distribution-for-mpi">Building and running a container image from a base Linux distribution for MPI</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/containers_on_frontier.html#pushing-your-apptainer-image-to-an-oci-registry-supporting-oras-e-g-dockerhub">Pushing your Apptainer image to an OCI Registry supporting ORAS (e.g. DockerHub)</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/containers_on_frontier.html#building-an-image-on-top-of-an-existing-image-local-docker-image-oci-artifact">Building an image on top of an existing image (local, docker image, OCI artifact)</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../software/containers_on_frontier.html#olcf-base-images-apptainer-modules">OLCF Base Images & Apptainer Modules</a><ul> <li class="toctree-l4"><a class="reference internal" href="../software/containers_on_frontier.html#base-images">Base Images</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/containers_on_frontier.html#apptainer-modules">Apptainer Modules</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/containers_on_frontier.html#example-workflow">Example Workflow</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../software/containers_on_frontier.html#some-restrictions-and-tips">Some Restrictions and Tips</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../software/debugging/index.html">Debugging</a><ul> <li class="toctree-l3"><a class="reference internal" href="../software/debugging/index.html#linaro-forge-ddt">Linaro Forge DDT</a><ul> <li class="toctree-l4"><a class="reference internal" href="../software/debugging/index.html#client-setup-and-usage">Client Setup and Usage</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/debugging/index.html#download">Download</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/debugging/index.html#installation">Installation</a></li> <li class="toctree-l4"><a class="reference internal" href="../software/debugging/index.html#configuration">Configuration</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../software/debugging/index.html#gnu-gdb">GNU GDB</a></li> <li class="toctree-l3"><a class="reference internal" href="../software/debugging/index.html#valgrind">Valgrind</a></li> </ul> </li> </ul> </li> </ul> <ul> <li class="toctree-l1"><a class="reference internal" href="../training/index.html">Training</a><ul> <li class="toctree-l2"><a class="reference external" href="https://www.olcf.ornl.gov/for-users/training/training-calendar" target="_blank">OLCF Training Calendar</a></li> <li class="toctree-l2"><a class="reference external" href="https://github.com/olcf-tutorials" target="_blank">OLCF Tutorials</a></li> <li class="toctree-l2"><a class="reference internal" href="../training/training_archive.html">OLCF Training Archive</a></li> <li class="toctree-l2"><a class="reference internal" href="../training/olcf_gpu_hackathons.html">OLCF GPU Hackathons</a></li> <li class="toctree-l2"><a class="reference internal" href="../training/olcf_gpu_hackathons.html#facultyhack">FacultyHack</a></li> <li class="toctree-l2"><a class="reference external" href="https://vimeo.com/channels/olcftraining" target="_blank">OLCF Vimeo Channel</a></li> <li class="toctree-l2"><a class="reference internal" href="../training/index.html#new-user-quick-start">New User Quick Start</a></li> </ul> </li> </ul> <ul> <li class="toctree-l1"><a class="reference internal" href="../quantum/index.html">Quantum</a><ul> <li class="toctree-l2"><a class="reference internal" href="../quantum/quantum_access.html">Quantum Computing User Program (QCUP) Access</a><ul> <li class="toctree-l3"><a class="reference internal" href="../quantum/quantum_access.html#qcup-priorities">QCUP Priorities</a></li> <li class="toctree-l3"><a class="reference internal" href="../quantum/quantum_access.html#project-allocations">Project Allocations</a><ul> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_access.html#what-happens-after-a-project-request-is-approved">What happens after a project request is approved?</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_access.html#project-renewals">Project Renewals</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_access.html#closeout-and-quarterly-reports">Closeout and Quarterly Reports</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../quantum/quantum_access.html#user-accounts">User Accounts</a><ul> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_access.html#checking-the-status-of-your-application">Checking the status of your application</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../quantum/quantum_access.html#accessing-quantum-resources">Accessing Quantum Resources</a><ul> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_access.html#ibm-quantum-computing">IBM Quantum Computing</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_access.html#quantinuum">Quantinuum</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_access.html#ionq">IonQ</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_access.html#iqm">IQM</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../quantum/quantum_access.html#publication-citations">Publication Citations</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../quantum/quantum_systems/index.html">Quantum Systems</a><ul> <li class="toctree-l3"><a class="reference internal" href="../quantum/quantum_systems/ibm_quantum.html">IBM Quantum</a><ul> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/ibm_quantum.html#overview">Overview</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/ibm_quantum.html#connecting">Connecting</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/ibm_quantum.html#running-jobs-queue-policies">Running Jobs & Queue Policies</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/ibm_quantum.html#checking-system-availability-capability">Checking System Availability & Capability</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/ibm_quantum.html#software">Software</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/ibm_quantum.html#additional-resources">Additional Resources</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../quantum/quantum_systems/quantinuum.html">Quantinuum</a><ul> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/quantinuum.html#overview">Overview</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/quantinuum.html#connecting">Connecting</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/quantinuum.html#running-jobs-queue-policies">Running Jobs & Queue Policies</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/quantinuum.html#default-quotas">Default Quotas</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/quantinuum.html#allocations-credit-usage">Allocations & Credit Usage</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/quantinuum.html#software">Software</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../quantum/quantum_systems/ionq.html">IonQ</a><ul> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/ionq.html#overview">Overview</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/ionq.html#ionq-systems">IonQ systems</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/ionq.html#connecting">Connecting</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/ionq.html#running-jobs-queue-policies">Running Jobs & Queue Policies</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/ionq.html#allocations-credit-usage">Allocations & Credit Usage</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/ionq.html#checking-system-availability-capability">Checking System Availability & Capability</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/ionq.html#additional-resources">Additional Resources</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../quantum/quantum_systems/iqm.html">IQM</a><ul> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/iqm.html#overview">Overview</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/iqm.html#connecting">Connecting</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/iqm.html#running-jobs-queue-policies">Running Jobs & Queue Policies</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/iqm.html#checking-system-availability">Checking System Availability</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/iqm.html#software">Software</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_systems/iqm.html#additional-resources">Additional Resources</a></li> </ul> </li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../quantum/quantum_software/index.html">Quantum Software</a><ul> <li class="toctree-l3"><a class="reference internal" href="../quantum/quantum_software/hybrid_hpc.html">Quantum Software on HPC Systems</a><ul> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_software/hybrid_hpc.html#overview">Overview</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_software/hybrid_hpc.html#qiskit">Qiskit</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_software/hybrid_hpc.html#pyquil-forest-sdk-rigetti">PyQuil/Forest SDK (Rigetti)</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_software/hybrid_hpc.html#pennylane">PennyLane</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_software/hybrid_hpc.html#pytket">Pytket</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_software/hybrid_hpc.html#cuda-q">CUDA-Q</a></li> <li class="toctree-l4"><a class="reference internal" href="../quantum/quantum_software/hybrid_hpc.html#batch-jobs">Batch Jobs</a></li> </ul> </li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../quantum/hello_qcup.html">Hello QCUP Scripts</a><ul> <li class="toctree-l3"><a class="reference internal" href="../quantum/hello_qcup.html#overview">Overview</a></li> <li class="toctree-l3"><a class="reference internal" href="../quantum/hello_qcup.html#ibm-quantum">IBM Quantum</a></li> <li class="toctree-l3"><a class="reference internal" href="../quantum/hello_qcup.html#quantinuum">Quantinuum</a></li> <li class="toctree-l3"><a class="reference internal" href="../quantum/hello_qcup.html#ionq">IonQ</a></li> <li class="toctree-l3"><a class="reference internal" href="../quantum/hello_qcup.html#iqm">IQM</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../quantum/quantum_faq.html">Frequently Asked Questions</a><ul> <li class="toctree-l3"><a class="reference internal" href="../quantum/quantum_faq.html#how-do-quantum-computers-differ-from-classical-computers">How do quantum computers differ from classical computers?</a></li> <li class="toctree-l3"><a class="reference internal" href="../quantum/quantum_faq.html#what-is-a-qubit">What is a qubit?</a></li> <li class="toctree-l3"><a class="reference internal" href="../quantum/quantum_faq.html#how-do-i-access-the-olcf-quantum-computing-resources">How do I access the OLCF quantum computing resources?</a></li> <li class="toctree-l3"><a class="reference internal" href="../quantum/quantum_faq.html#what-happens-after-i-apply-for-access-to-qcup">What happens after I apply for access to QCUP?</a></li> <li class="toctree-l3"><a class="reference internal" href="../quantum/quantum_faq.html#i-formerly-had-access-to-quantum-resources-but-my-backends-lattices-etc-have-disappeared-what-do-i-do">I formerly had access to quantum resources, but my backends/lattices/etc. have disappeared, what do I do?</a></li> <li class="toctree-l3"><a class="reference internal" href="../quantum/quantum_faq.html#i-applied-to-a-quantum-computing-resource-via-the-vendor-website-but-dont-have-access-what-do-i-do">I applied to a quantum computing resource via the vendor website, but don’t have access; what do I do?</a></li> </ul> </li> </ul> </li> </ul> <ul> <li class="toctree-l1"><a class="reference internal" href="../spi/index.html">Scalable Protected Infrastructure (SPI)</a><ul> <li class="toctree-l2"><a class="reference internal" href="../spi/index.html#what-is-spi">What is SPI</a><ul> <li class="toctree-l3"><a class="reference internal" href="../spi/index.html#what-is-citadel">What is Citadel</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../spi/index.html#new-user-quickstart">New User QuickStart</a><ul> <li class="toctree-l3"><a class="reference internal" href="../spi/index.html#notable-differences">Notable Differences</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../spi/index.html#allocations-and-user-accounts">Allocations and User Accounts</a><ul> <li class="toctree-l3"><a class="reference internal" href="../spi/index.html#allocations-projects">Allocations (Projects)</a><ul> <li class="toctree-l4"><a class="reference internal" href="../spi/index.html#requesting-a-new-allocation-project">Requesting a New Allocation (Project)</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../spi/index.html#user-accounts">User Accounts</a><ul> <li class="toctree-l4"><a class="reference internal" href="../spi/index.html#requesting-a-new-user-account">Requesting a New User Account</a></li> </ul> </li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../spi/index.html#available-resources">Available Resources</a><ul> <li class="toctree-l3"><a class="reference internal" href="../spi/index.html#compute">Compute</a></li> <li class="toctree-l3"><a class="reference internal" href="../spi/index.html#file-systems">File Systems</a></li> <li class="toctree-l3"><a class="reference internal" href="../spi/index.html#data-transfer">Data Transfer</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../spi/index.html#ip-whitelisting">IP Whitelisting</a><ul> <li class="toctree-l3"><a class="reference internal" href="../spi/index.html#whitelisting-an-ip-or-range">Whitelisting an IP or range</a></li> <li class="toctree-l3"><a class="reference internal" href="../spi/index.html#finding-your-ip">Finding your IP</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../spi/index.html#citadel">Citadel</a><ul> <li class="toctree-l3"><a class="reference internal" href="../spi/index.html#login-nodes">Login Nodes</a></li> <li class="toctree-l3"><a class="reference internal" href="../spi/index.html#connecting">Connecting</a></li> <li class="toctree-l3"><a class="reference internal" href="../spi/index.html#building-software">Building Software</a><ul> <li class="toctree-l4"><a class="reference internal" href="../spi/index.html#external-repositories">External Repositories</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../spi/index.html#running-batch-jobs">Running Batch Jobs</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../spi/index.html#spi-file-systems">File Systems</a></li> <li class="toctree-l2"><a class="reference internal" href="../spi/index.html#spi-data-transfer">Data Transfer</a></li> </ul> </li> </ul> <ul> <li class="toctree-l1"><a class="reference internal" href="../ace_testbed/index.html">Advanced Computing Ecosystem Testbed (ACE)</a><ul> <li class="toctree-l2"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html">Defiant Quick-Start Guide</a><ul> <li class="toctree-l3"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#system-overview">System Overview</a><ul> <li class="toctree-l4"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#defiant-compute-nodes">Defiant Compute Nodes</a></li> <li class="toctree-l4"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#system-interconnect">System Interconnect</a></li> <li class="toctree-l4"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#file-systems">File Systems</a></li> <li class="toctree-l4"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#gpus">GPUs</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#connecting">Connecting</a></li> <li class="toctree-l3"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#data-and-storage">Data and Storage</a><ul> <li class="toctree-l4"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#nfs-filesystem">NFS Filesystem</a></li> <li class="toctree-l4"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#lustre-filesystem-polis">Lustre Filesystem (Polis)</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#programming-environment">Programming Environment</a><ul> <li class="toctree-l4"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#environment-modules-lmod">Environment Modules (Lmod)</a></li> <li class="toctree-l4"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#compilers">Compilers</a></li> <li class="toctree-l4"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#mpi">MPI</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#compiling">Compiling</a><ul> <li class="toctree-l4"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#id3">MPI</a></li> <li class="toctree-l4"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#openmp">OpenMP</a></li> <li class="toctree-l4"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#openmp-gpu-offload">OpenMP GPU Offload</a></li> <li class="toctree-l4"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#hip">HIP</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#running-jobs">Running Jobs</a><ul> <li class="toctree-l4"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#slurm-workload-manager">Slurm Workload Manager</a></li> <li class="toctree-l4"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#slurm-compute-node-partitions">Slurm Compute Node Partitions</a></li> <li class="toctree-l4"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#process-and-thread-mapping">Process and Thread Mapping</a></li> <li class="toctree-l4"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#nvme-usage">NVMe Usage</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#container-usage">Container Usage</a><ul> <li class="toctree-l4"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#setup-before-building">Setup before Building</a></li> <li class="toctree-l4"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#build-and-run-workflow">Build and Run Workflow</a></li> </ul> </li> <li class="toctree-l3"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#getting-help">Getting Help</a></li> <li class="toctree-l3"><a class="reference internal" href="../ace_testbed/defiant_quick_start_guide.html#known-issues">Known Issues</a></li> </ul> </li> </ul> </li> </ul> <ul> <li class="toctree-l1"><a class="reference internal" href="../contributing/index.html">Contributing to these docs</a><ul> <li class="toctree-l2"><a class="reference internal" href="../contributing/index.html#submitting-suggestions">Submitting suggestions</a></li> <li class="toctree-l2"><a class="reference internal" href="../contributing/index.html#authoring-content">Authoring content</a><ul> <li class="toctree-l3"><a class="reference internal" href="../contributing/index.html#setup-authoring-environment">Setup authoring environment</a></li> <li class="toctree-l3"><a class="reference internal" href="../contributing/index.html#edit-the-docs">Edit the docs</a></li> <li class="toctree-l3"><a class="reference internal" href="../contributing/index.html#resources">Resources</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="../contributing/index.html#github-guidelines">GitHub Guidelines</a></li> </ul> </li> </ul> </div> </div> </nav> <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" style="background: #efefef" > <a href="../index.html">OLCF User Documentation</a> </nav> <div class="wy-nav-content"> <div class="rst-content style-external-links"> <div role="navigation" aria-label="Page navigation"> <ul class="wy-breadcrumbs"> <li><a href="../index.html" class="icon icon-home" aria-label="Home"></a></li> <li class="breadcrumb-item"><a href="index.html">Systems</a></li> <li class="breadcrumb-item active">Spock Quick-Start Guide</li> <li class="wy-breadcrumbs-aside"> <a href="https://github.com/olcf/olcf-user-docs/blob/master/systems/spock_quick_start_guide.rst" class="fa fa-github"> Edit on GitHub</a> </li> </ul> <hr/> </div> <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> <div itemprop="articleBody"> <section id="spock-quick-start-guide"> <h1>Spock Quick-Start Guide<a class="headerlink" href="#spock-quick-start-guide" title="Link to this heading"></a></h1> <div class="admonition warning"> Warning The Spock Early Access System was decommissioned on March 15, 2023. The file systems that were available on Spock are still accessible from the Home server and the Data Transfer Nodes (DTN), so all your data will remain accessible. If you do not have access to other OLCF systems, your project will move to data-only for 30-days. If you have any questions, please contact <a class="reference external" href="mailto:help%40olcf.ornl.gov" target="_blank">help@olcf.ornl.gov</a>. </div> <section id="system-overview"> <h2>System Overview<a class="headerlink" href="#system-overview" title="Link to this heading"></a></h2> Spock is an NCCS moderate-security system that contains similar hardware and software as the upcoming Frontier system. It is used as an early-access testbed for Center for Accelerated Application Readiness (CAAR) and Exascale Computing Project (ECP) teams as well as NCCS staff and our vendor partners. The system has 3 cabinets, each containing 12 compute nodes, for a total of 36 compute nodes. <section id="spock-compute-nodes"> <h3>Spock Compute Nodes<a class="headerlink" href="#spock-compute-nodes" title="Link to this heading"></a></h3> Each Spock compute node consists of [1x] 64-core AMD EPYC 7662 “Rome” CPU (with 2 hardware threads per physical core) with access to 256 GB of DDR4 memory and connected to [4x] AMD MI100 GPUs. The CPU is connected to all GPUs via PCIe Gen4, allowing peak host-to-device (H2D) and device-to-host (D2H) data transfers of 32+32 GB/s. The GPUs are connected in an all-to-all arrangement via Infinity Fabric (xGMI), allowing for a peak device-to-device bandwidth of 46+46 GB/s. Each compute node also has [2x] 3.2 TB NVMe devices (SSDs) with sequential read and write speeds of 6900 MB/s and 4200 MB/s, respectively. <div class="admonition note"> Note The X+X GB/s values for bandwidths above represent bi-directional bandwidths. So, for example, the Infinity Fabric connecting any two GPUs allows peak data transfers of 46 GB/s in both directions simultaneously. </div> <a class="reference internal image-reference" href="../_images/Spock_Node.jpg"><img alt="Spock node architecture diagram" class="align-center" src="../_images/Spock_Node.jpg" style="width: 100%;" /> </a> <div class="admonition note"> Note There are 4 NUMA domains per node, that are defined as follows: <ul class="simple"> <li>NUMA 0: hardware threads 000-015, 064-079 | GPU 0</li> <li>NUMA 1: hardware threads 016-031, 080-095 | GPU 1</li> <li>NUMA 2: hardware threads 032-047, 096-111 | GPU 2</li> <li>NUMA 3: hardware threads 048-063, 112-127 | GPU 3</li> </ul> </div> </section> <section id="system-interconnect"> <h3>System Interconnect<a class="headerlink" href="#system-interconnect" title="Link to this heading"></a></h3> The Spock nodes are connected with Slingshot-10 providing a node injection bandwidth of 12.5 GB/s. </section> <section id="file-systems"> <h3>File Systems<a class="headerlink" href="#file-systems" title="Link to this heading"></a></h3> Spock is connected to an IBM Spectrum Scale™ filesystem providing 250 PB of storage capacity with a peak write speed of 2.5 TB/s. Spock also has access to the center-wide NFS-based filesystem (which provides user and project home areas). While Spock does not have direct access to the center’s Nearline archival storage system (Kronos) - for user and project archival storage - users can log in to the moderate DTNs to move data to/from Kronos or use the “OLCF Kronos” Globus collection. For more information on using Kronos, see the <a class="reference internal" href="../data/index.html#kronos">Kronos Nearline Archival Storage System</a> section. </section> <section id="gpus"> <h3>GPUs<a class="headerlink" href="#gpus" title="Link to this heading"></a></h3> Spock contains a total of 144 AMD MI100 GPUs. The AMD MI100 GPU has a peak performance of up to 11.5 TFLOPS in double-precision for modeling & simulation and up to 184.6 TFLOPS in half-precision for machine learning and data analytics. Each GPU contains 120 compute units (7680 stream processors) and 32 GB of high-bandwidth memory (HBM2) which can be accessed at speeds of up to 1.2 TB/s. </section> </section> <hr class="docutils" /> <section id="connecting"> <h2>Connecting<a class="headerlink" href="#connecting" title="Link to this heading"></a></h2> To connect to Spock, <code class="docutils literal notranslate">ssh</code> to <code class="docutils literal notranslate">spock.olcf.ornl.gov</code>. For example: <div class="highlight-bash notranslate"><div class="highlight"><pre>$ ssh username@spock.olcf.ornl.gov </pre></div> </div> For more information on connecting to OLCF resources, see <a class="reference internal" href="../connecting/index.html#connecting-to-olcf">Connecting for the first time</a>. </section> <hr class="docutils" /> <section id="data-and-storage"> <h2>Data and Storage<a class="headerlink" href="#data-and-storage" title="Link to this heading"></a></h2> For more detailed information about center-wide file systems and data archiving available on Spock, please refer to the pages on <a class="reference internal" href="../data/index.html#data-storage-and-transfers">Data Storage and Transfers</a>, but the two subsections below give a quick overview of NFS and GPFS storage spaces. <section id="nfs"> <h3>NFS<a class="headerlink" href="#nfs" title="Link to this heading"></a></h3> <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Area</th> <th class="head">Path</th> <th class="head">Type</th> <th class="head">Permissions</th> <th class="head">Quota</th> <th class="head">Backups</th> <th class="head">Purged</th> <th class="head">Retention</th> <th class="head">On Compute Nodes</th> </tr> </thead> <tbody> <tr class="row-even"><td>User Home</td> <td><code class="docutils literal notranslate">/ccs/home/[userid]</code></td> <td>NFS</td> <td>User set</td> <td>50 GB</td> <td>Yes</td> <td>No</td> <td>90 days</td> <td>Read-only</td> </tr> <tr class="row-odd"><td>Project Home</td> <td><code class="docutils literal notranslate">/ccs/proj/[projid]</code></td> <td>NFS</td> <td>770</td> <td>50 GB</td> <td>Yes</td> <td>No</td> <td>90 days</td> <td>Read-only</td> </tr> </tbody> </table> </section> <section id="gpfs"> <h3>GPFS<a class="headerlink" href="#gpfs" title="Link to this heading"></a></h3> <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Area</th> <th class="head">Path</th> <th class="head">Type</th> <th class="head">Permissions</th> <th class="head">Quota</th> <th class="head">Backups</th> <th class="head">Purged</th> <th class="head">Retention</th> <th class="head">On Compute Nodes</th> </tr> </thead> <tbody> <tr class="row-even"><td>Member Work</td> <td><code class="docutils literal notranslate">/gpfs/alpine/[projid]/scratch/[userid]</code></td> <td>Spectrum Scale</td> <td>700</td> <td>50 TB</td> <td>No</td> <td>90 days</td> <td>N/A</td> <td>Yes</td> </tr> <tr class="row-odd"><td>Project Work</td> <td><code class="docutils literal notranslate">/gpfs/alpine/[projid]/proj-shared</code></td> <td>Spectrum Scale</td> <td>770</td> <td>50 TB</td> <td>No</td> <td>90 days</td> <td>N/A</td> <td>Yes</td> </tr> <tr class="row-even"><td>World Work</td> <td><code class="docutils literal notranslate">/gpfs/alpine/[projid]/world-shared</code></td> <td>Spectrum Scale</td> <td>775</td> <td>50 TB</td> <td>No</td> <td>90 days</td> <td>N/A</td> <td>Yes</td> </tr> </tbody> </table> </section> </section> <hr class="docutils" /> <section id="programming-environment"> <h2>Programming Environment<a class="headerlink" href="#programming-environment" title="Link to this heading"></a></h2> OLCF provides Spock users many pre-installed software packages and scientific libraries. To facilitate this, environment management tools are used to handle necessary changes to the shell. <section id="environment-modules-lmod"> <h3>Environment Modules (Lmod)<a class="headerlink" href="#environment-modules-lmod" title="Link to this heading"></a></h3> Environment modules are provided through <a class="reference external" href="https://lmod.readthedocs.io/en/latest/" target="_blank">Lmod</a>, a Lua-based module system for dynamically altering shell environments. By managing changes to the shell’s environment variables (such as <code class="docutils literal notranslate">PATH</code>, <code class="docutils literal notranslate">LD_LIBRARY_PATH</code>, and <code class="docutils literal notranslate">PKG_CONFIG_PATH</code>), Lmod allows you to alter the software available in your shell environment without the risk of creating package and version combinations that cannot coexist in a single environment. <section id="general-usage"> <h4>General Usage<a class="headerlink" href="#general-usage" title="Link to this heading"></a></h4> The interface to Lmod is provided by the <code class="docutils literal notranslate">module</code> command: <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Command</th> <th class="head">Description</th> </tr> </thead> <tbody> <tr class="row-even"><td><code class="docutils literal notranslate">module -t list</code></td> <td>Shows a terse list of the currently loaded modules</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">module avail</code></td> <td>Shows a table of the currently available modules</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">module help <modulename></code></td> <td>Shows help information about <code class="docutils literal notranslate"><modulename></code></td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">module show <modulename></code></td> <td>Shows the environment changes made by the <code class="docutils literal notranslate"><modulename></code> modulefile</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">module spider <string></code></td> <td>Searches all possible modules according to <code class="docutils literal notranslate"><string></code></td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">module load <modulename> [...]</code></td> <td>Loads the given <code class="docutils literal notranslate"><modulename></code>(s) into the current environment</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">module use <path></code></td> <td>Adds <code class="docutils literal notranslate"><path></code> to the modulefile search cache and <code class="docutils literal notranslate">MODULESPATH</code></td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">module unuse <path></code></td> <td>Removes <code class="docutils literal notranslate"><path></code> from the modulefile search cache and <code class="docutils literal notranslate">MODULESPATH</code></td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">module purge</code></td> <td>Unloads all modules</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">module reset</code></td> <td>Resets loaded modules to system defaults</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">module update</code></td> <td>Reloads all currently loaded modules</td> </tr> </tbody> </table> </section> <section id="searching-for-modules"> <h4>Searching for Modules<a class="headerlink" href="#searching-for-modules" title="Link to this heading"></a></h4> Modules with dependencies are only available when the underlying dependencies, such as compiler families, are loaded. Thus, module avail will only display modules that are compatible with the current state of the environment. To search the entire hierarchy across all possible dependencies, the <code class="docutils literal notranslate">spider</code> sub-command can be used as summarized in the following table. <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Command</th> <th class="head">Description</th> </tr> </thead> <tbody> <tr class="row-even"><td><code class="docutils literal notranslate">module spider</code></td> <td>Shows the entire possible graph of modules</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">module spider <modulename></code></td> <td>Searches for modules named <code class="docutils literal notranslate"><modulename></code> in the graph of possible modules</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">module spider <modulename>/<version></code></td> <td>Searches for a specific version of <code class="docutils literal notranslate"><modulename></code> in the graph of possible modules</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">module spider <string></code></td> <td>Searches for modulefiles containing <code class="docutils literal notranslate"><string></code></td> </tr> </tbody> </table> </section> </section> <section id="compilers"> <h3>Compilers<a class="headerlink" href="#compilers" title="Link to this heading"></a></h3> Cray, AMD, and GCC compilers are provided through modules on Spock. The Cray and AMD compilers are both based on LLVM/Clang. There are also system/OS versions of both Clang and GCC available in <code class="docutils literal notranslate">/usr/bin</code>. The table below lists details about each of the module-provided compilers. <div class="admonition note"> Note It is highly recommended to use the Cray compiler wrappers (<code class="docutils literal notranslate">cc</code>, <code class="docutils literal notranslate">CC</code>, and <code class="docutils literal notranslate">ftn</code>) whenever possible. See the next section for more details. </div> <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Vendor</th> <th class="head">Programming Environment</th> <th class="head">Compiler Module</th> <th class="head">Language</th> <th class="head">Compiler Wrapper</th> <th class="head">Compiler</th> </tr> </thead> <tbody> <tr class="row-even"><td rowspan="3">Cray</td> <td rowspan="3"><code class="docutils literal notranslate">PrgEnv-cray</code></td> <td rowspan="3"><code class="docutils literal notranslate">cce</code></td> <td>C</td> <td><code class="docutils literal notranslate">cc</code></td> <td><code class="docutils literal notranslate">craycc</code></td> </tr> <tr class="row-odd"><td>C++</td> <td><code class="docutils literal notranslate">CC</code></td> <td><code class="docutils literal notranslate">craycxx</code> or <code class="docutils literal notranslate">crayCC</code></td> </tr> <tr class="row-even"><td>Fortran</td> <td><code class="docutils literal notranslate">ftn</code></td> <td><code class="docutils literal notranslate">crayftn</code></td> </tr> <tr class="row-odd"><td rowspan="3">AMD</td> <td rowspan="3"><code class="docutils literal notranslate">PrgEnv-amd</code></td> <td rowspan="3"><code class="docutils literal notranslate">rocm</code></td> <td>C</td> <td><code class="docutils literal notranslate">cc</code></td> <td><code class="docutils literal notranslate">$ROCM_PATH/llvm/bin/clang</code></td> </tr> <tr class="row-even"><td>C++</td> <td><code class="docutils literal notranslate">CC</code></td> <td><code class="docutils literal notranslate">$ROCM_PATH/llvm/bin/clang++</code></td> </tr> <tr class="row-odd"><td>Fortran</td> <td><code class="docutils literal notranslate">ftn</code></td> <td><code class="docutils literal notranslate">$ROCM_PATH/llvm/bin/flang</code></td> </tr> <tr class="row-even"><td rowspan="3">GCC</td> <td rowspan="3"><code class="docutils literal notranslate">PrgEnv-gnu</code></td> <td rowspan="3"><code class="docutils literal notranslate">gcc</code></td> <td>C</td> <td><code class="docutils literal notranslate">cc</code></td> <td><code class="docutils literal notranslate">$GCC_PATH/bin/gcc</code></td> </tr> <tr class="row-odd"><td>C++</td> <td><code class="docutils literal notranslate">CC</code></td> <td><code class="docutils literal notranslate">$GCC_PATH/bin/g++</code></td> </tr> <tr class="row-even"><td>Fortran</td> <td><code class="docutils literal notranslate">ftn</code></td> <td><code class="docutils literal notranslate">$GCC_PATH/bin/gfortran</code></td> </tr> </tbody> </table> <section id="cray-programming-environment-and-compiler-wrappers"> <h4>Cray Programming Environment and Compiler Wrappers<a class="headerlink" href="#cray-programming-environment-and-compiler-wrappers" title="Link to this heading"></a></h4> Cray provides <code class="docutils literal notranslate">PrgEnv-<compiler></code> modules (e.g., <code class="docutils literal notranslate">PrgEnv-cray</code>) that load compatible components of a specific compiler toolchain. The components include the specified compiler as well as MPI, LibSci, and other libraries. Loading the <code class="docutils literal notranslate">PrgEnv-<compiler></code> modules also defines a set of compiler wrappers for that compiler toolchain that automatically add include paths and link in libraries for Cray software. Compiler wrappers are provided for C (<code class="docutils literal notranslate">cc</code>), C++ (<code class="docutils literal notranslate">CC</code>), and Fortran (<code class="docutils literal notranslate">ftn</code>). <div class="admonition note"> Note Use the <code class="docutils literal notranslate">-craype-verbose</code> flag to display the full include and link information used by the Cray compiler wrappers. This must be called on a file to see the full output (e.g., <code class="docutils literal notranslate">CC -craype-verbose test.cpp</code>). </div> </section> </section> <section id="mpi"> <h3>MPI<a class="headerlink" href="#mpi" title="Link to this heading"></a></h3> The MPI implementation available on Spock is Cray’s MPICH, which is “GPU-aware” so GPU buffers can be passed directly to MPI calls. </section> </section> <hr class="docutils" /> <section id="compiling"> <h2>Compiling<a class="headerlink" href="#compiling" title="Link to this heading"></a></h2> This section covers how to compile for different programming models using the different compilers covered in the previous section. <section id="id3"> <h3>MPI<a class="headerlink" href="#id3" title="Link to this heading"></a></h3> <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Implementation</th> <th class="head">Module</th> <th class="head">Compiler</th> <th class="head">Header Files & Linking</th> </tr> </thead> <tbody> <tr class="row-even"><td rowspan="2">Cray MPICH</td> <td rowspan="2"><code class="docutils literal notranslate">cray-mpich</code></td> <td><code class="docutils literal notranslate">cc</code>, <code class="docutils literal notranslate">CC</code>, <code class="docutils literal notranslate">ftn</code> (Cray compiler wrappers)</td> <td>MPI header files and linking is built into the Cray compiler wrappers</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">hipcc</code></td> <td><div class="line-block"> <div class="line"><code class="docutils literal notranslate">-L$(MPICH_DIR)/lib -lmpi</code></div> <div class="line"><code class="docutils literal notranslate">-I$(MPICH_DIR)/include</code></div> </div> </td> </tr> </tbody> </table> <section id="gpu-aware-mpi"> <h4>GPU-Aware MPI<a class="headerlink" href="#gpu-aware-mpi" title="Link to this heading"></a></h4> To use GPU-aware Cray MPICH, there are currently some extra steps needed in addition to the table above, which depend on the compiler that is used. <section id="compiling-with-the-cray-compiler-wrappers-cc-or-cc"> <h5>1. Compiling with the Cray compiler wrappers, <code class="docutils literal notranslate">cc</code> or <code class="docutils literal notranslate">CC</code><a class="headerlink" href="#compiling-with-the-cray-compiler-wrappers-cc-or-cc" title="Link to this heading"></a></h5> To use GPU-aware Cray MPICH with the Cray compiler wrappers, users must load specific modules, set some environment variables, and include appropriate headers and libraries. The following modules and environment variables must be set: <div class="admonition note"> Note Setting <code class="docutils literal notranslate">MPICH_SMP_SINGLE_COPY_MODE=CMA</code> is required as a temporary workaround due to a <a class="reference external" href="https://docs.olcf.ornl.gov/systems/spock_quick_start_guide.html#olcfdev-138-gpu-aware-cray-mpich-can-cause-hang-in-some-codes" target="_blank">known issue</a>. Users should make a note of where they set this environment variable (if e.g., set in a script) since it should NOT be set once the known issue has been resolved. </div> <div class="highlight-bash notranslate"><div class="highlight"><pre>module load craype-accel-amd-gfx908 module load PrgEnv-cray module load rocm ## These must be set before running export MPIR_CVAR_GPU_EAGER_DEVICE_MEM=0 export MPICH_GPU_SUPPORT_ENABLED=1 export MPICH_SMP_SINGLE_COPY_MODE=CMA </pre></div> </div> In addition, the following header files and libraries must be included: <div class="highlight-bash notranslate"><div class="highlight"><pre>-I${ROCM_PATH}/include -L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64 </pre></div> </div> where the include path implies that <code class="docutils literal notranslate">#include <hip/hip_runtime.h></code> is included in the source file. </section> <section id="compiling-with-hipcc"> <h5>2. Compiling with <code class="docutils literal notranslate">hipcc</code><a class="headerlink" href="#compiling-with-hipcc" title="Link to this heading"></a></h5> To use GPU-aware Cray MPICH with <code class="docutils literal notranslate">hipcc</code>, users must load specific modules, set some environment variables, and include appropriate headers and libraries. The following modules and environment variables must be set: <div class="highlight-bash notranslate"><div class="highlight"><pre>module load craype-accel-amd-gfx908 module load PrgEnv-cray module load rocm ## These must be set before running export MPIR_CVAR_GPU_EAGER_DEVICE_MEM=0 export MPICH_GPU_SUPPORT_ENABLED=1 export MPICH_SMP_SINGLE_COPY_MODE=CMA </pre></div> </div> In addition, the following header files and libraries must be included: <div class="highlight-bash notranslate"><div class="highlight"><pre>-I${MPICH_DIR}/include -L${MPICH_DIR}/lib -lmpi -L${CRAY_MPICH_ROOTDIR}/gtl/lib -lmpi_gtl_hsa </pre></div> </div> </section> </section> </section> <section id="openmp"> <h3>OpenMP<a class="headerlink" href="#openmp" title="Link to this heading"></a></h3> This section shows how to compile with OpenMP using the different compilers covered above. <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Vendor</th> <th class="head">Module</th> <th class="head">Language</th> <th class="head">Compiler</th> <th class="head">OpenMP flag (CPU thread)</th> </tr> </thead> <tbody> <tr class="row-even"><td rowspan="2">Cray</td> <td rowspan="2"><code class="docutils literal notranslate">cce</code></td> <td>C, C++</td> <td><div class="line-block"> <div class="line"><code class="docutils literal notranslate">cc</code></div> <div class="line"><code class="docutils literal notranslate">CC</code></div> </div> </td> <td><code class="docutils literal notranslate">-fopenmp</code></td> </tr> <tr class="row-odd"><td>Fortran</td> <td><code class="docutils literal notranslate">ftn</code></td> <td><div class="line-block"> <div class="line"><code class="docutils literal notranslate">-homp</code></div> <div class="line"><code class="docutils literal notranslate">-fopenmp</code> (alias)</div> </div> </td> </tr> <tr class="row-even"><td>AMD</td> <td><code class="docutils literal notranslate">rocm</code></td> <td><div class="line-block"> <div class="line">C</div> <div class="line">C++</div> <div class="line">Fortran</div> </div> </td> <td><div class="line-block"> <div class="line"><code class="docutils literal notranslate">$ROCM_PATH/llvm/bin/clang</code></div> <div class="line"><code class="docutils literal notranslate">$ROCM_PATH/llvm/bin/clang++</code></div> <div class="line"><code class="docutils literal notranslate">ROCM_PATH/llvm/bin/flang</code></div> </div> </td> <td><code class="docutils literal notranslate">-fopenmp</code></td> </tr> <tr class="row-odd"><td>GCC</td> <td><code class="docutils literal notranslate">gcc</code></td> <td><div class="line-block"> <div class="line">C</div> <div class="line">C++</div> <div class="line">Fortran</div> </div> </td> <td><div class="line-block"> <div class="line"><code class="docutils literal notranslate">$GCC_PATH/bin/gcc</code></div> <div class="line"><code class="docutils literal notranslate">$GCC_PATH/bin/g++</code></div> <div class="line"><code class="docutils literal notranslate">$GCC_PATH/bin/gfortran</code></div> </div> </td> <td><code class="docutils literal notranslate">-fopenmp</code></td> </tr> </tbody> </table> </section> <section id="openmp-gpu-offload"> <h3>OpenMP GPU Offload<a class="headerlink" href="#openmp-gpu-offload" title="Link to this heading"></a></h3> This section shows how to compile with OpenMP Offload using the different compilers covered above. <div class="admonition note"> Note Make sure the <code class="docutils literal notranslate">craype-accel-amd-gfx908</code> module is loaded when using OpenMP offload. </div> <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Vendor</th> <th class="head">Module</th> <th class="head">Language</th> <th class="head">Compiler</th> <th class="head">OpenMP flag (GPU)</th> </tr> </thead> <tbody> <tr class="row-even"><td rowspan="2">Cray</td> <td rowspan="2"><code class="docutils literal notranslate">cce</code></td> <td>C C++</td> <td><div class="line-block"> <div class="line"><code class="docutils literal notranslate">cc</code></div> <div class="line"><code class="docutils literal notranslate">CC</code></div> </div> </td> <td><code class="docutils literal notranslate">-fopenmp</code></td> </tr> <tr class="row-odd"><td>Fortran</td> <td><code class="docutils literal notranslate">ftn</code></td> <td><div class="line-block"> <div class="line"><code class="docutils literal notranslate">-homp</code></div> <div class="line"><code class="docutils literal notranslate">-fopenmp</code> (alias)</div> </div> </td> </tr> <tr class="row-even"><td>AMD</td> <td><code class="docutils literal notranslate">rocm</code></td> <td><div class="line-block"> <div class="line">C</div> <div class="line">C++</div> <div class="line">Fortran</div> </div> </td> <td><div class="line-block"> <div class="line"><code class="docutils literal notranslate">$ROCM_PATH/llvm/bin/clang</code></div> <div class="line"><code class="docutils literal notranslate">$ROCM_PATH/llvm/bin/clang++</code></div> <div class="line"><code class="docutils literal notranslate">ROCM_PATH/llvm/bin/flang</code></div> <div class="line"><code class="docutils literal notranslate">hipcc</code></div> </div> </td> <td><div class="line-block"> <div class="line"><code class="docutils literal notranslate">-fopenmp -target x86_64-pc-linux-gnu \</code></div> <div class="line"><code class="docutils literal notranslate">-fopenmp-targets=amdgcn-amd-amdhsa   \</code></div> <div class="line"><code class="docutils literal notranslate">-Xopenmp-target=amdgcn-amd-amdhsa    \</code></div> <div class="line"><code class="docutils literal notranslate">-march=gfx908</code></div> </div> </td> </tr> </tbody> </table> </section> <section id="hip"> <h3>HIP<a class="headerlink" href="#hip" title="Link to this heading"></a></h3> This section shows how to compile HIP codes using the Cray compiler wrappers and <code class="docutils literal notranslate">hipcc</code> compiler driver. <div class="admonition note"> Note Make sure the <code class="docutils literal notranslate">craype-accel-amd-gfx908</code> module is loaded when using HIP. </div> <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Compiler</th> <th class="head">Compile/Link Flags, Header Files, and Libraries</th> </tr> </thead> <tbody> <tr class="row-even"><td><code class="docutils literal notranslate">CC</code></td> <td><div class="line-block"> <div class="line"><code class="docutils literal notranslate">CFLAGS = -std=c++11 -D__HIP_ROCclr__ -D__HIP_ARCH_GFX908__=1 --rocm-path=${ROCM_PATH} --offload-arch=gfx908 -x hip</code></div> <div class="line"><code class="docutils literal notranslate">LFLAGS = -std=c++11 -D__HIP_ROCclr__ --rocm-path=${ROCM_PATH}</code></div> <div class="line"><code class="docutils literal notranslate">-I${HIP_PATH}/include</code></div> <div class="line"><code class="docutils literal notranslate">-L${HIP_PATH}/lib -lamdhip64</code></div> </div> </td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">hipcc</code></td> <td><div class="line-block"> <div class="line">Can be used directly to compile HIP source files.</div> <div class="line">To see what is being invoked within this compiler driver, issue the command, <code class="docutils literal notranslate">hipcc --verbose</code></div> </div> </td> </tr> </tbody> </table> </section> </section> <hr class="docutils" /> <section id="running-jobs"> <h2>Running Jobs<a class="headerlink" href="#running-jobs" title="Link to this heading"></a></h2> This section describes how to run programs on the Spock compute nodes, including a brief overview of Slurm and also how to map processes and threads to CPU cores and GPUs. <section id="slurm-workload-manager"> <h3>Slurm Workload Manager<a class="headerlink" href="#slurm-workload-manager" title="Link to this heading"></a></h3> <a class="reference external" href="https://slurm.schedmd.com/" target="_blank">Slurm</a> is the workload manager used to interact with the compute nodes on Spock. In the following subsections, the most commonly used Slurm commands for submitting, running, and monitoring jobs will be covered, but users are encouraged to visit the official documentation and man pages for more information. <section id="batch-scheduler-and-job-launcher"> <h4>Batch Scheduler and Job Launcher<a class="headerlink" href="#batch-scheduler-and-job-launcher" title="Link to this heading"></a></h4> Slurm provides 3 ways of submitting and launching jobs on Spock’s compute nodes: batch scripts, interactive, and single-command. The Slurm commands associated with these methods are shown in the table below and examples of their use can be found in the related subsections. <table class="docutils align-default"> <tbody> <tr class="row-odd"><td><code class="docutils literal notranslate">sbatch</code></td> <td><div class="line-block"> <div class="line">Used to submit a batch script to allocate a Slurm job allocation. The script contains options preceded with <code class="docutils literal notranslate">#SBATCH</code>.</div> <div class="line">(see Batch Scripts section below)</div> </div> </td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">salloc</code></td> <td><div class="line-block"> <div class="line">Used to allocate an interactive Slurm job allocation, where one or more job steps (i.e., <code class="docutils literal notranslate">srun</code> commands) can then be launched on the allocated resources (i.e., nodes).</div> <div class="line">(see Interactive Jobs section below)</div> </div> </td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">srun</code></td> <td><div class="line-block"> <div class="line">Used to run a parallel job (job step) on the resources allocated with sbatch or <code class="docutils literal notranslate">salloc</code>.</div> <div class="line">If necessary, srun will first create a resource allocation in which to run the parallel job(s).</div> <div class="line">(see Single Command section below)</div> </div> </td> </tr> </tbody> </table> <section id="batch-scripts"> <h5>Batch Scripts<a class="headerlink" href="#batch-scripts" title="Link to this heading"></a></h5> A batch script can be used to submit a job to run on the compute nodes at a later time. In this case, stdout and stderr will be written to a file(s) that can be opened after the job completes. Here is an example of a simple batch script: <div class="highlight-bash notranslate"><div class="highlight"><pre>1#!/bin/bash 2#SBATCH -A <project_id> 3#SBATCH -J <job_name> 4#SBATCH -o %x-%j.out 5#SBATCH -t 00:05:00 6#SBATCH -p <partition> 7#SBATCH -N 2 8 9srun -n4 --ntasks-per-node=2 ./a.out </pre></div> </div> The Slurm submission options are preceded by <code class="docutils literal notranslate">#SBATCH</code>, making them appear as comments to a shell (since comments begin with <code class="docutils literal notranslate">#</code>). Slurm will look for submission options from the first line through the first non-comment line. Options encountered after the first non-comment line will not be read by Slurm. In the example script, the lines are: <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Line</th> <th class="head">Description</th> </tr> </thead> <tbody> <tr class="row-even"><td>1</td> <td>[Optional] shell interpreter line</td> </tr> <tr class="row-odd"><td>2</td> <td>OLCF project to charge</td> </tr> <tr class="row-even"><td>3</td> <td>Job name</td> </tr> <tr class="row-odd"><td>4</td> <td>stdout file name ( <code class="docutils literal notranslate">%x</code> represents job name, <code class="docutils literal notranslate">%j</code> represents job id)</td> </tr> <tr class="row-even"><td>5</td> <td>Walltime requested (<code class="docutils literal notranslate">HH:MM:SS</code>)</td> </tr> <tr class="row-odd"><td>6</td> <td>Batch queue</td> </tr> <tr class="row-even"><td>7</td> <td>Number of compute nodes requested</td> </tr> <tr class="row-odd"><td>8</td> <td>Blank line</td> </tr> <tr class="row-even"><td>9</td> <td><code class="docutils literal notranslate">srun</code> command to launch parallel job (requesting 4 processes - 2 per node)</td> </tr> </tbody> </table> </section> <section id="interactive-jobs"> <h5>Interactive Jobs<a class="headerlink" href="#interactive-jobs" title="Link to this heading"></a></h5> To request an interactive job where multiple job steps (i.e., multiple srun commands) can be launched on the allocated compute node(s), the <code class="docutils literal notranslate">salloc</code> command can be used: <div class="highlight-bash notranslate"><div class="highlight"><pre>$ salloc -A <project_id> -J <job_name> -t 00:05:00 -p <partition> -N 2 salloc: Granted job allocation 4258 salloc: Waiting for resource configuration salloc: Nodes spock[10-11] are ready for job $ srun -n 4 --ntasks-per-node=2 ./a.out <output printed to terminal> $ srun -n 2 --ntasks-per-node=1 ./a.out <output printed to terminal> </pre></div> </div> Here, <code class="docutils literal notranslate">salloc</code> is used to request an allocation of 2 MI100 compute nodes for 5 minutes. Once the resources become available, the user is granted access to the compute nodes (<code class="docutils literal notranslate">spock10</code> and <code class="docutils literal notranslate">spock11</code> in this case) and can launch job steps on them using srun. </section> <section id="single-command-non-interactive"> <h5>Single Command (non-interactive)<a class="headerlink" href="#single-command-non-interactive" title="Link to this heading"></a></h5> <div class="highlight-bash notranslate"><div class="highlight"><pre>$ srun -A <project_id> -t 00:05:00 -p <partition> -N 2 -n 4 --ntasks-per-node=2 ./a.out <output printed to terminal> </pre></div> </div> The job name and output options have been removed since stdout/stderr are typically desired in the terminal window in this usage mode. </section> </section> <section id="common-slurm-submission-options"> <h4>Common Slurm Submission Options<a class="headerlink" href="#common-slurm-submission-options" title="Link to this heading"></a></h4> The table below summarizes commonly-used Slurm job submission options: <table class="docutils align-default"> <tbody> <tr class="row-odd"><td><code class="docutils literal notranslate">A <project_id></code></td> <td>Project ID to charge</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">-J <job_name></code></td> <td>Name of job</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">-p <partition></code></td> <td>Partition / batch queue</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">-t <time></code></td> <td>Wall clock time <<code class="docutils literal notranslate">HH:MM:SS</code>></td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">-N <number_of_nodes></code></td> <td>Number of compute nodes</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">-o <file_name></code></td> <td>Standard output file name</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">-e <file_name></code></td> <td>Standard error file name</td> </tr> </tbody> </table> For more information about these and/or other options, please see the <code class="docutils literal notranslate">sbatch</code> man page. </section> <section id="other-common-slurm-commands"> <h4>Other Common Slurm Commands<a class="headerlink" href="#other-common-slurm-commands" title="Link to this heading"></a></h4> The table below summarizes commonly-used Slurm commands: <table class="docutils align-default"> <tbody> <tr class="row-odd"><td><code class="docutils literal notranslate">sinfo</code></td> <td><div class="line-block"> <div class="line">Used to view partition and node information.</div> <div class="line">E.g., to view user-defined details about the caar queue:</div> <div class="line"><code class="docutils literal notranslate">sinfo -p caar -o "%15N %10D %10P %10a %10c %10z"</code></div> </div> </td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">squeue</code></td> <td><div class="line-block"> <div class="line">Used to view job and job step information for jobs in the scheduling queue.</div> <div class="line">E.g., to see all jobs from a specific user:</div> <div class="line"><code class="docutils literal notranslate">squeue -l -u <user_id></code></div> </div> </td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">sacct</code></td> <td><div class="line-block"> <div class="line">Used to view accounting data for jobs and job steps in the job accounting log (currently in the queue or recently completed).</div> <div class="line">E.g., to see a list of specified information about all jobs submitted/run by a users since 1 PM on January 4, 2021:</div> <div class="line"><code class="docutils literal notranslate">sacct -u <username> -S 2021-01-04T13:00:00 -o "jobid%5,jobname%25,user%15,nodelist%20" -X</code></div> </div> </td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">scancel</code></td> <td><div class="line-block"> <div class="line">Used to signal or cancel jobs or job steps.</div> <div class="line">E.g., to cancel a job:</div> <div class="line"><code class="docutils literal notranslate">scancel <jobid></code></div> </div> </td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">scontrol</code></td> <td><div class="line-block"> <div class="line">Used to view or modify job configuration.</div> <div class="line">E.g., to place a job on hold:</div> <div class="line"><code class="docutils literal notranslate">scontrol hold <jobid></code></div> </div> </td> </tr> </tbody> </table> </section> </section> <hr class="docutils" /> <section id="slurm-compute-node-partitions"> <h3>Slurm Compute Node Partitions<a class="headerlink" href="#slurm-compute-node-partitions" title="Link to this heading"></a></h3> Spock’s compute nodes are separated into 2 Slurm partitions (queues): 1 for CAAR projects and 1 for ECP projects. Please see the tables below for details. <div class="admonition note"> Note If CAAR or ECP teams require a temporary exception to this policy, please email <a class="reference external" href="mailto:help%40olcf.ornl.gov" target="_blank">help@olcf.ornl.gov</a> with your request and it will be given to the OLCF Resource Utilization Council (RUC) for review. </div> <section id="caar-partition"> <h4>CAAR Partition<a class="headerlink" href="#caar-partition" title="Link to this heading"></a></h4> The CAAR partition consists of 24 total compute nodes. On a per-project basis, each user can have 1 running and 1 eligible job at a time, with no limit on the number of jobs submitted. <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Number of Nodes</th> <th class="head">Max Walltime</th> </tr> </thead> <tbody> <tr class="row-even"><td>1 - 4</td> <td>3 hours</td> </tr> <tr class="row-odd"><td>5 - 16</td> <td>1 hour</td> </tr> </tbody> </table> </section> <section id="ecp-partition"> <h4>ECP Partition<a class="headerlink" href="#ecp-partition" title="Link to this heading"></a></h4> The ECP partition consists of 12 total compute nodes. On a per-project basis, each user can have 1 running and 1 eligible job at a time, with up to 5 jobs submitted. <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Number of Nodes</th> <th class="head">Max Walltime</th> </tr> </thead> <tbody> <tr class="row-even"><td>1 - 4</td> <td>3 hours</td> </tr> </tbody> </table> </section> </section> <section id="process-and-thread-mapping"> <h3>Process and Thread Mapping<a class="headerlink" href="#process-and-thread-mapping" title="Link to this heading"></a></h3> This section describes how to map processes (e.g., MPI ranks) and process threads (e.g., OpenMP threads) to the CPUs and GPUs on Spock. The <a class="reference internal" href="#spock-compute-nodes">Spock Compute Nodes</a> diagram will be helpful when reading this section to understand which hardware threads your processes and threads run on. <section id="cpu-mapping"> <h4>CPU Mapping<a class="headerlink" href="#cpu-mapping" title="Link to this heading"></a></h4> In this sub-section, a simple MPI+OpenMP “Hello, World” program (<a class="reference external" href="https://code.ornl.gov/olcf/hello_mpi_omp" target="_blank">hello_mpi_omp</a>) will be used to clarify the mappings. Slurm’s <a class="reference internal" href="#interactive-spock">Interactive Jobs</a> method was used to request an allocation of 1 compute node for these examples: <code class="docutils literal notranslate">salloc -A <project_id> -t 30 -p <parition> -N 1</code> The <code class="docutils literal notranslate">srun</code> options used in this section are (see <code class="docutils literal notranslate">man srun</code> for more information): <table class="docutils align-default"> <tbody> <tr class="row-odd"><td><code class="docutils literal notranslate">-c, --cpus-per-task=<ncpus></code></td> <td><div class="line-block"> <div class="line">Request that <code class="docutils literal notranslate">ncpus</code> be allocated per process (default is 1).</div> <div class="line">(<code class="docutils literal notranslate">ncpus</code> refers to hardware threads)</div> </div> </td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">--threads-per-core=<threads></code></td> <td><div class="line-block"> <div class="line">In task layout, use the specified maximum number of threads per core</div> <div class="line">(default is 1; there are 2 hardware threads per physical CPU core).</div> </div> </td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">--cpu-bind=threads</code></td> <td><div class="line-block"> <div class="line">Bind tasks to CPUs.</div> <div class="line"><code class="docutils literal notranslate">threads</code> - Automatically generate masks binding tasks to threads.</div> <div class="line">(Although this option is not explicitly used in these examples, it is the default CPU binding.)</div> </div> </td> </tr> </tbody> </table> <div class="admonition note"> Note In the <code class="docutils literal notranslate">srun</code> man page (and so the table above), threads refers to hardware threads. </div> <section id="mpi-ranks-each-with-2-openmp-threads"> <h5>2 MPI ranks - each with 2 OpenMP threads<a class="headerlink" href="#mpi-ranks-each-with-2-openmp-threads" title="Link to this heading"></a></h5> In this example, the intent is to launch 2 MPI ranks, each of which spawn 2 OpenMP threads, and have all of the 4 OpenMP threads run on different physical CPU cores. First (INCORRECT) attempt To set the number of OpenMP threads spawned per MPI rank, the <code class="docutils literal notranslate">OMP_NUM_THREADS</code> environment variable can be used. To set the number of MPI ranks launched, the <code class="docutils literal notranslate">srun</code> flag <code class="docutils literal notranslate">-n</code> can be used. <div class="highlight-bash notranslate"><div class="highlight"><pre>$ export OMP_NUM_THREADS=2 $ srun -n2 ./hello_mpi_omp | sort WARNING: Requested total thread count and/or thread affinity may result in oversubscription of available CPU resources! Performance may be degraded. Explicitly set OMP_WAIT_POLICY=PASSIVE or ACTIVE to suppress this message. Set CRAY_OMP_CHECK_AFFINITY=TRUE to print detailed thread-affinity messages. WARNING: Requested total thread count and/or thread affinity may result in oversubscription of available CPU resources! Performance may be degraded. Explicitly set OMP_WAIT_POLICY=PASSIVE or ACTIVE to suppress this message. Set CRAY_OMP_CHECK_AFFINITY=TRUE to print detailed thread-affinity messages. MPI 000 - OMP 000 - HWT 000 - Node spock01 MPI 000 - OMP 001 - HWT 000 - Node spock01 MPI 001 - OMP 000 - HWT 016 - Node spock01 MPI 001 - OMP 001 - HWT 016 - Node spock01 </pre></div> </div> The first thing to notice here is the <code class="docutils literal notranslate">WARNING</code> about oversubscribing the available CPU cores. Also, the output shows each MPI rank did spawn 2 OpenMP threads, but both OpenMP threads ran on the same hardware thread (for a given MPI rank). This was not the intended behavior; each OpenMP thread was meant to run on its own physical CPU core. Second (CORRECT) attempt By default, each MPI rank is allocated only 1 hardware thread, so both OpenMP threads only have that 1 hardware thread to run on - hence the WARNING and undesired behavior. In order for each OpenMP thread to run on its own physical CPU core, each MPI rank should be given 2 hardware thread (<code class="docutils literal notranslate">-c 2</code>) - since, by default, only 1 hardware thread per physical CPU core is enabled (this would need to be <code class="docutils literal notranslate">-c 4</code> if <code class="docutils literal notranslate">--threads-per-core=2</code> instead of the default of <code class="docutils literal notranslate">1</code>. The OpenMP threads will be mapped to unique physical CPU cores unless there are not enough physical CPU cores available, in which case the remaining OpenMP threads will share hardware threads and a WARNING will be issued as shown in the previous example. <div class="highlight-bash notranslate"><div class="highlight"><pre>$ export OMP_NUM_THREADS=2 $ srun -n2 -c2 ./hello_mpi_omp | sort MPI 000 - OMP 000 - HWT 000 - Node spock13 MPI 000 - OMP 001 - HWT 001 - Node spock13 MPI 001 - OMP 000 - HWT 016 - Node spock13 MPI 001 - OMP 001 - HWT 017 - Node spock13 </pre></div> </div> Now the output shows that each OpenMP thread ran on (one of the hardware threads of) its own physical CPU cores. More specifically (see the Spock Compute Node diagram), OpenMP thread 000 of MPI rank 000 ran on hardware thread 000 (i.e., physical CPU core 00), OpenMP thread 001 of MPI rank 000 ran on hardware thread 001 (i.e., physical CPU core 01), OpenMP thread 000 of MPI rank 001 ran on hardware thread 016 (i.e., physical CPU core 16), and OpenMP thread 001 of MPI rank 001 ran on hardware thread 017 (i.e., physical CPU core 17) - as expected. <div class="admonition note"> Note There are many different ways users might choose to perform these mappings, so users are encouraged to clone the <code class="docutils literal notranslate">hello_mpi_omp</code> program and test whether or not processes and threads are running where intended. </div> </section> </section> <section id="gpu-mapping"> <h4>GPU Mapping<a class="headerlink" href="#gpu-mapping" title="Link to this heading"></a></h4> In this sub-section, an MPI+OpenMP+HIP “Hello, World” program (<a class="reference external" href="https://code.ornl.gov/olcf/hello_jobstep" target="_blank">hello_jobstep</a>) will be used to clarify the GPU mappings. Again, Slurm’s <a class="reference internal" href="#interactive-spock">Interactive Jobs</a> method was used to request an allocation of 2 compute node for these examples: <code class="docutils literal notranslate">salloc -A <project_id> -t 30 -p <parition> -N 2</code>. The CPU mapping part of this example is very similar to the example used above in the CPU Mapping sub-section, so the focus here will be on the GPU mapping part. The following <code class="docutils literal notranslate">srun</code> options will be used in the examples below. See <code class="docutils literal notranslate">man srun</code> for a complete list of options and more information. <table class="docutils align-default"> <tbody> <tr class="row-odd"><td><code class="docutils literal notranslate">--gpus-per-task</code></td> <td>Specify the number of GPUs required for the job on each task to be spawned in the job’s resource allocation.</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">--gpu-bind=closest</code></td> <td>Binds each task to the GPU which is on the same NUMA domain as the CPU core the MPI rank is running on.</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">--gpu-bind=map_gpu:<list></code></td> <td>Bind tasks to specific GPUs by setting GPU masks on tasks (or ranks) as specified where <code class="docutils literal notranslate"><list></code> is <code class="docutils literal notranslate"><gpu_id_for_task_0>,<gpu_id_for_task_1>,...</code>. If the number of tasks (or ranks) exceeds the number of elements in this list, elements in the list will be reused as needed starting from the beginning of the list. To simplify support for large task counts, the lists may follow a map with an asterisk and repetition count. (For example <code class="docutils literal notranslate">map_gpu:0*4,1*4</code>)</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">--ntasks-per-gpu=<ntasks></code></td> <td>Request that there are ntasks tasks invoked for every GPU.</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">--distribution=<value>[:<value>][:<value>]</code></td> <td>Specifies the distribution of MPI ranks across compute nodes, sockets (NUMA domains on Spock), and cores, respectively. The default values are <code class="docutils literal notranslate">block:cyclic:cyclic</code></td> </tr> </tbody> </table> <div class="admonition note"> Note In general, GPU mapping can be accomplished in different ways. For example, an application might map MPI ranks to GPUs programmatically within the code using, say, <code class="docutils literal notranslate">hipSetDevice</code>. In this case, since all GPUs on a node are available to all MPI ranks on that node by default, there might not be a need to map to GPUs using Slurm (just do it in the code). However, in another application, there might be a reason to make only a subset of GPUs available to the MPI ranks on a node. It is this latter case that the following examples refer to. </div> <section id="mapping-1-task-per-gpu"> <h5>Mapping 1 task per GPU<a class="headerlink" href="#mapping-1-task-per-gpu" title="Link to this heading"></a></h5> In the following examples, each MPI rank (and its OpenMP threads) will be mapped to a single GPU. Example 1: 4 MPI ranks - each with 2 OpenMP threads and 1 GPU (single-node) This example launches 4 MPI ranks (<code class="docutils literal notranslate">-n4</code>), each with 2 physical CPU cores (<code class="docutils literal notranslate">-c2</code>) to launch 2 OpenMP threads (<code class="docutils literal notranslate">OMP_NUM_THREADS=2</code>) on. In addition, each MPI rank (and its 2 OpenMP threads) should have access to only 1 GPU. To accomplish the GPU mapping, two new <code class="docutils literal notranslate">srun</code> options will be used: <ul class="simple"> <li><code class="docutils literal notranslate">--gpus-per-task</code> specifies the number of GPUs required for the job on each task</li> <li><code class="docutils literal notranslate">--gpu-bind=closest</code> binds each task to the GPU which is closest.</li> </ul> <div class="admonition note"> Note To further clarify, <code class="docutils literal notranslate">--gpus-per-task</code> does not actually bind GPUs to MPI ranks. It allocates GPUs to the job step. The <code class="docutils literal notranslate">--gpu-bind=closest</code> is what actually maps a specific GPU to each rank; namely, the “closest” one, which is the GPU on the same NUMA domain as the CPU core the MPI rank is running on (see the <a class="reference internal" href="#spock-compute-nodes">Spock Compute Nodes</a> section). </div> <div class="admonition note"> Note Without these additional flags, all MPI ranks would have access to all GPUs (which is the default behavior). </div> <div class="highlight-bash notranslate"><div class="highlight"><pre>$ export OMP_NUM_THREADS=2 $ srun -N1 -n4 -c2 --gpus-per-task=1 --gpu-bind=closest ./hello_jobstep | sort MPI 000 - OMP 000 - HWT 000 - Node spock13 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 000 - OMP 001 - HWT 001 - Node spock13 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 001 - OMP 000 - HWT 016 - Node spock13 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 001 - OMP 001 - HWT 017 - Node spock13 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 002 - OMP 000 - HWT 032 - Node spock13 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 002 - OMP 001 - HWT 033 - Node spock13 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 003 - OMP 000 - HWT 048 - Node spock13 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 MPI 003 - OMP 001 - HWT 049 - Node spock13 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 </pre></div> </div> The output contains different IDs associated with the GPUs so it is important to first describe these IDs before moving on. <code class="docutils literal notranslate">GPU_ID</code> is the node-level (or global) GPU ID, which is labeled as one might expect from looking at a node diagram: 0, 1, 2, 3. <code class="docutils literal notranslate">RT_GPU_ID</code> is the HIP runtime GPU ID, which can be thought of as each MPI rank’s local GPU ID numbering (with zero-based indexing). So in the output above, each MPI rank has access to 1 unique GPU - where MPI 000 has access to GPU 0, MPI 001 has access to GPU 1, etc., but all MPI ranks show a HIP runtime GPU ID of 0. The reason is that each MPI rank only “sees” one GPU and so the HIP runtime labels it as “0”, even though it might be global GPU ID 0, 1, 2, or 3. The GPU’s bus ID is included to definitively show that different GPUs are being used. Here is a summary of the different GPU IDs reported by the example program: <ul class="simple"> <li><code class="docutils literal notranslate">GPU_ID</code> is the node-level (or global) GPU ID read from <code class="docutils literal notranslate">ROCR_VISIBLE_DEVICES</code>. If this environment variable is not set (either by the user or by Slurm), the value of <code class="docutils literal notranslate">GPU_ID</code> will be set to <code class="docutils literal notranslate">N/A</code>.</li> <li><code class="docutils literal notranslate">RT_GPU_ID</code> is the HIP runtime GPU ID (as reported from, say <code class="docutils literal notranslate">hipGetDevice</code>).</li> <li><code class="docutils literal notranslate">Bus_ID</code> is the physical bus ID associated with the GPUs. Comparing the bus IDs is meant to definitively show that different GPUs are being used.</li> </ul> So the job step (i.e., <code class="docutils literal notranslate">srun</code> command) used above gave the desired output. Each MPI rank spawned 2 OpenMP threads and had access to a unique GPU. The <code class="docutils literal notranslate">--gpus-per-task=1</code> allocated 1 GPU for each MPI rank and the <code class="docutils literal notranslate">--gpu-bind=closest</code> ensured that the closest GPU to each rank was the one used. Example 2: 8 MPI ranks - each with 2 OpenMP threads and 1 GPU (multi-node) This example will extend Example 1 to run on 2 nodes. As the output shows, it is a very straightforward exercise of changing the number of nodes to 2 (<code class="docutils literal notranslate">-N2</code>) and the number of MPI ranks to 8 (<code class="docutils literal notranslate">-n8</code>). <div class="highlight-bash notranslate"><div class="highlight"><pre>$ export OMP_NUM_THREADS=2 $ srun -N2 -n8 -c2 --gpus-per-task=1 --gpu-bind=closest ./hello_jobstep | sort MPI 000 - OMP 000 - HWT 000 - Node spock13 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 000 - OMP 001 - HWT 001 - Node spock13 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 001 - OMP 000 - HWT 016 - Node spock13 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 001 - OMP 001 - HWT 017 - Node spock13 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 002 - OMP 000 - HWT 032 - Node spock13 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 002 - OMP 001 - HWT 033 - Node spock13 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 003 - OMP 000 - HWT 048 - Node spock13 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 MPI 003 - OMP 001 - HWT 049 - Node spock13 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 MPI 004 - OMP 000 - HWT 000 - Node spock14 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 004 - OMP 001 - HWT 001 - Node spock14 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 005 - OMP 000 - HWT 016 - Node spock14 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 005 - OMP 001 - HWT 017 - Node spock14 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 006 - OMP 000 - HWT 032 - Node spock14 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 006 - OMP 001 - HWT 033 - Node spock14 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 007 - OMP 000 - HWT 048 - Node spock14 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 MPI 007 - OMP 001 - HWT 049 - Node spock14 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 </pre></div> </div> Example 3: 4 MPI ranks - each with 2 OpenMP threads and 1 *specific* GPU (single-node) This example will be very similar to Example 1, but instead of using <code class="docutils literal notranslate">--gpu-bind=closest</code> to map each MPI rank to the closest GPU, <code class="docutils literal notranslate">--gpu-bind=map_gpu</code> will be used to map each MPI rank to a specific GPU. The <code class="docutils literal notranslate">map_gpu</code> option takes a comma-separated list of GPU IDs to specify how the MPI ranks are mapped to GPUs, where the form of the comma-separated list is <code class="docutils literal notranslate"><gpu_id_for_task_0>, <gpu_id_for_task_1>,...</code>. <div class="highlight-bash notranslate"><div class="highlight"><pre>$ export OMP_NUM_THREADS=2 $ srun -N1 -n4 -c2 --gpus-per-task=1 --gpu-bind=map_gpu:0,1,2,3 ./hello_jobstep | sort MPI 000 - OMP 000 - HWT 000 - Node spock13 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 000 - OMP 001 - HWT 001 - Node spock13 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 001 - OMP 000 - HWT 016 - Node spock13 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 001 - OMP 001 - HWT 017 - Node spock13 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 002 - OMP 000 - HWT 032 - Node spock13 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 002 - OMP 001 - HWT 033 - Node spock13 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 003 - OMP 000 - HWT 048 - Node spock13 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 MPI 003 - OMP 001 - HWT 049 - Node spock13 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 </pre></div> </div> Here, the output is the same as the results from Example 1. This is because the 4 GPU IDs in the comma-separated list happen to specify the GPUs within the same NUMA domains that the MPI ranks are in. So MPI 000 is mapped to GPU 0, MPI 001 is mapped to GPU 1, etc. While this level of control over mapping MPI ranks to GPUs might be useful for some applications, it is always important to consider the implication of the mapping. For example, if the order of the GPU IDs in the <code class="docutils literal notranslate">map_gpu</code> option is reversed, the MPI ranks and the GPUs they are mapped to would be in different NUMA domains, which could potentially lead to poorer performance. <div class="highlight-bash notranslate"><div class="highlight"><pre>$ export OMP_NUM_THREADS=2 $ srun -N1 -n4 -c2 --gpus-per-task=1 --gpu-bind=map_gpu:3,2,1,0 ./hello_jobstep | sort MPI 000 - OMP 000 - HWT 000 - Node spock13 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 MPI 000 - OMP 001 - HWT 001 - Node spock13 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 MPI 001 - OMP 000 - HWT 016 - Node spock13 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 001 - OMP 001 - HWT 017 - Node spock13 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 002 - OMP 000 - HWT 032 - Node spock13 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 002 - OMP 001 - HWT 033 - Node spock13 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 003 - OMP 000 - HWT 048 - Node spock13 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 003 - OMP 001 - HWT 049 - Node spock13 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 </pre></div> </div> Here, notice that MPI 000 now maps to GPU 3, MPI 001 maps to GPU 2, etc., so the MPI ranks are not in the same NUMA domains as the GPUs they are mapped to. <div class="admonition note"> Note Again, this particular example would NOT be a very good mapping of GPUs to MPI ranks though. E.g., notice that MPI rank 000 is running on NUMA node 0, whereas GPU 3 is on NUMA node 3. Again, see the <a class="reference internal" href="#spock-compute-nodes">Spock Compute Nodes</a> section for NUMA descriptions. </div> Example 4: 8 MPI ranks - each with 2 OpenMP threads and 1 *specific* GPU (multi-node) Extending Examples 2 and 3 to run on 2 nodes is also a straightforward exercise by changing the number of nodes to 2 (<code class="docutils literal notranslate">-N2</code>) and the number of MPI ranks to 8 (<code class="docutils literal notranslate">-n8</code>). <div class="highlight-bash notranslate"><div class="highlight"><pre>$ export OMP_NUM_THREADS=2 $ srun -N2 -n8 -c2 --gpus-per-task=1 --gpu-bind=map_gpu:0,1,2,3 ./hello_jobstep | sort MPI 000 - OMP 000 - HWT 000 - Node spock13 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 000 - OMP 001 - HWT 001 - Node spock13 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 001 - OMP 000 - HWT 016 - Node spock13 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 001 - OMP 001 - HWT 017 - Node spock13 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 002 - OMP 000 - HWT 032 - Node spock13 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 002 - OMP 001 - HWT 033 - Node spock13 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 003 - OMP 000 - HWT 048 - Node spock13 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 MPI 003 - OMP 001 - HWT 049 - Node spock13 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 MPI 004 - OMP 000 - HWT 000 - Node spock14 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 004 - OMP 001 - HWT 001 - Node spock14 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 005 - OMP 000 - HWT 016 - Node spock14 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 005 - OMP 001 - HWT 017 - Node spock14 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 006 - OMP 000 - HWT 032 - Node spock14 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 006 - OMP 001 - HWT 033 - Node spock14 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 007 - OMP 000 - HWT 048 - Node spock14 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 MPI 007 - OMP 001 - HWT 049 - Node spock14 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 </pre></div> </div> </section> <section id="mapping-multiple-mpi-ranks-to-a-single-gpu"> <h5>Mapping multiple MPI ranks to a single GPU<a class="headerlink" href="#mapping-multiple-mpi-ranks-to-a-single-gpu" title="Link to this heading"></a></h5> In the following examples, 2 MPI ranks will be mapped to 1 GPU. For the sake of brevity, <code class="docutils literal notranslate">OMP_NUM_THREADS</code> will be set to <code class="docutils literal notranslate">1</code>, so <code class="docutils literal notranslate">-c1</code> will be used unless otherwise specified. <div class="admonition note"> Note On AMD’s MI100 GPUs, multi-process service (MPS) is not needed since multiple MPI ranks per GPU is supported natively. </div> Example 5: 8 MPI ranks - where 2 ranks share a GPU (round-robin, single-node) This example launches 8 MPI ranks (<code class="docutils literal notranslate">-n8</code>), each with 1 physical CPU core (<code class="docutils literal notranslate">-c1</code>) to launch 1 OpenMP thread (<code class="docutils literal notranslate">OMP_NUM_THREADS=1</code>) on. The MPI ranks will be assigned to GPUs in a round-robin fashion so that each of the 4 GPUs on the node are shared by 2 MPI ranks. To accomplish this GPU mapping, a new <code class="docutils literal notranslate">srun</code> option will be used: <ul class="simple"> <li><code class="docutils literal notranslate">--ntasks-per-gpu</code> specifies the number of MPI ranks that will share access to a GPU.</li> </ul> <div class="highlight-bash notranslate"><div class="highlight"><pre>$ export OMP_NUM_THREADS=1 $ srun -N1 -n8 -c1 --ntasks-per-gpu=2 --gpu-bind=closest ./hello_jobstep | sort MPI 000 - OMP 000 - HWT 000 - Node spock13 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 001 - OMP 000 - HWT 016 - Node spock13 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 002 - OMP 000 - HWT 032 - Node spock13 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 003 - OMP 000 - HWT 048 - Node spock13 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 MPI 004 - OMP 000 - HWT 001 - Node spock13 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 005 - OMP 000 - HWT 017 - Node spock13 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 006 - OMP 000 - HWT 033 - Node spock13 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 007 - OMP 000 - HWT 049 - Node spock13 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 </pre></div> </div> The output shows the round-robin (<code class="docutils literal notranslate">cyclic</code>) distribution of MPI ranks to GPUs. In fact, it is a round-robin distribution of MPI ranks to NUMA domains (the default distribution). The GPU mapping is a consequence of where the MPI ranks are distributed; <code class="docutils literal notranslate">--gpu-bind=closest</code> simply maps the GPU in a NUMA domain to the MPI ranks in the same NUMA domain. Example 6: 16 MPI ranks - where 2 ranks share a GPU (round-robin, multi-node) This example is an extension of Example 5 to run on 2 nodes. <div class="admonition warning"> Warning This example requires a workaround to run as expected. <code class="docutils literal notranslate">--ntasks-per-gpu=2</code> does not force MPI ranks 008-015 to run on the second node, so the number of physical CPU cores per MPI rank is increased to 8 (<code class="docutils literal notranslate">-c8</code>) to force the desired behavior due to the constraint of the number of physical CPU cores (64) on a node. </div> <div class="highlight-bash notranslate"><div class="highlight"><pre>$ export OMP_NUM_THREADS=1 $ srun -N2 -n16 -c8 --ntasks-per-gpu=2 --gpu-bind=closest ./hello_jobstep | sort MPI 000 - OMP 000 - HWT 005 - Node spock13 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 001 - OMP 000 - HWT 018 - Node spock13 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 002 - OMP 000 - HWT 032 - Node spock13 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 003 - OMP 000 - HWT 050 - Node spock13 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 MPI 004 - OMP 000 - HWT 010 - Node spock13 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 005 - OMP 000 - HWT 026 - Node spock13 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 006 - OMP 000 - HWT 040 - Node spock13 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 007 - OMP 000 - HWT 059 - Node spock13 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 MPI 008 - OMP 000 - HWT 003 - Node spock14 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 009 - OMP 000 - HWT 016 - Node spock14 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 010 - OMP 000 - HWT 032 - Node spock14 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 011 - OMP 000 - HWT 048 - Node spock14 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 MPI 012 - OMP 000 - HWT 008 - Node spock14 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 013 - OMP 000 - HWT 024 - Node spock14 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 014 - OMP 000 - HWT 042 - Node spock14 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 015 - OMP 000 - HWT 056 - Node spock14 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 </pre></div> </div> Example 7: 8 MPI ranks - where 2 ranks share a GPU (packed, single-node) This example launches 8 MPI ranks (<code class="docutils literal notranslate">-n8</code>), each with 8 physical CPU cores (<code class="docutils literal notranslate">-c8</code>) to launch 1 OpenMP thread (<code class="docutils literal notranslate">OMP_NUM_THREADS=1</code>) on. The MPI ranks will be assigned to GPUs in a packed fashion so that each of the 4 GPUs on the node are shared by 2 MPI ranks. Similar to Example 5, <code class="docutils literal notranslate">-ntasks-per-gpu=2</code> will be used, but a new <code class="docutils literal notranslate">srun</code> flag will be used to change the default round-robin (<code class="docutils literal notranslate">cyclic</code>) distribution of MPI ranks across NUMA domains: <ul class="simple"> <li><code class="docutils literal notranslate">--distribution=<value>:[<value>]:[<value>]</code> specifies the distribution of MPI ranks across compute nodes, sockets (NUMA domains on Spock), and cores, respectively. The default values are <code class="docutils literal notranslate">block:cyclic:cyclic</code>, which is where the <code class="docutils literal notranslate">cyclic</code> assignment comes from in the previous examples.</li> </ul> <div class="admonition note"> Note In the job step for this example, <code class="docutils literal notranslate">--distribution=*:block</code> is used, where <code class="docutils literal notranslate">*</code> represents the default value of <code class="docutils literal notranslate">block</code> for the distribution of MPI ranks across compute nodes and the distribution of MPI ranks across NUMA domains has been changed to <code class="docutils literal notranslate">block</code> from its default value of <code class="docutils literal notranslate">cyclic</code>. </div> <div class="admonition note"> Note Because the distribution across NUMA domains has been changed to a “packed” (<code class="docutils literal notranslate">block</code>) configuration, caution must be taken to ensure MPI ranks end up in the NUMA domains where the GPUs they intend to be mapped to are located. To accomplish this, the number of physical CPU cores assigned to an MPI rank was increased - in this case to 8. Doing so ensures that only 2 MPI ranks can fit into a single NUMA domain. If the value of <code class="docutils literal notranslate">-c</code> was left at <code class="docutils literal notranslate">1</code>, all 8 MPI ranks would be “packed” into the first NUMA domain, where the “closest” GPU would be GPU 0 - the only GPU in that NUMA domain. Notice that this is not a workaround like in Example 6, but a requirement due to the <code class="docutils literal notranslate">block</code> distribution of MPI ranks across NUMA domains. </div> <div class="highlight-bash notranslate"><div class="highlight"><pre>$ export OMP_NUM_THREADS=1 $ srun -N1 -n8 -c8 --ntasks-per-gpu=2 --gpu-bind=closest --distribution=*:block ./hello_jobstep | sort MPI 000 - OMP 000 - HWT 001 - Node spock13 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 001 - OMP 000 - HWT 008 - Node spock13 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 002 - OMP 000 - HWT 016 - Node spock13 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 003 - OMP 000 - HWT 024 - Node spock13 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 004 - OMP 000 - HWT 035 - Node spock13 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 005 - OMP 000 - HWT 043 - Node spock13 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 006 - OMP 000 - HWT 049 - Node spock13 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 MPI 007 - OMP 000 - HWT 057 - Node spock13 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 </pre></div> </div> The overall effect of using <code class="docutils literal notranslate">--distribution=*:block</code> and increasing the number of physical CPU cores available to each MPI rank is to place the first two MPI ranks in NUMA 0 with GPU 0, the next two MPI ranks in NUMA 1 with GPU 1, and so on. Example 8: 16 MPI ranks - where 2 ranks share a GPU (packed, multi-node) This example is an extension of Example 7 to use 2 compute nodes. With the appropriate changes put in place in Example 7, it is a straightforward exercise to change to using 2 nodes (<code class="docutils literal notranslate">-N2</code>) and 16 MPI ranks (<code class="docutils literal notranslate">-n16</code>). <div class="highlight-bash notranslate"><div class="highlight"><pre>$ export OMP_NUM_THREADS=1 $ srun -N2 -n16 -c8 --ntasks-per-gpu=2 --gpu-bind=closest --distribution=*:block ./hello_jobstep | sort MPI 000 - OMP 000 - HWT 005 - Node spock13 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 001 - OMP 000 - HWT 008 - Node spock13 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 002 - OMP 000 - HWT 017 - Node spock13 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 003 - OMP 000 - HWT 026 - Node spock13 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 004 - OMP 000 - HWT 033 - Node spock13 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 005 - OMP 000 - HWT 041 - Node spock13 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 006 - OMP 000 - HWT 048 - Node spock13 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 MPI 007 - OMP 000 - HWT 057 - Node spock13 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 MPI 008 - OMP 000 - HWT 002 - Node spock14 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 009 - OMP 000 - HWT 011 - Node spock14 - RT_GPU_ID 0 - GPU_ID 0 - Bus_ID c9 MPI 010 - OMP 000 - HWT 016 - Node spock14 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 011 - OMP 000 - HWT 026 - Node spock14 - RT_GPU_ID 0 - GPU_ID 1 - Bus_ID 87 MPI 012 - OMP 000 - HWT 033 - Node spock14 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 013 - OMP 000 - HWT 041 - Node spock14 - RT_GPU_ID 0 - GPU_ID 2 - Bus_ID 48 MPI 014 - OMP 000 - HWT 054 - Node spock14 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 MPI 015 - OMP 000 - HWT 063 - Node spock14 - RT_GPU_ID 0 - GPU_ID 3 - Bus_ID 09 </pre></div> </div> </section> <section id="multiple-gpus-per-mpi-rank"> <h5>Multiple GPUs per MPI rank<a class="headerlink" href="#multiple-gpus-per-mpi-rank" title="Link to this heading"></a></h5> As mentioned previously, all GPUs are accessible by all MPI ranks by default, so it is possible to programatically map any combination of MPI ranks to GPUs. However, there is currently no way to use Slurm to map multiple GPUs to a single MPI rank. If this functionality is needed for an application, please submit a ticket by emailing <a class="reference external" href="mailto:help%40olcf.ornl.gov" target="_blank">help@olcf.ornl.gov</a>. <div class="admonition note"> Note There are many different ways users might choose to perform these mappings, so users are encouraged to clone the <code class="docutils literal notranslate">hello_jobstep</code> program and test whether or not processes and threads are running where intended. </div> </section> </section> </section> <section id="nvme-usage"> <h3>NVMe Usage<a class="headerlink" href="#nvme-usage" title="Link to this heading"></a></h3> Each Spock compute node has [2x] 3.2 TB NVMe devices (SSDs) with a peak sequential performance of 6900 MB/s (read) and 4200 MB/s (write). To use the NVMe, users must request access during job allocation using the <code class="docutils literal notranslate">-C nvme</code> option to <code class="docutils literal notranslate">sbatch</code>, <code class="docutils literal notranslate">salloc</code>, or <code class="docutils literal notranslate">srun</code>. Once the devices have been granted to a job, users can access them at <code class="docutils literal notranslate">/mnt/bb/<userid></code>. Users are responsible for moving data to/from the NVMe before/after their jobs. Here is a simple example script: <div class="highlight-bash notranslate"><div class="highlight"><pre>#!/bin/bash #SBATCH -A <projid> #SBATCH -J nvme_test #SBATCH -o %x-%j.out #SBATCH -t 00:05:00 #SBATCH -p batch #SBATCH -N 1 #SBATCH -C nvme date # Change directory to user scratch space (GPFS) cd /gpfs/alpine/<projid>/scratch/<userid> echo " " echo "*****ORIGINAL FILE*****" cat test.txt echo "***********************" # Move file from GPFS to SSD mv test.txt /mnt/bb/<userid> # Edit file from compute node srun -n1 hostname >> /mnt/bb/<userid>/test.txt # Move file from SSD back to GPFS mv /mnt/bb/<userid>/test.txt . echo " " echo "*****UPDATED FILE******" cat test.txt echo "***********************" </pre></div> </div> And here is the output from the script: <div class="highlight-bash notranslate"><div class="highlight"><pre>$ cat nvme_test-<jobid>.out Mon May 17 12:28:18 EDT 2021 *****ORIGINAL FILE***** This is my file. There are many like it but this one is mine. *********************** *****UPDATED FILE****** This is my file. There are many like it but this one is mine. spock25 *********************** </pre></div> </div> </section> </section> <hr class="docutils" /> <section id="getting-help"> <h2>Getting Help<a class="headerlink" href="#getting-help" title="Link to this heading"></a></h2> If you have problems or need helping running on Spock, please submit a ticket by emailing <a class="reference external" href="mailto:help%40olcf.ornl.gov" target="_blank">help@olcf.ornl.gov</a>. </section> </section> </div> </div> <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer"> <a href="odo_user_guide.html" class="btn btn-neutral float-left" title="Odo" accesskey="p" rel="prev"> Previous</a> <a href="crusher_quick_start_guide.html" class="btn btn-neutral float-right" title="Crusher Quick-Start Guide" accesskey="n" rel="next">Next </a> </div> <hr/> <div role="contentinfo"> © Copyright 2025, OLCF. </div> Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. </footer> </div> </div> </section> </div> <script> jQuery(function () { SphinxRtdTheme.Navigation.enable(true); }); </script> </body> </html>