CINXE.COM

<!DOCTYPE html> <html class="writer-html5" lang="en" > <head> <meta charset="utf-8" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> <title>Process reference — Nextflow documentation</title> <link rel="stylesheet" href="../_static/pygments.css" type="text/css" /> <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> <link rel="stylesheet" href="../_static/copybutton.css" type="text/css" /> <link rel="stylesheet" href="../_static/tabs.css" type="text/css" /> <link rel="stylesheet" href="../_static/theme.css" type="text/css" /> <link rel="shortcut icon" href="../_static/favicon.ico"/>  <script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script> <script src="../_static/jquery.js"></script> <script src="../_static/underscore.js"></script> <script src="../_static/_sphinx_javascript_frameworks_compat.js"></script> <script src="../_static/doctools.js"></script> <script src="../_static/sphinx_highlight.js"></script> <script src="../_static/clipboard.min.js"></script> <script src="../_static/copybutton.js"></script> <script src="../_static/js/theme.js"></script> <link rel="index" title="Index" href="../genindex.html" /> <link rel="search" title="Search" href="../search.html" /> <link rel="next" title="Channel factories" href="channel.html" /> <link rel="prev" title="Standard library" href="stdlib.html" />  <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-TNCXSWG');</script>  </head> <body class="wy-body-for-nav"> <div class="wy-grid-for-nav"> <nav data-toggle="wy-nav-shift" class="wy-nav-side"> <div class="wy-side-scroll"> <div class="wy-side-nav-search" > <a href="../index.html"> <img src="../_static/nextflow-logo.png" class="logo" alt="Logo"/> </a> <div role="search"> <form id="rtd-search-form" class="wy-form" action="../search.html" method="get"> <input type="text" name="q" placeholder="Search docs" /> <input type="hidden" name="check_keywords" value="yes" /> <input type="hidden" name="area" value="default" /> </form> </div> <div class="sidebar_version"> Version dev <samp>(da5f75928)</samp> <select onchange="window.location = this.value;"> <option value="/docs/stable/index.html" >Stable</option> <option value="/docs/edge/index.html" >Edge</option> <option value="/docs/latest/index.html" selected>Latest</option> </select> </select> </div> </div> <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu"> Get started <ul> <li class="toctree-l1"><a class="reference internal" href="../overview.html">Overview</a></li> <li class="toctree-l1"><a class="reference internal" href="../install.html">Installation</a></li> <li class="toctree-l1"><a class="reference internal" href="../developer-env.html">Environment setup</a></li> <li class="toctree-l1"><a class="reference internal" href="../your-first-script.html">Your first script</a></li> </ul> Running pipelines <ul> <li class="toctree-l1"><a class="reference internal" href="../cli.html">Command line</a></li> <li class="toctree-l1"><a class="reference internal" href="../config.html">Configuration</a></li> <li class="toctree-l1"><a class="reference internal" href="../executor.html">Executors</a></li> <li class="toctree-l1"><a class="reference internal" href="../cache-and-resume.html">Caching and resuming</a></li> <li class="toctree-l1"><a class="reference internal" href="../reports.html">Reports</a></li> <li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li> </ul> Developing pipelines <ul> <li class="toctree-l1"><a class="reference internal" href="../script.html">Scripts</a></li> <li class="toctree-l1"><a class="reference internal" href="../working-with-files.html">Working with files</a></li> <li class="toctree-l1"><a class="reference internal" href="../process.html">Processes</a></li> <li class="toctree-l1"><a class="reference internal" href="../channel.html">Channels</a></li> <li class="toctree-l1"><a class="reference internal" href="../workflow.html">Workflows</a></li> <li class="toctree-l1"><a class="reference internal" href="../module.html">Modules</a></li> <li class="toctree-l1"><a class="reference internal" href="../notifications.html">Notifications</a></li> <li class="toctree-l1"><a class="reference internal" href="../secrets.html">Secrets</a></li> <li class="toctree-l1"><a class="reference internal" href="../sharing.html">Sharing pipelines</a></li> <li class="toctree-l1"><a class="reference internal" href="../vscode.html">VS Code integration</a></li> </ul> Software dependencies <ul> <li class="toctree-l1"><a class="reference internal" href="../git.html">Git</a></li> <li class="toctree-l1"><a class="reference internal" href="../container.html">Containers</a></li> <li class="toctree-l1"><a class="reference internal" href="../conda.html">Conda environments</a></li> <li class="toctree-l1"><a class="reference internal" href="../spack.html">Spack environments</a></li> <li class="toctree-l1"><a class="reference internal" href="../wave.html">Wave containers</a></li> </ul> Compute & storage <ul> <li class="toctree-l1"><a class="reference internal" href="../aws.html">Amazon Web Services</a></li> <li class="toctree-l1"><a class="reference internal" href="../amazons3.html">Amazon S3</a></li> <li class="toctree-l1"><a class="reference internal" href="../azure.html">Azure</a></li> <li class="toctree-l1"><a class="reference internal" href="../google.html">Google Cloud</a></li> <li class="toctree-l1"><a class="reference internal" href="../kubernetes.html">Kubernetes</a></li> <li class="toctree-l1"><a class="reference internal" href="../fusion.html">Fusion file system</a></li> </ul> Reference <ul class="current"> <li class="toctree-l1"><a class="reference internal" href="syntax.html">Syntax</a></li> <li class="toctree-l1"><a class="reference internal" href="cli.html">CLI reference</a></li> <li class="toctree-l1"><a class="reference internal" href="config.html">Configuration options</a></li> <li class="toctree-l1"><a class="reference internal" href="env-vars.html">Environment variables</a></li> <li class="toctree-l1"><a class="reference internal" href="feature-flags.html">Feature flags</a></li> <li class="toctree-l1"><a class="reference internal" href="stdlib.html">Standard library</a></li> <li class="toctree-l1 current"><a class="current reference internal" href="#">Process reference</a><ul> <li class="toctree-l2"><a class="reference internal" href="#task-properties">Task properties</a></li> <li class="toctree-l2"><a class="reference internal" href="#inputs">Inputs</a></li> <li class="toctree-l2"><a class="reference internal" href="#outputs">Outputs</a><ul> <li class="toctree-l3"><a class="reference internal" href="#generic-options">Generic options</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="#directives">Directives</a><ul> <li class="toctree-l3"><a class="reference internal" href="#accelerator">accelerator</a></li> <li class="toctree-l3"><a class="reference internal" href="#afterscript">afterScript</a></li> <li class="toctree-l3"><a class="reference internal" href="#arch">arch</a></li> <li class="toctree-l3"><a class="reference internal" href="#array">array</a></li> <li class="toctree-l3"><a class="reference internal" href="#beforescript">beforeScript</a></li> <li class="toctree-l3"><a class="reference internal" href="#cache">cache</a></li> <li class="toctree-l3"><a class="reference internal" href="#clusteroptions">clusterOptions</a></li> <li class="toctree-l3"><a class="reference internal" href="#conda">conda</a></li> <li class="toctree-l3"><a class="reference internal" href="#container">container</a></li> <li class="toctree-l3"><a class="reference internal" href="#containeroptions">containerOptions</a></li> <li class="toctree-l3"><a class="reference internal" href="#cpus">cpus</a></li> <li class="toctree-l3"><a class="reference internal" href="#debug">debug</a></li> <li class="toctree-l3"><a class="reference internal" href="#disk">disk</a></li> <li class="toctree-l3"><a class="reference internal" href="#echo">echo</a></li> <li class="toctree-l3"><a class="reference internal" href="#errorstrategy">errorStrategy</a></li> <li class="toctree-l3"><a class="reference internal" href="#executor">executor</a></li> <li class="toctree-l3"><a class="reference internal" href="#ext">ext</a></li> <li class="toctree-l3"><a class="reference internal" href="#fair">fair</a></li> <li class="toctree-l3"><a class="reference internal" href="#label">label</a></li> <li class="toctree-l3"><a class="reference internal" href="#machinetype">machineType</a></li> <li class="toctree-l3"><a class="reference internal" href="#maxsubmitawait">maxSubmitAwait</a></li> <li class="toctree-l3"><a class="reference internal" href="#maxerrors">maxErrors</a></li> <li class="toctree-l3"><a class="reference internal" href="#maxforks">maxForks</a></li> <li class="toctree-l3"><a class="reference internal" href="#maxretries">maxRetries</a></li> <li class="toctree-l3"><a class="reference internal" href="#memory">memory</a></li> <li class="toctree-l3"><a class="reference internal" href="#module">module</a></li> <li class="toctree-l3"><a class="reference internal" href="#penv">penv</a></li> <li class="toctree-l3"><a class="reference internal" href="#pod">pod</a></li> <li class="toctree-l3"><a class="reference internal" href="#publishdir">publishDir</a></li> <li class="toctree-l3"><a class="reference internal" href="#queue">queue</a></li> <li class="toctree-l3"><a class="reference internal" href="#resourcelabels">resourceLabels</a></li> <li class="toctree-l3"><a class="reference internal" href="#resourcelimits">resourceLimits</a></li> <li class="toctree-l3"><a class="reference internal" href="#scratch">scratch</a></li> <li class="toctree-l3"><a class="reference internal" href="#shell">shell</a></li> <li class="toctree-l3"><a class="reference internal" href="#spack">spack</a></li> <li class="toctree-l3"><a class="reference internal" href="#stageinmode">stageInMode</a></li> <li class="toctree-l3"><a class="reference internal" href="#stageoutmode">stageOutMode</a></li> <li class="toctree-l3"><a class="reference internal" href="#storedir">storeDir</a></li> <li class="toctree-l3"><a class="reference internal" href="#tag">tag</a></li> <li class="toctree-l3"><a class="reference internal" href="#time">time</a></li> </ul> </li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="channel.html">Channel factories</a></li> <li class="toctree-l1"><a class="reference internal" href="operator.html">Operators</a></li> </ul> Updates <ul> <li class="toctree-l1"><a class="reference internal" href="../updating-nextflow.html">Updating Nextflow</a></li> <li class="toctree-l1"><a class="reference internal" href="../updating-syntax.html">Updating Nextflow syntax</a></li> <li class="toctree-l1"><a class="reference internal" href="../dsl1.html">Migrating from DSL1</a></li> </ul> Contributing <ul> <li class="toctree-l1"><a class="reference internal" href="../developer/index.html">Overview</a></li> <li class="toctree-l1"><a class="reference internal" href="../developer/diagram.html">Workflow Diagram</a></li> <li class="toctree-l1"><a class="reference internal" href="../developer/packages.html">Packages</a></li> <li class="toctree-l1"><a class="reference internal" href="../developer/plugins.html">Plugins</a></li> </ul> Tutorials <ul> <li class="toctree-l1"><a class="reference internal" href="../flux.html">Using Nextflow with Flux</a></li> <li class="toctree-l1"><a class="reference internal" href="../metrics.html">Understanding task resource metrics</a></li> </ul> </div> <div class="nav-footer-logo"> <a href="https://seqera.io/" target="_blank" title="Developed by Seqera Labs"> Nextflow is developed by: <img src="../_static/seqera-logo.svg" alt="Seqera Labs"> </a> </div> </div> </nav> <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" > <a href="../index.html">Nextflow</a> </nav> <div class="wy-nav-content"> <div class="rst-content"> <div role="navigation" aria-label="Page navigation"> <ul class="wy-breadcrumbs"> <li><a href="../index.html" class="icon icon-home"></a></li> <li class="breadcrumb-item active">Process reference</li> <li class="wy-breadcrumbs-aside"> <a href="https://github.com/nextflow-io/nextflow/blob/master/docs/reference/process.md" class="fa fa-github"> Edit on GitHub</a> </li> </ul> <hr/> </div> <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> <div itemprop="articleBody"> <section class="tex2jax_ignore mathjax_ignore" id="process-reference"> <h1>Process reference<a class="headerlink" href="#process-reference" title="Permalink to this heading"></a></h1> This page lists the task properties, input/output methods, and directives available in <a class="reference internal" href="../process.html#process-page">process</a> definitions. <section id="task-properties"> <h2>Task properties<a class="headerlink" href="#task-properties" title="Permalink to this heading"></a></h2> The following task properties are defined in the process body: <dl class="simple myst"> <dt><code class="docutils literal notranslate">task.attempt</code></dt><dd>The current task attempt. </dd> <dt><code class="docutils literal notranslate">task.exitStatus</code></dt><dd>The exit code of the task script. Only applicable for processes with a <code class="docutils literal notranslate">script:</code> or <code class="docutils literal notranslate">shell:</code> block. </dd> <dd>Since the exit code is only available after the task has been executed, it can only be used by certain process directives such as <a class="reference internal" href="#errorstrategy">errorStrategy</a>. </dd> <dt><code class="docutils literal notranslate">task.hash</code></dt><dd>Available only in <code class="docutils literal notranslate">exec:</code> blocks </dd> <dd>The task unique hash ID. </dd> <dt><code class="docutils literal notranslate">task.id</code></dt><dd>The pipeline-level task index. Corresponds to <code class="docutils literal notranslate">task_id</code> in the <a class="reference internal" href="../reports.html#trace-report">execution trace</a>. </dd> <dt><code class="docutils literal notranslate">task.index</code></dt><dd>The process-level task index. </dd> <dt><code class="docutils literal notranslate">task.name</code></dt><dd>Available only in <code class="docutils literal notranslate">exec:</code> blocks </dd> <dd>The current task name. </dd> <dt><code class="docutils literal notranslate">task.previousException</code></dt><dd><div class="versionadded"> New in version 24.10.0. </div> </dd> <dd>The exception reported by the previous task attempt. </dd> <dd>Since the exception is available after a failed task attempt, it can only be accessed when retrying a failed task execution, and therefore when <code class="docutils literal notranslate">task.attempt</code> is greater than 1. </dd> <dt><code class="docutils literal notranslate">task.previousTrace</code></dt><dd><div class="versionadded"> New in version 24.10.0. </div> </dd> <dd>The trace record associated with the previous task attempt. </dd> <dd>Since the trace record is available after a failed task attempt, it can only be accessed when retrying a failed task execution, and therefore when <code class="docutils literal notranslate">task.attempt</code> is greater than 1. </dd> <dd>This is useful when retrying a task execution to access the previous task attempt runtime metrics e.g. used memory and CPUs. </dd> <dt><code class="docutils literal notranslate">task.process</code></dt><dd>The current process name. </dd> <dt><code class="docutils literal notranslate">task.workDir</code></dt><dd>Available only in <code class="docutils literal notranslate">exec:</code> blocks </dd> <dd>The task unique directory. </dd> </dl> Additionally, the <a class="reference internal" href="#directives">directive values</a> for the given task can be accessed via <code class="docutils literal notranslate">task.<directive></code>. </section> <section id="inputs"> <h2>Inputs<a class="headerlink" href="#inputs" title="Permalink to this heading"></a></h2> <dl class="simple myst"> <dt><code class="docutils literal notranslate">val( identifier )</code></dt><dd>Declare a variable input. The received value can be any type, and it will be made available to the process body (i.e. <code class="docutils literal notranslate">script</code>, <code class="docutils literal notranslate">shell</code>, <code class="docutils literal notranslate">exec</code>) as a variable given by <code class="docutils literal notranslate">identifier</code>. </dd> <dt><code class="docutils literal notranslate">file( identifier | stageName )</code></dt><dd><div class="deprecated"> Deprecated since version 19.10.0: Use <code class="docutils literal notranslate">path</code> instead. </div> </dd> <dd>Declare a file input. The received value can be any type, and it will be staged into the task directory. If the received value is not a file or collection of files, it is implicitly converted to a string and written to a file. </dd> <dd>The argument can be an identifier or string. If an identifier, the received value will be made available to the process body as a variable. If a string, the received value will be staged into the task directory under the given alias. </dd> <dt><code class="docutils literal notranslate">path( identifier | stageName )</code></dt><dd>Declare a file input. The received value should be a file or collection of files and will be staged into the task directory. </dd> <dd>The argument can be an identifier or string. If an identifier, the received value will be made available to the process body as a variable. If a string, the received value will be staged into the task directory under the given alias. </dd> <dd>Available options: <dl class="simple myst"> <dt><code class="docutils literal notranslate">arity</code></dt><dd><div class="versionadded"> New in version 23.09.0-edge. </div> </dd> <dd>Specify the number of expected files. Can be a number, e.g. <code class="docutils literal notranslate">'1'</code>, or a range, e.g. <code class="docutils literal notranslate">'1..*'</code>. If a task receives an invalid number of files for this <code class="docutils literal notranslate">path</code> input, it will fail. </dd> <dt><code class="docutils literal notranslate">name</code></dt><dd>Specify how the file should be named in the task work directory. Can be a name or a pattern. </dd> <dt><code class="docutils literal notranslate">stageAs</code></dt><dd>Alias of <code class="docutils literal notranslate">name</code>. </dd> </dl> </dd> <dt><code class="docutils literal notranslate">env( name )</code></dt><dd>Declare an environment variable input. The received value should be a string, and it will be exported to the task environment as an environment variable given by <code class="docutils literal notranslate">name</code>. </dd> <dt><code class="docutils literal notranslate">stdin</code></dt><dd>Declare a <code class="docutils literal notranslate">stdin</code> input. The received value should be a string, and it will be provided as the standard input (i.e. <code class="docutils literal notranslate">stdin</code>) to the task script. It should be declared only once for a process. </dd> <dt><code class="docutils literal notranslate">tuple( arg1, arg2, ... )</code></dt><dd>Declare a tuple input. Each argument should be an input declaration such as <code class="docutils literal notranslate">val</code>, <code class="docutils literal notranslate">path</code>, <code class="docutils literal notranslate">env</code>, or <code class="docutils literal notranslate">stdin</code>. </dd> <dd>The received value should be a tuple with the same number of elements as the <code class="docutils literal notranslate">tuple</code> declaration, and each received element should be compatible with the corresponding <code class="docutils literal notranslate">tuple</code> argument. Each tuple element is treated the same way as if it were a standalone input. </dd> </dl> </section> <section id="outputs"> <h2>Outputs<a class="headerlink" href="#outputs" title="Permalink to this heading"></a></h2> <dl class="simple myst"> <dt><code class="docutils literal notranslate">val( value )</code></dt><dd>Declare a variable output. The argument can be any value, and it can reference any output variables defined in the process body (i.e. variables declared without the <code class="docutils literal notranslate">def</code> keyword). </dd> <dt><code class="docutils literal notranslate">file( pattern )</code></dt><dd><div class="deprecated"> Deprecated since version 19.10.0: Use <code class="docutils literal notranslate">path</code> instead. </div> </dd> <dd>Declare a file output. It receives the output files from the task environment that match the given pattern. </dd> <dd>Multiple patterns can be specified using the colon separator (<code class="docutils literal notranslate">:</code>). The union of all files matched by each pattern will be collected. </dd> <dt><code class="docutils literal notranslate">path( pattern, [options] )</code></dt><dd>Declare a file output. It receives the output files from the task environment that match the given pattern. </dd> <dd>Available options: <dl class="simple myst"> <dt><code class="docutils literal notranslate">arity</code></dt><dd><div class="versionadded"> New in version 23.09.0-edge. </div> </dd> <dd>Specify the number of expected files. Can be a number or a range. If a task produces an invalid number of files for this <code class="docutils literal notranslate">path</code> output, it will fail. </dd> <dd>If the arity is <code class="docutils literal notranslate">1</code>, a single file will be emitted. Otherwise, a list will always be emitted, even if only one file is produced. </dd> <dd><div class="admonition warning"> Warning If the arity is not specified, a single file or list will be emitted based on whether a single file or multiple files are produced at runtime, resulting potentially in an output channel with a mixture of files and file collections. </div> </dd> <dt><code class="docutils literal notranslate">followLinks</code></dt><dd>When <code class="docutils literal notranslate">true</code>, target files are returned in place of any matching symlink (default: <code class="docutils literal notranslate">true</code>). </dd> <dt><code class="docutils literal notranslate">glob</code></dt><dd>When <code class="docutils literal notranslate">true</code>, the specified name is interpreted as a glob pattern (default: <code class="docutils literal notranslate">true</code>). </dd> <dt><code class="docutils literal notranslate">hidden</code></dt><dd>When <code class="docutils literal notranslate">true</code>, hidden files are included in the matching output files (default: <code class="docutils literal notranslate">false</code>). </dd> <dt><code class="docutils literal notranslate">includeInputs</code></dt><dd>When <code class="docutils literal notranslate">true</code> and the output path is a glob pattern, any input files matching the pattern are also included in the output (default: <code class="docutils literal notranslate">false</code>). </dd> <dt><code class="docutils literal notranslate">maxDepth</code></dt><dd>Maximum number of directory levels to visit (default: no limit). </dd> <dt><code class="docutils literal notranslate">type</code></dt><dd>Type of paths returned, either <code class="docutils literal notranslate">file</code>, <code class="docutils literal notranslate">dir</code> or <code class="docutils literal notranslate">any</code> (default: <code class="docutils literal notranslate">any</code>, or <code class="docutils literal notranslate">file</code> if the specified file name pattern contains a double star (<code class="docutils literal notranslate">**</code>)). </dd> </dl> </dd> <dt><code class="docutils literal notranslate">env( name )</code></dt><dd>Declare an environment variable output. It receives the value of the environment variable (given by <code class="docutils literal notranslate">name</code>) from the task environment. </dd> <dd><div class="versionchanged"> Changed in version 23.12.0-edge: Prior to this version, if the environment variable contained multiple lines of output, the output would be compressed to a single line by converting newlines to spaces. </div> </dd> <dt><code class="docutils literal notranslate">stdout</code></dt><dd>Declare a <code class="docutils literal notranslate">stdout</code> output. It receives the standard output of the task script. </dd> <dt><code class="docutils literal notranslate">eval( command )</code></dt><dd><div class="versionadded"> New in version 24.02.0-edge. </div> </dd> <dd>Declare an <code class="docutils literal notranslate">eval</code> output. It receives the standard output of the given command, which is executed in the task environment after the task script. </dd> <dd>If the command fails, the task will also fail. </dd> <dt><code class="docutils literal notranslate">tuple( arg1, arg2, ... )</code></dt><dd>Declare a tuple output. Each argument should be an output declaration such as <code class="docutils literal notranslate">val</code>, <code class="docutils literal notranslate">path</code>, <code class="docutils literal notranslate">env</code>, <code class="docutils literal notranslate">stdin</code>, or <code class="docutils literal notranslate">eval</code>. Each tuple element is treated the same way as if it were a standalone output. </dd> </dl> <section id="generic-options"> <h3>Generic options<a class="headerlink" href="#generic-options" title="Permalink to this heading"></a></h3> The following options are available for all process outputs: <dl class="simple myst"> <dt><code class="docutils literal notranslate">emit: <name></code></dt><dd>Defines the name of the output channel. </dd> <dt><code class="docutils literal notranslate">optional: true | false</code></dt><dd>When <code class="docutils literal notranslate">true</code>, the task will not fail if the specified output is missing (default: <code class="docutils literal notranslate">false</code>). </dd> <dt><code class="docutils literal notranslate">topic: <name></code></dt><dd><div class="versionadded"> New in version 23.11.0-edge. </div> </dd> <dd>Experimental: may change in a future release. </dd> <dd>Defines the <a class="reference internal" href="channel.html#channel-topic">channel topic</a> to which the output will be sent. </dd> </dl> </section> </section> <section id="directives"> <h2>Directives<a class="headerlink" href="#directives" title="Permalink to this heading"></a></h2> <section id="accelerator"> <h3>accelerator<a class="headerlink" href="#accelerator" title="Permalink to this heading"></a></h3> <div class="versionadded"> New in version 19.09.0-edge. </div> The <code class="docutils literal notranslate">accelerator</code> directive allows you to request hardware accelerators (e.g. GPUs) for the task execution. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process foo { accelerator 4, type: 'nvidia-tesla-k80' script: """ your_gpu_enabled --command --line """ } </pre></div> </div> The above examples will request 4 GPUs of type <code class="docutils literal notranslate">nvidia-tesla-k80</code>. <div class="admonition note"> Note This directive is only used by certain executors. Refer to the <a class="reference internal" href="../executor.html#executor-page">Executors</a> page to see which executors support this directive. </div> <div class="admonition note"> Note Additional options may be required to fully enable the use of accelerators. When using containers with GPUs, you must pass the GPU drivers through to the container. For Docker, this requires the option <code class="docutils literal notranslate">--gpus all</code> in the docker run command. For Apptainer/Singularity, this requires the option <code class="docutils literal notranslate">--nv</code>. The specific implementation details depend on the accelerator and container type being used. </div> <div class="admonition note"> Note The accelerator <code class="docutils literal notranslate">type</code> option depends on the target execution platform. Refer to the platform-specific documentation for details on the available accelerators: <ul class="simple"> <li><a class="reference external" href="https://cloud.google.com/compute/docs/gpus/">Google Cloud</a></li> <li><a class="reference external" href="https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/#clusters-containing-different-types-of-gpus">Kubernetes</a></li> </ul> The accelerator <code class="docutils literal notranslate">type</code> option is not supported for AWS Batch. You can control the accelerator type indirectly through the allowed instance types in your Compute Environment. See the <a class="reference external" href="https://aws.amazon.com/batch/faqs/?#GPU_Scheduling_">AWS Batch FAQs</a> for more information. </div> </section> <section id="afterscript"> <h3>afterScript<a class="headerlink" href="#afterscript" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">afterScript</code> directive allows you to execute a custom (Bash) snippet immediately after the main process has run. This may be useful to clean up your staging area. <div class="admonition note"> Note When combined with the <a class="reference internal" href="#process-container">container directive</a>, the <code class="docutils literal notranslate">afterScript</code> will be executed outside the specified container. In other words, the <code class="docutils literal notranslate">afterScript</code> is always executed in the host environment. </div> </section> <section id="arch"> <h3>arch<a class="headerlink" href="#arch" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">arch</code> directive allows you to define the CPU architecture to build the software in use by the process’ task. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process cpu_task { spack 'blast-plus@2.13.0' arch 'linux/x86_64', target: 'cascadelake' script: """ blastp -query input_sequence -num_threads ${task.cpus} """ } </pre></div> </div> The example above declares that the CPU generic architecture is <code class="docutils literal notranslate">linux/x86_64</code> (X86 64 bit), and more specifically that the microarchitecture is <code class="docutils literal notranslate">cascadelake</code> (a specific generation of Intel CPUs). This directive is currently used by the following Nextflow functionalities: <ul class="simple"> <li>by the <a class="reference internal" href="#spack">spack</a> directive, to build microarchitecture-optimised applications;</li> <li>by the <a class="reference internal" href="../wave.html#wave-page">Wave containers</a> service, to build containers for one of the generic families of CPU architectures (see below);</li> <li>by the <code class="docutils literal notranslate">spack</code> strategy within <a class="reference internal" href="../wave.html#wave-page">Wave containers</a>, to optimise the container builds for specific CPU microarchitectures.</li> </ul> Allowed values for the <code class="docutils literal notranslate">arch</code> directive are as follows, grouped by equivalent family (choices available for the sake of compatibility): <ul class="simple"> <li>X86 64 bit: <code class="docutils literal notranslate">linux/x86_64</code>, <code class="docutils literal notranslate">x86_64</code>, <code class="docutils literal notranslate">linux/amd64</code>, <code class="docutils literal notranslate">amd64</code></li> <li>ARM 64 bit: <code class="docutils literal notranslate">linux/aarch64</code>, <code class="docutils literal notranslate">aarch64</code>, <code class="docutils literal notranslate">linux/arm64</code>, <code class="docutils literal notranslate">arm64</code>, <code class="docutils literal notranslate">linux/arm64/v8</code></li> <li>ARM 64 bit, older generation: <code class="docutils literal notranslate">linux/arm64/v7</code></li> </ul> Examples of values for the architecture <code class="docutils literal notranslate">target</code> option are <code class="docutils literal notranslate">cascadelake</code>, <code class="docutils literal notranslate">icelake</code>, <code class="docutils literal notranslate">zen2</code> and <code class="docutils literal notranslate">zen3</code>. See the <a class="reference external" href="https://spack.readthedocs.io/en/latest/basic_usage.html#support-for-specific-microarchitectures">Spack documentation</a> for the full and up-to-date list of meaningful targets. </section> <section id="array"> <h3>array<a class="headerlink" href="#array" title="Permalink to this heading"></a></h3> <div class="versionadded"> New in version 24.04.0. </div> <div class="admonition warning"> Warning Experimental: may change in a future release. </div> The <code class="docutils literal notranslate">array</code> directive allows you to submit tasks as job arrays for executors that support it. A job array is a collection of jobs with the same resource requirements and the same script (parameterized by an index). Job arrays incur significantly less scheduling overhead compared to individual jobs, and as a result they are preferred by HPC schedulers where possible. The directive should be specified with a given array size, along with an executor that supports job arrays. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process cpu_task { executor 'slurm' array 100 script: """ your_command --here """ } </pre></div> </div> Nextflow currently supports job arrays for the following executors: <ul class="simple"> <li><a class="reference internal" href="../executor.html#awsbatch-executor">AWS Batch</a></li> <li><a class="reference internal" href="../executor.html#google-batch-executor">Google Cloud Batch</a></li> <li><a class="reference internal" href="../executor.html#lsf-executor">LSF</a></li> <li><a class="reference internal" href="../executor.html#pbs-executor">PBS/Torque</a></li> <li><a class="reference internal" href="../executor.html#pbspro-executor">PBS Pro</a></li> <li><a class="reference internal" href="../executor.html#sge-executor">SGE</a></li> <li><a class="reference internal" href="../executor.html#slurm-executor">SLURM</a></li> </ul> A process using job arrays will collect tasks and submit each batch as a job array when it is ready. Any “leftover” tasks will be submitted as a partial job array. Once a job array is submitted, each “child” task is executed as an independent job. Any tasks that fail (and can be retried) will be retried without interfering with the tasks that succeeded. Retried tasks are submitted individually rather than through a job array, in order to allow for the use of <a class="reference internal" href="../process.html#dynamic-task-resources">dynamic resources</a>. The following directives must be uniform across all tasks in a process that uses job arrays, because these directives are specified once for the entire job array: <ul class="simple"> <li><a class="reference internal" href="#process-accelerator">accelerator</a></li> <li><a class="reference internal" href="#process-clusteroptions">clusterOptions</a></li> <li><a class="reference internal" href="#process-cpus">cpus</a></li> <li><a class="reference internal" href="#process-disk">disk</a></li> <li><a class="reference internal" href="#process-machinetype">machineType</a></li> <li><a class="reference internal" href="#process-memory">memory</a></li> <li><a class="reference internal" href="#process-queue">queue</a></li> <li><a class="reference internal" href="#process-resourcelabels">resourceLabels</a></li> <li><a class="reference internal" href="#process-resourcelimits">resourceLimits</a></li> <li><a class="reference internal" href="#process-time">time</a></li> </ul> For cloud-based executors like AWS Batch, or when using Fusion with any executor, the following additional directives must be uniform: <ul class="simple"> <li><a class="reference internal" href="#process-container">container</a></li> <li><a class="reference internal" href="#process-containeroptions">containerOptions</a></li> </ul> When using Wave, the following additional directives must be uniform: <ul class="simple"> <li><a class="reference internal" href="#process-conda">conda</a></li> </ul> </section> <section id="beforescript"> <h3>beforeScript<a class="headerlink" href="#beforescript" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">beforeScript</code> directive allows you to execute a custom (Bash) snippet before the main process script is run. This may be useful to initialise the underlying cluster environment or for other custom initialisation. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process foo { beforeScript 'source /cluster/bin/setup' script: """ echo bar """ } </pre></div> </div> When the process is containerized (using the <a class="reference internal" href="#process-container">container</a> directive), the <code class="docutils literal notranslate">beforeScript</code> will be executed in the container only if the executor is container-native (e.g. cloud batch executors, Kubernetes). Otherwise, the <code class="docutils literal notranslate">beforeScript</code> will be executed outside the container. </section> <section id="cache"> <h3>cache<a class="headerlink" href="#cache" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">cache</code> directive allows you to store the process results to a local cache. When the cache is enabled and the pipeline is launched with the <a class="reference internal" href="../your-first-script.html#getstarted-resume">resume</a> option, any task executions that are already cached will be re-used. See the <a class="reference internal" href="../cache-and-resume.html#cache-resume-page">Caching and resuming</a> page for more information about how the cache works. The cache is enabled by default, but you can disable it for a specific process by setting the <code class="docutils literal notranslate">cache</code> directive to <code class="docutils literal notranslate">false</code>. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process noCacheThis { cache false // ... } </pre></div> </div> The following options are available: <dl class="simple myst"> <dt><code class="docutils literal notranslate">false</code></dt><dd>Disable caching. </dd> <dt><code class="docutils literal notranslate">true</code> (default)</dt><dd>Enable caching. Input file metadata (name, size, last updated timestamp) are included in the cache keys. </dd> <dt><code class="docutils literal notranslate">'deep'</code></dt><dd>Enable caching. Input file content is included in the cache keys. </dd> <dt><code class="docutils literal notranslate">'lenient'</code></dt><dd>Enable caching. Minimal input file metadata (name and size only) are included in the cache keys. </dd> <dd>This strategy provides a workaround for incorrect caching invalidation observed on shared file systems due to inconsistent file timestamps. </dd> </dl> </section> <section id="clusteroptions"> <h3>clusterOptions<a class="headerlink" href="#clusteroptions" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">clusterOptions</code> directive allows the usage of any native configuration option accepted by your cluster submit command. You can use it to request non-standard resources or use settings that are specific to your cluster and not supported out of the box by Nextflow. The cluster options can be a string: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process foo { clusterOptions '-x 1 -y 2' // ... } </pre></div> </div> <div class="versionchanged"> Changed in version 24.04.0: Prior to this version, grid executors that require each option to be on a separate line in the job script would attempt to split multiple options using a variety of different conventions. Multiple options can now be specified more clearly using a string list as shown below. </div> The cluster options can also be a string list: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process foo { clusterOptions '-x 1', '-y 2', '--flag' // ... } </pre></div> </div> Grid executors that require one option per line will write each option to a separate line, while grid executors that allow multiple options per line will write all options to a single line, the same as with a string. This form is useful to control how the options are split across lines when it is required by the scheduler. <div class="admonition note"> Note This directive is only used by grid executors. Refer to the <a class="reference internal" href="../executor.html#executor-page">Executors</a> page to see which executors support this directive. </div> <div class="admonition warning"> Warning While you can use the <code class="docutils literal notranslate">clusterOptions</code> directive to specify options that are supported as process directives (<code class="docutils literal notranslate">queue</code>, <code class="docutils literal notranslate">memory</code>, <code class="docutils literal notranslate">time</code>, etc), you should not use both at the same time, as it will cause undefined behavior. Most HPC schedulers will either fail or simply ignore one or the other. </div> </section> <section id="conda"> <h3>conda<a class="headerlink" href="#conda" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">conda</code> directive allows for the definition of the process dependencies using the <a class="reference external" href="https://conda.io">Conda</a> package manager. Nextflow automatically sets up an environment for the given package names listed by in the <code class="docutils literal notranslate">conda</code> directive. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process foo { conda 'bwa=0.7.15' script: """ your_command --here """ } </pre></div> </div> Multiple packages can be specified separating them with a blank space e.g. <code class="docutils literal notranslate">bwa=0.7.15 fastqc=0.11.5</code>. The name of the channel from where a specific package needs to be downloaded can be specified using the usual Conda notation i.e. prefixing the package with the channel name as shown here <code class="docutils literal notranslate">bioconda::bwa=0.7.15</code>. The <code class="docutils literal notranslate">conda</code> directive also allows the specification of a Conda environment file path or the path of an existing environment directory. See the <a class="reference internal" href="../conda.html#conda-page">Conda environments</a> page for further details. </section> <section id="container"> <h3>container<a class="headerlink" href="#container" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">container</code> directive allows you to execute the process script in a <a class="reference external" href="http://docker.io">Docker</a> container. It requires the Docker daemon to be running in machine where the pipeline is executed, i.e. the local machine when using the local executor or the cluster nodes when the pipeline is deployed through a grid executor. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process runThisInDocker { container 'dockerbox:tag' script: """ your_command --here """ } </pre></div> </div> Simply replace in the above script <code class="docutils literal notranslate">dockerbox:tag</code> with the name of the Docker image you want to use. <div class="admonition tip"> Tip Containers are a very useful way to execute your scripts in a reproducible self-contained environment or to run your pipeline in the cloud. </div> <div class="admonition note"> Note This directive is ignored for processes that are <a class="reference internal" href="../process.html#process-native">executed natively</a>. </div> </section> <section id="containeroptions"> <h3>containerOptions<a class="headerlink" href="#containeroptions" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">containerOptions</code> directive allows you to specify any container execution option supported by the underlying container engine (ie. Docker, Singularity, etc). This can be useful to provide container settings only for a specific process e.g. mount a custom path: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process runThisWithDocker { container 'busybox:latest' containerOptions '--volume /data/db:/db' output: path 'output.txt' script: """ your_command --data /db > output.txt """ } </pre></div> </div> <div class="admonition warning"> Warning This feature is not supported by the <a class="reference internal" href="../executor.html#k8s-executor">Kubernetes</a> and <a class="reference internal" href="../executor.html#google-lifesciences-executor">Google Life Sciences</a> executors. </div> </section> <section id="cpus"> <h3>cpus<a class="headerlink" href="#cpus" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">cpus</code> directive allows you to define the number of (logical) CPU required by the process’ task. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process big_job { cpus 8 executor 'sge' script: """ blastp -query input_sequence -num_threads ${task.cpus} """ } </pre></div> </div> This directive is required for tasks that execute multi-process or multi-threaded commands/tools and it is meant to reserve enough CPUs when a pipeline task is executed through a cluster resource manager. See also: <a class="reference internal" href="#penv">penv</a>, <a class="reference internal" href="#memory">memory</a>, <a class="reference internal" href="#time">time</a>, <a class="reference internal" href="#queue">queue</a>, <a class="reference internal" href="#maxforks">maxForks</a> </section> <section id="debug"> <h3>debug<a class="headerlink" href="#debug" title="Permalink to this heading"></a></h3> By default the <code class="docutils literal notranslate">stdout</code> produced by the commands executed in all processes is ignored. By setting the <code class="docutils literal notranslate">debug</code> directive to <code class="docutils literal notranslate">true</code>, you can forward the process <code class="docutils literal notranslate">stdout</code> to the current top running process <code class="docutils literal notranslate">stdout</code> file, showing it in the shell terminal. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process sayHello { debug true script: """ echo Hello """ } </pre></div> </div> <div class="highlight-default notranslate"><div class="highlight"><pre>Hello </pre></div> </div> Without specifying <code class="docutils literal notranslate">debug true</code>, you won’t see the <code class="docutils literal notranslate">Hello</code> string printed out when executing the above example. </section> <section id="disk"> <h3>disk<a class="headerlink" href="#disk" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">disk</code> directive allows you to define how much local disk storage the process is allowed to use. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process big_job { disk '2 GB' executor 'cirrus' script: """ your_command --here """ } </pre></div> </div> The following memory unit suffix can be used when specifying the disk value: <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Unit</th> <th class="head">Description</th> </tr> </thead> <tbody> <tr class="row-even"><td>B</td> <td>Bytes</td> </tr> <tr class="row-odd"><td>KB</td> <td>Kilobytes</td> </tr> <tr class="row-even"><td>MB</td> <td>Megabytes</td> </tr> <tr class="row-odd"><td>GB</td> <td>Gigabytes</td> </tr> <tr class="row-even"><td>TB</td> <td>Terabytes</td> </tr> </tbody> </table> See <a class="reference internal" href="stdlib.html#stdlib-types-memoryunit">MemoryUnit</a> for more information. <div class="admonition note"> Note This directive is only used by certain executors. Refer to the <a class="reference internal" href="../executor.html#executor-page">Executors</a> page to see which executors support this directive. </div> See also: <a class="reference internal" href="#cpus">cpus</a>, <a class="reference internal" href="#memory">memory</a> <a class="reference internal" href="#time">time</a>, <a class="reference internal" href="#queue">queue</a> and <a class="reference internal" href="../process.html#dynamic-task-resources">Dynamic task resources</a>. </section> <section id="echo"> <h3>echo<a class="headerlink" href="#echo" title="Permalink to this heading"></a></h3> <div class="deprecated"> Deprecated since version 22.04.0: Use <code class="docutils literal notranslate">debug</code> instead </div> </section> <section id="errorstrategy"> <h3>errorStrategy<a class="headerlink" href="#errorstrategy" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">errorStrategy</code> directive allows you to define how the process manages an error condition. By default, when an error status is returned by the executed script (i.e. when it ends with a non-zero exit status), the process stops immediately, forcing the entire pipeline to terminate. The following error strategies are available: <dl class="simple myst"> <dt><code class="docutils literal notranslate">terminate</code> (default)</dt><dd>When a task fails, terminate the pipeline immediately and report an error. Pending and running jobs are killed. </dd> <dt><code class="docutils literal notranslate">finish</code></dt><dd>When a task fails, wait for submitted and running tasks to finish and then terminate the pipeline, reporting an error. </dd> <dt><code class="docutils literal notranslate">ignore</code></dt><dd>When a task fails, ignore it and continue the pipeline execution. If the <code class="docutils literal notranslate">workflow.failOnIgnore</code> config option is set to <code class="docutils literal notranslate">true</code>, the pipeline will report an error (i.e. return a non-zero exit code) upon completion. Otherwise, the pipeline will complete successfully. </dd> <dd>See <a class="reference internal" href="stdlib.html#stdlib-constants">Constants</a> for more information on <code class="docutils literal notranslate">workflow.failOnIgnore</code>. </dd> <dt><code class="docutils literal notranslate">retry</code></dt><dd>When a task fails, retry it. </dd> </dl> When setting the <code class="docutils literal notranslate">errorStrategy</code> directive to <code class="docutils literal notranslate">ignore</code> the process doesn’t stop on an error condition, it just reports a message notifying you of the error event. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process ignoreAnyError { errorStrategy 'ignore' // ... } </pre></div> </div> In this case, the workflow will complete successfully and return an exit status of 0. However, if you set <code class="docutils literal notranslate">workflow.failOnIgnore = true</code> in your Nextflow configuration, the workflow will return a non-zero exit status and report the failed tasks as an error. The <code class="docutils literal notranslate">retry</code> error strategy allows you to re-submit for execution a process returning an error condition. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process retryIfFail { errorStrategy 'retry' // ... } </pre></div> </div> The number of times a failing process is re-executed is defined by the <a class="reference internal" href="#maxretries">maxRetries</a> and <a class="reference internal" href="#maxerrors">maxErrors</a> directives. <div class="admonition tip"> Tip More complex strategies depending on the task exit status or other parametric values can be defined using a dynamic <code class="docutils literal notranslate">errorStrategy</code>. See <a class="reference internal" href="../process.html#dynamic-directives">Dynamic directives</a> for details. </div> See also: <a class="reference internal" href="#maxerrors">maxErrors</a>, <a class="reference internal" href="#maxretries">maxRetries</a> and <a class="reference internal" href="../process.html#dynamic-task-resources">Dynamic task resources</a>. </section> <section id="executor"> <h3>executor<a class="headerlink" href="#executor" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">executor</code> defines the underlying system where processes are executed. By default a process uses the executor defined globally in the <code class="docutils literal notranslate">nextflow.config</code> file. The <code class="docutils literal notranslate">executor</code> directive allows you to configure what executor has to be used by the process, overriding the default configuration. The following executors are available: <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Name</th> <th class="head">Executor</th> </tr> </thead> <tbody> <tr class="row-even"><td><code class="docutils literal notranslate">awsbatch</code></td> <td><a class="reference external" href="https://aws.amazon.com/batch/">AWS Batch</a> service</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">azurebatch</code></td> <td><a class="reference external" href="https://azure.microsoft.com/en-us/services/batch/">Azure Batch</a> service</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">condor</code></td> <td><a class="reference external" href="https://research.cs.wisc.edu/htcondor/">HTCondor</a> job scheduler</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">google-lifesciences</code></td> <td><a class="reference external" href="https://cloud.google.com/life-sciences">Google Genomics Pipelines</a> service</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">k8s</code></td> <td><a class="reference external" href="https://kubernetes.io/">Kubernetes</a> cluster</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">local</code></td> <td>The computer where <code class="docutils literal notranslate">Nextflow</code> is launched</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">lsf</code></td> <td><a class="reference external" href="http://en.wikipedia.org/wiki/Platform_LSF">Platform LSF</a> job scheduler</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">moab</code></td> <td><a class="reference external" href="http://www.adaptivecomputing.com/moab-hpc-basic-edition/">Moab</a> job scheduler</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">nqsii</code></td> <td><a class="reference external" href="https://www.rz.uni-kiel.de/en/our-portfolio/hiperf/nec-linux-cluster">NQSII</a> job scheduler</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">oge</code></td> <td>Alias for the <code class="docutils literal notranslate">sge</code> executor</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">pbs</code></td> <td><a class="reference external" href="http://en.wikipedia.org/wiki/Portable_Batch_System">PBS/Torque</a> job scheduler</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">pbspro</code></td> <td><a class="reference external" href="https://www.pbsworks.com/">PBS Pro</a> job scheduler</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">sge</code></td> <td>Sun Grid Engine / <a class="reference external" href="http://gridscheduler.sourceforge.net/">Open Grid Engine</a></td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">slurm</code></td> <td><a class="reference external" href="https://en.wikipedia.org/wiki/Slurm_Workload_Manager">SLURM</a> workload manager</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">uge</code></td> <td>Alias for the <code class="docutils literal notranslate">sge</code> executor</td> </tr> </tbody> </table> The following example shows how to set the process’s executor: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process doSomething { executor 'sge' // ... } </pre></div> </div> <div class="admonition note"> Note Each executor supports additional directives and <code class="docutils literal notranslate">executor</code> configuration options. See <a class="reference internal" href="../executor.html#executor-page">Executors</a> for more information. </div> </section> <section id="ext"> <h3>ext<a class="headerlink" href="#ext" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">ext</code> is a special directive used as namespace for user custom process directives. This can be useful for advanced configuration options. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process mapping { container "biocontainers/star:${task.ext.version}" input: path genome tuple val(sampleId), path(reads) script: """ STAR --genomeDir $genome --readFilesIn $reads ${task.ext.args ?: ''} """ } </pre></div> </div> In the above example, the process container version is controlled by <code class="docutils literal notranslate">ext.version</code>, and the script supports additional command line arguments through <code class="docutils literal notranslate">ext.args</code>. The <code class="docutils literal notranslate">ext</code> directive can be set in the process definition: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process mapping { ext version: '2.5.3', args: '--foo --bar' // ... } </pre></div> </div> Or in the Nextflow configuration: <div class="highlight-groovy notranslate"><div class="highlight"><pre>process.ext.version = '2.5.3' process.ext.args = '--foo --bar' </pre></div> </div> </section> <section id="fair"> <h3>fair<a class="headerlink" href="#fair" title="Permalink to this heading"></a></h3> <div class="versionadded"> New in version 22.12.0-edge. </div> The <code class="docutils literal notranslate">fair</code> directive, when enabled, guarantees that process outputs will be emitted in the order in which they were received. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process foo { fair true input: val x output: tuple val(task.index), val(x) script: """ sleep \$((RANDOM % 3)) """ } workflow { channel.of('A','B','C','D') | foo | view } </pre></div> </div> The above example produces: <div class="highlight-default notranslate"><div class="highlight"><pre>[1, A] [2, B] [3, C] [4, D] </pre></div> </div> </section> <section id="label"> <h3>label<a class="headerlink" href="#label" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">label</code> directive allows the annotation of processes with mnemonic identifier of your choice. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process bigTask { label 'big_mem' script: """ your_command --here """ } </pre></div> </div> The same label can be applied to more than a process and multiple labels can be applied to the same process using the <code class="docutils literal notranslate">label</code> directive more than one time. <div class="admonition note"> Note A label must consist of alphanumeric characters or <code class="docutils literal notranslate">_</code>, must start with an alphabetic character and must end with an alphanumeric character. </div> Labels are useful to organize workflow processes in separate groups which can be referenced in the configuration file to select and configure a subset of processes having similar computing requirements. See <a class="reference internal" href="../config.html#config-process-selectors">Process selectors</a> for more information. See also: <a class="reference internal" href="#resourcelabels">resourceLabels</a> </section> <section id="machinetype"> <h3>machineType<a class="headerlink" href="#machinetype" title="Permalink to this heading"></a></h3> <div class="versionadded"> New in version 19.07.0. </div> The <code class="docutils literal notranslate">machineType</code> can be used to specify a predefined Google Compute Platform <a class="reference external" href="https://cloud.google.com/compute/docs/machine-types">machine type</a> when running using the <a class="reference internal" href="../executor.html#google-batch-executor">Google Batch</a> or <a class="reference internal" href="../executor.html#google-lifesciences-executor">Google Life Sciences</a> executor, or when using the autopools feature of the <a class="reference internal" href="../executor.html#azurebatch-executor">Azure Batch executor</a>. This directive is optional and if specified overrides the cpus and memory directives: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process foo { machineType 'n1-highmem-8' script: """ your_command --here """ } </pre></div> </div> See also: <a class="reference internal" href="#cpus">cpus</a> and <a class="reference internal" href="#memory">memory</a>. </section> <section id="maxsubmitawait"> <h3>maxSubmitAwait<a class="headerlink" href="#maxsubmitawait" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">maxSubmitAwait</code> directive allows you to specify how long a task can remain in submission queue without being executed. Elapsed this time the task execution will fail. When used along with <code class="docutils literal notranslate">retry</code> error strategy, it can be useful to re-schedule the task to a difference queue or resource requirement. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process foo { errorStrategy 'retry' maxSubmitAwait '10 mins' maxRetries 3 queue "${task.submitAttempt==1 : 'spot-compute' : 'on-demand-compute'}" script: """ your_command --here """ } </pre></div> </div> In the above example the task is submitted to the <code class="docutils literal notranslate">spot-compute</code> on the first attempt (<code class="docutils literal notranslate">task.submitAttempt==1</code>). If the task execution does not start in the 10 minutes, a failure is reported and a new submission is attempted using the queue named <code class="docutils literal notranslate">on-demand-compute</code>. </section> <section id="maxerrors"> <h3>maxErrors<a class="headerlink" href="#maxerrors" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">maxErrors</code> directive allows you to specify the maximum number of times a process can fail when using the <code class="docutils literal notranslate">retry</code> error strategy. By default this directive is disabled, you can set it as shown in the example below: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process retryIfFail { errorStrategy 'retry' maxErrors 5 script: """ echo 'do this as that .. ' """ } </pre></div> </div> <div class="admonition note"> Note This setting considers the total errors accumulated for a given process, across all instances. If you want to control the number of times a process instance (aka task) can fail, use <code class="docutils literal notranslate">maxRetries</code>. </div> See also: <a class="reference internal" href="#errorstrategy">errorStrategy</a> and <a class="reference internal" href="#maxretries">maxRetries</a>. </section> <section id="maxforks"> <h3>maxForks<a class="headerlink" href="#maxforks" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">maxForks</code> directive allows you to define the maximum number of process instances that can be executed in parallel. By default this value is equals to the number of CPU cores available minus 1. If you want to execute a process in a sequential manner, set this directive to one. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process doNotParallelizeIt { maxForks 1 script: """ your_command --here """ } </pre></div> </div> </section> <section id="maxretries"> <h3>maxRetries<a class="headerlink" href="#maxretries" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">maxRetries</code> directive allows you to define the maximum number of times a process instance can be re-submitted in case of failure. This value is applied only when using the <code class="docutils literal notranslate">retry</code> error strategy. By default only one retry is allowed, you can increase this value as shown below: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process retryIfFail { errorStrategy 'retry' maxRetries 3 script: """ echo 'do this as that .. ' """ } </pre></div> </div> <div class="admonition note"> Note There is a subtle but important difference between <code class="docutils literal notranslate">maxRetries</code> and the <code class="docutils literal notranslate">maxErrors</code> directive. The latter defines the total number of errors that are allowed during the process execution (the same process can launch different execution instances), while the <code class="docutils literal notranslate">maxRetries</code> defines the maximum number of times the same process execution can be retried in case of an error. </div> See also: <a class="reference internal" href="#errorstrategy">errorStrategy</a> and <a class="reference internal" href="#maxerrors">maxErrors</a>. </section> <section id="memory"> <h3>memory<a class="headerlink" href="#memory" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">memory</code> directive allows you to define how much memory the process is allowed to use. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process big_job { memory '2 GB' executor 'sge' script: """ your_command --here """ } </pre></div> </div> The following memory unit suffix can be used when specifying the memory value: <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Unit</th> <th class="head">Description</th> </tr> </thead> <tbody> <tr class="row-even"><td>B</td> <td>Bytes</td> </tr> <tr class="row-odd"><td>KB</td> <td>Kilobytes</td> </tr> <tr class="row-even"><td>MB</td> <td>Megabytes</td> </tr> <tr class="row-odd"><td>GB</td> <td>Gigabytes</td> </tr> <tr class="row-even"><td>TB</td> <td>Terabytes</td> </tr> </tbody> </table> See <a class="reference internal" href="stdlib.html#stdlib-types-memoryunit">MemoryUnit</a> for more information. See also: <a class="reference internal" href="#cpus">cpus</a>, <a class="reference internal" href="#time">time</a>, <a class="reference internal" href="#queue">queue</a> and <a class="reference internal" href="../process.html#dynamic-task-resources">Dynamic task resources</a>. </section> <section id="module"> <h3>module<a class="headerlink" href="#module" title="Permalink to this heading"></a></h3> <a class="reference external" href="http://modules.sourceforge.net/">Environment Modules</a> is a package manager that allows you to dynamically configure your execution environment and easily switch between multiple versions of the same software tool. If it is available in your system you can use it with Nextflow in order to configure the processes execution environment in your pipeline. In a process definition you can use the <code class="docutils literal notranslate">module</code> directive to load a specific module version to be used in the process execution environment. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process basicExample { module 'ncbi-blast/2.2.27' script: """ blastp -query <etc..> """ } </pre></div> </div> You can repeat the <code class="docutils literal notranslate">module</code> directive for each module you need to load. Alternatively multiple modules can be specified in a single <code class="docutils literal notranslate">module</code> directive by separating all the module names by using a <code class="docutils literal notranslate">:</code> (colon) character as shown below: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process manyModules { module 'ncbi-blast/2.2.27:t_coffee/10.0:clustalw/2.1' script: """ blastp -query <etc..> """ } </pre></div> </div> </section> <section id="penv"> <h3>penv<a class="headerlink" href="#penv" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">penv</code> directive allows you to define the parallel environment to be used when submitting a parallel task to the <a class="reference internal" href="../executor.html#sge-executor">SGE</a> resource manager. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process big_job { cpus 4 penv 'smp' executor 'sge' script: """ blastp -query input_sequence -num_threads ${task.cpus} """ } </pre></div> </div> This configuration depends on the parallel environment provided by your grid engine installation. Refer to your cluster documentation or contact your admin to learn more about this. See also: <a class="reference internal" href="#cpus">cpus</a>, <a class="reference internal" href="#memory">memory</a>, <a class="reference internal" href="#time">time</a> </section> <section id="pod"> <h3>pod<a class="headerlink" href="#pod" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">pod</code> directive allows the definition of pod specific settings, such as environment variables, secrets, and config maps, when using the <a class="reference internal" href="../executor.html#k8s-executor">Kubernetes</a> executor. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process your_task { pod env: 'FOO', value: 'bar' script: """ echo $FOO """ } </pre></div> </div> The above snippet defines an environment variable named <code class="docutils literal notranslate">FOO</code> whose value is <code class="docutils literal notranslate">bar</code>. When defined in the Nextflow configuration file, a pod setting can be defined as a map: <div class="highlight-groovy notranslate"><div class="highlight"><pre>process { pod = [env: 'FOO', value: 'bar'] } </pre></div> </div> Or as a list of maps: <div class="highlight-groovy notranslate"><div class="highlight"><pre>process { pod = [ [env: 'FOO', value: 'bar'], [secret: 'my-secret/key1', mountPath: '/etc/file.txt'] ] } </pre></div> </div> The following options are available: <dl class="simple myst"> <dt><code class="docutils literal notranslate">affinity: <config></code></dt><dd><div class="versionadded"> New in version 22.01.0-edge. </div> </dd> <dd>Specifies the pod <a class="reference external" href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity">affinity</a> with the given configuration. </dd> <dt><code class="docutils literal notranslate">annotation: '<name>', value: '<value>'</code></dt><dd>Can be specified multiple times </dd> <dd>Defines a pod <a class="reference external" href="https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/">annotation</a> with the given name and value. </dd> <dt><code class="docutils literal notranslate">automountServiceAccountToken: true | false</code></dt><dd><div class="versionadded"> New in version 22.01.0-edge. </div> </dd> <dd>Specifies whether to <a class="reference external" href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#opt-out-of-api-credential-automounting">automount service account token</a> into the pod (default: <code class="docutils literal notranslate">true</code>). </dd> <dt><code class="docutils literal notranslate">config: '<configMap>/<key>', mountPath: '</absolute/path>'</code></dt><dd>Can be specified multiple times </dd> <dd>Mounts a <a class="reference external" href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/">ConfigMap</a> with name and optional key to the given path. If the key is omitted, the path is interpreted as a directory and all entries in the <code class="docutils literal notranslate">ConfigMap</code> are exposed in that path. </dd> <dt><code class="docutils literal notranslate">csi: '<config>', mountPath: '</absolute/path>'</code></dt><dd><div class="versionadded"> New in version 22.11.0-edge. </div> </dd> <dd>Can be specified multiple times </dd> <dd>Mounts a <a class="reference external" href="https://kubernetes.io/docs/concepts/storage/ephemeral-volumes/#csi-ephemeral-volumes">CSI ephemeral volume</a> with the given configuration to the given path. </dd> <dt><code class="docutils literal notranslate">emptyDir: <config>, mountPath: '</absolute/path>'</code></dt><dd><div class="versionadded"> New in version 22.11.0-edge. </div> </dd> <dd>Can be specified multiple times </dd> <dd>Mounts an <a class="reference external" href="https://kubernetes.io/docs/concepts/storage/volumes/#emptydir">emptyDir</a> with the given configuration to the given path. </dd> <dt><code class="docutils literal notranslate">env: '<name>', config: '<configMap>/<key>'</code></dt><dd>Can be specified multiple times </dd> <dd>Defines an environment variable whose value is defined by the given <a class="reference external" href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/">ConfigMap</a> and key. </dd> <dt><code class="docutils literal notranslate">env: '<name>', fieldPath: '<fieldPath>'</code></dt><dd><div class="versionadded"> New in version 21.09.1-edge. </div> </dd> <dd>Can be specified multiple times </dd> <dd>Defines an environment variable whose value is defined by the given <a class="reference external" href="https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#use-pod-fields-as-values-for-environment-variables">field path</a> value. </dd> <dd>For example, the following pod option: <div class="highlight-groovy notranslate"><div class="highlight"><pre>pod = [env: 'MY_NODE_NAME', fieldPath: 'spec.nodeName'] </pre></div> </div> Maps to the following pod spec: <div class="highlight-yaml notranslate"><div class="highlight"><pre>env: - name: MY_NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName </pre></div> </div> </dd> <dt><code class="docutils literal notranslate">env: '<name>', secret: '<secret>/<key>'</code></dt><dd>Can be specified multiple times </dd> <dd>Defines an environment variable whose value is defined by the given <a class="reference external" href="https://kubernetes.io/docs/concepts/configuration/secret/">Secret</a> and key. </dd> <dt><code class="docutils literal notranslate">env: '<name>', value: '<value>'</code></dt><dd>Can be specified multiple times </dd> <dd>Defines an environment variable with the given name and value. </dd> <dt><code class="docutils literal notranslate">hostPath: '/host/absolute/path', mountPath: '</pod/absolute/path>'</code></dt><dd><div class="versionadded"> New in version 23.10.0. </div> </dd> <dd>Can be specified multiple times </dd> <dd>Allows creating <a class="reference external" href="https://kubernetes.io/docs/concepts/storage/volumes/#hostpath">hostPath</a> volume and access it with the specified <code class="docutils literal notranslate">mountPath</code> in the pod. </dd> <dt><code class="docutils literal notranslate">imagePullPolicy: 'IfNotPresent' | 'Always' | 'Never'</code></dt><dd>Specifies the <a class="reference external" href="https://kubernetes.io/docs/concepts/containers/images/#image-pull-policy">image pull policy</a> used by the pod to pull the container image. </dd> <dt><code class="docutils literal notranslate">imagePullSecret: '<name>'</code></dt><dd>Specifies the <a class="reference external" href="https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod">image pull secret</a> used to access a private container image registry. </dd> <dt><code class="docutils literal notranslate">label: '<name>', value: '<value>'</code></dt><dd>Can be specified multiple times </dd> <dd>Defines a pod <a class="reference external" href="https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/">label</a> with the given name and value. </dd> <dt><code class="docutils literal notranslate">nodeSelector: <config></code></dt><dd>Specifies the <a class="reference external" href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector">node selector</a> with the given configuration. </dd> <dd>The configuration can be a map or a string: <div class="highlight-groovy notranslate"><div class="highlight"><pre>// map pod = [nodeSelector: [disktype: 'ssd', cpu: 'intel']] // string pod = [nodeSelector: 'disktype=ssd,cpu=intel'] </pre></div> </div> </dd> <dt><code class="docutils literal notranslate">priorityClassName: '<name>'</code></dt><dd><div class="versionadded"> New in version 22.01.0-edge. </div> </dd> <dd>Specifies the <a class="reference external" href="https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/">priority class name</a> for pods. </dd> <dt><code class="docutils literal notranslate">privileged: true | false</code></dt><dd><div class="versionadded"> New in version 22.05.0-edge. </div> </dd> <dd>Specifies whether the pod should run as a privileged container (default: <code class="docutils literal notranslate">false</code>). </dd> <dt><code class="docutils literal notranslate">runAsUser: '<uid>'</code></dt><dd>Specifies the user ID with which to run the container. Shortcut for the <code class="docutils literal notranslate">securityContext</code> option. </dd> <dt><code class="docutils literal notranslate">schedulerName: '<name>'</code></dt><dd>Specifies which <a class="reference external" href="https://kubernetes.io/docs/tasks/extend-kubernetes/configure-multiple-schedulers/#specify-schedulers-for-pods">scheduler</a> is used to schedule the container. </dd> <dt><code class="docutils literal notranslate">secret: '<secret>/<key>', mountPath: '</absolute/path>'</code></dt><dd>Can be specified multiple times </dd> <dd>Mounts a <a class="reference external" href="https://kubernetes.io/docs/concepts/configuration/secret/">Secret</a> with name and optional key to the given path. If the key is omitted, the path is interpreted as a directory and all entries in the <code class="docutils literal notranslate">Secret</code> are exposed in that path. </dd> <dt><code class="docutils literal notranslate">securityContext: <config></code></dt><dd>Specifies the pod <a class="reference external" href="https://kubernetes.io/docs/tasks/configure-pod-container/security-context/">security context</a> with the given configuration. </dd> <dt><code class="docutils literal notranslate">toleration: <config></code></dt><dd><div class="versionadded"> New in version 22.04.0. </div> </dd> <dd>Can be specified multiple times </dd> <dd>Specifies the pod <a class="reference external" href="https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/">toleration</a> with the given configuration. </dd> <dd>The configuration should be a map corresponding to a single toleration rule. For example, the following pod options: <div class="highlight-groovy notranslate"><div class="highlight"><pre>pod = [ [toleration: [key: 'key1', operator: 'Equal', value: 'value1', effect: 'NoSchedule']], [toleration: [key: 'key1', operator: 'Exists', effect: 'NoSchedule']], ] </pre></div> </div> Maps to the following pod spec: <div class="highlight-yaml notranslate"><div class="highlight"><pre>tolerations: - key: "key1" operator: "Equal" value: "value1" effect: "NoSchedule" - key: "key1" operator: "Exists" effect: "NoSchedule" </pre></div> </div> </dd> <dt><code class="docutils literal notranslate">ttlSecondsAfterFinished</code></dt><dd><div class="versionadded"> New in version 24.02.0-edge. </div> </dd> <dd>Specifies the <a class="reference external" href="https://kubernetes.io/docs/concepts/workloads/controllers/job/#ttl-mechanism-for-finished-jobs">TTL mechanism</a> for finished jobs in seconds. Applies to both successful and failed jobs. </dd> <dt><code class="docutils literal notranslate">volumeClaim: '<name>', mountPath: '</absolute/path>' [, subPath: '<path>', readOnly: true | false]</code></dt><dd>Can be specified multiple times </dd> <dd>Mounts a <a class="reference external" href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/">Persistent volume claim</a> with the given name to the given path. </dd> <dd>The <code class="docutils literal notranslate">subPath</code> option can be used to mount a sub-directory of the volume instead of its root. </dd> <dd>The <code class="docutils literal notranslate">readOnly</code> option can be used to mount the volume as read-only (default: <code class="docutils literal notranslate">false</code>) </dd> </dl> </section> <section id="publishdir"> <h3>publishDir<a class="headerlink" href="#publishdir" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">publishDir</code> directive allows you to publish the process output files to a specified folder. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process foo { publishDir '/data/chunks' output: path 'chunk_*' script: """ printf 'Hola' | split -b 1 - chunk_ """ } </pre></div> </div> The above example splits the string <code class="docutils literal notranslate">Hola</code> into file chunks of a single byte. When complete the <code class="docutils literal notranslate">chunk_*</code> output files are published into the <code class="docutils literal notranslate">/data/chunks</code> folder. <div class="admonition note"> Note Only files that match the declaration in the <code class="docutils literal notranslate">output</code> block are published, not all the outputs of the process. </div> <div class="admonition tip"> Tip The <code class="docutils literal notranslate">publishDir</code> directive can be specified more than once in order to publish output files to different target directories based on different rules. </div> By default files are published to the target folder creating a symbolic link for each process output that links the file produced into the process working directory. This behavior can be modified using the <code class="docutils literal notranslate">mode</code> option, for example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process foo { publishDir '/data/chunks', mode: 'copy', overwrite: false output: path 'chunk_*' script: """ printf 'Hola' | split -b 1 - chunk_ """ } </pre></div> </div> <div class="admonition warning"> Warning Files are copied into the specified directory in an asynchronous manner, so they may not be immediately available in the publish directory at the end of the process execution. For this reason, downstream processes should not try to access output files through the publish directory, but through channels. </div> Available options: <dl class="simple myst"> <dt><code class="docutils literal notranslate">contentType</code></dt><dd><div class="versionadded"> New in version 22.10.0. </div> </dd> <dd>Experimental: currently only supported for S3. </dd> <dd>Allow specifying the media content type of the published file a.k.a. <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_Types">MIME type</a>. If set to <code class="docutils literal notranslate">true</code>, the content type is inferred from the file extension (default: <code class="docutils literal notranslate">false</code>). </dd> <dt><code class="docutils literal notranslate">enabled</code></dt><dd>Enable or disable the publish rule depending on the boolean value specified (default: <code class="docutils literal notranslate">true</code>). </dd> <dt><code class="docutils literal notranslate">failOnError</code></dt><dd><div class="versionchanged"> Changed in version 24.03.0-edge: The default value was changed from <code class="docutils literal notranslate">false</code> to <code class="docutils literal notranslate">true</code> </div> </dd> <dd>When <code class="docutils literal notranslate">true</code> abort the execution if some file can’t be published to the specified target directory or bucket for any cause (default: <code class="docutils literal notranslate">true</code>) </dd> <dt><code class="docutils literal notranslate">mode</code></dt><dd>The file publishing method. Can be one of the following values: <ul class="simple"> <li><code class="docutils literal notranslate">'copy'</code>: Copies the output files into the publish directory.</li> <li><code class="docutils literal notranslate">'copyNoFollow'</code>: Copies the output files into the publish directory without following symlinks ie. copies the links themselves.</li> <li><code class="docutils literal notranslate">'link'</code>: Creates a hard link in the publish directory for each output file.</li> <li><code class="docutils literal notranslate">'move'</code>: Moves the output files into the publish directory. Note: this is only supposed to be used for a terminal process i.e. a process whose output is not consumed by any other downstream process.</li> <li><code class="docutils literal notranslate">'rellink'</code>: Creates a relative symbolic link in the publish directory for each output file.</li> <li><code class="docutils literal notranslate">'symlink'</code>: Creates an absolute symbolic link in the publish directory for each output file (default).</li> </ul> </dd> <dt><code class="docutils literal notranslate">overwrite</code></dt><dd>When <code class="docutils literal notranslate">true</code> any existing file in the specified folder will be overridden (default: <code class="docutils literal notranslate">true</code> during normal pipeline execution and <code class="docutils literal notranslate">false</code> when pipeline execution is <code class="docutils literal notranslate">resumed</code>). </dd> <dt><code class="docutils literal notranslate">path</code></dt><dd>Specifies the directory where files need to be published. Note: the syntax <code class="docutils literal notranslate">publishDir '/some/dir'</code> is a shortcut for <code class="docutils literal notranslate">publishDir path: '/some/dir'</code>. </dd> <dt><code class="docutils literal notranslate">pattern</code></dt><dd>Specifies a [glob][glob] file pattern that selects which files to publish from the overall set of output files. </dd> <dt><code class="docutils literal notranslate">saveAs</code></dt><dd>A closure which, given the name of the file being published, returns the actual file name or a full path where the file is required to be stored. This can be used to rename or change the destination directory of the published files dynamically by using a custom strategy. Return the value <code class="docutils literal notranslate">null</code> from the closure to not publish a file. This is useful when the process has multiple output files, but you want to publish only some of them. </dd> <dt><code class="docutils literal notranslate">storageClass</code></dt><dd><div class="versionadded"> New in version 22.12.0-edge. </div> </dd> <dd>Experimental: currently only supported for S3. </dd> <dd>Allow specifying the storage class to be used for the published file. </dd> <dt><code class="docutils literal notranslate">tags</code></dt><dd><div class="versionadded"> New in version 21.12.0-edge. </div> </dd> <dd>Experimental: currently only supported for S3. </dd> <dd>Allow the association of arbitrary tags with the published file e.g. <code class="docutils literal notranslate">tags: [FOO: 'Hello world']</code>. </dd> </dl> </section> <section id="queue"> <h3>queue<a class="headerlink" href="#queue" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">queue</code> directive allows you to set the <code class="docutils literal notranslate">queue</code> where jobs are scheduled when using a grid based executor in your pipeline. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process grid_job { queue 'long' executor 'sge' script: """ your_command --here """ } </pre></div> </div> <div class="admonition tip"> Tip Some grid executors support multiple queue names as a comma-separated list: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>queue 'short,long,cn-el6' </pre></div> </div> However, this does not generally apply to other executors such as AWS Batch, Azure Batch, Google Batch. </div> <div class="admonition note"> Note This directive is only used by certain executors. Refer to the <a class="reference internal" href="../executor.html#executor-page">Executors</a> page to see which executors support this directive. </div> </section> <section id="resourcelabels"> <h3>resourceLabels<a class="headerlink" href="#resourcelabels" title="Permalink to this heading"></a></h3> <div class="versionadded"> New in version 22.09.1-edge. </div> The <code class="docutils literal notranslate">resourceLabels</code> directive allows you to specify custom name-value pairs that Nextflow applies to the computing resource used to carry out the process execution. Resource labels can be specified using the syntax shown below: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process my_task { resourceLabels region: 'some-region', user: 'some-username' script: """ your_command --here """ } </pre></div> </div> The limits and the syntax of the corresponding cloud provider should be taken into consideration when using resource labels. Resource labels are currently supported by the following executors: <ul class="simple"> <li><a class="reference internal" href="../executor.html#awsbatch-executor">AWS Batch</a></li> <li><a class="reference internal" href="../executor.html#azurebatch-executor">Azure Batch</a></li> <li><a class="reference internal" href="../executor.html#google-batch-executor">Google Cloud Batch</a></li> <li><a class="reference internal" href="../executor.html#google-lifesciences-executor">Google Life Sciences</a></li> <li><a class="reference internal" href="../executor.html#k8s-executor">Kubernetes</a></li> </ul> <div class="versionadded"> New in version 23.09.0-edge: Resource labels are supported for Azure Batch when using automatic pool creation. Resource labels in Azure are added to pools, rather than jobs, in order to facilitate cost analysis. A new pool will be created for each new set of resource labels, therefore it is recommended to also set <code class="docutils literal notranslate">azure.batch.deletePoolsOnCompletion = true</code> when using process-specific resource labels. </div> See also: <a class="reference internal" href="#label">label</a> </section> <section id="resourcelimits"> <h3>resourceLimits<a class="headerlink" href="#resourcelimits" title="Permalink to this heading"></a></h3> <div class="versionadded"> New in version 24.04.0. </div> The <code class="docutils literal notranslate">resourceLimits</code> directive allows you to specify environment-specific limits for task resource requests. Resource limits can be specified in a process as follows: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process my_task { resourceLimits cpus: 24, memory: 768.GB, time: 72.h script: """ your_command --here """ } </pre></div> </div> Or in the Nextflow configuration: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process { resourceLimits = [ cpus: 24, memory: 768.GB, time: 72.h ] } </pre></div> </div> Resource limits can be defined for the following directives: <ul class="simple"> <li><a class="reference internal" href="#cpus">cpus</a></li> <li><a class="reference internal" href="#disk">disk</a></li> <li><a class="reference internal" href="#memory">memory</a></li> <li><a class="reference internal" href="#time">time</a></li> </ul> Resource limits are a useful way to specify environment-specific limits alongside tasks with <a class="reference internal" href="../process.html#dynamic-task-resources">dynamic resources</a>. Normally, if a task requests more resources than can be provisioned (e.g. a task requests 32 cores but the largest node in the cluster has 24), the task will either fail or cause the pipeline to hang forever as it will never be scheduled. If the <code class="docutils literal notranslate">resourceLimits</code> directive is defined with these limits, the task resources will be automatically reduced to comply with these limits before the job is submitted. </section> <section id="scratch"> <h3>scratch<a class="headerlink" href="#scratch" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">scratch</code> directive allows you to execute the process in a temporary folder that is local to the execution node. This is useful when your pipeline is launched by using a grid executor, because it allows you to decrease the NFS overhead by running the pipeline processes in a temporary directory in the local disk of the actual execution node. Only the files declared as output in the process definition will be copied in the pipeline working area. In its basic form simply specify <code class="docutils literal notranslate">true</code> at the directive value, as shown below: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process simpleTask { scratch true output: path 'data_out' script: """ your_command --here """ } </pre></div> </div> By doing this, it tries to execute the script in the directory defined by the variable <code class="docutils literal notranslate">$TMPDIR</code> in the execution node. If this variable does not exist, it will create a new temporary directory by using the Linux command <code class="docutils literal notranslate">mktemp</code>. <div class="admonition note"> Note Cloud-based executors use <code class="docutils literal notranslate">scratch = true</code> by default, since the work directory resides in object storage. </div> The following values are supported: <dl class="simple myst"> <dt><code class="docutils literal notranslate">false</code></dt><dd>Do not use a scratch directory. </dd> <dt><code class="docutils literal notranslate">true</code></dt><dd>Create a scratch directory in the directory defined by the <code class="docutils literal notranslate">$TMPDIR</code> environment variable, or <code class="docutils literal notranslate">$(mktemp /tmp)</code> if <code class="docutils literal notranslate">$TMPDIR</code> is not set. </dd> <dt><code class="docutils literal notranslate">'$YOUR_VAR'</code></dt><dd>Create a scratch directory in the directory defined by the given environment variable, or <code class="docutils literal notranslate">$(mktemp /tmp)</code> if that variable is not set. The value must use single quotes, otherwise the environment variable will be evaluated in the pipeline script context. </dd> <dt><code class="docutils literal notranslate">'/my/tmp/path'</code></dt><dd>Create a scratch directory in the specified directory. </dd> <dt><code class="docutils literal notranslate">'ram-disk'</code></dt><dd>Create a scratch directory in the RAM disk <code class="docutils literal notranslate">/dev/shm/</code>. </dd> </dl> </section> <section id="shell"> <h3>shell<a class="headerlink" href="#shell" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">shell</code> directive allows you to define a custom shell command for process scripts. By default, script blocks are executed with <code class="docutils literal notranslate">/bin/bash -ue</code>. <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process doMoreThings { shell '/bin/bash', '-euo', 'pipefail' script: """ your_command --here """ } </pre></div> </div> The same directive could be specified in your Nextflow configuration as follows: <div class="highlight-groovy notranslate"><div class="highlight"><pre>process.shell = ['/bin/bash', '-euo', 'pipefail'] </pre></div> </div> </section> <section id="spack"> <h3>spack<a class="headerlink" href="#spack" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">spack</code> directive allows for the definition of the process dependencies using the <a class="reference external" href="https://spack.io">Spack</a> package manager. Nextflow automatically sets up an environment for the given package names listed by in the <code class="docutils literal notranslate">spack</code> directive. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process foo { spack 'bwa@0.7.15' script: """ your_command --here """ } </pre></div> </div> Multiple packages can be specified separating them with a blank space, e.g. <code class="docutils literal notranslate">bwa@0.7.15 fastqc@0.11.5</code>. The <code class="docutils literal notranslate">spack</code> directive also allows the specification of a Spack environment file path or the path of an existing environment directory. See the <a class="reference internal" href="../spack.html#spack-page">Spack environments</a> page for further details. </section> <section id="stageinmode"> <h3>stageInMode<a class="headerlink" href="#stageinmode" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">stageInMode</code> directive defines how input files are staged into the process work directory. The following values are allowed: <dl class="simple myst"> <dt><code class="docutils literal notranslate">'copy'</code></dt><dd>Input files are staged in the process work directory by creating a copy. </dd> <dt><code class="docutils literal notranslate">'link'</code></dt><dd>Input files are staged in the process work directory by creating a hard link for each of them. </dd> <dt><code class="docutils literal notranslate">'rellink'</code></dt><dd>Input files are staged in the process work directory by creating a symbolic link with a relative path for each of them. </dd> <dt><code class="docutils literal notranslate">'symlink'</code></dt><dd>Input files are staged in the process work directory by creating a symbolic link with an absolute path for each of them (default). </dd> </dl> </section> <section id="stageoutmode"> <h3>stageOutMode<a class="headerlink" href="#stageoutmode" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">stageOutMode</code> directive defines how output files are staged out from the scratch directory to the process work directory. The following values are allowed: <dl class="simple myst"> <dt><code class="docutils literal notranslate">'copy'</code></dt><dd>Output files are copied from the scratch directory to the work directory. </dd> <dt><code class="docutils literal notranslate">'fcp'</code></dt><dd><div class="versionadded"> New in version 23.02.0-edge. </div> </dd> <dd>Output files are copied from the scratch directory to the work directory by using the <a class="reference external" href="https://github.com/Svetlitski/fcp">fcp</a> utility (note: it must be available in your cluster computing nodes). </dd> <dt><code class="docutils literal notranslate">'move'</code></dt><dd>Output files are moved from the scratch directory to the work directory. </dd> <dt><code class="docutils literal notranslate">'rclone'</code></dt><dd><div class="versionadded"> New in version 23.01.0-edge. </div> </dd> <dd>Output files are copied from the scratch directory to the work directory by using the <a class="reference external" href="https://rclone.org">rclone</a> utility (note: it must be available in your cluster computing nodes). </dd> <dt><code class="docutils literal notranslate">'rsync'</code></dt><dd>Output files are copied from the scratch directory to the work directory by using the <code class="docutils literal notranslate">rsync</code> utility. </dd> </dl> See also: <a class="reference internal" href="#scratch">scratch</a>. </section> <section id="storedir"> <h3>storeDir<a class="headerlink" href="#storedir" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">storeDir</code> directive allows you to define a directory that is used as a permanent cache for your process results. In more detail, it affects the process execution in two main ways: <ol class="arabic simple"> <li>The process is executed only if the files declared in the <code class="docutils literal notranslate">output</code> block do not exist in the directory specified by the <code class="docutils literal notranslate">storeDir</code> directive. When the files exist the process execution is skipped and these files are used as the actual process result.</li> <li>Whenever a process is successfully completed the files listed in the <code class="docutils literal notranslate">output</code> block are moved into the directory specified by the <code class="docutils literal notranslate">storeDir</code> directive.</li> </ol> The following example shows how to use the <code class="docutils literal notranslate">storeDir</code> directive to create a directory containing a BLAST database for each species specified by an input parameter: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process formatBlastDatabases { storeDir '/db/genomes' input: path species output: path "${dbName}.*" script: dbName = species.baseName """ makeblastdb -dbtype nucl -in ${species} -out ${dbName} """ } </pre></div> </div> <div class="admonition warning"> Warning If a process uses <code class="docutils literal notranslate">storeDir</code> and all of its outputs are optional, the process will always be skipped, even if the store directory is empty. This issue can be avoided by specifying at least one required file output. </div> <div class="admonition warning"> Warning The <code class="docutils literal notranslate">storeDir</code> directive should not be used to publish workflow outputs. Use the <a class="reference internal" href="#publishdir">publishDir</a> directive or the <a class="reference internal" href="../workflow.html#workflow-output-def">workflow output definition</a> instead. </div> </section> <section id="tag"> <h3>tag<a class="headerlink" href="#tag" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">tag</code> directive allows you to associate each process execution with a custom label, so that it will be easier to identify them in the log file or in the trace execution report. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process foo { tag "$code" input: val code script: """ echo $code """ } workflow { Channel.of('alpha', 'gamma', 'omega') | foo } </pre></div> </div> The above snippet will print a log similar to the following one, where process names contain the tag value: <div class="highlight-default notranslate"><div class="highlight"><pre>[6e/28919b] Submitted process > foo (alpha) [d2/1c6175] Submitted process > foo (gamma) [1c/3ef220] Submitted process > foo (omega) </pre></div> </div> See also <a class="reference internal" href="../reports.html#trace-report">Trace execution report</a> </section> <section id="time"> <h3>time<a class="headerlink" href="#time" title="Permalink to this heading"></a></h3> The <code class="docutils literal notranslate">time</code> directive allows you to define how long a process is allowed to run. For example: <div class="highlight-nextflow notranslate"><div class="highlight"><pre>process big_job { time '1h' script: """ your_command --here """ } </pre></div> </div> The following time unit suffixes can be used when specifying the duration value: <table class="docutils align-default"> <thead> <tr class="row-odd"><th class="head">Unit</th> <th class="head">Description</th> </tr> </thead> <tbody> <tr class="row-even"><td><code class="docutils literal notranslate">ms</code>, <code class="docutils literal notranslate">milli</code>, <code class="docutils literal notranslate">millis</code></td> <td>Milliseconds</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">s</code>, <code class="docutils literal notranslate">sec</code>, <code class="docutils literal notranslate">second</code>, <code class="docutils literal notranslate">seconds</code></td> <td>Seconds</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">m</code>, <code class="docutils literal notranslate">min</code>, <code class="docutils literal notranslate">minute</code>, <code class="docutils literal notranslate">minutes</code></td> <td>Minutes</td> </tr> <tr class="row-odd"><td><code class="docutils literal notranslate">h</code>, <code class="docutils literal notranslate">hour</code>, <code class="docutils literal notranslate">hours</code></td> <td>Hours</td> </tr> <tr class="row-even"><td><code class="docutils literal notranslate">d</code>, <code class="docutils literal notranslate">day</code>, <code class="docutils literal notranslate">days</code></td> <td>Days</td> </tr> </tbody> </table> Multiple units can be used in a single declaration, for example: <code class="docutils literal notranslate">'1day 6hours 3minutes 30seconds'</code> See <a class="reference internal" href="stdlib.html#stdlib-types-duration">Duration</a> for more information. <div class="admonition note"> Note This directive is only used by certain executors. Refer to the <a class="reference internal" href="../executor.html#executor-page">Executors</a> page to see which executors support this directive. </div> See also: <a class="reference internal" href="#cpus">cpus</a>, <a class="reference internal" href="#memory">memory</a>, <a class="reference internal" href="#queue">queue</a> and <a class="reference internal" href="../process.html#dynamic-task-resources">Dynamic task resources</a>. </section> </section> </section> </div> </div> <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer"> <a href="stdlib.html" class="btn btn-neutral float-left" title="Standard library" accesskey="p" rel="prev"> Previous</a> <a href="channel.html" class="btn btn-neutral float-right" title="Channel factories" accesskey="n" rel="next">Next </a> </div> <hr/> <div role="contentinfo"> © Copyright 2025, Seqera Labs, S.L. </div> Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. </footer> </div> </div> </section> </div> <script> jQuery(function () { SphinxRtdTheme.Navigation.enable(true); }); </script>  <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-TNCXSWG" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>  </body> </html>