CINXE.COM

<!DOCTYPE html> <html class="writer-html5" lang="en"> <head><script type="text/javascript" src="/_static/js/bundle-playback.js?v=HxkREWBo" charset="utf-8"></script> <script type="text/javascript" src="/_static/js/wombat.js?v=txqj7nKC" charset="utf-8"></script> <script>window.RufflePlayer=window.RufflePlayer||{};window.RufflePlayer.config={"autoplay":"on","unmuteOverlay":"hidden"};</script> <script type="text/javascript" src="/_static/js/ruffle/ruffle.js"></script> <script type="text/javascript"> __wm.init("https://web.archive.org/web"); __wm.wombat("https://nextflow.io/docs/latest/google.html","20210410212021","https://web.archive.org/","web","/_static/", "1618089621"); </script> <link rel="stylesheet" type="text/css" href="/_static/css/banner-styles.css?v=S1zqJCYt" /> <link rel="stylesheet" type="text/css" href="/_static/css/iconochive.css?v=3PDvdIFv" />  <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <title>Google Cloud — Nextflow 20.10.0 documentation</title> <link rel="stylesheet" href="/web/20210410212021cs_/https://nextflow.io/docs/latest/_static/css/theme.css" type="text/css"/> <link rel="stylesheet" href="/web/20210410212021cs_/https://nextflow.io/docs/latest/_static/pygments.css" type="text/css"/> <link rel="shortcut icon" href="/web/20210410212021im_/https://nextflow.io/docs/latest/_static/favicon.ico"/>  <script type="text/javascript" id="documentation_options" data-url_root="./" src="/web/20210410212021js_/https://nextflow.io/docs/latest/_static/documentation_options.js"></script> <script src="/web/20210410212021js_/https://nextflow.io/docs/latest/_static/jquery.js"></script> <script src="/web/20210410212021js_/https://nextflow.io/docs/latest/_static/underscore.js"></script> <script src="/web/20210410212021js_/https://nextflow.io/docs/latest/_static/doctools.js"></script> <script async="async" src="https://web.archive.org/web/20210410212021js_/https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script> <script type="text/javascript" src="/web/20210410212021js_/https://nextflow.io/docs/latest/_static/js/theme.js"></script> <link rel="index" title="Index" href="genindex.html"/> <link rel="search" title="Search" href="search.html"/> <link rel="next" title="Conda environments" href="conda.html"/> <link rel="prev" title="Amazon S3 storage" href="amazons3.html"/> </head> <body class="wy-body-for-nav"> <div class="wy-grid-for-nav"> <nav data-toggle="wy-nav-shift" class="wy-nav-side"> <div class="wy-side-scroll"> <div class="wy-side-nav-search"> <a href="index.html" class="icon icon-home"> Nextflow </a> <div role="search"> <form id="rtd-search-form" class="wy-form" action="search.html" method="get"> <input type="text" name="q" placeholder="Search docs"/> <input type="hidden" name="check_keywords" value="yes"/> <input type="hidden" name="area" value="default"/> </form> </div> </div> <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> <ul class="current"> <li class="toctree-l1"><a class="reference internal" href="getstarted.html">Get started</a></li> <li class="toctree-l1"><a class="reference internal" href="basic.html">Basic concepts</a></li> <li class="toctree-l1"><a class="reference internal" href="script.html">Nextflow scripting</a></li> <li class="toctree-l1"><a class="reference internal" href="process.html">Processes</a></li> <li class="toctree-l1"><a class="reference internal" href="channel.html">Channels</a></li> <li class="toctree-l1"><a class="reference internal" href="operator.html">Operators</a></li> <li class="toctree-l1"><a class="reference internal" href="executor.html">Executors</a></li> <li class="toctree-l1"><a class="reference internal" href="config.html">Configuration</a></li> <li class="toctree-l1"><a class="reference internal" href="dsl2.html">DSL 2</a></li> <li class="toctree-l1"><a class="reference internal" href="cli.html">Command line interface (CLI)</a></li> <li class="toctree-l1"><a class="reference internal" href="awscloud.html">Amazon Cloud</a></li> <li class="toctree-l1"><a class="reference internal" href="amazons3.html">Amazon S3 storage</a></li> <li class="toctree-l1 current"><a class="current reference internal" href="#">Google Cloud</a><ul> <li class="toctree-l2"><a class="reference internal" href="#requirements">Requirements</a><ul> <li class="toctree-l3"><a class="reference internal" href="#nextflow">Nextflow</a></li> <li class="toctree-l3"><a class="reference internal" href="#credentials">Credentials</a></li> </ul> </li> <li class="toctree-l2"><a class="reference internal" href="#cloud-life-sciences">Cloud Life Sciences</a><ul> <li class="toctree-l3"><a class="reference internal" href="#configuration">Configuration</a></li> <li class="toctree-l3"><a class="reference internal" href="#process-definition">Process definition</a></li> <li class="toctree-l3"><a class="reference internal" href="#pipeline-execution">Pipeline execution</a></li> <li class="toctree-l3"><a class="reference internal" href="#preemptible-instances">Preemptible instances</a></li> <li class="toctree-l3"><a class="reference internal" href="#hybrid-execution">Hybrid execution</a></li> <li class="toctree-l3"><a class="reference internal" href="#limitation">Limitation</a></li> <li class="toctree-l3"><a class="reference internal" href="#troubleshooting">Troubleshooting</a></li> </ul> </li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="conda.html">Conda environments</a></li> <li class="toctree-l1"><a class="reference internal" href="docker.html">Docker containers</a></li> <li class="toctree-l1"><a class="reference internal" href="shifter.html">Shifter Containers</a></li> <li class="toctree-l1"><a class="reference internal" href="singularity.html">Singularity containers</a></li> <li class="toctree-l1"><a class="reference internal" href="podman.html">Podman containers</a></li> <li class="toctree-l1"><a class="reference internal" href="ignite.html">Apache Ignite</a></li> <li class="toctree-l1"><a class="reference internal" href="kubernetes.html">Kubernetes</a></li> <li class="toctree-l1"><a class="reference internal" href="tracing.html">Tracing & visualisation</a></li> <li class="toctree-l1"><a class="reference internal" href="sharing.html">Pipeline sharing</a></li> <li class="toctree-l1"><a class="reference internal" href="metadata.html">Workflow introspection</a></li> <li class="toctree-l1"><a class="reference internal" href="mail.html">Mail & Notifications</a></li> </ul> </div> </div> </nav> <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> <nav class="wy-nav-top" aria-label="top navigation"> <i data-toggle="wy-nav-top" class="fa fa-bars"></i> <a href="index.html">Nextflow</a> </nav> <div class="wy-nav-content"> <div class="rst-content"> <div role="navigation" aria-label="breadcrumbs navigation"> <ul class="wy-breadcrumbs"> <li><a href="index.html" class="icon icon-home"></a> »</li> <li>Google Cloud</li> <li class="wy-breadcrumbs-aside"> </li> </ul> <hr/> </div> <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> <div itemprop="articleBody"> <div class="section" id="google-cloud"> <span id="google-page"></span><h1>Google Cloud<a class="headerlink" href="#google-cloud" title="Permalink to this headline">¶</a></h1> <div class="section" id="requirements"> <h2>Requirements<a class="headerlink" href="#requirements" title="Permalink to this headline">¶</a></h2> <div class="section" id="nextflow"> <h3>Nextflow<a class="headerlink" href="#nextflow" title="Permalink to this headline">¶</a></h3> <p>The support for Google Cloud requires Nextflow version <code class="docutils literal notranslate"><span class="pre">20.01.0</span></code>. To install it define the following variables in your system environment:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">export</span> <span class="n">NXF_VER</span><span class="o">=</span><span class="mf">20.01</span><span class="o">.</span><span class="mi">0</span> <span class="n">export</span> <span class="n">NXF_MODE</span><span class="o">=</span><span class="n">google</span> </pre></div> </div> </div> <div class="section" id="credentials"> <h3>Credentials<a class="headerlink" href="#credentials" title="Permalink to this headline">¶</a></h3> <p>To allow the deployment in the Google Cloud you need to configure the security credentials using a <em>Security account key</em> JSON file.</p> <p>Nextflow looks for this file using the <code class="docutils literal notranslate"><span class="pre">GOOGLE_APPLICATION_CREDENTIALS</span></code> variable that has to be defined in the launching environment.</p> <p>If you don’t have it, download the credentials file from the Google Cloud Console following these steps:</p> <ul class="simple"> <li><p>Open the <a class="reference external" href="https://web.archive.org/web/20210410212021/https://console.cloud.google.com/">Google Cloud Console</a></p></li> <li><p>Go to APIs & Services → Credentials</p></li> <li><p>Click on the <em>Create credentials</em> (blue) drop-down and choose <em>Service account key</em>, in the following page</p></li> <li><p>Select an existing <em>Service account</em> or create a new one if needed</p></li> <li><p>Select JSON as <em>Key type</em></p></li> <li><p>Click the <em>Create</em> button and download the JSON file giving a name of your choice e.g. <code class="docutils literal notranslate"><span class="pre">creds.json</span></code>.</p></li> </ul> <p>Finally define the following variable replacing the path in the example with the one of your credentials file just downloaded:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">export</span> <span class="n">GOOGLE_APPLICATION_CREDENTIALS</span><span class="o">=/</span><span class="n">path</span><span class="o">/</span><span class="n">your</span><span class="o">/</span><span class="n">file</span><span class="o">/</span><span class="n">creds</span><span class="o">.</span><span class="n">json</span> </pre></div> </div> </div> </div> <div class="section" id="cloud-life-sciences"> <span id="google-lifesciences"></span><h2>Cloud Life Sciences<a class="headerlink" href="#cloud-life-sciences" title="Permalink to this headline">¶</a></h2> <p><a class="reference external" href="https://web.archive.org/web/20210410212021/https://cloud.google.com/life-sciences/">Cloud Life Sciences</a> is a managed computing service that allows the execution of containerized workloads in the Google Cloud Platform infrastructure.</p> <p>Nextflow provides built-in support for Cloud Life Sciences API which allows the seamless deployment of a Nextflow pipeline in the cloud, offloading the process executions through the Google Cloud service.</p> <div class="admonition note"> <p class="admonition-title">Note</p> <p>This features requires Nextflow <code class="docutils literal notranslate"><span class="pre">20.01.0-edge</span></code> or later.</p> </div> <div class="admonition warning"> <p class="admonition-title">Warning</p> <p>This API works well for coarse-grained workloads i.e. long running jobs. It’s not suggested the use this feature for pipelines spawning many short lived tasks.</p> </div> <div class="section" id="configuration"> <span id="google-lifesciences-config"></span><h3>Configuration<a class="headerlink" href="#configuration" title="Permalink to this headline">¶</a></h3> <p>Make sure to have defined in your environment the <code class="docutils literal notranslate"><span class="pre">GOOGLE_APPLICATION_CREDENTIALS</span></code> variable. See the section <a class="reference internal" href="#requirements">Requirements</a> for details.</p> <div class="admonition tip"> <p class="admonition-title">Tip</p> <p>Make sure to have enabled Cloud Life Sciences API to use this feature. To learn how to enable it follow <a class="reference external" href="https://web.archive.org/web/20210410212021/https://cloud.google.com/life-sciences/docs/quickstart">this link</a>.</p> </div> <p>Create a <code class="docutils literal notranslate"><span class="pre">nextflow.config</span></code> file in the project root directory. The config must specify the following parameters:</p> <ul class="simple"> <li><p>Google Life Sciences as Nextflow executor i.e. <code class="docutils literal notranslate"><span class="pre">process.executor</span> <span class="pre">=</span> <span class="pre">'google-lifesciences'</span></code>.</p></li> <li><p>The Docker container images to be used to run pipeline tasks e.g. <code class="docutils literal notranslate"><span class="pre">process.container</span> <span class="pre">=</span> <span class="pre">'biocontainers/salmon:0.8.2--1'</span></code>.</p></li> <li><p>The Google Cloud <cite>project</cite> ID to run in e.g. <code class="docutils literal notranslate"><span class="pre">google.project</span> <span class="pre">=</span> <span class="pre">'rare-lattice-222412'</span></code>.</p></li> <li><p>The Google Cloud <cite>region</cite> or <cite>zone</cite>. This is where the Compute Engine VMs will be started. You need to specify either one, <strong>not</strong> both. Multiple regions or zones can be specified by separating them with a comma e.g. <code class="docutils literal notranslate"><span class="pre">google.zone</span> <span class="pre">=</span> <span class="pre">'us-central1-f,us-central-1-b'</span></code>.</p></li> </ul> <p>Example:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">process</span> <span class="p">{</span> <span class="n">executor</span> <span class="o">=</span> <span class="s1">'google-lifesciences'</span> <span class="n">container</span> <span class="o">=</span> <span class="s1">'your/container:latest'</span> <span class="p">}</span> <span class="n">google</span> <span class="p">{</span> <span class="n">project</span> <span class="o">=</span> <span class="s1">'your-project-id'</span> <span class="n">zone</span> <span class="o">=</span> <span class="s1">'europe-west1-b'</span> <span class="p">}</span> </pre></div> </div> <div class="admonition warning"> <p class="admonition-title">Warning</p> <p>Make sure to specify in the above setting the project ID not the project name.</p> </div> <div class="admonition note"> <p class="admonition-title">Note</p> <p>A container image must be specified to deploy the process execution. You can use a different Docker image for each process using one or more <a class="reference internal" href="config.html#config-process-selectors"><span class="std std-ref">Process selectors</span></a>.</p> </div> <p>The following configuration options are available:</p> <table class="docutils align-default"> <colgroup> <col style="width: 12%"/> <col style="width: 88%"/> </colgroup> <thead> <tr class="row-odd"><th class="head"><p>Name</p></th> <th class="head"><p>Description</p></th> </tr> </thead> <tbody> <tr class="row-even"><td><p>google.project</p></td> <td><p>The Google Project Id to use for the pipeline execution.</p></td> </tr> <tr class="row-odd"><td><p>google.region</p></td> <td><p>The Google <em>region</em> where the computation is executed in Compute Engine VMs. Multiple regions can be provided separating them by a comma. Do not specify if a zone is provided. See <a class="reference external" href="https://web.archive.org/web/20210410212021/https://cloud.google.com/compute/docs/regions-zones/">available Compute Engine regions and zones</a></p></td> </tr> <tr class="row-even"><td><p>google.zone</p></td> <td><p>The Google <em>zone</em> where the computation is executed in Compute Engine VMs. Multiple zones can be provided separating them by a comma. Do not specify if a region is provided. See <a class="reference external" href="https://web.archive.org/web/20210410212021/https://cloud.google.com/compute/docs/regions-zones/">available Compute Engine regions and zones</a></p></td> </tr> <tr class="row-odd"><td><p>google.location</p></td> <td><p>The Google <em>location</em> where the job executions are deployed to Cloud Life Sciences API. See <a class="reference external" href="https://web.archive.org/web/20210410212021/https://cloud.google.com/life-sciences/docs/concepts/locations">available Cloud Life Sciences API locations</a> (default: the same as the region or the zone specified).</p></td> </tr> <tr class="row-even"><td><p>google.enableRequesterPaysBuckets</p></td> <td><p>When <code class="docutils literal notranslate"><span class="pre">true</span></code> uses the configured Google project id as the billing project for storage access. This is required when accessing data from <em>requester pays enabled</em> buckets. See <a class="reference external" href="https://web.archive.org/web/20210410212021/https://cloud.google.com/storage/docs/requester-pays">Requester Pays on Google Cloud Storage documentation</a> (default: <code class="docutils literal notranslate"><span class="pre">false</span></code>)</p></td> </tr> <tr class="row-odd"><td><p>google.lifeSciences.cpuPlatform</p></td> <td><p>Set the minimum CPU Platform e.g. <cite>‘Intel Skylake’</cite>. See <a class="reference external" href="https://web.archive.org/web/20210410212021/https://cloud.google.com/compute/docs/instances/specify-min-cpu-platform#specifications">Specifying a minimum CPU Platform for VM instances</a> (default: none).</p></td> </tr> <tr class="row-even"><td><p>google.lifeSciences.bootDiskSize</p></td> <td><p>Set the size of the virtual machine boot disk e.g <cite>50.GB</cite> (default: none).</p></td> </tr> <tr class="row-odd"><td><p>google.lifeSciences.copyImage</p></td> <td><p>The container image run to copy input and output files. It must include the <code class="docutils literal notranslate"><span class="pre">gsutil</span></code> tool (default: <code class="docutils literal notranslate"><span class="pre">google/cloud-sdk:alpine</span></code>).</p></td> </tr> <tr class="row-even"><td><p>google.lifeSciences.debug</p></td> <td><p>When <code class="docutils literal notranslate"><span class="pre">true</span></code> copies the <cite>/google</cite> debug directory in that task bucket directory (default: <code class="docutils literal notranslate"><span class="pre">false</span></code>)</p></td> </tr> <tr class="row-odd"><td><p>google.lifeSciences.preemptible</p></td> <td><p>When <code class="docutils literal notranslate"><span class="pre">true</span></code> enables the usage of <em>preemptible</em> virtual machines or <code class="docutils literal notranslate"><span class="pre">false</span></code> otherwise (default: <code class="docutils literal notranslate"><span class="pre">true</span></code>)</p></td> </tr> <tr class="row-even"><td><p>google.lifeSciences.usePrivateAddress</p></td> <td><p>When <code class="docutils literal notranslate"><span class="pre">true</span></code> the VM will NOT be provided with a public IP address, and only contain an internal IP. If this option is enabled, the associated job can only load docker images from Google Container Registry, and the job executable cannot use external services other than Google APIs (default: <code class="docutils literal notranslate"><span class="pre">false</span></code>). Requires version <cite>20.03.0-edge</cite> or later.</p></td> </tr> <tr class="row-odd"><td><p>google.lifeSciences.sshDaemon</p></td> <td><p>When <code class="docutils literal notranslate"><span class="pre">true</span></code> runs SSH daemon in the VM carrying out the job to which it’s possible to connect for debugging purposes (default: <code class="docutils literal notranslate"><span class="pre">false</span></code>).</p></td> </tr> <tr class="row-even"><td><p>google.lifeSciences.sshImage</p></td> <td><p>The container image used to run the SSH daemon (default: <code class="docutils literal notranslate"><span class="pre">gcr.io/cloud-genomics-pipelines/tools</span></code>).</p></td> </tr> </tbody> </table> </div> <div class="section" id="process-definition"> <h3>Process definition<a class="headerlink" href="#process-definition" title="Permalink to this headline">¶</a></h3> <p>Processes can be defined as usual and by default the <code class="docutils literal notranslate"><span class="pre">cpus</span></code> and <code class="docutils literal notranslate"><span class="pre">memory</span></code> directives are used to instantiate a custom machine type with the specified compute resources. If <code class="docutils literal notranslate"><span class="pre">memory</span></code> is not specified, 1GB of memory is allocated per cpu. A persistent disk will be created with size corresponding to the <code class="docutils literal notranslate"><span class="pre">disk</span></code> directive. If <code class="docutils literal notranslate"><span class="pre">disk</span></code> is not specified, the instance default is chosen to ensure reasonable I/O performance.</p> <p>The process <code class="docutils literal notranslate"><span class="pre">machineType</span></code> directive may optionally be used to specify a predefined Google Compute Platform <a class="reference external" href="https://web.archive.org/web/20210410212021/https://cloud.google.com/compute/docs/machine-types">machine type</a> If specified, this value overrides the <code class="docutils literal notranslate"><span class="pre">cpus</span></code> and <code class="docutils literal notranslate"><span class="pre">memory</span></code> directives. If the <code class="docutils literal notranslate"><span class="pre">cpus</span></code> and <code class="docutils literal notranslate"><span class="pre">memory</span></code> directives are used, the values must comply with the allowed custom machine type <a class="reference external" href="https://web.archive.org/web/20210410212021/https://cloud.google.com/compute/docs/instances/creating-instance-with-custom-machine-type#specifications">specifications</a> . Extended memory is not directly supported, however high memory or cpu predefined instances may be utilized using the <code class="docutils literal notranslate"><span class="pre">machineType</span></code> directive</p> <p>Examples:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">process</span> <span class="n">custom_resources_task</span> <span class="p">{</span> <span class="n">cpus</span> <span class="mi">8</span> <span class="n">memory</span> <span class="s1">'40 GB'</span> <span class="n">disk</span> <span class="s1">'200 GB'</span> <span class="sd">"""</span> <span class="sd"> <Your script here></span> <span class="sd"> """</span> <span class="p">}</span> <span class="n">process</span> <span class="n">predefined_resources_task</span> <span class="p">{</span> <span class="n">machineType</span> <span class="s1">'n1-highmem-8'</span> <span class="sd">"""</span> <span class="sd"> <Your script here></span> <span class="sd"> """</span> <span class="p">}</span> </pre></div> </div> <div class="admonition note"> <p class="admonition-title">Note</p> <p>This feature requires Nextflow 19.07.0 or later.</p> </div> </div> <div class="section" id="pipeline-execution"> <h3>Pipeline execution<a class="headerlink" href="#pipeline-execution" title="Permalink to this headline">¶</a></h3> <p>The pipeline can be launched either in a local computer or a cloud instance. Pipeline input data can be stored either locally or in a Google Storage bucket.</p> <p>The pipeline execution must specify a Google Storage bucket where the workflow’s intermediate results are stored using the <code class="docutils literal notranslate"><span class="pre">-work-dir</span></code> command line options. For example:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">nextflow</span> <span class="n">run</span> <span class="o"><</span><span class="n">script</span> <span class="ow">or</span> <span class="n">project</span> <span class="n">name</span><span class="o">></span> <span class="o">-</span><span class="n">work</span><span class="o">-</span><span class="nb">dir</span> <span class="n">gs</span><span class="p">:</span><span class="o">//</span><span class="n">my</span><span class="o">-</span><span class="n">bucket</span><span class="o">/</span><span class="n">some</span><span class="o">/</span><span class="n">path</span> </pre></div> </div> <div class="admonition tip"> <p class="admonition-title">Tip</p> <p>Any input data <strong>not</strong> stored in a Google Storage bucket will automatically be transferred to the pipeline work bucket. Use this feature with caution being careful to avoid unnecessary data transfers.</p> </div> </div> <div class="section" id="preemptible-instances"> <h3>Preemptible instances<a class="headerlink" href="#preemptible-instances" title="Permalink to this headline">¶</a></h3> <p>Preemptible instances are supported adding the following setting in the Nextflow config file:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">google</span> <span class="p">{</span> <span class="n">lifeSciences</span><span class="o">.</span><span class="n">preemptible</span> <span class="o">=</span> <span class="n">true</span> <span class="p">}</span> </pre></div> </div> <p>Since this type of virtual machines can be retired by the provider before the job completion, it is advisable to add the following retry strategy to your config file to instruct Nextflow to automatically re-execute a job if the virtual machine was terminated preemptively:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span>process { errorStrategy = { task.exitStatus==14 ? 'retry' : 'terminate' } maxRetries = 5 } </pre></div> </div> </div> <div class="section" id="hybrid-execution"> <h3>Hybrid execution<a class="headerlink" href="#hybrid-execution" title="Permalink to this headline">¶</a></h3> <p>Nextflow allows the use of multiple executors in the same workflow application. This feature enables the deployment of hybrid workloads in which some jobs are executed in the local computer or local computing cluster and some other jobs are offloaded to Google Pipelines service.</p> <p>To enable this feature use one or more <a class="reference internal" href="config.html#config-process-selectors"><span class="std std-ref">Process selectors</span></a> in your Nextflow configuration file to apply the Google Pipelines <em>executor</em> only to a subset of processes in your workflow. For example:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">process</span> <span class="p">{</span> <span class="n">withLabel</span><span class="p">:</span> <span class="n">bigTask</span> <span class="p">{</span> <span class="n">executor</span> <span class="o">=</span> <span class="s1">'google-lifesciences'</span> <span class="n">container</span> <span class="o">=</span> <span class="s1">'my/image:tag'</span> <span class="p">}</span> <span class="p">}</span> <span class="n">google</span> <span class="p">{</span> <span class="n">project</span> <span class="o">=</span> <span class="s1">'your-project-id'</span> <span class="n">zone</span> <span class="o">=</span> <span class="s1">'europe-west1-b'</span> <span class="p">}</span> </pre></div> </div> <p>Then deploy the workflow execution using the <code class="docutils literal notranslate"><span class="pre">-bucket-dir</span></code> to specify a Google Storage path for the jobs computed by the Google Pipeline service and, optionally, the <code class="docutils literal notranslate"><span class="pre">-work-dir</span></code> to specify the local storage for the jobs computed locally:</p> <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">nextflow</span> <span class="n">run</span> <span class="o"><</span><span class="n">script</span> <span class="ow">or</span> <span class="n">project</span> <span class="n">name</span><span class="o">></span> <span class="o">-</span><span class="n">bucket</span><span class="o">-</span><span class="nb">dir</span> <span class="n">gs</span><span class="p">:</span><span class="o">//</span><span class="n">my</span><span class="o">-</span><span class="n">bucket</span><span class="o">/</span><span class="n">some</span><span class="o">/</span><span class="n">path</span> </pre></div> </div> <div class="admonition warning"> <p class="admonition-title">Warning</p> <p>The Google Storage path needs to contain at least sub-directory. Don’t use only the bucket name e.g. <code class="docutils literal notranslate"><span class="pre">gs://my-bucket</span></code>.</p> </div> </div> <div class="section" id="limitation"> <h3>Limitation<a class="headerlink" href="#limitation" title="Permalink to this headline">¶</a></h3> <ul class="simple"> <li><p>Currently it’s not possible to specify a disk type different from the default one assigned by the service depending on the chosen instance type.</p></li> </ul> </div> <div class="section" id="troubleshooting"> <h3>Troubleshooting<a class="headerlink" href="#troubleshooting" title="Permalink to this headline">¶</a></h3> <ul class="simple"> <li><p>Make sure to have enabled Compute Engine API, Life Sciences API and Cloud Storage Service in the <a class="reference external" href="https://web.archive.org/web/20210410212021/https://console.cloud.google.com/apis/dashboard">APIs & Services Dashboard</a> page.</p></li> <li><p>Make sure to have enough compute resources to run your pipeline in your project <a class="reference external" href="https://web.archive.org/web/20210410212021/https://console.cloud.google.com/iam-admin/quotas">Quotas</a> (i.e. Compute Engine CPUs, Compute Engine Persistent Disk, Compute Engine In-use IP addresses, etc).</p></li> <li><p>Make sure your security credentials allows you to access any Google Storage bucket where input data and temporary files are stored.</p></li> <li><p>Check the directory <code class="docutils literal notranslate"><span class="pre">google/</span></code> created in the task work directory (in the bucket storage) created when on job failure and containing useful information of the job execution. The creation can be enabled as default setting the option <code class="docutils literal notranslate"><span class="pre">google.lifeSciences.debug</span> <span class="pre">=</span> <span class="pre">true</span></code> in the Nextflow config file</p></li> <li><p>Enable the optional SSH daemon in the job VM using the option <code class="docutils literal notranslate"><span class="pre">google.lifeSciences.sshDaemon</span> <span class="pre">=</span> <span class="pre">true</span></code></p></li> <li><p>Make sure you are choosing a <cite>location</cite> where <a class="reference external" href="https://web.archive.org/web/20210410212021/https://cloud.google.com/life-sciences/docs/concepts/locations">Cloud Life Sciences API is available</a>, and a <cite>region</cite> or <cite>zone</cite> where <a class="reference external" href="https://web.archive.org/web/20210410212021/https://cloud.google.com/compute/docs/regions-zones/">Compute Engine is available</a>.</p></li> </ul> </div> </div> </div> </div> </div> <footer> <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> <a href="conda.html" class="btn btn-neutral float-right" title="Conda environments" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a> <a href="amazons3.html" class="btn btn-neutral float-left" title="Amazon S3 storage" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a> </div> <hr/> <div role="contentinfo"> <p> © Copyright 2020, Seqera Labs. 2013-2019, Centre for Genomic Regulation (CRG). </p> </div> Built with <a href="https://web.archive.org/web/20210410212021/https://www.sphinx-doc.org/">Sphinx</a> using a <a href="https://web.archive.org/web/20210410212021/https://github.com/readthedocs/sphinx_rtd_theme">theme</a> provided by <a href="https://web.archive.org/web/20210410212021/https://readthedocs.org/">Read the Docs</a>. </footer> </div> </div> </section> </div> <script type="text/javascript"> jQuery(function () { SphinxRtdTheme.Navigation.enable(true); }); </script>  <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://web.archive.org/web/20210410212021/https://www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-364526-10', 'auto'); ga('send', 'pageview'); </script> </body> </html>