CINXE.COM

Apache Tez – Install and Deployment Instructions

<!DOCTYPE html> <!-- | Generated by Apache Maven Doxia Site Renderer 1.11.1 from src/site/markdown/install.md at 2024-12-23 | Rendered using Apache Maven Fluido Skin 1.9 --> <html xmlns="http://www.w3.org/1999/xhtml" lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <meta name="generator" content="Apache Maven Doxia Site Renderer 1.11.1" /> <title>Apache Tez &#x2013; Install and Deployment Instructions</title> <link rel="stylesheet" href="./css/apache-maven-fluido-1.9.min.css" /> <link rel="stylesheet" href="./css/site.css" /> <link rel="stylesheet" href="./css/print.css" media="print" /> <script src="./js/apache-maven-fluido-1.9.min.js"></script> <script> var _paq = window._paq = window._paq || []; /* tracker methods like "setCustomDimension" should be called before "trackPageView" */ _paq.push(["setDoNotTrack", true]); _paq.push(["disableCookies"]); _paq.push(['trackPageView']); _paq.push(['enableLinkTracking']); (function() { var u="https://analytics.apache.org/"; _paq.push(['setTrackerUrl', u+'matomo.php']); _paq.push(['setSiteId', '63']); var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s); })(); </script> <style> .well.sidebar-nav { background-color: #fff; } a.externalLink[href^=http] { background-image: none; padding-right: 0; } body.topBarEnabled { padding-top: 40px; } #leftColumn .nav-list .active a { background-color: #a0a0a0; } .nav-list .active a:hover { background-color: #a0a0a0; } </style> </head> <body class="topBarEnabled"> <header id="topbar" class="navbar navbar-fixed-top "> <div class="navbar-inner"> <div class="container-fluid"> <a data-target=".nav-collapse" data-toggle="collapse" class="btn btn-navbar"> <span class="icon-bar"></span> <span class="icon-bar"></span> <span class="icon-bar"></span> </a> <ul class="nav"> <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Getting Started <b class="caret"></b></a> <ul class="dropdown-menu"> <li><a href="index.html" title="Overview">Overview</a></li> <li><a href="talks.html" title="Talks and Meetup Recordings">Talks and Meetup Recordings</a></li> <li><a href="https://www.apache.org/licenses/LICENSE-2.0.txt" title="Project License">Project License</a></li> </ul> </li> <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation <b class="caret"></b></a> <ul class="dropdown-menu"> <li><a href="install.html" title="Install Guide">Install Guide</a></li> <li><a href="localmode.html" title="Local Mode">Local Mode</a></li> <li><a href="tez-ui.html" title="Tez UI">Tez UI</a></li> <li><a href="shuffle-handler.html" title="Tez Shuffle Handler">Tez Shuffle Handler</a></li> <li><a href="user_guides.html" title="User Guides">User Guides</a></li> </ul> </li> <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Community <b class="caret"></b></a> <ul class="dropdown-menu"> <li><a href="mail-lists.html" title="Mailing Lists">Mailing Lists</a></li> <li><a href="issue-tracking.html" title="Issue Tracking">Issue Tracking</a></li> <li><a href="team-list.html" title="Project Team">Project Team</a></li> <li><a href="by-laws.html" title="Project By-Laws">Project By-Laws</a></li> <li><a href="https://www.meetup.com/Apache-Tez-User-Group" title="User Meetup Group">User Meetup Group</a></li> <li><a href="https://cwiki.apache.org/confluence/display/TEZ" title="Tez Wiki">Tez Wiki</a></li> </ul> </li> <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Download Apache TEZ庐 Releases <b class="caret"></b></a> <ul class="dropdown-menu"> <li><a href="releases/apache-tez-0-8-5.html" title="0.8.5">0.8.5</a></li> <li><a href="releases/apache-tez-0-9-2.html" title="0.9.2">0.9.2</a></li> <li><a href="releases/apache-tez-0-10-3.html" title="0.10.3">0.10.3</a></li> <li><a href="releases/apache-tez-0-10-4.html" title="0.10.4">0.10.4</a></li> <li><a href="releases/index.html" title="All Releases">All Releases</a></li> </ul> </li> <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Contribute <b class="caret"></b></a> <ul class="dropdown-menu"> <li><a href="https://issues.apache.org/jira/browse/TEZ" title="JIRA">JIRA</a></li> <li><a href="https://gitbox.apache.org/repos/asf/tez.git" title="Git (Read/Write)">Git (Read/Write)</a></li> <li><a href="git://git.apache.org/tez.git" title="Git (Read-only)">Git (Read-only)</a></li> <li><a href="https://cwiki.apache.org/confluence/display/TEZ/How+to+Contribute+to+Tez" title="How to Contribute">How to Contribute</a></li> </ul> </li> <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">ASF <b class="caret"></b></a> <ul class="dropdown-menu"> <li><a href="https://www.apache.org/licenses/" title="License">License</a></li> <li><a href="https://www.apache.org/foundation/sponsorship.html" title="Donate">Donate</a></li> <li><a href="https://www.apache.org/foundation/thanks.html" title="Thanks">Thanks</a></li> <li><a href="https://www.apache.org/security/" title="Security">Security</a></li> </ul> </li> </ul> </div> </div> </header> <div class="container-fluid"> <header> <div id="banner"> <div class="pull-left"><a href="./" id="bannerLeft"><img src="images/ApacheTezLogo_lowres.png" alt="Apache Tez" width="25%" height="25%"/></a></div> <div class="pull-right"><a href="https://www.apache.org/" id="bannerRight"><img src="https://apache.org/foundation/press/kit/asf_logo_wide.png" alt="Apache Software Foundation"/></a></div> <div class="clear"><hr/></div> </div> <div id="breadcrumbs"> <ul class="breadcrumb"> <li class=""><a href="./" title="Apache TEZ庐">Apache TEZ庐</a><span class="divider">/</span></li> <li class="active ">Install and Deployment Instructions</li> <li id="publishDate" class="pull-right">Last Published: 2024-12-23</li> </ul> </div> </header> <div class="row-fluid"> <header id="leftColumn" class="span2"> <nav class="well sidebar-nav"> <ul class="nav nav-list"> <li class="nav-header">Getting Started</li> <li><a href="index.html" title="Overview"><span class="none"></span>Overview</a></li> <li><a href="talks.html" title="Talks and Meetup Recordings"><span class="none"></span>Talks and Meetup Recordings</a></li> <li><a href="https://www.apache.org/licenses/LICENSE-2.0.txt" class="externalLink" title="Project License"><span class="none"></span>Project License</a></li> <li class="nav-header">Documentation</li> <li class="active"><a href="#"><span class="none"></span>Install Guide</a></li> <li><a href="localmode.html" title="Local Mode"><span class="none"></span>Local Mode</a></li> <li><a href="tez-ui.html" title="Tez UI"><span class="none"></span>Tez UI</a></li> <li><a href="shuffle-handler.html" title="Tez Shuffle Handler"><span class="none"></span>Tez Shuffle Handler</a></li> <li><a href="user_guides.html" title="User Guides"><span class="none"></span>User Guides</a></li> <li class="nav-header">Community</li> <li><a href="mail-lists.html" title="Mailing Lists"><span class="none"></span>Mailing Lists</a></li> <li><a href="issue-tracking.html" title="Issue Tracking"><span class="none"></span>Issue Tracking</a></li> <li><a href="team-list.html" title="Project Team"><span class="none"></span>Project Team</a></li> <li><a href="by-laws.html" title="Project By-Laws"><span class="none"></span>Project By-Laws</a></li> <li><a href="https://www.meetup.com/Apache-Tez-User-Group" class="externalLink" title="User Meetup Group"><span class="none"></span>User Meetup Group</a></li> <li><a href="https://cwiki.apache.org/confluence/display/TEZ" class="externalLink" title="Tez Wiki"><span class="none"></span>Tez Wiki</a></li> <li class="nav-header">Download Apache TEZ庐 Releases</li> <li><a href="releases/apache-tez-0-8-5.html" title="0.8.5"><span class="none"></span>0.8.5</a></li> <li><a href="releases/apache-tez-0-9-2.html" title="0.9.2"><span class="none"></span>0.9.2</a></li> <li><a href="releases/apache-tez-0-10-3.html" title="0.10.3"><span class="none"></span>0.10.3</a></li> <li><a href="releases/apache-tez-0-10-4.html" title="0.10.4"><span class="none"></span>0.10.4</a></li> <li><a href="releases/index.html" title="All Releases"><span class="none"></span>All Releases</a></li> <li class="nav-header">Contribute</li> <li><a href="https://issues.apache.org/jira/browse/TEZ" class="externalLink" title="JIRA"><span class="none"></span>JIRA</a></li> <li><a href="https://gitbox.apache.org/repos/asf/tez.git" class="externalLink" title="Git (Read/Write)"><span class="none"></span>Git (Read/Write)</a></li> <li><a href="git://git.apache.org/tez.git" class="externalLink" title="Git (Read-only)"><span class="none"></span>Git (Read-only)</a></li> <li><a href="https://cwiki.apache.org/confluence/display/TEZ/How+to+Contribute+to+Tez" class="externalLink" title="How to Contribute"><span class="none"></span>How to Contribute</a></li> <li class="nav-header">ASF</li> <li><a href="https://www.apache.org/licenses/" class="externalLink" title="License"><span class="none"></span>License</a></li> <li><a href="https://www.apache.org/foundation/sponsorship.html" class="externalLink" title="Donate"><span class="none"></span>Donate</a></li> <li><a href="https://www.apache.org/foundation/thanks.html" class="externalLink" title="Thanks"><span class="none"></span>Thanks</a></li> <li><a href="https://www.apache.org/security/" class="externalLink" title="Security"><span class="none"></span>Security</a></li> </ul> </nav> <div class="well sidebar-nav"> <hr /> <div id="poweredBy"> <div class="clear"></div> <div id="twitter" style="border:none; margin-top: 10px"> <a href="https://twitter.com/ApacheTez" class="twitter-follow-button" data-show-count="false" data-align="left" data-size="medium" data-show-screen-name="true" data-lang="en">Follow ApacheTez</a> <script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");</script> </div> <div class="clear"></div> <div class="clear"></div> <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy"><img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png" /></a> </div> </div> </header> <main id="bodyColumn" class="span10" > <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <section> <h2><a name="Install.2FDeploy_Instructions_for_Tez"></a>Install/Deploy Instructions for Tez</h2> <p>Replace x.y.z with the tez release number that you are using. E.g. 0.5.0. For Tez versions 0.8.3 and higher, Tez needs Apache Hadoop to be of version 2.6.0 or higher. For Tez version 0.9.0 and higher, Tez needs Apache Hadoop to be version 2.7.0 or higher.</p> <ol style="list-style-type: decimal"> <li> <p>Deploy Apache Hadoop using version of 2.7.0 or higher.</p> <ul> <li>You need to change the value of the hadoop.version property in the top-level pom.xml to match the version of the hadoop branch being used.</li> </ul> <div class="source"><pre class="prettyprint"><code>$ hadoop version </code></pre></div> </li> <li> <p>Build tez using <code>mvn clean package -DskipTests=true -Dmaven.javadoc.skip=true</code></p> <ul> <li>This assumes that you have already installed JDK8 or later and Maven 3 or later.</li> <li>Tez also requires Protocol Buffers 3.19.4, including the protoc-compiler. <ul> <li>This can be downloaded from <a class="externalLink" href="https://github.com/google/protobuf/tags/">https://github.com/google/protobuf/tags/</a>.</li> <li>On Mac OS X with the homebrew package manager <code>brew install protobuf250</code></li> <li>For rpm-based linux systems, the yum repos may not have the 3.19.4 version. <code>rpm.pbone.net</code> has the protobuf-3.19.4 and protobuf-compiler-3.19.4 packages.</li> </ul> </li> <li>If you prefer to run the unit tests, remove skipTests from the command above.</li> <li>If you use Eclipse IDE, you can import the projects using &#x201c;Import/Maven/Existing Maven Projects&#x201d;. Eclipse does not automatically generate Java sources or include the generated sources into the projects. Please build using maven as described above and then use Project Properties to include &#x201c;target/generatedsources/java&#x201d; as a source directory into the &#x201c;Java Build Path&#x201d; for these projects: tez-api, tez-mapreduce, tez-runtime-internals and tez-runtime-library. This needs to be done just once after importing the project.</li> </ul> </li> <li> <p>Copy the relevant tez tarball into HDFS, and configure tez-site.xml</p> <ul> <li>A tez tarball containing tez and hadoop libraries will be found at tez-dist/target/tez-x.y.z-SNAPSHOT.tar.gz</li> <li>Assuming that the tez jars are put in /apps/ on HDFS, the command would be</li> </ul> <div class="source"><pre class="prettyprint"><code>hadoop fs -mkdir /apps/tez-x.y.z-SNAPSHOT hadoop fs -copyFromLocal tez-dist/target/tez-x.y.z-SNAPSHOT.tar.gz /apps/tez-x.y.z-SNAPSHOT/ </code></pre></div> <ul> <li>tez-site.xml configuration. <ul> <li>Set tez.lib.uris to point to the tar.gz uploaded to HDFS. Assuming the steps mentioned so far were followed, set tez.lib.uris to <code>${fs.defaultFS}/apps/tez-x.y.z-SNAPSHOT/tez-x.y.z-SNAPSHOT.tar.gz</code></li> <li>Ensure tez.use.cluster.hadoop-libs is not set in tez-site.xml, or if it is set, the value should be false</li> </ul> </li> <li>Please note that the tarball version should match the version of the client jars used when submitting Tez jobs to the cluster. Please refer to the <a class="externalLink" href="https://cwiki.apache.org/confluence/display/TEZ/Version+Compatibility">Version Compatibility Guide</a> for more details on version compatibility and detecting mismatches.</li> </ul> </li> <li> <p>Optional: If running existing MapReduce jobs on Tez. Modify mapred-site.xml to change &#x201c;mapreduce.framework.name&#x201d; property from its default value of &#x201c;yarn&#x201d; to &#x201c;yarn-tez&#x201d;</p> </li> <li> <p>Configure the client node to include the tez-libraries in the hadoop classpath</p> <ul> <li>Extract the tez minimal tarball created in step 2 to a local directory (assuming TEZ_JARS is where the files will be decompressed for the next steps)</li> </ul> <div class="source"><pre class="prettyprint"><code>tar -xvzf tez-dist/target/tez-x.y.z-minimal.tar.gz -C $TEZ_JARS </code></pre></div> <ul> <li>set TEZ_CONF_DIR to the location of tez-site.xml</li> <li>Add $TEZ_CONF_DIR, ${TEZ_JARS}/* and ${TEZ_JARS}/lib/* to the application classpath. For example, doing it via the standard Hadoop tool chain would use the following command to set up the application classpath:</li> </ul> <div class="source"><pre class="prettyprint"><code>export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/* </code></pre></div> <ul> <li>Please note the &#x201c;*&#x201d; which is an important requirement when setting up classpaths for directories containing jar files.</li> </ul> </li> <li> <p>There is a basic example of using an MRR job in the tez-examples.jar. Refer to OrderedWordCount.java in the source code. To run this example:</p> <div class="source"><pre class="prettyprint"><code>$HADOOP_PREFIX/bin/hadoop jar tez-examples.jar orderedwordcount &lt;input&gt; &lt;output&gt; </code></pre></div> <p>This will use the TEZ DAG ApplicationMaster to run the ordered word count job. This job is similar to the word count example except that it also orders all words based on the frequency of occurrence.</p> <p>Tez DAGs could be run separately as different applications or serially within a single TEZ session. There is a different variation of orderedwordcount in tez-tests that supports the use of Sessions and handling multiple input-output pairs. You can use it to run multiple DAGs serially on different inputs/outputs.</p> <div class="source"><pre class="prettyprint"><code>$HADOOP_PREFIX/bin/hadoop jar tez-tests.jar testorderedwordcount &lt;input1&gt; &lt;output1&gt; &lt;input2&gt; &lt;output2&gt; &lt;input3&gt; &lt;output3&gt; ... </code></pre></div> <p>The above will run multiple DAGs for each input-output pair.</p> <p>To use TEZ sessions, set -DUSE_TEZ_SESSION=true</p> <div class="source"><pre class="prettyprint"><code>$HADOOP_PREFIX/bin/hadoop jar tez-tests.jar testorderedwordcount -DUSE_TEZ_SESSION=true &lt;input1&gt; &lt;output1&gt; &lt;input2&gt; &lt;output2&gt; </code></pre></div> </li> <li> <p>Submit a MR job as you normally would using something like:</p> <div class="source"><pre class="prettyprint"><code>$HADOOP_PREFIX/bin/hadoop jar hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT-tests.jar sleep -mt 1 -rt 1 -m 1 -r 1 </code></pre></div> <p>This will use the TEZ DAG ApplicationMaster to run the MR job. This can be verified by looking at the AM&#x2019;s logs from the YARN ResourceManager UI. This needs mapred-site.xml to have &#x201c;mapreduce.framework.name&#x201d; set to &#x201c;yarn-tez&#x201d;</p> </li> </ol></section><section> <h2><a name="Various_ways_to_configure_tez.lib.uris"></a>Various ways to configure tez.lib.uris</h2> <p>The <code>tez.lib.uris</code> configuration property supports a comma-separated list of values. The types of values supported are:</p> <ul> <li>Path to simple file</li> <li>Path to a directory</li> <li>Path to a compressed archive ( tarball, zip, etc).</li> </ul> <p>For simple files and directories, Tez will add all these files and first-level entries in the directories (recursive traversal of dirs is not supported) into the working directory of the Tez runtime and they will automatically be included into the classpath. For archives i.e. files whose names end with generally known compressed archive suffixes such as &#x2018;tgz&#x2019;, &#x2018;tar.gz&#x2019;, &#x2018;zip&#x2019;, etc. will be uncompressed into the container working directory too. However, given that the archive structure is not known to the Tez framework, the user is expected to configure <code>tez.lib.uris.classpath</code> to ensure that the nested directory structure of an archive is added to the classpath. This classpath values should be relative i.e. the entries should start with &#x201c;./&#x201d;.</p></section><section> <h2><a name="Hadoop_Installation_dependent_Install.2FDeploy_Instructions"></a>Hadoop Installation dependent Install/Deploy Instructions</h2> <p>The above install instructions use Tez with pre-packaged Hadoop libraries included in the package and is the recommended method for installation. A full tarball with all dependencies is a better approach to ensure that existing jobs continue to run during a cluster's rolling upgrade.</p> <p>Although the <code>tez.lib.uris</code> configuration options enable a wide variety of usage patterns, there are 2 main alternative modes that are supported by the framework:</p> <ol style="list-style-type: decimal"> <li>Mode A: Using a tez tarball on HDFS along with Hadoop libraries available on the cluster.</li> <li>Mode B: Using a tez tarball along with the Hadoop tarball.</li> </ol> <p>Both these modes will require a tez build without Hadoop dependencies and that is available at tez-dist/target/tez-x.y.z-minimal.tar.gz.</p></section><section> <h2><a name="For_Mode_A:_Tez_tarball_with_using_existing_cluster_Hadoop_libraries_by_leveraging_yarn.application.classpath"></a>For Mode A: Tez tarball with using existing cluster Hadoop libraries by leveraging yarn.application.classpath</h2> <p>This mode is not recommended for clusters that use rolling upgrades. Additionally, it is the user's responsibility to ensure that the tez version being used is compatible with the version of Hadoop running on the cluster. Step 3 above changes as follows. Also subsequent steps should use tez-dist/target/tez-x.y.z-minimal.tar.gz instead of tez-dist/target/tez-x.y.z.tar.gz</p> <ul> <li> <p>A tez build without Hadoop dependencies will be available at tez-dist/target/tez-x.y.z-minimal.tar.gz Assuming that the tez jars are put in /apps/ on HDFS, the command would be</p> <div class="source"><pre class="prettyprint"><code>&quot;hadoop fs -mkdir /apps/tez-x.y.z&quot; &quot;hadoop fs -copyFromLocal tez-dist/target/tez-x.y.z-minimal.tar.gz /apps/tez-x.y.z&quot; </code></pre></div> </li> <li> <p>tez-site.xml configuration</p> <ul> <li>Set tez.lib.uris to point to the paths in HDFS containing the tez jars. Assuming the steps mentioned so far were followed, set tez.lib.uris to <code>${fs.defaultFS}/apps/tez-x.y.z/tez-x.y.z-minimal.tar.gz</code></li> <li>Set tez.use.cluster.hadoop-libs to true</li> </ul> </li> </ul></section><section> <h2><a name="For_Mode_B:_Tez_tarball_with_Hadoop_tarball"></a>For Mode B: Tez tarball with Hadoop tarball</h2> <p>This mode will support rolling upgrades. It is the user's responsibility to ensure that the versions of Tez and Hadoop being used are compatible. To do this configuration, we need to change Step 3 of the default instructions in the following ways.</p> <ul> <li>Assuming that the tez archives/jars are put in /apps/ on HDFS, the command to put this minimal Tez archive into HDFS would be:</li> </ul> <div class="source"><pre class="prettyprint"><code>&quot;hadoop fs -mkdir /apps/tez-x.y.z&quot; &quot;hadoop fs -copyFromLocal tez-dist/target/tez-x.y.z-minimal.tar.gz /apps/tez-x.y.z&quot; </code></pre></div> <ul> <li>Alternatively, you can put the minimal directory directly into HDFS and reference the jars, instead of using an archive. The command to put the minimal directory into HDFS would be:</li> </ul> <div class="source"><pre class="prettyprint"><code>&quot;hadoop fs -copyFromLocal tez-dist/target/tez-x.y.z-minimal/* /apps/tez-x.y.z&quot; </code></pre></div> <ul> <li>After building hadoop, the hadoop tarball will be available at hadoop/hadoop-dist/target/hadoop-x.y.z-SNAPSHOT.tar.gz</li> <li>Assuming that the hadoop jars are put in /apps/ on HDFS, the command to put this Hadoop archive into HDFS would be:</li> </ul> <div class="source"><pre class="prettyprint"><code>&quot;hadoop fs -mkdir /apps/hadoop-x.y.z&quot; &quot;hadoop fs -copyFromLocal hadoop-dist/target/hadoop-x.y.z-SNAPSHOT.tar.gz /apps/hadoop-x.y.z&quot; </code></pre></div> <ul> <li> <p>tez-site.xml configuration</p> <ul> <li> <p>Set tez.lib.uris to point to the the archives and jars that are needed for Tez/Hadoop.</p> </li> <li> <p>Example: When using both Tez and Hadoop archives, set tez.lib.uris to <code>${fs.defaultFS}/apps/tez-x.y.z/tez-x.y.z-minimal.tar.gz#tez,${fs.defaultFS}/apps/hadoop-x.y.z/hadoop-x.y.z-SNAPSHOT.tar.gz#hadoop-mapreduce</code></p> </li> <li> <p>Example: When using Tez jars with a Hadoop archive, set tez.lib.uris to: <code>${fs.defaultFS}/apps/tez-x.y.z,${fs.defaultFS}/apps/tez-x.y.z/lib,${fs.defaultFS}/apps/hadoop-x.y.z/hadoop-x.y.z-SNAPSHOT.tar.gz#hadoop-mapreduce</code></p> </li> <li> <p>In tez.lib.uris, the text immediately following the &#x2018;#&#x2019; symbol is the fragment that refers to the symlink that will be created for the archive. If no fragment is given, the symlink will be set to the name of the archive. Fragments should not be given to directories or jars.</p> </li> <li> <p>If any archives are specified in tez.lib.uris, then tez.lib.uris.classpath must be set to define the classpath for these archives as the archive structure is not known.</p> </li> <li> <p>Example: Classpath when using both Tez and Hadoop archives, set tez.lib.uris.classpath to:</p> </li> </ul> <div class="source"><pre class="prettyprint"><code></code></pre></div> </li> </ul> <p>./tez/<i>:./tez/lib/</i>:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/common/<i>:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/common/lib/</i>:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/hdfs/<i>:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/hdfs/lib/</i>:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/yarn/<i>:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/yarn/lib/</i>:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/mapreduce/<i>:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/mapreduce/lib/</i> ```</p> <div class="source"><pre class="prettyprint"><code>- Example: Classpath when using Tez jars with a Hadoop archive, set tez.lib.uris.classpath to: ``` </code></pre></div> <p>./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/common/<i>:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/common/lib/</i>:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/hdfs/<i>:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/hdfs/lib/</i>:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/yarn/<i>:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/yarn/lib/</i>:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/mapreduce/<i>:./hadoop-mapreduce/hadoop-x.y.z-SNAPSHOT/share/hadoop/mapreduce/lib/</i> ```</p></section><section> <h2><a name="Install_instructions_for_older_versions_of_Tez_.28pre_0.5.0.29"></a><a href="./install_pre_0_5_0.html">Install instructions for older versions of Tez (pre 0.5.0)</a></h2></section> </main> </div> </div> <hr/> <footer> <div class="container-fluid"> <div class="row-fluid"> <div class="row span12"> Apache Tez, Apache, the Apache feather logo, and the Apache Tez project logos are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners. </div> <br/> <div class="row span12"> <a href="https://privacy.apache.org/policies/privacy-policy-public.html">Privacy Policy</a> </div> </div> </div> </footer> </body> </html>

Pages: 1 2 3 4 5 6 7 8 9 10