CINXE.COM
Samza - Tests
<!DOCTYPE html> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <html lang="en"> <head> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <meta charset="utf-8"> <title>Samza - Tests</title> <link rel="apple-touch-icon-precomposed" sizes="57x57" href="/img/favicon/apple-touch-icon-57x57.png" /> <link rel="apple-touch-icon-precomposed" sizes="114x114" href="/img/favicon/apple-touch-icon-114x114.png" /> <link rel="apple-touch-icon-precomposed" sizes="72x72" href="/img/favicon/apple-touch-icon-72x72.png" /> <link rel="apple-touch-icon-precomposed" sizes="144x144" href="/img/favicon/apple-touch-icon-144x144.png" /> <link rel="apple-touch-icon-precomposed" sizes="60x60" href="/img/favicon/apple-touch-icon-60x60.png" /> <link rel="apple-touch-icon-precomposed" sizes="120x120" href="/img/favicon/apple-touch-icon-120x120.png" /> <link rel="apple-touch-icon-precomposed" sizes="76x76" href="/img/favicon/apple-touch-icon-76x76.png" /> <link rel="apple-touch-icon-precomposed" sizes="152x152" href="/img/favicon/apple-touch-icon-152x152.png" /> <link rel="icon" type="image/png" href="/img/favicon/favicon-196x196.png" sizes="196x196" /> <link rel="icon" type="image/png" href="/img/favicon/favicon-96x96.png" sizes="96x96" /> <link rel="icon" type="image/png" href="/img/favicon/favicon-32x32.png" sizes="32x32" /> <link rel="icon" type="image/png" href="/img/favicon/favicon-16x16.png" sizes="16x16" /> <link rel="icon" type="image/png" href="/img/favicon/favicon-128.png" sizes="128x128" /> <meta name="application-name" content="https://samza.apache.org" /> <meta name="msapplication-TileColor" content="#FFFFFF" /> <meta name="msapplication-TileImage" content="/img/favicon/mstile-144x144.png" /> <meta name="msapplication-square70x70logo" content="/img/favicon/mstile-70x70.png" /> <meta name="msapplication-square150x150logo" content="/img/favicon/mstile-150x150.png" /> <meta name="msapplication-wide310x150logo" content="/img/favicon/mstile-310x150.png" /> <meta name="msapplication-square310x310logo" content="/img/favicon/mstile-310x310.png" /> <link href="/css/ionicons.min.css" rel="stylesheet"> <link href="/css/google-fonts.css" rel="stylesheet"> <link href="/css/syntax.css" rel="stylesheet"/> <link rel="stylesheet" href="/css/main.new.css" /> </head> <body class="page"> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <div class="main-navigation" data-plugin="menu"> <div class="main-navigation__toggle" data-menu-closed> <i class="icon ion-md-menu"></i> </div> <div class="main-navigation__toggle main-navigation__toggle--opened" data-menu-opened> <i class="icon ion-md-close"></i> </div> <div class="main-navigation__inner"> <div class="main-navigation__logo"> <a href="/"> <img class="main-navigation__logo-img" src="/img/samza-logo.png" srcset="/img/samza-logo.png 1x, /img/samza-logo@2x.png 2x" alt="Samza Logo" /> </a> </div> <div class="main-navigation__items" data-menu-opened> <a class="main-navigation__item" href="/">Home</a> <a class="main-navigation__item" href="/learn/documentation/latest/core-concepts/core-concepts.html">Docs</a> <a class="main-navigation__item" href="/powered-by/">Powered By</a> <a class="main-navigation__item" href="/startup/download/">Downloads</a> <a class="main-navigation__item" href="/blog/">Blog</a> <div class="main-navigation__item main-navigation__item--group"> <div class="main-navigation__item-group-title"> Community <i class="icon ion-md-arrow-dropdown"></i> </div> <div class="main-navigation__item-group-list"> <a class="main-navigation__item" href="/community/contact-us.html">Contact Us</a> <a class="main-navigation__item" href="/contribute/contributors-corner.html">Contributor's Corner</a> <a class="main-navigation__item" href="/community/committers.html">PMC Members and committers</a> <a class="main-navigation__item" href="/meetups/">Talks and Meetups</a> </div> </div> </div> </div> </div> <div class="container"> <div class="container__toggle"> <i class="icon ion-md-arrow-dropleft-circle container__toggle-icon"></i> <i class="icon ion-md-arrow-dropright-circle container__toggle-icon container__toggle-icon--opened"></i> </div> <!-- There is only one menu, but made it as a no-output collection to grab data only --> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <div class="side-navigation"> <!-- Start Group --> <div class="side-navigation__group side-navigation__group--has-nested" data-plugin="sub-menu" data-sub-menu-show-class="side-navigation__group--has-nested-visible"> <!-- Make menu_title, and start items group if needed --> <div class="side-navigation__group-title"> <i class="side-navigation__group-title-icon icon ion-md-arrow-dropdown"></i> Getting Started </div> <div class="side-navigation__group-items " data-sub-menu > <!-- Handle sub navigation items from data --> <a class="side-navigation__group-item" data-match-active="" href="/startup/quick-start/latest/">QuickStart</a> <a class="side-navigation__group-item" data-match-active="" href="/startup/code-examples/latest/">Code Examples</a> <!-- Handle sub nagivation from site collections --> <!-- Close sub nav group --> </div> <!-- Close menu group --> </div> <!-- Start Group --> <div class="side-navigation__group side-navigation__group--has-nested" data-plugin="sub-menu" data-sub-menu-show-class="side-navigation__group--has-nested-visible"> <!-- Make menu_title, and start items group if needed --> <div class="side-navigation__group-title"> <i class="side-navigation__group-title-icon icon ion-md-arrow-dropdown"></i> Documentation </div> <div class="side-navigation__group-items side-navigation__group-has-submenus" data-sub-menu data-documentation="/learn/documentation/latest/"> <!-- Handle sub navigation items from data --> <!-- Handle sub nagivation from site collections --> <!-- Close sub nav group --> </div> <!-- Close menu group --> </div> <!-- Start Group --> <div class="side-navigation__group side-navigation__group--has-nested" data-plugin="sub-menu" data-sub-menu-show-class="side-navigation__group--has-nested-visible"> <!-- Make menu_title, and start items group if needed --> <div class="side-navigation__group-title"> <i class="side-navigation__group-title-icon icon ion-md-arrow-dropdown"></i> Releases </div> <div class="side-navigation__group-items " data-sub-menu > <!-- Handle sub navigation items from data --> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.8.0">1.8.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.7.0">1.7.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.6.0">1.6.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.5.1">1.5.1</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.5.0">1.5.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.4.0">1.4.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.3.1">1.3.1</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.3.0">1.3.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.2.0">1.2.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.1.0">1.1.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.0.0">1.0.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/0.14">0.14</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/0.13">0.13</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/0.12">0.12</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/0.11">0.11</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/0.10">0.10</a> <!-- Handle sub nagivation from site collections --> <!-- Close sub nav group --> </div> <!-- Close menu group --> </div> <!-- Start Group --> <div class="side-navigation__group"> <!-- Make menu_title, and start items group if needed --> <a class="side-navigation__group-title" data-plugin="top-menu" data-match-active="" href="/blog/"> Blog </a> <!-- Handle sub navigation items from data --> <!-- Handle sub nagivation from site collections --> <!-- Close sub nav group --> <!-- Close menu group --> </div> <!-- Start Group --> <div class="side-navigation__group side-navigation__group--has-nested" data-plugin="sub-menu" data-sub-menu-show-class="side-navigation__group--has-nested-visible"> <!-- Make menu_title, and start items group if needed --> <div class="side-navigation__group-title"> <i class="side-navigation__group-title-icon icon ion-md-arrow-dropdown"></i> Community </div> <div class="side-navigation__group-items " data-sub-menu > <!-- Handle sub navigation items from data --> <a class="side-navigation__group-item" data-match-active="" href="/community/contact-us.html">Contact Us</a> <a class="side-navigation__group-item" data-match-active="" href="/contribute/contributors-corner.html">Contributor's Corner</a> <a class="side-navigation__group-item" data-match-active="" href="/contribute/enhancement-proposal.html">Enhancement Proposal</a> <a class="side-navigation__group-item" data-match-active="" href="/community/committers.html">PMC members & Committers</a> <a class="side-navigation__group-item" data-match-active="" href="/meetups/">Talks and Meetups</a> <!-- Handle sub nagivation from site collections --> <!-- Close sub nav group --> </div> <!-- Close menu group --> </div> <!-- Start Group --> <div class="side-navigation__group side-navigation__group--has-nested" data-plugin="sub-menu" data-sub-menu-show-class="side-navigation__group--has-nested-visible"> <!-- Make menu_title, and start items group if needed --> <div class="side-navigation__group-title"> <i class="side-navigation__group-title-icon icon ion-md-arrow-dropdown"></i> Case Studies </div> <div class="side-navigation__group-items " data-sub-menu > <!-- Handle sub navigation items from data --> <a class="side-navigation__group-item" data-match-active="exact" href="/case-studies/">View All</a> <hr> <!-- Handle sub nagivation from site collections --> <a class="side-navigation__group-item" href="/case-studies/ebay" data-match-active="">eBay</a> <a class="side-navigation__group-item" href="/case-studies/tripadvisor" data-match-active="">TripAdvisor</a> <a class="side-navigation__group-item" href="/case-studies/slack" data-match-active="">Slack</a> <a class="side-navigation__group-item" href="/case-studies/optimizely" data-match-active="">Optimizely</a> <a class="side-navigation__group-item" href="/case-studies/redfin" data-match-active="">Redfin</a> <a class="side-navigation__group-item" href="/case-studies/linkedin" data-match-active="">LinkedIn</a> <!-- Close sub nav group --> </div> <!-- Close menu group --> </div> </div> <div class="section"> <div class="content"> <h2>Tests</h2> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <p>Samza鈥檚 unit tests are written on top of <a href="http://junit.org/">JUnit</a>, and license checking is done with <a href="http://creadur.apache.org/rat/">Apache Rat</a>. Samza鈥檚 integration tests are written on top of <a href="https://github.com/linkedin/Zopkio">Zopkio</a>. As of Samza 0.10, the integration tests are based on Zopkio 0.1.17.</p> <h3 id="running-unit-tests-locally">Running Unit Tests Locally</h3> <p>To run all tests, and license checks:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./gradlew clean check </code></pre></div></div> <p>To run a single test:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./gradlew clean :samza-core:test -Dtest.single=TestSamzaContainer </code></pre></div></div> <p>Test results are located in:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code><module name>/build/reports/tests/index.html </code></pre></div></div> <h4 id="testing-scala-and-java">Testing Scala and Java</h4> <p>Samza鈥檚 unit tests can also be run against all supported permutations of Scala and Java.</p> <p>To run the tests against a specific combination:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./gradlew -PscalaSuffix=2.11 -PyarnVersion=2.4.0 clean check </code></pre></div></div> <p>To run Samza鈥檚 unit tests against all permutations, run:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bin/check-all.sh </code></pre></div></div> <p>When run on Linux, this command requires you to set two environment variables:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>JAVA8_HOME is not set. </code></pre></div></div> <p>On Mac, check-all.sh will default to the appropriate path for each environment variable if it鈥檚 not already set:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>JAVA8_HOME is not set. JAVA8_HOME defaulted to /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Contents/Home </code></pre></div></div> <h3 id="running-integration-tests-locally">Running Integration Tests Locally</h3> <p>Samza uses <a href="https://github.com/linkedin/Zopkio">Zopkio</a> to deploy and execute its integration tests. Integration tests can be executed by running:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./bin/integration-tests.sh /tmp/samza-tests [yarn-integration-tests|standalone-integration-tests] </code></pre></div></div> <p>The parameter defines where the integration tests should install packages both locally and on remote systems. Executing this command will:</p> <ol> <li>Build a samza-test job tarball.</li> <li>Download and install YARN, Kafka, and ZooKeeper.</li> <li>Deploy the samza-test job tarball to all YARN NM machines.</li> <li>Start all Samza integration test jobs.</li> <li>Feed input data to the jobs, and verify the results.</li> <li>Open a report, and aggregate all remote logs.</li> </ol> <p>The default configurations that ship with Samza deploy all software, and run all tests locally on the machine from which the <code class="language-plaintext highlighter-rouge">integration-tests.sh</code> command was executed.</p> <p>If you have difficulties (<code class="language-plaintext highlighter-rouge">connection refused</code> error or <code class="language-plaintext highlighter-rouge">JAVA_HOME is not set and could not be found.</code> error), take a look the <code class="language-plaintext highlighter-rouge">Prerequisites for Running the Integration Tests Locally</code> section.</p> <p>The integration tests use SSH to interact with remote machines (and localhost). This means that you need an authentication mechanism when connecting to the machines. The two authentication mechanisms provided are:</p> <ol> <li>Interactive</li> <li>Public key</li> </ol> <h4 id="interactive">Interactive</h4> <p>Zopkio will prompt you for a password by default. This password will be used as the SSH password when trying to log into remote systems.</p> <h4 id="public-key">Public Key</h4> <p>Zopkio supports public key authentication if you prefer to use it, or if your environment doesn鈥檛 allow interactive authentication. To use public key authentication, add your public SSH key to ~/.ssh/authorized_keys, and SSH to all of the machines that you鈥檒l be deploying to (localhost by default). See <a href="http://www.linuxproblem.org/art_9.html">here</a> for details.</p> <p>Once this is done, you can run Zopkio with the --nopassword parameter:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./bin/integration-tests.sh /tmp/samza-tests [yarn-integration-tests|standalone-integration-tests] --nopassword </code></pre></div></div> <p>This will skip the password prompt, and force Zopkio to try public key authentication.</p> <h4 id="console-logging">Console Logging</h4> <p>The integration-tests.sh script will set the console log level to INFO by default. The level can be changed with:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./bin/integration-tests.sh /tmp/samza-tests [yarn-integration-tests|standalone-integration-tests] --console-log-level DEBUG </code></pre></div></div> <p>Changing this setting will define how verbose Zopkio is during test execution. It does not affect any of the log4j.xml settings in Samza, YARN, Kafka, or ZooKeeper.</p> <h4 id="prerequisites-for-running-the-integration-tests-locally">Prerequisites for Running the Integration Tests Locally</h4> <p>Before running the integration tests locally make sure that:</p> <ul> <li><code class="language-plaintext highlighter-rouge">ssh localhost</code> works</li> <li><code class="language-plaintext highlighter-rouge">ssh localhost 'echo $JAVA_HOME'</code> prints the correct path to Java installation on your system</li> </ul> <p>If <code class="language-plaintext highlighter-rouge">ssh localhost</code> does not works, you need to install an SSH server (enable <code class="language-plaintext highlighter-rouge">Remote login</code> on Mac or install <code class="language-plaintext highlighter-rouge">openssh-server</code> on Linux for example).</p> <p>If <code class="language-plaintext highlighter-rouge">ssh localhost 'echo $JAVA_HOME'</code> does not prints the path to Java installation on your system, you need to configure your login shell to set the <code class="language-plaintext highlighter-rouge">JAVA_HOME</code> environment variable regardless in non-interactive mode.<br /> Zopkio is doing the deployment in non-interactive mode therefore no configuration files (including <code class="language-plaintext highlighter-rouge">/etc/profile</code>) will be taken into account.<br /> For example if your system uses <code class="language-plaintext highlighter-rouge">bash</code> shell, you can insert <code class="language-plaintext highlighter-rouge">export JAVA_HOME=/path/to/java/home</code> at the beginning of the <code class="language-plaintext highlighter-rouge">~/.bashrc</code> file before the line <code class="language-plaintext highlighter-rouge"># If not running interactively, don't do anything</code>.<br /> More info on this subject can be found <a href="http://askubuntu.com/questions/247738/why-is-etc-profile-not-invoked-for-non-login-shells">here</a>.</p> </div> </div> </div> <!-- footer starts here --> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <footer> <div class="footer-inner"> <div class="side-by-side"> <div> <div class="footer__heading">Learn More</div> <div class="footer__items"> <a class="footer__item" href="/meetups/">Meetups</a> <a class="footer__item" href="/blog/">Blog</a> <a class="footer__item" href="/learn/documentation/latest/introduction/background.html">About</a> </div> </div> <div> <div class="footer__heading">Community</div> <div class="footer__items"> <a class="footer__item" href="/community/contact-us.html">Contact Us</a> <a class="footer__item" href="/contribute/contributors-corner.html">Contributors' Corner</a> <a class="footer__item" href="/community/committers.html">PMC members and committers</a> <a class="footer__item" href="/powered-by/">Powered By</a> </div> </div> <div> <div class="quick-links"> <a class="quick-link" href="/startup/download" target="_blank"> <i class="icon ion-md-download"></i> </a> <a class="quick-link" href="https://git-wip-us.apache.org/repos/asf?p=samza.git;a=tree" target="_blank"> <i class="icon ion-md-code"></i> </a> <a class="quick-link" href="https://twitter.com/samzastream" target="_blank"> <i class="icon ion-logo-twitter"></i> </a> </div> <p> <script>document.write(new Date().getFullYear());</script> © samza.apache.org</p> </div> </div> </div> </footer> <!-- Google Analytics --> <script> (function (i, s, o, g, r, a, m) { i['GoogleAnalyticsObject'] = r; i[r] = i[r] || function () { (i[r].q = i[r].q || []).push(arguments) }, i[r].l = 1 * new Date(); a = s.createElement(o), m = s.getElementsByTagName(o)[0]; a.async = 1; a.src = g; m.parentNode.insertBefore(a, m) })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga'); ga('create', 'UA-43122768-1', 'apache.org'); ga('send', 'pageview'); </script> <script src="/js/main.new.js"></script> </body> </html>