Apache Zeppelin 0.10.0 Documentation: Apache Zeppelin Tutorial

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <title>Apache Zeppelin 0.10.0 Documentation: Apache Zeppelin Tutorial</title> <meta name="description" content="This tutorial page contains a short walk-through tutorial that uses Apache Spark backend. Please note that this tutorial is valid for Spark 1.3 and higher."> <meta name="author" content="The Apache Software Foundation">  <meta name="viewport" content="width=device-width, initial-scale=1.0">   <link href="//maxcdn.bootstrapcdn.com/font-awesome/4.2.0/css/font-awesome.min.css" rel="stylesheet">  <link href="/docs/0.10.0/assets/themes/zeppelin/bootstrap/css/bootstrap.css" rel="stylesheet"> <link href="/docs/0.10.0/assets/themes/zeppelin/css/style.css?body=1" rel="stylesheet" type="text/css"> <link href="/docs/0.10.0/assets/themes/zeppelin/css/syntax.css" rel="stylesheet" type="text/css" media="screen" />    <script src="https://code.jquery.com/jquery-1.10.2.min.js"></script> <script src="/docs/0.10.0/assets/themes/zeppelin/bootstrap/js/bootstrap.min.js"></script> <script src="/docs/0.10.0/assets/themes/zeppelin/js/docs.js"></script> <script src="/docs/0.10.0/assets/themes/zeppelin/js/anchor.min.js"></script> <script src="/docs/0.10.0/assets/themes/zeppelin/js/toc.js"></script> <script src="/docs/0.10.0/assets/themes/zeppelin/js/lunr.min.js"></script> <script src="/docs/0.10.0/assets/themes/zeppelin/js/search.js"></script>  <link href="/docs/0.10.0/atom.xml" type="application/atom+xml" rel="alternate" title="Sitewide ATOM Feed"> <link href="/docs/0.10.0/rss.xml" type="application/rss+xml" rel="alternate" title="Sitewide RSS Feed"> </head> <body> <div id="menu" class="navbar navbar-inverse navbar-fixed-top" role="navigation"> <div class="container navbar-container"> <div class="navbar-header"> <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse"> Toggle navigation </button> <div class="navbar-brand"> <a class="navbar-brand-main" href="http://zeppelin.apache.org"> <img src="/docs/0.10.0/assets/themes/zeppelin/img/zeppelin_logo.png" width="50" style="margin-top: -2px;" alt="I'm zeppelin"> Zeppelin <a class="navbar-brand-version" href="/docs/0.10.0" style="font-size: 15px; color: white;"> 0.10.0 </a> </a> </div> </div> <nav class="navbar-collapse collapse" role="navigation"> <ul class="nav navbar-nav"> <li> <a href="#" data-toggle="dropdown" class="dropdown-toggle">Quick Start </a> <ul class="dropdown-menu"> <li class="title">Getting Started</li> <li><a href="/docs/0.10.0/quickstart/install.html">Install</a></li> <li><a href="/docs/0.10.0/quickstart/explore_ui.html">Explore UI</a></li> <li><a href="/docs/0.10.0/quickstart/tutorial.html">Tutorial</a></li> <li role="separator" class="divider"></li> <li class="title">Run Mode</li> <li><a href="/docs/0.10.0/quickstart/kubernetes.html">Kubernetes</a></li> <li><a href="/docs/0.10.0/quickstart/docker.html">Docker</a></li> <li><a href="/docs/0.10.0/quickstart/yarn.html">Yarn</a></li> <li role="separator" class="divider"></li> <li><a href="/docs/0.10.0/quickstart/spark_with_zeppelin.html">Spark with Zeppelin</a></li> <li><a href="/docs/0.10.0/quickstart/flink_with_zeppelin.html">Flink with Zeppelin</a></li> <li><a href="/docs/0.10.0/quickstart/sql_with_zeppelin.html">SQL with Zeppelin</a></li> <li><a href="/docs/0.10.0/quickstart/python_with_zeppelin.html">Python with Zeppelin</a></li> <li><a href="/docs/0.10.0/quickstart/r_with_zeppelin.html">R with Zeppelin</a></li> </ul> </li> <li> <a href="#" data-toggle="dropdown" class="dropdown-toggle">Usage</a> <ul class="dropdown-menu scrollable-menu"> <li class="title">Dynamic Form</li> <li><a href="/docs/0.10.0/usage/dynamic_form/intro.html">What is Dynamic Form?</a></li> <li role="separator" class="divider"></li> <li class="title">Display System</li> <li><a href="/docs/0.10.0/usage/display_system/basic.html#text">Text Display</a></li> <li><a href="/docs/0.10.0/usage/display_system/basic.html#html">HTML Display</a></li> <li><a href="/docs/0.10.0/usage/display_system/basic.html#table">Table Display</a></li> <li><a href="/docs/0.10.0/usage/display_system/basic.html#network">Network Display</a></li> <li><a href="/docs/0.10.0/usage/display_system/angular_backend.html">Angular Display using Backend API</a></li> <li><a href="/docs/0.10.0/usage/display_system/angular_frontend.html">Angular Display using Frontend API</a></li> <li role="separator" class="divider"></li> <li class="title">Interpreter</li> <li><a href="/docs/0.10.0/usage/interpreter/overview.html">Overview</a></li> <li><a href="/docs/0.10.0/usage/interpreter/interpreter_binding_mode.html">Interpreter Binding Mode</a></li> <li><a href="/docs/0.10.0/usage/interpreter/user_impersonation.html">User Impersonation</a></li> <li><a href="/docs/0.10.0/usage/interpreter/dependency_management.html">Dependency Management</a></li> <li><a href="/docs/0.10.0/usage/interpreter/installation.html">Installing Interpreters</a></li>  <li><a href="/docs/0.10.0/usage/interpreter/execution_hooks.html">Execution Hooks (Experimental)</a></li> <li role="separator" class="divider"></li> <li class="title">Other Features</li> <li><a href="/docs/0.10.0/usage/other_features/publishing_paragraphs.html">Publishing Paragraphs</a></li> <li><a href="/docs/0.10.0/usage/other_features/personalized_mode.html">Personalized Mode</a></li> <li><a href="/docs/0.10.0/usage/other_features/customizing_homepage.html">Customizing Zeppelin Homepage</a></li> <li><a href="/docs/0.10.0/usage/other_features/notebook_actions.html">Notebook Actions</a></li> <li><a href="/docs/0.10.0/usage/other_features/cron_scheduler.html">Cron Scheduler</a></li> <li><a href="/docs/0.10.0/usage/other_features/zeppelin_context.html">Zeppelin Context</a></li> <li role="separator" class="divider"></li> <li class="title">REST API</li> <li><a href="/docs/0.10.0/usage/rest_api/interpreter.html">Interpreter API</a></li> <li><a href="/docs/0.10.0/usage/rest_api/zeppelin_server.html">Zeppelin Server API</a></li> <li><a href="/docs/0.10.0/usage/rest_api/notebook.html">Notebook API</a></li> <li><a href="/docs/0.10.0/usage/rest_api/notebook_repository.html">Notebook Repository API</a></li> <li><a href="/docs/0.10.0/usage/rest_api/configuration.html">Configuration API</a></li> <li><a href="/docs/0.10.0/usage/rest_api/credential.html">Credential API</a></li> <li><a href="/docs/0.10.0/usage/rest_api/helium.html">Helium API</a></li> <li class="title">Zeppelin SDK</li> <li><a href="/docs/0.10.0/usage/zeppelin_sdk/client_api.html">Client API</a></li> <li><a href="/docs/0.10.0/usage/zeppelin_sdk/session_api.html">Session API</a></li> </ul> </li> <li> <a href="#" data-toggle="dropdown" class="dropdown-toggle">Setup</a> <ul class="dropdown-menu scrollable-menu"> <li class="title">Basics</li> <li><a href="/docs/0.10.0/setup/basics/how_to_build.html">How to Build Zeppelin</a></li> <li><a href="/docs/0.10.0/setup/basics/hadoop_integration.html">Hadoop Integration</a></li> <li><a href="/docs/0.10.0/setup/basics/multi_user_support.html">Multi-user Support</a></li> <li role="separator" class="divider"></li> <li class="title">Deployment</li>  <li><a href="/docs/0.10.0/setup/deployment/spark_cluster_mode.html#spark-standalone-mode">Spark Cluster Mode: Standalone</a></li> <li><a href="/docs/0.10.0/setup/deployment/spark_cluster_mode.html#spark-on-yarn-mode">Spark Cluster Mode: YARN</a></li> <li><a href="/docs/0.10.0/setup/deployment/spark_cluster_mode.html#spark-on-mesos-mode">Spark Cluster Mode: Mesos</a></li> <li><a href="/docs/0.10.0/setup/deployment/flink_and_spark_cluster.html">Zeppelin with Flink, Spark Cluster</a></li> <li><a href="/docs/0.10.0/setup/deployment/cdh.html">Zeppelin on CDH</a></li> <li><a href="/docs/0.10.0/setup/deployment/virtual_machine.html">Zeppelin on VM: Vagrant</a></li> <li role="separator" class="divider"></li> <li class="title">Security</li> <li><a href="/docs/0.10.0/setup/security/authentication_nginx.html">HTTP Basic Auth using NGINX</a></li> <li><a href="/docs/0.10.0/setup/security/shiro_authentication.html">Shiro Authentication</a></li> <li><a href="/docs/0.10.0/setup/security/notebook_authorization.html">Notebook Authorization</a></li> <li><a href="/docs/0.10.0/setup/security/datasource_authorization.html">Data Source Authorization</a></li> <li><a href="/docs/0.10.0/setup/security/http_security_headers.html">HTTP Security Headers</a></li> <li role="separator" class="divider"></li> <li class="title">Notebook Storage</li> <li><a href="/docs/0.10.0/setup/storage/storage.html#notebook-storage-in-local-git-repository">Git Storage</a></li> <li><a href="/docs/0.10.0/setup/storage/storage.html#notebook-storage-in-s3">S3 Storage</a></li> <li><a href="/docs/0.10.0/setup/storage/storage.html#notebook-storage-in-azure">Azure Storage</a></li> <li><a href="/docs/0.10.0/setup/storage/storage.html#notebook-storage-in-oss">OSS Storage</a></li> <li><a href="/docs/0.10.0/setup/storage/storage.html#notebook-storage-in-zeppelinhub">ZeppelinHub Storage</a></li> <li><a href="/docs/0.10.0/setup/storage/storage.html#notebook-storage-in-mongodb">MongoDB Storage</a></li> <li role="separator" class="divider"></li> <li class="title">Operation</li> <li><a href="/docs/0.10.0/setup/operation/configuration.html">Configuration</a></li> <li><a href="/docs/0.10.0/setup/operation/proxy_setting.html">Proxy Setting</a></li> <li><a href="/docs/0.10.0/setup/operation/upgrading.html">Upgrading</a></li> <li><a href="/docs/0.10.0/setup/operation/trouble_shooting.html">Trouble Shooting</a></li> </ul> </li> <li> <a href="#" data-toggle="dropdown" class="dropdown-toggle">Interpreter </a> <ul class="dropdown-menu scrollable-menu"> <li class="title">Interpreters</li> <li><a href="/docs/0.10.0/usage/interpreter/overview.html">Overview</a></li> <li role="separator" class="divider"></li> <li><a href="/docs/0.10.0/interpreter/spark.html">Spark</a></li> <li><a href="/docs/0.10.0/interpreter/flink.html">Flink</a></li> <li><a href="/docs/0.10.0/interpreter/jdbc.html">JDBC</a></li> <li><a href="/docs/0.10.0/interpreter/python.html">Python</a></li> <li><a href="/docs/0.10.0/interpreter/r.html">R</a></li> <li role="separator" class="divider"></li> <li><a href="/docs/0.10.0/interpreter/alluxio.html">Alluxio</a></li> <li><a href="/docs/0.10.0/interpreter/beam.html">Beam</a></li> <li><a href="/docs/0.10.0/interpreter/bigquery.html">BigQuery</a></li> <li><a href="/docs/0.10.0/interpreter/cassandra.html">Cassandra</a></li> <li><a href="/docs/0.10.0/interpreter/elasticsearch.html">Elasticsearch</a></li> <li><a href="/docs/0.10.0/interpreter/geode.html">Geode</a></li> <li><a href="/docs/0.10.0/interpreter/groovy.html">Groovy</a></li> <li><a href="/docs/0.10.0/interpreter/hazelcastjet.html">Hazelcast Jet</a></li> <li><a href="/docs/0.10.0/interpreter/hbase.html">HBase</a></li> <li><a href="/docs/0.10.0/interpreter/hdfs.html">HDFS</a></li> <li><a href="/docs/0.10.0/interpreter/hive.html">Hive</a></li> <li><a href="/docs/0.10.0/interpreter/ignite.html">Ignite</a></li> <li><a href="/docs/0.10.0/interpreter/influxdb.html">influxDB</a></li> <li><a href="/docs/0.10.0/interpreter/java.html">Java</a></li> <li><a href="/docs/0.10.0/interpreter/jupyter.html">Jupyter</a></li> <li><a href="/docs/0.10.0/interpreter/kotlin.html">Kotlin</a></li> <li><a href="/docs/0.10.0/interpreter/ksql.html">KSQL</a></li> <li><a href="/docs/0.10.0/interpreter/kylin.html">Kylin</a></li> <li><a href="/docs/0.10.0/interpreter/lens.html">Lens</a></li> <li><a href="/docs/0.10.0/interpreter/livy.html">Livy</a></li> <li><a href="/docs/0.10.0/interpreter/mahout.html">Mahout</a></li> <li><a href="/docs/0.10.0/interpreter/markdown.html">Markdown</a></li> <li><a href="/docs/0.10.0/interpreter/mongodb.html">MongoDB</a></li> <li><a href="/docs/0.10.0/interpreter/neo4j.html">Neo4j</a></li> <li><a href="/docs/0.10.0/interpreter/pig.html">Pig</a></li> <li><a href="/docs/0.10.0/interpreter/postgresql.html">Postgresql, HAWQ</a></li> <li><a href="/docs/0.10.0/interpreter/sap.html">SAP</a></li> <li><a href="/docs/0.10.0/interpreter/scalding.html">Scalding</a></li> <li><a href="/docs/0.10.0/interpreter/scio.html">Scio</a></li> <li><a href="/docs/0.10.0/interpreter/shell.html">Shell</a></li> <li><a href="/docs/0.10.0/interpreter/sparql.html">Sparql</a></li> <li><a href="/docs/0.10.0/interpreter/submarine.html">Submarine</a></li> </ul> </li> <li> <a href="#" data-toggle="dropdown" class="dropdown-toggle">More</a> <ul class="dropdown-menu scrollable-menu" style="right: 0; left: auto;"> <li class="title">Extending Zeppelin</li> <li><a href="/docs/0.10.0/development/writing_zeppelin_interpreter.html">Writing Zeppelin Interpreter</a></li> <li role="separator" class="divider"></li> <li class="title">Helium (Experimental)</li> <li><a href="/docs/0.10.0/development/helium/overview.html">Overview</a></li> <li><a href="/docs/0.10.0/development/helium/writing_application.html">Writing Helium Application</a></li> <li><a href="/docs/0.10.0/development/helium/writing_spell.html">Writing Helium Spell</a></li> <li><a href="/docs/0.10.0/development/helium/writing_visualization_basic.html">Writing Helium Visualization: Basics</a></li> <li><a href="/docs/0.10.0/development/helium/writing_visualization_transformation.html">Writing Helium Visualization: Transformation</a></li> <li role="separator" class="divider"></li> <li class="title">Contributing to Zeppelin</li> <li><a href="/docs/0.10.0/setup/basics/how_to_build.html">How to Build Zeppelin</a></li> <li><a href="/docs/0.10.0/development/contribution/useful_developer_tools.html">Useful Developer Tools</a></li> <li><a href="/docs/0.10.0/development/contribution/how_to_contribute_code.html">How to Contribute (code)</a></li> <li><a href="/docs/0.10.0/development/contribution/how_to_contribute_website.html">How to Contribute (website)</a></li> <li role="separator" class="divider"></li> <li class="title">External Resources</li> <li><a target="_blank" href="https://zeppelin.apache.org/community.html">Mailing List</a></li> <li><a target="_blank" href="https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Home">Apache Zeppelin Wiki</a></li> <li><a target="_blank" href="http://stackoverflow.com/questions/tagged/apache-zeppelin">Stackoverflow Questions about Zeppelin</a></li> </ul> </li> <li> <a href="/docs/0.10.0/search.html" class="nav-search-link"> </a> </li> </ul> </nav> </div> </div> <div class="content">  <div class="row"> <div class="col-md-12">  <h1>Zeppelin Tutorial</h1> <div id="toc"></div> This tutorial walks you through some of the fundamental Zeppelin concepts. We will assume you have already installed Zeppelin. If not, please see <a href="./install.html">here</a> first. Current main backend processing engine of Zeppelin is <a href="https://spark.apache.org">Apache Spark</a>. If you're new to this system, you might want to start by getting an idea of how it processes data to get the most out of Zeppelin. <h2>Tutorial with Local File</h2> <h3>Data Refine</h3> Before you start Zeppelin tutorial, you will need to download <a href="http://archive.ics.uci.edu/ml/machine-learning-databases/00222/bank.zip">bank.zip</a>. First, to transform csv format data into RDD of <code>Bank</code> objects, run following script. This will also remove header using <code>filter</code> function. <div class="highlight"><pre><code class="scala language-scala" data-lang="scala">val bankText = sc.textFile("yourPath/bank/bank-full.csv") case class Bank(age:Integer, job:String, marital : String, education : String, balance : Integer) // split each line, filter out header (starts with "age"), and map it into Bank case class val bank = bankText.map(s=>s.split(";")).filter(s=>s(0)!="\"age\"").map( s=>Bank(s(0).toInt, s(1).replaceAll("\"", ""), s(2).replaceAll("\"", ""), s(3).replaceAll("\"", ""), s(5).replaceAll("\"", "").toInt ) ) // convert to DataFrame and create temporal table bank.toDF().registerTempTable("bank") </code></pre></div> <h3>Data Retrieval</h3> Suppose we want to see age distribution from <code>bank</code>. To do this, run: <div class="highlight"><pre><code class="sql language-sql" data-lang="sql">%sql select age, count(1) from bank where age < 30 group by age order by age </code></pre></div> You can make input box for setting age condition by replacing <code>30</code> with <code>${maxAge=30}</code>. <div class="highlight"><pre><code class="sql language-sql" data-lang="sql">%sql select age, count(1) from bank where age < ${maxAge=30} group by age order by age </code></pre></div> Now we want to see age distribution with certain marital status and add combo box to select marital status. Run: <div class="highlight"><pre><code class="sql language-sql" data-lang="sql">%sql select age, count(1) from bank where marital="${marital=single,single|divorced|married}" group by age order by age </code></pre></div> <h2>Tutorial with Streaming Data</h2> <h3>Data Refine</h3> Since this tutorial is based on Twitter's sample tweet stream, you must configure authentication with a Twitter account. To do this, take a look at <a href="https://databricks-training.s3.amazonaws.com/realtime-processing-with-spark-streaming.html#twitter-credential-setup">Twitter Credential Setup</a>. After you get API keys, you should fill out credential related values(<code>apiKey</code>, <code>apiSecret</code>, <code>accessToken</code>, <code>accessTokenSecret</code>) with your API keys on following script. This will create a RDD of <code>Tweet</code> objects and register these stream data as a table: <div class="highlight"><pre><code class="scala language-scala" data-lang="scala">import org.apache.spark.streaming._ import org.apache.spark.streaming.twitter._ import org.apache.spark.storage.StorageLevel import scala.io.Source import scala.collection.mutable.HashMap import java.io.File import org.apache.log4j.Logger import org.apache.log4j.Level import sys.process.stringSeqToProcess /** Configures the Oauth Credentials for accessing Twitter */ def configureTwitterCredentials(apiKey: String, apiSecret: String, accessToken: String, accessTokenSecret: String) { val configs = new HashMap[String, String] ++= Seq( "apiKey" -> apiKey, "apiSecret" -> apiSecret, "accessToken" -> accessToken, "accessTokenSecret" -> accessTokenSecret) println("Configuring Twitter OAuth") configs.foreach{ case(key, value) => if (value.trim.isEmpty) { throw new Exception("Error setting authentication - value for " + key + " not set") } val fullKey = "twitter4j.oauth." + key.replace("api", "consumer") System.setProperty(fullKey, value.trim) println("\tProperty " + fullKey + " set as [" + value.trim + "]") } println() } // Configure Twitter credentials val apiKey = "xxxxxxxxxxxxxxxxxxxxxxxxx" val apiSecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" val accessToken = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" val accessTokenSecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" configureTwitterCredentials(apiKey, apiSecret, accessToken, accessTokenSecret) import org.apache.spark.streaming.twitter._ val ssc = new StreamingContext(sc, Seconds(2)) val tweets = TwitterUtils.createStream(ssc, None) val twt = tweets.window(Seconds(60)) case class Tweet(createdAt:Long, text:String) twt.map(status=> Tweet(status.getCreatedAt().getTime()/1000, status.getText()) ).foreachRDD(rdd=> // Below line works only in spark 1.3.0. // For spark 1.1.x and spark 1.2.x, // use rdd.registerTempTable("tweets") instead. rdd.toDF().registerAsTable("tweets") ) twt.print ssc.start() </code></pre></div> <h3>Data Retrieval</h3> For each following script, every time you click run button you will see different result since it is based on real-time data. Let's begin by extracting maximum 10 tweets which contain the word girl. <div class="highlight"><pre><code class="sql language-sql" data-lang="sql">%sql select * from tweets where text like '%girl%' limit 10 </code></pre></div> This time suppose we want to see how many tweets have been created per sec during last 60 sec. To do this, run: <div class="highlight"><pre><code class="sql language-sql" data-lang="sql">%sql select createdAt, count(1) from tweets group by createdAt order by createdAt </code></pre></div> You can make user-defined function and use it in Spark SQL. Let's try it by making function named <code>sentiment</code>. This function will return one of the three attitudes( positive, negative, neutral ) towards the parameter. <div class="highlight"><pre><code class="scala language-scala" data-lang="scala">def sentiment(s:String) : String = { val positive = Array("like", "love", "good", "great", "happy", "cool", "the", "one", "that") val negative = Array("hate", "bad", "stupid", "is") var st = 0; val words = s.split(" ") positive.foreach(p => words.foreach(w => if(p==w) st = st+1 ) ) negative.foreach(p=> words.foreach(w=> if(p==w) st = st-1 ) ) if(st>0) "positivie" else if(st<0) "negative" else "neutral" } // Below line works only in spark 1.3.0. // For spark 1.1.x and spark 1.2.x, // use sqlc.registerFunction("sentiment", sentiment _) instead. sqlc.udf.register("sentiment", sentiment _) </code></pre></div> To check how people think about girls using <code>sentiment</code> function we've made above, run this: <div class="highlight"><pre><code class="sql language-sql" data-lang="sql">%sql select sentiment(text), count(1) from tweets where text like '%girl%' group by sentiment(text) </code></pre></div> </div> </div> <hr> <footer>  </footer> </div> <script type="text/javascript"> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-45176241-5', 'zeppelin.apache.org'); ga('require', 'linkid', 'linkid.js'); ga('send', 'pageview'); </script> </body> </html>

CINXE.COM

Apache Zeppelin 0.10.0 Documentation: Apache Zeppelin Tutorial