CINXE.COM

<!doctype html> <html> <head> <meta charset="utf-8">  <meta content="IE=edge,chrome=1" http-equiv="X-UA-Compatible"> <meta name="google-site-verification" content="9N7qTOUYyX4kYfXYc0OIomWJku3PVvGrf6oTNWg2CHI" />  <title>Apache DataFu</title> <link href="/stylesheets/all.css" rel="stylesheet" /><link href="/stylesheets/highlight.css" rel="stylesheet" /> <script src="/javascripts/all.js"></script> <script type="text/javascript"> var _gaq = _gaq || []; _gaq.push(['_setAccount', 'UA-30533336-2']); _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })(); </script> </head> <body class="index"> <div class="container"> <div class="header"> <ul class="nav nav-pills pull-right"> <li><a href="/blog">Blog</a></li> </ul> <h3 class="header-title"><a href="/">Apache DataFu™</a></h3> </div> <div class="row"> <div class="col-md-3"> <h4>Getting Started</h4> <ul class="nav nav-pills nav-stacked"> <li><a href="/docs/download.html">Download</a></li> <li><a href="/docs/spark/getting-started.html">DataFu Spark</a></li> <li><a href="/docs/datafu/getting-started.html">DataFu Pig</a></li> <li><a href="/docs/hourglass/getting-started.html">DataFu Hourglass</a></li> </ul> <h4>DataFu Spark Docs</h4> <ul class="nav nav-pills nav-stacked"> <li><a href="/docs/spark/guide.html">Guide</a></li> <li><a href="https://datafu.apache.org/docs/spark/2.0.0/">Scaladocs</a></li> </ul> <h4>DataFu Pig Docs</h4> <ul class="nav nav-pills nav-stacked"> <li><a href="/docs/datafu/guide.html">Guide</a></li> <li><a href="https://datafu.apache.org/docs/datafu/1.6.1/">Javadocs</a></li> </ul> <h4>DataFu Hourglass Docs</h4> <ul class="nav nav-pills nav-stacked"> <li><a href="/docs/hourglass/concepts.html">Concepts</a></li> <li><a href="https://datafu.apache.org/docs/hourglass/1.6.1/">Javadocs</a></li> </ul> <h4>Community</h4> <ul class="nav nav-pills nav-stacked"> <li><a href="/community/mailing-lists.html">Mailing Lists</a></li> <li><a href="/community/contributing.html">Contributing</a></li> <li><a href="/community/issues.html">Issues</a></li> <li><a href="https://cwiki.apache.org/confluence/display/DATAFU/DataFu+Home">Wiki</a></li> </ul> <h4>Apache Software Foundation</h4> <ul class="nav nav-pills nav-stacked"> <li><a href="https://www.apache.org/" target="_blank">Homepage</a></li> <li><a href="https://www.apache.org/licenses/" target="_blank">License</a></li> <li><a href="https://www.apache.org/foundation/sponsorship.html" target="_blank">Sponsorship</a></li> <li><a href="https://www.apache.org/security/" target="_blank">Security</a></li> <li><a href="https://www.apache.org/foundation/thanks.html" target="_blank">Thanks</a></li> </ul> </div> <div class="col-md-7"> <h1 id="apache-datafu">Apache DataFu</h1> <p>Apache DataFu™ is a collection of libraries for working with large-scale data in Hadoop. The project was inspired by the need for stable, well-tested libraries for data mining and statistics.</p> <p>It consists of three libraries:</p> <ul> <li><strong><a href="/docs/spark/getting-started.html">Apache DataFu Spark</a></strong>: a collection of utils and user-defined functions for <a href="http://spark.apache.org/">Apache Spark</a></li> <li><strong>Apache DataFu Pig</strong>: a collection of user-defined functions and macros for <a href="http://pig.apache.org/">Apache Pig</a></li> <li><strong>Apache DataFu Hourglass</strong>: an incremental processing framework for <a href="http://hadoop.apache.org/">Apache Hadoop</a> in MapReduce</li> </ul> <p>To begin using it, see our <a href="/docs/download.html">Download</a> page. If you'd like to help contribute, see <a href="/community/contributing.html">Contributing</a>.</p> <h2 id="about-the-project">About the Project</h2> <h3 id="apache-datafu-spark">Apache DataFu Spark</h3> <p>Apache DataFu Spark is a collection of utils and user-defined functions for <a href="http://spark.apache.org/">Apache Spark</a>. This library is based on an internal PayPal project and was open sourced in 2019. It has been used by production workflows at PayPal since 2017. All of the codes is unit tested to ensure quality.</p> <p>Check out the <a href="/docs/spark/getting-started.html">Getting Started</a> guide to learn more.</p> <h3 id="apache-datafu-pig">Apache DataFu Pig</h3> <p>Apache DataFu Pig is a collection of useful user-defined functions for data analysis in <a href="http://pig.apache.org/">Apache Pig</a>. This library was open sourced in 2010 and continues to receive contributions, having reached 1.0 in September, 2013. It has been used by production workflows at LinkedIn since 2010. It is also included in Cloudera's <a href="http://www.cloudera.com/content/cloudera/en/products-and-services/cdh.html">CDH</a> and <a href="http://bigtop.apache.org/">Apache Bigtop</a>. All of the UDFs are unit tested to ensure quality.</p> <p>Check out the <a href="/docs/datafu/getting-started.html">Getting Started</a> guide to learn more.</p> <h3 id="apache-datafu-hourglass">Apache DataFu Hourglass</h3> <p>Apache DataFu Hourglass is a library for incrementally processing data using Hadoop MapReduce. This library was inspired by the prevalance of sliding window computations over daily tracking data at LinkedIn. Computations such as these typically happen at regular intervals (e.g. daily, weekly), and therefore the sliding nature of the computations means that much of the work is unnecessarily repeated. DataFu's Hourglass was created to make these computations more efficient, yielding sometimes 50-95% reductions in computational resources.</p> <p>Work on this library began in early 2013, which led to a <a href="http://www.slideshare.net/matthewterencehayes/hourglass-27038297">paper</a> <a href="http://www.slideshare.net/matthewterencehayes/hourglass-a-library-for-incremental-processing-on-hadoop">presented</a> at <a href="http://cci.drexel.edu/bigdata/bigdata2013/">IEEE BigData 2013</a>. It is currently in production use at LinkedIn.</p> <p>Check out the <a href="/docs/hourglass/getting-started.html">Getting Started</a> guide to learn more.</p> </div> </div> <div class="footer"> <div class="feather"> <a href="http://www.apache.org/" target="_blank"><img src="/images/feather.png" alt="Apache Feather" title="Apache Feather"/></a> </div> <div class="copyright"> Copyright © 2011-2024 The Apache Software Foundation, Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br> Apache DataFu, DataFu, Apache Pig, Apache Hadoop, Hadoop, Apache, and the Apache feather logo are either registered trademarks or trademarks of the <a href="http://www.apache.org/">Apache Software Foundation</a> in the United States and other countries. </div> </div> </div> </body> </html>