CINXE.COM
Samza - On using Samza at Optimizely to compute analytics over session windows.
<!DOCTYPE html> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <html lang="en"> <head> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <meta charset="utf-8"> <title>Samza - On using Samza at Optimizely to compute analytics over session windows.</title> <link rel="apple-touch-icon-precomposed" sizes="57x57" href="/img/favicon/apple-touch-icon-57x57.png" /> <link rel="apple-touch-icon-precomposed" sizes="114x114" href="/img/favicon/apple-touch-icon-114x114.png" /> <link rel="apple-touch-icon-precomposed" sizes="72x72" href="/img/favicon/apple-touch-icon-72x72.png" /> <link rel="apple-touch-icon-precomposed" sizes="144x144" href="/img/favicon/apple-touch-icon-144x144.png" /> <link rel="apple-touch-icon-precomposed" sizes="60x60" href="/img/favicon/apple-touch-icon-60x60.png" /> <link rel="apple-touch-icon-precomposed" sizes="120x120" href="/img/favicon/apple-touch-icon-120x120.png" /> <link rel="apple-touch-icon-precomposed" sizes="76x76" href="/img/favicon/apple-touch-icon-76x76.png" /> <link rel="apple-touch-icon-precomposed" sizes="152x152" href="/img/favicon/apple-touch-icon-152x152.png" /> <link rel="icon" type="image/png" href="/img/favicon/favicon-196x196.png" sizes="196x196" /> <link rel="icon" type="image/png" href="/img/favicon/favicon-96x96.png" sizes="96x96" /> <link rel="icon" type="image/png" href="/img/favicon/favicon-32x32.png" sizes="32x32" /> <link rel="icon" type="image/png" href="/img/favicon/favicon-16x16.png" sizes="16x16" /> <link rel="icon" type="image/png" href="/img/favicon/favicon-128.png" sizes="128x128" /> <meta name="application-name" content="https://samza.apache.org" /> <meta name="msapplication-TileColor" content="#FFFFFF" /> <meta name="msapplication-TileImage" content="/img/favicon/mstile-144x144.png" /> <meta name="msapplication-square70x70logo" content="/img/favicon/mstile-70x70.png" /> <meta name="msapplication-square150x150logo" content="/img/favicon/mstile-150x150.png" /> <meta name="msapplication-wide310x150logo" content="/img/favicon/mstile-310x150.png" /> <meta name="msapplication-square310x310logo" content="/img/favicon/mstile-310x310.png" /> <link href="/css/ionicons.min.css" rel="stylesheet"> <link href="/css/google-fonts.css" rel="stylesheet"> <link href="/css/syntax.css" rel="stylesheet"/> <link rel="stylesheet" href="/css/main.new.css" /> </head> <body class="page page--case-study"> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <div class="main-navigation" data-plugin="menu"> <div class="main-navigation__toggle" data-menu-closed> <i class="icon ion-md-menu"></i> </div> <div class="main-navigation__toggle main-navigation__toggle--opened" data-menu-opened> <i class="icon ion-md-close"></i> </div> <div class="main-navigation__inner"> <div class="main-navigation__logo"> <a href="/"> <img class="main-navigation__logo-img" src="/img/samza-logo.png" srcset="/img/samza-logo.png 1x, /img/samza-logo@2x.png 2x" alt="Samza Logo" /> </a> </div> <div class="main-navigation__items" data-menu-opened> <a class="main-navigation__item" href="/">Home</a> <a class="main-navigation__item" href="/learn/documentation/latest/core-concepts/core-concepts.html">Docs</a> <a class="main-navigation__item" href="/powered-by/">Powered By</a> <a class="main-navigation__item" href="/startup/download/">Downloads</a> <a class="main-navigation__item" href="/blog/">Blog</a> <div class="main-navigation__item main-navigation__item--group"> <div class="main-navigation__item-group-title"> Community <i class="icon ion-md-arrow-dropdown"></i> </div> <div class="main-navigation__item-group-list"> <a class="main-navigation__item" href="/community/contact-us.html">Contact Us</a> <a class="main-navigation__item" href="/contribute/contributors-corner.html">Contributor's Corner</a> <a class="main-navigation__item" href="/community/committers.html">PMC Members and committers</a> <a class="main-navigation__item" href="/meetups/">Talks and Meetups</a> </div> </div> </div> </div> </div> <div class="container"> <div class="container__toggle"> <i class="icon ion-md-arrow-dropleft-circle container__toggle-icon"></i> <i class="icon ion-md-arrow-dropright-circle container__toggle-icon container__toggle-icon--opened"></i> </div> <!-- There is only one menu, but made it as a no-output collection to grab data only --> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <div class="side-navigation"> <!-- Start Group --> <div class="side-navigation__group side-navigation__group--has-nested" data-plugin="sub-menu" data-sub-menu-show-class="side-navigation__group--has-nested-visible"> <!-- Make menu_title, and start items group if needed --> <div class="side-navigation__group-title"> <i class="side-navigation__group-title-icon icon ion-md-arrow-dropdown"></i> Getting Started </div> <div class="side-navigation__group-items " data-sub-menu > <!-- Handle sub navigation items from data --> <a class="side-navigation__group-item" data-match-active="" href="/startup/quick-start/latest/">QuickStart</a> <a class="side-navigation__group-item" data-match-active="" href="/startup/code-examples/latest/">Code Examples</a> <!-- Handle sub nagivation from site collections --> <!-- Close sub nav group --> </div> <!-- Close menu group --> </div> <!-- Start Group --> <div class="side-navigation__group side-navigation__group--has-nested" data-plugin="sub-menu" data-sub-menu-show-class="side-navigation__group--has-nested-visible"> <!-- Make menu_title, and start items group if needed --> <div class="side-navigation__group-title"> <i class="side-navigation__group-title-icon icon ion-md-arrow-dropdown"></i> Documentation </div> <div class="side-navigation__group-items side-navigation__group-has-submenus" data-sub-menu data-documentation="/learn/documentation/latest/"> <!-- Handle sub navigation items from data --> <!-- Handle sub nagivation from site collections --> <!-- Close sub nav group --> </div> <!-- Close menu group --> </div> <!-- Start Group --> <div class="side-navigation__group side-navigation__group--has-nested" data-plugin="sub-menu" data-sub-menu-show-class="side-navigation__group--has-nested-visible"> <!-- Make menu_title, and start items group if needed --> <div class="side-navigation__group-title"> <i class="side-navigation__group-title-icon icon ion-md-arrow-dropdown"></i> Releases </div> <div class="side-navigation__group-items " data-sub-menu > <!-- Handle sub navigation items from data --> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.8.0">1.8.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.7.0">1.7.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.6.0">1.6.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.5.1">1.5.1</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.5.0">1.5.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.4.0">1.4.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.3.1">1.3.1</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.3.0">1.3.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.2.0">1.2.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.1.0">1.1.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.0.0">1.0.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/0.14">0.14</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/0.13">0.13</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/0.12">0.12</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/0.11">0.11</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/0.10">0.10</a> <!-- Handle sub nagivation from site collections --> <!-- Close sub nav group --> </div> <!-- Close menu group --> </div> <!-- Start Group --> <div class="side-navigation__group"> <!-- Make menu_title, and start items group if needed --> <a class="side-navigation__group-title" data-plugin="top-menu" data-match-active="" href="/blog/"> Blog </a> <!-- Handle sub navigation items from data --> <!-- Handle sub nagivation from site collections --> <!-- Close sub nav group --> <!-- Close menu group --> </div> <!-- Start Group --> <div class="side-navigation__group side-navigation__group--has-nested" data-plugin="sub-menu" data-sub-menu-show-class="side-navigation__group--has-nested-visible"> <!-- Make menu_title, and start items group if needed --> <div class="side-navigation__group-title"> <i class="side-navigation__group-title-icon icon ion-md-arrow-dropdown"></i> Community </div> <div class="side-navigation__group-items " data-sub-menu > <!-- Handle sub navigation items from data --> <a class="side-navigation__group-item" data-match-active="" href="/community/contact-us.html">Contact Us</a> <a class="side-navigation__group-item" data-match-active="" href="/contribute/contributors-corner.html">Contributor's Corner</a> <a class="side-navigation__group-item" data-match-active="" href="/contribute/enhancement-proposal.html">Enhancement Proposal</a> <a class="side-navigation__group-item" data-match-active="" href="/community/committers.html">PMC members & Committers</a> <a class="side-navigation__group-item" data-match-active="" href="/meetups/">Talks and Meetups</a> <!-- Handle sub nagivation from site collections --> <!-- Close sub nav group --> </div> <!-- Close menu group --> </div> <!-- Start Group --> <div class="side-navigation__group side-navigation__group--has-nested" data-plugin="sub-menu" data-sub-menu-show-class="side-navigation__group--has-nested-visible"> <!-- Make menu_title, and start items group if needed --> <div class="side-navigation__group-title"> <i class="side-navigation__group-title-icon icon ion-md-arrow-dropdown"></i> Case Studies </div> <div class="side-navigation__group-items " data-sub-menu > <!-- Handle sub navigation items from data --> <a class="side-navigation__group-item" data-match-active="exact" href="/case-studies/">View All</a> <hr> <!-- Handle sub nagivation from site collections --> <a class="side-navigation__group-item" href="/case-studies/ebay" data-match-active="">eBay</a> <a class="side-navigation__group-item" href="/case-studies/tripadvisor" data-match-active="">TripAdvisor</a> <a class="side-navigation__group-item" href="/case-studies/slack" data-match-active="">Slack</a> <a class="side-navigation__group-item" href="/case-studies/optimizely" data-match-active="">Optimizely</a> <a class="side-navigation__group-item" href="/case-studies/redfin" data-match-active="">Redfin</a> <a class="side-navigation__group-item" href="/case-studies/linkedin" data-match-active="">LinkedIn</a> <!-- Close sub nav group --> </div> <!-- Close menu group --> </div> </div> <div class="section"> <div class="content"> <div class="case-study-header"> <div class="case-study-header__img" style="background-image: url('https://logo.clearbit.com/optimizely.com?size=256');"></div> <div class="case-study-header__headings"> <h3> <a href="https://optimizely.com" rel="nofollow" target="_blank"> optimizely.com </a> </h3> <h2>On using Samza at Optimizely to compute analytics over session windows.</h2> </div> </div> <!--more--> <p>Optimizely is the world’s leading experimentation platform, enabling businesses to deliver continuous experimentation and personalization across websites, mobile apps and connected devices. At Optimizely, billions of events are tracked on a daily basis and session metrics are provided to their users in real-time.</p> <p>Prior to introducing Samza for their realtime computation, the engineering team at Optimizely built their data-pipeline using a complex <a href="http://lambda-architecture.net/">Lambda architecture</a> using <a href="https://medium.com/engineers-optimizely/building-a-scalable-data-pipeline-bfe3f531eb38">Druid and Hbase</a>. Since some session metrics were computed using Map-Reduce jobs, they could be delayed up to hours after the events are received. As business requirements evolved, this solution became more and <a href="https://medium.com/engineers-optimizely/from-batching-to-streaming-real-time-session-metrics-using-samza-part-1-aed2051dd7a3">more challenging</a> to scale.</p> <p>The engineering team at Optimizely turned to stream processing to reduce latencies. In their solution, each up-stream client associates a <em>sessionId</em> with the events it generates. Upon receiving each event, the Samza job extracts various fields (e.g. ip address, location information, browser version, etc) and updates aggregated metrics for the session. At the end of a time-window, the merged metrics for that session are ingested to HBase.</p> <p>With the new solution <br /></p> <ul> <li>The median query latency was reduced from 40+ ms to 5 ms <br /></li> <li>Session metrics are now available in real-time <br /></li> <li>Write-rate to Hbase is reduced, since the metrics are pre-aggregated by Samza<br /></li> <li>Storage requirements on Hbase are drastically reduced <br /></li> <li>Lower development effort thanks to out-of-the-box Kafka integration <br /></li> </ul> <p>Here is a testimonial from Optimizely</p> <p>“At Optimizely, we have built the world’s leading experimentation platform, which ingests billions of click-stream events a day from millions of visitors for analysis. Apache Samza has been a great asset to Optimizely’s Event ingestion pipeline allowing us to perform large scale, real time stream computing such as aggregations (e.g. session computations) and data enrichment on a multiple billion events / day scale. The programming model, durability and the close integration with Apache Kafka fit our needs perfectly” says Vignesh Sukumar, Senior Engineering Manager at Optimizely.</p> <p>In addition to this case-study, Apache Samza is also leveraged for other usecases such as data-enrichment, re-partitioning of event streams and computing realtime metrics etc.</p> <p>Key Samza features: <em>Stateful processing</em>, <em>Windowing</em>, <em>Kafka-integration</em></p> <p>More information</p> <ul> <li><a href="https://medium.com/engineers-optimizely/from-batching-to-streaming-real-time-session-metrics-using-samza-part-1-aed2051dd7a3">From batching to streaming at Optimizely - Part 1</a></li> <li><a href="https://medium.com/engineers-optimizely/from-batching-to-streaming-real-time-session-metrics-using-samza-part-2-b596350a7820">From batching to streaming at Optimizely - Part 2</a></li> </ul> <hr/> <h3> More Case Studies... </h3> <div class="case-studies"> <div class="case-studies-list"> <div class="case-studies-list__item"> <a class="case-studies-list__item-logo" href="/case-studies/ebay" title="eBay"> <div class="case-studies-list__item-logo-bg" style="background-image: url('https://logo.clearbit.com/ebay.com?size=256');"></div> </a> </div> <div class="case-studies-list__item"> <a class="case-studies-list__item-logo" href="/case-studies/tripadvisor" title="TripAdvisor"> <div class="case-studies-list__item-logo-bg" style="background-image: url('https://logo.clearbit.com/tripadvisor.com?size=256');"></div> </a> </div> <div class="case-studies-list__item"> <a class="case-studies-list__item-logo" href="/case-studies/slack" title="Slack"> <div class="case-studies-list__item-logo-bg" style="background-image: url('https://logo.clearbit.com/slack.com?size=256');"></div> </a> </div> <div class="case-studies-list__item"> <a class="case-studies-list__item-logo" href="/case-studies/redfin" title="Redfin"> <div class="case-studies-list__item-logo-bg" style="background-image: url('https://logo.clearbit.com/redfin.com?size=256');"></div> </a> </div> <div class="case-studies-list__item"> <a class="case-studies-list__item-logo" href="/case-studies/linkedin" title="LinkedIn"> <div class="case-studies-list__item-logo-bg" style="background-image: url('https://logo.clearbit.com/linkedin.com?size=256');"></div> </a> </div> </div> </div> </div> </div> </div> <!-- footer starts here --> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <footer> <div class="footer-inner"> <div class="side-by-side"> <div> <div class="footer__heading">Learn More</div> <div class="footer__items"> <a class="footer__item" href="/meetups/">Meetups</a> <a class="footer__item" href="/blog/">Blog</a> <a class="footer__item" href="/learn/documentation/latest/introduction/background.html">About</a> </div> </div> <div> <div class="footer__heading">Community</div> <div class="footer__items"> <a class="footer__item" href="/community/contact-us.html">Contact Us</a> <a class="footer__item" href="/contribute/contributors-corner.html">Contributors' Corner</a> <a class="footer__item" href="/community/committers.html">PMC members and committers</a> <a class="footer__item" href="/powered-by/">Powered By</a> </div> </div> <div> <div class="quick-links"> <a class="quick-link" href="/startup/download" target="_blank"> <i class="icon ion-md-download"></i> </a> <a class="quick-link" href="https://git-wip-us.apache.org/repos/asf?p=samza.git;a=tree" target="_blank"> <i class="icon ion-md-code"></i> </a> <a class="quick-link" href="https://twitter.com/samzastream" target="_blank"> <i class="icon ion-logo-twitter"></i> </a> </div> <p> <script>document.write(new Date().getFullYear());</script> © samza.apache.org</p> </div> </div> </div> </footer> <!-- Google Analytics --> <script> (function (i, s, o, g, r, a, m) { i['GoogleAnalyticsObject'] = r; i[r] = i[r] || function () { (i[r].q = i[r].q || []).push(arguments) }, i[r].l = 1 * new Date(); a = s.createElement(o), m = s.getElementsByTagName(o)[0]; a.async = 1; a.src = g; m.parentNode.insertBefore(a, m) })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga'); ga('create', 'UA-43122768-1', 'apache.org'); ga('send', 'pageview'); </script> <script src="/js/main.new.js"></script> </body> </html>