CINXE.COM
Samza - Realtime Notifications at Redfin
<!DOCTYPE html> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <html lang="en"> <head> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <meta charset="utf-8"> <title>Samza - Realtime Notifications at Redfin</title> <link rel="apple-touch-icon-precomposed" sizes="57x57" href="/img/favicon/apple-touch-icon-57x57.png" /> <link rel="apple-touch-icon-precomposed" sizes="114x114" href="/img/favicon/apple-touch-icon-114x114.png" /> <link rel="apple-touch-icon-precomposed" sizes="72x72" href="/img/favicon/apple-touch-icon-72x72.png" /> <link rel="apple-touch-icon-precomposed" sizes="144x144" href="/img/favicon/apple-touch-icon-144x144.png" /> <link rel="apple-touch-icon-precomposed" sizes="60x60" href="/img/favicon/apple-touch-icon-60x60.png" /> <link rel="apple-touch-icon-precomposed" sizes="120x120" href="/img/favicon/apple-touch-icon-120x120.png" /> <link rel="apple-touch-icon-precomposed" sizes="76x76" href="/img/favicon/apple-touch-icon-76x76.png" /> <link rel="apple-touch-icon-precomposed" sizes="152x152" href="/img/favicon/apple-touch-icon-152x152.png" /> <link rel="icon" type="image/png" href="/img/favicon/favicon-196x196.png" sizes="196x196" /> <link rel="icon" type="image/png" href="/img/favicon/favicon-96x96.png" sizes="96x96" /> <link rel="icon" type="image/png" href="/img/favicon/favicon-32x32.png" sizes="32x32" /> <link rel="icon" type="image/png" href="/img/favicon/favicon-16x16.png" sizes="16x16" /> <link rel="icon" type="image/png" href="/img/favicon/favicon-128.png" sizes="128x128" /> <meta name="application-name" content="https://samza.apache.org" /> <meta name="msapplication-TileColor" content="#FFFFFF" /> <meta name="msapplication-TileImage" content="/img/favicon/mstile-144x144.png" /> <meta name="msapplication-square70x70logo" content="/img/favicon/mstile-70x70.png" /> <meta name="msapplication-square150x150logo" content="/img/favicon/mstile-150x150.png" /> <meta name="msapplication-wide310x150logo" content="/img/favicon/mstile-310x150.png" /> <meta name="msapplication-square310x310logo" content="/img/favicon/mstile-310x310.png" /> <link href="/css/ionicons.min.css" rel="stylesheet"> <link href="/css/google-fonts.css" rel="stylesheet"> <link href="/css/syntax.css" rel="stylesheet"/> <link rel="stylesheet" href="/css/main.new.css" /> </head> <body class="page page--case-study"> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <div class="main-navigation" data-plugin="menu"> <div class="main-navigation__toggle" data-menu-closed> <i class="icon ion-md-menu"></i> </div> <div class="main-navigation__toggle main-navigation__toggle--opened" data-menu-opened> <i class="icon ion-md-close"></i> </div> <div class="main-navigation__inner"> <div class="main-navigation__logo"> <a href="/"> <img class="main-navigation__logo-img" src="/img/samza-logo.png" srcset="/img/samza-logo.png 1x, /img/samza-logo@2x.png 2x" alt="Samza Logo" /> </a> </div> <div class="main-navigation__items" data-menu-opened> <a class="main-navigation__item" href="/">Home</a> <a class="main-navigation__item" href="/learn/documentation/latest/core-concepts/core-concepts.html">Docs</a> <a class="main-navigation__item" href="/powered-by/">Powered By</a> <a class="main-navigation__item" href="/startup/download/">Downloads</a> <a class="main-navigation__item" href="/blog/">Blog</a> <div class="main-navigation__item main-navigation__item--group"> <div class="main-navigation__item-group-title"> Community <i class="icon ion-md-arrow-dropdown"></i> </div> <div class="main-navigation__item-group-list"> <a class="main-navigation__item" href="/community/contact-us.html">Contact Us</a> <a class="main-navigation__item" href="/contribute/contributors-corner.html">Contributor's Corner</a> <a class="main-navigation__item" href="/community/committers.html">PMC Members and committers</a> <a class="main-navigation__item" href="/meetups/">Talks and Meetups</a> </div> </div> </div> </div> </div> <div class="container"> <div class="container__toggle"> <i class="icon ion-md-arrow-dropleft-circle container__toggle-icon"></i> <i class="icon ion-md-arrow-dropright-circle container__toggle-icon container__toggle-icon--opened"></i> </div> <!-- There is only one menu, but made it as a no-output collection to grab data only --> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <div class="side-navigation"> <!-- Start Group --> <div class="side-navigation__group side-navigation__group--has-nested" data-plugin="sub-menu" data-sub-menu-show-class="side-navigation__group--has-nested-visible"> <!-- Make menu_title, and start items group if needed --> <div class="side-navigation__group-title"> <i class="side-navigation__group-title-icon icon ion-md-arrow-dropdown"></i> Getting Started </div> <div class="side-navigation__group-items " data-sub-menu > <!-- Handle sub navigation items from data --> <a class="side-navigation__group-item" data-match-active="" href="/startup/quick-start/latest/">QuickStart</a> <a class="side-navigation__group-item" data-match-active="" href="/startup/code-examples/latest/">Code Examples</a> <!-- Handle sub nagivation from site collections --> <!-- Close sub nav group --> </div> <!-- Close menu group --> </div> <!-- Start Group --> <div class="side-navigation__group side-navigation__group--has-nested" data-plugin="sub-menu" data-sub-menu-show-class="side-navigation__group--has-nested-visible"> <!-- Make menu_title, and start items group if needed --> <div class="side-navigation__group-title"> <i class="side-navigation__group-title-icon icon ion-md-arrow-dropdown"></i> Documentation </div> <div class="side-navigation__group-items side-navigation__group-has-submenus" data-sub-menu data-documentation="/learn/documentation/latest/"> <!-- Handle sub navigation items from data --> <!-- Handle sub nagivation from site collections --> <!-- Close sub nav group --> </div> <!-- Close menu group --> </div> <!-- Start Group --> <div class="side-navigation__group side-navigation__group--has-nested" data-plugin="sub-menu" data-sub-menu-show-class="side-navigation__group--has-nested-visible"> <!-- Make menu_title, and start items group if needed --> <div class="side-navigation__group-title"> <i class="side-navigation__group-title-icon icon ion-md-arrow-dropdown"></i> Releases </div> <div class="side-navigation__group-items " data-sub-menu > <!-- Handle sub navigation items from data --> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.8.0">1.8.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.7.0">1.7.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.6.0">1.6.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.5.1">1.5.1</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.5.0">1.5.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.4.0">1.4.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.3.1">1.3.1</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.3.0">1.3.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.2.0">1.2.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.1.0">1.1.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/1.0.0">1.0.0</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/0.14">0.14</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/0.13">0.13</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/0.12">0.12</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/0.11">0.11</a> <a class="side-navigation__group-item" data-match-active="" href="/releases/0.10">0.10</a> <!-- Handle sub nagivation from site collections --> <!-- Close sub nav group --> </div> <!-- Close menu group --> </div> <!-- Start Group --> <div class="side-navigation__group"> <!-- Make menu_title, and start items group if needed --> <a class="side-navigation__group-title" data-plugin="top-menu" data-match-active="" href="/blog/"> Blog </a> <!-- Handle sub navigation items from data --> <!-- Handle sub nagivation from site collections --> <!-- Close sub nav group --> <!-- Close menu group --> </div> <!-- Start Group --> <div class="side-navigation__group side-navigation__group--has-nested" data-plugin="sub-menu" data-sub-menu-show-class="side-navigation__group--has-nested-visible"> <!-- Make menu_title, and start items group if needed --> <div class="side-navigation__group-title"> <i class="side-navigation__group-title-icon icon ion-md-arrow-dropdown"></i> Community </div> <div class="side-navigation__group-items " data-sub-menu > <!-- Handle sub navigation items from data --> <a class="side-navigation__group-item" data-match-active="" href="/community/contact-us.html">Contact Us</a> <a class="side-navigation__group-item" data-match-active="" href="/contribute/contributors-corner.html">Contributor's Corner</a> <a class="side-navigation__group-item" data-match-active="" href="/contribute/enhancement-proposal.html">Enhancement Proposal</a> <a class="side-navigation__group-item" data-match-active="" href="/community/committers.html">PMC members & Committers</a> <a class="side-navigation__group-item" data-match-active="" href="/meetups/">Talks and Meetups</a> <!-- Handle sub nagivation from site collections --> <!-- Close sub nav group --> </div> <!-- Close menu group --> </div> <!-- Start Group --> <div class="side-navigation__group side-navigation__group--has-nested" data-plugin="sub-menu" data-sub-menu-show-class="side-navigation__group--has-nested-visible"> <!-- Make menu_title, and start items group if needed --> <div class="side-navigation__group-title"> <i class="side-navigation__group-title-icon icon ion-md-arrow-dropdown"></i> Case Studies </div> <div class="side-navigation__group-items " data-sub-menu > <!-- Handle sub navigation items from data --> <a class="side-navigation__group-item" data-match-active="exact" href="/case-studies/">View All</a> <hr> <!-- Handle sub nagivation from site collections --> <a class="side-navigation__group-item" href="/case-studies/ebay" data-match-active="">eBay</a> <a class="side-navigation__group-item" href="/case-studies/tripadvisor" data-match-active="">TripAdvisor</a> <a class="side-navigation__group-item" href="/case-studies/slack" data-match-active="">Slack</a> <a class="side-navigation__group-item" href="/case-studies/optimizely" data-match-active="">Optimizely</a> <a class="side-navigation__group-item" href="/case-studies/redfin" data-match-active="">Redfin</a> <a class="side-navigation__group-item" href="/case-studies/linkedin" data-match-active="">LinkedIn</a> <!-- Close sub nav group --> </div> <!-- Close menu group --> </div> </div> <div class="section"> <div class="content"> <div class="case-study-header"> <div class="case-study-header__img" style="background-image: url('https://logo.clearbit.com/redfin.com?size=256');"></div> <div class="case-study-header__headings"> <h3> <a href="https://redfin.com" rel="nofollow" target="_blank"> redfin.com </a> </h3> <h2>Realtime Notifications at Redfin</h2> </div> </div> <!--more--> <p>Redfin is a leading full-service real estate brokerage that uses modern technology to help people buy and sell homes. Notification is the critical feature to communicate with Redfin鈥檚 customers, notification includes recommendations, instant emails, scheduled digests and push notifications. Thousands of emails are delivered to customers every minute at peak.</p> <p>The notification system used to be a monolithic system, which served the company well. However, as the business grew and requirements evolved, it became harder and harder to maintain and scale.</p> <p><img src="/img/case-studies/redfin.svg" alt="Samza pipeline at Redfin" /></p> <p>The engineering team at Redfin decided to replace the existing system with Samza primarily for Samza鈥檚 performance, scalability, support for stateful processing and Kafka-integration. A multi-stage stream processing pipeline was developed. At the <em>Identify</em> stage, external events such as new listings are identified as candidates for sending a new notification; Then potential recipients of notifications are determined by analyzing data in the events and customer profiles. The results are grouped by customer at the end of each time window during the <em>Match</em> Stage. Once notifications and recipients are identified, the <em>Organize</em> stage further joins them with additional data-sources (eg: notification settings, customer profiles) leveraging Samza鈥檚 support for local state. It makes heavy use of RocksDB to store and merge individual notifications before sending them to customers. Finally, the notifications are formatted at the <em>Format</em> stage and sent to the delivery system at the <em>Notify</em> stage.</p> <p>With the new notification system based on Apache Samza, Redfin observed that</p> <ul> <li>It is now easier to add support for new use cases</li> <li>The new system is more performant and horizontally scalable</li> <li>Reduced pressure on downstream services due to the use of local RocksDB state store</li> <li>Processing stages can be scaled individually since they are isolated</li> </ul> <p>In addition to the notifications platform, other engineering teams at Redfin also use Samza for calculating business metrics, document processing, event scheduling etc.,</p> <p>Key Samza Features: <em>Stateful processing</em>, <em>Windowing</em>, <em>Kafka-integration</em></p> <p>More information</p> <ul> <li><a href="https://www.youtube.com/watch?v=cfy0xjJJf7Y">https://www.youtube.com/watch?v=cfy0xjJJf7Y</a></li> <li><a href="https://www.redfin.com/">https://www.redfin.com/</a></li> </ul> <hr/> <h3> More Case Studies... </h3> <div class="case-studies"> <div class="case-studies-list"> <div class="case-studies-list__item"> <a class="case-studies-list__item-logo" href="/case-studies/ebay" title="eBay"> <div class="case-studies-list__item-logo-bg" style="background-image: url('https://logo.clearbit.com/ebay.com?size=256');"></div> </a> </div> <div class="case-studies-list__item"> <a class="case-studies-list__item-logo" href="/case-studies/tripadvisor" title="TripAdvisor"> <div class="case-studies-list__item-logo-bg" style="background-image: url('https://logo.clearbit.com/tripadvisor.com?size=256');"></div> </a> </div> <div class="case-studies-list__item"> <a class="case-studies-list__item-logo" href="/case-studies/slack" title="Slack"> <div class="case-studies-list__item-logo-bg" style="background-image: url('https://logo.clearbit.com/slack.com?size=256');"></div> </a> </div> <div class="case-studies-list__item"> <a class="case-studies-list__item-logo" href="/case-studies/optimizely" title="Optimizely"> <div class="case-studies-list__item-logo-bg" style="background-image: url('https://logo.clearbit.com/optimizely.com?size=256');"></div> </a> </div> <div class="case-studies-list__item"> <a class="case-studies-list__item-logo" href="/case-studies/linkedin" title="LinkedIn"> <div class="case-studies-list__item-logo-bg" style="background-image: url('https://logo.clearbit.com/linkedin.com?size=256');"></div> </a> </div> </div> </div> </div> </div> </div> <!-- footer starts here --> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <footer> <div class="footer-inner"> <div class="side-by-side"> <div> <div class="footer__heading">Learn More</div> <div class="footer__items"> <a class="footer__item" href="/meetups/">Meetups</a> <a class="footer__item" href="/blog/">Blog</a> <a class="footer__item" href="/learn/documentation/latest/introduction/background.html">About</a> </div> </div> <div> <div class="footer__heading">Community</div> <div class="footer__items"> <a class="footer__item" href="/community/contact-us.html">Contact Us</a> <a class="footer__item" href="/contribute/contributors-corner.html">Contributors' Corner</a> <a class="footer__item" href="/community/committers.html">PMC members and committers</a> <a class="footer__item" href="/powered-by/">Powered By</a> </div> </div> <div> <div class="quick-links"> <a class="quick-link" href="/startup/download" target="_blank"> <i class="icon ion-md-download"></i> </a> <a class="quick-link" href="https://git-wip-us.apache.org/repos/asf?p=samza.git;a=tree" target="_blank"> <i class="icon ion-md-code"></i> </a> <a class="quick-link" href="https://twitter.com/samzastream" target="_blank"> <i class="icon ion-logo-twitter"></i> </a> </div> <p> <script>document.write(new Date().getFullYear());</script> © samza.apache.org</p> </div> </div> </div> </footer> <!-- Google Analytics --> <script> (function (i, s, o, g, r, a, m) { i['GoogleAnalyticsObject'] = r; i[r] = i[r] || function () { (i[r].q = i[r].q || []).push(arguments) }, i[r].l = 1 * new Date(); a = s.createElement(o), m = s.getElementsByTagName(o)[0]; a.async = 1; a.src = g; m.parentNode.insertBefore(a, m) })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga'); ga('create', 'UA-43122768-1', 'apache.org'); ga('send', 'pageview'); </script> <script src="/js/main.new.js"></script> </body> </html>