Yelp Engineering and Product Blog

<!DOCTYPE html> <html > <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, minimum-scale=1, initial-scale=1"> <title>Yelp Engineering and Product Blog</title> <meta name="description" content="Loading data into Redshift with DBT Christopher Arnold, Software Engineer Nov 6, 2024 At Yelp, we embrace innovation and thrive on exploring new possibilities. With our consumers’ ever growing appetite..."> <link rel="amphtml" href="https://engineeringblog.yelp.com/amp/" /> <link rel="stylesheet" href="/css/main.css"> <link rel="canonical" href="https://engineeringblog.yelp.com/"> <link rel="alternate" type="application/rss+xml" title="Yelp Engineering and Product Blog" href="https://engineeringblog.yelp.com/feed.xml" /> <script>(function (d, w) { var supportsSVG = ( !!d.createElementNS && !!d.createElementNS( 'http://www.w3.org/2000/svg', 'svg' ).createSVGRect ); var cdnPath = 'https://s3-media0.fl.yelpcdn.com/assets/srv0/svg_icons/3e34a52fd3aa/assets/svg_sprite.js'; var head = d.getElementsByTagName('head')[0]; function fallback() { var link = d.createElement('link'); link.rel = 'stylesheet'; link.href = 'https://s3-media0.fl.yelpcdn.com/assets/srv0/svg_icons/addea686693c/assets/sprite.css'; head.appendChild(link); } if (!supportsSVG) { fallback(); return; } if (!w.yelp) { w.yelp = {}; } w.yelp.__injectSvgSpritesheet = function (svg) { document.body.insertAdjacentHTML('afterbegin', svg); delete window.yelp.__injectSvgSpritesheet; } function onError() { d.documentElement.className = [ d.documentElement.className, 'icon-svg-unavailable' ].join(' '); fallback(); return true; } var script = d.createElement('script'); script.async = true; script.onerror = onError; script.src = cdnPath + '?callback=window.yelp.__injectSvgSpritesheet'; head.appendChild(script); }(document, window));</script> <noscript> <link rel="stylesheet" href="https://s3-media0.fl.yelpcdn.com/assets/srv0/svg_icons/addea686693c/assets/sprite.css"> </noscript>  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.11.1/dist/katex.min.css" integrity="sha384-zB1R0rpPzHqg7Kpt0Aljp8JPLqbXI3bhnPWROx27a9N0Ll6ZP/+DiW/UqRcLbRjq" crossorigin="anonymous"> <script defer src="https://cdn.jsdelivr.net/npm/katex@0.11.1/dist/katex.min.js" integrity="sha384-y23I5Q6l+B6vatafAwxRu/0oK/79VlbSz7Q9aiSZUvyWYIYsd+qj+o24G5ZU2zJz" crossorigin="anonymous"></script> <script defer src="https://cdn.jsdelivr.net/npm/katex@0.11.1/dist/contrib/auto-render.min.js" integrity="sha384-kWPLUVMOks5AQFrykwIup5lo0m3iMkkHrD0uJ4H5cjeGihAutqP0yW0J6dpFiVkI" crossorigin="anonymous" onload="renderMathInElement(document.body);"></script> </head> <body class="ytype responsive front-page"> <div class="main-header main-content-wrap main-header--slim"> <div class="content-container clearfix"> <div class="arrange arrange--18 arrange--middle"> <div class="arrange_unit"> <h1 class="main-header_logo" id="logo"> <a href="http://yelp.com/engineering">Yelp</a> </h1> </div> <div class="arrange_unit nowrap"> <h2 class="main-header_title"><span>Engineering</span></h2> </div> <div class="arrange_unit arrange_unit--fill"> <div class="main-header_nav nowrap responsive-hidden-small responsive-hidden-medium"> <ul class="header-nav" id="header-nav"> <li class="header-nav_item"> <a class="header-nav_link" href="http://yelp.com/engineering">Home</a> </li> <li class="header-nav_item"> <a class="header-nav_link tabon" href="/">Blog</a> </li> <li class="header-nav_item"> <a class="header-nav_link" href="https://yelp.github.io">Open Source</a> </li> <li class="header-nav_item"> <a class="header-nav_link" href="https://yelp.com/engineering/awe">AWE</a> </li> <li class="header-nav_item"> <a class="header-nav_link" href="https://yelp.com/engineering/whats-it-like">What's it like?</a> </li> </ul> </div> </div> </div> </div> </div> <div class="main-content-wrap--full"> <div class="content-container main-heading"> <h1>Engineering Blog</h1> </div> <div class="content-container"> <div class="posts"> <article class="article-excert"> <div class="clearfix layout-block layout-a row--responsive"> <div class="column column-alpha column--responsive"> <h3 class="alternate"><a href="/2024/11/loading-data-into-redshift-with-dbt.html">Loading data into Redshift with DBT</a></h3> <div class="post-info"> <div class="ypassport ypassport-slim media-block"> <div class="media-avatar"> <div class="photo-box pb-30s"> <img src="/images/authors/christoarno.png" alt="Christopher Arnold, Software Engineer" class="photo-box-img"> </div> </div> <div class="media-story"> <ul class="user-passport-info"> <li class="user-name user-display-name"> Christopher Arnold, Software Engineer <br> </li> <li class="post-date"> Nov 6, 2024 </li> </ul> </div> </div> </div> <div class="post-preview"> <p>At Yelp, we embrace innovation and thrive on exploring new possibilities. With our consumers’ ever growing appetite for data, we recently revisited how we could load data into Redshift more efficiently. In this blog post, we explore how DBT can be used seamlessly with Redshift Spectrum to read data from Data Lake into Redshift to significantly reduce runtime, resolve data quality issues, and improve developer productivity. Starting Point Our method of loading batch data into Redshift had been effective for years, but we continually sought improvements. We primarily used Spark jobs to read S3 data and publish it to our...</p> <p><a href="/2024/11/loading-data-into-redshift-with-dbt.html">Continue reading</a></p> </div> </div> <div class="column column-beta column--responsive"> <img src="/images/previews/dbt-redshift-preview.png"> </div> </div> </article> <article class="article-excert"> <div class="clearfix layout-block layout-a row--responsive"> <div class="column column-alpha column--responsive"> <h3 class="alternate"><a href="/2024/10/how-we-improved-our-android-navigation-performance-by-~30.html">How we improved our Android navigation performance by ~30%</a></h3> <div class="post-info"> <div class="ypassport ypassport-slim media-block"> <div class="media-avatar"> <div class="photo-box pb-30s"> <img src="/images/authors/paulm.jpg" alt="Paul Martin, Core Android Tech Lead" class="photo-box-img"> </div> </div> <div class="media-story"> <ul class="user-passport-info"> <li class="user-name user-display-name"> Paul Martin, Core Android Tech Lead <br> </li> <li class="post-date"> Oct 8, 2024 </li> </ul> </div> </div> </div> <div class="post-preview"> <p>In 2019, Yelp’s Core Android team led an effort to boost navigation performance in Yelp’s Consumer app. We switched from building screens with multiple separate activities to using fragments inside a single activity. In this blog post, we’ll cover our solution, how we approached the migration and share learnings from along the way as well as performance wins. Where we started circa 2018 Navigating between screens in an Android app is often when the app and device are under the most strain. The new screen and its dependencies are quickly created, which can lead to slow or frozen frames. Prior...</p> <p><a href="/2024/10/how-we-improved-our-android-navigation-performance-by-~30.html">Continue reading</a></p> </div> </div> <div class="column column-beta column--responsive"> <img src="/images/previews/2024-10-08-android-logo.jpg"> </div> </div> </article> <article class="article-excert"> <div class="clearfix layout-block layout-a row--responsive"> <div class="column column-alpha column--responsive"> <h3 class="alternate"><a href="/2024/10/migrating-from-postgres-to-mysql.html">Migrating in-place from PostgreSQL to MySQL</a></h3> <div class="post-info"> <div class="ypassport ypassport-slim media-block"> <div class="media-avatar"> <div class="photo-box pb-30s"> <img src="/images/authors/alext.png" alt="Alex Toumazis, Software Engineer" class="photo-box-img"> </div> </div> <div class="media-story"> <ul class="user-passport-info"> <li class="user-name user-display-name"> Alex Toumazis, Software Engineer <br> </li> <li class="post-date"> Oct 7, 2024 </li> </ul> </div> </div> </div> <div class="post-preview"> <p>The Yelp Reservations service (yelp_res) is the service that powers reservations on Yelp. It was acquired along with Seatme in 2013, and is a Django service and webapp. It powers the reservation backend and logic for Yelp Guest Manager, our iPad app for restaurants, and handles diner and partner flows that create reservations. Along with that, it serves a web UI and backend API for our Yelp Reservations app, which has been superseded by Yelp Guest Manager but is still used by many of our restaurant customers. This service was built using a DB-centric architecture, and uses a “DB sync”...</p> <p><a href="/2024/10/migrating-from-postgres-to-mysql.html">Continue reading</a></p> </div> </div> <div class="column column-beta column--responsive"> <img src="/images/previews/2024-09-27-postgresmysql.jpg"> </div> </div> </article> <article class="article-excert"> <div class="clearfix layout-block layout-a row--responsive"> <div class="column column-alpha column--responsive"> <h3 class="alternate"><a href="/2024/09/boosting-ml-pipeline-efficiency-direct-cassandra-ingestion-from-spark.html">Boosting ML Pipeline Efficiency: Direct Cassandra Ingestion from Spark</a></h3> <div class="post-info"> <div class="ypassport ypassport-slim media-block"> <div class="media-avatar"> <div class="photo-box pb-30s"> <img src="/images/authors/darwin.png" alt="Muhammad Junaid Muzammil, Software Engineer; Arnold Ziesche-Blank, Machine Learning Engineer" class="photo-box-img darwin-image"> </div> </div> <div class="media-story"> <ul class="user-passport-info"> <li class="user-name user-display-name"> Muhammad Junaid Muzammil, Software Engineer; Arnold Ziesche-Blank, Machine Learning Engineer <br> </li> <li class="post-date"> Sep 19, 2024 </li> </ul> </div> </div> </div> <div class="post-preview"> <p>Machine Learning Feature Stores ML Feature Store at Yelp Many of Yelp’s core capabilities such as business search, ads, and reviews are powered by Machine Learning (ML). In order to ensure these capabilities are well supported, we have built a dedicated ML platform. One of the pillars of this infrastructure is the Feature Store, which is a centralized data store for ML Features that are the input of ML models. Having a centralized dedicated datastore for ML Features serves a number of purposes: Data Quality and Data Governance Feature discovery Improved operational efficiency Availability of Features in every required environment...</p> <p><a href="/2024/09/boosting-ml-pipeline-efficiency-direct-cassandra-ingestion-from-spark.html">Continue reading</a></p> </div> </div> <div class="column column-beta column--responsive"> <img src="/images/previews/2024-08-27-boosting-ml-pipeline-efficiency-direct-cassandra-ingestion-from-spark.png"> </div> </div> </article> <article class="article-excert"> <div class="clearfix layout-block layout-a row--responsive"> <div class="column column-alpha column--responsive"> <h3 class="alternate"><a href="/2024/08/dbt-Generic-Tests-in-Sessions-Validation-at-Yelp.html">dbt Generic Tests in Sessions Validation at Yelp</a></h3> <div class="post-info"> <div class="ypassport ypassport-slim media-block"> <div class="media-avatar"> <div class="photo-box pb-30s"> <img src="/images/authors/tyk.png" alt="Tian, Yukang, Software Engineer" class="photo-box-img"> </div> </div> <div class="media-story"> <ul class="user-passport-info"> <li class="user-name user-display-name"> Tian, Yukang, Software Engineer <br> </li> <li class="post-date"> Aug 14, 2024 </li> </ul> </div> </div> </div> <div class="post-preview"> <p>Sessions, Where Everything Started For the past few years, Yelp has been using dbt as one of the tools to develop data products that power data marts, which are one stop shops for high visibility dashboards pertaining to top level business metrics. One of the key data products that’s owned by my team, Clickstream Analytics, is the Sessions Data Mart. This product is our in-house solution to understand what consumers do during their session interaction with Yelp products and provide insights on top of it. This blog post will walk you through how dbt is used as an important test...</p> <p><a href="/2024/08/dbt-Generic-Tests-in-Sessions-Validation-at-Yelp.html">Continue reading</a></p> </div> </div> <div class="column column-beta column--responsive"> <img src="/images/previews/2024-08-14-dbt-Generic-Tests-in-Sessions-Validation-at-Yelp-preview-updated.png"> </div> </div> </article> <article class="article-excert"> <div class="clearfix layout-block layout-a row--responsive"> <div class="column column-alpha column--responsive"> <h3 class="alternate"><a href="/2024/08/multi-metric-paasta.html">Implementing multi-metric scaling: making changes to legacy code safely</a></h3> <div class="post-info"> <div class="ypassport ypassport-slim media-block"> <div class="media-avatar"> <div class="photo-box pb-30s"> <img src="/images/authors/darwin.png" alt="David R. Morrison (Contractor); Charan Gangaraju (Software Engineer)" class="photo-box-img darwin-image"> </div> </div> <div class="media-story"> <ul class="user-passport-info"> <li class="user-name user-display-name"> David R. Morrison (Contractor); Charan Gangaraju (Software Engineer) <br> </li> <li class="post-date"> Aug 7, 2024 </li> </ul> </div> </div> </div> <div class="post-preview"> <p>We’re excited to announce that multi-metric horizontal autoscaling is available for all services at Yelp. This allows us to scale services using multiple metrics, such as the number of in-flight requests and CPU utilization, rather than relying on a single metric. We expect this to provide us with better resilience and faster recovery during outages. This year, PaaSTA (Yelp’s platform-as-a-service, which we use to manage all of the applications running on our infrastructure) turns eleven years old! The first commit was on August 20th, 2013, and the first public commit was on October 22nd, 2015. That’s over half of Yelp’s...</p> <p><a href="/2024/08/multi-metric-paasta.html">Continue reading</a></p> </div> </div> <div class="column column-beta column--responsive"> <img src="/images/previews/2024-08-07-multi-metric-paasta.png"> </div> </div> </article> <article class="article-excert"> <div class="clearfix layout-block layout-a row--responsive"> <div class="column column-alpha column--responsive"> <h3 class="alternate"><a href="/2024/05/fine-tuning-AWS-ASGs-with-attribute-based-instance-selection.html">Fine-tuning AWS ASGs with Attribute Based Instance Selection</a></h3> <div class="post-info"> <div class="ypassport ypassport-slim media-block"> <div class="media-avatar"> <div class="photo-box pb-30s"> <img src="/images/authors/ajayprataps.png" alt="Ajay Pratap Singh, Site Reliability Engineer" class="photo-box-img"> </div> </div> <div class="media-story"> <ul class="user-passport-info"> <li class="user-name user-display-name"> Ajay Pratap Singh, Site Reliability Engineer <br> </li> <li class="post-date"> May 1, 2024 </li> </ul> </div> </div> </div> <div class="post-preview"> <p>This is the next installment of our blog series on improving our autoscaling infrastructure. In the previous blog posts (Open-sourcing Clusterman, Recycling kubernetes nodes) we explained the architecture and inner-working of Clusterman. This time we are discussing how attribute based instance selection in the autoscaling group has helped us make our infrastructure more reliable and cost effective, while also decreasing the operation overhead. This will also cover how these changes enabled us to migrate from Clusterman to Karpenter. (Spoiler alert: Karpenter blog post is coming soon!) Motivation At Yelp we run most of our workload on AWS spot instances, and...</p> <p><a href="/2024/05/fine-tuning-AWS-ASGs-with-attribute-based-instance-selection.html">Continue reading</a></p> </div> </div> <div class="column column-beta column--responsive"> <img src="/images/previews/2024-05-01-fine-tuning-AWS-ASGs-with-attribute-based-instance-selection.png"> </div> </div> </article> <article class="article-excert"> <div class="clearfix layout-block layout-a row--responsive"> <div class="column column-alpha column--responsive"> <h3 class="alternate"><a href="/2024/03/moderating-inappropriate-video-content-at-yelp.html">Moderating Inappropriate Video Content at Yelp</a></h3> <div class="post-info"> <div class="ypassport ypassport-slim media-block"> <div class="media-avatar"> <div class="photo-box pb-30s"> <img src="/images/authors/prateekyadav.jpg" alt="Prateek Yadav, Machine Learning Engineer" class="photo-box-img"> </div> </div> <div class="media-story"> <ul class="user-passport-info"> <li class="user-name user-display-name"> Prateek Yadav, Machine Learning Engineer <br> </li> <li class="post-date"> Mar 27, 2024 </li> </ul> </div> </div> </div> <div class="post-preview"> <p>One of Yelp’s top priorities is the trust and safety of our users. Yelp’s platform is most well-known for its reviews, and its moderation practices have been recognised in academic research for mitigating misinformation and building consumer trust. In addition to reviews, Yelp’s Trust and Safety team takes significant measures when it comes to protecting its users from inappropriate material posted through other content types. This blog post discusses how Yelp protects its users from inappropriate content in videos. Videos at Yelp Recently, Yelp revamped its review experience by giving users the ability to upload videos alongside their review text....</p> <p><a href="/2024/03/moderating-inappropriate-video-content-at-yelp.html">Continue reading</a></p> </div> </div> <div class="column column-beta column--responsive"> <img src="/images/previews/2024-03-27-moderating-inappropriate-video-content-at-yelp-video-moderation-pipeline.png"> </div> </div> </article> </div> <div class="pagination-block"> <div class="arrange arrange--stack arrange--baseline arrange--6"> <div class="page-of-pages arrange_unit arrange_unit--fill"> Page 1 of 40 </div> <div class="pagination-links arrange_unit"> <div class="arrange arrange--baseline"> <div class="arrange_unit page-option current"> <span class="pagination-links_anchor"> 1 </span> </div> <div class="arrange_unit page-option"> <a class="available-number pagination-links_anchor" href="/page/2"> 2 </a> </div> <div class="arrange_unit page-option"> <a class="available-number pagination-links_anchor" href="/page/3"> 3 </a> </div> <div class="arrange_unit page-option"> <a class="available-number pagination-links_anchor" href="/page/4"> 4 </a> </div> <div class="arrange_unit page-option"> <a class="available-number pagination-links_anchor" href="/page/5"> 5 </a> </div> <div class="arrange_unit"> <a class="u-decoration-none next pagination-links_anchor" href="/page/2"> <span class="pagination-label responsive-hidden-small">Next</span> <span aria-hidden="true" style="width: 24px; height: 24px;" class="icon icon--24-chevron-right icon--size-24 icon--currentColor"> <svg role="img" class="icon_svg"> <use xlink:href="#24x24_chevron_right" /> </svg> </span> </a> </div> </div> </div> </div> </div> </div> </div> <div class="main-content-wrap main-content-wrap--separated"> <div class="content-container"> <div class="main-footer"> <div class="main-footer_section main-footer_menu clearfix"> <div class="main-footer_item"> <div class="footer-menu"> <h3 class="footer-menu_header">About</h3> <ul class="footer-menu_list"> <li class="footer-menu_item"> <a href="http://yelp.com/about">About Yelp</a> </li> <li class="footer-menu_item"> <a href="http://yelp.com/careers?country=US">Careers</a> </li> <li class="footer-menu_item"> <a href="http://www.yelp-press.com/">Press</a> </li> <li class="footer-menu_item"> <a href="http://www.yelp-ir.com/">Investor Relations</a> </li> <li class="footer-menu_item"> <a href="http://yelp.com/guidelines">Content Guidelines</a> </li> <li class="footer-menu_item"> <a href="http://yelp.com/static?p=tos">Terms of Service</a> </li> <li class="footer-menu_item"> <a href="http://yelp.com/tos/privacy_policy">Privacy Policy</a> </li> <li class="footer-menu_item"> <a href="http://yelp.com/static?p=thirdpartyterms#adchoices">Ad Privacy Info</a> </li> </ul> </div> </div> <div class="main-footer_item"> <div class="footer-menu"> <h3 class="footer-menu_header">Discover</h3> <ul class="footer-menu_list"> <li class="footer-menu_item"> <a href="http://yelp.com/local_yelp">The Local Yelp</a> </li> <li class="footer-menu_item"> <a href="http://officialblog.yelp.com/">Yelp Blog</a> </li> <li class="footer-menu_item"> <a href="http://yelp.com/contact">Contact Yelp</a> </li> <li class="footer-menu_item"> <a href="http://yelp.com/faq">FAQ</a> </li> <li class="footer-menu_item"> <a href="http://yelp.com/yelpmobile">Yelp Mobile</a> </li> <li class="footer-menu_item"> <a href="http://yelp.com/developers">Developers</a> </li> <li class="footer-menu_item"> <a href="http://yelp.com/rss">RSS</a> </li> </ul> </div> </div> <div class="main-footer_item"> <div class="footer-menu"> <h3 class="footer-menu_header">Yelp for Business Owners</h3> <ul class="footer-menu_list"> <li class="footer-menu_item"> <a href="https://biz.yelp.com/">Claim your Business Page</a> </li> <li class="footer-menu_item"> <a href="http://yelp.com/advertise">Advertise on Yelp</a> </li> <li class="footer-menu_item"> <a href="https://www.seatme.yelp.com/">Yelp SeatMe</a> </li> <li class="footer-menu_item"> <a href="https://biz.yelp.com/support/case_studies">Business Success Stories</a> </li> <li class="footer-menu_item"> <a href="http://biz.yelp.com/support">Business Support</a> </li> <li class="footer-menu_item"> <a href="https://biz.yelp.com/blog">Yelp Blog for Business Owners</a> </li> </ul> </div> </div> </div> <div class="main-footer_city-landscape-img responsive-hidden-small" role="presentation"></div> <div class="main-footer_copyright"> Copyright © 2004–2024 Yelp </div> </div> </div> </div> <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','//www.google-analytics.com/analytics.js','ga'); ga('create', 'UA-30501-54', 'auto'); ga('send', 'pageview'); </script> </body> </html>

CINXE.COM

Yelp Engineering and Product Blog