Cerner Open Sources its Kafka Utilities

<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <title> Cerner Open Sources its Kafka Utilities | Engineering Health </title><meta name="description" content="a blog by engineers, for engineers"><meta name="author" content="Bryan Baugher"> <link rel="icon" href="/favicon.png"> <link rel="stylesheet" href="/dist/main.2b1315bd4c5d4f5eb804.min.css"> <link rel="canonical" href="https://engineering.cerner.com/blog/cerner-open-sources-its-kafka-utilities/"><script src="/js/jquery-3.5.1.min.js"></script><meta property="og:title" content="Cerner Open Sources its Kafka Utilities" /> <meta property="og:description" content="At Cerner, we often make use of many open source projects in our infrastructure. I work on a team responsible for Cerner’s Ingestion Platform, a critical piece of infrastructure that takes in TBs of data and over a billion messages per day. The platform’s responsibility is then to make this data available for downstream teams to consume. When designing the Ingestion Platform, we felt Apache Kafka was perfect for ingesting and consuming these massive streams of data." /> <meta property="og:type" content="article" /> <meta property="og:url" content="https://engineering.cerner.com/blog/cerner-open-sources-its-kafka-utilities/" /><meta property="og:image" content="https://engineering.cerner.com/images/alan-grace.png"/><meta property="article:section" content="post" /> <meta property="article:published_time" content="2018-02-06T00:00:00+00:00" /> <meta property="article:modified_time" content="2018-02-06T00:00:00+00:00" /><meta property="og:site_name" content="Cerner Engineering" /> <meta itemprop="name" content="Cerner Open Sources its Kafka Utilities"> <meta itemprop="description" content="At Cerner, we often make use of many open source projects in our infrastructure. I work on a team responsible for Cerner’s Ingestion Platform, a critical piece of infrastructure that takes in TBs of data and over a billion messages per day. The platform’s responsibility is then to make this data available for downstream teams to consume. When designing the Ingestion Platform, we felt Apache Kafka was perfect for ingesting and consuming these massive streams of data."><meta itemprop="datePublished" content="2018-02-06T00:00:00+00:00" /> <meta itemprop="dateModified" content="2018-02-06T00:00:00+00:00" /> <meta itemprop="wordCount" content="258"><meta itemprop="image" content="https://engineering.cerner.com/images/alan-grace.png"/> <meta itemprop="keywords" content="engineering,kafka,ingestion," /><meta name="twitter:card" content="summary_large_image"/> <meta name="twitter:image" content="https://engineering.cerner.com/images/alan-grace.png"/> <meta name="twitter:title" content="Cerner Open Sources its Kafka Utilities"/> <meta name="twitter:description" content="At Cerner, we often make use of many open source projects in our infrastructure. I work on a team responsible for Cerner’s Ingestion Platform, a critical piece of infrastructure that takes in TBs of data and over a billion messages per day. The platform’s responsibility is then to make this data available for downstream teams to consume. When designing the Ingestion Platform, we felt Apache Kafka was perfect for ingesting and consuming these massive streams of data."/> </head> <body> <nav class="navbar navbar-expand-md navbar-light bg-light fixed-top shadow-sm" id="navbar-main-menu"> <div class="container"> <a class="navbar-brand" href="/"><img src="/logo.png" style='height: 40px;' /></a> <button class="navbar-toggler collapsed" type="button" data-toggle="collapse" data-target="#main-menu" aria-controls="main-menu" aria-expanded="false" aria-label="Toggle navigation"> <span class="navbar-toggler-icon"></span> </button> <div id="main-menu" class="collapse navbar-collapse" > <ul class="nav navbar-nav ml-auto"> <li class="nav-item"><a class="nav-link" href="/">Home</a></li> <li class="nav-item"><a class="nav-link" href="/culture/">Culture</a></li> <li class="nav-item"><a class="nav-link" href="/open-source/">Open Source</a></li> <li class="nav-item"><a class="nav-link" href="/tech-talks/">Tech Talks</a></li> <li class="nav-item"><a class="nav-link" href="/careers/">Careers</a></li> </ul> <div class="form-inline my-2 my-lg-0 searchbox"> <input id="search-by" type="search" class="form-control mr-sm-2" placeholder="Search" aria-label="Search"> </div> <script type="text/javascript" src="/js/lunr.min.js"></script> <script type="text/javascript" src="/js/auto-complete.min.js"></script> <script type="text/javascript"> var baseurl = "https:\/\/engineering.cerner.com"; </script> <script type="text/javascript" src="/js/search.js"></script> </div> </div> </nav> <main class="content-page container pt-7 pb-5"> <div class="row"> <div class="col"> <article class="article-content"> <div class="row justify-content-center"> <div class="col-lg-8"> <div class="meta text-muted mb-3"> <p class="created text-muted text-uppercase font-weight-bold mb-1">February 6, 2018</p> <span class="mr-2"><i class="fas fa-book-open mr-2"></i>258 words</span> <span><i class="fas fa-clock mr-2"></i>2 mins read</span> </div> <h1>Cerner Open Sources its Kafka Utilities</h1> <ul class="authors list-inline"> By:  <li class="list-inline-item mr-3"> <div class="media author"> 👤 <div class="media-body"> <h5 class="name my-0 "><a href="/authors/bryan-baugher/" class="small">Bryan Baugher</a> </h5> </div> </div> </li></ul> </div> </div> <div class="row justify-content-center"> <div class="col-lg-8"> <div class="content"> <p>At Cerner, we often make use of many open source projects in our infrastructure. I work on a team responsible for Cerner’s Ingestion Platform, a critical piece of infrastructure that takes in TBs of data and over a billion messages per day. The platform’s responsibility is then to make this data available for downstream teams to consume. When designing the Ingestion Platform, we felt Apache Kafka was perfect for ingesting and consuming these massive streams of data. We originally built the Ingestion Platform in 2014 and it has grown with Kafka ever since.</p> <p>Along the way, we’ve become experienced with Apache Kafka and how it works. We know how to monitor it, perform common administrative operations, and more. We’ve even made contributions to Kafka for various improvements [<a href="https://issues.apache.org/jira/browse/KAFKA-3297">1</a>][<a href="https://issues.apache.org/jira/browse/KAFKA-4687">2</a>][<a href="https://issues.apache.org/jira/browse/KAFKA-2434">3</a>] and Apache Crunch, our processing framework, to <a href="https://issues.apache.org/jira/browse/CRUNCH-606">add Kafka support</a>. We have also given <a href="https://www.confluent.io/resources/kafka-summit-2016/ingesting-complex-healthcare-data-apache-kafka/">talks</a> about our usage of Apache Kafka at Kafka Summit and we’ve written a <a href="http://blog.cloudera.com/blog/2014/11/how-cerner-uses-cdh-with-apache-kafka/">blog post</a> about it as well.</p> <p>We are still interested in supporting the Kafka community, and today we hit another milestone as we open source some of our <a href="https://github.com/cerner/common-kafka">utilities for Kafka</a>. This includes:</p> <ul> <li>Lightweight wrappers over producer and consumer providing additional functionality</li> <li>A client to simplify admin operations</li> <li>A Kafka connect sink for delivering data to another Kafka cluster</li> <li>Kafka test infrastructure for integration testing</li> </ul> <p>Try it and let me know what you think! You can log issues for improvements, or email me your thoughts: <a href="mailto:bryan.baugher@cerner.com">bryan.baugher@cerner.com</a>. We hope to continue to improve the library and always welcome contributions and ideas from others.</p> </div><div class="tags my-3"><a class="badge badge-pill badge-light border mr-2" href="/tags/engineering"> <i class="fas fa-tag mr-2"></i>engineering </a><a class="badge badge-pill badge-light border mr-2" href="/tags/kafka"> <i class="fas fa-tag mr-2"></i>kafka </a><a class="badge badge-pill badge-light border mr-2" href="/tags/ingestion"> <i class="fas fa-tag mr-2"></i>ingestion </a></div><ul class="share nav my-3 justify-content-end"> <li class="nav-item"> <a class="nav-link" target="_blank" href="https://twitter.com/intent/tweet?url=https%3a%2f%2fengineering.cerner.com%2fblog%2fcerner-open-sources-its-kafka-utilities%2f&text=Cerner%20Open%20Sources%20its%20Kafka%20Utilities"> <i class="fa-fw fab fa-twitter"></i> </a> </li> <li class="nav-item"> <a class="nav-link" target="_blank" href="https://www.linkedin.com/shareArticle?url=https%3a%2f%2fengineering.cerner.com%2fblog%2fcerner-open-sources-its-kafka-utilities%2f&title=Cerner%20Open%20Sources%20its%20Kafka%20Utilities"> <i class="fa-fw fab fa-linkedin-in"></i> </a> </li> <li class="nav-item"> <a class="nav-link" target="_blank" href="https://www.facebook.com/sharer.php?u=https%3a%2f%2fengineering.cerner.com%2fblog%2fcerner-open-sources-its-kafka-utilities%2f&t=Cerner%20Open%20Sources%20its%20Kafka%20Utilities"> <i class="fa-fw fab fa-facebook-f"></i> </a> </li> <li class="nav-item"> <a class="nav-link" target="_blank" href="https://reddit.com/submit?url=https%3a%2f%2fengineering.cerner.com%2fblog%2fcerner-open-sources-its-kafka-utilities%2f&title=Cerner%20Open%20Sources%20its%20Kafka%20Utilities"> <i class="fa-fw fab fa-reddit-alien"></i> </a> </li> </nav> </div> </div> <div class="row justify-content-center"> <div class="col-lg-8"> </div> </div></article> </div> </div> <div class="related-content row mt-5 row-cols-1 row-cols-lg-3"><div class="col mb-3"> <div class="card h-100"> <a href="/blog/automated-deployment-with-apache-kafka/" class="d-block"><div class="card-body"> <h4 class="card-title">Automated Deployment with Apache Kafka</h4> <p class="card-text text-muted text-uppercase">February 16, 2017</p> <div class="card-text"> It’s likely not a surprise that Cerner would use Apache Kafka as we have used a number of related technologies like Apache Hadoop along with its Map/Reduce, HDFS and even Apache HBase. Our team first started using Apache Kafka in 2014 when Kafka 0.8 first came out. Since then we’ve expanded to using Kafka for a number of different use cases (1, 2) and it has become a core piece of Cerner’s infrastructure. </div> </div> </a> </div> </div><div class="col mb-3"> <div class="card h-100"> <a href="/blog/cerner-and-the-sdlc/" class="d-block"><div class="card-body"> <h4 class="card-title">Cerner and the SDLC</h4> <p class="card-text text-muted text-uppercase">January 23, 2018</p> <div class="card-text"> Cerner has a very compelling story that many want to learn from. Cerner’s Agile Champions are regular presenters at local, national, and global conferences. We routinely host calls and site visits for interested companies around the world who want to figure out how we were able to adopt Agile so quickly and sustain it so successfully. Despite our success, there has been a general feeling at the engineering leadership and Agile Champion tiers that we aren’t getting the benefits from Agile that we once were. </div> </div> </a> </div> </div><div class="col mb-3"> <div class="card h-100"> <a href="/blog/2-to-the-5th-coding-competition-2017/" class="d-block"><div class="card-body"> <h4 class="card-title">2^5 Coding Competition 2017: 32 lines or less</h4> <p class="card-text text-muted text-uppercase">December 19, 2017</p> <div class="card-text"> The Competition Part of Cerner’s engineering culture is to celebrate technology. This past fall, we had our annual celebration for Programmers’ Day. We celebrated the day with great food (taco bar and ice cream) and an evening of trivia. Topping off Programmers' Day with a taco bar and trivia night with @geekswhodrink pic.twitter.com/pSjqBVBHxK — Cerner Engineering (@CernerEng) September 13, 2017 Happy Programmers' Day! Celebrating with an ice cream social. pic. </div> </div> </a> </div> </div></div> </main> <footer class="footer text-center bg-dark py-6"> <div class="container"> <div class="row"> <div class="col"> <ul class="list-inline"> <li class="list-inline-item"><a href="https://engineering.cerner.com/index.xml" rel="alternate" type="application/rss+xml" class="icons d-block"> <span class="fa-stack fa-lg"> <i class="fa fa-circle fa-stack-2x"></i> <i class="fa fa-rss fa-stack-1x fa-inverse"></i> </span> </a></li><li class="list-inline-item"> <a href="https://github.com/cerner" class="icons d-block"> <span class="fa-stack fa-lg"> <i class="fa fa-circle fa-stack-2x"></i> <i class="fab fa-github fa-stack-1x fa-inverse"></i> </span> </a> </li><li class="list-inline-item"> <a href="https://www.instagram.com/cernercorporation/" class="icons d-block"> <span class="fa-stack fa-lg"> <i class="fa fa-circle fa-stack-2x"></i> <i class="fab fa-instagram fa-stack-1x fa-inverse"></i> </span> </a> </li><li class="list-inline-item"> <a href="https://www.linkedin.com/company/cerner-corporation" class="icons d-block"> <span class="fa-stack fa-lg"> <i class="fa fa-circle fa-stack-2x"></i> <i class="fab fa-linkedin fa-stack-1x fa-inverse"></i> </span> </a> </li><li class="list-inline-item"> <a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fengineering.cerner.com%2F&screen_name=CernerEng" class="icons d-block"> <span class="fa-stack fa-lg"> <i class="fa fa-circle fa-stack-2x"></i> <i class="fab fa-twitter fa-stack-1x fa-inverse"></i> </span> </a> </li><li class="list-inline-item"> <a href="https://www.youtube.com/user/cernereng?sub_confirmation=1" class="icons d-block"> <span class="fa-stack fa-lg"> <i class="fa fa-circle fa-stack-2x"></i> <i class="fab fa-youtube fa-stack-1x fa-inverse"></i> </span> </a> </li> </ul> <p class="text-light"> Copyright © 2023 </p> <p class="text-light"> Made with ❤️ by Cerner engineers. </p> </div> </div> </div> </footer> <script src="/dist/main.0f92af9103b4f0550a4b.min.js"></script> </body> </html>

CINXE.COM

Cerner Open Sources its Kafka Utilities | Engineering Health