CINXE.COM
Automated Deployment with Apache Kafka | Engineering Health
<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <title> Automated Deployment with Apache Kafka | Engineering Health </title><meta name="description" content="a blog by engineers, for engineers"><meta name="author" content="Bryan Baugher"> <link rel="icon" href="/favicon.png"> <link rel="stylesheet" href="/dist/main.2b1315bd4c5d4f5eb804.min.css"> <link rel="canonical" href="https://engineering.cerner.com/blog/automated-deployment-with-apache-kafka/"><script src="/js/jquery-3.5.1.min.js"></script><meta property="og:title" content="Automated Deployment with Apache Kafka" /> <meta property="og:description" content="It’s likely not a surprise that Cerner would use Apache Kafka as we have used a number of related technologies like Apache Hadoop along with its Map/Reduce, HDFS and even Apache HBase. Our team first started using Apache Kafka in 2014 when Kafka 0.8 first came out. Since then we’ve expanded to using Kafka for a number of different use cases (1, 2) and it has become a core piece of Cerner’s infrastructure." /> <meta property="og:type" content="article" /> <meta property="og:url" content="https://engineering.cerner.com/blog/automated-deployment-with-apache-kafka/" /><meta property="og:image" content="https://engineering.cerner.com/images/alan-grace.png"/><meta property="article:section" content="post" /> <meta property="article:published_time" content="2017-02-16T00:00:00+00:00" /> <meta property="article:modified_time" content="2017-02-16T00:00:00+00:00" /><meta property="og:site_name" content="Cerner Engineering" /> <meta itemprop="name" content="Automated Deployment with Apache Kafka"> <meta itemprop="description" content="It’s likely not a surprise that Cerner would use Apache Kafka as we have used a number of related technologies like Apache Hadoop along with its Map/Reduce, HDFS and even Apache HBase. Our team first started using Apache Kafka in 2014 when Kafka 0.8 first came out. Since then we’ve expanded to using Kafka for a number of different use cases (1, 2) and it has become a core piece of Cerner’s infrastructure."><meta itemprop="datePublished" content="2017-02-16T00:00:00+00:00" /> <meta itemprop="dateModified" content="2017-02-16T00:00:00+00:00" /> <meta itemprop="wordCount" content="197"><meta itemprop="image" content="https://engineering.cerner.com/images/alan-grace.png"/> <meta itemprop="keywords" content="engineering,kafka," /><meta name="twitter:card" content="summary_large_image"/> <meta name="twitter:image" content="https://engineering.cerner.com/images/alan-grace.png"/> <meta name="twitter:title" content="Automated Deployment with Apache Kafka"/> <meta name="twitter:description" content="It’s likely not a surprise that Cerner would use Apache Kafka as we have used a number of related technologies like Apache Hadoop along with its Map/Reduce, HDFS and even Apache HBase. Our team first started using Apache Kafka in 2014 when Kafka 0.8 first came out. Since then we’ve expanded to using Kafka for a number of different use cases (1, 2) and it has become a core piece of Cerner’s infrastructure."/> </head> <body> <nav class="navbar navbar-expand-md navbar-light bg-light fixed-top shadow-sm" id="navbar-main-menu"> <div class="container"> <a class="navbar-brand" href="/"><img src="/logo.png" style='height: 40px;' /></a> <button class="navbar-toggler collapsed" type="button" data-toggle="collapse" data-target="#main-menu" aria-controls="main-menu" aria-expanded="false" aria-label="Toggle navigation"> <span class="navbar-toggler-icon"></span> </button> <div id="main-menu" class="collapse navbar-collapse" > <ul class="nav navbar-nav ml-auto"> <li class="nav-item"><a class="nav-link" href="/">Home</a></li> <li class="nav-item"><a class="nav-link" href="/culture/">Culture</a></li> <li class="nav-item"><a class="nav-link" href="/open-source/">Open Source</a></li> <li class="nav-item"><a class="nav-link" href="/tech-talks/">Tech Talks</a></li> <li class="nav-item"><a class="nav-link" href="/careers/">Careers</a></li> </ul> <div class="form-inline my-2 my-lg-0 searchbox"> <input id="search-by" type="search" class="form-control mr-sm-2" placeholder="Search" aria-label="Search"> </div> <script type="text/javascript" src="/js/lunr.min.js"></script> <script type="text/javascript" src="/js/auto-complete.min.js"></script> <script type="text/javascript"> var baseurl = "https:\/\/engineering.cerner.com"; </script> <script type="text/javascript" src="/js/search.js"></script> </div> </div> </nav> <main class="content-page container pt-7 pb-5"> <div class="row"> <div class="col"> <article class="article-content"> <div class="row justify-content-center"> <div class="col-lg-8"> <div class="meta text-muted mb-3"> <p class="created text-muted text-uppercase font-weight-bold mb-1">February 16, 2017</p> <span class="mr-2"><i class="fas fa-book-open mr-2"></i>197 words</span> <span><i class="fas fa-clock mr-2"></i>1 min read</span> </div> <h1>Automated Deployment with Apache Kafka</h1> <ul class="authors list-inline"> By: <li class="list-inline-item mr-3"> <div class="media author"> 馃懁 <div class="media-body"> <h5 class="name my-0 "><a href="/authors/bryan-baugher/" class="small">Bryan Baugher</a> </h5> </div> </div> </li></ul> </div> </div> <div class="row justify-content-center"> <div class="col-lg-8"> <div class="content"> <p>It’s likely not a surprise that Cerner would use <a href="http://kafka.apache.org/">Apache Kafka</a> as we have used a number of related technologies like <a href="http://engineering.cerner.com/2013/02/composable-mapreduce-with-hadoop-and-crunch/">Apache Hadoop</a> along with its Map/Reduce, HDFS and even <a href="http://engineering.cerner.com/2013/02/near-real-time-processing-over-hadoop-and-hbase/">Apache HBase</a>. Our team first started using Apache Kafka in 2014 when Kafka 0.8 first came out. Since then we’ve expanded to using Kafka for a number of different use cases (<a href="http://blog.cloudera.com/blog/2014/11/how-cerner-uses-cdh-with-apache-kafka/">1</a>, <a href="https://www.confluent.io/kafka-summit-2016-users-ingesting-complex-healthcare-data-with-apache-kafka">2</a>) and it has become a core piece of Cerner’s infrastructure.</p> <p>Just like the <a href="http://engineering.cerner.com/blog/deploying-web-services-with-apache-tomcat-and-chef/">applications we create</a>, we also needed to automate the deployment of Kafka to handle the ever growing amount of operations work and to ensure a high level of consistency with our environments. <a href="https://www.chef.io/chef/">Chef</a> was an obvious choice for us since its been our deployment tool of choice for the last few years. We quickly put together a <a href="https://github.com/cerner/cerner_kafka">cookbook</a> to help us automate the deployment of the many Kafka clusters here at Cerner. We have <a href="https://github.com/cerner/cerner_kafka/blob/master/CHANGELOG.md">continued to support and use this cookbook</a> after open-sourcing it upgrading it to handle newer versions of Kafka (0.9, 0.10), handle Kerberos authentication, and <a href="https://github.com/cerner/cerner_kafka/issues?q=is%3Aissue%20is%3Aclosed%20">many other improvements</a>.</p> <p>If you use Apache Kafka, feel free to <a href="https://github.com/cerner/cerner_kafka">try it out</a> and let us know how it works for you.</p> </div><div class="tags my-3"><a class="badge badge-pill badge-light border mr-2" href="/tags/engineering"> <i class="fas fa-tag mr-2"></i>engineering </a><a class="badge badge-pill badge-light border mr-2" href="/tags/kafka"> <i class="fas fa-tag mr-2"></i>kafka </a></div><ul class="share nav my-3 justify-content-end"> <li class="nav-item"> <a class="nav-link" target="_blank" href="https://twitter.com/intent/tweet?url=https%3a%2f%2fengineering.cerner.com%2fblog%2fautomated-deployment-with-apache-kafka%2f&text=Automated%20Deployment%20with%20Apache%20Kafka"> <i class="fa-fw fab fa-twitter"></i> </a> </li> <li class="nav-item"> <a class="nav-link" target="_blank" href="https://www.linkedin.com/shareArticle?url=https%3a%2f%2fengineering.cerner.com%2fblog%2fautomated-deployment-with-apache-kafka%2f&title=Automated%20Deployment%20with%20Apache%20Kafka"> <i class="fa-fw fab fa-linkedin-in"></i> </a> </li> <li class="nav-item"> <a class="nav-link" target="_blank" href="https://www.facebook.com/sharer.php?u=https%3a%2f%2fengineering.cerner.com%2fblog%2fautomated-deployment-with-apache-kafka%2f&t=Automated%20Deployment%20with%20Apache%20Kafka"> <i class="fa-fw fab fa-facebook-f"></i> </a> </li> <li class="nav-item"> <a class="nav-link" target="_blank" href="https://reddit.com/submit?url=https%3a%2f%2fengineering.cerner.com%2fblog%2fautomated-deployment-with-apache-kafka%2f&title=Automated%20Deployment%20with%20Apache%20Kafka"> <i class="fa-fw fab fa-reddit-alien"></i> </a> </li> </nav> </div> </div> <div class="row justify-content-center"> <div class="col-lg-8"> </div> </div></article> </div> </div> <div class="related-content row mt-5 row-cols-1 row-cols-lg-3"><div class="col mb-3"> <div class="card h-100"> <a href="/blog/beadledom-simple-java-framework-for-building-rest-apis/" class="d-block"><img data-src="/blog/beadledom-simple-java-framework-for-building-rest-apis/beadledom-01_hu6c6e9842af9f4e1245d581c9f3dd2816_13527764_700x350_fill_box_smart1_3.png" class="card-img-top mx-auto d-block" alt="Beadledom - Simple Java framework for building REST APIs"><div class="card-body"> <h4 class="card-title">Beadledom - Simple Java framework for building REST APIs</h4> <p class="card-text text-muted text-uppercase">February 15, 2017</p> <div class="card-text"> Beadledom is a framework for building Java based web services. It bundles several open sourced components required for building JAX-RS services. Why? HealtheIntent, Cerner鈥檚 population health management platform, started 3.5 years ago. We went through the process of investigating different technologies for the platform. We decided on using Java for building services and arrived on a set of libraries that we believed work well together. The long history of Java has led to an ocean of libraries that are available to Java developers. </div> </div> </a> </div> </div><div class="col mb-3"> <div class="card h-100"> <a href="/blog/shipit-vii-day-winter-2016/" class="d-block"><img data-src="/blog/shipit-vii-day-winter-2016/shipit-01_hu806bbef559da793620cc96755cf4bc3e_29390_700x350_fill_q75_box_smart1.jpg" class="card-img-top mx-auto d-block" alt="ShipIt VII Day: Winter 2016"><div class="card-body"> <h4 class="card-title">ShipIt VII Day: Winter 2016</h4> <p class="card-text text-muted text-uppercase">January 18, 2017</p> <div class="card-text"> Cerner鈥檚 7th ShipIt Day took place on December 8th and 9th. ShipIt is a 24-hour hackathon with the goal of building something awesome, usable, and value-adding within a single day. The event was hosted at our DevCenter in the Realization Campus, a large open space that hosts our DevAcademy (to learn more about our DevAcademy, check out this earlier post). We had 18 teams made up of associates from different Cerner organizations. </div> </div> </a> </div> </div><div class="col mb-3"> <div class="card h-100"> <a href="/blog/alan-and-grace-an-origin-story/" class="d-block"><img data-src="/blog/alan-and-grace-an-origin-story/alan-grace_hu1e4d4896897642c980dddf5de6f3f991_21157_700x350_fill_box_smart1_3.png" class="card-img-top mx-auto d-block" alt="Alan and Grace: An Origin Story"><div class="card-body"> <h4 class="card-title">Alan and Grace: An Origin Story</h4> <p class="card-text text-muted text-uppercase">November 23, 2016</p> <div class="card-text"> Meet Alan and Grace. These lighthearted, 8-bit characters were nominated in early 2014 to become the iconic mascots of Engineering at Cerner. Have you ever wondered who these 80鈥檚 video gaming inspired characters on our Engineering t-shirts are? Why were they chosen as mascots? What do they represent? The countless hours spent pondering these questions will be no more. Pull up a chair as we dive into the mystery behind the dynamic duo known as Alan and Grace. </div> </div> </a> </div> </div></div> </main> <footer class="footer text-center bg-dark py-6"> <div class="container"> <div class="row"> <div class="col"> <ul class="list-inline"> <li class="list-inline-item"><a href="https://engineering.cerner.com/index.xml" rel="alternate" type="application/rss+xml" class="icons d-block"> <span class="fa-stack fa-lg"> <i class="fa fa-circle fa-stack-2x"></i> <i class="fa fa-rss fa-stack-1x fa-inverse"></i> </span> </a></li><li class="list-inline-item"> <a href="https://github.com/cerner" class="icons d-block"> <span class="fa-stack fa-lg"> <i class="fa fa-circle fa-stack-2x"></i> <i class="fab fa-github fa-stack-1x fa-inverse"></i> </span> </a> </li><li class="list-inline-item"> <a href="https://www.instagram.com/cernercorporation/" class="icons d-block"> <span class="fa-stack fa-lg"> <i class="fa fa-circle fa-stack-2x"></i> <i class="fab fa-instagram fa-stack-1x fa-inverse"></i> </span> </a> </li><li class="list-inline-item"> <a href="https://www.linkedin.com/company/cerner-corporation" class="icons d-block"> <span class="fa-stack fa-lg"> <i class="fa fa-circle fa-stack-2x"></i> <i class="fab fa-linkedin fa-stack-1x fa-inverse"></i> </span> </a> </li><li class="list-inline-item"> <a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fengineering.cerner.com%2F&screen_name=CernerEng" class="icons d-block"> <span class="fa-stack fa-lg"> <i class="fa fa-circle fa-stack-2x"></i> <i class="fab fa-twitter fa-stack-1x fa-inverse"></i> </span> </a> </li><li class="list-inline-item"> <a href="https://www.youtube.com/user/cernereng?sub_confirmation=1" class="icons d-block"> <span class="fa-stack fa-lg"> <i class="fa fa-circle fa-stack-2x"></i> <i class="fab fa-youtube fa-stack-1x fa-inverse"></i> </span> </a> </li> </ul> <p class="text-light"> Copyright © 2023 </p> <p class="text-light"> Made with 鉂わ笍 by Cerner engineers. </p> </div> </div> </div> </footer> <script src="/dist/main.0f92af9103b4f0550a4b.min.js"></script> </body> </html>