CINXE.COM

<!DOCTYPE html> <html id="caseStudies" lang="en" class=""> <head> <meta name="robots" content="noindex, nofollow"> <link rel="alternate" hreflang="zh-cn" href="https://kubernetes.io/zh-cn/case-studies/cern/"> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <meta name="generator" content="Hugo 0.111.3"> <link rel="shortcut icon" type="image/png" href="/images/favicon.png"> <link rel="apple-touch-icon" href="/favicons/apple-touch-icon-180x180.png" sizes="180x180"> <link rel="manifest" href="/manifest.webmanifest"> <link rel="apple-touch-icon" href="/images/kubernetes-192x192.png"> <title>CERN Case Study | Kubernetes</title><meta property="og:title" content="CERN Case Study" /> <meta property="og:description" content="Challenge At CERN, the European Organization for Nuclear Research, physicists conduct experiments to learn about fundamental science. In its particle accelerators, "we accelerate protons to very high energy, close to the speed of light, and we make the two beams of protons collide," says CERN Software Engineer Ricardo Rocha. "The end result is a lot of data that we have to process." CERN currently stores 330 petabytes of data in its data centers, and an upgrade of its accelerators expected in the next few years will drive that number up by 10x." /> <meta property="og:type" content="article" /> <meta property="og:url" content="https://kubernetes.io/case-studies/cern/" /><meta property="og:image" content="https://kubernetes.io/case-studies/cern/cern_featured_logo.png"/><meta property="article:section" content="case-studies" /> <meta property="article:modified_time" content="2020-09-06T18:13:40+07:00" /><meta property="og:site_name" content="Kubernetes" /> <meta itemprop="name" content="CERN Case Study"> <meta itemprop="description" content="Challenge At CERN, the European Organization for Nuclear Research, physicists conduct experiments to learn about fundamental science. In its particle accelerators, "we accelerate protons to very high energy, close to the speed of light, and we make the two beams of protons collide," says CERN Software Engineer Ricardo Rocha. "The end result is a lot of data that we have to process." CERN currently stores 330 petabytes of data in its data centers, and an upgrade of its accelerators expected in the next few years will drive that number up by 10x."> <meta itemprop="dateModified" content="2020-09-06T18:13:40+07:00" /> <meta itemprop="wordCount" content="1776"><meta itemprop="image" content="https://kubernetes.io/case-studies/cern/cern_featured_logo.png"> <meta itemprop="keywords" content="" /><meta name="twitter:card" content="summary_large_image"/> <meta name="twitter:image" content="https://kubernetes.io/case-studies/cern/cern_featured_logo.png"/> <meta name="twitter:title" content="CERN Case Study"/> <meta name="twitter:description" content="Challenge At CERN, the European Organization for Nuclear Research, physicists conduct experiments to learn about fundamental science. In its particle accelerators, "we accelerate protons to very high energy, close to the speed of light, and we make the two beams of protons collide," says CERN Software Engineer Ricardo Rocha. "The end result is a lot of data that we have to process." CERN currently stores 330 petabytes of data in its data centers, and an upgrade of its accelerators expected in the next few years will drive that number up by 10x."/> <link href="/scss/main.css" rel="stylesheet"> <script type="application/ld+json"> { "@context": "https://schema.org", "@type": "Organization", "url": "https://kubernetes.io", "logo": "https://kubernetes.io/images/favicon.png", } </script> <meta name="theme-color" content="#326ce5"> <link rel="stylesheet" href="/scss/_case-studies.462831e860769d5e833a56d0e121e5a5ec385ad2be6f06f4b802751277d7c7ba.css" integrity="sha256-Rigx6GB2nV6DOlbQ4SHlpew4WtK+bwb0uAJ1EnfXx7o="> <link rel="stylesheet" href="/css/feature-states.css"> <link rel="stylesheet" href="/css/new-case-studies.css"> <meta name="description" content="Challenge At CERN, the European Organization for Nuclear Research, physicists conduct experiments to learn about fundamental science. In its particle accelerators, "we accelerate protons to very high energy, close to the speed of light, and we make the two beams of protons collide," says CERN Software Engineer Ricardo Rocha. "The end result is a lot of data that we have to process." CERN currently stores 330 petabytes of data in its data centers, and an upgrade of its accelerators expected in the next few years will drive that number up by 10x."> <meta property="og:description" content="Challenge At CERN, the European Organization for Nuclear Research, physicists conduct experiments to learn about fundamental science. In its particle accelerators, "we accelerate protons to very high energy, close to the speed of light, and we make the two beams of protons collide," says CERN Software Engineer Ricardo Rocha. "The end result is a lot of data that we have to process." CERN currently stores 330 petabytes of data in its data centers, and an upgrade of its accelerators expected in the next few years will drive that number up by 10x."> <meta name="twitter:description" content="Challenge At CERN, the European Organization for Nuclear Research, physicists conduct experiments to learn about fundamental science. In its particle accelerators, "we accelerate protons to very high energy, close to the speed of light, and we make the two beams of protons collide," says CERN Software Engineer Ricardo Rocha. "The end result is a lot of data that we have to process." CERN currently stores 330 petabytes of data in its data centers, and an upgrade of its accelerators expected in the next few years will drive that number up by 10x."> <meta property="og:url" content="https://kubernetes.io/case-studies/cern/"> <meta property="og:title" content="CERN Case Study"> <meta name="twitter:title" content="CERN Case Study"> <meta name="twitter:image" content="https://kubernetes.io/images/favicon.png" /> <meta name="twitter:image:alt" content="Kubernetes"> <meta property="og:image" content="/images/kubernetes-horizontal-color.png"> <meta property="og:type" content="article"> <script src="/js/jquery-3.6.0.min.js" intregrity="sha384-vtXRMe3mGCbOeY7l30aIg8H9p3GdeSe4IFlP6G8JMa7o7lXvnz3GFKzPxzJdPfGK" crossorigin="anonymous"></script> </head> <body> <nav class="js-navbar-scroll navbar navbar-expand navbar-dark flex-column flex-md-row td-navbar" data-auto-burger="primary"> <a class="navbar-brand img-fluid" href="/"></a> <div class="td-navbar-nav-scroll ml-md-auto" id="main_navbar"> <ul class="navbar-nav mt-2 mt-lg-0"> <li class="nav-item mr-2 mb-lg-0"> <a class="nav-link" href="/docs/" >Documentation</a> </li> <li class="nav-item mr-2 mb-lg-0"> <a class="nav-link" href="/blog/" >Kubernetes Blog</a> </li> <li class="nav-item mr-2 mb-lg-0"> <a class="nav-link" href="/training/" >Training</a> </li> <li class="nav-item mr-2 mb-lg-0"> <a class="nav-link" href="/partners/" >Partners</a> </li> <li class="nav-item mr-2 mb-lg-0"> <a class="nav-link" href="/community/" >Community</a> </li> <li class="nav-item mr-2 mb-lg-0"> <a class="nav-link active" href="/case-studies/" >Case Studies</a> </li> <li class="nav-item mr-n3 mr-lg-0 dropdown"> <a class="nav-link dropdown-toggle" href="#" id="navbarDropdown" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false"> Versions </a> <div class="dropdown-menu dropdown-menu-right" aria-labelledby="navbarDropdownMenuLink"> <a class="dropdown-item" href="/releases">Release Information</a> <a class="dropdown-item" href="https://kubernetes.io/case-studies/cern/">v1.31</a> <a class="dropdown-item" href="https://v1-30.docs.kubernetes.io/case-studies/cern/">v1.30</a> <a class="dropdown-item" href="https://v1-29.docs.kubernetes.io/case-studies/cern/">v1.29</a> <a class="dropdown-item" href="https://v1-28.docs.kubernetes.io/case-studies/cern/">v1.28</a> <a class="dropdown-item" href="https://v1-27.docs.kubernetes.io/case-studies/cern/">v1.27</a> </div> </li> <li class="nav-item mr-n4 mr-lg-0 dropdown"> <a class="nav-link dropdown-toggle" href="#" id="navbarDropdownMenuLink" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false"> English </a> <div class="dropdown-menu dropdown-menu-right" aria-labelledby="navbarDropdownMenuLink"> <a class="dropdown-item" href="/zh-cn/case-studies/cern/">中文 (Chinese)</a> </div> </li> </ul> </div> <button id="hamburger" onclick="kub.toggleMenu()" data-auto-burger-exclude><div></div></button> </nav> <section id="deprecation-warning"> <div class="content deprecation-warning pageinfo"> <h3> You are viewing documentation for Kubernetes version: v1.27 </h3> <p> Kubernetes v1.27 documentation is no longer actively maintained. The version you are currently viewing is a static snapshot. For up-to-date information, see the <a href="https://kubernetes.io/docs/home/">latest version.</a> </p> </div> </section> <div class="banner " style="background-image: url(/images/case-studies/cern/banner1.jpg)"> <h1> <span class="heading">CASE STUDY:</span>CERN<span class="subheading">CERN: Processing Petabytes of Data More Efficiently with Kubernetes </span> </h1> </div> <div class="details"> <span class="item">Company <strong>CERN</strong></span> <span class="item">Location <strong>Geneva, Switzerland</strong></span> <span class="item">Industry <strong>Particle physics research</strong></span> </div> <div class="content"> <h2>Challenge</h2> <p>At CERN, the European Organization for Nuclear Research, physicists conduct experiments to learn about fundamental science. In its particle accelerators, "we accelerate protons to very high energy, close to the speed of light, and we make the two beams of protons collide," says CERN Software Engineer Ricardo Rocha. "The end result is a lot of data that we have to process." CERN currently stores 330 petabytes of data in its data centers, and an upgrade of its accelerators expected in the next few years will drive that number up by 10x. Additionally, the organization experiences extreme peaks in its workloads during periods prior to big conferences, and needs its infrastructure to scale to those peaks. "We want to have a more hybrid infrastructure, where we have our on premise infrastructure but can make use of public clouds temporarily when these peaks come up," says Rocha. "We've been looking to new technologies that can help improve our efficiency in our infrastructure so that we can dedicate more of our resources to the actual processing of the data."</p> <h2>Solution</h2> <p>CERN's technology team embraced containerization and cloud native practices, choosing Kubernetes for orchestration, Helm for deployment, Prometheus for monitoring, and CoreDNS for DNS resolution inside the clusters. Kubernetes federation has allowed the organization to run some production workloads both on premise and in public clouds.</p> <h2>Impact</h2> <p>"Kubernetes gives us the full automation of the application," says Rocha. "It comes with built-in monitoring and logging for all the applications and the workloads that deploy in Kubernetes. This is a massive simplification of our current deployments." The time to deploy a new cluster for a complex distributed storage system has gone from more than 3 hours to less than 15 minutes. Adding new nodes to a cluster used to take more than an hour; now it takes less than 2 minutes. The time it takes to autoscale replicas for system components has decreased from more than an hour to less than 2 minutes. Initially, virtualization gave 20% overhead, but with tuning this was reduced to ~5%. Moving to Kubernetes on bare metal would get this to 0%. Not having to host virtual machines is expected to also get 10% of memory capacity back.</p> <div class="quote banner"> <div class="quote-text"> "Kubernetes is something we can relate to very much because it's naturally distributed. What it gives us is a uniform API across heterogeneous resources to define our workloads. This is something we struggled with a lot in the past when we want to expand our resources outside our infrastructure." <span class="quote-author">— Ricardo Rocha, Software Engineer, CERN</span> </div> </div> <div class="lead"> With a mission of researching fundamental science, and a stable of extremely large machines, the European Organization for Nuclear Research (CERN) operates at what can only be described as hyperscale. </div> <p>Experiments are conducted in particle accelerators, the biggest of which is 27 kilometers in circumference. "We accelerate protons to very high energy, to close to the speed of light, and we make the two beams of protons collide in well-defined places," says CERN Software Engineer Ricardo Rocha. "We build experiments around these places where we do the collisions. The end result is a lot of data that we have to process."</p> <p>And he does mean a lot: CERN currently stores and processes 330 petabytes of data—gathered from 4,300 projects and 3,300 users—using 10,000 hypervisors and 320,000 cores in its data centers.</p> <p>Over the years, the CERN technology department has built a large computing infrastructure, based on OpenStack private clouds, to help the organization's physicists analyze and treat all this data. The organization experiences extreme peaks in its workloads. "Very often, just before conferences, physicists want to do an enormous amount of extra analysis to publish their papers, and we have to scale to these peaks, which means overcommitting resources in some cases," says Rocha. "We want to have a more hybrid infrastructure, where we have our on premise infrastructure but can make use of public clouds temporarily when these peaks come up."</p> <p>Additionally, few years ago, CERN announced that it would be doing a big upgrade of its accelerators, which will mean a ten-fold increase in the amount of data that can be collected. "So we've been looking to new technologies that can help improve our efficiency in our infrastructure, so that we can dedicate more of our resources to the actual processing of the data," says Rocha.</p> <div class="quote banner " style="background-image: url(/images/case-studies/cern/banner3.jpg)"> <div class="quote-text"> "Before, the tendency was always: 'I need this, I get a couple of developers, and I implement it.' Right now it's 'I need this, I'm sure other people also need this, so I'll go and ask around.' The CNCF is a good source because there's a very large catalog of applications available. It's very hard right now to justify developing a new product in-house. There is really no real reason to keep doing that. It's much easier for us to try it out, and if we see it's a good solution, we try to reach out to the community and start working with that community." <span class="quote-author">— Ricardo Rocha, Software Engineer, CERN</span> </div> </div> <p>Rocha's team started looking at Kubernetes and containerization in the second half of 2015. "We've been using distributed infrastructures for decades now," says Rocha. "Kubernetes is something we can relate to very much because it's naturally distributed. What it gives us is a uniform API across heterogeneous resources to define our workloads. This is something we struggled with a lot in the past when we want to expand our resources outside our infrastructure."</p> <p>The team created a prototype system for users to deploy their own Kubernetes cluster in CERN's infrastructure, and spent six months validating the use cases and making sure that Kubernetes integrated with CERN's internal systems. The main use case is batch workloads, which represent more than 80% of resource usage at CERN. (One single project that does most of the physics data processing and analysis alone consumes 250,000 cores.) "This is something where the investment in simplification of the deployment, logging, and monitoring pays off very quickly," says Rocha. Other use cases include Spark-based data analysis and machine learning to improve physics analysis. "The fact that most of these technologies integrate very well with Kubernetes makes our lives easier," he adds.</p> <p>The system went into production in October 2016, also using Helm for deployment, Prometheus for monitoring, and CoreDNS for DNS resolution within the cluster. "One thing that Kubernetes gives us is the full automation of the application," says Rocha. "So it comes with built-in monitoring and logging for all the applications and the workloads that deploy in Kubernetes. This is a massive simplification of our current deployments." The time to deploy a new cluster for a complex distributed storage system has gone from more than 3 hours to less than 15 minutes.</p> <p>Adding new nodes to a cluster used to take more than an hour; now it takes less than 2 minutes. The time it takes to autoscale replicas for system components has decreased from more than an hour to less than 2 minutes.</p> <div class="quote banner " style="background-image: url(/images/case-studies/cern/banner4.jpg)"> <div class="quote-text"> "With Kubernetes, there's a well-established technology and a big community that we can contribute to. It allows us to do our physics analysis without having to focus so much on the lower level software. This is just exciting. We are looking forward to keep contributing to the community and collaborating with everyone." <span class="quote-author">— Ricardo Rocha, Software Engineer, CERN</span> </div> </div> <p>Rocha points out that the metric used in the particle accelerators may be events per second, but in reality "it's how fast and how much of the data we can process that actually counts." And efficiency has certainly been improved with Kubernetes. Initially, virtualization gave 20% overhead, but with tuning this was reduced to ~5%. Moving to Kubernetes on bare metal would get this to 0%. Not having to host virtual machines is expected to also get 10% of memory capacity back.</p> <p>Kubernetes federation, which CERN has been using for a portion of its production workloads since February 2018, has allowed the organization to adopt a hybrid cloud strategy. And it was remarkably simple to do. "We had a summer intern working on federation," says Rocha. "For many years, I've been developing distributed computing software, which took like a decade and a lot of effort from a lot of people to stabilize and make sure it works. And for our intern, in a couple of days he was able to demo to me and my team that we had a cluster at CERN and a few clusters outside in public clouds that were federated together and that we could submit workloads to. This was shocking for us. It really shows the power of using this kind of well-established technologies."</p> <p>With such results, adoption of Kubernetes has made rapid gains at CERN, and the team is eager to give back to the community. "If we look back into the '90s and early 2000s, there were not a lot of companies focusing on systems that have to scale to this kind of size, storing petabytes of data, analyzing petabytes of data," says Rocha. "The fact that Kubernetes is supported by such a wide community and different backgrounds, it motivates us to contribute back."</p> <div class="quote banner"> <div class="quote-text"> This means that the physicist can build his or her analysis and publish it in a repository, share it with colleagues, and in 10 years redo the same analysis with new data. If we looked back even 10 years, this was just a dream." <span class="quote-author">— Ricardo Rocha, Software Engineer, CERN</span> </div> </div> <p>These new technologies aren't just enabling infrastructure improvements. CERN also uses the Kubernetes-based <a href="https://github.com/recast-hep">Reana/Recast</a> platform for reusable analysis, which is "the ability to define physics analysis as a set of workflows that are fully containerized in one single entry point," says Rocha. "This means that the physicist can build his or her analysis and publish it in a repository, share it with colleagues, and in 10 years redo the same analysis with new data. If we looked back even 10 years, this was just a dream."</p> <p>All of these things have changed the culture at CERN considerably. A decade ago, "The tendency was always: 'I need this, I get a couple of developers, and I implement it,'" says Rocha. "Right now it's 'I need this, I'm sure other people also need this, so I'll go and ask around.' The CNCF is a good source because there's a very large catalog of applications available. It's very hard right now to justify developing a new product in-house. There is really no real reason to keep doing that. It's much easier for us to try it out, and if we see it's a good solution, we try to reach out to the community and start working with that community."</p> </div> <footer class="d-print-none"> <div class="footer__links"> <nav> <a class="text-white" href="/docs/home/">Home</a> <a class="text-white" href="/blog/">Blog</a> <a class="text-white" href="/training/">Training</a> <a class="text-white" href="/partners/">Partners</a> <a class="text-white" href="/community/">Community</a> <a class="text-white" href="/case-studies/">Case Studies</a> </nav> </div> <div class="container-fluid"> <div class="row"> <div class="col-6 col-sm-2 text-xs-center order-sm-2"> <ul class="list-inline mb-0"> <li class="list-inline-item mx-2 h3" data-toggle="tooltip" data-placement="top" title="User mailing list" aria-label="User mailing list"> <a class="text-white" target="_blank" href="https://discuss.kubernetes.io"> <i class="fa fa-envelope"></i> </a> </li> <li class="list-inline-item mx-2 h3" data-toggle="tooltip" data-placement="top" title="Twitter" aria-label="Twitter"> <a class="text-white" target="_blank" href="https://twitter.com/kubernetesio"> <i class="fab fa-twitter"></i> </a> </li> <li class="list-inline-item mx-2 h3" data-toggle="tooltip" data-placement="top" title="Calendar" aria-label="Calendar"> <a class="text-white" target="_blank" href="https://calendar.google.com/calendar/embed?src=calendar%40kubernetes.io"> <i class="fas fa-calendar-alt"></i> </a> </li> <li class="list-inline-item mx-2 h3" data-toggle="tooltip" data-placement="top" title="Youtube" aria-label="Youtube"> <a class="text-white" target="_blank" href="https://youtube.com/kubernetescommunity"> <i class="fab fa-youtube"></i> </a> </li> </ul> </div> <div class="col-6 col-sm-2 text-right text-xs-center order-sm-3"> <ul class="list-inline mb-0"> <li class="list-inline-item mx-2 h3" data-toggle="tooltip" data-placement="top" title="GitHub" aria-label="GitHub"> <a class="text-white" target="_blank" href="https://github.com/kubernetes/kubernetes"> <i class="fab fa-github"></i> </a> </li> <li class="list-inline-item mx-2 h3" data-toggle="tooltip" data-placement="top" title="Slack" aria-label="Slack"> <a class="text-white" target="_blank" href="https://slack.k8s.io"> <i class="fab fa-slack"></i> </a> </li> <li class="list-inline-item mx-2 h3" data-toggle="tooltip" data-placement="top" title="Contribute" aria-label="Contribute"> <a class="text-white" target="_blank" href="https://git.k8s.io/community/contributors/guide"> <i class="fas fa-edit"></i> </a> </li> <li class="list-inline-item mx-2 h3" data-toggle="tooltip" data-placement="top" title="Stack Overflow" aria-label="Stack Overflow"> <a class="text-white" target="_blank" href="https://stackoverflow.com/questions/tagged/kubernetes"> <i class="fab fa-stack-overflow"></i> </a> </li> </ul> </div> <div class="col-12 col-sm-8 text-center order-sm-2"> <small class="text-white">© 2024 The Kubernetes Authors | Documentation Distributed under <a href="https://git.k8s.io/website/LICENSE" class="light-text">CC BY 4.0</a></small> <br/> <small class="text-white">Copyright © 2024 The Linux Foundation ®. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our <a href="https://www.linuxfoundation.org/trademark-usage" class="light-text">Trademark Usage page</a></small> <br/> <small class="text-white">ICP license: 京ICP备17074266号-3</small> </div> </div> </div> </footer> <script src="/js/jquery-3.6.0.min.js" integrity="sha384-vtXRMe3mGCbOeY7l30aIg8H9p3GdeSe4IFlP6G8JMa7o7lXvnz3GFKzPxzJdPfGK" crossorigin="anonymous"></script> <script src="/js/popper-1.16.1.min.js" intregrity="sha384-9/reFTGAW83EW2RDu2S0VKaIzap3H66lZH81PoYlFhbGU+6BZp6G7niu735Sk7lN" crossorigin="anonymous"></script> <script src="/js/bootstrap-4.6.1.min.js" integrity="sha384-VHvPCCyXqtD5DqJeNxl2dtTyhF78xXNXdkwX1CZeRusQfRKp+tA7hAShOK/B/fQ2" crossorigin="anonymous"></script> <script src="/js/script.js"></script> <script async src="/js/mermaid-8.13.4.min.js" integrity="sha384-5hHNvPeMrNH14oM3IcQofDoBhiclNK3g2+hnEinKzQ07C4AliMeVpnvxuiwEGpaO" crossorigin="anonymous"></script> <script src="/js/main.min.5c0bf7f21dc4f66485f74efbbeeff28a7e4f8cddaac1bae47043159c922ff3a3.js" integrity="sha256-XAv38h3E9mSF9077vu/yin5PjN2qwbrkcEMVnJIv86M=" crossorigin="anonymous"></script> </body> </html>