CINXE.COM

CERN Case Study | Kubernetes

<!DOCTYPE html> <html id="caseStudies" lang="zh-cn" class=""> <head> <meta name="robots" content="noindex, nofollow"> <link rel="alternate" hreflang="en" href="https://kubernetes.io/case-studies/cern/"> <link rel="alternate" hreflang="bn" href="https://kubernetes.io/bn/case-studies/cern/"> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <meta name="generator" content="Hugo 0.121.2"> <link rel="shortcut icon" type="image/png" href="/images/kubernetes.png" /><link rel="icon" type="image/png" sizes="64x64" href="/icons/favicon-64.png" /><link rel="icon" type="image/png" sizes="32x32" href="/icons/favicon-32.png" /><link rel="icon" type="image/png" sizes="16x16" href="/icons/favicon-16.png" /><link rel="apple-touch-icon-256x256" href="/icons/apple-touch-icon-256x256.png" /><link rel="apple-touch-icon-196x196" href="/icons/apple-touch-icon-196x196.png" /><link rel="apple-touch-icon-192x192" href="/icons/apple-touch-icon-192x192.png" /><link rel="apple-touch-icon-180x180" href="/icons/apple-touch-icon-180x180.png" /><link rel="apple-touch-icon-167x167" href="/icons/apple-touch-icon-167x167.png" /><link rel="apple-touch-icon-160x160" href="/icons/apple-touch-icon-160x160.png" /><link rel="apple-touch-icon-152x152" href="/icons/apple-touch-icon-152x152.png" /><link rel="apple-touch-icon-120x120" href="/icons/apple-touch-icon-120x120.png" /><link rel="apple-touch-icon-76x76" href="/icons/apple-touch-icon-76x76.png" /><link rel="icon" type="image/png" href="/icons/icon-128x128.png" sizes=128x128><meta name="theme-color" content="#326de6" /> <title>CERN Case Study | Kubernetes</title><meta property="og:title" content="CERN Case Study" /> <meta property="og:description" content=" CASE STUDY: CERN CERN: Processing Petabytes of Data More Efficiently with Kubernetes Company &nbsp;CERN&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Location &nbsp;Geneva, Switzerland &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Industry &nbsp;Particle physics research " /> <meta property="og:type" content="article" /> <meta property="og:url" content="https://kubernetes.io/zh-cn/case-studies/cern/" /><meta property="og:image" content="https://kubernetes.io/zh-cn/case-studies/cern/cern_featured_logo.png" /><meta property="article:section" content="case-studies" /> <meta property="article:modified_time" content="2022-06-10T20:25:27+08:00" /> <meta itemprop="name" content="CERN Case Study"> <meta itemprop="description" content=" CASE STUDY: CERN CERN: Processing Petabytes of Data More Efficiently with Kubernetes Company &nbsp;CERN&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Location &nbsp;Geneva, Switzerland &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Industry &nbsp;Particle physics research "> <meta itemprop="dateModified" content="2022-06-10T20:25:27+08:00" /> <meta itemprop="wordCount" content="20"><meta itemprop="image" content="https://kubernetes.io/zh-cn/case-studies/cern/cern_featured_logo.png" /> <meta itemprop="keywords" content="" /><meta name="twitter:card" content="summary_large_image" /> <meta name="twitter:image" content="https://kubernetes.io/zh-cn/case-studies/cern/cern_featured_logo.png" /><meta name="twitter:title" content="CERN Case Study"/> <meta name="twitter:description" content=" CASE STUDY: CERN CERN: Processing Petabytes of Data More Efficiently with Kubernetes Company &nbsp;CERN&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Location &nbsp;Geneva, Switzerland &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Industry &nbsp;Particle physics research "/> <link href="/scss/main.css" rel="stylesheet"> <link rel="manifest" href="/manifest.webmanifest"> <script type="application/ld+json"> { "@context": "https://schema.org", "@type": "Organization", "url": "https://kubernetes.io", "logo": "https://kubernetes.io/images/favicon.png", } </script> <meta name="theme-color" content="#326ce5"> <style> .gutter { background-color: #eee; background-repeat: no-repeat; background-position: 50%; } .gutter.gutter-horizontal { background-image: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUAAAAeCAYAAADkftS9AAAAIklEQVQoU2M4c+bMfxAGAgYYmwGrIIiDjrELjpo5aiZeMwF+yNnOs5KSvgAAAABJRU5ErkJggg=='); cursor: col-resize; } #sidebarnav, #maindoc { max-width: 100%; } #maindoc { overflow-wrap: break-word; } @media (max-width: 768px) { #sidebarnav { padding-left: 15px; padding-right: 15px; } } </style> <link rel="stylesheet" href="/scss/_case-studies.f79eaadb37c5ce0b99b43075c83703b9e8d6e0b058a516ceb4cc46f9c1d98741.css" integrity="sha256-956q2zfFzguZtDB1yDcDuejW4LBYpRbOtMxG&#43;cHZh0E="> <link rel="stylesheet" href="/css/style_case_studies.css"> <meta name="description" content=" CASE STUDY: CERN CERN: Processing Petabytes of Data More Efficiently with Kubernetes Company &nbsp;CERN&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Location &nbsp;Geneva, Switzerland &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Industry &nbsp;Particle physics research "> <meta property="og:description" content=" CASE STUDY: CERN CERN: Processing Petabytes of Data More Efficiently with Kubernetes Company &nbsp;CERN&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Location &nbsp;Geneva, Switzerland &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Industry &nbsp;Particle physics research "> <meta name="twitter:description" content=" CASE STUDY: CERN CERN: Processing Petabytes of Data More Efficiently with Kubernetes Company &nbsp;CERN&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Location &nbsp;Geneva, Switzerland &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Industry &nbsp;Particle physics research "> <meta property="og:url" content="https://kubernetes.io/zh-cn/case-studies/cern/"> <meta property="og:title" content="CERN Case Study"> <meta name="twitter:title" content="CERN Case Study"> <meta name="twitter:image" content="https://kubernetes.io/images/favicon.png" /> <meta name="twitter:image:alt" content="Kubernetes"> <meta property="og:image" content="/images/kubernetes-horizontal-color.png"> <meta property="og:type" content="article"> <script src="/js/jquery-3.6.0.min.js" intregrity="sha384-vtXRMe3mGCbOeY7l30aIg8H9p3GdeSe4IFlP6G8JMa7o7lXvnz3GFKzPxzJdPfGK" crossorigin="anonymous"></script> <script src="/js/split-1.6.0.js" intregrity="sha384-0blL3GqHy6+9fw0cyY2Aoiwg4onHAtslAs4OkqZY7UQBrR65/K4gI+hxLdWDrjpz"></script> <script defer src="/js/dismiss_banner.js"></script> </head> <body> <nav class="js-navbar-scroll navbar navbar-expand navbar-dark flex-column flex-md-row td-navbar" data-auto-burger="primary"> <a class="navbar-brand img-fluid" href="/zh-cn/"></a> <div class="td-navbar-nav-scroll ml-md-auto" id="main_navbar"> <ul class="navbar-nav mt-2 mt-lg-0"> <li class="nav-item mr-2 mb-lg-0"> <a class="nav-link" href="/zh-cn/docs/" >文档</a> </li> <li class="nav-item mr-2 mb-lg-0"> <a class="nav-link" href="/zh-cn/blog/" >Kubernetes 博客</a> </li> <li class="nav-item mr-2 mb-lg-0"> <a class="nav-link" href="/zh-cn/training/" >培训</a> </li> <li class="nav-item mr-2 mb-lg-0"> <a class="nav-link" href="/zh-cn/partners/" >合作伙伴</a> </li> <li class="nav-item mr-2 mb-lg-0"> <a class="nav-link" href="/zh-cn/community/" >社区</a> </li> <li class="nav-item mr-2 mb-lg-0"> <a class="nav-link active" href="/zh-cn/case-studies/" >案例分析</a> </li> <li class="nav-item mr-n3 mr-lg-0 dropdown"> <a class="nav-link dropdown-toggle" href="#" id="navbarDropdown" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false"> 版本列表 </a> <div class="dropdown-menu dropdown-menu-right" aria-labelledby="navbarDropdownMenuLink"> <a class="dropdown-item" href="/zh-cn/releases">发布信息</a> <a class="dropdown-item" href="https://kubernetes.io/zh-cn/case-studies/cern/">v1.31</a> <a class="dropdown-item" href="https://v1-30.docs.kubernetes.io/zh-cn/case-studies/cern/">v1.30</a> <a class="dropdown-item" href="https://v1-29.docs.kubernetes.io/zh-cn/case-studies/cern/">v1.29</a> <a class="dropdown-item" href="https://v1-28.docs.kubernetes.io/zh-cn/case-studies/cern/">v1.28</a> <a class="dropdown-item" href="https://v1-27.docs.kubernetes.io/zh-cn/case-studies/cern/">v1.27</a> </div> </li> <li class="nav-item mr-n4 mr-lg-0 dropdown"> <a class="nav-link dropdown-toggle" href="#" id="navbarDropdownMenuLink" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false"> 中文 (Chinese) </a> <div class="dropdown-menu dropdown-menu-right" aria-labelledby="navbarDropdownMenuLink"> <a class="dropdown-item" href="/case-studies/cern/">English</a> <a class="dropdown-item" href="/bn/case-studies/cern/">বাংলা (Bengali)</a> </div> </li> <li class="search-item nav-item mr-n4 mr-lg-0"> <div class="search-bar"> <i class="search-icon fas fa-search"></i> <input type="search" name="q" data-search-page="/zh-cn/search/" class="search-input td-search-input" placeholder="搜索" aria-label="搜索" autocomplete="off" > </div> </li> </ul> </div> <button id="hamburger" onclick="kub.toggleMenu()" data-auto-burger-exclude><div></div></button> </nav> <section id="deprecation-warning"> <div class="content deprecation-warning pageinfo"> <h3> 你正在查看的文档所针对的是 Kubernetes 版本: v1.30 </h3> <p> Kubernetes v1.30 版本的文档已不再维护。你现在看到的版本来自于一份静态的快照。如需查阅最新文档,请点击 <a href="https://kubernetes.io/docs/home/">最新版本。</a> </p> </div> </section> <div data-pagefind-body> <div class="banner1" style="background-image: url('/images/case-studies/cern/banner1.jpg')"> <h1> CASE STUDY: CERN<br> <div class="subhead" style="margin-top:1%">CERN: Processing Petabytes of Data More Efficiently with Kubernetes </div></h1> </div> <div class="details"> Company &nbsp;<b>CERN</b>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Location &nbsp;<b>Geneva, Switzerland </b>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Industry &nbsp;<b>Particle physics research</b> </div> <hr> <section class="section1"> <div class="cols"> <div class="col1" style="width:100%""> <h2>Challenge</h2> At CERN, the European Organization for Nuclear Research, physicists conduct experiments to learn about fundamental science. In its particle accelerators, "we accelerate protons to very high energy, close to the speed of light, and we make the two beams of protons collide," says CERN Software Engineer Ricardo Rocha. "The end result is a lot of data that we have to process." CERN currently stores 330 petabytes of data in its data centers, and an upgrade of its accelerators expected in the next few years will drive that number up by 10x. Additionally, the organization experiences extreme peaks in its workloads during periods prior to big conferences, and needs its infrastructure to scale to those peaks. "We want to have a more hybrid infrastructure, where we have our on premise infrastructure but can make use of public clouds temporarily when these peaks come up," says Rocha. "We’ve been looking to new technologies that can help improve our efficiency in our infrastructure so that we can dedicate more of our resources to the actual processing of the data." <br><br> <h2>Solution</h2> CERN’s technology team embraced containerization and cloud native practices, choosing Kubernetes for orchestration, Helm for deployment, Prometheus for monitoring, and CoreDNS for DNS resolution inside the clusters. Kubernetes federation has allowed the organization to run some production workloads both on premise and in public clouds. <br><br> <h2>Impact</h2> "Kubernetes gives us the full automation of the application," says Rocha. "It comes with built-in monitoring and logging for all the applications and the workloads that deploy in Kubernetes. This is a massive simplification of our current deployments." The time to deploy a new cluster for a complex distributed storage system has gone from more than 3 hours to less than 15 minutes. Adding new nodes to a cluster used to take more than an hour; now it takes less than 2 minutes. The time it takes to autoscale replicas for system components has decreased from more than an hour to less than 2 minutes. Initially, virtualization gave 20% overhead, but with tuning this was reduced to ~5%. Moving to Kubernetes on bare metal would get this to 0%. Not having to host virtual machines is expected to also get 10% of memory capacity back. </div> </div> </section> <div class="banner2"> <div class="banner2text"> "Kubernetes is something we can relate to very much because it’s naturally distributed. What it gives us is a uniform API across heterogeneous resources to define our workloads. This is something we struggled with a lot in the past when we want to expand our resources outside our infrastructure." <br style="height:25px"><span style="font-size:14px;letter-spacing:2px;text-transform:uppercase;margin-top:5% !important;"><br>- Ricardo Rocha, Software Engineer, CERN</span> </div> </div> <section class="section2"> <div class="fullcol"> <h2>With a mission of researching fundamental science, and a stable of extremely large machines, the European Organization for Nuclear Research (CERN) operates at what can only be described as hyperscale. </h2> Experiments are conducted in particle accelerators, the biggest of which is 27 kilometers in circumference. "We accelerate protons to very high energy, to close to the speed of light, and we make the two beams of protons collide in well-defined places," says CERN Software Engineer Ricardo Rocha. "We build experiments around these places where we do the collisions. The end result is a lot of data that we have to process."<br><br> And he does mean a lot: CERN currently stores and processes 330 petabytes of data—gathered from 4,300 projects and 3,300 users—using 10,000 hypervisors and 320,000 cores in its data centers. <br><br> Over the years, the CERN technology department has built a large computing infrastructure, based on OpenStack private clouds, to help the organization’s physicists analyze and treat all this data. The organization experiences extreme peaks in its workloads. "Very often, just before conferences, physicists want to do an enormous amount of extra analysis to publish their papers, and we have to scale to these peaks, which means overcommitting resources in some cases," says Rocha. "We want to have a more hybrid infrastructure, where we have our on premise infrastructure but can make use of public clouds temporarily when these peaks come up."<br><br> Additionally, few years ago, CERN announced that it would be doing a big upgrade of its accelerators, which will mean a ten-fold increase in the amount of data that can be collected. "So we’ve been looking to new technologies that can help improve our efficiency in our infrastructure, so that we can dedicate more of our resources to the actual processing of the data," says Rocha. </div> </section> <div class="banner3" style="background-image: url('/images/case-studies/cern/banner3.jpg')"> <div class="banner3text"> "Before, the tendency was always: ‘I need this, I get a couple of developers, and I implement it.’ Right now it’s ‘I need this, I’m sure other people also need this, so I’ll go and ask around.’ The CNCF is a good source because there’s a very large catalog of applications available. It’s very hard right now to justify developing a new product in-house. There is really no real reason to keep doing that. It’s much easier for us to try it out, and if we see it’s a good solution, we try to reach out to the community and start working with that community." <br style="height:25px"><span style="font-size:14px;letter-spacing:2px;text-transform:uppercase;margin-top:5% !important;"><br>- Ricardo Rocha, Software Engineer, CERN</span> </div> </div> <section class="section3"> <div class="fullcol"> Rocha’s team started looking at Kubernetes and containerization in the second half of 2015. "We’ve been using distributed infrastructures for decades now," says Rocha. "Kubernetes is something we can relate to very much because it’s naturally distributed. What it gives us is a uniform API across heterogeneous resources to define our workloads. This is something we struggled with a lot in the past when we want to expand our resources outside our infrastructure."<br><br> The team created a prototype system for users to deploy their own Kubernetes cluster in CERN’s infrastructure, and spent six months validating the use cases and making sure that Kubernetes integrated with CERN’s internal systems. The main use case is batch workloads, which represent more than 80% of resource usage at CERN. (One single project that does most of the physics data processing and analysis alone consumes 250,000 cores.) "This is something where the investment in simplification of the deployment, logging, and monitoring pays off very quickly," says Rocha. Other use cases include Spark-based data analysis and machine learning to improve physics analysis. "The fact that most of these technologies integrate very well with Kubernetes makes our lives easier," he adds.<br><br> The system went into production in October 2016, also using Helm for deployment, Prometheus for monitoring, and CoreDNS for DNS resolution within the cluster. "One thing that Kubernetes gives us is the full automation of the application," says Rocha. "So it comes with built-in monitoring and logging for all the applications and the workloads that deploy in Kubernetes. This is a massive simplification of our current deployments." The time to deploy a new cluster for a complex distributed storage system has gone from more than 3 hours to less than 15 minutes.<br><br> Adding new nodes to a cluster used to take more than an hour; now it takes less than 2 minutes. The time it takes to autoscale replicas for system components has decreased from more than an hour to less than 2 minutes. </div> </section> <div class="banner4" style="background-image: url('/images/case-studies/cern/banner4.jpg')"> <div class="banner4text"> "With Kubernetes, there’s a well-established technology and a big community that we can contribute to. It allows us to do our physics analysis without having to focus so much on the lower level software. This is just exciting. We are looking forward to keep contributing to the community and collaborating with everyone."<br style="height:25px"><span style="font-size:14px;letter-spacing:2px;text-transform:uppercase;margin-top:5% !important;"><br>- Ricardo Rocha, Software Engineer, CERN</span> </div> </div> <section class="section5" style="padding:0px !important"> <div class="fullcol"> Rocha points out that the metric used in the particle accelerators may be events per second, but in reality "it’s how fast and how much of the data we can process that actually counts." And efficiency has certainly been improved with Kubernetes. Initially, virtualization gave 20% overhead, but with tuning this was reduced to ~5%. Moving to Kubernetes on bare metal would get this to 0%. Not having to host virtual machines is expected to also get 10% of memory capacity back.<br><br> Kubernetes federation, which CERN has been using for a portion of its production workloads since February 2018, has allowed the organization to adopt a hybrid cloud strategy. And it was remarkably simple to do. "We had a summer intern working on federation," says Rocha. "For many years, I’ve been developing distributed computing software, which took like a decade and a lot of effort from a lot of people to stabilize and make sure it works. And for our intern, in a couple of days he was able to demo to me and my team that we had a cluster at CERN and a few clusters outside in public clouds that were federated together and that we could submit workloads to. This was shocking for us. It really shows the power of using this kind of well-established technologies." <br><br> With such results, adoption of Kubernetes has made rapid gains at CERN, and the team is eager to give back to the community. "If we look back into the ’90s and early 2000s, there were not a lot of companies focusing on systems that have to scale to this kind of size, storing petabytes of data, analyzing petabytes of data," says Rocha. "The fact that Kubernetes is supported by such a wide community and different backgrounds, it motivates us to contribute back." </div> <div class="banner5" > <div class="banner5text"> This means that the physicist can build his or her analysis and publish it in a repository, share it with colleagues, and in 10 years redo the same analysis with new data. If we looked back even 10 years, this was just a dream."<br style="height:25px"><span style="font-size:14px;letter-spacing:2px;text-transform:uppercase;margin-top:5% !important;"><br>- Ricardo Rocha, Software Engineer, CERN</span></div> </div> <div class="fullcol"> These new technologies aren’t just enabling infrastructure improvements. CERN also uses the Kubernetes-based <a href="https://github.com/recast-hep">Reana/Recast</a> platform for reusable analysis, which is "the ability to define physics analysis as a set of workflows that are fully containerized in one single entry point," says Rocha. "This means that the physicist can build his or her analysis and publish it in a repository, share it with colleagues, and in 10 years redo the same analysis with new data. If we looked back even 10 years, this was just a dream."<br><br> All of these things have changed the culture at CERN considerably. A decade ago, "The tendency was always: ‘I need this, I get a couple of developers, and I implement it,’" says Rocha. "Right now it’s ‘I need this, I’m sure other people also need this, so I’ll go and ask around.’ The CNCF is a good source because there’s a very large catalog of applications available. It’s very hard right now to justify developing a new product in-house. There is really no real reason to keep doing that. It’s much easier for us to try it out, and if we see it’s a good solution, we try to reach out to the community and start working with that community." </div> </section> </div> <footer class="d-print-none"> <div class="footer__links"> <nav> <a class="text-white" href="/zh-cn/docs/home/">文档</a> <a class="text-white" href="/zh-cn/blog/">博客</a> <a class="text-white" href="/zh-cn/training/">培训</a> <a class="text-white" href="/zh-cn/partners/">合作伙伴</a> <a class="text-white" href="/zh-cn/community/">社区</a> <a class="text-white" href="/zh-cn/case-studies/">案例分析</a> </nav> </div> <div class="container-fluid"> <div class="row"> <div class="col-6 col-sm-2 text-xs-center order-sm-2"> <ul class="list-inline mb-0"> <li class="list-inline-item mx-2 h3" data-toggle="tooltip" data-placement="top" title="User mailing list" aria-label="User mailing list"> <a class="text-white" target="_blank" href="https://discuss.kubernetes.io"> <i class="fa fa-envelope"></i> </a> </li> <li class="list-inline-item mx-2 h3" data-toggle="tooltip" data-placement="top" title="X(Twitter)" aria-label="X(Twitter)"> <a class="text-white" target="_blank" href="https://twitter.com/kubernetesio"> <i class="fab fa-x-twitter"></i> </a> </li> <li class="list-inline-item mx-2 h3" data-toggle="tooltip" data-placement="top" title="Calendar" aria-label="Calendar"> <a class="text-white" target="_blank" href="https://calendar.google.com/calendar/embed?src=calendar%40kubernetes.io"> <i class="fas fa-calendar-alt"></i> </a> </li> <li class="list-inline-item mx-2 h3" data-toggle="tooltip" data-placement="top" title="Youtube" aria-label="Youtube"> <a class="text-white" target="_blank" href="https://youtube.com/kubernetescommunity"> <i class="fab fa-youtube"></i> </a> </li> </ul> </div> <div class="col-6 col-sm-2 text-right text-xs-center order-sm-3"> <ul class="list-inline mb-0"> <li class="list-inline-item mx-2 h3" data-toggle="tooltip" data-placement="top" title="GitHub" aria-label="GitHub"> <a class="text-white" target="_blank" href="https://github.com/kubernetes/kubernetes"> <i class="fab fa-github"></i> </a> </li> <li class="list-inline-item mx-2 h3" data-toggle="tooltip" data-placement="top" title="Slack" aria-label="Slack"> <a class="text-white" target="_blank" href="https://slack.k8s.io"> <i class="fab fa-slack"></i> </a> </li> <li class="list-inline-item mx-2 h3" data-toggle="tooltip" data-placement="top" title="Contribute" aria-label="Contribute"> <a class="text-white" target="_blank" href="https://git.k8s.io/community/contributors/guide"> <i class="fas fa-edit"></i> </a> </li> <li class="list-inline-item mx-2 h3" data-toggle="tooltip" data-placement="top" title="Stack Overflow" aria-label="Stack Overflow"> <a class="text-white" target="_blank" href="https://stackoverflow.com/questions/tagged/kubernetes"> <i class="fab fa-stack-overflow"></i> </a> </li> </ul> </div> <div class="col-12 col-sm-8 text-center order-sm-2"> <small class="text-white">&copy; 2024 The Kubernetes 作者 | 文档发布基于 <a href="https://git.k8s.io/website/LICENSE" class="light-text">CC BY 4.0</a> 授权许可</small> <br/> <small class="text-white">Copyright &copy; 2024 Linux 基金会&reg;。保留所有权利。Linux 基金会已注册并使用商标。如需了解 Linux 基金会的商标列表,请访问<a href="https://www.linuxfoundation.org/trademark-usage" class="light-text">商标使用页面</a></small> <br/> <small class="text-white">ICP 许可: 京ICP备17074266号-3</small> </div> </div> </div> </footer> <script src="/js/jquery-3.6.0.min.js" integrity="sha384-vtXRMe3mGCbOeY7l30aIg8H9p3GdeSe4IFlP6G8JMa7o7lXvnz3GFKzPxzJdPfGK" crossorigin="anonymous"></script> <script src="/js/popper-1.16.1.min.js" intregrity="sha384-9/reFTGAW83EW2RDu2S0VKaIzap3H66lZH81PoYlFhbGU+6BZp6G7niu735Sk7lN" crossorigin="anonymous"></script> <script src="/js/bootstrap-4.6.1.min.js" integrity="sha384-VHvPCCyXqtD5DqJeNxl2dtTyhF78xXNXdkwX1CZeRusQfRKp+tA7hAShOK/B/fQ2" crossorigin="anonymous"></script> <script src="/js/script.js"></script> <script src="/js/main.min.b8b5957f4ba9b582bc3696c8cbed20dde33f789d0fe3255f10fdc7adecf38b7e.js" integrity="sha256-uLWVf0uptYK8NpbIy&#43;0g3eM/eJ0P4yVfEP3Hrezzi34=" crossorigin="anonymous"></script> <script> let splitInstance = null; function enableSplitter(mediaQuery) { if (mediaQuery.matches) { if (!splitInstance) { splitInstance = Split(["#sidebarnav", "#maindoc"], { sizes: [20, 80], minSize: 100, }); } } else { if (splitInstance) { splitInstance.destroy(); splitInstance = null; } } } const screenWidthMediaQuery = window.matchMedia("(min-width: 768px)"); const eleNav = document.getElementById("sidebarnav"); if (eleNav !== null) { enableSplitter(screenWidthMediaQuery); screenWidthMediaQuery.addListener(enableSplitter); } </script> </body> </html>

Pages: 1 2 3 4 5 6 7 8 9 10