Baidu Research

<!doctype html> <html> <head> <meta charset="utf-8"> <meta name="renderer" content="webkit" /> <meta name="format-detection" content="telephone=no"/> <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" /> <meta content="width=device-width, initial-scale=1, maximum-scale=1, minimum-scale=1" name="viewport" /> <title>Baidu Research</title> <link rel="shortcut icon" href="../web/images/favicon.png" type="image/x-icon"> <link rel="stylesheet" href="../web/css/reset.css"> <link rel="stylesheet" href="../web/css/animate.css"> <link rel="stylesheet" href="../web/css/Bootstrap.css"> <link rel="stylesheet" href="../web/css/style.css"> <link rel="stylesheet" href="../web/css/media.css?3333"> <script src="../web/js/jquery.min.js"></script> </head> <body class="n_body" ondragstart="return false">  <div class="n_header"> <div class="container"> <div class="header01"> <div class="logo"> <a class="h_logo" href="/Index" style="background-image: url(../web/images/logo.png);"><img src="../web/images/logo.png" alt="Baidu Research"></a> </div> <div class="nav"> <ul> <li><a href="/Index">Home</a></li> <li><a href="/Publications">Publications</a></li> <li><a href="/Research_Areas?id=55">Research Areas</a> <div class="nav_er"> <ul class="div_dl "> <li><a href="/Research_Areas/index-view?id=55">Data Science and Data Mining</a></li> <li><a href="/Research_Areas/index-view?id=56">Natural Language and Speech</a></li> <li><a href="/Research_Areas/index-view?id=57">Business Intelligence</a></li> <li><a href="/Research_Areas/index-view?id=58">Robotics and Autonomous Driving</a></li> <li><a href="/Research_Areas/index-view?id=59">Computer Vision</a></li> <li><a href="/Research_Areas/index-view?id=60">Machine Learning and Deep Learning</a></li> <li><a href="/Research_Areas/index-view?id=61">Computational Biology and Bioinformatics</a></li> <li><a href="/Research_Areas/index-view?id=62">High Performance Computing</a></li> <li><a href="/Research_Areas/index-view?id=75">Quantum Computing</a></li> </ul> </div> </li> <li><a class="active" href="/Blog">Blog</a></li> <li><a href="/Career">Careers</a></li> <li><a href="/Downloads">Downloads</a></li> <li><a href="/AI_Colloquium">AI Colloquium</a></li> </ul> <div class="nav_btn"><span></span></div> </div> </div> </div> <div class="header03 "> <div class="logo"><a href="/Index"><img src="../web/images/logo.png" alt="Baidu Research"></a></div> <div class="nav"> <ul> <li><a href="/Index">Home</a></li> <li><a href="/Publications">Publications</a></li> <li><a href="/Research_Areas?id=55">Research Areas</a> <div class="nav_er"> <ul class="div_dl "> <li><a href="/Research_Areas/index-view?id=55">Data Science and Data Mining</a></li> <li><a href="/Research_Areas/index-view?id=56">Natural Language and Speech</a></li> <li><a href="/Research_Areas/index-view?id=57">Business Intelligence</a></li> <li><a href="/Research_Areas/index-view?id=58">Robotics and Autonomous Driving</a></li> <li><a href="/Research_Areas/index-view?id=59">Computer Vision</a></li> <li><a href="/Research_Areas/index-view?id=60">Machine Learning and Deep Learning</a></li> <li><a href="/Research_Areas/index-view?id=61">Computational Biology and Bioinformatics</a></li> <li><a href="/Research_Areas/index-view?id=62">High Performance Computing</a></li> <li><a href="/Research_Areas/index-view?id=75">Quantum Computing</a></li> </ul> </div> </li> <li><a href="/Blog">Blog</a></li> <li><a href="/Career">Careers</a></li> <li><a href="/Downloads">Downloads</a></li> <li><a href="/AI_Colloquium">AI Colloquium</a></li> </ul> </div> <div class="nav_btn"><span></span></div> </div> </div> <div class="baidu-page-banner blog-side" style="background: url(/Public/uploads/5ae96c0a7676c.png);"> <div class="container"> <div class="baidu-page-title wow fadeIn">Blog</div> </div> </div> <div class="content-info"> <div class="container-details-er"> <div class="blog-details-title">Baidu Releases PaddlePaddle 2.0 with New Features Including Dynamic Graphs, Reorganized APIs and Heterogeneous Parameter Server</div> <div class="blog-details-date"><p>2021-04-01</p><a href="/Blog">Back to list</a></div> <p style="text-align:center"><br/></p><p style="text-align:center"><img src="/ueditor/upload/20210401/1617257221139814.png" title="1617257221139814.png" alt="46cb9dd942a5ace791e6e4ca97c7e535.png" width="1113" height="163"/></p><p style="text-align: center; font-size: 14px; font-family: DengXian; white-space: normal;"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; text-align: justify; color: rgb(0, 0, 0);"><br/></span></p><p style="text-align: justify; font-size: 14px; font-family: DengXian; white-space: normal;"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; text-align: justify; color: rgb(0, 0, 0);">We are excited to announce that PaddlePaddle 2.0 is now available. The deep learning platform's latest version features dynamic (computational) graphs, a new API system, distributed training for trillion-parameter models, and better hardware support.<br/></span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-size: 16px; font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 0);"> </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-size: 16px; font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 0);">Since the release in 2016, both PaddlePaddle and Baidu Brain, our open AI platform and core AI technology engine, have attracted over 2.65 million developers to join in as of December 31st, 2020, and we have owned the largest AI developer community among Chinese companies.</span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);"> </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);">GitHub: </span><a href="https://github.com/PaddlePaddle" style="text-decoration: underline; font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);">https://github.com/PaddlePaddle</span></a></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);">Gitee: </span><a href="https://gitee.com/paddlepaddle" style="text-decoration: underline; font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);">https://gitee.com/paddlepaddle</span></a></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);"> </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-size: 16px; font-family: arial, helvetica, sans-serif;"><strong><span style="font-size: 16px; color: rgb(14, 16, 26);">Dynamic graphs</span></strong></span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="color: rgb(14, 16, 26); font-size: 16px; font-family: arial, helvetica, sans-serif;"> </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="color: rgb(14, 16, 26); font-size: 16px; font-family: arial, helvetica, sans-serif;">PaddlePaddle’s earlier versions added support of dynamic graphs. Compared with static graphs, dynamic graphs demonstrate more flexible programming and better debug mechanisms. In version 2.0, PaddlePaddle has switched its default programming paradigm from static graphs to dynamic graphs, allowing developers to check inputs and outputs of tensors at any time and to quickly debug their codes. Dynamic graphs in version 2.0 support automatic mixed precision and quantization aware training.</span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="color: rgb(14, 16, 26); font-size: 16px; font-family: arial, helvetica, sans-serif;"> </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="color: rgb(14, 16, 26); font-size: 16px; font-family: arial, helvetica, sans-serif;">PaddlePaddle 2.0 also provides a smooth conversion of dynamic graphs to static graphs with unified interfaces to accelerate model training and deployment, as static graphs enable optimized training and inference performance once a graph is already defined. To use the static graph programming paradigm, developers can switch to static graph mode by running paddle.enable_static(). </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="color: rgb(14, 16, 26); font-size: 16px; font-family: arial, helvetica, sans-serif;"> </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="color: rgb(14, 16, 26); font-size: 16px; font-family: arial, helvetica, sans-serif;">PaddlePaddle's development kits have received a dynamic graph upgrade to fully facilitate industrial AI applications. For example, PaddleSeg, PaddlePaddle's image segmentation dev kit, has collected numerous high-precision and light-weight segmentation models, powering both configuration and API driven development through a modular structure. </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-size: 15px; color: rgb(14, 16, 26); font-family: arial, helvetica, sans-serif;"> </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);">Dynamic graphs: </span><a href="https://github.com/PaddlePaddle/models/tree/develop/dygraph" style="text-decoration: underline; font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);">https://github.com/PaddlePaddle/models/tree/develop/dygraph</span></a></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-size: 15px; color: rgb(14, 16, 26); font-family: arial, helvetica, sans-serif;"> </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);"><strong><span style="font-size: 16px; font-family: arial, helvetica, sans-serif;">New API upgrade</span></strong><strong><span style="font-size: 16px; font-family: arial, helvetica, sans-serif;"></span></strong></span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);"> </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);">In version 2.0, we have improved the API namespaces to help developers quickly locate target API, allowing them to build their neural networks with stateful class APIs and stateless functional APIs. 217 new APIs have been added to the system, while 195 APIs have been reoptimized. </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);"> </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);">PaddlePaddle also provides high-level APIs in data augmentation, data pipeline, batch training, and more tasks tailored for low-code programming. Thanks to these high-level APIs, developers can train a model with only 10 lines of code. Most importantly, the hierarchical design of high-level and low-level APIs leads to seamless programming, when both types of APIs are used, allowing developers to simply leverage these two API types and to promptly switch between simple implementation and detailed customization. The new API system is fully compatible with previous versions. PaddlePaddle also provides upgrade tools to help developers reduce upgrade and migration costs.</span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-size: 15px; color: rgb(14, 16, 26); font-family: arial, helvetica, sans-serif;"> </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);">PaddlePaddle API Documentation: </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><a href="https://www.paddlepaddle.org.cn/documentation/docs/en/api/index_en.html" style="text-decoration: underline; font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);">https://www.paddlepaddle.org.cn/documentation/docs/en/api/index_en.html</span></a></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);"> </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);"><strong><span style="font-size: 16px; font-family: arial, helvetica, sans-serif;">Large-scale model training</span></strong><strong><span style="font-size: 16px; font-family: arial, helvetica, sans-serif;"></span></strong></span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);"> </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);">The acronym “Paddle” stands for “PArallel Distributed Deep LEarning”. PaddlePaddle is designed to perform large-scale deep network training, using data sources distributed across hundreds of nodes. PaddlePaddle 2.0 introduces a new hybrid parallel mode, with which developers can combine data-parallel, model-parallel, and pipeline-parallel training. As a result, each layer of a network's parameters can be distributed to multiple GPUs, making the training of a trillion-parameter model computationally effective.</span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-size: 15px; color: rgb(14, 16, 26); font-family: arial, helvetica, sans-serif;"> </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);"><strong><span style="font-size: 16px; font-family: arial, helvetica, sans-serif;">General-purpose heterogeneous parameter server</span></strong><strong><span style="font-size: 16px; font-family: arial, helvetica, sans-serif;"></span></strong></span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);"> </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);">PaddlePaddle 2.0 has released the industry's first general-purpose heterogeneous parameter server technology, which supports different hardware types – CPU, GPU, and AI chips – efficiently in a single training job. This tackles the inefficiency of computing resources’ utilization, caused by high IO usage, while large-scale sparse-feature models of search and recommendation applications are trained. Developers can deploy distributed training tasks on heterogeneous hardware clusters and leverage different kinds of chips efficiently.</span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);"><br/></span></p><p style="text-align: center; font-size: 14px; font-family: DengXian; white-space: normal;"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);"><img src="/ueditor/upload/20210331/1617157078167791.png" title="1617157078167791.png" alt="eae1614794d1a547d40f6462f.png" width="693" height="436"/></span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);"> </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);">The general-purpose heterogeneous parameter server architecture is compatible with traditional parameter servers composed of CPUs, parameter servers consisting of GPUs or other AI accelerators, and heterogeneous parameter servers that mix CPUs with GPUs or AI accelerators. It has also been proven to be cost-effective. As shown in the figure below, two CPUs and two GPUs can achieve a training speed similar to 4 GPUs, while saving at least 35% of the cost.</span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);"><br/></span></p><p style="text-align:center"><span style="font-family: arial, helvetica, sans-serif; font-size: 16px; color: rgb(0, 0, 0);"><img src="/ueditor/upload/20210330/1617063165619991.png" title="1617063165619991.png" alt="图片 1.png" width="688" height="617"/></span></p><p><br/></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-size: 16px; font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 0);"><strong>New Hardware Support</strong></span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-size: 16px; font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 0);"> </span></p><p style=";text-align: justify;font-size: 14px;font-family: DengXian;white-space: normal"><span style="font-size: 16px; font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 0);">There are 29 categories of processors from semiconductor manufacturers like Intel, NVIDIA, and ARM, which have already been compatible with or are in the process of being compatible with PaddlePaddle 2.0. </span></p><p><br/></p> <div class="pager"> <a href="/Blog/index-view?id=157"> <i class="glyphicon glyphicon-menu-up"></i>Previous One：PaddlePaddle Receives Full Upgrade to Facilitate Integrated Innovation and Lower Thresholds </a> <a href="/Blog/index-view?id=155"> <i class="glyphicon glyphicon-menu-down"></i>Next One：PaddlePaddle’s Graph Neural Networks Advance Drug Design </a> </div> </div> </div> <footer> <div class="baidu-bottom"> <div class="container"> <div class="col-md-6 col-xs-12"> <h2>Baidu Research</h2> <p>1195 Bordeaux Drive Sunnyvale, CA 94089<br>Baidu Technology Park, No. 10 Xibeiwang East Road, Haidian District, Beijing, China<br>Media Inquiries: <a href="mailto:intlcomm@baidu.com">intlcomm@baidu.com</a><br>General Inquries: <a href="mailto:air-info@baidu.com">air-info@baidu.com</a></p> </div> <div class="col-md-6 col-xs-12"> <ul class="social-icons"> <li><a href="https://twitter.com/baiduresearch" target="_blank"><img src="../web/images/ico-2.png"></a> </li> <li><a href="https://www.linkedin.com/company/baidu-usa" target="_blank"><img src="../web/images/ico-3.png"></a> </li> </ul> <div class="baidu-weibu"> <div class="baidu-img"><img src="../web/images/f-logo.png"></div> <div class="baidu-links"> <a class="baidu-links-title" href="javascript:;">Links</a> <ul class="baidu-links-friends"> <li><a href="http://ai.baidu.com/" target="_blank">Baidu AI Open Platform</a> </li> <li><a href="http://www.dlnel.org/" target="_blank">DLNEL</a> </li> </ul> </div> </div> </div> </div> </div> <div class="baidu-foot">© 2018 Baidu Research</div> </footer> <script src="../web/js/bootstrap.min.js"></script> <script src="../web/js/wow.js"></script> <script src="../web/js/base.js"></script> </body> </html>

CINXE.COM

Baidu Research