Top Five Lessons from Running A/B Tests on the World's Largest Professional Network

<!DOCTYPE html> <html lang="en"> <head> <meta name="pageKey" content="d_lithograph_content">  <meta name="locale" content="en_US"> <meta id="config" data-app-id="lithograph" data-app-version="0.0.1084" data-call-tree-id="AAYn2eUxwxFsn2+A41k1pA==" data-multiproduct-name="lithograph-publish-frontend" data-service-name="lithograph-publish-frontend" data-browser-id="4c5b32e0-9396-4922-886b-413e171e4cb5" data-page-instance="urn:li:page:d_lithograph_content;uvYghLH7TbqbPuqAyYPDlA==" data-disable-jsbeacon-pagekey-suffix="false"> <link rel="canonical" href="https://www.linkedin.com/blog/engineering/archive/top-five-lessons-from-running-a-b-tests-on-the-world-s-largest-p">     <link rel="manifest" href="/homepage-guest/manifest.json" crossorigin="use-credentials"> <link rel="icon" href="https://static.licdn.com/aero-v1/sc/h/al2o9zrvru7aqj8e1x2rzsrca"> <script> function getDfd() {let yFn,nFn;const p=new Promise(function(y, n){yFn=y;nFn=n;});p.resolve=yFn;p.reject=nFn;return p;} window.lazyloader = getDfd(); window.tracking = getDfd(); window.impressionTracking = getDfd(); window.ingraphTracking = getDfd(); window.appDetection = getDfd(); window.pemTracking = getDfd(); </script>  <meta name="viewport" content="width=device-width, initial-scale=1.0"> <title>Top Five Lessons from Running A/B Tests on the World's Largest Professional Network</title>  <meta property="og:type" content="website"> <meta property="og:title" content="Top Five Lessons from Running A/B Tests on the World's Largest Professional Network"> <meta property="og:description" content="We run thousands of experiments each year at LinkedIn. Thousands of features, big and small, have been launched, refined, or discarded through our robust A/B testing platform. As a result of these experiments, we have learned a tremendous amount about how to make our products better and our members happier. At the same time, we have accumulated knowledge on how to run better experiments and how to leverage experimentation for better decision making. Here are the top five lessons on my ever-growing list."> <meta property="og:url" content="https://www.linkedin.com/blog/engineering/archive/top-five-lessons-from-running-a-b-tests-on-the-world-s-largest-p"> <meta property="og:image" content="https://media.licdn.com/dms/image/v2/D4D08AQFwzKhP8QTkcA/croft-frontend-shrinkToFit1024/croft-frontend-shrinkToFit1024/0/1700688407866?e=2147483647&v=beta&t=JrXwYwPdy7NmPtylqLz_HZZTZ53F0UwsCyMjYhma8Kw"> <meta name="twitter:title" content="Top Five Lessons from Running A/B Tests on the World's Largest Professional Network"> <meta name="twitter:site" content="@linkedin">   <meta name="robots" content="noarchive"> <meta name="linkedin:pageTag" content="/blog/engineering/archive/top-five-lessons-from-running-a-b-tests-on-the-world-s-largest-p"> <meta name="litmsProfileName" content="lithograph">  <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="asset-url" id="artdeco/static/images/icons.svg" content="https://static.licdn.com/aero-v1/sc/h/6sz1e821vgereyxaakx87tt4z "> <link rel="stylesheet" href="https://static.licdn.com/aero-v1/sc/h/crjcj3vgj0tgpae2eu4u9y7eb">   </head> <body dir="ltr">  <a href="#lithograph-app" class="skip-link btn-md btn-primary absolute z-11 -top-[100vh] focus:top-0"> Skip to main content </a>  <div class="page-header__wrapper page-header__wrapper--margin" data-custom-nav-wrapper="true"> <div class="page-header__container"> <header id="page-header" class="page-header page-header--hide-dropshadow"> <div class="page-header__content-container"> <div class="header__logo"> <a href="https://www.linkedin.com/blog/engineering" class="custom-nav__logo-link"> <img class="header__linkedin-logo header__full-logo" src="https://static.licdn.com/aero-v1/sc/h/6zm111mce7vohqze950ilreo4" alt="LinkedIn Logo"> <img class="header__linkedin-logo header__in-bug" src="https://static.licdn.com/aero-v1/sc/h/5g0hjlcng3j5pgn50n2et1ca2" alt="LinkedIn Logo"> <span class="t-17 t-black t-bold default-header-title"> Engineering Blog </span> </a> </div>   <button class="stripped header__expand-nav-button" data-mobile-header-navigation-button data-header-expand-navigation-button aria-expanded="false"> <li-icon class="header__expand-nav-icon" a11y-text="Open navigation" type="hamburger-icon"></li-icon> </button> <button class="stripped header__expand-nav-button hidden" data-mobile-header-navigation-button data-header-collapse-navigation-button aria-expanded="false"> <li-icon class="header__expand-nav-icon" a11y-text="Close navigation" type="cancel-icon"></li-icon> </button> </div> </header> </div> <nav class="header-nav fixed" aria-label="main" data-header-navigation> <ul class="header-nav__list"> <li> <div class="artdeco-dropdown"> <div class="artdeco-dropdown__trigger"> <a href="https://www.linkedin.com/blog/engineering/data" class="t-sans t-black--light t-semibold header-nav__toplink" target="_self"> Data </a> <button class="header-navigation__button" aria-label="Expand to show more links for Data" aria-expanded="false" data-trigger-dropdown-menu type="button"> <li-icon size="small" type="caret-filled-down-icon"></li-icon> </button> <div class="artdeco-dropdown__content header-nav__dropdown-offset"> <ul> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/graph-systems" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Graph System</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/economic-graph" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Economic Graph</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/skills-graph" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Skills Graph</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/knowledge" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Knowledge Graph</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/data-management" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Data Management</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/data-streaming-processing" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Data Streams/Processing</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/data-science" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Data Science</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/research" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Research</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/ab-testing-experimentation" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">A/B Testing/Experimentation</a> </div> </li> </ul> </div> </div> </div> </li> <li> <div class="artdeco-dropdown"> <div class="artdeco-dropdown__trigger"> <a href="https://www.linkedin.com/blog/engineering/artificial-intelligence" class="t-sans t-black--light t-semibold header-nav__toplink" target="_self"> AI </a> <button class="header-navigation__button" aria-label="Expand to show more links for AI" aria-expanded="false" data-trigger-dropdown-menu type="button"> <li-icon size="small" type="caret-filled-down-icon"></li-icon> </button> <div class="artdeco-dropdown__content header-nav__dropdown-offset"> <ul> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/machine-learning" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Machine Learning</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/natural-language-processing" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Natural Language Processing</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/generative-ai" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Generative AI</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/data-modeling" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Data Modeling</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/responsible-ai" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Responsible AI</a> </div> </li> </ul> </div> </div> </div> </li> <li> <div class="artdeco-dropdown"> <div class="artdeco-dropdown__trigger"> <a href="https://www.linkedin.com/blog/engineering/trust-and-safety" class="t-sans t-black--light t-semibold header-nav__toplink" target="_self"> Trust & Safety </a> <button class="header-navigation__button" aria-label="Expand to show more links for Trust &amp; Safety" aria-expanded="false" data-trigger-dropdown-menu type="button"> <li-icon size="small" type="caret-filled-down-icon"></li-icon> </button> <div class="artdeco-dropdown__content header-nav__dropdown-offset"> <ul> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/fairness" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Fairness</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/verification" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Verification </a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/security" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Security</a> </div> </li> </ul> </div> </div> </div> </li> <li> <div class="artdeco-dropdown"> <div class="artdeco-dropdown__trigger"> <a href="https://www.linkedin.com/blog/engineering/product-design" class="t-sans t-black--light t-semibold header-nav__toplink" target="_self"> Product Design </a> <button class="header-navigation__button" aria-label="Expand to show more links for Product Design" aria-expanded="false" data-trigger-dropdown-menu type="button"> <li-icon size="small" type="caret-filled-down-icon"></li-icon> </button> <div class="artdeco-dropdown__content header-nav__dropdown-offset"> <ul> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/marketing" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Marketing </a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/sales" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Sales</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/learning" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Learning</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/hiring" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Hiring</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/profile" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Profile</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/messaging-notifications" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Messaging/Notifications</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/feed" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Feed</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/profile" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Profile</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/groups" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Groups</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/accessibility" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Accessibility</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/member-customer-experience" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Member/Customer Experience</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/search" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Search</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/recommendations" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Recommendations</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/skills" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Skills</a> </div> </li> </ul> </div> </div> </div> </li> <li> <div class="artdeco-dropdown"> <div class="artdeco-dropdown__trigger"> <a href="https://www.linkedin.com/blog/engineering/infrastructure" class="t-sans t-black--light t-semibold header-nav__toplink" target="_self"> Infrastructure </a> <button class="header-navigation__button" aria-label="Expand to show more links for Infrastructure" aria-expanded="false" data-trigger-dropdown-menu type="button"> <li-icon size="small" type="caret-filled-down-icon"></li-icon> </button> <div class="artdeco-dropdown__content header-nav__dropdown-offset"> <ul> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/code" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Code</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/scalability" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Scalability</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/distributed-systems" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Distributed Systems</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/graph-systems" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Graph Systems</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/architecture" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Architecture</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/analytics" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Analytics</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/automation" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Automation</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/platform-platformization" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Platforms/Platformization</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/cloud-computing" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Cloud Computing</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/open-source" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Open Source</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/additional-innovations" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Additional Innovations</a> </div> </li> </ul> </div> </div> </div> </li> <li> <div class="artdeco-dropdown"> <div class="artdeco-dropdown__trigger"> <a href="https://www.linkedin.com/blog/engineering/talent" class="t-sans t-black--light t-semibold header-nav__toplink" target="_self"> Talent </a> <button class="header-navigation__button" aria-label="Expand to show more links for Talent" aria-expanded="false" data-trigger-dropdown-menu type="button"> <li-icon size="small" type="caret-filled-down-icon"></li-icon> </button> <div class="artdeco-dropdown__content header-nav__dropdown-offset"> <ul> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/culture" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Culture</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/developer-experience-productivity" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Developer Experience/Productivity</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/leadership" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Leadership</a> </div> </li> </ul> </div> </div> </div> </li> </ul> </nav> <nav class="custom-header__nav--mobile invisible" data-mobile-navigation="true" aria-label="main"> <ul class="custom-header__list--mobile"> <li class="custom-header__list-item--mobile" data-mobile-navigation-list-item="true"> <a href="https://www.linkedin.com/blog/engineering/data" class="t-16 t-sans t-black t-semibold header-nav__toplink" target="_self"> Data </a> <button class="header-navigation__button" aria-label="Expand to show more links for Data" aria-expanded="false" data-mobile-dropdown-trigger type="button"> <li-icon size="small" type="chevron-down-icon"></li-icon> </button> <ul class="custom-header__dropdownlink-list--mobile hidden" data-dropdownlinks-list="true"> <li> <a href="https://www.linkedin.com/blog/engineering/graph-systems" class="t-sans t-black t-normal header-nav__link" target="_self">Graph System</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/economic-graph" class="t-sans t-black t-normal header-nav__link" target="_self">Economic Graph</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/skills-graph" class="t-sans t-black t-normal header-nav__link" target="_self">Skills Graph</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/knowledge" class="t-sans t-black t-normal header-nav__link" target="_self">Knowledge Graph</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/data-management" class="t-sans t-black t-normal header-nav__link" target="_self">Data Management</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/data-streaming-processing" class="t-sans t-black t-normal header-nav__link" target="_self">Data Streams/Processing</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/data-science" class="t-sans t-black t-normal header-nav__link" target="_self">Data Science</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/research" class="t-sans t-black t-normal header-nav__link" target="_self">Research</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/ab-testing-experimentation" class="t-sans t-black t-normal header-nav__link" target="_self">A/B Testing/Experimentation</a> </li> </ul> </li> <li class="custom-header__list-item--mobile" data-mobile-navigation-list-item="true"> <a href="https://www.linkedin.com/blog/engineering/artificial-intelligence" class="t-16 t-sans t-black t-semibold header-nav__toplink" target="_self"> AI </a> <button class="header-navigation__button" aria-label="Expand to show more links for AI" aria-expanded="false" data-mobile-dropdown-trigger type="button"> <li-icon size="small" type="chevron-down-icon"></li-icon> </button> <ul class="custom-header__dropdownlink-list--mobile hidden" data-dropdownlinks-list="true"> <li> <a href="https://www.linkedin.com/blog/engineering/machine-learning" class="t-sans t-black t-normal header-nav__link" target="_self">Machine Learning</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/natural-language-processing" class="t-sans t-black t-normal header-nav__link" target="_self">Natural Language Processing</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/generative-ai" class="t-sans t-black t-normal header-nav__link" target="_self">Generative AI</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/data-modeling" class="t-sans t-black t-normal header-nav__link" target="_self">Data Modeling</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/responsible-ai" class="t-sans t-black t-normal header-nav__link" target="_self">Responsible AI</a> </li> </ul> </li> <li class="custom-header__list-item--mobile" data-mobile-navigation-list-item="true"> <a href="https://www.linkedin.com/blog/engineering/trust-and-safety" class="t-16 t-sans t-black t-semibold header-nav__toplink" target="_self"> Trust & Safety </a> <button class="header-navigation__button" aria-label="Expand to show more links for Trust &amp; Safety" aria-expanded="false" data-mobile-dropdown-trigger type="button"> <li-icon size="small" type="chevron-down-icon"></li-icon> </button> <ul class="custom-header__dropdownlink-list--mobile hidden" data-dropdownlinks-list="true"> <li> <a href="https://www.linkedin.com/blog/engineering/fairness" class="t-sans t-black t-normal header-nav__link" target="_self">Fairness</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/verification" class="t-sans t-black t-normal header-nav__link" target="_self">Verification </a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/security" class="t-sans t-black t-normal header-nav__link" target="_self">Security</a> </li> </ul> </li> <li class="custom-header__list-item--mobile" data-mobile-navigation-list-item="true"> <a href="https://www.linkedin.com/blog/engineering/product-design" class="t-16 t-sans t-black t-semibold header-nav__toplink" target="_self"> Product Design </a> <button class="header-navigation__button" aria-label="Expand to show more links for Product Design" aria-expanded="false" data-mobile-dropdown-trigger type="button"> <li-icon size="small" type="chevron-down-icon"></li-icon> </button> <ul class="custom-header__dropdownlink-list--mobile hidden" data-dropdownlinks-list="true"> <li> <a href="https://www.linkedin.com/blog/engineering/marketing" class="t-sans t-black t-normal header-nav__link" target="_self">Marketing </a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/sales" class="t-sans t-black t-normal header-nav__link" target="_self">Sales</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/learning" class="t-sans t-black t-normal header-nav__link" target="_self">Learning</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/hiring" class="t-sans t-black t-normal header-nav__link" target="_self">Hiring</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/profile" class="t-sans t-black t-normal header-nav__link" target="_self">Profile</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/messaging-notifications" class="t-sans t-black t-normal header-nav__link" target="_self">Messaging/Notifications</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/feed" class="t-sans t-black t-normal header-nav__link" target="_self">Feed</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/profile" class="t-sans t-black t-normal header-nav__link" target="_self">Profile</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/groups" class="t-sans t-black t-normal header-nav__link" target="_self">Groups</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/accessibility" class="t-sans t-black t-normal header-nav__link" target="_self">Accessibility</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/member-customer-experience" class="t-sans t-black t-normal header-nav__link" target="_self">Member/Customer Experience</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/search" class="t-sans t-black t-normal header-nav__link" target="_self">Search</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/recommendations" class="t-sans t-black t-normal header-nav__link" target="_self">Recommendations</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/skills" class="t-sans t-black t-normal header-nav__link" target="_self">Skills</a> </li> </ul> </li> <li class="custom-header__list-item--mobile" data-mobile-navigation-list-item="true"> <a href="https://www.linkedin.com/blog/engineering/infrastructure" class="t-16 t-sans t-black t-semibold header-nav__toplink" target="_self"> Infrastructure </a> <button class="header-navigation__button" aria-label="Expand to show more links for Infrastructure" aria-expanded="false" data-mobile-dropdown-trigger type="button"> <li-icon size="small" type="chevron-down-icon"></li-icon> </button> <ul class="custom-header__dropdownlink-list--mobile hidden" data-dropdownlinks-list="true"> <li> <a href="https://www.linkedin.com/blog/engineering/code" class="t-sans t-black t-normal header-nav__link" target="_self">Code</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/scalability" class="t-sans t-black t-normal header-nav__link" target="_self">Scalability</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/distributed-systems" class="t-sans t-black t-normal header-nav__link" target="_self">Distributed Systems</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/graph-systems" class="t-sans t-black t-normal header-nav__link" target="_self">Graph Systems</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/architecture" class="t-sans t-black t-normal header-nav__link" target="_self">Architecture</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/analytics" class="t-sans t-black t-normal header-nav__link" target="_self">Analytics</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/automation" class="t-sans t-black t-normal header-nav__link" target="_self">Automation</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/platform-platformization" class="t-sans t-black t-normal header-nav__link" target="_self">Platforms/Platformization</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/cloud-computing" class="t-sans t-black t-normal header-nav__link" target="_self">Cloud Computing</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/open-source" class="t-sans t-black t-normal header-nav__link" target="_self">Open Source</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/additional-innovations" class="t-sans t-black t-normal header-nav__link" target="_self">Additional Innovations</a> </li> </ul> </li> <li class="custom-header__list-item--mobile" data-mobile-navigation-list-item="true"> <a href="https://www.linkedin.com/blog/engineering/talent" class="t-16 t-sans t-black t-semibold header-nav__toplink" target="_self"> Talent </a> <button class="header-navigation__button" aria-label="Expand to show more links for Talent" aria-expanded="false" data-mobile-dropdown-trigger type="button"> <li-icon size="small" type="chevron-down-icon"></li-icon> </button> <ul class="custom-header__dropdownlink-list--mobile hidden" data-dropdownlinks-list="true"> <li> <a href="https://www.linkedin.com/blog/engineering/culture" class="t-sans t-black t-normal header-nav__link" target="_self">Culture</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/developer-experience-productivity" class="t-sans t-black t-normal header-nav__link" target="_self">Developer Experience/Productivity</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/leadership" class="t-sans t-black t-normal header-nav__link" target="_self">Leadership</a> </li> </ul> </li> </ul> <div class="custom-header__button-share-container--mobile">  </div> </nav> </div> <main role="main"> <div id="lithograph-app"> <section id="component-container" data-attach-collapse-behavior="true"> <div id class="component component-articleHeadline" data-component-type="articleHeadline"> <div class="article-headline-container">  <h1 class="article-headline t-40" data-article-title="Top Five Lessons from Running A/B Tests on the World's Largest Professional Network">Top Five Lessons from Running A/B Tests on the World's Largest Professional Network</h1> <div class="article-meta t-14 article-meta__hairline"> <div class="article-meta__byline-container"> <p class="t-14 t-black--light" data-published-date="2015-11-10"> November 10, 2015 </p> </div> <div class="social-share"> <ul class="social-share__list share-list" title="Share via" data-share-list> <li class="share-list__item share-item"> <a href="#" class="share-item__link medium-round-muted-tertiary-button" role="button" data-tracking-control-name="_linkedin_share" data-share-type="linkedin" aria-label="Share on LinkedIn, New window will open"> <li-icon class="share-item__icon" type="linkedin-icon"></li-icon> </a> </li> <li class="share-list__item share-item"> <a href="#" class="share-item__link medium-round-muted-tertiary-button" role="button" data-tracking-control-name="_facebook_share" data-share-type="facebook" aria-label="Share on Facebook, New window will open"> <li-icon class="share-item__icon" type="facebook-icon"></li-icon> </a> </li> <li class="share-list__item share-item"> <a href="#" class="share-item__link medium-round-muted-tertiary-button" role="button" data-tracking-control-name="_twitter_share" data-share-type="twitter" aria-label="Share on Twitter, New window will open"> <li-icon class="share-item__icon" type="twitter-icon"></li-icon> </a> </li> </ul> </div> </div> </div> </div> <div id class="component component-migratedContent" data-component-type="migratedContent"> <div class="rich-text"> <p>We run thousands of experiments each year at LinkedIn. Thousands of features, big and small, have been launched, refined, or discarded through <a href="https://engineering.linkedin.com/ab-testing/xlnt-platform-driving-ab-testing-linkedin" target="_blank" data-tracking-control-name="_anchor_link">our robust A/B testing platform</a>. As a result of these experiments, we have learned a tremendous amount about how to make our products better and our members happier. At the same time, we have accumulated knowledge on how to run better experiments and how to leverage experimentation for better decision making. Here are the top five lessons on my ever-growing list.</p> <p><b>#1. Measure one change at a time.</b></p> <p>This is not to say that you can only test one thing at a time, but that you have to design your experiment properly so that you are able to measure one change at a time. At LinkedIn, a product launch usually involves multiple features/components. One big upgrade to LinkedIn Search in 2013 introduced unified search across different product categories. With this functionality, the search box is smart enough to figure out query intent without explicit input on categories such as “People,” or “Jobs,” or “Companies.”</p> <p>However, that was not all. Almost every single component on the search landing-page was touched, from the left rail navigation to snippets and action buttons. The first experiment was run with all changes lumped together. To our surprise, many key metrics tanked. It was a lengthy process to bring back one feature at a time in order to figure out the true culprit. In the end, we realized that several small changes, not the unified search itself, were responsible for bringing down clicks and revenue. After restoring these features, unified search was shown to be positive to user experience and deployed to everyone.  </p> <p><b>#2. Decide on triggered users, but report on all users.</b></p> <p>It is very common that an experiment only impacts a small fraction of your user base. For example, we want to automatically help people fill in their patents on their LinkedIn profiles, but not every member has a patent. So the experiment would only be affecting those ~5% of members who have filed patents. To measure how much benefit this is bringing to our members, we have to focus on this small subsegment, the “triggered” users. Otherwise, the signal from that 5% of users would be lost in the 95% noise. However, once we determined that patents are a beneficial feature, we needed to have a “realistic” estimate of the overall impact. How is LinkedIn’s bottom line going to change once this feature is rolled out universally? Having such a “site-wide” impact not only makes it possible to compare impacts across experiments, but also easy to quantify ROI.</p> <p><b>#3. The experimental group should not be influenced by the experiment outcomes.</b></p> <p>The fundamental assumption of A/B testing is that the difference between the A and B groups is only caused by the treatment we impose. It may be obvious that we need to make sure the users in A and B are similar enough to begin with. The standard approach to check for any pre-existing differences is to run an A/A test before the actual A/B test, where both groups of users receive identical treatments. However, it is equally important to make sure the user groups stay “similar” during the experiment especially in the online world because the experimental population is usually “dynamic”. As an example, we tested a new feature where members received a small banner on their LinkedIn profile page to encourage them to explore our new homepage. Only users who had not visited the homepage recently were eligible to be in the experiment, and the eligibility was dynamically updated after a user visited the homepage. Because the banner brought more users in the treatment group to visit the homepage, more treatment users became ineligible over time. Because these “additionally” removed users tend to be more active than the rest, we artificially created a difference between users in A and B as the test continued. In general, if the experimental population is directly influenced by the experiment outcomes, we are likely to see a bias. Such bias could void the experiment results because it usually overwhelms any real signal resulting from the treatment itself.</p> <p><b>#4. Avoid coupling a marketing campaign with an A/B test.</b></p> <p>We have recently revamped the <a href="https://www.linkedin.com/wvmx/profile" target="_blank" data-tracking-control-name="_anchor_link">Who Viewed My Profile page</a>. The product team wanted to measure through an A/B test if the changes are indeed better, and if so, by how much. The marketing team wanted to create buzz around the new page with an email campaign. This is a very common scenario, but how can the A/B test and the email campaign coexist? Clearly, we can only send campaign emails to the treatment group, since there is nothing new for members in control. However, such a campaign would contaminate the online A/B test because it encourages more members from the treatment to visit. These additional users tend to be less engaged, therefore we are likely to see an artificial drop in key metrics. It is best to measure the A/B test first before launching the campaign.</p> <p><b>#5. Use a simple rule of thumb to address multiple testing problems.</b></p> <p>Multiple testing problems are extremely prevalent in online A/B testing. The symptom is that irrelevant metrics appear to be statistically significant. The root cause is usually because too many metrics are examined simultaneously (keep in mind that we compute over 1000 metrics for each experiment). Even though we have tried to educate people on the topic of multiple testing, many are still clueless about what they should do when a metric is unexpectedly significant. Should they trust it or treat it as noise? Instead, we have found it very effective to introduce a simple rule of thumb: Use the standard 0.05 <a href="https://en.wikipedia.org/wiki/P-value" target="_blank" data-tracking-control-name="_anchor_link">p-value</a> cutoff for metrics that are expected to be impacted, but use a smaller cutoff, say 0.001, for metrics that are not. The rule-of-thumb is based on an interesting Bayesian interpretation. It boils down to how much we believe a metric will be impacted before we even run the experiment. In particular, if using 0.05 reflects a prior probability of 50%, then using 0.001 means a much weaker belief - at about 2%.</p> <p>These are only a few best practices for experimentation, but they've proven crucial for product development at LinkedIn. As I’ve said before, A/B testing and making data driven decisions through experimentation is an <a href="https://engineering.linkedin.com/ab-testing/why-experimentation-so-important-linkedin" target="_blank" data-tracking-control-name="_anchor_link">extremely important part of the culture</a> at LinkedIn. It guides how and why we build products for our users by giving us crucial data on how they actually use our services. By following these five lessons, developers across all companies and industries can not only make more informed decisions about their products, but also create a better experience for the people using them.</p> </div> </div> <div id class="component component-postList" data-component-type="postList"> <section id="postList0FocusPoint" class="component__content-container list-layout" data-post-list> <section class="simple-topic-list"> <p class="t-14 t-bold">Topics: <a class="simple-topic-list__topic" href="https://www.linkedin.com/blog/engineering/culture"> Culture </a> <a class="simple-topic-list__topic" href="https://www.linkedin.com/blog/engineering/ab-testing-experimentation"> A/B Testing/Experimentation </a> </p> </section> <hr class="artdeco-divider"> <section class="list-layout__headline-container"> <p class="list-layout__headline t-32 t-sans">Related articles</p>  </section> <ul class="list-layout__posts"> <li class="post-list__item list-post"> <div class="list-post__content-container"> <p class="list-post__content-container__topic"> <a class="t-14 t-bold" href="https://www.linkedin.com/blog/engineering/leadership"> Leadership </a> </p> <div class="list-post__content-container__title"> <a class="list-post__link t-20 t-black" href="https://www.linkedin.com/blog/engineering/leadership/why-representation-matters-in-shaping-the-future-of-engineering"> Why Representation Matters in Shaping The Future of Engineering </a> </div> <div class="list-post__content-container__byline"> <p class="list-post__content-container__author t-14 t-bold t-black"> Sabry Tozin </p> <p class="list-post__content-container__date t-14 t-black--light"> Feb 23, 2024 </p> </div> </div> <div class="list-post__image-container"> <figure class="post__image-figure"> <img class="post__image" alt data-delayed-url="https://media.licdn.com/dms/image/v2/D4D08AQFq-uxWo61UPQ/croft-frontend-shrinkToFit1024/croft-frontend-shrinkToFit1024/0/1708119705134?e=2147483647&v=beta&t=28QTXbM-BYHuCEfK4EP8Pq9dfgSX3--p8C8R3EU1zAI" data-ghost-url="https://static.licdn.com/aero-v1/sc/h/8pouyb1ly746xtru0x7p2axm0" sizes="(max-width: 480px) 480px, (max-width: 767px) 767px, (max-width: 1024px) 1024px, 480px"> </figure> </div> </li> <li class="post-list__item list-post"> <div class="list-post__content-container"> <p class="list-post__content-container__topic"> <a class="t-14 t-bold" href="https://www.linkedin.com/blog/engineering/culture"> Culture </a> </p> <div class="list-post__content-container__title"> <a class="list-post__link t-20 t-black" href="https://www.linkedin.com/blog/engineering/culture/announcing-our-linkedin-cornell-2023-grant-recipients"> Announcing Our LinkedIn-Cornell 2023 Grant Recipients </a> </div> <div class="list-post__content-container__byline"> <p class="list-post__content-container__author t-14 t-bold t-black"> Souvik Ghosh </p> <p class="list-post__content-container__date t-14 t-black--light"> Aug 2, 2023 </p> </div> </div> <div class="list-post__image-container"> <figure class="post__image-figure"> <img class="post__image" alt data-delayed-url="https://media.licdn.com/dms/image/v2/D4D08AQHE431HIyorRA/croft-frontend-shrinkToFit1024/croft-frontend-shrinkToFit1024/0/1704752231739?e=2147483647&v=beta&t=SRn5B7e7j7AqCoHv7vgGPElI0G76jeiO8A3zrSyaj3Y" data-ghost-url="https://static.licdn.com/aero-v1/sc/h/8pouyb1ly746xtru0x7p2axm0" sizes="(max-width: 480px) 480px, (max-width: 767px) 767px, (max-width: 1024px) 1024px, 480px"> </figure> </div> </li> <li class="post-list__item list-post"> <div class="list-post__content-container"> <p class="list-post__content-container__topic"> <a class="t-14 t-bold" href="https://www.linkedin.com/blog/engineering/data-science"> Data Science </a> </p> <div class="list-post__content-container__title"> <a class="list-post__link t-20 t-black" href="https://www.linkedin.com/blog/engineering/data-science/ocelot-scaling-observational-causal-inference-at-linkedin"> Ocelot: Scaling observational causal inference at LinkedIn </a> </div> <div class="list-post__content-container__byline">  <p class="list-post__content-container__date t-14 t-black--light"> Dec 13, 2022 </p> </div> </div> <div class="list-post__image-container"> <figure class="post__image-figure"> <img class="post__image" alt data-delayed-url="https://media.licdn.com/dms/image/v2/D4D08AQF6eyJc0YTcZA/croft-frontend-shrinkToFit1024/croft-frontend-shrinkToFit1024/0/1700687404243?e=2147483647&v=beta&t=jWvodl4T00EWDaqLdb7e7t9OsZX1UP9JP6DqfuxRaM8" data-ghost-url="https://static.licdn.com/aero-v1/sc/h/8pouyb1ly746xtru0x7p2axm0" sizes="(max-width: 480px) 480px, (max-width: 767px) 767px, (max-width: 1024px) 1024px, 480px"> </figure> </div> </li> </ul>  </section> </div>  </section> </div> <script src="https://static.licdn.com/aero-v1/sc/h/8hfbuq1ftcvnnx4dd5067pi0t" async></script> <script src="https://static.licdn.com/aero-v1/sc/h/dev5px9xu71l67oqmrrbimi6b" async></script> <script src="https://static.licdn.com/aero-v1/sc/h/3e9epkft4bs8unu9ky8gss8ya" async></script> </main> <footer class="li-footer bg-transparent w-full "> <ul class="li-footer__list flex flex-wrap flex-row items-start justify-start w-full h-auto min-h-[50px] my-[0px] mx-auto py-3 px-2 papabear:w-[1128px] papabear:p-0"> <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <span class="sr-only">LinkedIn</span> <icon class="li-footer__copy-logo text-color-logo-brand-alt inline-block self-center h-[14px] w-[56px] mr-1" data-delayed-url="https://static.licdn.com/aero-v1/sc/h/e12h2cd8ac580qen9qdd0qks8"></icon> <span class="li-footer__copy-text flex items-center">© 2024</span> </li> <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <a class="li-footer__item-link flex items-center font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover focus:text-color-link-focus" href="https://about.linkedin.com?trk=content_footer-about" data-tracking-control-name="content_footer-about" data-tracking-will-navigate> About </a> </li> <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <a class="li-footer__item-link flex items-center font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover focus:text-color-link-focus" href="https://www.linkedin.com/accessibility?trk=content_footer-accessibility" data-tracking-control-name="content_footer-accessibility" data-tracking-will-navigate> Accessibility </a> </li> <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <a class="li-footer__item-link flex items-center font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover focus:text-color-link-focus" href="https://www.linkedin.com/legal/user-agreement?trk=content_footer-user-agreement" data-tracking-control-name="content_footer-user-agreement" data-tracking-will-navigate> User Agreement </a> </li> <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <a class="li-footer__item-link flex items-center font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover focus:text-color-link-focus" href="https://www.linkedin.com/legal/privacy-policy?trk=content_footer-privacy-policy" data-tracking-control-name="content_footer-privacy-policy" data-tracking-will-navigate> Privacy Policy </a> </li>  <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <a class="li-footer__item-link flex items-center font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover focus:text-color-link-focus" href="https://www.linkedin.com/legal/cookie-policy?trk=content_footer-cookie-policy" data-tracking-control-name="content_footer-cookie-policy" data-tracking-will-navigate> Cookie Policy </a> </li> <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <a class="li-footer__item-link flex items-center font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover focus:text-color-link-focus" href="https://www.linkedin.com/legal/copyright-policy?trk=content_footer-copyright-policy" data-tracking-control-name="content_footer-copyright-policy" data-tracking-will-navigate> Copyright Policy </a> </li> <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <a class="li-footer__item-link flex items-center font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover focus:text-color-link-focus" href="https://brand.linkedin.com/policies?trk=content_footer-brand-policy" data-tracking-control-name="content_footer-brand-policy" data-tracking-will-navigate> Brand Policy </a> </li> <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <a class="li-footer__item-link flex items-center font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover focus:text-color-link-focus" href="https://www.linkedin.com/psettings/guest-controls?trk=content_footer-guest-controls" data-tracking-control-name="content_footer-guest-controls" data-tracking-will-navigate> Guest Controls </a> </li> <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <a class="li-footer__item-link flex items-center font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover focus:text-color-link-focus" href="https://www.linkedin.com/help/linkedin/answer/34593?trk=content_footer-community-guide" data-tracking-control-name="content_footer-community-guide" data-tracking-will-navigate> Community Guidelines </a> </li>  <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <div class="collapsible-dropdown collapsible-dropdown--footer collapsible-dropdown--up flex items-center relative hyphens-auto language-selector z-2">  <ul class="collapsible-dropdown__list hidden container-raised absolute w-auto overflow-y-auto flex-col items-stretch z-1 bottom-[100%] top-auto" role="menu" tabindex="-1"> <li class="language-selector__item" role="presentation">  <button aria-label="العربية (Arabic)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-ar_AE" data-locale="ar_AE" role="menuitem" lang="ar_AE"> العربية (Arabic) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="বাংলা (Bangla)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-bn_IN" data-locale="bn_IN" role="menuitem" lang="bn_IN"> বাংলা (Bangla) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Čeština (Czech)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-cs_CZ" data-locale="cs_CZ" role="menuitem" lang="cs_CZ"> Čeština (Czech) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Dansk (Danish)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-da_DK" data-locale="da_DK" role="menuitem" lang="da_DK"> Dansk (Danish) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Deutsch (German)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-de_DE" data-locale="de_DE" role="menuitem" lang="de_DE"> Deutsch (German) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Ελληνικά (Greek)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-el_GR" data-locale="el_GR" role="menuitem" lang="el_GR"> Ελληνικά (Greek) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="English (English) selected" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link--selected" data-tracking-control-name="language-selector-en_US" data-locale="en_US" role="menuitem" lang="en_US"> <strong>English (English)</strong> </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Español (Spanish)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-es_ES" data-locale="es_ES" role="menuitem" lang="es_ES"> Español (Spanish) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="فارسی (Persian)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-fa_IR" data-locale="fa_IR" role="menuitem" lang="fa_IR"> فارسی (Persian) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Suomi (Finnish)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-fi_FI" data-locale="fi_FI" role="menuitem" lang="fi_FI"> Suomi (Finnish) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Français (French)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-fr_FR" data-locale="fr_FR" role="menuitem" lang="fr_FR"> Français (French) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="हिंदी (Hindi)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-hi_IN" data-locale="hi_IN" role="menuitem" lang="hi_IN"> हिंदी (Hindi) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Magyar (Hungarian)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-hu_HU" data-locale="hu_HU" role="menuitem" lang="hu_HU"> Magyar (Hungarian) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Bahasa Indonesia (Indonesian)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-in_ID" data-locale="in_ID" role="menuitem" lang="in_ID"> Bahasa Indonesia (Indonesian) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Italiano (Italian)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-it_IT" data-locale="it_IT" role="menuitem" lang="it_IT"> Italiano (Italian) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="עברית (Hebrew)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-iw_IL" data-locale="iw_IL" role="menuitem" lang="iw_IL"> עברית (Hebrew) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="日本語 (Japanese)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-ja_JP" data-locale="ja_JP" role="menuitem" lang="ja_JP"> 日本語 (Japanese) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="한국어 (Korean)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-ko_KR" data-locale="ko_KR" role="menuitem" lang="ko_KR"> 한국어 (Korean) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="मराठी (Marathi)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-mr_IN" data-locale="mr_IN" role="menuitem" lang="mr_IN"> मराठी (Marathi) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Bahasa Malaysia (Malay)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-ms_MY" data-locale="ms_MY" role="menuitem" lang="ms_MY"> Bahasa Malaysia (Malay) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Nederlands (Dutch)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-nl_NL" data-locale="nl_NL" role="menuitem" lang="nl_NL"> Nederlands (Dutch) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Norsk (Norwegian)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-no_NO" data-locale="no_NO" role="menuitem" lang="no_NO"> Norsk (Norwegian) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="ਪੰਜਾਬੀ (Punjabi)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-pa_IN" data-locale="pa_IN" role="menuitem" lang="pa_IN"> ਪੰਜਾਬੀ (Punjabi) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Polski (Polish)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-pl_PL" data-locale="pl_PL" role="menuitem" lang="pl_PL"> Polski (Polish) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Português (Portuguese)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-pt_BR" data-locale="pt_BR" role="menuitem" lang="pt_BR"> Português (Portuguese) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Română (Romanian)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-ro_RO" data-locale="ro_RO" role="menuitem" lang="ro_RO"> Română (Romanian) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Русский (Russian)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-ru_RU" data-locale="ru_RU" role="menuitem" lang="ru_RU"> Русский (Russian) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Svenska (Swedish)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-sv_SE" data-locale="sv_SE" role="menuitem" lang="sv_SE"> Svenska (Swedish) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="తెలుగు (Telugu)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-te_IN" data-locale="te_IN" role="menuitem" lang="te_IN"> తెలుగు (Telugu) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="ภาษาไทย (Thai)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-th_TH" data-locale="th_TH" role="menuitem" lang="th_TH"> ภาษาไทย (Thai) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Tagalog (Tagalog)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-tl_PH" data-locale="tl_PH" role="menuitem" lang="tl_PH"> Tagalog (Tagalog) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Türkçe (Turkish)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-tr_TR" data-locale="tr_TR" role="menuitem" lang="tr_TR"> Türkçe (Turkish) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Українська (Ukrainian)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-uk_UA" data-locale="uk_UA" role="menuitem" lang="uk_UA"> Українська (Ukrainian) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Tiếng Việt (Vietnamese)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-vi_VN" data-locale="vi_VN" role="menuitem" lang="vi_VN"> Tiếng Việt (Vietnamese) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="简体中文 (Chinese (Simplified))" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-zh_CN" data-locale="zh_CN" role="menuitem" lang="zh_CN"> 简体中文 (Chinese (Simplified)) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="正體中文 (Chinese (Traditional))" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-zh_TW" data-locale="zh_TW" role="menuitem" lang="zh_TW"> 正體中文 (Chinese (Traditional)) </button> </li>  </ul> <button class="language-selector__button select-none relative pr-2 font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover hover:cursor-pointer focus:text-color-link-focus focus:outline-dotted focus:outline-1" aria-expanded="false" data-tracking-control-name="footer-lang-dropdown_trigger"> <span class="language-selector__label-text mr-0.5 break-words"> Language </span> <icon class="language-selector__label-chevron w-2 h-2 absolute top-0 right-0" data-delayed-url="https://static.licdn.com/aero-v1/sc/h/cyolgscd0imw2ldqppkrb84vo"></icon> </button> </div> </li> </ul>  </footer> <script src="https://static.licdn.com/aero-v1/sc/h/eh08muqvrde4h3hc6koyij5ti" async></script>  </body> </html>

CINXE.COM

Top Five Lessons from Running A/B Tests on the World's Largest Professional Network