Building inclusive products through A/B testing

<!DOCTYPE html> <html lang="en"> <head> <meta name="pageKey" content="d_lithograph_content">  <meta name="locale" content="en_US"> <meta id="config" data-app-id="lithograph" data-app-version="0.0.1098" data-call-tree-id="AAYubmeqYr3K4xRc1lnvhA==" data-multiproduct-name="lithograph-publish-frontend" data-service-name="lithograph-publish-frontend" data-browser-id="ecb42799-703c-4cf0-8e38-fd838772c890" data-page-instance="urn:li:page:d_lithograph_content;Jdayy+hST+KF3DN97+K+UQ==" data-disable-jsbeacon-pagekey-suffix="false"> <link rel="canonical" href="https://www.linkedin.com/blog/engineering/ab-testing-experimentation/building-inclusive-products-through-a-b-testing">     <link rel="manifest" href="/homepage-guest/manifest.json" crossorigin="use-credentials"> <link rel="icon" href="https://static.licdn.com/aero-v1/sc/h/al2o9zrvru7aqj8e1x2rzsrca"> <script> function getDfd() {let yFn,nFn;const p=new Promise(function(y, n){yFn=y;nFn=n;});p.resolve=yFn;p.reject=nFn;return p;} window.lazyloader = getDfd(); window.tracking = getDfd(); window.impressionTracking = getDfd(); window.ingraphTracking = getDfd(); window.appDetection = getDfd(); window.pemTracking = getDfd(); </script>  <meta name="viewport" content="width=device-width, initial-scale=1.0"> <title>Building inclusive products through A/B testing</title>  <meta property="og:type" content="website"> <meta property="og:title" content="Building inclusive products through A/B testing">  <meta property="og:url" content="https://www.linkedin.com/blog/engineering/ab-testing-experimentation/building-inclusive-products-through-a-b-testing"> <meta property="og:image" content="https://media.licdn.com/dms/image/v2/D4D08AQGEZDDl3lq5NQ/croft-frontend-shrinkToFit1024/croft-frontend-shrinkToFit1024/0/1700688417416?e=2147483647&v=beta&t=v4wqxQcywtR8MtJrtChL36rdKiKrUbw7bPsox98DsUE"> <meta name="twitter:title" content="Building inclusive products through A/B testing"> <meta name="twitter:site" content="@linkedin">   <meta name="robots" content="noarchive"> <meta name="linkedin:pageTag" content="/blog/engineering/ab-testing-experimentation/building-inclusive-products-through-a-b-testing"> <meta name="litmsProfileName" content="lithograph">  <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="asset-url" id="artdeco/static/images/icons.svg" content="https://static.licdn.com/aero-v1/sc/h/6sz1e821vgereyxaakx87tt4z "> <link rel="stylesheet" href="https://static.licdn.com/aero-v1/sc/h/crjcj3vgj0tgpae2eu4u9y7eb">   </head> <body dir="ltr">  <a href="#lithograph-app" class="skip-link btn-md btn-primary absolute z-11 -top-[100vh] focus:top-0"> Skip to main content </a>  <div class="page-header__wrapper page-header__wrapper--margin" data-custom-nav-wrapper="true"> <div class="page-header__container"> <header id="page-header" class="page-header page-header--hide-dropshadow"> <div class="page-header__content-container"> <div class="header__logo"> <a href="https://www.linkedin.com/blog/engineering" class="custom-nav__logo-link"> <img class="header__linkedin-logo header__full-logo" src="https://static.licdn.com/aero-v1/sc/h/6zm111mce7vohqze950ilreo4" alt="LinkedIn Logo"> <img class="header__linkedin-logo header__in-bug" src="https://static.licdn.com/aero-v1/sc/h/5g0hjlcng3j5pgn50n2et1ca2" alt="LinkedIn Logo"> <span class="t-17 t-black t-bold default-header-title"> Engineering Blog </span> </a> </div>   <button class="stripped header__expand-nav-button" data-mobile-header-navigation-button data-header-expand-navigation-button aria-expanded="false"> <li-icon class="header__expand-nav-icon" a11y-text="Open navigation" type="hamburger-icon"></li-icon> </button> <button class="stripped header__expand-nav-button hidden" data-mobile-header-navigation-button data-header-collapse-navigation-button aria-expanded="false"> <li-icon class="header__expand-nav-icon" a11y-text="Close navigation" type="cancel-icon"></li-icon> </button> </div> </header> </div> <nav class="header-nav fixed" aria-label="main" data-header-navigation> <ul class="header-nav__list"> <li> <div class="artdeco-dropdown"> <div class="artdeco-dropdown__trigger"> <a href="https://www.linkedin.com/blog/engineering/data" class="t-sans t-black--light t-semibold header-nav__toplink" target="_self"> Data </a> <button class="header-navigation__button" aria-label="Expand to show more links for Data" aria-expanded="false" data-trigger-dropdown-menu type="button"> <li-icon size="small" type="caret-filled-down-icon"></li-icon> </button> <div class="artdeco-dropdown__content header-nav__dropdown-offset"> <ul> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/graph-systems" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Graph System</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/economic-graph" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Economic Graph</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/skills-graph" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Skills Graph</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/knowledge" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Knowledge Graph</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/data-management" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Data Management</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/data-streaming-processing" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Data Streams/Processing</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/data-science" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Data Science</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/research" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Research</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/ab-testing-experimentation" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">A/B Testing/Experimentation</a> </div> </li> </ul> </div> </div> </div> </li> <li> <div class="artdeco-dropdown"> <div class="artdeco-dropdown__trigger"> <a href="https://www.linkedin.com/blog/engineering/artificial-intelligence" class="t-sans t-black--light t-semibold header-nav__toplink" target="_self"> AI </a> <button class="header-navigation__button" aria-label="Expand to show more links for AI" aria-expanded="false" data-trigger-dropdown-menu type="button"> <li-icon size="small" type="caret-filled-down-icon"></li-icon> </button> <div class="artdeco-dropdown__content header-nav__dropdown-offset"> <ul> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/machine-learning" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Machine Learning</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/natural-language-processing" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Natural Language Processing</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/generative-ai" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Generative AI</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/data-modeling" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Data Modeling</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/responsible-ai" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Responsible AI</a> </div> </li> </ul> </div> </div> </div> </li> <li> <div class="artdeco-dropdown"> <div class="artdeco-dropdown__trigger"> <a href="https://www.linkedin.com/blog/engineering/trust-and-safety" class="t-sans t-black--light t-semibold header-nav__toplink" target="_self"> Trust & Safety </a> <button class="header-navigation__button" aria-label="Expand to show more links for Trust &amp; Safety" aria-expanded="false" data-trigger-dropdown-menu type="button"> <li-icon size="small" type="caret-filled-down-icon"></li-icon> </button> <div class="artdeco-dropdown__content header-nav__dropdown-offset"> <ul> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/fairness" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Fairness</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/verification" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Verification </a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/security" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Security</a> </div> </li> </ul> </div> </div> </div> </li> <li> <div class="artdeco-dropdown"> <div class="artdeco-dropdown__trigger"> <a href="https://www.linkedin.com/blog/engineering/product-design" class="t-sans t-black--light t-semibold header-nav__toplink" target="_self"> Product Design </a> <button class="header-navigation__button" aria-label="Expand to show more links for Product Design" aria-expanded="false" data-trigger-dropdown-menu type="button"> <li-icon size="small" type="caret-filled-down-icon"></li-icon> </button> <div class="artdeco-dropdown__content header-nav__dropdown-offset"> <ul> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/marketing" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Marketing </a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/sales" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Sales</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/learning" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Learning</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/hiring" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Hiring</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/profile" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Profile</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/messaging-notifications" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Messaging/Notifications</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/feed" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Feed</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/profile" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Profile</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/groups" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Groups</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/accessibility" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Accessibility</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/member-customer-experience" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Member/Customer Experience</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/search" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Search</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/recommendations" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Recommendations</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/skills" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Skills</a> </div> </li> </ul> </div> </div> </div> </li> <li> <div class="artdeco-dropdown"> <div class="artdeco-dropdown__trigger"> <a href="https://www.linkedin.com/blog/engineering/infrastructure" class="t-sans t-black--light t-semibold header-nav__toplink" target="_self"> Infrastructure </a> <button class="header-navigation__button" aria-label="Expand to show more links for Infrastructure" aria-expanded="false" data-trigger-dropdown-menu type="button"> <li-icon size="small" type="caret-filled-down-icon"></li-icon> </button> <div class="artdeco-dropdown__content header-nav__dropdown-offset"> <ul> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/code" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Code</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/scalability" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Scalability</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/distributed-systems" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Distributed Systems</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/graph-systems" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Graph Systems</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/architecture" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Architecture</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/analytics" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Analytics</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/automation" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Automation</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/platform-platformization" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Platforms/Platformization</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/cloud-computing" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Cloud Computing</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/open-source" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Open Source</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/additional-innovations" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Additional Innovations</a> </div> </li> </ul> </div> </div> </div> </li> <li> <div class="artdeco-dropdown"> <div class="artdeco-dropdown__trigger"> <a href="https://www.linkedin.com/blog/engineering/talent" class="t-sans t-black--light t-semibold header-nav__toplink" target="_self"> Talent </a> <button class="header-navigation__button" aria-label="Expand to show more links for Talent" aria-expanded="false" data-trigger-dropdown-menu type="button"> <li-icon size="small" type="caret-filled-down-icon"></li-icon> </button> <div class="artdeco-dropdown__content header-nav__dropdown-offset"> <ul> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/culture" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Culture</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/developer-experience-productivity" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Developer Experience/Productivity</a> </div> </li> <li> <div class="artdeco-dropdown__item"> <a href="https://www.linkedin.com/blog/engineering/leadership" class="t-14 t-sans t-black t-bold header-nav__link" target="_self">Leadership</a> </div> </li> </ul> </div> </div> </div> </li> </ul> </nav> <nav class="custom-header__nav--mobile invisible" data-mobile-navigation="true" aria-label="main"> <ul class="custom-header__list--mobile"> <li class="custom-header__list-item--mobile" data-mobile-navigation-list-item="true"> <a href="https://www.linkedin.com/blog/engineering/data" class="t-16 t-sans t-black t-semibold header-nav__toplink" target="_self"> Data </a> <button class="header-navigation__button" aria-label="Expand to show more links for Data" aria-expanded="false" data-mobile-dropdown-trigger type="button"> <li-icon size="small" type="chevron-down-icon"></li-icon> </button> <ul class="custom-header__dropdownlink-list--mobile hidden" data-dropdownlinks-list="true"> <li> <a href="https://www.linkedin.com/blog/engineering/graph-systems" class="t-sans t-black t-normal header-nav__link" target="_self">Graph System</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/economic-graph" class="t-sans t-black t-normal header-nav__link" target="_self">Economic Graph</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/skills-graph" class="t-sans t-black t-normal header-nav__link" target="_self">Skills Graph</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/knowledge" class="t-sans t-black t-normal header-nav__link" target="_self">Knowledge Graph</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/data-management" class="t-sans t-black t-normal header-nav__link" target="_self">Data Management</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/data-streaming-processing" class="t-sans t-black t-normal header-nav__link" target="_self">Data Streams/Processing</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/data-science" class="t-sans t-black t-normal header-nav__link" target="_self">Data Science</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/research" class="t-sans t-black t-normal header-nav__link" target="_self">Research</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/ab-testing-experimentation" class="t-sans t-black t-normal header-nav__link" target="_self">A/B Testing/Experimentation</a> </li> </ul> </li> <li class="custom-header__list-item--mobile" data-mobile-navigation-list-item="true"> <a href="https://www.linkedin.com/blog/engineering/artificial-intelligence" class="t-16 t-sans t-black t-semibold header-nav__toplink" target="_self"> AI </a> <button class="header-navigation__button" aria-label="Expand to show more links for AI" aria-expanded="false" data-mobile-dropdown-trigger type="button"> <li-icon size="small" type="chevron-down-icon"></li-icon> </button> <ul class="custom-header__dropdownlink-list--mobile hidden" data-dropdownlinks-list="true"> <li> <a href="https://www.linkedin.com/blog/engineering/machine-learning" class="t-sans t-black t-normal header-nav__link" target="_self">Machine Learning</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/natural-language-processing" class="t-sans t-black t-normal header-nav__link" target="_self">Natural Language Processing</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/generative-ai" class="t-sans t-black t-normal header-nav__link" target="_self">Generative AI</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/data-modeling" class="t-sans t-black t-normal header-nav__link" target="_self">Data Modeling</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/responsible-ai" class="t-sans t-black t-normal header-nav__link" target="_self">Responsible AI</a> </li> </ul> </li> <li class="custom-header__list-item--mobile" data-mobile-navigation-list-item="true"> <a href="https://www.linkedin.com/blog/engineering/trust-and-safety" class="t-16 t-sans t-black t-semibold header-nav__toplink" target="_self"> Trust & Safety </a> <button class="header-navigation__button" aria-label="Expand to show more links for Trust &amp; Safety" aria-expanded="false" data-mobile-dropdown-trigger type="button"> <li-icon size="small" type="chevron-down-icon"></li-icon> </button> <ul class="custom-header__dropdownlink-list--mobile hidden" data-dropdownlinks-list="true"> <li> <a href="https://www.linkedin.com/blog/engineering/fairness" class="t-sans t-black t-normal header-nav__link" target="_self">Fairness</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/verification" class="t-sans t-black t-normal header-nav__link" target="_self">Verification </a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/security" class="t-sans t-black t-normal header-nav__link" target="_self">Security</a> </li> </ul> </li> <li class="custom-header__list-item--mobile" data-mobile-navigation-list-item="true"> <a href="https://www.linkedin.com/blog/engineering/product-design" class="t-16 t-sans t-black t-semibold header-nav__toplink" target="_self"> Product Design </a> <button class="header-navigation__button" aria-label="Expand to show more links for Product Design" aria-expanded="false" data-mobile-dropdown-trigger type="button"> <li-icon size="small" type="chevron-down-icon"></li-icon> </button> <ul class="custom-header__dropdownlink-list--mobile hidden" data-dropdownlinks-list="true"> <li> <a href="https://www.linkedin.com/blog/engineering/marketing" class="t-sans t-black t-normal header-nav__link" target="_self">Marketing </a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/sales" class="t-sans t-black t-normal header-nav__link" target="_self">Sales</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/learning" class="t-sans t-black t-normal header-nav__link" target="_self">Learning</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/hiring" class="t-sans t-black t-normal header-nav__link" target="_self">Hiring</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/profile" class="t-sans t-black t-normal header-nav__link" target="_self">Profile</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/messaging-notifications" class="t-sans t-black t-normal header-nav__link" target="_self">Messaging/Notifications</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/feed" class="t-sans t-black t-normal header-nav__link" target="_self">Feed</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/profile" class="t-sans t-black t-normal header-nav__link" target="_self">Profile</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/groups" class="t-sans t-black t-normal header-nav__link" target="_self">Groups</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/accessibility" class="t-sans t-black t-normal header-nav__link" target="_self">Accessibility</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/member-customer-experience" class="t-sans t-black t-normal header-nav__link" target="_self">Member/Customer Experience</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/search" class="t-sans t-black t-normal header-nav__link" target="_self">Search</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/recommendations" class="t-sans t-black t-normal header-nav__link" target="_self">Recommendations</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/skills" class="t-sans t-black t-normal header-nav__link" target="_self">Skills</a> </li> </ul> </li> <li class="custom-header__list-item--mobile" data-mobile-navigation-list-item="true"> <a href="https://www.linkedin.com/blog/engineering/infrastructure" class="t-16 t-sans t-black t-semibold header-nav__toplink" target="_self"> Infrastructure </a> <button class="header-navigation__button" aria-label="Expand to show more links for Infrastructure" aria-expanded="false" data-mobile-dropdown-trigger type="button"> <li-icon size="small" type="chevron-down-icon"></li-icon> </button> <ul class="custom-header__dropdownlink-list--mobile hidden" data-dropdownlinks-list="true"> <li> <a href="https://www.linkedin.com/blog/engineering/code" class="t-sans t-black t-normal header-nav__link" target="_self">Code</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/scalability" class="t-sans t-black t-normal header-nav__link" target="_self">Scalability</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/distributed-systems" class="t-sans t-black t-normal header-nav__link" target="_self">Distributed Systems</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/graph-systems" class="t-sans t-black t-normal header-nav__link" target="_self">Graph Systems</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/architecture" class="t-sans t-black t-normal header-nav__link" target="_self">Architecture</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/analytics" class="t-sans t-black t-normal header-nav__link" target="_self">Analytics</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/automation" class="t-sans t-black t-normal header-nav__link" target="_self">Automation</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/platform-platformization" class="t-sans t-black t-normal header-nav__link" target="_self">Platforms/Platformization</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/cloud-computing" class="t-sans t-black t-normal header-nav__link" target="_self">Cloud Computing</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/open-source" class="t-sans t-black t-normal header-nav__link" target="_self">Open Source</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/additional-innovations" class="t-sans t-black t-normal header-nav__link" target="_self">Additional Innovations</a> </li> </ul> </li> <li class="custom-header__list-item--mobile" data-mobile-navigation-list-item="true"> <a href="https://www.linkedin.com/blog/engineering/talent" class="t-16 t-sans t-black t-semibold header-nav__toplink" target="_self"> Talent </a> <button class="header-navigation__button" aria-label="Expand to show more links for Talent" aria-expanded="false" data-mobile-dropdown-trigger type="button"> <li-icon size="small" type="chevron-down-icon"></li-icon> </button> <ul class="custom-header__dropdownlink-list--mobile hidden" data-dropdownlinks-list="true"> <li> <a href="https://www.linkedin.com/blog/engineering/culture" class="t-sans t-black t-normal header-nav__link" target="_self">Culture</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/developer-experience-productivity" class="t-sans t-black t-normal header-nav__link" target="_self">Developer Experience/Productivity</a> </li> <li> <a href="https://www.linkedin.com/blog/engineering/leadership" class="t-sans t-black t-normal header-nav__link" target="_self">Leadership</a> </li> </ul> </li> </ul> <div class="custom-header__button-share-container--mobile">  </div> </nav> </div> <main role="main"> <div id="lithograph-app"> <section id="component-container" data-attach-collapse-behavior="true"> <div id class="component component-articleHeadline" data-component-type="articleHeadline"> <div class="article-headline-container"> <p class="article-headline__topic"> <a class="t-14" href="https://www.linkedin.com/blog/engineering/ab-testing-experimentation">A/B Testing/Experimentation</a> </p> <h1 class="article-headline t-40" data-article-title="Building inclusive products through A/B testing">Building inclusive products through A/B testing</h1> <div class="article-meta t-14 article-meta__hairline"> <div class="article-meta__byline-container"> <div class="author-profile__author-container"> <img class="author-profile__author-image" srcset="https://media.licdn.com/dms/image/v2/C5603AQFv284H9xed_g/profile-displayphoto-shrink_100_100/profile-displayphoto-shrink_100_100/0/1517188788754?e=2147483647&v=beta&t=9NJdG5ZSv5PBkhBC16dKGO-OWUTIFRwR2FSnPxjlmIA 100w,https://media.licdn.com/dms/image/v2/C5603AQFv284H9xed_g/profile-displayphoto-shrink_200_200/profile-displayphoto-shrink_200_200/0/1517188788791?e=2147483647&v=beta&t=L7ShP5XUnXnA_Z_cQYJta_G5j5ujxev92qDxoSoV9BY 200w,https://media.licdn.com/dms/image/v2/C5603AQFv284H9xed_g/profile-displayphoto-shrink_400_400/profile-displayphoto-shrink_400_400/0/1517188788755?e=2147483647&v=beta&t=U2kTegmn_GF_vvHg2TgXn5CFOTZHDBbbpg33GnCnD24 400w,https://media.licdn.com/dms/image/v2/C5603AQFv284H9xed_g/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1517188788756?e=2147483647&v=beta&t=Vi0bA7sXSYVTEvXaqSExeW658xHFqqL7SNflu_U6al8 450w," sizes="(max-width: 480px) 480px, (max-width: 767px) 767px, (max-width: 1024px) 1024px, 1920px" src="https://media.licdn.com/dms/image/v2/C5603AQFv284H9xed_g/profile-displayphoto-shrink_800_800/profile-displayphoto-shrink_800_800/0/1517188788756?e=2147483647&v=beta&t=Vi0bA7sXSYVTEvXaqSExeW658xHFqqL7SNflu_U6al8" alt="Guillaume Saint-Jacques"> <div class="author-profile__author-text-container"> <a href="https://www.linkedin.com/in/gsaintjacques" target="_blank" class="t-14 t-bold"><span class="screen-reader">Authored by</span>Guillaume Saint-Jacques</a> <p class="t-14">Sr Manager, AIML@Apple</p> <p class="t-14 t-black--light" data-published-date="2020-3-31"> March 31, 2020 </p> </div> </div>  </div> <div class="social-share"> <ul class="social-share__list share-list" title="Share via" data-share-list> <li class="share-list__item share-item"> <a href="#" class="share-item__link medium-round-muted-tertiary-button" role="button" data-tracking-control-name="_linkedin_share" data-share-type="linkedin" aria-label="Share on LinkedIn, New window will open"> <li-icon class="share-item__icon" type="linkedin-icon"></li-icon> </a> </li> <li class="share-list__item share-item"> <a href="#" class="share-item__link medium-round-muted-tertiary-button" role="button" data-tracking-control-name="_facebook_share" data-share-type="facebook" aria-label="Share on Facebook, New window will open"> <li-icon class="share-item__icon" type="facebook-icon"></li-icon> </a> </li> <li class="share-list__item share-item"> <a href="#" class="share-item__link medium-round-muted-tertiary-button" role="button" data-tracking-control-name="_twitter_share" data-share-type="twitter" aria-label="Share on Twitter, New window will open"> <li-icon class="share-item__icon" type="twitter-icon"></li-icon> </a> </li> </ul> </div> </div> </div> </div> <div id class="component component-migratedContent" data-component-type="migratedContent"> <div class="rich-text"> <p><i>Co-authors: <a href="https://www.linkedin.com/in/guillaumesaintjacques/" target="_blank" data-tracking-control-name="_anchor_link">Guillaume Saint-Jacques</a>, <a href="https://www.linkedin.com/in/amir-sepehri-b86b5a45/" target="_blank" data-tracking-control-name="_anchor_link">Amir Sepehri</a>, <a href="https://www.linkedin.com/in/nicole-li-a118b813/" target="_blank" data-tracking-control-name="_anchor_link">Nicole Li</a>, and <a href="https://www.linkedin.com/in/igorperisic/" target="_blank" data-tracking-control-name="_anchor_link">Igor Perisic</a></i></p> <h2>Introduction</h2> <p>Previously on this blog, we’ve shared information on <a href="https://engineering.linkedin.com/blog/topic/data-science" target="_blank" data-tracking-control-name="_anchor_link">best practices in data science</a>, particularly <a href="https://engineering.linkedin.com/ab-testing/why-experimentation-so-important-linkedin" target="_blank" data-tracking-control-name="_anchor_link">in areas such as A/B testing</a>. We’ve also discussed the <a href="https://www.youtube.com/watch?v=Dj6-oE6dAtA&feature=share" target="_blank" data-tracking-control-name="_anchor_link">importance of ethics in fields such as data science</a>, early implementations of <a href="https://engineering.linkedin.com/blog/2019/fairness-privacy-transparency-by-design" target="_blank" data-tracking-control-name="_anchor_link">“fairness by design” principles in our products</a>, and our commitment to <a href="https://engineering.linkedin.com/blog/2018/07/EGR_Program_Insights_and_Updates" target="_blank" data-tracking-control-name="_anchor_link">sharing our research</a> in order to further the industry conversation about designing systems that spread economic opportunity. These findings are shared with the aim of highlighting the real-world positive impacts of data science and encouraging further industry discussion around best practices in responsible product design.</p> <p>In this post, we discuss a novel approach to integrating product A/B testing and inequality measurement concepts from the field of economics. We also discuss the methodology we have adopted for lowering barriers to economic opportunity in how different groups of members use our products. Finally, we provide examples of how it is helping to reshape research and design practices at LinkedIn, through a few choice case studies from the thousands of network A/B tests that we have already analyzed.</p> <p>It is worth emphasizing that the term “inequality” is used throughout this blog post in the following ways:</p> <ul> <li><b>To establish inequality baselines</b>, we use the Atkinson <i>inequality index</i>, which can be applied to any metric, and captures how unequally it is distributed (if everyone has the same amount of that metric, inequality is 0; if some people have a large amount and others nothing, inequality is high). It is routinely applied to income or wealth by economists. Here, we are applying it to metrics that capture economic opportunity for our members on LinkedIn.</li> <li><b>To measure the impact of our experiments on inequality baselines</b>, we use <i>inequality impact</i>, which is used to measure the effect an experiment has on baseline inequality in our metrics. For example, if job applications are very unequally distributed, and an intervention makes them more equally distributed (e.g., by helping people who normally apply to few jobs apply to more of them), we say that there is an <i>inequality reduction impact on job applications</i>.</li> </ul> <h2>Product design, fairness in AI, and A/B testing</h2> <p>In recent years, researchers and industry experts have devoted a great deal of time to exploring the unintended consequences of applied technologies. Three primary areas of concern to many of us in the technology industry include:</p> <ol> <li>The tendency for algorithmic systems to “learn” or otherwise encode real-world biases in their operation (and then further amplify/reinforce those biases);</li> <li>The potential for product design to differentially benefit some groups of users more than others; </li> <li>Sparse or poor data quality that leads to objective-setting errors and system designs that lead to suboptimal outcomes for many groups of end users.</li> </ol> <p>While there are many well-documented examples of these and other types of problems in the technology industry, developing a data-driven solution is not a straightforward task (see recent publications from <a href="https://dblp.org/db/conf/aaai/safeai2019" target="_blank" data-tracking-control-name="_anchor_link">SafeAI@AAAI</a>, <a href="https://facctconference.org/2020/programschedule.html" target="_blank" data-tracking-control-name="_anchor_link">FAccT</a>, and <a href="https://datasociety.net/library/fairness-in-precision-medicine/" target="_blank" data-tracking-control-name="_anchor_link">others</a>).</p> <p><b>Towards a framework for addressing fairness issues in products<br> </b>Given the complexity of this topic, there are likely many ways we could go about ensuring that our members benefit as equally as possible from our products. Before showing our solution to the problem, we want to also advance a set of principles that underpin our thinking:</p> <ul> <li>First, considering what the end result is after people have engaged with a product should be as important as understanding whether an algorithm is “intrinsically” representative or fair. For example, any system that seems to be treating men and women similarly, but still results in women getting disengaged over time, is generally undesirable. These kinds of outcomes may be due to a host of reasons that exhibit patterns of structural inequality in the real world, such as social biases, cultural norms, etc.</li> <li>Secondly, collecting such data may be otherwise problematic. Tracking all <a href="https://www.eeoc.gov/laws/types/" target="_blank" data-tracking-control-name="_anchor_link">protected categories for discrimination</a> would require collecting sensitive data, potentially at odds with members’ expectations of privacy and with data security best practices (e.g., minimization, etc.) that are subject to complex, overlapping privacy laws and regulations. Currently, LinkedIn does not use sensitive demographic data (as defined by GDPR, e.g., race, ethnicity, religion, political preferences, etc.) for our Recruiter product or for marketing services; members in some regions can <a href="https://www.linkedin.com/psettings/demographics" target="_blank" data-tracking-control-name="_anchor_link">opt-in</a> to provide limited demographic data for aggregate reporting purposes.</li> <li>Finally, using existing demographic categories may not map to or reflect all kinds of inequality directly. It is possible that we may be overlooking many opportunities to improve our products if we look solely at the categories we are explicitly monitoring. We would like a way to identify instances of these kinds of functional inequalities that do not map to existing categories of users during the product testing process.</li> </ul> <p>In summary, even if data on members’ demographic categories is available, it is not a panacea for identifying inequality impact. Even if a product may seem to have been designed in a “responsible” or “fair” manner based on assumptions of demographic parity, it can still drive a wedge between different groups of users. For instance, an app update that improves overall engagement but runs slowly on older mobile devices might dramatically affect members across many demographic categories in a manner that does not appear in a typical product A/B test. </p> <p>Traditional A/B testing looks at averages, focusing on an idealized “average user.” However, people may respond to new products in ways that a designer never intended. In order to be inclusive, we need to look beyond the average. The approach that we’ve developed, outlined below, instead empowers leaders to design products that are more inclusive and equitable, regardless of the causes of an underlying disparity. This helps to overcome the “average user” problem of traditional A/B testing. Building more equitable products is also good business, as making sure no one is inadvertently left behind is key to long-term growth.</p> </div> </div> <div id class="component component-standaloneImage" data-component-type="standaloneImage"> <div class="standalone-image-component standalone-image-component--article-width"> <figure> <img class="standalone-image-component__image" sizes="(max-width: 480px) 480px, (max-width: 767px) 767px, (max-width: 1024px) 1024px, 1920px" data-delayed-url="https://media.licdn.com/dms/image/v2/D4D08AQG820bamMAS_w/croft-frontend-shrinkToFit1024/croft-frontend-shrinkToFit1024/0/1700688417474?e=2147483647&v=beta&t=EaymMstRr9pJgAknCVTWjV40xEErICdPpH9tr5uRWFM" alt="showing-the-inequality-impact-of-a-b-testing">  </figure> </div> </div> <div id class="component component-migratedContent" data-component-type="migratedContent"> <div class="rich-text"> <p><i>A stylized example of traditional A/B testing assumptions vs. results that show inequality impact</i></p> <p><b>An A/B testing approach to measuring fairness and inequality impact<br> </b>For several years at LinkedIn, we have used a series of <a href="https://content.linkedin.comhttps://business.linkedin.com/content/dam/engineering/site-assets/pdfs/ABTestingSocialNetwork_share.pdf" target="_blank" data-tracking-control-name="_anchor_link">scalable experimentation platforms</a> to analyze <a href="https://engineering.linkedin.com/mobile/mobile-ab-testing-linkedin-how-members-shape-our-apps" target="_blank" data-tracking-control-name="_anchor_link">product changes</a>, AI model revisions, and many of the <a href="https://business.linkedin.com/marketing-solutions/blog/linkedin-b2b-marketing/2016/linkedin-advertising-101--targeting" target="_blank" data-tracking-control-name="_anchor_link">business decisions</a> at our company. Instead of characterizing a feature in the void, experimentation measures the effect it has on real users. As we explored the <a href="https://blog.linkedin.com/2019/september/26/closing-the-network-gap" target="_blank" data-tracking-control-name="_anchor_link">implications of social network composition</a> on economic outcomes, our team identified a unique opportunity to apply this methodology to LinkedIn products. We then started measuring the inequality impact of all new features and product changes (whether they are algorithms, UI tweaks, or infrastructure changes, for example) and flagging experiments that have a notably positive or negative inequality impact for users that are more or less well-connected on LinkedIn. Subsequently, we applied this method to every single experiment that LinkedIn has conducted over the past year and analyzed the ones with the highest inequality impact-increasing or -reducing effect.</p> </div> </div> <div id class="component component-standaloneImage" data-component-type="standaloneImage"> <div class="standalone-image-component standalone-image-component--article-width"> <figure> <img class="standalone-image-component__image" sizes="(max-width: 480px) 480px, (max-width: 767px) 767px, (max-width: 1024px) 1024px, 1920px" data-delayed-url="https://media.licdn.com/dms/image/v2/D4D08AQHdhU0lfdqqgA/croft-frontend-shrinkToFit1024/croft-frontend-shrinkToFit1024/0/1700688417572?e=2147483647&v=beta&t=wgeX72Dnhnmw4LtH4ck1Ge6qmoWw_WFRln0JILLg-n4" alt="a-b-testing-in-the-product-development-lifecycle">  </figure> </div> </div> <div id class="component component-migratedContent" data-component-type="migratedContent"> <div class="rich-text"> <p><i>An example of A/B testing within the product development lifecycle</i></p> <p><b>Inequality measures using the Atkinson index<br> </b>The Atkinson index is a standard measure of economic inequality that is useful in determining which end of the distribution contributed most to the observed inequality. It is often used by economists for comparisons of income distributions. For example, the United Nations Human Development Programme <a href="https://www.un.org/en/development/desa/policy/wess/wess_dev_issues/dsp_policy_02.pdf" target="_blank" data-tracking-control-name="_anchor_link">leverages the Atkinson index</a> to allow for meaningful comparisons of policy impacts across countries with <a href="http://hdr.undp.org/enhttps://business.linkedin.com/content/gini-coefficient-not-sufficient-measure-inequality-what-difference-between-gini-and-atkinson" target="_blank" data-tracking-control-name="_anchor_link">widely differing ranges of income distributions</a>, while the U.S. Census <a href="https://www.census.gov/topics/income-poverty/income-inequality/about/metrics/atkinson-index.html" target="_blank" data-tracking-control-name="_anchor_link">uses the Index</a> for income comparisons within American society.</p> <p>The Atkinson index can be formulated thusly: Given a sample x<sub>1</sub>,…,x<sub>n</sub>, representing a metric (such as sessions, page views, connections accepted, etc.) for a group of users indexed by <i>i</i>, and an inequality-aversion parameter <i>ϵ</i>. It is computed as follows for ϵ≠1:</p> </div> </div> <div id class="component component-standaloneImage" data-component-type="standaloneImage"> <div class="standalone-image-component standalone-image-component--article-width"> <figure> <img class="standalone-image-component__image" sizes="(max-width: 480px) 480px, (max-width: 767px) 767px, (max-width: 1024px) 1024px, 1920px" data-delayed-url="https://media.licdn.com/dms/image/v2/D4D08AQGwdtwGK90ONg/croft-frontend-shrinkToFit1024/croft-frontend-shrinkToFit1024/0/1700688417550?e=2147483647&v=beta&t=E8dsyOi58nbjzXHJwAF83S3M5s3zmawCGSPoLaS7jj4" alt="formula-of-the-atkinson-index">  </figure> </div> </div> <div id class="component component-migratedContent" data-component-type="migratedContent"> <div class="rich-text"> <p>Intuitively, for a fixed <i>ϵ</i> , a low value of <i>A<sub>ϵ</sub></i> is obtained when all individuals have almost the exact same metric value. When this is exact, the Atkinson index is zero. A higher value reflects more inequality. The <i>ϵ</i> parameter can be tuned to reflect the inequality preferences of a decision maker, as will be shown in the next section.</p> <p>The Atkinson index has several desirable properties for our application, namely:</p> <ul> <li><i>It is zero only if all individual metrics are the same.</i> This is helpful in comparing distributions to the “pure equality” baseline, as shown in the above figure.</li> <li><i>It satisfies the population replication axiom.</i> If we create a new population by replicating the existing population any number of times, the inequality remains the same. This is particularly helpful because our member base continues to increase, and the simple fact that the number of users is growing should not change the inequality index, unless actual inequality increases are happening at the same time.</li> <li><i>It satisfies the principle of transfers.</i> Any redistribution (i.e., reducing one user’s metrics to increase the metrics of another, lower-ranking user) results in a decrease of the Atkinson index, as long as this intervention does not change their rank in the distribution. This can be useful to specifically assess the impact of “redistribution” experiments, such as redistributing attention on the feed.</li> <li><i>If all metrics are multiplied by a positive constant, the index remains the same.</i> This is useful because it allows us to compare the inequality of distributions across different time horizons meaningfully. For example, if all users have the same number of sessions every day, the inequality will be the same, whether measured as daily, weekly, or monthly sessions.</li> <li><i>It is scalable.</i> Looking at the formula above, one can see that the Atkinson index lends itself to distributed computation, using map-reduce or Spark, for example.</li> </ul> <p><b>Scaling this approach with existing design processes<br> </b>As mentioned <a href="https://engineering.linkedin.com/ab-testing/why-experimentation-so-important-linkedin" target="_blank" data-tracking-control-name="_anchor_link">previously</a>, experimentation is deeply embedded in LinkedIn’s decision-making processes and company culture. Every day, we run hundreds of A/B tests tracking thousands of metrics—everything from minor visual changes in an app to improvements to our AI-powered recommendation algorithms. Starting last year, we began tracking the inequality impact of our experiments on core business and member value metrics. Since our addition of this analysis to the typical A/B testing process, we’ve also created a special multidisciplinary team at the company that discusses the impact of notable experiments. This team then invites the owners of these products to a working session to discuss the impact. </p> <p>In essence, along with asking "What would be the total number of sessions or contributions on our platform if feature A (vs. B) were rolled out?” we can also ask "If feature A were to be rolled out, what would be the share of contributions from the top 1% of members, in terms of engagement and contributions? Would inequality impact go up or down between our most and least engaged members?" After the meeting, experiment owners are encouraged to do a deep-dive to better understand why the change may be having a disproportionate impact among different subgroups, and to share their learnings. The findings from these experiments have also gradually made their way into regular updates that are shared throughout the product and data science organizations at LinkedIn. </p> <p>Combining measures of inequality (drawn from economics literature) and A/B testing provides us two distinct advantages.</p> <ul> <li>First, instead of only measuring inequality impact, we can also trace it back to its causes: a specific set of features and product decisions. We can ask, “Are our products responsive to the needs of every LinkedIn member?” </li> <li>Second, unlike classical algorithmic fairness approaches, it helps us identify features that increase inequality impact without having to rely only on explicitly protected categories. A feature that increases the gap between any two groups of members is likely to be detected.</li> </ul> <h2>Example of an experiment: Instant job notifications</h2> <p>The effect that this methodological change has had at LinkedIn can be seen in our products. Earlier this year, we rolled out a new feature called “Instant Job Notifications” that sends a push notification to active job seekers as soon as a new, relevant job is posted. These kinds of features can be a contentious issue, as we need to be selective about the kind of events on our platform that <a href="https://engineering.linkedin.com/blog/2018/03/air-traffic-controller--member-first-notifications-at-linkedin" target="_blank" data-tracking-control-name="_anchor_link">trigger a notification</a>. During A/B testing, our review team found that this new feature had a significant equalizing effect—it matched the right people to the right job, regardless of how new they were to the site, where they are in their career, or the relative strength of their online network.</p> <p>We were able to identify two main reasons for this change. First, prior to this feature being rolled out, members with less well-connected networks (“social capital” in the image below, a measure of how many connections a member has and how likely those connections are to bridge different clusters of members) were not as likely to be referred to these jobs or to hear about them through their network. Second, sharing the job opportunity before it had received a high number of applicants made members more likely to apply; there is a wealth of research on how self-censorship and other socially normative behaviors can disproportionately impact some groups more than others.</p> </div> </div> <div id class="component component-standaloneImage" data-component-type="standaloneImage"> <div class="standalone-image-component standalone-image-component--article-width"> <figure> <img class="standalone-image-component__image" sizes="(max-width: 480px) 480px, (max-width: 767px) 767px, (max-width: 1024px) 1024px, 1920px" data-delayed-url="https://media.licdn.com/dms/image/v2/D4D08AQGhoHPV3nWYFg/croft-frontend-shrinkToFit1024/croft-frontend-shrinkToFit1024/0/1700688417663?e=2147483647&v=beta&t=ehT_enkJ2HAlPZepgtS1adeyDR60J1UYyU3xjQhMOnU" alt="example-of-a-feedback-panel-from-inequality-impact-testing-process">  </figure> </div> </div> <div id class="component component-migratedContent" data-component-type="migratedContent"> <div class="rich-text"> <p><i>An example feedback panel from the inequality impact testing process, using obfuscated data</i></p> <p>Our analysis found that Instant Job Notifications had several positive effects for members. In terms of our <a href="https://engineering.linkedin.com/blog/2018/10/an-introduction-to-ai-at-linkedin" target="_blank" data-tracking-control-name="_anchor_link">core metrics</a>, it increased the number of job applications we saw for a given posting and also increased the chance of an application receiving interaction from the prospective employer. Finally, the illustration above shows that engagement from members with “low social capital” also increased. </p> <h2>Analyzing thousands of experiments: What we’ve learned so far</h2> <p>Fairness through experimentation should not only be about setting guardrails to detect new features that could be potentially damaging to our members; it should also be about finding inspiration to create a more inclusive product. Contrary to “top-down” fairness criteria or ex-ante requirements, we try to learn from our colleagues and engineers as they experiment on the features they are building. Through this, we are compiling a knowledge base of types of interventions that seem to reduce inequality impact, so that it may be used as a guide for the development of our next products. Below, we share a few of the things we have been able to learn so far.</p> <ul> <li><b>Metric-neutral interventions are often not neutral for everyone:</b> In many situations, teams may try to implement “neutral” interventions—for example, when performing a back-end infrastructure change, or when trying to “boost” or promote a specific product on the platform while making sure no other product suffers. The principal method of monitoring this neutrality is looking at a treatment-control comparison of the average impact (often called lift) in the experimentation platform. However, even if no average impact is detected, we found many instances where there was an inequality impact. In other words, metrics were not affected on average, but some members were. This makes inequality impact a critical aspect to monitor in such experiments, as “neutral” impact should mean neutral for everyone. </li> <li><b>Notifications are a powerful tool:</b> Notifications have a strong impact on inequality of engagement, as they are a powerful tool to help orient less-engaged members towards useful features on the site. Prior work has also shown that our efforts to strategically batch notifications for highly-engaged members resulted in a qualitatively better user experience for that group of users. In short, an inequality-aware design pattern may be to handle notifications differently for different groups of members, based on their engagement levels.</li> <li><b>New member onboarding is extremely important:</b> Making sure more people benefit from LinkedIn requires helping new members familiarize themselves with the platform and its value proposition. Providing a richer onboarding experience had a positive impact on average engagement, but also on equality of engagement, since it primarily helped members who were at the highest risk of dropping off or overlooking useful features on the site.</li> <li><b>Site speed and availability matter to inclusiveness:</b> We found that many interventions relating to site speed and reliability had a disproportionately positive impact on the least engaged members, and reduced inequality. This makes sense, as a suboptimal experience could lead to lower engagement, and members who only have access to slower devices and connections may also experience other structural inequalities that limit their opportunity. </li> <li><b>The low-bandwidth LinkedIn app:</b> A corollary to the above point is that inclusive product design needs to account for members with slower devices or limited data plans. Several experiments on the <a href="https://engineering.linkedin.com/blog/2018/03/linkedin-lite--a-lightweight-mobile-web-experience" target="_blank" data-tracking-control-name="_anchor_link">low-bandwidth optimized LinkedIn app</a> showed a strong positive impact, both on average engagement and on equality in engagement. Adding features that brought the low-bandwidth app experience closer to the experience of members using the default app or the desktop experience had positive inclusiveness effects.</li> <li><b>On social network platforms, social capital matters for inclusiveness:</b> Once an inequality impact (positive or negative) is found, we seek to understand it, in particular by asking whether we can identify two or more groups that are being affected differently by the experiment. We have repeatedly found that a member’s “social capital” often has an impact on how much value she can get out of a social network.</li> <li><b>When it comes to inequality, unintended consequences are the norm:</b> Throughout over a year of experiment review meetings and learning from experiment owners, we have often found that both negative and positive inequality impacts are unintended by product managers. Designers and product managers may often think about members in an idealized fashion: as a representative, average user that does not actually exist. This may pose inclusiveness challenges, as it runs the risk of leaving members who do not resemble the idealized average behind.</li> </ul> <h2>A/B testing at scale for large-scale change</h2> <p>Our approach to fairness is to use A/B testing as a complement, not a substitute, to additional approaches to creating more equitable products that are already implemented at LinkedIn, such as evaluating bias in datasets, various approaches to eliminating biases in AI systems, and qualitative member research. The most important thing in any design process (AI modeling, product design, experimental design, etc.) is to properly define the problem you are trying to solve. To do this, we need to collaborate with experts across domains and look for new ideas outside of the typical best practices in the technology industry. Similarly, it truly takes a variety of approaches and viewpoints to understand how these systems will fully impact an end user. In the end, this requires an investment in tools, culture, and processes that make our engineers talk to their product/design partners—the domain knowledge experts.</p> <p>As the world’s largest professional network, LinkedIn has a unique opportunity to help close opportunity gaps, such as the skills gap and the network gap. Since neither skills nor networks are the basis for legally protected categories, utilizing a new, inequality impact-based approach to A/B testing that helps us detect the unintended consequences of new products and features is just one of the ways that we are making this happen.</p> <p>Using a number of LinkedIn examples, we have argued that instead of just looking at the average effect of decisions, leaders should also consider the inequality impact when designing products. For each potential innovation, this means asking: “Does this proposal increase or decrease inequality among our members?” or, more specifically, “Does this proposal benefit members with low social capital as much as it benefits well-connected members?” We hope that  increased understanding of the underlying causes of inequality can lead to similar approaches to ethical product design across several different industries.</p> <p><i>Read the full paper on <a href="https://arxiv.org/abs/2002.05819" data-tracking-control-name="_anchor_link">arXiv</a>.</i></p> <p><i>Updated April 2, 2020 to add a reference to opt-in data on the personal settings page. </i></p> </div> </div> <div id class="component component-postList" data-component-type="postList"> <section id="postList0FocusPoint" class="component__content-container list-layout" data-post-list> <section class="simple-topic-list"> <p class="t-14 t-bold">Topics: <a class="simple-topic-list__topic" href="https://www.linkedin.com/blog/engineering/ab-testing-experimentation"> A/B Testing/Experimentation </a> <a class="simple-topic-list__topic" href="https://www.linkedin.com/blog/engineering/data"> Data </a> <a class="simple-topic-list__topic" href="https://www.linkedin.com/blog/engineering/product-design"> Product Design </a> <a class="simple-topic-list__topic" href="https://www.linkedin.com/blog/engineering/research"> Research </a> <a class="simple-topic-list__topic" href="https://www.linkedin.com/blog/engineering/data-science"> Data Science </a> <a class="simple-topic-list__topic" href="https://www.linkedin.com/blog/engineering/machine-learning"> Machine Learning </a> </p> </section> <hr class="artdeco-divider"> <section class="list-layout__headline-container"> <p class="list-layout__headline t-32 t-sans">Related articles</p>  </section> <ul class="list-layout__posts"> <li class="post-list__item list-post"> <div class="list-post__content-container"> <p class="list-post__content-container__topic"> <a class="t-14 t-bold" href="https://www.linkedin.com/blog/engineering/product-design"> Product Design </a> </p> <div class="list-post__content-container__title"> <a class="list-post__link t-20 t-black" href="https://www.linkedin.com/blog/engineering/product-design/building-collaborative-prompt-engineering-playgrounds-using-jupyter-notebook"> Building collaborative prompt engineering playgrounds using Ju... </a> </div> <div class="list-post__content-container__byline"> <p class="list-post__content-container__author t-14 t-bold t-black"> Ajay Prakash </p>  <p class="list-post__content-container__date t-14 t-black--light"> Feb 13, 2025 </p> </div> </div> <div class="list-post__image-container"> <figure class="post__image-figure"> <img class="post__image" data-delayed-url="https://media.licdn.com/dms/image/v2/D4D08AQGjTYbN5UtiOg/croft-frontend-shrinkToFit767/B4DZT8ALjpGkAQ-/0/1739394694227?e=2147483647&v=beta&t=LJd4SIgJx9IY4Rdw7d2Rk7ZiiCRkZDBBMsbWQ8KF_5w" data-ghost-url="https://static.licdn.com/aero-v1/sc/h/8pouyb1ly746xtru0x7p2axm0" sizes="(max-width: 480px) 480px, (max-width: 767px) 767px, (max-width: 1024px) 1024px, 480px"> </figure> </div> </li> <li class="post-list__item list-post"> <div class="list-post__content-container"> <p class="list-post__content-container__topic"> <a class="t-14 t-bold" href="https://www.linkedin.com/blog/engineering/marketing"> Marketing </a> </p> <div class="list-post__content-container__title"> <a class="list-post__link t-20 t-black" href="https://www.linkedin.com/blog/engineering/marketing/buyer-journey-insights-with-data-driven-attribution"> Buyer journey insights with data-driven attribution </a> </div> <div class="list-post__content-container__byline"> <p class="list-post__content-container__author t-14 t-bold t-black"> John Bencina </p>  <p class="list-post__content-container__date t-14 t-black--light"> Jan 22, 2025 </p> </div> </div> <div class="list-post__image-container"> <figure class="post__image-figure"> <img class="post__image" data-delayed-url="https://media.licdn.com/dms/image/v2/D4D08AQGkfqQ8hTSyDQ/croft-frontend-shrinkToFit1024/croft-frontend-shrinkToFit1024/0/1727903865702?e=2147483647&v=beta&t=G6jwkQKs9ofiIlXDmks9sa7HK-CWm_OMYJ7D0cLAJ-I" data-ghost-url="https://static.licdn.com/aero-v1/sc/h/8pouyb1ly746xtru0x7p2axm0" sizes="(max-width: 480px) 480px, (max-width: 767px) 767px, (max-width: 1024px) 1024px, 480px"> </figure> </div> </li> <li class="post-list__item list-post"> <div class="list-post__content-container"> <p class="list-post__content-container__topic"> <a class="t-14 t-bold" href="https://www.linkedin.com/blog/engineering/talent"> Talent </a> </p> <div class="list-post__content-container__title"> <a class="list-post__link t-20 t-black" href="https://www.linkedin.com/blog/engineering/culture/announcing-our-linkedin-cornell-2024-grant-recipients"> Announcing Our LinkedIn-Cornell 2024 Grant Recipients </a> </div> <div class="list-post__content-container__byline"> <p class="list-post__content-container__author t-14 t-bold t-black"> Natesh Pillai </p> <p class="list-post__content-container__date t-14 t-black--light"> Aug 13, 2024 </p> </div> </div> <div class="list-post__image-container"> <figure class="post__image-figure"> <img class="post__image" alt data-delayed-url="https://media.licdn.com/dms/image/v2/D4D08AQENv5zuGz0oWQ/croft-frontend-shrinkToFit1024/croft-frontend-shrinkToFit1024/0/1723502284196?e=2147483647&v=beta&t=3hexsH2DgbYsh_mi9fBXB7vLXm23bI4B2HuF1YcOVYg" data-ghost-url="https://static.licdn.com/aero-v1/sc/h/8pouyb1ly746xtru0x7p2axm0" sizes="(max-width: 480px) 480px, (max-width: 767px) 767px, (max-width: 1024px) 1024px, 480px"> </figure> </div> </li> </ul>  </section> </div>  </section> </div> <script src="https://static.licdn.com/aero-v1/sc/h/8hfbuq1ftcvnnx4dd5067pi0t" async></script> <script src="https://static.licdn.com/aero-v1/sc/h/dev5px9xu71l67oqmrrbimi6b" async></script> <script src="https://static.licdn.com/aero-v1/sc/h/tx4xi374guj6fou99j3hlnlj" async></script> </main> <footer class="li-footer bg-transparent w-full "> <ul class="li-footer__list flex flex-wrap flex-row items-start justify-start w-full h-auto min-h-[50px] my-[0px] mx-auto py-3 px-2 papabear:w-[1128px] papabear:p-0"> <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <span class="sr-only">LinkedIn</span> <icon class="li-footer__copy-logo text-color-logo-brand-alt inline-block self-center h-[14px] w-[56px] mr-1" data-delayed-url="https://static.licdn.com/aero-v1/sc/h/e12h2cd8ac580qen9qdd0qks8"></icon> <span class="li-footer__copy-text flex items-center">© 2025</span> </li> <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <a class="li-footer__item-link flex items-center font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover focus:text-color-link-focus" href="https://about.linkedin.com?trk=content_footer-about" data-tracking-control-name="content_footer-about" data-tracking-will-navigate> About </a> </li> <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <a class="li-footer__item-link flex items-center font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover focus:text-color-link-focus" href="https://www.linkedin.com/accessibility?trk=content_footer-accessibility" data-tracking-control-name="content_footer-accessibility" data-tracking-will-navigate> Accessibility </a> </li> <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <a class="li-footer__item-link flex items-center font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover focus:text-color-link-focus" href="https://www.linkedin.com/legal/user-agreement?trk=content_footer-user-agreement" data-tracking-control-name="content_footer-user-agreement" data-tracking-will-navigate> User Agreement </a> </li> <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <a class="li-footer__item-link flex items-center font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover focus:text-color-link-focus" href="https://www.linkedin.com/legal/privacy-policy?trk=content_footer-privacy-policy" data-tracking-control-name="content_footer-privacy-policy" data-tracking-will-navigate> Privacy Policy </a> </li>  <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <a class="li-footer__item-link flex items-center font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover focus:text-color-link-focus" href="https://www.linkedin.com/legal/cookie-policy?trk=content_footer-cookie-policy" data-tracking-control-name="content_footer-cookie-policy" data-tracking-will-navigate> Cookie Policy </a> </li> <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <a class="li-footer__item-link flex items-center font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover focus:text-color-link-focus" href="https://www.linkedin.com/legal/copyright-policy?trk=content_footer-copyright-policy" data-tracking-control-name="content_footer-copyright-policy" data-tracking-will-navigate> Copyright Policy </a> </li> <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <a class="li-footer__item-link flex items-center font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover focus:text-color-link-focus" href="https://brand.linkedin.com/policies?trk=content_footer-brand-policy" data-tracking-control-name="content_footer-brand-policy" data-tracking-will-navigate> Brand Policy </a> </li> <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <a class="li-footer__item-link flex items-center font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover focus:text-color-link-focus" href="https://www.linkedin.com/psettings/guest-controls?trk=content_footer-guest-controls" data-tracking-control-name="content_footer-guest-controls" data-tracking-will-navigate> Guest Controls </a> </li> <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <a class="li-footer__item-link flex items-center font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover focus:text-color-link-focus" href="https://www.linkedin.com/help/linkedin/answer/34593?trk=content_footer-community-guide" data-tracking-control-name="content_footer-community-guide" data-tracking-will-navigate> Community Guidelines </a> </li>  <li class="li-footer__item font-sans text-xs text-color-text-low-emphasis flex flex-shrink-0 justify-start p-1 relative w-50% papabear:justify-center papabear:w-auto"> <div class="collapsible-dropdown collapsible-dropdown--footer collapsible-dropdown--up flex items-center relative hyphens-auto language-selector z-2">  <ul class="collapsible-dropdown__list hidden container-raised absolute w-auto overflow-y-auto flex-col items-stretch z-1 bottom-[100%] top-auto" role="menu" tabindex="-1"> <li class="language-selector__item" role="presentation">  <button aria-label="العربية (Arabic)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-ar_AE" data-locale="ar_AE" role="menuitem" lang="ar_AE"> العربية (Arabic) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="বাংলা (Bangla)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-bn_IN" data-locale="bn_IN" role="menuitem" lang="bn_IN"> বাংলা (Bangla) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Čeština (Czech)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-cs_CZ" data-locale="cs_CZ" role="menuitem" lang="cs_CZ"> Čeština (Czech) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Dansk (Danish)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-da_DK" data-locale="da_DK" role="menuitem" lang="da_DK"> Dansk (Danish) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Deutsch (German)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-de_DE" data-locale="de_DE" role="menuitem" lang="de_DE"> Deutsch (German) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Ελληνικά (Greek)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-el_GR" data-locale="el_GR" role="menuitem" lang="el_GR"> Ελληνικά (Greek) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="English (English) selected" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link--selected" data-tracking-control-name="language-selector-en_US" data-locale="en_US" role="menuitem" lang="en_US"> <strong>English (English)</strong> </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Español (Spanish)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-es_ES" data-locale="es_ES" role="menuitem" lang="es_ES"> Español (Spanish) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="فارسی (Persian)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-fa_IR" data-locale="fa_IR" role="menuitem" lang="fa_IR"> فارسی (Persian) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Suomi (Finnish)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-fi_FI" data-locale="fi_FI" role="menuitem" lang="fi_FI"> Suomi (Finnish) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Français (French)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-fr_FR" data-locale="fr_FR" role="menuitem" lang="fr_FR"> Français (French) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="हिंदी (Hindi)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-hi_IN" data-locale="hi_IN" role="menuitem" lang="hi_IN"> हिंदी (Hindi) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Magyar (Hungarian)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-hu_HU" data-locale="hu_HU" role="menuitem" lang="hu_HU"> Magyar (Hungarian) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Bahasa Indonesia (Indonesian)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-in_ID" data-locale="in_ID" role="menuitem" lang="in_ID"> Bahasa Indonesia (Indonesian) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Italiano (Italian)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-it_IT" data-locale="it_IT" role="menuitem" lang="it_IT"> Italiano (Italian) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="עברית (Hebrew)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-iw_IL" data-locale="iw_IL" role="menuitem" lang="iw_IL"> עברית (Hebrew) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="日本語 (Japanese)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-ja_JP" data-locale="ja_JP" role="menuitem" lang="ja_JP"> 日本語 (Japanese) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="한국어 (Korean)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-ko_KR" data-locale="ko_KR" role="menuitem" lang="ko_KR"> 한국어 (Korean) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="मराठी (Marathi)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-mr_IN" data-locale="mr_IN" role="menuitem" lang="mr_IN"> मराठी (Marathi) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Bahasa Malaysia (Malay)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-ms_MY" data-locale="ms_MY" role="menuitem" lang="ms_MY"> Bahasa Malaysia (Malay) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Nederlands (Dutch)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-nl_NL" data-locale="nl_NL" role="menuitem" lang="nl_NL"> Nederlands (Dutch) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Norsk (Norwegian)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-no_NO" data-locale="no_NO" role="menuitem" lang="no_NO"> Norsk (Norwegian) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="ਪੰਜਾਬੀ (Punjabi)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-pa_IN" data-locale="pa_IN" role="menuitem" lang="pa_IN"> ਪੰਜਾਬੀ (Punjabi) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Polski (Polish)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-pl_PL" data-locale="pl_PL" role="menuitem" lang="pl_PL"> Polski (Polish) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Português (Portuguese)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-pt_BR" data-locale="pt_BR" role="menuitem" lang="pt_BR"> Português (Portuguese) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Română (Romanian)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-ro_RO" data-locale="ro_RO" role="menuitem" lang="ro_RO"> Română (Romanian) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Русский (Russian)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-ru_RU" data-locale="ru_RU" role="menuitem" lang="ru_RU"> Русский (Russian) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Svenska (Swedish)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-sv_SE" data-locale="sv_SE" role="menuitem" lang="sv_SE"> Svenska (Swedish) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="తెలుగు (Telugu)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-te_IN" data-locale="te_IN" role="menuitem" lang="te_IN"> తెలుగు (Telugu) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="ภาษาไทย (Thai)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-th_TH" data-locale="th_TH" role="menuitem" lang="th_TH"> ภาษาไทย (Thai) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Tagalog (Tagalog)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-tl_PH" data-locale="tl_PH" role="menuitem" lang="tl_PH"> Tagalog (Tagalog) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Türkçe (Turkish)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-tr_TR" data-locale="tr_TR" role="menuitem" lang="tr_TR"> Türkçe (Turkish) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Українська (Ukrainian)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-uk_UA" data-locale="uk_UA" role="menuitem" lang="uk_UA"> Українська (Ukrainian) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="Tiếng Việt (Vietnamese)" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-vi_VN" data-locale="vi_VN" role="menuitem" lang="vi_VN"> Tiếng Việt (Vietnamese) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="简体中文 (Chinese (Simplified))" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-zh_CN" data-locale="zh_CN" role="menuitem" lang="zh_CN"> 简体中文 (Chinese (Simplified)) </button> </li> <li class="language-selector__item" role="presentation">  <button aria-label="正體中文 (Chinese (Traditional))" class="font-sans text-xs link block py-[5px] px-2 w-full hover:cursor-pointer hover:bg-color-action hover:text-color-text-on-dark focus:bg-color-action focus:text-color-text-on-dark language-selector__link !font-regular" data-tracking-control-name="language-selector-zh_TW" data-locale="zh_TW" role="menuitem" lang="zh_TW"> 正體中文 (Chinese (Traditional)) </button> </li>  </ul> <button class="language-selector__button select-none relative pr-2 font-sans text-xs font-bold text-color-text-low-emphasis hover:text-color-link-hover hover:cursor-pointer focus:text-color-link-focus focus:outline-dotted focus:outline-1" aria-expanded="false" data-tracking-control-name="footer-lang-dropdown_trigger"> <span class="language-selector__label-text mr-0.5 break-words"> Language </span> <icon class="language-selector__label-chevron w-2 h-2 absolute top-0 right-0" data-delayed-url="https://static.licdn.com/aero-v1/sc/h/cyolgscd0imw2ldqppkrb84vo"></icon> </button> </div> </li> </ul>  </footer> <script src="https://static.licdn.com/aero-v1/sc/h/eh08muqvrde4h3hc6koyij5ti" async></script>  </body> </html>

CINXE.COM

Building inclusive products through A/B testing