CINXE.COM

<!DOCTYPE html><html lang="en-GB"><head><script type="text/javascript" src="/_static/js/bundle-playback.js?v=HxkREWBo" charset="utf-8"></script> <script type="text/javascript" src="/_static/js/wombat.js?v=txqj7nKC" charset="utf-8"></script> <script>window.RufflePlayer=window.RufflePlayer||{};window.RufflePlayer.config={"autoplay":"on","unmuteOverlay":"hidden"};</script> <script type="text/javascript" src="/_static/js/ruffle/ruffle.js"></script> <script type="text/javascript"> __wm.init("http://web.archive.org/web"); __wm.wombat("https://www.jisc.ac.uk/guides/developing-high-quality-question-and-answer-sets-for-chatbots","20240227045235","http://web.archive.org/","web","/_static/", "1709009555"); </script> <link rel="stylesheet" type="text/css" href="/_static/css/banner-styles.css?v=S1zqJCYt" /> <link rel="stylesheet" type="text/css" href="/_static/css/iconochive.css?v=3PDvdIFv" />  <meta charset="utf-8"/><meta name="viewport" content="width=device-width, initial-scale=1"/><title>Developing high-quality question-and-answer sets for chatbots - Jisc</title><meta name="description" content="You've started work on introducing chatbots to your organisation, now find out about what's involved in development and implementation."/><meta name="twitter:card" content="summary_large_image"/><meta name="twitter:site" content="@jisc"/><meta property="twitter:image"/><meta property="twitter:image:alt" content=""/><meta property="og:title" content="Developing high-quality question-and-answer sets for chatbots - Jisc"/><meta property="og:description" content="You've started work on introducing chatbots to your organisation, now find out about what's involved in development and implementation."/><meta property="og:type" content="website"/><meta property="og:site_name" content="Jisc"/><meta property="og:url" content="http://web.archive.org/web/20240227045235/https://beta.jisc.ac.uk/guides/developing-high-quality-question-and-answer-sets-for-chatbots"/><meta property="og:image"/><meta property="og:image:alt" content=""/><meta name="next-head-count" content="15"/><meta http-equiv="Content-Security-Policy" content=""/><meta name="robots" content="index, follow"/><meta name="google-site-verification" content="UNqcoVblscrmuOfPyGUTeRMyNRcroUK_U9ghx-PFl0Q"/><link rel="apple-touch-icon" sizes="180x180" href="/web/20240227045235im_/https://www.jisc.ac.uk/apple-touch-icon.png"/><link rel="manifest" href="/web/20240227045235/https://www.jisc.ac.uk/site.webmanifest"/><link rel="preconnect" href="http://web.archive.org/web/20240227045235/https://fonts.gstatic.com/" crossorigin/><link rel="preload" href="/web/20240227045235/https://www.jisc.ac.uk/_next/static/css/c71e2eb9d5f58a34.css" as="style"/><link rel="stylesheet" href="/web/20240227045235cs_/https://www.jisc.ac.uk/_next/static/css/c71e2eb9d5f58a34.css" data-n-g=""/><noscript data-n-css=""></noscript><script defer="" nomodule="" src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js"></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/webpack-cefddfb7403e4fd5.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/framework-92a422f151f77ddb.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/main-b7b1d23e11800ea1.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/pages/_app-f82791795eb58916.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/653-32bafc607c9d7b18.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/5785-48f49357a2a50076.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/6649-e004daf2b30cf83b.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/7758-d4bdcd49824ecc1b.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/7433-394f70b2d63a646a.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/8935-6c1bf2111ed2b8bd.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/542-885fdfe9a39a717d.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/3722-9fac8acde93f3fec.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/8239-a6c3d989f1e5f9c9.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/814-fa0f70416c3d6cea.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/3028-62cbdb2ea442f939.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/9177-b0f7a5239df17233.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/4031-44b8f159d6287c42.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/607-53a83fe5e28837da.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/3262-d06b629fdbcc83f6.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/4210-c8dd2d790233c3cc.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/9633-6bc6b50128d81548.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/4477-c133a61fe09fe714.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/7418-901562356cac876e.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/1190-ec52796f1b3aadac.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/8558-fe58da3bbf2ca2b0.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/1785-776435c2ae403e04.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/8812-64189dc81f86ae25.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/1093-91278d6800533446.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/chunks/pages/%5B...path%5D-faf574cb0fbad4d3.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/62phW-AXuCLFFYt9M070l/_buildManifest.js" defer=""></script><script src="/web/20240227045235js_/https://www.jisc.ac.uk/_next/static/62phW-AXuCLFFYt9M070l/_ssgManifest.js" defer=""></script><style id="jss-server-side">.MuiPaper-root { color: #333; transition: box-shadow 300ms cubic-bezier(0.4, 0, 0.2, 1) 0ms; background-color: #fff; } .MuiPaper-rounded { border-radius: 4px; } .MuiPaper-outlined { border: 1px solid rgba(0, 0, 0, 0.12); } .MuiPaper-elevation0 { box-shadow: none; } .MuiPaper-elevation1 { box-shadow: 0px 2px 1px -1px rgba(0,0,0,0.2),0px 1px 1px 0px rgba(0,0,0,0.14),0px 1px 3px 0px rgba(0,0,0,0.12); } .MuiPaper-elevation2 { box-shadow: 0px 3px 1px -2px rgba(0,0,0,0.2),0px 2px 2px 0px rgba(0,0,0,0.14),0px 1px 5px 0px rgba(0,0,0,0.12); } .MuiPaper-elevation3 { box-shadow: 0px 3px 3px -2px rgba(0,0,0,0.2),0px 3px 4px 0px rgba(0,0,0,0.14),0px 1px 8px 0px rgba(0,0,0,0.12); } .MuiPaper-elevation4 { box-shadow: 0px 2px 4px -1px rgba(0,0,0,0.2),0px 4px 5px 0px rgba(0,0,0,0.14),0px 1px 10px 0px rgba(0,0,0,0.12); } .MuiPaper-elevation5 { box-shadow: 0px 3px 5px -1px rgba(0,0,0,0.2),0px 5px 8px 0px rgba(0,0,0,0.14),0px 1px 14px 0px rgba(0,0,0,0.12); } .MuiPaper-elevation6 { box-shadow: 0px 3px 5px -1px rgba(0,0,0,0.2),0px 6px 10px 0px rgba(0,0,0,0.14),0px 1px 18px 0px rgba(0,0,0,0.12); } .MuiPaper-elevation7 { box-shadow: 0px 4px 5px -2px rgba(0,0,0,0.2),0px 7px 10px 1px rgba(0,0,0,0.14),0px 2px 16px 1px rgba(0,0,0,0.12); } .MuiPaper-elevation8 { box-shadow: 0px 5px 5px -3px rgba(0,0,0,0.2),0px 8px 10px 1px rgba(0,0,0,0.14),0px 3px 14px 2px rgba(0,0,0,0.12); } .MuiPaper-elevation9 { box-shadow: 0px 5px 6px -3px rgba(0,0,0,0.2),0px 9px 12px 1px rgba(0,0,0,0.14),0px 3px 16px 2px rgba(0,0,0,0.12); } .MuiPaper-elevation10 { box-shadow: 0px 6px 6px -3px rgba(0,0,0,0.2),0px 10px 14px 1px rgba(0,0,0,0.14),0px 4px 18px 3px rgba(0,0,0,0.12); } .MuiPaper-elevation11 { box-shadow: 0px 6px 7px -4px rgba(0,0,0,0.2),0px 11px 15px 1px rgba(0,0,0,0.14),0px 4px 20px 3px rgba(0,0,0,0.12); } .MuiPaper-elevation12 { box-shadow: 0px 7px 8px -4px rgba(0,0,0,0.2),0px 12px 17px 2px rgba(0,0,0,0.14),0px 5px 22px 4px rgba(0,0,0,0.12); } .MuiPaper-elevation13 { box-shadow: 0px 7px 8px -4px rgba(0,0,0,0.2),0px 13px 19px 2px rgba(0,0,0,0.14),0px 5px 24px 4px rgba(0,0,0,0.12); } .MuiPaper-elevation14 { box-shadow: 0px 7px 9px -4px rgba(0,0,0,0.2),0px 14px 21px 2px rgba(0,0,0,0.14),0px 5px 26px 4px rgba(0,0,0,0.12); } .MuiPaper-elevation15 { box-shadow: 0px 8px 9px -5px rgba(0,0,0,0.2),0px 15px 22px 2px rgba(0,0,0,0.14),0px 6px 28px 5px rgba(0,0,0,0.12); } .MuiPaper-elevation16 { box-shadow: 0px 8px 10px -5px rgba(0,0,0,0.2),0px 16px 24px 2px rgba(0,0,0,0.14),0px 6px 30px 5px rgba(0,0,0,0.12); } .MuiPaper-elevation17 { box-shadow: 0px 8px 11px -5px rgba(0,0,0,0.2),0px 17px 26px 2px rgba(0,0,0,0.14),0px 6px 32px 5px rgba(0,0,0,0.12); } .MuiPaper-elevation18 { box-shadow: 0px 9px 11px -5px rgba(0,0,0,0.2),0px 18px 28px 2px rgba(0,0,0,0.14),0px 7px 34px 6px rgba(0,0,0,0.12); } .MuiPaper-elevation19 { box-shadow: 0px 9px 12px -6px rgba(0,0,0,0.2),0px 19px 29px 2px rgba(0,0,0,0.14),0px 7px 36px 6px rgba(0,0,0,0.12); } .MuiPaper-elevation20 { box-shadow: 0px 10px 13px -6px rgba(0,0,0,0.2),0px 20px 31px 3px rgba(0,0,0,0.14),0px 8px 38px 7px rgba(0,0,0,0.12); } .MuiPaper-elevation21 { box-shadow: 0px 10px 13px -6px rgba(0,0,0,0.2),0px 21px 33px 3px rgba(0,0,0,0.14),0px 8px 40px 7px rgba(0,0,0,0.12); } .MuiPaper-elevation22 { box-shadow: 0px 10px 14px -6px rgba(0,0,0,0.2),0px 22px 35px 3px rgba(0,0,0,0.14),0px 8px 42px 7px rgba(0,0,0,0.12); } .MuiPaper-elevation23 { box-shadow: 0px 11px 14px -7px rgba(0,0,0,0.2),0px 23px 36px 3px rgba(0,0,0,0.14),0px 9px 44px 8px rgba(0,0,0,0.12); } .MuiPaper-elevation24 { box-shadow: 0px 11px 15px -7px rgba(0,0,0,0.2),0px 24px 38px 3px rgba(0,0,0,0.14),0px 9px 46px 8px rgba(0,0,0,0.12); } .MuiTypography-root { margin: 0; } .MuiTypography-body2 { font-size: .875rem; font-family: "Roboto", "Helvetica", "Arial", sans-serif; font-weight: 400; line-height: 1.428571428571429rem; letter-spacing: unset; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } .MuiTypography-body2 b { font-weight: bolder; } .MuiTypography-body2 strong { font-weight: bolder; } .MuiTypography-body1 { font-size: 1rem; font-family: "Roboto", "Helvetica", "Arial", sans-serif; font-weight: 400; line-height: 1.5rem; letter-spacing: unset; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } .MuiTypography-body1 b { font-weight: bolder; } .MuiTypography-body1 strong { font-weight: bolder; } .MuiTypography-caption { font-size: 0.75rem; font-family: "Roboto", "Helvetica", "Arial", sans-serif; font-weight: 400; line-height: 1.66; letter-spacing: 0.03333em; } .MuiTypography-button { font-size: 0.875rem; font-family: "Roboto", "Helvetica", "Arial", sans-serif; font-weight: 500; line-height: 1.75; letter-spacing: 0.02857em; text-transform: uppercase; } .MuiTypography-h1 { font-size: 2.5rem; font-family: "Roboto", "Helvetica", "Arial", sans-serif; font-weight: 400; line-height: 3.5rem; letter-spacing: unset; } @media (min-width:600px) { .MuiTypography-h1 { font-size: 2.75rem; line-height: 3.5rem; } } @media (min-width:960px) { .MuiTypography-h1 { font-size: 3rem; line-height: 4rem; } } .MuiTypography-h2 { font-size: 2rem; font-family: "Roboto", "Helvetica", "Arial", sans-serif; font-weight: 400; line-height: 3rem; letter-spacing: unset; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } @media (min-width:600px) { .MuiTypography-h2 { font-size: 2.25rem; line-height: 3rem; } } @media (min-width:960px) { .MuiTypography-h2 { font-size: 2.5rem; line-height: 3.5rem; } } .MuiTypography-h2 b { font-weight: bolder; } .MuiTypography-h2 strong { font-weight: bolder; } .MuiTypography-h3 { font-size: 1.5rem; font-family: "Roboto", "Helvetica", "Arial", sans-serif; font-weight: 500; line-height: 2rem; letter-spacing: unset; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } @media (min-width:600px) { .MuiTypography-h3 { font-size: 1.75rem; line-height: 2.5rem; } } @media (min-width:960px) { .MuiTypography-h3 { font-size: 2rem; line-height: 3rem; } } .MuiTypography-h3 b { font-weight: bolder; } .MuiTypography-h3 strong { font-weight: bolder; } .MuiTypography-h4 { font-size: 1.25rem; font-family: "Roboto", "Helvetica", "Arial", sans-serif; font-weight: 400; line-height: 2rem; letter-spacing: unset; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } @media (min-width:960px) { .MuiTypography-h4 { font-size: 1.5rem; line-height: 2rem; } } .MuiTypography-h4 b { font-weight: bolder; } .MuiTypography-h4 strong { font-weight: bolder; } .MuiTypography-h5 { font-size: 1rem; font-family: "Roboto", "Helvetica", "Arial", sans-serif; font-weight: 500; line-height: 1.5rem; letter-spacing: unset; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } @media (min-width:960px) { .MuiTypography-h5 { font-size: 1.25rem; line-height: 2rem; } } .MuiTypography-h5 b { font-weight: bolder; } .MuiTypography-h5 strong { font-weight: bolder; } .MuiTypography-h6 { font-size: .875rem; font-family: "Roboto", "Helvetica", "Arial", sans-serif; font-weight: 700; line-height: 1.5rem; letter-spacing: unset; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } .MuiTypography-h6 b { font-weight: bolder; } .MuiTypography-h6 strong { font-weight: bolder; } .MuiTypography-subtitle1 { font-size: 1rem; font-family: "Roboto", "Helvetica", "Arial", sans-serif; font-weight: 400; line-height: 1.75; letter-spacing: 0.00938em; } .MuiTypography-subtitle2 { font-size: 1.25rem; font-family: "Roboto", "Helvetica", "Arial", sans-serif; font-weight: 400; line-height: 2rem; letter-spacing: unset; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } .MuiTypography-subtitle2 b { font-weight: bolder; } .MuiTypography-subtitle2 strong { font-weight: bolder; } .MuiTypography-overline { font-size: 0.75rem; font-family: "Roboto", "Helvetica", "Arial", sans-serif; font-weight: 400; line-height: 2.66; letter-spacing: 0.08333em; text-transform: uppercase; } .MuiTypography-srOnly { width: 1px; height: 1px; overflow: hidden; position: absolute; } .MuiTypography-alignLeft { text-align: left; } .MuiTypography-alignCenter { text-align: center; } .MuiTypography-alignRight { text-align: right; } .MuiTypography-alignJustify { text-align: justify; } .MuiTypography-noWrap { overflow: hidden; white-space: nowrap; text-overflow: ellipsis; } .MuiTypography-gutterBottom { margin-bottom: 1.5rem; } .MuiTypography-paragraph { margin-bottom: 16px; } .MuiTypography-colorInherit { color: inherit; } .MuiTypography-colorPrimary { color: #007dba; } .MuiTypography-colorSecondary { color: #00857d; } .MuiTypography-colorTextPrimary { color: #333; } .MuiTypography-colorTextSecondary { color: #666; } .MuiTypography-colorError { color: #f44336; } .MuiTypography-displayInline { display: inline; } .MuiTypography-displayBlock { display: block; } .MuiContainer-root { width: 100%; display: block; box-sizing: border-box; margin-left: auto; margin-right: auto; padding-left: 1rem; padding-right: 1rem; } @media (min-width:600px) { .MuiContainer-root { padding-left: 1.5rem; padding-right: 1.5rem; } } .MuiContainer-disableGutters { padding-left: 0; padding-right: 0; } @media (min-width:600px) { .MuiContainer-fixed { max-width: 600px; } } @media (min-width:960px) { .MuiContainer-fixed { max-width: 960px; } } @media (min-width:1280px) { .MuiContainer-fixed { max-width: 1280px; } } @media (min-width:1920px) { .MuiContainer-fixed { max-width: 1920px; } } @media (min-width:0px) { .MuiContainer-maxWidthXs { max-width: 444px; } } @media (min-width:600px) { .MuiContainer-maxWidthSm { max-width: 600px; } } @media (min-width:960px) { .MuiContainer-maxWidthMd { max-width: 960px; } } @media (min-width:1280px) { .MuiContainer-maxWidthLg { max-width: 1280px; } } @media (min-width:1920px) { .MuiContainer-maxWidthXl { max-width: 1920px; } } .MuiGrid-container { width: 100%; display: flex; flex-wrap: wrap; box-sizing: border-box; } .MuiGrid-item { margin: 0; box-sizing: border-box; } .MuiGrid-zeroMinWidth { min-width: 0; } .MuiGrid-direction-xs-column { flex-direction: column; } .MuiGrid-direction-xs-column-reverse { flex-direction: column-reverse; } .MuiGrid-direction-xs-row-reverse { flex-direction: row-reverse; } .MuiGrid-wrap-xs-nowrap { flex-wrap: nowrap; } .MuiGrid-wrap-xs-wrap-reverse { flex-wrap: wrap-reverse; } .MuiGrid-align-items-xs-center { align-items: center; } .MuiGrid-align-items-xs-flex-start { align-items: flex-start; } .MuiGrid-align-items-xs-flex-end { align-items: flex-end; } .MuiGrid-align-items-xs-baseline { align-items: baseline; } .MuiGrid-align-content-xs-center { align-content: center; } .MuiGrid-align-content-xs-flex-start { align-content: flex-start; } .MuiGrid-align-content-xs-flex-end { align-content: flex-end; } .MuiGrid-align-content-xs-space-between { align-content: space-between; } .MuiGrid-align-content-xs-space-around { align-content: space-around; } .MuiGrid-justify-content-xs-center { justify-content: center; } .MuiGrid-justify-content-xs-flex-end { justify-content: flex-end; } .MuiGrid-justify-content-xs-space-between { justify-content: space-between; } .MuiGrid-justify-content-xs-space-around { justify-content: space-around; } .MuiGrid-justify-content-xs-space-evenly { justify-content: space-evenly; } .MuiGrid-spacing-xs-0 { width: calc(100% + 0rem); margin: -0rem; } .MuiGrid-spacing-xs-0 > .MuiGrid-item { padding: 0rem; } .MuiGrid-spacing-xs-1 { width: calc(100% + 0.5rem); margin: -0.25rem; } .MuiGrid-spacing-xs-1 > .MuiGrid-item { padding: 0.25rem; } .MuiGrid-spacing-xs-2 { width: calc(100% + 1rem); margin: -0.5rem; } .MuiGrid-spacing-xs-2 > .MuiGrid-item { padding: 0.5rem; } .MuiGrid-spacing-xs-3 { width: calc(100% + 1.5rem); margin: -0.75rem; } .MuiGrid-spacing-xs-3 > .MuiGrid-item { padding: 0.75rem; } .MuiGrid-spacing-xs-4 { width: calc(100% + 2rem); margin: -1rem; } .MuiGrid-spacing-xs-4 > .MuiGrid-item { padding: 1rem; } .MuiGrid-spacing-xs-5 { width: calc(100% + 2.5rem); margin: -1.25rem; } .MuiGrid-spacing-xs-5 > .MuiGrid-item { padding: 1.25rem; } .MuiGrid-spacing-xs-6 { width: calc(100% + 3rem); margin: -1.5rem; } .MuiGrid-spacing-xs-6 > .MuiGrid-item { padding: 1.5rem; } .MuiGrid-spacing-xs-7 { width: calc(100% + 3.5rem); margin: -1.75rem; } .MuiGrid-spacing-xs-7 > .MuiGrid-item { padding: 1.75rem; } .MuiGrid-spacing-xs-8 { width: calc(100% + 4rem); margin: -2rem; } .MuiGrid-spacing-xs-8 > .MuiGrid-item { padding: 2rem; } .MuiGrid-spacing-xs-9 { width: calc(100% + 4.5rem); margin: -2.25rem; } .MuiGrid-spacing-xs-9 > .MuiGrid-item { padding: 2.25rem; } .MuiGrid-spacing-xs-10 { width: calc(100% + 5rem); margin: -2.5rem; } .MuiGrid-spacing-xs-10 > .MuiGrid-item { padding: 2.5rem; } .MuiGrid-grid-xs-auto { flex-grow: 0; max-width: none; flex-basis: auto; } .MuiGrid-grid-xs-true { flex-grow: 1; max-width: 100%; flex-basis: 0; } .MuiGrid-grid-xs-1 { flex-grow: 0; max-width: 8.333333%; flex-basis: 8.333333%; } .MuiGrid-grid-xs-2 { flex-grow: 0; max-width: 16.666667%; flex-basis: 16.666667%; } .MuiGrid-grid-xs-3 { flex-grow: 0; max-width: 25%; flex-basis: 25%; } .MuiGrid-grid-xs-4 { flex-grow: 0; max-width: 33.333333%; flex-basis: 33.333333%; } .MuiGrid-grid-xs-5 { flex-grow: 0; max-width: 41.666667%; flex-basis: 41.666667%; } .MuiGrid-grid-xs-6 { flex-grow: 0; max-width: 50%; flex-basis: 50%; } .MuiGrid-grid-xs-7 { flex-grow: 0; max-width: 58.333333%; flex-basis: 58.333333%; } .MuiGrid-grid-xs-8 { flex-grow: 0; max-width: 66.666667%; flex-basis: 66.666667%; } .MuiGrid-grid-xs-9 { flex-grow: 0; max-width: 75%; flex-basis: 75%; } .MuiGrid-grid-xs-10 { flex-grow: 0; max-width: 83.333333%; flex-basis: 83.333333%; } .MuiGrid-grid-xs-11 { flex-grow: 0; max-width: 91.666667%; flex-basis: 91.666667%; } .MuiGrid-grid-xs-12 { flex-grow: 0; max-width: 100%; flex-basis: 100%; } @media (min-width:600px) { .MuiGrid-grid-sm-auto { flex-grow: 0; max-width: none; flex-basis: auto; } .MuiGrid-grid-sm-true { flex-grow: 1; max-width: 100%; flex-basis: 0; } .MuiGrid-grid-sm-1 { flex-grow: 0; max-width: 8.333333%; flex-basis: 8.333333%; } .MuiGrid-grid-sm-2 { flex-grow: 0; max-width: 16.666667%; flex-basis: 16.666667%; } .MuiGrid-grid-sm-3 { flex-grow: 0; max-width: 25%; flex-basis: 25%; } .MuiGrid-grid-sm-4 { flex-grow: 0; max-width: 33.333333%; flex-basis: 33.333333%; } .MuiGrid-grid-sm-5 { flex-grow: 0; max-width: 41.666667%; flex-basis: 41.666667%; } .MuiGrid-grid-sm-6 { flex-grow: 0; max-width: 50%; flex-basis: 50%; } .MuiGrid-grid-sm-7 { flex-grow: 0; max-width: 58.333333%; flex-basis: 58.333333%; } .MuiGrid-grid-sm-8 { flex-grow: 0; max-width: 66.666667%; flex-basis: 66.666667%; } .MuiGrid-grid-sm-9 { flex-grow: 0; max-width: 75%; flex-basis: 75%; } .MuiGrid-grid-sm-10 { flex-grow: 0; max-width: 83.333333%; flex-basis: 83.333333%; } .MuiGrid-grid-sm-11 { flex-grow: 0; max-width: 91.666667%; flex-basis: 91.666667%; } .MuiGrid-grid-sm-12 { flex-grow: 0; max-width: 100%; flex-basis: 100%; } } @media (min-width:960px) { .MuiGrid-grid-md-auto { flex-grow: 0; max-width: none; flex-basis: auto; } .MuiGrid-grid-md-true { flex-grow: 1; max-width: 100%; flex-basis: 0; } .MuiGrid-grid-md-1 { flex-grow: 0; max-width: 8.333333%; flex-basis: 8.333333%; } .MuiGrid-grid-md-2 { flex-grow: 0; max-width: 16.666667%; flex-basis: 16.666667%; } .MuiGrid-grid-md-3 { flex-grow: 0; max-width: 25%; flex-basis: 25%; } .MuiGrid-grid-md-4 { flex-grow: 0; max-width: 33.333333%; flex-basis: 33.333333%; } .MuiGrid-grid-md-5 { flex-grow: 0; max-width: 41.666667%; flex-basis: 41.666667%; } .MuiGrid-grid-md-6 { flex-grow: 0; max-width: 50%; flex-basis: 50%; } .MuiGrid-grid-md-7 { flex-grow: 0; max-width: 58.333333%; flex-basis: 58.333333%; } .MuiGrid-grid-md-8 { flex-grow: 0; max-width: 66.666667%; flex-basis: 66.666667%; } .MuiGrid-grid-md-9 { flex-grow: 0; max-width: 75%; flex-basis: 75%; } .MuiGrid-grid-md-10 { flex-grow: 0; max-width: 83.333333%; flex-basis: 83.333333%; } .MuiGrid-grid-md-11 { flex-grow: 0; max-width: 91.666667%; flex-basis: 91.666667%; } .MuiGrid-grid-md-12 { flex-grow: 0; max-width: 100%; flex-basis: 100%; } } @media (min-width:1280px) { .MuiGrid-grid-lg-auto { flex-grow: 0; max-width: none; flex-basis: auto; } .MuiGrid-grid-lg-true { flex-grow: 1; max-width: 100%; flex-basis: 0; } .MuiGrid-grid-lg-1 { flex-grow: 0; max-width: 8.333333%; flex-basis: 8.333333%; } .MuiGrid-grid-lg-2 { flex-grow: 0; max-width: 16.666667%; flex-basis: 16.666667%; } .MuiGrid-grid-lg-3 { flex-grow: 0; max-width: 25%; flex-basis: 25%; } .MuiGrid-grid-lg-4 { flex-grow: 0; max-width: 33.333333%; flex-basis: 33.333333%; } .MuiGrid-grid-lg-5 { flex-grow: 0; max-width: 41.666667%; flex-basis: 41.666667%; } .MuiGrid-grid-lg-6 { flex-grow: 0; max-width: 50%; flex-basis: 50%; } .MuiGrid-grid-lg-7 { flex-grow: 0; max-width: 58.333333%; flex-basis: 58.333333%; } .MuiGrid-grid-lg-8 { flex-grow: 0; max-width: 66.666667%; flex-basis: 66.666667%; } .MuiGrid-grid-lg-9 { flex-grow: 0; max-width: 75%; flex-basis: 75%; } .MuiGrid-grid-lg-10 { flex-grow: 0; max-width: 83.333333%; flex-basis: 83.333333%; } .MuiGrid-grid-lg-11 { flex-grow: 0; max-width: 91.666667%; flex-basis: 91.666667%; } .MuiGrid-grid-lg-12 { flex-grow: 0; max-width: 100%; flex-basis: 100%; } } @media (min-width:1920px) { .MuiGrid-grid-xl-auto { flex-grow: 0; max-width: none; flex-basis: auto; } .MuiGrid-grid-xl-true { flex-grow: 1; max-width: 100%; flex-basis: 0; } .MuiGrid-grid-xl-1 { flex-grow: 0; max-width: 8.333333%; flex-basis: 8.333333%; } .MuiGrid-grid-xl-2 { flex-grow: 0; max-width: 16.666667%; flex-basis: 16.666667%; } .MuiGrid-grid-xl-3 { flex-grow: 0; max-width: 25%; flex-basis: 25%; } .MuiGrid-grid-xl-4 { flex-grow: 0; max-width: 33.333333%; flex-basis: 33.333333%; } .MuiGrid-grid-xl-5 { flex-grow: 0; max-width: 41.666667%; flex-basis: 41.666667%; } .MuiGrid-grid-xl-6 { flex-grow: 0; max-width: 50%; flex-basis: 50%; } .MuiGrid-grid-xl-7 { flex-grow: 0; max-width: 58.333333%; flex-basis: 58.333333%; } .MuiGrid-grid-xl-8 { flex-grow: 0; max-width: 66.666667%; flex-basis: 66.666667%; } .MuiGrid-grid-xl-9 { flex-grow: 0; max-width: 75%; flex-basis: 75%; } .MuiGrid-grid-xl-10 { flex-grow: 0; max-width: 83.333333%; flex-basis: 83.333333%; } .MuiGrid-grid-xl-11 { flex-grow: 0; max-width: 91.666667%; flex-basis: 91.666667%; } .MuiGrid-grid-xl-12 { flex-grow: 0; max-width: 100%; flex-basis: 100%; } } .jss2 { color: #fff; z-index: 1300; position: relative; padding-top: 1.5rem; padding-bottom: 1.5rem; background-color: #0d224c; } @media (max-width:599.95px) { .jss3 { display: flex; flex-wrap: wrap; align-items: center; justify-content: space-between; } } @media (min-width:600px) { .jss3 { flex-shrink: 0; } } @media (max-width:599.95px) { .jss4 { margin-top: 1.5rem; flex-shrink: 0; } } @media (min-width:600px) { .jss5 { display: flex; align-items: center; justify-content: space-between; } } @media (min-width:600px) { .jss6 { margin-right: 1.5rem; } } @media (max-width:599.95px) { .jss7 { margin-top: 1.5rem; flex-shrink: 0; margin-right: 1rem; } } @media (min-width:600px) { .jss7 { margin-right: 1.5rem; } } .jss8 { margin-top: 1rem; } .jss1 { clip: rect(0 0 0 0); color: #000; width: 1px; height: 1px; display: block; overflow: hidden; position: absolute; clip-path: inset(50%); padding-top: 0.5rem; white-space: nowrap; padding-bottom: 0.5rem; text-decoration: unset; background-color: #fd6; } .jss1:focus { clip: unset; width: unset; height: unset; outline: .25rem solid transparent; overflow: unset; position: unset; clip-path: unset; white-space: unset; } .jss35 > * { box-sizing: border-box; font-family: Roboto; } .jss36 { padding-top: 1.5rem; padding-bottom: 2.5rem; } .jss37 { padding-top: 2.5rem; padding-bottom: 2.5rem; } .jss38 { padding-top: 1.5rem; padding-bottom: 2.5rem; } .jss39 { color: #fff; } .jss40 { background-color: #4c4c4c; } .jss41 { background-color: #404040; } .jss42 { background-color: #333; } .jss43 { display: inline; } .jss44:link { color: #fff; text-decoration: none; } .jss44:visited { color: #fff; } .jss44:focus { color: #000; outline: .25rem solid transparent; box-shadow: 0 .25rem 0 0 #fd6, 0 -.25rem 0 0 #fd6; text-decoration: none; background-color: #fd6; } .jss44:hover { color: #d9d9d9; transition: color .2s ease; text-decoration: underline; } .jss44:focus:hover { color: #000; outline: .25rem solid transparent; box-shadow: 0 .25rem 0 0 #fd6, 0 -.25rem 0 0 #fd6; text-decoration: none; background-color: #fd6; } .jss44:active { color: #fff; } .jss44:focus:active { color: #fff; outline: none; box-shadow: none; text-decoration: underline; background-color: transparent; } .jss45 { margin: 0; display: inline; padding: 0; list-style: none; } .jss46 { display: inline-block; font-size: .875rem; margin-top: 1rem; font-weight: 400; line-height: 1.428571428571429rem; letter-spacing: unset; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } .jss46 b { font-weight: bolder; } .jss46 strong { font-weight: bolder; } .jss46:not(:last-child) { margin-right: 1rem; } .jss47::before { content: ">"; margin-right: 1rem; } .jss48 { width: calc(100% + 1.5rem); margin-left: -0.75rem; margin-right: -0.75rem; } .jss48 .MuiGrid-item { padding-left: 0.75rem; padding-right: 0.75rem; } @media (max-width:959.95px) { .jss48 { width: calc(100% + 1rem); margin-left: -0.5rem; margin-right: -0.5rem; } .jss48 .MuiGrid-item { padding-left: 0.5rem; padding-right: 0.5rem; } } .jss49 { padding-top: 0; padding-bottom: 0; } .jss50 { font-size: .875rem; font-weight: 400; line-height: 1.428571428571429rem; padding-top: 1rem; letter-spacing: unset; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } .jss50 b { font-weight: bolder; } .jss50 strong { font-weight: bolder; } @media (max-width:599.95px) { .jss50 { border-top: .0625rem solid #707070; padding-top: 1.5rem; } } .jss51 { list-style: none; margin-top: 1.5rem; padding-left: 0; margin-bottom: 0; } @media (max-width:599.95px) { .jss52 { margin-top: 3rem; } } .jss53 { font-size: 1rem; font-family: "Roboto", "Helvetica", "Arial", sans-serif; font-weight: 500; line-height: 1.5rem; letter-spacing: unset; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } @media (min-width:960px) { .jss53 { font-size: 1.25rem; line-height: 2rem; } } .jss53 b { font-weight: bolder; } .jss53 strong { font-weight: bolder; } .jss54 { font-size: .875rem; font-weight: 400; line-height: 1.428571428571429rem; letter-spacing: unset; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } .jss54 b { font-weight: bolder; } .jss54 strong { font-weight: bolder; } .jss54:not(:first-child) { margin-top: 1rem; } .jss55 { margin-right: 0.5rem; } .jss56 { display: flex; flex-wrap: wrap; list-style: none; margin-top: 0; padding-left: 0; margin-bottom: 0; } .jss57 { margin-bottom: 0; } @media (max-width:599.95px) { .jss57 { margin-bottom: 1.5rem; } } .jss58 { font-size: .875rem; margin-top: 1rem; font-weight: 400; line-height: 1.428571428571429rem; letter-spacing: unset; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } .jss58 b { font-weight: bolder; } .jss58 strong { font-weight: bolder; } .jss58:not(:last-child) { margin-right: 1.5rem; } .jss59 { font-size: .875rem; font-weight: 400; line-height: 1.428571428571429rem; padding-top: 1.5rem; letter-spacing: unset; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } .jss59 b { font-weight: bolder; } .jss59 strong { font-weight: bolder; } @media (max-width:599.95px) { .jss59 { border-top: .0625rem solid #707070; } } .jss60:link { color: #fff; text-decoration: underline; } .jss60:visited { color: #d9d9d9; } .jss60:focus { color: #000; outline: .25rem solid transparent; box-shadow: 0 .25rem 0 0 #fd6, 0 -.25rem 0 0 #fd6; text-decoration: none; background-color: #fd6; } .jss60:hover { color: #d9d9d9; transition: color .2s ease; text-decoration: underline; } .jss60:focus:hover { color: #000; outline: .25rem solid transparent; box-shadow: 0 .25rem 0 0 #fd6, 0 -.25rem 0 0 #fd6; text-decoration: none; background-color: #fd6; } .jss60:active { color: #fff; } .jss60:focus:active { color: #fff; outline: none; box-shadow: none; text-decoration: underline; background-color: transparent; } .jss9 { z-index: 1100; position: relative; } .jss10 { color: #666; background-color: #fff; } @media (min-width:600px) { .jss10 { padding-top: 0.5rem; padding-bottom: 0.5rem; } } @media (max-width:599.95px) { .jss11::before { content: ''; display: table; } .jss11::after { clear: both; content: ''; display: table; } } @media (min-width:600px) { .jss11 { display: flex; justify-content: space-between; } } @media (max-width:599.95px) { .jss12 { float: left; } } .jss13 { display: flex; } .jss14 { line-height: 0; } .jss15 { margin-left: 16px; } @media (max-width:599.95px) { .jss15 { margin-top: 1rem; } } @media (min-width:600px) { .jss15 { margin-top: .625rem; } } @media (max-width:599.95px) { .jss16 { font-size: 1rem; font-family: "Roboto", "Helvetica", "Arial", sans-serif; font-weight: 400; line-height: 1.5rem; letter-spacing: unset; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } .jss16 b { font-weight: bolder; } .jss16 strong { font-weight: bolder; } } @media (min-width:600px) { .jss16 { font-size: 1.25rem; font-family: "Roboto", "Helvetica", "Arial", sans-serif; font-weight: 400; line-height: 2rem; letter-spacing: unset; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } .jss16 b { font-weight: bolder; } .jss16 strong { font-weight: bolder; } } .jss17 { margin-right: 3px; } @media (max-width:599.95px) { .jss18 { clip: rect(0 0 0 0); width: 1px; height: 1px; overflow: hidden; position: absolute; clip-path: inset(50%); white-space: nowrap; } } @media (min-width:600px) { .jss18 { font-size: 1.25rem; font-family: "Roboto", "Helvetica", "Arial", sans-serif; font-weight: 400; line-height: 2rem; margin-left: 3px; letter-spacing: unset; -webkit-font-smoothing: antialiased; -moz-osx-font-smoothing: grayscale; } .jss18 b { font-weight: bolder; } .jss18 strong { font-weight: bolder; } } .jss19 { clip: rect(0 0 0 0); width: 1px; height: 1px; overflow: hidden; position: absolute; clip-path: inset(50%); white-space: nowrap; } .jss20:link { color: #666; text-decoration: unset; } .jss20:visited { color: #666; } .jss20:focus { color: #000; outline: .25rem solid transparent; box-shadow: 0 .25rem 0 0 #fd6, 0 -.25rem 0 0 #fd6; text-decoration: none; background-color: #fd6; } .jss20:focus:not(:focus-visible) { outline: .25rem solid transparent; box-shadow: unset; text-decoration: underline; background-color: unset; } .jss20:hover { color: #004777; text-decoration: underline; } .jss20:active { color: #000; text-decoration: underline; } @media (hover: none) { .jss20:hover { color: #666; } } @media (max-width:599.95px) { .jss21 { width: 2.5rem; height: 2.5rem; } } @media (min-width:600px) { .jss21 { width: 3.25rem; height: 3.25rem; } } .jss22 { display: inline-block; } @media (max-width:599.95px) { .jss22 { margin-top: 0.5rem; margin-bottom: 0.5rem; } } .jss22:focus { outline: .25rem solid #fd6; } .jss22:focus:not(:focus-visible) { outline: none; } .jss22:hover { opacity: 0.7; } .jss22:active { opacity: 1; } @media (hover: none) { .jss22:hover { opacity: 1; } } .jss23 { margin-left: 8px; } @media (max-width:599.95px) { .jss23 { float: right; margin-top: .625rem; margin-bottom: 0.5rem; } } .jss26 { display: flex; position: relative; align-items: center; margin-left: auto; } @media (max-width: 757px) { .jss26 { top: 100%; right: 0; display: none; padding: 1.8rem; position: absolute; border-top: 1px solid #ccc; box-shadow: 0px 5px 4px rgb(0 0 0 / 20%); -moz-box-shadow: 0px 5px 4px rgba(0,0,0,0.2); background-color: #fff; -webkit-box-shadow: 0px 5px 4px rgb(0 0 0 / 20%); } } @media (max-width: 400px) { .jss26 { left: 0; } } @media (max-width: 757px) { .jss26.visible { display: block; } } .jss27 { color: #666; display: block; font-size: 18px; text-decoration: unset; } .jss27:hover { color: #004777; text-decoration: underline; } .jss27:focus { color: #000; outline: transparent solid 0.25rem; box-shadow: rgb(255 221 102) 0px 0.25rem 0px 0px, rgb(255 221 102) 0px -0.25rem 0px 0px; text-decoration: none; background-color: rgb(255, 221, 102); } .jss28 { top: 100%; right: 0; width: 600px; display: none; padding: 20px 30px; position: absolute; border-top: 2px solid #00557f; box-shadow: 0px 5px 4px rgb(0 0 0 / 20%); text-align: center; -moz-box-shadow: 0px 5px 4px rgba(0,0,0,0.2); background-color: #fff; -webkit-box-shadow: 0px 5px 4px rgb(0 0 0 / 20%); } @media (max-width: 757px) { .jss28 { display: none; } } .jss29 { top: 50%; color: #2c3841; right: 1.8rem; width: 32px; border: none; bottom: 0; cursor: pointer; height: 32px; display: flex; position: absolute; transform: translateY(-50%); align-items: center; justify-content: center; background-color: transparent; } @media (min-width: 757px) { .jss29 { right: 10px; } } .jss29:hover { opacity: 0.5; } .jss29:focus { outline: .25rem solid #fd6; background-color: #fff; } @media (max-width: 757px) { .jss30 { width: 100%; } } @media (min-width: 757px) { .jss30:focus-within #predictive-results.visible { display: block; } } @media (min-width: 757px) { .jss31 { padding: 10px 10px; position: relative; margin-right: -10px; } .jss31.has-search-term:focus-within { box-shadow: 0 0 2px rgb(0 0 0 / 20%); -moz-box-shadow: 0 0 2px rgba(0,0,0,0.2); background-image: linear-gradient(to bottom,#e4e9ec 0%,#fff 100%); -webkit-box-shadow: 0 0 2px rgb(0 0 0 / 20%); } } .jss32 { width: 210px; border: 2px solid #E4E9EC; display: block; padding: 5px 32px 4px 9px; min-height: 31px; background-color: #fff; } .jss32:focus { outline: .25rem solid #fd6; background-color: #fff; } .jss32::-webkit-search-cancel-button, .jss32::-webkit-search-decoration, .jss32 -webkit-search-results-button, .jss32 -webkit-search-results-decoration { display: none; } .jss32::-ms-clear { width: 0; height: 0; display: none; } .jss32::-ms-reveal { width: 0; height: 0; display: none; } .jss32:-webkit-autofill, .jss32:-webkit-autofill:hover, .jss32:-webkit-autofill:active, .jss32:-webkit-autofill:focus { -webkit-box-shadow: 0 0 0px 1000px #fff inset; } @media (max-width: 757px) { .jss32 { width: 100%; } } .jss33 span { clip: rect(0 0 0 0); width: 1px; height: 1px; overflow: hidden; position: absolute; clip-path: inset(50%); white-space: nowrap; } .jss24 { display: none; align-items: center; margin-left: auto; justify-content: center; } @media (max-width: 757px) { .jss24 { gap: 8px; display: flex; } } @media (min-width:600px) { .jss24 { margin-top: 0.5rem; } } .jss25 { color: #069; width: 36px; border: none; height: 36px; justify-content: center; background-color: #e4e9ec; } .jss25:hover { color: #fff; cursor: pointer; background-color: #069; } .jss25:focus { color: #fff; outline: .25rem solid #fd6; outline-offset: -2px; background-color: #069; }</style><style data-href="https://fonts.googleapis.com/css2?family=Roboto:ital,wght@0,300;0,400;0,500;0,700;0,900;1,300;1,400;1,500&display=swap">@font-face{font-family:'Roboto';font-style:italic;font-weight:300;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOjCnqEu92Fr1Mu51TjARc-.woff) format('woff')}@font-face{font-family:'Roboto';font-style:italic;font-weight:400;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOkCnqEu92Fr1Mu52xM.woff) format('woff')}@font-face{font-family:'Roboto';font-style:italic;font-weight:500;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOjCnqEu92Fr1Mu51S7ABc-.woff) format('woff')}@font-face{font-family:'Roboto';font-style:normal;font-weight:300;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmSU5vAA.woff) format('woff')}@font-face{font-family:'Roboto';font-style:normal;font-weight:400;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOmCnqEu92Fr1Me5g.woff) format('woff')}@font-face{font-family:'Roboto';font-style:normal;font-weight:500;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmEU9vAA.woff) format('woff')}@font-face{font-family:'Roboto';font-style:normal;font-weight:700;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmWUlvAA.woff) format('woff')}@font-face{font-family:'Roboto';font-style:normal;font-weight:900;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmYUtvAA.woff) format('woff')}@font-face{font-family:'Roboto';font-style:italic;font-weight:300;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOjCnqEu92Fr1Mu51TjASc3CsTYl4BOQ3o.woff2) format('woff2');unicode-range:U+0460-052F,U+1C80-1C88,U+20B4,U+2DE0-2DFF,U+A640-A69F,U+FE2E-FE2F}@font-face{font-family:'Roboto';font-style:italic;font-weight:300;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOjCnqEu92Fr1Mu51TjASc-CsTYl4BOQ3o.woff2) format('woff2');unicode-range:U+0301,U+0400-045F,U+0490-0491,U+04B0-04B1,U+2116}@font-face{font-family:'Roboto';font-style:italic;font-weight:300;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOjCnqEu92Fr1Mu51TjASc2CsTYl4BOQ3o.woff2) format('woff2');unicode-range:U+1F00-1FFF}@font-face{font-family:'Roboto';font-style:italic;font-weight:300;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOjCnqEu92Fr1Mu51TjASc5CsTYl4BOQ3o.woff2) format('woff2');unicode-range:U+0370-0377,U+037A-037F,U+0384-038A,U+038C,U+038E-03A1,U+03A3-03FF}@font-face{font-family:'Roboto';font-style:italic;font-weight:300;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOjCnqEu92Fr1Mu51TjASc1CsTYl4BOQ3o.woff2) format('woff2');unicode-range:U+0102-0103,U+0110-0111,U+0128-0129,U+0168-0169,U+01A0-01A1,U+01AF-01B0,U+0300-0301,U+0303-0304,U+0308-0309,U+0323,U+0329,U+1EA0-1EF9,U+20AB}@font-face{font-family:'Roboto';font-style:italic;font-weight:300;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOjCnqEu92Fr1Mu51TjASc0CsTYl4BOQ3o.woff2) format('woff2');unicode-range:U+0100-02AF,U+0304,U+0308,U+0329,U+1E00-1E9F,U+1EF2-1EFF,U+2020,U+20A0-20AB,U+20AD-20C0,U+2113,U+2C60-2C7F,U+A720-A7FF}@font-face{font-family:'Roboto';font-style:italic;font-weight:300;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOjCnqEu92Fr1Mu51TjASc6CsTYl4BO.woff2) format('woff2');unicode-range:U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,U+02C6,U+02DA,U+02DC,U+0304,U+0308,U+0329,U+2000-206F,U+2074,U+20AC,U+2122,U+2191,U+2193,U+2212,U+2215,U+FEFF,U+FFFD}@font-face{font-family:'Roboto';font-style:italic;font-weight:400;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOkCnqEu92Fr1Mu51xFIzIXKMnyrYk.woff2) format('woff2');unicode-range:U+0460-052F,U+1C80-1C88,U+20B4,U+2DE0-2DFF,U+A640-A69F,U+FE2E-FE2F}@font-face{font-family:'Roboto';font-style:italic;font-weight:400;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOkCnqEu92Fr1Mu51xMIzIXKMnyrYk.woff2) format('woff2');unicode-range:U+0301,U+0400-045F,U+0490-0491,U+04B0-04B1,U+2116}@font-face{font-family:'Roboto';font-style:italic;font-weight:400;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOkCnqEu92Fr1Mu51xEIzIXKMnyrYk.woff2) format('woff2');unicode-range:U+1F00-1FFF}@font-face{font-family:'Roboto';font-style:italic;font-weight:400;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOkCnqEu92Fr1Mu51xLIzIXKMnyrYk.woff2) format('woff2');unicode-range:U+0370-0377,U+037A-037F,U+0384-038A,U+038C,U+038E-03A1,U+03A3-03FF}@font-face{font-family:'Roboto';font-style:italic;font-weight:400;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOkCnqEu92Fr1Mu51xHIzIXKMnyrYk.woff2) format('woff2');unicode-range:U+0102-0103,U+0110-0111,U+0128-0129,U+0168-0169,U+01A0-01A1,U+01AF-01B0,U+0300-0301,U+0303-0304,U+0308-0309,U+0323,U+0329,U+1EA0-1EF9,U+20AB}@font-face{font-family:'Roboto';font-style:italic;font-weight:400;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOkCnqEu92Fr1Mu51xGIzIXKMnyrYk.woff2) format('woff2');unicode-range:U+0100-02AF,U+0304,U+0308,U+0329,U+1E00-1E9F,U+1EF2-1EFF,U+2020,U+20A0-20AB,U+20AD-20C0,U+2113,U+2C60-2C7F,U+A720-A7FF}@font-face{font-family:'Roboto';font-style:italic;font-weight:400;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOkCnqEu92Fr1Mu51xIIzIXKMny.woff2) format('woff2');unicode-range:U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,U+02C6,U+02DA,U+02DC,U+0304,U+0308,U+0329,U+2000-206F,U+2074,U+20AC,U+2122,U+2191,U+2193,U+2212,U+2215,U+FEFF,U+FFFD}@font-face{font-family:'Roboto';font-style:italic;font-weight:500;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOjCnqEu92Fr1Mu51S7ACc3CsTYl4BOQ3o.woff2) format('woff2');unicode-range:U+0460-052F,U+1C80-1C88,U+20B4,U+2DE0-2DFF,U+A640-A69F,U+FE2E-FE2F}@font-face{font-family:'Roboto';font-style:italic;font-weight:500;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOjCnqEu92Fr1Mu51S7ACc-CsTYl4BOQ3o.woff2) format('woff2');unicode-range:U+0301,U+0400-045F,U+0490-0491,U+04B0-04B1,U+2116}@font-face{font-family:'Roboto';font-style:italic;font-weight:500;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOjCnqEu92Fr1Mu51S7ACc2CsTYl4BOQ3o.woff2) format('woff2');unicode-range:U+1F00-1FFF}@font-face{font-family:'Roboto';font-style:italic;font-weight:500;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOjCnqEu92Fr1Mu51S7ACc5CsTYl4BOQ3o.woff2) format('woff2');unicode-range:U+0370-0377,U+037A-037F,U+0384-038A,U+038C,U+038E-03A1,U+03A3-03FF}@font-face{font-family:'Roboto';font-style:italic;font-weight:500;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOjCnqEu92Fr1Mu51S7ACc1CsTYl4BOQ3o.woff2) format('woff2');unicode-range:U+0102-0103,U+0110-0111,U+0128-0129,U+0168-0169,U+01A0-01A1,U+01AF-01B0,U+0300-0301,U+0303-0304,U+0308-0309,U+0323,U+0329,U+1EA0-1EF9,U+20AB}@font-face{font-family:'Roboto';font-style:italic;font-weight:500;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOjCnqEu92Fr1Mu51S7ACc0CsTYl4BOQ3o.woff2) format('woff2');unicode-range:U+0100-02AF,U+0304,U+0308,U+0329,U+1E00-1E9F,U+1EF2-1EFF,U+2020,U+20A0-20AB,U+20AD-20C0,U+2113,U+2C60-2C7F,U+A720-A7FF}@font-face{font-family:'Roboto';font-style:italic;font-weight:500;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOjCnqEu92Fr1Mu51S7ACc6CsTYl4BO.woff2) format('woff2');unicode-range:U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,U+02C6,U+02DA,U+02DC,U+0304,U+0308,U+0329,U+2000-206F,U+2074,U+20AC,U+2122,U+2191,U+2193,U+2212,U+2215,U+FEFF,U+FFFD}@font-face{font-family:'Roboto';font-style:normal;font-weight:300;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmSU5fCRc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0460-052F,U+1C80-1C88,U+20B4,U+2DE0-2DFF,U+A640-A69F,U+FE2E-FE2F}@font-face{font-family:'Roboto';font-style:normal;font-weight:300;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmSU5fABc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0301,U+0400-045F,U+0490-0491,U+04B0-04B1,U+2116}@font-face{font-family:'Roboto';font-style:normal;font-weight:300;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmSU5fCBc4AMP6lbBP.woff2) format('woff2');unicode-range:U+1F00-1FFF}@font-face{font-family:'Roboto';font-style:normal;font-weight:300;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmSU5fBxc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0370-0377,U+037A-037F,U+0384-038A,U+038C,U+038E-03A1,U+03A3-03FF}@font-face{font-family:'Roboto';font-style:normal;font-weight:300;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmSU5fCxc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0102-0103,U+0110-0111,U+0128-0129,U+0168-0169,U+01A0-01A1,U+01AF-01B0,U+0300-0301,U+0303-0304,U+0308-0309,U+0323,U+0329,U+1EA0-1EF9,U+20AB}@font-face{font-family:'Roboto';font-style:normal;font-weight:300;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmSU5fChc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0100-02AF,U+0304,U+0308,U+0329,U+1E00-1E9F,U+1EF2-1EFF,U+2020,U+20A0-20AB,U+20AD-20C0,U+2113,U+2C60-2C7F,U+A720-A7FF}@font-face{font-family:'Roboto';font-style:normal;font-weight:300;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmSU5fBBc4AMP6lQ.woff2) format('woff2');unicode-range:U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,U+02C6,U+02DA,U+02DC,U+0304,U+0308,U+0329,U+2000-206F,U+2074,U+20AC,U+2122,U+2191,U+2193,U+2212,U+2215,U+FEFF,U+FFFD}@font-face{font-family:'Roboto';font-style:normal;font-weight:400;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOmCnqEu92Fr1Mu72xKKTU1Kvnz.woff2) format('woff2');unicode-range:U+0460-052F,U+1C80-1C88,U+20B4,U+2DE0-2DFF,U+A640-A69F,U+FE2E-FE2F}@font-face{font-family:'Roboto';font-style:normal;font-weight:400;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOmCnqEu92Fr1Mu5mxKKTU1Kvnz.woff2) format('woff2');unicode-range:U+0301,U+0400-045F,U+0490-0491,U+04B0-04B1,U+2116}@font-face{font-family:'Roboto';font-style:normal;font-weight:400;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOmCnqEu92Fr1Mu7mxKKTU1Kvnz.woff2) format('woff2');unicode-range:U+1F00-1FFF}@font-face{font-family:'Roboto';font-style:normal;font-weight:400;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOmCnqEu92Fr1Mu4WxKKTU1Kvnz.woff2) format('woff2');unicode-range:U+0370-0377,U+037A-037F,U+0384-038A,U+038C,U+038E-03A1,U+03A3-03FF}@font-face{font-family:'Roboto';font-style:normal;font-weight:400;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOmCnqEu92Fr1Mu7WxKKTU1Kvnz.woff2) format('woff2');unicode-range:U+0102-0103,U+0110-0111,U+0128-0129,U+0168-0169,U+01A0-01A1,U+01AF-01B0,U+0300-0301,U+0303-0304,U+0308-0309,U+0323,U+0329,U+1EA0-1EF9,U+20AB}@font-face{font-family:'Roboto';font-style:normal;font-weight:400;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOmCnqEu92Fr1Mu7GxKKTU1Kvnz.woff2) format('woff2');unicode-range:U+0100-02AF,U+0304,U+0308,U+0329,U+1E00-1E9F,U+1EF2-1EFF,U+2020,U+20A0-20AB,U+20AD-20C0,U+2113,U+2C60-2C7F,U+A720-A7FF}@font-face{font-family:'Roboto';font-style:normal;font-weight:400;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOmCnqEu92Fr1Mu4mxKKTU1Kg.woff2) format('woff2');unicode-range:U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,U+02C6,U+02DA,U+02DC,U+0304,U+0308,U+0329,U+2000-206F,U+2074,U+20AC,U+2122,U+2191,U+2193,U+2212,U+2215,U+FEFF,U+FFFD}@font-face{font-family:'Roboto';font-style:normal;font-weight:500;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmEU9fCRc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0460-052F,U+1C80-1C88,U+20B4,U+2DE0-2DFF,U+A640-A69F,U+FE2E-FE2F}@font-face{font-family:'Roboto';font-style:normal;font-weight:500;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmEU9fABc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0301,U+0400-045F,U+0490-0491,U+04B0-04B1,U+2116}@font-face{font-family:'Roboto';font-style:normal;font-weight:500;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmEU9fCBc4AMP6lbBP.woff2) format('woff2');unicode-range:U+1F00-1FFF}@font-face{font-family:'Roboto';font-style:normal;font-weight:500;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmEU9fBxc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0370-0377,U+037A-037F,U+0384-038A,U+038C,U+038E-03A1,U+03A3-03FF}@font-face{font-family:'Roboto';font-style:normal;font-weight:500;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmEU9fCxc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0102-0103,U+0110-0111,U+0128-0129,U+0168-0169,U+01A0-01A1,U+01AF-01B0,U+0300-0301,U+0303-0304,U+0308-0309,U+0323,U+0329,U+1EA0-1EF9,U+20AB}@font-face{font-family:'Roboto';font-style:normal;font-weight:500;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmEU9fChc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0100-02AF,U+0304,U+0308,U+0329,U+1E00-1E9F,U+1EF2-1EFF,U+2020,U+20A0-20AB,U+20AD-20C0,U+2113,U+2C60-2C7F,U+A720-A7FF}@font-face{font-family:'Roboto';font-style:normal;font-weight:500;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmEU9fBBc4AMP6lQ.woff2) format('woff2');unicode-range:U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,U+02C6,U+02DA,U+02DC,U+0304,U+0308,U+0329,U+2000-206F,U+2074,U+20AC,U+2122,U+2191,U+2193,U+2212,U+2215,U+FEFF,U+FFFD}@font-face{font-family:'Roboto';font-style:normal;font-weight:700;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmWUlfCRc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0460-052F,U+1C80-1C88,U+20B4,U+2DE0-2DFF,U+A640-A69F,U+FE2E-FE2F}@font-face{font-family:'Roboto';font-style:normal;font-weight:700;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmWUlfABc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0301,U+0400-045F,U+0490-0491,U+04B0-04B1,U+2116}@font-face{font-family:'Roboto';font-style:normal;font-weight:700;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmWUlfCBc4AMP6lbBP.woff2) format('woff2');unicode-range:U+1F00-1FFF}@font-face{font-family:'Roboto';font-style:normal;font-weight:700;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmWUlfBxc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0370-0377,U+037A-037F,U+0384-038A,U+038C,U+038E-03A1,U+03A3-03FF}@font-face{font-family:'Roboto';font-style:normal;font-weight:700;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmWUlfCxc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0102-0103,U+0110-0111,U+0128-0129,U+0168-0169,U+01A0-01A1,U+01AF-01B0,U+0300-0301,U+0303-0304,U+0308-0309,U+0323,U+0329,U+1EA0-1EF9,U+20AB}@font-face{font-family:'Roboto';font-style:normal;font-weight:700;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmWUlfChc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0100-02AF,U+0304,U+0308,U+0329,U+1E00-1E9F,U+1EF2-1EFF,U+2020,U+20A0-20AB,U+20AD-20C0,U+2113,U+2C60-2C7F,U+A720-A7FF}@font-face{font-family:'Roboto';font-style:normal;font-weight:700;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmWUlfBBc4AMP6lQ.woff2) format('woff2');unicode-range:U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,U+02C6,U+02DA,U+02DC,U+0304,U+0308,U+0329,U+2000-206F,U+2074,U+20AC,U+2122,U+2191,U+2193,U+2212,U+2215,U+FEFF,U+FFFD}@font-face{font-family:'Roboto';font-style:normal;font-weight:900;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmYUtfCRc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0460-052F,U+1C80-1C88,U+20B4,U+2DE0-2DFF,U+A640-A69F,U+FE2E-FE2F}@font-face{font-family:'Roboto';font-style:normal;font-weight:900;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmYUtfABc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0301,U+0400-045F,U+0490-0491,U+04B0-04B1,U+2116}@font-face{font-family:'Roboto';font-style:normal;font-weight:900;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmYUtfCBc4AMP6lbBP.woff2) format('woff2');unicode-range:U+1F00-1FFF}@font-face{font-family:'Roboto';font-style:normal;font-weight:900;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmYUtfBxc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0370-0377,U+037A-037F,U+0384-038A,U+038C,U+038E-03A1,U+03A3-03FF}@font-face{font-family:'Roboto';font-style:normal;font-weight:900;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmYUtfCxc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0102-0103,U+0110-0111,U+0128-0129,U+0168-0169,U+01A0-01A1,U+01AF-01B0,U+0300-0301,U+0303-0304,U+0308-0309,U+0323,U+0329,U+1EA0-1EF9,U+20AB}@font-face{font-family:'Roboto';font-style:normal;font-weight:900;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmYUtfChc4AMP6lbBP.woff2) format('woff2');unicode-range:U+0100-02AF,U+0304,U+0308,U+0329,U+1E00-1E9F,U+1EF2-1EFF,U+2020,U+20A0-20AB,U+20AD-20C0,U+2113,U+2C60-2C7F,U+A720-A7FF}@font-face{font-family:'Roboto';font-style:normal;font-weight:900;font-display:swap;src:url(http://web.archive.org/web/20240227045235im_/https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmYUtfBBc4AMP6lQ.woff2) format('woff2');unicode-range:U+0000-00FF,U+0131,U+0152-0153,U+02BB-02BC,U+02C6,U+02DA,U+02DC,U+0304,U+0308,U+0329,U+2000-206F,U+2074,U+20AC,U+2122,U+2191,U+2193,U+2212,U+2215,U+FEFF,U+FFFD}</style></head><body style="margin:0"><div id="__next"><nav aria-labelledby="main-content-skip-link"><a class="MuiTypography-root jss1 MuiTypography-subtitle2" href="#main-content" id="main-content-skip-link"><div class="MuiContainer-root">Skip to main content</div></a></nav><header class="MuiPaper-root jss9 MuiPaper-elevation3" aria-label="Site header"><div class="jss10"><div class="MuiContainer-root MuiContainer-maxWidthLg"><div class="jss11"><div><nav aria-label="Site header breadcrumb" class="jss12"><span class="jss13" role="list"><span class="jss14"><a aria-label="Jisc homepage" class="jss22" title="Jisc homepage" href="/web/20240227045235/https://www.jisc.ac.uk/"><img alt="Jisc" class="jss21" src="data:image/svg+xml;base64,PHN2ZyBlbmFibGUtYmFja2dyb3VuZD0ibmV3IDAgMCA5OTkuOSA5OTkuOSIgdmlld0JveD0iMCAwIDk5OS45IDk5OS45IiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPgogICAgPHBhdGggZD0ibTAgMGg5OTkuOXY5OTkuOWgtOTk5Ljl6IiBmaWxsPSIjZTg1ZTEzIi8+CiAgICA8cGF0aCBkPSJtMzA1LjggNTU0YzAgODAuMS00Ni44IDExOS43LTExMy40IDExOS43LTUyLjEgMC04NS45LTI0LjEtMTA4LjEtNTQuNWwzOS4xLTQwYzE5LjggMjUuMSAzOS41IDM5LjYgNjggMzkuNiAzMS44IDAgNTMuNi0yMS4yIDUzLjYtNjYuMXYtMjIxLjZoNjAuOHptMTA3LjYgMTE0LjhoLTU4LjR2LTI1NC44aDU4LjR6bS0yOS41LTI4My42Yy0yMSAwLTM4LTE3LTM4LjEtMzcuOSAwLTIxIDE3LTM4IDM3LjktMzguMXMzOCAxNyAzOC4xIDM3Ljl2LjFjLjEgMjEtMTYuOSAzOC0zNy45IDM4em0yNjMuNCAyMDguM2MwIDUzLjUtNDEuNSA4MC4xLTk0LjYgODAuMS0zNi4yIDAtNzYuMi0xMy0xMDguMS0zOC4xbDI2LjEtMzkuNWMyNy41IDIwLjcgNTcuNCAzMS40IDgzLjUgMzEuNCAyNS4xIDAgMzkuNS0xMC42IDM5LjUtMjcuNXYtMWMwLTE5LjgtMjctMjYuNS01Ni45LTM1LjctMzcuNi0xMC42LTc5LjYtMjYtNzkuNi03NC44di0xYzAtNDguMiA0MC4xLTc3LjcgOTAuNy03Ny43IDMxLjggMCA2Ni42IDExLjEgOTMuNiAyOWwtMjMuMiA0MS41Yy0yNC42LTE0LjktNTAuNy0yNC4xLTcxLjktMjQuMS0yMi43IDAtMzUuNyAxMC42LTM1LjcgMjUuNnYxYzAgMTguMyAyNy41IDI2IDU3LjQgMzUuNyAzNy4xIDExLjYgNzkuMSAyOC41IDc5LjEgNzQuM3ptMTU3LjggODEuMWMtNzUuOCAwLTEzMS43LTU5LjMtMTMxLjctMTMyLjJzNTYtMTMzLjYgMTMyLjItMTMzLjZjNDguMyAwIDc4LjIgMTcuOSAxMDEuOCA0My45bC0zNi4yIDM5LjFjLTE3LjgtMTguOC0zNi43LTMxLjgtNjYuMS0zMS44LTQyLjUgMC03My4zIDM2LjctNzMuMyA4MS41IDAgNDUuOCAzMS40IDgyLjUgNzYuMiA4Mi41IDI3LjUgMCA0Ny44LTEzIDY2LjEtMzEuOGwzNS4yIDM0LjhjLTI1LjEgMjcuOC01NC41IDQ3LjYtMTA0LjIgNDcuNnoiCiAgICAgICAgICBmaWxsPSIjZmZmIi8+Cjwvc3ZnPgo="/></a></span><span class="jss15"></span></span></nav></div><div class="jss23"><div class="jss24"><button aria-controls="search-form" aria-expanded="false" aria-label="Show search box" class="jss25" title="Show search box" type="button"><svg aria-hidden="true" focusable="false" data-prefix="fas" data-icon="magnifying-glass" class="svg-inline--fa fa-magnifying-glass " role="img" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 512 512"><path fill="currentColor" d="M416 208c0 45.9-14.9 88.3-40 122.7L502.6 457.4c12.5 12.5 12.5 32.8 0 45.3s-32.8 12.5-45.3 0L330.7 376c-34.4 25.2-76.8 40-122.7 40C93.1 416 0 322.9 0 208S93.1 0 208 0S416 93.1 416 208zM208 352a144 144 0 1 0 0-288 144 144 0 1 0 0 288z"></path></svg></button></div><div class="jss26" id="search-form"><div class="jss31"><form action="/web/20240227045235/https://www.jisc.ac.uk/search" class="jss30" method="get"><label class="jss33" for="search-term"><span>Search Jisc</span><input class="jss32" id="search-term" name="q" placeholder="Search Jisc" title="Search Jisc" type="search"/></label><button aria-label="Search" class="jss29" title="Search" type="submit"><svg aria-hidden="true" focusable="false" data-prefix="fas" data-icon="magnifying-glass" class="svg-inline--fa fa-magnifying-glass " role="img" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 512 512"><path fill="currentColor" d="M416 208c0 45.9-14.9 88.3-40 122.7L502.6 457.4c12.5 12.5 12.5 32.8 0 45.3s-32.8 12.5-45.3 0L330.7 376c-34.4 25.2-76.8 40-122.7 40C93.1 416 0 322.9 0 208S93.1 0 208 0S416 93.1 416 208zM208 352a144 144 0 1 0 0-288 144 144 0 1 0 0 288z"></path></svg></button></form></div></div></div></div></div></div></header><main id="main-content"><article><header aria-labelledby="pageHeaderTitle" class="pb-10 text-jiscMineShaft bg-jiscConcrete" data-testid="legacy-infobox-page-header"><div class="border-0 border-t-2 border-solid border-jiscHibiscus"></div><div class="box-border block max-w-screen-md px-4 mx-auto md:px-6" data-testclass="container"><span class="null inline-block px-3 py-1 font-sans text-[1rem] font-normal antialiased leading-[1.5rem] text-jiscWhite bg-jiscHibiscus">Guide</span><div class="md:box-border md:flex md:w-[calc(100%+1rem)] md:-mx-2"><div class="md:box-border md:w-9/12 md:px-2"><header><h1 class="my-6 font-sans text-[2.5rem] font-normal antialiased leading-[3.5rem] sm:text-[2.75rem] md:text-[3rem] md:leading-[4rem]" id="pageHeaderTitle">Developing high-quality question-and-answer sets for chatbots</h1><p class="mt-6 mb-[unset] font-sans text-[1.25rem] font-normal antialiased leading-[2rem] sm:w-1/2 sm:min-w-full sm:mt-[unset]">You've started work on introducing chatbots to your organisation, now find out about what's involved in development and implementation.</p></header></div><div class="mt-6 md:box-border md:w-3/12 md:px-2 md:mt-2"><aside class="px-4 bg-jiscWhite border-2 border-solid border-jiscHibiscus "><p class="my-4 font-sans font-normal antialiased text-[1.25rem] leading-[2rem] ">About this guide</p><p class="my-4 font-sans text-[1rem] font-normal antialiased ">Published: <time datetime="2023-01-23T16:50:00Z">23 January 2023</time></p></aside></div></div></div></header><div class=" clearfix pt-10 pb-10 bg-jiscWhite " data-testid="content-area"><div class="box-border block max-w-screen-md px-4 mx-auto md:px-6" data-testclass="container"><div class="sm:box-border sm:flex sm:w-[calc(100%+1rem)] sm:-mx-2"><div class="hidden sm:box-border sm:px-2 sm:block sm:w-1/12" data-testid="replace-material-ui-hidden--1/12@sm"></div><div class="sm:box-border sm:px-2 sm:w-10/12"><nav class="py-2 mb-8 border-t-0 border-b-2 border-solid border-x-0 bg-jiscWhite border-jiscConcrete "><div class="flex items-center"><span class="antialiased font-normal text-2xl text-jiscMineShaft font-sans leading-[2rem] text-center pl-2">Contents</span></div><div><ul class=" list-none cursor-pointer font-normal text-jiscBahamaBlue underline p-1 visited:text-jiscPurple focus:text-jiscBlack focus:bg-jiscDandelion focus:outline focus:outline-4 focus:outline-transparent focus:no-underline focus:shadow-[0_.25rem_0_0_#fd6,_0_-.25rem_0_0_#fd6] hover:text-jiscRegalBlue active:text-jiscBlack focus:active:underline focus:active:bg-transparent focus:active:shadow-none prose-strong:text-jiscBahamaBlue prose-strong:visited:text-jiscPurple prose-strong:focus:text-jiscBlack prose-strong:hover:text-jiscRegalBlue prose-strong:active:text-jiscBlack "><li><button type="button" class="antialiased font-normal text-lg font-sans leading-[2rem] text-left cursor-pointer outline-none bg-[unset] border-none text-jiscBahamaBlue hover:underline visited:text-jiscPurple focus:text-jiscBlack focus:bg-jiscDandelion focus:outline focus:outline-4 focus:outline-transparent focus:no-underline focus:shadow-[0_.25rem_0_0_#fd6,_0_-.25rem_0_0_#fd6] hover:text-jiscRegalBlue active:text-jiscBlack focus:active:unde rline focus:active:bg-transparent focus:active:shadow-none prose-strong:text-jiscBahamaBlue prose-strong:visited:text-jiscPurple prose-strong:focus:text-jiscBlack prose-strong:hover:text-jiscRegalBlue prose-strong:active:text-jiscBlack">Introduction</button></li><li><button type="button" class="antialiased font-normal text-lg font-sans leading-[2rem] text-left cursor-pointer outline-none bg-[unset] border-none text-jiscBahamaBlue hover:underline visited:text-jiscPurple focus:text-jiscBlack focus:bg-jiscDandelion focus:outline focus:outline-4 focus:outline-transparent focus:no-underline focus:shadow-[0_.25rem_0_0_#fd6,_0_-.25rem_0_0_#fd6] hover:text-jiscRegalBlue active:text-jiscBlack focus:active:unde rline focus:active:bg-transparent focus:active:shadow-none prose-strong:text-jiscBahamaBlue prose-strong:visited:text-jiscPurple prose-strong:focus:text-jiscBlack prose-strong:hover:text-jiscRegalBlue prose-strong:active:text-jiscBlack">The importance of question-and-answer sets</button></li><li><button type="button" class="antialiased font-normal text-lg font-sans leading-[2rem] text-left cursor-pointer outline-none bg-[unset] border-none text-jiscBahamaBlue hover:underline visited:text-jiscPurple focus:text-jiscBlack focus:bg-jiscDandelion focus:outline focus:outline-4 focus:outline-transparent focus:no-underline focus:shadow-[0_.25rem_0_0_#fd6,_0_-.25rem_0_0_#fd6] hover:text-jiscRegalBlue active:text-jiscBlack focus:active:unde rline focus:active:bg-transparent focus:active:shadow-none prose-strong:text-jiscBahamaBlue prose-strong:visited:text-jiscPurple prose-strong:focus:text-jiscBlack prose-strong:hover:text-jiscRegalBlue prose-strong:active:text-jiscBlack">The structure of question-and-answer sets</button></li><li><button type="button" class="antialiased font-normal text-lg font-sans leading-[2rem] text-left cursor-pointer outline-none bg-[unset] border-none text-jiscBahamaBlue hover:underline visited:text-jiscPurple focus:text-jiscBlack focus:bg-jiscDandelion focus:outline focus:outline-4 focus:outline-transparent focus:no-underline focus:shadow-[0_.25rem_0_0_#fd6,_0_-.25rem_0_0_#fd6] hover:text-jiscRegalBlue active:text-jiscBlack focus:active:unde rline focus:active:bg-transparent focus:active:shadow-none prose-strong:text-jiscBahamaBlue prose-strong:visited:text-jiscPurple prose-strong:focus:text-jiscBlack prose-strong:hover:text-jiscRegalBlue prose-strong:active:text-jiscBlack">First steps</button></li><li><button type="button" class="antialiased font-normal text-lg font-sans leading-[2rem] text-left cursor-pointer outline-none bg-[unset] border-none text-jiscBahamaBlue hover:underline visited:text-jiscPurple focus:text-jiscBlack focus:bg-jiscDandelion focus:outline focus:outline-4 focus:outline-transparent focus:no-underline focus:shadow-[0_.25rem_0_0_#fd6,_0_-.25rem_0_0_#fd6] hover:text-jiscRegalBlue active:text-jiscBlack focus:active:unde rline focus:active:bg-transparent focus:active:shadow-none prose-strong:text-jiscBahamaBlue prose-strong:visited:text-jiscPurple prose-strong:focus:text-jiscBlack prose-strong:hover:text-jiscRegalBlue prose-strong:active:text-jiscBlack">Rules of thumb</button></li><li><button type="button" class="antialiased font-normal text-lg font-sans leading-[2rem] text-left cursor-pointer outline-none bg-[unset] border-none text-jiscBahamaBlue hover:underline visited:text-jiscPurple focus:text-jiscBlack focus:bg-jiscDandelion focus:outline focus:outline-4 focus:outline-transparent focus:no-underline focus:shadow-[0_.25rem_0_0_#fd6,_0_-.25rem_0_0_#fd6] hover:text-jiscRegalBlue active:text-jiscBlack focus:active:unde rline focus:active:bg-transparent focus:active:shadow-none prose-strong:text-jiscBahamaBlue prose-strong:visited:text-jiscPurple prose-strong:focus:text-jiscBlack prose-strong:hover:text-jiscRegalBlue prose-strong:active:text-jiscBlack">Refining question-and-answer sets through pre-launch testing</button></li><li><button type="button" class="antialiased font-normal text-lg font-sans leading-[2rem] text-left cursor-pointer outline-none bg-[unset] border-none text-jiscBahamaBlue hover:underline visited:text-jiscPurple focus:text-jiscBlack focus:bg-jiscDandelion focus:outline focus:outline-4 focus:outline-transparent focus:no-underline focus:shadow-[0_.25rem_0_0_#fd6,_0_-.25rem_0_0_#fd6] hover:text-jiscRegalBlue active:text-jiscBlack focus:active:unde rline focus:active:bg-transparent focus:active:shadow-none prose-strong:text-jiscBahamaBlue prose-strong:visited:text-jiscPurple prose-strong:focus:text-jiscBlack prose-strong:hover:text-jiscRegalBlue prose-strong:active:text-jiscBlack">About our work in AI</button></li></ul></div></nav><div class=" clearfix | prose | antialiased | font-sans | max-w-full | prose-h2:text-[2rem] prose-h2:font-normal prose-h2:leading-[3.0rem] prose-h2:sm:text-[2.25rem] prose-h2:md:text-[2.5rem] prose-h2:md:leading-[3.5rem] | prose-h2:mt-10 prose-h2:mb-6 first:prose-h2:mt-0 last:prose-h2:mb-0 prose-h3:text-[1.5rem] prose-h3:font-medium prose-h3:leading-[2rem] first:prose-h3:mt-0 last:prose-h3:mb-0 prose-h3:mt-10 prose-h3:mb-6 prose-h3:sm:text-[1.75rem] prose-h3:sm:leading-[2.5rem] prose-h3:md:text-[2rem] prose-h3:md:leading-[3rem] | prose-h4:text-[1.25rem] prose-h4:font-normal prose-h4:leading-[2rem] first:prose-h4:mt-0 last:prose-h4:mb-0 prose-h4:mt-10 prose-h4:mb-6 prose-h4:md:text-[1.5rem] prose-h4:md:leading-[2rem] | prose-h5:text-[1rem] prose-h5:font-medium prose-h5:leading-[1.5rem] prose-h5:mt-10 prose-h5:mb-6 first:prose-h5:mt-[unset] last:prose-h5:mb-[unset] prose-h5:md:text-[1.25rem] prose-h5:md:leading-[2rem] prose-h6:text-[1rem] prose-h6:font-semibold prose-h6:leading-[1.5rem] prose-h6:mt-10 prose-h6:mb-6 first:prose-h6:mt-[unset] last:prose-h6:mb-[unset] prose-p:my-6 prose-p:text-[1rem] prose-p:font-normal prose-p:leading-[1.5rem] first:prose-p:mt-[unset] last:prose-p:mb-[unset] empty:first:prose-p:-mt-6 prose-blockquote:my-[unset] prose-blockquote:ml-6 prose-blockquote:text-[1rem] prose-blockquote:not-italic prose-blockquote:font-normal prose-blockquote:leading-[1.5rem] prose-blockquote:border-solid prose-blockquote:border-0 prose-blockquote:border-l-[.0625rem] prose-p:prose-blockquote:py-3 before:prose-p:prose-blockquote:content-[unset] after:prose-p:prose-blockquote:content-[unset] prose-li:text-[1rem] prose-li:font-normal prose-li:leading-[1.5rem] | first:prose-li:mt-2 prose-li:pl-2 | first:prose-ol:mt-0 last:prose-ol:mb-0 prose-ol:my-6 prose-ol:pl-8 prose-ol:leading-[1.5rem] | first:prose-ul:mt-0 last:prose-ul:mb-0 prose-ul:my-6 prose-ul:pl-8 prose-ul:leading-[1.5rem] | prose-table:leading-[1.5rem] | prose-table:mt-[unset] prose-table:mb-[unset] | prose-table:border-collapse | prose-p:prose-td:text-[0.875rem] | prose-p:prose-th:text-[0.875rem] | prose-li:prose-th:text-[0.875rem] | prose-li:prose-td:text-[0.875rem] | prose-td:border-solid prose-td:border-l-[0rem] prose-td:border-r-[0rem] prose-td:border-jiscAlto | prose-td:border-b-[0.0625rem] prose-td:border-t-[0.0625rem] | prose-td:p-4 prose-td:align-middle prose-td:leading-[1.5rem]| prose-th:border-solid prose-th:border-l-[0rem] prose-th:border-r-[0rem] prose-th:border-jiscAlto | prose-th:border-b-[0.0625rem] prose-th:border-t-[0.0625rem] prose-th:border-b-jiscAlto prose-th:border-t-jiscAlto | prose-th:p-4 prose-th:align-middle prose-th:leading-[1.5rem] | prose-th:font-medium | prose-tr:border-b-jiscAlto prose-tr:border-t-jiscAlto | prose-img:block prose-img:max-w-full prose-img:h-auto prose-img:my-6 prose-img:leading-none | prose-a:font-normal prose-a:text-jiscBahamaBlue prose-a:underline | visited:prose-a:text-jiscPurple | focus:prose-a:text-jiscBlack focus:prose-a:bg-jiscDandelion focus:prose-a:outline focus:prose-a:outline-4 focus:prose-a:outline-transparent focus:prose-a:no-underline focus:prose-a:shadow-[0_.25rem_0_0_#fd6,_0_-.25rem_0_0_#fd6] | hover:prose-a:text-jiscRegalBlue | active:prose-a:text-jiscBlack | focus:active:prose-a:underline focus:active:prose-a:bg-transparent focus:active:prose-a:shadow-none | prose-strong:prose-a:text-jiscBahamaBlue | prose-strong:visited:prose-a:text-jiscPurple | prose-strong:focus:prose-a:text-jiscBlack | prose-strong:hover:prose-a:text-jiscRegalBlue | prose-strong:active:prose-a:text-jiscBlack || prose-blockquote:border-l-jiscAlto || marker:text-jiscMineShaft | prose-blockquote:text-jiscDoveGrey | prose-headings:text-jiscMineShaft | prose-li:text-jiscMineShaft | prose-ol:text-jiscMineShaft | prose-p:text-jiscMineShaft | prose-strong:text-jiscMineShaft | prose-table:text-jiscMineShaft | prose-ul:text-jiscMineShaft | | mb-6 |"><h2 id="introduction-66cow" data-block-key="66cow">Introduction</h2><p data-block-key="4l8cp">This guide is for anyone who has begun work on implementing chatbots and would like guidance on how to develop high-quality question-and-answer sets. It may also be of use to those who are strongly considering implementing a chatbot and want to find out more about the work involved in development and implementation.</p><p data-block-key="abnfj">This guide begins by explaining the importance of high-quality question-and-answer sets for chatbot performance. It then outlines the structure of question-and-answer sets, so that readers are:</p><ul><li data-block-key="buqf6">Introduced to the key terms used when describing chatbot question-and-answer sets</li><li data-block-key="9gl0b">Able to understand central concepts underpinning chatbot use</li><li data-block-key="qadk">Able to decide whether the guidance in this document is relevant to the type of chatbot they are using</li></ul><p data-block-key="7lf71">We then look at the initial steps that should be taken to develop high-quality question-and-answer sets and provides a few ‘rules of thumb’ that will help in developing them. Finally, we look at the steps that can be taken to refine question-and-answer sets and manage their ongoing development once the chatbot has gone live.</p></div><div class=" clearfix | prose | antialiased | font-sans | max-w-full | prose-h2:text-[2rem] prose-h2:font-normal prose-h2:leading-[3.0rem] prose-h2:sm:text-[2.25rem] prose-h2:md:text-[2.5rem] prose-h2:md:leading-[3.5rem] | prose-h2:mt-10 prose-h2:mb-6 first:prose-h2:mt-0 last:prose-h2:mb-0 prose-h3:text-[1.5rem] prose-h3:font-medium prose-h3:leading-[2rem] first:prose-h3:mt-0 last:prose-h3:mb-0 prose-h3:mt-10 prose-h3:mb-6 prose-h3:sm:text-[1.75rem] prose-h3:sm:leading-[2.5rem] prose-h3:md:text-[2rem] prose-h3:md:leading-[3rem] | prose-h4:text-[1.25rem] prose-h4:font-normal prose-h4:leading-[2rem] first:prose-h4:mt-0 last:prose-h4:mb-0 prose-h4:mt-10 prose-h4:mb-6 prose-h4:md:text-[1.5rem] prose-h4:md:leading-[2rem] | prose-h5:text-[1rem] prose-h5:font-medium prose-h5:leading-[1.5rem] prose-h5:mt-10 prose-h5:mb-6 first:prose-h5:mt-[unset] last:prose-h5:mb-[unset] prose-h5:md:text-[1.25rem] prose-h5:md:leading-[2rem] prose-h6:text-[1rem] prose-h6:font-semibold prose-h6:leading-[1.5rem] prose-h6:mt-10 prose-h6:mb-6 first:prose-h6:mt-[unset] last:prose-h6:mb-[unset] prose-p:my-6 prose-p:text-[1rem] prose-p:font-normal prose-p:leading-[1.5rem] first:prose-p:mt-[unset] last:prose-p:mb-[unset] empty:first:prose-p:-mt-6 prose-blockquote:my-[unset] prose-blockquote:ml-6 prose-blockquote:text-[1rem] prose-blockquote:not-italic prose-blockquote:font-normal prose-blockquote:leading-[1.5rem] prose-blockquote:border-solid prose-blockquote:border-0 prose-blockquote:border-l-[.0625rem] prose-p:prose-blockquote:py-3 before:prose-p:prose-blockquote:content-[unset] after:prose-p:prose-blockquote:content-[unset] prose-li:text-[1rem] prose-li:font-normal prose-li:leading-[1.5rem] | first:prose-li:mt-2 prose-li:pl-2 | first:prose-ol:mt-0 last:prose-ol:mb-0 prose-ol:my-6 prose-ol:pl-8 prose-ol:leading-[1.5rem] | first:prose-ul:mt-0 last:prose-ul:mb-0 prose-ul:my-6 prose-ul:pl-8 prose-ul:leading-[1.5rem] | prose-table:leading-[1.5rem] | prose-table:mt-[unset] prose-table:mb-[unset] | prose-table:border-collapse | prose-p:prose-td:text-[0.875rem] | prose-p:prose-th:text-[0.875rem] | prose-li:prose-th:text-[0.875rem] | prose-li:prose-td:text-[0.875rem] | prose-td:border-solid prose-td:border-l-[0rem] prose-td:border-r-[0rem] prose-td:border-jiscAlto | prose-td:border-b-[0.0625rem] prose-td:border-t-[0.0625rem] | prose-td:p-4 prose-td:align-middle prose-td:leading-[1.5rem]| prose-th:border-solid prose-th:border-l-[0rem] prose-th:border-r-[0rem] prose-th:border-jiscAlto | prose-th:border-b-[0.0625rem] prose-th:border-t-[0.0625rem] prose-th:border-b-jiscAlto prose-th:border-t-jiscAlto | prose-th:p-4 prose-th:align-middle prose-th:leading-[1.5rem] | prose-th:font-medium | prose-tr:border-b-jiscAlto prose-tr:border-t-jiscAlto | prose-img:block prose-img:max-w-full prose-img:h-auto prose-img:my-6 prose-img:leading-none | prose-a:font-normal prose-a:text-jiscBahamaBlue prose-a:underline | visited:prose-a:text-jiscPurple | focus:prose-a:text-jiscBlack focus:prose-a:bg-jiscDandelion focus:prose-a:outline focus:prose-a:outline-4 focus:prose-a:outline-transparent focus:prose-a:no-underline focus:prose-a:shadow-[0_.25rem_0_0_#fd6,_0_-.25rem_0_0_#fd6] | hover:prose-a:text-jiscRegalBlue | active:prose-a:text-jiscBlack | focus:active:prose-a:underline focus:active:prose-a:bg-transparent focus:active:prose-a:shadow-none | prose-strong:prose-a:text-jiscBahamaBlue | prose-strong:visited:prose-a:text-jiscPurple | prose-strong:focus:prose-a:text-jiscBlack | prose-strong:hover:prose-a:text-jiscRegalBlue | prose-strong:active:prose-a:text-jiscBlack || prose-blockquote:border-l-jiscAlto || marker:text-jiscMineShaft | prose-blockquote:text-jiscDoveGrey | prose-headings:text-jiscMineShaft | prose-li:text-jiscMineShaft | prose-ol:text-jiscMineShaft | prose-p:text-jiscMineShaft | prose-strong:text-jiscMineShaft | prose-table:text-jiscMineShaft | prose-ul:text-jiscMineShaft | | mt-6 mb-6 |"><h2 id="the-importance-of-question-and-answer-sets-68zpo" data-block-key="68zpo">The importance of question-and-answer sets</h2><p data-block-key="419ii">For users to benefit from chatbot use, those developing chatbot content must anticipate the kinds of questions that users are likely to ask and the information they would need in response. As such, chatbot performance depends on more than just the effectiveness of the software being used: performance depends on the calibre of the content too. Ultimately, the quality of a question-and-answer set depends on:</p><ul><li data-block-key="f5t9t">How well it captures the information that users want to find out from using the chatbot</li><li data-block-key="5nh9p">How well its structure aligns with the logic of how the chatbot works</li></ul></div><div class=" clearfix | prose | antialiased | font-sans | max-w-full | prose-h2:text-[2rem] prose-h2:font-normal prose-h2:leading-[3.0rem] prose-h2:sm:text-[2.25rem] prose-h2:md:text-[2.5rem] prose-h2:md:leading-[3.5rem] | prose-h2:mt-10 prose-h2:mb-6 first:prose-h2:mt-0 last:prose-h2:mb-0 prose-h3:text-[1.5rem] prose-h3:font-medium prose-h3:leading-[2rem] first:prose-h3:mt-0 last:prose-h3:mb-0 prose-h3:mt-10 prose-h3:mb-6 prose-h3:sm:text-[1.75rem] prose-h3:sm:leading-[2.5rem] prose-h3:md:text-[2rem] prose-h3:md:leading-[3rem] | prose-h4:text-[1.25rem] prose-h4:font-normal prose-h4:leading-[2rem] first:prose-h4:mt-0 last:prose-h4:mb-0 prose-h4:mt-10 prose-h4:mb-6 prose-h4:md:text-[1.5rem] prose-h4:md:leading-[2rem] | prose-h5:text-[1rem] prose-h5:font-medium prose-h5:leading-[1.5rem] prose-h5:mt-10 prose-h5:mb-6 first:prose-h5:mt-[unset] last:prose-h5:mb-[unset] prose-h5:md:text-[1.25rem] prose-h5:md:leading-[2rem] prose-h6:text-[1rem] prose-h6:font-semibold prose-h6:leading-[1.5rem] prose-h6:mt-10 prose-h6:mb-6 first:prose-h6:mt-[unset] last:prose-h6:mb-[unset] prose-p:my-6 prose-p:text-[1rem] prose-p:font-normal prose-p:leading-[1.5rem] first:prose-p:mt-[unset] last:prose-p:mb-[unset] empty:first:prose-p:-mt-6 prose-blockquote:my-[unset] prose-blockquote:ml-6 prose-blockquote:text-[1rem] prose-blockquote:not-italic prose-blockquote:font-normal prose-blockquote:leading-[1.5rem] prose-blockquote:border-solid prose-blockquote:border-0 prose-blockquote:border-l-[.0625rem] prose-p:prose-blockquote:py-3 before:prose-p:prose-blockquote:content-[unset] after:prose-p:prose-blockquote:content-[unset] prose-li:text-[1rem] prose-li:font-normal prose-li:leading-[1.5rem] | first:prose-li:mt-2 prose-li:pl-2 | first:prose-ol:mt-0 last:prose-ol:mb-0 prose-ol:my-6 prose-ol:pl-8 prose-ol:leading-[1.5rem] | first:prose-ul:mt-0 last:prose-ul:mb-0 prose-ul:my-6 prose-ul:pl-8 prose-ul:leading-[1.5rem] | prose-table:leading-[1.5rem] | prose-table:mt-[unset] prose-table:mb-[unset] | prose-table:border-collapse | prose-p:prose-td:text-[0.875rem] | prose-p:prose-th:text-[0.875rem] | prose-li:prose-th:text-[0.875rem] | prose-li:prose-td:text-[0.875rem] | prose-td:border-solid prose-td:border-l-[0rem] prose-td:border-r-[0rem] prose-td:border-jiscAlto | prose-td:border-b-[0.0625rem] prose-td:border-t-[0.0625rem] | prose-td:p-4 prose-td:align-middle prose-td:leading-[1.5rem]| prose-th:border-solid prose-th:border-l-[0rem] prose-th:border-r-[0rem] prose-th:border-jiscAlto | prose-th:border-b-[0.0625rem] prose-th:border-t-[0.0625rem] prose-th:border-b-jiscAlto prose-th:border-t-jiscAlto | prose-th:p-4 prose-th:align-middle prose-th:leading-[1.5rem] | prose-th:font-medium | prose-tr:border-b-jiscAlto prose-tr:border-t-jiscAlto | prose-img:block prose-img:max-w-full prose-img:h-auto prose-img:my-6 prose-img:leading-none | prose-a:font-normal prose-a:text-jiscBahamaBlue prose-a:underline | visited:prose-a:text-jiscPurple | focus:prose-a:text-jiscBlack focus:prose-a:bg-jiscDandelion focus:prose-a:outline focus:prose-a:outline-4 focus:prose-a:outline-transparent focus:prose-a:no-underline focus:prose-a:shadow-[0_.25rem_0_0_#fd6,_0_-.25rem_0_0_#fd6] | hover:prose-a:text-jiscRegalBlue | active:prose-a:text-jiscBlack | focus:active:prose-a:underline focus:active:prose-a:bg-transparent focus:active:prose-a:shadow-none | prose-strong:prose-a:text-jiscBahamaBlue | prose-strong:visited:prose-a:text-jiscPurple | prose-strong:focus:prose-a:text-jiscBlack | prose-strong:hover:prose-a:text-jiscRegalBlue | prose-strong:active:prose-a:text-jiscBlack || prose-blockquote:border-l-jiscAlto || marker:text-jiscMineShaft | prose-blockquote:text-jiscDoveGrey | prose-headings:text-jiscMineShaft | prose-li:text-jiscMineShaft | prose-ol:text-jiscMineShaft | prose-p:text-jiscMineShaft | prose-strong:text-jiscMineShaft | prose-table:text-jiscMineShaft | prose-ul:text-jiscMineShaft | | mt-6 mb-6 |"><h2 id="the-structure-of-question-and-answer-sets-68zpo" data-block-key="68zpo">The structure of question-and-answer sets</h2><p data-block-key="1uuqb">A typical question-and-answer set is built upon three fundamental elements:</p><ul><li data-block-key="d7fm1"><strong>Intents</strong> – The underlying meaning or intention behind a question</li><li data-block-key="9lgli"><strong>Utterances</strong> – Different ways of wording a particular question</li><li data-block-key="d1nkt"><strong>Fulfilments</strong> – The answer to a user's question</li></ul><p data-block-key="7uf4">For example:</p><ul><li data-block-key="eu1s4"><strong>Intent</strong> – A user wants to find out if the cafeteria caters for vegans</li><li data-block-key="2klvn"><strong>Utterances</strong> – “Do you serve vegan food in college?”, “Is the food in the café suitable for vegans?”, “Do you cater for vegans?”, “I’m vegan, will I be able to get lunch in college?”</li><li data-block-key="56o0f"><strong>Fulfilment</strong> – “The college cafeteria offers a range of dishes that are suitable for vegans. To see the menu for the next two weeks, please visit...”</li></ul><p data-block-key="f2djh">Giving the chatbot multiple examples of how a question could be asked (in other words, inputting numerous utterances), enables the chatbot’s machine learning to build models of the intent so that the chatbot can respond to questions that have been worded in a way that was not anticipated. Take the example above of the intent relating to vegan food. It would be very difficult – potentially impossible – to anticipate all the ways in which a user might express their intention of finding out whether the cafeteria caters for vegans.</p><p data-block-key="e64u0">That said, if the chatbot is given a large enough sample of example questions (ie utterances), it will be able to use machine learning models to establish the underlying meaning that unites them. This means that if a user were to ask “is there vegan food in the college cafeteria?”, the chatbot should be able to recognise the meaning (intent) behind this question, and hence respond with the answer (fulfilment) given in the table above.</p></div><div class=" clearfix | prose | antialiased | font-sans | max-w-full | prose-h2:text-[2rem] prose-h2:font-normal prose-h2:leading-[3.0rem] prose-h2:sm:text-[2.25rem] prose-h2:md:text-[2.5rem] prose-h2:md:leading-[3.5rem] | prose-h2:mt-10 prose-h2:mb-6 first:prose-h2:mt-0 last:prose-h2:mb-0 prose-h3:text-[1.5rem] prose-h3:font-medium prose-h3:leading-[2rem] first:prose-h3:mt-0 last:prose-h3:mb-0 prose-h3:mt-10 prose-h3:mb-6 prose-h3:sm:text-[1.75rem] prose-h3:sm:leading-[2.5rem] prose-h3:md:text-[2rem] prose-h3:md:leading-[3rem] | prose-h4:text-[1.25rem] prose-h4:font-normal prose-h4:leading-[2rem] first:prose-h4:mt-0 last:prose-h4:mb-0 prose-h4:mt-10 prose-h4:mb-6 prose-h4:md:text-[1.5rem] prose-h4:md:leading-[2rem] | prose-h5:text-[1rem] prose-h5:font-medium prose-h5:leading-[1.5rem] prose-h5:mt-10 prose-h5:mb-6 first:prose-h5:mt-[unset] last:prose-h5:mb-[unset] prose-h5:md:text-[1.25rem] prose-h5:md:leading-[2rem] prose-h6:text-[1rem] prose-h6:font-semibold prose-h6:leading-[1.5rem] prose-h6:mt-10 prose-h6:mb-6 first:prose-h6:mt-[unset] last:prose-h6:mb-[unset] prose-p:my-6 prose-p:text-[1rem] prose-p:font-normal prose-p:leading-[1.5rem] first:prose-p:mt-[unset] last:prose-p:mb-[unset] empty:first:prose-p:-mt-6 prose-blockquote:my-[unset] prose-blockquote:ml-6 prose-blockquote:text-[1rem] prose-blockquote:not-italic prose-blockquote:font-normal prose-blockquote:leading-[1.5rem] prose-blockquote:border-solid prose-blockquote:border-0 prose-blockquote:border-l-[.0625rem] prose-p:prose-blockquote:py-3 before:prose-p:prose-blockquote:content-[unset] after:prose-p:prose-blockquote:content-[unset] prose-li:text-[1rem] prose-li:font-normal prose-li:leading-[1.5rem] | first:prose-li:mt-2 prose-li:pl-2 | first:prose-ol:mt-0 last:prose-ol:mb-0 prose-ol:my-6 prose-ol:pl-8 prose-ol:leading-[1.5rem] | first:prose-ul:mt-0 last:prose-ul:mb-0 prose-ul:my-6 prose-ul:pl-8 prose-ul:leading-[1.5rem] | prose-table:leading-[1.5rem] | prose-table:mt-[unset] prose-table:mb-[unset] | prose-table:border-collapse | prose-p:prose-td:text-[0.875rem] | prose-p:prose-th:text-[0.875rem] | prose-li:prose-th:text-[0.875rem] | prose-li:prose-td:text-[0.875rem] | prose-td:border-solid prose-td:border-l-[0rem] prose-td:border-r-[0rem] prose-td:border-jiscAlto | prose-td:border-b-[0.0625rem] prose-td:border-t-[0.0625rem] | prose-td:p-4 prose-td:align-middle prose-td:leading-[1.5rem]| prose-th:border-solid prose-th:border-l-[0rem] prose-th:border-r-[0rem] prose-th:border-jiscAlto | prose-th:border-b-[0.0625rem] prose-th:border-t-[0.0625rem] prose-th:border-b-jiscAlto prose-th:border-t-jiscAlto | prose-th:p-4 prose-th:align-middle prose-th:leading-[1.5rem] | prose-th:font-medium | prose-tr:border-b-jiscAlto prose-tr:border-t-jiscAlto | prose-img:block prose-img:max-w-full prose-img:h-auto prose-img:my-6 prose-img:leading-none | prose-a:font-normal prose-a:text-jiscBahamaBlue prose-a:underline | visited:prose-a:text-jiscPurple | focus:prose-a:text-jiscBlack focus:prose-a:bg-jiscDandelion focus:prose-a:outline focus:prose-a:outline-4 focus:prose-a:outline-transparent focus:prose-a:no-underline focus:prose-a:shadow-[0_.25rem_0_0_#fd6,_0_-.25rem_0_0_#fd6] | hover:prose-a:text-jiscRegalBlue | active:prose-a:text-jiscBlack | focus:active:prose-a:underline focus:active:prose-a:bg-transparent focus:active:prose-a:shadow-none | prose-strong:prose-a:text-jiscBahamaBlue | prose-strong:visited:prose-a:text-jiscPurple | prose-strong:focus:prose-a:text-jiscBlack | prose-strong:hover:prose-a:text-jiscRegalBlue | prose-strong:active:prose-a:text-jiscBlack || prose-blockquote:border-l-jiscAlto || marker:text-jiscMineShaft | prose-blockquote:text-jiscDoveGrey | prose-headings:text-jiscMineShaft | prose-li:text-jiscMineShaft | prose-ol:text-jiscMineShaft | prose-p:text-jiscMineShaft | prose-strong:text-jiscMineShaft | prose-table:text-jiscMineShaft | prose-ul:text-jiscMineShaft | | mt-6 mb-6 |"><h2 id="first-steps-68zpo" data-block-key="68zpo">First steps</h2><h3 data-block-key="2vj21">Step 1: clarify the purpose of your chatbot</h3><p data-block-key="2d65k">Before beginning work on the question-and-answer set, it is important to pinpoint the aims and objectives of the chatbot itself. Doing so will help you to shape the topics you want the question-and-answer set to focus on, while also providing useful steers on the chatbot’s tone and the kind of information that should be conveyed through its responses to users.</p><h3 data-block-key="767cq">Step 2: identify the questions your chatbot needs to be able to answer</h3><p data-block-key="e73q7">The use case for the chatbot will determine, in general terms, the topics that need to be captured in the question-and-answer set. After this, the next step is to think carefully about the kinds of queries that users are likely to raise, and the kinds of responses that will satisfy these queries. Where possible, a useful place to start is to gather evidence on what kinds of queries users are already raising. For instance, if a chatbot’s purpose is to answer new students’ queries around enrolment – queries that might normally be raised to staff at a student help desk – then evidence of the kinds of questions that are normally asked could be gained by:</p><ul><li data-block-key="5js7f">Using the experience of staff at the help desk</li><li data-block-key="eej54">Reviewing relevant emails sent by students to the help desk</li></ul><p data-block-key="bd89o">In some cases, it may not be possible to gather an evidence base around the queries students are likely to have (the chatbot may be responding to student queries about a new process, for example). Where this is the case, it is suggested that professional judgement is used to anticipate the kinds of questions that students might raise.</p><p data-block-key="5943i">For instance, if a chatbot’s purpose is to answer student questions about a new set of courses on offer, then those developing the question-and-answer set (alongside those with expertise in these new courses, perhaps) should review how students are receiving information and anticipate where they may want more detail or clarification on certain points.</p><p data-block-key="b8o6q">Whatever means are used to predict which questions will be asked, it is important to capture them systematically so that they can be referred back to by all individuals involved in the question-and-answer set development.</p><h3 data-block-key="bcaso">Step 3: establish how users are likely to ask their questions</h3><p data-block-key="diqsj">Having established which questions the chatbot will need to answer, the next step is to identify the variety of ways in which these questions could be worded. This is because the chatbot needs to be given a number of example wordings (utterances) of a particular question (intent) so that it can build machine learning-based models of the full range of ways in which that question could be worded.</p><p data-block-key="d8vpt">It is difficult to predict precisely how users will word their questions. That said, there are a number of methods that can be employed to make the task easier:</p><ul><li data-block-key="ai6g1">Where possible, draw upon evidence of how students have raised such queries in the past (for example, ask relevant members of staff, review appropriate email records)</li><li data-block-key="8gt43">Work collaboratively within teams when generating multiple wordings for the same question, to draw upon different colleagues’ insights</li><li data-block-key="d9gt7">Where appropriate, get input from users on how they might word questions (this can be particularly useful for anticipating the jargon users might use)</li></ul><p data-block-key="a8aeh">When generating different wordings for a question, it is important to know how many different wordings (utterances) should be input into the question-and-answer set. From experience, it has been found that using five to ten utterances (different ways of wording the same question) per intent (group of questions that all share the same meaning), tends to be more effective than using less than five utterances. It has also been found that using more than ten utterances can yield diminishing returns or be counterproductive.</p></div><div class=" clearfix | prose | antialiased | font-sans | max-w-full | prose-h2:text-[2rem] prose-h2:font-normal prose-h2:leading-[3.0rem] prose-h2:sm:text-[2.25rem] prose-h2:md:text-[2.5rem] prose-h2:md:leading-[3.5rem] | prose-h2:mt-10 prose-h2:mb-6 first:prose-h2:mt-0 last:prose-h2:mb-0 prose-h3:text-[1.5rem] prose-h3:font-medium prose-h3:leading-[2rem] first:prose-h3:mt-0 last:prose-h3:mb-0 prose-h3:mt-10 prose-h3:mb-6 prose-h3:sm:text-[1.75rem] prose-h3:sm:leading-[2.5rem] prose-h3:md:text-[2rem] prose-h3:md:leading-[3rem] | prose-h4:text-[1.25rem] prose-h4:font-normal prose-h4:leading-[2rem] first:prose-h4:mt-0 last:prose-h4:mb-0 prose-h4:mt-10 prose-h4:mb-6 prose-h4:md:text-[1.5rem] prose-h4:md:leading-[2rem] | prose-h5:text-[1rem] prose-h5:font-medium prose-h5:leading-[1.5rem] prose-h5:mt-10 prose-h5:mb-6 first:prose-h5:mt-[unset] last:prose-h5:mb-[unset] prose-h5:md:text-[1.25rem] prose-h5:md:leading-[2rem] prose-h6:text-[1rem] prose-h6:font-semibold prose-h6:leading-[1.5rem] prose-h6:mt-10 prose-h6:mb-6 first:prose-h6:mt-[unset] last:prose-h6:mb-[unset] prose-p:my-6 prose-p:text-[1rem] prose-p:font-normal prose-p:leading-[1.5rem] first:prose-p:mt-[unset] last:prose-p:mb-[unset] empty:first:prose-p:-mt-6 prose-blockquote:my-[unset] prose-blockquote:ml-6 prose-blockquote:text-[1rem] prose-blockquote:not-italic prose-blockquote:font-normal prose-blockquote:leading-[1.5rem] prose-blockquote:border-solid prose-blockquote:border-0 prose-blockquote:border-l-[.0625rem] prose-p:prose-blockquote:py-3 before:prose-p:prose-blockquote:content-[unset] after:prose-p:prose-blockquote:content-[unset] prose-li:text-[1rem] prose-li:font-normal prose-li:leading-[1.5rem] | first:prose-li:mt-2 prose-li:pl-2 | first:prose-ol:mt-0 last:prose-ol:mb-0 prose-ol:my-6 prose-ol:pl-8 prose-ol:leading-[1.5rem] | first:prose-ul:mt-0 last:prose-ul:mb-0 prose-ul:my-6 prose-ul:pl-8 prose-ul:leading-[1.5rem] | prose-table:leading-[1.5rem] | prose-table:mt-[unset] prose-table:mb-[unset] | prose-table:border-collapse | prose-p:prose-td:text-[0.875rem] | prose-p:prose-th:text-[0.875rem] | prose-li:prose-th:text-[0.875rem] | prose-li:prose-td:text-[0.875rem] | prose-td:border-solid prose-td:border-l-[0rem] prose-td:border-r-[0rem] prose-td:border-jiscAlto | prose-td:border-b-[0.0625rem] prose-td:border-t-[0.0625rem] | prose-td:p-4 prose-td:align-middle prose-td:leading-[1.5rem]| prose-th:border-solid prose-th:border-l-[0rem] prose-th:border-r-[0rem] prose-th:border-jiscAlto | prose-th:border-b-[0.0625rem] prose-th:border-t-[0.0625rem] prose-th:border-b-jiscAlto prose-th:border-t-jiscAlto | prose-th:p-4 prose-th:align-middle prose-th:leading-[1.5rem] | prose-th:font-medium | prose-tr:border-b-jiscAlto prose-tr:border-t-jiscAlto | prose-img:block prose-img:max-w-full prose-img:h-auto prose-img:my-6 prose-img:leading-none | prose-a:font-normal prose-a:text-jiscBahamaBlue prose-a:underline | visited:prose-a:text-jiscPurple | focus:prose-a:text-jiscBlack focus:prose-a:bg-jiscDandelion focus:prose-a:outline focus:prose-a:outline-4 focus:prose-a:outline-transparent focus:prose-a:no-underline focus:prose-a:shadow-[0_.25rem_0_0_#fd6,_0_-.25rem_0_0_#fd6] | hover:prose-a:text-jiscRegalBlue | active:prose-a:text-jiscBlack | focus:active:prose-a:underline focus:active:prose-a:bg-transparent focus:active:prose-a:shadow-none | prose-strong:prose-a:text-jiscBahamaBlue | prose-strong:visited:prose-a:text-jiscPurple | prose-strong:focus:prose-a:text-jiscBlack | prose-strong:hover:prose-a:text-jiscRegalBlue | prose-strong:active:prose-a:text-jiscBlack || prose-blockquote:border-l-jiscAlto || marker:text-jiscMineShaft | prose-blockquote:text-jiscDoveGrey | prose-headings:text-jiscMineShaft | prose-li:text-jiscMineShaft | prose-ol:text-jiscMineShaft | prose-p:text-jiscMineShaft | prose-strong:text-jiscMineShaft | prose-table:text-jiscMineShaft | prose-ul:text-jiscMineShaft | | mt-6 mb-6 |"><h2 id="rules-of-thumb-68zpo" data-block-key="68zpo">Rules of thumb</h2><p data-block-key="fa2n1">High-quality question-and-answer sets need to be structured in a way that aligns with the logic of how the chatbot works. Chatbots use utterances (different wordings for the same question) that have been inputted in order to build machine learning models, which then enables them to establish whether a user’s query has the same meaning/intention as a question in the question-and-answer set. This underlying mechanism gives rise to several rules of thumb for how to develop question-and-answer sets:</p><h3 data-block-key="cbptr">Make sure all utterances within a given intent are equivalent</h3><p data-block-key="1advi">The chatbot’s machine learning models will assume that, within a given intent, all the utterances are equivalent. In other words, it will assume that you are grouping questions into differently worded versions of the same question. Based on this assumption, the chatbot is then able to build models, which will establish the connections between these utterances.</p><p data-block-key="3jhp7">One implication of this is that if those developing the question-and-answer sets input non-equivalent utterances (for instance, “where is the library?” and “are there lifts I can take to get to the library?” are not equivalent wordings of the same question), then the chatbot will build its model based on incorrect assumptions. This is likely to diminish the chatbot’s performance.</p><p data-block-key="1oova">It is suggested that when listing possible ways of wording a question, consider whether you are in fact capturing multiple related questions. If you are, the best approach may be to split up the intent into multiple intents. With the example above, for instance, it may be advantageous to have two separate intents: one for finding out where the library is, and one for finding out whether the library can be accessed via lift.</p><h3 data-block-key="3gr2k">Utterances should be unique to a particular intent</h3><p data-block-key="4hp8g">As well as all utterances within an intent needing to be equivalent, it is also essential that utterances are unique to one intent.</p><p data-block-key="59eo4">Consider the following two groups of questions:</p><ul><li data-block-key="48lkk">The first is a grouping of questions about getting support with UCAS applications.</li><li data-block-key="94a0r">The second is a grouping of questions about getting support with personal statements.</li></ul><p data-block-key="3r301">Someone inputting utterances for these intents, may consider that the utterance “support with university application” may be relevant to either.</p><p data-block-key="a3tl8">If this utterance were inputted into both intents and a student asked, “I need support with my uni application”, the chatbot would then identify that the intention behind the user question is similar to both the intents on UCAS and personal statements. It would likely struggle to establish which was most relevant – which increases the overall chance of inaccuracy.</p><p data-block-key="6j3d9">One solution in the above case would be to have a separate intent for generic queries about university support, in addition to having intents for specific queries (ie around UCAS and personal statements – or even more granular).</p></div><div class=" clearfix | prose | antialiased | font-sans | max-w-full | prose-h2:text-[2rem] prose-h2:font-normal prose-h2:leading-[3.0rem] prose-h2:sm:text-[2.25rem] prose-h2:md:text-[2.5rem] prose-h2:md:leading-[3.5rem] | prose-h2:mt-10 prose-h2:mb-6 first:prose-h2:mt-0 last:prose-h2:mb-0 prose-h3:text-[1.5rem] prose-h3:font-medium prose-h3:leading-[2rem] first:prose-h3:mt-0 last:prose-h3:mb-0 prose-h3:mt-10 prose-h3:mb-6 prose-h3:sm:text-[1.75rem] prose-h3:sm:leading-[2.5rem] prose-h3:md:text-[2rem] prose-h3:md:leading-[3rem] | prose-h4:text-[1.25rem] prose-h4:font-normal prose-h4:leading-[2rem] first:prose-h4:mt-0 last:prose-h4:mb-0 prose-h4:mt-10 prose-h4:mb-6 prose-h4:md:text-[1.5rem] prose-h4:md:leading-[2rem] | prose-h5:text-[1rem] prose-h5:font-medium prose-h5:leading-[1.5rem] prose-h5:mt-10 prose-h5:mb-6 first:prose-h5:mt-[unset] last:prose-h5:mb-[unset] prose-h5:md:text-[1.25rem] prose-h5:md:leading-[2rem] prose-h6:text-[1rem] prose-h6:font-semibold prose-h6:leading-[1.5rem] prose-h6:mt-10 prose-h6:mb-6 first:prose-h6:mt-[unset] last:prose-h6:mb-[unset] prose-p:my-6 prose-p:text-[1rem] prose-p:font-normal prose-p:leading-[1.5rem] first:prose-p:mt-[unset] last:prose-p:mb-[unset] empty:first:prose-p:-mt-6 prose-blockquote:my-[unset] prose-blockquote:ml-6 prose-blockquote:text-[1rem] prose-blockquote:not-italic prose-blockquote:font-normal prose-blockquote:leading-[1.5rem] prose-blockquote:border-solid prose-blockquote:border-0 prose-blockquote:border-l-[.0625rem] prose-p:prose-blockquote:py-3 before:prose-p:prose-blockquote:content-[unset] after:prose-p:prose-blockquote:content-[unset] prose-li:text-[1rem] prose-li:font-normal prose-li:leading-[1.5rem] | first:prose-li:mt-2 prose-li:pl-2 | first:prose-ol:mt-0 last:prose-ol:mb-0 prose-ol:my-6 prose-ol:pl-8 prose-ol:leading-[1.5rem] | first:prose-ul:mt-0 last:prose-ul:mb-0 prose-ul:my-6 prose-ul:pl-8 prose-ul:leading-[1.5rem] | prose-table:leading-[1.5rem] | prose-table:mt-[unset] prose-table:mb-[unset] | prose-table:border-collapse | prose-p:prose-td:text-[0.875rem] | prose-p:prose-th:text-[0.875rem] | prose-li:prose-th:text-[0.875rem] | prose-li:prose-td:text-[0.875rem] | prose-td:border-solid prose-td:border-l-[0rem] prose-td:border-r-[0rem] prose-td:border-jiscAlto | prose-td:border-b-[0.0625rem] prose-td:border-t-[0.0625rem] | prose-td:p-4 prose-td:align-middle prose-td:leading-[1.5rem]| prose-th:border-solid prose-th:border-l-[0rem] prose-th:border-r-[0rem] prose-th:border-jiscAlto | prose-th:border-b-[0.0625rem] prose-th:border-t-[0.0625rem] prose-th:border-b-jiscAlto prose-th:border-t-jiscAlto | prose-th:p-4 prose-th:align-middle prose-th:leading-[1.5rem] | prose-th:font-medium | prose-tr:border-b-jiscAlto prose-tr:border-t-jiscAlto | prose-img:block prose-img:max-w-full prose-img:h-auto prose-img:my-6 prose-img:leading-none | prose-a:font-normal prose-a:text-jiscBahamaBlue prose-a:underline | visited:prose-a:text-jiscPurple | focus:prose-a:text-jiscBlack focus:prose-a:bg-jiscDandelion focus:prose-a:outline focus:prose-a:outline-4 focus:prose-a:outline-transparent focus:prose-a:no-underline focus:prose-a:shadow-[0_.25rem_0_0_#fd6,_0_-.25rem_0_0_#fd6] | hover:prose-a:text-jiscRegalBlue | active:prose-a:text-jiscBlack | focus:active:prose-a:underline focus:active:prose-a:bg-transparent focus:active:prose-a:shadow-none | prose-strong:prose-a:text-jiscBahamaBlue | prose-strong:visited:prose-a:text-jiscPurple | prose-strong:focus:prose-a:text-jiscBlack | prose-strong:hover:prose-a:text-jiscRegalBlue | prose-strong:active:prose-a:text-jiscBlack || prose-blockquote:border-l-jiscAlto || marker:text-jiscMineShaft | prose-blockquote:text-jiscDoveGrey | prose-headings:text-jiscMineShaft | prose-li:text-jiscMineShaft | prose-ol:text-jiscMineShaft | prose-p:text-jiscMineShaft | prose-strong:text-jiscMineShaft | prose-table:text-jiscMineShaft | prose-ul:text-jiscMineShaft | | mt-6 mb-6 |"><h2 id="refining-question-and-answer-sets-through-pre-launch-testing-68zpo" data-block-key="68zpo">Refining question-and-answer sets through pre-launch testing</h2><p data-block-key="a75q8">The first draft of the question-and-answer set will have made many assumptions about the kinds of questions users will want answered, and the ways in which those questions will be asked. As such, it is unlikely to enable the chatbot to perform optimally.</p><p data-block-key="6mbi5">Fortunately, pre-launch testing will provide useful insights that can improve the quality of the question-and-answer set, and thus the performance of the chatbot.</p><h3 data-block-key="5nhat">Step 1: selecting testing subjects</h3><p data-block-key="6so00">One way to diagnose areas for improvement in the question-and-answer set is to test the chatbot before it is launched, using either students, staff or a mixture of both as testers.</p><p data-block-key="bhdb7">An advantage of testing the chatbot on students is that the questions they put to the chatbot are more likely to be representative of what students would ask the live chatbot (particularly where students are the intended users).</p><p data-block-key="ab6bo">An advantage of testing the chatbot on staff is that this can be easier and quicker to organise, which also makes it straightforward to do multiple iterations of testing and improvement.</p><h3 data-block-key="6t73q">Step 2: collecting data from testing</h3><p data-block-key="68g8h">Two core things to learn through testing are:</p><ul><li data-block-key="o3nm">Is the chatbot able to respond to queries with relevant information? (in other words, is the chatbot accurate?)</li><li data-block-key="1gf8k">Are the chatbot’s responses useful? (Are the answers clear and informative?)</li></ul><p data-block-key="6ph02">A formalised way of collecting this information would be to ask testing subjects these questions after each query they raise with the chatbot, and then to collate the overall results.</p><p data-block-key="btsnf">Because chatbot accuracy/usefulness may vary between topic areas, it may also be helpful to collect data on the topic areas asked for each question, so that you can judge whether this influences the two measures of success.</p><h3 data-block-key="brnl4">Step 3: responding to pre-launch testing, and conducting further testing</h3><p data-block-key="bflg1">The main purpose of pre-launch testing is to diagnose areas for development in the question-and-answer set. As such, it is important to make changes based on the feedback received.</p><p data-block-key="2jbvc">Once the first round of changes has been made it is sensible to conduct further testing to evaluate whether any shortcomings have been addressed. This poses the question of when the cycle ends, and at what point it becomes appropriate to launch the chatbot.</p><p data-block-key="ffh8u">One point to consider here is that the version of the chatbot that is launched will not be the finished product. Notably, there should be updates to the question-and-answer set as more and more data is received from the live queries raised by users, and the way in which the chatbot responds.</p><p data-block-key="6bjp9">As such, consider the following as steers for when to deploy the chatbot:</p><ul><li data-block-key="4dj1t">The main risks of deploying the chatbot too late (ie after lengthy and extensive testing) are the opportunity costs of not having use of the chatbot during the testing period and the effort involved in ongoing testing.</li><li data-block-key="7qc56">The main risk of deploying the chatbot too early (ie after only minimal testing) is that it may perform poorly and put off users. Note, however, that users’ expectations can be managed in order to reduce the risk of this outcome.</li></ul></div><div class=" clearfix | prose | antialiased | font-sans | max-w-full | prose-h2:text-[2rem] prose-h2:font-normal prose-h2:leading-[3.0rem] prose-h2:sm:text-[2.25rem] prose-h2:md:text-[2.5rem] prose-h2:md:leading-[3.5rem] | prose-h2:mt-10 prose-h2:mb-6 first:prose-h2:mt-0 last:prose-h2:mb-0 prose-h3:text-[1.5rem] prose-h3:font-medium prose-h3:leading-[2rem] first:prose-h3:mt-0 last:prose-h3:mb-0 prose-h3:mt-10 prose-h3:mb-6 prose-h3:sm:text-[1.75rem] prose-h3:sm:leading-[2.5rem] prose-h3:md:text-[2rem] prose-h3:md:leading-[3rem] | prose-h4:text-[1.25rem] prose-h4:font-normal prose-h4:leading-[2rem] first:prose-h4:mt-0 last:prose-h4:mb-0 prose-h4:mt-10 prose-h4:mb-6 prose-h4:md:text-[1.5rem] prose-h4:md:leading-[2rem] | prose-h5:text-[1rem] prose-h5:font-medium prose-h5:leading-[1.5rem] prose-h5:mt-10 prose-h5:mb-6 first:prose-h5:mt-[unset] last:prose-h5:mb-[unset] prose-h5:md:text-[1.25rem] prose-h5:md:leading-[2rem] prose-h6:text-[1rem] prose-h6:font-semibold prose-h6:leading-[1.5rem] prose-h6:mt-10 prose-h6:mb-6 first:prose-h6:mt-[unset] last:prose-h6:mb-[unset] prose-p:my-6 prose-p:text-[1rem] prose-p:font-normal prose-p:leading-[1.5rem] first:prose-p:mt-[unset] last:prose-p:mb-[unset] empty:first:prose-p:-mt-6 prose-blockquote:my-[unset] prose-blockquote:ml-6 prose-blockquote:text-[1rem] prose-blockquote:not-italic prose-blockquote:font-normal prose-blockquote:leading-[1.5rem] prose-blockquote:border-solid prose-blockquote:border-0 prose-blockquote:border-l-[.0625rem] prose-p:prose-blockquote:py-3 before:prose-p:prose-blockquote:content-[unset] after:prose-p:prose-blockquote:content-[unset] prose-li:text-[1rem] prose-li:font-normal prose-li:leading-[1.5rem] | first:prose-li:mt-2 prose-li:pl-2 | first:prose-ol:mt-0 last:prose-ol:mb-0 prose-ol:my-6 prose-ol:pl-8 prose-ol:leading-[1.5rem] | first:prose-ul:mt-0 last:prose-ul:mb-0 prose-ul:my-6 prose-ul:pl-8 prose-ul:leading-[1.5rem] | prose-table:leading-[1.5rem] | prose-table:mt-[unset] prose-table:mb-[unset] | prose-table:border-collapse | prose-p:prose-td:text-[0.875rem] | prose-p:prose-th:text-[0.875rem] | prose-li:prose-th:text-[0.875rem] | prose-li:prose-td:text-[0.875rem] | prose-td:border-solid prose-td:border-l-[0rem] prose-td:border-r-[0rem] prose-td:border-jiscAlto | prose-td:border-b-[0.0625rem] prose-td:border-t-[0.0625rem] | prose-td:p-4 prose-td:align-middle prose-td:leading-[1.5rem]| prose-th:border-solid prose-th:border-l-[0rem] prose-th:border-r-[0rem] prose-th:border-jiscAlto | prose-th:border-b-[0.0625rem] prose-th:border-t-[0.0625rem] prose-th:border-b-jiscAlto prose-th:border-t-jiscAlto | prose-th:p-4 prose-th:align-middle prose-th:leading-[1.5rem] | prose-th:font-medium | prose-tr:border-b-jiscAlto prose-tr:border-t-jiscAlto | prose-img:block prose-img:max-w-full prose-img:h-auto prose-img:my-6 prose-img:leading-none | prose-a:font-normal prose-a:text-jiscBahamaBlue prose-a:underline | visited:prose-a:text-jiscPurple | focus:prose-a:text-jiscBlack focus:prose-a:bg-jiscDandelion focus:prose-a:outline focus:prose-a:outline-4 focus:prose-a:outline-transparent focus:prose-a:no-underline focus:prose-a:shadow-[0_.25rem_0_0_#fd6,_0_-.25rem_0_0_#fd6] | hover:prose-a:text-jiscRegalBlue | active:prose-a:text-jiscBlack | focus:active:prose-a:underline focus:active:prose-a:bg-transparent focus:active:prose-a:shadow-none | prose-strong:prose-a:text-jiscBahamaBlue | prose-strong:visited:prose-a:text-jiscPurple | prose-strong:focus:prose-a:text-jiscBlack | prose-strong:hover:prose-a:text-jiscRegalBlue | prose-strong:active:prose-a:text-jiscBlack || prose-blockquote:border-l-jiscAlto || marker:text-jiscMineShaft | prose-blockquote:text-jiscDoveGrey | prose-headings:text-jiscMineShaft | prose-li:text-jiscMineShaft | prose-ol:text-jiscMineShaft | prose-p:text-jiscMineShaft | prose-strong:text-jiscMineShaft | prose-table:text-jiscMineShaft | prose-ul:text-jiscMineShaft | | mt-6 |"><h2 id="about-our-work-in-ai-68zpo" data-block-key="68zpo">About our work in AI</h2><p data-block-key="i03m">Our <a href="/web/20240227045235/https://www.jisc.ac.uk/innovation/artificial-intelligence/">work in AI</a> accelerates the adoption of effective and ethical AI solutions within colleges and universities. As part of working towards this goal, we have conducted a pilot in which chatbots have been deployed in four UK further education colleges.</p><p data-block-key="378hn">Conducting this pilot has enabled us to develop expertise in educational chatbots, which has, in turn, motivated the publication of a series of guidance documents on the subject of chatbots. This guide is the third in that series. The additional guides are:</p><ul><li data-block-key="drrqh"><a href="/web/20240227045235/https://www.jisc.ac.uk/guides/an-introduction-to-chatbots/">An introduction to chatbots</a> – the first guide in the series, which is intended for those who want to learn the basics of how chatbots function and how they can be used in education.</li><li data-block-key="l2i9"><a href="/web/20240227045235/https://www.jisc.ac.uk/guides/preparing-for-chatbots-by-engaging-key-internal-stakeholders/">Preparing for chatbots by engaging key internal stakeholders</a> – the second guide in the series, which is intended for those who have (firmly or tentatively) decided to deploy a chatbot, and want guidance on what to do next.</li></ul></div><div class="p-10 my-10 font-sans text-sm antialiased font-normal bg-jiscConcrete text-jiscMineShaft">This guide is made available under <a href="http://web.archive.org/web/20240227045235/https://creativecommons.org/licenses/by-nc-nd/4.0/">Creative Commons License (CC BY-NC-ND).</a></div></div></div></div></div></article></main><footer aria-labelledby="site-footer" class="jss35"><div class="jss39 jss36 jss40"><div class="MuiContainer-root MuiContainer-maxWidthMd"><div class="MuiGrid-root jss48 MuiGrid-container"><div class="MuiGrid-root jss49 MuiGrid-item MuiGrid-grid-xs-12 MuiGrid-grid-sm-9"><h2 class="MuiTypography-root MuiTypography-srOnly" id="site-footer">Site footer</h2><div class=""><nav aria-label="Site footer breadcrumb" class="jss43"><ul class="jss45"><li class="jss46"><a aria-label="Jisc homepage" class="jss44" href="/web/20240227045235/https://www.jisc.ac.uk/" title="Jisc homepage"><svg aria-hidden="true" focusable="false" data-prefix="fas" data-icon="house" class="svg-inline--fa fa-house " role="img" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 576 512"><path fill="currentColor" d="M575.8 255.5c0 18-15 32.1-32 32.1h-32l.7 160.2c0 2.7-.2 5.4-.5 8.1V472c0 22.1-17.9 40-40 40H456c-1.1 0-2.2 0-3.3-.1c-1.4 .1-2.8 .1-4.2 .1H416 392c-22.1 0-40-17.9-40-40V448 384c0-17.7-14.3-32-32-32H256c-17.7 0-32 14.3-32 32v64 24c0 22.1-17.9 40-40 40H160 128.1c-1.5 0-3-.1-4.5-.2c-1.2 .1-2.4 .2-3.6 .2H104c-22.1 0-40-17.9-40-40V360c0-.9 0-1.9 .1-2.8V287.6H32c-18 0-32-14-32-32.1c0-9 3-17 10-24L266.4 8c7-7 15-8 22-8s15 2 21 7L564.8 231.5c8 7 12 15 11 24z"></path></svg></a></li><li class="jss46"><span class="jss47"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/guides/">Guides</a></span></li><li aria-current="page" class="jss46"><span class="jss47">Developing high-quality question-and-answer sets for chatbots</span></li></ul></nav></div></div><div class="MuiGrid-root jss49 MuiGrid-item MuiGrid-grid-xs-12 MuiGrid-grid-sm-3"></div></div></div></div><div class="jss39 jss37 jss41"><div class="MuiContainer-root MuiContainer-maxWidthMd"><h3 class="MuiTypography-root MuiTypography-srOnly">Jisc product areas</h3><div class="MuiGrid-root jss48 MuiGrid-container"><div class="MuiGrid-root jss49 MuiGrid-item MuiGrid-grid-xs-12 MuiGrid-grid-sm-3"><h4 class="MuiTypography-root jss53 MuiTypography-body1">Areas</h4><ul class="jss51"><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/connectivity">Connectivity</a></li><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/cyber-security">Cyber security</a></li><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/cloud">Cloud</a></li><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/data-analytics">Data and analytics</a></li><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/libraries-learning-resources-and-research">Libraries, learning resources and research</a></li><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/student-experience">Student experience</a></li><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/trust-and-identity">Trust and identity</a></li><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/advice-and-guidance">Advice and guidance</a></li></ul></div><div class="MuiGrid-root jss49 MuiGrid-item MuiGrid-grid-xs-12 MuiGrid-grid-sm-3"><h4 class="MuiTypography-root jss53 jss52 MuiTypography-body1">Explore</h4><ul class="jss51"><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/guides">Guides</a></li><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/training">Training</a></li><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/consultancy">Consultancy</a></li><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/events">Events</a></li><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/innovation">Innovation</a></li></ul></div><div class="MuiGrid-root jss49 MuiGrid-item MuiGrid-grid-xs-12 MuiGrid-grid-sm-3"><h4 class="MuiTypography-root jss53 jss52 MuiTypography-body1">Useful</h4><ul class="jss51"><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/about-us">About</a></li><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/membership">Membership</a></li><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/get-involved">Get involved</a></li><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/news">News</a></li><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/jobs">Jobs</a></li></ul></div><div class="MuiGrid-root jss49 MuiGrid-item MuiGrid-grid-xs-12 MuiGrid-grid-sm-3"><h4 class="MuiTypography-root jss53 jss52 MuiTypography-body1">Get in touch</h4><ul class="jss51"><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/contact">Contact us</a></li><li class="jss54"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/forms/headlines">Sign up to our newsletter</a></li><li class="jss54"><svg aria-hidden="true" focusable="false" data-prefix="fab" data-icon="twitter" class="svg-inline--fa fa-twitter jss55" role="img" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 512 512"><path fill="currentColor" d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"></path></svg><a class="jss60" href="http://web.archive.org/web/20240227045235/https://twitter.com/jisc">Twitter</a></li><li class="jss54"><svg aria-hidden="true" focusable="false" data-prefix="fab" data-icon="facebook" class="svg-inline--fa fa-facebook jss55" role="img" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 512 512"><path fill="currentColor" d="M512 256C512 114.6 397.4 0 256 0S0 114.6 0 256C0 376 82.7 476.8 194.2 504.5V334.2H141.4V256h52.8V222.3c0-87.1 39.4-127.5 125-127.5c16.2 0 44.2 3.2 55.7 6.4V172c-6-.6-16.5-1-29.6-1c-42 0-58.2 15.9-58.2 57.2V256h83.6l-14.4 78.2H287V510.1C413.8 494.8 512 386.9 512 256h0z"></path></svg><a class="jss60" href="http://web.archive.org/web/20240227045235/https://www.facebook.com/jiscsocial">Facebook</a></li><li class="jss54"><svg aria-hidden="true" focusable="false" data-prefix="fab" data-icon="linkedin" class="svg-inline--fa fa-linkedin jss55" role="img" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 448 512"><path fill="currentColor" d="M416 32H31.9C14.3 32 0 46.5 0 64.3v383.4C0 465.5 14.3 480 31.9 480H416c17.6 0 32-14.5 32-32.3V64.3c0-17.8-14.4-32.3-32-32.3zM135.4 416H69V202.2h66.5V416zm-33.2-243c-21.3 0-38.5-17.3-38.5-38.5S80.9 96 102.2 96c21.2 0 38.5 17.3 38.5 38.5 0 21.3-17.2 38.5-38.5 38.5zm282.1 243h-66.4V312c0-24.8-.5-56.7-34.5-56.7-34.6 0-39.9 27-39.9 54.9V416h-66.4V202.2h63.7v29.2h.9c8.9-16.8 30.6-34.5 62.9-34.5 67.2 0 79.7 44.3 79.7 101.9V416z"></path></svg><a class="jss60" href="http://web.archive.org/web/20240227045235/https://www.linkedin.com/company/jisc">LinkedIn</a></li><li class="jss54"><svg aria-hidden="true" focusable="false" data-prefix="fab" data-icon="youtube" class="svg-inline--fa fa-youtube jss55" role="img" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 576 512"><path fill="currentColor" d="M549.655 124.083c-6.281-23.65-24.787-42.276-48.284-48.597C458.781 64 288 64 288 64S117.22 64 74.629 75.486c-23.497 6.322-42.003 24.947-48.284 48.597-11.412 42.867-11.412 132.305-11.412 132.305s0 89.438 11.412 132.305c6.281 23.65 24.787 41.5 48.284 47.821C117.22 448 288 448 288 448s170.78 0 213.371-11.486c23.497-6.321 42.003-24.171 48.284-47.821 11.412-42.867 11.412-132.305 11.412-132.305s0-89.438-11.412-132.305zm-317.51 213.508V175.185l142.739 81.205-142.739 81.201z"></path></svg><a class="jss60" href="http://web.archive.org/web/20240227045235/http://www.youtube.com/user/JISCmedia">YouTube</a></li></ul></div></div></div></div><div class="jss39 jss38 jss42"><div class="MuiContainer-root MuiContainer-maxWidthMd"><div class="MuiGrid-root jss48 MuiGrid-container"><div class="MuiGrid-root jss49 MuiGrid-item MuiGrid-grid-xs-12 MuiGrid-grid-sm-9"><h3 class="MuiTypography-root MuiTypography-srOnly">Site utilities</h3><ul class="jss56"><li class="jss58"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/cookies">Cookies</a></li><li class="jss58"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/website/privacy-notice">Privacy</a></li><li class="jss58"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/about-us/modern-slavery-statement">Modern slavery</a></li><li class="jss58"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/about-us/carbon-reduction-plan">Carbon reduction plan</a></li><li class="jss58"><a class="jss60" href="/web/20240227045235/https://www.jisc.ac.uk/accessibility-statement">Accessibility</a></li></ul></div></div></div></div></footer></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"path":["guides","developing-high-quality-question-and-answer-sets-for-chatbots"],"pageData":{"id":486,"meta":{"type":"base.GuidePage","detail_url":"http://web.archive.org/web/20240227045235/http://backend:8000/api/v2/pages/486/","html_url":"http://web.archive.org/web/20240227045235/https://www.jisc.ac.uk/guides/developing-high-quality-question-and-answer-sets-for-chatbots/","slug":"developing-high-quality-question-and-answer-sets-for-chatbots","show_in_menus":false,"seo_title":"Developing high-quality question-and-answer sets for chatbots","search_description":"You've started work on introducing chatbots to your organisation, now find out about what's involved in development and implementation.","first_published_at":"2023-01-23T16:50:00Z","alias_of":null,"parent":{"id":241,"meta":{"type":"base.GuideListingPage","detail_url":"http://web.archive.org/web/20240227045235/http://backend:8000/api/v2/pages/241/","html_url":"http://web.archive.org/web/20240227045235/https://www.jisc.ac.uk/guides/"},"title":"Guides"}},"title":"Developing high-quality question-and-answer sets for chatbots","categories":[{"id":9,"name":"Student experience"},{"id":14,"name":"Research"},{"id":20,"name":"Artificial intelligence"}],"authors":[],"breadcrumbs":[{"id":1,"name":"Root","url":null},{"id":3,"name":"Jisc","url":"/"},{"id":241,"name":"Guides","url":"/guides/"},{"id":486,"name":"Developing high-quality question-and-answer sets for chatbots","url":"/guides/developing-high-quality-question-and-answer-sets-for-chatbots/"}],"copyright_banner":true,"last_published_at":"2024-02-26T14:09:38.662888Z","open_graph_image":null,"open_graph_image_alt":"","override_last_published_at":"2023-01-23T16:50:00Z","page_body":[{"type":"content_block","value":"\u003ch2 id=\"introduction-66cow\" data-block-key=\"66cow\"\u003eIntroduction\u003c/h2\u003e\u003cp data-block-key=\"4l8cp\"\u003eThis guide is for anyone who has begun work on implementing chatbots and would like guidance on how to develop high-quality question-and-answer sets. It may also be of use to those who are strongly considering implementing a chatbot and want to find out more about the work involved in development and implementation.\u003c/p\u003e\u003cp data-block-key=\"abnfj\"\u003eThis guide begins by explaining the importance of high-quality question-and-answer sets for chatbot performance. It then outlines the structure of question-and-answer sets, so that readers are:\u003c/p\u003e\u003cul\u003e\u003cli data-block-key=\"buqf6\"\u003eIntroduced to the key terms used when describing chatbot question-and-answer sets\u003c/li\u003e\u003cli data-block-key=\"9gl0b\"\u003eAble to understand central concepts underpinning chatbot use\u003c/li\u003e\u003cli data-block-key=\"qadk\"\u003eAble to decide whether the guidance in this document is relevant to the type of chatbot they are using\u003c/li\u003e\u003c/ul\u003e\u003cp data-block-key=\"7lf71\"\u003eWe then look at the initial steps that should be taken to develop high-quality question-and-answer sets and provides a few ‘rules of thumb’ that will help in developing them. Finally, we look at the steps that can be taken to refine question-and-answer sets and manage their ongoing development once the chatbot has gone live.\u003c/p\u003e","id":"0e0db4bf-b41d-4cd9-9b1e-f0f5f6879520"},{"type":"content_block","value":"\u003ch2 id=\"the-importance-of-question-and-answer-sets-68zpo\" data-block-key=\"68zpo\"\u003eThe importance of question-and-answer sets\u003c/h2\u003e\u003cp data-block-key=\"419ii\"\u003eFor users to benefit from chatbot use, those developing chatbot content must anticipate the kinds of questions that users are likely to ask and the information they would need in response. As such, chatbot performance depends on more than just the effectiveness of the software being used: performance depends on the calibre of the content too. Ultimately, the quality of a question-and-answer set depends on:\u003c/p\u003e\u003cul\u003e\u003cli data-block-key=\"f5t9t\"\u003eHow well it captures the information that users want to find out from using the chatbot\u003c/li\u003e\u003cli data-block-key=\"5nh9p\"\u003eHow well its structure aligns with the logic of how the chatbot works\u003c/li\u003e\u003c/ul\u003e","id":"e585a7b2-7130-43b1-a343-fc8391f068c0"},{"type":"content_block","value":"\u003ch2 id=\"the-structure-of-question-and-answer-sets-68zpo\" data-block-key=\"68zpo\"\u003eThe structure of question-and-answer sets\u003c/h2\u003e\u003cp data-block-key=\"1uuqb\"\u003eA typical question-and-answer set is built upon three fundamental elements:\u003c/p\u003e\u003cul\u003e\u003cli data-block-key=\"d7fm1\"\u003e\u003cstrong\u003eIntents\u003c/strong\u003e – The underlying meaning or intention behind a question\u003c/li\u003e\u003cli data-block-key=\"9lgli\"\u003e\u003cstrong\u003eUtterances\u003c/strong\u003e – Different ways of wording a particular question\u003c/li\u003e\u003cli data-block-key=\"d1nkt\"\u003e\u003cstrong\u003eFulfilments\u003c/strong\u003e – The answer to a user\u0026#x27;s question\u003c/li\u003e\u003c/ul\u003e\u003cp data-block-key=\"7uf4\"\u003eFor example:\u003c/p\u003e\u003cul\u003e\u003cli data-block-key=\"eu1s4\"\u003e\u003cstrong\u003eIntent\u003c/strong\u003e – A user wants to find out if the cafeteria caters for vegans\u003c/li\u003e\u003cli data-block-key=\"2klvn\"\u003e\u003cstrong\u003eUtterances\u003c/strong\u003e – “Do you serve vegan food in college?”, “Is the food in the café suitable for vegans?”, “Do you cater for vegans?”, “I’m vegan, will I be able to get lunch in college?”\u003c/li\u003e\u003cli data-block-key=\"56o0f\"\u003e\u003cstrong\u003eFulfilment\u003c/strong\u003e – “The college cafeteria offers a range of dishes that are suitable for vegans. To see the menu for the next two weeks, please visit...”\u003c/li\u003e\u003c/ul\u003e\u003cp data-block-key=\"f2djh\"\u003eGiving the chatbot multiple examples of how a question could be asked (in other words, inputting numerous utterances), enables the chatbot’s machine learning to build models of the intent so that the chatbot can respond to questions that have been worded in a way that was not anticipated. Take the example above of the intent relating to vegan food. It would be very difficult – potentially impossible – to anticipate all the ways in which a user might express their intention of finding out whether the cafeteria caters for vegans.\u003c/p\u003e\u003cp data-block-key=\"e64u0\"\u003eThat said, if the chatbot is given a large enough sample of example questions (ie utterances), it will be able to use machine learning models to establish the underlying meaning that unites them. This means that if a user were to ask “is there vegan food in the college cafeteria?”, the chatbot should be able to recognise the meaning (intent) behind this question, and hence respond with the answer (fulfilment) given in the table above.\u003c/p\u003e","id":"e4a0112f-9cbb-4193-a53c-6e8c851a21cc"},{"type":"content_block","value":"\u003ch2 id=\"first-steps-68zpo\" data-block-key=\"68zpo\"\u003eFirst steps\u003c/h2\u003e\u003ch3 data-block-key=\"2vj21\"\u003eStep 1: clarify the purpose of your chatbot\u003c/h3\u003e\u003cp data-block-key=\"2d65k\"\u003eBefore beginning work on the question-and-answer set, it is important to pinpoint the aims and objectives of the chatbot itself. Doing so will help you to shape the topics you want the question-and-answer set to focus on, while also providing useful steers on the chatbot’s tone and the kind of information that should be conveyed through its responses to users.\u003c/p\u003e\u003ch3 data-block-key=\"767cq\"\u003eStep 2: identify the questions your chatbot needs to be able to answer\u003c/h3\u003e\u003cp data-block-key=\"e73q7\"\u003eThe use case for the chatbot will determine, in general terms, the topics that need to be captured in the question-and-answer set. After this, the next step is to think carefully about the kinds of queries that users are likely to raise, and the kinds of responses that will satisfy these queries. Where possible, a useful place to start is to gather evidence on what kinds of queries users are already raising. For instance, if a chatbot’s purpose is to answer new students’ queries around enrolment – queries that might normally be raised to staff at a student help desk – then evidence of the kinds of questions that are normally asked could be gained by:\u003c/p\u003e\u003cul\u003e\u003cli data-block-key=\"5js7f\"\u003eUsing the experience of staff at the help desk\u003c/li\u003e\u003cli data-block-key=\"eej54\"\u003eReviewing relevant emails sent by students to the help desk\u003c/li\u003e\u003c/ul\u003e\u003cp data-block-key=\"bd89o\"\u003eIn some cases, it may not be possible to gather an evidence base around the queries students are likely to have (the chatbot may be responding to student queries about a new process, for example). Where this is the case, it is suggested that professional judgement is used to anticipate the kinds of questions that students might raise.\u003c/p\u003e\u003cp data-block-key=\"5943i\"\u003eFor instance, if a chatbot’s purpose is to answer student questions about a new set of courses on offer, then those developing the question-and-answer set (alongside those with expertise in these new courses, perhaps) should review how students are receiving information and anticipate where they may want more detail or clarification on certain points.\u003c/p\u003e\u003cp data-block-key=\"b8o6q\"\u003eWhatever means are used to predict which questions will be asked, it is important to capture them systematically so that they can be referred back to by all individuals involved in the question-and-answer set development.\u003c/p\u003e\u003ch3 data-block-key=\"bcaso\"\u003eStep 3: establish how users are likely to ask their questions\u003c/h3\u003e\u003cp data-block-key=\"diqsj\"\u003eHaving established which questions the chatbot will need to answer, the next step is to identify the variety of ways in which these questions could be worded. This is because the chatbot needs to be given a number of example wordings (utterances) of a particular question (intent) so that it can build machine learning-based models of the full range of ways in which that question could be worded.\u003c/p\u003e\u003cp data-block-key=\"d8vpt\"\u003eIt is difficult to predict precisely how users will word their questions. That said, there are a number of methods that can be employed to make the task easier:\u003c/p\u003e\u003cul\u003e\u003cli data-block-key=\"ai6g1\"\u003eWhere possible, draw upon evidence of how students have raised such queries in the past (for example, ask relevant members of staff, review appropriate email records)\u003c/li\u003e\u003cli data-block-key=\"8gt43\"\u003eWork collaboratively within teams when generating multiple wordings for the same question, to draw upon different colleagues’ insights\u003c/li\u003e\u003cli data-block-key=\"d9gt7\"\u003eWhere appropriate, get input from users on how they might word questions (this can be particularly useful for anticipating the jargon users might use)\u003c/li\u003e\u003c/ul\u003e\u003cp data-block-key=\"a8aeh\"\u003eWhen generating different wordings for a question, it is important to know how many different wordings (utterances) should be input into the question-and-answer set. From experience, it has been found that using five to ten utterances (different ways of wording the same question) per intent (group of questions that all share the same meaning), tends to be more effective than using less than five utterances. It has also been found that using more than ten utterances can yield diminishing returns or be counterproductive.\u003c/p\u003e","id":"7cdac756-fe8c-4b3c-8ba5-f3cf3a4f9e3b"},{"type":"content_block","value":"\u003ch2 id=\"rules-of-thumb-68zpo\" data-block-key=\"68zpo\"\u003eRules of thumb\u003c/h2\u003e\u003cp data-block-key=\"fa2n1\"\u003eHigh-quality question-and-answer sets need to be structured in a way that aligns with the logic of how the chatbot works. Chatbots use utterances (different wordings for the same question) that have been inputted in order to build machine learning models, which then enables them to establish whether a user’s query has the same meaning/intention as a question in the question-and-answer set. This underlying mechanism gives rise to several rules of thumb for how to develop question-and-answer sets:\u003c/p\u003e\u003ch3 data-block-key=\"cbptr\"\u003eMake sure all utterances within a given intent are equivalent\u003c/h3\u003e\u003cp data-block-key=\"1advi\"\u003eThe chatbot’s machine learning models will assume that, within a given intent, all the utterances are equivalent. In other words, it will assume that you are grouping questions into differently worded versions of the same question. Based on this assumption, the chatbot is then able to build models, which will establish the connections between these utterances.\u003c/p\u003e\u003cp data-block-key=\"3jhp7\"\u003eOne implication of this is that if those developing the question-and-answer sets input non-equivalent utterances (for instance, “where is the library?” and “are there lifts I can take to get to the library?” are not equivalent wordings of the same question), then the chatbot will build its model based on incorrect assumptions. This is likely to diminish the chatbot’s performance.\u003c/p\u003e\u003cp data-block-key=\"1oova\"\u003eIt is suggested that when listing possible ways of wording a question, consider whether you are in fact capturing multiple related questions. If you are, the best approach may be to split up the intent into multiple intents. With the example above, for instance, it may be advantageous to have two separate intents: one for finding out where the library is, and one for finding out whether the library can be accessed via lift.\u003c/p\u003e\u003ch3 data-block-key=\"3gr2k\"\u003eUtterances should be unique to a particular intent\u003c/h3\u003e\u003cp data-block-key=\"4hp8g\"\u003eAs well as all utterances within an intent needing to be equivalent, it is also essential that utterances are unique to one intent.\u003c/p\u003e\u003cp data-block-key=\"59eo4\"\u003eConsider the following two groups of questions:\u003c/p\u003e\u003cul\u003e\u003cli data-block-key=\"48lkk\"\u003eThe first is a grouping of questions about getting support with UCAS applications.\u003c/li\u003e\u003cli data-block-key=\"94a0r\"\u003eThe second is a grouping of questions about getting support with personal statements.\u003c/li\u003e\u003c/ul\u003e\u003cp data-block-key=\"3r301\"\u003eSomeone inputting utterances for these intents, may consider that the utterance “support with university application” may be relevant to either.\u003c/p\u003e\u003cp data-block-key=\"a3tl8\"\u003eIf this utterance were inputted into both intents and a student asked, “I need support with my uni application”, the chatbot would then identify that the intention behind the user question is similar to both the intents on UCAS and personal statements. It would likely struggle to establish which was most relevant – which increases the overall chance of inaccuracy.\u003c/p\u003e\u003cp data-block-key=\"6j3d9\"\u003eOne solution in the above case would be to have a separate intent for generic queries about university support, in addition to having intents for specific queries (ie around UCAS and personal statements – or even more granular).\u003c/p\u003e","id":"f76a0686-2924-4e7b-ba0f-c76499257c8a"},{"type":"content_block","value":"\u003ch2 id=\"refining-question-and-answer-sets-through-pre-launch-testing-68zpo\" data-block-key=\"68zpo\"\u003eRefining question-and-answer sets through pre-launch testing\u003c/h2\u003e\u003cp data-block-key=\"a75q8\"\u003eThe first draft of the question-and-answer set will have made many assumptions about the kinds of questions users will want answered, and the ways in which those questions will be asked. As such, it is unlikely to enable the chatbot to perform optimally.\u003c/p\u003e\u003cp data-block-key=\"6mbi5\"\u003eFortunately, pre-launch testing will provide useful insights that can improve the quality of the question-and-answer set, and thus the performance of the chatbot.\u003c/p\u003e\u003ch3 data-block-key=\"5nhat\"\u003eStep 1: selecting testing subjects\u003c/h3\u003e\u003cp data-block-key=\"6so00\"\u003eOne way to diagnose areas for improvement in the question-and-answer set is to test the chatbot before it is launched, using either students, staff or a mixture of both as testers.\u003c/p\u003e\u003cp data-block-key=\"bhdb7\"\u003eAn advantage of testing the chatbot on students is that the questions they put to the chatbot are more likely to be representative of what students would ask the live chatbot (particularly where students are the intended users).\u003c/p\u003e\u003cp data-block-key=\"ab6bo\"\u003eAn advantage of testing the chatbot on staff is that this can be easier and quicker to organise, which also makes it straightforward to do multiple iterations of testing and improvement.\u003c/p\u003e\u003ch3 data-block-key=\"6t73q\"\u003eStep 2: collecting data from testing\u003c/h3\u003e\u003cp data-block-key=\"68g8h\"\u003eTwo core things to learn through testing are:\u003c/p\u003e\u003cul\u003e\u003cli data-block-key=\"o3nm\"\u003eIs the chatbot able to respond to queries with relevant information? (in other words, is the chatbot accurate?)\u003c/li\u003e\u003cli data-block-key=\"1gf8k\"\u003eAre the chatbot’s responses useful? (Are the answers clear and informative?)\u003c/li\u003e\u003c/ul\u003e\u003cp data-block-key=\"6ph02\"\u003eA formalised way of collecting this information would be to ask testing subjects these questions after each query they raise with the chatbot, and then to collate the overall results.\u003c/p\u003e\u003cp data-block-key=\"btsnf\"\u003eBecause chatbot accuracy/usefulness may vary between topic areas, it may also be helpful to collect data on the topic areas asked for each question, so that you can judge whether this influences the two measures of success.\u003c/p\u003e\u003ch3 data-block-key=\"brnl4\"\u003eStep 3: responding to pre-launch testing, and conducting further testing\u003c/h3\u003e\u003cp data-block-key=\"bflg1\"\u003eThe main purpose of pre-launch testing is to diagnose areas for development in the question-and-answer set. As such, it is important to make changes based on the feedback received.\u003c/p\u003e\u003cp data-block-key=\"2jbvc\"\u003eOnce the first round of changes has been made it is sensible to conduct further testing to evaluate whether any shortcomings have been addressed. This poses the question of when the cycle ends, and at what point it becomes appropriate to launch the chatbot.\u003c/p\u003e\u003cp data-block-key=\"ffh8u\"\u003eOne point to consider here is that the version of the chatbot that is launched will not be the finished product. Notably, there should be updates to the question-and-answer set as more and more data is received from the live queries raised by users, and the way in which the chatbot responds.\u003c/p\u003e\u003cp data-block-key=\"6bjp9\"\u003eAs such, consider the following as steers for when to deploy the chatbot:\u003c/p\u003e\u003cul\u003e\u003cli data-block-key=\"4dj1t\"\u003eThe main risks of deploying the chatbot too late (ie after lengthy and extensive testing) are the opportunity costs of not having use of the chatbot during the testing period and the effort involved in ongoing testing.\u003c/li\u003e\u003cli data-block-key=\"7qc56\"\u003eThe main risk of deploying the chatbot too early (ie after only minimal testing) is that it may perform poorly and put off users. Note, however, that users’ expectations can be managed in order to reduce the risk of this outcome.\u003c/li\u003e\u003c/ul\u003e","id":"e05d4e97-912c-43bf-a072-7b58aba97c97"},{"type":"content_block","value":"\u003ch2 id=\"about-our-work-in-ai-68zpo\" data-block-key=\"68zpo\"\u003eAbout our work in AI\u003c/h2\u003e\u003cp data-block-key=\"i03m\"\u003eOur \u003ca href=\"/innovation/artificial-intelligence/\"\u003ework in AI\u003c/a\u003e accelerates the adoption of effective and ethical AI solutions within colleges and universities. As part of working towards this goal, we have conducted a pilot in which chatbots have been deployed in four UK further education colleges.\u003c/p\u003e\u003cp data-block-key=\"378hn\"\u003eConducting this pilot has enabled us to develop expertise in educational chatbots, which has, in turn, motivated the publication of a series of guidance documents on the subject of chatbots. This guide is the third in that series. The additional guides are:\u003c/p\u003e\u003cul\u003e\u003cli data-block-key=\"drrqh\"\u003e\u003ca href=\"/guides/an-introduction-to-chatbots/\"\u003eAn introduction to chatbots\u003c/a\u003e – the first guide in the series, which is intended for those who want to learn the basics of how chatbots function and how they can be used in education.\u003c/li\u003e\u003cli data-block-key=\"l2i9\"\u003e\u003ca href=\"/guides/preparing-for-chatbots-by-engaging-key-internal-stakeholders/\"\u003ePreparing for chatbots by engaging key internal stakeholders\u003c/a\u003e – the second guide in the series, which is intended for those who have (firmly or tentatively) decided to deploy a chatbot, and want guidance on what to do next.\u003c/li\u003e\u003c/ul\u003e","id":"d134022d-a39c-4df7-90df-4ebeee4abce1"}],"redirect_url":"","secondary_page_type":"guide","summary":"You've started work on introducing chatbots to your organisation, now find out about what's involved in development and implementation.","table_of_contents":"simple","topics":[{"id":7,"name":"Research and innovation"},{"id":18,"name":"Artificial intelligence"}]}},"__N_SSP":true},"page":"/[...path]","query":{"path":["guides","developing-high-quality-question-and-answer-sets-for-chatbots"]},"buildId":"62phW-AXuCLFFYt9M070l","isFallback":false,"isExperimentalCompile":false,"gssp":true,"scriptLoader":[]}</script></body></html>