5 Identifying the evidence: literature searching and evidence submission | Developing NICE guidelines: the manual | Guidance

<!DOCTYPE html> <html lang="en-GB" class="no-js"> <head> <meta charset="utf-8" /> <meta content="IE=edge,chrome=1" name="X-UA-Compatible" /> <meta content="width=device-width, initial-scale=1" name="viewport" /> <title>5 Identifying the evidence: literature searching and evidence submission | Developing NICE guidelines: the manual | Guidance | NICE</title> <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin /> <link rel="preconnect" href="https://fonts.googleapis.com" crossorigin /> <link rel="preconnect" href="https://code.jquery.com" crossorigin /> <link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;600&family=Lora:ital,wght@0,600;1,600&display=swap" rel="stylesheet" /> <link href="https://cdn.nice.org.uk/V2/NICE.bootstrap.updated.brand.css" rel="stylesheet" type="text/css" /> <link href="/Themes/NICE.Bootstrap/styles/nice/NICE.base.css" rel="stylesheet" type="text/css" /> <link href="//cdn.nice.org.uk/V2/Content/fontawesome.css" rel="stylesheet" type="text/css" /> <link href="//cdn.nice.org.uk/V2/Content/NICE.glyphs.css" rel="stylesheet" type="text/css" /> <link href="/css/guidance.web.css?v=MfI3fZ7eulA9Kx8MLWnew5A4TpPacjutlQulEuCgaBs" rel="stylesheet" type="text/css" /> <meta content="This manual explains the processes and methods used to develop and update NICE guidelines, the guidance that NICE develops covering topics across clinical care (in primary, secondary and community care settings), social care and public health. For more information on the other types of NICE guidance and advice (including technology appraisal guidance), see about NICE" name="Description" /> <meta content="NICE" name="DC.Publisher" /> <meta content="All content on this site is NICE copyright unless otherwise stated. You can download material for private research, study or in-house use only. Do not distribute or publish any material from this site without first obtaining NICE's permission. Where Crown copyright applies, see the Office of Public Sector Information (formerly HMSO) website for information." name="DC.Rights.Copyright" /> <meta content="eng" name="DC.Language" scheme="DCTERMS.ISO639-2T" /> <meta content="Health, well-being and care" name="DC.Subject" scheme="eGMS.IPSV" /> <meta content="Double-A" name="eGMS.accessibility" scheme="eGMS.WCAG10" /> <meta content="5 Identifying the evidence: literature searching and evidence submission | Developing NICE guidelines: the manual | Guidance | NICE" name="DC.Title" /> <meta content="#f6f6f6" name="msapplication-TileColor" /> <meta content="//cdn.nice.org.uk/V3/Content/nice/favicon-144.png" name="msapplication-TileImage" /> <meta content="#333" name="theme-color" /> <link rel="shortcut icon" href="//cdn.nice.org.uk/V3/Content/nice/favicon.ico" /> <link rel="apple-touch-icon-precomposed" href="//cdn.nice.org.uk/V3/Content/nice/favicon-152.png"> <link rel="apple-touch-icon" href="//cdn.nice.org.uk/V3/Content/nice/favicon-152.png"> <link rel="apple-touch-icon-precomposed" sizes="120x120" href="//cdn.nice.org.uk/V3/Content/nice/favicon-120.png"> <link rel="apple-touch-icon" sizes="120x120" href="//cdn.nice.org.uk/V3/Content/nice/favicon-120.png"> <link rel="apple-touch-icon-precomposed" sizes="152x152" href="//cdn.nice.org.uk/V3/Content/nice/favicon-152.png"> <link rel="apple-touch-icon" sizes="152x152" href="//cdn.nice.org.uk/V3/Content/nice/favicon-152.png"> <script src="//cdn.nice.org.uk/cookie-banner/cookie-banner.min.js" type="text/javascript"></script> <script src="https://code.jquery.com/jquery-3.7.1.min.js" integrity="sha384-1H217gwSVyLSIfaLxHbE7dRb3v4mYCKbpQvzx0cegeju1MVsGrX5xXxAvs/HgeFs" crossorigin="anonymous"></script> <script src="https://code.jquery.com/jquery-migrate-3.4.1.min.js" integrity="sha384-ez4i72z0XhJqm0ZRoQkpPN0bDV8Kw8PK3n8kjEZsWhRDrfICLd0AGghHJB+vytx0" crossorigin="anonymous"></script> <script type="text/javascript" src="/js/bundle.min.js?v=3w6GfX1dYYlm8GobS-ucsLbXKPIAW90zg06zFfevsHw"></script> <meta name="theme-color" content="#333"><link href="/Media/Default/css/default.css" rel="stylesheet" type="text/css" /><script src="/Media/Default/js/default.js" type="text/javascript"></script> </head> <body data-spy="scroll" data-target=".aside-primary" class=""> <noscript> <iframe src="//www.googletagmanager.com/ns.html?id=GTM-M55QTQ" height="0" width="0" style="display:none;visibility:hidden"></iframe> </noscript> <script> (function (w, d, s, l, i) { w[l] = w[l] || []; w[l].push({ 'gtm.start': new Date().getTime(), event: 'gtm.js' }); var f = d.getElementsByTagName(s)[0], j = d.createElement(s), dl = l != 'dataLayer' ? '&l=' + l : ''; j.async = true; j.src = '//www.googletagmanager.com/gtm.js?id=' + i + dl; f.parentNode.insertBefore(j, f); })(window, document, 'script', 'dataLayer', 'GTM-M55QTQ'); </script> <div id="global-nav-header"></div> <div id="printpane"></div> <main> <div class="_ page-header" aria-label="Page header"> <div class="zone zone-breadcrumb"> <div class="container" id="global-breadcrumb"> <ol class="breadcrumb"> <li> <a href="/" data-track="guidancebreadcrumb">Home</a> <span class="divider icon-chevron-right"></span> </li> </ol> <h1 class="_ h1-alt" id="content-start">Developing NICE guidelines: the manual </h1> <div class="product-heading "> <div class="product-heading"> <p class="product-title lead inline"> <span class="prod-title">NICE process and methods [PMG20]</span><wbr> <span class="published-date"> Published: <time datetime="2014-10-31">31 October 2014</time> </span> <wbr /> <span class="published-date"> Last updated: <time datetime="2024-05-29">29 May 2024</time> </span> </p> </div> </div> </div> </div> </div> <div class="_ layout " aria-label="Main page content"> <div id="toTop"><span aria-hidden="true" class="icon-nice-chevron-up"></span></div> <div class="_"> <div class="content"> <div class="zone zone-content"> <div id="guidance-tabs-container" class="tabbable"> <ul class="nav nav-tabs" id="guidanceTabs-product"> <li class="active"> <a href="/process/pmg20">Process</a> </li> <li class=""> <a href="/process/pmg20/resources">Tools and resources</a> </li> </ul> <div class="tab-content"> <div class="tab-pane active gtm-guideline-page-tab" id="NICE-process-and-methods-Tab"> <div class="product product-guideline row"> <div class="sections span3 toresponsive"> <div id="overview-menu"> <div class="menu-container"> <ul class="nav nav-list" id="Overview-Menu"> <li class="sync "> <a href="/process/pmg20">Overview</a> </li> </ul> </div> </div> <div id='guidance-menu'> <div class='menu-container'> <ul class="nav nav-list" id="Guidance-Menu"> <li> <a href="/process/pmg20/chapter/introduction">1 Introduction</a> </li> <li> <a href="/process/pmg20/chapter/the-scope">2 The scope</a> </li> <li> <a href="/process/pmg20/chapter/decision-making-committees">3 Decision-making committees</a> </li> <li> <a href="/process/pmg20/chapter/developing-review-questions-and-planning-the-evidence-review">4 Developing review questions and planning the evidence review</a> </li> <li class="active"> <a href="/process/pmg20/chapter/identifying-the-evidence-literature-searching-and-evidence-submission">5 Identifying the evidence: literature searching and evidence submission</a> </li> <li> <a href="/process/pmg20/chapter/reviewing-evidence">6 Reviewing evidence</a> </li> <li> <a href="/process/pmg20/chapter/incorporating-economic-evaluation">7 Incorporating economic evaluation</a> </li> <li> <a href="/process/pmg20/chapter/linking-to-other-guidance">8 Linking to other guidance</a> </li> <li> <a href="/process/pmg20/chapter/interpreting-the-evidence-and-writing-the-guideline">9 Interpreting the evidence and writing the guideline</a> </li> <li> <a href="/process/pmg20/chapter/the-validation-process-for-draft-guidelines-and-dealing-with-stakeholder-comments">10 The validation process for draft guidelines, and dealing with stakeholder comments</a> </li> <li> <a href="/process/pmg20/chapter/finalising-and-publishing-the-guideline-recommendations">11 Finalising and publishing the guideline recommendations</a> </li> <li> <a href="/process/pmg20/chapter/support-for-putting-the-guideline-recommendations-into-practice">12 Support for putting the guideline recommendations into practice</a> </li> <li> <a href="/process/pmg20/chapter/ensuring-that-published-guidelines-are-current-and-accurate">13 Ensuring that published guidelines are current and accurate</a> </li> <li> <a href="/process/pmg20/chapter/updating-guideline-recommendations">14 Updating guideline recommendations</a> </li> <li> <a href="/process/pmg20/chapter/appendices">15 Appendices</a> </li> <li> <a href="/process/pmg20/chapter/glossary">Glossary</a> </li> <li> <a href="/process/pmg20/chapter/update-information">Update information</a> </li> </ul> </div> </div> </div> <div class="span9 tab-header"> <div class="share-links navbar pull-right"> <div class="navbar-inner"> <div class="unstyled"> <div class="dropdown"> <a href="/guidance/pmg20/resources/developing-nice-guidelines-the-manual-pdf-72286708700869" id="nice-download" target="_blank" class="track-event btn-sharelink dropdown-toggle" title="Download (PDF)" data-track="guidancedownload" data-track-category="guidance download" data-track-action="clicked" data-track-label="/guidance/pmg20/resources/developing-nice-guidelines-the-manual-pdf-72286708700869"> <span>Download (PDF)</span> </a> </div> </div> </div> </div> <h2 id="tab-title" class="h3">NICE process and methods</h2> </div> <div class="content span9"> <div class="product-page summary"> <div class="navbar navbar-resources"> </div> </div> <div class="section-heading"> <ul class="pagination "> <li class="sectionmenu hidden-desktop hidden-tablet span8"></li> <li class="previous"><a href="/process/pmg20/chapter/developing-review-questions-and-planning-the-evidence-review" aria-label="Previous chapter"><span class="icon-nice-chevron-left"></span></a></li> <li class="next"><a href="/process/pmg20/chapter/reviewing-evidence" aria-label="Next chapter"><span class="hidden-phone">Next </span><span class="icon-nice-chevron-right"></span></a></li> </ul> <h3 class="h2">5 Identifying the evidence: literature searching and evidence submission</h3> <ul class="unstyled sectiontitles"> <li><a href="#introduction-3">5.1 Introduction</a></li> <li><a href="#searches-during-guideline-recommendation-scoping-and-surveillance">5.2 Searches during guideline recommendation scoping and surveillance</a></li> <li><a href="#searches-during-guideline-recommendation-development">5.3 Searches during guideline recommendation development</a></li> <li><a href="#health-inequalities-and-equality-and-diversity">5.4 Health inequalities and equality and diversity</a></li> <li><a href="#quality-assurance">5.5 Quality assurance</a></li> <li><a href="#documenting-the-search">5.6 Documenting the search</a></li> <li><a href="#re-running-searches">5.7 Re-running searches</a></li> <li><a href="#calls-for-evidence-from-stakeholders">5.8 Calls for evidence from stakeholders</a></li> <li><a href="#references-and-further-reading-4">5.9 References and further reading</a></li> </ul> </div> <div class="section-summary web-viewer-content"> <div class="chapter" title="5 Identifying the evidence: literature searching and evidence submission" id="pmg20-identifying-the-evidence-literature-searching-and-evidence-submission" xmlns="http://www.w3.org/1999/xhtml"> <h2 class="title" id="identifying-the-evidence-literature-searching-and-evidence-submission">5 Identifying the evidence: literature searching and evidence submission</h2> <div class="section" title="5.1 Introduction" id="pmg20-introduction-3"> <h3 class="title" id="introduction-3">5.1 Introduction</h3> <p>The systematic identification of <a class="link" href="https://www.nice.org.uk/Glossary?letter=E#Evidence" target="_top">evidence</a> is an essential step in developing NICE guideline recommendations. </p> <p>This chapter sets out how evidence is identified at each stage of the guideline development cycle. It provides details of the systematic literature searching methods used to identify the best available evidence for NICE guidelines. It also provides details of associated information management processes including <a class="link" href="https://www.nice.org.uk/process/pmg20/chapter/introduction#quality-assurance" target="_top">quality assurance</a> (peer review), re‑running searches, and documenting the search process. </p> <p>Our searching methods are informed by the <a class="link" href="https://training.cochrane.org/handbook/current" target="_top">chapter on searching & selecting studies in the Cochrane Handbook for Systematic Reviews of Interventions</a> and the <a class="link" href="https://onlinelibrary.wiley.com/doi/10.4073/cmg.2016.1?msockid=1cad8cc0e8626012171d981de982614d" target="_top">Campbell Collaboration's searching for studies guide</a>. The <a class="link" href="https://sites.google.com/york.ac.uk/sureinfo/home" target="_top">Summarized Research in Information Retrieval for HTA (SuRe Info)</a> resource also provides research-based advice on information retrieval for systematic reviews. </p> <p>Our literature searches are designed to be systematic, transparent, and reproducible, and minimise dissemination bias. Dissemination bias may affect the results of reviews and includes publication bias and database bias. </p> <p>We use search methods that balance recall and precision. When the need to reduce the number of records for sifting requires pragmatic search approaches that may increase the risk of missing relevant studies, the context and trade-offs are discussed and agreed within the development team and made explicit in the reported search methods. </p> <p>A flexible approach to identifying evidence is adopted, guided by the subject of the review question (see the <a class="link" href="https://www.nice.org.uk/process/pmg20/chapter/developing-review-questions-and-planning-the-evidence-review" target="_top">chapter on developing review questions and planning the evidence review</a>), type of evidence sought, and the resource constraints of the evidence review. Often an evidence review will be an update of our earlier work, therefore the approach can be informed by previous searches and surveillance reviews (see the <a class="link" href="https://www.nice.org.uk/process/pmg20/chapter/ensuring-that-published-guidelines-are-current-and-accurate" target="_top">chapter on ensuring that published guidelines are current and accurate</a>). </p> </div> <div class="section numbered-paragraph recommendation" title="5.2 Searches during guideline recommendation scoping and surveillance" id="pmg20-searches-during-guideline-recommendation-scoping-and-surveillance"> <h3 class="title" id="searches-during-guideline-recommendation-scoping-and-surveillance">5.2 Searches during guideline recommendation scoping and surveillance</h3> <div class="section" title="Scoping searches" id="pmg20-scoping-searches"> <h4 class="title" id="scoping-searches">Scoping searches</h4> <p>Scoping searches are top-level searches to support scope development. The purpose of the searches is to investigate the current evidence around the topic, and to identify any areas where an evidence review may be beneficial and any research gaps. The results of the searches are used to draft the scope of the upcoming guideline or update and to inform the discussions at scoping workshops (if held). Scoping searches do not aim to be exhaustive. </p> <p>In some cases, scoping searches are not required when it is more efficient to use the surveillance review (see the <a class="link" href="https://www.nice.org.uk/process/pmg20/chapter/the-scope" target="_top">chapter on the scope</a>). </p> <p>The sources searched at scoping stage will vary according to the topic, type of review questions the guideline or update will seek to address, and type of evidence sought. Each scoping search is tailored using combinations of the following types of information: </p> <div class="itemizedlist"> <ul class="itemizedlist"> <li class="listitem"> <p>NICE guidance and guidance from other organisations</p> </li> <li class="listitem"> <p>policy and legislation guides</p> </li> <li class="listitem"> <p>key systematic reviews and epidemiological reviews</p> </li> <li class="listitem"> <p>economic evaluations</p> </li> <li class="listitem"> <p>current practice data, including costs and resource use and any safety concerns</p> </li> <li class="listitem"> <p>views and experiences of people using services, their family members or carers, or the public</p> </li> <li class="listitem"> <p>other real-world health and social care data (for example audits, surveys, registries, electronic health records, patient-generated health data), if appropriate</p> </li> <li class="listitem"> <p>summaries of interventions that may be appropriate, including any national safety advice</p> </li> <li class="listitem"> <p>statistics (for example on epidemiology, natural history of the condition, service configuration or national prevalence data).</p> </li> </ul> </div> <p>All scoping searches are fully documented and if new issues are identified at a scoping workshop, the search is updated. A range of possible sources considered for scoping searches is provided in the <a class="link" href="https://www.nice.org.uk/process/pmg20/resources/developing-nice-guidelines-the-manual-appendices-2549710189/chapter/appendix-f-suggested-sources-for-scoping" target="_top">appendix on suggested sources for scoping</a>.</p> </div> <div class="section" title="Health inequalities searches" id="pmg20-health-inequalities-searches"> <h4 class="title" id="health-inequalities-searches">Health inequalities searches</h4> <p>The purpose of these searches is to identify evidence to help inform the scope, health inequalities briefing, or the equality and health inequalities assessment (EHIA). They help identify key issues relevant to health inequalities on the topic, for example covering protected characteristics, groups experiencing or at risk of inequalities, or wider determinants of health.</p> <p>The searches involve finding key data sources, such as routinely available national databases, audits or published reports by charities, non-governmental bodies, or government organisations. </p> </div> <div class="section" title="Surveillance searches" id="pmg20-surveillance-searches"> <h4 class="title" id="surveillance-searches">Surveillance searches</h4> <p>Surveillance determines whether published recommendations remain current. The searches are tailored to the evidence required. This may include searches for new or updated policies, legislation, guidance from other organisations, or ongoing studies in the area covered by the evidence review.</p> <p>If required, published evidence is identified by searching a range of bibliographic databases relevant to the topic. Surveillance searches generally use the same core set of databases used during the development of the original evidence review. A list of sources is given in the <a class="link" href="https://www.nice.org.uk/process/pmg20/resources/developing-nice-guidelines-the-manual-appendices-2549710189/chapter/appendix-g-sources-for-evidence-reviews" target="_top">appendix on sources for evidence reviews</a>.</p> <p>The search approach and sources will vary between topics and may include:</p> <div class="itemizedlist"> <ul class="itemizedlist"> <li class="listitem"> <p>population and intervention searches</p> </li> <li class="listitem"> <p>focused searches for specific question areas</p> </li> <li class="listitem"> <p>forward and backward citation searching.</p> </li> </ul> </div> <p>Searches usually focus on randomised controlled trials and systematic reviews, although other study types will be considered where appropriate, for example for diagnostic questions. </p> <p>The search period starts at either the end of the search for the last update of a guideline evidence review, or at the last search date for any previous surveillance check. Where appropriate, living evidence surveillance could be set up to continuously monitor the publication of new evidence over a period of time until impact reaches the threshold for actions. For more information on NICE guideline recommendation surveillance, see the <a class="link" href="https://www.nice.org.uk/process/pmg20/chapter/ensuring-that-published-guidelines-are-current-and-accurate" target="_top">chapter on ensuring that guideline recommendations are current and accurate</a> and <a class="link" href="https://www.nice.org.uk/process/pmg20/resources/developing-nice-guidelines-the-manual-appendices-a-to-p-2549710189/chapter/appendix-o-surveillance-interim-principles-for-monitoring-approaches-of-guideline" target="_top">appendix on surveillance - interim principles for monitoring approaches of guideline recommendations</a>.</p> </div> </div> <div class="section numbered-paragraph recommendation" title="5.3 Searches during guideline recommendation development" id="pmg20-searches-during-guideline-recommendation-development"> <h3 class="title" id="searches-during-guideline-recommendation-development">5.3 Searches during guideline recommendation development</h3> <div class="section" title="Search protocols" id="pmg20-search-protocols"> <h4 class="title" id="search-protocols">Search protocols</h4> <p>Search protocols form part of the wider guideline review protocol (see the <a class="link" href="https://www.nice.org.uk/process/pmg20/resources" target="_top">appendix on the review protocol template</a>). They pre‑define how the evidence is identified and provide a basis for developing the search strategies.</p> <p>Once the final scope is agreed, the information specialist develops the search protocols and agrees them with the development team before the evidence search begins.</p> <p>A search protocol includes the following elements:</p> <div class="itemizedlist"> <ul class="itemizedlist"> <li class="listitem"> <p>approach to the search strategy, tailored to the review question and eligibility criteria</p> </li> <li class="listitem"> <p>sources to be searched </p> </li> <li class="listitem"> <p>plans to use any <a class="link" href="identifying-the-evidence-literature-searching-and-evidence-submission#additional-search-techniques" target="_top" rel="internal" data-chapter="identifying-the-evidence-literature-searching-and-evidence-submission" data-original-url="#additional-search-techniques">additional or alternative search techniques</a>, when known at the protocol development stage, and the reasons for their use</p> </li> <li class="listitem"> <p>details of any <a class="link" href="identifying-the-evidence-literature-searching-and-evidence-submission#search-limits" target="_top" rel="internal" data-chapter="identifying-the-evidence-literature-searching-and-evidence-submission" data-original-url="#search-limits">limits to be applied to the search</a></p> </li> <li class="listitem"> <p>references to any key papers used to inform the search approach.</p> </li> </ul> </div> </div> <div class="section" title="Sources" id="pmg20-sources"> <h4 class="title" id="sources">Sources</h4> <p>Searches are done on a mix of bibliographic databases, websites and other sources, depending on the subject of the review question and the type of evidence sought. </p> <p>For most searches there are key sources that are prioritised, and other potentially relevant sources that can be considered. It is important to ensure adequate coverage of the relevant literature and to search a range of sources. However, there are practical limits to the number of sources that can be searched in the standard time available for an evidence review.</p> <p>The selection of sources varies according to the requirements of the review question. </p> </div> <div class="section" title="Clinical intervention sources" id="pmg20-clinical-intervention-sources"> <h4 class="title" id="clinical-intervention-sources">Clinical intervention sources</h4> <p>For reviews of the effectiveness of clinical interventions the following sources are prioritised for searching:</p> <div class="itemizedlist"> <ul class="itemizedlist"> <li class="listitem"> <p>the Cochrane Central Register of Controlled Trials (CENTRAL)</p> </li> <li class="listitem"> <p>the Cochrane Database of Systematic Reviews (CDSR)</p> </li> <li class="listitem"> <p>Embase</p> </li> <li class="listitem"> <p>MEDLINE.</p> </li> </ul> </div> </div> <div class="section" title="Clinical safety sources" id="pmg20-clinical-safety-sources"> <h4 class="title" id="clinical-safety-sources">Clinical safety sources</h4> <p>In addition to the sources searched for clinical interventions, the following should be prioritised for clinical safety review questions:</p> <div class="itemizedlist"> <ul class="itemizedlist"> <li class="listitem"> <p> <a class="link" href="https://www.gov.uk/drug-safety-update" target="_top">MHRA drug safety updates</a> </p> </li> <li class="listitem"> <p> <a class="link" href="https://www.cas.mhra.gov.uk/Home.aspx" target="_top">National patient safety alerts</a>.</p> </li> </ul> </div> </div> <div class="section" title="Antimicrobial resistance sources" id="pmg20-antimicrobial-resistance-sources"> <h4 class="title" id="antimicrobial-resistance-sources">Antimicrobial resistance sources</h4> <p>For reviews of antimicrobial resistance, the following sources should be prioritised:</p> <div class="itemizedlist"> <ul class="itemizedlist"> <li class="listitem"> <p> <a class="link" href="https://www.gov.uk/government/publications/english-surveillance-programme-antimicrobial-utilisation-and-resistance-espaur-report" target="_top">UK Health Security Agency's English surveillance programme for antimicrobial utilisation and resistance (ESPAUR) report</a> </p> </li> <li class="listitem"> <p> <a class="link" href="https://fingertips.phe.org.uk/profile/amr-local-indicators#:~:text=AMR%20local%20indicators%20are%20publically,development%20of%20local%20action%20plans." target="_top">UK Health Security Agency's antimicrobial resistance local indicators</a>.</p> </li> </ul> </div> </div> <div class="section" title="Cost-effectiveness sources" id="pmg20-cost-effectiveness-sources"> <h4 class="title" id="cost-effectiveness-sources">Cost-effectiveness sources</h4> <p>For reviews of cost effectiveness, economic databases are used in combination with general bibliographic databases, such as MEDLINE and Embase (see <a class="link" href="https://www.nice.org.uk/process/pmg20/resources/developing-nice-guidelines-the-manual-appendices-2549710189/chapter/appendix-g-sources-for-evidence-reviews" target="_top">appendix G on sources for economic reviews</a>). </p> <p>Economic evaluations of social care interventions may be published in journals that are not identified through standard searches. Targeted searches based on references of key articles and contacting authors can be considered to identify relevant papers.</p> </div> <div class="section" title="Topic-specific sources" id="pmg20-topic-specific-sources"> <h4 class="title" id="topic-specific-sources">Topic-specific sources</h4> <p>Some topics we cover may require the use of topic-specific sources. Examples include:</p> <div class="itemizedlist"> <ul class="itemizedlist"> <li class="listitem"> <p>PsycINFO (psychology and psychiatry)</p> </li> <li class="listitem"> <p>CINAHL (nursing and allied health professions) </p> </li> <li class="listitem"> <p>ASSIA (Applied Social Sciences Index and Abstracts)</p> </li> <li class="listitem"> <p> <a class="link" href="https://healthtalk.org/" target="_top">HealthTalk</a>, and other sources to identify the views and experiences of people using services, carers and the public</p> </li> <li class="listitem"> <p>social policy and practice</p> </li> <li class="listitem"> <p>sociological abstracts</p> </li> <li class="listitem"> <p>transport database</p> </li> <li class="listitem"> <p>Greenfile (environmental literature)</p> </li> <li class="listitem"> <p>HMIC (Health Management Information Consortium).</p> </li> </ul> </div> </div> <div class="section" title="Searching for model inputs" id="pmg20-searching-for-model-inputs"> <h4 class="title" id="searching-for-model-inputs">Searching for model inputs</h4> <p>Evidence searches may be needed to inform design-oriented conceptual models. Examples include precise searches to find representative NHS costs for an intervention or finding out the proportion of people offered an intervention who take up the offer. </p> <p>Some model inputs, such as costs, use national sources such as national list prices or national audit data. In some cases, it may be more appropriate to identify costs from the academic literature. Further advice on methods to identify model inputs are also informed by Paisley (2016) and Kaltenhaler et al. (2011). See also the <a class="link" href="https://www.nice.org.uk/process/pmg20/chapter/incorporating-economic-evaluation" target="_top">chapter on incorporating economic evaluation</a>. </p> </div> <div class="section" title="Real-world data" id="pmg20-real-world-data"> <h4 class="title" id="real-world-data">Real-world data</h4> <p>Information specialists can identify sources of real-world data (such as electronic health records, registries, and audits) for data analysts to explore further. The <a class="link" href="https://www.hdruk.ac.uk/access-to-health-data/health-data-research-innovation-gateway/" target="_top">Health Data Research Innovation Gateway</a> can be used to identify datasets. The <a class="link" href="https://www.nice.org.uk/corporate/ecd9/chapter/overview" target="_top">NICE real-world evidence framework (2022)</a> has additional guidance on searching for and selecting real-world data sources.</p> </div> <div class="section" title="Grey literature" id="pmg20-grey-literature"> <h4 class="title" id="grey-literature">Grey literature</h4> <p>For some review questions, for example, where significant evidence is likely to be published in non-journal sources and there is a paucity of evidence in published journal sources, it may be appropriate to search for <a class="link" href="https://www.nice.org.uk/Glossary/Grey-literature" target="_top">grey literature</a>. Useful sources of grey literature include:</p> <div class="itemizedlist"> <ul class="itemizedlist"> <li class="listitem"> <p>HMIC (Health Management Information Consortium)</p> </li> <li class="listitem"> <p>GOV.uk</p> </li> <li class="listitem"> <p>TRIP database</p> </li> <li class="listitem"> <p>social policy and practice</p> </li> <li class="listitem"> <p> <a class="link" href="https://greymatters.cadth.ca/about#:~:text=About%20Grey%20Matters%20Background%20CADTH%20%E2%80%94%20the%20Canadian,and%20medical%20devices%20in%20our%20health%20care%20system." target="_top">Canadian Agency for Drugs and Technology in Health (CADTH) Grey Matters resource</a>.</p> </li> </ul> </div> <p>Committee members may also be able to suggest additional appropriate sources for grey literature.</p> <p>A list containing potential relevant sources is provided in the <a class="link" href="https://www.nice.org.uk/process/pmg20/resources/developing-nice-guidelines-the-manual-appendices-2549710189/chapter/appendix-g-sources-for-evidence-reviews" target="_top">appendix on sources for evidence reviews</a>.</p> </div> <div class="section" title="Developing search strategies" id="pmg20-developing-search-strategies"> <h4 class="title" id="developing-search-strategies">Developing search strategies</h4> <p>The approach to devising and structuring search strategies is informed by the review protocol. The PICO (population, intervention, comparator and outcome) or SPICE (setting, perspective, intervention, comparison, evaluation) frameworks may be used to structure a search strategy for intervention review questions. For other types of review questions, alternative frameworks may be more suitable. </p> <p>It is sometimes more efficient to conduct a single search for multiple review questions, rather than conducting a separate search for each question.</p> <p>Some topics may not easily lend themselves to PICO- or SPICE-type frameworks. In these cases, it may be better to combine multiple, shorter searches rather than attempting to capture the entire topic using a single search. This is often referred to as multi-stranded searching.</p> <p>In some instances, for example where the terminology around a topic is diffuse or ill defined, it may be difficult to specify the most appropriate search terms in advance. In these cases, an iterative approach to searching can be used. </p> <p>In an iterative approach, searching is done in several stages, with each search considering the evidence that has already been retrieved (for example, see <a class="link" href="https://onlinelibrary.wiley.com/doi/abs/10.1002/jrsm.1386" target="_top">Booth et al. 2020</a>). Searching in stages allows the reviewers to review the most relevant, high-quality information first and then make decisions for identifying additional evidence if needed. </p> <p>Decisions to use iterative approaches are agreed by the development team and staff with responsibility for quality assurance because it can affect timelines. </p> </div> <div class="section" title="Updating previous work" id="pmg20-updating-previous-work"> <h4 class="title" id="updating-previous-work">Updating previous work</h4> <p>Where high-quality review-level evidence is available on a topic, the review team may choose to update or expand this previous work rather than duplicating the existing findings. In these cases, the original review searches are re-run and expanded to account for any differences in scope and inclusion criteria between the original review and the update. </p> </div> <div class="section" title="Cost-effectiveness searches" id="pmg20-cost-effectiveness-searches"> <h4 class="title" id="cost-effectiveness-searches">Cost-effectiveness searches</h4> <p>There are several methods that can be used to identify economic evaluations:</p> <div class="itemizedlist"> <ul class="itemizedlist"> <li class="listitem"> <p>All relevant review questions can be covered by a single search using the population search terms, combined with a search filter, to identify economic evidence.</p> </li> <li class="listitem"> <p>The search strategies for individual review questions can be combined with search filters to identify economic evidence. If using this approach, it may be necessary to adapt strategies for some databases to ensure adequate sensitivity.</p> </li> <li class="listitem"> <p>Economic evidence can be manually sifted while screening evidence from a general literature search (so no separate searches are required).</p> </li> </ul> </div> <p>The rationale for the selected approach is recorded in the search protocol.</p> <p>Where searches are needed to populate an economic model, these are usually done separately.</p> </div> <div class="section" title="Identifying search terms" id="pmg20-identifying-search-terms"> <h4 class="title" id="identifying-search-terms">Identifying search terms</h4> <p>Search terms usually consist of a combination of subject headings and free‑text terms from the titles and abstracts of relevant references. </p> <p>When identifying subject headings, variations in thesaurus and indexing terms for each database should be considered, for example MeSH (Medical Subject Headings) in MEDLINE and Emtree in Embase. Not all databases have indexing terms and some contain records that have not yet been indexed. </p> <p>Free‑text terms may include synonyms, acronyms and abbreviations, spelling variants, old and new terminology, brand and generic medicine names, and lay and medical terminology. </p> <p>For updates, previous search terms, including those from surveillance searches, are reviewed and used to inform new search terms. New or changed terms are identified, as well as any changes to indexing terms. This also applies when an existing review, for example a Cochrane review, is being updated to answer a review question. </p> <p>Key studies can be a useful source of search terms, as can reports, guidelines, topic-specific websites, committee members and topic experts.</p> <p>Some websites and databases have limited search functionality. It may be necessary to use fewer search terms or do multiple searches of the same resource with different search term combinations.</p> <p>It may be helpful to use frequency analysis or text mining to develop the search-term strategy. Tools such as <a class="link" href="http://hgserver2.amc.nl/cgi-bin/miner/miner2.cgi" target="_top">PubReMiner</a> and <a class="link" href="http://cbdm-01.zdv.uni-mainz.de/~jfontain/" target="_top">Medline Ranker</a> can help, either by highlighting search terms that might not otherwise be apparent, or by flagging terms of high value when exhaustive synonym searching is unfeasible or inadvisable. </p> </div> <div class="section" title="Search limits" id="pmg20-search-limits"> <h4 class="title" id="search-limits">Search limits</h4> <p>The application of limits to search strategies will reflect the eligibility criteria in the review protocol. Typically, English language limits, date limits, and the exclusion of conference abstracts and animal studies are usually done as a matter of routine. </p> </div> <div class="section" title="Search filters" id="pmg20-search-filters"> <h4 class="title" id="search-filters">Search filters</h4> <p>A <a class="link" href="https://www.nice.org.uk/Glossary/search-filter" target="_top">search filter</a> is a string of search terms with known (validated) performance. When a particular study design is required for a review question, relevant search filters are usually applied to literature search strategies. </p> <p>Other search filters relating to age, setting, geography, and health inequalities are also applied as relevant. The most comprehensive list of available search filters is the <a class="link" href="https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/home" target="_top">search filter resource of the InterTASC Information Specialists' SubGroup</a>. This resource also includes critical appraisal tools, which are used for filter selection. </p> </div> <div class="section" title="Economics-related filters" id="pmg20-economics-related-filters"> <h4 class="title" id="economics-related-filters">Economics-related filters</h4> <p>A variety of search filters of relevance to cost effectiveness are available. These include filters for economic evaluations, quality of life data, and cost-utilities data. It may be necessary to use more than 1 filter to identify relevant data. In addition, it may be appropriate to add geographic search filters, such as those for the UK or Organisation for Economic Co-operation and Development (OECD) countries, to retrieve economic studies relevant to the UK or OECD (Ayiku et al. 2017, 2019, 2021). </p> </div> <div class="section" title="Use of machine learning-based classifiers" id="pmg20-use-of-machine-learning-based-classifiers"> <h4 class="title" id="use-of-machine-learning-based-classifiers">Use of machine learning-based classifiers</h4> <p>Machine learning-based classification software has been developed for some study types (for example the Cochrane RCT classifier, <a class="link" href="https://doi.org/10.1016/j.jclinepi.2020.11.003" target="_top">Thomas et al. 2020</a>). These classifiers apply a probability weighting to each bibliographical reference within a set of search results. The weighting relates to the reference's likelihood to be a particular study type, based on a model created from analysis of known, relevant papers. The weightings can then be used to either order references for screening or be used with a fixed cut-off value to divide a list of references into those more likely to be included, and those that can be excluded without manual screening. </p> <p>We support the use of machine classifiers if their performance characteristics are known, and if they improve efficiency in the search and screening process. However, caution is needed when using classifiers, because they may not be as effective if used on data that is different to the type of data for which they were originally developed. For example, the Cochrane RCT classifier is reported to have over 99% recall for health studies but showed "unacceptably low" recall for educational research (<a class="link" href="https://doi.org/10.1002/jrsm.1537" target="_top">Stansfield et al. 2022</a>). </p> <p>Priority screening, a type of machine classifier that orders references for manual sifting based on previous sifting decisions, is considered in the <a class="link" href="https://www.nice.org.uk/process/pmg20/chapter/reviewing-evidence" target="_top">chapter on reviewing evidence</a>.</p> </div> <div class="section" title="Additional search techniques" id="pmg20-additional-search-techniques"> <h4 class="title" id="additional-search-techniques">Additional search techniques</h4> <p>Additional search techniques are used alongside database searching when it is known, or reasonably likely, that relevant evidence is not indexed in bibliographic databases, or when it will be difficult to retrieve relevant evidence from databases in a way that adequately balances recall and precision. Additional search techniques include forward and backward citation searching, journal hand-searches and contacting experts and stakeholders.</p> <p>Existing reviews may provide an additional source of primary studies, with reference lists being used as an indirect method of identifying primary research. </p> <p>Various tools, including <a class="link" href="https://doi.org/10.1002/jrsm.1563" target="_top">Citationchaser</a> and Web of Science, are available to speed up the process of citation searching. These may not be as comprehensive as manual reference list checking (due to limitations of the underlying data sources), but the trade-off in terms of speed is generally acceptable.</p> <p>All search techniques should follow the same principles of transparency, rigour and reproducibility as other search methods. </p> <p>If possible, additional search techniques should be considered at the outset and documented in the search protocol. They should also be documented in the supporting appendices for the final evidence review.</p> </div> </div> <div class="section" title="5.4 Health inequalities and equality and diversity" id="pmg20-health-inequalities-and-equality-and-diversity"> <h3 class="title" id="health-inequalities-and-equality-and-diversity">5.4 Health inequalities and equality and diversity</h3> <p>All searches aim to be inclusive. This may mean not specifying any population groups. </p> <p>Searches should avoid inadvertently excluding relevant groups. For example, if the population group is older people, a search for older people should pick up subpopulations such as disabled older people. </p> <p>Additional search strategies may be needed to target evidence about people with protected characteristics or people experiencing or at risk from other inequalities. </p> <p>Searches may need to be developed iteratively to ensure coverage of the health inequalities issues or evidence on the impacts of an intervention on equality. </p> <p>Appropriate terminology for the search should be used, considering how language has evolved.</p> </div> <div class="section" title="5.5 Quality assurance" id="pmg20-quality-assurance"> <h3 class="title" id="quality-assurance">5.5 Quality assurance</h3> <p>Quality assuring the literature search is an important step in developing guideline recommendations. Studies have shown that errors do occur. </p> <p>For each search (including economic searches), the initial MEDLINE search strategy is quality assured by a second information specialist. A standardised checklist, based on the <a class="link" href="https://www.sciencedirect.com/science/article/pii/S0895435616000585" target="_top">PRESS peer review of electronic search strategies: 2015 guideline statement</a>, is used to ensure clarity and consistency when quality assuring search strategies. </p> <p>The information specialist carrying out the quality assurance process also considers how appropriate the overall search approach is to the parameters of the evidence review (for example, the time available to carry out the review). The quality assurance comments are recorded and the information specialist who conducted the search should respond to the comments and revise the search strategy as needed. </p> <p>Search strategy translations across the remaining databases are also checked by a second information specialist to ensure that the strategies have been adapted appropriately, in accordance with the interfaces and search functionality of the sources used. </p> </div> <div class="section" title="5.6 Documenting the search" id="pmg20-documenting-the-search"> <h3 class="title" id="documenting-the-search">5.6 Documenting the search</h3> <p>Details of the evidence search are included as appendices to the individual evidence reviews. They are published for consultation alongside the draft evidence review and included in the final version. </p> <p>Records are kept of the searches undertaken during guideline recommendation development for all review questions to ensure that the process for identifying the evidence is transparent and reproducible. </p> <p>We use the <a class="link" href="https://www.equator-network.org/reporting-guidelines/prisma-s/" target="_top">PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews</a> to inform search reporting. The search documentation is an audit trail that allows the reader to understand both the technical aspect of what was done (such as which sources were searched; what platform was used and on what date; any deviations from the original search protocol) and the underlying rationale for the search approach where this may not be immediately apparent. </p> <p>Documenting the search begins with creating the search protocol (see the <a class="link" href="identifying-the-evidence-literature-searching-and-evidence-submission#search-protocols" target="_top" rel="internal" data-chapter="identifying-the-evidence-literature-searching-and-evidence-submission" data-original-url="#search-protocols">section on search protocols</a>). If using an iterative or emergent stepped approach, initial search strategies, key decision points and the reasons for subsequent search steps are clearly documented in the search protocol and final evidence review. When using a proprietary search engine such as Google, whose underlying algorithm adapts to different users, the search is reported in a way that should allow the reader to understand what was done. </p> </div> <div class="section" title="5.7 Re-running searches" id="pmg20-re-running-searches"> <h3 class="title" id="re-running-searches">5.7 Re-running searches</h3> <p>Searches undertaken to identify evidence for each review question (including economics searches) may be re-run before consultation or before publication. For example, searches are re‑run if the evidence changes quickly, there is reason to believe that substantial new evidence exists, or the development time is longer than usual. </p> <p>A decision to re‑run searches is taken by the development team and staff with responsibility for quality assurance. </p> <p>If undertaken, searches are re‑run at least 6 to 8 weeks before the final committee meeting before consultation.</p> <p>If evidence is identified after the last cut‑off date for searching but before publication, a judgement on its impact is made by the development team and staff with responsibility for quality assurance. In exceptional circumstances, this evidence can be considered if its impact is judged as potentially substantial.</p> </div> <div class="section" title="5.8 Calls for evidence from stakeholders" id="pmg20-calls-for-evidence-from-stakeholders"> <h3 class="title" id="calls-for-evidence-from-stakeholders">5.8 Calls for evidence from stakeholders</h3> <p>In some topic areas or for some review questions, staff with responsibility for quality assurance, the development team or the committee may believe that there is relevant evidence in addition to that identified by the searches. In these situations, the development team may invite stakeholders, and possibly also other relevant organisations or individuals with a significant role or interest (see <a class="link" href="decision-making-committees#expert-witnesses" target="_top" rel="internal" data-chapter="decision-making-committees" data-original-url="#expert-witnesses">expert witnesses in the section on other attendees at committee meetings in the chapter on decision-making committees</a>), to submit evidence. A call for evidence is issued directly to registered stakeholders on the NICE website. Examples and details of process are included in the <a class="link" href="https://www.nice.org.uk/process/pmg20/resources/developing-nice-guidelines-the-manual-appendices-2549710189/chapter/appendix-j-call-for-evidence" target="_top">appendix on call for evidence and expert witnesses</a>. Confidential information should be kept to an absolute minimum.</p> </div> <div class="section" title="5.9 References and further reading" id="pmg20-references-and-further-reading-4"> <h3 class="title" id="references-and-further-reading-4">5.9 References and further reading</h3> <p>Ayiku L, Levay P, Hudson T et al. (2017) <a class="link" href="https://onlinelibrary.wiley.com/doi/10.1111/hir.12187" target="_top">The medline UK filter: development and validation of a geographic search filter to retrieve research about the UK from OVID medline</a>. Health Information and Libraries Journal 34(3): 200–216</p> <p>Ayiku L, Levay P, Hudson T et al. (2019) <a class="link" href="https://onlinelibrary.wiley.com/doi/10.1111/hir.12252" target="_top">The Embase UK filter: validation of a geographic search filter to retrieve research about the UK from OVID Embase</a>. Health Information and Libraries Journal 36(2): 121–133</p> <p>Ayiku L, Hudson T, Williams C et al. (2021) <a class="link" href="https://jmla.pitt.edu/ojs/jmla/article/view/1224" target="_top">The NICE OECD countries' geographic search filters: Part 2-validation of the MEDLINE and Embase (Ovid) filters</a>. Journal of the Medical Library Association 109(4): 583–9 </p> <p>Booth A, Briscoe S, Wright JM (2020) <a class="link" href="https://onlinelibrary.wiley.com/doi/abs/10.1002/jrsm.1386" target="_top">The "realist search": a systematic review of current practice and reporting</a>. Research Synthesis Methods 11: 14–35</p> <p>Canadian Agency for Drugs and Technologies in Health (2019) <a class="link" href="https://www.cadth.ca/resources/finding-evidence/grey-matters" target="_top">Grey Matters: a practical tool for searching health-related grey literature</a> [online; accessed 24 July 2023]</p> <p>Glanville J, Lefebvre C, Wright K (editors) (2008, updated 2017) <a class="link" href="https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/home" target="_top">The InterTASC Information Specialists' Subgroup Search Filters Resource</a> [online; accessed 24 July 2023]</p> <p>Kaltenthaler E, Tappenden P, Paisley S (2011) <a class="link" href="https://www.sheffield.ac.uk/nice-dsu/tsds/full-list" target="_top">NICE DSU Technical support document 13: identifying and reviewing evidence to inform the conceptualisation and population of cost-effectiveness models</a> [online; accessed 24 July 2023]</p> <p>Kugley S, Wade A, Thomas J et al. (2017) <a class="link" href="https://pubmed.ncbi.nlm.nih.gov/39258215/" target="_top">Searching for studies: a guide to information retrieval for Campbell systematic reviews</a>. Oslo: The Campbell Collaboration</p> <p>Lefebvre C, Glanville J, Briscoe S et al. <a class="link" href="https://training.cochrane.org/handbook/current/chapter-04" target="_top">Chapter 4: Searching for and selecting studies</a>. In: Higgins JPT, Thomas J, Cumpston M et al. (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021 </p> <p>McGowan J, Sampson M, Salzwedel DM et al. (2016) <a class="link" href="http://www.sciencedirect.com/science/article/pii/S0895435616000585" target="_top">PRESS Peer Review of Electronic Search Strategies: 2015 guideline statement</a>. Journal of Clinical Epidemiology 75: 40–6</p> <p>National Institute for Health and Care Excellence (2022) <a class="link" href="https://www.nice.org.uk/corporate/ecd9/chapter/overview" target="_top">NICE real-world evidence framework</a> [online; accessed 24 July 2023]</p> <p>Paisley S (2016) <a class="link" href="https://link.springer.com/article/10.1007/s40273-015-0372-x" target="_top">Identification of key parameters in decision-analytic models of cost-effectiveness: a description of sources and a recommended minimum search requirement</a>. Pharmacoeconomics 34: 597–8</p> <p>Rethlefsen M, Kirtley S, Waffenschmidt S et al. (2021) <a class="link" href="https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-020-01542-z" target="_top">PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews</a>. Systematic Reviews 10: 39</p> <p>Stansfield C, Stokes G, Thoman J (2022) <a class="link" href="https://onlinelibrary.wiley.com/doi/10.1002/jrsm.1537" target="_top">Applying machine classifiers to update searches: analysis from two case studies</a>. Research Synthesis Methods 13: 121–33</p> <p> <a class="link" href="https://sites.google.com/york.ac.uk/sureinfo/home" target="_top">Summarized research for Information Retrieval in HTA</a> (SuRe Info) [online; accessed 24 July 2023]</p> </div> </div> </div> <div class="product-footer"> </div> <br/> <div class="section-footer"> <ul class="pagination "> <li class="sectionmenu hidden-desktop hidden-tablet span8"></li> <li class="previous"><a href="/process/pmg20/chapter/developing-review-questions-and-planning-the-evidence-review" aria-label="Previous chapter"><span class="icon-nice-chevron-left"></span></a></li> <li class="next"><a href="/process/pmg20/chapter/reviewing-evidence" aria-label="Next chapter"><span class="hidden-phone">Next </span><span class="icon-nice-chevron-right"></span></a></li> </ul> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </main> <div id="global-nav-footer"></div> <script> var global_nav_config = { service: "", header: { cookie: false, skipLinkId: "content-start", search: { url: "/search", autocomplete: "/autocomplete?ajax=ajax", }, auth: { environment: "live" } } }; </script>  <script src="https://cdn.nice.org.uk/global-nav/global-nav.min.js" type="text/javascript"></script> <script src="//cdn.nice.org.uk/V2/Scripts/twitter.bootstrap.min.js" type="text/javascript"></script> <script src="/Themes/NICE.Bootstrap/scripts/niceorg/NICE.TopScroll.js" type="text/javascript"></script> <script src="//cdn.nice.org.uk/V2/Scripts/NICE.bootstrap.min.js" type="text/javascript"></script> </body> </html>

CINXE.COM

5 Identifying the evidence: literature searching and evidence submission | Developing NICE guidelines: the manual | Guidance | NICE