CINXE.COM

<!DOCTYPE html> <html lang="en" class="no-js"> <head> <meta charset="UTF-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="applicable-device" content="pc,mobile"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="robots" content="max-image-preview:large"> <meta name="access" content="Yes"> <meta name="360-site-verification" content="1268d79b5e96aecf3ff2a7dac04ad990" /> <title>The promise of automated machine learning for the genetic analysis of complex traits | Human Genetics</title> <meta name="twitter:site" content="@SpringerLink"/> <meta name="twitter:card" content="summary_large_image"/> <meta name="twitter:image:alt" content="Content cover image"/> <meta name="twitter:title" content="The promise of automated machine learning for the genetic analysis of complex traits"/> <meta name="twitter:description" content="Human Genetics - The genetic analysis of complex traits has been dominated by parametric statistical methods due to their theoretical properties, ease of use, computational efficiency, and..."/> <meta name="twitter:image" content="https://static-content.springer.com/image/art%3A10.1007%2Fs00439-021-02393-x/MediaObjects/439_2021_2393_Fig1_HTML.png"/> <meta name="journal_id" content="439"/> <meta name="dc.title" content="The promise of automated machine learning for the genetic analysis of complex traits"/> <meta name="dc.source" content="Human Genetics 2021 141:9"/> <meta name="dc.format" content="text/html"/> <meta name="dc.publisher" content="Springer"/> <meta name="dc.date" content="2021-10-28"/> <meta name="dc.type" content="ReviewPaper"/> <meta name="dc.language" content="En"/> <meta name="dc.copyright" content="2021 The Author(s)"/> <meta name="dc.rights" content="2021 The Author(s)"/> <meta name="dc.rightsAgent" content="journalpermissions@springernature.com"/> <meta name="dc.description" content="The genetic analysis of complex traits has been dominated by parametric statistical methods due to their theoretical properties, ease of use, computational efficiency, and intuitive interpretation. However, there are likely to be patterns arising from complex genetic architectures which are more easily detected and modeled using machine learning methods. Unfortunately, selecting the right machine learning algorithm and tuning its hyperparameters can be daunting for experts and non-experts alike. The goal of automated machine learning (AutoML) is to let a computer algorithm identify the right algorithms and hyperparameters thus taking the guesswork out of the optimization process. We review the promises and challenges of AutoML for the genetic analysis of complex traits and give an overview of several approaches and some example applications to omics data. It is our hope that this review will motivate studies to develop and evaluate novel AutoML methods and software in the genetics and genomics space. The promise of AutoML is to enable anyone, regardless of training or expertise, to apply machine learning as part of their genetic analysis strategy."/> <meta name="prism.issn" content="1432-1203"/> <meta name="prism.publicationName" content="Human Genetics"/> <meta name="prism.publicationDate" content="2021-10-28"/> <meta name="prism.volume" content="141"/> <meta name="prism.number" content="9"/> <meta name="prism.section" content="ReviewPaper"/> <meta name="prism.startingPage" content="1529"/> <meta name="prism.endingPage" content="1544"/> <meta name="prism.copyright" content="2021 The Author(s)"/> <meta name="prism.rightsAgent" content="journalpermissions@springernature.com"/> <meta name="prism.url" content="https://link.springer.com/article/10.1007/s00439-021-02393-x"/> <meta name="prism.doi" content="doi:10.1007/s00439-021-02393-x"/> <meta name="citation_pdf_url" content="https://link.springer.com/content/pdf/10.1007/s00439-021-02393-x.pdf"/> <meta name="citation_fulltext_html_url" content="https://link.springer.com/article/10.1007/s00439-021-02393-x"/> <meta name="citation_journal_title" content="Human Genetics"/> <meta name="citation_journal_abbrev" content="Hum Genet"/> <meta name="citation_publisher" content="Springer Berlin Heidelberg"/> <meta name="citation_issn" content="1432-1203"/> <meta name="citation_title" content="The promise of automated machine learning for the genetic analysis of complex traits"/> <meta name="citation_volume" content="141"/> <meta name="citation_issue" content="9"/> <meta name="citation_publication_date" content="2022/09"/> <meta name="citation_online_date" content="2021/10/28"/> <meta name="citation_firstpage" content="1529"/> <meta name="citation_lastpage" content="1544"/> <meta name="citation_article_type" content="Review"/> <meta name="citation_fulltext_world_readable" content=""/> <meta name="citation_language" content="en"/> <meta name="dc.identifier" content="doi:10.1007/s00439-021-02393-x"/> <meta name="DOI" content="10.1007/s00439-021-02393-x"/> <meta name="size" content="240777"/> <meta name="citation_doi" content="10.1007/s00439-021-02393-x"/> <meta name="citation_springer_api_url" content="http://api.springer.com/xmldata/jats?q=doi:10.1007/s00439-021-02393-x&api_key="/> <meta name="description" content="The genetic analysis of complex traits has been dominated by parametric statistical methods due to their theoretical properties, ease of use, computational"/> <meta name="dc.creator" content="Manduchi, Elisabetta"/> <meta name="dc.creator" content="Romano, Joseph D."/> <meta name="dc.creator" content="Moore, Jason H."/> <meta name="dc.subject" content="Human Genetics"/> <meta name="dc.subject" content="Molecular Medicine"/> <meta name="dc.subject" content="Gene Function"/> <meta name="dc.subject" content="Metabolic Diseases"/> <meta name="citation_reference" content="citation_journal_title=J Pers Med; citation_title=Genome wide epistasis study of on-statin cardiovascular events with iterative feature reduction and selection; citation_author=SM Adams, H Feroze, T Nguyen; citation_publication_date=2020; citation_doi=10.3390/jpm10040212; citation_id=CR1"/> <meta name="citation_reference" content="citation_journal_title=PLoS One; citation_title=Cardiovascular disease risk prediction using automated machine learning: a prospective study of 423,604 UK Biobank participants; citation_author=AM Alaa, T Bolton, ED Angelantonio; citation_volume=14; citation_publication_date=2019; citation_doi=10.1371/journal.pone.0213653; citation_id=CR2"/> <meta name="citation_reference" content="Alaa AM, van der Schaar M (2018a) AutoPrognosis: automated clinical prognostic modeling via Bayesian optimization with structured kernel learning. In: International conference on machine learning. PMLR, pp 139–148. http://proceedings.mlr.press/v80/alaa18b.html "/> <meta name="citation_reference" content="citation_journal_title=Sci Rep; citation_title=Prognostication and risk factors for cystic fibrosis via automated machine learning; citation_author=AM Alaa, M Schaar; citation_volume=8; citation_publication_date=2018; citation_pages=11242; citation_doi=10.1038/s41598-018-29523-2; citation_id=CR4"/> <meta name="citation_reference" content="citation_journal_title=J Proteome Res; citation_title=Deep learning accurately predicts estrogen receptor status in breast cancer metabolomics data; citation_author=FM Alakwaa, K Chaudhary, LX Garmire; citation_volume=17; citation_publication_date=2018; citation_pages=337-347; citation_doi=10.1021/acs.jproteome.7b00595; citation_id=CR5"/> <meta name="citation_reference" content="citation_journal_title=Sci Rep; citation_title=Genome-wide investigation of gene-cancer associations for the prediction of novel therapeutic targets in oncology; citation_author=A Bazaga, D Leggate, H Weisser; citation_volume=10; citation_publication_date=2020; citation_pages=10787; citation_doi=10.1038/s41598-020-67846-1; citation_id=CR6"/> <meta name="citation_reference" content="citation_journal_title=Comput Sci Discov; citation_title=Hyperopt: a Python library for model selection and hyperparameter optimization; citation_author=J Bergstra, B Komer, C Eliasmith; citation_volume=8; citation_publication_date=2015; citation_doi=10.1088/1749-4699/8/1/014008; citation_id=CR7"/> <meta name="citation_reference" content="Brochu E, Cora VM, de Freitas N (2010) A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. http://arxiv.org/abs/1012.2599 [cs]"/> <meta name="citation_reference" content="citation_journal_title=Nature; citation_title=The UK Biobank resource with deep phenotyping and genomic data; citation_author=C Bycroft, C Freeman, D Petkova; citation_volume=562; citation_publication_date=2018; citation_pages=203-209; citation_doi=10.1038/s41586-018-0579-z; citation_id=CR9"/> <meta name="citation_reference" content="citation_title=Deep learning with HO; citation_publication_date=2021; citation_id=CR10; citation_author=A Candel; citation_author=E LeDell; citation_publisher=H2O.ai, Inc."/> <meta name="citation_reference" content="Caruana R, Niculescu-Mizil A, Crew G, Ksikes A (2004) Ensemble selection from libraries of models. In: Proceedings of the twenty-first international conference on Machine learning. Association for Computing Machinery, New York, NY, USA, p 18"/> <meta name="citation_reference" content="citation_title=Visualizing the feature importance for black box models; citation_inbook_title=Machine learning and knowledge discovery in databases; citation_publication_date=2019; citation_pages=655-670; citation_id=CR12; citation_author=G Casalicchio; citation_author=C Molnar; citation_author=B Bischl; citation_publisher=Springer International Publishing"/> <meta name="citation_reference" content="Chen T, Guestrin C (2016) XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, New York, NY, USA, pp 785–794"/> <meta name="citation_reference" content="citation_journal_title=Abdom Radiol (NY); citation_title=Preoperative prediction of the stage, size, grade, and necrosis score in clear cell renal cell carcinoma using MRI-based radiomics; citation_author=JW Choi, R Hu, Y Zhao; citation_publication_date=2021; citation_doi=10.1007/s00261-020-02876-x; citation_id=CR14"/> <meta name="citation_reference" content="citation_title=A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: NSGA-II; citation_inbook_title=Parallel problem solving from nature PPSN VI; citation_publication_date=2000; citation_pages=849-858; citation_id=CR15; citation_author=K Deb; citation_author=S Agrawal; citation_author=A Pratap; citation_author=T Meyarivan; citation_publisher=Springer"/> <meta name="citation_reference" content="citation_journal_title=IEEE Trans Evol Comput; citation_title=A fast and elitist multiobjective genetic algorithm: NSGA-II; citation_author=K Deb, A Pratap, S Agarwal, T Meyarivan; citation_volume=6; citation_publication_date=2002; citation_pages=182-197; citation_doi=10.1109/4235.996017; citation_id=CR16"/> <meta name="citation_reference" content="citation_title=Neural architecture search; citation_inbook_title=Automated machine learning; citation_publication_date=2019; citation_pages=63-77; citation_id=CR17; citation_author=T Elsken; citation_author=JH Metzen; citation_author=F Hutter; citation_publisher=Springer International Publishing"/> <meta name="citation_reference" content="Erickson N, Mueller J, Shirkov A, et al (2020) AutoGluon-tabular: robust and accurate AutoML for structured data. http://arxiv.org/abs/20030.6505 [cs, stat]"/> <meta name="citation_reference" content="citation_title=Efficient and robust automated machine learning; citation_inbook_title=Advances in neural information processing systems; citation_publication_date=2015; citation_pages=2962-2970; citation_id=CR19; citation_author=M Feurer; citation_author=A Klein; citation_author=K Eggensperger; citation_publisher=Curran Associates, Inc."/> <meta name="citation_reference" content="Feurer M, Eggensperger K, Falkner S et al (2018) Practical automated machine learning for the AutoML challenge. ICML 2018 AutoML Workshop. https://ml.informatik.uni-freiburg.de/wp-content/uploads/papers/18-AUTOML-AutoChallenge.pdf "/> <meta name="citation_reference" content="Frank E, Hall MA, Witten IH (2016) The Weka Workbench. Online Appendix for “Data mining: practical machine learning tools and techniques”, Morgan Kaufmann, Fourth Edition. https://www.cs.waikato.ac.nz/ml/weka/Witten_et_al_2016_appendix.pdf . Accessed 19 Apr 2021"/> <meta name="citation_reference" content="citation_journal_title=Nat Commun; citation_title=Annotating pathogenic non-coding variants in genic regions; citation_author=S Gelfman, Q Wang, KM McSweeney; citation_volume=8; citation_publication_date=2017; citation_pages=236; citation_doi=10.1038/s41467-017-00141-2; citation_id=CR22"/> <meta name="citation_reference" content="Gijsbers P, LeDell E, Thomas J, et al (2019) An open source AutoML benchmark. http://arxiv.org/abs/1907.00909 [cs, stat]"/> <meta name="citation_reference" content="citation_journal_title=Int J Biostat; citation_title=Super learner for survival data prediction; citation_author=MK Golmakani, EC Polley; citation_publication_date=2020; citation_doi=10.1515/ijb-2019-0065; citation_id=CR24"/> <meta name="citation_reference" content="citation_title=Analysis of the AutoML challenge series 2015–2018; citation_inbook_title=Automated machine learning: methods, systems, challenges; citation_publication_date=2019; citation_pages=177-219; citation_id=CR25; citation_author=I Guyon; citation_author=L Sun-Hosoya; citation_author=M Boullé; citation_publisher=Springer International Publishing"/> <meta name="citation_reference" content="citation_journal_title=J Dent Res; citation_title=Metabolomics insights in early childhood caries; citation_author=LH Heimisdottir, BM Lin, H Cho; citation_publication_date=2021; citation_doi=10.1177/0022034520982963; citation_id=CR26"/> <meta name="citation_reference" content="Hooker G, Mentch L, Zhou S (2021) Unrestricted permutation forces extrapolation: variable importance requires at least one more model, or there is no free variable importance. http://arxiv.org/abs/1905.03151 [cs, stat]"/> <meta name="citation_reference" content="citation_journal_title=J Med Internet Res; citation_title=Transfer learning for risk classification of social media posts: model evaluation study; citation_author=D Howard, MM Maslej, J Lee; citation_volume=22; citation_publication_date=2020; citation_doi=10.2196/15371; citation_id=CR28"/> <meta name="citation_reference" content="Huang S, Ji X, Cho M, et al (2021) DL-PRS: a novel deep learning approach to polygenic risk scores. BMC Bioinformatics. https://doi.org/10.21203/rs.3.rs-423764/v1 "/> <meta name="citation_reference" content="citation_title=Sequential model-based optimization for general algorithm configuration; citation_inbook_title=Learning and intelligent optimization; citation_publication_date=2011; citation_pages=507-523; citation_id=CR30; citation_author=F Hutter; citation_author=HH Hoos; citation_author=K Leyton-Brown; citation_publisher=Springer"/> <meta name="citation_reference" content="citation_title=Automated machine learning: methods, systems; citation_publication_date=2019; citation_id=CR31; citation_publisher=Springer International Publishing"/> <meta name="citation_reference" content="citation_journal_title=Am J Hum Genet; citation_title=REVEL: an ensemble method for predicting the pathogenicity of rare missense variants; citation_author=NM Ioannidis, JH Rothstein, V Pejaver; citation_volume=99; citation_publication_date=2016; citation_pages=877-885; citation_doi=10.1016/j.ajhg.2016.08.016; citation_id=CR32"/> <meta name="citation_reference" content="citation_journal_title=Electron J Statist; citation_title=Fast Bayesian hyperparameter optimization on large datasets; citation_author=A Klein, S Falkner, S Bartels; citation_publication_date=2017; citation_doi=10.1214/17-EJS1335SI; citation_id=CR33"/> <meta name="citation_reference" content="citation_title=Hyperopt-sklearn; citation_inbook_title=Automated machine learning: methods, systems, challenges; citation_publication_date=2019; citation_pages=97-111; citation_id=CR34; citation_author=B Komer; citation_author=J Bergstra; citation_author=C Eliasmith; citation_publisher=Springer International Publishing"/> <meta name="citation_reference" content="citation_journal_title=Appl Intell; citation_title=Overcoming the myopia of inductive learning algorithms with RELIEFF; citation_author=I Kononenko, E Šimec, M Robnik-Šikonja; citation_volume=7; citation_publication_date=1997; citation_pages=39-55; citation_doi=10.1023/A:1008280620621; citation_id=CR35"/> <meta name="citation_reference" content="citation_journal_title=J Mach Learn Res; citation_title=Auto-WEKA 2.0: automatic model selection and hyperparameter optimization in WEKA; citation_author=L Kotthoff, C Thornton, HH Hoos; citation_volume=18; citation_publication_date=2017; citation_pages=1-5; citation_id=CR36"/> <meta name="citation_reference" content="citation_title=Auto-WEKA: automatic model selection and hyperparameter optimization in WEKA; citation_inbook_title=Automated machine learning; citation_publication_date=2019; citation_pages=81-95; citation_id=CR37; citation_author=L Kotthoff; citation_author=C Thornton; citation_author=HH Hoos; citation_publisher=Springer International Publishing"/> <meta name="citation_reference" content="citation_title=Applied predictive modeling; citation_publication_date=2013; citation_id=CR38; citation_author=M Kuhn; citation_author=K Johnson; citation_publisher=Springer-Verlag"/> <meta name="citation_reference" content="citation_journal_title=Nature; citation_title=Integrative analysis of 111 reference human epigenomes; citation_author=A Kundaje, W Meuleman, J Ernst; citation_volume=518; citation_publication_date=2015; citation_pages=317-330; citation_doi=10.1038/nature14248; citation_id=CR39"/> <meta name="citation_reference" content="citation_journal_title=Bioinformatics; citation_title=Evaluating recommender systems for AI-driven biomedical informatics; citation_author=W Cava, H Williams, W Fu; citation_publication_date=2020; citation_doi=10.1093/bioinformatics/btaa698; citation_id=CR40"/> <meta name="citation_reference" content="citation_journal_title=Bioinformatics; citation_title=Scaling tree-based automated machine learning to biomedical big data with a feature set selector; citation_author=TT Le, W Fu, JH Moore; citation_volume=36; citation_publication_date=2020; citation_pages=250-256; citation_doi=10.1093/bioinformatics/btz470; citation_id=CR41"/> <meta name="citation_reference" content="LeDell E, Poirier S (2020) H2O AutoML: scalable automatic machine learning. In: 7th ICML workshop on automated machine learning. https://www.automl.org/wp-content/uploads/2020/07/AutoML_2020_paper_61.pdf "/> <meta name="citation_reference" content="citation_journal_title=Behav Neurol; citation_title=Use of deep-learning genomics to discriminate healthy individuals from those with Alzheimer’s disease or mild cognitive impairment; citation_author=L Li, Y Yang, Q Zhang; citation_volume=2021; citation_publication_date=2021; citation_pages=3359103; citation_doi=10.1155/2021/3359103; citation_id=CR43"/> <meta name="citation_reference" content="citation_journal_title=Adv Neural Inf Process Syst; citation_title=A unified approach to interpreting model predictions; citation_author=SM Lundberg, S-I Lee; citation_volume=30; citation_publication_date=2017; citation_pages=4765-4774; citation_id=CR44"/> <meta name="citation_reference" content="citation_journal_title=Int J Public Health; citation_title=Leveraging automated machine learning for the analysis of global public health data: a case study in malaria; citation_author=E Manduchi, JH Moore; citation_publication_date=2021; citation_doi=10.3389/ijph.2021.614296; citation_id=CR45"/> <meta name="citation_reference" content="citation_journal_title=BMC Bioinform; citation_title=Embedding covariate adjustments in tree-based automated machine learning for biomedical big data analyses; citation_author=E Manduchi, W Fu, JD Romano; citation_volume=36; citation_publication_date=2020; citation_pages=1772; citation_doi=10.1093/bioinformatics/btz796; citation_id=CR46"/> <meta name="citation_reference" content="citation_journal_title=IEEE/ACM Trans Comput Biol Bioinform; citation_title=Genetic analysis of coronary artery disease using tree-based automated machine learning informed by biology-based feature selection; citation_author=E Manduchi, T Le, W Fu, JH Moore; citation_publication_date=2021; citation_doi=10.1109/TCBB.2021.3099068; citation_id=CR47"/> <meta name="citation_reference" content="citation_title=Towards automatically-tuned deep neural networks; citation_inbook_title=Automated machine learning; citation_publication_date=2019; citation_pages=135-149; citation_id=CR48; citation_author=H Mendoza; citation_author=A Klein; citation_author=M Feurer; citation_publisher=Springer International Publishing"/> <meta name="citation_reference" content="citation_journal_title=Mach Learn; citation_title=ML-Plan: automated machine learning via hierarchical planning; citation_author=F Mohr, M Wever, E Hüllermeier; citation_volume=107; citation_publication_date=2018; citation_pages=1495-1515; citation_doi=10.1007/s10994-018-5735-z; citation_id=CR49"/> <meta name="citation_reference" content="Molnar C, König G, Herbinger J, et al (2021) General pitfalls of model-agnostic interpretation methods for machine learning models. http://arxiv.org/abs/2007.04131 [cs, stat]"/> <meta name="citation_reference" content="Molnar C (2021) Interpretable machine learning"/> <meta name="citation_reference" content="citation_title=AutoWeka: toward an automated data mining software for QSAR and QSPR studies; citation_inbook_title=Artificial neural networks; citation_publication_date=2015; citation_pages=119-147; citation_id=CR52; citation_author=C Nantasenamat; citation_author=A Worachartcheewan; citation_author=S Jamsak; citation_publisher=Springer"/> <meta name="citation_reference" content="citation_journal_title=Front Genet; citation_title=Reaching the end-game for GWAS: machine learning approaches for the prioritization of complex disease loci; citation_author=HL Nicholls, CR John, DS Watson; citation_volume=11; citation_publication_date=2020; citation_pages=350; citation_doi=10.3389/fgene.2020.00350; citation_id=CR53"/> <meta name="citation_reference" content="citation_title=TPOT: a tree-based pipeline optimization tool for automating machine learning; citation_inbook_title=Automated machine learning: methods, systems, challenges; citation_publication_date=2019; citation_pages=151-160; citation_id=CR54; citation_author=RS Olson; citation_author=JH Moore; citation_publisher=Springer International Publishing"/> <meta name="citation_reference" content="citation_title=Automating biomedical data science through tree-based pipeline optimization; citation_inbook_title=Applications of evolutionary computation; citation_publication_date=2016; citation_pages=123-137; citation_id=CR55; citation_author=RS Olson; citation_author=RJ Urbanowicz; citation_author=PC Andrews; citation_publisher=Springer International Publishing"/> <meta name="citation_reference" content="citation_title=A system for accessible artificial intelligence; citation_inbook_title=Genetic programming theory and practice XV; citation_publication_date=2018; citation_pages=121-134; citation_id=CR56; citation_author=RS Olson; citation_author=M Sipper; citation_author=WL Cava; citation_publisher=Springer International Publishing"/> <meta name="citation_reference" content="citation_journal_title=BioData Min; citation_title=A comparison of methods for interpreting random forest models of genetic association in the presence of non-additive interactions; citation_author=A Orlenko, JH Moore; citation_volume=14; citation_publication_date=2021; citation_pages=9; citation_doi=10.1186/s13040-021-00243-0; citation_id=CR57"/> <meta name="citation_reference" content="citation_journal_title=Pac Symp Biocomput; citation_title=onsiderations for automated machine learning in clinical metabolic profiling: altered homocysteine plasma concentration associated with metformin exposure; citation_author=A Orlenko, JH Moore, P Orzechowski; citation_volume=23; citation_publication_date=2018; citation_pages=460-471; citation_id=CR59"/> <meta name="citation_reference" content="citation_journal_title=Bioinformatics; citation_title=Model selection for metabolomics: predicting diagnosis of coronary artery disease using automated machine learning; citation_author=A Orlenko, D Kofink, L-P Lyytikäinen; citation_volume=36; citation_publication_date=2020; citation_pages=1772-1778; citation_doi=10.1093/bioinformatics/btz796; citation_id=CR58"/> <meta name="citation_reference" content="citation_journal_title=Int J Prod Res; citation_title=Filtered beam search in scheduling†; citation_author=PS Ow, TE Morton; citation_volume=26; citation_publication_date=1988; citation_pages=35-62; citation_doi=10.1080/00207548808947840; citation_id=CR60"/> <meta name="citation_reference" content="citation_journal_title=J Clin Med; citation_title=Physician-friendly machine learning: a case study with cardiovascular disease risk prediction; citation_author=M Padmanabhan, P Yuan, G Chada, HV Nguyen; citation_volume=8; citation_publication_date=2019; citation_pages=1050; citation_doi=10.3390/jcm8071050; citation_id=CR61"/> <meta name="citation_reference" content="citation_journal_title=J Mach Learn Res; citation_title=Scikit-learn: machine learning in python; citation_author=F Pedregosa, G Varoquaux, A Gramfort; citation_volume=12; citation_publication_date=2011; citation_pages=2825-2830; citation_id=CR62"/> <meta name="citation_reference" content="citation_journal_title=Nucleic Acids Res; citation_title=CADD: predicting the deleteriousness of variants throughout the human genome; citation_author=P Rentzsch, D Witten, GM Cooper; citation_volume=47; citation_publication_date=2019; citation_pages=D886-D894; citation_doi=10.1093/nar/gky1016; citation_id=CR63"/> <meta name="citation_reference" content="citation_journal_title=Am J Hum Genet; citation_title=Multifactor-dimensionality reduction reveals high-order interactions among estrogen-metabolism genes in sporadic breast cancer; citation_author=MD Ritchie, LW Hahn, N Roodi; citation_volume=69; citation_publication_date=2001; citation_pages=138-147; citation_doi=10.1086/321276; citation_id=CR64"/> <meta name="citation_reference" content="citation_journal_title=Nat Methods; citation_title=Functional annotation of noncoding sequence variants; citation_author=GRS Ritchie, I Dunham, E Zeggini, P Flicek; citation_volume=11; citation_publication_date=2014; citation_pages=294-296; citation_doi=10.1038/nmeth.2832; citation_id=CR65"/> <meta name="citation_reference" content="citation_journal_title=Genet Program Evol Mach; citation_title=TPOT-NN: augmenting tree-based automated machine learning with neural network estimators; citation_author=JD Romano, TT Le, W Fu, JH Moore; citation_publication_date=2021; citation_doi=10.1007/s10710-021-09401-z; citation_id=CR66"/> <meta name="citation_reference" content="citation_journal_title=Sci Rep; citation_title=Imbalance-aware machine learning for predicting rare and common disease-associated non-coding variants; citation_author=M Schubach, M Re, PN Robinson, G Valentini; citation_volume=7; citation_publication_date=2017; citation_pages=2959; citation_doi=10.1038/s41598-017-03011-5; citation_id=CR67"/> <meta name="citation_reference" content="citation_journal_title=Diagnostics (basel); citation_title=Forecasting the walking assistance rehabilitation level of stroke patients using artificial intelligence; citation_author=K Seo, B Chung, HP Panchaseelan; citation_volume=11; citation_publication_date=2021; citation_pages=1096; citation_doi=10.3390/diagnostics11061096; citation_id=CR68"/> <meta name="citation_reference" content="citation_journal_title=HHE; citation_title=Comprehensive assessment of genotype imputation performance; citation_author=S Shi, N Yuan, M Yang; citation_volume=83; citation_publication_date=2018; citation_pages=107-116; citation_doi=10.1159/000489758; citation_id=CR69"/> <meta name="citation_reference" content="citation_journal_title=Jpn J Infect Dis; citation_title=Data mining and machine learning algorithms using IL28B genotype and biochemical markers best predicted advanced liver fibrosis in chronic Hepatitis C; citation_author=HI Shousha, AH Awad, DA Omran; citation_volume=71; citation_publication_date=2018; citation_pages=51-57; citation_doi=10.7883/yoken.JJID.2017.089; citation_id=CR70"/> <meta name="citation_reference" content="citation_journal_title=Stat Appl Genet Mol Biol; citation_title=Super learning: an application to the prediction of HIV-1 drug resistance; citation_author=SE Sinisi, EC Polley, ML Petersen; citation_publication_date=2007; citation_doi=10.2202/1544-6115.1240; citation_id=CR71"/> <meta name="citation_reference" content="citation_title=Toward the automated analysis of complex diseases in genome-wide association studies using genetic programming; citation_inbook_title=Proceedings of the genetic and evolutionary computation conference; citation_publication_date=2017; citation_pages=489-496; citation_id=CR72; citation_author=A Sohn; citation_author=RS Olson; citation_author=JH Moore; citation_publisher=Association for Computing Machinery"/> <meta name="citation_reference" content="citation_journal_title=Proc Natl Acad Sci USA; citation_title=Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles; citation_author=A Subramanian, P Tamayo, VK Mootha; citation_volume=102; citation_publication_date=2005; citation_pages=15545-15550; citation_doi=10.1073/pnas.0506580102; citation_id=CR73"/> <meta name="citation_reference" content="Thornton C, Hutter F, Hoos HH, Leyton-Brown K (2013) Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms. In: Proceeding of KDD-2013, pp 847–855. https://doi.org/10.1145/2487575.2487629 "/> <meta name="citation_reference" content="citation_journal_title=Circulation; citation_title=Druggability of coronary artery disease risk loci; citation_author=V Tragante, D Hemerich, M Alshabeeb; citation_volume=11; citation_publication_date=2018; citation_doi=10.1161/CIRCGEN.117.001977; citation_id=CR75"/> <meta name="citation_reference" content="citation_journal_title=Front Neural Circ; citation_title=Automated curation of CNMF-E-extracted ROI spatial footprints and calcium traces using open-source AutoML tools; citation_author=LM Tran, AJ Mocle, AI Ramsaran; citation_publication_date=2020; citation_doi=10.3389/fncir.2020.00042; citation_id=CR76"/> <meta name="citation_reference" content="citation_journal_title=Stat Appl Genetics Mol Biol; citation_title=Super learner; citation_author=MJ Laan, EC Polley, AE Hubbard; citation_publication_date=2007; citation_doi=10.2202/1544-6115.1309; citation_id=CR77"/> <meta name="citation_reference" content="citation_journal_title=SIGKDD Explor Newsl; citation_title=OpenML: networked science in machine learning; citation_author=J Vanschoren, JN Rijn, B Bischl, L Torgo; citation_volume=15; citation_publication_date=2014; citation_pages=49-60; citation_doi=10.1145/2641190.2641198; citation_id=CR78"/> <meta name="citation_reference" content="citation_journal_title=Sci Rep; citation_title=Multimodal deep learning models for early detection of Alzheimer’s disease stage; citation_author=J Venugopalan, L Tong, HR Hassanzadeh, MD Wang; citation_volume=11; citation_publication_date=2021; citation_pages=3254; citation_doi=10.1038/s41598-020-74399-w; citation_id=CR79"/> <meta name="citation_reference" content="citation_journal_title=Front Neurol; citation_title=Automatic machine-learning-based outcome prediction in patients with primary intracerebral hemorrhage; citation_author=H-L Wang, W-Y Hsu, M-H Lee; citation_volume=10; citation_publication_date=2019; citation_pages=910; citation_doi=10.3389/fneur.2019.00910; citation_id=CR80"/> <meta name="citation_reference" content="Wang F, Jiang M, Qian C, et al (2017) Residual attention network for image classification. http://arxiv.org/abs/1704.06904 [cs]"/> <meta name="citation_reference" content="Wang C, Wu Q, Weimer M, Zhu E (2021) FLAML: a fast and lightweight AutoML Library. http://arxiv.org/abs/1911.04706 [cs, stat]"/> <meta name="citation_reference" content="citation_journal_title=Artif Intell Med; citation_title=Automated machine learning: review of the state-of-the-art and opportunities for healthcare; citation_author=J Waring, C Lindvall, R Umeton; citation_volume=104; citation_publication_date=2020; citation_doi=10.1016/j.artmed.2020.101822; citation_id=CR83"/> <meta name="citation_reference" content="citation_journal_title=Neural Comput; citation_title=The lack of a priori distinctions between learning algorithms; citation_author=DH Wolpert; citation_volume=8; citation_publication_date=1996; citation_pages=1341-1390; citation_doi=10.1162/neco.1996.8.7.1341; citation_id=CR84"/> <meta name="citation_reference" content="citation_journal_title=IEEE Trans Evol Comput; citation_title=No free lunch theorems for optimization; citation_author=DH Wolpert, WG Macready; citation_volume=1; citation_publication_date=1997; citation_pages=67-82; citation_doi=10.1109/4235.585893; citation_id=CR85"/> <meta name="citation_reference" content="Yang C, Akimoto Y, Kim DW, Udell M (2019) OBOE: collaborative filtering for AutoML model selection. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp 1173–1183. https://doi.org/10.1145/3292500.3330909 "/> <meta name="citation_reference" content="citation_title=Evaluating machine learning models: a beginner’s guide to key concepts and pitfalls; citation_publication_date=2015; citation_id=CR87; citation_author=A Zheng; citation_publisher=O’Reilly Media"/> <meta name="citation_author" content="Manduchi, Elisabetta"/> <meta name="citation_author_institution" content="Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, USA"/> <meta name="citation_author_institution" content="Institute for Biomedical Informatics, University of Pennsylvania, Philadelphia, USA"/> <meta name="citation_author" content="Romano, Joseph D."/> <meta name="citation_author_institution" content="Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, USA"/> <meta name="citation_author" content="Moore, Jason H."/> <meta name="citation_author_email" content="jhmoore@upenn.edu"/> <meta name="citation_author_institution" content="Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, USA"/> <meta name="citation_author_institution" content="Institute for Biomedical Informatics, University of Pennsylvania, Philadelphia, USA"/> <meta name="format-detection" content="telephone=no"/> <meta name="citation_cover_date" content="2022/09/01"/> <meta property="og:url" content="https://link.springer.com/article/10.1007/s00439-021-02393-x"/> <meta property="og:type" content="article"/> <meta property="og:site_name" content="SpringerLink"/> <meta property="og:title" content="The promise of automated machine learning for the genetic analysis of complex traits - Human Genetics"/> <meta property="og:description" content="The genetic analysis of complex traits has been dominated by parametric statistical methods due to their theoretical properties, ease of use, computational efficiency, and intuitive interpretation. However, there are likely to be patterns arising from complex genetic architectures which are more easily detected and modeled using machine learning methods. Unfortunately, selecting the right machine learning algorithm and tuning its hyperparameters can be daunting for experts and non-experts alike. The goal of automated machine learning (AutoML) is to let a computer algorithm identify the right algorithms and hyperparameters thus taking the guesswork out of the optimization process. We review the promises and challenges of AutoML for the genetic analysis of complex traits and give an overview of several approaches and some example applications to omics data. It is our hope that this review will motivate studies to develop and evaluate novel AutoML methods and software in the genetics and genomics space. The promise of AutoML is to enable anyone, regardless of training or expertise, to apply machine learning as part of their genetic analysis strategy."/> <meta property="og:image" content="https://static-content.springer.com/image/art%3A10.1007%2Fs00439-021-02393-x/MediaObjects/439_2021_2393_Fig1_HTML.png"/> <meta name="format-detection" content="telephone=no"> <link rel="apple-touch-icon" sizes="180x180" href=/oscar-static/img/favicons/darwin/apple-touch-icon-92e819bf8a.png> <link rel="icon" type="image/png" sizes="192x192" href=/oscar-static/img/favicons/darwin/android-chrome-192x192-6f081ca7e5.png> <link rel="icon" type="image/png" sizes="32x32" href=/oscar-static/img/favicons/darwin/favicon-32x32-1435da3e82.png> <link rel="icon" type="image/png" sizes="16x16" href=/oscar-static/img/favicons/darwin/favicon-16x16-ed57f42bd2.png> <link rel="shortcut icon" data-test="shortcut-icon" href=/oscar-static/img/favicons/darwin/favicon-c6d59aafac.ico> <meta name="theme-color" content="#e6e6e6">   <link rel="stylesheet" media="print" href=/oscar-static/app-springerlink/css/print-b8af42253b.css> <style> html{text-size-adjust:100%;line-height:1.15}body{font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif;line-height:1.8;margin:0}details,main{display:block}h1{font-size:2em;margin:.67em 0}a{background-color:transparent;color:#025e8d}sub{bottom:-.25em;font-size:75%;line-height:0;position:relative;vertical-align:baseline}img{border:0;height:auto;max-width:100%;vertical-align:middle}button,input{font-family:inherit;font-size:100%;line-height:1.15;margin:0;overflow:visible}button{text-transform:none}[type=button],[type=submit],button{-webkit-appearance:button}[type=search]{-webkit-appearance:textfield;outline-offset:-2px}summary{display:list-item}[hidden]{display:none}button{cursor:pointer}svg{height:1rem;width:1rem} </style> <style>@media only print, only all and (prefers-color-scheme: no-preference), only all and (prefers-color-scheme: light), only all and (prefers-color-scheme: dark) { body{background:#fff;color:#222;font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif;line-height:1.8;min-height:100%}a{color:#025e8d;text-decoration:underline;text-decoration-skip-ink:auto}button{cursor:pointer}img{border:0;height:auto;max-width:100%;vertical-align:middle}html{box-sizing:border-box;font-size:100%;height:100%;overflow-y:scroll}h1{font-size:2.25rem}h2{font-size:1.75rem}h1,h2,h4{font-weight:700;line-height:1.2}h4{font-size:1.25rem}body{font-size:1.125rem}*{box-sizing:inherit}p{margin-bottom:2rem;margin-top:0}p:last-of-type{margin-bottom:0}.c-ad{text-align:center}@media only screen and (min-width:480px){.c-ad{padding:8px}}.c-ad--728x90{display:none}.c-ad--728x90 .c-ad__inner{min-height:calc(1.5em + 94px)}@media only screen and (min-width:876px){.js .c-ad--728x90{display:none}}.c-ad__label{color:#333;font-size:.875rem;font-weight:400;line-height:1.5;margin-bottom:4px}.c-ad__label,.c-status-message{font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif}.c-status-message{align-items:center;box-sizing:border-box;display:flex;position:relative;width:100%}.c-status-message :last-child{margin-bottom:0}.c-status-message--boxed{background-color:#fff;border:1px solid #ccc;line-height:1.4;padding:16px}.c-status-message__heading{font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif;font-size:.875rem;font-weight:700}.c-status-message__icon{fill:currentcolor;display:inline-block;flex:0 0 auto;height:1.5em;margin-right:8px;transform:translate(0);vertical-align:text-top;width:1.5em}.c-status-message__icon--top{align-self:flex-start}.c-status-message--info .c-status-message__icon{color:#003f8d}.c-status-message--boxed.c-status-message--info{border-bottom:4px solid #003f8d}.c-status-message--error .c-status-message__icon{color:#c40606}.c-status-message--boxed.c-status-message--error{border-bottom:4px solid #c40606}.c-status-message--success .c-status-message__icon{color:#00b8b0}.c-status-message--boxed.c-status-message--success{border-bottom:4px solid #00b8b0}.c-status-message--warning .c-status-message__icon{color:#edbc53}.c-status-message--boxed.c-status-message--warning{border-bottom:4px solid #edbc53}.eds-c-header{background-color:#fff;border-bottom:2px solid #01324b;font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif;font-size:1rem;line-height:1.5;padding:8px 0 0}.eds-c-header__container{align-items:center;display:flex;flex-wrap:nowrap;gap:8px 16px;justify-content:space-between;margin:0 auto 8px;max-width:1280px;padding:0 8px;position:relative}.eds-c-header__nav{border-top:2px solid #c5e0f4;padding-top:4px;position:relative}.eds-c-header__nav-container{align-items:center;display:flex;flex-wrap:wrap;margin:0 auto 4px;max-width:1280px;padding:0 8px;position:relative}.eds-c-header__nav-container>:not(:last-child){margin-right:32px}.eds-c-header__link-container{align-items:center;display:flex;flex:1 0 auto;gap:8px 16px;justify-content:space-between}.eds-c-header__list{list-style:none;margin:0;padding:0}.eds-c-header__list-item{font-weight:700;margin:0 auto;max-width:1280px;padding:8px}.eds-c-header__list-item:not(:last-child){border-bottom:2px solid #c5e0f4}.eds-c-header__item{color:inherit}@media only screen and (min-width:768px){.eds-c-header__item--menu{display:none;visibility:hidden}.eds-c-header__item--menu:first-child+*{margin-block-start:0}}.eds-c-header__item--inline-links{display:none;visibility:hidden}@media only screen and (min-width:768px){.eds-c-header__item--inline-links{display:flex;gap:16px 16px;visibility:visible}}.eds-c-header__item--divider:before{border-left:2px solid #c5e0f4;content:"";height:calc(100% - 16px);margin-left:-15px;position:absolute;top:8px}.eds-c-header__brand{padding:16px 8px}.eds-c-header__brand a{display:block;line-height:1;text-decoration:none}.eds-c-header__brand img{height:1.5rem;width:auto}.eds-c-header__link{color:inherit;display:inline-block;font-weight:700;padding:16px 8px;position:relative;text-decoration-color:transparent;white-space:nowrap;word-break:normal}.eds-c-header__icon{fill:currentcolor;display:inline-block;font-size:1.5rem;height:1em;transform:translate(0);vertical-align:bottom;width:1em}.eds-c-header__icon+*{margin-left:8px}.eds-c-header__expander{background-color:#f0f7fc}.eds-c-header__search{display:block;padding:24px 0}@media only screen and (min-width:768px){.eds-c-header__search{max-width:70%}}.eds-c-header__search-container{position:relative}.eds-c-header__search-label{color:inherit;display:inline-block;font-weight:700;margin-bottom:8px}.eds-c-header__search-input{background-color:#fff;border:1px solid #000;padding:8px 48px 8px 8px;width:100%}.eds-c-header__search-button{background-color:transparent;border:0;color:inherit;height:100%;padding:0 8px;position:absolute;right:0}.has-tethered.eds-c-header__expander{border-bottom:2px solid #01324b;left:0;margin-top:-2px;top:100%;width:100%;z-index:10}@media only screen and (min-width:768px){.has-tethered.eds-c-header__expander--menu{display:none;visibility:hidden}}.has-tethered .eds-c-header__heading{display:none;visibility:hidden}.has-tethered .eds-c-header__heading:first-child+*{margin-block-start:0}.has-tethered .eds-c-header__search{margin:auto}.eds-c-header__heading{margin:0 auto;max-width:1280px;padding:16px 16px 0}.eds-c-pagination{align-items:center;display:flex;flex-wrap:wrap;font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif;font-size:.875rem;gap:16px 0;justify-content:center;line-height:1.4;list-style:none;margin:0;padding:32px 0}@media only screen and (min-width:480px){.eds-c-pagination{padding:32px 16px}}.eds-c-pagination__item{margin-right:8px}.eds-c-pagination__item--prev{margin-right:16px}.eds-c-pagination__item--next .eds-c-pagination__link,.eds-c-pagination__item--prev .eds-c-pagination__link{padding:16px 8px}.eds-c-pagination__item--next{margin-left:8px}.eds-c-pagination__item:last-child{margin-right:0}.eds-c-pagination__link{align-items:center;color:#222;cursor:pointer;display:inline-block;font-size:1rem;margin:0;padding:16px 24px;position:relative;text-align:center;transition:all .2s ease 0s}.eds-c-pagination__link:visited{color:#222}.eds-c-pagination__link--disabled{border-color:#555;color:#555;cursor:default}.eds-c-pagination__link--active{background-color:#01324b;background-image:none;border-radius:8px;color:#fff}.eds-c-pagination__link--active:focus,.eds-c-pagination__link--active:hover,.eds-c-pagination__link--active:visited{color:#fff}.eds-c-pagination__link-container{align-items:center;display:flex}.eds-c-pagination__icon{fill:#222;height:1.5rem;width:1.5rem}.eds-c-pagination__icon--disabled{fill:#555}.eds-c-pagination__visually-hidden{clip:rect(0,0,0,0);border:0;clip-path:inset(50%);height:1px;overflow:hidden;padding:0;position:absolute!important;white-space:nowrap;width:1px}.c-breadcrumbs{color:#333;font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif;font-size:1rem;list-style:none;margin:0;padding:0}.c-breadcrumbs>li{display:inline}svg.c-breadcrumbs__chevron{fill:#333;height:10px;margin:0 .25rem;width:10px}.c-breadcrumbs--contrast,.c-breadcrumbs--contrast .c-breadcrumbs__link{color:#fff}.c-breadcrumbs--contrast svg.c-breadcrumbs__chevron{fill:#fff}@media only screen and (max-width:479px){.c-breadcrumbs .c-breadcrumbs__item{display:none}.c-breadcrumbs .c-breadcrumbs__item:last-child,.c-breadcrumbs .c-breadcrumbs__item:nth-last-child(2){display:inline}}.c-skip-link{background:#01324b;bottom:auto;color:#fff;font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif;font-size:1rem;padding:8px;position:absolute;text-align:center;transform:translateY(-100%);width:100%;z-index:9999}@media (prefers-reduced-motion:reduce){.c-skip-link{transition:top .3s ease-in-out 0s}}@media print{.c-skip-link{display:none}}.c-skip-link:active,.c-skip-link:hover,.c-skip-link:link,.c-skip-link:visited{color:#fff}.c-skip-link:focus{transform:translateY(0)}.l-with-sidebar{display:flex;flex-wrap:wrap}.l-with-sidebar>*{margin:0}.l-with-sidebar__sidebar{flex-basis:var(--with-sidebar--basis,400px);flex-grow:1}.l-with-sidebar>:not(.l-with-sidebar__sidebar){flex-basis:0px;flex-grow:999;min-width:var(--with-sidebar--min,53%)}.l-with-sidebar>:first-child{padding-right:4rem}@supports (gap:1em){.l-with-sidebar>:first-child{padding-right:0}.l-with-sidebar{gap:var(--with-sidebar--gap,4rem)}}.c-header__link{color:inherit;display:inline-block;font-weight:700;padding:16px 8px;position:relative;text-decoration-color:transparent;white-space:nowrap;word-break:normal}.app-masthead__colour-4{--background-color:#ff9500;--gradient-light:rgba(0,0,0,.5);--gradient-dark:rgba(0,0,0,.8)}.app-masthead{background:var(--background-color,#0070a8);position:relative}.app-masthead:after{background:radial-gradient(circle at top right,var(--gradient-light,rgba(0,0,0,.4)),var(--gradient-dark,rgba(0,0,0,.7)));bottom:0;content:"";left:0;position:absolute;right:0;top:0}@media only screen and (max-width:479px){.app-masthead:after{background:linear-gradient(225deg,var(--gradient-light,rgba(0,0,0,.4)),var(--gradient-dark,rgba(0,0,0,.7)))}}.app-masthead__container{color:var(--masthead-color,#fff);margin:0 auto;max-width:1280px;padding:0 16px;position:relative;z-index:1}.u-button{align-items:center;background-color:#01324b;background-image:none;border:4px solid transparent;border-radius:32px;cursor:pointer;display:inline-flex;font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif;font-size:.875rem;font-weight:700;justify-content:center;line-height:1.3;margin:0;padding:16px 32px;position:relative;transition:all .2s ease 0s;width:auto}.u-button svg,.u-button--contrast svg,.u-button--primary svg,.u-button--secondary svg,.u-button--tertiary svg{fill:currentcolor}.u-button,.u-button:visited{color:#fff}.u-button,.u-button:hover{box-shadow:0 0 0 1px #01324b;text-decoration:none}.u-button:hover{border:4px solid #fff}.u-button:focus{border:4px solid #fc0;box-shadow:none;outline:0;text-decoration:none}.u-button:focus,.u-button:hover{background-color:#fff;background-image:none;color:#01324b}.app-masthead--pastel .c-pdf-download .u-button--primary:focus svg path,.app-masthead--pastel .c-pdf-download .u-button--primary:hover svg path,.c-context-bar--sticky .c-context-bar__container .c-pdf-download .u-button--primary:focus svg path,.c-context-bar--sticky .c-context-bar__container .c-pdf-download .u-button--primary:hover svg path,.u-button--primary:focus svg path,.u-button--primary:hover svg path,.u-button:focus svg path,.u-button:hover svg path{fill:#01324b}.u-button--primary{background-color:#01324b;background-image:none;border:4px solid transparent;box-shadow:0 0 0 1px #01324b;color:#fff;font-weight:700}.u-button--primary:visited{color:#fff}.u-button--primary:hover{border:4px solid #fff;box-shadow:0 0 0 1px #01324b;text-decoration:none}.u-button--primary:focus{border:4px solid #fc0;box-shadow:none;outline:0;text-decoration:none}.u-button--primary:focus,.u-button--primary:hover{background-color:#fff;background-image:none;color:#01324b}.u-button--secondary{background-color:#fff;border:4px solid #fff;color:#01324b;font-weight:700}.u-button--secondary:visited{color:#01324b}.u-button--secondary:hover{border:4px solid #01324b;box-shadow:none}.u-button--secondary:focus,.u-button--secondary:hover{background-color:#01324b;color:#fff}.app-masthead--pastel .c-pdf-download .u-button--secondary:focus svg path,.app-masthead--pastel .c-pdf-download .u-button--secondary:hover svg path,.c-context-bar--sticky .c-context-bar__container .c-pdf-download .u-button--secondary:focus svg path,.c-context-bar--sticky .c-context-bar__container .c-pdf-download .u-button--secondary:hover svg path,.u-button--secondary:focus svg path,.u-button--secondary:hover svg path,.u-button--tertiary:focus svg path,.u-button--tertiary:hover svg path{fill:#fff}.u-button--tertiary{background-color:#ebf1f5;border:4px solid transparent;box-shadow:none;color:#666;font-weight:700}.u-button--tertiary:visited{color:#666}.u-button--tertiary:hover{border:4px solid #01324b;box-shadow:none}.u-button--tertiary:focus,.u-button--tertiary:hover{background-color:#01324b;color:#fff}.u-button--contrast{background-color:transparent;background-image:none;color:#fff;font-weight:400}.u-button--contrast:visited{color:#fff}.u-button--contrast,.u-button--contrast:focus,.u-button--contrast:hover{border:4px solid #fff}.u-button--contrast:focus,.u-button--contrast:hover{background-color:#fff;background-image:none;color:#000}.u-button--contrast:focus svg path,.u-button--contrast:hover svg path{fill:#000}.u-button--disabled,.u-button:disabled{background-color:transparent;background-image:none;border:4px solid #ccc;color:#000;cursor:default;font-weight:400;opacity:.7}.u-button--disabled svg,.u-button:disabled svg{fill:currentcolor}.u-button--disabled:visited,.u-button:disabled:visited{color:#000}.u-button--disabled:focus,.u-button--disabled:hover,.u-button:disabled:focus,.u-button:disabled:hover{border:4px solid #ccc;text-decoration:none}.u-button--disabled:focus,.u-button--disabled:hover,.u-button:disabled:focus,.u-button:disabled:hover{background-color:transparent;background-image:none;color:#000}.u-button--disabled:focus svg path,.u-button--disabled:hover svg path,.u-button:disabled:focus svg path,.u-button:disabled:hover svg path{fill:#000}.u-button--small,.u-button--xsmall{font-size:.875rem;padding:2px 8px}.u-button--small{padding:8px 16px}.u-button--large{font-size:1.125rem;padding:10px 35px}.u-button--full-width{display:flex;width:100%}.u-button--icon-left svg{margin-right:8px}.u-button--icon-right svg{margin-left:8px}.u-clear-both{clear:both}.u-container{margin:0 auto;max-width:1280px;padding:0 16px}.u-justify-content-space-between{justify-content:space-between}.u-display-none{display:none}.js .u-js-hide,.u-hide{display:none;visibility:hidden}.u-visually-hidden{clip:rect(0,0,0,0);border:0;clip-path:inset(50%);height:1px;overflow:hidden;padding:0;position:absolute!important;white-space:nowrap;width:1px}.u-icon{fill:currentcolor;display:inline-block;height:1em;transform:translate(0);vertical-align:text-top;width:1em}.u-list-reset{list-style:none;margin:0;padding:0}.u-ma-16{margin:16px}.u-mt-0{margin-top:0}.u-mt-24{margin-top:24px}.u-mt-32{margin-top:32px}.u-mb-8{margin-bottom:8px}.u-mb-32{margin-bottom:32px}.u-button-reset{background-color:transparent;border:0;padding:0}.u-sans-serif{font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif}.u-serif{font-family:Merriweather,serif}h1,h2,h4{-webkit-font-smoothing:antialiased}p{overflow-wrap:break-word;word-break:break-word}.u-h4{font-size:1.25rem;font-weight:700;line-height:1.2}.u-mbs-0{margin-block-start:0!important}.c-article-header{font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif}.c-article-identifiers{color:#6f6f6f;display:flex;flex-wrap:wrap;font-size:1rem;line-height:1.3;list-style:none;margin:0 0 8px;padding:0}.c-article-identifiers__item{border-right:1px solid #6f6f6f;list-style:none;margin-right:8px;padding-right:8px}.c-article-identifiers__item:last-child{border-right:0;margin-right:0;padding-right:0}@media only screen and (min-width:876px){.c-article-title{font-size:1.875rem;line-height:1.2}}.c-article-author-list{display:inline;font-size:1rem;list-style:none;margin:0 8px 0 0;padding:0;width:100%}.c-article-author-list__item{display:inline;padding-right:0}.c-article-author-list__show-more{display:none;margin-right:4px}.c-article-author-list__button,.js .c-article-author-list__item--hide,.js .c-article-author-list__show-more{display:none}.js .c-article-author-list--long .c-article-author-list__show-more,.js .c-article-author-list--long+.c-article-author-list__button{display:inline}@media only screen and (max-width:767px){.js .c-article-author-list__item--hide-small-screen{display:none}.js .c-article-author-list--short .c-article-author-list__show-more,.js .c-article-author-list--short+.c-article-author-list__button{display:inline}}#uptodate-client,.js .c-article-author-list--expanded .c-article-author-list__show-more{display:none!important}.js .c-article-author-list--expanded .c-article-author-list__item--hide-small-screen{display:inline!important}.c-article-author-list__button,.c-button-author-list{background:#ebf1f5;border:4px solid #ebf1f5;border-radius:20px;color:#666;font-size:.875rem;line-height:1.4;padding:2px 11px 2px 8px;text-decoration:none}.c-article-author-list__button svg,.c-button-author-list svg{margin:1px 4px 0 0}.c-article-author-list__button:hover,.c-button-author-list:hover{background:#025e8d;border-color:transparent;color:#fff}.c-article-body .c-article-access-provider{padding:8px 16px}.c-article-body .c-article-access-provider,.c-notes{border:1px solid #d5d5d5;border-image:initial;border-left:none;border-right:none;margin:24px 0}.c-article-body .c-article-access-provider__text{color:#555}.c-article-body .c-article-access-provider__text,.c-notes__text{font-size:1rem;margin-bottom:0;padding-bottom:2px;padding-top:2px;text-align:center}.c-article-body .c-article-author-affiliation__address{color:inherit;font-weight:700;margin:0}.c-article-body .c-article-author-affiliation__authors-list{list-style:none;margin:0;padding:0}.c-article-body .c-article-author-affiliation__authors-item{display:inline;margin-left:0}.c-article-authors-search{margin-bottom:24px;margin-top:0}.c-article-authors-search__item,.c-article-authors-search__title{font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif}.c-article-authors-search__title{color:#626262;font-size:1.05rem;font-weight:700;margin:0;padding:0}.c-article-authors-search__item{font-size:1rem}.c-article-authors-search__text{margin:0}.c-code-block{border:1px solid #fff;font-family:monospace;margin:0 0 24px;padding:20px}.c-code-block__heading{font-weight:400;margin-bottom:16px}.c-code-block__line{display:block;overflow-wrap:break-word;white-space:pre-wrap}.c-article-share-box{font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif;margin-bottom:24px}.c-article-share-box__description{font-size:1rem;margin-bottom:8px}.c-article-share-box__no-sharelink-info{font-size:.813rem;font-weight:700;margin-bottom:24px;padding-top:4px}.c-article-share-box__only-read-input{border:1px solid #d5d5d5;box-sizing:content-box;display:inline-block;font-size:.875rem;font-weight:700;height:24px;margin-bottom:8px;padding:8px 10px}.c-article-share-box__additional-info{color:#626262;font-size:.813rem}.c-article-share-box__button{background:#fff;box-sizing:content-box;text-align:center}.c-article-share-box__button--link-like{background-color:transparent;border:0;color:#025e8d;cursor:pointer;font-size:.875rem;margin-bottom:8px;margin-left:10px}.c-article-associated-content__container .c-article-associated-content__collection-label{font-size:.875rem;line-height:1.4}.c-article-associated-content__container .c-article-associated-content__collection-title{line-height:1.3}.c-reading-companion{clear:both;min-height:389px}.c-reading-companion__figures-list,.c-reading-companion__references-list{list-style:none;min-height:389px;padding:0}.c-reading-companion__references-list--numeric{list-style:decimal inside}.c-reading-companion__figure-item{border-top:1px solid #d5d5d5;font-size:1rem;padding:16px 8px 16px 0}.c-reading-companion__figure-item:first-child{border-top:none;padding-top:8px}.c-reading-companion__reference-item{font-size:1rem}.c-reading-companion__reference-item:first-child{border-top:none}.c-reading-companion__reference-item a{word-break:break-word}.c-reading-companion__reference-citation{display:inline}.c-reading-companion__reference-links{font-size:.813rem;font-weight:700;list-style:none;margin:8px 0 0;padding:0;text-align:right}.c-reading-companion__reference-links>a{display:inline-block;padding-left:8px}.c-reading-companion__reference-links>a:first-child{display:inline-block;padding-left:0}.c-reading-companion__figure-title{display:block;font-size:1.25rem;font-weight:700;line-height:1.2;margin:0 0 8px}.c-reading-companion__figure-links{display:flex;justify-content:space-between;margin:8px 0 0}.c-reading-companion__figure-links>a{align-items:center;display:flex}.c-article-section__figure-caption{display:block;margin-bottom:8px;word-break:break-word}.c-article-section__figure .video,p.app-article-masthead__access--above-download{margin:0 0 16px}.c-article-section__figure-description{font-size:1rem}.c-article-section__figure-description>*{margin-bottom:0}.c-cod{display:block;font-size:1rem;width:100%}.c-cod__form{background:#ebf0f3}.c-cod__prompt{font-size:1.125rem;line-height:1.3;margin:0 0 24px}.c-cod__label{display:block;margin:0 0 4px}.c-cod__row{display:flex;margin:0 0 16px}.c-cod__row:last-child{margin:0}.c-cod__input{border:1px solid #d5d5d5;border-radius:2px;flex-shrink:0;margin:0;padding:13px}.c-cod__input--submit{background-color:#025e8d;border:1px solid #025e8d;color:#fff;flex-shrink:1;margin-left:8px;transition:background-color .2s ease-out 0s,color .2s ease-out 0s}.c-cod__input--submit-single{flex-basis:100%;flex-shrink:0;margin:0}.c-cod__input--submit:focus,.c-cod__input--submit:hover{background-color:#fff;color:#025e8d}.save-data .c-article-author-institutional-author__sub-division,.save-data .c-article-equation__number,.save-data .c-article-figure-description,.save-data .c-article-fullwidth-content,.save-data .c-article-main-column,.save-data .c-article-satellite-article-link,.save-data .c-article-satellite-subtitle,.save-data .c-article-table-container,.save-data .c-blockquote__body,.save-data .c-code-block__heading,.save-data .c-reading-companion__figure-title,.save-data .c-reading-companion__reference-citation,.save-data .c-site-messages--nature-briefing-email-variant .serif,.save-data .c-site-messages--nature-briefing-email-variant.serif,.save-data .serif,.save-data .u-serif,.save-data h1,.save-data h2,.save-data h3{font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif}.c-pdf-download__link{display:flex;flex:1 1 0%;padding:13px 24px}.c-pdf-download__link:hover{text-decoration:none}@media only screen and (min-width:768px){.c-context-bar--sticky .c-pdf-download__link{align-items:center;flex:1 1 183px}}@media only screen and (max-width:320px){.c-context-bar--sticky .c-pdf-download__link{padding:16px}}.c-article-body .c-article-recommendations-list,.c-book-body .c-article-recommendations-list{display:flex;flex-direction:row;gap:16px 16px;margin:0;max-width:100%;padding:16px 0 0}.c-article-body .c-article-recommendations-list__item,.c-book-body .c-article-recommendations-list__item{flex:1 1 0%}@media only screen and (max-width:767px){.c-article-body .c-article-recommendations-list,.c-book-body .c-article-recommendations-list{flex-direction:column}}.c-article-body .c-article-recommendations-card__authors{display:none;font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif;font-size:.875rem;line-height:1.5;margin:0 0 8px}@media only screen and (max-width:767px){.c-article-body .c-article-recommendations-card__authors{display:block;margin:0}}.c-article-body .c-article-history{margin-top:24px}.app-article-metrics-bar p{margin:0}.app-article-masthead{display:flex;flex-direction:column;gap:16px 16px;padding:16px 0 24px}.app-article-masthead__info{display:flex;flex-direction:column;flex-grow:1}.app-article-masthead__brand{border-top:1px solid hsla(0,0%,100%,.8);display:flex;flex-direction:column;flex-shrink:0;gap:8px 8px;min-height:96px;padding:16px 0 0}.app-article-masthead__brand img{border:1px solid #fff;border-radius:8px;box-shadow:0 4px 15px 0 hsla(0,0%,50%,.25);height:auto;left:0;position:absolute;width:72px}.app-article-masthead__journal-link{display:block;font-size:1.125rem;font-weight:700;margin:0 0 8px;max-width:400px;padding:0 0 0 88px;position:relative}.app-article-masthead__journal-title{-webkit-box-orient:vertical;-webkit-line-clamp:3;display:-webkit-box;overflow:hidden}.app-article-masthead__submission-link{align-items:center;display:flex;font-size:1rem;gap:4px 4px;margin:0 0 0 88px}.app-article-masthead__access{align-items:center;display:flex;flex-wrap:wrap;font-size:.875rem;font-weight:300;gap:4px 4px;margin:0}.app-article-masthead__buttons{display:flex;flex-flow:column wrap;gap:16px 16px}.app-article-masthead__access svg,.app-masthead--pastel .c-pdf-download .u-button--primary svg,.app-masthead--pastel .c-pdf-download .u-button--secondary svg,.c-context-bar--sticky .c-context-bar__container .c-pdf-download .u-button--primary svg,.c-context-bar--sticky .c-context-bar__container .c-pdf-download .u-button--secondary svg{fill:currentcolor}.app-article-masthead a{color:#fff}.app-masthead--pastel .c-pdf-download .u-button--primary,.c-context-bar--sticky .c-context-bar__container .c-pdf-download .u-button--primary{background-color:#025e8d;background-image:none;border:2px solid transparent;box-shadow:none;color:#fff;font-weight:700}.app-masthead--pastel .c-pdf-download .u-button--primary:visited,.c-context-bar--sticky .c-context-bar__container .c-pdf-download .u-button--primary:visited{color:#fff}.app-masthead--pastel .c-pdf-download .u-button--primary:hover,.c-context-bar--sticky .c-context-bar__container .c-pdf-download .u-button--primary:hover{text-decoration:none}.app-masthead--pastel .c-pdf-download .u-button--primary:focus,.c-context-bar--sticky .c-context-bar__container .c-pdf-download .u-button--primary:focus{border:4px solid #fc0;box-shadow:none;outline:0;text-decoration:none}.app-masthead--pastel .c-pdf-download .u-button--primary:focus,.app-masthead--pastel .c-pdf-download .u-button--primary:hover,.c-context-bar--sticky .c-context-bar__container .c-pdf-download .u-button--primary:focus,.c-context-bar--sticky .c-context-bar__container .c-pdf-download .u-button--primary:hover{background-color:#fff;background-image:none;color:#01324b}.app-masthead--pastel .c-pdf-download .u-button--primary:hover,.c-context-bar--sticky .c-context-bar__container .c-pdf-download .u-button--primary:hover{background:0 0;border:2px solid #025e8d;box-shadow:none;color:#025e8d}.app-masthead--pastel .c-pdf-download .u-button--secondary,.c-context-bar--sticky .c-context-bar__container .c-pdf-download .u-button--secondary{background:0 0;border:2px solid #025e8d;color:#025e8d;font-weight:700}.app-masthead--pastel .c-pdf-download .u-button--secondary:visited,.c-context-bar--sticky .c-context-bar__container .c-pdf-download .u-button--secondary:visited{color:#01324b}.app-masthead--pastel .c-pdf-download .u-button--secondary:hover,.c-context-bar--sticky .c-context-bar__container .c-pdf-download .u-button--secondary:hover{background-color:#01324b;background-color:#025e8d;border:2px solid transparent;box-shadow:none;color:#fff}.app-masthead--pastel .c-pdf-download .u-button--secondary:focus,.c-context-bar--sticky .c-context-bar__container .c-pdf-download .u-button--secondary:focus{background-color:#fff;background-image:none;border:4px solid #fc0;color:#01324b}@media only screen and (min-width:768px){.app-article-masthead{flex-direction:row;gap:64px 64px;padding:24px 0}.app-article-masthead__brand{border:0;padding:0}.app-article-masthead__brand img{height:auto;position:static;width:auto}.app-article-masthead__buttons{align-items:center;flex-direction:row;margin-top:auto}.app-article-masthead__journal-link{display:flex;flex-direction:column;gap:24px 24px;margin:0 0 8px;padding:0}.app-article-masthead__submission-link{margin:0}}@media only screen and (min-width:1024px){.app-article-masthead__brand{flex-basis:400px}}.app-article-masthead .c-article-identifiers{font-size:.875rem;font-weight:300;line-height:1;margin:0 0 8px;overflow:hidden;padding:0}.app-article-masthead .c-article-identifiers--cite-list{margin:0 0 16px}.app-article-masthead .c-article-identifiers *{color:#fff}.app-article-masthead .c-cod{display:none}.app-article-masthead .c-article-identifiers__item{border-left:1px solid #fff;border-right:0;margin:0 17px 8px -9px;padding:0 0 0 8px}.app-article-masthead .c-article-identifiers__item--cite{border-left:0}.app-article-metrics-bar{display:flex;flex-wrap:wrap;font-size:1rem;padding:16px 0 0;row-gap:24px}.app-article-metrics-bar__item{padding:0 16px 0 0}.app-article-metrics-bar__count{font-weight:700}.app-article-metrics-bar__label{font-weight:400;padding-left:4px}.app-article-metrics-bar__icon{height:auto;margin-right:4px;margin-top:-4px;width:auto}.app-article-metrics-bar__arrow-icon{margin:4px 0 0 4px}.app-article-metrics-bar a{color:#000}.app-article-metrics-bar .app-article-metrics-bar__item--metrics{padding-right:0}.app-overview-section .c-article-author-list,.app-overview-section__authors{line-height:2}.app-article-metrics-bar{margin-top:8px}.c-book-toc-pagination+.c-book-section__back-to-top{margin-top:0}.c-article-body .c-article-access-provider__text--chapter{color:#222;font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif;padding:20px 0}.c-article-body .c-article-access-provider__text--chapter svg.c-status-message__icon{fill:#003f8d;vertical-align:middle}.c-article-body-section__content--separator{padding-top:40px}.c-pdf-download__link{max-height:44px}.app-article-access .u-button--primary,.app-article-access .u-button--primary:visited{color:#fff}.c-article-sidebar{display:none}@media only screen and (min-width:1024px){.c-article-sidebar{display:block}}.c-cod__form{border-radius:12px}.c-cod__label{font-size:.875rem}.c-cod .c-status-message{align-items:center;justify-content:center;margin-bottom:16px;padding-bottom:16px}@media only screen and (min-width:1024px){.c-cod .c-status-message{align-items:inherit}}.c-cod .c-status-message__icon{margin-top:4px}.c-cod .c-cod__prompt{font-size:1rem;margin-bottom:16px}.c-article-body .app-article-access,.c-book-body .app-article-access{display:block}@media only screen and (min-width:1024px){.c-article-body .app-article-access,.c-book-body .app-article-access{display:none}}.c-article-body .app-card-service{margin-bottom:32px}@media only screen and (min-width:1024px){.c-article-body .app-card-service{display:none}}.app-article-access .buybox__buy .u-button--secondary,.app-article-access .u-button--primary,.c-cod__row .u-button--primary{background-color:#025e8d;border:2px solid #025e8d;box-shadow:none;font-size:1rem;font-weight:700;gap:8px 8px;justify-content:center;line-height:1.5;padding:8px 24px}.app-article-access .buybox__buy .u-button--secondary,.app-article-access .u-button--primary:hover,.c-cod__row .u-button--primary:hover{background-color:#fff;color:#025e8d}.app-article-access .buybox__buy .u-button--secondary:hover{background-color:#025e8d;color:#fff}.buybox__buy .c-notes__text{color:#666;font-size:.875rem;padding:0 16px 8px}.c-cod__input{flex-basis:auto;width:100%}.c-article-title{font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif;font-size:2.25rem;font-weight:700;line-height:1.2;margin:12px 0}.c-reading-companion__figure-item figure{margin:0}@media only screen and (min-width:768px){.c-article-title{margin:16px 0}}.app-article-access{border:1px solid #c5e0f4;border-radius:12px}.app-article-access__heading{border-bottom:1px solid #c5e0f4;font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif;font-size:1.125rem;font-weight:700;margin:0;padding:16px;text-align:center}.app-article-access .buybox__info svg{vertical-align:middle}.c-article-body .app-article-access p{margin-bottom:0}.app-article-access .buybox__info{font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif;font-size:1rem;margin:0}.app-article-access{margin:0 0 32px}@media only screen and (min-width:1024px){.app-article-access{margin:0 0 24px}}.c-status-message{font-size:1rem}.c-article-body{font-size:1.125rem}.c-article-body dl,.c-article-body ol,.c-article-body p,.c-article-body ul{margin-bottom:32px;margin-top:0}.c-article-access-provider__text:last-of-type,.c-article-body .c-notes__text:last-of-type{margin-bottom:0}.c-article-body ol p,.c-article-body ul p{margin-bottom:16px}.c-article-section__figure-caption{font-family:Merriweather Sans,Helvetica Neue,Helvetica,Arial,sans-serif}.c-reading-companion__figure-item{border-top-color:#c5e0f4}.c-reading-companion__sticky{max-width:400px}.c-article-section .c-article-section__figure-description>*{font-size:1rem;margin-bottom:16px}.c-reading-companion__reference-item{border-top:1px solid #d5d5d5;padding:16px 0}.c-reading-companion__reference-item:first-child{padding-top:0}.c-article-share-box__button,.js .c-article-authors-search__item .c-article-button{background:0 0;border:2px solid #025e8d;border-radius:32px;box-shadow:none;color:#025e8d;font-size:1rem;font-weight:700;line-height:1.5;margin:0;padding:8px 24px;transition:all .2s ease 0s}.c-article-authors-search__item .c-article-button{width:100%}.c-pdf-download .u-button{background-color:#fff;border:2px solid #fff;color:#01324b;justify-content:center}.c-context-bar__container .c-pdf-download .u-button svg,.c-pdf-download .u-button svg{fill:currentcolor}.c-pdf-download .u-button:visited{color:#01324b}.c-pdf-download .u-button:hover{border:4px solid #01324b;box-shadow:none}.c-pdf-download .u-button:focus,.c-pdf-download .u-button:hover{background-color:#01324b}.c-pdf-download .u-button:focus svg path,.c-pdf-download .u-button:hover svg path{fill:#fff}.c-context-bar__container .c-pdf-download .u-button{background-image:none;border:2px solid;color:#fff}.c-context-bar__container .c-pdf-download .u-button:visited{color:#fff}.c-context-bar__container .c-pdf-download .u-button:hover{text-decoration:none}.c-context-bar__container .c-pdf-download .u-button:focus{box-shadow:none;outline:0;text-decoration:none}.c-context-bar__container .c-pdf-download .u-button:focus,.c-context-bar__container .c-pdf-download .u-button:hover{background-color:#fff;background-image:none;color:#01324b}.c-context-bar__container .c-pdf-download .u-button:focus svg path,.c-context-bar__container .c-pdf-download .u-button:hover svg path{fill:#01324b}.c-context-bar__container .c-pdf-download .u-button,.c-pdf-download .u-button{box-shadow:none;font-size:1rem;font-weight:700;line-height:1.5;padding:8px 24px}.c-context-bar__container .c-pdf-download .u-button{background-color:#025e8d}.c-pdf-download .u-button:hover{border:2px solid #fff}.c-pdf-download .u-button:focus,.c-pdf-download .u-button:hover{background:0 0;box-shadow:none;color:#fff}.c-context-bar__container .c-pdf-download .u-button:hover{border:2px solid #025e8d;box-shadow:none;color:#025e8d}.c-context-bar__container .c-pdf-download .u-button:focus,.c-pdf-download .u-button:focus{border:2px solid #025e8d}.c-article-share-box__button:focus:focus,.c-article__pill-button:focus:focus,.c-context-bar__container .c-pdf-download .u-button:focus:focus,.c-pdf-download .u-button:focus:focus{outline:3px solid #08c;will-change:transform}.c-pdf-download__link .u-icon{padding-top:0}.c-bibliographic-information__column button{margin-bottom:16px}.c-article-body .c-article-author-affiliation__list p,.c-article-body .c-article-author-information__list p,figure{margin:0}.c-article-share-box__button{margin-right:16px}.c-status-message--boxed{border-radius:12px}.c-article-associated-content__collection-title{font-size:1rem}.app-card-service__description,.c-article-body .app-card-service__description{color:#222;margin-bottom:0;margin-top:8px}.app-article-access__subscriptions a,.app-article-access__subscriptions a:visited,.app-book-series-listing__item a,.app-book-series-listing__item a:hover,.app-book-series-listing__item a:visited,.c-article-author-list a,.c-article-author-list a:visited,.c-article-buy-box a,.c-article-buy-box a:visited,.c-article-peer-review a,.c-article-peer-review a:visited,.c-article-satellite-subtitle a,.c-article-satellite-subtitle a:visited,.c-breadcrumbs__link,.c-breadcrumbs__link:hover,.c-breadcrumbs__link:visited{color:#000}.c-article-author-list svg{height:24px;margin:0 0 0 6px;width:24px}.c-article-header{margin-bottom:32px}@media only screen and (min-width:876px){.js .c-ad--conditional{display:block}}.u-lazy-ad-wrapper{background-color:#fff;display:none;min-height:149px}@media only screen and (min-width:876px){.u-lazy-ad-wrapper{display:block}}p.c-ad__label{margin-bottom:4px}.c-ad--728x90{background-color:#fff;border-bottom:2px solid #cedbe0} } </style> <style>@media only print, only all and (prefers-color-scheme: no-preference), only all and (prefers-color-scheme: light), only all and (prefers-color-scheme: dark) { .eds-c-header__brand img{height:24px;width:203px}.app-article-masthead__journal-link img{height:93px;width:72px}@media only screen and (min-width:769px){.app-article-masthead__journal-link img{height:161px;width:122px}} } </style> <link rel="stylesheet" data-test="critical-css-handler" data-inline-css-source="critical-css" href=/oscar-static/app-springerlink/css/core-darwin-3c86549cfc.css media="print" onload="this.media='all';this.onload=null"> <link rel="stylesheet" data-test="critical-css-handler" data-inline-css-source="critical-css" href="/oscar-static/app-springerlink/css/enhanced-darwin-article-72ba046d97.css" media="print" onload="this.media='only print, only all and (prefers-color-scheme: no-preference), only all and (prefers-color-scheme: light), only all and (prefers-color-scheme: dark)';this.onload=null"> <script type="text/javascript"> config = { env: 'live', site: '439.springer.com', siteWithPath: '439.springer.com' + window.location.pathname, twitterHashtag: '439', cmsPrefix: 'https://studio-cms.springernature.com/studio/', publisherBrand: 'Springer', mustardcut: false }; </script> <script> window.dataLayer = [{"GA Key":"UA-26408784-1","DOI":"10.1007/s00439-021-02393-x","Page":"article","springerJournal":true,"Publishing Model":"Hybrid Access","page":{"attributes":{"environment":"live"}},"Country":"HK","japan":false,"doi":"10.1007-s00439-021-02393-x","Journal Id":439,"Journal Title":"Human Genetics","imprint":"Springer","Keywords":"","kwrd":[],"Labs":"Y","ksg":"Krux.segments","kuid":"Krux.uid","Has Body":"Y","Features":[],"Open Access":"Y","hasAccess":"Y","bypassPaywall":"N","user":{"license":{"businessPartnerID":[],"businessPartnerIDString":""}},"Access Type":"open","Bpids":"","Bpnames":"","BPID":["1"],"VG Wort Identifier":"vgzm.415900-10.1007-s00439-021-02393-x","Full HTML":"Y","Subject Codes":["SCB","SCB12008","SCB1700X","SCB12030","SCH33118"],"pmc":["B","B12008","B1700X","B12030","H33118"],"session":{"authentication":{"loginStatus":"N"},"attributes":{"edition":"academic"}},"content":{"serial":{"eissn":"1432-1203","pissn":"0340-6717"},"type":"Article","category":{"pmc":{"primarySubject":"Biomedicine","primarySubjectCode":"B","secondarySubjects":{"1":"Human Genetics","2":"Molecular Medicine","3":"Gene Function","4":"Metabolic Diseases"},"secondarySubjectCodes":{"1":"B12008","2":"B1700X","3":"B12030","4":"H33118"}},"sucode":"SC3","articleType":"Review"},"attributes":{"deliveryPlatform":"oscar"}},"Event Category":"Article"}]; </script> <script data-test="springer-link-article-datalayer"> window.dataLayer = window.dataLayer || []; window.dataLayer.push({ ga4MeasurementId: 'G-B3E4QL2TPR', ga360TrackingId: 'UA-26408784-1', twitterId: 'o47a7', baiduId: 'aef3043f025ccf2305af8a194652d70b', ga4ServerUrl: 'https://collect.springer.com', imprint: 'springerlink', page: { attributes:{ featureFlags: [{ name: 'darwin-orion', active: true }, { name: 'chapter-books-recs', active: true } ], darwinAvailable: true } } }); </script> <script> (function(w, d) { w.config = w.config || {}; w.config.mustardcut = false; if (w.matchMedia && w.matchMedia('only print, only all and (prefers-color-scheme: no-preference), only all and (prefers-color-scheme: light), only all and (prefers-color-scheme: dark)').matches) { w.config.mustardcut = true; d.classList.add('js'); d.classList.remove('grade-c'); d.classList.remove('no-js'); } })(window, document.documentElement); </script> <script class="js-entry"> if (window.config.mustardcut) { (function(w, d) { window.Component = {}; window.suppressShareButton = false; window.onArticlePage = true; var currentScript = d.currentScript || d.head.querySelector('script.js-entry'); function catchNoModuleSupport() { var scriptEl = d.createElement('script'); return (!('noModule' in scriptEl) && 'onbeforeload' in scriptEl) } var headScripts = [ {'src': '/oscar-static/js/polyfill-es5-bundle-572d4fec60.js', 'async': false} ]; var bodyScripts = [ {'src': '/oscar-static/js/global-article-es5-bundle-dad1690b0d.js', 'async': false, 'module': false}, {'src': '/oscar-static/js/global-article-es6-bundle-e7d03c4cb3.js', 'async': false, 'module': true} ]; function createScript(script) { var scriptEl = d.createElement('script'); scriptEl.src = script.src; scriptEl.async = script.async; if (script.module === true) { scriptEl.type = "module"; if (catchNoModuleSupport()) { scriptEl.src = ''; } } else if (script.module === false) { scriptEl.setAttribute('nomodule', true) } if (script.charset) { scriptEl.setAttribute('charset', script.charset); } return scriptEl; } for (var i = 0; i < headScripts.length; ++i) { var scriptEl = createScript(headScripts[i]); currentScript.parentNode.insertBefore(scriptEl, currentScript.nextSibling); } d.addEventListener('DOMContentLoaded', function() { for (var i = 0; i < bodyScripts.length; ++i) { var scriptEl = createScript(bodyScripts[i]); d.body.appendChild(scriptEl); } }); // Webfont repeat view var config = w.config; if (config && config.publisherBrand && sessionStorage.fontsLoaded === 'true') { d.documentElement.className += ' webfonts-loaded'; } })(window, document); } </script> <script data-src="https://cdn.optimizely.com/js/27195530232.js" data-cc-script="C03"></script> <script data-test="gtm-head"> window.initGTM = function() { if (window.config.mustardcut) { (function (w, d, s, l, i) { w[l] = w[l] || []; w[l].push({'gtm.start': new Date().getTime(), event: 'gtm.js'}); var f = d.getElementsByTagName(s)[0], j = d.createElement(s), dl = l != 'dataLayer' ? '&l=' + l : ''; j.async = true; j.src = 'https://www.googletagmanager.com/gtm.js?id=' + i + dl; f.parentNode.insertBefore(j, f); })(window, document, 'script', 'dataLayer', 'GTM-MRVXSHQ'); } } </script> <script> (function (w, d, t) { function cc() { var h = w.location.hostname; var e = d.createElement(t), s = d.getElementsByTagName(t)[0]; if (h.indexOf('springer.com') > -1 && h.indexOf('biomedcentral.com') === -1 && h.indexOf('springeropen.com') === -1) { if (h.indexOf('link-qa.springer.com') > -1 || h.indexOf('test-www.springer.com') > -1) { e.src = 'https://cmp.springer.com/production_live/en/consent-bundle-17-52.js'; e.setAttribute('onload', "initGTM(window,document,'script','dataLayer','GTM-MRVXSHQ')"); } else { e.src = 'https://cmp.springer.com/production_live/en/consent-bundle-17-52.js'; e.setAttribute('onload', "initGTM(window,document,'script','dataLayer','GTM-MRVXSHQ')"); } } else if (h.indexOf('biomedcentral.com') > -1) { if (h.indexOf('biomedcentral.com.qa') > -1) { e.src = 'https://cmp.biomedcentral.com/production_live/en/consent-bundle-15-36.js'; e.setAttribute('onload', "initGTM(window,document,'script','dataLayer','GTM-MRVXSHQ')"); } else { e.src = 'https://cmp.biomedcentral.com/production_live/en/consent-bundle-15-36.js'; e.setAttribute('onload', "initGTM(window,document,'script','dataLayer','GTM-MRVXSHQ')"); } } else if (h.indexOf('springeropen.com') > -1) { if (h.indexOf('springeropen.com.qa') > -1) { e.src = 'https://cmp.springernature.com/production_live/en/consent-bundle-16-34.js'; e.setAttribute('onload', "initGTM(window,document,'script','dataLayer','GTM-MRVXSHQ')"); } else { e.src = 'https://cmp.springernature.com/production_live/en/consent-bundle-16-34.js'; e.setAttribute('onload', "initGTM(window,document,'script','dataLayer','GTM-MRVXSHQ')"); } } else if (h.indexOf('springernature.com') > -1) { if (h.indexOf('beta-qa.springernature.com') > -1) { e.src = 'https://cmp.springernature.com/production_live/en/consent-bundle-49-43.js'; e.setAttribute('onload', "initGTM(window,document,'script','dataLayer','GTM-NK22KLS')"); } else { e.src = 'https://cmp.springernature.com/production_live/en/consent-bundle-49-43.js'; e.setAttribute('onload', "initGTM(window,document,'script','dataLayer','GTM-NK22KLS')"); } } else { e.src = '/oscar-static/js/cookie-consent-es5-bundle-cb57c2c98a.js'; e.setAttribute('data-consent', h); } s.insertAdjacentElement('afterend', e); } cc(); })(window, document, 'script'); </script> <link rel="canonical" href="https://link.springer.com/article/10.1007/s00439-021-02393-x"/> <script type="application/ld+json">{"mainEntity":{"headline":"The promise of automated machine learning for the genetic analysis of complex traits","description":"The genetic analysis of complex traits has been dominated by parametric statistical methods due to their theoretical properties, ease of use, computational efficiency, and intuitive interpretation. However, there are likely to be patterns arising from complex genetic architectures which are more easily detected and modeled using machine learning methods. Unfortunately, selecting the right machine learning algorithm and tuning its hyperparameters can be daunting for experts and non-experts alike. The goal of automated machine learning (AutoML) is to let a computer algorithm identify the right algorithms and hyperparameters thus taking the guesswork out of the optimization process. We review the promises and challenges of AutoML for the genetic analysis of complex traits and give an overview of several approaches and some example applications to omics data. It is our hope that this review will motivate studies to develop and evaluate novel AutoML methods and software in the genetics and genomics space. The promise of AutoML is to enable anyone, regardless of training or expertise, to apply machine learning as part of their genetic analysis strategy.","datePublished":"2021-10-28T00:00:00Z","dateModified":"2021-10-28T00:00:00Z","pageStart":"1529","pageEnd":"1544","license":"http://creativecommons.org/licenses/by/4.0/","sameAs":"https://doi.org/10.1007/s00439-021-02393-x","keywords":["Human Genetics","Molecular Medicine","Gene Function","Metabolic Diseases"],"image":["https://media.springernature.com/lw1200/springer-static/image/art%3A10.1007%2Fs00439-021-02393-x/MediaObjects/439_2021_2393_Fig1_HTML.png","https://media.springernature.com/lw1200/springer-static/image/art%3A10.1007%2Fs00439-021-02393-x/MediaObjects/439_2021_2393_Fig2_HTML.png","https://media.springernature.com/lw1200/springer-static/image/art%3A10.1007%2Fs00439-021-02393-x/MediaObjects/439_2021_2393_Fig3_HTML.png","https://media.springernature.com/lw1200/springer-static/image/art%3A10.1007%2Fs00439-021-02393-x/MediaObjects/439_2021_2393_Fig4_HTML.png"],"isPartOf":{"name":"Human Genetics","issn":["1432-1203","0340-6717"],"volumeNumber":"141","@type":["Periodical","PublicationVolume"]},"publisher":{"name":"Springer Berlin Heidelberg","logo":{"url":"https://www.springernature.com/app-sn/public/images/logo-springernature.png","@type":"ImageObject"},"@type":"Organization"},"author":[{"name":"Elisabetta Manduchi","url":"http://orcid.org/0000-0002-4110-3714","affiliation":[{"name":"University of Pennsylvania","address":{"name":"Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, USA","@type":"PostalAddress"},"@type":"Organization"},{"name":"University of Pennsylvania","address":{"name":"Institute for Biomedical Informatics, University of Pennsylvania, Philadelphia, USA","@type":"PostalAddress"},"@type":"Organization"}],"@type":"Person"},{"name":"Joseph D. Romano","url":"http://orcid.org/0000-0002-7999-4399","affiliation":[{"name":"University of Pennsylvania","address":{"name":"Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, USA","@type":"PostalAddress"},"@type":"Organization"}],"@type":"Person"},{"name":"Jason H. Moore","url":"http://orcid.org/0000-0002-5015-1099","affiliation":[{"name":"University of Pennsylvania","address":{"name":"Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, USA","@type":"PostalAddress"},"@type":"Organization"},{"name":"University of Pennsylvania","address":{"name":"Institute for Biomedical Informatics, University of Pennsylvania, Philadelphia, USA","@type":"PostalAddress"},"@type":"Organization"}],"email":"jhmoore@upenn.edu","@type":"Person"}],"isAccessibleForFree":true,"@type":"ScholarlyArticle"},"@context":"https://schema.org","@type":"WebPage"}</script> </head> <body class="" >  <noscript> <iframe src="https://www.googletagmanager.com/ns.html?id=GTM-MRVXSHQ" height="0" width="0" style="display:none;visibility:hidden"></iframe> </noscript>   <noscript data-test="gtm-body"> <iframe src="https://www.googletagmanager.com/ns.html?id=GTM-MRVXSHQ" height="0" width="0" style="display:none;visibility:hidden"></iframe> </noscript>  <div class="u-visually-hidden" aria-hidden="true" data-test="darwin-icons"> <?xml version="1.0" encoding="UTF-8"?><!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><symbol id="icon-eds-i-accesses-medium" viewBox="0 0 24 24"><path d="M15.59 1a1 1 0 0 1 .706.291l5.41 5.385a1 1 0 0 1 .294.709v13.077c0 .674-.269 1.32-.747 1.796a2.549 2.549 0 0 1-1.798.742H15a1 1 0 0 1 0-2h4.455a.549.549 0 0 0 .387-.16.535.535 0 0 0 .158-.378V7.8L15.178 3H5.545a.543.543 0 0 0-.538.451L5 3.538v8.607a1 1 0 0 1-2 0V3.538A2.542 2.542 0 0 1 5.545 1h10.046ZM8 13c2.052 0 4.66 1.61 6.36 3.4l.124.141c.333.41.516.925.516 1.459 0 .6-.232 1.178-.64 1.599C12.666 21.388 10.054 23 8 23c-2.052 0-4.66-1.61-6.353-3.393A2.31 2.31 0 0 1 1 18c0-.6.232-1.178.64-1.6C3.34 14.61 5.948 13 8 13Zm0 2c-1.369 0-3.552 1.348-4.917 2.785A.31.31 0 0 0 3 18c0 .083.031.161.09.222C4.447 19.652 6.631 21 8 21c1.37 0 3.556-1.35 4.917-2.785A.31.31 0 0 0 13 18a.32.32 0 0 0-.048-.17l-.042-.052C11.553 16.348 9.369 15 8 15Zm0 1a2 2 0 1 1 0 4 2 2 0 0 1 0-4Z"/></symbol><symbol id="icon-eds-i-altmetric-medium" viewBox="0 0 24 24"><path d="M12 1c5.978 0 10.843 4.77 10.996 10.712l.004.306-.002.022-.002.248C22.843 18.23 17.978 23 12 23 5.925 23 1 18.075 1 12S5.925 1 12 1Zm-1.726 9.246L8.848 12.53a1 1 0 0 1-.718.461L8.003 13l-4.947.014a9.001 9.001 0 0 0 17.887-.001L16.553 13l-2.205 3.53a1 1 0 0 1-1.735-.068l-.05-.11-2.289-6.106ZM12 3a9.001 9.001 0 0 0-8.947 8.013l4.391-.012L9.652 7.47a1 1 0 0 1 1.784.179l2.288 6.104 1.428-2.283a1 1 0 0 1 .722-.462l.129-.008 4.943.012A9.001 9.001 0 0 0 12 3Z"/></symbol><symbol id="icon-eds-i-arrow-bend-down-medium" viewBox="0 0 24 24"><path d="m11.852 20.989.058.007L12 21l.075-.003.126-.017.111-.03.111-.044.098-.052.104-.074.082-.073 6-6a1 1 0 0 0-1.414-1.414L13 17.585v-12.2C13 4.075 11.964 3 10.667 3H4a1 1 0 1 0 0 2h6.667c.175 0 .333.164.333.385v12.2l-4.293-4.292a1 1 0 0 0-1.32-.083l-.094.083a1 1 0 0 0 0 1.414l6 6c.035.036.073.068.112.097l.11.071.114.054.105.035.118.025Z"/></symbol><symbol id="icon-eds-i-arrow-bend-down-small" viewBox="0 0 16 16"><path d="M1 2a1 1 0 0 0 1 1h5v8.585L3.707 8.293a1 1 0 0 0-1.32-.083l-.094.083a1 1 0 0 0 0 1.414l5 5 .063.059.093.069.081.048.105.048.104.035.105.022.096.01h.136l.122-.018.113-.03.103-.04.1-.053.102-.07.052-.043 5.04-5.037a1 1 0 1 0-1.415-1.414L9 11.583V3a2 2 0 0 0-2-2H2a1 1 0 0 0-1 1Z"/></symbol><symbol id="icon-eds-i-arrow-bend-up-medium" viewBox="0 0 24 24"><path d="m11.852 3.011.058-.007L12 3l.075.003.126.017.111.03.111.044.098.052.104.074.082.073 6 6a1 1 0 1 1-1.414 1.414L13 6.415v12.2C13 19.925 11.964 21 10.667 21H4a1 1 0 0 1 0-2h6.667c.175 0 .333-.164.333-.385v-12.2l-4.293 4.292a1 1 0 0 1-1.32.083l-.094-.083a1 1 0 0 1 0-1.414l6-6c.035-.036.073-.068.112-.097l.11-.071.114-.054.105-.035.118-.025Z"/></symbol><symbol id="icon-eds-i-arrow-bend-up-small" viewBox="0 0 16 16"><path d="M1 13.998a1 1 0 0 1 1-1h5V4.413L3.707 7.705a1 1 0 0 1-1.32.084l-.094-.084a1 1 0 0 1 0-1.414l5-5 .063-.059.093-.068.081-.05.105-.047.104-.035.105-.022L7.94 1l.136.001.122.017.113.03.103.04.1.053.102.07.052.043 5.04 5.037a1 1 0 1 1-1.415 1.414L9 4.415v8.583a2 2 0 0 1-2 2H2a1 1 0 0 1-1-1Z"/></symbol><symbol id="icon-eds-i-arrow-diagonal-medium" viewBox="0 0 24 24"><path d="M14 3h6l.075.003.126.017.111.03.111.044.098.052.096.067.09.08c.036.035.068.073.097.112l.071.11.054.114.035.105.03.148L21 4v6a1 1 0 0 1-2 0V6.414l-4.293 4.293a1 1 0 0 1-1.414-1.414L17.584 5H14a1 1 0 0 1-.993-.883L13 4a1 1 0 0 1 1-1ZM4 13a1 1 0 0 1 1 1v3.584l4.293-4.291a1 1 0 1 1 1.414 1.414L6.414 19H10a1 1 0 0 1 .993.883L11 20a1 1 0 0 1-1 1l-6.075-.003-.126-.017-.111-.03-.111-.044-.098-.052-.096-.067-.09-.08a1.01 1.01 0 0 1-.097-.112l-.071-.11-.054-.114-.035-.105-.025-.118-.007-.058L3 20v-6a1 1 0 0 1 1-1Z"/></symbol><symbol id="icon-eds-i-arrow-diagonal-small" viewBox="0 0 16 16"><path d="m2 15-.082-.004-.119-.016-.111-.03-.111-.044-.098-.052-.096-.067-.09-.08a1.008 1.008 0 0 1-.097-.112l-.071-.11-.031-.062-.034-.081-.024-.076-.025-.118-.007-.058L1 14.02V9a1 1 0 1 1 2 0v2.584l2.793-2.791a1 1 0 1 1 1.414 1.414L4.414 13H7a1 1 0 0 1 .993.883L8 14a1 1 0 0 1-1 1H2ZM14 1l.081.003.12.017.111.03.111.044.098.052.096.067.09.08c.036.035.068.073.097.112l.071.11.031.062.034.081.024.076.03.148L15 2v5a1 1 0 0 1-2 0V4.414l-2.96 2.96A1 1 0 1 1 8.626 5.96L11.584 3H9a1 1 0 0 1-.993-.883L8 2a1 1 0 0 1 1-1h5Z"/></symbol><symbol id="icon-eds-i-arrow-down-medium" viewBox="0 0 24 24"><path d="m20.707 12.728-7.99 7.98a.996.996 0 0 1-.561.281l-.157.011a.998.998 0 0 1-.788-.384l-7.918-7.908a1 1 0 0 1 1.414-1.416L11 17.576V4a1 1 0 0 1 2 0v13.598l6.293-6.285a1 1 0 0 1 1.32-.082l.095.083a1 1 0 0 1-.001 1.414Z"/></symbol><symbol id="icon-eds-i-arrow-down-small" viewBox="0 0 16 16"><path d="m1.293 8.707 6 6 .063.059.093.069.081.048.105.049.104.034.056.013.118.017L8 15l.076-.003.122-.017.113-.03.085-.032.063-.03.098-.058.06-.043.05-.043 6.04-6.037a1 1 0 0 0-1.414-1.414L9 11.583V2a1 1 0 1 0-2 0v9.585L2.707 7.293a1 1 0 0 0-1.32-.083l-.094.083a1 1 0 0 0 0 1.414Z"/></symbol><symbol id="icon-eds-i-arrow-left-medium" viewBox="0 0 24 24"><path d="m11.272 3.293-7.98 7.99a.996.996 0 0 0-.281.561L3 12.001c0 .32.15.605.384.788l7.908 7.918a1 1 0 0 0 1.416-1.414L6.424 13H20a1 1 0 0 0 0-2H6.402l6.285-6.293a1 1 0 0 0 .082-1.32l-.083-.095a1 1 0 0 0-1.414.001Z"/></symbol><symbol id="icon-eds-i-arrow-left-small" viewBox="0 0 16 16"><path d="m7.293 1.293-6 6-.059.063-.069.093-.048.081-.049.105-.034.104-.013.056-.017.118L1 8l.003.076.017.122.03.113.032.085.03.063.058.098.043.06.043.05 6.037 6.04a1 1 0 0 0 1.414-1.414L4.417 9H14a1 1 0 0 0 0-2H4.415l4.292-4.293a1 1 0 0 0 .083-1.32l-.083-.094a1 1 0 0 0-1.414 0Z"/></symbol><symbol id="icon-eds-i-arrow-right-medium" viewBox="0 0 24 24"><path d="m12.728 3.293 7.98 7.99a.996.996 0 0 1 .281.561l.011.157c0 .32-.15.605-.384.788l-7.908 7.918a1 1 0 0 1-1.416-1.414L17.576 13H4a1 1 0 0 1 0-2h13.598l-6.285-6.293a1 1 0 0 1-.082-1.32l.083-.095a1 1 0 0 1 1.414.001Z"/></symbol><symbol id="icon-eds-i-arrow-right-small" viewBox="0 0 16 16"><path d="m8.707 1.293 6 6 .059.063.069.093.048.081.049.105.034.104.013.056.017.118L15 8l-.003.076-.017.122-.03.113-.032.085-.03.063-.058.098-.043.06-.043.05-6.037 6.04a1 1 0 0 1-1.414-1.414L11.583 9H2a1 1 0 1 1 0-2h9.585L7.293 2.707a1 1 0 0 1-.083-1.32l.083-.094a1 1 0 0 1 1.414 0Z"/></symbol><symbol id="icon-eds-i-arrow-up-medium" viewBox="0 0 24 24"><path d="m3.293 11.272 7.99-7.98a.996.996 0 0 1 .561-.281L12.001 3c.32 0 .605.15.788.384l7.918 7.908a1 1 0 0 1-1.414 1.416L13 6.424V20a1 1 0 0 1-2 0V6.402l-6.293 6.285a1 1 0 0 1-1.32.082l-.095-.083a1 1 0 0 1 .001-1.414Z"/></symbol><symbol id="icon-eds-i-arrow-up-small" viewBox="0 0 16 16"><path d="m1.293 7.293 6-6 .063-.059.093-.069.081-.048.105-.049.104-.034.056-.013.118-.017L8 1l.076.003.122.017.113.03.085.032.063.03.098.058.06.043.05.043 6.04 6.037a1 1 0 0 1-1.414 1.414L9 4.417V14a1 1 0 0 1-2 0V4.415L2.707 8.707a1 1 0 0 1-1.32.083l-.094-.083a1 1 0 0 1 0-1.414Z"/></symbol><symbol id="icon-eds-i-article-medium" viewBox="0 0 24 24"><path d="M8 7a1 1 0 0 0 0 2h4a1 1 0 1 0 0-2H8ZM8 11a1 1 0 1 0 0 2h8a1 1 0 1 0 0-2H8ZM7 16a1 1 0 0 1 1-1h8a1 1 0 1 1 0 2H8a1 1 0 0 1-1-1Z"/><path d="M5.545 1A2.542 2.542 0 0 0 3 3.538v16.924A2.542 2.542 0 0 0 5.545 23h12.91A2.542 2.542 0 0 0 21 20.462V3.5A2.5 2.5 0 0 0 18.5 1H5.545ZM5 3.538C5 3.245 5.24 3 5.545 3H18.5a.5.5 0 0 1 .5.5v16.962c0 .293-.24.538-.546.538H5.545A.542.542 0 0 1 5 20.462V3.538Z" clip-rule="evenodd"/></symbol><symbol id="icon-eds-i-book-medium" viewBox="0 0 24 24"><path d="M18.5 1A2.5 2.5 0 0 1 21 3.5v12c0 1.16-.79 2.135-1.86 2.418l-.14.031V21h1a1 1 0 0 1 .993.883L21 22a1 1 0 0 1-1 1H6.5A3.5 3.5 0 0 1 3 19.5v-15A3.5 3.5 0 0 1 6.5 1h12ZM17 18H6.5a1.5 1.5 0 0 0-1.493 1.356L5 19.5A1.5 1.5 0 0 0 6.5 21H17v-3Zm1.5-15h-12A1.5 1.5 0 0 0 5 4.5v11.837l.054-.025a3.481 3.481 0 0 1 1.254-.307L6.5 16h12a.5.5 0 0 0 .492-.41L19 15.5v-12a.5.5 0 0 0-.5-.5ZM15 6a1 1 0 0 1 0 2H9a1 1 0 1 1 0-2h6Z"/></symbol><symbol id="icon-eds-i-book-series-medium" viewBox="0 0 24 24"><path fill-rule="evenodd" d="M1 3.786C1 2.759 1.857 2 2.82 2H6.18c.964 0 1.82.759 1.82 1.786V4h3.168c.668 0 1.298.364 1.616.938.158-.109.333-.195.523-.252l3.216-.965c.923-.277 1.962.204 2.257 1.187l4.146 13.82c.296.984-.307 1.957-1.23 2.234l-3.217.965c-.923.277-1.962-.203-2.257-1.187L13 10.005v10.21c0 1.04-.878 1.785-1.834 1.785H7.833c-.291 0-.575-.07-.83-.195A1.849 1.849 0 0 1 6.18 22H2.821C1.857 22 1 21.241 1 20.214V3.786ZM3 4v11h3V4H3Zm0 16v-3h3v3H3Zm15.075-.04-.814-2.712 2.874-.862.813 2.712-2.873.862Zm1.485-5.49-2.874.862-2.634-8.782 2.873-.862 2.635 8.782ZM8 20V6h3v14H8Z" clip-rule="evenodd"/></symbol><symbol id="icon-eds-i-calendar-acceptance-medium" viewBox="0 0 24 24"><path d="M17 2a1 1 0 0 1 1 1v1h1.5C20.817 4 22 5.183 22 6.5v13c0 1.317-1.183 2.5-2.5 2.5h-15C3.183 22 2 20.817 2 19.5v-13C2 5.183 3.183 4 4.5 4a1 1 0 1 1 0 2c-.212 0-.5.288-.5.5v13c0 .212.288.5.5.5h15c.212 0 .5-.288.5-.5v-13c0-.212-.288-.5-.5-.5H18v1a1 1 0 0 1-2 0V3a1 1 0 0 1 1-1Zm-.534 7.747a1 1 0 0 1 .094 1.412l-4.846 5.538a1 1 0 0 1-1.352.141l-2.77-2.076a1 1 0 0 1 1.2-1.6l2.027 1.519 4.236-4.84a1 1 0 0 1 1.411-.094ZM7.5 2a1 1 0 0 1 1 1v1H14a1 1 0 0 1 0 2H8.5v1a1 1 0 1 1-2 0V3a1 1 0 0 1 1-1Z"/></symbol><symbol id="icon-eds-i-calendar-date-medium" viewBox="0 0 24 24"><path d="M17 2a1 1 0 0 1 1 1v1h1.5C20.817 4 22 5.183 22 6.5v13c0 1.317-1.183 2.5-2.5 2.5h-15C3.183 22 2 20.817 2 19.5v-13C2 5.183 3.183 4 4.5 4a1 1 0 1 1 0 2c-.212 0-.5.288-.5.5v13c0 .212.288.5.5.5h15c.212 0 .5-.288.5-.5v-13c0-.212-.288-.5-.5-.5H18v1a1 1 0 0 1-2 0V3a1 1 0 0 1 1-1ZM8 15a1 1 0 1 1 0 2 1 1 0 0 1 0-2Zm4 0a1 1 0 1 1 0 2 1 1 0 0 1 0-2Zm-4-4a1 1 0 1 1 0 2 1 1 0 0 1 0-2Zm4 0a1 1 0 1 1 0 2 1 1 0 0 1 0-2Zm4 0a1 1 0 1 1 0 2 1 1 0 0 1 0-2ZM7.5 2a1 1 0 0 1 1 1v1H14a1 1 0 0 1 0 2H8.5v1a1 1 0 1 1-2 0V3a1 1 0 0 1 1-1Z"/></symbol><symbol id="icon-eds-i-calendar-decision-medium" viewBox="0 0 24 24"><path d="M17 2a1 1 0 0 1 1 1v1h1.5C20.817 4 22 5.183 22 6.5v13c0 1.317-1.183 2.5-2.5 2.5h-15C3.183 22 2 20.817 2 19.5v-13C2 5.183 3.183 4 4.5 4a1 1 0 1 1 0 2c-.212 0-.5.288-.5.5v13c0 .212.288.5.5.5h15c.212 0 .5-.288.5-.5v-13c0-.212-.288-.5-.5-.5H18v1a1 1 0 0 1-2 0V3a1 1 0 0 1 1-1Zm-2.935 8.246 2.686 2.645c.34.335.34.883 0 1.218l-2.686 2.645a.858.858 0 0 1-1.213-.009.854.854 0 0 1 .009-1.21l1.05-1.035H7.984a.992.992 0 0 1-.984-1c0-.552.44-1 .984-1h5.928l-1.051-1.036a.854.854 0 0 1-.085-1.121l.076-.088a.858.858 0 0 1 1.213-.009ZM7.5 2a1 1 0 0 1 1 1v1H14a1 1 0 0 1 0 2H8.5v1a1 1 0 1 1-2 0V3a1 1 0 0 1 1-1Z"/></symbol><symbol id="icon-eds-i-calendar-impact-factor-medium" viewBox="0 0 24 24"><path d="M17 2a1 1 0 0 1 1 1v1h1.5C20.817 4 22 5.183 22 6.5v13c0 1.317-1.183 2.5-2.5 2.5h-15C3.183 22 2 20.817 2 19.5v-13C2 5.183 3.183 4 4.5 4a1 1 0 1 1 0 2c-.212 0-.5.288-.5.5v13c0 .212.288.5.5.5h15c.212 0 .5-.288.5-.5v-13c0-.212-.288-.5-.5-.5H18v1a1 1 0 0 1-2 0V3a1 1 0 0 1 1-1Zm-3.2 6.924a.48.48 0 0 1 .125.544l-1.52 3.283h2.304c.27 0 .491.215.491.483a.477.477 0 0 1-.13.327l-4.18 4.484a.498.498 0 0 1-.69.031.48.48 0 0 1-.125-.544l1.52-3.284H9.291a.487.487 0 0 1-.491-.482c0-.121.047-.238.13-.327l4.18-4.484a.498.498 0 0 1 .69-.031ZM7.5 2a1 1 0 0 1 1 1v1H14a1 1 0 0 1 0 2H8.5v1a1 1 0 1 1-2 0V3a1 1 0 0 1 1-1Z"/></symbol><symbol id="icon-eds-i-call-papers-medium" viewBox="0 0 24 24"><g><path d="m20.707 2.883-1.414 1.414a1 1 0 0 0 1.414 1.414l1.414-1.414a1 1 0 0 0-1.414-1.414Z"/><path d="M6 16.054c0 2.026 1.052 2.943 3 2.943a1 1 0 1 1 0 2c-2.996 0-5-1.746-5-4.943v-1.227a4.068 4.068 0 0 1-1.83-1.189 4.553 4.553 0 0 1-.87-1.455 4.868 4.868 0 0 1-.3-1.686c0-1.17.417-2.298 1.17-3.14.38-.426.834-.767 1.338-1 .51-.237 1.06-.36 1.617-.36L6.632 6H7l7.932-2.895A2.363 2.363 0 0 1 18 5.36v9.28a2.36 2.36 0 0 1-3.069 2.25l.084.03L7 14.997H6v1.057Zm9.637-11.057a.415.415 0 0 0-.083.008L8 7.638v5.536l7.424 1.786.104.02c.035.01.072.02.109.02.2 0 .363-.16.363-.36V5.36c0-.2-.163-.363-.363-.363Zm-9.638 3h-.874a1.82 1.82 0 0 0-.625.111l-.15.063a2.128 2.128 0 0 0-.689.517c-.42.47-.661 1.123-.661 1.81 0 .34.06.678.176.992.114.308.28.585.485.816.4.447.925.691 1.464.691h.874v-5Z" clip-rule="evenodd"/><path d="M20 8.997h2a1 1 0 1 1 0 2h-2a1 1 0 1 1 0-2ZM20.707 14.293l1.414 1.414a1 1 0 0 1-1.414 1.414l-1.414-1.414a1 1 0 0 1 1.414-1.414Z"/></g></symbol><symbol id="icon-eds-i-card-medium" viewBox="0 0 24 24"><path d="M19.615 2c.315 0 .716.067 1.14.279.76.38 1.245 1.107 1.245 2.106v15.23c0 .315-.067.716-.279 1.14-.38.76-1.107 1.245-2.106 1.245H4.385a2.56 2.56 0 0 1-1.14-.279C2.485 21.341 2 20.614 2 19.615V4.385c0-.315.067-.716.279-1.14C2.659 2.485 3.386 2 4.385 2h15.23Zm0 2H4.385c-.213 0-.265.034-.317.14A.71.71 0 0 0 4 4.385v15.23c0 .213.034.265.14.317a.71.71 0 0 0 .245.068h15.23c.213 0 .265-.034.317-.14a.71.71 0 0 0 .068-.245V4.385c0-.213-.034-.265-.14-.317A.71.71 0 0 0 19.615 4ZM17 16a1 1 0 0 1 0 2H7a1 1 0 0 1 0-2h10Zm0-3a1 1 0 0 1 0 2H7a1 1 0 0 1 0-2h10Zm-.5-7A1.5 1.5 0 0 1 18 7.5v3a1.5 1.5 0 0 1-1.5 1.5h-9A1.5 1.5 0 0 1 6 10.5v-3A1.5 1.5 0 0 1 7.5 6h9ZM16 8H8v2h8V8Z"/></symbol><symbol id="icon-eds-i-cart-medium" viewBox="0 0 24 24"><path d="M5.76 1a1 1 0 0 1 .994.902L7.155 6h13.34c.18 0 .358.02.532.057l.174.045a2.5 2.5 0 0 1 1.693 3.103l-2.069 7.03c-.36 1.099-1.398 1.823-2.49 1.763H8.65c-1.272.015-2.352-.927-2.546-2.244L4.852 3H2a1 1 0 0 1-.993-.883L1 2a1 1 0 0 1 1-1h3.76Zm2.328 14.51a.555.555 0 0 0 .55.488l9.751.001a.533.533 0 0 0 .527-.357l2.059-7a.5.5 0 0 0-.48-.642H7.351l.737 7.51ZM18 19a2 2 0 1 1 0 4 2 2 0 0 1 0-4ZM8 19a2 2 0 1 1 0 4 2 2 0 0 1 0-4Z"/></symbol><symbol id="icon-eds-i-check-circle-medium" viewBox="0 0 24 24"><path d="M12 1c6.075 0 11 4.925 11 11s-4.925 11-11 11S1 18.075 1 12 5.925 1 12 1Zm0 2a9 9 0 1 0 0 18 9 9 0 0 0 0-18Zm5.125 4.72a1 1 0 0 1 .156 1.405l-6 7.5a1 1 0 0 1-1.421.143l-3-2.5a1 1 0 0 1 1.28-1.536l2.217 1.846 5.362-6.703a1 1 0 0 1 1.406-.156Z"/></symbol><symbol id="icon-eds-i-check-filled-medium" viewBox="0 0 24 24"><path d="M12 1c6.075 0 11 4.925 11 11s-4.925 11-11 11S1 18.075 1 12 5.925 1 12 1Zm5.125 6.72a1 1 0 0 0-1.406.155l-5.362 6.703-2.217-1.846a1 1 0 1 0-1.28 1.536l3 2.5a1 1 0 0 0 1.42-.143l6-7.5a1 1 0 0 0-.155-1.406Z"/></symbol><symbol id="icon-eds-i-chevron-down-medium" viewBox="0 0 24 24"><path d="M3.305 8.28a1 1 0 0 0-.024 1.415l7.495 7.762c.314.345.757.543 1.224.543.467 0 .91-.198 1.204-.522l7.515-7.783a1 1 0 1 0-1.438-1.39L12 15.845l-7.28-7.54A1 1 0 0 0 3.4 8.2l-.096.082Z"/></symbol><symbol id="icon-eds-i-chevron-down-small" viewBox="0 0 16 16"><path d="M13.692 5.278a1 1 0 0 1 .03 1.414L9.103 11.51a1.491 1.491 0 0 1-2.188.019L2.278 6.692a1 1 0 0 1 1.444-1.384L8 9.771l4.278-4.463a1 1 0 0 1 1.318-.111l.096.081Z"/></symbol><symbol id="icon-eds-i-chevron-left-medium" viewBox="0 0 24 24"><path d="M15.72 3.305a1 1 0 0 0-1.415-.024l-7.762 7.495A1.655 1.655 0 0 0 6 12c0 .467.198.91.522 1.204l7.783 7.515a1 1 0 1 0 1.39-1.438L8.155 12l7.54-7.28A1 1 0 0 0 15.8 3.4l-.082-.096Z"/></symbol><symbol id="icon-eds-i-chevron-left-small" viewBox="0 0 16 16"><path d="M10.722 2.308a1 1 0 0 0-1.414-.03L4.49 6.897a1.491 1.491 0 0 0-.019 2.188l4.838 4.637a1 1 0 1 0 1.384-1.444L6.229 8l4.463-4.278a1 1 0 0 0 .111-1.318l-.081-.096Z"/></symbol><symbol id="icon-eds-i-chevron-right-medium" viewBox="0 0 24 24"><path d="M8.28 3.305a1 1 0 0 1 1.415-.024l7.762 7.495c.345.314.543.757.543 1.224 0 .467-.198.91-.522 1.204l-7.783 7.515a1 1 0 1 1-1.39-1.438L15.845 12l-7.54-7.28A1 1 0 0 1 8.2 3.4l.082-.096Z"/></symbol><symbol id="icon-eds-i-chevron-right-small" viewBox="0 0 16 16"><path d="M5.278 2.308a1 1 0 0 1 1.414-.03l4.819 4.619a1.491 1.491 0 0 1 .019 2.188l-4.838 4.637a1 1 0 1 1-1.384-1.444L9.771 8 5.308 3.722a1 1 0 0 1-.111-1.318l.081-.096Z"/></symbol><symbol id="icon-eds-i-chevron-up-medium" viewBox="0 0 24 24"><path d="M20.695 15.72a1 1 0 0 0 .024-1.415l-7.495-7.762A1.655 1.655 0 0 0 12 6c-.467 0-.91.198-1.204.522l-7.515 7.783a1 1 0 1 0 1.438 1.39L12 8.155l7.28 7.54a1 1 0 0 0 1.319.106l.096-.082Z"/></symbol><symbol id="icon-eds-i-chevron-up-small" viewBox="0 0 16 16"><path d="M13.692 10.722a1 1 0 0 0 .03-1.414L9.103 4.49a1.491 1.491 0 0 0-2.188-.019L2.278 9.308a1 1 0 0 0 1.444 1.384L8 6.229l4.278 4.463a1 1 0 0 0 1.318.111l.096-.081Z"/></symbol><symbol id="icon-eds-i-citations-medium" viewBox="0 0 24 24"><path d="M15.59 1a1 1 0 0 1 .706.291l5.41 5.385a1 1 0 0 1 .294.709v13.077c0 .674-.269 1.32-.747 1.796a2.549 2.549 0 0 1-1.798.742h-5.843a1 1 0 1 1 0-2h5.843a.549.549 0 0 0 .387-.16.535.535 0 0 0 .158-.378V7.8L15.178 3H5.545a.543.543 0 0 0-.538.451L5 3.538v8.607a1 1 0 0 1-2 0V3.538A2.542 2.542 0 0 1 5.545 1h10.046ZM5.483 14.35c.197.26.17.62-.049.848l-.095.083-.016.011c-.36.24-.628.45-.804.634-.393.409-.59.93-.59 1.562.077-.019.192-.028.345-.028.442 0 .84.158 1.195.474.355.316.532.716.532 1.2 0 .501-.173.9-.518 1.198-.345.298-.767.446-1.266.446-.672 0-1.209-.195-1.612-.585-.403-.39-.604-.976-.604-1.757 0-.744.11-1.39.33-1.938.222-.549.49-1.009.807-1.38a4.28 4.28 0 0 1 .992-.88c.07-.043.148-.087.232-.133a.881.881 0 0 1 1.121.245Zm5 0c.197.26.17.62-.049.848l-.095.083-.016.011c-.36.24-.628.45-.804.634-.393.409-.59.93-.59 1.562.077-.019.192-.028.345-.028.442 0 .84.158 1.195.474.355.316.532.716.532 1.2 0 .501-.173.9-.518 1.198-.345.298-.767.446-1.266.446-.672 0-1.209-.195-1.612-.585-.403-.39-.604-.976-.604-1.757 0-.744.11-1.39.33-1.938.222-.549.49-1.009.807-1.38a4.28 4.28 0 0 1 .992-.88c.07-.043.148-.087.232-.133a.881.881 0 0 1 1.121.245Z"/></symbol><symbol id="icon-eds-i-clipboard-check-medium" viewBox="0 0 24 24"><path d="M14.4 1c1.238 0 2.274.865 2.536 2.024L18.5 3C19.886 3 21 4.14 21 5.535v14.93C21 21.86 19.886 23 18.5 23h-13C4.114 23 3 21.86 3 20.465V5.535C3 4.14 4.114 3 5.5 3h1.57c.27-1.147 1.3-2 2.53-2h4.8Zm4.115 4-1.59.024A2.601 2.601 0 0 1 14.4 7H9.6c-1.23 0-2.26-.853-2.53-2H5.5c-.27 0-.5.234-.5.535v14.93c0 .3.23.535.5.535h13c.27 0 .5-.234.5-.535V5.535c0-.3-.23-.535-.485-.535Zm-1.909 4.205a1 1 0 0 1 .19 1.401l-5.334 7a1 1 0 0 1-1.344.23l-2.667-1.75a1 1 0 1 1 1.098-1.672l1.887 1.238 4.769-6.258a1 1 0 0 1 1.401-.19ZM14.4 3H9.6a.6.6 0 0 0-.6.6v.8a.6.6 0 0 0 .6.6h4.8a.6.6 0 0 0 .6-.6v-.8a.6.6 0 0 0-.6-.6Z"/></symbol><symbol id="icon-eds-i-clipboard-report-medium" viewBox="0 0 24 24"><path d="M14.4 1c1.238 0 2.274.865 2.536 2.024L18.5 3C19.886 3 21 4.14 21 5.535v14.93C21 21.86 19.886 23 18.5 23h-13C4.114 23 3 21.86 3 20.465V5.535C3 4.14 4.114 3 5.5 3h1.57c.27-1.147 1.3-2 2.53-2h4.8Zm4.115 4-1.59.024A2.601 2.601 0 0 1 14.4 7H9.6c-1.23 0-2.26-.853-2.53-2H5.5c-.27 0-.5.234-.5.535v14.93c0 .3.23.535.5.535h13c.27 0 .5-.234.5-.535V5.535c0-.3-.23-.535-.485-.535Zm-2.658 10.929a1 1 0 0 1 0 2H8a1 1 0 0 1 0-2h7.857Zm0-3.929a1 1 0 0 1 0 2H8a1 1 0 0 1 0-2h7.857ZM14.4 3H9.6a.6.6 0 0 0-.6.6v.8a.6.6 0 0 0 .6.6h4.8a.6.6 0 0 0 .6-.6v-.8a.6.6 0 0 0-.6-.6Z"/></symbol><symbol id="icon-eds-i-close-medium" viewBox="0 0 24 24"><path d="M12 1c6.075 0 11 4.925 11 11s-4.925 11-11 11S1 18.075 1 12 5.925 1 12 1Zm0 2a9 9 0 1 0 0 18 9 9 0 0 0 0-18ZM8.707 7.293 12 10.585l3.293-3.292a1 1 0 0 1 1.414 1.414L13.415 12l3.292 3.293a1 1 0 0 1-1.414 1.414L12 13.415l-3.293 3.292a1 1 0 1 1-1.414-1.414L10.585 12 7.293 8.707a1 1 0 0 1 1.414-1.414Z"/></symbol><symbol id="icon-eds-i-cloud-upload-medium" viewBox="0 0 24 24"><path d="m12.852 10.011.028-.004L13 10l.075.003.126.017.086.022.136.052.098.052.104.074.082.073 3 3a1 1 0 0 1 0 1.414l-.094.083a1 1 0 0 1-1.32-.083L14 13.416V20a1 1 0 0 1-2 0v-6.586l-1.293 1.293a1 1 0 0 1-1.32.083l-.094-.083a1 1 0 0 1 0-1.414l3-3 .112-.097.11-.071.114-.054.105-.035.118-.025Zm.587-7.962c3.065.362 5.497 2.662 5.992 5.562l.013.085.207.073c2.117.782 3.496 2.845 3.337 5.097l-.022.226c-.297 2.561-2.503 4.491-5.124 4.502a1 1 0 1 1-.009-2c1.619-.007 2.967-1.186 3.147-2.733.179-1.542-.86-2.979-2.487-3.353-.512-.149-.894-.579-.981-1.165-.21-2.237-2-4.035-4.308-4.308-2.31-.273-4.497 1.06-5.25 3.19l-.049.113c-.234.468-.718.756-1.176.743-1.418.057-2.689.857-3.32 2.084a3.668 3.668 0 0 0 .262 3.798c.796 1.136 2.169 1.764 3.583 1.635a1 1 0 1 1 .182 1.992c-2.125.194-4.193-.753-5.403-2.48a5.668 5.668 0 0 1-.403-5.86c.85-1.652 2.449-2.79 4.323-3.092l.287-.039.013-.028c1.207-2.741 4.125-4.404 7.186-4.042Z"/></symbol><symbol id="icon-eds-i-collection-medium" viewBox="0 0 24 24"><path d="M21 7a1 1 0 0 1 1 1v12.5a2.5 2.5 0 0 1-2.5 2.5H8a1 1 0 0 1 0-2h11.5a.5.5 0 0 0 .5-.5V8a1 1 0 0 1 1-1Zm-5.5-5A2.5 2.5 0 0 1 18 4.5v12a2.5 2.5 0 0 1-2.5 2.5h-11A2.5 2.5 0 0 1 2 16.5v-12A2.5 2.5 0 0 1 4.5 2h11Zm0 2h-11a.5.5 0 0 0-.5.5v12a.5.5 0 0 0 .5.5h11a.5.5 0 0 0 .5-.5v-12a.5.5 0 0 0-.5-.5ZM13 13a1 1 0 0 1 0 2H7a1 1 0 0 1 0-2h6Zm0-3.5a1 1 0 0 1 0 2H7a1 1 0 0 1 0-2h6ZM13 6a1 1 0 0 1 0 2H7a1 1 0 1 1 0-2h6Z"/></symbol><symbol id="icon-eds-i-conference-series-medium" viewBox="0 0 24 24"><path fill-rule="evenodd" d="M4.5 2A2.5 2.5 0 0 0 2 4.5v11A2.5 2.5 0 0 0 4.5 18h2.37l-2.534 2.253a1 1 0 0 0 1.328 1.494L9.88 18H11v3a1 1 0 1 0 2 0v-3h1.12l4.216 3.747a1 1 0 0 0 1.328-1.494L17.13 18h2.37a2.5 2.5 0 0 0 2.5-2.5v-11A2.5 2.5 0 0 0 19.5 2h-15ZM20 6V4.5a.5.5 0 0 0-.5-.5h-15a.5.5 0 0 0-.5.5V6h16ZM4 8v7.5a.5.5 0 0 0 .5.5h15a.5.5 0 0 0 .5-.5V8H4Z" clip-rule="evenodd"/></symbol><symbol id="icon-eds-i-delivery-medium" viewBox="0 0 24 24"><path d="M8.51 20.598a3.037 3.037 0 0 1-3.02 0A2.968 2.968 0 0 1 4.161 19L3.5 19A2.5 2.5 0 0 1 1 16.5v-11A2.5 2.5 0 0 1 3.5 3h10a2.5 2.5 0 0 1 2.45 2.004L16 5h2.527c.976 0 1.855.585 2.27 1.49l2.112 4.62a1 1 0 0 1 .091.416v4.856C23 17.814 21.889 19 20.484 19h-.523a1.01 1.01 0 0 1-.121-.007 2.96 2.96 0 0 1-1.33 1.605 3.037 3.037 0 0 1-3.02 0A2.968 2.968 0 0 1 14.161 19H9.838a2.968 2.968 0 0 1-1.327 1.597Zm-2.024-3.462a.955.955 0 0 0-.481.73L5.999 18l.001.022a.944.944 0 0 0 .388.777l.098.065c.316.181.712.181 1.028 0A.97.97 0 0 0 8 17.978a.95.95 0 0 0-.486-.842 1.037 1.037 0 0 0-1.028 0Zm10 0a.955.955 0 0 0-.481.73l-.005.156a.944.944 0 0 0 .388.777l.098.065c.316.181.712.181 1.028 0a.97.97 0 0 0 .486-.886.95.95 0 0 0-.486-.842 1.037 1.037 0 0 0-1.028 0ZM21 12h-5v3.17a3.038 3.038 0 0 1 2.51.232 2.993 2.993 0 0 1 1.277 1.45l.058.155.058-.005.581-.002c.27 0 .516-.263.516-.618V12Zm-7.5-7h-10a.5.5 0 0 0-.5.5v11a.5.5 0 0 0 .5.5h.662a2.964 2.964 0 0 1 1.155-1.491l.172-.107a3.037 3.037 0 0 1 3.022 0A2.987 2.987 0 0 1 9.843 17H13.5a.5.5 0 0 0 .5-.5v-11a.5.5 0 0 0-.5-.5Zm5.027 2H16v3h4.203l-1.224-2.677a.532.532 0 0 0-.375-.316L18.527 7Z"/></symbol><symbol id="icon-eds-i-download-medium" viewBox="0 0 24 24"><path d="M22 18.5a3.5 3.5 0 0 1-3.5 3.5h-13A3.5 3.5 0 0 1 2 18.5V18a1 1 0 0 1 2 0v.5A1.5 1.5 0 0 0 5.5 20h13a1.5 1.5 0 0 0 1.5-1.5V18a1 1 0 0 1 2 0v.5Zm-3.293-7.793-6 6-.063.059-.093.069-.081.048-.105.049-.104.034-.056.013-.118.017L12 17l-.076-.003-.122-.017-.113-.03-.085-.032-.063-.03-.098-.058-.06-.043-.05-.043-6.04-6.037a1 1 0 0 1 1.414-1.414l4.294 4.29L11 3a1 1 0 0 1 2 0l.001 10.585 4.292-4.292a1 1 0 0 1 1.32-.083l.094.083a1 1 0 0 1 0 1.414Z"/></symbol><symbol id="icon-eds-i-edit-medium" viewBox="0 0 24 24"><path d="M17.149 2a2.38 2.38 0 0 1 1.699.711l2.446 2.46a2.384 2.384 0 0 1 .005 3.38L10.01 19.906a1 1 0 0 1-.434.257l-6.3 1.8a1 1 0 0 1-1.237-1.237l1.8-6.3a1 1 0 0 1 .257-.434L15.443 2.718A2.385 2.385 0 0 1 17.15 2Zm-3.874 5.689-7.586 7.536-1.234 4.319 4.318-1.234 7.54-7.582-3.038-3.039ZM17.149 4a.395.395 0 0 0-.286.126L14.695 6.28l3.029 3.029 2.162-2.173a.384.384 0 0 0 .106-.197L20 6.864c0-.103-.04-.2-.119-.278l-2.457-2.47A.385.385 0 0 0 17.149 4Z"/></symbol><symbol id="icon-eds-i-education-medium" viewBox="0 0 24 24"><path fill-rule="evenodd" d="M12.41 2.088a1 1 0 0 0-.82 0l-10 4.5a1 1 0 0 0 0 1.824L3 9.047v7.124A3.001 3.001 0 0 0 4 22a3 3 0 0 0 1-5.83V9.948l1 .45V14.5a1 1 0 0 0 .087.408L7 14.5c-.913.408-.912.41-.912.41l.001.003.003.006.007.015a1.988 1.988 0 0 0 .083.16c.054.097.131.225.236.373.21.297.53.68.993 1.057C8.351 17.292 9.824 18 12 18c2.176 0 3.65-.707 4.589-1.476.463-.378.783-.76.993-1.057a4.162 4.162 0 0 0 .319-.533l.007-.015.003-.006v-.003h.002s0-.002-.913-.41l.913.408A1 1 0 0 0 18 14.5v-4.103l4.41-1.985a1 1 0 0 0 0-1.824l-10-4.5ZM16 11.297l-3.59 1.615a1 1 0 0 1-.82 0L8 11.297v2.94a3.388 3.388 0 0 0 .677.739C9.267 15.457 10.294 16 12 16s2.734-.543 3.323-1.024a3.388 3.388 0 0 0 .677-.739v-2.94ZM4.437 7.5 12 4.097 19.563 7.5 12 10.903 4.437 7.5ZM3 19a1 1 0 1 1 2 0 1 1 0 0 1-2 0Z" clip-rule="evenodd"/></symbol><symbol id="icon-eds-i-error-diamond-medium" viewBox="0 0 24 24"><path d="M12.002 1c.702 0 1.375.279 1.871.775l8.35 8.353a2.646 2.646 0 0 1 .001 3.744l-8.353 8.353a2.646 2.646 0 0 1-3.742 0l-8.353-8.353a2.646 2.646 0 0 1 0-3.744l8.353-8.353.156-.142c.424-.362.952-.58 1.507-.625l.21-.008Zm0 2a.646.646 0 0 0-.38.123l-.093.08-8.34 8.34a.646.646 0 0 0-.18.355L3 12c0 .171.068.336.19.457l8.353 8.354a.646.646 0 0 0 .914 0l8.354-8.354a.646.646 0 0 0-.001-.914l-8.351-8.354A.646.646 0 0 0 12.002 3ZM12 14.5a1.5 1.5 0 0 1 .144 2.993L12 17.5a1.5 1.5 0 0 1 0-3ZM12 6a1 1 0 0 1 1 1v5a1 1 0 0 1-2 0V7a1 1 0 0 1 1-1Z"/></symbol><symbol id="icon-eds-i-error-filled-medium" viewBox="0 0 24 24"><path d="M12.002 1c.702 0 1.375.279 1.871.775l8.35 8.353a2.646 2.646 0 0 1 .001 3.744l-8.353 8.353a2.646 2.646 0 0 1-3.742 0l-8.353-8.353a2.646 2.646 0 0 1 0-3.744l8.353-8.353.156-.142c.424-.362.952-.58 1.507-.625l.21-.008ZM12 14.5a1.5 1.5 0 0 0 0 3l.144-.007A1.5 1.5 0 0 0 12 14.5ZM12 6a1 1 0 0 0-1 1v5a1 1 0 0 0 2 0V7a1 1 0 0 0-1-1Z"/></symbol><symbol id="icon-eds-i-external-link-medium" viewBox="0 0 24 24"><path d="M9 2a1 1 0 1 1 0 2H4.6c-.371 0-.6.209-.6.5v15c0 .291.229.5.6.5h14.8c.371 0 .6-.209.6-.5V15a1 1 0 0 1 2 0v4.5c0 1.438-1.162 2.5-2.6 2.5H4.6C3.162 22 2 20.938 2 19.5v-15C2 3.062 3.162 2 4.6 2H9Zm6 0h6l.075.003.126.017.111.03.111.044.098.052.096.067.09.08c.036.035.068.073.097.112l.071.11.054.114.035.105.03.148L22 3v6a1 1 0 0 1-2 0V5.414l-6.693 6.693a1 1 0 0 1-1.414-1.414L18.584 4H15a1 1 0 0 1-.993-.883L14 3a1 1 0 0 1 1-1Z"/></symbol><symbol id="icon-eds-i-external-link-small" viewBox="0 0 16 16"><path d="M5 1a1 1 0 1 1 0 2l-2-.001V13L13 13v-2a1 1 0 0 1 2 0v2c0 1.15-.93 2-2.067 2H3.067C1.93 15 1 14.15 1 13V3c0-1.15.93-2 2.067-2H5Zm4 0h5l.075.003.126.017.111.03.111.044.098.052.096.067.09.08.044.047.073.093.051.083.054.113.035.105.03.148L15 2v5a1 1 0 0 1-2 0V4.414L9.107 8.307a1 1 0 0 1-1.414-1.414L11.584 3H9a1 1 0 0 1-.993-.883L8 2a1 1 0 0 1 1-1Z"/></symbol><symbol id="icon-eds-i-file-download-medium" viewBox="0 0 24 24"><path d="M14.5 1a1 1 0 0 1 .707.293l5.5 5.5A1 1 0 0 1 21 7.5v12.962A2.542 2.542 0 0 1 18.455 23H5.545A2.542 2.542 0 0 1 3 20.462V3.538A2.542 2.542 0 0 1 5.545 1H14.5Zm-.415 2h-8.54A.542.542 0 0 0 5 3.538v16.924c0 .296.243.538.545.538h12.91a.542.542 0 0 0 .545-.538V7.915L14.085 3ZM12 7a1 1 0 0 1 1 1v6.585l2.293-2.292a1 1 0 0 1 1.32-.083l.094.083a1 1 0 0 1 0 1.414l-4 4a1.008 1.008 0 0 1-.112.097l-.11.071-.114.054-.105.035-.149.03L12 18l-.075-.003-.126-.017-.111-.03-.111-.044-.098-.052-.096-.067-.09-.08-4-4a1 1 0 0 1 1.414-1.414L11 14.585V8a1 1 0 0 1 1-1Z"/></symbol><symbol id="icon-eds-i-file-report-medium" viewBox="0 0 24 24"><path d="M14.5 1a1 1 0 0 1 .707.293l5.5 5.5A1 1 0 0 1 21 7.5v12.962c0 .674-.269 1.32-.747 1.796a2.549 2.549 0 0 1-1.798.742H5.545c-.674 0-1.32-.267-1.798-.742A2.535 2.535 0 0 1 3 20.462V3.538A2.542 2.542 0 0 1 5.545 1H14.5Zm-.415 2h-8.54A.542.542 0 0 0 5 3.538v16.924c0 .142.057.278.158.379.102.102.242.159.387.159h12.91a.549.549 0 0 0 .387-.16.535.535 0 0 0 .158-.378V7.915L14.085 3ZM16 17a1 1 0 0 1 0 2H8a1 1 0 0 1 0-2h8Zm0-3a1 1 0 0 1 0 2H8a1 1 0 0 1 0-2h8Zm-4.793-6.207L13 9.585l1.793-1.792a1 1 0 0 1 1.32-.083l.094.083a1 1 0 0 1 0 1.414l-2.5 2.5a1 1 0 0 1-1.414 0L10.5 9.915l-1.793 1.792a1 1 0 0 1-1.32.083l-.094-.083a1 1 0 0 1 0-1.414l2.5-2.5a1 1 0 0 1 1.414 0Z"/></symbol><symbol id="icon-eds-i-file-text-medium" viewBox="0 0 24 24"><path d="M14.5 1a1 1 0 0 1 .707.293l5.5 5.5A1 1 0 0 1 21 7.5v12.962A2.542 2.542 0 0 1 18.455 23H5.545A2.542 2.542 0 0 1 3 20.462V3.538A2.542 2.542 0 0 1 5.545 1H14.5Zm-.415 2h-8.54A.542.542 0 0 0 5 3.538v16.924c0 .296.243.538.545.538h12.91a.542.542 0 0 0 .545-.538V7.915L14.085 3ZM16 15a1 1 0 0 1 0 2H8a1 1 0 0 1 0-2h8Zm0-4a1 1 0 0 1 0 2H8a1 1 0 0 1 0-2h8Zm-5-4a1 1 0 0 1 0 2H8a1 1 0 1 1 0-2h3Z"/></symbol><symbol id="icon-eds-i-file-upload-medium" viewBox="0 0 24 24"><path d="M14.5 1a1 1 0 0 1 .707.293l5.5 5.5A1 1 0 0 1 21 7.5v12.962A2.542 2.542 0 0 1 18.455 23H5.545A2.542 2.542 0 0 1 3 20.462V3.538A2.542 2.542 0 0 1 5.545 1H14.5Zm-.415 2h-8.54A.542.542 0 0 0 5 3.538v16.924c0 .296.243.538.545.538h12.91a.542.542 0 0 0 .545-.538V7.915L14.085 3Zm-2.233 4.011.058-.007L12 7l.075.003.126.017.111.03.111.044.098.052.104.074.082.073 4 4a1 1 0 0 1 0 1.414l-.094.083a1 1 0 0 1-1.32-.083L13 10.415V17a1 1 0 0 1-2 0v-6.585l-2.293 2.292a1 1 0 0 1-1.32.083l-.094-.083a1 1 0 0 1 0-1.414l4-4 .112-.097.11-.071.114-.054.105-.035.118-.025Z"/></symbol><symbol id="icon-eds-i-filter-medium" viewBox="0 0 24 24"><path d="M21 2a1 1 0 0 1 .82 1.573L15 13.314V18a1 1 0 0 1-.31.724l-.09.076-4 3A1 1 0 0 1 9 21v-7.684L2.18 3.573a1 1 0 0 1 .707-1.567L3 2h18Zm-1.921 2H4.92l5.9 8.427a1 1 0 0 1 .172.45L11 13v6l2-1.5V13a1 1 0 0 1 .117-.469l.064-.104L19.079 4Z"/></symbol><symbol id="icon-eds-i-funding-medium" viewBox="0 0 24 24"><path fill-rule="evenodd" d="M23 8A7 7 0 1 0 9 8a7 7 0 0 0 14 0ZM9.006 12.225A4.07 4.07 0 0 0 6.12 11.02H2a.979.979 0 1 0 0 1.958h4.12c.558 0 1.094.222 1.489.617l2.207 2.288c.27.27.27.687.012.944a.656.656 0 0 1-.928 0L7.744 15.67a.98.98 0 0 0-1.386 1.384l1.157 1.158c.535.536 1.244.791 1.946.765l.041.002h6.922c.874 0 1.597.748 1.597 1.688 0 .203-.146.354-.309.354H7.755c-.487 0-.96-.178-1.339-.504L2.64 17.259a.979.979 0 0 0-1.28 1.482L5.137 22c.733.631 1.66.979 2.618.979h9.957c1.26 0 2.267-1.043 2.267-2.312 0-2.006-1.584-3.646-3.555-3.646h-4.529a2.617 2.617 0 0 0-.681-2.509l-2.208-2.287ZM16 3a5 5 0 1 0 0 10 5 5 0 0 0 0-10Zm.979 3.5a.979.979 0 1 0-1.958 0v3a.979.979 0 1 0 1.958 0v-3Z" clip-rule="evenodd"/></symbol><symbol id="icon-eds-i-hashtag-medium" viewBox="0 0 24 24"><path d="M12 1c6.075 0 11 4.925 11 11s-4.925 11-11 11S1 18.075 1 12 5.925 1 12 1Zm0 2a9 9 0 1 0 0 18 9 9 0 0 0 0-18ZM9.52 18.189a1 1 0 1 1-1.964-.378l.437-2.274H6a1 1 0 1 1 0-2h2.378l.592-3.076H6a1 1 0 0 1 0-2h3.354l.51-2.65a1 1 0 1 1 1.964.378l-.437 2.272h3.04l.51-2.65a1 1 0 1 1 1.964.378l-.438 2.272H18a1 1 0 0 1 0 2h-1.917l-.592 3.076H18a1 1 0 0 1 0 2h-2.893l-.51 2.652a1 1 0 1 1-1.964-.378l.437-2.274h-3.04l-.51 2.652Zm.895-4.652h3.04l.591-3.076h-3.04l-.591 3.076Z"/></symbol><symbol id="icon-eds-i-home-medium" viewBox="0 0 24 24"><path d="M5 22a1 1 0 0 1-1-1v-8.586l-1.293 1.293a1 1 0 0 1-1.32.083l-.094-.083a1 1 0 0 1 0-1.414l10-10a1 1 0 0 1 1.414 0l10 10a1 1 0 0 1-1.414 1.414L20 12.415V21a1 1 0 0 1-1 1H5Zm7-17.585-6 5.999V20h5v-4a1 1 0 0 1 2 0v4h5v-9.585l-6-6Z"/></symbol><symbol id="icon-eds-i-image-medium" viewBox="0 0 24 24"><path d="M19.615 2A2.385 2.385 0 0 1 22 4.385v15.23A2.385 2.385 0 0 1 19.615 22H4.385A2.385 2.385 0 0 1 2 19.615V4.385A2.385 2.385 0 0 1 4.385 2h15.23Zm0 2H4.385A.385.385 0 0 0 4 4.385v15.23c0 .213.172.385.385.385h1.244l10.228-8.76a1 1 0 0 1 1.254-.037L20 13.392V4.385A.385.385 0 0 0 19.615 4Zm-3.07 9.283L8.703 20h10.912a.385.385 0 0 0 .385-.385v-3.713l-3.455-2.619ZM9.5 6a3.5 3.5 0 1 1 0 7 3.5 3.5 0 0 1 0-7Zm0 2a1.5 1.5 0 1 0 0 3 1.5 1.5 0 0 0 0-3Z"/></symbol><symbol id="icon-eds-i-impact-factor-medium" viewBox="0 0 24 24"><path d="M16.49 2.672c.74.694.986 1.765.632 2.712l-.04.1-1.549 3.54h1.477a2.496 2.496 0 0 1 2.485 2.34l.005.163c0 .618-.23 1.21-.642 1.675l-7.147 7.961a2.48 2.48 0 0 1-3.554.165 2.512 2.512 0 0 1-.633-2.712l.042-.103L9.108 15H7.46c-1.393 0-2.379-1.11-2.455-2.369L5 12.473c0-.593.142-1.145.628-1.692l7.307-7.944a2.48 2.48 0 0 1 3.555-.165ZM14.43 4.164l-7.33 7.97c-.083.093-.101.214-.101.34 0 .277.19.526.46.526h4.163l.097-.009c.015 0 .03.003.046.009.181.078.264.32.186.5l-2.554 5.817a.512.512 0 0 0 .127.552.48.48 0 0 0 .69-.033l7.155-7.97a.513.513 0 0 0 .13-.34.497.497 0 0 0-.49-.502h-3.988a.355.355 0 0 1-.328-.497l2.555-5.844a.512.512 0 0 0-.127-.552.48.48 0 0 0-.69.033Z"/></symbol><symbol id="icon-eds-i-info-circle-medium" viewBox="0 0 24 24"><path d="M12 1c6.075 0 11 4.925 11 11s-4.925 11-11 11S1 18.075 1 12 5.925 1 12 1Zm0 2a9 9 0 1 0 0 18 9 9 0 0 0 0-18Zm0 7a1 1 0 0 1 1 1v5h1.5a1 1 0 0 1 0 2h-5a1 1 0 0 1 0-2H11v-4h-.5a1 1 0 0 1-.993-.883L9.5 11a1 1 0 0 1 1-1H12Zm0-4.5a1.5 1.5 0 0 1 .144 2.993L12 8.5a1.5 1.5 0 0 1 0-3Z"/></symbol><symbol id="icon-eds-i-info-filled-medium" viewBox="0 0 24 24"><path d="M12 1c6.075 0 11 4.925 11 11s-4.925 11-11 11S1 18.075 1 12 5.925 1 12 1Zm0 9h-1.5a1 1 0 0 0-1 1l.007.117A1 1 0 0 0 10.5 12h.5v4H9.5a1 1 0 0 0 0 2h5a1 1 0 0 0 0-2H13v-5a1 1 0 0 0-1-1Zm0-4.5a1.5 1.5 0 0 0 0 3l.144-.007A1.5 1.5 0 0 0 12 5.5Z"/></symbol><symbol id="icon-eds-i-journal-medium" viewBox="0 0 24 24"><path d="M18.5 1A2.5 2.5 0 0 1 21 3.5v14a2.5 2.5 0 0 1-2.5 2.5h-13a.5.5 0 1 0 0 1H20a1 1 0 0 1 0 2H5.5A2.5 2.5 0 0 1 3 20.5v-17A2.5 2.5 0 0 1 5.5 1h13ZM7 3H5.5a.5.5 0 0 0-.5.5v14.549l.016-.002c.104-.02.211-.035.32-.042L5.5 18H7V3Zm11.5 0H9v15h9.5a.5.5 0 0 0 .5-.5v-14a.5.5 0 0 0-.5-.5ZM16 5a1 1 0 0 1 1 1v4a1 1 0 0 1-1 1h-5a1 1 0 0 1-1-1V6a1 1 0 0 1 1-1h5Zm-1 2h-3v2h3V7Z"/></symbol><symbol id="icon-eds-i-mail-medium" viewBox="0 0 24 24"><path d="M20.462 3C21.875 3 23 4.184 23 5.619v12.762C23 19.816 21.875 21 20.462 21H3.538C2.125 21 1 19.816 1 18.381V5.619C1 4.184 2.125 3 3.538 3h16.924ZM21 8.158l-7.378 6.258a2.549 2.549 0 0 1-3.253-.008L3 8.16v10.222c0 .353.253.619.538.619h16.924c.285 0 .538-.266.538-.619V8.158ZM20.462 5H3.538c-.264 0-.5.228-.534.542l8.65 7.334c.2.165.492.165.684.007l8.656-7.342-.001-.025c-.044-.3-.274-.516-.531-.516Z"/></symbol><symbol id="icon-eds-i-mail-send-medium" viewBox="0 0 24 24"><path d="M20.444 5a2.562 2.562 0 0 1 2.548 2.37l.007.078.001.123v7.858A2.564 2.564 0 0 1 20.444 18H9.556A2.564 2.564 0 0 1 7 15.429l.001-7.977.007-.082A2.561 2.561 0 0 1 9.556 5h10.888ZM21 9.331l-5.46 3.51a1 1 0 0 1-1.08 0L9 9.332v6.097c0 .317.251.571.556.571h10.888a.564.564 0 0 0 .556-.571V9.33ZM20.444 7H9.556a.543.543 0 0 0-.32.105l5.763 3.706 5.766-3.706a.543.543 0 0 0-.32-.105ZM4.308 5a1 1 0 1 1 0 2H2a1 1 0 1 1 0-2h2.308Zm0 5.5a1 1 0 0 1 0 2H2a1 1 0 0 1 0-2h2.308Zm0 5.5a1 1 0 0 1 0 2H2a1 1 0 0 1 0-2h2.308Z"/></symbol><symbol id="icon-eds-i-mentions-medium" viewBox="0 0 24 24"><path d="m9.452 1.293 5.92 5.92 2.92-2.92a1 1 0 0 1 1.415 1.414l-2.92 2.92 5.92 5.92a1 1 0 0 1 0 1.415 10.371 10.371 0 0 1-10.378 2.584l.652 3.258A1 1 0 0 1 12 23H2a1 1 0 0 1-.874-1.486l4.789-8.62C4.194 9.074 4.9 4.43 8.038 1.292a1 1 0 0 1 1.414 0Zm-2.355 13.59L3.699 21h7.081l-.689-3.442a10.392 10.392 0 0 1-2.775-2.396l-.22-.28Zm1.69-11.427-.07.09a8.374 8.374 0 0 0 11.737 11.737l.089-.071L8.787 3.456Z"/></symbol><symbol id="icon-eds-i-menu-medium" viewBox="0 0 24 24"><path d="M21 4a1 1 0 0 1 0 2H3a1 1 0 1 1 0-2h18Zm-4 7a1 1 0 0 1 0 2H3a1 1 0 0 1 0-2h14Zm4 7a1 1 0 0 1 0 2H3a1 1 0 0 1 0-2h18Z"/></symbol><symbol id="icon-eds-i-metrics-medium" viewBox="0 0 24 24"><path d="M3 22a1 1 0 0 1-1-1V3a1 1 0 0 1 1-1h6a1 1 0 0 1 1 1v7h4V8a1 1 0 0 1 1-1h6a1 1 0 0 1 1 1v13a1 1 0 0 1-.883.993L21 22H3Zm17-2V9h-4v11h4Zm-6-8h-4v8h4v-8ZM8 4H4v16h4V4Z"/></symbol><symbol id="icon-eds-i-news-medium" viewBox="0 0 24 24"><path d="M17.384 3c.975 0 1.77.787 1.77 1.762v13.333c0 .462.354.846.815.899l.107.006.109-.006a.915.915 0 0 0 .809-.794l.006-.105V8.19a1 1 0 0 1 2 0v9.905A2.914 2.914 0 0 1 20.077 21H3.538a2.547 2.547 0 0 1-1.644-.601l-.147-.135A2.516 2.516 0 0 1 1 18.476V4.762C1 3.787 1.794 3 2.77 3h14.614Zm-.231 2H3v13.476c0 .11.035.216.1.304l.054.063c.101.1.24.157.384.157l13.761-.001-.026-.078a2.88 2.88 0 0 1-.115-.655l-.004-.17L17.153 5ZM14 15.021a.979.979 0 1 1 0 1.958H6a.979.979 0 1 1 0-1.958h8Zm0-8c.54 0 .979.438.979.979v4c0 .54-.438.979-.979.979H6A.979.979 0 0 1 5.021 12V8c0-.54.438-.979.979-.979h8Zm-.98 1.958H6.979v2.041h6.041V8.979Z"/></symbol><symbol id="icon-eds-i-newsletter-medium" viewBox="0 0 24 24"><path d="M21 10a1 1 0 0 1 1 1v9.5a2.5 2.5 0 0 1-2.5 2.5h-15A2.5 2.5 0 0 1 2 20.5V11a1 1 0 0 1 2 0v.439l8 4.888 8-4.889V11a1 1 0 0 1 1-1Zm-1 3.783-7.479 4.57a1 1 0 0 1-1.042 0l-7.48-4.57V20.5a.5.5 0 0 0 .501.5h15a.5.5 0 0 0 .5-.5v-6.717ZM15 9a1 1 0 0 1 0 2H9a1 1 0 0 1 0-2h6Zm2.5-8A2.5 2.5 0 0 1 20 3.5V9a1 1 0 0 1-2 0V3.5a.5.5 0 0 0-.5-.5h-11a.5.5 0 0 0-.5.5V9a1 1 0 1 1-2 0V3.5A2.5 2.5 0 0 1 6.5 1h11ZM15 5a1 1 0 0 1 0 2H9a1 1 0 1 1 0-2h6Z"/></symbol><symbol id="icon-eds-i-notifcation-medium" viewBox="0 0 24 24"><path d="M14 20a1 1 0 0 1 0 2h-4a1 1 0 0 1 0-2h4ZM3 18l-.133-.007c-1.156-.124-1.156-1.862 0-1.986l.3-.012C4.32 15.923 5 15.107 5 14V9.5C5 5.368 8.014 2 12 2s7 3.368 7 7.5V14c0 1.107.68 1.923 1.832 1.995l.301.012c1.156.124 1.156 1.862 0 1.986L21 18H3Zm9-14C9.17 4 7 6.426 7 9.5V14c0 .671-.146 1.303-.416 1.858L6.51 16h10.979l-.073-.142a4.192 4.192 0 0 1-.412-1.658L17 14V9.5C17 6.426 14.83 4 12 4Z"/></symbol><symbol id="icon-eds-i-publish-medium" viewBox="0 0 24 24"><g><path d="M16.296 1.291A1 1 0 0 0 15.591 1H5.545A2.542 2.542 0 0 0 3 3.538V13a1 1 0 1 0 2 0V3.538l.007-.087A.543.543 0 0 1 5.545 3h9.633L20 7.8v12.662a.534.534 0 0 1-.158.379.548.548 0 0 1-.387.159H11a1 1 0 1 0 0 2h8.455c.674 0 1.32-.267 1.798-.742A2.534 2.534 0 0 0 22 20.462V7.385a1 1 0 0 0-.294-.709l-5.41-5.385Z"/><path d="M10.762 16.647a1 1 0 0 0-1.525-1.294l-4.472 5.271-2.153-1.665a1 1 0 1 0-1.224 1.582l2.91 2.25a1 1 0 0 0 1.374-.144l5.09-6ZM16 10a1 1 0 1 1 0 2H8a1 1 0 1 1 0-2h8ZM12 7a1 1 0 0 0-1-1H8a1 1 0 1 0 0 2h3a1 1 0 0 0 1-1Z"/></g></symbol><symbol id="icon-eds-i-refresh-medium" viewBox="0 0 24 24"><g><path d="M7.831 5.636H6.032A8.76 8.76 0 0 1 9 3.631 8.549 8.549 0 0 1 12.232 3c.603 0 1.192.063 1.76.182C17.979 4.017 21 7.632 21 12a1 1 0 1 0 2 0c0-5.296-3.674-9.746-8.591-10.776A10.61 10.61 0 0 0 5 3.851V2.805a1 1 0 0 0-.987-1H4a1 1 0 0 0-1 1v3.831a1 1 0 0 0 1 1h3.831a1 1 0 0 0 .013-2h-.013ZM17.968 18.364c-1.59 1.632-3.784 2.636-6.2 2.636C6.948 21 3 16.993 3 12a1 1 0 1 0-2 0c0 6.053 4.799 11 10.768 11 2.788 0 5.324-1.082 7.232-2.85v1.045a1 1 0 1 0 2 0v-3.831a1 1 0 0 0-1-1h-3.831a1 1 0 0 0 0 2h1.799Z"/></g></symbol><symbol id="icon-eds-i-search-medium" viewBox="0 0 24 24"><path d="M11 1c5.523 0 10 4.477 10 10 0 2.4-.846 4.604-2.256 6.328l3.963 3.965a1 1 0 0 1-1.414 1.414l-3.965-3.963A9.959 9.959 0 0 1 11 21C5.477 21 1 16.523 1 11S5.477 1 11 1Zm0 2a8 8 0 1 0 0 16 8 8 0 0 0 0-16Z"/></symbol><symbol id="icon-eds-i-settings-medium" viewBox="0 0 24 24"><path d="M11.382 1h1.24a2.508 2.508 0 0 1 2.334 1.63l.523 1.378 1.59.933 1.444-.224c.954-.132 1.89.3 2.422 1.101l.095.155.598 1.066a2.56 2.56 0 0 1-.195 2.848l-.894 1.161v1.896l.92 1.163c.6.768.707 1.812.295 2.674l-.09.17-.606 1.08a2.504 2.504 0 0 1-2.531 1.25l-1.428-.223-1.589.932-.523 1.378a2.512 2.512 0 0 1-2.155 1.625L12.65 23h-1.27a2.508 2.508 0 0 1-2.334-1.63l-.524-1.379-1.59-.933-1.443.225c-.954.132-1.89-.3-2.422-1.101l-.095-.155-.598-1.066a2.56 2.56 0 0 1 .195-2.847l.891-1.161v-1.898l-.919-1.162a2.562 2.562 0 0 1-.295-2.674l.09-.17.606-1.08a2.504 2.504 0 0 1 2.531-1.25l1.43.223 1.618-.938.524-1.375.07-.167A2.507 2.507 0 0 1 11.382 1Zm.003 2a.509.509 0 0 0-.47.338l-.65 1.71a1 1 0 0 1-.434.51L7.6 6.85a1 1 0 0 1-.655.123l-1.762-.275a.497.497 0 0 0-.498.252l-.61 1.088a.562.562 0 0 0 .04.619l1.13 1.43a1 1 0 0 1 .216.62v2.585a1 1 0 0 1-.207.61L4.15 15.339a.568.568 0 0 0-.036.634l.601 1.072a.494.494 0 0 0 .484.26l1.78-.278a1 1 0 0 1 .66.126l2.2 1.292a1 1 0 0 1 .43.507l.648 1.71a.508.508 0 0 0 .467.338h1.263a.51.51 0 0 0 .47-.34l.65-1.708a1 1 0 0 1 .428-.507l2.201-1.292a1 1 0 0 1 .66-.126l1.763.275a.497.497 0 0 0 .498-.252l.61-1.088a.562.562 0 0 0-.04-.619l-1.13-1.43a1 1 0 0 1-.216-.62v-2.585a1 1 0 0 1 .207-.61l1.105-1.437a.568.568 0 0 0 .037-.634l-.601-1.072a.494.494 0 0 0-.484-.26l-1.78.278a1 1 0 0 1-.66-.126l-2.2-1.292a1 1 0 0 1-.43-.507l-.649-1.71A.508.508 0 0 0 12.62 3h-1.234ZM12 8a4 4 0 1 1 0 8 4 4 0 0 1 0-8Zm0 2a2 2 0 1 0 0 4 2 2 0 0 0 0-4Z"/></symbol><symbol id="icon-eds-i-shipping-medium" viewBox="0 0 24 24"><path d="M16.515 2c1.406 0 2.706.728 3.352 1.902l2.02 3.635.02.042.036.089.031.105.012.058.01.073.004.075v11.577c0 .64-.244 1.255-.683 1.713a2.356 2.356 0 0 1-1.701.731H4.386a2.356 2.356 0 0 1-1.702-.731 2.476 2.476 0 0 1-.683-1.713V7.948c.01-.217.083-.43.22-.6L4.2 3.905C4.833 2.755 6.089 2.032 7.486 2h9.029ZM20 9H4v10.556a.49.49 0 0 0 .075.26l.053.07a.356.356 0 0 0 .257.114h15.23c.094 0 .186-.04.258-.115a.477.477 0 0 0 .127-.33V9Zm-2 7.5a1 1 0 0 1 0 2h-4a1 1 0 0 1 0-2h4ZM16.514 4H13v3h6.3l-1.183-2.13c-.288-.522-.908-.87-1.603-.87ZM11 3.999H7.51c-.679.017-1.277.36-1.566.887L4.728 7H11V3.999Z"/></symbol><symbol id="icon-eds-i-step-guide-medium" viewBox="0 0 24 24"><path d="M11.394 9.447a1 1 0 1 0-1.788-.894l-.88 1.759-.019-.02a1 1 0 1 0-1.414 1.415l1 1a1 1 0 0 0 1.601-.26l1.5-3ZM12 11a1 1 0 0 1 1-1h3a1 1 0 1 1 0 2h-3a1 1 0 0 1-1-1ZM12 17a1 1 0 0 1 1-1h3a1 1 0 1 1 0 2h-3a1 1 0 0 1-1-1ZM10.947 14.105a1 1 0 0 1 .447 1.342l-1.5 3a1 1 0 0 1-1.601.26l-1-1a1 1 0 1 1 1.414-1.414l.02.019.879-1.76a1 1 0 0 1 1.341-.447Z"/><path d="M5.545 1A2.542 2.542 0 0 0 3 3.538v16.924A2.542 2.542 0 0 0 5.545 23h12.91A2.542 2.542 0 0 0 21 20.462V7.5a1 1 0 0 0-.293-.707l-5.5-5.5A1 1 0 0 0 14.5 1H5.545ZM5 3.538C5 3.245 5.24 3 5.545 3h8.54L19 7.914v12.547c0 .294-.24.539-.546.539H5.545A.542.542 0 0 1 5 20.462V3.538Z" clip-rule="evenodd"/></symbol><symbol id="icon-eds-i-submission-medium" viewBox="0 0 24 24"><g><path d="M5 3.538C5 3.245 5.24 3 5.545 3h9.633L20 7.8v12.662a.535.535 0 0 1-.158.379.549.549 0 0 1-.387.159H6a1 1 0 0 1-1-1v-2.5a1 1 0 1 0-2 0V20a3 3 0 0 0 3 3h13.455c.673 0 1.32-.266 1.798-.742A2.535 2.535 0 0 0 22 20.462V7.385a1 1 0 0 0-.294-.709l-5.41-5.385A1 1 0 0 0 15.591 1H5.545A2.542 2.542 0 0 0 3 3.538V7a1 1 0 0 0 2 0V3.538Z"/><path d="m13.707 13.707-4 4a1 1 0 0 1-1.414 0l-.083-.094a1 1 0 0 1 .083-1.32L10.585 14 2 14a1 1 0 1 1 0-2l8.583.001-2.29-2.294a1 1 0 0 1 1.414-1.414l4.037 4.04.043.05.043.06.059.098.03.063.031.085.03.113.017.122L14 13l-.004.087-.017.118-.013.056-.034.104-.049.105-.048.081-.07.093-.058.063Z"/></g></symbol><symbol id="icon-eds-i-table-1-medium" viewBox="0 0 24 24"><path d="M4.385 22a2.56 2.56 0 0 1-1.14-.279C2.485 21.341 2 20.614 2 19.615V4.385c0-.315.067-.716.279-1.14C2.659 2.485 3.386 2 4.385 2h15.23c.315 0 .716.067 1.14.279.76.38 1.245 1.107 1.245 2.106v15.23c0 .315-.067.716-.279 1.14-.38.76-1.107 1.245-2.106 1.245H4.385ZM4 19.615c0 .213.034.265.14.317a.71.71 0 0 0 .245.068H8v-4H4v3.615ZM20 16H10v4h9.615c.213 0 .265-.034.317-.14a.71.71 0 0 0 .068-.245V16Zm0-2v-4H10v4h10ZM4 14h4v-4H4v4ZM19.615 4H10v4h10V4.385c0-.213-.034-.265-.14-.317A.71.71 0 0 0 19.615 4ZM8 4H4.385l-.082.002c-.146.01-.19.047-.235.138A.71.71 0 0 0 4 4.385V8h4V4Z"/></symbol><symbol id="icon-eds-i-table-2-medium" viewBox="0 0 24 24"><path d="M4.384 22A2.384 2.384 0 0 1 2 19.616V4.384A2.384 2.384 0 0 1 4.384 2h15.232A2.384 2.384 0 0 1 22 4.384v15.232A2.384 2.384 0 0 1 19.616 22H4.384ZM10 15H4v4.616c0 .212.172.384.384.384H10v-5Zm5 0h-3v5h3v-5Zm5 0h-3v5h2.616a.384.384 0 0 0 .384-.384V15ZM10 9H4v4h6V9Zm5 0h-3v4h3V9Zm5 0h-3v4h3V9Zm-.384-5H4.384A.384.384 0 0 0 4 4.384V7h16V4.384A.384.384 0 0 0 19.616 4Z"/></symbol><symbol id="icon-eds-i-tag-medium" viewBox="0 0 24 24"><path d="m12.621 1.998.127.004L20.496 2a1.5 1.5 0 0 1 1.497 1.355L22 3.5l-.005 7.669c.038.456-.133.905-.447 1.206l-9.02 9.018a2.075 2.075 0 0 1-2.932 0l-6.99-6.99a2.075 2.075 0 0 1 .001-2.933L11.61 2.47c.246-.258.573-.418.881-.46l.131-.011Zm.286 2-8.885 8.886a.075.075 0 0 0 0 .106l6.987 6.988c.03.03.077.03.106 0l8.883-8.883L19.999 4l-7.092-.002ZM16 6.5a1.5 1.5 0 0 1 .144 2.993L16 9.5a1.5 1.5 0 0 1 0-3Z"/></symbol><symbol id="icon-eds-i-trash-medium" viewBox="0 0 24 24"><path d="M12 1c2.717 0 4.913 2.232 4.997 5H21a1 1 0 0 1 0 2h-1v12.5c0 1.389-1.152 2.5-2.556 2.5H6.556C5.152 23 4 21.889 4 20.5V8H3a1 1 0 1 1 0-2h4.003l.001-.051C7.114 3.205 9.3 1 12 1Zm6 7H6v12.5c0 .238.19.448.454.492l.102.008h10.888c.315 0 .556-.232.556-.5V8Zm-4 3a1 1 0 0 1 1 1v6.005a1 1 0 0 1-2 0V12a1 1 0 0 1 1-1Zm-4 0a1 1 0 0 1 1 1v6a1 1 0 0 1-2 0v-6a1 1 0 0 1 1-1Zm2-8c-1.595 0-2.914 1.32-2.996 3h5.991v-.02C14.903 4.31 13.589 3 12 3Z"/></symbol><symbol id="icon-eds-i-user-account-medium" viewBox="0 0 24 24"><path d="M12 1c6.075 0 11 4.925 11 11s-4.925 11-11 11S1 18.075 1 12 5.925 1 12 1Zm0 16c-1.806 0-3.52.994-4.664 2.698A8.947 8.947 0 0 0 12 21a8.958 8.958 0 0 0 4.664-1.301C15.52 17.994 13.806 17 12 17Zm0-14a9 9 0 0 0-6.25 15.476C7.253 16.304 9.54 15 12 15s4.747 1.304 6.25 3.475A9 9 0 0 0 12 3Zm0 3a4 4 0 1 1 0 8 4 4 0 0 1 0-8Zm0 2a2 2 0 1 0 0 4 2 2 0 0 0 0-4Z"/></symbol><symbol id="icon-eds-i-user-add-medium" viewBox="0 0 24 24"><path d="M9 1a5 5 0 1 1 0 10A5 5 0 0 1 9 1Zm0 2a3 3 0 1 0 0 6 3 3 0 0 0 0-6Zm9 10a1 1 0 0 1 1 1v3h3a1 1 0 0 1 0 2h-3v3a1 1 0 0 1-2 0v-3h-3a1 1 0 0 1 0-2h3v-3a1 1 0 0 1 1-1Zm-5.545-.15a1 1 0 1 1-.91 1.78 5.713 5.713 0 0 0-5.705.282c-1.67 1.068-2.728 2.927-2.832 4.956L3.004 20 11.5 20a1 1 0 0 1 .993.883L12.5 21a1 1 0 0 1-1 1H2a1 1 0 0 1-1-1v-.876c.028-2.812 1.446-5.416 3.763-6.897a7.713 7.713 0 0 1 7.692-.378Z"/></symbol><symbol id="icon-eds-i-user-assign-medium" viewBox="0 0 24 24"><path d="M16.226 13.298a1 1 0 0 1 1.414-.01l.084.093a1 1 0 0 1-.073 1.32L15.39 17H22a1 1 0 0 1 0 2h-6.611l2.262 2.298a1 1 0 0 1-1.425 1.404l-3.939-4a1 1 0 0 1 0-1.404l3.94-4Zm-3.771-.449a1 1 0 1 1-.91 1.781 5.713 5.713 0 0 0-5.705.282c-1.67 1.068-2.728 2.927-2.832 4.956L3.004 20 10.5 20a1 1 0 0 1 .993.883L11.5 21a1 1 0 0 1-1 1H2a1 1 0 0 1-1-1v-.876c.028-2.812 1.446-5.416 3.763-6.897a7.713 7.713 0 0 1 7.692-.378ZM9 1a5 5 0 1 1 0 10A5 5 0 0 1 9 1Zm0 2a3 3 0 1 0 0 6 3 3 0 0 0 0-6Z"/></symbol><symbol id="icon-eds-i-user-block-medium" viewBox="0 0 24 24"><path d="M9 1a5 5 0 1 1 0 10A5 5 0 0 1 9 1Zm0 2a3 3 0 1 0 0 6 3 3 0 0 0 0-6Zm9 10a5 5 0 1 1 0 10 5 5 0 0 1 0-10Zm-5.545-.15a1 1 0 1 1-.91 1.78 5.713 5.713 0 0 0-5.705.282c-1.67 1.068-2.728 2.927-2.832 4.956L3.004 20 11.5 20a1 1 0 0 1 .993.883L12.5 21a1 1 0 0 1-1 1H2a1 1 0 0 1-1-1v-.876c.028-2.812 1.446-5.416 3.763-6.897a7.713 7.713 0 0 1 7.692-.378ZM15 18a3 3 0 0 0 4.294 2.707l-4.001-4c-.188.391-.293.83-.293 1.293Zm3-3c-.463 0-.902.105-1.294.293l4.001 4A3 3 0 0 0 18 15Z"/></symbol><symbol id="icon-eds-i-user-check-medium" viewBox="0 0 24 24"><path d="M9 1a5 5 0 1 1 0 10A5 5 0 0 1 9 1Zm0 2a3 3 0 1 0 0 6 3 3 0 0 0 0-6Zm13.647 12.237a1 1 0 0 1 .116 1.41l-5.091 6a1 1 0 0 1-1.375.144l-2.909-2.25a1 1 0 1 1 1.224-1.582l2.153 1.665 4.472-5.271a1 1 0 0 1 1.41-.116Zm-8.139-.977c.22.214.428.44.622.678a1 1 0 1 1-1.548 1.266 6.025 6.025 0 0 0-1.795-1.49.86.86 0 0 1-.163-.048l-.079-.036a5.721 5.721 0 0 0-2.62-.63l-.194.006c-2.76.134-5.022 2.177-5.592 4.864l-.035.175-.035.213c-.03.201-.05.405-.06.61L3.003 20 10 20a1 1 0 0 1 .993.883L11 21a1 1 0 0 1-1 1H2a1 1 0 0 1-1-1v-.876l.005-.223.02-.356.02-.222.03-.248.022-.15c.02-.133.044-.265.071-.397.44-2.178 1.725-4.105 3.595-5.301a7.75 7.75 0 0 1 3.755-1.215l.12-.004a7.908 7.908 0 0 1 5.87 2.252Z"/></symbol><symbol id="icon-eds-i-user-delete-medium" viewBox="0 0 24 24"><path d="M9 1a5 5 0 1 1 0 10A5 5 0 0 1 9 1Zm0 2a3 3 0 1 0 0 6 3 3 0 0 0 0-6ZM4.763 13.227a7.713 7.713 0 0 1 7.692-.378 1 1 0 1 1-.91 1.781 5.713 5.713 0 0 0-5.705.282c-1.67 1.068-2.728 2.927-2.832 4.956L3.004 20H11.5a1 1 0 0 1 .993.883L12.5 21a1 1 0 0 1-1 1H2a1 1 0 0 1-1-1v-.876c.028-2.812 1.446-5.416 3.763-6.897Zm11.421 1.543 2.554 2.553 2.555-2.553a1 1 0 0 1 1.414 1.414l-2.554 2.554 2.554 2.555a1 1 0 0 1-1.414 1.414l-2.555-2.554-2.554 2.554a1 1 0 0 1-1.414-1.414l2.553-2.555-2.553-2.554a1 1 0 0 1 1.414-1.414Z"/></symbol><symbol id="icon-eds-i-user-edit-medium" viewBox="0 0 24 24"><path d="m19.876 10.77 2.831 2.83a1 1 0 0 1 0 1.415l-7.246 7.246a1 1 0 0 1-.572.284l-3.277.446a1 1 0 0 1-1.125-1.13l.461-3.277a1 1 0 0 1 .283-.567l7.23-7.246a1 1 0 0 1 1.415-.001Zm-7.421 2.08a1 1 0 1 1-.91 1.78 5.713 5.713 0 0 0-5.705.282c-1.67 1.068-2.728 2.927-2.832 4.956L3.004 20 7.5 20a1 1 0 0 1 .993.883L8.5 21a1 1 0 0 1-1 1H2a1 1 0 0 1-1-1v-.876c.028-2.812 1.446-5.416 3.763-6.897a7.713 7.713 0 0 1 7.692-.378Zm6.715.042-6.29 6.3-.23 1.639 1.633-.222 6.302-6.302-1.415-1.415ZM9 1a5 5 0 1 1 0 10A5 5 0 0 1 9 1Zm0 2a3 3 0 1 0 0 6 3 3 0 0 0 0-6Z"/></symbol><symbol id="icon-eds-i-user-linked-medium" viewBox="0 0 24 24"><path d="M15.65 6c.31 0 .706.066 1.122.274C17.522 6.65 18 7.366 18 8.35v12.3c0 .31-.066.706-.274 1.122-.375.75-1.092 1.228-2.076 1.228H3.35a2.52 2.52 0 0 1-1.122-.274C1.478 22.35 1 21.634 1 20.65V8.35c0-.31.066-.706.274-1.122C1.65 6.478 2.366 6 3.35 6h12.3Zm0 2-12.376.002c-.134.007-.17.04-.21.12A.672.672 0 0 0 3 8.35v12.3c0 .198.028.24.122.287.09.044.2.063.228.063h.887c.788-2.269 2.814-3.5 5.263-3.5 2.45 0 4.475 1.231 5.263 3.5h.887c.198 0 .24-.028.287-.122.044-.09.063-.2.063-.228V8.35c0-.198-.028-.24-.122-.287A.672.672 0 0 0 15.65 8ZM9.5 19.5c-1.36 0-2.447.51-3.06 1.5h6.12c-.613-.99-1.7-1.5-3.06-1.5ZM20.65 1A2.35 2.35 0 0 1 23 3.348V15.65A2.35 2.35 0 0 1 20.65 18H20a1 1 0 0 1 0-2h.65a.35.35 0 0 0 .35-.35V3.348A.35.35 0 0 0 20.65 3H8.35a.35.35 0 0 0-.35.348V4a1 1 0 1 1-2 0v-.652A2.35 2.35 0 0 1 8.35 1h12.3ZM9.5 10a3.5 3.5 0 1 1 0 7 3.5 3.5 0 0 1 0-7Zm0 2a1.5 1.5 0 1 0 0 3 1.5 1.5 0 0 0 0-3Z"/></symbol><symbol id="icon-eds-i-user-multiple-medium" viewBox="0 0 24 24"><path d="M9 1a5 5 0 1 1 0 10A5 5 0 0 1 9 1Zm6 0a5 5 0 0 1 0 10 1 1 0 0 1-.117-1.993L15 9a3 3 0 0 0 0-6 1 1 0 0 1 0-2ZM9 3a3 3 0 1 0 0 6 3 3 0 0 0 0-6Zm8.857 9.545a7.99 7.99 0 0 1 2.651 1.715A8.31 8.31 0 0 1 23 20.134V21a1 1 0 0 1-1 1h-3a1 1 0 0 1 0-2h1.995l-.005-.153a6.307 6.307 0 0 0-1.673-3.945l-.204-.209a5.99 5.99 0 0 0-1.988-1.287 1 1 0 1 1 .732-1.861Zm-3.349 1.715A8.31 8.31 0 0 1 17 20.134V21a1 1 0 0 1-1 1H2a1 1 0 0 1-1-1v-.877c.044-4.343 3.387-7.908 7.638-8.115a7.908 7.908 0 0 1 5.87 2.252ZM9.016 14l-.285.006c-3.104.15-5.58 2.718-5.725 5.9L3.004 20h11.991l-.005-.153a6.307 6.307 0 0 0-1.673-3.945l-.204-.209A5.924 5.924 0 0 0 9.3 14.008L9.016 14Z"/></symbol><symbol id="icon-eds-i-user-notify-medium" viewBox="0 0 24 24"><path d="M9 1a5 5 0 1 1 0 10A5 5 0 0 1 9 1Zm0 2a3 3 0 1 0 0 6 3 3 0 0 0 0-6Zm10 18v1a1 1 0 0 1-2 0v-1h-3a1 1 0 0 1 0-2v-2.818C14 13.885 15.777 12 18 12s4 1.885 4 4.182V19a1 1 0 0 1 0 2h-3Zm-6.545-8.15a1 1 0 1 1-.91 1.78 5.713 5.713 0 0 0-5.705.282c-1.67 1.068-2.728 2.927-2.832 4.956L3.004 20 11.5 20a1 1 0 0 1 .993.883L12.5 21a1 1 0 0 1-1 1H2a1 1 0 0 1-1-1v-.876c.028-2.812 1.446-5.416 3.763-6.897a7.713 7.713 0 0 1 7.692-.378ZM18 14c-1.091 0-2 .964-2 2.182V19h4v-2.818c0-1.165-.832-2.098-1.859-2.177L18 14Z"/></symbol><symbol id="icon-eds-i-user-remove-medium" viewBox="0 0 24 24"><path d="M9 1a5 5 0 1 1 0 10A5 5 0 0 1 9 1Zm0 2a3 3 0 1 0 0 6 3 3 0 0 0 0-6Zm3.455 9.85a1 1 0 1 1-.91 1.78 5.713 5.713 0 0 0-5.705.282c-1.67 1.068-2.728 2.927-2.832 4.956L3.004 20 11.5 20a1 1 0 0 1 .993.883L12.5 21a1 1 0 0 1-1 1H2a1 1 0 0 1-1-1v-.876c.028-2.812 1.446-5.416 3.763-6.897a7.713 7.713 0 0 1 7.692-.378ZM22 17a1 1 0 0 1 0 2h-8a1 1 0 0 1 0-2h8Z"/></symbol><symbol id="icon-eds-i-user-single-medium" viewBox="0 0 24 24"><path d="M12 1a5 5 0 1 1 0 10 5 5 0 0 1 0-10Zm0 2a3 3 0 1 0 0 6 3 3 0 0 0 0-6Zm-.406 9.008a8.965 8.965 0 0 1 6.596 2.494A9.161 9.161 0 0 1 21 21.025V22a1 1 0 0 1-1 1H4a1 1 0 0 1-1-1v-.985c.05-4.825 3.815-8.777 8.594-9.007Zm.39 1.992-.299.006c-3.63.175-6.518 3.127-6.678 6.775L5 21h13.998l-.009-.268a7.157 7.157 0 0 0-1.97-4.573l-.214-.213A6.967 6.967 0 0 0 11.984 14Z"/></symbol><symbol id="icon-eds-i-warning-circle-medium" viewBox="0 0 24 24"><path d="M12 1c6.075 0 11 4.925 11 11s-4.925 11-11 11S1 18.075 1 12 5.925 1 12 1Zm0 2a9 9 0 1 0 0 18 9 9 0 0 0 0-18Zm0 11.5a1.5 1.5 0 0 1 .144 2.993L12 17.5a1.5 1.5 0 0 1 0-3ZM12 6a1 1 0 0 1 1 1v5a1 1 0 0 1-2 0V7a1 1 0 0 1 1-1Z"/></symbol><symbol id="icon-eds-i-warning-filled-medium" viewBox="0 0 24 24"><path d="M12 1c6.075 0 11 4.925 11 11s-4.925 11-11 11S1 18.075 1 12 5.925 1 12 1Zm0 13.5a1.5 1.5 0 0 0 0 3l.144-.007A1.5 1.5 0 0 0 12 14.5ZM12 6a1 1 0 0 0-1 1v5a1 1 0 0 0 2 0V7a1 1 0 0 0-1-1Z"/></symbol><symbol id="icon-chevron-left-medium" viewBox="0 0 24 24"><path d="M15.7194 3.3054C15.3358 2.90809 14.7027 2.89699 14.3054 3.28061L6.54342 10.7757C6.19804 11.09 6 11.5335 6 12C6 12.4665 6.19804 12.91 6.5218 13.204L14.3054 20.7194C14.7027 21.103 15.3358 21.0919 15.7194 20.6946C16.103 20.2973 16.0919 19.6642 15.6946 19.2806L8.155 12L15.6946 4.71939C16.0614 4.36528 16.099 3.79863 15.8009 3.40105L15.7194 3.3054Z"/></symbol><symbol id="icon-chevron-right-medium" viewBox="0 0 24 24"><path d="M8.28061 3.3054C8.66423 2.90809 9.29729 2.89699 9.6946 3.28061L17.4566 10.7757C17.802 11.09 18 11.5335 18 12C18 12.4665 17.802 12.91 17.4782 13.204L9.6946 20.7194C9.29729 21.103 8.66423 21.0919 8.28061 20.6946C7.89699 20.2973 7.90809 19.6642 8.3054 19.2806L15.845 12L8.3054 4.71939C7.93865 4.36528 7.90098 3.79863 8.19908 3.40105L8.28061 3.3054Z"/></symbol><symbol id="icon-eds-alerts" viewBox="0 0 32 32"><path d="M28 12.667c.736 0 1.333.597 1.333 1.333v13.333A3.333 3.333 0 0 1 26 30.667H6a3.333 3.333 0 0 1-3.333-3.334V14a1.333 1.333 0 1 1 2.666 0v1.252L16 21.769l10.667-6.518V14c0-.736.597-1.333 1.333-1.333Zm-1.333 5.71-9.972 6.094c-.427.26-.963.26-1.39 0l-9.972-6.094v8.956c0 .368.299.667.667.667h20a.667.667 0 0 0 .667-.667v-8.956ZM19.333 12a1.333 1.333 0 1 1 0 2.667h-6.666a1.333 1.333 0 1 1 0-2.667h6.666Zm4-10.667a3.333 3.333 0 0 1 3.334 3.334v6.666a1.333 1.333 0 1 1-2.667 0V4.667A.667.667 0 0 0 23.333 4H8.667A.667.667 0 0 0 8 4.667v6.666a1.333 1.333 0 1 1-2.667 0V4.667a3.333 3.333 0 0 1 3.334-3.334h14.666Zm-4 5.334a1.333 1.333 0 0 1 0 2.666h-6.666a1.333 1.333 0 1 1 0-2.666h6.666Z"/></symbol><symbol id="icon-eds-arrow-up" viewBox="0 0 24 24"><path fill-rule="evenodd" d="m13.002 7.408 4.88 4.88a.99.99 0 0 0 1.32.08l.09-.08c.39-.39.39-1.03 0-1.42l-6.58-6.58a1.01 1.01 0 0 0-1.42 0l-6.58 6.58a1 1 0 0 0-.09 1.32l.08.1a1 1 0 0 0 1.42-.01l4.88-4.87v11.59a.99.99 0 0 0 .88.99l.12.01c.55 0 1-.45 1-1V7.408z" class="layer"/></symbol><symbol id="icon-eds-checklist" viewBox="0 0 32 32"><path d="M19.2 1.333a3.468 3.468 0 0 1 3.381 2.699L24.667 4C26.515 4 28 5.52 28 7.38v19.906c0 1.86-1.485 3.38-3.333 3.38H7.333c-1.848 0-3.333-1.52-3.333-3.38V7.38C4 5.52 5.485 4 7.333 4h2.093A3.468 3.468 0 0 1 12.8 1.333h6.4ZM9.426 6.667H7.333c-.36 0-.666.312-.666.713v19.906c0 .401.305.714.666.714h17.334c.36 0 .666-.313.666-.714V7.38c0-.4-.305-.713-.646-.714l-2.121.033A3.468 3.468 0 0 1 19.2 9.333h-6.4a3.468 3.468 0 0 1-3.374-2.666Zm12.715 5.606c.586.446.7 1.283.253 1.868l-7.111 9.334a1.333 1.333 0 0 1-1.792.306l-3.556-2.333a1.333 1.333 0 1 1 1.463-2.23l2.517 1.651 6.358-8.344a1.333 1.333 0 0 1 1.868-.252ZM19.2 4h-6.4a.8.8 0 0 0-.8.8v1.067a.8.8 0 0 0 .8.8h6.4a.8.8 0 0 0 .8-.8V4.8a.8.8 0 0 0-.8-.8Z"/></symbol><symbol id="icon-eds-citation" viewBox="0 0 36 36"><path d="M23.25 1.5a1.5 1.5 0 0 1 1.06.44l8.25 8.25a1.5 1.5 0 0 1 .44 1.06v19.5c0 2.105-1.645 3.75-3.75 3.75H18a1.5 1.5 0 0 1 0-3h11.25c.448 0 .75-.302.75-.75V11.873L22.628 4.5H8.31a.811.811 0 0 0-.8.68l-.011.13V16.5a1.5 1.5 0 0 1-3 0V5.31A3.81 3.81 0 0 1 8.31 1.5h14.94ZM8.223 20.358a.984.984 0 0 1-.192 1.378l-.048.034c-.54.36-.942.676-1.206.951-.59.614-.885 1.395-.885 2.343.115-.028.288-.042.518-.042.662 0 1.26.237 1.791.711.533.474.799 1.074.799 1.799 0 .753-.259 1.352-.777 1.799-.518.446-1.151.669-1.9.669-1.006 0-1.812-.293-2.417-.878C3.302 28.536 3 27.657 3 26.486c0-1.115.165-2.085.496-2.907.331-.823.734-1.513 1.209-2.071.475-.558.971-.997 1.49-1.318a6.01 6.01 0 0 1 .347-.2 1.321 1.321 0 0 1 1.681.368Zm7.5 0a.984.984 0 0 1-.192 1.378l-.048.034c-.54.36-.942.676-1.206.951-.59.614-.885 1.395-.885 2.343.115-.028.288-.042.518-.042.662 0 1.26.237 1.791.711.533.474.799 1.074.799 1.799 0 .753-.259 1.352-.777 1.799-.518.446-1.151.669-1.9.669-1.006 0-1.812-.293-2.417-.878-.604-.586-.906-1.465-.906-2.636 0-1.115.165-2.085.496-2.907.331-.823.734-1.513 1.209-2.071.475-.558.971-.997 1.49-1.318a6.01 6.01 0 0 1 .347-.2 1.321 1.321 0 0 1 1.681.368Z"/></symbol><symbol id="icon-eds-i-access-indicator" viewBox="0 0 16 16"><circle cx="4.5" cy="11.5" r="3.5" style="fill:currentColor"/><path fill-rule="evenodd" d="M4 3v3a1 1 0 0 1-2 0V2.923C2 1.875 2.84 1 3.909 1h5.909a1 1 0 0 1 .713.298l3.181 3.231a1 1 0 0 1 .288.702v7.846c0 .505-.197.993-.554 1.354a1.902 1.902 0 0 1-1.355.569H10a1 1 0 1 1 0-2h2V5.64L9.4 3H4Z" clip-rule="evenodd" style="fill:#222"/></symbol><symbol id="icon-eds-i-github-medium" viewBox="0 0 24 24"><path d="M 11.964844 0 C 5.347656 0 0 5.269531 0 11.792969 C 0 17.003906 3.425781 21.417969 8.179688 22.976562 C 8.773438 23.09375 8.992188 22.722656 8.992188 22.410156 C 8.992188 22.136719 8.972656 21.203125 8.972656 20.226562 C 5.644531 20.929688 4.953125 18.820312 4.953125 18.820312 C 4.417969 17.453125 3.625 17.101562 3.625 17.101562 C 2.535156 16.378906 3.703125 16.378906 3.703125 16.378906 C 4.914062 16.457031 5.546875 17.589844 5.546875 17.589844 C 6.617188 19.386719 8.339844 18.878906 9.03125 18.566406 C 9.132812 17.804688 9.449219 17.277344 9.785156 16.984375 C 7.132812 16.710938 4.339844 15.695312 4.339844 11.167969 C 4.339844 9.878906 4.8125 8.824219 5.566406 8.003906 C 5.445312 7.710938 5.03125 6.5 5.683594 4.878906 C 5.683594 4.878906 6.695312 4.566406 8.972656 6.089844 C 9.949219 5.832031 10.953125 5.703125 11.964844 5.699219 C 12.972656 5.699219 14.003906 5.835938 14.957031 6.089844 C 17.234375 4.566406 18.242188 4.878906 18.242188 4.878906 C 18.898438 6.5 18.480469 7.710938 18.363281 8.003906 C 19.136719 8.824219 19.589844 9.878906 19.589844 11.167969 C 19.589844 15.695312 16.796875 16.691406 14.125 16.984375 C 14.558594 17.355469 14.933594 18.058594 14.933594 19.171875 C 14.933594 20.753906 14.914062 22.019531 14.914062 22.410156 C 14.914062 22.722656 15.132812 23.09375 15.726562 22.976562 C 20.480469 21.414062 23.910156 17.003906 23.910156 11.792969 C 23.929688 5.269531 18.558594 0 11.964844 0 Z M 11.964844 0 "/></symbol><symbol id="icon-eds-i-limited-access" viewBox="0 0 16 16"><path fill-rule="evenodd" d="M4 3v3a1 1 0 0 1-2 0V2.923C2 1.875 2.84 1 3.909 1h5.909a1 1 0 0 1 .713.298l3.181 3.231a1 1 0 0 1 .288.702V6a1 1 0 1 1-2 0v-.36L9.4 3H4ZM3 8a1 1 0 0 1 1 1v1a1 1 0 1 1-2 0V9a1 1 0 0 1 1-1Zm10 0a1 1 0 0 1 1 1v1a1 1 0 1 1-2 0V9a1 1 0 0 1 1-1Zm-3.5 6a1 1 0 0 1-1 1h-1a1 1 0 1 1 0-2h1a1 1 0 0 1 1 1Zm2.441-1a1 1 0 0 1 2 0c0 .73-.246 1.306-.706 1.664a1.61 1.61 0 0 1-.876.334l-.032.002H11.5a1 1 0 1 1 0-2h.441ZM4 13a1 1 0 0 0-2 0c0 .73.247 1.306.706 1.664a1.609 1.609 0 0 0 .876.334l.032.002H4.5a1 1 0 1 0 0-2H4Z" clip-rule="evenodd"/></symbol><symbol id="icon-eds-i-subjects-medium" viewBox="0 0 24 24"><g id="icon-subjects-copy" stroke="none" stroke-width="1" fill-rule="evenodd"><path d="M13.3846154,2 C14.7015971,2 15.7692308,3.06762994 15.7692308,4.38461538 L15.7692308,7.15384615 C15.7692308,8.47082629 14.7015955,9.53846154 13.3846154,9.53846154 L13.1038388,9.53925278 C13.2061091,9.85347965 13.3815528,10.1423885 13.6195822,10.3804178 C13.9722182,10.7330539 14.436524,10.9483278 14.9293854,10.9918129 L15.1153846,11 C16.2068332,11 17.2535347,11.433562 18.0254647,12.2054189 C18.6411944,12.8212361 19.0416785,13.6120766 19.1784166,14.4609738 L19.6153846,14.4615385 C20.932386,14.4615385 22,15.5291672 22,16.8461538 L22,19.6153846 C22,20.9323924 20.9323924,22 19.6153846,22 L16.8461538,22 C15.5291672,22 14.4615385,20.932386 14.4615385,19.6153846 L14.4615385,16.8461538 C14.4615385,15.5291737 15.5291737,14.4615385 16.8461538,14.4615385 L17.126925,14.460779 C17.0246537,14.1465537 16.8492179,13.857633 16.6112344,13.6196157 C16.2144418,13.2228606 15.6764136,13 15.1153846,13 C14.0239122,13 12.9771569,12.5664197 12.2053686,11.7946314 C12.1335167,11.7227795 12.0645962,11.6485444 11.9986839,11.5721119 C11.9354038,11.6485444 11.8664833,11.7227795 11.7946314,11.7946314 C11.0228431,12.5664197 9.97608778,13 8.88461538,13 C8.323576,13 7.78552852,13.2228666 7.38881294,13.6195822 C7.15078359,13.8576115 6.97533988,14.1465203 6.8730696,14.4607472 L7.15384615,14.4615385 C8.47082629,14.4615385 9.53846154,15.5291737 9.53846154,16.8461538 L9.53846154,19.6153846 C9.53846154,20.932386 8.47083276,22 7.15384615,22 L4.38461538,22 C3.06762347,22 2,20.9323876 2,19.6153846 L2,16.8461538 C2,15.5291721 3.06762994,14.4615385 4.38461538,14.4615385 L4.8215823,14.4609378 C4.95831893,13.6120029 5.3588057,12.8211623 5.97459937,12.2053686 C6.69125996,11.488708 7.64500941,11.0636656 8.6514968,11.0066017 L8.88461538,11 C9.44565477,11 9.98370225,10.7771334 10.3804178,10.3804178 C10.6184472,10.1423885 10.7938909,9.85347965 10.8961612,9.53925278 L10.6153846,9.53846154 C9.29840448,9.53846154 8.23076923,8.47082629 8.23076923,7.15384615 L8.23076923,4.38461538 C8.23076923,3.06762994 9.29840286,2 10.6153846,2 L13.3846154,2 Z M7.15384615,16.4615385 L4.38461538,16.4615385 C4.17220099,16.4615385 4,16.63374 4,16.8461538 L4,19.6153846 C4,19.8278134 4.17218833,20 4.38461538,20 L7.15384615,20 C7.36626945,20 7.53846154,19.8278103 7.53846154,19.6153846 L7.53846154,16.8461538 C7.53846154,16.6337432 7.36625679,16.4615385 7.15384615,16.4615385 Z M19.6153846,16.4615385 L16.8461538,16.4615385 C16.6337432,16.4615385 16.4615385,16.6337432 16.4615385,16.8461538 L16.4615385,19.6153846 C16.4615385,19.8278103 16.6337306,20 16.8461538,20 L19.6153846,20 C19.8278229,20 20,19.8278229 20,19.6153846 L20,16.8461538 C20,16.6337306 19.8278103,16.4615385 19.6153846,16.4615385 Z M13.3846154,4 L10.6153846,4 C10.4029708,4 10.2307692,4.17220099 10.2307692,4.38461538 L10.2307692,7.15384615 C10.2307692,7.36625679 10.402974,7.53846154 10.6153846,7.53846154 L13.3846154,7.53846154 C13.597026,7.53846154 13.7692308,7.36625679 13.7692308,7.15384615 L13.7692308,4.38461538 C13.7692308,4.17220099 13.5970292,4 13.3846154,4 Z" id="Shape" fill-rule="nonzero"/></g></symbol><symbol id="icon-eds-small-arrow-left" viewBox="0 0 16 17"><path stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M14 8.092H2m0 0L8 2M2 8.092l6 6.035"/></symbol><symbol id="icon-eds-small-arrow-right" viewBox="0 0 16 16"><g fill-rule="evenodd" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2"><path d="M2 8.092h12M8 2l6 6.092M8 14.127l6-6.035"/></g></symbol><symbol id="icon-orcid-logo" viewBox="0 0 40 40"><path fill-rule="evenodd" d="M12.281 10.453c.875 0 1.578-.719 1.578-1.578 0-.86-.703-1.578-1.578-1.578-.875 0-1.578.703-1.578 1.578 0 .86.703 1.578 1.578 1.578Zm-1.203 18.641h2.406V12.359h-2.406v16.735Z"/><path fill-rule="evenodd" d="M17.016 12.36h6.5c6.187 0 8.906 4.421 8.906 8.374 0 4.297-3.36 8.375-8.875 8.375h-6.531V12.36Zm6.234 14.578h-3.828V14.53h3.703c4.688 0 6.828 2.844 6.828 6.203 0 2.063-1.25 6.203-6.703 6.203Z" clip-rule="evenodd"/></symbol></svg> </div> <a class="c-skip-link" href="#main">Skip to main content</a> <div class="u-lazy-ad-wrapper u-mbs-0"> <div class="c-ad c-ad--728x90 c-ad--conditional" data-test="springer-doubleclick-ad"> <div class="c-ad c-ad__inner" > <p class="c-ad__label">Advertisement</p> <div id="div-gpt-ad-LB1" class="div-gpt-ad grade-c-hide" data-gpt data-gpt-unitpath="/270604982/springerlink/439/article" data-gpt-sizes="728x90" data-gpt-targeting="pos=top;articleid=s00439-021-02393-x;" data-ad-type="top" style="min-width:728px;min-height:90px"> <noscript> <a href="//pubads.g.doubleclick.net/gampad/jump?iu=/270604982/springerlink/439/article&sz=728x90&pos=top&articleid=s00439-021-02393-x"> <img data-test="gpt-advert-fallback-img" src="//pubads.g.doubleclick.net/gampad/ad?iu=/270604982/springerlink/439/article&sz=728x90&pos=top&articleid=s00439-021-02393-x" alt="Advertisement" width="728" height="90"> </a> </noscript> </div> </div> </div> </div> <header class="eds-c-header" data-eds-c-header> <div class="eds-c-header__container" data-eds-c-header-expander-anchor> <div class="eds-c-header__brand"> <a href="https://link.springer.com" data-test=springerlink-logo data-track="click_imprint_logo" data-track-context="unified header" data-track-action="click logo link" data-track-category="unified header" data-track-label="link" > <img src="/oscar-static/images/darwin/header/img/logo-springer-nature-link-3149409f62.svg" alt="Springer Nature Link"> </a> </div> <a class="c-header__link eds-c-header__link" id="identity-account-widget" href='https://idp.springer.com/auth/personal/springernature?redirect_uri=https://link.springer.com/article/10.1007/s00439-021-02393-x?'><span class="eds-c-header__widget-fragment-title">Log in</span></a> </div> <nav class="eds-c-header__nav" aria-label="header navigation"> <div class="eds-c-header__nav-container"> <div class="eds-c-header__item eds-c-header__item--menu"> <a href="#eds-c-header-nav" class="eds-c-header__link" data-eds-c-header-expander> <svg class="eds-c-header__icon" width="24" height="24" aria-hidden="true" focusable="false"> <use xlink:href="#icon-eds-i-menu-medium"></use> </svg><span>Menu</span> </a> </div> <div class="eds-c-header__item eds-c-header__item--inline-links"> <a class="eds-c-header__link" href="https://link.springer.com/journals/" data-track="nav_find_a_journal" data-track-context="unified header" data-track-action="click find a journal" data-track-category="unified header" data-track-label="link" > Find a journal </a> <a class="eds-c-header__link" href="https://www.springernature.com/gp/authors" data-track="nav_how_to_publish" data-track-context="unified header" data-track-action="click publish with us link" data-track-category="unified header" data-track-label="link" > Publish with us </a> <a class="eds-c-header__link" href="https://link.springernature.com/home/" data-track="nav_track_your_research" data-track-context="unified header" data-track-action="click track your research" data-track-category="unified header" data-track-label="link" > Track your research </a> </div> <div class="eds-c-header__link-container"> <div class="eds-c-header__item eds-c-header__item--divider"> <a href="#eds-c-header-popup-search" class="eds-c-header__link" data-eds-c-header-expander data-eds-c-header-test-search-btn> <svg class="eds-c-header__icon" width="24" height="24" aria-hidden="true" focusable="false"> <use xlink:href="#icon-eds-i-search-medium"></use> </svg><span>Search</span> </a> </div> <div id="ecommerce-header-cart-icon-link" class="eds-c-header__item ecommerce-cart" style="display:inline-block"> <a class="eds-c-header__link" href="https://order.springer.com/public/cart" style="appearance:none;border:none;background:none;color:inherit;position:relative"> <svg id="eds-i-cart" class="eds-c-header__icon" xmlns="http://www.w3.org/2000/svg" height="24" width="24" viewBox="0 0 24 24" aria-hidden="true" focusable="false"> <path fill="currentColor" fill-rule="nonzero" d="M2 1a1 1 0 0 0 0 2l1.659.001 2.257 12.808a2.599 2.599 0 0 0 2.435 2.185l.167.004 9.976-.001a2.613 2.613 0 0 0 2.61-1.748l.03-.106 1.755-7.82.032-.107a2.546 2.546 0 0 0-.311-1.986l-.108-.157a2.604 2.604 0 0 0-2.197-1.076L6.042 5l-.56-3.17a1 1 0 0 0-.864-.82l-.12-.007L2.001 1ZM20.35 6.996a.63.63 0 0 1 .54.26.55.55 0 0 1 .082.505l-.028.1L19.2 15.63l-.022.05c-.094.177-.282.299-.526.317l-10.145.002a.61.61 0 0 1-.618-.515L6.394 6.999l13.955-.003ZM18 19a2 2 0 1 0 0 4 2 2 0 0 0 0-4ZM8 19a2 2 0 1 0 0 4 2 2 0 0 0 0-4Z"></path> </svg><span>Cart</span><span class="cart-info" style="display:none;position:absolute;top:10px;right:45px;background-color:#C65301;color:#fff;width:18px;height:18px;font-size:11px;border-radius:50%;line-height:17.5px;text-align:center"></span></a> <script>(function () { var exports = {}; if (window.fetch) { "use strict"; Object.defineProperty(exports, "__esModule", { value: true }); exports.headerWidgetClientInit = void 0; var headerWidgetClientInit = function (getCartInfo) { document.body.addEventListener("updatedCart", function () { updateCartIcon(); }, false); return updateCartIcon(); function updateCartIcon() { return getCartInfo() .then(function (res) { return res.json(); }) .then(refreshCartState) .catch(function (_) { }); } function refreshCartState(json) { var indicator = document.querySelector("#ecommerce-header-cart-icon-link .cart-info"); /* istanbul ignore else */ if (indicator && json.itemCount) { indicator.style.display = 'block'; indicator.textContent = json.itemCount > 9 ? '9+' : json.itemCount.toString(); var moreThanOneItem = json.itemCount > 1; indicator.setAttribute('title', "there ".concat(moreThanOneItem ? "are" : "is", " ").concat(json.itemCount, " item").concat(moreThanOneItem ? "s" : "", " in your cart")); } return json; } }; exports.headerWidgetClientInit = headerWidgetClientInit; headerWidgetClientInit( function () { return window.fetch("https://cart.springer.com/cart-info", { credentials: "include", headers: { Accept: "application/json" } }) } ) }})()</script> </div> </div> </div> </nav> </header> <article lang="en" id="main" class="app-masthead__colour-5"> <section class="app-masthead " aria-label="article masthead"> <div class="app-masthead__container"> <div class="app-article-masthead u-sans-serif js-context-bar-sticky-point-masthead" data-track-component="article" data-test="masthead-component"> <div class="app-article-masthead__info"> <nav aria-label="breadcrumbs" data-test="breadcrumbs"> <ol class="c-breadcrumbs c-breadcrumbs--contrast" itemscope itemtype="https://schema.org/BreadcrumbList"> <li class="c-breadcrumbs__item" id="breadcrumb0" itemprop="itemListElement" itemscope="" itemtype="https://schema.org/ListItem"> <a href="/" class="c-breadcrumbs__link" itemprop="item" data-track="click_breadcrumb" data-track-context="article page" data-track-category="article" data-track-action="breadcrumbs" data-track-label="breadcrumb1"><span itemprop="name">Home</span></a><meta itemprop="position" content="1"> <svg class="c-breadcrumbs__chevron" role="img" aria-hidden="true" focusable="false" width="10" height="10" viewBox="0 0 10 10"> <path d="m5.96738168 4.70639573 2.39518594-2.41447274c.37913917-.38219212.98637524-.38972225 1.35419292-.01894278.37750606.38054586.37784436.99719163-.00013556 1.37821513l-4.03074001 4.06319683c-.37758093.38062133-.98937525.38100976-1.367372-.00003075l-4.03091981-4.06337806c-.37759778-.38063832-.38381821-.99150444-.01600053-1.3622839.37750607-.38054587.98772445-.38240057 1.37006824.00302197l2.39538588 2.4146743.96295325.98624457z" fill-rule="evenodd" transform="matrix(0 -1 1 0 0 10)"/> </svg> </li> <li class="c-breadcrumbs__item" id="breadcrumb1" itemprop="itemListElement" itemscope="" itemtype="https://schema.org/ListItem"> <a href="/journal/439" class="c-breadcrumbs__link" itemprop="item" data-track="click_breadcrumb" data-track-context="article page" data-track-category="article" data-track-action="breadcrumbs" data-track-label="breadcrumb2"><span itemprop="name">Human Genetics</span></a><meta itemprop="position" content="2"> <svg class="c-breadcrumbs__chevron" role="img" aria-hidden="true" focusable="false" width="10" height="10" viewBox="0 0 10 10"> <path d="m5.96738168 4.70639573 2.39518594-2.41447274c.37913917-.38219212.98637524-.38972225 1.35419292-.01894278.37750606.38054586.37784436.99719163-.00013556 1.37821513l-4.03074001 4.06319683c-.37758093.38062133-.98937525.38100976-1.367372-.00003075l-4.03091981-4.06337806c-.37759778-.38063832-.38381821-.99150444-.01600053-1.3622839.37750607-.38054587.98772445-.38240057 1.37006824.00302197l2.39538588 2.4146743.96295325.98624457z" fill-rule="evenodd" transform="matrix(0 -1 1 0 0 10)"/> </svg> </li> <li class="c-breadcrumbs__item" id="breadcrumb2" itemprop="itemListElement" itemscope="" itemtype="https://schema.org/ListItem"> <span itemprop="name">Article</span><meta itemprop="position" content="3"> </li> </ol> </nav> <h1 class="c-article-title" data-test="article-title" data-article-title="">The promise of automated machine learning for the genetic analysis of complex traits</h1> <ul class="c-article-identifiers"> <li class="c-article-identifiers__item" data-test="article-category">Review</li> <li class="c-article-identifiers__item"> <a href="https://www.springernature.com/gp/open-research/about/the-fundamentals-of-open-access-and-open-research" data-track="click" data-track-action="open access" data-track-label="link" class="u-color-open-access" data-test="open-access">Open access</a> </li> <li class="c-article-identifiers__item"> Published: <time datetime="2021-10-28">28 October 2021</time> </li> </ul> <ul class="c-article-identifiers c-article-identifiers--cite-list"> <li class="c-article-identifiers__item"> <span data-test="journal-volume">Volume 141</span>, pages 1529–1544, (<span data-test="article-publication-year">2022</span>) </li> <li class="c-article-identifiers__item c-article-identifiers__item--cite"> <a href="#citeas" data-track="click" data-track-action="cite this article" data-track-category="article body" data-track-label="link">Cite this article</a> </li> </ul> <div class="app-article-masthead__buttons" data-test="download-article-link-wrapper" data-track-context="masthead"> <div class="c-pdf-container"> <div class="c-pdf-download u-clear-both u-mb-16"> <a href="/content/pdf/10.1007/s00439-021-02393-x.pdf" class="u-button u-button--full-width u-button--primary u-justify-content-space-between c-pdf-download__link" data-article-pdf="true" data-readcube-pdf-url="true" data-test="pdf-link" data-draft-ignore="true" data-track="content_download" data-track-type="article pdf download" data-track-action="download pdf" data-track-label="button" data-track-external download> <span class="c-pdf-download__text">Download PDF</span> <svg aria-hidden="true" focusable="false" width="16" height="16" class="u-icon"><use xlink:href="#icon-eds-i-download-medium"/></svg> </a> </div> </div> <p class="app-article-masthead__access"> <svg width="16" height="16" focusable="false" role="img" aria-hidden="true"><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#icon-eds-i-check-filled-medium"></use></svg> You have full access to this <a href="https://www.springernature.com/gp/open-research/about/the-fundamentals-of-open-access-and-open-research" data-track="click" data-track-action="open access" data-track-label="link">open access</a> article</p> </div> </div> <div class="app-article-masthead__brand"> <a href="/journal/439" class="app-article-masthead__journal-link" data-track="click_journal_home" data-track-action="journal homepage" data-track-context="article page" data-track-label="link"> <picture> <source type="image/webp" media="(min-width: 768px)" width="120" height="159" srcset="https://media.springernature.com/w120/springer-static/cover-hires/journal/439?as=webp, https://media.springernature.com/w316/springer-static/cover-hires/journal/439?as=webp 2x"> <img width="72" height="95" src="https://media.springernature.com/w72/springer-static/cover-hires/journal/439?as=webp" srcset="https://media.springernature.com/w144/springer-static/cover-hires/journal/439?as=webp 2x" alt=""> </picture> <span class="app-article-masthead__journal-title">Human Genetics</span> </a> <a href="https://link.springer.com/journal/439/aims-and-scope" class="app-article-masthead__submission-link" data-track="click_aims_and_scope" data-track-action="aims and scope" data-track-context="article page" data-track-label="link"> Aims and scope <svg width="16" height="16" focusable="false" role="img" aria-hidden="true" class="u-icon"><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#icon-eds-i-arrow-right-medium"></use></svg> </a> <a href="https://submission.springernature.com/new-submission/439/3" class="app-article-masthead__submission-link" data-track="click_submit_manuscript" data-track-context="article masthead on springerlink article page" data-track-action="submit manuscript" data-track-label="link"> Submit manuscript <svg width="16" height="16" focusable="false" role="img" aria-hidden="true" class="u-icon"><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#icon-eds-i-arrow-right-medium"></use></svg> </a> </div> </div> </div> </section> <div class="c-article-main u-container u-mt-24 u-mb-32 l-with-sidebar" id="main-content" data-component="article-container"> <main class="u-serif js-main-column" data-track-component="article body"> <div class="c-context-bar u-hide" data-test="context-bar" data-context-bar aria-hidden="true"> <div class="c-context-bar__container u-container"> <div class="c-context-bar__title"> The promise of automated machine learning for the genetic analysis of complex traits </div> <div data-test="inCoD" data-track-context="sticky banner"> <div class="c-pdf-container"> <div class="c-pdf-download u-clear-both u-mb-16"> <a href="/content/pdf/10.1007/s00439-021-02393-x.pdf" class="u-button u-button--full-width u-button--primary u-justify-content-space-between c-pdf-download__link" data-article-pdf="true" data-readcube-pdf-url="true" data-test="pdf-link" data-draft-ignore="true" data-track="content_download" data-track-type="article pdf download" data-track-action="download pdf" data-track-label="button" data-track-external download> <span class="c-pdf-download__text">Download PDF</span> <svg aria-hidden="true" focusable="false" width="16" height="16" class="u-icon"><use xlink:href="#icon-eds-i-download-medium"/></svg> </a> </div> </div> </div> </div> </div> <div class="c-article-header"> <header> <ul class="c-article-author-list c-article-author-list--short" data-test="authors-list" data-component-authors-activator="authors-list"><li class="c-article-author-list__item"><a data-test="author-name" data-track="click" data-track-action="open author" data-track-label="link" href="#auth-Elisabetta-Manduchi-Aff1-Aff2" data-author-popup="auth-Elisabetta-Manduchi-Aff1-Aff2" data-author-search="Manduchi, Elisabetta">Elisabetta Manduchi</a><span class="u-js-hide"> <a class="js-orcid" href="http://orcid.org/0000-0002-4110-3714"><span class="u-visually-hidden">ORCID: </span>orcid.org/0000-0002-4110-3714</a></span><sup class="u-js-hide"><a href="#Aff1">1</a>,<a href="#Aff2">2</a></sup>, </li><li class="c-article-author-list__item"><a data-test="author-name" data-track="click" data-track-action="open author" data-track-label="link" href="#auth-Joseph_D_-Romano-Aff1" data-author-popup="auth-Joseph_D_-Romano-Aff1" data-author-search="Romano, Joseph D.">Joseph D. Romano</a><span class="u-js-hide"> <a class="js-orcid" href="http://orcid.org/0000-0002-7999-4399"><span class="u-visually-hidden">ORCID: </span>orcid.org/0000-0002-7999-4399</a></span><sup class="u-js-hide"><a href="#Aff1">1</a></sup> & </li><li class="c-article-author-list__item"><a data-test="author-name" data-track="click" data-track-action="open author" data-track-label="link" href="#auth-Jason_H_-Moore-Aff1-Aff2" data-author-popup="auth-Jason_H_-Moore-Aff1-Aff2" data-author-search="Moore, Jason H." data-corresp-id="c1">Jason H. Moore<svg width="16" height="16" focusable="false" role="img" aria-hidden="true" class="u-icon"><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#icon-eds-i-mail-medium"></use></svg></a><span class="u-js-hide"> <a class="js-orcid" href="http://orcid.org/0000-0002-5015-1099"><span class="u-visually-hidden">ORCID: </span>orcid.org/0000-0002-5015-1099</a></span><sup class="u-js-hide"><a href="#Aff1">1</a>,<a href="#Aff2">2</a></sup> </li></ul> <div data-test="article-metrics"> <ul class="app-article-metrics-bar u-list-reset"> <li class="app-article-metrics-bar__item"> <p class="app-article-metrics-bar__count"><svg class="u-icon app-article-metrics-bar__icon" width="24" height="24" aria-hidden="true" focusable="false"> <use xlink:href="#icon-eds-i-accesses-medium"></use> </svg>7345 <span class="app-article-metrics-bar__label">Accesses</span></p> </li> <li class="app-article-metrics-bar__item"> <p class="app-article-metrics-bar__count"><svg class="u-icon app-article-metrics-bar__icon" width="24" height="24" aria-hidden="true" focusable="false"> <use xlink:href="#icon-eds-i-citations-medium"></use> </svg>12 <span class="app-article-metrics-bar__label">Citations</span></p> </li> <li class="app-article-metrics-bar__item"> <p class="app-article-metrics-bar__count"><svg class="u-icon app-article-metrics-bar__icon" width="24" height="24" aria-hidden="true" focusable="false"> <use xlink:href="#icon-eds-i-altmetric-medium"></use> </svg>16 <span class="app-article-metrics-bar__label">Altmetric</span></p> </li> <li class="app-article-metrics-bar__item app-article-metrics-bar__item--metrics"> <p class="app-article-metrics-bar__details"><a href="/article/10.1007/s00439-021-02393-x/metrics" data-track="click" data-track-action="view metrics" data-track-label="link" rel="nofollow">Explore all metrics <svg class="u-icon app-article-metrics-bar__arrow-icon" width="24" height="24" aria-hidden="true" focusable="false"> <use xlink:href="#icon-eds-i-arrow-right-medium"></use> </svg></a></p> </li> </ul> </div> <div class="u-mt-32"> </div> </header> </div> <div data-article-body="true" data-track-component="article body" class="c-article-body"> <section aria-labelledby="Abs1" data-title="Abstract" lang="en"><div class="c-article-section" id="Abs1-section"><h2 class="c-article-section__title js-section-title js-c-reading-companion-sections-item" id="Abs1">Abstract</h2><div class="c-article-section__content" id="Abs1-content"><p>The genetic analysis of complex traits has been dominated by parametric statistical methods due to their theoretical properties, ease of use, computational efficiency, and intuitive interpretation. However, there are likely to be patterns arising from complex genetic architectures which are more easily detected and modeled using machine learning methods. Unfortunately, selecting the right machine learning algorithm and tuning its hyperparameters can be daunting for experts and non-experts alike. The goal of automated machine learning (AutoML) is to let a computer algorithm identify the right algorithms and hyperparameters thus taking the guesswork out of the optimization process. We review the promises and challenges of AutoML for the genetic analysis of complex traits and give an overview of several approaches and some example applications to omics data. It is our hope that this review will motivate studies to develop and evaluate novel AutoML methods and software in the genetics and genomics space. The promise of AutoML is to enable anyone, regardless of training or expertise, to apply machine learning as part of their genetic analysis strategy.</p></div></div></section> <div data-test="cobranding-download"> </div> <section aria-labelledby="inline-recommendations" data-title="Inline Recommendations" class="c-article-recommendations" data-track-component="inline-recommendations"> <h3 class="c-article-recommendations-title" id="inline-recommendations">Similar content being viewed by others</h3> <div class="c-article-recommendations-list"> <div class="c-article-recommendations-list__item"> <article class="c-article-recommendations-card" itemscope itemtype="http://schema.org/ScholarlyArticle"> <div class="c-article-recommendations-card__img"><img src="https://media.springernature.com/w215h120/springer-static/image/art%3A10.1038%2Fnrg3920/MediaObjects/41576_2015_Article_BFnrg3920_Fig1_HTML.jpg" loading="lazy" alt=""></div> <div class="c-article-recommendations-card__main"> <h3 class="c-article-recommendations-card__heading" itemprop="name headline"> <a class="c-article-recommendations-card__link" itemprop="url" href="https://link.springer.com/10.1038/nrg3920?fromPaywallRec=false" data-track="select_recommendations_1" data-track-context="inline recommendations" data-track-action="click recommendations inline - 1" data-track-label="10.1038/nrg3920">Machine learning applications in genetics and genomics </a> </h3> <div class="c-article-meta-recommendations" data-test="recommendation-info"> <span class="c-article-meta-recommendations__item-type">Article</span> <span class="c-article-meta-recommendations__date">07 May 2015</span> </div> </div> </article> </div> <div class="c-article-recommendations-list__item"> <article class="c-article-recommendations-card" itemscope itemtype="http://schema.org/ScholarlyArticle"> <div class="c-article-recommendations-card__img"><img src="https://media.springernature.com/w215h120/springer-static/image/art%3A10.1007%2Fs00439-019-01996-9/MediaObjects/439_2019_1996_Fig1_HTML.png" loading="lazy" alt=""></div> <div class="c-article-recommendations-card__main"> <h3 class="c-article-recommendations-card__heading" itemprop="name headline"> <a class="c-article-recommendations-card__link" itemprop="url" href="https://link.springer.com/10.1007/s00439-019-01996-9?fromPaywallRec=false" data-track="select_recommendations_2" data-track-context="inline recommendations" data-track-action="click recommendations inline - 2" data-track-label="10.1007/s00439-019-01996-9">Statistical learning approaches in the genetic epidemiology of complex diseases </a> </h3> <div class="c-article-meta-recommendations" data-test="recommendation-info"> <span class="c-article-meta-recommendations__item-type">Article</span> <span class="c-article-meta-recommendations__date">02 May 2019</span> </div> </div> </article> </div> <div class="c-article-recommendations-list__item"> <article class="c-article-recommendations-card" itemscope itemtype="http://schema.org/ScholarlyArticle"> <div class="c-article-recommendations-card__img"><img src="https://media.springernature.com/w215h120/springer-static/image/art%3A10.1186%2Fs12864-015-1616-z/MediaObjects/12864_2015_1616_Fig1_HTML.gif" loading="lazy" alt=""></div> <div class="c-article-recommendations-card__main"> <h3 class="c-article-recommendations-card__heading" itemprop="name headline"> <a class="c-article-recommendations-card__link" itemprop="url" href="https://link.springer.com/10.1186/s12864-015-1616-z?fromPaywallRec=false" data-track="select_recommendations_3" data-track-context="inline recommendations" data-track-action="click recommendations inline - 3" data-track-label="10.1186/s12864-015-1616-z">Assessing models for genetic prediction of complex traits: a comparison of visualization and quantitative methods </a> </h3> <div class="c-article-meta-recommendations" data-test="recommendation-info"> <span class="c-article-meta-recommendations__item-type">Article</span> <span class="c-article-meta-recommendations__access-type">Open access</span> <span class="c-article-meta-recommendations__date">22 May 2015</span> </div> </div> </article> </div> </div> </section> <script> window.dataLayer = window.dataLayer || []; window.dataLayer.push({ recommendations: { recommender: 'semantic', model: 'specter', policy_id: 'NA', timestamp: 1732280880, embedded_user: 'null' } }); </script> <div class="app-card-service" data-test="article-checklist-banner"> <div> <a class="app-card-service__link" data-track="click_presubmission_checklist" data-track-context="article page top of reading companion" data-track-category="pre-submission-checklist" data-track-action="clicked article page checklist banner test 2 old version" data-track-label="link" href="https://beta.springernature.com/pre-submission?journalId=439" data-test="article-checklist-banner-link"> <span class="app-card-service__link-text">Use our pre-submission checklist</span> <svg class="app-card-service__link-icon" aria-hidden="true" focusable="false"><use xlink:href="#icon-eds-i-arrow-right-small"></use></svg> </a> <p class="app-card-service__description">Avoid common mistakes on your manuscript.</p> </div> <div class="app-card-service__icon-container"> <svg class="app-card-service__icon" aria-hidden="true" focusable="false"> <use xlink:href="#icon-eds-i-clipboard-check-medium"></use> </svg> </div> </div> <div class="main-content"> <section data-title="Outline"><div class="c-article-section" id="Sec1-section"><h2 class="c-article-section__title js-section-title js-c-reading-companion-sections-item" id="Sec1">Outline</h2><div class="c-article-section__content" id="Sec1-content"><p>We provide a review of current Automated Machine Learning (AutoML) systems with a special emphasis on their applicability to the biomedical domain and, in particular, on omics and genetic applications. The content is organized as follows. In the next section, we describe Machine Learning (ML) in general, including open-source ML tools, feature importance, and biomedical applications. In the following section, we focus on AutoML, describing first three widely used open-source solutions (Auto-WEKA, Auto-sklearn, and TPOT). Then, we present AutoML solutions that include neural networks or allow for neural architecture search. Finally, we briefly discuss a few additional systems, including commercial ones. The next section focuses on TPOT applications to omics, since this tool has been particularly used in this context and some of its refinements were motivated by this type of applications. We conclude in the next section with a discussion of promises and challenges of AutoML in the genetics domain, including the typically large dimensionality of genetics data and class imbalance.</p></div></div></section><section data-title="Machine learning"><div class="c-article-section" id="Sec2-section"><h2 class="c-article-section__title js-section-title js-c-reading-companion-sections-item" id="Sec2">Machine learning</h2><div class="c-article-section__content" id="Sec2-content"><h3 class="c-article__sub-heading" id="Sec3">Generalities</h3><p>Machine Learning (ML) refers to approaches by which computers learn from data to accomplish certain tasks, without a programmer having to specify every single algorithmic instruction. In supervised ML, which will be the focus of what follows, the task is to learn a predictive model from <i>training data</i> that provide examples of inputs and their corresponding outputs. This means learning a general rule that can then be used to predict the outputs for new inputs of the same type as the training data. Each input consists of the values that a collection of <i>features</i> (the independent/explanatory variables, also referred to as <i>predictors</i>) have for a particular sample (e.g. an individual or subject, also referred to as <i>observation</i>), and the output is the value of a <i>target outcome</i> of interest (the dependent/response variable) for that sample. Inputs can be represented by a matrix <i>X</i> whose columns correspond to <i>p</i> features and rows to <i>n</i> samples and the output by a vector <i>y</i> of <i>n</i> target values for those samples. In a <i>classification</i> problem the target can take finitely many possible values (class labels), whereas in a <i>regression</i> problem, the target is continuous. Any given algorithm in ML can have <i>parameters</i> and <i>hyperparameters</i>. Parameters are internal configuration variables learned from the data during training. For example, the coefficients in a linear or logistic regression model are parameters for that model. Hyperparameters are instead values that are specified before the learning starts. For example, in regularized regression, a penalty hyperparameter λ is specified to discourage complex models; in a random forest (RF), the number of trees is a hyperparameter, etc.</p><p>There are several metrics that can be used to assess the effectiveness of a predictive model. In classification problems, a frequently used metric is the accuracy (i.e., the percentage of correct predictions) or its variants which are appropriate in the presence of imbalance between the number of samples from the different classes, such as the balanced accuracy. Metrics frequently used for regression problems include the root mean squared error (RMSE), i.e. the square root of the average squared difference between true and predicted values, and the coefficient of determination, which reflects the proportion of target variance explained by the model. But there are many other choices of metrics that can be selected when optimizing algorithms and hyperparameters in an ML application (Kuhn and Johnson <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2013" title="Kuhn M, Johnson K (2013) Applied predictive modeling. Springer-Verlag, New York" href="/article/10.1007/s00439-021-02393-x#ref-CR38" id="ref-link-section-d56742713e561">2013</a>; Zheng <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2015" title="Zheng A (2015) Evaluating machine learning models: a beginner’s guide to key concepts and pitfalls. O’Reilly Media, Newton" href="/article/10.1007/s00439-021-02393-x#ref-CR87" id="ref-link-section-d56742713e564">2015</a>). An ML model’s performance should be evaluated by computing its score for the chosen metric on a data set separate from that used for training. The score on the training set is not a good indicator of the generalizability of the learned model as it could be affected by <i>overfitting</i>, a phenomenon by which the model has adapted to characteristics which are specific to the training set. Thus, the typical flow is to use a training set to learn the model parameters, i.e. to <i>fit</i> the model, and then to assess the model performance on a hold-out <i>testing set</i>, with samples drawn from the same population. Often, it is desirable to tune the choice of algorithm and its hyperparameters. This can be done using an independent <i>validation set</i>, with samples drawn from the same population as the training set. Essentially, for each of different choices of algorithm and of its hyperparameter settings, one fits the model with those selections to the training set and then evaluates its performance on the validation set. The model with the choices that optimize performance on the validation set is then selected and evaluated on the hold-out testing set. In practice, more complex schemes are employed. For example, a common approach is <i>k</i>-fold cross-validation (CV), where the input data are randomly partitioned into <i>k</i> equal sized subsets. For each of different choices of algorithm and of its hyperparameter settings, in turn one of the subsets serves as the validation set and the model is fit to the union of the remaining <i>k </i>– 1 subsets and evaluated on the validation set. Then, the selection yielding the best average performance across the <i>k</i> folds is adopted and the corresponding model is fit to the entire training set and evaluated on a hold-out testing set (see Fig. <a data-track="click" data-track-label="link" data-track-action="figure anchor" href="/article/10.1007/s00439-021-02393-x#Fig1">1</a>).</p><div class="c-article-section__figure js-c-reading-companion-figures-item" data-test="figure" data-container-section="figure" id="figure-1" data-title="Fig. 1"><figure><figcaption><b id="Fig1" class="c-article-section__figure-caption" data-test="figure-caption-text">Fig. 1</b></figcaption><div class="c-article-section__figure-content"><div class="c-article-section__figure-item"><a class="c-article-section__figure-link" data-test="img-link" data-track="click" data-track-label="image" data-track-action="view figure" href="/article/10.1007/s00439-021-02393-x/figures/1" rel="nofollow"><picture><img aria-describedby="Fig1" src="//media.springernature.com/lw685/springer-static/image/art%3A10.1007%2Fs00439-021-02393-x/MediaObjects/439_2021_2393_Fig1_HTML.png" alt="figure 1" loading="lazy" width="685" height="275"></picture></a></div><div class="c-article-section__figure-description" data-test="bottom-caption" id="figure-1-desc"><p>Flow for <i>k</i>-fold CV on algorithm/hyperparameter selection and evaluation. <i>A</i><sub><i>i</i></sub> indicates the selection of an algorithm with specified hyperparameters</p></div></div><div class="u-text-right u-hide-print"><a class="c-article__pill-button" data-test="article-link" data-track="click" data-track-label="button" data-track-action="view figure" href="/article/10.1007/s00439-021-02393-x/figures/1" data-track-dest="link:Figure1 Full size image" aria-label="Full size image figure 1" rel="nofollow"><span>Full size image</span><svg width="16" height="16" focusable="false" role="img" aria-hidden="true" class="u-icon"><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#icon-eds-i-chevron-right-small"></use></svg></a></div></figure></div><p>There are several steps involved in setting up a full ML solution for a given task, from pre-processing to predictive model generation. First, the data must be cleaned as appropriate; this includes decisions on how to handle missing values, how to encode nominal variables, etc. Then, prior to running a classifier or regressor ML algorithm, one needs to decide whether to use all or a subset of the features and, in the latter case, what algorithm to use to select such features. Moreover, one needs to decide on possible transformations to apply to the features and creation of new features, which is referred to as <i>feature engineering</i>. Thus, in effect, the typical solution consists of a pipeline of feature selector, feature transformation, and estimator algorithms (classifiers or regressors), where the output of a step becomes the input of the following step. Each pipeline could involve more than one feature selector, feature transformer, and estimator. Indeed, multiple pipelines could be combined in a workflow, yielding an even more complex architecture. The ultimate goal is to obtain a solution with good predictive performance, but at the same time there is a tradeoff between complexity and interpretability; a simpler solution may be preferable to a very complex one if the latter is only slightly better in terms of performance. Thus, many decisions must be made, different options assessed (including hyperparameter settings), and in general, the process is labor intensive and requires considerable domain expertise.</p><h3 class="c-article__sub-heading" id="Sec4">Open-source software</h3><p>Along with scientific software used in other domains, ML has benefitted substantially from the “free and open-source software” (FOSS) movement. “Free” software (or libre software) is computer software that can be used for any purpose without restrictions, including modifying and/or redistributing the software. “Open-source” specifically refers to making the source code for the software publicly available, either by distributing the software directly as source code at no cost, or by maintaining a separate source code repository (e.g., GitHub, Sourceforge, or similar) that the public can use to browse the source code directly through a web browser or other interfaces. Most popular ML software today is open-source, and even many ML frameworks developed by large corporations are released as open-source and developed publicly, often allowing for external contributions and improvements from end-users.</p><p>WEKA (Frank et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2016" title="Frank E, Hall MA, Witten IH (2016) The Weka Workbench. Online Appendix for “Data mining: practical machine learning tools and techniques”, Morgan Kaufmann, Fourth Edition. 
 https://www.cs.waikato.ac.nz/ml/weka/Witten_et_al_2016_appendix.pdf
 
 . Accessed 19 Apr 2021" href="/article/10.1007/s00439-021-02393-x#ref-CR21" id="ref-link-section-d56742713e640">2016</a>) is one of the earliest open-source ML software still in common use, originally developed in 1993 at the University of Waikato. WEKA supports classification, regression, pre-processing, and other common data mining tasks, and provides a graphical user interface catered towards users with little to no programming experience or users who prefer not to work in a command line environment. Deep learning (described below) is supported via the Deeplearning4j library for the Java programming language.</p><p>Scikit-learn (Pedregosa et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2011" title="Pedregosa F, Varoquaux G, Gramfort A et al (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12:2825–2830" href="/article/10.1007/s00439-021-02393-x#ref-CR62" id="ref-link-section-d56742713e646">2011</a>) is one of the most popular ML libraries and acting as one of the main interfaces for ML in the Python programming language. Scikit-learn focuses on providing a common, easy-to-use application programming interface (API) for a wide range of ML tasks, and supports advanced features including pipeline construction, semi-supervised learning, and others. Many other ML toolkits imitate or extend the scikit-learn API due to its familiarity and simplicity.</p><p>Several open-source programming languages have ML features implemented either as part of the core language or in the language’s standard library. The R programming language—designed mainly for statistical computing—implements a number of ML algorithms as core functions that are actively loaded at all times. For example, the <i>k</i>-nearest neighbors algorithm can be trained on a dataset by calling knn(<i>X</i>_train, <i>X</i>_test, <i>y</i>_train), where <i>X</i> and <i>y</i> are features and targets (class labels), respectively, and ‘train’ and ‘test’ refer to training and testing datasets, respectively (note that no import statements or other external libraries are needed). The Julia programming language offers similar basic functionality as part of the language’s standard library, but most Julia users apply ML algorithms using the MLJ.jl library, developed and released as free and open-source by the Alan Turing Institute.</p><h3 class="c-article__sub-heading" id="Sec5">Feature importance</h3><p>After a predictive model has been built, it is usually of interest to explore which features are driving the model. There are various approaches to rank features in terms of their importance for the model. Some estimators naturally yield quantities that can serve this purpose. For example, the coefficients of a linear or logistic regression model reflect feature importance when the features have the same scale; in decision trees (and their ensembles, such as RFs), the criteria used to select the split points yield importance scores. A general method which can be employed with any estimator (i.e., <i>model-agnostic</i>) is <i>permutation importance</i> (see <a href="https://github.com/TeamHG-Memex/eli5">https://github.com/TeamHG-Memex/eli5</a> for an implementation). The idea is to measure feature importance for each feature by examining how much the selected performance score (e.g., accuracy, or RMSE, etc.) degrades when that feature is not available. However, removing one feature at a time, retraining the model, and computing the new score would be too intensive computationally. Instead, after the model is fit, for each feature, its values in the hold-out testing set are permuted and the model evaluated on the resulting set (typically multiple permutations are applied and the resulting scores averaged). Essentially, instead of removing that feature, one replaces it by random noise. Features can then be ranked according to how much worse performance on the permuted set is as compared to performance on the unpermuted set. We note that this approach should be used with care, as discussed in Molnar et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Molnar C (2021) Interpretable machine learning" href="/article/10.1007/s00439-021-02393-x#ref-CR51" id="ref-link-section-d56742713e692">2021</a>). For example, if features are correlated, permuting a feature could produce unrealistic data instances which could lead to misleading results. The effects of breaking feature dependencies in various model setups are explored in Hooker et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Hooker G, Mentch L, Zhou S (2021) Unrestricted permutation forces extrapolation: variable importance requires at least one more model, or there is no free variable importance. 
 http://arxiv.org/abs/1905.03151
 
 [cs, stat]" href="/article/10.1007/s00439-021-02393-x#ref-CR27" id="ref-link-section-d56742713e695">2021</a>) and alternative approaches are discussed, which, however, require more computation. Another point to keep in mind to avoid misleading interpretations, is that permutation importance does not separate between main and interaction effects, but reflects both the importance of a feature as well as that of its interactions with the other features (Casalicchio et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2019" title="Casalicchio G, Molnar C, Bischl B (2019) Visualizing the feature importance for black box models. In: Berlingerio M, Bonchi F, Gärtner T et al (eds) Machine learning and knowledge discovery in databases. Springer International Publishing, Cham, pp 655–670" href="/article/10.1007/s00439-021-02393-x#ref-CR12" id="ref-link-section-d56742713e699">2019</a>; Molnar et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Molnar C (2021) Interpretable machine learning" href="/article/10.1007/s00439-021-02393-x#ref-CR51" id="ref-link-section-d56742713e702">2021</a>).</p><p>Permutation importance measures the overall relevance of a feature to a model; it is a so-called ‘global’ method. However, it is also of interest to examine how each feature contributes to the individual predictions. This is particularly relevant in the context of precision medicine and when there is heterogeneity among subjects (i.e., when different features are responsible for the same outcome in different subjects). Approaches have recently been developed with this goal in mind; these are termed ‘local’ methods. For example, SHAP (Lundberg and Lee <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2017" title="Lundberg SM, Lee S-I (2017) A unified approach to interpreting model predictions. Adv Neural Inf Process Syst 30:4765–4774" href="/article/10.1007/s00439-021-02393-x#ref-CR44" id="ref-link-section-d56742713e708">2017</a>) are game-theory-derived metrics indicating, for each sample, how much each feature contributed to the model prediction for that sample. These values can also be summarized across samples to rank features according to their overall contribution to all predictions. SHAP values need to be used with care too. For example, KernelSHAP, a model-agnostic method to compute them, ignores feature dependencies. TreeSHAP, a method to compute SHAP values for tree-based models, does not have this problem but could assign nonzero values to features that have no influence on the prediction.</p><p>Feature importance falls within the umbrella of interpretability in ML, a relevant and complex research area. There are other model-agnostic global and local methods besides those mentioned above. An overview of these and a guide to interpretable ML in general is provided by Molnar (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Molnar C, König G, Herbinger J, et al (2021) General pitfalls of model-agnostic interpretation methods for machine learning models. 
 http://arxiv.org/abs/2007.04131
 
 [cs, stat]" href="/article/10.1007/s00439-021-02393-x#ref-CR50" id="ref-link-section-d56742713e714">2021</a>).</p><h3 class="c-article__sub-heading" id="Sec6">Biomedical applications</h3><p>ML approaches are now routinely used in biomedical applications, as an alternative or a complement to statistical approaches. This includes applications leveraging omics data. For example, Bazaga et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2020" title="Bazaga A, Leggate D, Weisser H (2020) Genome-wide investigation of gene-cancer associations for the prediction of novel therapeutic targets in oncology. Sci Rep 10:10787. 
 https://doi.org/10.1038/s41598-020-67846-1
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR6" id="ref-link-section-d56742713e725">2020</a>) utilize multiple ML algorithms to build drug-target prediction models for a variety of cancer types. The features include genetic mutations, gene expression, gene essentiality, and gene interactions. The resulting models are then applied to more than 15,000 protein coding genes to identify novel cancer type-specific drug-target candidates. In Adams et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2020" title="Adams SM, Feroze H, Nguyen T et al (2020) Genome wide epistasis study of on-statin cardiovascular events with iterative feature reduction and selection. J Pers Med. 
 https://doi.org/10.3390/jpm10040212
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR1" id="ref-link-section-d56742713e728">2020</a>), an RF-driven method is applied to a genome-wide association study (GWAS) to identify epistasis-networks that may provide insights into the risk for on-statin major adverse cardiovascular events. More generally, RFs have been an effective ML approach to identify epistasis, i.e. interactions between two or more genetic loci which are associated to a given phenotype (Orlenko and Moore <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Orlenko A, Kofink D, Lyytikäinen L-P et al (2020) Model selection for metabolomics: predicting diagnosis of coronary artery disease using automated machine learning. Bioinformatics 36:1772–1778. 
 https://doi.org/10.1093/bioinformatics/btz796
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR58" id="ref-link-section-d56742713e731">2021</a>). Another type of ML application specific to genomics consists of building classifiers for deleterious versus non-deleterious genetic variants. For example, CADD (Rentzsch et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2019" title="Rentzsch P, Witten D, Cooper GM et al (2019) CADD: predicting the deleteriousness of variants throughout the human genome. Nucleic Acids Res 47:D886–D894. 
 https://doi.org/10.1093/nar/gky1016
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR63" id="ref-link-section-d56742713e734">2019</a>) is based on a logistic regression model trained on more than 30 million variants and leveraging features from 60 different annotations, including conservation, epigenetic modifications, functional predictions, and genetic context. Using this model CADD then generated deleteriousness scores for variants throughout the human genome reference assembly. Another example is GWAVA (Ritchie et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2014" title="Ritchie MD, Hahn LW, Roodi N et al (2001) Multifactor-dimensionality reduction reveals high-order interactions among estrogen-metabolism genes in sporadic breast cancer. Am J Hum Genet 69:138–147. 
 https://doi.org/10.1086/321276
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR64" id="ref-link-section-d56742713e737">2014</a>), which employs RFs trained on functional genomics features to build a variant prioritization tool for non-coding variants. RFs are also used in TraP (Gelfman et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2017" title="Gelfman S, Wang Q, McSweeney KM et al (2017) Annotating pathogenic non-coding variants in genic regions. Nat Commun 8:236. 
 https://doi.org/10.1038/s41467-017-00141-2
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR22" id="ref-link-section-d56742713e741">2017</a>) to build an annotator for pathogenic non-coding variants in genic regions, and in REVEL (Ioannidis et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2016" title="Ioannidis NM, Rothstein JH, Pejaver V et al (2016) REVEL: an ensemble method for predicting the pathogenicity of rare missense variants. Am J Hum Genet 99:877–885. 
 https://doi.org/10.1016/j.ajhg.2016.08.016
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR32" id="ref-link-section-d56742713e744">2016</a>), to predict the pathogenicity of rare missense variants.</p><p>One important consideration for ML applications to omics data is the ‘big <i>p</i>, little <i>n</i>’ (<i>p</i> > > <i>n</i>) issue. In the omics context, unlike the more traditional tabular data to which ML is often applied, the number <i>p</i> of predictors is usually much larger than the number of observations. This <i>curse of dimensionality</i> makes it challenging to have a sufficiently representative sample of the <i>p</i>-dimensional feature domain, needed to build a good predictive model which does not suffer from overfitting. Thus, applications of ML to omics typically require pre-processing steps to reduce the number of predictors in the input to the ML. These include feature selection, based on expert knowledge or computationally based, and feature transformations aimed at dimensionality reduction.</p></div></div></section><section data-title="Automated machine learning"><div class="c-article-section" id="Sec7-section"><h2 class="c-article-section__title js-section-title js-c-reading-companion-sections-item" id="Sec7">Automated machine learning</h2><div class="c-article-section__content" id="Sec7-content"><p>According to the No Free Lunch Theorems (Wolpert <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 1996" title="Wolpert DH (1996) The lack of a priori distinctions between learning algorithms. Neural Comput 8:1341–1390. 
 https://doi.org/10.1162/neco.1996.8.7.1341
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR84" id="ref-link-section-d56742713e781">1996</a>; Wolpert and Macready <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 1997" title="Wolpert DH, Macready WG (1997) No free lunch theorems for optimization. IEEE Trans Evol Comput 1:67–82. 
 https://doi.org/10.1109/4235.585893
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR85" id="ref-link-section-d56742713e784">1997</a>), there is no single ML algorithm that works well on all tasks. Every aspect of an ML application needs careful configuration and, as we have indicated above, setting up a pipeline requires considerable ML experience to best tune it for the specific task at hand. Therefore, methods which can assist in the design and optimization of ML pipelines, referred to as Automated Machine Learning (AutoML), are particularly appealing to biomedical investigators with limited data science expertise. There are approaches aimed at automating single tasks of an ML pipeline, such as feature engineering or hyperparameter optimization for a specified algorithm, but AutoML methods that can handle multiple tasks are particularly appealing as they provide off-the-shelf options for non-expert users. Below, we discuss several AutoML approaches, summarized in Table <a data-track="click" data-track-label="link" data-track-action="table anchor" href="/article/10.1007/s00439-021-02393-x#Tab1">1</a>. For an in-depth description of fundamentals and an extensive review of state-of-the-art AutoML methods, we refer the reader to Hutter et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2019" title="Hutter F, Kotthoff L, Vanschoren J (eds) (2019) Automated machine learning: methods, systems. Springer International Publishing, Challenges" href="/article/10.1007/s00439-021-02393-x#ref-CR31" id="ref-link-section-d56742713e790">2019</a>) and, with a healthcare perspective, to Waring et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2020" title="Waring J, Lindvall C, Umeton R (2020) Automated machine learning: review of the state-of-the-art and opportunities for healthcare. Artif Intell Med 104:101822. 
 https://doi.org/10.1016/j.artmed.2020.101822
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR83" id="ref-link-section-d56742713e793">2020</a>).</p><div class="c-article-table" data-test="inline-table" data-container-section="table" id="table-1"><figure><figcaption class="c-article-table__figcaption"><b id="Tab1" data-test="table-caption">Table 1 For each of the AutoML systems that we discussed, the architecture type of the resulting pipeline and optimization method are indicated together with the type of applications described in this work</b></figcaption><div class="u-text-right u-hide-print"><a class="c-article__pill-button" data-test="table-link" data-track="click" data-track-action="view table" data-track-label="button" rel="nofollow" href="/article/10.1007/s00439-021-02393-x/tables/1" aria-label="Full size table 1"><span>Full size table</span><svg width="16" height="16" focusable="false" role="img" aria-hidden="true" class="u-icon"><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#icon-eds-i-chevron-right-small"></use></svg></a></div></figure></div><h3 class="c-article__sub-heading" id="Sec8">Open-source AutoML pipeline optimization methods</h3><p>Here, we adopt the terminology by Waring et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2020" title="Waring J, Lindvall C, Umeton R (2020) Automated machine learning: review of the state-of-the-art and opportunities for healthcare. Artif Intell Med 104:101822. 
 https://doi.org/10.1016/j.artmed.2020.101822
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR83" id="ref-link-section-d56742713e1073">2020</a>) and refer to pipeline optimization AutoML as those methods which address more than one task in an ML pipeline. The three most popular open-source pipeline optimization AutoML systems to date are Auto-WEKA (Thornton et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2013" title="Thornton C, Hutter F, Hoos HH, Leyton-Brown K (2013) Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms. In: Proceeding of KDD-2013, pp 847–855. 
 https://doi.org/10.1145/2487575.2487629
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR74" id="ref-link-section-d56742713e1076">2013</a>; Kotthoff et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2019" title="Kotthoff L, Thornton C, Hoos HH et al (2017) Auto-WEKA 2.0: automatic model selection and hyperparameter optimization in WEKA. J Mach Learn Res 18:1–5" href="/article/10.1007/s00439-021-02393-x#ref-CR36" id="ref-link-section-d56742713e1079">2019</a>), built on top of the WEKA package; Auto-sklearn (Feurer et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2015" title="Feurer M, Eggensperger K, Falkner S et al (2018) Practical automated machine learning for the AutoML challenge. ICML 2018 AutoML Workshop. 
 https://ml.informatik.uni-freiburg.de/wp-content/uploads/papers/18-AUTOML-AutoChallenge.pdf
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR20" id="ref-link-section-d56742713e1082">2015</a>), built on top of the scikit-learn package; and Tree-based Pipeline Optimization Tool (TPOT) (Olson et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2016" title="Olson RS, Sipper M, Cava WL et al (2018) A system for accessible artificial intelligence. In: Banzhaf W, Olson RS, Tozier W, Riolo R (eds) Genetic programming theory and practice XV. Springer International Publishing, Cham, pp 121–134" href="/article/10.1007/s00439-021-02393-x#ref-CR56" id="ref-link-section-d56742713e1085">2016</a>; Olson and Moore <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2019" title="Olson RS, Moore JH (2019) TPOT: a tree-based pipeline optimization tool for automating machine learning. In: Hutter F, Kotthoff L, Vanschoren J (eds) Automated machine learning: methods, systems, challenges. Springer International Publishing, Cham, pp 151–160" href="/article/10.1007/s00439-021-02393-x#ref-CR54" id="ref-link-section-d56742713e1089">2019</a>), which also leverages scikit-learn. All three of these approaches aim to solve the Combined Algorithm Selection and Hyperparameter (CASH) optimization problem. The idea is to also model the choices of algorithms for pipeline steps as hyperparameters and then consider conditional dependencies between hyperparameters, i.e., a hyperparameter may be relevant only when other hyperparameters have certain values. For example, if the hyperparameter ‘estimator algorithm’ takes the value ‘random forest’, then the RF hyperparameters become relevant. Thus, the entire pipeline optimization task can be formulated in terms of a structured hyperparameter optimization problem. Auto-WEKA and Auto-sklearn optimize pipelines with a fixed architecture, in terms of the number and type of pipeline steps, whereas TPOT supports arbitrarily sized and complex pipelines by leveraging genetic programming as we illustrate below. Both Auto-WEKA and Auto-sklearn are based on Bayesian optimization (Brochu et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2010" title="Brochu E, Cora VM, de Freitas N (2010) A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. 
 http://arxiv.org/abs/1012.2599
 
 [cs]" href="/article/10.1007/s00439-021-02393-x#ref-CR8" id="ref-link-section-d56742713e1092">2010</a>). Bayesian optimization aims to find the optimal architecture quickly without reaching a premature sub-optimal architecture, by trading off exploration of new (hence high-uncertainty) regions of the search space with exploitation of known good regions. This is achieved by generating a probabilistic model capturing the relationship between hyperparameter settings and performance, using this model to select the next most promising hyperparameter setting, updating the model with the result from evaluation at the new setting, and iterating.</p><h4 class="c-article__sub-heading c-article__sub-heading--small" id="Sec9">Auto-WEKA</h4><p>Auto-WEKA is an Auto ML extension built on top of WEKA (discussed above), designed primarily for users without the technical expertise to know which particular ML algorithm or hyperparameter settings are ideal for the specific task they are performing. Auto-WEKA is a level of abstraction that treats the entirety of WEKA as a single ML algorithm comprised of other, specific ML algorithms:</p><div id="Equa" class="c-article-equation"><div class="c-article-equation__content"><span class="mathjax-tex">$$\mathcal{A}=\{{A}^{\left(1\right)},\dots ,{A}^{\left(k\right)}\}.$$</span></div></div><p>Briefly, the goal of Auto-WEKA is to find the correct algorithm <span class="mathjax-tex">${A}^{\left(i\right)}\in \mathcal{A}$</span>—and the correct hyperparameter settings for that algorithm—resulting in the best CV performance on a user-provided training data set. This is done using the previously mentioned Bayesian optimization approach for the CASH problem. A Bayesian approach provides the flexibility for specific applications to choose their own optimization model, but an effective one should be able to handle the tradeoff between model complexity and computational performance, while simultaneously choosing sensible prior distributions and initial parameterizations. The specific Bayesian optimization algorithm used in Auto-WEKA is Sequential Model-based Algorithm Configuration (SMAC), which is one of several evaluated during the project’s original development. From a high level, SMAC iteratively selects candidate algorithms for evaluation and uses both prior knowledge (based on similar problems) and results from previous iterations to propose new sets of hyperparameters for them that are likely to perform well on the learning task. At the beginning of an experiment, SMAC evaluates all candidate algorithms to identify their best initial parameter set <span class="mathjax-tex">${\varvec{\lambda}}$</span> based on their <i>expected positive improvement</i>, which is computed by optimizing the expectation of a loss function defined over the means and standard deviations of the parameters in <span class="mathjax-tex">${\varvec{\lambda}}$</span>. At each iteration, this value is maximized based on the current model, the models are evaluated using these new optimal parameters, the average CV loss of those parameters on a training data set is computed, and the model is updated based on the actual improved value. Specifically, this expectation is defined over <span class="mathjax-tex">${M}_{L}$</span>—a predictive model for the best algorithm along with its optimal hyperparameter configuration, which is implemented in SMAC as a random forest.</p><p>In the case of Auto-WEKA, the candidate algorithms consist of 39 (as of Auto-WEKA 2.0’s initial release) algorithms contained within WEKA, each of which falls into one of 4 categories: <i>learners</i>, <i>ensemble methods</i>, <i>meta-methods</i>, and <i>attribute selection methods</i> (i.e., feature selectors). A complete list of the candidate algorithms is given in Kotthoff et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2017" title="Kotthoff L, Thornton C, Hoos HH et al (2019) Auto-WEKA: automatic model selection and hyperparameter optimization in WEKA. In: Hutter F, Kotthoff L, Vanschoren J (eds) Automated machine learning. Springer International Publishing, Cham, pp 81–95" href="/article/10.1007/s00439-021-02393-x#ref-CR37" id="ref-link-section-d56742713e1286">2017</a>). “Prior knowledge” consists of a database of candidate algorithms and hyperparameter configurations previously evaluated within the same run of Auto-WEKA, and accordingly the estimation capacity of SMAC improves at each iteration of the algorithm’s main loop. At the beginning of an Auto-WEKA experiment, all parameters for all algorithms are assigned either a uniform or log-uniform prior (as semantically appropriate for each algorithm). Comprehensive details on SMAC and how it is used in conjunction with Auto-WEKA are given in Hutter et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2011" title="Hutter F, Hoos HH, Leyton-Brown K (2011) Sequential model-based optimization for general algorithm configuration. In: Coello CAC (ed) Learning and intelligent optimization. Springer, Berlin, pp 507–523" href="/article/10.1007/s00439-021-02393-x#ref-CR30" id="ref-link-section-d56742713e1290">2011</a>) and Kotthoff et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2019" title="Kotthoff L, Thornton C, Hoos HH et al (2017) Auto-WEKA 2.0: automatic model selection and hyperparameter optimization in WEKA. J Mach Learn Res 18:1–5" href="/article/10.1007/s00439-021-02393-x#ref-CR36" id="ref-link-section-d56742713e1293">2019</a>). When run on a set of training data, the output of Auto-WEKA is a trained model comprised of a specific ML algorithm with hyperparameters identified by SMAC. For example, this might consist of an RF for the algorithm and certain values for number of iterations, minimum number of instances per leaf, maximum depth of trees, and others as the optimized hyperparameters (these are specific to RF, and would change if a different ML algorithm was selected). In contrast, competing AutoML software implementations (such as Auto-sklearn and TPOT; see below) often produce pipelines comprising multiple steps (data preprocessors, estimators, and meta-operators) that manipulate the data in various ways that are often crucial in real-world applications of ML.</p><p>Biomedical applications of Auto-WEKA currently in the literature include using genotypes and biochemical laboratory values to predict liver fibrosis in patients with hepatitis C (Shousha et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2018" title="Shousha HI, Awad AH, Omran DA et al (2018) Data mining and machine learning algorithms using IL28B genotype and biochemical markers best predicted advanced liver fibrosis in chronic Hepatitis C. Jpn J Infect Dis 71:51–57. 
 https://doi.org/10.7883/yoken.JJID.2017.089
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR70" id="ref-link-section-d56742713e1299">2018</a>), predicting functional outcome scores for brain hemorrhage patients using combined demographic, laboratory, and radiometric imaging data (Wang et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2019" title="Wang C, Wu Q, Weimer M, Zhu E (2021) FLAML: a fast and lightweight AutoML Library. 
 http://arxiv.org/abs/1911.04706
 
 [cs, stat]" href="/article/10.1007/s00439-021-02393-x#ref-CR82" id="ref-link-section-d56742713e1302">2019</a>), and various applications in quantitative structure/activity relationship (QSAR) modeling (Nantasenamat et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2015" title="Nantasenamat C, Worachartcheewan A, Jamsak S et al (2015) AutoWeka: toward an automated data mining software for QSAR and QSPR studies. In: Cartwright H (ed) Artificial neural networks. Springer, New York, pp 119–147" href="/article/10.1007/s00439-021-02393-x#ref-CR52" id="ref-link-section-d56742713e1305">2015</a>).</p><h4 class="c-article__sub-heading c-article__sub-heading--small" id="Sec10">Auto-sklearn</h4><p>Auto-sklearn, like Auto-WEKA, tackles the CASH problem using the SMAC Bayesian optimization method, but it combines it with an initial warm starting step to improve efficiency and a final (optional) ensemble step to improve performance and reduce overfitting. The warm starting step consists of initializing the Bayesian optimizer with hyperparameter settings based on meta-learning. More precisely, an a priori step, performed just once by the Auto-sklearn maintainers, uses Bayesian optimization to determine optimal settings for a large number of data sets in the OpenML repository (Vanschoren et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2014" title="Vanschoren J, van Rijn JN, Bischl B, Torgo L (2014) OpenML: networked science in machine learning. SIGKDD Explor Newsl 15:49–60. 
 https://doi.org/10.1145/2641190.2641198
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR78" id="ref-link-section-d56742713e1316">2014</a>). Each data set in this repository is summarized by a set of meta-features, such as the number of data points, features, and classes, the data skewness, the entropy of the targets, etc. When Auto-sklearn is run on a new dataset, its meta-features are computed and the precomputed hyperparameter settings for the (25) most similar (based on meta-features) data sets in the repository are used to initialize the optimizer. Once warm started, the optimizer searches trough pipelines whose architecture consists of zero or one <i>feature preprocessors</i> (feature selectors or transformers which change the set of input features), up to three <i>data preprocessors</i> (transformers which change the feature values) and an estimator. There are several possible choices for the algorithms for each of these steps drawn from scikit-learn. At the end of optimization an optional post hoc step allows the user to request, instead of the best pipeline from the optimizer, an ensemble of the pipelines stored during optimization constructed using a method described in Caruana et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2004" title="Caruana R, Niculescu-Mizil A, Crew G, Ksikes A (2004) Ensemble selection from libraries of models. In: Proceedings of the twenty-first international conference on Machine learning. Association for Computing Machinery, New York, NY, USA, p 18" href="/article/10.1007/s00439-021-02393-x#ref-CR11" id="ref-link-section-d56742713e1325">2004</a>). When running Auto-sklearn, a user specifies the resource limits (memory and time) which is necessary, especially when working with large data sets. Of course, there is a tradeoff between resource limits and number of pipelines that can be tested.</p><p>Auto-sklearn has been applied very successfully to data sets from the ChaLearn AutoML challenges (Guyon et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2019" title="Guyon I, Sun-Hosoya L, Boullé M et al (2019) Analysis of the AutoML challenge series 2015–2018. In: Hutter F, Kotthoff L, Vanschoren J (eds) Automated machine learning: methods, systems, challenges. Springer International Publishing, Cham, pp 177–219" href="/article/10.1007/s00439-021-02393-x#ref-CR25" id="ref-link-section-d56742713e1331">2019</a>), winning in several phases of these challenges. A search on PubMed (<a href="http://pubmed.ncbi.nlm.nih.gov">http://pubmed.ncbi.nlm.nih.gov</a>) has yielded a few works employing Auto-sklearn in the biomedical context (Padmanabhan et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2019" title="Padmanabhan M, Yuan P, Chada G, Nguyen HV (2019) Physician-friendly machine learning: a case study with cardiovascular disease risk prediction. J Clin Med 8:1050. 
 https://doi.org/10.3390/jcm8071050
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR61" id="ref-link-section-d56742713e1341">2019</a>; Howard et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2020" title="Howard D, Maslej MM, Lee J et al (2020) Transfer learning for risk classification of social media posts: model evaluation study. J Med Internet Res 22:e15371. 
 https://doi.org/10.2196/15371
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR28" id="ref-link-section-d56742713e1344">2020</a>; Tran et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2020" title="Tran LM, Mocle AJ, Ramsaran AI et al (2020) Automated curation of CNMF-E-extracted ROI spatial footprints and calcium traces using open-source AutoML tools. Front Neural Circ. 
 https://doi.org/10.3389/fncir.2020.00042
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR76" id="ref-link-section-d56742713e1347">2020</a>). We have not identified in the literature any Auto-sklearn application to omics data, likely due to the challenging size of these types of data sets. As progress is being made towards handling large data sets more effectively, such as in the recent extension PoSH Auto-sklearn (Feurer et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2018" title="Feurer M, Klein A, Eggensperger K et al (2015) Efficient and robust automated machine learning. In: Cortes C, Lawrence N, Lee D et al (eds) Advances in neural information processing systems. Curran Associates, Inc., pp 2962–2970" href="/article/10.1007/s00439-021-02393-x#ref-CR19" id="ref-link-section-d56742713e1351">2018</a>), we expect to see the use of this system also in the omics field.</p><h4 class="c-article__sub-heading c-article__sub-heading--small" id="Sec11">TPOT</h4><p>Whereas Auto-WEKA and Auto-sklearn optimize pipelines with a fixed architecture, TPOT allows for arbitrarily sized ML pipelines. These pipelines involve operators (e.g. feature selectors, feature transformers, and ML estimators) drawn from scikit-learn and XGBoost (Chen and Guestrin <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2016" title="Chen T, Guestrin C (2016) XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, New York, NY, USA, pp 785–794" href="/article/10.1007/s00439-021-02393-x#ref-CR13" id="ref-link-section-d56742713e1362">2016</a>), as illustrated by the example in Fig. <a data-track="click" data-track-label="link" data-track-action="figure anchor" href="/article/10.1007/s00439-021-02393-x#Fig2">2</a>.</p><div class="c-article-section__figure js-c-reading-companion-figures-item" data-test="figure" data-container-section="figure" id="figure-2" data-title="Fig. 2"><figure><figcaption><b id="Fig2" class="c-article-section__figure-caption" data-test="figure-caption-text">Fig. 2</b></figcaption><div class="c-article-section__figure-content"><div class="c-article-section__figure-item"><a class="c-article-section__figure-link" data-test="img-link" data-track="click" data-track-label="image" data-track-action="view figure" href="/article/10.1007/s00439-021-02393-x/figures/2" rel="nofollow"><picture><img aria-describedby="Fig2" src="//media.springernature.com/lw685/springer-static/image/art%3A10.1007%2Fs00439-021-02393-x/MediaObjects/439_2021_2393_Fig2_HTML.png" alt="figure 2" loading="lazy" width="685" height="194"></picture></a></div><div class="c-article-section__figure-description" data-test="bottom-caption" id="figure-2-desc"><p>A hypothetical machine learning pipeline which could be discovered by TPOT. In the top branch of the pipeline, features are selected from a random forest (RF) analysis according to their importance scores and then subjected to a polynomial transformation. The transformed features are then analyzed using a <i>k</i>-nearest neighbors (kNN) algorithm with the output given to a decision tree (DT) as a new engineered feature. In the bottom branch, principal components (PCA) are analyzed by a support vector machine (SVM) with the output given to the DT. The DT performs the final classification using the newly engineered features from the kNN and SVM</p></div></div><div class="u-text-right u-hide-print"><a class="c-article__pill-button" data-test="article-link" data-track="click" data-track-label="button" data-track-action="view figure" href="/article/10.1007/s00439-021-02393-x/figures/2" data-track-dest="link:Figure2 Full size image" aria-label="Full size image figure 2" rel="nofollow"><span>Full size image</span><svg width="16" height="16" focusable="false" role="img" aria-hidden="true" class="u-icon"><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#icon-eds-i-chevron-right-small"></use></svg></a></div></figure></div><p>TPOT tackles the CASH problem using genetic programming (GP). It starts by generating an initial population of <i>N</i> random pipelines (the population size <i>N</i> defaults to 100 but can be user-specified) and evaluates them using the average <i>k</i>-fold CV score on the input data set (<i>k</i> defaults to 5 but can be user-specified), where the metric to be used for the score can be chosen from several available options. For each of <i>G</i> generations (<i>G</i> defaults to 100 but can be user-specified), the GP algorithm selects the top 20 pipelines in the current population according to a specific scheme (Deb et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2002" title="Deb K, Pratap A, Agarwal S, Meyarivan T (2002) A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans Evol Comput 6:182–197. 
 https://doi.org/10.1109/4235.996017
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR16" id="ref-link-section-d56742713e1411">2002</a>) that aims at optimizing the average CV score and minimizing the complexity, i.e. the number of steps. These pipelines produce the next generation of the population via transformations that mimic genetics, such as point mutations (random change of one of the pipeline operators) and cross-over of two pipelines (Fig. <a data-track="click" data-track-label="link" data-track-action="figure anchor" href="/article/10.1007/s00439-021-02393-x#Fig3">3</a>). At every generation, the algorithm updates a <i>Pareto front</i> of the pipelines discovered at any point in the GP run, where the Pareto front consists of those pipelines for which there is no other pipeline with both a better average CV score and a smaller complexity. This process iterates for the <i>G</i> generations, whereby adding and tuning pipeline operators that improve the average CV score and pruning those that degrade it. At the end, the algorithm selects the pipeline from the Pareto front with the best average CV score as the optimized pipeline and retrains it on the entire data set (i.e., without CV splits). As indicated earlier, it is good practice to evaluate the score of this pipeline on a hold-out testing set. Typically, one approach is to split the original data set into two portions; one (e.g., 75%) to be used as input to TPOT, and the other (e.g., the remaining 25%) as a hold-out set on which to assess the optimized pipeline. Actually, due to the stochasticity inherent to GP, it is also good practice to run TPOT multiple times with different such splits of the original data. Each such run yields a pipeline, and these pipelines can then be explored to get insights into the data. In particular, for each such pipeline, one can rank the features in terms of their importance, and the results can be combined across pipelines.</p><div class="c-article-section__figure js-c-reading-companion-figures-item" data-test="figure" data-container-section="figure" id="figure-3" data-title="Fig. 3"><figure><figcaption><b id="Fig3" class="c-article-section__figure-caption" data-test="figure-caption-text">Fig. 3</b></figcaption><div class="c-article-section__figure-content"><div class="c-article-section__figure-item"><a class="c-article-section__figure-link" data-test="img-link" data-track="click" data-track-label="image" data-track-action="view figure" href="/article/10.1007/s00439-021-02393-x/figures/3" rel="nofollow"><picture><img aria-describedby="Fig3" src="//media.springernature.com/lw685/springer-static/image/art%3A10.1007%2Fs00439-021-02393-x/MediaObjects/439_2021_2393_Fig3_HTML.png" alt="figure 3" loading="lazy" width="685" height="187"></picture></a></div><div class="c-article-section__figure-description" data-test="bottom-caption" id="figure-3-desc"><p>The essence of genetic programming-based optimization is the selection of good AutoML pipelines (parents) and the introduction of variability to generate new pipelines (children) for evaluation. On the left are two selected parental pipelines. In the first pipeline, features are selected according to their importance scores from a random forest (RF) analysis and then given to a decision tree (DT) which performs the classification. The second pipeline performs a k-nearest neighbors (kNN) and gradient boosting (GB) analysis with the output given to a naïve Bayes (NB) algorithm for classification. Two new pipelines are created by randomly swapping or recombining the RF and kNN algorithms and mutating the NB algorithm to a logistic regression (LR) algorithm. This results in two new pipelines to be evaluated</p></div></div><div class="u-text-right u-hide-print"><a class="c-article__pill-button" data-test="article-link" data-track="click" data-track-label="button" data-track-action="view figure" href="/article/10.1007/s00439-021-02393-x/figures/3" data-track-dest="link:Figure3 Full size image" aria-label="Full size image figure 3" rel="nofollow"><span>Full size image</span><svg width="16" height="16" focusable="false" role="img" aria-hidden="true" class="u-icon"><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#icon-eds-i-chevron-right-small"></use></svg></a></div></figure></div><p>Since TPOT was first introduced in Olson et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2016" title="Olson RS, Sipper M, Cava WL et al (2018) A system for accessible artificial intelligence. In: Banzhaf W, Olson RS, Tozier W, Riolo R (eds) Genetic programming theory and practice XV. Springer International Publishing, Cham, pp 121–134" href="/article/10.1007/s00439-021-02393-x#ref-CR56" id="ref-link-section-d56742713e1443">2016</a>), several specializations and extensions have been developed which were motivated by biomedical informatics applications. In Sohn et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2017" title="Sohn A, Olson RS, Moore JH (2017) Toward the automated analysis of complex diseases in genome-wide association studies using genetic programming. Proceedings of the genetic and evolutionary computation conference. Association for Computing Machinery, New York, pp 489–496" href="/article/10.1007/s00439-021-02393-x#ref-CR72" id="ref-link-section-d56742713e1446">2017</a>), a specialized version of TPOT is introduced that focuses on genetic analysis studies, named TPOT-MDR, where the TPOT search space is constrained to utilize pipelines whose steps use some special operators. The main special operator is an implementation of Multifactor Dimensionality Reduction (MDR), a non-parametric method that combines two or more features to create a single feature that captures their interaction effects (Ritchie et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2001" title="Ritchie GRS, Dunham I, Zeggini E, Flicek P (2014) Functional annotation of noncoding sequence variants. Nat Methods 11:294–296. 
 https://doi.org/10.1038/nmeth.2832
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR65" id="ref-link-section-d56742713e1449">2001</a>) and is, therefore, particularly suited to study epistasis in genetic analyses. Another useful operator employed in TPOT-MDR pipelines is the Expert Knowledge Filter (EKF), which allows feature selection based on statistical or biological filters. This is particularly relevant for applications to data sets comprising a large number of features.</p><p>Scalability is an important consideration for the applicability of AutoML to data sets stemming from the omics world. This has inspired two useful extensions of the standard TPOT framework that reduce TPOT computation time (Le et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2020" title="Le TT, Fu W, Moore JH (2020) Scaling tree-based automated machine learning to biomedical big data with a feature set selector. Bioinformatics 36:250–256. 
 https://doi.org/10.1093/bioinformatics/btz470
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR41" id="ref-link-section-d56742713e1455">2020</a>). One is the Template which lets the user specify the architecture of pipelines to be searched by TPOT and imposes restrictions on which operator can be chosen at each node. The other is the Feature Set Selector (FSS), which is used in combination with Template, whereby at the very first stage of each pipeline FSS passes only a specific subset of the features onwards. This essentially corresponds to slicing a potentially large original data set into smaller ones allowing TPOT to identify the feature subset that optimizes the <i>k</i>-fold CV score. Besides rendering the analyses of large data sets more feasible, the combination of Template and FSS also serves to generate more interpretable models.</p><p>Another extension of TPOT which was motivated by applications to biomedical informatics is the ability to adjust for covariates affecting features and/or target (resAdj TPOT). In Manduchi et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2020" title="Manduchi E, Le T, Fu W, Moore JH (2021) Genetic analysis of coronary artery disease using tree-based automated machine learning informed by biology-based feature selection. IEEE/ACM Trans Comput Biol Bioinform. 
 https://doi.org/10.1109/TCBB.2021.3099068
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR47" id="ref-link-section-d56742713e1465">2020</a>), an approach is presented to do this in a ‘leakage-free’ manner, meaning that the correction is applied in a way that prevents models built on the training part of a CV split from accessing information involving the validation part of the same split. Two other recent TPOT extensions are TPOT-NN, discussed below, and TPOT-cuML which provides a restricted configuration with GPU-accelerated estimators. Among the AutoML pipeline optimizers, TPOT has the most applications to date within the biomedical field, in particular omics applications, so we will discuss these in a separate section below.</p><h3 class="c-article__sub-heading" id="Sec12">Neural network AutoML</h3><p>Traditionally, AutoML software tends to build ML pipelines out of relatively simple candidate algorithms. This is due to various reasons, including the relative computational efficiency of training less complex models, portability of simple models to many domains, and sometimes ease of model introspection and interpretability. Despite these reasons, there is increasing interest in using artificial neural networks (ANNs/NNs) within the context of AutoML (Mendoza et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2019" title="Mendoza H, Klein A, Feurer M et al (2019) Towards automatically-tuned deep neural networks. In: Hutter F, Kotthoff L, Vanschoren J (eds) Automated machine learning. Springer International Publishing, Cham, pp 135–149" href="/article/10.1007/s00439-021-02393-x#ref-CR48" id="ref-link-section-d56742713e1477">2019</a>). NNs are mathematical approximations of biological neural networks, where sets of neurons (simple linear transformations composed with pointwise nonlinearities called <i>activation functions</i>) act by accepting input from one or more data points or other neurons, and potentially propagating a signal to subsequent groups of neurons based on whether the inputs pass a threshold specified by the activation function. Neurons are organized into <i>layers</i> that are stacked in a serial configuration. <i>Deep learning</i> is a branch of ML that uses NNs with many stacked layers; often tens or hundreds of layers. Specific arrangements of neurons result in different NN architectures, each of which has various performance advantages and disadvantages on certain tasks. NNs have exploded in popularity in recent decades, largely due to their ability to approximate any arbitrary function given sufficient size of the network, but also because computers have reached processing speeds that can adequately deal with the very large numbers of tunable parameters these networks contain (billions, in some cases).</p><p>The significant flexibility of NN architectures allows for a more general AutoML paradigm to be used when compared to non-NN applications of AutoML. As covered above, most AutoML can be summarized as pipeline optimization, which itself consists of several subtasks, including hyperparameter optimization, feature selection, and others. AutoML software that utilizes NNs can generally fall into two categories: (1) systems that aim to discover a larger NN architecture composed of smaller NN units, an approach known as Neural Architecture Search (NAS) (Elsken et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2019" title="Elsken T, Metzen JH, Hutter F (2019) Neural architecture search. In: Hutter F, Kotthoff L, Vanschoren J (eds) Automated machine learning. Springer International Publishing, Cham, pp 63–77" href="/article/10.1007/s00439-021-02393-x#ref-CR17" id="ref-link-section-d56742713e1492">2019</a>); and (2) systems that incorporate simpler, pre-specified NN architectures (such as multilayer perceptrons) as individual operators within a larger ML pipeline. In either of these approaches, NN layers can simultaneously accomplish feature selection (either by ‘dropping out’ or filtering uninformative features, or by explicitly highlighting important features through attention mechanisms (Wang et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2017" title="Wang F, Jiang M, Qian C, et al (2017) Residual attention network for image classification. 
 http://arxiv.org/abs/1704.06904
 
 [cs]" href="/article/10.1007/s00439-021-02393-x#ref-CR81" id="ref-link-section-d56742713e1495">2017</a>)), classification/regression, dimensionality reduction, and many other tasks that usually need to be explicitly modeled in non-NN systems. A major benefit of NAS is that it can implicitly perform any of these during a single optimization problem. The actual search process of NAS can be accomplished in a number of ways, including via random search, evolutionary methods, Bayesian optimization, and others, which comprise some of the main differences between existing NN-based AutoML systems. The second (non-NAS) approach to NN-based AutoML simply predefines neural networks, possibly with a dynamic number of layers and layer sizes encoded as hyperparameters.</p><p>There are several noteworthy AutoML systems that have successfully incorporated ANNs. Here, we provide a brief survey of some of these; we direct readers to (Mendoza et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2019" title="Mendoza H, Klein A, Feurer M et al (2019) Towards automatically-tuned deep neural networks. In: Hutter F, Kotthoff L, Vanschoren J (eds) Automated machine learning. Springer International Publishing, Cham, pp 135–149" href="/article/10.1007/s00439-021-02393-x#ref-CR48" id="ref-link-section-d56742713e1501">2019</a>) for a detailed technical analysis of some of these NN AutoML systems, as well as several others. AutoGluon—which is discussed more extensively below—is a Python AutoML tool developed by Amazon Web Services that supports NAS, and employs an ensemble stacking and bootstrap aggregation approach to constructing its estimators (Erickson et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2020" title="Erickson N, Mueller J, Shirkov A, et al (2020) AutoGluon-tabular: robust and accurate AutoML for structured data. 
 http://arxiv.org/abs/20030.6505
 
 [cs, stat]" href="/article/10.1007/s00439-021-02393-x#ref-CR18" id="ref-link-section-d56742713e1504">2020</a>). H2O, also discussed below, is a general-purpose AutoML system that uses a nearly identical approach to build estimators, and also supports NNs (specifically, a simple type of NN known as a multilayer perceptron) through its included deep learning module (Candel and LeDell <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Candel A, LeDell E (2021) Deep learning with H2O, 6th edn. H2O.ai, Inc., Mountain View" href="/article/10.1007/s00439-021-02393-x#ref-CR10" id="ref-link-section-d56742713e1507">2021</a>). H2O’s deep learning capabilities have been used successfully within several biomedical studies, most notably in the context of predicting estrogen receptor status using breast cancer metabolomics data, where the H2O deep learning approach outperformed a wide variety of alternate ML algorithms (Alakwaa et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2018" title="Alakwaa FM, Chaudhary K, Garmire LX (2018) Deep learning accurately predicts estrogen receptor status in breast cancer metabolomics data. J Proteome Res 17:337–347. 
 https://doi.org/10.1021/acs.jproteome.7b00595
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR5" id="ref-link-section-d56742713e1510">2018</a>). TPOT—which was discussed above—contains a submodule, named TPOT-NN, that provides a flexible framework for defining new NN operators (Romano et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Romano JD, Le TT, Fu W, Moore JH (2021) TPOT-NN: augmenting tree-based automated machine learning with neural network estimators. Genet Program Evol Mach. 
 https://doi.org/10.1007/s10710-021-09401-z
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR66" id="ref-link-section-d56742713e1513">2021</a>) that can be incorporated into AutoML pipelines aside non-NN operators supported by base-TPOT. TPOT-NN currently only provides NN implementations of logistic regression and multilayer perceptrons, but the developers have been testing complex neural architectures, including convolutional neural networks (e.g., for image classification) and recurrent neural networks (e.g., for text classification or application to time series data). Furthermore, TPOT-NN provides tools for users to extend the software to make use of any arbitrary NN-based operator that suits their needs.</p><p>Google has recently released a new NAS-based tool named Model Search, which is an open-source AutoML tool built on the neural computing library Tensorflow (which is, itself, another open-source software platform written by Google) that combines various strengths of previous AutoML systems. Briefly, Model Search’s strategy for identifying optimal architectures consists of training multiple candidate architectures in parallel, and the results for each candidate are saved in a database. The system then uses a heuristic search algorithm named beam search (Ow and Morton <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 1988" title="Ow PS, Morton TE (1988) Filtered beam search in scheduling†. Int J Prod Res 26:35–62. 
 https://doi.org/10.1080/00207548808947840
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR60" id="ref-link-section-d56742713e1519">1988</a>) to compare and rank the results of all candidate models. The best model is then mutated, and the process is repeated for a number of cycles until the system reaches some stopping criterion. Overall, this approach is similar to TPOT/TPOT-NN, where the ranking and mutation strategy of Model Search is analogous to TPOT’s GP optimization approach, but rather than constructing a pipeline from a pool of multi-purpose operators (which may consist of preprocessors, transformers, and a wide variety of ML algorithms), each of the blocks consists of a single neural network “motif” (e.g., a convolutional block, an LSTM block, a ResNet block, or others) that is composed into a larger neural network architecture. In other words, rather than a pipeline, Model Search finds a neural network architecture comprised of smaller blocks of neural network layers, potentially arranged in a branching tree-like configuration.</p><p>Several other examples of AutoML applications of NNs exist, but to lesser degrees of popularity. In general, NN applications within AutoML come with several important caveats. Due to the aforementioned large number of parameters, the time required to learn these NNs can be substantially greater than when the AutoML software only considers simpler (i.e., non-NN) estimators. This is sometimes alleviated by performing training on computers with certain hardware that can accelerate the training process (e.g., using CUDA-enabled graphics processing units), but such computing resources are costly and may not be available to all users. Furthermore, the fact that NNs are challenging to introspect and interpret may make these applications unsuitable when it is important to understand why the AutoML system made specific predictions. For example, consider an AutoML analysis of genotype data, where the goal is to predict phenotypic outcomes using SNPs of possibly unknown effects as inputs. If the AutoML system performs exceptionally well, the user may want to understand which SNPs are most predictive of the phenotypic outcome. With simpler ML building blocks (e.g., logistic regression, decision trees, etc.), it can be easy to interpret the contributions of these kinds of input features, but NNs could effectively obscure the effects of individual SNPs due to the complex nonlinear relationships between inputs and outputs of the network. Ultimately, these factors need to be considered on a case-by-case basis, and new research is needed to potentially alleviate these shortcomings.</p><h3 class="c-article__sub-heading" id="Sec13">Other AutoML approaches</h3><p>There are several other AutoML approaches, some of which we discuss in this section. We also point the interested readers to MLPlan (Mohr et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2018" title="Mohr F, Wever M, Hüllermeier E (2018) ML-Plan: automated machine learning via hierarchical planning. Mach Learn 107:1495–1515. 
 https://doi.org/10.1007/s10994-018-5735-z
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR49" id="ref-link-section-d56742713e1534">2018</a>), OBOE (Yang et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2019" title="Yang C, Akimoto Y, Kim DW, Udell M (2019) OBOE: collaborative filtering for AutoML model selection. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp 1173–1183. 
 https://doi.org/10.1145/3292500.3330909
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR86" id="ref-link-section-d56742713e1537">2019</a>), and Microsoft’s AutoML projects that include FLAML (Wang et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Wang H-L, Hsu W-Y, Lee M-H et al (2019) Automatic machine-learning-based outcome prediction in patients with primary intracerebral hemorrhage. Front Neurol 10:910. 
 https://doi.org/10.3389/fneur.2019.00910
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR80" id="ref-link-section-d56742713e1540">2021</a>) and NNI (<a href="https://www.microsoft.com/en-us/research/project/neural-network-intelligence/">https://www.microsoft.com/en-us/research/project/neural-network-intelligence/</a>).</p><h4 class="c-article__sub-heading c-article__sub-heading--small" id="Sec14">AutoGluon</h4><p>Recently, Amazon has open-sourced an AutoML tool named AutoGluon (<a href="https://auto.gluon.ai">https://auto.gluon.ai</a>), for Text, Image, and Tabular data. The AutoGluon Tabular (Erickson et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2020" title="Erickson N, Mueller J, Shirkov A, et al (2020) AutoGluon-tabular: robust and accurate AutoML for structured data. 
 http://arxiv.org/abs/20030.6505
 
 [cs, stat]" href="/article/10.1007/s00439-021-02393-x#ref-CR18" id="ref-link-section-d56742713e1564">2020</a>), which is the most relevant for applications of interest to this readership, does not focus on CASH optimization. Instead, it uses a custom set of base estimators (including RFs and NNs) in a multilayer stacked ensemble scheme. More precisely, in the base layer, these estimators are individually trained. Then, their aggregate predictions are added to the initial features and become the inputs of the next stacked layer, which consists of the same base estimators, and so on until the final step where an ensemble selection like that used in Auto-Sklearn (Caruana et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2004" title="Caruana R, Niculescu-Mizil A, Crew G, Ksikes A (2004) Ensemble selection from libraries of models. In: Proceedings of the twenty-first international conference on Machine learning. Association for Computing Machinery, New York, NY, USA, p 18" href="/article/10.1007/s00439-021-02393-x#ref-CR11" id="ref-link-section-d56742713e1567">2004</a>) is employed to aggregate the predictions from the last stacked layer in a weighted manner. AutoGluon automatically recognizes the data type for each feature and the type of prediction problem (e.g. regression, classification) and applies model-agnostic pre-processing that transforms the inputs to all estimators followed by model-specific pre-processing that is only applied to the inputs used in a particular estimator. Moreover, the layer-wise training is done in such a way to obtain high-quality data within an allotted time constraint. AutoGluon first estimates the required training time for each estimator in a layer and if this exceeds the remaining time for that layer (based on the allotted time constraint), it skips to the next layer. Base estimators have a predefined order so that the more reliable and less expensive models are trained prior to the less reliable and more expensive ones. Overfitting is mitigated throughout via a careful approach termed ‘repeated <i>k</i>-fold ensemble bagging’ which utilizes all the available data for both training and validation, ensuring that the higher layer models are only trained upon lower layer validation predictions. AutoGluon is among the methods evaluated by Seo et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Seo K, Chung B, Panchaseelan HP et al (2021) Forecasting the walking assistance rehabilitation level of stroke patients using artificial intelligence. Diagnostics (basel) 11:1096. 
 https://doi.org/10.3390/diagnostics11061096
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR68" id="ref-link-section-d56742713e1573">2021</a>) for forecasting the walking assistance rehabilitation level of stroke patients based on 82 features in 6 categories (anthropometry, stroke, blood tests, functional assessment, biosignal ward, and disease). We are not aware of omics applications of AutoGluon to date, probably due to its relatively recent release, but it is a promising method for this field.</p><h4 class="c-article__sub-heading c-article__sub-heading--small" id="Sec15">AutoPrognosis</h4><p>AutoPrognosis is an autoML system tailored to clinical prognosis and able to handle a diversity of clinical data types (including longitudinal and survival data). The approach uses an advanced Bayesian optimization technique to design a prognostic model consisting of a weighted ensemble of ML pipelines (Alaa and van der Schaar <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2018a" title="Alaa AM, van der Schaar M (2018a) AutoPrognosis: automated clinical prognostic modeling via Bayesian optimization with structured kernel learning. In: International conference on machine learning. PMLR, pp 139–148. 
 http://proceedings.mlr.press/v80/alaa18b.html
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR3" id="ref-link-section-d56742713e1584">2018a</a>). The system also provides explanations of its predictions in the form of logical association rules linking patients’ features to predicted risk strata. AutoPrognosis was used (Alaa et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2019" title="Alaa AM, van der Schaar M (2018a) AutoPrognosis: automated clinical prognostic modeling via Bayesian optimization with structured kernel learning. In: International conference on machine learning. PMLR, pp 139–148. 
 http://proceedings.mlr.press/v80/alaa18b.html
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR3" id="ref-link-section-d56742713e1587">2019</a>) to develop a cardiovascular disease risk predictor based on ~ 500 features, using a study of 423,604 UK Biobank (Bycroft et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2018" title="Bycroft C, Freeman C, Petkova D et al (2018) The UK Biobank resource with deep phenotyping and genomic data. Nature 562:203–209. 
 https://doi.org/10.1038/s41586-018-0579-z
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR9" id="ref-link-section-d56742713e1590">2018</a>) participants. Another application was prediction of short-term survival of cystic fibrosis (CF) patients using data from the UK CF registry (Alaa and van der Schaar <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2018b" title="Alaa AM, van der Schaar M (2018b) Prognostication and risk factors for cystic fibrosis via automated machine learning. Sci Rep 8:11242. 
 https://doi.org/10.1038/s41598-018-29523-2
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR4" id="ref-link-section-d56742713e1593">2018b</a>).</p><h4 class="c-article__sub-heading c-article__sub-heading--small" id="Sec16">H2O</h4><p>H2O is a commercial entity providing cloud-based machine learning services. Components of their software are freely available and open-source including software for AutoML. The H2O AutoML software includes a grid search and Bayesian optimization algorithms for hyperparameter tuning and the use of the Super Learner algorithm (van der Laan et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2007" title="van der Laan MJ, Polley EC, Hubbard AE (2007) Super learner. Stat Appl Genetics Mol Biol. 
 https://doi.org/10.2202/1544-6115.1309
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR77" id="ref-link-section-d56742713e1604">2007</a>) which combines multiple machine learning algorithms as an ensemble for prediction (LeDell and Poirier <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2020" title="LeDell E, Poirier S (2020) H2O AutoML: scalable automatic machine learning. In: 7th ICML workshop on automated machine learning. 
 https://www.automl.org/wp-content/uploads/2020/07/AutoML_2020_paper_61.pdf
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR42" id="ref-link-section-d56742713e1607">2020</a>). Super Learner is more extensively discussed below.</p><p>H2O, Auto-WEKA, Auto-sklearn, and TPOT are part of an open-source, extensible, and ongoing benchmark for AutoML frameworks publishing online the latest results on the performance of these tools on public datasets (Gijsbers et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2019" title="Gijsbers P, LeDell E, Thomas J, et al (2019) An open source AutoML benchmark. 
 http://arxiv.org/abs/1907.00909
 
 [cs, stat]" href="/article/10.1007/s00439-021-02393-x#ref-CR23" id="ref-link-section-d56742713e1613">2019</a>).</p><h4 class="c-article__sub-heading c-article__sub-heading--small" id="Sec17">PennAI</h4><p>PennAI was designed as an accessible and user-friendly AutoML software package for non-experts (Olson et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2018" title="Olson RS, Urbanowicz RJ, Andrews PC et al (2016) Automating biomedical data science through tree-based pipeline optimization. In: Squillero G, Burelli P (eds) Applications of evolutionary computation. Springer International Publishing, Cham, pp 123–137" href="/article/10.1007/s00439-021-02393-x#ref-CR55" id="ref-link-section-d56742713e1625">2018</a>). It features the scikit-learn library for machine learning, a controller for launching jobs, a database for storing machine learning results as a memory of the system, a singular-value decomposition (SVD) algorithm-based recommender system, and a user-friendly interface accessible via web browser. The recommender system analyzes previous machine learning results from the database and automatically launches and runs new analyses. PennAI has been shown to be competitive with Auto-Sklearn and HyperOpt, an automated hyperparameter tuner (Bergstra et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2015" title="Bergstra J, Komer B, Eliasmith C et al (2015) Hyperopt: a Python library for model selection and hyperparameter optimization. Comput Sci Discov 8:014008. 
 https://doi.org/10.1088/1749-4699/8/1/014008
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR7" id="ref-link-section-d56742713e1628">2015</a>; Komer et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2019" title="Komer B, Bergstra J, Eliasmith C (2019) Hyperopt-sklearn. In: Hutter F, Kotthoff L, Vanschoren J (eds) Automated machine learning: methods, systems, challenges. Springer International Publishing, Cham, pp 97–111" href="/article/10.1007/s00439-021-02393-x#ref-CR34" id="ref-link-section-d56742713e1631">2019</a>). PennAI has been applied to biomedical data in (La Cava et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2020" title="La Cava W, Williams H, Fu W et al (2020) Evaluating recommender systems for AI-driven biomedical informatics. Bioinformatics. 
 https://doi.org/10.1093/bioinformatics/btaa698
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR40" id="ref-link-section-d56742713e1634">2020</a>). We are not aware of any omics applications of PennAI.</p></div></div></section><section data-title="TPOT omics applications"><div class="c-article-section" id="Sec18-section"><h2 class="c-article-section__title js-section-title js-c-reading-companion-sections-item" id="Sec18">TPOT omics applications</h2><div class="c-article-section__content" id="Sec18-content"><p>TPOT has been applied in several omics contexts. Transcriptomics was one of the motivations for the FSS and Template extensions described above and a first application to RNAseq data from individuals with or without major depressive disorder was presented in that paper (Le et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2020" title="Le TT, Fu W, Moore JH (2020) Scaling tree-based automated machine learning to biomedical big data with a feature set selector. Bioinformatics 36:250–256. 
 https://doi.org/10.1093/bioinformatics/btz470
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR41" id="ref-link-section-d56742713e1647">2020</a>). In Manduchi et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2020" title="Manduchi E, Le T, Fu W, Moore JH (2021) Genetic analysis of coronary artery disease using tree-based automated machine learning informed by biology-based feature selection. IEEE/ACM Trans Comput Biol Bioinform. 
 https://doi.org/10.1109/TCBB.2021.3099068
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR47" id="ref-link-section-d56742713e1650">2020</a>), two more extensive transcriptomics applications can be found. The first used a toxicogenomics Affymetrics microarray data set (1693 features and 933 samples) to build models leading to the identification of pathways and genes whose expression is associated with creatinine levels in rat kidney, after utilizing the covariate adjustment extension introduced in that paper to remove confounding effects such as study batch, compound treatment, dose, and sacrifice time. The second used an RNAseq expression data set (4952 features and 1072 samples) to build models leading to the identification of pathways associated with differential expression between schizophrenic and control individuals. Interestingly, the latter yielded known pathways which could not be detected by the more popular gene set enrichment tool GSEA (Subramanian et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2005" title="Subramanian A, Tamayo P, Mootha VK et al (2005) Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles. Proc Natl Acad Sci USA 102:15545–15550. 
 https://doi.org/10.1073/pnas.0506580102
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR73" id="ref-link-section-d56742713e1653">2005</a>).</p><p>Metabolomics is another area where TPOT has been successfully applied. The first application in this field was presented in Orlenko et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2018" title="Orlenko A, Moore JH, Orzechowski P et al (2018) Considerations for automated machine learning in clinical metabolic profiling: altered homocysteine plasma concentration associated with metformin exposure. Pac Symp Biocomput 23:460–471" href="/article/10.1007/s00439-021-02393-x#ref-CR59" id="ref-link-section-d56742713e1659">2018</a>) to study type 2 diabetes patients with glycemic control exposed to metformin monotherapy as compared to matched healthy controls. In a second metabolomics TPOT application, described in Orlenko et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2020" title="Orlenko A, Moore JH (2021) A comparison of methods for interpreting random forest models of genetic association in the presence of non-additive interactions. BioData Min 14:9. 
 https://doi.org/10.1186/s13040-021-00243-0
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR57" id="ref-link-section-d56742713e1662">2020</a>), a cohort of 925 patients is analyzed using 73 metabolic and 27 demographic and clinical features with respect to an endpoint of obstructive, non-obstructive, and no Coronary Artery Disease (CAD). Interestingly, in this work, the pipeline discovered by TPOT as having the best classification performance (see Fig. <a data-track="click" data-track-label="link" data-track-action="figure anchor" href="/article/10.1007/s00439-021-02393-x#Fig4">4</a>) includes the Bernoulli Naïve Bayes classifier. The latter is typically employed in text analyses for spam detection and is rarely considered for biomedical predictive analyses, so it would unlikely be used in a manually configured ML pipeline for this task. The most recent metabolomics application of TPOT utilizes > 500 measurements to build predictive models of early childhood caries, in Heimisdottir et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Heimisdottir LH, Lin BM, Cho H et al (2021) Metabolomics insights in early childhood caries. J Dent Res. 
 https://doi.org/10.1177/0022034520982963
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR26" id="ref-link-section-d56742713e1668">2021</a>).</p><div class="c-article-section__figure js-c-reading-companion-figures-item" data-test="figure" data-container-section="figure" id="figure-4" data-title="Fig. 4"><figure><figcaption><b id="Fig4" class="c-article-section__figure-caption" data-test="figure-caption-text">Fig. 4</b></figcaption><div class="c-article-section__figure-content"><div class="c-article-section__figure-item"><a class="c-article-section__figure-link" data-test="img-link" data-track="click" data-track-label="image" data-track-action="view figure" href="/article/10.1007/s00439-021-02393-x/figures/4" rel="nofollow"><picture><img aria-describedby="Fig4" src="//media.springernature.com/lw685/springer-static/image/art%3A10.1007%2Fs00439-021-02393-x/MediaObjects/439_2021_2393_Fig4_HTML.png" alt="figure 4" loading="lazy" width="685" height="286"></picture></a></div><div class="c-article-section__figure-description" data-test="bottom-caption" id="figure-4-desc"><p>An optimal TPOT pipeline derived from the analysis of metabolomics data (Orlenko et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2020" title="Orlenko A, Moore JH (2021) A comparison of methods for interpreting random forest models of genetic association in the presence of non-additive interactions. BioData Min 14:9. 
 https://doi.org/10.1186/s13040-021-00243-0
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR57" id="ref-link-section-d56742713e1681">2020</a>). In the first step of the pipeline, an Extra Trees analysis is performed with recursive feature elimination to select a subset of most informative features. These selected features are then analyzed using Logistic Regression (LR) and the output included in the data set as a newly engineered feature. This same process is then repeated using a multinomial naïve Bayes (MNB) algorithm. The selected and engineered features are then scaled by subtracting the mean and dividing by the standard deviation. These newly transformed features are then used to classify subjects as cases with coronary artery disease (CAD) or controls with no CAD using a Bernoulli Naïve Bayes (BNB) classifier</p></div></div><div class="u-text-right u-hide-print"><a class="c-article__pill-button" data-test="article-link" data-track="click" data-track-label="button" data-track-action="view figure" href="/article/10.1007/s00439-021-02393-x/figures/4" data-track-dest="link:Figure4 Full size image" aria-label="Full size image figure 4" rel="nofollow"><span>Full size image</span><svg width="16" height="16" focusable="false" role="img" aria-hidden="true" class="u-icon"><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#icon-eds-i-chevron-right-small"></use></svg></a></div></figure></div><p>As for genomics, besides applications of TPOT (Olson et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2016" title="Olson RS, Sipper M, Cava WL et al (2018) A system for accessible artificial intelligence. In: Banzhaf W, Olson RS, Tozier W, Riolo R (eds) Genetic programming theory and practice XV. Springer International Publishing, Cham, pp 121–134" href="/article/10.1007/s00439-021-02393-x#ref-CR56" id="ref-link-section-d56742713e1695">2016</a>) and TPOT-MDR (Sohn et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2017" title="Sohn A, Olson RS, Moore JH (2017) Toward the automated analysis of complex diseases in genome-wide association studies using genetic programming. Proceedings of the genetic and evolutionary computation conference. Association for Computing Machinery, New York, pp 489–496" href="/article/10.1007/s00439-021-02393-x#ref-CR72" id="ref-link-section-d56742713e1698">2017</a>) to a data set extracted from a GWAS on prostate cancer aggressiveness, more recently (Manduchi et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Manduchi E, Moore JH (2021) Leveraging automated machine learning for the analysis of global public health data: a case study in malaria. Int J Public Health. 
 https://doi.org/10.3389/ijph.2021.614296
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR45" id="ref-link-section-d56742713e1701">2021</a>) TPOT and resAdj TPOT were used to analyze a large CAD data set extracted from the UK Biobank, consisting of > 19,000 cases and > 320,000 controls. Functional genomics data from Roadmap Epigenomics (Kundaje et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2015" title="Kundaje A, Meuleman W, Ernst J et al (2015) Integrative analysis of 111 reference human epigenomes. Nature 518:317–330. 
 https://doi.org/10.1038/nature14248
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR39" id="ref-link-section-d56742713e1704">2015</a>), previous results on putative CAD druggable genes (Tragante et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2018" title="Tragante V, Hemerich D, Alshabeeb M et al (2018) Druggability of coronary artery disease risk loci. Circulation 11:e001977. 
 https://doi.org/10.1161/CIRCGEN.117.001977
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR75" id="ref-link-section-d56742713e1707">2018</a>), and an integrative network resource (<a href="https://het.io/">https://het.io/</a>) were employed as a biology-based feature filter. A 2-stage TPOT approach yielded the identification of a recurrent signal from a subset of 28 SNPs and feature importance analyses uncovered links between the top SNPs in this subset and genes related to atherosclerotic plaques and myocardial infarction. We will return to this example below where we discuss the challenges presented to AutoML by full-fledged GWAS data sets and potential directions for future development.</p><p>Radiomics has also seen an application where TPOT was employed to determine the prognosis of clear cell renal cell carcinoma prior to any invasive therapy on the basis of > 6000 MRI-based features in a cohort of 374 patients (Choi et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Choi JW, Hu R, Zhao Y et al (2021) Preoperative prediction of the stage, size, grade, and necrosis score in clear cell renal cell carcinoma using MRI-based radiomics. Abdom Radiol (NY). 
 https://doi.org/10.1007/s00261-020-02876-x
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR14" id="ref-link-section-d56742713e1721">2021</a>). Aside from the omics area, TPOT is suitable for other types of biomedical predictive tasks, including public health, as recently highlighted in Manduchi and Moore (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Manduchi E, Fu W, Romano JD et al (2020) Embedding covariate adjustments in tree-based automated machine learning for biomedical big data analyses. BMC Bioinform 36:1772" href="/article/10.1007/s00439-021-02393-x#ref-CR46" id="ref-link-section-d56742713e1724">2021</a>).</p></div></div></section><section data-title="AutoML in genetics; promises, challenges, future directions"><div class="c-article-section" id="Sec19-section"><h2 class="c-article-section__title js-section-title js-c-reading-companion-sections-item" id="Sec19">AutoML in genetics; promises, challenges, future directions</h2><div class="c-article-section__content" id="Sec19-content"><p>Potential applications of AutoML to genetics and genomics are not limited to predictive models for traits based on features derived from genotyping assays. For example, AutoML could be used to possibly improve on methods such as the SNP deleteriousness scorers described earlier. However, the possibility of using AutoML to study phenotype-genotype relationships is probably the most appealing as it offers a means to automatically explore associations that go beyond those investigated with traditional GWAS approaches. It might be ideally suited to genetic analysis in the presence of epistasis, genetic heterogeneity, and gene–environment interactions. These are all genetic phenomena which tend to be non-additive thus requiring machine learning to complement parametric approaches. The challenge for modeling interactions and heterogeneity is knowing which pre-processing and machine learning algorithms are the right ones to use. AutoML has the potential to improve machine learning results by making fewer assumptions about the right methods to use.</p><p>There are two main purposes that may drive ML, hence AutoML, applications in the study of phenotype–genotype relationships. In the first and most frequent scenario, the main goal is to identify the genes associated to a given trait. This includes applications where ML is directly used to discover relevant features (e.g., Adams et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2020" title="Adams SM, Feroze H, Nguyen T et al (2020) Genome wide epistasis study of on-statin cardiovascular events with iterative feature reduction and selection. J Pers Med. 
 https://doi.org/10.3390/jpm10040212
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR1" id="ref-link-section-d56742713e1738">2020</a>; Manduchi et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Manduchi E, Moore JH (2021) Leveraging automated machine learning for the analysis of global public health data: a case study in malaria. Int J Public Health. 
 https://doi.org/10.3389/ijph.2021.614296
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR45" id="ref-link-section-d56742713e1741">2021</a>), in which case the emphasis is on feature importance and, more generally, interpretability. It also includes applications where ML is used for post-GWAS prioritization; we are not focusing on these but refer to (Nicholls et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2020" title="Nicholls HL, John CR, Watson DS et al (2020) Reaching the end-game for GWAS: machine learning approaches for the prioritization of complex disease loci. Front Genet 11:350. 
 https://doi.org/10.3389/fgene.2020.00350
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR53" id="ref-link-section-d56742713e1744">2020</a>) for a review. In the second scenario, often motivated by precision medicine applications, the main interest is in the immediate output of the ML i.e., the predictive model itself (Li et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Li L, Yang Y, Zhang Q et al (2021) Use of deep-learning genomics to discriminate healthy individuals from those with Alzheimer’s disease or mild cognitive impairment. Behav Neurol 2021:3359103. 
 https://doi.org/10.1155/2021/3359103
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR43" id="ref-link-section-d56742713e1747">2021</a>; Huang et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Huang S, Ji X, Cho M, et al (2021) DL-PRS: a novel deep learning approach to polygenic risk scores. BMC Bioinformatics. 
 https://doi.org/10.21203/rs.3.rs-423764/v1
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR29" id="ref-link-section-d56742713e1750">2021</a>), in which case the emphasis is on predictive ability.</p><p>A GWAS data set can present many challenges to AutoML analyses. First, the search space is very large, both in terms of number of samples and features (SNPs). Second, the signal is usually weak as common variants typically have small effect sizes. In fact, one of the reasons GWAS comprise very large numbers of samples (often combining several cohorts under the umbrella of consortia) is to provide enough power to detect such signals. Third, a trait may present genetic heterogeneity, meaning that different variants may be responsible for that trait in different individuals, and this increases the difficulty in detecting the signals. Finally, for binary traits, often there is a large imbalance between the number of cases and controls. This is, for example, true for data collected as part of national biobanks, like the UK Biobank (<a href="https://www.ukbiobank.ac.uk/">https://www.ukbiobank.ac.uk/</a>) or All of Us (<a href="https://allofus.nih.gov/">https://allofus.nih.gov/</a>). To date, applications of AutoML to GWAS data are very few and we have indicated those that we know of above. Computational feasibility is certainly a major hurdle. Reducing the number of SNPs is generally the first step and is also needed to handle the <i>p</i> > > <i>n</i> issue. Systems like Auto-sklearn and TPOT comprise various (computationally based) feature selectors and transformers among the operators of the explored pipelines, and TPOT Template allows to specify that one of these operators should be the first step of any explored pipeline. In these pipelines, the dimensionality of what is passed to subsequent classifier/regressor operators is, therefore, reduced. Moreover, the entire pipeline, including the feature selector/transformer step, is assessed via CV. However, with GWAS data containing millions of SNPs (the large majority of which have values in {0, 1, 2}), in current AutoML systems, it is still typically necessary to filter SNPs prior to running the program. Employing computationally based filters to this end may not be ideal, also because this process would not be leakage free. Filters based on domain-specific knowledge are independent of the training data and provide an attractive approach, especially when the emphasis is on interpretability. When using this type of filter, the goal is no longer to look for all possible signals of interest but to focus on a promising subset. SNP filters can be based on biological knowledge, e.g. restricting the analyses to SNPs in pathways that are known to be relevant to the trait, as is done in Olson et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2016" title="Olson RS, Sipper M, Cava WL et al (2018) A system for accessible artificial intelligence. In: Banzhaf W, Olson RS, Tozier W, Riolo R (eds) Genetic programming theory and practice XV. Springer International Publishing, Cham, pp 121–134" href="/article/10.1007/s00439-021-02393-x#ref-CR56" id="ref-link-section-d56742713e1776">2016</a>) and Sohn et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2017" title="Sohn A, Olson RS, Moore JH (2017) Toward the automated analysis of complex diseases in genome-wide association studies using genetic programming. Proceedings of the genetic and evolutionary computation conference. Association for Computing Machinery, New York, pp 489–496" href="/article/10.1007/s00439-021-02393-x#ref-CR72" id="ref-link-section-d56742713e1780">2017</a>). Another approach, as in Manduchi et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Manduchi E, Moore JH (2021) Leveraging automated machine learning for the analysis of global public health data: a case study in malaria. Int J Public Health. 
 https://doi.org/10.3389/ijph.2021.614296
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR45" id="ref-link-section-d56742713e1783">2021</a>), is to integrate various resources, including functional genomics data from tissues that are relevant to the trait as well as the computational scorers for variant deleteriousness described above. But there are difficulties that come along with this. For example, for some traits, the culprit tissue(s) may not be fully known or the functional genomics data for known culprit tissues may not be available. Even when the tissues are known and the data are available, the reliability of the derived regulatory regions (e.g., enhancers) or SNP functional scores are tied to the computational methods used and it is possible that the trait-relevant SNPs are erroneously removed from the search space. In addition, filtering of SNPs as just described, typically still yields more features than an AutoML system can handle. To overcome this, the FSS option in TPOT can be valuable. For example, in Manduchi et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Manduchi E, Moore JH (2021) Leveraging automated machine learning for the analysis of global public health data: a case study in malaria. Int J Public Health. 
 https://doi.org/10.3389/ijph.2021.614296
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR45" id="ref-link-section-d56742713e1786">2021</a>), SNP sets are created based on pairs of connected genes in the Hetionet integrative network and each searched pipeline starts with the step of selecting one of these sets. An additional advantage of using the FSS is that it is also a way of facilitating interpretability. More efficient handling in AutoML of feature selection and feature transformation approaches in general, represents an avenue for future developments to improve AutoML applicability to genomics. Including selectors that are not based on main effects, like the RELIEFF based selectors (Kononenko et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 1997" title="Kononenko I, Šimec E, Robnik-Šikonja M (1997) Overcoming the myopia of inductive learning algorithms with RELIEFF. Appl Intell 7:39–55. 
 https://doi.org/10.1023/A:1008280620621
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR35" id="ref-link-section-d56742713e1789">1997</a>) used in TPOT-MDR, is desirable to better explore potential epistatic effects. As for feature transformations, efficient incorporation within the AutoML of feature fusion approaches such as, for example, that discussed in Venugopalan et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Venugopalan J, Tong L, Hassanzadeh HR, Wang MD (2021) Multimodal deep learning models for early detection of Alzheimer’s disease stage. Sci Rep 11:3254. 
 https://doi.org/10.1038/s41598-020-74399-w
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR79" id="ref-link-section-d56742713e1792">2021</a>), is another interesting area for development.</p><p>While filtering and engineering SNPs to reduce dimensionality is an important area of investigation, it is important to note that there are machine learning algorithms such as deep learning neural networks and gradient boosting methods which can scale to genome-wide genetic and genomics data. These are especially attractive when the emphasis is on predictive ability. For example, deep learning has been applied to the analysis of GWAS data from studies of Alzheimer’s disease (Li et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Li L, Yang Y, Zhang Q et al (2021) Use of deep-learning genomics to discriminate healthy individuals from those with Alzheimer’s disease or mild cognitive impairment. Behav Neurol 2021:3359103. 
 https://doi.org/10.1155/2021/3359103
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR43" id="ref-link-section-d56742713e1798">2021</a>). One of the challenges of deep learning is that there are a number of hyperparameters which need to be tuned. The Auto-Net methods was designed to automate the construction and hyperparameter tuning of deep learning models (Mendoza et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2019" title="Mendoza H, Klein A, Feurer M et al (2019) Towards automatically-tuned deep neural networks. In: Hutter F, Kotthoff L, Vanschoren J (eds) Automated machine learning. Springer International Publishing, Cham, pp 135–149" href="/article/10.1007/s00439-021-02393-x#ref-CR48" id="ref-link-section-d56742713e1801">2019</a>). This approach is synergistic with Auto-Sklearn and, as discussed above, opens the door to AutoML using neural networks. The scaling of deep learning to GWAS and other genome-wide studies with AutoML methods such as Auto-Net needs to be evaluated. Another approach to address scalability is to take advantage of an increasing number of cloud-based commercial AutoML solutions. These are quickly becoming mature and are integrated with cloud-based high-performance computing which makes scaling easy. The advantages and disadvantages of these solutions for AutoML analysis of genome-wide data should also be explored.</p><p>Even after filtering SNPs and grouping them in feature sets, the number of samples from GWAS is typically too large for current AutoML capabilities. In this case though, rather than filtering subjects, one could instead run the AutoML several times using different subsets of the subjects. For example, in Manduchi et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2021" title="Manduchi E, Moore JH (2021) Leveraging automated machine learning for the analysis of global public health data: a case study in malaria. Int J Public Health. 
 https://doi.org/10.3389/ijph.2021.614296
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR45" id="ref-link-section-d56742713e1808">2021</a>), TPOT is run multiple times, each time using all available cases and a subset of the controls of the same size as the cases, which also has the effect of balancing the input of each run. Other schemes are possible and could involve imbalance-aware approaches such as that described in Schubach et al. (<a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2017" title="Schubach M, Re M, Robinson PN, Valentini G (2017) Imbalance-aware machine learning for predicting rare and common disease-associated non-coding variants. Sci Rep 7:2959. 
 https://doi.org/10.1038/s41598-017-03011-5
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR67" id="ref-link-section-d56742713e1811">2017</a>), leveraging packages like the Python imbalanced-learn (<a href="https://imbalanced-learn.org/stable/">https://imbalanced-learn.org/stable/</a>). Another interesting avenue to explore, for better tailoring of AutoML to genomics in view of large sample sizes, is incorporation of approaches like FABOLAS (Klein et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2017" title="Klein A, Falkner S, Bartels S et al (2017) Fast Bayesian hyperparameter optimization on large datasets. Electron J Statist. 
 https://doi.org/10.1214/17-EJS1335SI
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR33" id="ref-link-section-d56742713e1821">2017</a>), a Bayesian optimization method that evaluates models on subsets of the data to learn good hyperparameter settings, aiming to assess configurations that yield, per time spent, the most information about the globally best hyperparameters for the full dataset.</p><p>We note that, even with SNP filtering and multiple runs on subsets of the samples, the computational resource requirements can be demanding and having the availability of a computer cluster for parallel runs of the AutoML is highly desirable. As more progress is made to leverage GPU for AutoML to improve scalability and GPU cost decrease, we expect analysis of GWAS data using these systems to become more common.</p><p>When a given AutoML is run multiple times, either for the reasons indicated in the previous example or simply due to inherent stochasticity (like in TPOT), multiple optimal pipelines are generated. When the emphasis is on interpretability, feature importance can be computed for each such pipeline and the results combined to get insights. But it is also possible to combine these pipelines with ensemble methods, either simple approaches such as voting or more sophisticated approaches such as Super Learner, so to obtain a single model at the end, which may outperform the individual models upon which it is built. (This is useful both when the emphasis is on interpretability and predictive ability.) Super Learner is an ensemble-based algorithm that is particularly attractive for AutoML for two primary reasons. First, it is compatible with any arbitrary machine learning algorithm to comprise its individual components. Other ensemble methods often do not have this flexibility—for example, gradient tree boosting algorithms rely on individual learning functions with differentiable losses (many sophisticated AutoML approaches generate learners that might not have computable gradients). Furthermore, there are often technical limitations related to the complexity of the individual candidate learners, such as AdaBoost relying on simple and quickly trained candidate learners that perform classification or regression only slightly better than average. Second, Super Learner is analytically known to (asymptotically) produce an ensemble learner that is as accurate as the best possible prediction algorithm. Briefly, Super Learner trains <i>V</i> arbitrary candidate learners, each on a collection of data with a different validation block of samples held out from the learning process. Each candidate learner is then used to predict outcomes on samples from their individual validation blocks, and the overall body of predictions is then used to train a regression model that assigns relative weights to the predictions of each candidate algorithm. Super Learner—out of the context of AutoML—has been successfully used to a substantial degree in biomedical data science (Sinisi et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2007" title="Sinisi SE, Polley EC, Petersen ML et al (2007) Super learning: an application to the prediction of HIV-1 drug resistance. Stat Appl Genet Mol Biol. 
 https://doi.org/10.2202/1544-6115.1240
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR71" id="ref-link-section-d56742713e1833">2007</a>; van der Laan et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2007" title="van der Laan MJ, Polley EC, Hubbard AE (2007) Super learner. Stat Appl Genetics Mol Biol. 
 https://doi.org/10.2202/1544-6115.1309
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR77" id="ref-link-section-d56742713e1836">2007</a>; Golmakani and Polley <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2020" title="Golmakani MK, Polley EC (2020) Super learner for survival data prediction. Int J Biostat. 
 https://doi.org/10.1515/ijb-2019-0065
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR24" id="ref-link-section-d56742713e1839">2020</a>). TPOT is currently in the process of developing an extension that combines individual AutoML pipelines into a Super Learner.</p><p>One of the important limitations of many machine learning methods and applications is that only one objective (e.g., classifier accuracy or area under the ROC curve) is used to evaluate the quality of the model. This may work well for some problems. However, there may be additional objectives of importance for problems in genetic and genomics. For example, a machine learning model designed to identify genes representing new drug targets might care about whether there is evidence that the protein products are druggable. A model with no druggable genes might predict a phenotype with a high accuracy but might not be useful or interesting from a pharmacology perspective. Fortunately, there are methods for assessing the quality of a model using multiple objectives. An example of a multi-objective algorithm is Pareto optimization which ranks models according to two or more objectives. Models are referred to as Pareto optimal if one objective cannot be improved by selecting another model without being worse for one or more other objectives. The TPOT AutoML approach uses a type of Pareto optimization called non-dominated sorting (Deb et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2000" title="Deb K, Agrawal S, Pratap A, Meyarivan T (2000) A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: NSGA-II. In: Schoenauer M, Deb K, Rudolph G et al (eds) Parallel problem solving from nature PPSN VI. Springer, Berlin, pp 849–858" href="/article/10.1007/s00439-021-02393-x#ref-CR15" id="ref-link-section-d56742713e1845">2000</a>) and by default selects Pareto optimal models using accuracy and pipeline complexity. These objectives could be modified to include biological criteria such as druggability. It will be important to evaluate whether biology-based objectives improve AutoML for genetic and genomics problems beyond standard metrics such as accuracy.</p><p>Automated machine learning shows tremendous promise for the genetic analysis of complex traits. Particularly important is the ability of AutoML to bring machine learning to non-experts by taking much of the guesswork out of selecting algorithms and hyperparameters. High-throughput genetic data have specific characteristics that distinguish them not only from the tabular data which are more typically used in ML and AutoML, but also from other types of omics data. They are much larger scale both in terms of number of features and samples, they can be highly imbalanced (e.g., when derived from biobanks), imputation of missing values requires sophisticated and time consuming methods (Shi et al. <a data-track="click" data-track-action="reference anchor" data-track-label="link" data-test="citation-ref" aria-label="Reference 2018" title="Shi S, Yuan N, Yang M et al (2018) Comprehensive assessment of genotype imputation performance. HHE 83:107–116. 
 https://doi.org/10.1159/000489758
 
 " href="/article/10.1007/s00439-021-02393-x#ref-CR69" id="ref-link-section-d56742713e1851">2018</a>), it is typically important to identify relevant mechanisms (e.g., genes) hence interpretability is often a requirement. We note that the latter is further complicated by the fact that, even when relevant SNPs are identified, subsequent identification of the genes affected by these SNPs is non-trivial, since the affected genes are not necessarily those closest to the SNP in the linear genome and typically integration of additional data sources (other types of functional genomics data) is necessary to further elucidate mechanisms. Thus, several areas need to be developed further before the full potential of AutoML methods can be realized in the genetics domain. First, many of these approaches are computationally intensive, because they are iterating over many different algorithms. Additional work is needed to scale AutoML to genome-wide data, for example. Second, genetic analysis has the potential to be informed by functional genomics data. It will be important to develop powerful methods which allow AutoML to take advantage of these data for model building and search. Third, interpretation is always an important challenge for any ML result. AutoML has the potential to develop complex ML pipelines. The combination of multiple different ML algorithms in a pipeline can make interpretation more challenging. Finally, enabling geneticists with no ML experience to use AutoML will be key to maximizing the value of omics data we have collected for the study of complex traits. An emphasis on accessible and user-friendly software will be essential. We look forward to a time when anyone who wants to use ML for genetic analysis can do so with ease of running a <i>t</i> test.</p></div></div></section> </div> <section data-title="Availability of data and materials"><div class="c-article-section" id="availability-of-data-and-materials-section"><h2 class="c-article-section__title js-section-title js-c-reading-companion-sections-item" id="availability-of-data-and-materials">Availability of data and materials</h2><div class="c-article-section__content" id="availability-of-data-and-materials-content"> <p>Not applicable.</p> </div></div></section><section data-title="Code availability"><div class="c-article-section" id="code-availability-section"><h2 class="c-article-section__title js-section-title js-c-reading-companion-sections-item" id="code-availability">Code availability</h2><div class="c-article-section__content" id="code-availability-content"> <p>Not applicable.</p> </div></div></section><div id="MagazineFulltextArticleBodySuffix"><section aria-labelledby="Bib1" data-title="References"><div class="c-article-section" id="Bib1-section"><h2 class="c-article-section__title js-section-title js-c-reading-companion-sections-item" id="Bib1">References</h2><div class="c-article-section__content" id="Bib1-content"><div data-container-section="references"><ul class="c-article-references" data-track-component="outbound reference" data-track-context="references section"><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR1">Adams SM, Feroze H, Nguyen T et al (2020) Genome wide epistasis study of on-statin cardiovascular events with iterative feature reduction and selection. J Pers Med. <a href="https://doi.org/10.3390/jpm10040212" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.3390/jpm10040212">https://doi.org/10.3390/jpm10040212</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.3390/jpm10040212" data-track-item_id="10.3390/jpm10040212" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.3390%2Fjpm10040212" aria-label="Article reference 1" data-doi="10.3390/jpm10040212">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=33171725" aria-label="PubMed reference 1">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7712544" aria-label="PubMed Central reference 1">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 1" href="http://scholar.google.com/scholar_lookup?&title=Genome%20wide%20epistasis%20study%20of%20on-statin%20cardiovascular%20events%20with%20iterative%20feature%20reduction%20and%20selection&journal=J%20Pers%20Med&doi=10.3390%2Fjpm10040212&publication_year=2020&author=Adams%2CSM&author=Feroze%2CH&author=Nguyen%2CT"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR2">Alaa AM, Bolton T, Angelantonio ED et al (2019) Cardiovascular disease risk prediction using automated machine learning: a prospective study of 423,604 UK Biobank participants. PLoS One 14:e0213653. <a href="https://doi.org/10.1371/journal.pone.0213653" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1371/journal.pone.0213653">https://doi.org/10.1371/journal.pone.0213653</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1371/journal.pone.0213653" data-track-item_id="10.1371/journal.pone.0213653" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1371%2Fjournal.pone.0213653" aria-label="Article reference 2" data-doi="10.1371/journal.pone.0213653">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BC1MXht1SmtbfN" aria-label="CAS reference 2">CAS</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=31091238" aria-label="PubMed reference 2">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC6519796" aria-label="PubMed Central reference 2">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 2" href="http://scholar.google.com/scholar_lookup?&title=Cardiovascular%20disease%20risk%20prediction%20using%20automated%20machine%20learning%3A%20a%20prospective%20study%20of%20423%2C604%20UK%20Biobank%20participants&journal=PLoS%C2%A0One&doi=10.1371%2Fjournal.pone.0213653&volume=14&publication_year=2019&author=Alaa%2CAM&author=Bolton%2CT&author=Angelantonio%2CED"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR3">Alaa AM, van der Schaar M (2018a) AutoPrognosis: automated clinical prognostic modeling via Bayesian optimization with structured kernel learning. In: International conference on machine learning. PMLR, pp 139–148. <a href="http://proceedings.mlr.press/v80/alaa18b.html" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="http://proceedings.mlr.press/v80/alaa18b.html">http://proceedings.mlr.press/v80/alaa18b.html</a></p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR4">Alaa AM, van der Schaar M (2018b) Prognostication and risk factors for cystic fibrosis via automated machine learning. Sci Rep 8:11242. <a href="https://doi.org/10.1038/s41598-018-29523-2" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1038/s41598-018-29523-2">https://doi.org/10.1038/s41598-018-29523-2</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1038/s41598-018-29523-2" data-track-item_id="10.1038/s41598-018-29523-2" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1038%2Fs41598-018-29523-2" aria-label="Article reference 4" data-doi="10.1038/s41598-018-29523-2">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BC1MXltFaisQ%3D%3D" aria-label="CAS reference 4">CAS</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=30050169" aria-label="PubMed reference 4">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC6062529" aria-label="PubMed Central reference 4">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 4" href="http://scholar.google.com/scholar_lookup?&title=Prognostication%20and%20risk%20factors%20for%20cystic%20fibrosis%20via%20automated%20machine%20learning&journal=Sci%20Rep&doi=10.1038%2Fs41598-018-29523-2&volume=8&publication_year=2018&author=Alaa%2CAM&author=Schaar%2CM"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR5">Alakwaa FM, Chaudhary K, Garmire LX (2018) Deep learning accurately predicts estrogen receptor status in breast cancer metabolomics data. J Proteome Res 17:337–347. <a href="https://doi.org/10.1021/acs.jproteome.7b00595" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1021/acs.jproteome.7b00595">https://doi.org/10.1021/acs.jproteome.7b00595</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1021/acs.jproteome.7b00595" data-track-item_id="10.1021/acs.jproteome.7b00595" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1021%2Facs.jproteome.7b00595" aria-label="Article reference 5" data-doi="10.1021/acs.jproteome.7b00595">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BC2sXhslGrtrzL" aria-label="CAS reference 5">CAS</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=29110491" aria-label="PubMed reference 5">PubMed</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 5" href="http://scholar.google.com/scholar_lookup?&title=Deep%20learning%20accurately%20predicts%20estrogen%20receptor%20status%20in%20breast%20cancer%20metabolomics%20data&journal=J%20Proteome%20Res&doi=10.1021%2Facs.jproteome.7b00595&volume=17&pages=337-347&publication_year=2018&author=Alakwaa%2CFM&author=Chaudhary%2CK&author=Garmire%2CLX"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR6">Bazaga A, Leggate D, Weisser H (2020) Genome-wide investigation of gene-cancer associations for the prediction of novel therapeutic targets in oncology. Sci Rep 10:10787. <a href="https://doi.org/10.1038/s41598-020-67846-1" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1038/s41598-020-67846-1">https://doi.org/10.1038/s41598-020-67846-1</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1038/s41598-020-67846-1" data-track-item_id="10.1038/s41598-020-67846-1" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1038%2Fs41598-020-67846-1" aria-label="Article reference 6" data-doi="10.1038/s41598-020-67846-1">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BB3cXhtlajsr7F" aria-label="CAS reference 6">CAS</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=32612205" aria-label="PubMed reference 6">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7330039" aria-label="PubMed Central reference 6">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 6" href="http://scholar.google.com/scholar_lookup?&title=Genome-wide%20investigation%20of%20gene-cancer%20associations%20for%20the%20prediction%20of%20novel%20therapeutic%20targets%20in%20oncology&journal=Sci%20Rep&doi=10.1038%2Fs41598-020-67846-1&volume=10&publication_year=2020&author=Bazaga%2CA&author=Leggate%2CD&author=Weisser%2CH"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR7">Bergstra J, Komer B, Eliasmith C et al (2015) Hyperopt: a Python library for model selection and hyperparameter optimization. Comput Sci Discov 8:014008. <a href="https://doi.org/10.1088/1749-4699/8/1/014008" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1088/1749-4699/8/1/014008">https://doi.org/10.1088/1749-4699/8/1/014008</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1088/1749-4699/8/1/014008" data-track-item_id="10.1088/1749-4699/8/1/014008" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1088%2F1749-4699%2F8%2F1%2F014008" aria-label="Article reference 7" data-doi="10.1088/1749-4699/8/1/014008">Article</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 7" href="http://scholar.google.com/scholar_lookup?&title=Hyperopt%3A%20a%20Python%20library%20for%20model%20selection%20and%20hyperparameter%20optimization&journal=Comput%20Sci%20Discov&doi=10.1088%2F1749-4699%2F8%2F1%2F014008&volume=8&publication_year=2015&author=Bergstra%2CJ&author=Komer%2CB&author=Eliasmith%2CC"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR8">Brochu E, Cora VM, de Freitas N (2010) A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. <a href="http://arxiv.org/abs/1012.2599" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="http://arxiv.org/abs/1012.2599">http://arxiv.org/abs/1012.2599</a> [cs]</p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR9">Bycroft C, Freeman C, Petkova D et al (2018) The UK Biobank resource with deep phenotyping and genomic data. Nature 562:203–209. <a href="https://doi.org/10.1038/s41586-018-0579-z" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1038/s41586-018-0579-z">https://doi.org/10.1038/s41586-018-0579-z</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1038/s41586-018-0579-z" data-track-item_id="10.1038/s41586-018-0579-z" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1038%2Fs41586-018-0579-z" aria-label="Article reference 9" data-doi="10.1038/s41586-018-0579-z">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BC1cXhvV2qsbvI" aria-label="CAS reference 9">CAS</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=30305743" aria-label="PubMed reference 9">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC6786975" aria-label="PubMed Central reference 9">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 9" href="http://scholar.google.com/scholar_lookup?&title=The%20UK%20Biobank%20resource%20with%20deep%20phenotyping%20and%20genomic%20data&journal=Nature&doi=10.1038%2Fs41586-018-0579-z&volume=562&pages=203-209&publication_year=2018&author=Bycroft%2CC&author=Freeman%2CC&author=Petkova%2CD"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR10">Candel A, LeDell E (2021) Deep learning with H<sub>2</sub>O, 6th edn. H2O.ai, Inc., Mountain View</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 10" href="http://scholar.google.com/scholar_lookup?&title=Deep%20learning%20with%20H2O&publication_year=2021&author=Candel%2CA&author=LeDell%2CE"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR11">Caruana R, Niculescu-Mizil A, Crew G, Ksikes A (2004) Ensemble selection from libraries of models. In: Proceedings of the twenty-first international conference on Machine learning. Association for Computing Machinery, New York, NY, USA, p 18</p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR12">Casalicchio G, Molnar C, Bischl B (2019) Visualizing the feature importance for black box models. In: Berlingerio M, Bonchi F, Gärtner T et al (eds) Machine learning and knowledge discovery in databases. Springer International Publishing, Cham, pp 655–670</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="noopener" data-track-label="10.1007/978-3-030-10925-7_40" data-track-item_id="10.1007/978-3-030-10925-7_40" data-track-value="chapter reference" data-track-action="chapter reference" href="https://link.springer.com/doi/10.1007/978-3-030-10925-7_40" aria-label="Chapter reference 12" data-doi="10.1007/978-3-030-10925-7_40">Chapter</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 12" href="http://scholar.google.com/scholar_lookup?&title=Visualizing%20the%20feature%20importance%20for%20black%20box%20models&doi=10.1007%2F978-3-030-10925-7_40&pages=655-670&publication_year=2019&author=Casalicchio%2CG&author=Molnar%2CC&author=Bischl%2CB"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR13">Chen T, Guestrin C (2016) XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, New York, NY, USA, pp 785–794</p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR14">Choi JW, Hu R, Zhao Y et al (2021) Preoperative prediction of the stage, size, grade, and necrosis score in clear cell renal cell carcinoma using MRI-based radiomics. Abdom Radiol (NY). <a href="https://doi.org/10.1007/s00261-020-02876-x" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1007/s00261-020-02876-x">https://doi.org/10.1007/s00261-020-02876-x</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="noopener" data-track-label="10.1007/s00261-020-02876-x" data-track-item_id="10.1007/s00261-020-02876-x" data-track-value="article reference" data-track-action="article reference" href="https://link.springer.com/doi/10.1007/s00261-020-02876-x" aria-label="Article reference 14" data-doi="10.1007/s00261-020-02876-x">Article</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 14" href="http://scholar.google.com/scholar_lookup?&title=Preoperative%20prediction%20of%20the%20stage%2C%20size%2C%20grade%2C%20and%20necrosis%20score%20in%20clear%20cell%20renal%20cell%20carcinoma%20using%20MRI-based%20radiomics&journal=Abdom%20Radiol%20%28NY%29&doi=10.1007%2Fs00261-020-02876-x&publication_year=2021&author=Choi%2CJW&author=Hu%2CR&author=Zhao%2CY"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR15">Deb K, Agrawal S, Pratap A, Meyarivan T (2000) A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: NSGA-II. In: Schoenauer M, Deb K, Rudolph G et al (eds) Parallel problem solving from nature PPSN VI. Springer, Berlin, pp 849–858</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="noopener" data-track-label="10.1007/3-540-45356-3_83" data-track-item_id="10.1007/3-540-45356-3_83" data-track-value="chapter reference" data-track-action="chapter reference" href="https://link.springer.com/doi/10.1007/3-540-45356-3_83" aria-label="Chapter reference 15" data-doi="10.1007/3-540-45356-3_83">Chapter</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 15" href="http://scholar.google.com/scholar_lookup?&title=A%20fast%20elitist%20non-dominated%20sorting%20genetic%20algorithm%20for%20multi-objective%20optimization%3A%20NSGA-II&doi=10.1007%2F3-540-45356-3_83&pages=849-858&publication_year=2000&author=Deb%2CK&author=Agrawal%2CS&author=Pratap%2CA&author=Meyarivan%2CT"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR16">Deb K, Pratap A, Agarwal S, Meyarivan T (2002) A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans Evol Comput 6:182–197. <a href="https://doi.org/10.1109/4235.996017" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1109/4235.996017">https://doi.org/10.1109/4235.996017</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1109/4235.996017" data-track-item_id="10.1109/4235.996017" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1109%2F4235.996017" aria-label="Article reference 16" data-doi="10.1109/4235.996017">Article</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 16" href="http://scholar.google.com/scholar_lookup?&title=A%20fast%20and%20elitist%20multiobjective%20genetic%20algorithm%3A%20NSGA-II&journal=IEEE%20Trans%20Evol%20Comput&doi=10.1109%2F4235.996017&volume=6&pages=182-197&publication_year=2002&author=Deb%2CK&author=Pratap%2CA&author=Agarwal%2CS&author=Meyarivan%2CT"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR17">Elsken T, Metzen JH, Hutter F (2019) Neural architecture search. In: Hutter F, Kotthoff L, Vanschoren J (eds) Automated machine learning. Springer International Publishing, Cham, pp 63–77</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="noopener" data-track-label="10.1007/978-3-030-05318-5_3" data-track-item_id="10.1007/978-3-030-05318-5_3" data-track-value="chapter reference" data-track-action="chapter reference" href="https://link.springer.com/doi/10.1007/978-3-030-05318-5_3" aria-label="Chapter reference 17" data-doi="10.1007/978-3-030-05318-5_3">Chapter</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 17" href="http://scholar.google.com/scholar_lookup?&title=Neural%20architecture%20search&doi=10.1007%2F978-3-030-05318-5_3&pages=63-77&publication_year=2019&author=Elsken%2CT&author=Metzen%2CJH&author=Hutter%2CF"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR18">Erickson N, Mueller J, Shirkov A, et al (2020) AutoGluon-tabular: robust and accurate AutoML for structured data. <a href="http://arxiv.org/abs/20030.6505" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="http://arxiv.org/abs/20030.6505">http://arxiv.org/abs/20030.6505</a> [cs, stat]</p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR19">Feurer M, Klein A, Eggensperger K et al (2015) Efficient and robust automated machine learning. In: Cortes C, Lawrence N, Lee D et al (eds) Advances in neural information processing systems. Curran Associates, Inc., pp 2962–2970</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 19" href="http://scholar.google.com/scholar_lookup?&title=Efficient%20and%20robust%20automated%20machine%20learning&pages=2962-2970&publication_year=2015&author=Feurer%2CM&author=Klein%2CA&author=Eggensperger%2CK"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR20">Feurer M, Eggensperger K, Falkner S et al (2018) Practical automated machine learning for the AutoML challenge. ICML 2018 AutoML Workshop. <a href="https://ml.informatik.uni-freiburg.de/wp-content/uploads/papers/18-AUTOML-AutoChallenge.pdf" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="https://ml.informatik.uni-freiburg.de/wp-content/uploads/papers/18-AUTOML-AutoChallenge.pdf">https://ml.informatik.uni-freiburg.de/wp-content/uploads/papers/18-AUTOML-AutoChallenge.pdf</a></p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR21">Frank E, Hall MA, Witten IH (2016) The Weka Workbench. Online Appendix for “Data mining: practical machine learning tools and techniques”, Morgan Kaufmann, Fourth Edition. <a href="https://www.cs.waikato.ac.nz/ml/weka/Witten_et_al_2016_appendix.pdf" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="https://www.cs.waikato.ac.nz/ml/weka/Witten_et_al_2016_appendix.pdf">https://www.cs.waikato.ac.nz/ml/weka/Witten_et_al_2016_appendix.pdf</a>. Accessed 19 Apr 2021</p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR22">Gelfman S, Wang Q, McSweeney KM et al (2017) Annotating pathogenic non-coding variants in genic regions. Nat Commun 8:236. <a href="https://doi.org/10.1038/s41467-017-00141-2" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1038/s41467-017-00141-2">https://doi.org/10.1038/s41467-017-00141-2</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1038/s41467-017-00141-2" data-track-item_id="10.1038/s41467-017-00141-2" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1038%2Fs41467-017-00141-2" aria-label="Article reference 22" data-doi="10.1038/s41467-017-00141-2">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BC1cXosVWksbY%3D" aria-label="CAS reference 22">CAS</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=28794409" aria-label="PubMed reference 22">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC5550444" aria-label="PubMed Central reference 22">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 22" href="http://scholar.google.com/scholar_lookup?&title=Annotating%20pathogenic%20non-coding%20variants%20in%20genic%20regions&journal=Nat%20Commun&doi=10.1038%2Fs41467-017-00141-2&volume=8&publication_year=2017&author=Gelfman%2CS&author=Wang%2CQ&author=McSweeney%2CKM"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR23">Gijsbers P, LeDell E, Thomas J, et al (2019) An open source AutoML benchmark. <a href="http://arxiv.org/abs/1907.00909" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="http://arxiv.org/abs/1907.00909">http://arxiv.org/abs/1907.00909</a> [cs, stat]</p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR24">Golmakani MK, Polley EC (2020) Super learner for survival data prediction. Int J Biostat. <a href="https://doi.org/10.1515/ijb-2019-0065" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1515/ijb-2019-0065">https://doi.org/10.1515/ijb-2019-0065</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1515/ijb-2019-0065" data-track-item_id="10.1515/ijb-2019-0065" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1515%2Fijb-2019-0065" aria-label="Article reference 24" data-doi="10.1515/ijb-2019-0065">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=32097120" aria-label="PubMed reference 24">PubMed</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 24" href="http://scholar.google.com/scholar_lookup?&title=Super%20learner%20for%20survival%20data%20prediction&journal=Int%20J%20Biostat&doi=10.1515%2Fijb-2019-0065&publication_year=2020&author=Golmakani%2CMK&author=Polley%2CEC"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR25">Guyon I, Sun-Hosoya L, Boullé M et al (2019) Analysis of the AutoML challenge series 2015–2018. In: Hutter F, Kotthoff L, Vanschoren J (eds) Automated machine learning: methods, systems, challenges. Springer International Publishing, Cham, pp 177–219</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="noopener" data-track-label="10.1007/978-3-030-05318-5_10" data-track-item_id="10.1007/978-3-030-05318-5_10" data-track-value="chapter reference" data-track-action="chapter reference" href="https://link.springer.com/doi/10.1007/978-3-030-05318-5_10" aria-label="Chapter reference 25" data-doi="10.1007/978-3-030-05318-5_10">Chapter</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 25" href="http://scholar.google.com/scholar_lookup?&title=Analysis%20of%20the%20AutoML%20challenge%20series%202015%E2%80%932018&doi=10.1007%2F978-3-030-05318-5_10&pages=177-219&publication_year=2019&author=Guyon%2CI&author=Sun-Hosoya%2CL&author=Boull%C3%A9%2CM"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR26">Heimisdottir LH, Lin BM, Cho H et al (2021) Metabolomics insights in early childhood caries. J Dent Res. <a href="https://doi.org/10.1177/0022034520982963" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1177/0022034520982963">https://doi.org/10.1177/0022034520982963</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1177/0022034520982963" data-track-item_id="10.1177/0022034520982963" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1177%2F0022034520982963" aria-label="Article reference 26" data-doi="10.1177/0022034520982963">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=33423574" aria-label="PubMed reference 26">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC8142089" aria-label="PubMed Central reference 26">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 26" href="http://scholar.google.com/scholar_lookup?&title=Metabolomics%20insights%20in%20early%20childhood%20caries&journal=J%20Dent%20Res&doi=10.1177%2F0022034520982963&publication_year=2021&author=Heimisdottir%2CLH&author=Lin%2CBM&author=Cho%2CH"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR27">Hooker G, Mentch L, Zhou S (2021) Unrestricted permutation forces extrapolation: variable importance requires at least one more model, or there is no free variable importance. <a href="http://arxiv.org/abs/1905.03151" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="http://arxiv.org/abs/1905.03151">http://arxiv.org/abs/1905.03151</a> [cs, stat]</p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR28">Howard D, Maslej MM, Lee J et al (2020) Transfer learning for risk classification of social media posts: model evaluation study. J Med Internet Res 22:e15371. <a href="https://doi.org/10.2196/15371" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.2196/15371">https://doi.org/10.2196/15371</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.2196/15371" data-track-item_id="10.2196/15371" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.2196%2F15371" aria-label="Article reference 28" data-doi="10.2196/15371">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=32401222" aria-label="PubMed reference 28">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7254287" aria-label="PubMed Central reference 28">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 28" href="http://scholar.google.com/scholar_lookup?&title=Transfer%20learning%20for%20risk%20classification%20of%20social%20media%20posts%3A%20model%20evaluation%20study&journal=J%20Med%20Internet%20Res&doi=10.2196%2F15371&volume=22&publication_year=2020&author=Howard%2CD&author=Maslej%2CMM&author=Lee%2CJ"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR29">Huang S, Ji X, Cho M, et al (2021) DL-PRS: a novel deep learning approach to polygenic risk scores. BMC Bioinformatics. <a href="https://doi.org/10.21203/rs.3.rs-423764/v1" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.21203/rs.3.rs-423764/v1">https://doi.org/10.21203/rs.3.rs-423764/v1</a></p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR30">Hutter F, Hoos HH, Leyton-Brown K (2011) Sequential model-based optimization for general algorithm configuration. In: Coello CAC (ed) Learning and intelligent optimization. Springer, Berlin, pp 507–523</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="noopener" data-track-label="10.1007/978-3-642-25566-3_40" data-track-item_id="10.1007/978-3-642-25566-3_40" data-track-value="chapter reference" data-track-action="chapter reference" href="https://link.springer.com/doi/10.1007/978-3-642-25566-3_40" aria-label="Chapter reference 30" data-doi="10.1007/978-3-642-25566-3_40">Chapter</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 30" href="http://scholar.google.com/scholar_lookup?&title=Sequential%20model-based%20optimization%20for%20general%20algorithm%20configuration&doi=10.1007%2F978-3-642-25566-3_40&pages=507-523&publication_year=2011&author=Hutter%2CF&author=Hoos%2CHH&author=Leyton-Brown%2CK"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR31">Hutter F, Kotthoff L, Vanschoren J (eds) (2019) Automated machine learning: methods, systems. Springer International Publishing, Challenges</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 31" href="http://scholar.google.com/scholar_lookup?&title=Automated%20machine%20learning%3A%20methods%2C%20systems&publication_year=2019"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR32">Ioannidis NM, Rothstein JH, Pejaver V et al (2016) REVEL: an ensemble method for predicting the pathogenicity of rare missense variants. Am J Hum Genet 99:877–885. <a href="https://doi.org/10.1016/j.ajhg.2016.08.016" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1016/j.ajhg.2016.08.016">https://doi.org/10.1016/j.ajhg.2016.08.016</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1016/j.ajhg.2016.08.016" data-track-item_id="10.1016/j.ajhg.2016.08.016" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1016%2Fj.ajhg.2016.08.016" aria-label="Article reference 32" data-doi="10.1016/j.ajhg.2016.08.016">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BC28XhsFGrurnM" aria-label="CAS reference 32">CAS</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=27666373" aria-label="PubMed reference 32">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC5065685" aria-label="PubMed Central reference 32">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 32" href="http://scholar.google.com/scholar_lookup?&title=REVEL%3A%20an%20ensemble%20method%20for%20predicting%20the%20pathogenicity%20of%20rare%20missense%20variants&journal=Am%20J%20Hum%20Genet&doi=10.1016%2Fj.ajhg.2016.08.016&volume=99&pages=877-885&publication_year=2016&author=Ioannidis%2CNM&author=Rothstein%2CJH&author=Pejaver%2CV"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR33">Klein A, Falkner S, Bartels S et al (2017) Fast Bayesian hyperparameter optimization on large datasets. Electron J Statist. <a href="https://doi.org/10.1214/17-EJS1335SI" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1214/17-EJS1335SI">https://doi.org/10.1214/17-EJS1335SI</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1214/17-EJS1335SI" data-track-item_id="10.1214/17-EJS1335SI" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1214%2F17-EJS1335SI" aria-label="Article reference 33" data-doi="10.1214/17-EJS1335SI">Article</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 33" href="http://scholar.google.com/scholar_lookup?&title=Fast%20Bayesian%20hyperparameter%20optimization%20on%20large%20datasets&journal=Electron%20J%20Statist&doi=10.1214%2F17-EJS1335SI&publication_year=2017&author=Klein%2CA&author=Falkner%2CS&author=Bartels%2CS"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR34">Komer B, Bergstra J, Eliasmith C (2019) Hyperopt-sklearn. In: Hutter F, Kotthoff L, Vanschoren J (eds) Automated machine learning: methods, systems, challenges. Springer International Publishing, Cham, pp 97–111</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="noopener" data-track-label="10.1007/978-3-030-05318-5_5" data-track-item_id="10.1007/978-3-030-05318-5_5" data-track-value="chapter reference" data-track-action="chapter reference" href="https://link.springer.com/doi/10.1007/978-3-030-05318-5_5" aria-label="Chapter reference 34" data-doi="10.1007/978-3-030-05318-5_5">Chapter</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 34" href="http://scholar.google.com/scholar_lookup?&title=Hyperopt-sklearn&doi=10.1007%2F978-3-030-05318-5_5&pages=97-111&publication_year=2019&author=Komer%2CB&author=Bergstra%2CJ&author=Eliasmith%2CC"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR35">Kononenko I, Šimec E, Robnik-Šikonja M (1997) Overcoming the myopia of inductive learning algorithms with RELIEFF. Appl Intell 7:39–55. <a href="https://doi.org/10.1023/A:1008280620621" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1023/A:1008280620621">https://doi.org/10.1023/A:1008280620621</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1023/A:1008280620621" data-track-item_id="10.1023/A:1008280620621" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1023%2FA%3A1008280620621" aria-label="Article reference 35" data-doi="10.1023/A:1008280620621">Article</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 35" href="http://scholar.google.com/scholar_lookup?&title=Overcoming%20the%20myopia%20of%20inductive%20learning%20algorithms%20with%20RELIEFF&journal=Appl%20Intell&doi=10.1023%2FA%3A1008280620621&volume=7&pages=39-55&publication_year=1997&author=Kononenko%2CI&author=%C5%A0imec%2CE&author=Robnik-%C5%A0ikonja%2CM"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR36">Kotthoff L, Thornton C, Hoos HH et al (2017) Auto-WEKA 2.0: automatic model selection and hyperparameter optimization in WEKA. J Mach Learn Res 18:1–5</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 36" href="http://scholar.google.com/scholar_lookup?&title=Auto-WEKA%202.0%3A%20automatic%20model%20selection%20and%20hyperparameter%20optimization%20in%20WEKA&journal=J%20Mach%20Learn%20Res&volume=18&pages=1-5&publication_year=2017&author=Kotthoff%2CL&author=Thornton%2CC&author=Hoos%2CHH"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR37">Kotthoff L, Thornton C, Hoos HH et al (2019) Auto-WEKA: automatic model selection and hyperparameter optimization in WEKA. In: Hutter F, Kotthoff L, Vanschoren J (eds) Automated machine learning. Springer International Publishing, Cham, pp 81–95</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="noopener" data-track-label="10.1007/978-3-030-05318-5_4" data-track-item_id="10.1007/978-3-030-05318-5_4" data-track-value="chapter reference" data-track-action="chapter reference" href="https://link.springer.com/doi/10.1007/978-3-030-05318-5_4" aria-label="Chapter reference 37" data-doi="10.1007/978-3-030-05318-5_4">Chapter</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 37" href="http://scholar.google.com/scholar_lookup?&title=Auto-WEKA%3A%20automatic%20model%20selection%20and%20hyperparameter%20optimization%20in%20WEKA&doi=10.1007%2F978-3-030-05318-5_4&pages=81-95&publication_year=2019&author=Kotthoff%2CL&author=Thornton%2CC&author=Hoos%2CHH"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR38">Kuhn M, Johnson K (2013) Applied predictive modeling. Springer-Verlag, New York</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="noopener" data-track-label="10.1007/978-1-4614-6849-3" data-track-item_id="10.1007/978-1-4614-6849-3" data-track-value="book reference" data-track-action="book reference" href="https://link.springer.com/doi/10.1007/978-1-4614-6849-3" aria-label="Book reference 38" data-doi="10.1007/978-1-4614-6849-3">Book</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 38" href="http://scholar.google.com/scholar_lookup?&title=Applied%20predictive%20modeling&doi=10.1007%2F978-1-4614-6849-3&publication_year=2013&author=Kuhn%2CM&author=Johnson%2CK"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR39">Kundaje A, Meuleman W, Ernst J et al (2015) Integrative analysis of 111 reference human epigenomes. Nature 518:317–330. <a href="https://doi.org/10.1038/nature14248" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1038/nature14248">https://doi.org/10.1038/nature14248</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1038/nature14248" data-track-item_id="10.1038/nature14248" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1038%2Fnature14248" aria-label="Article reference 39" data-doi="10.1038/nature14248">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BC2MXjtVSktbc%3D" aria-label="CAS reference 39">CAS</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=25693563" aria-label="PubMed reference 39">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4530010" aria-label="PubMed Central reference 39">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 39" href="http://scholar.google.com/scholar_lookup?&title=Integrative%20analysis%20of%20111%20reference%20human%20epigenomes&journal=Nature&doi=10.1038%2Fnature14248&volume=518&pages=317-330&publication_year=2015&author=Kundaje%2CA&author=Meuleman%2CW&author=Ernst%2CJ"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR40">La Cava W, Williams H, Fu W et al (2020) Evaluating recommender systems for AI-driven biomedical informatics. Bioinformatics. <a href="https://doi.org/10.1093/bioinformatics/btaa698" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1093/bioinformatics/btaa698">https://doi.org/10.1093/bioinformatics/btaa698</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1093/bioinformatics/btaa698" data-track-item_id="10.1093/bioinformatics/btaa698" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1093%2Fbioinformatics%2Fbtaa698" aria-label="Article reference 40" data-doi="10.1093/bioinformatics/btaa698">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC8055228" aria-label="PubMed Central reference 40">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 40" href="http://scholar.google.com/scholar_lookup?&title=Evaluating%20recommender%20systems%20for%20AI-driven%20biomedical%20informatics&journal=Bioinformatics&doi=10.1093%2Fbioinformatics%2Fbtaa698&publication_year=2020&author=Cava%2CW&author=Williams%2CH&author=Fu%2CW"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR41">Le TT, Fu W, Moore JH (2020) Scaling tree-based automated machine learning to biomedical big data with a feature set selector. Bioinformatics 36:250–256. <a href="https://doi.org/10.1093/bioinformatics/btz470" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1093/bioinformatics/btz470">https://doi.org/10.1093/bioinformatics/btz470</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1093/bioinformatics/btz470" data-track-item_id="10.1093/bioinformatics/btz470" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1093%2Fbioinformatics%2Fbtz470" aria-label="Article reference 41" data-doi="10.1093/bioinformatics/btz470">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BB3cXisVOrtLnN" aria-label="CAS reference 41">CAS</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=31165141" aria-label="PubMed reference 41">PubMed</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 41" href="http://scholar.google.com/scholar_lookup?&title=Scaling%20tree-based%20automated%20machine%20learning%20to%20biomedical%20big%20data%20with%20a%20feature%20set%20selector&journal=Bioinformatics&doi=10.1093%2Fbioinformatics%2Fbtz470&volume=36&pages=250-256&publication_year=2020&author=Le%2CTT&author=Fu%2CW&author=Moore%2CJH"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR42">LeDell E, Poirier S (2020) H2O AutoML: scalable automatic machine learning. In: 7th ICML workshop on automated machine learning. <a href="https://www.automl.org/wp-content/uploads/2020/07/AutoML_2020_paper_61.pdf" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="https://www.automl.org/wp-content/uploads/2020/07/AutoML_2020_paper_61.pdf">https://www.automl.org/wp-content/uploads/2020/07/AutoML_2020_paper_61.pdf</a></p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR43">Li L, Yang Y, Zhang Q et al (2021) Use of deep-learning genomics to discriminate healthy individuals from those with Alzheimer’s disease or mild cognitive impairment. Behav Neurol 2021:3359103. <a href="https://doi.org/10.1155/2021/3359103" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1155/2021/3359103">https://doi.org/10.1155/2021/3359103</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1155/2021/3359103" data-track-item_id="10.1155/2021/3359103" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1155%2F2021%2F3359103" aria-label="Article reference 43" data-doi="10.1155/2021/3359103">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=34336000" aria-label="PubMed reference 43">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC8298161" aria-label="PubMed Central reference 43">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 43" href="http://scholar.google.com/scholar_lookup?&title=Use%20of%20deep-learning%20genomics%20to%20discriminate%20healthy%20individuals%20from%20those%20with%20Alzheimer%E2%80%99s%20disease%20or%20mild%20cognitive%20impairment&journal=Behav%20Neurol&doi=10.1155%2F2021%2F3359103&volume=2021&publication_year=2021&author=Li%2CL&author=Yang%2CY&author=Zhang%2CQ"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR44">Lundberg SM, Lee S-I (2017) A unified approach to interpreting model predictions. Adv Neural Inf Process Syst 30:4765–4774</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 44" href="http://scholar.google.com/scholar_lookup?&title=A%20unified%20approach%20to%20interpreting%20model%20predictions&journal=Adv%20Neural%20Inf%20Process%20Syst&volume=30&pages=4765-4774&publication_year=2017&author=Lundberg%2CSM&author=Lee%2CS-I"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR45">Manduchi E, Moore JH (2021) Leveraging automated machine learning for the analysis of global public health data: a case study in malaria. Int J Public Health. <a href="https://doi.org/10.3389/ijph.2021.614296" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.3389/ijph.2021.614296">https://doi.org/10.3389/ijph.2021.614296</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.3389/ijph.2021.614296" data-track-item_id="10.3389/ijph.2021.614296" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.3389%2Fijph.2021.614296" aria-label="Article reference 45" data-doi="10.3389/ijph.2021.614296">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=34744577" aria-label="PubMed reference 45">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC8565284" aria-label="PubMed Central reference 45">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 45" href="http://scholar.google.com/scholar_lookup?&title=Leveraging%20automated%20machine%20learning%20for%20the%20analysis%20of%20global%20public%20health%20data%3A%20a%20case%20study%20in%20malaria&journal=Int%20J%20Public%20Health&doi=10.3389%2Fijph.2021.614296&publication_year=2021&author=Manduchi%2CE&author=Moore%2CJH"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR46">Manduchi E, Fu W, Romano JD et al (2020) Embedding covariate adjustments in tree-based automated machine learning for biomedical big data analyses. BMC Bioinform 36:1772</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1093/bioinformatics/btz796" data-track-item_id="10.1093/bioinformatics/btz796" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1093%2Fbioinformatics%2Fbtz796" aria-label="Article reference 46" data-doi="10.1093/bioinformatics/btz796">Article</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 46" href="http://scholar.google.com/scholar_lookup?&title=Embedding%20covariate%20adjustments%20in%20tree-based%20automated%20machine%20learning%20for%20biomedical%20big%20data%20analyses&journal=BMC%20Bioinform&doi=10.1093%2Fbioinformatics%2Fbtz796&volume=36&publication_year=2020&author=Manduchi%2CE&author=Fu%2CW&author=Romano%2CJD"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR47">Manduchi E, Le T, Fu W, Moore JH (2021) Genetic analysis of coronary artery disease using tree-based automated machine learning informed by biology-based feature selection. IEEE/ACM Trans Comput Biol Bioinform. <a href="https://doi.org/10.1109/TCBB.2021.3099068" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1109/TCBB.2021.3099068">https://doi.org/10.1109/TCBB.2021.3099068</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1109/TCBB.2021.3099068" data-track-item_id="10.1109/TCBB.2021.3099068" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1109%2FTCBB.2021.3099068" aria-label="Article reference 47" data-doi="10.1109/TCBB.2021.3099068">Article</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 47" href="http://scholar.google.com/scholar_lookup?&title=Genetic%20analysis%20of%20coronary%20artery%20disease%20using%20tree-based%20automated%20machine%20learning%20informed%20by%20biology-based%20feature%20selection&journal=IEEE%2FACM%20Trans%20Comput%20Biol%20Bioinform&doi=10.1109%2FTCBB.2021.3099068&publication_year=2021&author=Manduchi%2CE&author=Le%2CT&author=Fu%2CW&author=Moore%2CJH"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR48">Mendoza H, Klein A, Feurer M et al (2019) Towards automatically-tuned deep neural networks. In: Hutter F, Kotthoff L, Vanschoren J (eds) Automated machine learning. Springer International Publishing, Cham, pp 135–149</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="noopener" data-track-label="10.1007/978-3-030-05318-5_7" data-track-item_id="10.1007/978-3-030-05318-5_7" data-track-value="chapter reference" data-track-action="chapter reference" href="https://link.springer.com/doi/10.1007/978-3-030-05318-5_7" aria-label="Chapter reference 48" data-doi="10.1007/978-3-030-05318-5_7">Chapter</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 48" href="http://scholar.google.com/scholar_lookup?&title=Towards%20automatically-tuned%20deep%20neural%20networks&doi=10.1007%2F978-3-030-05318-5_7&pages=135-149&publication_year=2019&author=Mendoza%2CH&author=Klein%2CA&author=Feurer%2CM"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR49">Mohr F, Wever M, Hüllermeier E (2018) ML-Plan: automated machine learning via hierarchical planning. Mach Learn 107:1495–1515. <a href="https://doi.org/10.1007/s10994-018-5735-z" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1007/s10994-018-5735-z">https://doi.org/10.1007/s10994-018-5735-z</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="noopener" data-track-label="10.1007/s10994-018-5735-z" data-track-item_id="10.1007/s10994-018-5735-z" data-track-value="article reference" data-track-action="article reference" href="https://link.springer.com/doi/10.1007/s10994-018-5735-z" aria-label="Article reference 49" data-doi="10.1007/s10994-018-5735-z">Article</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 49" href="http://scholar.google.com/scholar_lookup?&title=ML-Plan%3A%20automated%20machine%20learning%20via%20hierarchical%20planning&journal=Mach%20Learn&doi=10.1007%2Fs10994-018-5735-z&volume=107&pages=1495-1515&publication_year=2018&author=Mohr%2CF&author=Wever%2CM&author=H%C3%BCllermeier%2CE"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR50">Molnar C, König G, Herbinger J, et al (2021) General pitfalls of model-agnostic interpretation methods for machine learning models. <a href="http://arxiv.org/abs/2007.04131" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="http://arxiv.org/abs/2007.04131">http://arxiv.org/abs/2007.04131</a> [cs, stat]</p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR51">Molnar C (2021) Interpretable machine learning</p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR52">Nantasenamat C, Worachartcheewan A, Jamsak S et al (2015) AutoWeka: toward an automated data mining software for QSAR and QSPR studies. In: Cartwright H (ed) Artificial neural networks. Springer, New York, pp 119–147</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 52" href="http://scholar.google.com/scholar_lookup?&title=AutoWeka%3A%20toward%20an%20automated%20data%20mining%20software%20for%20QSAR%20and%20QSPR%20studies&pages=119-147&publication_year=2015&author=Nantasenamat%2CC&author=Worachartcheewan%2CA&author=Jamsak%2CS"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR53">Nicholls HL, John CR, Watson DS et al (2020) Reaching the end-game for GWAS: machine learning approaches for the prioritization of complex disease loci. Front Genet 11:350. <a href="https://doi.org/10.3389/fgene.2020.00350" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.3389/fgene.2020.00350">https://doi.org/10.3389/fgene.2020.00350</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.3389/fgene.2020.00350" data-track-item_id="10.3389/fgene.2020.00350" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.3389%2Ffgene.2020.00350" aria-label="Article reference 53" data-doi="10.3389/fgene.2020.00350">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=32351543" aria-label="PubMed reference 53">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7174742" aria-label="PubMed Central reference 53">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 53" href="http://scholar.google.com/scholar_lookup?&title=Reaching%20the%20end-game%20for%20GWAS%3A%20machine%20learning%20approaches%20for%20the%20prioritization%20of%20complex%20disease%20loci&journal=Front%20Genet&doi=10.3389%2Ffgene.2020.00350&volume=11&publication_year=2020&author=Nicholls%2CHL&author=John%2CCR&author=Watson%2CDS"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR54">Olson RS, Moore JH (2019) TPOT: a tree-based pipeline optimization tool for automating machine learning. In: Hutter F, Kotthoff L, Vanschoren J (eds) Automated machine learning: methods, systems, challenges. Springer International Publishing, Cham, pp 151–160</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="noopener" data-track-label="10.1007/978-3-030-05318-5_8" data-track-item_id="10.1007/978-3-030-05318-5_8" data-track-value="chapter reference" data-track-action="chapter reference" href="https://link.springer.com/doi/10.1007/978-3-030-05318-5_8" aria-label="Chapter reference 54" data-doi="10.1007/978-3-030-05318-5_8">Chapter</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 54" href="http://scholar.google.com/scholar_lookup?&title=TPOT%3A%20a%20tree-based%20pipeline%20optimization%20tool%20for%20automating%20machine%20learning&doi=10.1007%2F978-3-030-05318-5_8&pages=151-160&publication_year=2019&author=Olson%2CRS&author=Moore%2CJH"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR55">Olson RS, Urbanowicz RJ, Andrews PC et al (2016) Automating biomedical data science through tree-based pipeline optimization. In: Squillero G, Burelli P (eds) Applications of evolutionary computation. Springer International Publishing, Cham, pp 123–137</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="noopener" data-track-label="10.1007/978-3-319-31204-0_9" data-track-item_id="10.1007/978-3-319-31204-0_9" data-track-value="chapter reference" data-track-action="chapter reference" href="https://link.springer.com/doi/10.1007/978-3-319-31204-0_9" aria-label="Chapter reference 55" data-doi="10.1007/978-3-319-31204-0_9">Chapter</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 55" href="http://scholar.google.com/scholar_lookup?&title=Automating%20biomedical%20data%20science%20through%20tree-based%20pipeline%20optimization&doi=10.1007%2F978-3-319-31204-0_9&pages=123-137&publication_year=2016&author=Olson%2CRS&author=Urbanowicz%2CRJ&author=Andrews%2CPC"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR56">Olson RS, Sipper M, Cava WL et al (2018) A system for accessible artificial intelligence. In: Banzhaf W, Olson RS, Tozier W, Riolo R (eds) Genetic programming theory and practice XV. Springer International Publishing, Cham, pp 121–134</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="noopener" data-track-label="10.1007/978-3-319-90512-9_8" data-track-item_id="10.1007/978-3-319-90512-9_8" data-track-value="chapter reference" data-track-action="chapter reference" href="https://link.springer.com/doi/10.1007/978-3-319-90512-9_8" aria-label="Chapter reference 56" data-doi="10.1007/978-3-319-90512-9_8">Chapter</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 56" href="http://scholar.google.com/scholar_lookup?&title=A%20system%20for%20accessible%20artificial%20intelligence&doi=10.1007%2F978-3-319-90512-9_8&pages=121-134&publication_year=2018&author=Olson%2CRS&author=Sipper%2CM&author=Cava%2CWL"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR57">Orlenko A, Moore JH (2021) A comparison of methods for interpreting random forest models of genetic association in the presence of non-additive interactions. BioData Min 14:9. <a href="https://doi.org/10.1186/s13040-021-00243-0" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1186/s13040-021-00243-0">https://doi.org/10.1186/s13040-021-00243-0</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="noopener" data-track-label="10.1186/s13040-021-00243-0" data-track-item_id="10.1186/s13040-021-00243-0" data-track-value="article reference" data-track-action="article reference" href="https://link.springer.com/doi/10.1186/s13040-021-00243-0" aria-label="Article reference 57" data-doi="10.1186/s13040-021-00243-0">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=33514397" aria-label="PubMed reference 57">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7847145" aria-label="PubMed Central reference 57">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 57" href="http://scholar.google.com/scholar_lookup?&title=A%20comparison%20of%20methods%20for%20interpreting%20random%20forest%20models%20of%20genetic%20association%20in%20the%20presence%20of%20non-additive%20interactions&journal=BioData%20Min&doi=10.1186%2Fs13040-021-00243-0&volume=14&publication_year=2021&author=Orlenko%2CA&author=Moore%2CJH"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR59">Orlenko A, Moore JH, Orzechowski P et al (2018) Considerations for automated machine learning in clinical metabolic profiling: altered homocysteine plasma concentration associated with metformin exposure. Pac Symp Biocomput 23:460–471</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=29218905" aria-label="PubMed reference 58">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC5882490" aria-label="PubMed Central reference 58">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 58" href="http://scholar.google.com/scholar_lookup?&title=onsiderations%20for%20automated%20machine%20learning%20in%20clinical%20metabolic%20profiling%3A%20altered%20homocysteine%20plasma%20concentration%20associated%20with%20metformin%20exposure&journal=Pac%20Symp%20Biocomput&volume=23&pages=460-471&publication_year=2018&author=Orlenko%2CA&author=Moore%2CJH&author=Orzechowski%2CP"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR58">Orlenko A, Kofink D, Lyytikäinen L-P et al (2020) Model selection for metabolomics: predicting diagnosis of coronary artery disease using automated machine learning. Bioinformatics 36:1772–1778. <a href="https://doi.org/10.1093/bioinformatics/btz796" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1093/bioinformatics/btz796">https://doi.org/10.1093/bioinformatics/btz796</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1093/bioinformatics/btz796" data-track-item_id="10.1093/bioinformatics/btz796" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1093%2Fbioinformatics%2Fbtz796" aria-label="Article reference 59" data-doi="10.1093/bioinformatics/btz796">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BB3cXhslCisbzK" aria-label="CAS reference 59">CAS</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=31702773" aria-label="PubMed reference 59">PubMed</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 59" href="http://scholar.google.com/scholar_lookup?&title=Model%20selection%20for%20metabolomics%3A%20predicting%20diagnosis%20of%20coronary%20artery%20disease%20using%20automated%20machine%20learning&journal=Bioinformatics&doi=10.1093%2Fbioinformatics%2Fbtz796&volume=36&pages=1772-1778&publication_year=2020&author=Orlenko%2CA&author=Kofink%2CD&author=Lyytik%C3%A4inen%2CL-P"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR60">Ow PS, Morton TE (1988) Filtered beam search in scheduling†. Int J Prod Res 26:35–62. <a href="https://doi.org/10.1080/00207548808947840" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1080/00207548808947840">https://doi.org/10.1080/00207548808947840</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1080/00207548808947840" data-track-item_id="10.1080/00207548808947840" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1080%2F00207548808947840" aria-label="Article reference 60" data-doi="10.1080/00207548808947840">Article</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 60" href="http://scholar.google.com/scholar_lookup?&title=Filtered%20beam%20search%20in%20scheduling%E2%80%A0&journal=Int%20J%20Prod%20Res&doi=10.1080%2F00207548808947840&volume=26&pages=35-62&publication_year=1988&author=Ow%2CPS&author=Morton%2CTE"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR61">Padmanabhan M, Yuan P, Chada G, Nguyen HV (2019) Physician-friendly machine learning: a case study with cardiovascular disease risk prediction. J Clin Med 8:1050. <a href="https://doi.org/10.3390/jcm8071050" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.3390/jcm8071050">https://doi.org/10.3390/jcm8071050</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.3390/jcm8071050" data-track-item_id="10.3390/jcm8071050" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.3390%2Fjcm8071050" aria-label="Article reference 61" data-doi="10.3390/jcm8071050">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC6678298" aria-label="PubMed Central reference 61">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 61" href="http://scholar.google.com/scholar_lookup?&title=Physician-friendly%20machine%20learning%3A%20a%20case%20study%20with%20cardiovascular%20disease%20risk%20prediction&journal=J%20Clin%20Med&doi=10.3390%2Fjcm8071050&volume=8&publication_year=2019&author=Padmanabhan%2CM&author=Yuan%2CP&author=Chada%2CG&author=Nguyen%2CHV"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR62">Pedregosa F, Varoquaux G, Gramfort A et al (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12:2825–2830</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 62" href="http://scholar.google.com/scholar_lookup?&title=Scikit-learn%3A%20machine%20learning%20in%20python&journal=J%20Mach%20Learn%20Res&volume=12&pages=2825-2830&publication_year=2011&author=Pedregosa%2CF&author=Varoquaux%2CG&author=Gramfort%2CA"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR63">Rentzsch P, Witten D, Cooper GM et al (2019) CADD: predicting the deleteriousness of variants throughout the human genome. Nucleic Acids Res 47:D886–D894. <a href="https://doi.org/10.1093/nar/gky1016" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1093/nar/gky1016">https://doi.org/10.1093/nar/gky1016</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1093/nar/gky1016" data-track-item_id="10.1093/nar/gky1016" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1093%2Fnar%2Fgky1016" aria-label="Article reference 63" data-doi="10.1093/nar/gky1016">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BC1MXhs1CgtL%2FI" aria-label="CAS reference 63">CAS</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=30371827" aria-label="PubMed reference 63">PubMed</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 63" href="http://scholar.google.com/scholar_lookup?&title=CADD%3A%20predicting%20the%20deleteriousness%20of%20variants%20throughout%20the%20human%20genome&journal=Nucleic%20Acids%20Res&doi=10.1093%2Fnar%2Fgky1016&volume=47&pages=D886-D894&publication_year=2019&author=Rentzsch%2CP&author=Witten%2CD&author=Cooper%2CGM"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR64">Ritchie MD, Hahn LW, Roodi N et al (2001) Multifactor-dimensionality reduction reveals high-order interactions among estrogen-metabolism genes in sporadic breast cancer. Am J Hum Genet 69:138–147. <a href="https://doi.org/10.1086/321276" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1086/321276">https://doi.org/10.1086/321276</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1086/321276" data-track-item_id="10.1086/321276" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1086%2F321276" aria-label="Article reference 64" data-doi="10.1086/321276">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BD3MXls1emt7k%3D" aria-label="CAS reference 64">CAS</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=11404819" aria-label="PubMed reference 64">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1226028" aria-label="PubMed Central reference 64">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 64" href="http://scholar.google.com/scholar_lookup?&title=Multifactor-dimensionality%20reduction%20reveals%20high-order%20interactions%20among%20estrogen-metabolism%20genes%20in%20sporadic%20breast%20cancer&journal=Am%20J%20Hum%20Genet&doi=10.1086%2F321276&volume=69&pages=138-147&publication_year=2001&author=Ritchie%2CMD&author=Hahn%2CLW&author=Roodi%2CN"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR65">Ritchie GRS, Dunham I, Zeggini E, Flicek P (2014) Functional annotation of noncoding sequence variants. Nat Methods 11:294–296. <a href="https://doi.org/10.1038/nmeth.2832" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1038/nmeth.2832">https://doi.org/10.1038/nmeth.2832</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1038/nmeth.2832" data-track-item_id="10.1038/nmeth.2832" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1038%2Fnmeth.2832" aria-label="Article reference 65" data-doi="10.1038/nmeth.2832">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BC2cXhs1Shsr8%3D" aria-label="CAS reference 65">CAS</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=24487584" aria-label="PubMed reference 65">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC5015703" aria-label="PubMed Central reference 65">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 65" href="http://scholar.google.com/scholar_lookup?&title=Functional%20annotation%20of%20noncoding%20sequence%20variants&journal=Nat%20Methods&doi=10.1038%2Fnmeth.2832&volume=11&pages=294-296&publication_year=2014&author=Ritchie%2CGRS&author=Dunham%2CI&author=Zeggini%2CE&author=Flicek%2CP"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR66">Romano JD, Le TT, Fu W, Moore JH (2021) TPOT-NN: augmenting tree-based automated machine learning with neural network estimators. Genet Program Evol Mach. <a href="https://doi.org/10.1007/s10710-021-09401-z" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1007/s10710-021-09401-z">https://doi.org/10.1007/s10710-021-09401-z</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="noopener" data-track-label="10.1007/s10710-021-09401-z" data-track-item_id="10.1007/s10710-021-09401-z" data-track-value="article reference" data-track-action="article reference" href="https://link.springer.com/doi/10.1007/s10710-021-09401-z" aria-label="Article reference 66" data-doi="10.1007/s10710-021-09401-z">Article</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 66" href="http://scholar.google.com/scholar_lookup?&title=TPOT-NN%3A%20augmenting%20tree-based%20automated%20machine%20learning%20with%20neural%20network%20estimators&journal=Genet%20Program%20Evol%20Mach&doi=10.1007%2Fs10710-021-09401-z&publication_year=2021&author=Romano%2CJD&author=Le%2CTT&author=Fu%2CW&author=Moore%2CJH"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR67">Schubach M, Re M, Robinson PN, Valentini G (2017) Imbalance-aware machine learning for predicting rare and common disease-associated non-coding variants. Sci Rep 7:2959. <a href="https://doi.org/10.1038/s41598-017-03011-5" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1038/s41598-017-03011-5">https://doi.org/10.1038/s41598-017-03011-5</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1038/s41598-017-03011-5" data-track-item_id="10.1038/s41598-017-03011-5" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1038%2Fs41598-017-03011-5" aria-label="Article reference 67" data-doi="10.1038/s41598-017-03011-5">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BC1cXhtlKgsbrF" aria-label="CAS reference 67">CAS</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=28592878" aria-label="PubMed reference 67">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC5462751" aria-label="PubMed Central reference 67">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 67" href="http://scholar.google.com/scholar_lookup?&title=Imbalance-aware%20machine%20learning%20for%20predicting%20rare%20and%20common%20disease-associated%20non-coding%20variants&journal=Sci%20Rep&doi=10.1038%2Fs41598-017-03011-5&volume=7&publication_year=2017&author=Schubach%2CM&author=Re%2CM&author=Robinson%2CPN&author=Valentini%2CG"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR68">Seo K, Chung B, Panchaseelan HP et al (2021) Forecasting the walking assistance rehabilitation level of stroke patients using artificial intelligence. Diagnostics (basel) 11:1096. <a href="https://doi.org/10.3390/diagnostics11061096" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.3390/diagnostics11061096">https://doi.org/10.3390/diagnostics11061096</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.3390/diagnostics11061096" data-track-item_id="10.3390/diagnostics11061096" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.3390%2Fdiagnostics11061096" aria-label="Article reference 68" data-doi="10.3390/diagnostics11061096">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BB3MXisF2isb3J" aria-label="CAS reference 68">CAS</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 68" href="http://scholar.google.com/scholar_lookup?&title=Forecasting%20the%20walking%20assistance%20rehabilitation%20level%20of%20stroke%20patients%20using%20artificial%20intelligence&journal=Diagnostics%20%28basel%29&doi=10.3390%2Fdiagnostics11061096&volume=11&publication_year=2021&author=Seo%2CK&author=Chung%2CB&author=Panchaseelan%2CHP"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR69">Shi S, Yuan N, Yang M et al (2018) Comprehensive assessment of genotype imputation performance. HHE 83:107–116. <a href="https://doi.org/10.1159/000489758" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1159/000489758">https://doi.org/10.1159/000489758</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1159/000489758" data-track-item_id="10.1159/000489758" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1159%2F000489758" aria-label="Article reference 69" data-doi="10.1159/000489758">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BC1MXjsVCqsbk%3D" aria-label="CAS reference 69">CAS</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 69" href="http://scholar.google.com/scholar_lookup?&title=Comprehensive%20assessment%20of%20genotype%20imputation%20performance&journal=HHE&doi=10.1159%2F000489758&volume=83&pages=107-116&publication_year=2018&author=Shi%2CS&author=Yuan%2CN&author=Yang%2CM"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR70">Shousha HI, Awad AH, Omran DA et al (2018) Data mining and machine learning algorithms using IL28B genotype and biochemical markers best predicted advanced liver fibrosis in chronic Hepatitis C. Jpn J Infect Dis 71:51–57. <a href="https://doi.org/10.7883/yoken.JJID.2017.089" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.7883/yoken.JJID.2017.089">https://doi.org/10.7883/yoken.JJID.2017.089</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.7883/yoken.JJID.2017.089" data-track-item_id="10.7883/yoken.JJID.2017.089" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.7883%2Fyoken.JJID.2017.089" aria-label="Article reference 70" data-doi="10.7883/yoken.JJID.2017.089">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BC1MXkslKgu7c%3D" aria-label="CAS reference 70">CAS</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=29279441" aria-label="PubMed reference 70">PubMed</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 70" href="http://scholar.google.com/scholar_lookup?&title=Data%20mining%20and%20machine%20learning%20algorithms%20using%20IL28B%20genotype%20and%20biochemical%20markers%20best%20predicted%20advanced%20liver%20fibrosis%20in%20chronic%20Hepatitis%20C&journal=Jpn%20J%20Infect%20Dis&doi=10.7883%2Fyoken.JJID.2017.089&volume=71&pages=51-57&publication_year=2018&author=Shousha%2CHI&author=Awad%2CAH&author=Omran%2CDA"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR71">Sinisi SE, Polley EC, Petersen ML et al (2007) Super learning: an application to the prediction of HIV-1 drug resistance. Stat Appl Genet Mol Biol. <a href="https://doi.org/10.2202/1544-6115.1240" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.2202/1544-6115.1240">https://doi.org/10.2202/1544-6115.1240</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.2202/1544-6115.1240" data-track-item_id="10.2202/1544-6115.1240" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.2202%2F1544-6115.1240" aria-label="Article reference 71" data-doi="10.2202/1544-6115.1240">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=17402922" aria-label="PubMed reference 71">PubMed</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 71" href="http://scholar.google.com/scholar_lookup?&title=Super%20learning%3A%20an%20application%20to%20the%20prediction%20of%20HIV-1%20drug%20resistance&journal=Stat%20Appl%20Genet%20Mol%20Biol&doi=10.2202%2F1544-6115.1240&publication_year=2007&author=Sinisi%2CSE&author=Polley%2CEC&author=Petersen%2CML"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR72">Sohn A, Olson RS, Moore JH (2017) Toward the automated analysis of complex diseases in genome-wide association studies using genetic programming. Proceedings of the genetic and evolutionary computation conference. Association for Computing Machinery, New York, pp 489–496</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1145/3071178.3071212" data-track-item_id="10.1145/3071178.3071212" data-track-value="chapter reference" data-track-action="chapter reference" href="https://doi.org/10.1145%2F3071178.3071212" aria-label="Chapter reference 72" data-doi="10.1145/3071178.3071212">Chapter</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 72" href="http://scholar.google.com/scholar_lookup?&title=Toward%20the%20automated%20analysis%20of%20complex%20diseases%20in%20genome-wide%20association%20studies%20using%20genetic%20programming&doi=10.1145%2F3071178.3071212&pages=489-496&publication_year=2017&author=Sohn%2CA&author=Olson%2CRS&author=Moore%2CJH"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR73">Subramanian A, Tamayo P, Mootha VK et al (2005) Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles. Proc Natl Acad Sci USA 102:15545–15550. <a href="https://doi.org/10.1073/pnas.0506580102" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1073/pnas.0506580102">https://doi.org/10.1073/pnas.0506580102</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1073/pnas.0506580102" data-track-item_id="10.1073/pnas.0506580102" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1073%2Fpnas.0506580102" aria-label="Article reference 73" data-doi="10.1073/pnas.0506580102">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BD2MXht1ShtrnO" aria-label="CAS reference 73">CAS</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=16199517" aria-label="PubMed reference 73">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1239896" aria-label="PubMed Central reference 73">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 73" href="http://scholar.google.com/scholar_lookup?&title=Gene%20set%20enrichment%20analysis%3A%20a%20knowledge-based%20approach%20for%20interpreting%20genome-wide%20expression%20profiles&journal=Proc%20Natl%20Acad%20Sci%20USA&doi=10.1073%2Fpnas.0506580102&volume=102&pages=15545-15550&publication_year=2005&author=Subramanian%2CA&author=Tamayo%2CP&author=Mootha%2CVK"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR74">Thornton C, Hutter F, Hoos HH, Leyton-Brown K (2013) Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms. In: Proceeding of KDD-2013, pp 847–855. <a href="https://doi.org/10.1145/2487575.2487629" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1145/2487575.2487629">https://doi.org/10.1145/2487575.2487629</a></p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR75">Tragante V, Hemerich D, Alshabeeb M et al (2018) Druggability of coronary artery disease risk loci. Circulation 11:e001977. <a href="https://doi.org/10.1161/CIRCGEN.117.001977" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1161/CIRCGEN.117.001977">https://doi.org/10.1161/CIRCGEN.117.001977</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1161/CIRCGEN.117.001977" data-track-item_id="10.1161/CIRCGEN.117.001977" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1161%2FCIRCGEN.117.001977" aria-label="Article reference 75" data-doi="10.1161/CIRCGEN.117.001977">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BC1cXhslOru77F" aria-label="CAS reference 75">CAS</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=30354342" aria-label="PubMed reference 75">PubMed</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 75" href="http://scholar.google.com/scholar_lookup?&title=Druggability%20of%20coronary%20artery%20disease%20risk%20loci&journal=Circulation&doi=10.1161%2FCIRCGEN.117.001977&volume=11&publication_year=2018&author=Tragante%2CV&author=Hemerich%2CD&author=Alshabeeb%2CM"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR76">Tran LM, Mocle AJ, Ramsaran AI et al (2020) Automated curation of CNMF-E-extracted ROI spatial footprints and calcium traces using open-source AutoML tools. Front Neural Circ. <a href="https://doi.org/10.3389/fncir.2020.00042" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.3389/fncir.2020.00042">https://doi.org/10.3389/fncir.2020.00042</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.3389/fncir.2020.00042" data-track-item_id="10.3389/fncir.2020.00042" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.3389%2Ffncir.2020.00042" aria-label="Article reference 76" data-doi="10.3389/fncir.2020.00042">Article</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 76" href="http://scholar.google.com/scholar_lookup?&title=Automated%20curation%20of%20CNMF-E-extracted%20ROI%20spatial%20footprints%20and%20calcium%20traces%20using%20open-source%20AutoML%20tools&journal=Front%20Neural%20Circ&doi=10.3389%2Ffncir.2020.00042&publication_year=2020&author=Tran%2CLM&author=Mocle%2CAJ&author=Ramsaran%2CAI"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR77">van der Laan MJ, Polley EC, Hubbard AE (2007) Super learner. Stat Appl Genetics Mol Biol. <a href="https://doi.org/10.2202/1544-6115.1309" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.2202/1544-6115.1309">https://doi.org/10.2202/1544-6115.1309</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.2202/1544-6115.1309" data-track-item_id="10.2202/1544-6115.1309" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.2202%2F1544-6115.1309" aria-label="Article reference 77" data-doi="10.2202/1544-6115.1309">Article</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 77" href="http://scholar.google.com/scholar_lookup?&title=Super%20learner&journal=Stat%20Appl%20Genetics%20Mol%20Biol&doi=10.2202%2F1544-6115.1309&publication_year=2007&author=Laan%2CMJ&author=Polley%2CEC&author=Hubbard%2CAE"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR78">Vanschoren J, van Rijn JN, Bischl B, Torgo L (2014) OpenML: networked science in machine learning. SIGKDD Explor Newsl 15:49–60. <a href="https://doi.org/10.1145/2641190.2641198" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1145/2641190.2641198">https://doi.org/10.1145/2641190.2641198</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1145/2641190.2641198" data-track-item_id="10.1145/2641190.2641198" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1145%2F2641190.2641198" aria-label="Article reference 78" data-doi="10.1145/2641190.2641198">Article</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 78" href="http://scholar.google.com/scholar_lookup?&title=OpenML%3A%20networked%20science%20in%20machine%20learning&journal=SIGKDD%20Explor%20Newsl&doi=10.1145%2F2641190.2641198&volume=15&pages=49-60&publication_year=2014&author=Vanschoren%2CJ&author=Rijn%2CJN&author=Bischl%2CB&author=Torgo%2CL"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR79">Venugopalan J, Tong L, Hassanzadeh HR, Wang MD (2021) Multimodal deep learning models for early detection of Alzheimer’s disease stage. Sci Rep 11:3254. <a href="https://doi.org/10.1038/s41598-020-74399-w" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1038/s41598-020-74399-w">https://doi.org/10.1038/s41598-020-74399-w</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1038/s41598-020-74399-w" data-track-item_id="10.1038/s41598-020-74399-w" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1038%2Fs41598-020-74399-w" aria-label="Article reference 79" data-doi="10.1038/s41598-020-74399-w">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="cas reference" data-track-action="cas reference" href="/articles/cas-redirect/1:CAS:528:DC%2BB3MXjvFOlsLw%3D" aria-label="CAS reference 79">CAS</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=33547343" aria-label="PubMed reference 79">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC7864942" aria-label="PubMed Central reference 79">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 79" href="http://scholar.google.com/scholar_lookup?&title=Multimodal%20deep%20learning%20models%20for%20early%20detection%20of%20Alzheimer%E2%80%99s%20disease%20stage&journal=Sci%20Rep&doi=10.1038%2Fs41598-020-74399-w&volume=11&publication_year=2021&author=Venugopalan%2CJ&author=Tong%2CL&author=Hassanzadeh%2CHR&author=Wang%2CMD"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR80">Wang H-L, Hsu W-Y, Lee M-H et al (2019) Automatic machine-learning-based outcome prediction in patients with primary intracerebral hemorrhage. Front Neurol 10:910. <a href="https://doi.org/10.3389/fneur.2019.00910" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.3389/fneur.2019.00910">https://doi.org/10.3389/fneur.2019.00910</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.3389/fneur.2019.00910" data-track-item_id="10.3389/fneur.2019.00910" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.3389%2Ffneur.2019.00910" aria-label="Article reference 80" data-doi="10.3389/fneur.2019.00910">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=31496988" aria-label="PubMed reference 80">PubMed</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed central reference" data-track-action="pubmed central reference" href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC6713018" aria-label="PubMed Central reference 80">PubMed Central</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 80" href="http://scholar.google.com/scholar_lookup?&title=Automatic%20machine-learning-based%20outcome%20prediction%20in%20patients%20with%20primary%20intracerebral%20hemorrhage&journal=Front%20Neurol&doi=10.3389%2Ffneur.2019.00910&volume=10&publication_year=2019&author=Wang%2CH-L&author=Hsu%2CW-Y&author=Lee%2CM-H"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR81">Wang F, Jiang M, Qian C, et al (2017) Residual attention network for image classification. <a href="http://arxiv.org/abs/1704.06904" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="http://arxiv.org/abs/1704.06904">http://arxiv.org/abs/1704.06904</a> [cs]</p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR82">Wang C, Wu Q, Weimer M, Zhu E (2021) FLAML: a fast and lightweight AutoML Library. <a href="http://arxiv.org/abs/1911.04706" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="http://arxiv.org/abs/1911.04706">http://arxiv.org/abs/1911.04706</a> [cs, stat]</p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR83">Waring J, Lindvall C, Umeton R (2020) Automated machine learning: review of the state-of-the-art and opportunities for healthcare. Artif Intell Med 104:101822. <a href="https://doi.org/10.1016/j.artmed.2020.101822" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1016/j.artmed.2020.101822">https://doi.org/10.1016/j.artmed.2020.101822</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1016/j.artmed.2020.101822" data-track-item_id="10.1016/j.artmed.2020.101822" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1016%2Fj.artmed.2020.101822" aria-label="Article reference 83" data-doi="10.1016/j.artmed.2020.101822">Article</a> <a data-track="click_references" rel="nofollow noopener" data-track-label="link" data-track-item_id="link" data-track-value="pubmed reference" data-track-action="pubmed reference" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=32499001" aria-label="PubMed reference 83">PubMed</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 83" href="http://scholar.google.com/scholar_lookup?&title=Automated%20machine%20learning%3A%20review%20of%20the%20state-of-the-art%20and%20opportunities%20for%20healthcare&journal=Artif%20Intell%20Med&doi=10.1016%2Fj.artmed.2020.101822&volume=104&publication_year=2020&author=Waring%2CJ&author=Lindvall%2CC&author=Umeton%2CR"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR84">Wolpert DH (1996) The lack of a priori distinctions between learning algorithms. Neural Comput 8:1341–1390. <a href="https://doi.org/10.1162/neco.1996.8.7.1341" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1162/neco.1996.8.7.1341">https://doi.org/10.1162/neco.1996.8.7.1341</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1162/neco.1996.8.7.1341" data-track-item_id="10.1162/neco.1996.8.7.1341" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1162%2Fneco.1996.8.7.1341" aria-label="Article reference 84" data-doi="10.1162/neco.1996.8.7.1341">Article</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 84" href="http://scholar.google.com/scholar_lookup?&title=The%20lack%20of%20a%20priori%20distinctions%20between%20learning%20algorithms&journal=Neural%20Comput&doi=10.1162%2Fneco.1996.8.7.1341&volume=8&pages=1341-1390&publication_year=1996&author=Wolpert%2CDH"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR85">Wolpert DH, Macready WG (1997) No free lunch theorems for optimization. IEEE Trans Evol Comput 1:67–82. <a href="https://doi.org/10.1109/4235.585893" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1109/4235.585893">https://doi.org/10.1109/4235.585893</a></p><p class="c-article-references__links u-hide-print"><a data-track="click_references" rel="nofollow noopener" data-track-label="10.1109/4235.585893" data-track-item_id="10.1109/4235.585893" data-track-value="article reference" data-track-action="article reference" href="https://doi.org/10.1109%2F4235.585893" aria-label="Article reference 85" data-doi="10.1109/4235.585893">Article</a> <a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 85" href="http://scholar.google.com/scholar_lookup?&title=No%20free%20lunch%20theorems%20for%20optimization&journal=IEEE%20Trans%20Evol%20Comput&doi=10.1109%2F4235.585893&volume=1&pages=67-82&publication_year=1997&author=Wolpert%2CDH&author=Macready%2CWG"> Google Scholar</a> </p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR86">Yang C, Akimoto Y, Kim DW, Udell M (2019) OBOE: collaborative filtering for AutoML model selection. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp 1173–1183. <a href="https://doi.org/10.1145/3292500.3330909" data-track="click_references" data-track-action="external reference" data-track-value="external reference" data-track-label="10.1145/3292500.3330909">https://doi.org/10.1145/3292500.3330909</a></p></li><li class="c-article-references__item js-c-reading-companion-references-item"><p class="c-article-references__text" id="ref-CR87">Zheng A (2015) Evaluating machine learning models: a beginner’s guide to key concepts and pitfalls. O’Reilly Media, Newton</p><p class="c-article-references__links u-hide-print"><a data-track="click_references" data-track-action="google scholar reference" data-track-value="google scholar reference" data-track-label="link" data-track-item_id="link" rel="nofollow noopener" aria-label="Google Scholar reference 87" href="http://scholar.google.com/scholar_lookup?&title=Evaluating%20machine%20learning%20models%3A%20a%20beginner%E2%80%99s%20guide%20to%20key%20concepts%20and%20pitfalls&publication_year=2015&author=Zheng%2CA"> Google Scholar</a> </p></li></ul><p class="c-article-references__download u-hide-print"><a data-track="click" data-track-action="download citation references" data-track-label="link" rel="nofollow" href="https://citation-needed.springer.com/v2/references/10.1007/s00439-021-02393-x?format=refman&flavour=references">Download references<svg width="16" height="16" focusable="false" role="img" aria-hidden="true" class="u-icon"><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#icon-eds-i-download-medium"></use></svg></a></p></div></div></div></section></div><section data-title="Acknowledgements"><div class="c-article-section" id="Ack1-section"><h2 class="c-article-section__title js-section-title js-c-reading-companion-sections-item" id="Ack1">Acknowledgements</h2><div class="c-article-section__content" id="Ack1-content"><p>We thank the anonymous reviewers for their constructive comments.</p></div></div></section><section data-title="Funding"><div class="c-article-section" id="Fun-section"><h2 class="c-article-section__title js-section-title js-c-reading-companion-sections-item" id="Fun">Funding</h2><div class="c-article-section__content" id="Fun-content"><p>We acknowledge support from NIH Grants LM010098 and AG066833. JDR also received support from NIH training Grant T32-ES019851.</p></div></div></section><section aria-labelledby="author-information" data-title="Author information"><div class="c-article-section" id="author-information-section"><h2 class="c-article-section__title js-section-title js-c-reading-companion-sections-item" id="author-information">Author information</h2><div class="c-article-section__content" id="author-information-content"><h3 class="c-article__sub-heading" id="affiliations">Authors and Affiliations</h3><ol class="c-article-author-affiliation__list"><li id="Aff1"><p class="c-article-author-affiliation__address">Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, PA, 19104, USA</p><p class="c-article-author-affiliation__authors-list">Elisabetta Manduchi, Joseph D. Romano & Jason H. Moore</p></li><li id="Aff2"><p class="c-article-author-affiliation__address">Institute for Biomedical Informatics, University of Pennsylvania, Philadelphia, PA, 19104, USA</p><p class="c-article-author-affiliation__authors-list">Elisabetta Manduchi & Jason H. Moore</p></li></ol><div class="u-js-hide u-hide-print" data-test="author-info"><span class="c-article__sub-heading">Authors</span><ol class="c-article-authors-search u-list-reset"><li id="auth-Elisabetta-Manduchi-Aff1-Aff2"><span class="c-article-authors-search__title u-h3 js-search-name">Elisabetta Manduchi</span><div class="c-article-authors-search__list"><div class="c-article-authors-search__item c-article-authors-search__list-item--left"><a href="/search?dc.creator=Elisabetta%20Manduchi" class="c-article-button" data-track="click" data-track-action="author link - publication" data-track-label="link" rel="nofollow">View author publications</a></div><div class="c-article-authors-search__item c-article-authors-search__list-item--right"><p class="search-in-title-js c-article-authors-search__text">You can also search for this author in <span class="c-article-identifiers"><a class="c-article-identifiers__item" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Elisabetta%20Manduchi" data-track="click" data-track-action="author link - pubmed" data-track-label="link" rel="nofollow">PubMed</a><span class="u-hide"> </span><a class="c-article-identifiers__item" href="http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Elisabetta%20Manduchi%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en" data-track="click" data-track-action="author link - scholar" data-track-label="link" rel="nofollow">Google Scholar</a></span></p></div></div></li><li id="auth-Joseph_D_-Romano-Aff1"><span class="c-article-authors-search__title u-h3 js-search-name">Joseph D. Romano</span><div class="c-article-authors-search__list"><div class="c-article-authors-search__item c-article-authors-search__list-item--left"><a href="/search?dc.creator=Joseph%20D.%20Romano" class="c-article-button" data-track="click" data-track-action="author link - publication" data-track-label="link" rel="nofollow">View author publications</a></div><div class="c-article-authors-search__item c-article-authors-search__list-item--right"><p class="search-in-title-js c-article-authors-search__text">You can also search for this author in <span class="c-article-identifiers"><a class="c-article-identifiers__item" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Joseph%20D.%20Romano" data-track="click" data-track-action="author link - pubmed" data-track-label="link" rel="nofollow">PubMed</a><span class="u-hide"> </span><a class="c-article-identifiers__item" href="http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Joseph%20D.%20Romano%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en" data-track="click" data-track-action="author link - scholar" data-track-label="link" rel="nofollow">Google Scholar</a></span></p></div></div></li><li id="auth-Jason_H_-Moore-Aff1-Aff2"><span class="c-article-authors-search__title u-h3 js-search-name">Jason H. Moore</span><div class="c-article-authors-search__list"><div class="c-article-authors-search__item c-article-authors-search__list-item--left"><a href="/search?dc.creator=Jason%20H.%20Moore" class="c-article-button" data-track="click" data-track-action="author link - publication" data-track-label="link" rel="nofollow">View author publications</a></div><div class="c-article-authors-search__item c-article-authors-search__list-item--right"><p class="search-in-title-js c-article-authors-search__text">You can also search for this author in <span class="c-article-identifiers"><a class="c-article-identifiers__item" href="http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Jason%20H.%20Moore" data-track="click" data-track-action="author link - pubmed" data-track-label="link" rel="nofollow">PubMed</a><span class="u-hide"> </span><a class="c-article-identifiers__item" href="http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Jason%20H.%20Moore%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en" data-track="click" data-track-action="author link - scholar" data-track-label="link" rel="nofollow">Google Scholar</a></span></p></div></div></li></ol></div><h3 class="c-article__sub-heading" id="corresponding-author">Corresponding author</h3><p id="corresponding-author-list">Correspondence to <a id="corresp-c1" href="mailto:jhmoore@upenn.edu">Jason H. Moore</a>.</p></div></div></section><section data-title="Ethics declarations"><div class="c-article-section" id="ethics-section"><h2 class="c-article-section__title js-section-title js-c-reading-companion-sections-item" id="ethics">Ethics declarations</h2><div class="c-article-section__content" id="ethics-content"> <h3 class="c-article__sub-heading" id="FPar1">Conflict of interest</h3> <p>On behalf of all authors, the corresponding author states that there is no conflict of interest.</p> </div></div></section><section data-title="Additional information"><div class="c-article-section" id="additional-information-section"><h2 class="c-article-section__title js-section-title js-c-reading-companion-sections-item" id="additional-information">Additional information</h2><div class="c-article-section__content" id="additional-information-content"><h3 class="c-article__sub-heading">Publisher's Note</h3><p>Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.</p></div></div></section><section data-title="Rights and permissions"><div class="c-article-section" id="rightslink-section"><h2 class="c-article-section__title js-section-title js-c-reading-companion-sections-item" id="rightslink">Rights and permissions</h2><div class="c-article-section__content" id="rightslink-content"> <p><b>Open Access</b> This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit <a href="http://creativecommons.org/licenses/by/4.0/" rel="license">http://creativecommons.org/licenses/by/4.0/</a>.</p> <p class="c-article-rights"><a data-track="click" data-track-action="view rights and permissions" data-track-label="link" href="https://s100.copyright.com/AppDispatchServlet?title=The%20promise%20of%20automated%20machine%20learning%20for%20the%20genetic%20analysis%20of%20complex%20traits&author=Elisabetta%20Manduchi%20et%20al&contentID=10.1007%2Fs00439-021-02393-x&copyright=The%20Author%28s%29&publication=0340-6717&publicationDate=2021-10-28&publisherName=SpringerNature&orderBeanReset=true&oa=CC%20BY">Reprints and permissions</a></p></div></div></section><section aria-labelledby="article-info" data-title="About this article"><div class="c-article-section" id="article-info-section"><h2 class="c-article-section__title js-section-title js-c-reading-companion-sections-item" id="article-info">About this article</h2><div class="c-article-section__content" id="article-info-content"><div class="c-bibliographic-information"><div class="u-hide-print c-bibliographic-information__column c-bibliographic-information__column--border"><a data-crossmark="10.1007/s00439-021-02393-x" target="_blank" rel="noopener" href="https://crossmark.crossref.org/dialog/?doi=10.1007/s00439-021-02393-x" data-track="click" data-track-action="Click Crossmark" data-track-label="link" data-test="crossmark"><img loading="lazy" width="57" height="81" alt="Check for updates. Verify currency and authenticity via CrossMark" src="data:image/svg+xml;base64,PHN2ZyBoZWlnaHQ9IjgxIiB3aWR0aD0iNTciIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+PGcgZmlsbD0ibm9uZSIgZmlsbC1ydWxlPSJldmVub2RkIj48cGF0aCBkPSJtMTcuMzUgMzUuNDUgMjEuMy0xNC4ydi0xNy4wM2gtMjEuMyIgZmlsbD0iIzk4OTg5OCIvPjxwYXRoIGQ9Im0zOC42NSAzNS40NS0yMS4zLTE0LjJ2LTE3LjAzaDIxLjMiIGZpbGw9IiM3NDc0NzQiLz48cGF0aCBkPSJtMjggLjVjLTEyLjk4IDAtMjMuNSAxMC41Mi0yMy41IDIzLjVzMTAuNTIgMjMuNSAyMy41IDIzLjUgMjMuNS0xMC41MiAyMy41LTIzLjVjMC02LjIzLTIuNDgtMTIuMjEtNi44OC0xNi42Mi00LjQxLTQuNC0xMC4zOS02Ljg4LTE2LjYyLTYuODh6bTAgNDEuMjVjLTkuOCAwLTE3Ljc1LTcuOTUtMTcuNzUtMTcuNzVzNy45NS0xNy43NSAxNy43NS0xNy43NSAxNy43NSA3Ljk1IDE3Ljc1IDE3Ljc1YzAgNC43MS0xLjg3IDkuMjItNS4yIDEyLjU1cy03Ljg0IDUuMi0xMi41NSA1LjJ6IiBmaWxsPSIjNTM1MzUzIi8+PHBhdGggZD0ibTQxIDM2Yy01LjgxIDYuMjMtMTUuMjMgNy40NS0yMi40MyAyLjktNy4yMS00LjU1LTEwLjE2LTEzLjU3LTcuMDMtMjEuNWwtNC45Mi0zLjExYy00Ljk1IDEwLjctMS4xOSAyMy40MiA4Ljc4IDI5LjcxIDkuOTcgNi4zIDIzLjA3IDQuMjIgMzAuNi00Ljg2eiIgZmlsbD0iIzljOWM5YyIvPjxwYXRoIGQ9Im0uMiA1OC40NWMwLS43NS4xMS0xLjQyLjMzLTIuMDFzLjUyLTEuMDkuOTEtMS41Yy4zOC0uNDEuODMtLjczIDEuMzQtLjk0LjUxLS4yMiAxLjA2LS4zMiAxLjY1LS4zMi41NiAwIDEuMDYuMTEgMS41MS4zNS40NC4yMy44MS41IDEuMS44MWwtLjkxIDEuMDFjLS4yNC0uMjQtLjQ5LS40Mi0uNzUtLjU2LS4yNy0uMTMtLjU4LS4yLS45My0uMi0uMzkgMC0uNzMuMDgtMS4wNS4yMy0uMzEuMTYtLjU4LjM3LS44MS42Ni0uMjMuMjgtLjQxLjYzLS41MyAxLjA0LS4xMy40MS0uMTkuODgtLjE5IDEuMzkgMCAxLjA0LjIzIDEuODYuNjggMi40Ni40NS41OSAxLjA2Ljg4IDEuODQuODguNDEgMCAuNzctLjA3IDEuMDctLjIzcy41OS0uMzkuODUtLjY4bC45MSAxYy0uMzguNDMtLjguNzYtMS4yOC45OS0uNDcuMjItMSAuMzQtMS41OC4zNC0uNTkgMC0xLjEzLS4xLTEuNjQtLjMxLS41LS4yLS45NC0uNTEtMS4zMS0uOTEtLjM4LS40LS42Ny0uOS0uODgtMS40OC0uMjItLjU5LS4zMy0xLjI2LS4zMy0yLjAyem04LjQtNS4zM2gxLjYxdjIuNTRsLS4wNSAxLjMzYy4yOS0uMjcuNjEtLjUxLjk2LS43MnMuNzYtLjMxIDEuMjQtLjMxYy43MyAwIDEuMjcuMjMgMS42MS43MS4zMy40Ny41IDEuMTQuNSAyLjAydjQuMzFoLTEuNjF2LTQuMWMwLS41Ny0uMDgtLjk3LS4yNS0xLjIxLS4xNy0uMjMtLjQ1LS4zNS0uODMtLjM1LS4zIDAtLjU2LjA4LS43OS4yMi0uMjMuMTUtLjQ5LjM2LS43OC42NHY0LjhoLTEuNjF6bTcuMzcgNi40NWMwLS41Ni4wOS0xLjA2LjI2LTEuNTEuMTgtLjQ1LjQyLS44My43MS0xLjE0LjI5LS4zLjYzLS41NCAxLjAxLS43MS4zOS0uMTcuNzgtLjI1IDEuMTgtLjI1LjQ3IDAgLjg4LjA4IDEuMjMuMjQuMzYuMTYuNjUuMzguODkuNjdzLjQyLjYzLjU0IDEuMDNjLjEyLjQxLjE4Ljg0LjE4IDEuMzIgMCAuMzItLjAyLjU3LS4wNy43NmgtNC4zNmMuMDcuNjIuMjkgMS4xLjY1IDEuNDQuMzYuMzMuODIuNSAxLjM4LjUuMjkgMCAuNTctLjA0LjgzLS4xM3MuNTEtLjIxLjc2LS4zN2wuNTUgMS4wMWMtLjMzLjIxLS42OS4zOS0xLjA5LjUzLS40MS4xNC0uODMuMjEtMS4yNi4yMS0uNDggMC0uOTItLjA4LTEuMzQtLjI1LS40MS0uMTYtLjc2LS40LTEuMDctLjctLjMxLS4zMS0uNTUtLjY5LS43Mi0xLjEzLS4xOC0uNDQtLjI2LS45NS0uMjYtMS41MnptNC42LS42MmMwLS41NS0uMTEtLjk4LS4zNC0xLjI4LS4yMy0uMzEtLjU4LS40Ny0xLjA2LS40Ny0uNDEgMC0uNzcuMTUtMS4wNy40NS0uMzEuMjktLjUuNzMtLjU4IDEuM3ptMi41LjYyYzAtLjU3LjA5LTEuMDguMjgtMS41My4xOC0uNDQuNDMtLjgyLjc1LTEuMTNzLjY5LS41NCAxLjEtLjcxYy40Mi0uMTYuODUtLjI0IDEuMzEtLjI0LjQ1IDAgLjg0LjA4IDEuMTcuMjNzLjYxLjM0Ljg1LjU3bC0uNzcgMS4wMmMtLjE5LS4xNi0uMzgtLjI4LS41Ni0uMzctLjE5LS4wOS0uMzktLjE0LS42MS0uMTQtLjU2IDAtMS4wMS4yMS0xLjM1LjYzLS4zNS40MS0uNTIuOTctLjUyIDEuNjcgMCAuNjkuMTcgMS4yNC41MSAxLjY2LjM0LjQxLjc4LjYyIDEuMzIuNjIuMjggMCAuNTQtLjA2Ljc4LS4xNy4yNC0uMTIuNDUtLjI2LjY0LS40MmwuNjcgMS4wM2MtLjMzLjI5LS42OS41MS0xLjA4LjY1LS4zOS4xNS0uNzguMjMtMS4xOC4yMy0uNDYgMC0uOS0uMDgtMS4zMS0uMjQtLjQtLjE2LS43NS0uMzktMS4wNS0uN3MtLjUzLS42OS0uNy0xLjEzYy0uMTctLjQ1LS4yNS0uOTYtLjI1LTEuNTN6bTYuOTEtNi40NWgxLjU4djYuMTdoLjA1bDIuNTQtMy4xNmgxLjc3bC0yLjM1IDIuOCAyLjU5IDQuMDdoLTEuNzVsLTEuNzctMi45OC0xLjA4IDEuMjN2MS43NWgtMS41OHptMTMuNjkgMS4yN2MtLjI1LS4xMS0uNS0uMTctLjc1LS4xNy0uNTggMC0uODcuMzktLjg3IDEuMTZ2Ljc1aDEuMzR2MS4yN2gtMS4zNHY1LjZoLTEuNjF2LTUuNmgtLjkydi0xLjJsLjkyLS4wN3YtLjcyYzAtLjM1LjA0LS42OC4xMy0uOTguMDgtLjMxLjIxLS41Ny40LS43OXMuNDItLjM5LjcxLS41MWMuMjgtLjEyLjYzLS4xOCAxLjA0LS4xOC4yNCAwIC40OC4wMi42OS4wNy4yMi4wNS40MS4xLjU3LjE3em0uNDggNS4xOGMwLS41Ny4wOS0xLjA4LjI3LTEuNTMuMTctLjQ0LjQxLS44Mi43Mi0xLjEzLjMtLjMxLjY1LS41NCAxLjA0LS43MS4zOS0uMTYuOC0uMjQgMS4yMy0uMjRzLjg0LjA4IDEuMjQuMjRjLjQuMTcuNzQuNCAxLjA0Ljcxcy41NC42OS43MiAxLjEzYy4xOS40NS4yOC45Ni4yOCAxLjUzcy0uMDkgMS4wOC0uMjggMS41M2MtLjE4LjQ0LS40Mi44Mi0uNzIgMS4xM3MtLjY0LjU0LTEuMDQuNy0uODEuMjQtMS4yNC4yNC0uODQtLjA4LTEuMjMtLjI0LS43NC0uMzktMS4wNC0uN2MtLjMxLS4zMS0uNTUtLjY5LS43Mi0xLjEzLS4xOC0uNDUtLjI3LS45Ni0uMjctMS41M3ptMS42NSAwYzAgLjY5LjE0IDEuMjQuNDMgMS42Ni4yOC40MS42OC42MiAxLjE4LjYyLjUxIDAgLjktLjIxIDEuMTktLjYyLjI5LS40Mi40NC0uOTcuNDQtMS42NiAwLS43LS4xNS0xLjI2LS40NC0xLjY3LS4yOS0uNDItLjY4LS42My0xLjE5LS42My0uNSAwLS45LjIxLTEuMTguNjMtLjI5LjQxLS40My45Ny0uNDMgMS42N3ptNi40OC0zLjQ0aDEuMzNsLjEyIDEuMjFoLjA1Yy4yNC0uNDQuNTQtLjc5Ljg4LTEuMDIuMzUtLjI0LjctLjM2IDEuMDctLjM2LjMyIDAgLjU5LjA1Ljc4LjE0bC0uMjggMS40LS4zMy0uMDljLS4xMS0uMDEtLjIzLS4wMi0uMzgtLjAyLS4yNyAwLS41Ni4xLS44Ni4zMXMtLjU1LjU4LS43NyAxLjF2NC4yaC0xLjYxem0tNDcuODcgMTVoMS42MXY0LjFjMCAuNTcuMDguOTcuMjUgMS4yLjE3LjI0LjQ0LjM1LjgxLjM1LjMgMCAuNTctLjA3LjgtLjIyLjIyLS4xNS40Ny0uMzkuNzMtLjczdi00LjdoMS42MXY2Ljg3aC0xLjMybC0uMTItMS4wMWgtLjA0Yy0uMy4zNi0uNjMuNjQtLjk4Ljg2LS4zNS4yMS0uNzYuMzItMS4yNC4zMi0uNzMgMC0xLjI3LS4yNC0xLjYxLS43MS0uMzMtLjQ3LS41LTEuMTQtLjUtMi4wMnptOS40NiA3LjQzdjIuMTZoLTEuNjF2LTkuNTloMS4zM2wuMTIuNzJoLjA1Yy4yOS0uMjQuNjEtLjQ1Ljk3LS42My4zNS0uMTcuNzItLjI2IDEuMS0uMjYuNDMgMCAuODEuMDggMS4xNS4yNC4zMy4xNy42MS40Ljg0LjcxLjI0LjMxLjQxLjY4LjUzIDEuMTEuMTMuNDIuMTkuOTEuMTkgMS40NCAwIC41OS0uMDkgMS4xMS0uMjUgMS41Ny0uMTYuNDctLjM4Ljg1LS42NSAxLjE2LS4yNy4zMi0uNTguNTYtLjk0LjczLS4zNS4xNi0uNzIuMjUtMS4xLjI1LS4zIDAtLjYtLjA3LS45LS4ycy0uNTktLjMxLS44Ny0uNTZ6bTAtMi4zYy4yNi4yMi41LjM3LjczLjQ1LjI0LjA5LjQ2LjEzLjY2LjEzLjQ2IDAgLjg0LS4yIDEuMTUtLjYuMzEtLjM5LjQ2LS45OC40Ni0xLjc3IDAtLjY5LS4xMi0xLjIyLS4zNS0xLjYxLS4yMy0uMzgtLjYxLS41Ny0xLjEzLS41Ny0uNDkgMC0uOTkuMjYtMS41Mi43N3ptNS44Ny0xLjY5YzAtLjU2LjA4LTEuMDYuMjUtMS41MS4xNi0uNDUuMzctLjgzLjY1LTEuMTQuMjctLjMuNTgtLjU0LjkzLS43MXMuNzEtLjI1IDEuMDgtLjI1Yy4zOSAwIC43My4wNyAxIC4yLjI3LjE0LjU0LjMyLjgxLjU1bC0uMDYtMS4xdi0yLjQ5aDEuNjF2OS44OGgtMS4zM2wtLjExLS43NGgtLjA2Yy0uMjUuMjUtLjU0LjQ2LS44OC42NC0uMzMuMTgtLjY5LjI3LTEuMDYuMjctLjg3IDAtMS41Ni0uMzItMi4wNy0uOTVzLS43Ni0xLjUxLS43Ni0yLjY1em0xLjY3LS4wMWMwIC43NC4xMyAxLjMxLjQgMS43LjI2LjM4LjY1LjU4IDEuMTUuNTguNTEgMCAuOTktLjI2IDEuNDQtLjc3di0zLjIxYy0uMjQtLjIxLS40OC0uMzYtLjctLjQ1LS4yMy0uMDgtLjQ2LS4xMi0uNy0uMTItLjQ1IDAtLjgyLjE5LTEuMTMuNTktLjMxLjM5LS40Ni45NS0uNDYgMS42OHptNi4zNSAxLjU5YzAtLjczLjMyLTEuMy45Ny0xLjcxLjY0LS40IDEuNjctLjY4IDMuMDgtLjg0IDAtLjE3LS4wMi0uMzQtLjA3LS41MS0uMDUtLjE2LS4xMi0uMy0uMjItLjQzcy0uMjItLjIyLS4zOC0uM2MtLjE1LS4wNi0uMzQtLjEtLjU4LS4xLS4zNCAwLS42OC4wNy0xIC4ycy0uNjMuMjktLjkzLjQ3bC0uNTktMS4wOGMuMzktLjI0LjgxLS40NSAxLjI4LS42My40Ny0uMTcuOTktLjI2IDEuNTQtLjI2Ljg2IDAgMS41MS4yNSAxLjkzLjc2cy42MyAxLjI1LjYzIDIuMjF2NC4wN2gtMS4zMmwtLjEyLS43NmgtLjA1Yy0uMy4yNy0uNjMuNDgtLjk4LjY2cy0uNzMuMjctMS4xNC4yN2MtLjYxIDAtMS4xLS4xOS0xLjQ4LS41Ni0uMzgtLjM2LS41Ny0uODUtLjU3LTEuNDZ6bTEuNTctLjEyYzAgLjMuMDkuNTMuMjcuNjcuMTkuMTQuNDIuMjEuNzEuMjEuMjggMCAuNTQtLjA3Ljc3LS4ycy40OC0uMzEuNzMtLjU2di0xLjU0Yy0uNDcuMDYtLjg2LjEzLTEuMTguMjMtLjMxLjA5LS41Ny4xOS0uNzYuMzFzLS4zMy4yNS0uNDEuNGMtLjA5LjE1LS4xMy4zMS0uMTMuNDh6bTYuMjktMy42M2gtLjk4di0xLjJsMS4wNi0uMDcuMi0xLjg4aDEuMzR2MS44OGgxLjc1djEuMjdoLTEuNzV2My4yOGMwIC44LjMyIDEuMi45NyAxLjIuMTIgMCAuMjQtLjAxLjM3LS4wNC4xMi0uMDMuMjQtLjA3LjM0LS4xMWwuMjggMS4xOWMtLjE5LjA2LS40LjEyLS42NC4xNy0uMjMuMDUtLjQ5LjA4LS43Ni4wOC0uNCAwLS43NC0uMDYtMS4wMi0uMTgtLjI3LS4xMy0uNDktLjMtLjY3LS41Mi0uMTctLjIxLS4zLS40OC0uMzctLjc4LS4wOC0uMy0uMTItLjY0LS4xMi0xLjAxem00LjM2IDIuMTdjMC0uNTYuMDktMS4wNi4yNy0xLjUxcy40MS0uODMuNzEtMS4xNGMuMjktLjMuNjMtLjU0IDEuMDEtLjcxLjM5LS4xNy43OC0uMjUgMS4xOC0uMjUuNDcgMCAuODguMDggMS4yMy4yNC4zNi4xNi42NS4zOC44OS42N3MuNDIuNjMuNTQgMS4wM2MuMTIuNDEuMTguODQuMTggMS4zMiAwIC4zMi0uMDIuNTctLjA3Ljc2aC00LjM3Yy4wOC42Mi4yOSAxLjEuNjUgMS40NC4zNi4zMy44Mi41IDEuMzguNS4zIDAgLjU4LS4wNC44NC0uMTMuMjUtLjA5LjUxLS4yMS43Ni0uMzdsLjU0IDEuMDFjLS4zMi4yMS0uNjkuMzktMS4wOS41M3MtLjgyLjIxLTEuMjYuMjFjLS40NyAwLS45Mi0uMDgtMS4zMy0uMjUtLjQxLS4xNi0uNzctLjQtMS4wOC0uNy0uMy0uMzEtLjU0LS42OS0uNzItMS4xMy0uMTctLjQ0LS4yNi0uOTUtLjI2LTEuNTJ6bTQuNjEtLjYyYzAtLjU1LS4xMS0uOTgtLjM0LTEuMjgtLjIzLS4zMS0uNTgtLjQ3LTEuMDYtLjQ3LS40MSAwLS43Ny4xNS0xLjA4LjQ1LS4zMS4yOS0uNS43My0uNTcgMS4zem0zLjAxIDIuMjNjLjMxLjI0LjYxLjQzLjkyLjU3LjMuMTMuNjMuMi45OC4yLjM4IDAgLjY1LS4wOC44My0uMjNzLjI3LS4zNS4yNy0uNmMwLS4xNC0uMDUtLjI2LS4xMy0uMzctLjA4LS4xLS4yLS4yLS4zNC0uMjgtLjE0LS4wOS0uMjktLjE2LS40Ny0uMjNsLS41My0uMjJjLS4yMy0uMDktLjQ2LS4xOC0uNjktLjMtLjIzLS4xMS0uNDQtLjI0LS42Mi0uNHMtLjMzLS4zNS0uNDUtLjU1Yy0uMTItLjIxLS4xOC0uNDYtLjE4LS43NSAwLS42MS4yMy0xLjEuNjgtMS40OS40NC0uMzggMS4wNi0uNTcgMS44My0uNTcuNDggMCAuOTEuMDggMS4yOS4yNXMuNzEuMzYuOTkuNTdsLS43NC45OGMtLjI0LS4xNy0uNDktLjMyLS43My0uNDItLjI1LS4xMS0uNTEtLjE2LS43OC0uMTYtLjM1IDAtLjYuMDctLjc2LjIxLS4xNy4xNS0uMjUuMzMtLjI1LjU0IDAgLjE0LjA0LjI2LjEyLjM2cy4xOC4xOC4zMS4yNmMuMTQuMDcuMjkuMTQuNDYuMjFsLjU0LjE5Yy4yMy4wOS40Ny4xOC43LjI5cy40NC4yNC42NC40Yy4xOS4xNi4zNC4zNS40Ni41OC4xMS4yMy4xNy41LjE3LjgyIDAgLjMtLjA2LjU4LS4xNy44My0uMTIuMjYtLjI5LjQ4LS41MS42OC0uMjMuMTktLjUxLjM0LS44NC40NS0uMzQuMTEtLjcyLjE3LTEuMTUuMTctLjQ4IDAtLjk1LS4wOS0xLjQxLS4yNy0uNDYtLjE5LS44Ni0uNDEtMS4yLS42OHoiIGZpbGw9IiM1MzUzNTMiLz48L2c+PC9zdmc+"></a></div><div class="c-bibliographic-information__column"><h3 class="c-article__sub-heading" id="citeas">Cite this article</h3><p class="c-bibliographic-information__citation">Manduchi, E., Romano, J.D. & Moore, J.H. The promise of automated machine learning for the genetic analysis of complex traits. <i>Hum Genet</i> <b>141</b>, 1529–1544 (2022). https://doi.org/10.1007/s00439-021-02393-x</p><p class="c-bibliographic-information__download-citation u-hide-print"><a data-test="citation-link" data-track="click" data-track-action="download article citation" data-track-label="link" data-track-external="" rel="nofollow" href="https://citation-needed.springer.com/v2/references/10.1007/s00439-021-02393-x?format=refman&flavour=citation">Download citation<svg width="16" height="16" focusable="false" role="img" aria-hidden="true" class="u-icon"><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#icon-eds-i-download-medium"></use></svg></a></p><ul class="c-bibliographic-information__list" data-test="publication-history"><li class="c-bibliographic-information__list-item"><p>Received<span class="u-hide">: </span><span class="c-bibliographic-information__value"><time datetime="2021-04-21">21 April 2021</time></span></p></li><li class="c-bibliographic-information__list-item"><p>Accepted<span class="u-hide">: </span><span class="c-bibliographic-information__value"><time datetime="2021-10-22">22 October 2021</time></span></p></li><li class="c-bibliographic-information__list-item"><p>Published<span class="u-hide">: </span><span class="c-bibliographic-information__value"><time datetime="2021-10-28">28 October 2021</time></span></p></li><li class="c-bibliographic-information__list-item"><p>Issue Date<span class="u-hide">: </span><span class="c-bibliographic-information__value"><time datetime="2022-09">September 2022</time></span></p></li><li class="c-bibliographic-information__list-item c-bibliographic-information__list-item--full-width"><p><abbr title="Digital Object Identifier">DOI</abbr><span class="u-hide">: </span><span class="c-bibliographic-information__value">https://doi.org/10.1007/s00439-021-02393-x</span></p></li></ul><div data-component="share-box"><div class="c-article-share-box u-display-none" hidden=""><h3 class="c-article__sub-heading">Share this article</h3><p class="c-article-share-box__description">Anyone you share the following link with will be able to read this content:</p><button class="js-get-share-url c-article-share-box__button" type="button" id="get-share-url" data-track="click" data-track-label="button" data-track-external="" data-track-action="get shareable link">Get shareable link</button><div class="js-no-share-url-container u-display-none" hidden=""><p class="js-c-article-share-box__no-sharelink-info c-article-share-box__no-sharelink-info">Sorry, a shareable link is not currently available for this article.</p></div><div class="js-share-url-container u-display-none" hidden=""><p class="js-share-url c-article-share-box__only-read-input" id="share-url" data-track="click" data-track-label="button" data-track-action="select share url"></p><button class="js-copy-share-url c-article-share-box__button--link-like" type="button" id="copy-share-url" data-track="click" data-track-label="button" data-track-action="copy share url" data-track-external="">Copy to clipboard</button></div><p class="js-c-article-share-box__additional-info c-article-share-box__additional-info"> Provided by the Springer Nature SharedIt content-sharing initiative </p></div></div><div data-component="article-info-list"></div></div></div></div></div></section> </div> </main> <div class="c-article-sidebar u-text-sm u-hide-print l-with-sidebar__sidebar" id="sidebar" data-container-type="reading-companion" data-track-component="reading companion"> <aside> <div class="app-card-service" data-test="article-checklist-banner"> <div> <a class="app-card-service__link" data-track="click_presubmission_checklist" data-track-context="article page top of reading companion" data-track-category="pre-submission-checklist" data-track-action="clicked article page checklist banner test 2 old version" data-track-label="link" href="https://beta.springernature.com/pre-submission?journalId=439" data-test="article-checklist-banner-link"> <span class="app-card-service__link-text">Use our pre-submission checklist</span> <svg class="app-card-service__link-icon" aria-hidden="true" focusable="false"><use xlink:href="#icon-eds-i-arrow-right-small"></use></svg> </a> <p class="app-card-service__description">Avoid common mistakes on your manuscript.</p> </div> <div class="app-card-service__icon-container"> <svg class="app-card-service__icon" aria-hidden="true" focusable="false"> <use xlink:href="#icon-eds-i-clipboard-check-medium"></use> </svg> </div> </div> <div data-test="collections"> <aside> <div class="c-article-associated-content__container"> <h2 class="c-article-associated-content__title u-h3 u-mb-24 u-visually-hidden">Associated Content</h2> <div class="c-article-associated-content__collection collection u-mb-24"> <p class="c-article-associated-content__collection-label u-sans-serif u-text-bold u-mb-8">Part of a collection:</p> <h3 class="c-article-associated-content__collection-title u-mt-0 u-h3 u-mb-8" itemprop="name headline"> <a href="/collections/fafcjhfhfd" data-track="click" data-track-action="view collection" data-track-label="link">Artificial Intelligence</a> </h3> </div> </div> </aside> <script> window.dataLayer = window.dataLayer || []; window.dataLayer[0] = window.dataLayer[0] || {}; window.dataLayer[0].content = window.dataLayer[0].content || {}; window.dataLayer[0].content.collections = 'fafcjhfhfd'; </script> </div> <div data-test="editorial-summary"> </div> <div class="c-reading-companion"> <div class="c-reading-companion__sticky" data-component="reading-companion-sticky" data-test="reading-companion-sticky"> <div class="c-reading-companion__panel c-reading-companion__sections c-reading-companion__panel--active" id="tabpanel-sections"> <div class="u-lazy-ad-wrapper u-mt-16 u-hide" data-component-mpu><div class="c-ad c-ad--300x250"> <div class="c-ad__inner"> <p class="c-ad__label">Advertisement</p> <div id="div-gpt-ad-MPU1" class="div-gpt-ad grade-c-hide" data-pa11y-ignore data-gpt data-gpt-unitpath="/270604982/springerlink/439/article" data-gpt-sizes="300x250" data-test="MPU1-ad" data-gpt-targeting="pos=MPU1;articleid=s00439-021-02393-x;"> </div> </div> </div> </div> </div> <div class="c-reading-companion__panel c-reading-companion__figures c-reading-companion__panel--full-width" id="tabpanel-figures"></div> <div class="c-reading-companion__panel c-reading-companion__references c-reading-companion__panel--full-width" id="tabpanel-references"></div> </div> </div> </aside> </div> </div> </article> <div class="app-elements"> <div class="eds-c-header__expander eds-c-header__expander--search" id="eds-c-header-popup-search"> <h2 class="eds-c-header__heading">Search</h2> <div class="u-container"> <search class="eds-c-header__search" role="search" aria-label="Search from the header"> <form method="GET" action="//link.springer.com/search" data-test="header-search" data-track="search" data-track-context="search from header" data-track-action="submit search form" data-track-category="unified header" data-track-label="form" > <label for="eds-c-header-search" class="eds-c-header__search-label">Search by keyword or author</label> <div class="eds-c-header__search-container"> <input id="eds-c-header-search" class="eds-c-header__search-input" autocomplete="off" name="query" type="search" value="" required> <button class="eds-c-header__search-button" type="submit"> <svg class="eds-c-header__icon" aria-hidden="true" focusable="false"> <use xlink:href="#icon-eds-i-search-medium"></use> </svg> <span class="u-visually-hidden">Search</span> </button> </div> </form> </search> </div> </div> <div class="eds-c-header__expander eds-c-header__expander--menu" id="eds-c-header-nav"> <h2 class="eds-c-header__heading">Navigation</h2> <ul class="eds-c-header__list"> <li class="eds-c-header__list-item"> <a class="eds-c-header__link" href="https://link.springer.com/journals/" data-track="nav_find_a_journal" data-track-context="unified header" data-track-action="click find a journal" data-track-category="unified header" data-track-label="link" > Find a journal </a> </li> <li class="eds-c-header__list-item"> <a class="eds-c-header__link" href="https://www.springernature.com/gp/authors" data-track="nav_how_to_publish" data-track-context="unified header" data-track-action="click publish with us link" data-track-category="unified header" data-track-label="link" > Publish with us </a> </li> <li class="eds-c-header__list-item"> <a class="eds-c-header__link" href="https://link.springernature.com/home/" data-track="nav_track_your_research" data-track-context="unified header" data-track-action="click track your research" data-track-category="unified header" data-track-label="link" > Track your research </a> </li> </ul> </div> <footer > <div class="eds-c-footer" > <div class="eds-c-footer__container"> <div class="eds-c-footer__grid eds-c-footer__group--separator"> <div class="eds-c-footer__group"> <h3 class="eds-c-footer__heading">Discover content</h3> <ul class="eds-c-footer__list"> <li class="eds-c-footer__item"><a class="eds-c-footer__link" href="https://link.springer.com/journals/a/1" data-track="nav_journals_a_z" data-track-action="journals a-z" data-track-context="unified footer" data-track-label="link">Journals A-Z</a></li> <li class="eds-c-footer__item"><a class="eds-c-footer__link" href="https://link.springer.com/books/a/1" data-track="nav_books_a_z" data-track-action="books a-z" data-track-context="unified footer" data-track-label="link">Books A-Z</a></li> </ul> </div> <div class="eds-c-footer__group"> <h3 class="eds-c-footer__heading">Publish with us</h3> <ul class="eds-c-footer__list"> <li class="eds-c-footer__item"><a class="eds-c-footer__link" href="https://link.springer.com/journals" data-track="nav_journal_finder" data-track-action="journal finder" data-track-context="unified footer" data-track-label="link">Journal finder</a></li> <li class="eds-c-footer__item"><a class="eds-c-footer__link" href="https://www.springernature.com/gp/authors" data-track="nav_publish_your_research" data-track-action="publish your research" data-track-context="unified footer" data-track-label="link">Publish your research</a></li> <li class="eds-c-footer__item"><a class="eds-c-footer__link" href="https://www.springernature.com/gp/open-research/about/the-fundamentals-of-open-access-and-open-research" data-track="nav_open_access_publishing" data-track-action="open access publishing" data-track-context="unified footer" data-track-label="link">Open access publishing</a></li> </ul> </div> <div class="eds-c-footer__group"> <h3 class="eds-c-footer__heading">Products and services</h3> <ul class="eds-c-footer__list"> <li class="eds-c-footer__item"><a class="eds-c-footer__link" href="https://www.springernature.com/gp/products" data-track="nav_our_products" data-track-action="our products" data-track-context="unified footer" data-track-label="link">Our products</a></li> <li class="eds-c-footer__item"><a class="eds-c-footer__link" href="https://www.springernature.com/gp/librarians" data-track="nav_librarians" data-track-action="librarians" data-track-context="unified footer" data-track-label="link">Librarians</a></li> <li class="eds-c-footer__item"><a class="eds-c-footer__link" href="https://www.springernature.com/gp/societies" data-track="nav_societies" data-track-action="societies" data-track-context="unified footer" data-track-label="link">Societies</a></li> <li class="eds-c-footer__item"><a class="eds-c-footer__link" href="https://www.springernature.com/gp/partners" data-track="nav_partners_and_advertisers" data-track-action="partners and advertisers" data-track-context="unified footer" data-track-label="link">Partners and advertisers</a></li> </ul> </div> <div class="eds-c-footer__group"> <h3 class="eds-c-footer__heading">Our imprints</h3> <ul class="eds-c-footer__list"> <li class="eds-c-footer__item"><a class="eds-c-footer__link" href="https://www.springer.com/" data-track="nav_imprint_Springer" data-track-action="Springer" data-track-context="unified footer" data-track-label="link">Springer</a></li> <li class="eds-c-footer__item"><a class="eds-c-footer__link" href="https://www.nature.com/" data-track="nav_imprint_Nature_Portfolio" data-track-action="Nature Portfolio" data-track-context="unified footer" data-track-label="link">Nature Portfolio</a></li> <li class="eds-c-footer__item"><a class="eds-c-footer__link" href="https://www.biomedcentral.com/" data-track="nav_imprint_BMC" data-track-action="BMC" data-track-context="unified footer" data-track-label="link">BMC</a></li> <li class="eds-c-footer__item"><a class="eds-c-footer__link" href="https://www.palgrave.com/" data-track="nav_imprint_Palgrave_Macmillan" data-track-action="Palgrave Macmillan" data-track-context="unified footer" data-track-label="link">Palgrave Macmillan</a></li> <li class="eds-c-footer__item"><a class="eds-c-footer__link" href="https://www.apress.com/" data-track="nav_imprint_Apress" data-track-action="Apress" data-track-context="unified footer" data-track-label="link">Apress</a></li> </ul> </div> </div> </div> <div class="eds-c-footer__container"> <nav aria-label="footer navigation"> <ul class="eds-c-footer__links"> <li class="eds-c-footer__item"> <button class="eds-c-footer__link" data-cc-action="preferences" data-track="dialog_manage_cookies" data-track-action="Manage cookies" data-track-context="unified footer" data-track-label="link"><span class="eds-c-footer__button-text">Your privacy choices/Manage cookies</span></button> </li> <li class="eds-c-footer__item"> <a class="eds-c-footer__link" href="https://www.springernature.com/gp/legal/ccpa" data-track="nav_california_privacy_statement" data-track-action="california privacy statement" data-track-context="unified footer" data-track-label="link">Your US state privacy rights</a> </li> <li class="eds-c-footer__item"> <a class="eds-c-footer__link" href="https://www.springernature.com/gp/info/accessibility" data-track="nav_accessibility_statement" data-track-action="accessibility statement" data-track-context="unified footer" data-track-label="link">Accessibility statement</a> </li> <li class="eds-c-footer__item"> <a class="eds-c-footer__link" href="https://link.springer.com/termsandconditions" data-track="nav_terms_and_conditions" data-track-action="terms and conditions" data-track-context="unified footer" data-track-label="link">Terms and conditions</a> </li> <li class="eds-c-footer__item"> <a class="eds-c-footer__link" href="https://link.springer.com/privacystatement" data-track="nav_privacy_policy" data-track-action="privacy policy" data-track-context="unified footer" data-track-label="link">Privacy policy</a> </li> <li class="eds-c-footer__item"> <a class="eds-c-footer__link" href="https://support.springernature.com/en/support/home" data-track="nav_help_and_support" data-track-action="help and support" data-track-context="unified footer" data-track-label="link">Help and support</a> </li> <li class="eds-c-footer__item"> <a class="eds-c-footer__link" href="https://support.springernature.com/en/support/solutions/articles/6000255911-subscription-cancellations" data-track-action="cancel contracts here">Cancel contracts here</a> </li> </ul> </nav> <div class="eds-c-footer__user"> <p class="eds-c-footer__user-info"> <span data-test="footer-user-ip">8.222.208.146</span> </p> <p class="eds-c-footer__user-info" data-test="footer-business-partners">Not affiliated</p> </div> <a href="https://www.springernature.com/" class="eds-c-footer__link"> <img src="/oscar-static/images/logo-springernature-white-19dd4ba190.svg" alt="Springer Nature" loading="lazy" width="200" height="20"/> </a> <p class="eds-c-footer__legal" data-test="copyright">© 2024 Springer Nature</p> </div> </div> </footer> </div> </body> </html>