CINXE.COM
An environment for sustainable research software in... | F1000Research
<!DOCTYPE html> <html class=""> <head> <title>An environment for sustainable research software in... | F1000Research</title> <meta charset="utf-8"> <!--<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>--> <!--<meta lang="$locale">--> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="google-site-verification" content="9-QVycOO2_ob3Z9QzRmXv2CF08A9oyYXqWyTiVdKPlU" /> <!-- This is commented out to fix display problems on mobile devices. We may use it again once we implement a responsive design that supports native device resolutions. <meta name="viewport" content="width=device-width"> --> <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1"> <link rel="alternate" title="Recent articles published in F1000Research" href="/rss" type="application/rss+xml"> <link rel="alternate" type="application/pdf" title="PDF" href="https://f1000research.com/articles/9-295/pdf"/> <link rel="canonical" href="https://f1000research.com/articles/9-295" /> <meta name="description" content="Read the original article in full on F1000Research: An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action" /> <meta name="og:title" content="F1000Research Article: An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action."> <meta name="og:description" content="Read the latest article version by Hartwig Anzt, Felix Bach, Stephan Druskat, Frank Löffler, Axel Loewe, Bernhard Y. Renard, Gunnar Seemann, Alexander Struck, Elke Achhammer, Piush Aggarwal, Franziska Appel, Michael Bader, Lutz Brusch, Christian Busse, Gerasimos Chourdakis, Piotr Wojciech Dabrowski, Peter Ebert, Bernd Flemisch, Sven Friedl, Bernadette Fritzsch, Maximilian D. Funk, Volker Gast, Florian Goth, Jean-Noël Grad, Jan Hegewald, Sibylle Hermann, Florian Hohmann, Stephan Janosch, Dominik Kutra, Jan Linxweiler, Thilo Muth, Wolfgang Peters-Kottig, Fabian Rack, Fabian H.C. Raters, Stephan Rave, Guido Reina, Malte Reißig, Timo Ropinski, Joerg Schaarschmidt, Heidi Seibold, Jan P. Thiele, Benjamin Uekermann, Stefan Unger, Rudolf Weeber, at F1000Research."> <meta name="og:image" content="/img/share/og_research.png"> <meta name="version-id" content="31504"> <meta name="article-id" content="23224"> <meta name="dc.title" content="An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action"> <meta name="dc.description" content="Research software has become a central asset in academic research. It optimizes existing and enables new research methods, implements and embeds research knowledge, and constitutes an essential research product in itself. Research software must be sustainable in order to understand, replicate, reproduce, and build upon existing research or conduct new research effectively. In other words, software must be available, discoverable, usable, and adaptable to new needs, both now and in the future. Research software therefore requires an environment that supports sustainability. Hence, a change is needed in the way research software development and maintenance are currently motivated, incentivized, funded, structurally and infrastructurally supported, and legally treated. Failing to do so will threaten the quality and validity of research. In this paper, we identify challenges for research software sustainability in Germany and beyond, in terms of motivation, selection, research software engineering personnel, funding, infrastructure, and legal aspects. Besides researchers, we specifically address political and academic decision-makers to increase awareness of the importance and needs of sustainable research software practices. In particular, we recommend strategies and measures to create an environment for sustainable research software, with the ultimate goal to ensure that software-driven research is valid, reproducible and sustainable, and that software is recognized as a first class citizen in research. This paper is the outcome of two workshops run in Germany in 2019, at deRSE19 - the first International Conference of Research Software Engineers in Germany - and a dedicated DFG-supported follow-up workshop in Berlin."> <meta name="dc.subject" content="Sustainable Software Development, Academic Software, Software Infrastructure, Software Training, Software Licensing, Research Software"> <meta name="dc.creator" content="Anzt, Hartwig"> <meta name="dc.creator" content="Bach, Felix"> <meta name="dc.creator" content="Druskat, Stephan"> <meta name="dc.creator" content="Löffler, Frank"> <meta name="dc.creator" content="Loewe, Axel"> <meta name="dc.creator" content="Renard, Bernhard Y."> <meta name="dc.creator" content="Seemann, Gunnar"> <meta name="dc.creator" content="Struck, Alexander"> <meta name="dc.creator" content="Achhammer, Elke"> <meta name="dc.creator" content="Aggarwal, Piush"> <meta name="dc.creator" content="Appel, Franziska"> <meta name="dc.creator" content="Bader, Michael"> <meta name="dc.creator" content="Brusch, Lutz"> <meta name="dc.creator" content="Busse, Christian"> <meta name="dc.creator" content="Chourdakis, Gerasimos"> <meta name="dc.creator" content="Dabrowski, Piotr Wojciech"> <meta name="dc.creator" content="Ebert, Peter"> <meta name="dc.creator" content="Flemisch, Bernd"> <meta name="dc.creator" content="Friedl, Sven"> <meta name="dc.creator" content="Fritzsch, Bernadette"> <meta name="dc.creator" content="Funk, Maximilian D."> <meta name="dc.creator" content="Gast, Volker"> <meta name="dc.creator" content="Goth, Florian"> <meta name="dc.creator" content="Grad, Jean-Noël"> <meta name="dc.creator" content="Hegewald, Jan"> <meta name="dc.creator" content="Hermann, Sibylle"> <meta name="dc.creator" content="Hohmann, Florian"> <meta name="dc.creator" content="Janosch, Stephan"> <meta name="dc.creator" content="Kutra, Dominik"> <meta name="dc.creator" content="Linxweiler, Jan"> <meta name="dc.creator" content="Muth, Thilo"> <meta name="dc.creator" content="Peters-Kottig, Wolfgang"> <meta name="dc.creator" content="Rack, Fabian"> <meta name="dc.creator" content="Raters, Fabian H.C."> <meta name="dc.creator" content="Rave, Stephan"> <meta name="dc.creator" content="Reina, Guido"> <meta name="dc.creator" content="Reißig, Malte"> <meta name="dc.creator" content="Ropinski, Timo"> <meta name="dc.creator" content="Schaarschmidt, Joerg"> <meta name="dc.creator" content="Seibold, Heidi"> <meta name="dc.creator" content="Thiele, Jan P."> <meta name="dc.creator" content="Uekermann, Benjamin"> <meta name="dc.creator" content="Unger, Stefan"> <meta name="dc.creator" content="Weeber, Rudolf"> <meta name="dc.date" content="2021/01/26"> <meta name="dc.identifier" content="doi:10.12688/f1000research.23224.2"> <meta name="dc.source" content="F1000Research 2021 9:295"> <meta name="dc.format" content="text/html"> <meta name="dc.language" content="en"> <meta name="dc.publisher" content="F1000 Research Limited"> <meta name="dc.rights" content="https://creativecommons.org/licenses/by/4.0/"> <meta name="dc.relation" content="doi:10.12688/f1000research.23224.1"> <meta name="dc.type" content="text"> <meta name="prism.keyword" content="Sustainable Software Development"> <meta name="prism.keyword" content="Academic Software"> <meta name="prism.keyword" content="Software Infrastructure"> <meta name="prism.keyword" content="Software Training"> <meta name="prism.keyword" content="Software Licensing"> <meta name="prism.keyword" content="Research Software"> <meta name="prism.publication.Name" content="F1000Research"> <meta name="prism.publicationDate" content="2021/01/26"> <meta name="prism.volume" content="9"> <meta name="prism.number" content="295"> <meta name="prism.versionIdentifier" content="2"> <meta name="prism.doi" content="10.12688/f1000research.23224.2"> <meta name="prism.url" content="https://f1000research.com/articles/9-295"> <meta name="citation_title" content="An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action"> <meta name="citation_abstract" content="Research software has become a central asset in academic research. It optimizes existing and enables new research methods, implements and embeds research knowledge, and constitutes an essential research product in itself. Research software must be sustainable in order to understand, replicate, reproduce, and build upon existing research or conduct new research effectively. In other words, software must be available, discoverable, usable, and adaptable to new needs, both now and in the future. Research software therefore requires an environment that supports sustainability. Hence, a change is needed in the way research software development and maintenance are currently motivated, incentivized, funded, structurally and infrastructurally supported, and legally treated. Failing to do so will threaten the quality and validity of research. In this paper, we identify challenges for research software sustainability in Germany and beyond, in terms of motivation, selection, research software engineering personnel, funding, infrastructure, and legal aspects. Besides researchers, we specifically address political and academic decision-makers to increase awareness of the importance and needs of sustainable research software practices. In particular, we recommend strategies and measures to create an environment for sustainable research software, with the ultimate goal to ensure that software-driven research is valid, reproducible and sustainable, and that software is recognized as a first class citizen in research. This paper is the outcome of two workshops run in Germany in 2019, at deRSE19 - the first International Conference of Research Software Engineers in Germany - and a dedicated DFG-supported follow-up workshop in Berlin."> <meta name="citation_description" content="Research software has become a central asset in academic research. It optimizes existing and enables new research methods, implements and embeds research knowledge, and constitutes an essential research product in itself. Research software must be sustainable in order to understand, replicate, reproduce, and build upon existing research or conduct new research effectively. In other words, software must be available, discoverable, usable, and adaptable to new needs, both now and in the future. Research software therefore requires an environment that supports sustainability. Hence, a change is needed in the way research software development and maintenance are currently motivated, incentivized, funded, structurally and infrastructurally supported, and legally treated. Failing to do so will threaten the quality and validity of research. In this paper, we identify challenges for research software sustainability in Germany and beyond, in terms of motivation, selection, research software engineering personnel, funding, infrastructure, and legal aspects. Besides researchers, we specifically address political and academic decision-makers to increase awareness of the importance and needs of sustainable research software practices. In particular, we recommend strategies and measures to create an environment for sustainable research software, with the ultimate goal to ensure that software-driven research is valid, reproducible and sustainable, and that software is recognized as a first class citizen in research. This paper is the outcome of two workshops run in Germany in 2019, at deRSE19 - the first International Conference of Research Software Engineers in Germany - and a dedicated DFG-supported follow-up workshop in Berlin."> <meta name="citation_keywords" content="Sustainable Software Development, Academic Software, Software Infrastructure, Software Training, Software Licensing, Research Software"> <meta name="citation_journal_title" content="F1000Research"> <meta name="citation_author" content="Hartwig Anzt"> <meta name="citation_author_institution" content="Karlsruhe Institute of Technology (KIT), Karlsruhe"> <meta name="citation_author_institution" content="University of Tennessee, Knoxville, TN"> <meta name="citation_author" content="Felix Bach"> <meta name="citation_author_institution" content="Karlsruhe Institute of Technology (KIT), Karlsruhe"> <meta name="citation_author" content="Stephan Druskat"> <meta name="citation_author_institution" content="Friedrich Schiller University, Jena"> <meta name="citation_author_institution" content="German Aerospace Center (DLR), Berlin"> <meta name="citation_author_institution" content="Humboldt-Universität zu Berlin, Berlin"> <meta name="citation_author" content="Frank Löffler"> <meta name="citation_author_institution" content="Friedrich Schiller University, Jena"> <meta name="citation_author_institution" content="Louisiana State University, Baton Rouge, LA"> <meta name="citation_author" content="Axel Loewe"> <meta name="citation_author_institution" content="Karlsruhe Institute of Technology (KIT), Karlsruhe"> <meta name="citation_author" content="Bernhard Y. Renard"> <meta name="citation_author_institution" content="Hasso Plattner Institute, Potsdam"> <meta name="citation_author" content="Gunnar Seemann"> <meta name="citation_author_institution" content="University Heart Centre Freiburg Bad Krozingen, Freiburg"> <meta name="citation_author" content="Alexander Struck"> <meta name="citation_author_institution" content="Humboldt-Universität zu Berlin, Berlin"> <meta name="citation_author" content="Elke Achhammer"> <meta name="citation_author_institution" content="Technische Universität München, München"> <meta name="citation_author" content="Piush Aggarwal"> <meta name="citation_author_institution" content="Universität Duisburg-Essen, Duisburg"> <meta name="citation_author" content="Franziska Appel"> <meta name="citation_author_institution" content="Leibniz Institute of Agricultural Development in Transition Economies (IAMO), Halle (Saale)"> <meta name="citation_author" content="Michael Bader"> <meta name="citation_author_institution" content="Technische Universität München, München"> <meta name="citation_author" content="Lutz Brusch"> <meta name="citation_author_institution" content="Technische Universität Dresden, Dresden"> <meta name="citation_author" content="Christian Busse"> <meta name="citation_author_institution" content="Deutsches Krebsforschungszentrum, Heidelberg"> <meta name="citation_author" content="Gerasimos Chourdakis"> <meta name="citation_author_institution" content="Technische Universität München, München"> <meta name="citation_author" content="Piotr Wojciech Dabrowski"> <meta name="citation_author_institution" content="Hochschule für Technik und Wirtschaft Berlin, Berlin"> <meta name="citation_author" content="Peter Ebert"> <meta name="citation_author_institution" content="Saarland Informatics Campus, Saarbrücken"> <meta name="citation_author" content="Bernd Flemisch"> <meta name="citation_author_institution" content="University of Stuttgart, Stuttgart"> <meta name="citation_author" content="Sven Friedl"> <meta name="citation_author_institution" content="Berlin Institute of Health, Berlin"> <meta name="citation_author" content="Bernadette Fritzsch"> <meta name="citation_author_institution" content="Alfred Wegener Institute, Bremerhaven"> <meta name="citation_author" content="Maximilian D. Funk"> <meta name="citation_author_institution" content="Max-Planck-Gesellschaft e.V., München"> <meta name="citation_author" content="Volker Gast"> <meta name="citation_author_institution" content="Friedrich Schiller University, Jena"> <meta name="citation_author" content="Florian Goth"> <meta name="citation_author_institution" content="Universität Würzburg, Würzburg"> <meta name="citation_author" content="Jean-Noël Grad"> <meta name="citation_author_institution" content="University of Stuttgart, Stuttgart"> <meta name="citation_author" content="Jan Hegewald"> <meta name="citation_author_institution" content="Alfred Wegener Institute, Bremerhaven"> <meta name="citation_author" content="Sibylle Hermann"> <meta name="citation_author_institution" content="University of Stuttgart, Stuttgart"> <meta name="citation_author" content="Florian Hohmann"> <meta name="citation_author_institution" content="Universität Bremen, Bremen"> <meta name="citation_author" content="Stephan Janosch"> <meta name="citation_author_institution" content="Max Planck Institute of Molecular Cell Biology and Genetics, Dresden"> <meta name="citation_author" content="Dominik Kutra"> <meta name="citation_author_institution" content="European Molecular Biology Laboratory, Heidelberg"> <meta name="citation_author" content="Jan Linxweiler"> <meta name="citation_author_institution" content="Technische Universität Braunschweig, Braunschweig"> <meta name="citation_author" content="Thilo Muth"> <meta name="citation_author_institution" content="Federal Institute for Materials Research and Testing, Berlin"> <meta name="citation_author" content="Wolfgang Peters-Kottig"> <meta name="citation_author_institution" content="Konrad-Zuse-Zentrum für Informationstechnik Berlin (ZIB), Berlin"> <meta name="citation_author" content="Fabian Rack"> <meta name="citation_author_institution" content="FIZ Karlsruhe - Leibniz Institute for Information Infrastructure, Karlsruhe"> <meta name="citation_author" content="Fabian H.C. Raters"> <meta name="citation_author_institution" content="University of Goettingen, Göttingen"> <meta name="citation_author" content="Stephan Rave"> <meta name="citation_author_institution" content="University of Münster, Münster"> <meta name="citation_author" content="Guido Reina"> <meta name="citation_author_institution" content="University of Stuttgart, Stuttgart"> <meta name="citation_author" content="Malte Reißig"> <meta name="citation_author_institution" content="Institute for Advanced Sustainability Studies, Potsdam"> <meta name="citation_author" content="Timo Ropinski"> <meta name="citation_author_institution" content="Ulm University, Ulm"> <meta name="citation_author_institution" content="Linköping University, Linköping"> <meta name="citation_author" content="Joerg Schaarschmidt"> <meta name="citation_author_institution" content="Karlsruhe Institute of Technology (KIT), Karlsruhe"> <meta name="citation_author" content="Heidi Seibold"> <meta name="citation_author_institution" content="Ludwig Maximilian University of Munich, München"> <meta name="citation_author" content="Jan P. Thiele"> <meta name="citation_author_institution" content="Leibniz University Hannover, Hannover"> <meta name="citation_author" content="Benjamin Uekermann"> <meta name="citation_author_institution" content="Eindhoven University of Technology, Eindhoven"> <meta name="citation_author" content="Stefan Unger"> <meta name="citation_author_institution" content="Julius Kühn-Institut (JKI), Quedlinburg"> <meta name="citation_author" content="Rudolf Weeber"> <meta name="citation_author_institution" content="University of Stuttgart, Stuttgart"> <meta name="citation_publication_date" content="2021/01/26"> <meta name="citation_volume" content="9"> <meta name="citation_publication_number" content="295"> <meta name="citation_version_number" content="2"> <meta name="citation_doi" content="10.12688/f1000research.23224.2"> <meta name="citation_firstpage" content="295"> <meta name="citation_pdf_url" content="https://f1000research.com/articles/9-295/pdf"> <meta name="citation_xml_url" content="https://f1000research.com/articles/9-295/xml"> <link href="/img/favicon-research.ico" rel="shortcut icon" type="image/ico"> <link href="/img/favicon-research.ico" rel="icon" type="image/ico"> <link rel="stylesheet" href="/1740487202716/css/mdl/material-design-lite-research.css" type="text/css" media="all" /> <link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet"> <link href="https://fonts.googleapis.com/css?family=Roboto:100,100i,300,300i,400,400i,500,500i,700,700i,900,900i" rel="stylesheet"> <link rel="stylesheet" href="/1740487202716/css/Research.css" type="text/css" media="all" /> <link rel="stylesheet" href="/css/F1000ResearchFontIcons/F1000ResearchFontIcons.css" type="text/css" media="all" /> <link rel="stylesheet" href="/css/F1000ResearchFontIcons/animation.css" type="text/css" media="all" /> <!--[if IE 7]><link rel="stylesheet" href="/css/F1000ResearchFontIcons/F1000ResearchFontIcons-ie7.css" media="all" /><![endif]--> <!-- Transcend cookies consent for F1000Research --> <script type="text/javascript">"use strict";function _typeof(t){return(_typeof="function"==typeof Symbol&&"symbol"==typeof Symbol.iterator?function(t){return typeof t}:function(t){return t&&"function"==typeof Symbol&&t.constructor===Symbol&&t!==Symbol.prototype?"symbol":typeof t})(t)}!function(){var t=function(){var t,e,o=[],n=window,r=n;for(;r;){try{if(r.frames.__tcfapiLocator){t=r;break}}catch(t){}if(r===n.top)break;r=r.parent}t||(!function t(){var e=n.document,o=!!n.frames.__tcfapiLocator;if(!o)if(e.body){var r=e.createElement("iframe");r.style.cssText="display:none",r.name="__tcfapiLocator",e.body.appendChild(r)}else setTimeout(t,5);return!o}(),n.__tcfapi=function(){for(var t=arguments.length,n=new Array(t),r=0;r<t;r++)n[r]=arguments[r];if(!n.length)return o;"setGdprApplies"===n[0]?n.length>3&&2===parseInt(n[1],10)&&"boolean"==typeof n[3]&&(e=n[3],"function"==typeof n[2]&&n[2]("set",!0)):"ping"===n[0]?"function"==typeof n[2]&&n[2]({gdprApplies:e,cmpLoaded:!1,cmpStatus:"stub"}):o.push(n)},n.addEventListener("message",(function(t){var e="string"==typeof t.data,o={};if(e)try{o=JSON.parse(t.data)}catch(t){}else o=t.data;var n="object"===_typeof(o)&&null!==o?o.__tcfapiCall:null;n&&window.__tcfapi(n.command,n.version,(function(o,r){var a={__tcfapiReturn:{returnValue:o,success:r,callId:n.callId}};t&&t.source&&t.source.postMessage&&t.source.postMessage(e?JSON.stringify(a):a,"*")}),n.parameter)}),!1))};"undefined"!=typeof module?module.exports=t:t()}();</script> <script data-cfasync="false" data-tcf-ui-dismissed-view-state="Collapsed" src="https://transcend-cdn.com/cm/ef49a3f1-d8c1-47d6-88fc-50e41130631f/airgap.js"></script> <script> dataLayer = []; </script> <!-- Google Tag Manager --> <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0], j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src= 'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f); })(window,document,'script','dataLayer','GTM-PCBS9JK');</script> <!-- End Google Tag Manager --> <script type="text/javascript"> ;window.NREUM||(NREUM={});NREUM.init={distributed_tracing:{enabled:true},privacy:{cookies_enabled:true},ajax:{deny_list:["bam.nr-data.net"]}}; ;NREUM.loader_config={accountID:"438030",trustKey:"438030",agentID:"772317073",licenseKey:"97f8f67f26",applicationID:"772317073"} ;NREUM.info={beacon:"bam.nr-data.net",errorBeacon:"bam.nr-data.net",licenseKey:"97f8f67f26",applicationID:"772317073",sa:1} ;/*! For license information please see nr-loader-spa-1.236.0.min.js.LICENSE.txt */ (()=>{"use strict";var e,t,r={5763:(e,t,r)=>{r.d(t,{P_:()=>l,Mt:()=>g,C5:()=>s,DL:()=>v,OP:()=>T,lF:()=>D,Yu:()=>y,Dg:()=>h,CX:()=>c,GE:()=>b,sU:()=>_});var n=r(8632),i=r(9567);const o={beacon:n.ce.beacon,errorBeacon:n.ce.errorBeacon,licenseKey:void 0,applicationID:void 0,sa:void 0,queueTime:void 0,applicationTime:void 0,ttGuid:void 0,user:void 0,account:void 0,product:void 0,extra:void 0,jsAttributes:{},userAttributes:void 0,atts:void 0,transactionName:void 0,tNamePlain:void 0},a={};function s(e){if(!e)throw new Error("All info objects require an agent identifier!");if(!a[e])throw new Error("Info for ".concat(e," was never set"));return a[e]}function c(e,t){if(!e)throw new Error("All info objects require an agent identifier!");a[e]=(0,i.D)(t,o),(0,n.Qy)(e,a[e],"info")}var u=r(7056);const d=()=>{const e={blockSelector:"[data-nr-block]",maskInputOptions:{password:!0}};return{allow_bfcache:!0,privacy:{cookies_enabled:!0},ajax:{deny_list:void 0,enabled:!0,harvestTimeSeconds:10},distributed_tracing:{enabled:void 0,exclude_newrelic_header:void 0,cors_use_newrelic_header:void 0,cors_use_tracecontext_headers:void 0,allowed_origins:void 0},session:{domain:void 0,expiresMs:u.oD,inactiveMs:u.Hb},ssl:void 0,obfuscate:void 0,jserrors:{enabled:!0,harvestTimeSeconds:10},metrics:{enabled:!0},page_action:{enabled:!0,harvestTimeSeconds:30},page_view_event:{enabled:!0},page_view_timing:{enabled:!0,harvestTimeSeconds:30,long_task:!1},session_trace:{enabled:!0,harvestTimeSeconds:10},harvest:{tooManyRequestsDelay:60},session_replay:{enabled:!1,harvestTimeSeconds:60,sampleRate:.1,errorSampleRate:.1,maskTextSelector:"*",maskAllInputs:!0,get blockClass(){return"nr-block"},get ignoreClass(){return"nr-ignore"},get maskTextClass(){return"nr-mask"},get blockSelector(){return e.blockSelector},set blockSelector(t){e.blockSelector+=",".concat(t)},get maskInputOptions(){return e.maskInputOptions},set maskInputOptions(t){e.maskInputOptions={...t,password:!0}}},spa:{enabled:!0,harvestTimeSeconds:10}}},f={};function l(e){if(!e)throw new Error("All configuration objects require an agent identifier!");if(!f[e])throw new Error("Configuration for ".concat(e," was never set"));return f[e]}function h(e,t){if(!e)throw new Error("All configuration objects require an agent identifier!");f[e]=(0,i.D)(t,d()),(0,n.Qy)(e,f[e],"config")}function g(e,t){if(!e)throw new Error("All configuration objects require an agent identifier!");var r=l(e);if(r){for(var n=t.split("."),i=0;i<n.length-1;i++)if("object"!=typeof(r=r[n[i]]))return;r=r[n[n.length-1]]}return r}const p={accountID:void 0,trustKey:void 0,agentID:void 0,licenseKey:void 0,applicationID:void 0,xpid:void 0},m={};function v(e){if(!e)throw new Error("All loader-config objects require an agent identifier!");if(!m[e])throw new Error("LoaderConfig for ".concat(e," was never set"));return m[e]}function b(e,t){if(!e)throw new Error("All loader-config objects require an agent identifier!");m[e]=(0,i.D)(t,p),(0,n.Qy)(e,m[e],"loader_config")}const y=(0,n.mF)().o;var w=r(385),x=r(6818);const A={buildEnv:x.Re,bytesSent:{},queryBytesSent:{},customTransaction:void 0,disabled:!1,distMethod:x.gF,isolatedBacklog:!1,loaderType:void 0,maxBytes:3e4,offset:Math.floor(w._A?.performance?.timeOrigin||w._A?.performance?.timing?.navigationStart||Date.now()),onerror:void 0,origin:""+w._A.location,ptid:void 0,releaseIds:{},session:void 0,xhrWrappable:"function"==typeof w._A.XMLHttpRequest?.prototype?.addEventListener,version:x.q4},E={};function T(e){if(!e)throw new Error("All runtime objects require an agent identifier!");if(!E[e])throw new Error("Runtime for ".concat(e," was never set"));return E[e]}function _(e,t){if(!e)throw new Error("All runtime objects require an agent identifier!");E[e]=(0,i.D)(t,A),(0,n.Qy)(e,E[e],"runtime")}function D(e){return function(e){try{const t=s(e);return!!t.licenseKey&&!!t.errorBeacon&&!!t.applicationID}catch(e){return!1}}(e)}},9567:(e,t,r)=>{r.d(t,{D:()=>i});var n=r(50);function i(e,t){try{if(!e||"object"!=typeof e)return(0,n.Z)("Setting a Configurable requires an object as input");if(!t||"object"!=typeof t)return(0,n.Z)("Setting a Configurable requires a model to set its initial properties");const r=Object.create(Object.getPrototypeOf(t),Object.getOwnPropertyDescriptors(t)),o=0===Object.keys(r).length?e:r;for(let a in o)if(void 0!==e[a])try{"object"==typeof e[a]&&"object"==typeof t[a]?r[a]=i(e[a],t[a]):r[a]=e[a]}catch(e){(0,n.Z)("An error occurred while setting a property of a Configurable",e)}return r}catch(e){(0,n.Z)("An error occured while setting a Configurable",e)}}},6818:(e,t,r)=>{r.d(t,{Re:()=>i,gF:()=>o,q4:()=>n});const n="1.236.0",i="PROD",o="CDN"},385:(e,t,r)=>{r.d(t,{FN:()=>a,IF:()=>u,Nk:()=>f,Tt:()=>s,_A:()=>o,il:()=>n,pL:()=>c,v6:()=>i,w1:()=>d});const n="undefined"!=typeof window&&!!window.document,i="undefined"!=typeof WorkerGlobalScope&&("undefined"!=typeof self&&self instanceof WorkerGlobalScope&&self.navigator instanceof WorkerNavigator||"undefined"!=typeof globalThis&&globalThis instanceof WorkerGlobalScope&&globalThis.navigator instanceof WorkerNavigator),o=n?window:"undefined"!=typeof WorkerGlobalScope&&("undefined"!=typeof self&&self instanceof WorkerGlobalScope&&self||"undefined"!=typeof globalThis&&globalThis instanceof WorkerGlobalScope&&globalThis),a=""+o?.location,s=/iPad|iPhone|iPod/.test(navigator.userAgent),c=s&&"undefined"==typeof SharedWorker,u=(()=>{const e=navigator.userAgent.match(/Firefox[/\s](\d+\.\d+)/);return Array.isArray(e)&&e.length>=2?+e[1]:0})(),d=Boolean(n&&window.document.documentMode),f=!!navigator.sendBeacon},1117:(e,t,r)=>{r.d(t,{w:()=>o});var n=r(50);const i={agentIdentifier:"",ee:void 0};class o{constructor(e){try{if("object"!=typeof e)return(0,n.Z)("shared context requires an object as input");this.sharedContext={},Object.assign(this.sharedContext,i),Object.entries(e).forEach((e=>{let[t,r]=e;Object.keys(i).includes(t)&&(this.sharedContext[t]=r)}))}catch(e){(0,n.Z)("An error occured while setting SharedContext",e)}}}},8e3:(e,t,r)=>{r.d(t,{L:()=>d,R:()=>c});var n=r(2177),i=r(1284),o=r(4322),a=r(3325);const s={};function c(e,t){const r={staged:!1,priority:a.p[t]||0};u(e),s[e].get(t)||s[e].set(t,r)}function u(e){e&&(s[e]||(s[e]=new Map))}function d(){let e=arguments.length>0&&void 0!==arguments[0]?arguments[0]:"",t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:"feature";if(u(e),!e||!s[e].get(t))return a(t);s[e].get(t).staged=!0;const r=[...s[e]];function a(t){const r=e?n.ee.get(e):n.ee,a=o.X.handlers;if(r.backlog&&a){var s=r.backlog[t],c=a[t];if(c){for(var u=0;s&&u<s.length;++u)f(s[u],c);(0,i.D)(c,(function(e,t){(0,i.D)(t,(function(t,r){r[0].on(e,r[1])}))}))}delete a[t],r.backlog[t]=null,r.emit("drain-"+t,[])}}r.every((e=>{let[t,r]=e;return r.staged}))&&(r.sort(((e,t)=>e[1].priority-t[1].priority)),r.forEach((e=>{let[t]=e;a(t)})))}function f(e,t){var r=e[1];(0,i.D)(t[r],(function(t,r){var n=e[0];if(r[0]===n){var i=r[1],o=e[3],a=e[2];i.apply(o,a)}}))}},2177:(e,t,r)=>{r.d(t,{c:()=>f,ee:()=>u});var n=r(8632),i=r(2210),o=r(1284),a=r(5763),s="nr@context";let c=(0,n.fP)();var u;function d(){}function f(e){return(0,i.X)(e,s,l)}function l(){return new d}function h(){u.aborted=!0,u.backlog={}}c.ee?u=c.ee:(u=function e(t,r){var n={},c={},f={},g=!1;try{g=16===r.length&&(0,a.OP)(r).isolatedBacklog}catch(e){}var p={on:b,addEventListener:b,removeEventListener:y,emit:v,get:x,listeners:w,context:m,buffer:A,abort:h,aborted:!1,isBuffering:E,debugId:r,backlog:g?{}:t&&"object"==typeof t.backlog?t.backlog:{}};return p;function m(e){return e&&e instanceof d?e:e?(0,i.X)(e,s,l):l()}function v(e,r,n,i,o){if(!1!==o&&(o=!0),!u.aborted||i){t&&o&&t.emit(e,r,n);for(var a=m(n),s=w(e),d=s.length,f=0;f<d;f++)s[f].apply(a,r);var l=T()[c[e]];return l&&l.push([p,e,r,a]),a}}function b(e,t){n[e]=w(e).concat(t)}function y(e,t){var r=n[e];if(r)for(var i=0;i<r.length;i++)r[i]===t&&r.splice(i,1)}function w(e){return n[e]||[]}function x(t){return f[t]=f[t]||e(p,t)}function A(e,t){var r=T();p.aborted||(0,o.D)(e,(function(e,n){t=t||"feature",c[n]=t,t in r||(r[t]=[])}))}function E(e){return!!T()[c[e]]}function T(){return p.backlog}}(void 0,"globalEE"),c.ee=u)},5546:(e,t,r)=>{r.d(t,{E:()=>n,p:()=>i});var n=r(2177).ee.get("handle");function i(e,t,r,i,o){o?(o.buffer([e],i),o.emit(e,t,r)):(n.buffer([e],i),n.emit(e,t,r))}},4322:(e,t,r)=>{r.d(t,{X:()=>o});var n=r(5546);o.on=a;var i=o.handlers={};function o(e,t,r,o){a(o||n.E,i,e,t,r)}function a(e,t,r,i,o){o||(o="feature"),e||(e=n.E);var a=t[o]=t[o]||{};(a[r]=a[r]||[]).push([e,i])}},3239:(e,t,r)=>{r.d(t,{bP:()=>s,iz:()=>c,m$:()=>a});var n=r(385);let i=!1,o=!1;try{const e={get passive(){return i=!0,!1},get signal(){return o=!0,!1}};n._A.addEventListener("test",null,e),n._A.removeEventListener("test",null,e)}catch(e){}function a(e,t){return i||o?{capture:!!e,passive:i,signal:t}:!!e}function s(e,t){let r=arguments.length>2&&void 0!==arguments[2]&&arguments[2],n=arguments.length>3?arguments[3]:void 0;window.addEventListener(e,t,a(r,n))}function c(e,t){let r=arguments.length>2&&void 0!==arguments[2]&&arguments[2],n=arguments.length>3?arguments[3]:void 0;document.addEventListener(e,t,a(r,n))}},4402:(e,t,r)=>{r.d(t,{Ht:()=>u,M:()=>c,Rl:()=>a,ky:()=>s});var n=r(385);const i="xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx";function o(e,t){return e?15&e[t]:16*Math.random()|0}function a(){const e=n._A?.crypto||n._A?.msCrypto;let t,r=0;return e&&e.getRandomValues&&(t=e.getRandomValues(new Uint8Array(31))),i.split("").map((e=>"x"===e?o(t,++r).toString(16):"y"===e?(3&o()|8).toString(16):e)).join("")}function s(e){const t=n._A?.crypto||n._A?.msCrypto;let r,i=0;t&&t.getRandomValues&&(r=t.getRandomValues(new Uint8Array(31)));const a=[];for(var s=0;s<e;s++)a.push(o(r,++i).toString(16));return a.join("")}function c(){return s(16)}function u(){return s(32)}},7056:(e,t,r)=>{r.d(t,{Bq:()=>n,Hb:()=>o,oD:()=>i});const n="NRBA",i=144e5,o=18e5},7894:(e,t,r)=>{function n(){return Math.round(performance.now())}r.d(t,{z:()=>n})},7243:(e,t,r)=>{r.d(t,{e:()=>o});var n=r(385),i={};function o(e){if(e in i)return i[e];if(0===(e||"").indexOf("data:"))return{protocol:"data"};let t;var r=n._A?.location,o={};if(n.il)t=document.createElement("a"),t.href=e;else try{t=new URL(e,r.href)}catch(e){return o}o.port=t.port;var a=t.href.split("://");!o.port&&a[1]&&(o.port=a[1].split("/")[0].split("@").pop().split(":")[1]),o.port&&"0"!==o.port||(o.port="https"===a[0]?"443":"80"),o.hostname=t.hostname||r.hostname,o.pathname=t.pathname,o.protocol=a[0],"/"!==o.pathname.charAt(0)&&(o.pathname="/"+o.pathname);var s=!t.protocol||":"===t.protocol||t.protocol===r.protocol,c=t.hostname===r.hostname&&t.port===r.port;return o.sameOrigin=s&&(!t.hostname||c),"/"===o.pathname&&(i[e]=o),o}},50:(e,t,r)=>{function n(e,t){"function"==typeof console.warn&&(console.warn("New Relic: ".concat(e)),t&&console.warn(t))}r.d(t,{Z:()=>n})},2587:(e,t,r)=>{r.d(t,{N:()=>c,T:()=>u});var n=r(2177),i=r(5546),o=r(8e3),a=r(3325);const s={stn:[a.D.sessionTrace],err:[a.D.jserrors,a.D.metrics],ins:[a.D.pageAction],spa:[a.D.spa],sr:[a.D.sessionReplay,a.D.sessionTrace]};function c(e,t){const r=n.ee.get(t);e&&"object"==typeof e&&(Object.entries(e).forEach((e=>{let[t,n]=e;void 0===u[t]&&(s[t]?s[t].forEach((e=>{n?(0,i.p)("feat-"+t,[],void 0,e,r):(0,i.p)("block-"+t,[],void 0,e,r),(0,i.p)("rumresp-"+t,[Boolean(n)],void 0,e,r)})):n&&(0,i.p)("feat-"+t,[],void 0,void 0,r),u[t]=Boolean(n))})),Object.keys(s).forEach((e=>{void 0===u[e]&&(s[e]?.forEach((t=>(0,i.p)("rumresp-"+e,[!1],void 0,t,r))),u[e]=!1)})),(0,o.L)(t,a.D.pageViewEvent))}const u={}},2210:(e,t,r)=>{r.d(t,{X:()=>i});var n=Object.prototype.hasOwnProperty;function i(e,t,r){if(n.call(e,t))return e[t];var i=r();if(Object.defineProperty&&Object.keys)try{return Object.defineProperty(e,t,{value:i,writable:!0,enumerable:!1}),i}catch(e){}return e[t]=i,i}},1284:(e,t,r)=>{r.d(t,{D:()=>n});const n=(e,t)=>Object.entries(e||{}).map((e=>{let[r,n]=e;return t(r,n)}))},4351:(e,t,r)=>{r.d(t,{P:()=>o});var n=r(2177);const i=()=>{const e=new WeakSet;return(t,r)=>{if("object"==typeof r&&null!==r){if(e.has(r))return;e.add(r)}return r}};function o(e){try{return JSON.stringify(e,i())}catch(e){try{n.ee.emit("internal-error",[e])}catch(e){}}}},3960:(e,t,r)=>{r.d(t,{K:()=>a,b:()=>o});var n=r(3239);function i(){return"undefined"==typeof document||"complete"===document.readyState}function o(e,t){if(i())return e();(0,n.bP)("load",e,t)}function a(e){if(i())return e();(0,n.iz)("DOMContentLoaded",e)}},8632:(e,t,r)=>{r.d(t,{EZ:()=>u,Qy:()=>c,ce:()=>o,fP:()=>a,gG:()=>d,mF:()=>s});var n=r(7894),i=r(385);const o={beacon:"bam.nr-data.net",errorBeacon:"bam.nr-data.net"};function a(){return i._A.NREUM||(i._A.NREUM={}),void 0===i._A.newrelic&&(i._A.newrelic=i._A.NREUM),i._A.NREUM}function s(){let e=a();return e.o||(e.o={ST:i._A.setTimeout,SI:i._A.setImmediate,CT:i._A.clearTimeout,XHR:i._A.XMLHttpRequest,REQ:i._A.Request,EV:i._A.Event,PR:i._A.Promise,MO:i._A.MutationObserver,FETCH:i._A.fetch}),e}function c(e,t,r){let i=a();const o=i.initializedAgents||{},s=o[e]||{};return Object.keys(s).length||(s.initializedAt={ms:(0,n.z)(),date:new Date}),i.initializedAgents={...o,[e]:{...s,[r]:t}},i}function u(e,t){a()[e]=t}function d(){return function(){let e=a();const t=e.info||{};e.info={beacon:o.beacon,errorBeacon:o.errorBeacon,...t}}(),function(){let e=a();const t=e.init||{};e.init={...t}}(),s(),function(){let e=a();const t=e.loader_config||{};e.loader_config={...t}}(),a()}},7956:(e,t,r)=>{r.d(t,{N:()=>i});var n=r(3239);function i(e){let t=arguments.length>1&&void 0!==arguments[1]&&arguments[1],r=arguments.length>2?arguments[2]:void 0,i=arguments.length>3?arguments[3]:void 0;return void(0,n.iz)("visibilitychange",(function(){if(t)return void("hidden"==document.visibilityState&&e());e(document.visibilityState)}),r,i)}},1214:(e,t,r)=>{r.d(t,{em:()=>v,u5:()=>N,QU:()=>S,_L:()=>I,Gm:()=>L,Lg:()=>M,gy:()=>U,BV:()=>Q,Kf:()=>ee});var n=r(2177);const i="nr@original";var o=Object.prototype.hasOwnProperty,a=!1;function s(e,t){return e||(e=n.ee),r.inPlace=function(e,t,n,i,o){n||(n="");var a,s,c,u="-"===n.charAt(0);for(c=0;c<t.length;c++)d(a=e[s=t[c]])||(e[s]=r(a,u?s+n:n,i,s,o))},r.flag=i,r;function r(t,r,n,a,s){return d(t)?t:(r||(r=""),nrWrapper[i]=t,u(t,nrWrapper,e),nrWrapper);function nrWrapper(){var i,u,d,f;try{u=this,i=[...arguments],d="function"==typeof n?n(i,u):n||{}}catch(t){c([t,"",[i,u,a],d],e)}o(r+"start",[i,u,a],d,s);try{return f=t.apply(u,i)}catch(e){throw o(r+"err",[i,u,e],d,s),e}finally{o(r+"end",[i,u,f],d,s)}}}function o(r,n,i,o){if(!a||t){var s=a;a=!0;try{e.emit(r,n,i,t,o)}catch(t){c([t,r,n,i],e)}a=s}}}function c(e,t){t||(t=n.ee);try{t.emit("internal-error",e)}catch(e){}}function u(e,t,r){if(Object.defineProperty&&Object.keys)try{return Object.keys(e).forEach((function(r){Object.defineProperty(t,r,{get:function(){return e[r]},set:function(t){return e[r]=t,t}})})),t}catch(e){c([e],r)}for(var n in e)o.call(e,n)&&(t[n]=e[n]);return t}function d(e){return!(e&&e instanceof Function&&e.apply&&!e[i])}var f=r(2210),l=r(385);const h={},g=XMLHttpRequest,p="addEventListener",m="removeEventListener";function v(e){var t=function(e){return(e||n.ee).get("events")}(e);if(h[t.debugId]++)return t;h[t.debugId]=1;var r=s(t,!0);function i(e){r.inPlace(e,[p,m],"-",o)}function o(e,t){return e[1]}return"getPrototypeOf"in Object&&(l.il&&b(document,i),b(l._A,i),b(g.prototype,i)),t.on(p+"-start",(function(e,t){var n=e[1];if(null!==n&&("function"==typeof n||"object"==typeof n)){var i=(0,f.X)(n,"nr@wrapped",(function(){var e={object:function(){if("function"!=typeof n.handleEvent)return;return n.handleEvent.apply(n,arguments)},function:n}[typeof n];return e?r(e,"fn-",null,e.name||"anonymous"):n}));this.wrapped=e[1]=i}})),t.on(m+"-start",(function(e){e[1]=this.wrapped||e[1]})),t}function b(e,t){let r=e;for(;"object"==typeof r&&!Object.prototype.hasOwnProperty.call(r,p);)r=Object.getPrototypeOf(r);for(var n=arguments.length,i=new Array(n>2?n-2:0),o=2;o<n;o++)i[o-2]=arguments[o];r&&t(r,...i)}var y="fetch-",w=y+"body-",x=["arrayBuffer","blob","json","text","formData"],A=l._A.Request,E=l._A.Response,T="prototype",_="nr@context";const D={};function N(e){const t=function(e){return(e||n.ee).get("fetch")}(e);if(!(A&&E&&l._A.fetch))return t;if(D[t.debugId]++)return t;function r(e,r,n){var i=e[r];"function"==typeof i&&(e[r]=function(){var e,r=[...arguments],o={};t.emit(n+"before-start",[r],o),o[_]&&o[_].dt&&(e=o[_].dt);var a=i.apply(this,r);return t.emit(n+"start",[r,e],a),a.then((function(e){return t.emit(n+"end",[null,e],a),e}),(function(e){throw t.emit(n+"end",[e],a),e}))})}return D[t.debugId]=1,x.forEach((e=>{r(A[T],e,w),r(E[T],e,w)})),r(l._A,"fetch",y),t.on(y+"end",(function(e,r){var n=this;if(r){var i=r.headers.get("content-length");null!==i&&(n.rxSize=i),t.emit(y+"done",[null,r],n)}else t.emit(y+"done",[e],n)})),t}const O={},j=["pushState","replaceState"];function S(e){const t=function(e){return(e||n.ee).get("history")}(e);return!l.il||O[t.debugId]++||(O[t.debugId]=1,s(t).inPlace(window.history,j,"-")),t}var P=r(3239);const C={},R=["appendChild","insertBefore","replaceChild"];function I(e){const t=function(e){return(e||n.ee).get("jsonp")}(e);if(!l.il||C[t.debugId])return t;C[t.debugId]=!0;var r=s(t),i=/[?&](?:callback|cb)=([^&#]+)/,o=/(.*)\.([^.]+)/,a=/^(\w+)(\.|$)(.*)$/;function c(e,t){var r=e.match(a),n=r[1],i=r[3];return i?c(i,t[n]):t[n]}return r.inPlace(Node.prototype,R,"dom-"),t.on("dom-start",(function(e){!function(e){if(!e||"string"!=typeof e.nodeName||"script"!==e.nodeName.toLowerCase())return;if("function"!=typeof e.addEventListener)return;var n=(a=e.src,s=a.match(i),s?s[1]:null);var a,s;if(!n)return;var u=function(e){var t=e.match(o);if(t&&t.length>=3)return{key:t[2],parent:c(t[1],window)};return{key:e,parent:window}}(n);if("function"!=typeof u.parent[u.key])return;var d={};function f(){t.emit("jsonp-end",[],d),e.removeEventListener("load",f,(0,P.m$)(!1)),e.removeEventListener("error",l,(0,P.m$)(!1))}function l(){t.emit("jsonp-error",[],d),t.emit("jsonp-end",[],d),e.removeEventListener("load",f,(0,P.m$)(!1)),e.removeEventListener("error",l,(0,P.m$)(!1))}r.inPlace(u.parent,[u.key],"cb-",d),e.addEventListener("load",f,(0,P.m$)(!1)),e.addEventListener("error",l,(0,P.m$)(!1)),t.emit("new-jsonp",[e.src],d)}(e[0])})),t}var k=r(5763);const H={};function L(e){const t=function(e){return(e||n.ee).get("mutation")}(e);if(!l.il||H[t.debugId])return t;H[t.debugId]=!0;var r=s(t),i=k.Yu.MO;return i&&(window.MutationObserver=function(e){return this instanceof i?new i(r(e,"fn-")):i.apply(this,arguments)},MutationObserver.prototype=i.prototype),t}const z={};function M(e){const t=function(e){return(e||n.ee).get("promise")}(e);if(z[t.debugId])return t;z[t.debugId]=!0;var r=n.c,o=s(t),a=k.Yu.PR;return a&&function(){function e(r){var n=t.context(),i=o(r,"executor-",n,null,!1);const s=Reflect.construct(a,[i],e);return t.context(s).getCtx=function(){return n},s}l._A.Promise=e,Object.defineProperty(e,"name",{value:"Promise"}),e.toString=function(){return a.toString()},Object.setPrototypeOf(e,a),["all","race"].forEach((function(r){const n=a[r];e[r]=function(e){let i=!1;[...e||[]].forEach((e=>{this.resolve(e).then(a("all"===r),a(!1))}));const o=n.apply(this,arguments);return o;function a(e){return function(){t.emit("propagate",[null,!i],o,!1,!1),i=i||!e}}}})),["resolve","reject"].forEach((function(r){const n=a[r];e[r]=function(e){const r=n.apply(this,arguments);return e!==r&&t.emit("propagate",[e,!0],r,!1,!1),r}})),e.prototype=a.prototype;const n=a.prototype.then;a.prototype.then=function(){var e=this,i=r(e);i.promise=e;for(var a=arguments.length,s=new Array(a),c=0;c<a;c++)s[c]=arguments[c];s[0]=o(s[0],"cb-",i,null,!1),s[1]=o(s[1],"cb-",i,null,!1);const u=n.apply(this,s);return i.nextPromise=u,t.emit("propagate",[e,!0],u,!1,!1),u},a.prototype.then[i]=n,t.on("executor-start",(function(e){e[0]=o(e[0],"resolve-",this,null,!1),e[1]=o(e[1],"resolve-",this,null,!1)})),t.on("executor-err",(function(e,t,r){e[1](r)})),t.on("cb-end",(function(e,r,n){t.emit("propagate",[n,!0],this.nextPromise,!1,!1)})),t.on("propagate",(function(e,r,n){this.getCtx&&!r||(this.getCtx=function(){if(e instanceof Promise)var r=t.context(e);return r&&r.getCtx?r.getCtx():this})}))}(),t}const B={},F="requestAnimationFrame";function U(e){const t=function(e){return(e||n.ee).get("raf")}(e);if(!l.il||B[t.debugId]++)return t;B[t.debugId]=1;var r=s(t);return r.inPlace(window,[F],"raf-"),t.on("raf-start",(function(e){e[0]=r(e[0],"fn-")})),t}const q={},G="setTimeout",V="setInterval",X="clearTimeout",W="-start",Z="-",$=[G,"setImmediate",V,X,"clearImmediate"];function Q(e){const t=function(e){return(e||n.ee).get("timer")}(e);if(q[t.debugId]++)return t;q[t.debugId]=1;var r=s(t);return r.inPlace(l._A,$.slice(0,2),G+Z),r.inPlace(l._A,$.slice(2,3),V+Z),r.inPlace(l._A,$.slice(3),X+Z),t.on(V+W,(function(e,t,n){e[0]=r(e[0],"fn-",null,n)})),t.on(G+W,(function(e,t,n){this.method=n,this.timerDuration=isNaN(e[1])?0:+e[1],e[0]=r(e[0],"fn-",this,n)})),t}var Y=r(50);const K={},J=["open","send"];function ee(e){var t=e||n.ee;const r=function(e){return(e||n.ee).get("xhr")}(t);if(K[r.debugId]++)return r;K[r.debugId]=1,v(t);var i=s(r),o=k.Yu.XHR,a=k.Yu.MO,c=k.Yu.PR,u=k.Yu.SI,d="readystatechange",f=["onload","onerror","onabort","onloadstart","onloadend","onprogress","ontimeout"],h=[],g=l._A.XMLHttpRequest.listeners,p=l._A.XMLHttpRequest=function(e){var t=new o(e);function n(){try{r.emit("new-xhr",[t],t),t.addEventListener(d,b,(0,P.m$)(!1))}catch(e){(0,Y.Z)("An error occured while intercepting XHR",e);try{r.emit("internal-error",[e])}catch(e){}}}return this.listeners=g?[...g,n]:[n],this.listeners.forEach((e=>e())),t};function m(e,t){i.inPlace(t,["onreadystatechange"],"fn-",E)}function b(){var e=this,t=r.context(e);e.readyState>3&&!t.resolved&&(t.resolved=!0,r.emit("xhr-resolved",[],e)),i.inPlace(e,f,"fn-",E)}if(function(e,t){for(var r in e)t[r]=e[r]}(o,p),p.prototype=o.prototype,i.inPlace(p.prototype,J,"-xhr-",E),r.on("send-xhr-start",(function(e,t){m(e,t),function(e){h.push(e),a&&(y?y.then(A):u?u(A):(w=-w,x.data=w))}(t)})),r.on("open-xhr-start",m),a){var y=c&&c.resolve();if(!u&&!c){var w=1,x=document.createTextNode(w);new a(A).observe(x,{characterData:!0})}}else t.on("fn-end",(function(e){e[0]&&e[0].type===d||A()}));function A(){for(var e=0;e<h.length;e++)m(0,h[e]);h.length&&(h=[])}function E(e,t){return t}return r}},7825:(e,t,r)=>{r.d(t,{t:()=>n});const n=r(3325).D.ajax},6660:(e,t,r)=>{r.d(t,{A:()=>i,t:()=>n});const n=r(3325).D.jserrors,i="nr@seenError"},3081:(e,t,r)=>{r.d(t,{gF:()=>o,mY:()=>i,t9:()=>n,vz:()=>s,xS:()=>a});const n=r(3325).D.metrics,i="sm",o="cm",a="storeSupportabilityMetrics",s="storeEventMetrics"},4649:(e,t,r)=>{r.d(t,{t:()=>n});const n=r(3325).D.pageAction},7633:(e,t,r)=>{r.d(t,{Dz:()=>i,OJ:()=>a,qw:()=>o,t9:()=>n});const n=r(3325).D.pageViewEvent,i="firstbyte",o="domcontent",a="windowload"},9251:(e,t,r)=>{r.d(t,{t:()=>n});const n=r(3325).D.pageViewTiming},3614:(e,t,r)=>{r.d(t,{BST_RESOURCE:()=>i,END:()=>s,FEATURE_NAME:()=>n,FN_END:()=>u,FN_START:()=>c,PUSH_STATE:()=>d,RESOURCE:()=>o,START:()=>a});const n=r(3325).D.sessionTrace,i="bstResource",o="resource",a="-start",s="-end",c="fn"+a,u="fn"+s,d="pushState"},7836:(e,t,r)=>{r.d(t,{BODY:()=>A,CB_END:()=>E,CB_START:()=>u,END:()=>x,FEATURE_NAME:()=>i,FETCH:()=>_,FETCH_BODY:()=>v,FETCH_DONE:()=>m,FETCH_START:()=>p,FN_END:()=>c,FN_START:()=>s,INTERACTION:()=>l,INTERACTION_API:()=>d,INTERACTION_EVENTS:()=>o,JSONP_END:()=>b,JSONP_NODE:()=>g,JS_TIME:()=>T,MAX_TIMER_BUDGET:()=>a,REMAINING:()=>f,SPA_NODE:()=>h,START:()=>w,originalSetTimeout:()=>y});var n=r(5763);const i=r(3325).D.spa,o=["click","submit","keypress","keydown","keyup","change"],a=999,s="fn-start",c="fn-end",u="cb-start",d="api-ixn-",f="remaining",l="interaction",h="spaNode",g="jsonpNode",p="fetch-start",m="fetch-done",v="fetch-body-",b="jsonp-end",y=n.Yu.ST,w="-start",x="-end",A="-body",E="cb"+x,T="jsTime",_="fetch"},5938:(e,t,r)=>{r.d(t,{W:()=>o});var n=r(5763),i=r(2177);class o{constructor(e,t,r){this.agentIdentifier=e,this.aggregator=t,this.ee=i.ee.get(e,(0,n.OP)(this.agentIdentifier).isolatedBacklog),this.featureName=r,this.blocked=!1}}},9144:(e,t,r)=>{r.d(t,{j:()=>m});var n=r(3325),i=r(5763),o=r(5546),a=r(2177),s=r(7894),c=r(8e3),u=r(3960),d=r(385),f=r(50),l=r(3081),h=r(8632);function g(){const e=(0,h.gG)();["setErrorHandler","finished","addToTrace","inlineHit","addRelease","addPageAction","setCurrentRouteName","setPageViewName","setCustomAttribute","interaction","noticeError","setUserId"].forEach((t=>{e[t]=function(){for(var r=arguments.length,n=new Array(r),i=0;i<r;i++)n[i]=arguments[i];return function(t){for(var r=arguments.length,n=new Array(r>1?r-1:0),i=1;i<r;i++)n[i-1]=arguments[i];let o=[];return Object.values(e.initializedAgents).forEach((e=>{e.exposed&&e.api[t]&&o.push(e.api[t](...n))})),o.length>1?o:o[0]}(t,...n)}}))}var p=r(2587);function m(e){let t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:{},m=arguments.length>2?arguments[2]:void 0,v=arguments.length>3?arguments[3]:void 0,{init:b,info:y,loader_config:w,runtime:x={loaderType:m},exposed:A=!0}=t;const E=(0,h.gG)();y||(b=E.init,y=E.info,w=E.loader_config),(0,i.Dg)(e,b||{}),(0,i.GE)(e,w||{}),(0,i.sU)(e,x),y.jsAttributes??={},d.v6&&(y.jsAttributes.isWorker=!0),(0,i.CX)(e,y),g();const T=function(e,t){t||(0,c.R)(e,"api");const h={};var g=a.ee.get(e),p=g.get("tracer"),m="api-",v=m+"ixn-";function b(t,r,n,o){const a=(0,i.C5)(e);return null===r?delete a.jsAttributes[t]:(0,i.CX)(e,{...a,jsAttributes:{...a.jsAttributes,[t]:r}}),x(m,n,!0,o||null===r?"session":void 0)(t,r)}function y(){}["setErrorHandler","finished","addToTrace","inlineHit","addRelease"].forEach((e=>h[e]=x(m,e,!0,"api"))),h.addPageAction=x(m,"addPageAction",!0,n.D.pageAction),h.setCurrentRouteName=x(m,"routeName",!0,n.D.spa),h.setPageViewName=function(t,r){if("string"==typeof t)return"/"!==t.charAt(0)&&(t="/"+t),(0,i.OP)(e).customTransaction=(r||"http://custom.transaction")+t,x(m,"setPageViewName",!0)()},h.setCustomAttribute=function(e,t){let r=arguments.length>2&&void 0!==arguments[2]&&arguments[2];if("string"==typeof e){if(["string","number"].includes(typeof t)||null===t)return b(e,t,"setCustomAttribute",r);(0,f.Z)("Failed to execute setCustomAttribute.\nNon-null value must be a string or number type, but a type of <".concat(typeof t,"> was provided."))}else(0,f.Z)("Failed to execute setCustomAttribute.\nName must be a string type, but a type of <".concat(typeof e,"> was provided."))},h.setUserId=function(e){if("string"==typeof e||null===e)return b("enduser.id",e,"setUserId",!0);(0,f.Z)("Failed to execute setUserId.\nNon-null value must be a string type, but a type of <".concat(typeof e,"> was provided."))},h.interaction=function(){return(new y).get()};var w=y.prototype={createTracer:function(e,t){var r={},i=this,a="function"==typeof t;return(0,o.p)(v+"tracer",[(0,s.z)(),e,r],i,n.D.spa,g),function(){if(p.emit((a?"":"no-")+"fn-start",[(0,s.z)(),i,a],r),a)try{return t.apply(this,arguments)}catch(e){throw p.emit("fn-err",[arguments,this,"string"==typeof e?new Error(e):e],r),e}finally{p.emit("fn-end",[(0,s.z)()],r)}}}};function x(e,t,r,i){return function(){return(0,o.p)(l.xS,["API/"+t+"/called"],void 0,n.D.metrics,g),i&&(0,o.p)(e+t,[(0,s.z)(),...arguments],r?null:this,i,g),r?void 0:this}}function A(){r.e(439).then(r.bind(r,7438)).then((t=>{let{setAPI:r}=t;r(e),(0,c.L)(e,"api")})).catch((()=>(0,f.Z)("Downloading runtime APIs failed...")))}return["actionText","setName","setAttribute","save","ignore","onEnd","getContext","end","get"].forEach((e=>{w[e]=x(v,e,void 0,n.D.spa)})),h.noticeError=function(e,t){"string"==typeof e&&(e=new Error(e)),(0,o.p)(l.xS,["API/noticeError/called"],void 0,n.D.metrics,g),(0,o.p)("err",[e,(0,s.z)(),!1,t],void 0,n.D.jserrors,g)},d.il?(0,u.b)((()=>A()),!0):A(),h}(e,v);return(0,h.Qy)(e,T,"api"),(0,h.Qy)(e,A,"exposed"),(0,h.EZ)("activatedFeatures",p.T),T}},3325:(e,t,r)=>{r.d(t,{D:()=>n,p:()=>i});const n={ajax:"ajax",jserrors:"jserrors",metrics:"metrics",pageAction:"page_action",pageViewEvent:"page_view_event",pageViewTiming:"page_view_timing",sessionReplay:"session_replay",sessionTrace:"session_trace",spa:"spa"},i={[n.pageViewEvent]:1,[n.pageViewTiming]:2,[n.metrics]:3,[n.jserrors]:4,[n.ajax]:5,[n.sessionTrace]:6,[n.pageAction]:7,[n.spa]:8,[n.sessionReplay]:9}}},n={};function i(e){var t=n[e];if(void 0!==t)return t.exports;var o=n[e]={exports:{}};return r[e](o,o.exports,i),o.exports}i.m=r,i.d=(e,t)=>{for(var r in t)i.o(t,r)&&!i.o(e,r)&&Object.defineProperty(e,r,{enumerable:!0,get:t[r]})},i.f={},i.e=e=>Promise.all(Object.keys(i.f).reduce(((t,r)=>(i.f[r](e,t),t)),[])),i.u=e=>(({78:"page_action-aggregate",147:"metrics-aggregate",242:"session-manager",317:"jserrors-aggregate",348:"page_view_timing-aggregate",412:"lazy-feature-loader",439:"async-api",538:"recorder",590:"session_replay-aggregate",675:"compressor",733:"session_trace-aggregate",786:"page_view_event-aggregate",873:"spa-aggregate",898:"ajax-aggregate"}[e]||e)+"."+{78:"ac76d497",147:"3dc53903",148:"1a20d5fe",242:"2a64278a",317:"49e41428",348:"bd6de33a",412:"2f55ce66",439:"30bd804e",538:"1b18459f",590:"cf0efb30",675:"ae9f91a8",733:"83105561",786:"06482edd",860:"03a8b7a5",873:"e6b09d52",898:"998ef92b"}[e]+"-1.236.0.min.js"),i.o=(e,t)=>Object.prototype.hasOwnProperty.call(e,t),e={},t="NRBA:",i.l=(r,n,o,a)=>{if(e[r])e[r].push(n);else{var s,c;if(void 0!==o)for(var u=document.getElementsByTagName("script"),d=0;d<u.length;d++){var f=u[d];if(f.getAttribute("src")==r||f.getAttribute("data-webpack")==t+o){s=f;break}}s||(c=!0,(s=document.createElement("script")).charset="utf-8",s.timeout=120,i.nc&&s.setAttribute("nonce",i.nc),s.setAttribute("data-webpack",t+o),s.src=r),e[r]=[n];var l=(t,n)=>{s.onerror=s.onload=null,clearTimeout(h);var i=e[r];if(delete e[r],s.parentNode&&s.parentNode.removeChild(s),i&&i.forEach((e=>e(n))),t)return t(n)},h=setTimeout(l.bind(null,void 0,{type:"timeout",target:s}),12e4);s.onerror=l.bind(null,s.onerror),s.onload=l.bind(null,s.onload),c&&document.head.appendChild(s)}},i.r=e=>{"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},i.j=364,i.p="https://js-agent.newrelic.com/",(()=>{var e={364:0,953:0};i.f.j=(t,r)=>{var n=i.o(e,t)?e[t]:void 0;if(0!==n)if(n)r.push(n[2]);else{var o=new Promise(((r,i)=>n=e[t]=[r,i]));r.push(n[2]=o);var a=i.p+i.u(t),s=new Error;i.l(a,(r=>{if(i.o(e,t)&&(0!==(n=e[t])&&(e[t]=void 0),n)){var o=r&&("load"===r.type?"missing":r.type),a=r&&r.target&&r.target.src;s.message="Loading chunk "+t+" failed.\n("+o+": "+a+")",s.name="ChunkLoadError",s.type=o,s.request=a,n[1](s)}}),"chunk-"+t,t)}};var t=(t,r)=>{var n,o,[a,s,c]=r,u=0;if(a.some((t=>0!==e[t]))){for(n in s)i.o(s,n)&&(i.m[n]=s[n]);if(c)c(i)}for(t&&t(r);u<a.length;u++)o=a[u],i.o(e,o)&&e[o]&&e[o][0](),e[o]=0},r=window.webpackChunkNRBA=window.webpackChunkNRBA||[];r.forEach(t.bind(null,0)),r.push=t.bind(null,r.push.bind(r))})();var o={};(()=>{i.r(o);var e=i(3325),t=i(5763);const r=Object.values(e.D);function n(e){const n={};return r.forEach((r=>{n[r]=function(e,r){return!1!==(0,t.Mt)(r,"".concat(e,".enabled"))}(r,e)})),n}var a=i(9144);var s=i(5546),c=i(385),u=i(8e3),d=i(5938),f=i(3960),l=i(50);class h extends d.W{constructor(e,t,r){let n=!(arguments.length>3&&void 0!==arguments[3])||arguments[3];super(e,t,r),this.auto=n,this.abortHandler,this.featAggregate,this.onAggregateImported,n&&(0,u.R)(e,r)}importAggregator(){let e=arguments.length>0&&void 0!==arguments[0]?arguments[0]:{};if(this.featAggregate||!this.auto)return;const r=c.il&&!0===(0,t.Mt)(this.agentIdentifier,"privacy.cookies_enabled");let n;this.onAggregateImported=new Promise((e=>{n=e}));const o=async()=>{let t;try{if(r){const{setupAgentSession:e}=await Promise.all([i.e(860),i.e(242)]).then(i.bind(i,3228));t=e(this.agentIdentifier)}}catch(e){(0,l.Z)("A problem occurred when starting up session manager. This page will not start or extend any session.",e)}try{if(!this.shouldImportAgg(this.featureName,t))return void(0,u.L)(this.agentIdentifier,this.featureName);const{lazyFeatureLoader:r}=await i.e(412).then(i.bind(i,8582)),{Aggregate:o}=await r(this.featureName,"aggregate");this.featAggregate=new o(this.agentIdentifier,this.aggregator,e),n(!0)}catch(e){(0,l.Z)("Downloading and initializing ".concat(this.featureName," failed..."),e),this.abortHandler?.(),n(!1)}};c.il?(0,f.b)((()=>o()),!0):o()}shouldImportAgg(r,n){return r!==e.D.sessionReplay||!1!==(0,t.Mt)(this.agentIdentifier,"session_trace.enabled")&&(!!n?.isNew||!!n?.state.sessionReplay)}}var g=i(7633),p=i(7894);class m extends h{static featureName=g.t9;constructor(r,n){let i=!(arguments.length>2&&void 0!==arguments[2])||arguments[2];if(super(r,n,g.t9,i),("undefined"==typeof PerformanceNavigationTiming||c.Tt)&&"undefined"!=typeof PerformanceTiming){const n=(0,t.OP)(r);n[g.Dz]=Math.max(Date.now()-n.offset,0),(0,f.K)((()=>n[g.qw]=Math.max((0,p.z)()-n[g.Dz],0))),(0,f.b)((()=>{const t=(0,p.z)();n[g.OJ]=Math.max(t-n[g.Dz],0),(0,s.p)("timing",["load",t],void 0,e.D.pageViewTiming,this.ee)}))}this.importAggregator()}}var v=i(1117),b=i(1284);class y extends v.w{constructor(e){super(e),this.aggregatedData={}}store(e,t,r,n,i){var o=this.getBucket(e,t,r,i);return o.metrics=function(e,t){t||(t={count:0});return t.count+=1,(0,b.D)(e,(function(e,r){t[e]=w(r,t[e])})),t}(n,o.metrics),o}merge(e,t,r,n,i){var o=this.getBucket(e,t,n,i);if(o.metrics){var a=o.metrics;a.count+=r.count,(0,b.D)(r,(function(e,t){if("count"!==e){var n=a[e],i=r[e];i&&!i.c?a[e]=w(i.t,n):a[e]=function(e,t){if(!t)return e;t.c||(t=x(t.t));return t.min=Math.min(e.min,t.min),t.max=Math.max(e.max,t.max),t.t+=e.t,t.sos+=e.sos,t.c+=e.c,t}(i,a[e])}}))}else o.metrics=r}storeMetric(e,t,r,n){var i=this.getBucket(e,t,r);return i.stats=w(n,i.stats),i}getBucket(e,t,r,n){this.aggregatedData[e]||(this.aggregatedData[e]={});var i=this.aggregatedData[e][t];return i||(i=this.aggregatedData[e][t]={params:r||{}},n&&(i.custom=n)),i}get(e,t){return t?this.aggregatedData[e]&&this.aggregatedData[e][t]:this.aggregatedData[e]}take(e){for(var t={},r="",n=!1,i=0;i<e.length;i++)t[r=e[i]]=A(this.aggregatedData[r]),t[r].length&&(n=!0),delete this.aggregatedData[r];return n?t:null}}function w(e,t){return null==e?function(e){e?e.c++:e={c:1};return e}(t):t?(t.c||(t=x(t.t)),t.c+=1,t.t+=e,t.sos+=e*e,e>t.max&&(t.max=e),e<t.min&&(t.min=e),t):{t:e}}function x(e){return{t:e,min:e,max:e,sos:e*e,c:1}}function A(e){return"object"!=typeof e?[]:(0,b.D)(e,E)}function E(e,t){return t}var T=i(8632),_=i(4402),D=i(4351);var N=i(7956),O=i(3239),j=i(9251);class S extends h{static featureName=j.t;constructor(e,r){let n=!(arguments.length>2&&void 0!==arguments[2])||arguments[2];super(e,r,j.t,n),c.il&&((0,t.OP)(e).initHidden=Boolean("hidden"===document.visibilityState),(0,N.N)((()=>(0,s.p)("docHidden",[(0,p.z)()],void 0,j.t,this.ee)),!0),(0,O.bP)("pagehide",(()=>(0,s.p)("winPagehide",[(0,p.z)()],void 0,j.t,this.ee))),this.importAggregator())}}var P=i(3081);class C extends h{static featureName=P.t9;constructor(e,t){let r=!(arguments.length>2&&void 0!==arguments[2])||arguments[2];super(e,t,P.t9,r),this.importAggregator()}}var R,I=i(2210),k=i(1214),H=i(2177),L={};try{R=localStorage.getItem("__nr_flags").split(","),console&&"function"==typeof console.log&&(L.console=!0,-1!==R.indexOf("dev")&&(L.dev=!0),-1!==R.indexOf("nr_dev")&&(L.nrDev=!0))}catch(e){}function z(e){try{L.console&&z(e)}catch(e){}}L.nrDev&&H.ee.on("internal-error",(function(e){z(e.stack)})),L.dev&&H.ee.on("fn-err",(function(e,t,r){z(r.stack)})),L.dev&&(z("NR AGENT IN DEVELOPMENT MODE"),z("flags: "+(0,b.D)(L,(function(e,t){return e})).join(", ")));var M=i(6660);class B extends h{static featureName=M.t;constructor(r,n){let i=!(arguments.length>2&&void 0!==arguments[2])||arguments[2];super(r,n,M.t,i),this.skipNext=0;try{this.removeOnAbort=new AbortController}catch(e){}const o=this;o.ee.on("fn-start",(function(e,t,r){o.abortHandler&&(o.skipNext+=1)})),o.ee.on("fn-err",(function(t,r,n){o.abortHandler&&!n[M.A]&&((0,I.X)(n,M.A,(function(){return!0})),this.thrown=!0,(0,s.p)("err",[n,(0,p.z)()],void 0,e.D.jserrors,o.ee))})),o.ee.on("fn-end",(function(){o.abortHandler&&!this.thrown&&o.skipNext>0&&(o.skipNext-=1)})),o.ee.on("internal-error",(function(t){(0,s.p)("ierr",[t,(0,p.z)(),!0],void 0,e.D.jserrors,o.ee)})),this.origOnerror=c._A.onerror,c._A.onerror=this.onerrorHandler.bind(this),c._A.addEventListener("unhandledrejection",(t=>{const r=function(e){let t="Unhandled Promise Rejection: ";if(e instanceof Error)try{return e.message=t+e.message,e}catch(t){return e}if(void 0===e)return new Error(t);try{return new Error(t+(0,D.P)(e))}catch(e){return new Error(t)}}(t.reason);(0,s.p)("err",[r,(0,p.z)(),!1,{unhandledPromiseRejection:1}],void 0,e.D.jserrors,this.ee)}),(0,O.m$)(!1,this.removeOnAbort?.signal)),(0,k.gy)(this.ee),(0,k.BV)(this.ee),(0,k.em)(this.ee),(0,t.OP)(r).xhrWrappable&&(0,k.Kf)(this.ee),this.abortHandler=this.#e,this.importAggregator()}#e(){this.removeOnAbort?.abort(),this.abortHandler=void 0}onerrorHandler(t,r,n,i,o){"function"==typeof this.origOnerror&&this.origOnerror(...arguments);try{this.skipNext?this.skipNext-=1:(0,s.p)("err",[o||new F(t,r,n),(0,p.z)()],void 0,e.D.jserrors,this.ee)}catch(t){try{(0,s.p)("ierr",[t,(0,p.z)(),!0],void 0,e.D.jserrors,this.ee)}catch(e){}}return!1}}function F(e,t,r){this.message=e||"Uncaught error with no additional information",this.sourceURL=t,this.line=r}let U=1;const q="nr@id";function G(e){const t=typeof e;return!e||"object"!==t&&"function"!==t?-1:e===c._A?0:(0,I.X)(e,q,(function(){return U++}))}function V(e){if("string"==typeof e&&e.length)return e.length;if("object"==typeof e){if("undefined"!=typeof ArrayBuffer&&e instanceof ArrayBuffer&&e.byteLength)return e.byteLength;if("undefined"!=typeof Blob&&e instanceof Blob&&e.size)return e.size;if(!("undefined"!=typeof FormData&&e instanceof FormData))try{return(0,D.P)(e).length}catch(e){return}}}var X=i(7243);class W{constructor(e){this.agentIdentifier=e,this.generateTracePayload=this.generateTracePayload.bind(this),this.shouldGenerateTrace=this.shouldGenerateTrace.bind(this)}generateTracePayload(e){if(!this.shouldGenerateTrace(e))return null;var r=(0,t.DL)(this.agentIdentifier);if(!r)return null;var n=(r.accountID||"").toString()||null,i=(r.agentID||"").toString()||null,o=(r.trustKey||"").toString()||null;if(!n||!i)return null;var a=(0,_.M)(),s=(0,_.Ht)(),c=Date.now(),u={spanId:a,traceId:s,timestamp:c};return(e.sameOrigin||this.isAllowedOrigin(e)&&this.useTraceContextHeadersForCors())&&(u.traceContextParentHeader=this.generateTraceContextParentHeader(a,s),u.traceContextStateHeader=this.generateTraceContextStateHeader(a,c,n,i,o)),(e.sameOrigin&&!this.excludeNewrelicHeader()||!e.sameOrigin&&this.isAllowedOrigin(e)&&this.useNewrelicHeaderForCors())&&(u.newrelicHeader=this.generateTraceHeader(a,s,c,n,i,o)),u}generateTraceContextParentHeader(e,t){return"00-"+t+"-"+e+"-01"}generateTraceContextStateHeader(e,t,r,n,i){return i+"@nr=0-1-"+r+"-"+n+"-"+e+"----"+t}generateTraceHeader(e,t,r,n,i,o){if(!("function"==typeof c._A?.btoa))return null;var a={v:[0,1],d:{ty:"Browser",ac:n,ap:i,id:e,tr:t,ti:r}};return o&&n!==o&&(a.d.tk=o),btoa((0,D.P)(a))}shouldGenerateTrace(e){return this.isDtEnabled()&&this.isAllowedOrigin(e)}isAllowedOrigin(e){var r=!1,n={};if((0,t.Mt)(this.agentIdentifier,"distributed_tracing")&&(n=(0,t.P_)(this.agentIdentifier).distributed_tracing),e.sameOrigin)r=!0;else if(n.allowed_origins instanceof Array)for(var i=0;i<n.allowed_origins.length;i++){var o=(0,X.e)(n.allowed_origins[i]);if(e.hostname===o.hostname&&e.protocol===o.protocol&&e.port===o.port){r=!0;break}}return r}isDtEnabled(){var e=(0,t.Mt)(this.agentIdentifier,"distributed_tracing");return!!e&&!!e.enabled}excludeNewrelicHeader(){var e=(0,t.Mt)(this.agentIdentifier,"distributed_tracing");return!!e&&!!e.exclude_newrelic_header}useNewrelicHeaderForCors(){var e=(0,t.Mt)(this.agentIdentifier,"distributed_tracing");return!!e&&!1!==e.cors_use_newrelic_header}useTraceContextHeadersForCors(){var e=(0,t.Mt)(this.agentIdentifier,"distributed_tracing");return!!e&&!!e.cors_use_tracecontext_headers}}var Z=i(7825),$=["load","error","abort","timeout"],Q=$.length,Y=t.Yu.REQ,K=c._A.XMLHttpRequest;class J extends h{static featureName=Z.t;constructor(r,n){let i=!(arguments.length>2&&void 0!==arguments[2])||arguments[2];super(r,n,Z.t,i),(0,t.OP)(r).xhrWrappable&&(this.dt=new W(r),this.handler=(e,t,r,n)=>(0,s.p)(e,t,r,n,this.ee),(0,k.u5)(this.ee),(0,k.Kf)(this.ee),function(r,n,i,o){function a(e){var t=this;t.totalCbs=0,t.called=0,t.cbTime=0,t.end=E,t.ended=!1,t.xhrGuids={},t.lastSize=null,t.loadCaptureCalled=!1,t.params=this.params||{},t.metrics=this.metrics||{},e.addEventListener("load",(function(r){_(t,e)}),(0,O.m$)(!1)),c.IF||e.addEventListener("progress",(function(e){t.lastSize=e.loaded}),(0,O.m$)(!1))}function s(e){this.params={method:e[0]},T(this,e[1]),this.metrics={}}function u(e,n){var i=(0,t.DL)(r);i.xpid&&this.sameOrigin&&n.setRequestHeader("X-NewRelic-ID",i.xpid);var a=o.generateTracePayload(this.parsedOrigin);if(a){var s=!1;a.newrelicHeader&&(n.setRequestHeader("newrelic",a.newrelicHeader),s=!0),a.traceContextParentHeader&&(n.setRequestHeader("traceparent",a.traceContextParentHeader),a.traceContextStateHeader&&n.setRequestHeader("tracestate",a.traceContextStateHeader),s=!0),s&&(this.dt=a)}}function d(e,t){var r=this.metrics,i=e[0],o=this;if(r&&i){var a=V(i);a&&(r.txSize=a)}this.startTime=(0,p.z)(),this.listener=function(e){try{"abort"!==e.type||o.loadCaptureCalled||(o.params.aborted=!0),("load"!==e.type||o.called===o.totalCbs&&(o.onloadCalled||"function"!=typeof t.onload)&&"function"==typeof o.end)&&o.end(t)}catch(e){try{n.emit("internal-error",[e])}catch(e){}}};for(var s=0;s<Q;s++)t.addEventListener($[s],this.listener,(0,O.m$)(!1))}function f(e,t,r){this.cbTime+=e,t?this.onloadCalled=!0:this.called+=1,this.called!==this.totalCbs||!this.onloadCalled&&"function"==typeof r.onload||"function"!=typeof this.end||this.end(r)}function l(e,t){var r=""+G(e)+!!t;this.xhrGuids&&!this.xhrGuids[r]&&(this.xhrGuids[r]=!0,this.totalCbs+=1)}function h(e,t){var r=""+G(e)+!!t;this.xhrGuids&&this.xhrGuids[r]&&(delete this.xhrGuids[r],this.totalCbs-=1)}function g(){this.endTime=(0,p.z)()}function m(e,t){t instanceof K&&"load"===e[0]&&n.emit("xhr-load-added",[e[1],e[2]],t)}function v(e,t){t instanceof K&&"load"===e[0]&&n.emit("xhr-load-removed",[e[1],e[2]],t)}function b(e,t,r){t instanceof K&&("onload"===r&&(this.onload=!0),("load"===(e[0]&&e[0].type)||this.onload)&&(this.xhrCbStart=(0,p.z)()))}function y(e,t){this.xhrCbStart&&n.emit("xhr-cb-time",[(0,p.z)()-this.xhrCbStart,this.onload,t],t)}function w(e){var t,r=e[1]||{};"string"==typeof e[0]?t=e[0]:e[0]&&e[0].url?t=e[0].url:c._A?.URL&&e[0]&&e[0]instanceof URL&&(t=e[0].href),t&&(this.parsedOrigin=(0,X.e)(t),this.sameOrigin=this.parsedOrigin.sameOrigin);var n=o.generateTracePayload(this.parsedOrigin);if(n&&(n.newrelicHeader||n.traceContextParentHeader))if("string"==typeof e[0]||c._A?.URL&&e[0]&&e[0]instanceof URL){var i={};for(var a in r)i[a]=r[a];i.headers=new Headers(r.headers||{}),s(i.headers,n)&&(this.dt=n),e.length>1?e[1]=i:e.push(i)}else e[0]&&e[0].headers&&s(e[0].headers,n)&&(this.dt=n);function s(e,t){var r=!1;return t.newrelicHeader&&(e.set("newrelic",t.newrelicHeader),r=!0),t.traceContextParentHeader&&(e.set("traceparent",t.traceContextParentHeader),t.traceContextStateHeader&&e.set("tracestate",t.traceContextStateHeader),r=!0),r}}function x(e,t){this.params={},this.metrics={},this.startTime=(0,p.z)(),this.dt=t,e.length>=1&&(this.target=e[0]),e.length>=2&&(this.opts=e[1]);var r,n=this.opts||{},i=this.target;"string"==typeof i?r=i:"object"==typeof i&&i instanceof Y?r=i.url:c._A?.URL&&"object"==typeof i&&i instanceof URL&&(r=i.href),T(this,r);var o=(""+(i&&i instanceof Y&&i.method||n.method||"GET")).toUpperCase();this.params.method=o,this.txSize=V(n.body)||0}function A(t,r){var n;this.endTime=(0,p.z)(),this.params||(this.params={}),this.params.status=r?r.status:0,"string"==typeof this.rxSize&&this.rxSize.length>0&&(n=+this.rxSize);var o={txSize:this.txSize,rxSize:n,duration:(0,p.z)()-this.startTime};i("xhr",[this.params,o,this.startTime,this.endTime,"fetch"],this,e.D.ajax)}function E(t){var r=this.params,n=this.metrics;if(!this.ended){this.ended=!0;for(var o=0;o<Q;o++)t.removeEventListener($[o],this.listener,!1);r.aborted||(n.duration=(0,p.z)()-this.startTime,this.loadCaptureCalled||4!==t.readyState?null==r.status&&(r.status=0):_(this,t),n.cbTime=this.cbTime,i("xhr",[r,n,this.startTime,this.endTime,"xhr"],this,e.D.ajax))}}function T(e,t){var r=(0,X.e)(t),n=e.params;n.hostname=r.hostname,n.port=r.port,n.protocol=r.protocol,n.host=r.hostname+":"+r.port,n.pathname=r.pathname,e.parsedOrigin=r,e.sameOrigin=r.sameOrigin}function _(e,t){e.params.status=t.status;var r=function(e,t){var r=e.responseType;return"json"===r&&null!==t?t:"arraybuffer"===r||"blob"===r||"json"===r?V(e.response):"text"===r||""===r||void 0===r?V(e.responseText):void 0}(t,e.lastSize);if(r&&(e.metrics.rxSize=r),e.sameOrigin){var n=t.getResponseHeader("X-NewRelic-App-Data");n&&(e.params.cat=n.split(", ").pop())}e.loadCaptureCalled=!0}n.on("new-xhr",a),n.on("open-xhr-start",s),n.on("open-xhr-end",u),n.on("send-xhr-start",d),n.on("xhr-cb-time",f),n.on("xhr-load-added",l),n.on("xhr-load-removed",h),n.on("xhr-resolved",g),n.on("addEventListener-end",m),n.on("removeEventListener-end",v),n.on("fn-end",y),n.on("fetch-before-start",w),n.on("fetch-start",x),n.on("fn-start",b),n.on("fetch-done",A)}(r,this.ee,this.handler,this.dt),this.importAggregator())}}var ee=i(3614);const{BST_RESOURCE:te,RESOURCE:re,START:ne,END:ie,FEATURE_NAME:oe,FN_END:ae,FN_START:se,PUSH_STATE:ce}=ee;var ue=i(7836);const{FEATURE_NAME:de,START:fe,END:le,BODY:he,CB_END:ge,JS_TIME:pe,FETCH:me,FN_START:ve,CB_START:be,FN_END:ye}=ue;var we=i(4649);class xe extends h{static featureName=we.t;constructor(e,t){let r=!(arguments.length>2&&void 0!==arguments[2])||arguments[2];super(e,t,we.t,r),this.importAggregator()}}new class{constructor(e){let t=arguments.length>1&&void 0!==arguments[1]?arguments[1]:(0,_.ky)(16);c._A?(this.agentIdentifier=t,this.sharedAggregator=new y({agentIdentifier:this.agentIdentifier}),this.features={},this.desiredFeatures=new Set(e.features||[]),this.desiredFeatures.add(m),Object.assign(this,(0,a.j)(this.agentIdentifier,e,e.loaderType||"agent")),this.start()):(0,l.Z)("Failed to initial the agent. Could not determine the runtime environment.")}get config(){return{info:(0,t.C5)(this.agentIdentifier),init:(0,t.P_)(this.agentIdentifier),loader_config:(0,t.DL)(this.agentIdentifier),runtime:(0,t.OP)(this.agentIdentifier)}}start(){const t="features";try{const r=n(this.agentIdentifier),i=[...this.desiredFeatures];i.sort(((t,r)=>e.p[t.featureName]-e.p[r.featureName])),i.forEach((t=>{if(r[t.featureName]||t.featureName===e.D.pageViewEvent){const n=function(t){switch(t){case e.D.ajax:return[e.D.jserrors];case e.D.sessionTrace:return[e.D.ajax,e.D.pageViewEvent];case e.D.sessionReplay:return[e.D.sessionTrace];case e.D.pageViewTiming:return[e.D.pageViewEvent];default:return[]}}(t.featureName);n.every((e=>r[e]))||(0,l.Z)("".concat(t.featureName," is enabled but one or more dependent features has been disabled (").concat((0,D.P)(n),"). This may cause unintended consequences or missing data...")),this.features[t.featureName]=new t(this.agentIdentifier,this.sharedAggregator)}})),(0,T.Qy)(this.agentIdentifier,this.features,t)}catch(e){(0,l.Z)("Failed to initialize all enabled instrument classes (agent aborted) -",e);for(const e in this.features)this.features[e].abortHandler?.();const r=(0,T.fP)();return delete r.initializedAgents[this.agentIdentifier]?.api,delete r.initializedAgents[this.agentIdentifier]?.[t],delete this.sharedAggregator,r.ee?.abort(),delete r.ee?.get(this.agentIdentifier),!1}}}({features:[J,m,S,class extends h{static featureName=oe;constructor(t,r){if(super(t,r,oe,!(arguments.length>2&&void 0!==arguments[2])||arguments[2]),!c.il)return;const n=this.ee;let i;(0,k.QU)(n),this.eventsEE=(0,k.em)(n),this.eventsEE.on(se,(function(e,t){this.bstStart=(0,p.z)()})),this.eventsEE.on(ae,(function(t,r){(0,s.p)("bst",[t[0],r,this.bstStart,(0,p.z)()],void 0,e.D.sessionTrace,n)})),n.on(ce+ne,(function(e){this.time=(0,p.z)(),this.startPath=location.pathname+location.hash})),n.on(ce+ie,(function(t){(0,s.p)("bstHist",[location.pathname+location.hash,this.startPath,this.time],void 0,e.D.sessionTrace,n)}));try{i=new PerformanceObserver((t=>{const r=t.getEntries();(0,s.p)(te,[r],void 0,e.D.sessionTrace,n)})),i.observe({type:re,buffered:!0})}catch(e){}this.importAggregator({resourceObserver:i})}},C,xe,B,class extends h{static featureName=de;constructor(e,r){if(super(e,r,de,!(arguments.length>2&&void 0!==arguments[2])||arguments[2]),!c.il)return;if(!(0,t.OP)(e).xhrWrappable)return;try{this.removeOnAbort=new AbortController}catch(e){}let n,i=0;const o=this.ee.get("tracer"),a=(0,k._L)(this.ee),s=(0,k.Lg)(this.ee),u=(0,k.BV)(this.ee),d=(0,k.Kf)(this.ee),f=this.ee.get("events"),l=(0,k.u5)(this.ee),h=(0,k.QU)(this.ee),g=(0,k.Gm)(this.ee);function m(e,t){h.emit("newURL",[""+window.location,t])}function v(){i++,n=window.location.hash,this[ve]=(0,p.z)()}function b(){i--,window.location.hash!==n&&m(0,!0);var e=(0,p.z)();this[pe]=~~this[pe]+e-this[ve],this[ye]=e}function y(e,t){e.on(t,(function(){this[t]=(0,p.z)()}))}this.ee.on(ve,v),s.on(be,v),a.on(be,v),this.ee.on(ye,b),s.on(ge,b),a.on(ge,b),this.ee.buffer([ve,ye,"xhr-resolved"],this.featureName),f.buffer([ve],this.featureName),u.buffer(["setTimeout"+le,"clearTimeout"+fe,ve],this.featureName),d.buffer([ve,"new-xhr","send-xhr"+fe],this.featureName),l.buffer([me+fe,me+"-done",me+he+fe,me+he+le],this.featureName),h.buffer(["newURL"],this.featureName),g.buffer([ve],this.featureName),s.buffer(["propagate",be,ge,"executor-err","resolve"+fe],this.featureName),o.buffer([ve,"no-"+ve],this.featureName),a.buffer(["new-jsonp","cb-start","jsonp-error","jsonp-end"],this.featureName),y(l,me+fe),y(l,me+"-done"),y(a,"new-jsonp"),y(a,"jsonp-end"),y(a,"cb-start"),h.on("pushState-end",m),h.on("replaceState-end",m),window.addEventListener("hashchange",m,(0,O.m$)(!0,this.removeOnAbort?.signal)),window.addEventListener("load",m,(0,O.m$)(!0,this.removeOnAbort?.signal)),window.addEventListener("popstate",(function(){m(0,i>1)}),(0,O.m$)(!0,this.removeOnAbort?.signal)),this.abortHandler=this.#e,this.importAggregator()}#e(){this.removeOnAbort?.abort(),this.abortHandler=void 0}}],loaderType:"spa"})})(),window.NRBA=o})(); </script> <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.8.1/jquery.min.js"></script> <script>window.jQuery || document.write('<script src="/js/vendor/jquery-1.8.1.min.js"><\/script>')</script> <script src="https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.22.1/moment.min.js"></script> <script src="/1740487202716/js/vendor/modernizr-2.6.1-respond-1.1.0.min.js"></script> <script src="/1740487202716/js/shared_scripts/sticky.js"></script> <script src="/1740487202716/js/shared_scripts/helpers.js"></script> <script src="/1740487202716/js/shared_scripts/menu.js"></script> <script src="/1740487202716/js/shared_scripts/navbar.js"></script> <script src="/1740487202716/js/shared_scripts/platforms.js"></script> <script src="/1740487202716/js/shared_scripts/object-polyfills.js"></script> <script src="/1740487202716/js/vendor/lodash.min.js"></script> <script>CKEDITOR_BASEPATH='https://f1000research.com/js/vendor/ckeditor/'</script> <script src="/1740487202716/js/plugins.js"></script> <script src="/1740487202716/js/shared_scripts/helpers.js"></script> <script src="/1740487202716/js/app/research.js"></script> <script>window.reactTheme = 'research';</script> <script src="/js/public/bundle.js"></script> <script src="/1740487202716/js/app/research.ui.js"></script> <script src="/1740487202716/js/app/login.js"></script> <script src="/1740487202716/js/app/main.js"></script> <script src="/1740487202716/js/app/js-date-format.min.js"></script> <script src="/1740487202716/js/app/search.js"></script> <script src="/1740487202716/js/mdl/mdl.min.js"></script> <script> window.MathJax = { CommonHTML: { linebreaks: { automatic: true } }, 'HTML-CSS': { linebreaks: { automatic: true } }, SVG: { linebreaks: { automatic: true } }, AuthorInit: function() { MathJax.Hub.Register.MessageHook('End Process', function () { let timeout = false; // holder for timeout id const delay = 250; // delay after event is "complete" to run callback const reflowMath = function() { const dispFormulas = document.querySelectorAll('.disp-formula.panel'); if (!dispFormulas) { return; } for (const dispFormula of dispFormulas) { const child = dispFormula.querySelector('.MathJax_Preview').nextSibling.firstChild; const isMultiline = MathJax.Hub.getAllJax(dispFormula)[0].root.isMultiline; if (dispFormula.offsetWidth < child.offsetWidth || isMultiline) { MathJax.Hub.Queue(['Rerender', MathJax.Hub, dispFormula]); } } }; window.addEventListener('resize', function() { clearTimeout(timeout); // clear the timeout timeout = setTimeout(reflowMath, delay); // start timing for event "completion" }); }); }, }; </script> <script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script> <script src="/js/namespace.js"></script> <script src="/js/constants.js"></script> <script src="/js/utilities.js"></script> <script src="/js/shared_scripts/referee_timeline_pagination.js"></script> <script src="/js/article/article_scrolling_module.js"></script> <script src="/js/article/article.js"></script> <script src="/js/article/article_metrics.js"></script> <script src="/js/article/article_stats.js"></script> <script> if (window.location.hash == '#_=_'){ window.location = window.location.href.split('#')[0] } </script> <!-- pixelId: 1641728616063202 :: assetPixelId: 6034867600215 :: funderPixelId: $!fbFunderPixelId --> <!-- Facebook pixel code (merged with EP GTM code) --> <script > !function(f,b,e,v,n,t,s){if(f.fbq)return;n=f.fbq=function() {n.callMethod? n.callMethod.apply(n,arguments):n.queue.push(arguments)} ;if(!f._fbq)f._fbq=n; n.push=n;n.loaded=!0;n.version='2.0';n.queue=[];t=b.createElement(e);t.async=!0; t.src=v;s=b.getElementsByTagName(e)[0];s.parentNode.insertBefore(t,s)}(window, document,'script','https://connect.facebook.net/en_US/fbevents.js'); fbq('init', '1641728616063202'); fbq('track', "PixelInitialized", {}); </script> <noscript><img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=1641728616063202&noscript=1&ev=PixelInitialized" /></noscript> <!-- End Facebook Pixel Code --> <meta name="facebook-domain-verification" content="0ba5yc8np4c67n85z5ma2vtck11by4" /> <script> (function(h,o,t,j,a,r){ h.hj=h.hj||function(){(h.hj.q=h.hj.q||[]).push(arguments)}; h._hjSettings={hjid:2318163,hjsv:6}; a=o.getElementsByTagName('head')[0]; r=o.createElement('script');r.async=1; r.src=t+h._hjSettings.hjid+j+h._hjSettings.hjsv; a.appendChild(r); })(window,document,'https://static.hotjar.com/c/hotjar-','.js?sv='); </script> </head> <body class="o-page-container no-js p-article o-layout-reset "> <!-- Google Tag Manager (noscript) --> <noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-PCBS9JK" height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript> <!-- End Google Tag Manager (noscript) --> <div class="o-page"> <div id="notify-container"></div> <div id="pageWarning"></div> <div id="pageMessage"></div> <div id="pageFooterMessage"></div> <div class="header-wrapper js-navbar-space "> <nav class="c-navbar js-navbar js-mini-nav js-sticky c-navbar--js-sticky c-navbar--userSite c-navbar__platform-bgcolor c-navbar--bg-f1000research "> <div class="c-navbar__content"> <div class="c-navbar__extras"> <div class="o-wrapper"> <div class="o-actions o-actions--middle c-navbar__extras-row"> <div class="o-actions__primary"> </div> </div> </div> </div> <div class="o-wrapper t-inverted js-sticky-start"> <div class="c-navbar__branding-row"> <div class="c-navbar__row"> <div class="c-navbar__primary u-mr--2"> <a href="/" class="c-navbar__branding u-ib u-middle" data-test-id="nav_branding" > <img class="u-ib u-middle" src="/img/research/F1000Research_white_solid.svg" alt="F1000Research"> </a> </div> <div class="c-navbar__secondary c-navbar__row"> <form action="/search" class="-navbar__secondary u-mr--2 c-search-form js-search-form u-hide u-show@navbar"> <label for="searchInput" class="c-search-form__label _mdl-layout"> <input name="q" type="search" class="c-search-form__input" id="searchInput" placeholder="Search"> <button type="submit" class="c-search-form__submit mdl-button mdl-js-button mdl-button--icon"><i class="material-icons">search</i></button> </label> </form> <div class="c-navbar__primary u-hide u-show@navbar"> <div class="_mdl-layout c-navbar__cta"> <a class="mdl-button mdl-js-button mdl-button--raised mdl-button--no-shadow mdl-button--multi-line mdl-js-ripple-effect mdl-button--inverted c-navbar__submit" href="/for-authors/publish-your-research" data-test-id="nav_submit_research" ><i class="material-icons">file_upload</i>Submit your research</a> </div> </div> <span class="u-hide@navbar _mdl-layout u-nowrap"> <button type="button" class="mdl-button mdl-js-button mdl-js-ripple-effect mdl-button--icon c-navbar__toggle c-navbar__toggle--menu js-navbar-toggle" data-focus="#navbar_mob_search_input" data-toggle="navbarMenu" data-target="navbarMenu" aria-controls="navbarMenu" aria-expanded="false" aria-label="Toggle navigation" data-test-id="nav_menu_search_mob" ><i class="material-icons">search</i></button> <button class="mdl-button mdl-js-button mdl-js-ripple-effect mdl-button--icon mdl-button--multi-line c-navbar__toggle c-navbar__toggle--menu js-navbar-toggle" type="button" data-toggle="navbarMenu" data-target="navbarMenu" aria-controls="navbarMenu" aria-expanded="false" aria-label="Toggle navigation" data-test-id="nav_menu_toggle_mob" > <i class="material-icons c-navbar__toggle-open">menu</i> <i class="material-icons c-navbar__toggle-close">close</i> </button> </span> </div> </div> </div> <div class="c-navbar__menu-row js-navbar-block is-collapsed" id="navbarMenu"> <div class="c-navbar__menu-row-content"> <div class="u-hide@navbar c-navbar__menu-bar-spacing"> <form action="/search" class="c-search-form js-search-form"> <label for="navbar_mob_search_input" class="c-search-form__label _mdl-layout"> <input id="navbar_mob_search_input" name="q" type="search" class="c-search-form__input" placeholder="Search"> <button type="submit" class="c-search-form__submit mdl-button mdl-js-button mdl-button--icon"><i class="material-icons">search</i></button> </label> </form> </div> <div class="o-actions o-actions--middle"> <div class="o-actions__primary"> <ul class="c-menubar c-navbar__menu-bar js-main-menu" id="main-menu" role="menubar" aria-label="Main Navigation" data-menu-group="navbar" > <li role="none" data-index="0" class="c-menubar__item js-menu-item c-navbar__menu-bar-item "> <a href="/browse/articles" class="c-navbar__menu-bar-link js-menu-link " role="menuitem" tabindex="0" data-test-id="nav_browse" >Browse</a> </li> <li role="none" data-index="1" class="c-menubar__item js-menu-item c-navbar__menu-bar-item "> <a href="/gateways" class="c-navbar__menu-bar-link js-menu-link " role="menuitem" tabindex="-1" data-test-id="nav_gatewaysViewAndBrowse" >Gateways & Collections</a> </li> <li role="none" data-index="2" class="c-menubar__item js-menu-item c-navbar__menu-bar-item c-menubar__item--selected c-navbar__menu-bar-item--parent "> <a href="#" class="c-navbar__menu-bar-link js-menu-link " role="menuitem" aria-haspopup="true" aria-expanded="false" tabindex="-1" data-test-id="nav_for-authors" >How to Publish</a> <ul class="c-menu js-menu is-collapsed c-menubar__menu c-navbar__menu" role="menu" aria-label="How to Publish"> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_submit-manuscript" href="/for-authors/publish-your-research" role="menuitem" tabindex="0">Submit your Research</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_my-submissions" href="/for-authors/my-submissions" role="menuitem" tabindex="-1">My Submissions</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_article-guidelines" href="/for-authors/article-guidelines" role="menuitem" tabindex="-1">Article Guidelines</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_article-guidelines-new-versions" href="/for-authors/article-guidelines-new-versions" role="menuitem" tabindex="-1">Article Guidelines (New Versions)</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_article-guidelines-the-production-process" href="/for-authors/article-guidelines-the-production-process" role="menuitem" tabindex="-1">Article Guidelines - The Production Process</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_data-guidelines" href="/for-authors/data-guidelines" role="menuitem" tabindex="-1">Open Data, Software and Code Guidelines</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_data-guidelines-hss" href="/for-authors/data-guidelines-hss" role="menuitem" tabindex="-1">Open Data and Accessible Source Materials Guidelines (HSS)</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_data-guidelines-pse" href="/for-authors/data-guidelines-pse" role="menuitem" tabindex="-1">Open Data, Software and Code Guidelines (PSE)</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_prepublication-checks" href="/for-authors/prepublication-checks" role="menuitem" tabindex="-1">Prepublication Checks</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_asset-guidelines" href="/for-authors/posters-and-slides-guidelines" role="menuitem" tabindex="-1">Posters and Slides Guidelines</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_document-guidelines" href="/for-authors/document-guidelines" role="menuitem" tabindex="-1">Document Guidelines</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_article-processing-charges" href="/for-authors/article-processing-charges" role="menuitem" tabindex="-1">Article Processing Charges</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_peer-review" href="/for-authors/peer-review" role="menuitem" tabindex="-1">Peer Review</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_finding-referees" href="/for-authors/tips-for-finding-referees" role="menuitem" tabindex="-1">Finding Article Reviewers</a> </li> </ul> </li> <li role="none" data-index="3" class="c-menubar__item js-menu-item c-navbar__menu-bar-item c-menubar__item--selected c-navbar__menu-bar-item--parent "> <a href="#" class="c-navbar__menu-bar-link js-menu-link " role="menuitem" aria-haspopup="true" aria-expanded="false" tabindex="-1" data-test-id="nav_about-contact" >About</a> <ul class="c-menu js-menu is-collapsed c-menubar__menu c-navbar__menu" role="menu" aria-label="About"> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_about-page" href="/about" role="menuitem" tabindex="0">How it Works</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_referee-guidelines" href="/for-referees/guidelines" role="menuitem" tabindex="-1">For Reviewers</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_advisoryPanel" href="/advisors" role="menuitem" tabindex="-1">Our Advisors</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_policy-page" href="/about/policies" role="menuitem" tabindex="-1">Policies</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_glossary-page" href="/glossary" role="menuitem" tabindex="-1">Glossary</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_faqs-page" href="/faqs" role="menuitem" tabindex="-1">FAQs</a> </li> <li class="c-menu__item js-menu-item u-hide u-show@navbar" role="none"> <a class="c-menu__link js-menu-link u-hide u-show@navbar" data-test-id="nav_for-developers" href="/developers" role="menuitem" tabindex="-1">For Developers</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_newsroom-page" href="/newsroom" role="menuitem" tabindex="-1">Newsroom</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_contact-page" href="/contact" role="menuitem" tabindex="-1">Contact</a> </li> </ul> </li> </ul> </div> <div class="o-actions__secondary"> <ul class="c-menubar c-navbar__menu-bar js-main-menu" id="secondary-items" role="menubar" aria-label="My Account" data-menu-group="navbar" > <li role="none" data-index="0" class="c-menubar__item js-menu-item c-navbar__menu-bar-item c-menubar__item--selected c-navbar__menu-bar-item--parent "> <a href="#" class="c-navbar__menu-bar-link js-menu-link " role="menuitem" aria-haspopup="true" aria-expanded="false" tabindex="0" data-test-id="nav_my-research" >My Research</a> <ul class="c-menu js-menu is-collapsed c-menubar__menu c-navbar__menu" role="menu" aria-label="My Research"> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_my-submissions" href="/login?originalPath=%2Fmy%2Fsubmissions" role="menuitem" tabindex="0">Submissions</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_my-email-alerts" href="/login?originalPath=%2Fmy%2Femail-alerts" role="menuitem" tabindex="-1">Content and Tracking Alerts</a> </li> <li class="c-menu__item js-menu-item " role="none"> <a class="c-menu__link js-menu-link " data-test-id="nav_my-user-details" href="/login?originalPath=%2Fmy%2Fuser-details" role="menuitem" tabindex="-1">My Details</a> </li> </ul> </li> <li role="none" data-index="1" class="c-menubar__item js-menu-item c-navbar__menu-bar-item "> <a href="/login?originalPath=%2Farticles%2F9-295%2Fv2" class="c-navbar__menu-bar-link js-menu-link " role="menuitem" tabindex="-1" data-test-id="nav_sign-in" >Sign In</a> </li> </ul> </div> <div class="_mdl-layout c-navbar__cta u-hide@navbar c-navbar__menu-bar-spacing"> <a class="mdl-button mdl-js-button mdl-button--raised mdl-button--multi-line mdl-button--no-shadow mdl-js-ripple-effect mdl-button--inverted c-navbar__submit" href="/for-authors/publish-your-research" data-test-id="nav_submit_research_mob" ><i class="material-icons">file_upload</i>Submit your research</a> </div> </div> </div> </div> </div> </div> </nav> </div> <div class="content-wrapper o-page__main row "> <div id="highlight-area" class="content "> <div id=article-metadata class=hidden> <input type=hidden name=versionId value=31504 /> <input type=hidden id=articleId name=articleId value=23224 /> <input type=hidden id=xmlUrl value="/articles/9-295/v2/xml"/> <input type=hidden id=xmlFileName value="-9-295-v2.xml"> <input type=hidden id=article_uuid value=15417a87-a637-4761-bc45-e2c4e71a7858 /> <input type=hidden id=referer value=""/> <input type=hidden id=meta-article-title value="An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action"/> <input type=hidden id=workspace-export-url value="https://sciwheel.com/work/api/import/external?doi=10.12688/f1000research.23224.2"/> <input type=hidden id=versionDoi value="10.12688/f1000research.23224.2"/> <input type=hidden id=usePmcStats value=true /> </div> <main class="o-wrapper p-article__wrapper js-wrapper"> <script type="application/ld+json"> { "@context": "https://schema.org", "@type": "ScholarlyArticle", "mainEntityOfPage": { "@type": "WebPage", "@id": "https://f1000research.com/articles/9-295" }, "headline": "An environment for sustainable research software in Germany and beyond: current state, open challenges, and...", "datePublished": "2020-04-27T12:36:23", "dateModified": "2021-01-26T17:50:25", "author": [ { "@type": "Person", "name": "Hartwig Anzt" }, { "@type": "Person", "name": "Felix Bach" }, { "@type": "Person", "name": "Stephan Druskat" }, { "@type": "Person", "name": "Frank Löffler" }, { "@type": "Person", "name": "Axel Loewe" }, { "@type": "Person", "name": "Bernhard Y. Renard" }, { "@type": "Person", "name": "Gunnar Seemann" }, { "@type": "Person", "name": "Alexander Struck" }, { "@type": "Person", "name": "Elke Achhammer" }, { "@type": "Person", "name": "Piush Aggarwal" }, { "@type": "Person", "name": "Franziska Appel" }, { "@type": "Person", "name": "Michael Bader" }, { "@type": "Person", "name": "Lutz Brusch" }, { "@type": "Person", "name": "Christian Busse" }, { "@type": "Person", "name": "Gerasimos Chourdakis" }, { "@type": "Person", "name": "Piotr Wojciech Dabrowski" }, { "@type": "Person", "name": "Peter Ebert" }, { "@type": "Person", "name": "Bernd Flemisch" }, { "@type": "Person", "name": "Sven Friedl" }, { "@type": "Person", "name": "Bernadette Fritzsch" }, { "@type": "Person", "name": "Maximilian D. Funk" }, { "@type": "Person", "name": "Volker Gast" }, { "@type": "Person", "name": "Florian Goth" }, { "@type": "Person", "name": "Jean-Noël Grad" }, { "@type": "Person", "name": "Jan Hegewald" }, { "@type": "Person", "name": "Sibylle Hermann" }, { "@type": "Person", "name": "Florian Hohmann" }, { "@type": "Person", "name": "Stephan Janosch" }, { "@type": "Person", "name": "Dominik Kutra" }, { "@type": "Person", "name": "Jan Linxweiler" }, { "@type": "Person", "name": "Thilo Muth" }, { "@type": "Person", "name": "Wolfgang Peters-Kottig" }, { "@type": "Person", "name": "Fabian Rack" }, { "@type": "Person", "name": "Fabian H.C. Raters" }, { "@type": "Person", "name": "Stephan Rave" }, { "@type": "Person", "name": "Guido Reina" }, { "@type": "Person", "name": "Malte Reißig" }, { "@type": "Person", "name": "Timo Ropinski" }, { "@type": "Person", "name": "Joerg Schaarschmidt" }, { "@type": "Person", "name": "Heidi Seibold" }, { "@type": "Person", "name": "Jan P. Thiele" }, { "@type": "Person", "name": "Benjamin Uekermann" }, { "@type": "Person", "name": "Stefan Unger" }, { "@type": "Person", "name": "Rudolf Weeber" } ], "publisher": { "@type": "Organization", "name": "F1000Research", "logo": { "@type": "ImageObject", "url": "https://f1000research.com/img/AMP/F1000Research_image.png", "height": 480, "width": 60 } }, "image": { "@type": "ImageObject", "url": "https://f1000research.com/img/AMP/F1000Research_image.png", "height": 1200, "width": 150 }, "description": "Research software has become a central asset in academic research. It optimizes existing and enables new research methods, implements and embeds research knowledge, and constitutes an essential research product in itself. Research software must be sustainable in order to understand, replicate, reproduce, and build upon existing research or conduct new research effectively. In other words, software must be available, discoverable, usable, and adaptable to new needs, both now and in the future. Research software therefore requires an environment that supports sustainability. Hence, a change is needed in the way research software development and maintenance are currently motivated, incentivized, funded, structurally and infrastructurally supported, and legally treated. Failing to do so will threaten the quality and validity of research. In this paper, we identify challenges for research software sustainability in Germany and beyond, in terms of motivation, selection, research software engineering personnel, funding, infrastructure, and legal aspects. Besides researchers, we specifically address political and academic decision-makers to increase awareness of the importance and needs of sustainable research software practices. In particular, we recommend strategies and measures to create an environment for sustainable research software, with the ultimate goal to ensure that software-driven research is valid, reproducible and sustainable, and that software is recognized as a first class citizen in research. This paper is the outcome of two workshops run in Germany in 2019, at deRSE19 - the first International Conference of Research Software Engineers in Germany - and a dedicated DFG-supported follow-up workshop in Berlin." } </script> <div class="o-layout o-layout--right-gutter"> <div id=article_secondary-column class="p-article__main o-layout__item u-font-size--legal u-2/3@article not-expanded " data-version-id=31504> <div class=float-left> <script type="application/ld+json"> { "@context": "http://schema.org", "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "position": "1", "item": { "@id": "https://f1000research.com/", "name": "Home" } }, { "@type": "ListItem", "position": "2", "item": { "@id": "https://f1000research.com/browse/articles", "name": "Browse" } }, { "@type": "ListItem", "position": "3", "item": { "@id": "https://f1000research.com/articles/9-295/v2", "name": "An environment for sustainable research software in Germany and beyond:..." } } ] } </script> <div class="breadcrumbs js-breadcrumbs"> <a href="/" class=f1r-standard-link>Home</a> <span class=item_separator></span> <a href="/browse/articles" class=f1r-standard-link>Browse</a> <span class=item_separator></span> An environment for sustainable research software in Germany and beyond:... </div> </div> <div class="article-badges-container u-mb--2"> <div class=crossmark-new> <script src="https://crossmark-cdn.crossref.org/widget/v2.0/widget.js"></script> <a data-target=crossmark><img height=30 width=150 src="https://crossmark-cdn.crossref.org/widget/v2.0/logos/CROSSMARK_Color_horizontal.svg"/></a> </div> <div id=crossmark-dialog style="display: none;" title=""> <iframe id=crossmark-dialog-frame frameborder=0></iframe> </div> <div class=clearfix></div> </div> <div class=article-interaction-container> <div id=main-article-count-box class=article-count-box> <div class="article-metrics-wrapper metrics-icon-wrapper" data-version-id=31504 data-id=23224 data-downloads="" data-views="" data-scholar="10.12688/f1000research.23224.2" data-recommended="" data-doi="10.12688/f1000research.23224.2" data-f1r-ga-helper="Article Page Metrics (Desktop)"> <span class="metrics-on-browse article-metrics-icon f1r-icon icon-89_metrics"></span> <div class="count-title article-metrics-text">ALL Metrics</div> <div class=js-article-metrics-container></div> </div> <div> <div class=count-delimiter></div> <div title="Total views from F1000Research and PubMed Central"> <div class="count-container view-count js-views-count">-</div> <div class=count-title><span class="count-title-icon count-title-views-icon"></span>Views</div> </div> <div class=download-counts hidden> <div class=count-delimiter></div> <div title="Total downloads from F1000Research and PubMed Central"> <div class="count-container js-downloads-count"></div> <div class=count-title><span class="count-title-icon f1r-icon icon-76_download_file"></span>Downloads</div> </div> </div> </div> </div> <div id=main-article-interaction-box class="article-interaction-box has-control-tab"> <div class="article-interaction-info article-page"> <div class=article-interaction-button> <span class="f1r-icon icon-102_download_pdf"></span> <a href="https://f1000research.com/articles/9-295/v2/pdf?article_uuid=15417a87-a637-4761-bc45-e2c4e71a7858" title="Download PDF" class="button-link download pdf-download-helper" target=_blank>Get PDF</a> </div> </div> <div class="article-interaction-info article-page"> <div class=article-interaction-button> <span class="f1r-icon icon-103_download_xml"></span> <a id=download-xml href="#" class="button-link download" title="Download XML">Get XML</a> </div> </div> <div class="article-interaction-info article-page"> <div class="cite-article-popup-wrapper article-page-interaction-box"> <div class="article-interaction-button cite-article-button" title="Cite this article" data-windowref=cite-article-popup-23224-2> <span class="f1r-icon icon-82_quote"></span> <a href="#" class="button-link cite-article-popup-link" title="Cite Article">Cite</a> </div> <div id=cite-article-popup-23224-2 class="popup-window-wrapper is-hidden"> <div class=cite-popup-background></div> <div class="popup-window top-popup cite-this-article-box research-layout"> <div class="popup-window-title small cite-title">How to cite this article</div> <span id=cite-article-text-23224-2 data-test-id=copy-citation_text> <span class="article-title-and-info in-popup">Anzt H, Bach F, Druskat S <em>et al.</em> An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action [version 2; peer review: 2 approved]</span>. <i>F1000Research</i> 2021, <b>9</b>:295 (<a class=new-orange href="https://doi.org/10.12688/f1000research.23224.2" target=_blank>https://doi.org/10.12688/f1000research.23224.2</a>) </span> <div class="popup-window-title small margin-top-20 margin-bottom-20 note"> <strong>NOTE:</strong> If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article. </div> <div class=float-right> <button class="secondary no-fill orange-text-and-border margin-right-20 close-cite-popup uppercase">Close</button> <button id=copy-citation-details class="secondary orange copy-cite-article-version uppercase js-clipboard" title="Copy the current citation details to the clipboard." data-clipboard-target="#cite-article-text-23224-2" data-test-id=copy-citation_button>Copy Citation Details</button> </div> </div> </div> </div> </div> <div class="article-interaction-info article-page"> <div class=article-interaction-button> <span class="f1r-icon icon-76_download_file"></span> <a id=export-citation href="#" class="button-link download" title="Export Citation">Export</a> </div> <div class="modal-window-wrapper is-hidden"> <div id=export-citation-popup class="modal-window padding-20"> <div class=modal-window-close-button></div> <div class=modal-window-title>Export Citation</div> <div class=modal-window-row> <div> <input type=radio name=export-citation-option value=WORKSPACE /> <span class=radio-label>Sciwheel</span> </div> <div> <input type=radio name=export-citation-option value=ENDNOTE /> <span class=radio-label>EndNote</span> </div> <div> <input type=radio name=export-citation-option value=REF_MANAGER /> <span class=radio-label>Ref. Manager</span> </div> <div> <input type=radio name=export-citation-option value=BIBTEX /> <span class=radio-label>Bibtex</span> </div> <div> <input type=radio name=export-citation-option value=PROCITE /> <span class=radio-label>ProCite</span> </div> <div> <input type=radio name=export-citation-option value=SENTE /> <span class=radio-label>Sente</span> </div> </div> <div class=modal-window-footer> <button class=general-white-orange-button id=export-citation-submit>EXPORT</button> </div> <div class=default-error style="display: none;">Select a format first</div> </div> </div> </div> <div class="article-interaction-info article-page"> <div class=article-interaction-button> <span class="f1r-icon icon-90_track"></span> <a class="button-link track-article" data-article-id=23224 id=track-article-signin-23224 title="Receive updates on new activity such as publication of new versions, peer reviews or author responses." href="/login?originalPath=/trackArticle/23224?target=/articles/9-295/v2">Track</a> </div> </div> <div class="article-interaction-info article-page"> <div class="article-interaction-button share-article-button"> <span class="f1r-icon icon-34_share"></span> <a href="#" class="button-link last share-article share-article-button" title="Share this article" data-windowref=share-article-popup-23224-2>Share</a> </div> <div id=share-article-popup-23224-2 class="popup-window-wrapper is-hidden"> <div class=cite-popup-background></div> <div class="popup-window top-popup share-this-article-box research-layout"> <div class=f1r-shares-wrapper> <a target=_blank class="f1r-shares-icon-square f1r-shares-twitter" title="Share on Twitter"></a> <a target=_blank class="f1r-shares-icon-square f1r-shares-facebook" title="Share on Facebook"></a> <a target=_blank class="f1r-shares-icon-square f1r-shares-linkedin" title="Share on LinkedIn"></a> <a target=_blank class="f1r-shares-icon-square f1r-shares-reddit" title="Share on Reddit"></a> <a target=_blank class="f1r-shares-icon-square f1r-shares-mendelay" title="Share on Mendeley"></a> </div> </div> </div> </div> </div> <div id=article-interaction-control-tab class=article-interaction-control-tab> <div id=hide-article-interaction class=article-interaction-control title="Hide Toolbox">▬</div> <div id=show-article-interaction class="article-interaction-control open" title="Show Toolbox">✚</div> </div> </div> <div class="article-header-information article-page"> <div class="f1r-article-mobile article-heading-bar"></div> <div class="article-type article-display">Opinion Article </div> <div class="article-title-and-info article-view highlighted-article js-article-title-container" id=anchor-title> <span title="A revision, usually following reviewer/reader feedback" class=" updated-box updated-box-bg orange-bg revised helper">Revised</span> <h1 class=js-article-title>An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action</h1> <div class="additional-info js-article-title-additional-info" id=anchor-title-additional-info> [version 2; peer review: 2 approved] </div> </div> <div class="article-subtitle js-article-subtitle"></div> <div class=f1r-article-desk> <div class="authors _mdl-layout js-article-authors"><span class=""><span class=js-article-author>Hartwig Anzt</span><sup>1,2</sup><sup>*</sup><span class=js-article-author-separator>, </span></span><span class=""><span class=js-article-author>Felix Bach</span><a href="https://orcid.org/0000-0002-5035-7978" target=_blank id=author-orcid-1><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-1><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-5035-7978</div><sup>1</sup><sup>*</sup><span class=js-article-author-separator>, </span></span><span class=""><span class=js-article-author>Stephan Druskat</span><a href="https://orcid.org/0000-0003-4925-7248" target=_blank id=author-orcid-2><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-2><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0003-4925-7248</div><sup>3-5</sup><sup>*</sup><span class=js-article-author-separator>, </span></span><a href="#" class=article-page-expand-authors>[...] </a><span class=article-page-hidden-authors><span class=js-article-author>Frank Löffler</span><a href="https://orcid.org/0000-0001-6643-6323" target=_blank id=author-orcid-3><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-3><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0001-6643-6323</div><sup>3,6</sup><sup>*</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><a href="mailto:axel.loewe@kit.edu" title="Send email" class="cauthor research-layout"><span class='f1r-icon icon-6_email orange'></span><span class=js-article-author>Axel Loewe</span></a><a href="https://orcid.org/0000-0002-2487-4744" target=_blank id=author-orcid-4><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-4><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-2487-4744</div><sup>1</sup><sup>*</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Bernhard Y. Renard</span><sup>7</sup><sup>*</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><a href="mailto:gunnar.seemann@universitaets-herzzentrum.de" title="Send email" class="cauthor research-layout"><span class='f1r-icon icon-6_email orange'></span><span class=js-article-author>Gunnar Seemann</span></a><a href="https://orcid.org/0000-0001-7111-7992" target=_blank id=author-orcid-6><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-6><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0001-7111-7992</div><sup>8</sup><sup>*</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Alexander Struck</span><a href="https://orcid.org/0000-0002-1173-9228" target=_blank id=author-orcid-7><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-7><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-1173-9228</div><sup>5</sup><sup>*</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Elke Achhammer</span><sup>9</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Piush Aggarwal</span><a href="https://orcid.org/0000-0003-1339-0549" target=_blank id=author-orcid-9><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-9><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0003-1339-0549</div><sup>10</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Franziska Appel</span><sup>11</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Michael Bader</span><sup>9</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Lutz Brusch</span><a href="https://orcid.org/0000-0003-0137-5106" target=_blank id=author-orcid-12><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-12><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0003-0137-5106</div><sup>12</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Christian Busse</span><a href="https://orcid.org/0000-0001-7553-905X" target=_blank id=author-orcid-13><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-13><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0001-7553-905X</div><sup>13</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Gerasimos Chourdakis</span><a href="https://orcid.org/0000-0002-3977-1385" target=_blank id=author-orcid-14><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-14><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-3977-1385</div><sup>9</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Piotr Wojciech Dabrowski</span><a href="https://orcid.org/0000-0003-4893-805X" target=_blank id=author-orcid-15><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-15><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0003-4893-805X</div><sup>14</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Peter Ebert</span><sup>15</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Bernd Flemisch</span><sup>16</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Sven Friedl</span><a href="https://orcid.org/0000-0002-9345-1593" target=_blank id=author-orcid-18><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-18><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-9345-1593</div><sup>17</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Bernadette Fritzsch</span><a href="https://orcid.org/0000-0002-0690-7151" target=_blank id=author-orcid-19><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-19><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-0690-7151</div><sup>18</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Maximilian D. Funk</span><sup>19</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Volker Gast</span><sup>3</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Florian Goth</span><sup>20</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Jean-Noël Grad</span><a href="https://orcid.org/0000-0002-5821-4912" target=_blank id=author-orcid-23><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-23><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-5821-4912</div><sup>16</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Jan Hegewald</span><a href="https://orcid.org/0000-0002-3675-0968" target=_blank id=author-orcid-24><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-24><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-3675-0968</div><sup>18</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Sibylle Hermann</span><sup>16</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Florian Hohmann</span><sup>21</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Stephan Janosch</span><sup>22</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Dominik Kutra</span><a href="https://orcid.org/0000-0003-4202-3908" target=_blank id=author-orcid-28><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-28><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0003-4202-3908</div><sup>23</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Jan Linxweiler</span><a href="https://orcid.org/0000-0002-2755-5087" target=_blank id=author-orcid-29><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-29><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-2755-5087</div><sup>24</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Thilo Muth</span><a href="https://orcid.org/0000-0001-8304-2684" target=_blank id=author-orcid-30><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-30><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0001-8304-2684</div><sup>25</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Wolfgang Peters-Kottig</span><a href="https://orcid.org/0000-0003-4486-2422" target=_blank id=author-orcid-31><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-31><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0003-4486-2422</div><sup>26</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Fabian Rack</span><sup>27</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Fabian H.C. Raters</span><a href="https://orcid.org/0000-0003-0549-6167" target=_blank id=author-orcid-33><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-33><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0003-0549-6167</div><sup>28</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Stephan Rave</span><a href="https://orcid.org/0000-0003-0439-7212" target=_blank id=author-orcid-34><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-34><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0003-0439-7212</div><sup>29</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Guido Reina</span><a href="https://orcid.org/0000-0003-4127-1897" target=_blank id=author-orcid-35><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-35><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0003-4127-1897</div><sup>16</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Malte Reißig</span><a href="https://orcid.org/0000-0002-8830-0758" target=_blank id=author-orcid-36><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-36><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-8830-0758</div><sup>30</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Timo Ropinski</span><sup>31,32</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Joerg Schaarschmidt</span><sup>1</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Heidi Seibold</span><a href="https://orcid.org/0000-0002-8960-9642" target=_blank id=author-orcid-39><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-39><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-8960-9642</div><sup>33</sup><span class=js-article-author-separator>, </span></span><span class=article-page-hidden-authors><span class=js-article-author>Jan P. Thiele</span><a href="https://orcid.org/0000-0002-8901-6660" target=_blank id=author-orcid-40><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-40><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-8901-6660</div><sup>34</sup><span class=js-article-author-separator>, </span></span><span class=""><span class=js-article-author>Benjamin Uekermann</span><a href="https://orcid.org/0000-0002-1314-9969" target=_blank id=author-orcid-41><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=author-orcid-41><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-1314-9969</div><sup>35</sup><span class=js-article-author-separator>, </span></span><span class=""><span class=js-article-author>Stefan Unger</span><sup>36</sup><span class=js-article-author-separator>, </span></span><span class=""><span class=js-article-author>Rudolf Weeber</span><sup>16</sup></span><span class="author-display-control article-page-expand-authors f1r-icon icon-20_plus_big"></span><span class="author-display-control article-page-hidden-authors f1r-icon icon-12_minus_big"></span></div> </div> <div class=f1r-article-mobile> <div class="authors _mdl-layout js-article-authors"><span class=""><span class=js-article-author>Hartwig Anzt</span><sup>1,2</sup><sup>*</sup>, </span><span class=""><span class=js-article-author>Felix Bach</span><a href="http://orcid.org/0000-0002-5035-7978" target=_blank id=mauthor-orcid-1><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-1><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-5035-7978</div><sup>1</sup><sup>*</sup>, </span><a href="#" class=article-page-expand-authors>[...] </a><span class=article-page-hidden-authors><span class=js-article-author>Stephan Druskat</span><a href="http://orcid.org/0000-0003-4925-7248" target=_blank id=mauthor-orcid-2><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-2><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0003-4925-7248</div><sup>3-5</sup><sup>*</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Frank Löffler</span><a href="http://orcid.org/0000-0001-6643-6323" target=_blank id=mauthor-orcid-3><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-3><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0001-6643-6323</div><sup>3,6</sup><sup>*</sup>, </span><span class=article-page-hidden-authors><a href="mailto:axel.loewe@kit.edu" title="Send email" class="cauthor research-layout js-article-author"><span class='f1r-icon icon-6_email orange'></span><span>Axel Loewe</span></a><a href="http://orcid.org/0000-0002-2487-4744" target=_blank id=mauthor-orcid-4><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-4><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-2487-4744</div><sup>1</sup><sup>*</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Bernhard Y. Renard</span><sup>7</sup><sup>*</sup>, </span><span class=article-page-hidden-authors><a href="mailto:gunnar.seemann@universitaets-herzzentrum.de" title="Send email" class="cauthor research-layout js-article-author"><span class='f1r-icon icon-6_email orange'></span><span>Gunnar Seemann</span></a><a href="http://orcid.org/0000-0001-7111-7992" target=_blank id=mauthor-orcid-6><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-6><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0001-7111-7992</div><sup>8</sup><sup>*</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Alexander Struck</span><a href="http://orcid.org/0000-0002-1173-9228" target=_blank id=mauthor-orcid-7><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-7><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-1173-9228</div><sup>5</sup><sup>*</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Elke Achhammer</span><sup>9</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Piush Aggarwal</span><a href="http://orcid.org/0000-0003-1339-0549" target=_blank id=mauthor-orcid-9><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-9><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0003-1339-0549</div><sup>10</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Franziska Appel</span><sup>11</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Michael Bader</span><sup>9</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Lutz Brusch</span><a href="http://orcid.org/0000-0003-0137-5106" target=_blank id=mauthor-orcid-12><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-12><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0003-0137-5106</div><sup>12</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Christian Busse</span><a href="http://orcid.org/0000-0001-7553-905X" target=_blank id=mauthor-orcid-13><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-13><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0001-7553-905X</div><sup>13</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Gerasimos Chourdakis</span><a href="http://orcid.org/0000-0002-3977-1385" target=_blank id=mauthor-orcid-14><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-14><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-3977-1385</div><sup>9</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Piotr Wojciech Dabrowski</span><a href="http://orcid.org/0000-0003-4893-805X" target=_blank id=mauthor-orcid-15><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-15><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0003-4893-805X</div><sup>14</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Peter Ebert</span><sup>15</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Bernd Flemisch</span><sup>16</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Sven Friedl</span><a href="http://orcid.org/0000-0002-9345-1593" target=_blank id=mauthor-orcid-18><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-18><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-9345-1593</div><sup>17</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Bernadette Fritzsch</span><a href="http://orcid.org/0000-0002-0690-7151" target=_blank id=mauthor-orcid-19><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-19><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-0690-7151</div><sup>18</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Maximilian D. Funk</span><sup>19</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Volker Gast</span><sup>3</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Florian Goth</span><sup>20</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Jean-Noël Grad</span><a href="http://orcid.org/0000-0002-5821-4912" target=_blank id=mauthor-orcid-23><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-23><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-5821-4912</div><sup>16</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Jan Hegewald</span><a href="http://orcid.org/0000-0002-3675-0968" target=_blank id=mauthor-orcid-24><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-24><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-3675-0968</div><sup>18</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Sibylle Hermann</span><sup>16</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Florian Hohmann</span><sup>21</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Stephan Janosch</span><sup>22</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Dominik Kutra</span><a href="http://orcid.org/0000-0003-4202-3908" target=_blank id=mauthor-orcid-28><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-28><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0003-4202-3908</div><sup>23</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Jan Linxweiler</span><a href="http://orcid.org/0000-0002-2755-5087" target=_blank id=mauthor-orcid-29><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-29><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-2755-5087</div><sup>24</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Thilo Muth</span><a href="http://orcid.org/0000-0001-8304-2684" target=_blank id=mauthor-orcid-30><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-30><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0001-8304-2684</div><sup>25</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Wolfgang Peters-Kottig</span><a href="http://orcid.org/0000-0003-4486-2422" target=_blank id=mauthor-orcid-31><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-31><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0003-4486-2422</div><sup>26</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Fabian Rack</span><sup>27</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Fabian H.C. Raters</span><a href="http://orcid.org/0000-0003-0549-6167" target=_blank id=mauthor-orcid-33><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-33><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0003-0549-6167</div><sup>28</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Stephan Rave</span><a href="http://orcid.org/0000-0003-0439-7212" target=_blank id=mauthor-orcid-34><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-34><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0003-0439-7212</div><sup>29</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Guido Reina</span><a href="http://orcid.org/0000-0003-4127-1897" target=_blank id=mauthor-orcid-35><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-35><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0003-4127-1897</div><sup>16</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Malte Reißig</span><a href="http://orcid.org/0000-0002-8830-0758" target=_blank id=mauthor-orcid-36><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-36><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-8830-0758</div><sup>30</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Timo Ropinski</span><sup>31,32</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Joerg Schaarschmidt</span><sup>1</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Heidi Seibold</span><a href="http://orcid.org/0000-0002-8960-9642" target=_blank id=mauthor-orcid-39><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-39><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-8960-9642</div><sup>33</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Jan P. Thiele</span><a href="http://orcid.org/0000-0002-8901-6660" target=_blank id=mauthor-orcid-40><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-40><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-8901-6660</div><sup>34</sup>, </span><span class=article-page-hidden-authors><span class=js-article-author>Benjamin Uekermann</span><a href="http://orcid.org/0000-0002-1314-9969" target=_blank id=mauthor-orcid-41><span class=orcid-logo-for-author-list></span></a><div class="mdl-tooltip mdl-tooltip--wider" for=mauthor-orcid-41><span class=orcid-logo-for-author-list></span> https://orcid.org/0000-0002-1314-9969</div><sup>35</sup>, </span><span class=""><span class=js-article-author>Stefan Unger</span><sup>36</sup>, </span><span class=""><span class=js-article-author>Rudolf Weeber</span><sup>16</sup></span><span class="author-display-control article-page-expand-authors f1r-icon icon-20_plus_big"></span><span class="author-display-control article-page-hidden-authors f1r-icon icon-12_minus_big"></span></div> </div> <div class="authors extra-info js-authors-extra-info"> <sup>*</sup> <span class=js-authors-extra-info-equal>Equal contributors</span> </div> <div class=f1r-article-mobile> <div class=article-pubinfo-mobile> PUBLISHED 26 Jan 2021 </div> </div> <span class=Z3988 title="ctx_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft_id=info:doi/10.12688%2Ff1000research.23224.2"></span> <div class=f1r-article-desk> <div class="contracted-details first"> <a href="#" class="contracted-details-label author-affiliations"><span class=contracted></span>Author details</a> <a href="#" class=section-title>Author details</a> <span class="f1r-icon icon-14_more_small section-control"></span> <span class="f1r-icon icon-10_less_small section-control"></span> <div class="expanded-details affiliations is-hidden js-article-affiliations"> <sup>1</sup> <span class=js-article-affiliation>Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany</span><br/> <sup>2</sup> <span class=js-article-affiliation>University of Tennessee, Knoxville, TN, USA</span><br/> <sup>3</sup> <span class=js-article-affiliation>Friedrich Schiller University, Jena, Germany</span><br/> <sup>4</sup> <span class=js-article-affiliation>German Aerospace Center (DLR), Berlin, Germany</span><br/> <sup>5</sup> <span class=js-article-affiliation>Humboldt-Universität zu Berlin, Berlin, Germany</span><br/> <sup>6</sup> <span class=js-article-affiliation>Louisiana State University, Baton Rouge, LA, USA</span><br/> <sup>7</sup> <span class=js-article-affiliation>Hasso Plattner Institute, Potsdam, Germany</span><br/> <sup>8</sup> <span class=js-article-affiliation>University Heart Centre Freiburg Bad Krozingen, Freiburg, Germany</span><br/> <sup>9</sup> <span class=js-article-affiliation>Technische Universität München, München, Germany</span><br/> <sup>10</sup> <span class=js-article-affiliation>Universität Duisburg-Essen, Duisburg, Germany</span><br/> <sup>11</sup> <span class=js-article-affiliation>Leibniz Institute of Agricultural Development in Transition Economies (IAMO), Halle (Saale), Germany</span><br/> <sup>12</sup> <span class=js-article-affiliation>Technische Universität Dresden, Dresden, Germany</span><br/> <sup>13</sup> <span class=js-article-affiliation>Deutsches Krebsforschungszentrum, Heidelberg, Germany</span><br/> <sup>14</sup> <span class=js-article-affiliation>Hochschule für Technik und Wirtschaft Berlin, Berlin, Germany</span><br/> <sup>15</sup> <span class=js-article-affiliation>Saarland Informatics Campus, Saarbrücken, Germany</span><br/> <sup>16</sup> <span class=js-article-affiliation>University of Stuttgart, Stuttgart, Germany</span><br/> <sup>17</sup> <span class=js-article-affiliation>Berlin Institute of Health, Berlin, Germany</span><br/> <sup>18</sup> <span class=js-article-affiliation>Alfred Wegener Institute, Bremerhaven, Germany</span><br/> <sup>19</sup> <span class=js-article-affiliation>Max-Planck-Gesellschaft e.V., München, Germany</span><br/> <sup>20</sup> <span class=js-article-affiliation>Universität Würzburg, Würzburg, Germany</span><br/> <sup>21</sup> <span class=js-article-affiliation>Universität Bremen, Bremen, Germany</span><br/> <sup>22</sup> <span class=js-article-affiliation>Max Planck Institute of Molecular Cell Biology and Genetics, Dresden, Germany</span><br/> <sup>23</sup> <span class=js-article-affiliation>European Molecular Biology Laboratory, Heidelberg, Germany</span><br/> <sup>24</sup> <span class=js-article-affiliation>Technische Universität Braunschweig, Braunschweig, Germany</span><br/> <sup>25</sup> <span class=js-article-affiliation>Federal Institute for Materials Research and Testing, Berlin, Germany</span><br/> <sup>26</sup> <span class=js-article-affiliation>Konrad-Zuse-Zentrum für Informationstechnik Berlin (ZIB), Berlin, Germany</span><br/> <sup>27</sup> <span class=js-article-affiliation>FIZ Karlsruhe - Leibniz Institute for Information Infrastructure, Karlsruhe, Germany</span><br/> <sup>28</sup> <span class=js-article-affiliation>University of Goettingen, Göttingen, Germany</span><br/> <sup>29</sup> <span class=js-article-affiliation>University of Münster, Münster, Germany</span><br/> <sup>30</sup> <span class=js-article-affiliation>Institute for Advanced Sustainability Studies, Potsdam, Germany</span><br/> <sup>31</sup> <span class=js-article-affiliation>Ulm University, Ulm, Germany</span><br/> <sup>32</sup> <span class=js-article-affiliation>Linköping University, Linköping, Sweden</span><br/> <sup>33</sup> <span class=js-article-affiliation>Ludwig Maximilian University of Munich, München, Germany</span><br/> <sup>34</sup> <span class=js-article-affiliation>Leibniz University Hannover, Hannover, Germany</span><br/> <sup>35</sup> <span class=js-article-affiliation>Eindhoven University of Technology, Eindhoven, The Netherlands</span><br/> <sup>36</sup> <span class=js-article-affiliation>Julius Kühn-Institut (JKI), Quedlinburg, Germany</span><br/> <p> <div class=margin-bottom> Hartwig Anzt <br/> <span>Roles: </span> Conceptualization, Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Felix Bach <br/> <span>Roles: </span> Conceptualization, Investigation, Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Stephan Druskat <br/> <span>Roles: </span> Conceptualization, Investigation, Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Frank Löffler <br/> <span>Roles: </span> Conceptualization, Investigation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Axel Loewe <br/> <span>Roles: </span> Conceptualization, Funding Acquisition, Investigation, Project Administration, Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Bernhard Y. Renard <br/> <span>Roles: </span> Conceptualization, Investigation, Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Gunnar Seemann <br/> <span>Roles: </span> Conceptualization, Funding Acquisition, Investigation, Project Administration, Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Alexander Struck <br/> <span>Roles: </span> Conceptualization, Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Elke Achhammer <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Piush Aggarwal <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Franziska Appel <br/> <span>Roles: </span> Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Michael Bader <br/> <span>Roles: </span> Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Lutz Brusch <br/> <span>Roles: </span> Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Christian Busse <br/> <span>Roles: </span> Writing – Review & Editing </div> <div class=margin-bottom> Gerasimos Chourdakis <br/> <span>Roles: </span> Writing – Review & Editing </div> <div class=margin-bottom> Piotr Wojciech Dabrowski <br/> <span>Roles: </span> Writing – Review & Editing </div> <div class=margin-bottom> Peter Ebert <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Bernd Flemisch <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Sven Friedl <br/> <span>Roles: </span> Visualization, Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Bernadette Fritzsch <br/> <span>Roles: </span> Writing – Review & Editing </div> <div class=margin-bottom> Maximilian D. Funk <br/> <span>Roles: </span> Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Volker Gast <br/> <span>Roles: </span> Writing – Review & Editing </div> <div class=margin-bottom> Florian Goth <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Jean-Noël Grad <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Jan Hegewald <br/> <span>Roles: </span> Writing – Review & Editing </div> <div class=margin-bottom> Sibylle Hermann <br/> <span>Roles: </span> Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Florian Hohmann <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Stephan Janosch <br/> <span>Roles: </span> Writing – Review & Editing </div> <div class=margin-bottom> Dominik Kutra <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Jan Linxweiler <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Thilo Muth <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Wolfgang Peters-Kottig <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Fabian Rack <br/> <span>Roles: </span> Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Fabian H.C. Raters <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Stephan Rave <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Guido Reina <br/> <span>Roles: </span> Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Malte Reißig <br/> <span>Roles: </span> Writing – Review & Editing </div> <div class=margin-bottom> Timo Ropinski <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Joerg Schaarschmidt <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Heidi Seibold <br/> <span>Roles: </span> Writing – Review & Editing </div> <div class=margin-bottom> Jan P. Thiele <br/> <span>Roles: </span> Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Benjamin Uekermann <br/> <span>Roles: </span> Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Stefan Unger <br/> <span>Roles: </span> Visualization, Writing – Original Draft Preparation </div> <div class=margin-bottom> Rudolf Weeber <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> </p> </div> </div> </div> <div class=f1r-article-mobile> <div class="article-page-section-box margin-bottom-40 research-layout"> <span class=box-title> <span class="f1r-icon icon-85_peer_review"></span> OPEN PEER REVIEW </span> <button class="tertiary grey float-right" data-scrollto=article-reports>DETAILS</button> <div class="status-row referee-reports-container"> REVIEWER STATUS <span class=status-icons> <span class="f1r-icon icon-86_approved status-green smaller" title=Approved data-refInfo=66679-78211></span> <span class="f1r-icon icon-86_approved status-green smaller" title=Approved data-refInfo=66678-62872></span> </span> </div> </div> </div> <div class="f1r-article-desk padding-top"> <div class=article-collection-wrapper> <div class="o-media o-media--sm o-media--center"> <div class=o-media__img> <a href="https://f1000research.com/gateways/research_on_research/about-this-gateway" title="Open Gateway"><img src="https://f1000research.s3.amazonaws.com/thumbnails/e463ad88-80f5-41e8-90e6-d1fb24a94c5e_collection.thumbnail"></a> </div> <div class=o-media__body> <p class="u-mt--0 u-mb--0">This article is included in the <a href="https://f1000research.com/gateways/research_on_research/about-this-gateway">Research on Research, Policy & Culture</a> gateway.</p> </div> </div> </div> <div class=article-collection-wrapper> <div class="o-media o-media--sm o-media--center"> <div class=o-media__img> <a href="https://f1000research.com/collections/maxplancksociety/about-this-collection" title="Open Collection"><img src="https://f1000research.s3.amazonaws.com/thumbnails/781fa3fe-8200-46c8-8667-7a7902750949_collection.thumbnail"></a> </div> <div class=o-media__body> <p class="u-mt--0 u-mb--0">This article is included in the <a href="https://f1000research.com/collections/maxplancksociety/about-this-collection">Max Planck Society</a> collection.</p> </div> </div> </div> </div> </div> <div id=anchor-abstract> <h2 class="article-headings article-page-abstract"> <span class="f1r-article-mobile-inline abstract-heading-border"></span> <span>Abstract</span> </h2> <div class="article-abstract article-page-general-text-mobile research-layout"> <div class="abstract-text is-expanded"> <div>Research software has become a central asset in academic research. It optimizes existing and enables new research methods, implements and embeds research knowledge, and constitutes an essential research product in itself. Research software must be sustainable in order to understand, replicate, reproduce, and build upon existing research or conduct new research effectively. In other words, software must be available, discoverable, usable, and adaptable to new needs, both now and in the future. Research software therefore requires an environment that supports sustainability.<br/> <br/> Hence, a change is needed in the way research software development and maintenance are currently motivated, incentivized, funded, structurally and infrastructurally supported, and legally treated. Failing to do so will threaten the quality and validity of research. In this paper, we identify challenges for research software sustainability in Germany and beyond, in terms of motivation, selection, research software engineering personnel, funding, infrastructure, and legal aspects. Besides researchers, we specifically address political and academic decision-makers to increase awareness of the importance and needs of sustainable research software practices. In particular, we recommend strategies and measures to create an environment for sustainable research software, with the ultimate goal to ensure that software-driven research is valid, reproducible and sustainable, and that software is recognized as a first class citizen in research. This paper is the outcome of two workshops run in Germany in 2019, at deRSE19 - the first International Conference of Research Software Engineers in Germany - and a dedicated DFG-supported follow-up workshop in Berlin.</div> </div> <div class=abstract-for-mobile> <div class="margin-top-30 padding-bottom-30 research-layout is-centered"> <button class="primary orange-text white-bg bigger-text abstract-expand-button-mobile with-border show" style="display: none;"> READ ALL <span class="f1r-icon icon-14_more_small orange vmiddle big"></span> </button> <button class="primary orange-text white-bg bigger-text abstract-expand-button-mobile with-border hide"> READ LESS <span class="f1r-icon icon-10_less_small orange vmiddle big"></span> </button> </div> </div> </div> </div> <div class=clearfix></div> <div class="article-context no-divider"> <div class="article-abstract article-page-general-text-mobile research-layout generated-article-body"> <h2 class=main-title>Keywords</h2> <p class="u-mb--0 u-pb--2"> Sustainable Software Development, Academic Software, Software Infrastructure, Software Training, Software Licensing, Research Software </p> </div> </div> <div class=article-information> <span class="info-separation padding-bottom"> <div id=corresponding-author-icon class="email-icon float-left"> <span class="f1r-icon icon-6_email orange"></span> <div id=corresponding-author-window class="margin-top-20 popup-window-wrapper is-hidden"> <div class="popup-window corresponding-authors-popup"> <div class=corresponding-author-container> <div class="popup-window-title small">Corresponding Author(s)</div> <div class=authors> Axel Loewe (<a href="mailto:axel.loewe@kit.edu">axel.loewe@kit.edu</a>) <br> Gunnar Seemann (<a href="mailto:gunnar.seemann@universitaets-herzzentrum.de">gunnar.seemann@universitaets-herzzentrum.de</a>) </div> </div> <div class="margin-top margin-bottom float-left"> <button id=close-popup-window class=general-white-orange-button>Close</button> </div> </div> </div> </div> <span class="icon-text float-left" data-test-id=box-corresponding-author> <b>Corresponding authors:</b> Axel Loewe, Gunnar Seemann </span> <div class=clearfix></div> </span> <span class="info-separation padding-bottom competing-interests-display"> <span class=competing-interests-title>Competing interests:</span> No competing interests were disclosed. </span> <div class="info-separation padding-bottom grant-information-display"> <span class=grant-information-title>Grant information:</span> The authors thank the Deutsche Forschungsgemeinschaft (DFG) for funding a meeting (Rundgespräch, grants LO 2093/3-1 and SE 1758/6-1) during which the initial draft of this paper has been created. We are particularly grateful for the support from Dr. Matthias Katerbow (DFG). This work was additionally supported by Research Software Sustainability grants funded by the DFG: Aggarwal: 390886566; PI: Zesch. Appel: 391099391; PI: Balmann. Bach & Loewe & Seemann: 391128822; PIs: Loewe/Scholze/Seemann/Selzer/Streit/Upmeier.Bader: 391134334; PIs: Bader/Gabriel/Frank. Brusch: 391070520; PI: Brusch. Druskat & Gast: 391160252; PI: Gast/Lüdeling. Ebert: 391137747; PI: Marschall.Flemisch & Hermann: 391049448; PIs: Boehringer/Flemisch/Hermann.Hohmann: 391054082; PI: Hepp. Goth: 390966303; PI: Assaad. Grad & Weeber: 391126171; PI: Holm. Kutra: 391125810; PI: Kreshuk.Mehl & Uekermann: 391150578; PIs: Bungartz/Mehl/Uekermann. Peters-Kottig: 391087700; PIs: Gleixner/Peters-Kottig/Shinano/Sperber. Raters: 39099699; PI:Herwartz. Reina: 391302154; PIs: Ertl/Reina. Muth & Renard: 391179955; PIs Renard/Fuchs. Ropinski:391107954; PI: Ropinski.<br>Alexander Struck acknowledges the support of the Cluster of Excellence Matters of Activity. Image Space Material funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany´s Excellence Strategy – EXC 2025.<br>We acknowledge support by the KIT-Publication Fund of the Karlsruhe Institute of Technology. <br/> <i>The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.</i> </div> <span class="f1r-article-desk info-separation padding-bottom"> <span class="copywrite-icon float-left"> <span class="f1r-icon icon-100_open_access"></span> </span> <span class="icon-text float-left" data-test-id=box-copyright-text> <b>Copyright:</b> © 2021 Anzt H <em>et al</em>. This is an open access article distributed under the terms of the <a href="https://creativecommons.org/licenses/by/4.0/" target=_blank data-test-id=box-licence-link>Creative Commons Attribution License</a>, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. </span> <div class=clearfix></div> </span> <span class="info-separation padding-bottom" data-test-id=box-how-to-cite> <b>How to cite:</b> <span class="article-title-and-info in-article-box"> Anzt H, Bach F, Druskat S <em>et al.</em> An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action [version 2; peer review: 2 approved]</span>. <i>F1000Research</i> 2021, <b>9</b>:295 (<a href="https://doi.org/10.12688/f1000research.23224.2" target=_blank>https://doi.org/10.12688/f1000research.23224.2</a>) </span> <span class=info-separation data-test-id=box-first-published><b>First published:</b> 27 Apr 2020, <b>9</b>:295 (<a href="https://doi.org/10.12688/f1000research.23224.1" target=_blank>https://doi.org/10.12688/f1000research.23224.1</a>)</span> <span class=info-separation data-test-id=box-latest-published><b>Latest published:</b> 26 Jan 2021, <b>9</b>:295 (<a href="https://doi.org/10.12688/f1000research.23224.2" target=_blank>https://doi.org/10.12688/f1000research.23224.2</a>)</span> </div> <div class="update-box-wrapper update-text-view-control article-updated-box "> <h2 class="h2-title no-top-margin"> <span title="A revision, usually following reviewer/reader feedback" class=" updated-box updated-box-bg orange-bg revised ">Revised</span> Amendments from Version 1 <span class="research-layout f1r-article-mobile-inline update-box-control"> <span class="f1r-icon icon-14_more_small big midgrey"></span> <span class="f1r-icon icon-10_less_small big midgrey" style="display: none;"></span> </span> </h2> <div class=f1r-article-desk> <p>Besides fixing some typographic errors and adding references as suggested by the reviewers, <br/> we separated the legal decision tress from this manuscript. As they were not the focus of this position paper and diluted its key messages, they were published separately: <a href="https://protect-us.mimecast.com/s/xw4bC0R296f29L4NAi28JlY?domain=doi.org">https://doi.org/</a><a href="https://protect-us.mimecast.com/s/qSPMC82o95fjAKQNyhnn8EO?domain=doi.org">10.5281/zenodo.4327147</a><br/> Other aspects that were elaborated on include: testing, infrastructure for cross-institutional use, sustainable funding, the relation between software quality and transparency, a clear statement pro open source, the potential role of legal help desks. <br/> <br/> Jan Hegewald was added to the list of authors. He already contributed to initial submission but unfortunately his name was missing in the list.</p> </div> <div class="f1r-article-mobile research-layout article-page-general-text-mobile padding-top-20" style="display: none;"> Besides fixing some typographic errors and adding references as suggested by the reviewers, <br/> we separated the legal decision tress from this manuscript. As they were not the focus of this position paper and diluted its key messages, they were published separately: <a href="https://protect-us.mimecast.com/s/xw4bC0R296f29L4NAi28JlY?domain=doi.org">https://doi.org/</a><a href="https://protect-us.mimecast.com/s/qSPMC82o95fjAKQNyhnn8EO?domain=doi.org">10.5281/zenodo.4327147</a><br/> Other aspects that were elaborated on include: testing, infrastructure for cross-institutional use, sustainable funding, the relation between software quality and transparency, a clear statement pro open source, the potential role of legal help desks. <br/> <br/> Jan Hegewald was added to the list of authors. He already contributed to initial submission but unfortunately his name was missing in the list. </div> <p class=f1r-article-desk> <a href="https://f1000research.com/articles/9-295#referee-response-62873" data-path="/reports/62873/responses" class="black-link bold">See the authors' detailed response to the review by Wilhelm Hasselbring</a><br/> <a href="https://f1000research.com/articles/9-295#referee-response-62872" data-path="/reports/62872/responses" class="black-link bold">See the authors' detailed response to the review by Radovan Bast</a><br/> </p> <span class=f1r-article-mobile> <div class="margin-top-30 margin-bottom research-layout is-centered" style="display: none;"> <a href="#article-reports"> <button class="primary orange extra-padding">READ REVIEWER RESPONSES</button> </a> </div> </span> </div> <div class=clearfix></div> <div id=article-context class=article-context> <div id=article1-body class=generated-article-body><h2 class=main-title id=d11061e1018>Background</h2><a name=B1 class=n-a></a><div class="boxed-text panel clearfix"><div class=caption><h3></h3></div><p class=first id=d11061e1023>Meet Kim, who is currently a post-grad PhD student in researchonomy at the University of Arcadia (UofA). We will follow Kim’s fictional career in order to understand different aspects of research software sustainability. Note that in Kim’s world, many of the changes this paper calls for have already been implemented. (In our example, Kim is a female person. Of course, research software engineers (RSEs) can be of any gender.)</p></div><p class="" id=d11061e1027>Computational analysis of large data sets, computer-based simulations, and software technology in general play a central role for virtually all scientific breakthroughs of at least the 21st century. The first image of a black hole may be the most prominent recent example where astrophysical experiments and the collection and processing of data had to be complemented with sophisticated algorithms and software to enable research excellence<sup><a href="#ref-1">1</a>,<a href="#ref-2">2</a></sup>. Similarly, it is research software that allows us to get a glimpse of the consequences our actions today have on the climate of tomorrow. However, an implication of computer-based research is that findings and data can only be reproduced, understood, and validated if the software that was used in the research process is sustained and their functionality maintained.</p><p class="" id=d11061e1037>At the same time, sustaining research software, and in particular open research software, comes with a number of challenges. Commercial research software often has revenue flows that can facilitate sustainable software development, maintenance, and documentation as well as the operation of adequate infrastructure. However, a large share of researchers base their research on software that was developed in-house or as a community effort. Many of these software stacks can not be sustained – often because research software was not a first class deliverable in a research project and hence remained in a prototype state, or because of missing incentives and resources to maintain the software after project funding ended. Another fundamental difference to industrial software development is that most developers of academic research software (often doctoral students or postdoctoral researchers) never receive training in sustainable software development<sup><a href="#ref-3">3</a></sup>. In particular, as they see themselves usually as the primary user of a software product, there are virtually no incentives to invest in sustainability measures such as code documentation or portability. In combination with the predominance of temporary positions in research, this results in a highly inefficient system where millions of lines of code are generated every year that will not be re-used after the termination of the developer’s position. Part of the problem is the reluctance to accept research software engineering as an academic profession that results in a lack of incentives to produce high-quality software: producing high software quality needs sufficient resources, and although the <a target=xrefwindow href="http://www.sfdora.org/read" id=d11061e1043>San Francisco Declaration on Research Assessment (DORA)</a> demands a change in the academic credit system, many institutions base promotion and appointments on traditional metrics like the Hirsch index<sup><a href="#ref-4">4</a></sup>. It is obvious that an extraordinary amount of idealism is required to write sustainable code, including documentation and installation routines, as well as running infrastructure and giving support to others when resources can be used more profitably in writing scientific publications based on fragile prototype software<sup><a href="#ref-5">5</a>,<a href="#ref-6">6</a></sup>.</p><p class="" id=d11061e1058>Thus, one main factor for the poor sustainability of research software is the lack of long-term funding for research software engineers (RSEs)<sup><a href="#ref-7">7</a>,<a href="#ref-8">8</a></sup> who take care of the appropriate architecture, organization, implementation, documentation, and community interaction for the software, paired with the implementation of measures towards making the software sustainable during and beyond the development process<sup><a href="#ref-9">9</a></sup>.</p><p class="" id=d11061e1073>In this paper, we describe the state of the practice and current challenges for research software sustainability and suggest measures towards improvements that can solve these challenges. The paper is the result of a community effort, with work undertaken during two workshops and subsequent collaborative work across the larger RSE community in Germany. It has been initiated during a half-day workshop at <a target=xrefwindow href="http://de-rse.org/en/conf2019/" id=d11061e1075>first International Conference for Research Software Engineers in Germany (deRSE19)</a> in Potsdam, Germany on June 5th, 2019, and continued during a dedicated two-day workshop in Berlin, Germany on November 7th and 8th, 2019, which was funded by the German Research Foundation (Deutsche Forschungsgemeinschaft, DFG). Subsequently, the draft produced during the latter event was opened up for collaborative discussion by the German RSE community through <i><a target=xrefwindow href="https://de-rse.orgf" id=d11061e1079>de-RSE</a> e.V. - Society for Research Software</i>.</p><p class="" id=d11061e1084>We mainly focus on the situation of research software and RSEs in Germany, where funding bodies increasingly acknowledge the importance and value of sustainable research software and related infrastructures. The DFG, the largest funding body for fundamental research in Germany, for example, opened a <a target=xrefwindow href="https://www.dfg.de/foerderung/info_wissenschaft/2016/info_wissenschaft_16_71/" id=d11061e1086>call for sustainable research software development</a> at the end of 2016 and a second <a target=xrefwindow href="https://www.dfg.de/foerderung/info_wissenschaft/2019/info_wissenschaft_19_44/index.html" id=d11061e1089>call for quality management in research software</a> in June 2019. The first call was oversubscribed by a factor of 10-15, a strong indicator of unmet demand. As another example, the <a target=xrefwindow href="https://www.dfg.de/en/research_funding/principles_dfg_funding/good_scientific_practice/" id=d11061e1092>2019 “Guidelines for Safeguarding Good Research Practice” codex of the DFG</a> now explicitly lists software side-by-side with other research results and data. The FAIR principles for research data<sup><a href="#ref-10">10</a></sup> provide guidelines for data archiving, but enabling full reproducibility and traceability of research software requires additional steps<sup><a href="#ref-11">11</a></sup>. In consequence, there are ongoing discussions on whether software should be considered as a specific kind of research data or as a separate entity<sup><a href="#ref-12">12</a></sup>.</p><p class="" id=d11061e1109>These positive developments notwithstanding, guidelines and policies for sustainable research software development in Germany are unfortunately still lacking, and long-term funding strategies are missing. This all leads to unmet requirements and unsolved challenges that we want to highlight in this paper by elaborating on (1) why research software engineering needs to be considered an integral part of academic research; (2) how to decide which software to sustain; (3) who sustains research software; (4) how software can be funded sustainably; (5) what infrastructure is needed for sustainable software development; and (6) legal aspects of research software development in academia. While we specifically focus on the research software landscape in Germany, we are convinced that many of the analyses, findings, and recommendations may carry beyond. We want to address RSEs who are experiencing similar challenges and newcomers to the field of research software development, but first and foremost political and academic decision makers to raise awareness of the importance of and requirements for sustainable software development. As a community, we work hard on overcoming the challenges of software development in an academic setting, but we need support – and reliable funding options and institutional recognition in particular – for the sake of better research.</p></div><div id=article1-body class=generated-article-body><h2 class=main-title id=d11061e1115>Why sustainable research software in the first place?</h2><a name=B2 class=n-a></a><div class="boxed-text panel clearfix"><div class=caption><h3></h3></div><p class=first id=d11061e1120>After graduation, Kim joins a fixed-term researchonomical research project. For her PhD thesis, she wants to crunch some data. Her colleague recommends learning some Boa, which is an all-purpose programming language often used in researchonomy. Luckily, the UofA runs regular Software Plumbery courses for researchers, including a Boa course. Kim takes the course and gains a solid understanding of the basics of the Hash shell, version control with Tig, and the basics of Boa. She starts writing scripts, which help her a lot with the data processing. Unfortunately, Kim’s scripts are quite slow and actually break after she installs a newer version of Boa. She visits the weekly Code Café organized by her university’s central RSE team. The RSEs not only help her update her scripts but also suggest some changes which speed up the computation by a factor of 25.</p><p class="" id=d11061e1123>During the next meeting with her PhD supervisor, Kim presents her collection of scripts. The supervisor encourages Kim to create a Boa library from them, as they will be very useful to other researchonomists. Thankfully, Kim’s project PI had applied for three RSE person months in their grant, so the project enlists an RSE from the central team. Over the next three months, Kim and the RSE work together to build the library, document it, test it, license it under the permissive Comanche license, update the TigLab repository to let others contribute, introduce automated builds for every code change via a continuous integration platform, and make the library citable. Finally, they release the first major version of the library, named <i>hal9k</i> and publish it through the university library’s software portal, where they get a DOI (Digital Object Identifier) for the version as well as a concept DOI for any future versions of the library. Working with the RSE, Kim has gained a good understanding of some methods in software engineering, and she’s thrilled because this also means she’ll be able to get a job with a local tech company once her fixed-term contract has run out.</p><p class="" id=d11061e1129>Kim passes her PhD - of which <i>hal9k</i> is an important part - with flying colors, and soon citations to her library start appearing in the researchonomic literature. To Kim’s surprise, she also reads a blog post about a citizen science maker project which has used <i>hal9k</i> to process researchonomic data measured in a neighborhood of her hometown. She is invited to give a talk at the local office of Siren, a global tech company, which look to adopt <i>hal9k</i>, and pay Kim a generous speaker honorarium. So generous in fact, that Kim can pay a student assistant for a full year from the money.</p></div><p class="" id=d11061e1142>Our credibility as researchers in society hinges on the notion of proper research conduct, also known as “good research practice”. The digitalization of research has introduced complex digital research outputs, such as software and data sets. Although first recommendations<sup><a href="#ref-13">13</a></sup> and policies<sup><a href="#ref-14">14</a></sup> exist, they are far from being widely adopted. It is still somewhat unclear how to translate good research practice into good research software practice, for example in terms of validity and reproducibility, but also pertaining to the responsible use of resources. The damage that failing to do so is causing both to the progress of the research community and to the credibility of academic research in society is becoming increasingly clear with the growth of the replication crisis - while the lack of universally agreed-upon and supported good research software practice is not the main reason for that crisis, it clearly is a contributing factor.</p><p class="" id=d11061e1153>While it is obvious that software qualifies as a potentially re-usable digital artifact, the additional benefit of not just reproducing a given scenario, but transferring software use to new problems, domains, and/or applications, justifies developing research software with a long-term perspective as <i>sustainable research software</i>.</p><p class="" id=d11061e1159>In order to support research, a sustainable software must ideally be correct<sup><a href="#ref-15">15</a>–<a href="#ref-17">17</a></sup> and validatable. Due to the experimental nature of some research software, this may not be possible in all cases, e.g., due to lack, or infeasible implementation, of a test oracle<sup><a href="#ref-18">18</a></sup>, vast configuration spaces, or large and heterogeneous data inputs<sup><a href="#ref-19">19</a></sup>. While it must be accepted that precise, oracle-based testing may not be possible here, alternative solutions should be implemented, such as metamorphic testing, runtime assertions, test input sampling and generation (e.g. via machine learning), and input data modeling. Sustainable software must also be understandable, documented, publicly released, adequately published (i.e. in persistently identifiable form as software source code<sup><a href="#ref-20">20</a></sup>, and potentially in an additional paper which describes the software concept, design decisions, and development rationale), actively maintained, and (re-)usable<sup><a href="#ref-21">21</a>–<a href="#ref-23">23</a></sup>. We also argue that truly sustainable research software should ideally be published under a Free/Libre Open Source Software (FLOSS) license, and follow an open development model, to (1) enable the validation of research results that have been produced using the software, (2) enable the reproducibility of software-based research, (3) enable improvement and (re-) use of the software to support more and better research, and reduce resources to be spent on software development, (4) reduce legal issues (see section below), (5) meet ethical obligations from public funding, and (6) open research software to the general public, i.e., the stakeholder group with arguably the greatest interest in furthering research knowledge and improving research for the benefit of all.</p><p class="" id=d11061e1189>To make software-based research (and with that almost any research) reproducible, the used software must continue to exist. Furthermore, it must continue to be usable, understandable, and return consistent results (or potential changes to results and bug fixes must be clearly documented) in the evolving software and hardware environment. Moreover, the software should support reuse scenarios to avoid duplication of efforts and drain of resources. Therefore, if research software is publicly funded, it should be freely available under a FLOSS license.</p><p class="" id=d11061e1192>Currently, creating and using sustainable research software is not sufficiently incentivized. To evaluate in which area this shortcoming should be addressed, we have identified the following challenges: <div class=list><a name=d11061e1194 class=n-a></a><ul><li><p id=d11061e1198><b>Lack of benefit for the individual</b>: Currently, the primary motivation for sustainable research software is the common benefit, rather than the individual benefit. It is clearly beneficial for the research community as a whole to direct resources towards sustainable research software, as it enables better and more research by freeing funds for domain research rather than (repetitive) software development. But the developers are often even at a disadvantage (e.g., they publish fewer papers<sup><a href="#ref-5">5</a>,<a href="#ref-6">6</a></sup>), which in turn prevents sustainable research software.</p></li><li><p id=d11061e1213><b>Lack of suitable incentive systems</b>: Contributions to research that are not traditional text-based products (i.e., papers or monographs) are still not sufficiently rewarded, or not rewarded at all, due to the missing implementation of mandatory software citation<sup><a href="#ref-20">20</a>,<a href="#ref-24">24</a>–<a href="#ref-32">32</a></sup>, among other reasons. Interestingly, one third of research software repositories have a lifespan (defined as the time from the first time any code was uploaded to the last contribution) of less than one day (median: 15 days<sup><a href="#ref-11">11</a></sup>), indicating that many codes are only made available publicly for the publication in a journal (as increasingly encouraged or required by journals<sup><a href="#ref-33">33</a></sup> and associated with higher impact<sup><a href="#ref-34">34</a></sup>) but are not maintained thereafter.</p></li><li><p id=d11061e1243><b>Lack of awareness</b>: Research software sustainability and its importance is lacking visibility as well as acceptance<sup><a href="#ref-35">35</a>–<a href="#ref-38">38</a></sup>, and research software engineering in its implementation as sustainable software development and software maintenance is not sufficiently supported, both in Germany and beyond<sup><a href="#ref-9">9</a>,<a href="#ref-39">39</a>,<a href="#ref-40">40</a></sup>.</p></li><li><p id=d11061e1268><b>Lack of expertise</b>: Knowledge about how to create, maintain, and support sustainable research software is emerging<sup><a href="#ref-41">41</a>–<a href="#ref-43">43</a></sup> but has not yet permeated related activities within organizations - specifically teaching, mentoring, and consultancy. This lack of expertise can also lead to divergence between software design and community uptake, e.g., if the software fails to meet the needs of the target group, or is insufficiently usable. RSEs combine sustainable software engineering expertise with experience in one or more research domains.</p></li><li><p id=d11061e1283><b>Heterogeneous research community</b>: There are significant differences with respect to how software is developed, published, used, and valued in the different academic disciplines. Additionally, there is even heterogeneity within a community in terms of application and approach. This also makes it hard to train researchers for sustainable software development, as beyond basic training in computational research such as provided by <a target=xrefwindow href="https://carpentries.org" id=d11061e1287>The Carpentries</a>, advanced courses for research software engineering are not widely available (with the notable exception of the <a target=xrefwindow href="http://coderefinery.org/lessons/" id=d11061e1290>CodeRefinery project</a>). Targeted curricula must be developed and updated regularly, and specialized instructors need to be trained.</p></li><li><p id=d11061e1298><b>Lack of impact measures</b>: It is unclear how to measure the impact of research software with respect to its quality, reusability, and benefit for the research community. This exceeds the implementation of research software citation (which is work in progress<sup><a href="#ref-20">20</a>,<a href="#ref-31">31</a>,<a href="#ref-32">32</a>,<a href="#ref-44">44</a></sup>), and pertains to sustainability and policy studies.</p></li><li><p id=d11061e1319><b>Infrastructure issues</b>: Due to a lack of knowledge about how sustainability features impact the application of research software, there is not yet enough evidence for whether centralized or decentralized facilities should be favored to further research software sustainability<sup><a href="#ref-45">45</a>–<a href="#ref-47">47</a></sup>. Commonly, local infrastructure hinders cross-institutional collaboration, whereas cross-organizational infrastructures often suffer from lack of authentication and authorization implementations, or legal constraints. This in turn leads to a lack of infrastructure as a whole.</p></li><li><p id=d11061e1334><b>Legal issues</b>: Many obstacles for research software pertain to legal issues, such as applicable licensing and compatibility of licenses<sup><a href="#ref-48">48</a></sup>, and decisions about license types.</p></li><li><p id=d11061e1346><b>Funding issues</b>: Despite some individual initiatives<sup><a href="#ref-49">49</a>–<a href="#ref-52">52</a></sup>, funding for the creation, maintenance, and support of sustainable research software is still scarce. Additionally, existing models usually supply seed funding only, which disregards the support and maintenance steps in the software development lifecycle. Instead, a potential “market” is relied upon to support these, which may only develop long after the initial project has ended. With regard to the funding of infrastructure which underpins modern development approaches such as DevOps and continuous deployment, cloud infrastructure providers and their pricing models do not work well with current funding models, due to lack of knowledge of how to target them with traditional academic funding and budgeting, compliance issues, or rigid bureaucracy.</p></li><li><p id=d11061e1361><b>Slow adoption of research software engineering as a profession</b>: Career options for research software work are not fully determined, although career paths are emerging in some regions. Initially, the RSE initiative in the UK has made progress in this area, and RSE groups have been installed in many institutions. In <a target=xrefwindow href="http://www.de-rse.org/" id=d11061e1365>Germany</a>, the <a target=xrefwindow href="http://www.us-rse.org/" id=d11061e1368>US</a>, and the <a target=xrefwindow href="http://www.nl-rse.org/" id=d11061e1371>Netherlands</a>, this is still work in progress. It is also not yet determined how to match research software engineering roles in public institutions with industry roles<sup><a href="#ref-53">53</a></sup>.</p></li></ul></div> </p><p class="" id=d11061e1382>In summary, the necessary but resource-intensive practice of creating, maintaining, supporting, and funding sustainable research software is not yet sufficiently incentivized and enabled by research institutions and funding agencies, nor does it align well with the publish-or-perish culture that is still prominent in most fields.</p><p class="" id=d11061e1385>Therefore, it is necessary to comprehensively motivate sustainable research software practice. In the following, we identify stakeholders of research software<sup><a href="#ref-54">54</a>–<a href="#ref-56">56</a></sup>, and explicate their particular motivations for sustainable research software. Subsequently, we specify challenges towards satisfying the demands of the individual stakeholders.</p><div class=section><a name=d11061e1395 class=n-a></a><h3 class=section-title>Stakeholder motivations for research software sustainability</h3><p class="" id=d11061e1400>While a wide range of stakeholders share interest in sustainable software, we argue that their individual motivation can differ quite significantly:</p><p class="" id=d11061e1403>The <i>general public</i> benefits from research which supports the common good, in other terms: creates a better world, faster. Taxpayers have an interest in economical use of their tax money, to which duplicated or flawed efforts to create research software – in contrast to software reuse – is contrary. A subset of this group may be interested in sustainable, i.e., re-usable and understandable, software as part of citizen science.</p><p class="" id=d11061e1409><i>Domain researchers</i> benefit from better software to do more, better, and faster research. Sustainable research software supports this through validated functionality (e.g., correct algorithms), the potential for reuse, and general availability. Sustainable software also potentially simplifies building upon previous research results by reusing the involved software to produce additional data or by extending the software’s functionality. In light of recent updates to <a target=xrefwindow href="https://www.dfg.de/en/research_funding/principles_dfg_funding/good_scientific_practice/" id=d11061e1413>definitions of good research practice</a>, sustainable research software also allows domain researchers to comply with guidelines and best practices. Additionally, using software that is sustainable enough to establish itself as a standard tool in a field signifies inclusion in a research community. Less directly, researchers may benefit from the existence of sustainable standard tools as they yield standard formats, which in themselves facilitate reuse of research data.</p><p class="" id=d11061e1417><i>Research software engineers (RSEs)</i> have an intrinsic interest in sustainable research software. They create better software for research, which enables more and better research. RSEs have an inherent interest in developing and working with high quality software, as part of professional ethics as well as good research practice. RSEs build their reputation on high quality software and software citation<sup><a href="#ref-20">20</a>,<a href="#ref-31">31</a></sup>, which will open up new career paths. Finally, for RSEs, creating sustainable research software is part of an attractive, intellectually challenging, and satisfying work environment.</p><p class="" id=d11061e1430><i>Research leaders</i> as well as <i>research performing organizations</i> mainly focus on the economic aspects and management of research, i.e., available funds, people, and time employed to optimize research output. Both need to make sure that their employees continually improve their qualification and generate impact to improve their standing in the various research communities and ensure continued funding. Overseeing and enabling the creation of sustainable research software advances their visibility in the field and makes their research endeavors both more future-proof and more easily traceable, reproducible, and verifiable and thus more likely to attract additional resources (including human resources). <i>Research performing organizations</i> can additionally benefit from sustainable research software if it can be reused in other areas, creating synergies between different research disciplines. These synergies typically free resources that can then be used in areas other than software development and maintenance. Finally, organizations can gain highly competitive positions in terms of funding and hiring opportunities, as well as a reputation for being on the cutting edge of research, through early adoption of research software engineering units, and the implementation of sustainable research software policy and practice.</p><p class="" id=d11061e1441><i>Research funding organizations</i> have inherent interest in – and directly benefit from – the existence of sustainable research software as it allows them to direct more resources towards actual research (rather than recreation of software) and increase return on investment. At the same time, funding organizations can create incentives for sustainable software by imposing policies that reflect the necessity of research software sustainability and creating respective funding opportunities.</p><p class="" id=d11061e1446><i>Geopolitical units</i> have a strategic interest to be independent of other geopolitical units to ensure that research can continue seamlessly regardless of geopolitical developments and ensuing embargoes on information flow. Reuse of sustainable software additionally frees up funding for uses other than software development. Well-established, sustainable software systems can also attract researchers and companies in the research and technology sector.</p><p class="" id=d11061e1451><i>Libraries (also registries, indices)</i> benefit from sustainable research software, as it will undergo a formal publishing process and be properly described in its metadata. Libraries can extend their portfolio beyond text-based research objects and stake claims as organizations harnessing the digitalization of research. In turn, they help to increase visibility and discoverability for research software through their services and advance the competitiveness of their organization or geopolitical unit. In addition, libraries also use research software and would thus benefit directly from a more sustainable research software landscape. Last but not least, by using FLOSS research software, libraries could avoid expensive licenses and often insufficiently adapted commercial software.</p><p class="" id=d11061e1456><i>Infrastructure units</i>, such as supercomputing facilities and university computing centers, benefit from sustainable software as it makes their daily work in terms of software installation and user support easier. Additionally, they can position themselves at the forefront of research by bundling expertise on the creation and maintenance of sustainable research software and installing research software engineering teams.</p><p class="" id=d11061e1461><i>Industry</i> benefits from sustainable research software, as the process of creating and maintaining research software produces a highly-skilled workforce. Depending on the employed licensing model, sustainable research software can also be adopted by industry partners to reduce cost in corporate research and development. Helping to sustain research software may also enable positive outreach for companies across industry and into society.</p><p class="" id=d11061e1467><i>Independent (open source) developers</i> can get involved in research software, even if they are not employed by a research institution. This can help them get in contact with other developers in the field and may potentially lead to collaborations or job opportunities in research based on this extended experience.</p></div></div><div id=article1-body class=generated-article-body><h2 class=main-title id=d11061e1476>How to decide which software to sustain?</h2><a name=B3 class=n-a></a><div class="boxed-text panel clearfix"><div class=caption><h3></h3></div><p class=first id=d11061e1481>Kim’s PI is happy because Kim writes a longer section on <i>hal9k</i> for the final project report and provides a software management plan alongside it, which ticks off a box in the template that the PI had previously worried about. The PI does not want to let Kim go and instead offers her to be co-PI on a follow-up project to test new methods on the data, and integrate them into <i>hal9k</i> as well. They are positive that such a project proposal has a good chance to be funded, as they can show impact of their first project via their university’s current research information system (CRIS) and through the number of citations of <i>hal9k</i> and the publications for which it was used. While they write the proposal, the faculty dean approaches the two to tell them that based on Kim’s work, they will now negotiate about two new RSEs for the central RSE team with the university’s provost for research and plan to consider candidates with a background in researchonomics.</p><p class="" id=d11061e1493>When they get the decision letter from the research funding organization, Kim and her co-PI are happy to learn that their new project has won the grant. The reviewers specifically point out the value of extending Kim’s Boa library to include the proposed new methods, as well as the significant reuse potential of <i>hal9k</i> for the researchonomic community as a direct effect of its well-engineered architecture and modularity. Additionally, they stress that it was really easy to evaluate the software due to the comprehensive test suite, documentation, and example data. In fact, during the first month of the new project, three other researchonomic research projects approach them to ask whether they can contribute to Kim’s library and offer to fund six months of RSE work for this. Kim uses this money to also parallelize <i>hal9k</i> together with the RSEs and works with her university’s computing center to offer it as a standard tool for researchonomic supercomputing.</p></div><div class=section><a name=d11061e1503 class=n-a></a><h3 class=section-title>Requirements and challenges</h3><p class="" id=d11061e1508>The sustained funding of all existing software efforts is not only impossible but would risk overly splintering the community and eventually become counterproductive to the efficiency of the research community. Therefore, it is important to agree on a list of transparent criteria that qualify a software product for sustained funding. We recognize that defining research software engineering criteria for software evaluation will also lead to activities aiming at optimizing scores to achieve these criteria. Hence, the criteria have to be designed such that all score-pushing effort truly advances the value of the software. Criteria that can be manipulated without effectively adding value, i.e., wasting resources, should be excluded. The list of criteria presented in this section could be the basis for a structured review process that facilitates an unbiased evaluation of software tools from various fields. Therefore, this list must be general enough to be applied to research software from various research disciplines while also respecting differences between fields (e.g. citation rates between humanities and life sciences). The challenge to do justice to a wide spectrum is e.g. reflected by suggesting criteria comprising different levels<sup><a href="#ref-57">57</a></sup>. One of the major challenges in the endeavor to define a selection scheme for sustainable funding of research software is to organize a fair and transparent review process. We believe that it is important that the review process is conducted by experts, or teams of experts, that have a strong background both on software engineering as well as on the domain-specific aspects, the latter because certain criteria often exist on a spectrum that is most likely shaped by the specific demands of the respective research community.</p><p class="" id=d11061e1515>While an assessment based purely on quantitative metrics would allow for seemingly objective comparisons between programs, the definition of valid and robust quantitative metrics that can be evaluated with reasonable effort is a major challenge. On the other hand, a structured qualitative assessment with scores for groups of criteria can provide a middle ground. It is clear that both preparing an application for a review against these criteria from the applicant side as well as the evaluation by the reviewers requires significant effort. We believe that the added value significantly outweighs the investment but appropriate resources need to be factored in. Sustainability of research software should be considered from the beginning for new projects. The criteria listed below, or a subset such as the “good enough” practices proposed by Wilson <i>et al.</i><sup><a href="#ref-43">43</a></sup> and artifact review approaches<sup><a href="#ref-58">58</a>,<a href="#ref-59">59</a></sup>, are valuable throughout the development process (including early phases) for almost all types of research software applications. “Classical” research funding schemes should acknowledge the need to follow best practices during the development of new software and allow factoring in appropriate resources to design and implement for sustainability. In this section, we focus on the question which software to support in dedicated sustainability funding schemes. For such sustained funding, only software in application class 2 or 3 as defined by Schlauch <i>et al</i>.<sup><a href="#ref-60">60</a></sup>, i.e., with significant use beyond personal or institutional purposes, would likely be considered. Excellence as reflected in funded projects, publications, and software adoption, i.e., backing by a community, should be considered during selection. Nevertheless, we believe a good scheme should strike a balance between consolidating the field to few well-established software packages on one side and stimulating innovation and cooperation promoting diversity in terms of more than one monopolistic package on the other side. Last but not least, there is an inherent conflict between the long-term goals of sustainability funding software and the necessary reevaluation to monitor the state of the software over time.</p></div><div class=section><a name=d11061e1539 class=n-a></a><h3 class=section-title>Selection criteria</h3><p class="" id=d11061e1544>Several evaluation schemes for research software have been proposed before and led to the formulation of first recommendations<sup><a href="#ref-13">13</a>,<a href="#ref-14">14</a></sup>. Gomez-Diaz & Recio suggested the CDUR scheme based on Citation, Dissemination (including aspects like license, web site, contact point), Use, and Research (output)<sup><a href="#ref-61">61</a></sup>. Lamprecht <i>et al</i>. rephrased the FAIR data principles<sup><a href="#ref-10">10</a></sup> for research software<sup><a href="#ref-12">12</a></sup>. Hasselbring <i>et al</i>. found that the adoption of FAIR principles is different between fields with an emphasis on reuse in computer science as opposed to a reproducibility focus in computational science<sup><a href="#ref-11">11</a></sup>. Fehr <i>et al</i>. collected a set of best practices for the setup and publication of numerical experiments<sup><a href="#ref-62">62</a></sup>. Jiménez e<i>t al</i>. boiled it down to four best practices<sup><a href="#ref-63">63</a></sup>: public source code, community registry, license, and governance. Hsu <i>et al</i>.<sup><a href="#ref-64">64</a></sup> proposed a framework of seven sustainability influences (outputs modified, code repository used, champion present, workforce stability, support from other organizations, collaboration/partnership, and integration with policy). They found that the various outputs are widely accessible but not necessarily sustained or maintained. Projects with most sustainability influences often became institutionalized and met required needs of the community<sup><a href="#ref-64">64</a></sup>. In the field of open source software, the <a target=xrefwindow href="http://chaoss.community/metrics" id=d11061e1602>CHAOSS (Community Health Analytics Open Source Software) project</a> has developed metrics to evaluate sustainability. One objective of CHAOSS is to automatically generate project health reports based on software that evaluates the metrics, with most of the metrics already covered. The UK Software Sustainability Institute (SSI) suggested both a subjective tutorial-based and a more objective criteria-based software evaluation scheme<sup><a href="#ref-65">65</a></sup>, the latter being available as an <a target=xrefwindow href="https://www.software.ac.uk/resources/online-sustainability-evaluation" id=d11061e1609>online form</a>. ROpenSci<sup><a href="#ref-66">66</a></sup> provides software reviews for R developers, which have been very successful in the community. The <a target=xrefwindow href="https://joss.readthedocs.io/en/latest/review_criteria.html" id=d11061e1617>review criteria of the Journal of Open Source Software (JOSS)</a> focus on the aspects license, documentation, functionality, and tests. This list of essential items should be fulfilled by all research software that wants to be considered not only for publication but also for sustained funding.</p><p class="" id=d11061e1621>We drew inspiration from all these works and suggest a set of criteria on which to base reviews for sustainable funding. This set comprises mandatory, hard criteria that we think have to be fulfilled across domains (highlighted in italics) and additional desirable, soft criteria that can be implemented to different degrees depending on the use case and domain-specific software development requirements. The soft criteria should be evaluated in a structured way by the reviewers with a specific response for each section rather than one running text. The fact that most of these criteria will be considered in any software management plan (<a target=xrefwindow href="https://www.software.ac.uk/resources/guides/software-management-plans" id=d11061e1623>SMP</a>) highlights its importance for sustainable research software.</p><p class="" id=d11061e1627><b><i>Usage and impact.</i></b> Requirements qualifying software for sustained funding are (1) its <i>use beyond a single research group</i>, (2) the scientific relevance and validity of the software documented in <i>at least one peer-reviewed scientific publication</i>. Ideally a paper also describes the scope, performance, and design of the software. (3) The use of the software in publications is a measure of impact but quantitative assessment brings about additional challenges<sup><a href="#ref-27">27</a></sup>. Therefore, other, potentially domain-specific, impact measures, such as influence on policy and practice as well as use in other software and products should be considered as well to evaluate relevance for academia and society. Considerable attendance at training and networking events can be considered as a proof of use as well. (4) A <i>market analysis</i> needs to show that the software is important to a user base of relevant size and either unique or one of the main players in a field with several existing solutions. Geographical or political aspects can be considered as well, e.g. to support the maintenance of a European solution. A convergence process of (parts of) a research community towards a specific software stack, i.e., documented transition of several research groups to a common software, would be a strong indicator of impact. (5) As community uptake and benefits are a central goal of sustained software funding, outreach and <i>appropriate training material</i> for new users of the software are essential.</p><p class="" id=d11061e1649><b><i>Software transparency and quality.</i></b> As mandatory criteria of software transparency and quality that have to be fulfilled, we consider (6) the <i>public availability of the source code</i> in both a code repository and an archive (for long term availability), developed using (7) <i>version control</i> with meaningful commit messages and linked to an issue tracker (ideally maintained, but at least mirrored on a public platform). (8) <i>Documentation</i> of the software needs to be publicly available comprising both user documentation (requirements, installation, getting started, user manual, release notes) and developer documentation (with a development guide and API documentation within the code, e.g. using Doxygen)<sup><a href="#ref-67">67</a></sup>. (9) The <i>license</i> under which the software is distributed must be defined. Publicly funded software should be published under a Free/Libre Open Source Software (FLOSS) license by default, although exceptions to this might apply (e.g. excluding commercial use). (10) <i>Dependencies</i> on libraries and technologies must be defined.</p><p class="" id=d11061e1676>We acknowledge that some additional criteria have to be evaluated under consideration of the research domain. These comprise (11) the availability of examples (comprising input data and reference results), (12) mechanisms for extensibility (software modularity) as one aspect of software architecture<sup><a href="#ref-68">68</a></sup> and (13) interoperability (APIs / common and open data formats for input and output), (14) a test suite (including at least some of the following: unit tests, regression tests, integration tests, end-to-end tests, performance tests; ideally run in an automated fashion in a continuous integration environment), (15) tagged releases (considering their frequency, and availability for end users in terms of binary packages for major operating systems, or availability via package managers or containers), (16) no large-scale re-implementations for functionality for which good solutions already exist. Many of these aspects require appropriate infrastructure (see page 12).</p><p class="" id=d11061e1683><b><i>Maturity.</i></b> The research software applying for sustained funding must have already reached a certain level of maturity (typically class 2 or 3 as defined by Schlauch <i>et al.</i><sup><a href="#ref-60">60</a></sup>). A mandatory requirement is (17) a comprehensive and up--to-date <i>software management plan</i><sup><a href="#ref-69">69</a></sup>. The software should (18) be maintainable with an appropriate amount of resources as detailed in a sustainability section of the software management plan. The software has (19) a well maintained website with a clearly defined <i>point of contact</i> and a communication channel to inform users about news regarding the software such as new releases. Besides an active user community, sustainable software requires (20) a group of developers (i.e., definitely <i>more than 1 developer</i>) documented, e.g. by contributions to the code base or participation in documented, public discussions or issue tracking. Another criterion is (21) whether potential contributors are invited to participate in a clearly defined process (e.g., a CONTRIBUTING document). The group of developers should have defined a governance model for their project and easy ways for users to provide input regarding their needs.</p></div><div class=section><a name=d11061e1708 class=n-a></a><h3 class=section-title>Recommendations</h3><p class="" id=d11061e1713>Given the diversity in the software technology landscape, and the domain-specific software development cultures<sup><a href="#ref-70">70</a></sup>, some of the above-mentioned criteria have to be evaluated against domain-specific requirements. Therefore, we highly recommend to base the selection process on a combination of (1) a software quality-based review and (2) a domain-specific scientific review. In particular, the former should be ideally performed by a central institution (e.g. at funding bodies or other independent agencies such as a software sustainability institute). Only criteria for which improvement truly advances the value of the software should be considered in evaluation schemes, i.e. no criteria that can be gamed. After rejecting software not fulfilling the mandatory criteria in a first stage of the review process, the second stage of the selection process should be realized as a transparent procedure ideally allowing the reviewers to interact with the PIs of the software (e.g. remote meetings, forum-like discussions) and put the software quality and development efforts into the domain-specific context. The outcome of this second stage should be a structured review assessing each criterion explicitly and a rating for each of the dimensions <i>Usage and impact, Software quality, and Maturity</i>. For sustained software funding, it is important to audit the performance, relevance, impact, progress, and level of sustainability of funded software after reasonable time frames. Such a reevaluation should revisit the criteria under consideration of evolving software technology and scientific standards, without requiring a completely new proposal being submitted. We envision funding periods of 5 years to provide sufficient security for funded software projects, while allowing for adaptation of the portfolio of funded software to novel research directions and community needs. Failure to meet the reevaluation criteria should lead to the decision to phase-out sustainable funding. The phase-out process may come with a 1-year funding program based on a consolidation plan with clear goals regarding the archiving and preservation of the software, documentation, and all existing resources.</p></div></div><div id=article1-body class=generated-article-body><h2 class=main-title id=d11061e1727>Who sustains research software?</h2><a name=B4 class=n-a></a><div class="boxed-text panel clearfix"><div class=caption><h3></h3></div><p class=first id=d11061e1732>Kim wants to broaden her research portfolio within researchonomics and applies for postdoctoral positions at other institutions. Her library <i>hal9k</i> is growing in popularity within researchonomics, and she wants to continue working on it. As her university has adopted an open science policy, <i>hal9k</i> is free software under a Free/Libre Open Source Software (FLOSS) license, and Kim is free to continue her work on the library even after moving away from UofA. Due to her involvement in the creation of <i>hal9k</i> as well as her previous success in attracting funding, Kim has the choice between multiple, attractive positions and decides to move to the researchonomics group at Eden University (EdU). She has already extended <i>hal9k</i> in multiple directions in the past and plans to continue this work at EdU. Her group leader at EdU would like to continue funding her but due to a law called the Fixed-term Research Contract Bill, EdU is not allowed to extend her contract, and neither third-party funding for her own position nor a permanent position are available. After having developed a now widely-used research tool, several publications in software and paper form, as well as having attracted funding, Kim finds herself looking for a job again.</p></div><p class="" id=d11061e1748>Research relies on software and software relies on the people developing and maintaining it. Sustainable research requires sustainable software, and this in turn requires continuity for those who develop and maintain it.</p><div class=section><a name=d11061e1751 class=n-a></a><h3 class=section-title>Requirements</h3><p class="" id=d11061e1756>Possibly the most important demand is the need for an increase in <i>recognition and awareness</i> of research software as a first class citizen in research<sup><a href="#ref-14">14</a>,<a href="#ref-71">71</a>,<a href="#ref-72">72</a></sup>. For sustainability of research software, long-term commitments of the respective software leads are crucial, but very few <i>professional RSE profiles</i> currently exist. In consequence, it is essential to create career paths for RSEs that are attractive and include permanency perspectives. While creating permanent positions in the German academic system below the faculty level is an actively discussed topic overall<sup><a href="#ref-73">73</a></sup>, we specifically focus on the needs originating from the development and maintenance of research software here.</p><p class="" id=d11061e1779>As already mentioned, research software development not only requires domain expertise, but also software development <i>education, skills, and competence</i>. Currently, most of the domain researchers developing and maintaining domain-specific software technology have not received professional training on software development<sup><a href="#ref-3">3</a>,<a href="#ref-41">41</a></sup>. To enhance the productivity and sustainability of computer-based research, it is essential to integrate software development training into the education of domain researchers.</p><p class="" id=d11061e1792>Currently, a significant portion of the existing research software is developed by individuals or in small groups, primarily to serve their own requirements. This situation is unsatisfying in terms of collaboration and inefficient in terms of several groups spending resources on generating similar or even the same functionality. To enable and promote synergies, it is important to allocate resources for research software development and to build <i>communities</i>, as described in <a href="#ref-74">74</a>.</p></div><div class=section><a name=d11061e1802 class=n-a></a><h3 class=section-title>Challenges</h3><p class="" id=d11061e1807>We are currently facing a <i>lack of awareness</i> for the importance of research software as discussed above. Moreover, there is <i>little recognition</i> for the efforts put into software development and maintenance. In consequence, software development in academic settings is mostly considered as a means to an end and sustainability is often not considered in project planning and grant proposals and contributes little to progressing research careers<sup><a href="#ref-75">75</a></sup>. The main challenge here is the continued use of metrics that primarily leverage traditionally published articles and article citation numbers.</p><p class="" id=d11061e1820>In academia, developers of research software are typically domain researchers, and in particular if new areas are explored, the software development process itself has research character. Obviously, developing research software requires not only domain knowledge but also software development skills, and the researchers leading the software development process are often domain experts with substantial software development experience, making them extremely valuable members of the research community. However, the current academic system in Germany does not provide a defined <i>RSE role</i>. Fixed-term positions are, at least currently within the German academic system, often effectively the end of a Research Software Engineer's career path, sometimes even a dead end. The challenge here is the lack of available permanent positions within the non-professorial academic faculty (“Mittelbau”) in Germany, compounded by a lack of access to these few permanent positions for RSEs. This in turn is due to the already mentioned lack of recognition for efforts concerning research software for faculty appointments within domain sciences.</p><p class="" id=d11061e1826>In order to develop sustainable software, researchers need to have the <i>skills and expertise</i> to build software that is easy to maintain and extend<sup><a href="#ref-76">76</a></sup>. However, most of the researchers are self-taught developers<sup><a href="#ref-3">3</a>,<a href="#ref-41">41</a></sup>. Ideally, these skills have to be built into the domain science curricula, which could generally be done in two different ways (or a combination of them). One obvious solution attempt are additional courses that focus on these topics. The main challenge here is to decide which other topic(s) to possibly drop due to the limited volume of any given curriculum. A different approach is to incorporate software-related topics into existing domain science courses. While this would provide the benefit of show-casing the usage of specific software skills directly within the domain science, the challenge here is the amount of work necessary to change existing lecture material, let alone the need of the lecturers to acquire those skills themselves in the first place.</p><p class="" id=d11061e1843>As long as the necessary software skills within domain sciences are not yet wide-spread, building a network from those that have acquired relevant skills is difficult. <i>Community</i> efforts, that concentrate on questions regarding research software, can help to fill this gap. Examples of such efforts include the Software Carpentries, national and international RSE societies (e.g., within Germany <a target=xrefwindow href="https://de-rse.org" id=d11061e1848>deRSE</a> e.V.). However, since research software is such an interdisciplinary topic, it is hard to get recognition and find funding within any specific discipline. As a result, existing communities often have to rely heavily on volunteers. This is challenging because despite benefits to domain science, volunteers hardly receive recognition for their work “back home”, i.e., within their domain, underlining again the importance of our first demand.</p></div><div class=section><a name=d11061e1854 class=n-a></a><h3 class=section-title>Recommendations</h3><p class="" id=d11061e1859>Increasing <i>recognition and awareness</i> is a challenge that calls for both immediate action and perseverance. Nevertheless, some measures will likely show positive effects comparatively soon.</p><p class="" id=d11061e1865>Similarly to plans for research data management, funding agencies should request that applicants include considerations about how software developed in a project can be sustained beyond the end of the funded project. A follow up on these plans during and after the project lifetime, i.e., a dedicated software management plan, is crucial.</p><p class="" id=d11061e1868>Another recommendation is aimed at decision makers concerning recruitment for academic positions: broaden the definition of research impact beyond traditional scientific publications to also include other impactful results. Not all researchers that think of themselves as RSEs pursue a faculty position as their main career goal. However, permanent academic non-faculty positions are rare within the German academic system, also due to the lack of a defined <i>RSE role</i>. We recommend research institutions to leverage the benefit of dedicated RSEs by establishing attractive long-term career options in the academic environment. The long-term solution in order to gain sufficient software development <i>skills</i> should be education that is included early in the career path, ideally already at the Bachelor level. For the time being however, efforts involving workshops and seminars that provide easy access to hands-on training on software-related questions should be promoted and supported as much as possible.</p><p class="" id=d11061e1877>It is important to provide an environment where <i>communities</i> can form and flourish by allocating resources for research software development and for building communities around it<sup><a href="#ref-63">63</a>,<a href="#ref-74">74</a>,<a href="#ref-77">77</a></sup>. The identification with a community of like-minded people and personal action<sup><a href="#ref-78">78</a></sup> can lead to a permanent establishment of sustainable research software as a valuable research output. Thus, research institutions as well as funding agencies should not only be open-minded regarding existing volunteer organizations, but should actively promote the creation of such groups.</p></div></div><div id=article1-body class=generated-article-body><h2 class=main-title id=d11061e1901>How can research software be sustainably funded?</h2><a name=B5 class=n-a></a><div class="boxed-text panel clearfix"><div class=caption><h3></h3></div><p class=first id=d11061e1906><i>Hal9k</i> has grown into a widely used software in researchonomics, and Kim is proactively asked to apply for - and is subsequently awarded - a permanent RSE position at the institute for researchonomy at UofA, based on her work on the library. She works closely with the central RSE team, but mostly due to bureaucracy and the high demand for her library, Kim does not have enough time to maintain and further develop <i>hal9k</i> alone anymore. Together with the dean she develops a course for the researchonomics curriculum which teaches data processing with <i>hal9k</i>. As a lesson from her own career, she starts the course with sessions on the Hash shell, version control with Tig, Boa, and two whole sessions on basics of sustainable software development. This is very fruitful, and due to the implementation of a new research software funding scheme at UofA, Kim is able to hire one of the course students, who has shown great RSE skills, straight into a long-term position at her institute, where they focus on the maintenance and development of <i>hal9k</i>, work with the computing center to support <i>hal9k</i>-based supercomputing on a new, dedicated FGPA cluster, develop training materials for external users, and organize the yearly <i>hal9k</i> users and developers conference. Kim gets to travel the world to visit researchonomics groups who are using <i>hal9k</i>.</p></div><div class=section><a name=d11061e1931 class=n-a></a><h3 class=section-title>Requirements</h3><p class="" id=d11061e1936>Sustainable funding for research software boils down to funding the four main pillars enabling sustainable software development: (1) Personnel with expertise in research software development; (2) Infrastructure for developing, testing, validating, and benchmarking research software; (3) Training in software design and sustainable software development; and (4) Community management and events for creating synergies between research groups and software efforts.</p></div><div class=section><a name=d11061e1940 class=n-a></a><h3 class=section-title>Challenges</h3><p class="" id=d11061e1945>Short-term engagement of (early career) researchers raises the question of how to maintain a constant level of expertise within a developer team and prevent knowledge drain concerning domain knowledge and software engineering skills. Conversely, the permanent engagement of qualified personnel requires to offer career perspectives, especially due to the fact that academia competes with industry for the same people. A challenge specific to Germany is posed by the shortage of permanent positions and by the restrictions for temporary positions due to the German <i>Wissenschaftszeitvertragsgesetz</i><sup><a href="#ref-79">79</a></sup>.</p><p class="" id=d11061e1954>Sustainable software development requires hardware technology to develop, test, validate, and benchmark features in a continuous integration cycle. The challenge in this context is the persistent evolution of the hardware landscape. Hence, for creating an environment promoting sustainable software development, it is important to provide access to a wide hardware portfolio and to support a development cycle based on continuous integration.</p><p class="" id=d11061e1957>Expertise in sustainable research software development is a scarce resource, and training is heavily needed as one way of building up more expertise. However, while integrating interdisciplinary software engineering courses into the education curriculum can build up basic skills, some expertise is domain-specific and requires interinstitutional training activities. Furthermore, there exist no financial incentives for creating software-specific documentation and tutorials nor to provide other forms of support.</p><p class="" id=d11061e1960>While the creation of research software communities is one major asset in sustaining research software technology, promoting this process requires the installation of new funding instruments. Traditionally, research grants are limited to rather short time frames and support personnel, material, hardware, and to a limited degree also travel and research visits. Creating a research software community however requires funding for community and training events as well as “virtual hardware” such as webspace, versioning systems, task-managing systems, and compute cycles. These demands can hardly be met without third-party funding<sup><a href="#ref-45">45</a>,<a href="#ref-80">80</a>–<a href="#ref-82">82</a></sup>.</p></div><div class=section><a name=d11061e1974 class=n-a></a><h3 class=section-title>Recommendation: creation of adequate funding schemes</h3><p class="" id=d11061e1979>Funding is a crucial factor for sustaining research software. Currently available sources and instruments are not adequately shaped for the challenges and solutions outlined above. We recommend actions on the individual, organizational, and national level.</p><p class="" id=d11061e1982>Existing project-focused funding instruments on the local, national, and international level need to be complemented with funding instruments specifically designed for research software development and sustained research software maintenance to make research software a first class citizen in the research landscape. For example, software projects enhancing research and fulfilling the sustainability criteria detailed in section <i>How to decide which software to sustain?</i> may be entitled for sustained funding as long as they live up to the standards and remain a central component of the research landscape.</p><p class="" id=d11061e1988>Computing centers and supercomputing facilities for research need to receive earmarked resources for the support of sustainable software development. This funding is necessary to provide continuous integration services, a hardware portfolio for development, testing and benchmarking software, as well as personnel for training domain researchers in software design and the proper usage of the services.</p><p class="" id=d11061e1991>The creation and maintenance of training materials for general research software engineering education and the software-specific documentation and tutorial creation needs to be reflected in funding opportunities. This can either happen by dedicating modules of research or software grants to providing support and the generation of training material, or by opening funding schemes focusing on interdisciplinary software development education. The latter may include research that looks at research software development as a process to analyze which measures, interactions, and team compositions make research software successful. Additionally, funding instruments fostering the formation of research software communities have to be established.</p></div></div><div id=article1-body class=generated-article-body><h2 class=main-title id=d11061e1999>Which infrastructure is needed to sustain research software?</h2><a name=B6 class=n-a></a><div class="boxed-text panel clearfix"><div class=caption><h3></h3></div><p class=first id=d11061e2004>As the <i>hal9k</i> community grows, so does the need for infrastructure. Kim and her team collaborate with the National RSE Consortium to set up <i>hal9k</i> on the Consortium’s distributed TigHub instance and organize world-wide access to it via the NRSEC-AAI federation. Going forward, the Consortium’s Research Software Hub - a registry and <a target=xrefwindow href="http://www.softwareheritage.org/" id=d11061e2012>Software Heritage Archive-based</a> long-term repository for research software on a national level - ingests <i>hal9k</i> releases with complete metadata: citation information, the <i>hal9k</i> provenance graph and computational environment information, <a target=xrefwindow href="https://orcid.org/" id=d11061e2022>ORCID</a> iDs, etc. and provides its own DOIs for versions under a concept (umbrella) DOI. The community reviews all code and documentation changes that are contributed to <i>hal9k</i> via the central TigHub instance. The Hub’s CI system Alfred builds, tests, and pushes new releases automatically to the registered supercomputing clusters. Community efforts become better and more streamlined by the day, as research software development training is now offered as part of most curricula, and skilled RSEs are now much easier to find and hire by research institutions.</p></div><div class=section><a name=d11061e2030 class=n-a></a><h3 class=section-title>Project management tools</h3><p class="" id=d11061e2035>Research software is developed by individual researchers, in small teams within a single institution, or in larger teams distributed across multiple institutions. In particular if software development is distributed across institutions, there exists an urgent need for frameworks and tools enabling collaborative code development, software feature planning, and software management. As research software development typically includes bleeding-edge research and in some cases development that the researchers do not want to disclose for a certain time to preserve intellectual property, distributed research software development also needs a global Authentication and Authorization Infrastructure (AAI). We recommend the development and/or deployment of tools for distributed software development and software management as central research infrastructure. An important aspect in this context is the cataloging of research software to reduce the duplication of development efforts. This can efficiently be realized by promoting the registration of all research software with a unique identifier and developing a tool that allows to explore the research software landscape. Research software contributors should have an <a target=xrefwindow href="https://orcid.org/" id=d11061e2037>ORCID</a> iD to be uniquely identifiable and referable. While some funding for such tools and software repositories is emerging (e.g. the bio.tools catalogue of bioinformatics tools funded as part of the European ELIXIR project<sup><a href="#ref-83">83</a></sup>), a standardized extension of such efforts to the RSE community as a whole is necessary. However, as the experiences from ELIXIR demonstrate, this is a non-trivial effort that requires significant dedicated and long-term funding.</p></div><div class=section><a name=d11061e2046 class=n-a></a><h3 class=section-title>Developer training, motivation, and knowledge exchange</h3><p class="" id=d11061e2051>As elaborated, training in sustainable software development is key to achieve sustainability in research software. At the same time, it is not clear how such training should be facilitated and institutionalized. Furthermore, for deriving software quality standards, evaluating the quality of software, and providing a code review service, central resources are necessary that individuals and groups in the research software landscape can draw from.</p><p class="" id=d11061e2054>We consider Software Carpentry and similar efforts like the creation of the Data Science Academy <a target=xrefwindow href="https://www.helmholtz-hida.de/" id=d11061e2056>HIDA</a> in the Helmholtz Association of German Research Centers helpful solutions to exchange and distribute knowledge. Local chapters of RSE groups and (inter-)national conferences will further foster networking and community building. We strongly recommend the creation of a national Software Sustainability Institute (involving funded positions to establish web platforms and training material) similar to the <a target=xrefwindow href="https://www.software.ac.uk/" id=d11061e2059>UK Software Sustainability Institute (SSI)</a>, which serves as a national contact for all aspects related to research software. The UK SSI also publishes <a target=xrefwindow href="https://www.software.ac.uk/resources" id=d11061e2062>best practice guidelines</a> for research software engineering.</p></div><div class=section><a name=d11061e2067 class=n-a></a><h3 class=section-title>Research software discovery and publication</h3><p class="" id=d11061e2072>Proper software publication and possibilities for the community to find existing software solutions for a given problem are a prerequisite to optimally exploit synergies and avoid redundant development. However, we observe that today, many funding proposals lack a thorough state-of-the-art report of software that could possibly be reused. This is most often caused by insufficient information retrieval strategies, lack of knowledge about relevant repositories, and an abundance of locations where software is collaboratively developed and stored<sup><a href="#ref-84">84</a></sup>. Discovery requires publication in a globally accessible location with appropriate metadata, e.g. Citation File Format (CFF)<sup><a href="#ref-85">85</a></sup> and <a target=xrefwindow href="https://codemeta.github.io/" id=d11061e2082>CodeMeta</a>. Comprehensive metadata (e.g. contributors, contact, keywords, linked publications, etc.) and publishing platforms have to enable persistent citing, which in turn benefits research evaluation. Selection and curation of software (probably by a data/software librarian) for publication and discovery are certainly challenging.</p><p class="" id=d11061e2086>We consider GitLab or GitHub as collaborative working environments and repositories like Zenodo appropriate publication platforms, because the latter mint DOIs, allow versioning and are publicly funded for long-term access. GitHub, Figshare, and Mendeley Data are examples of commercial enterprises with business cases in the background, which leverage research results. Besides the aforementioned metadata standards, it is advisable to document source code, e.g. using MarkDown (with Doxygen tooling). Metadata and citations play a role in beneficial tools like PIDgraph, DataCite.org, CrossRef, which utilize Persistent Identifiers (PIDs) like DOIs. Another solution to discovery are (mostly) disciplinary software indices like <a target=xrefwindow href="http://swmath.org/" id=d11061e2088>swMATH</a> or the <a target=xrefwindow href="http://ascl.net/" id=d11061e2091>Astronomy Source Code Library</a> as well as language focused systems like <a target=xrefwindow href="http://cran.r-project.org/" id=d11061e2094>CRAN</a> for R. Most of them started as national endeavors and became platforms of global importance. For Germany, we assume that the Nationale Forschungsdateninfrastruktur (NFDI) will put effort into creating or supporting discovery platforms at a central point that ease information retrieval. At the same time, all stakeholders should be aware of and counteract potential institutional “fear” of losing “their” data, software, and intellectual property.</p><p class="" id=d11061e2098>Especially in interdisciplinary environments, it would be helpful to have access to a meta software repository index, similar to what re3data<sup><a href="#ref-86">86</a></sup> does for research data repositories. We recommend the creation of such a meta index covering important (disciplinary) software indexes in order to ease discovery of relevant software locations. Evaluation of discovered software is an unsolved problem. Here, anonymous telemetry of usage may provide information for the selection of relevant software. Publishing software, their dependencies, and environment in containers may also ease evaluation and further reuse. These suggestions require significant investment in longterm infrastructure. When publishing research software it is recommended to make use of integration schemes like GitHub with Zenodo or local GitLab instances with publication platforms. Such indices and publication outlets may benefit national federated research indexing & archiving systems, similar to the hierarchy of library catalogs<sup><a href="#ref-87">87</a></sup>.</p></div><div class=section><a name=d11061e2111 class=n-a></a><h3 class=section-title>Archiving</h3><p class="" id=d11061e2116>Software preservation aims to extend the lifetime of software that is no longer actively maintained. There are different approaches, which vary in the effort required and the likelihood of success. Software archiving is one important aspect of software preservation: the process of storing a copy of a software package so that it may be referred to in the future. The publication of a certain software version for reference in research articles requires simple ways to archive research software on a long-term basis. Furthermore, its integration with collaborative software development environments such as GitLab or GitHub and with publication repositories is needed to facilitate archiving of referenced software versions based on sustainable frameworks (e.g. <a target=xrefwindow href="http://www.invenio-software.org/" id=d11061e2118>Invenio</a> for GitHub to Zenodo integration).</p><p class="" id=d11061e2122>A challenge for software archiving is the need to (ideally) preserve the runtime environment and all dependencies of the software. This could improve reproducibility, especially when running the software in its original state. If research data are needed to reproduce results, they should also be archived with the software or the publication. Specialized and unique hardware like high performance computing resources can be part of the runtime environment, which may not be accessible in the future. To overcome this, an emulation of hardware may be a (challenging) solution. Emulation involves the encapsulation and distribution of the complete hardware and software stacks, including the operating system and driver interdependencies. This can result in intellectual property issues when offered as a service.</p><p class="" id=d11061e2125>There are both local and global approaches to software conservation. One solution to keep the software in an executable state by preserving its context and runtime environment is to use containers such as Docker. However, to archive the Docker containers, additional metadata should be added and stored with the software in an archive container format that allows exchange between repositories and exit strategies, such as the BagIt container format<sup><a href="#ref-88">88</a></sup>. Application or platform conservation is also achieved by conservational efforts where unmaintainable (virtual) machines are sandboxed to keep the platform in a secure but running state. Other notable efforts in this direction include for example Singularity and Guix HPC. Another threat is losing project repositories on global platforms like Github or BitBucket. Here, global platforms like <a target=xrefwindow href="https://www.softwareheritage.org" id=d11061e2131>Software Heritage</a> harvest those repositories and prevent loss by long-term archiving.</p></div></div><div id=article1-body class=generated-article-body><h2 class=main-title id=d11061e2139>Legal aspects</h2><a name=B7 class=n-a></a><div class="boxed-text panel clearfix"><div class=caption><h3></h3></div><p class=first id=d11061e2144>More and more industrial partners enter the <i>hal9k</i> community, and they bring their lawyers. Together with UofA’s research software task force, the RSE team, the researchonomy institute, the corporate lawyers, and community representatives, Kim decides to create a foundation to govern <i>hal9k</i> and its environment: the Fullest Possible Use Foundation for Open Researchonomy, funded by the Ministry of Research and Education and a consortium of corporate partners. As a first step, they re-license <i>hal9k</i> under the OSI-approved <a target=xrefwindow href="https://opensource.org/licenses/MIT" id=d11061e2155>MIT license</a>.</p></div><p class="" id=d11061e2160>A common situation in research software creation is that the developer has no knowledge or awareness of legal aspects and therefore did not consider them early enough. As seen in Kim’s example, re-licensing later in the project can be not only legally, but also organizationally very tricky, in particular for projects which developed over many years and involved many contributors from different organizations. Thus, we think the main legal demands for research software development are raising awareness and empowering all levels of responsible persons in academia (from researchers and RSEs over PIs to research performing organizations and research funding organizations) in legal aspects. This will hopefully lead to a general legal certainty before, during, and after the research software development process and thus enable better options for collaborations between universities, non-commercial research institutions, and other national or international partners. Legal aspects always have to be considered regarding the relevant jurisdiction. Though similar issues arise in all jurisdictions, the following will focus on the European and specifically German legal framework.</p><div class=section><a name=d11061e2163 class=n-a></a><h3 class=section-title>Challenges and clarifications</h3><p class="" id=d11061e2168><b><i>Clarification of rights</i></b>. Software development is a creative activity. The main relevant law governing legal aspects is therefore the copyright law. It regulates the rights and obligations of the parties involved. Chapter 8 of the German Act on Copyright and Related Rights (UrhG) contains specific provisions applicable to computer programs and is based on the EU computer programs directive. Copyright law protecting the creator of software in similar ways exist in nearly all legal systems. It is important for the identification of rights that software, in the sense of (German) law, includes not only the source code but also the design materials<sup><a href="#ref-89">89</a></sup>. The challenge in the use, distribution, and commercialization of software is to determine the chain of rights and to identify all right holders. The owner of the copyright is not necessarily the owner of the right of use. For Germany, the Copyright Act regulates the rights for employment relationships<sup><a href="#ref-90">90</a></sup>. In such cases, the right of use is automatically transferred to the employer. This means that in most cases of employed software developers and research staff, the institution holds the rights of use for the software work. This is not automatically the case for students, freelancers, and individual external cooperation partners. Employment and service contracts with contributors could contain regulations regarding the transfer of rights of use. For researchers who conduct free research not subject to directives, in Germany the constitution guarantees freedom of research so that the rights of use for their work remains initially with the natural person. In addition to the rights of the people directly involved, other rights of third parties may also be relevant. Existing source code (e.g., other Free/Libre Open Source Software (FLOSS)), external libraries, and contributions from institutional cooperation partners are published and provided under certain licenses and their conditions must be observed (which, due to incompatibilities even among FLOSS licenses, may well mean that individually reusable pieces of software cannot be reused together or in a new context). The nature of research careers often brings additional complications to the chain of rights. It happens that researchers take their software with them when they change institutions and develop it further during their career. Here, the former employer may be entitled to some rights of use. In third-party funded projects, in particular with industry but also with public funding, rules regarding rights of use are often defined. Last but not least, the software can also be affected by other (intellectual) property rights such as patents or trademarks. Software itself is usually not patentable but it may implement a technical invention covered by patents. When using or distributing such software, an additional matching patent license may be necessary. Licenses exist (for example: GNU GPL v3) which automatically grant related patent licenses while using the software license. That should be considered when exploitation of the patent is planned.</p><p class="" id=d11061e2182><b><i>Liability</i></b>. Issues of warranty and liability for faulty software must be taken into account. We consider the possibilities of contractual limitation of liability in licenses. Full exclusions of liability are generally invalid in the German law. Limitations of liability usually depend on the form of distribution: The limitation options are larger if the rights of use are granted free of charge, e.g. provision “as is” as defined, e.g. in the BSD 3-clause license.</p></div><div class=section><a name=d11061e2189 class=n-a></a><h3 class=section-title>Ideas for solutions</h3><p class="" id=d11061e2194>In order to meet the legal challenges mentioned, it is absolutely necessary for the software developer (team) to document the rights chain comprehensively during the software development (one possible solution is presented in the accompanying report<sup><a href="#ref-91">91</a></sup>). Contributions of individual persons must be traceable and their (labor law) status must be named. At best, contracts with rules on the transfer of rights of use should be concluded before work begins. Declarations of assignment of rights can be made for existing works. License conditions for external contributions must be evaluated with regard to further rights of use and possible sub-licensing. Contracts and funding conditions must be conscientiously documented and analyzed with regard to rules on rights of use. In case that different parts of the software are based on different conditions and rights of third parties, individual modules of the new software could be published under different licenses and merged accordingly.</p><p class="" id=d11061e2201>A national research software sustainability institute could be established. This institute supports local research software task forces and thereby respective researchers and research teams in the licensing of research software and related legal issues. For this purpose, a legal help desk will be set up, to which all members of their respective research performing organization can apply. Such a legal help desk should be seen as an infrastructural investment to avoid any uncertainty about re-use of existing research software and to support research-friendly licensing. If researchers want to publish the research software under a Free/Libre Open Source Software (FLOSS) license, the organization could bundle the necessary rights beforehand. This is particularly useful when teams of researchers, often international, write software. In addition, the sustainability institute may serve as a one-stop-shop for the licensing of research software.</p></div><div class=section><a name=d11061e2206 class=n-a></a><h3 class=section-title>Recommendations</h3><p class="" id=d11061e2211>We see it as an essential part of the sustainability of research to enable the free distribution of research software. There are a variety of open source software licensing models (ranging from permissive to copyleft; for further information, see <a target=xrefwindow href="http://tldrlegal.com/" id=d11061e2213>tldrlegal</a>, the <a target=xrefwindow href="https://ifross.github.io/ifrOSS/Lizenzcenter" id=d11061e2216>ifrOSS Lizenz-Center</a>, or Morin <i>et al.</i>, 2012<sup><a href="#ref-48">48</a></sup>). The use of an FSF- or OSI-approved FLOSS license for example would enable a truly free model and also reduce legal issues. We recommend that research funding organizations such as the DFG discuss if they expect publishing all funded software under these licenses, following the paradigm of <a target=xrefwindow href="https://publiccode.eu/" id=d11061e2226>“public money, public code”</a>. If licenses such as Apache or MIT are applied, the research institutions may later still commercialize the software if appropriate. Such open source licensing is also beneficial for start-ups that intend to provide professional services for the software.</p><p class="" id=d11061e2230>Also for legal aspects, we believe it is important that all (German) research performing organizations install a research software task force, especially in light of the new <a target=xrefwindow href="https://www.dfg.de/en/research_funding/principles_dfg_funding/good_scientific_practice/" id=d11061e2232>DFG Code of Conduct</a>. Besides organization and bundling of technical and infrastructural support for local RSEs and researchers (see previous sections), this group should organize a local legal help desk, organize educational offers e.g. for the legal topics presented, and (if not implemented yet) develop the software policy of the research performing organization. As an example, with the help of on-boarding processes performed by the research software task force, RSEs should be able to keep the clearance of rights as simple as possible right from the start. This helps to avoid that - out of uncertainty and fear to make a legal mistake - some research groups end up not choosing any license at all, which may hinder reuse of the software. We suggest that the local task forces build a network with the other research performing organizations for exchange of ideas but also for generating a bottom-up strategy to organize RSE standards for Germany and beyond and possibly be the origin of the aforementioned software sustainability institute.</p></div></div><div id=article1-body class=generated-article-body><h2 class=main-title id=d11061e2240>Conclusions</h2><p class="" id=d11061e2243>We find that the research software ecosystem is notoriously lacking resources despite its strategic importance. If funding and support does not improve, the success story of science based on academic research software may be at stake. We recommend the installation of infrastructure that enables sustainable software development including platforms for collaboration, continuous integration, testing, discovery, and long-term preservation. We suggest the establishment of a nationwide institution similar to the Software Sustainability Institute (SSI) to provide project consulting and code review services as well as sustainable software development training. We think that sustainable software development should become an integral component of the universities’ teaching curriculum. We encourage the research funding bodies to reflect the licensing models for academic software development, and to decide whether the “public money, public code” paradigm justifies the requirement that all publicly funded software has to be publicly available under a Free/Libre Open Source Software (FLOSS) license. Ultimately, we strongly advise the implementation of funding schemes for sustainably supporting the development and maintenance of research software based on clear and transparent criteria, for creating incentives to produce high quality community software, and for enabling career paths as research software engineer (RSE).</p></div><div id=article1-body class=generated-article-body><h2 class=main-title id=d11061e2249>Glossary</h2><p class="" id=d11061e2252><b>domain researchers</b> The people doing the research to advance knowledge in a field.</p><p class="" id=d11061e2257><b>general public</b> Lay people that do not necessarily have specific insight regarding a research domain.</p><p class="" id=d11061e2262><b>geopolitical units</b> Governed public units, ranging from cities and councils, over federal states and countries, up to political unions such as the EU. In the context of this paper, the discussion usually focuses on the larger units (countries and political unions).</p><p class="" id=d11061e2267><b>independent (open source) developers</b> Project-external software developers who are not employed by the institution(s) carrying out the project.</p><p class="" id=d11061e2273><b>industry</b> Companies conducting research or profit from available academic research software which they can directly or indirectly apply to their field.</p><p class="" id=d11061e2278><b>infrastructure units</b> Computing centers of research bodies such as universities and other research centers, as well as high-performance computing facilities.</p><p class="" id=d11061e2283><b>libraries (also registries, indices)</b> Infrastructure units of research bodies such as universities, or independent organizations, which gather research outputs and their structured metadata, and provide indices, search, etc.</p><p class="" id=d11061e2288><b>research funding organizations</b> Public research funding bodies but potentially also companies, foundations, associations, etc.</p><p class="" id=d11061e2293><b>research leaders</b> Heads of research groups, such as professors and other people with staff responsibility.</p><p class="" id=d11061e2298><b>research performing organizations</b> Research groups, departments, faculties, research institutions (universities, national labs, cross-institutional research groups, etc.), umbrella organizations, such as Helmholtz-Gemeinschaft Deutscher Forschungszentren, Max-Planck-Gesellschaft zur Förderung der Wissenschaften, Leibniz-Gemeinschaft, etc.</p><p class="" id=d11061e2304><b>research software engineers (RSEs)</b> People creating and maintaining research software; this group ranges from research-focused software developers, to software engineers with a focus on research; other definitions include other roles, such as research software managers.</p></div><div id=article1-body class=generated-article-body><h2 class=main-title id=d11061e2312>Data availability</h2><div class=section><a name=d11061e2315 class=n-a></a><h3 class=section-title>Underlying data</h3><p class="" id=d11061e2320>No underlying data are associated with this article.</p></div><div class=section><a name=d11061e2324 class=n-a></a><h3 class=section-title>Extended data</h3><p class="" id=d11061e2329>Decision trees and documentation templates for the legal topics are available on Zenodo<sup><a href="#ref-91">91</a></sup>: doi: <a target=xrefwindow href="10.5281/zenodo.4327147" id=d11061e2335>10.5281/zenodo.4327147</a> </p></div></div><div id=article1-body class=generated-article-body><h2 class=main-title id=d11061e2343>Author information</h2><p class="" id=d11061e2346>We are a group of software-providing researchers, RSEs, and infrastructural as well as legal supporters. Initially, a group of representatives of funded projects of <a target=xrefwindow href="https://gepris.dfg.de/gepris/suche/projekt/research_software?context=projekt&predefinedSearch=research_software&task=doSearchDirect&" id=d11061e2348>funded projects of the first DFG sustainability call</a> met during the <a target=xrefwindow href="https://de-rse.org/en/conf2019/" id=d11061e2351>first German RSE conference (deRSE19)</a> in June 2019 in a grass-roots workshop on sustainable research software addressing the software-based research community. During this workshop, we realized that a lot of valuable experience and good ideas are present in the group, and we decided to start working on this paper together with other interested practitioners. We followed the generous invitation of the DFG for the above-mentioned two-day meeting at the Robert Koch Institute in Berlin in November 2019 to sharpen the focus of this paper.</p></div><div id=article1-back class=generated-article-footer><div class=back-section><a name=d11061e2359 class=n-a></a><span class="research-layout prime-recommended-wrapper reference-heading is-hidden"><span class=faculty-opinion-icon></span><span class="prime-red big">Faculty Opinions recommended</span></span><h2 class=main-title id=d11619>References</h2><div class="section ref-list"><a name=d11061e2359 class=n-a></a><ul><li><a name=ref-1 class=n-a></a><span class=label>1. </span> <span class=citation><a name=d11061e2366 class=n-a></a>The Event Horizon Telescope Collaboration, Akiyama K, Alberdi A, <i> et al.</i>: First M87 Event Horizon Telescope Results. IV. Imaging the Central Supermassive Black Hole. <i>Astrophys J.</i> 2019; <b>875</b>(1): L4. <a target=xrefwindow id=d11061e2377 href="https://doi.org/10.3847/2041-8213/ab0e85">Publisher Full Text </a></span></li><li><a name=ref-2 class=n-a></a><span class=label>2. </span> <span class=citation><a name=d11061e2386 class=n-a></a>Nowogrodzki A: How to support open-source software and stay sane. <i>Nature.</i> 2019; <b>571</b>(7763): 133–134. <a target=xrefwindow id=d11061e2394 href="http://www.ncbi.nlm.nih.gov/pubmed/31263262">PubMed Abstract </a> | <a target=xrefwindow id=d11061e2397 href="https://doi.org/10.1038/d41586-019-02046-0">Publisher Full Text </a></span></li><li><a name=ref-3 class=n-a></a><span class=label>3. </span> <span class=citation><a name=d11061e2406 class=n-a></a>Philippe O, Hammitzsch M, Janosch S, <i> et al.</i>: softwaresaved/international-survey: Public release for 2018 results. 2019. <a target=xrefwindow id=d11061e2411 href="https://doi.org/10.5281/zenodo.2585783">Publisher Full Text </a></span></li><li><a name=ref-4 class=n-a></a><span class=label>4. </span> <span class=citation><a name=d11061e2420 class=n-a></a>Hirsch JE: An index to quantify an individual’s scientific research output. <i>Proc Natl Acad Sci U S A.</i> 2005; <b>102</b>(46): 16569–16572. <a target=xrefwindow id=d11061e2428 href="http://www.ncbi.nlm.nih.gov/pubmed/16275915">PubMed Abstract </a> | <a target=xrefwindow id=d11061e2431 href="https://doi.org/10.1073/pnas.0507655102">Publisher Full Text </a> | <a target=xrefwindow id=d11061e2434 href="http://www.ncbi.nlm.nih.gov/pmc/articles/1283832">Free Full Text </a></span></li><li><a name=ref-5 class=n-a></a><span class=label>5. </span> <span class=citation><a name=d11061e2443 class=n-a></a>Bangerth W, Heister T: Quo Vadis, Scientific Software? <i>SIAM News.</i> 2014; <b>47</b>(1): 8. <a target=xrefwindow id=d11061e2451 href="https://sinews.siam.org/Details-Page/quo-vadis-scientific-software-1">Reference Source</a></span></li><li><a name=ref-6 class=n-a></a><span class=label>6. </span> <span class=citation><a name=d11061e2461 class=n-a></a>Prins P, de Ligt J, Tarasov A, <i> et al.</i>: Toward effective software solutions for big biology. <i>Nat Biotechnol.</i> 2015; <b>33</b>(7): 686–687. <a target=xrefwindow id=d11061e2472 href="http://www.ncbi.nlm.nih.gov/pubmed/26154002">PubMed Abstract </a> | <a target=xrefwindow id=d11061e2475 href="https://doi.org/10.1038/nbt.3240">Publisher Full Text </a></span></li><li><a name=ref-7 class=n-a></a><span class=label>7. </span> <span class=citation><a name=d11061e2484 class=n-a></a>Richardson C, Croucher M: Research Software Engineer: A New Career Track? 2018. <a target=xrefwindow id=d11061e2486 href="https://sinews.siam.org/Details-Page/research-software-engineer-a-new-career-track">Reference Source</a></span></li><li><a name=ref-8 class=n-a></a><span class=label>8. </span> <span class=citation><a name=d11061e2495 class=n-a></a>Cohen J, Katz DS, Barker M, <i> et al.</i>: The Four Pillars of Research Software Engineering. <i>IEEE Software.</i> 2020; <b>38</b>(1): 97–105. <a target=xrefwindow id=d11061e2506 href="https://doi.org/10.1109/MS.2020.2973362">Publisher Full Text </a></span></li><li><a name=ref-9 class=n-a></a><span class=label>9. </span> <span class=citation><a name=d11061e2515 class=n-a></a>Brett A, Croucher M, Haines R, <i> et al.</i>: Research Software Engineers: State of the Nation Report 2017. 2017. <a target=xrefwindow id=d11061e2520 href="https://doi.org/10.5281/zenodo.495360">Publisher Full Text </a></span></li><li><a name=ref-10 class=n-a></a><span class=label>10. </span> <span class=citation><a name=d11061e2529 class=n-a></a>Wilkinson MD, Dumontier M, Aalbersberg IJ, <i> et al.</i>: The FAIR Guiding Principles for scientific data management and stewardship. <i>Sci Data.</i> 2016; <b>3</b>(1): 160018. <a target=xrefwindow id=d11061e2540 href="http://www.ncbi.nlm.nih.gov/pubmed/26978244">PubMed Abstract </a> | <a target=xrefwindow id=d11061e2543 href="https://doi.org/10.1038/sdata.2016.18">Publisher Full Text </a> | <a target=xrefwindow id=d11061e2547 href="http://www.ncbi.nlm.nih.gov/pmc/articles/4792175">Free Full Text </a></span></li><li><a name=ref-11 class=n-a></a><span class=label>11. </span> <span class=citation><a name=d11061e2556 class=n-a></a>Hasselbring W, Carr L, Hettrick S, <i> et al.</i>: FAIR and Open Computer Science Research Software. arXiv: 1908.05986. 2019. <a target=xrefwindow id=d11061e2561 href="https://arxiv.org/abs/1908.05986">Reference Source</a></span></li><li><a name=ref-12 class=n-a></a><span class=label>12. </span> <span class=citation><a name=d11061e2571 class=n-a></a>Lamprecht AL, Garcia L, Kuzak M, <i> et al.</i>: Towards FAIR principles for research software. <i>Data Sci.</i> 2019; <b>3</b>(1): 37–59. <a target=xrefwindow id=d11061e2582 href="https://doi.org/10.3233/DS-190026">Publisher Full Text </a></span></li><li><a name=ref-13 class=n-a></a><span class=label>13. </span> <span class=citation><a name=d11061e2591 class=n-a></a>Katerbow M, Feulner G: Recommendations on the development,use and provision of Research Software. 2018. <a target=xrefwindow id=d11061e2593 href="https://doi.org/10.5281/zenodo.1172988">Publisher Full Text </a></span></li><li><a name=ref-14 class=n-a></a><span class=label>14. </span> <span class=citation><a name=d11061e2602 class=n-a></a>Scheliga K, Pampel H, Konrad U, <i> et al.</i>: Dealing with research software:Recommendations for best practices. Helmholtz Open Science Coordination Office. 2019. <a target=xrefwindow id=d11061e2607 href="https://doi.org/10.2312/OS.HELMHOLTZ.003">Publisher Full Text </a></span></li><li><a name=ref-15 class=n-a></a><span class=label>15. </span> <span class=citation><a name=d11061e2616 class=n-a></a>Hatton L: The Chimera of Software Quality. <i>Computer.</i> 2007; <b>40</b>(8): 104–103. <a target=xrefwindow id=d11061e2624 href="https://doi.org/10.1109/MC.2007.292">Publisher Full Text </a></span></li><li><a name=ref-16 class=n-a></a><span class=label>16. </span> <span class=citation><a name=d11061e2633 class=n-a></a>Chang G, Roth CB, Reyes CL, <i> et al.</i>: Retraction. <i>Science.</i> 2006; <b>314</b>(5807): 1875–1875. <a target=xrefwindow id=d11061e2644 href="http://www.ncbi.nlm.nih.gov/pubmed/17185584">PubMed Abstract </a> | <a target=xrefwindow id=d11061e2647 href="https://doi.org/10.1126/science.314.5807.1875b">Publisher Full Text </a></span></li><li><a name=ref-17 class=n-a></a><span class=label>17. </span> <span class=citation><a name=d11061e2656 class=n-a></a>Matthews BW: Five retracted structure reports: inverted or incorrect? <i>Protein Sci.</i> 2007; <b>16</b>(6): 1013–1016. <a target=xrefwindow id=d11061e2664 href="http://www.ncbi.nlm.nih.gov/pubmed/17473006">PubMed Abstract </a> | <a target=xrefwindow id=d11061e2667 href="https://doi.org/10.1110/ps.072888607">Publisher Full Text </a> | <a target=xrefwindow id=d11061e2670 href="http://www.ncbi.nlm.nih.gov/pmc/articles/2206657">Free Full Text </a></span></li><li><a name=ref-18 class=n-a></a><span class=label>18. </span> <span class=citation><a name=d11061e2680 class=n-a></a>Kanewala U, Bieman JM: Techniques for testing scientific programs without an oracle. In: <i>2013 5th International Workshop on Software Engineering for Computational Science and Engineering (SE-CSE)</i>. 2013; 48–57. <a target=xrefwindow id=d11061e2685 href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.1082.1609&rep=rep1&type=pdf">Reference Source</a></span></li><li><a name=ref-19 class=n-a></a><span class=label>19. </span> <span class=citation><a name=d11061e2694 class=n-a></a>Vogel T, Druskat S, Scheidgen M, <i> et al.</i>: Challenges for Verifying and Validating Scientific Software in Computational Materials Science. In: <i>International Workshop on Software Engineering for Science</i>. 2019; 25–32. <a target=xrefwindow id=d11061e2702 href="https://doi.org/10.1109/SE4Science.2019.00010">Publisher Full Text </a></span></li><li><a name=ref-20 class=n-a></a><span class=label>20. </span> <span class=citation><a name=d11061e2711 class=n-a></a>Smith AM, Katz DS, Niemeyer KE, <i> et al.</i>: Software Citation Principles. <i>PeerJ Comput Sci.</i> 2016; <b>2</b>: e86. <a target=xrefwindow id=d11061e2722 href="https://doi.org/10.7287/PEERJ.PREPRINTS.2169V2 ">Publisher Full Text </a></span></li><li><a name=ref-21 class=n-a></a><span class=label>21. </span> <span class=citation><a name=d11061e2731 class=n-a></a>Merali Z: Computational science: ...Error. <i>Nature.</i> 2010; <b>467</b>(7317): 775–777. <a target=xrefwindow id=d11061e2739 href="http://www.ncbi.nlm.nih.gov/pubmed/20944712">PubMed Abstract </a> | <a target=xrefwindow id=d11061e2742 href="https://doi.org/10.1038/467775a">Publisher Full Text </a></span></li><li><a name=ref-22 class=n-a></a><span class=label>22. </span> <span class=citation><a name=d11061e2751 class=n-a></a>Barnes N: Publish your computer code: it is good enough. <i>Nature.</i> 2010; <b>467</b>(7317): 753. <a target=xrefwindow id=d11061e2759 href="http://www.ncbi.nlm.nih.gov/pubmed/20944687">PubMed Abstract </a> | <a target=xrefwindow id=d11061e2762 href="https://doi.org/10.1038/467753a">Publisher Full Text </a></span></li><li><a name=ref-23 class=n-a></a><span class=label>23. </span> <span class=citation><a name=d11061e2771 class=n-a></a>Tse H: Computer code: more credit needed. <i>Nature.</i> 2010; <b>468</b>(7320): 37. <a target=xrefwindow id=d11061e2779 href="http://www.ncbi.nlm.nih.gov/pubmed/21048748">PubMed Abstract </a> | <a target=xrefwindow id=d11061e2782 href="https://doi.org/10.1038/468037a">Publisher Full Text </a></span></li><li><a name=ref-24 class=n-a></a><span class=label>24. </span> <span class=citation><a name=d11061e2792 class=n-a></a>Hafer L, Kirkpatrick AE: Assessing Open Source Software As a Scholarly Contribution. <i>Commun ACM.</i> 2009; <b>52</b>(12): 126–129. <a target=xrefwindow id=d11061e2800 href="https://doi.org/10.1145/1610252.1610285">Publisher Full Text </a></span></li><li><a name=ref-25 class=n-a></a><span class=label>25. </span> <span class=citation><a name=d11061e2809 class=n-a></a>Howison J, Bullard J: Software in the Scientific Literature: Problems with Seeing, Finding, and Using Software Mentioned in the Biology Literature. <i>J Assoc Inf Sci Technol.</i> 2016; <b>67</b>(9): 137–2155. <a target=xrefwindow id=d11061e2817 href="https://doi.org/10.1002/asi.23538">Publisher Full Text </a></span></li><li><a name=ref-26 class=n-a></a><span class=label>26. </span> <span class=citation><a name=d11061e2826 class=n-a></a>Li K, Yan E, Feng Y: How Is R Cited in Research Outputs? Structure, Impacts, and Citation Standard. <i>J Informetr.</i> 2017; <b>11</b>(4): 989–1002. <a target=xrefwindow id=d11061e2834 href="https://doi.org/10.1016/j.joi.2017.08.003">Publisher Full Text </a></span></li><li><a name=ref-27 class=n-a></a><span class=label>27. </span> <span class=citation><a name=d11061e2843 class=n-a></a>Li K, Chen PY, Yan E: Challenges of measuring software impact through citations: An examination of the lme4 R package. <i>J Informetr.</i> 2019; <b>13</b>(1): 449–461. <a target=xrefwindow id=d11061e2851 href="https://econpapers.repec.org/article/eeeinfome/v_3a13_3ay_3a2019_3ai_3a1_3ap_3a449-461.htm">Reference Source</a></span></li><li><a name=ref-28 class=n-a></a><span class=label>28. </span> <span class=citation><a name=d11061e2860 class=n-a></a>Park H, Wolfram D: Research software citation in the Data Citation Index: Current practices and implications for research software sharing and reuse. <i>J Informetr.</i> 2019; <b>13</b>(2): 574–582. <a target=xrefwindow id=d11061e2868 href="https://doi.org/10.1016/j.joi.2019.03.005">Publisher Full Text </a></span></li><li><a name=ref-29 class=n-a></a><span class=label>29. </span> <span class=citation><a name=d11061e2877 class=n-a></a>Pan X, Yan E, Cui M, <i> et al.</i>: How Important Is Software to Library and Information Science Research? A Content Analysis of Full-Text Publications. <i>J Informetr.</i> 2019; <b>13</b>(1): 397–406. <a target=xrefwindow id=d11061e2888 href="https://doi.org/10.1016/j.joi.2019.02.002">Publisher Full Text </a></span></li><li><a name=ref-30 class=n-a></a><span class=label>30. </span> <span class=citation><a name=d11061e2898 class=n-a></a>Doerr A, Rusk N, Vogt N, <i> et al.</i>: Giving Software Its Due. <i>Nat Methods.</i> 2019; <b>16</b>(3): 207–207. <a target=xrefwindow id=d11061e2909 href="http://www.ncbi.nlm.nih.gov/pubmed/30814703">PubMed Abstract </a> | <a target=xrefwindow id=d11061e2912 href="https://doi.org/10.1038/s41592-019-0350-x">Publisher Full Text </a></span></li><li><a name=ref-31 class=n-a></a><span class=label>31. </span> <span class=citation><a name=d11061e2921 class=n-a></a>Druskat S: Software and Dependencies in Research Citation Graphs. <i>Comput Sci Eng.</i> 2020; <b>22</b>(2): 8–21. <a target=xrefwindow id=d11061e2929 href="https://doi.org/10.1109/MCSE.2019.2952840">Publisher Full Text </a><a target=xrefwindow id=d11061e2931 href="https://arxiv.org/abs/1906.06141">Free Full Text</a></span></li><li><a name=ref-32 class=n-a></a><span class=label>32. </span> <span class=citation><a name=d11061e2940 class=n-a></a>Katz DS, Bouquin D, Hong NP, <i> et al.</i>: Software Citation Implementation Challenges. arXiv: 1905.08674. 2019. <a target=xrefwindow id=d11061e2945 href="https://ui.adsabs.harvard.edu/abs/2019arXiv190508674K/abstract">Reference Source</a></span></li><li><a name=ref-33 class=n-a></a><span class=label>33. </span> <span class=citation><a name=d11061e2954 class=n-a></a>Resnik DB, Morales M, Landrum R, <i> et al.</i>: Effect of impact factor and discipline on journal data sharing policies. <i>Account Res.</i> 2019; <b>26</b>(3): 139–156. <a target=xrefwindow id=d11061e2965 href="http://www.ncbi.nlm.nih.gov/pubmed/30841755">PubMed Abstract </a> | <a target=xrefwindow id=d11061e2968 href="https://doi.org/10.1080/08989621.2019.1591277">Publisher Full Text </a> | <a target=xrefwindow id=d11061e2972 href="http://www.ncbi.nlm.nih.gov/pmc/articles/6533124">Free Full Text </a></span></li><li><a name=ref-34 class=n-a></a><span class=label>34. </span> <span class=citation><a name=d11061e2981 class=n-a></a>Vandewalle P: Code Sharing Is Associated with Research Impact in Image Processing. <i>Comput Sci Eng.</i> 2012; <b>14</b>(4): 42–47. <a target=xrefwindow id=d11061e2989 href="https://doi.org/10.1109/MCSE.2012.63">Publisher Full Text </a></span></li><li><a name=ref-35 class=n-a></a><span class=label>35. </span> <span class=citation><a name=d11061e2998 class=n-a></a>Venters CC, Jay C, Lau LMS, <i> et al.</i>: Software Sustainability: The Modern Tower of Babel. In: <i>Proceedings of the Third International Workshop on Requirements Engineering for Sustainable Systems Co-Located with 22nd International Conference on Requirements Engineering (RE 2014), Karlskrona, Sweden: CEUR-WS</i>. 2014; <b>1216</b>: 7–12. <a target=xrefwindow id=d11061e3009 href="http://eprints.whiterose.ac.uk/84941/1/Venters.pdf">Reference Source</a></span></li><li><a name=ref-36 class=n-a></a><span class=label>36. </span> <span class=citation><a name=d11061e3019 class=n-a></a>Goble C: Better Software, Better Research. <i>IEEE Internet Comput.</i> 2014; <b>18</b>(5): 4–8. <a target=xrefwindow id=d11061e3027 href="https://doi.org/10.1109/MIC.2014.88">Publisher Full Text </a></span></li><li><a name=ref-37 class=n-a></a><span class=label>37. </span> <span class=citation><a name=d11061e3036 class=n-a></a>Druskat S: A Proposal for the Measurement and Documentation of Research Software Sustainability in Interactive Metadata Repositories. In <i>Proceedings of the Fourth Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE4)</i>, Manchester, UK: CEUR-WS, 2016; <b>1686</b>. <a target=xrefwindow id=d11061e3044 href="https://arxiv.org/pdf/1608.04529.pdf">Reference Source</a></span></li><li><a name=ref-38 class=n-a></a><span class=label>38. </span> <span class=citation><a name=d11061e3053 class=n-a></a>Katz DS: Fundamentals of Software Sustainability. 2018. <a target=xrefwindow id=d11061e3055 href="https://zenodo.org/record/3378572/files/Raising the Profile of Research Software.pdf?download=1">Reference Source</a></span></li><li><a name=ref-39 class=n-a></a><span class=label>39. </span> <span class=citation><a name=d11061e3064 class=n-a></a>Akhmerov A, Cruz M, Drost N, <i> et al.</i>: Raising the Profile of Research Software: Recommendations for Funding Agencies and Research Institutions. NWO (The Netherlands Organisation for Scientific Research). 2019. <a target=xrefwindow id=d11061e3069 href="https://econpapers.repec.org/article/eeeinfome/v_3a13_3ay_3a2019_3ai_3a1_3ap_3a449-461.htm">Reference Source</a></span></li><li><a name=ref-40 class=n-a></a><span class=label>40. </span> <span class=citation><a name=d11061e3078 class=n-a></a>Casties R, Czmiel A, Damerow J, <i> et al.</i>: DH Research Software Engineers - For We Are Many. 2019. <a target=xrefwindow id=d11061e3083 href="http://web.archive.org/">Reference Source</a></span></li><li><a name=ref-41 class=n-a></a><span class=label>41. </span> <span class=citation><a name=d11061e3092 class=n-a></a>Wilson G, Aruliah DA, Brown CT, <i> et al.</i>: Best practices for scientific computing. <i>PLoS Biol.</i> 2014; <b>12</b>(1): e1001745. <a target=xrefwindow id=d11061e3103 href="http://www.ncbi.nlm.nih.gov/pubmed/24415924">PubMed Abstract </a> | <a target=xrefwindow id=d11061e3106 href="https://doi.org/10.1371/journal.pbio.1001745">Publisher Full Text </a> | <a target=xrefwindow id=d11061e3110 href="http://www.ncbi.nlm.nih.gov/pmc/articles/3886731">Free Full Text </a></span></li><li><a name=ref-42 class=n-a></a><span class=label>42. </span> <span class=citation><a name=d11061e3120 class=n-a></a>Stodden V, Miguez S: Best Practices for Computational Science: Software Infrastructure and Environments for Reproducible and Extensible Research. <i>J Open Res Softw.</i> 2014; <b>2</b>(1): e21. <a target=xrefwindow id=d11061e3128 href="https://doi.org/10.5334/jors.ay">Publisher Full Text </a></span></li><li><a name=ref-43 class=n-a></a><span class=label>43. </span> <span class=citation><a name=d11061e3137 class=n-a></a>Wilson G, Bryan J, Cranston K, <i> et al.</i>: Good enough practices in scientific computing. <i>PLoS Comput Biol.</i> 2017; <b>13</b>(6): e1005510. <a target=xrefwindow id=d11061e3148 href="http://www.ncbi.nlm.nih.gov/pubmed/28640806">PubMed Abstract </a> | <a target=xrefwindow id=d11061e3151 href="https://doi.org/10.1371/journal.pcbi.1005510">Publisher Full Text </a> | <a target=xrefwindow id=d11061e3155 href="http://www.ncbi.nlm.nih.gov/pmc/articles/5480810">Free Full Text </a></span></li><li><a name=ref-44 class=n-a></a><span class=label>44. </span> <span class=citation><a name=d11061e3164 class=n-a></a>Li K, Lin X, Greenberg J: Software Citation, Reuse and Metadata Considerations: An Exploratory Study Examining LAMMPS. <i>Proc Assoc Infor Sci Tech.</i> 2016; <b>53</b>(1): 1–10. <a target=xrefwindow id=d11061e3172 href="https://doi.org/10.1002/pra2.2016.14505301072">Publisher Full Text </a></span></li><li><a name=ref-45 class=n-a></a><span class=label>45. </span> <span class=citation><a name=d11061e3181 class=n-a></a>Kuchinke W, Ohmann C, Stenzhorn H, <i> et al.</i>: Ensuring sustainability of software tools and services by cooperation with a research infrastructure. <i>Per Med.</i> 2016; <b>13</b>(1): 43–55. <a target=xrefwindow id=d11061e3192 href="http://www.ncbi.nlm.nih.gov/pubmed/29749867">PubMed Abstract </a> | <a target=xrefwindow id=d11061e3195 href="https://doi.org/10.2217/pme.15.43">Publisher Full Text </a></span></li><li><a name=ref-46 class=n-a></a><span class=label>46. </span> <span class=citation><a name=d11061e3204 class=n-a></a>Loewe A, Seemann G, Wülfers EM, <i> et al.</i>: SuLMaSS - Sustainable Lifecycle Management for Scientific Software. E-Science-Tage 2019: Data to Knowledge. 2019. <a target=xrefwindow id=d11061e3209 href="https://doi.org/10.11588/heidok.00026843">Publisher Full Text </a></span></li><li><a name=ref-47 class=n-a></a><span class=label>47. </span> <span class=citation><a name=d11061e3218 class=n-a></a>Druskat S, Krause T, Lüdeling A, <i> et al.</i>: Infrastrukturstrategien für nachhaltige Forschungssoftware in befristeten Projekten. deRSE19 - Conference for Research Software Engineers in Germany. Potsdam, Germany. 2019. <a target=xrefwindow id=d11061e3223 href="https://doi.org/10.6084/m9.figshare.11277764.v1">Publisher Full Text </a></span></li><li><a name=ref-48 class=n-a></a><span class=label>48. </span> <span class=citation><a name=d11061e3233 class=n-a></a>Morin A, Urban J, Sliz P: A Quick Guide to Software Licensing for the Scientist-Programmer. <i>PLoS Comput Biol.</i> 2012; <b>8</b>(7): e1002598. <a target=xrefwindow id=d11061e3241 href="http://www.ncbi.nlm.nih.gov/pubmed/22844236">PubMed Abstract </a> | <a target=xrefwindow id=d11061e3244 href="https://doi.org/10.1371/journal.pcbi.1002598">Publisher Full Text </a> | <a target=xrefwindow id=d11061e3247 href="http://www.ncbi.nlm.nih.gov/pmc/articles/3406002">Free Full Text </a></span></li><li><a name=ref-49 class=n-a></a><span class=label>49. </span> <span class=citation><a name=d11061e3256 class=n-a></a>Katz DS, Ramnath R: Looking at Software Sustainability and Productivity Challenges from NSF. 2015. arXiv: 1508.03348. <a target=xrefwindow id=d11061e3258 href="https://www.researchgate.net/publication/281084477_Looking_at_Software_Sustainability_and_Productivity_Challenges_from_NSF">Reference Source</a></span></li><li><a name=ref-50 class=n-a></a><span class=label>50. </span> <span class=citation><a name=d11061e3267 class=n-a></a>DFG: Nachhaltigkeit von Forschungssoftware. 2016. <a target=xrefwindow id=d11061e3269 href="https://www.dfg.de/foerderung/info_wissenschaft/2016/info_wissenschaft_16_71/">Reference Source</a></span></li><li><a name=ref-51 class=n-a></a><span class=label>51. </span> <span class=citation><a name=d11061e3278 class=n-a></a>DFG: Qualitätssicherung von Forschungssoftware durch ihre nachhaltige Nutzbarmachung. 2019. <a target=xrefwindow id=d11061e3280 href="https://www.dfg.de/service/error/404.jsp?redirectedURL=/foerderung/info_wissenschaft/info_wissenschaft_19_44/index.jsp">Reference Source</a></span></li><li><a name=ref-52 class=n-a></a><span class=label>52. </span> <span class=citation><a name=d11061e3289 class=n-a></a>Chan Zuckerberg Initiative: Essential Open Source Software for Science. <a target=xrefwindow id=d11061e3291 href="web.archive.org/web/20191213112602/chanzuckerberg.com/rfa/essential-open-source-software-for-science">Reference Source</a></span></li><li><a name=ref-53 class=n-a></a><span class=label>53. </span> <span class=citation><a name=d11061e3300 class=n-a></a>Rodríguez-Sánchez F, Marwick B, Lazowska E, <i> et al.</i>: Academia's failure to retain data scientists. <i>Science.</i> 2017; <b>355</b>(6323): 357–358. <a target=xrefwindow id=d11061e3311 href="http://www.ncbi.nlm.nih.gov/pubmed/28126779">PubMed Abstract </a> | <a target=xrefwindow id=d11061e3314 href="https://doi.org/10.1126/science.aam6116">Publisher Full Text </a></span></li><li><a name=ref-54 class=n-a></a><span class=label>54. </span> <span class=citation><a name=d11061e3324 class=n-a></a>Katz DS, Druskat S, Haines R, <i> et al.</i>: The State of Sustainable Research Software: Learning from the Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE5.1). <i>J Open Res Softw.</i> 2019; <b>7</b>(1): 11. <a target=xrefwindow id=d11061e3335 href="https://doi.org/10.5334/jors.242">Publisher Full Text </a></span></li><li><a name=ref-55 class=n-a></a><span class=label>55. </span> <span class=citation><a name=d11061e3344 class=n-a></a>Druskat S, Katz DS: Mapping the Research Software Sustainability Space. In: <i>2018 IEEE 14th International Conference on E-Science (e-Science). </i>2018; 25–30. <a target=xrefwindow id=d11061e3349 href="https://doi.org/10.1109/eScience.2018.00014">Publisher Full Text </a></span></li><li><a name=ref-56 class=n-a></a><span class=label>56. </span> <span class=citation><a name=d11061e3358 class=n-a></a>Ye Y, Boyce RD, Davis MK, <i> et al.</i>: Open Source Software Sustainability Models: Initial White Paper from the Informatics Technology for Cancer Research Sustainability and Industry Partnership Work Group. 2019. <a target=xrefwindow id=d11061e3363 href="https://arxiv.org/ftp/arxiv/papers/1912/1912.12371.pdf">Reference Source</a></span></li><li><a name=ref-57 class=n-a></a><span class=label>57. </span> <span class=citation><a name=d11061e3372 class=n-a></a>Hong NC: Minimal information for reusable scientific software. In: <i>2nd Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE2). </i>2014. <a target=xrefwindow id=d11061e3377 href="https://doi.org/10.6084/m9.figshare.1112528.v1">Publisher Full Text </a></span></li><li><a name=ref-58 class=n-a></a><span class=label>58. </span> <span class=citation><a name=d11061e3386 class=n-a></a>Hasselbring W, Carr L, Hettrick S, <i> et al.</i>: From FAIR research data toward FAIR and open research software. <i>it - Information Technology.</i> 2020; <b>62</b>(1): 39–47. <a target=xrefwindow id=d11061e3397 href="https://doi.org/10.1515/itit-2019-0040">Publisher Full Text </a></span></li><li><a name=ref-59 class=n-a></a><span class=label>59. </span> <span class=citation><a name=d11061e3406 class=n-a></a><a target=xrefwindow id=d11061e3407 href="https://www.acm.org/publications/policies/artifact-review-badging">https://www.acm.org/publications/policies/artifact-review-badging</a>. </span></li><li><a name=ref-60 class=n-a></a><span class=label>60. </span> <span class=citation><a name=d11061e3418 class=n-a></a>Schlauch T, Meinel M, Haupt C: DLR Software Engineering Guidelines. Deutsches Zentrum für Luft- und Raumfahrt (DLR). 2018. <a target=xrefwindow id=d11061e3420 href="https://doi.org/10.5281/ZENODO.1344612">Publisher Full Text </a></span></li><li><a name=ref-61 class=n-a></a><span class=label>61. </span> <span class=citation><a name=d11061e3429 class=n-a></a>Gomez-Diaz T, Recio T: On the evaluation of research software: the CDUR procedure [version 2; peer review: 2 approved]. <i>F1000Res.</i> 2019; <b>8</b>: 1353. <a target=xrefwindow id=d11061e3437 href="http://www.ncbi.nlm.nih.gov/pubmed/31814965">PubMed Abstract </a> | <a target=xrefwindow id=d11061e3440 href="https://doi.org/10.12688/f1000research.19994.2">Publisher Full Text </a> | <a target=xrefwindow id=d11061e3443 href="http://www.ncbi.nlm.nih.gov/pmc/articles/6883399">Free Full Text </a></span></li><li><a name=ref-62 class=n-a></a><span class=label>62. </span> <span class=citation><a name=d11061e3452 class=n-a></a>Fehr J, Heiland J, Himpe C, <i> et al.</i>: Best practices for replicability, reproducibility and reusability of computer-based experiments exemplified by model reduction software. <i>AIMS Mathematics.</i> 2016; <b>1</b>(3): 261–281. <a target=xrefwindow id=d11061e3463 href="https://doi.org/10.3934/Math.2016.3.261">Publisher Full Text </a></span></li><li><a name=ref-63 class=n-a></a><span class=label>63. </span> <span class=citation><a name=d11061e3472 class=n-a></a>Jiménez RC, Kuzak M, Alhamdoosh M, <i> et al.</i>: Four simple recommendations to encourage best practices in research software [version 1; peer review: 3 approved]. <i>F1000Res.</i> 2017; <b>6</b>: pii: ELIXIR-876. <a target=xrefwindow id=d11061e3483 href="http://www.ncbi.nlm.nih.gov/pubmed/28751965">PubMed Abstract </a> | <a target=xrefwindow id=d11061e3486 href="https://doi.org/10.12688/f1000research.11407.1">Publisher Full Text </a> | <a target=xrefwindow id=d11061e3490 href="http://www.ncbi.nlm.nih.gov/pmc/articles/5490478">Free Full Text </a></span></li><li><a name=ref-64 class=n-a></a><span class=label>64. </span> <span class=citation><a name=d11061e3499 class=n-a></a>Hsu L, Hutchison VB, Langseth ML: Measuring sustainability of seed-funded earth science informatics projects. <i>PLoS One.</i> 2019; <b>14</b>(10): e0222807. <a target=xrefwindow id=d11061e3507 href="http://www.ncbi.nlm.nih.gov/pubmed/31644537">PubMed Abstract </a> | <a target=xrefwindow id=d11061e3510 href="https://doi.org/10.1371/journal.pone.0222807">Publisher Full Text </a> | <a target=xrefwindow id=d11061e3513 href="http://www.ncbi.nlm.nih.gov/pmc/articles/6808333">Free Full Text </a></span></li><li><a name=ref-65 class=n-a></a><span class=label>65. </span> <span class=citation><a name=d11061e3522 class=n-a></a>Jackson M, Crouch S, Baxter R: Software Evaluation Guide. 2019. <a target=xrefwindow id=d11061e3524 href="https://www.software.ac.uk/resources/guides-everything/software-evaluation-guide">Reference Source</a></span></li><li><a name=ref-66 class=n-a></a><span class=label>66. </span> <span class=citation><a name=d11061e3534 class=n-a></a>rOpenSci; Anderson B, Chamberlain S, <i> et al.</i>: Software Peer Review, Why? What? In: <i>rOpenSci Packages: Development, Maintenance, and Peer Review Zenodo. </i>2019. <a target=xrefwindow id=d11061e3542 href="https://doi.org/10.5281/zenodo.2554759">Publisher Full Text </a></span></li><li><a name=ref-67 class=n-a></a><span class=label>67. </span> <span class=citation><a name=d11061e3551 class=n-a></a>Lee BD: Ten simple rules for documenting scientific software. <i>PLoS Comput Biol.</i> 2018; <b>14</b>(12): e1006561. <a target=xrefwindow id=d11061e3559 href="http://www.ncbi.nlm.nih.gov/pubmed/30571677">PubMed Abstract </a> | <a target=xrefwindow id=d11061e3562 href="https://doi.org/10.1371/journal.pcbi.1006561">Publisher Full Text </a> | <a target=xrefwindow id=d11061e3565 href="http://www.ncbi.nlm.nih.gov/pmc/articles/6301674">Free Full Text </a></span></li><li><a name=ref-68 class=n-a></a><span class=label>68. </span> <span class=citation><a name=d11061e3574 class=n-a></a>Venters CC, Capilla R, Betz S, <i> et al.</i>: Software sustainability: Research and practice from a software architecture viewpoint. <i>J Syst Software.</i> 2018; <b>138</b>: 174–188. <a target=xrefwindow id=d11061e3585 href="https://doi.org/10.1016/j.jss.2017.12.026">Publisher Full Text </a></span></li><li><a name=ref-69 class=n-a></a><span class=label>69. </span> <span class=citation><a name=d11061e3594 class=n-a></a>SSI: Writing and using a software management plan. 2019. <a target=xrefwindow id=d11061e3596 href="https://www.software.ac.uk/resources/guides/software-management-plans">Reference Source</a></span></li><li><a name=ref-70 class=n-a></a><span class=label>70. </span> <span class=citation><a name=d11061e3605 class=n-a></a>Johanson A, Hasselbring W: Software engineering for computational science: Past, present, future. <i>Comput Sci Eng.</i> 2018; <b>20</b>(2): 90–109. <a target=xrefwindow id=d11061e3613 href="https://doi.org/10.1109/MCSE.2018.021651343">Publisher Full Text </a></span></li><li><a name=ref-71 class=n-a></a><span class=label>71. </span> <span class=citation><a name=d11061e3622 class=n-a></a>Akhmerov A, Cruz M, Drost N, <i> et al.</i>: Making Research Software a First-Class Citizen in Research. 2019. <a target=xrefwindow id=d11061e3627 href="https://doi.org/10.5281/zenodo.2647436">Publisher Full Text </a></span></li><li><a name=ref-72 class=n-a></a><span class=label>72. </span> <span class=citation><a name=d11061e3637 class=n-a></a>Hong NC: Making Software A First-Class Citizen. 2019. <a target=xrefwindow id=d11061e3639 href="https://doi.org/10.6084/m9.figshare.9862835.v1">Publisher Full Text </a></span></li><li><a name=ref-73 class=n-a></a><span class=label>73. </span> <span class=citation><a name=d11061e3648 class=n-a></a>Vereinigung der Kanzlerinnen und Kanzler der Universitäten Deutschlands: Bayreuther Erklärung zu befristeten Beschäftigungsverhältnissen mit wissenschaftlichem und künstlerischem Personal in Universitäten. 2019. <a target=xrefwindow id=d11061e3650 href="https://www.uni-kanzler.de/fileadmin/user_upload/05_Publikationen/2017_-_2010/20190919_Bayreuther_Erklaerung_der_Universitaetskanzler_final.pdf">Reference Source</a></span></li><li><a name=ref-74 class=n-a></a><span class=label>74. </span> <span class=citation><a name=d11061e3659 class=n-a></a>Katz DS, McInnes LC, Bernholdt DE, <i> et al.</i>: Community Organizations: Changing the Culture in Which Research Software Is Developed and Sustained. <i>Comput Sci Eng.</i> 2019; <b>21</b>(2): 8–24. <a target=xrefwindow id=d11061e3670 href="https://doi.org/10.1109/MCSE.2018.2883051">Publisher Full Text </a></span></li><li><a name=ref-75 class=n-a></a><span class=label>75. </span> <span class=citation><a name=d11061e3679 class=n-a></a>Science Guide: Room for everyone’s talent. 2019. <a target=xrefwindow id=d11061e3681 href="https://www.scienceguide.nl/wp-content/uploads/2019/11/283.002-Erkennen-en-Waarderen-Position-Paper_EN_web.pdf">Reference Source</a></span></li><li><a name=ref-76 class=n-a></a><span class=label>76. </span> <span class=citation><a name=d11061e3690 class=n-a></a>Carver JC, Hong NPC, Thiruvathukal GK: Software engineering for science. CRC Press, 2016; 274. <a target=xrefwindow id=d11061e3692 href="https://www.routledge.com/Software-Engineering-for-Science/Carver-Hong-Thiruvathukal/p/book/9781498743853">Reference Source</a></span></li><li><a name=ref-77 class=n-a></a><span class=label>77. </span> <span class=citation><a name=d11061e3701 class=n-a></a>Iaffaldano G, Steinmacher I, Calefato F, <i> et al.</i>: Why do developers take breaks from contributing to OSS projects? A preliminary analysis. <i>arXiv: 1903.09528.</i> 2019. <a target=xrefwindow id=d11061e3709 href="https://arxiv.org/abs/1903.09528">Reference Source</a></span></li><li><a name=ref-78 class=n-a></a><span class=label>78. </span> <span class=citation><a name=d11061e3719 class=n-a></a>Allen A, Aragon C, Becker C, <i> et al.</i>: Engineering Academic Software (Dagstuhl Perspectives Workshop 16252). <i>Dagstuhl Manifestos</i> 2017; <b>6</b>(1): 1–20. <a target=xrefwindow id=d11061e3730 href="https://drops.dagstuhl.de/opus/volltexte/2017/7146/pdf/dagman-v006-i001-p001-16252.pdf">Reference Source</a></span></li><li><a name=ref-79 class=n-a></a><span class=label>79. </span> <span class=citation><a name=d11061e3739 class=n-a></a>Bundesministerium der Justiz und für Verbraucherschutz: Gesetz über befristete Arbeitsverträge in der Wissenschaft. 2017. <a target=xrefwindow id=d11061e3741 href="https://www.gesetze-im-internet.de/wisszeitvg">Reference Source</a></span></li><li><a name=ref-80 class=n-a></a><span class=label>80. </span> <span class=citation><a name=d11061e3750 class=n-a></a>Chang V, Mills H, Newhouse S: From Open Source to long-term sustainability: Review of Business Models and Case studies. In: Proceedings of the UK e-Science All Hands Meeting 2007 University of Edinburgh/University of Glasgow (acting through the NeSC) 2007. <a target=xrefwindow id=d11061e3752 href="http://eprints.leedsbeckett.ac.uk/649/">Reference Source</a></span></li><li><a name=ref-81 class=n-a></a><span class=label>81. </span> <span class=citation><a name=d11061e3761 class=n-a></a>Aartsen W, Peeters P, Wagers S, <i> et al.</i>: Getting Digital Assets from Public-Private Partnership Research Projects through “The Valley of Death,” and Making Them Sustainable. <i>Front Med (Lausanne).</i> 2018; <b>5</b>: 65. <a target=xrefwindow id=d11061e3772 href="http://www.ncbi.nlm.nih.gov/pubmed/29594123">PubMed Abstract </a> | <a target=xrefwindow id=d11061e3775 href="https://doi.org/10.3389/fmed.2018.00065">Publisher Full Text </a> | <a target=xrefwindow id=d11061e3779 href="http://www.ncbi.nlm.nih.gov/pmc/articles/5855043">Free Full Text </a></span></li><li><a name=ref-82 class=n-a></a><span class=label>82. </span> <span class=citation><a name=d11061e3788 class=n-a></a>Gabella C, Durinx C, Appel R: Funding knowledgebases: Towards a sustainable funding model for the UniProt use case [version 2; peer review: 3 approved]. <i>F1000Res.</i> 2018; <b>6</b>: pii: ELIXIR-2051. <a target=xrefwindow id=d11061e3796 href="http://www.ncbi.nlm.nih.gov/pubmed/29333230">PubMed Abstract </a> | <a target=xrefwindow id=d11061e3799 href="https://doi.org/10.12688/f1000research.12989.2">Publisher Full Text </a> | <a target=xrefwindow id=d11061e3802 href="http://www.ncbi.nlm.nih.gov/pmc/articles/5747334 ">Free Full Text </a></span></li><li><a name=ref-83 class=n-a></a><span class=label>83. </span> <span class=citation><a name=d11061e3811 class=n-a></a>Ison J, Rapacki K, Ménager H, <i> et al.</i>: Tools and data services registry: a community effort to document bioinformatics resources. <i>Nucleic Acids Res.</i> 2016; <b>44</b>(D1): D38–D47. <a target=xrefwindow id=d11061e3822 href="http://www.ncbi.nlm.nih.gov/pubmed/26538599">PubMed Abstract </a> | <a target=xrefwindow id=d11061e3825 href="https://doi.org/10.1093/nar/gkv1116">Publisher Full Text </a> | <a target=xrefwindow id=d11061e3829 href="http://www.ncbi.nlm.nih.gov/pmc/articles/4702812 ">Free Full Text </a></span></li><li><a name=ref-84 class=n-a></a><span class=label>84. </span> <span class=citation><a name=d11061e3839 class=n-a></a>Struck A: Research Software Discovery: An Overview. In: <i>2018 IEEE 14th International Conference on e-Science IEEE</i>. 2018. <a target=xrefwindow id=d11061e3844 href="https://doi.org/10.1109/escience.2018.00016">Publisher Full Text </a></span></li><li><a name=ref-85 class=n-a></a><span class=label>85. </span> <span class=citation><a name=d11061e3853 class=n-a></a>Druskat S, Spaaks JH, Chue Hong N, <i> et al.</i>: Citation File Format (CFF) - Specifications. 2019. <a target=xrefwindow id=d11061e3858 href="https://doi.org/10.5281/zenodo.3515946">Publisher Full Text </a></span></li><li><a name=ref-86 class=n-a></a><span class=label>86. </span> <span class=citation><a name=d11061e3867 class=n-a></a>re3data.org – Registry of Research Data Repositories. <a target=xrefwindow id=d11061e3869 href="https://doi.org/10.17616/R3D">Publisher Full Text </a></span></li><li><a name=ref-87 class=n-a></a><span class=label>87. </span> <span class=citation><a name=d11061e3878 class=n-a></a>Mönnich MW: KVK - a meta catalog of libraries. <i>LIBER Quarterly.</i> 2001; <b>11</b>(2): 121–127. <a target=xrefwindow id=d11061e3886 href="https://doi.org/10.18352/lq.7638">Publisher Full Text </a></span></li><li><a name=ref-88 class=n-a></a><span class=label>88. </span> <span class=citation><a name=d11061e3895 class=n-a></a>Kunze J, Scancella J, Adams C, <i> et al.</i>: The bagIt file packaging format (v1. 0). RFC Editor. 2018; 8493. </span></li><li><a name=ref-89 class=n-a></a><span class=label>89. </span> <span class=citation><a name=d11061e3907 class=n-a></a>Bundesministerium der Justiz und für Verbraucherschutz, § 69a subsection (1) UrhG. 2014. <a target=xrefwindow id=d11061e3909 href="https://www.gesetze-im-internet.de/urhg/__69a.html">Reference Source</a></span></li><li><a name=ref-90 class=n-a></a><span class=label>90. </span> <span class=citation><a name=d11061e3919 class=n-a></a>Bundesministerium der Justiz und für Verbraucherschutz, § 69b UrhG. 2014. <a target=xrefwindow id=d11061e3921 href="https://www.gesetze-im-internet.de/urhg/__69b.html">Reference Source</a></span></li><li><a name=ref-91 class=n-a></a><span class=label>91. </span> <span class=citation><a name=d11061e3930 class=n-a></a>Struck A, Loewe A, Achhammer E, <i> et al.</i>: A Guide for Publishing, Using, and Licensing Research Software in Germany. <i>Zenodo.</i> 2020. <a target=xrefwindow id=d11061e3938 href="http://www.doi.org/10.5281/zenodo.4327147">http://www.doi.org/10.5281/zenodo.4327147</a></span></li></ul></div></div></div> </div> <div id=article-comments class="article-comments padding-bottom-20"> <div class=current-article-comment-section> <h2 class=main-title name=add-new-comment id=add-new-comment> <span class="research-layout f1r-article-mobile-inline valign-middle"> <span class="f1r-icon icon-104_comments size30"></span> </span> <span class=f1r-article-desk-inline>Comments on this article</span> <span class=f1r-article-mobile-inline>Comments (3)</span> <span class="f1r-article-mobile-inline float-right"> <span class="f1r-icon icon-14_more_small"></span> <span class="f1r-icon icon-10_less_small"></span> </span> </h2> </div> <div class=comments-list-container> <div class="f1r-article-desk-inline referee-report-info-box referee-report-version-box"> Version 2 </div> <div class="f1r-article-mobile research-layout mobile-version-info padding-top-30"> <span class="mversion ">VERSION 2</span> <span class="details u-ml--1">PUBLISHED 26 Jan 2021</span> <span class="article-pubinfo-mobile versions-section"> <div title="#springMessage("NewVersionReasonType.REVISED.tooltip")" class="version-info-box revised ">Revised</div> </span> </div> <ul class=all-comment-wrapper data-reportID=NONE data-versionID=2> <li class="comment-wrapper not-sortable " data-commentID=6376> <div class="one-comment version-comments "> <a name=article-comment-6376></a> <div class="f1r-article-desk commenter-role"> Reader Comment 22 Feb 2021 </div> <div> <span class=name>Dirk Feuchter</span><span class=name>, </span><span class=commenter-affiliation>Karlsruhe Institute of Technology, Karlsruhe, Germany</span> </div> <div class="f1r-article-mobile the-comment-date"> 22 Feb 2021 </div> <div class=f1r-article-mobile> <div class="article-pubinfo-mobile comments-section"> <span class="version-info-box comments"> Reader Comment </span> </div> </div> <div class="comment-content article-comment-list"> <span class=truncated-comment> Dear Mr. Anzt, dear Mr. Loewe, dear Mr. Bach, dear Mr. Seemann, as well as dear other authors and reviewers,<br/> <br/> just discovered your feedback to my Comment. Thank you ... <span class="continue-reading-container response-and-comment">Continue reading</span> </span> <span class="full-comment hidden"> <span id=full-comment-original-6376 class=hidden>Dear Mr. Anzt, dear Mr. Loewe, dear Mr. Bach, dear Mr. Seemann, as well as dear other authors and reviewers,<br/> <br/> just discovered your feedback to my Comment. Thank you very much for that.<br/> <br/> <i>I can well understand that your community has in mind proprietary licensing at most in exceptional situations.</i><br/> <br/> <i>Please note and obviously you keep this already in mind, that even in the case of pure F(L)OSS licensing, i.e. completely without proprietary licensing intentions the topic of "F(L)OSS-right compliance" should be considered especially in the sense of sustainability of the software.</i><br/> <br/> <i>Hence,and only as a suggestion, the „OpenChain movement“ and the new ISO 5230 standard resulting from it at the end of 2020 should possibly be mentioned and cited in your paper; </i>possibly in „Legal Aspects -> Challenges and clarifications“. after your sentence „…The challenge in the use, distribution, and commercialization of software is to determine the chain of rights and to identify all right holders….“<br/> <br/> <i>Best regards</i><br/> <br/> <i>Dirk Feuchter</i></span> <div class=padding-bottom id=full-comment-6376>Dear Mr. Anzt, dear Mr. Loewe, dear Mr. Bach, dear Mr. Seemann, as well as dear other authors and reviewers,<br/> <br/> just discovered your feedback to my Comment. Thank you very much for that.<br/> <br/> <i>I can well understand that your community has in mind proprietary licensing at most in exceptional situations.</i><br/> <br/> <i>Please note and obviously you keep this already in mind, that even in the case of pure F(L)OSS licensing, i.e. completely without proprietary licensing intentions the topic of "F(L)OSS-right compliance" should be considered especially in the sense of sustainability of the software.</i><br/> <br/> <i>Hence,and only as a suggestion, the „OpenChain movement“ and the new ISO 5230 standard resulting from it at the end of 2020 should possibly be mentioned and cited in your paper; </i>possibly in „Legal Aspects -> Challenges and clarifications“. after your sentence „…The challenge in the use, distribution, and commercialization of software is to determine the chain of rights and to identify all right holders….“<br/> <br/> <i>Best regards</i><br/> <br/> <i>Dirk Feuchter</i></div> <span class="disclosures competing-interests-display padding-top"> <strong>Competing Interests:</strong> <span class=c-i-d>No competing interests were disclosed.</span> </span> <span class="hide-comment-container response-and-comment">Close</span> </span> <div class="report-concern-wrapper last-comment"> <a href="mailto:research@f1000.com?subject=Report a concern (23224) " class=report-a-concern-link>Report a concern</a> </div> </div> </div> </li> <li class="comment-wrapper not-sortable button-wrapper"> <div class="one-comment version-comments add-the-comment"> <a href="/login?originalPath=/articles/9-295/v2&scrollTo=add-new-comments" class=register-report-comment-button data-test-id=add-comment> <span class=f1r-article-desk><span class=contracted></span>Comment</span> <span class="f1r-article-mobile research-layout "> <button class="primary orange extra-padding comment-on-this-article">ADD YOUR COMMENT</button> </span> </a> </div> </li> </ul> <div class="f1r-article-mobile research-layout"> <div class="mobile-sections-divider versions-divider"></div> </div> <div class="f1r-article-desk-inline referee-report-info-box referee-report-version-box"> Version 1 </div> <div class="f1r-article-mobile research-layout mobile-version-info padding-top-30"> <span class="mversion previous-version">VERSION 1</span> <span class="details u-ml--1">PUBLISHED 27 Apr 2020</span> <span class="article-pubinfo-mobile versions-section"> </span> </div> <div class="comments-closed f1r-article-mobile"> Discussion is closed on this version, please comment on the latest version above. </div> <ul class=all-comment-wrapper data-reportID=NONE data-versionID=1> <li class="comment-wrapper not-sortable " data-commentID=6211> <div class="one-comment version-comments "> <a name=article-comment-6211></a> <div class="f1r-article-desk commenter-role"> Author Response 26 Jan 2021 </div> <div> <span class=name>Axel Loewe</span><span class=name>, </span><span class=commenter-affiliation>Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany</span> </div> <div class="f1r-article-mobile the-comment-date"> 26 Jan 2021 </div> <div class=f1r-article-mobile> <div class="article-pubinfo-mobile comments-section"> <span class="version-info-box comments"> Author Response </span> </div> </div> <div class="comment-content article-comment-list"> <span class=truncated-comment> <i>We thank you for your thorough review and constructive feedback regarding the manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> Dear ... <span class="continue-reading-container response-and-comment">Continue reading</span> </span> <span class="full-comment hidden"> <span id=full-comment-original-6211 class=hidden><i>We thank you for your thorough review and constructive feedback regarding the manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> Dear Mr. Anzt, dear Mr. Loewe, dear Mr. Bach, dear Mr. Seemann, dear Elke, and dear Sven, as well as dear authors as yet unknown to me,<br/> I am working at KIT in the field of Innovation and Relations Management, especially Licensing of Intellectual Property Rights from KIT to Free Maket Economy. My particual focus is to Out-license Computerprograms to Third Parties and Industry.<br/> Thank you for your great efforts with your extensive and intersting article including the nicely written and ever-recurring story of Kim.<br/> Your FLOSS-based approach for sustainable software devlopment is holding immense savings potential. That’s great.<br/> Following are my comments to your article from the perspective of a TTO license manager, typically supporting RSEs in cases of proprietary licensing e.g. to spin-offs or industrial companies.<br/> Hence, most of my comments might go in a slightly different direction than the main focus of your article, but in my point of view these comments are complementary. Hence, I wonder what you think of them. Looking forward to your opinion and you feedback to my comments, if you like also gladly by telephone<br/> Best regards<br/> Dirk Feuchter<br/> <i>Thank you very much for your comments regarding distribution issues, IP compliance and commercialization. We have generally subsumed these issues under the more general term “availability”. We believe that this implies that software is legally available only if it is licensed, either under an open source license or proprietarily. This impacts possible modes of distribution, but does not concern actual (commercial) distribution processes. As a community of Research Software Engineers, we strongly believe that publicly funded research software should be F(L)OSS-licensed per default, although we recognize that this may not always be possible (and hence have weakened “must” to “should” as you suggested in the respective paragraph under “Why sustainable research software in the first place”). We have further avoided any changes to the text that would weaken this point, as we see no central obstacles to making publicly funded research software open source in general. Contrarily, we do not accept that IP should override public interest (both intellectually and fiscally) here, some corner cases excluded. Concurrently, we purposefully do not focus on marketability and commercialization of research software. Instead, we see commercial opportunities, e.g., in the provision of services for a research software product, whereas the product itself should remain free and open source.<br/> Thank you, also, for notifying us of some errors in the text itself and the figures, which we are fixing in the next version.<br/> Below, we address some of your concrete suggestions in more detail:</i><br/> <br/> Abstract:<br/> I would like to suggest to extend<br/> „Research software must be sustainable in order to<br/> understand, replicate, reproduce, and…“<br/> as follows<br/> „Research software must be sustainable in order to<br/> understand, replicate, reproduce, distribute and…“ <br/> <i>We thank you for the comment. The development of research software does not focus on distribution of software results, which is more of a business aspect. The availability, regardless of the actual distribution, is noted in the next sentence of the abstract.</i><br/> <br/> Abstract:<br/> I would like to suggest to extend<br/> „In other words, software must be available, discoverable, usable, and adaptable to new needs, both now and in the future.“<br/> as follows<br/> „In other words, software must be available, (IP-/FLOSS)compliant*, discoverable, usable, and adaptable to new needs, both now and in the future.“<br/> *By „(IP-/FLOSS)compliant“ I mean<br/> in compliance<br/> with intellectual property of third-party suppliers,<br/> with the terms of free/libre open source licenses and with the aim to protect own intellectual property from unintended disclosure <br/> <i>We thank you for the comment. As a community of RSEs we are aiming for FLOSS whenever feasible but acknowledge that there are scenarios in which non-FLOSS licenses need to be considered.</i><br/> <br/> Suggestion of a new penultimate paragraph in subchapter „Stakeholder motivations for research software sustainability“ as follows: RSEs, Research leaders and research performing organisations are interested in software sustainability also in the sense that their (research)software is sustainable concerning legal compliance.This is an important issue distributing (research)software for both acaedmic puposes as well as commerical purposes. For the latter RSEs and their research leaders (typically contacting their TTO) are interested to marktet (parts of) their research software and/or additional connectable closed software (which they prevent from unintended disclosure) to a spin-off or an industrial company using a proprietary or a dual licensing model. For both, acaedmic and commerical purposes RSEs and their research leaders and their research performing organisation are interested that their (research) software plus any connectable closed software is compliant with intellectual property of third-party suppliers and compliant with the terms of free/libre open source licenses. An important stakeholder motivation is therefore „software sustainbility with the aim of clarification of all software rights ownerships“. <br/> <i>We thank you for the suggestion. While we aim for reusable software in terms of licenses we as a community do not focus on commercial purposes.</i><br/> <br/> Please replace Subchapter heading„Challenges and clarifications Clarification of rights“by „Challenges and Clarification of rights“ <br/> <i>We thank you for pointing this out and corrected the structure (section and subsection headings, respectively).</i><br/> <br/> The term "research institution" appears in the glossary as a duplicate. Therefore, "research institution" should be deleted in the brackets.<br/> <i>Thank you, we have replaced it by “national labs”.</i></span> <div class=padding-bottom id=full-comment-6211><i>We thank you for your thorough review and constructive feedback regarding the manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> Dear Mr. Anzt, dear Mr. Loewe, dear Mr. Bach, dear Mr. Seemann, dear Elke, and dear Sven, as well as dear authors as yet unknown to me,<br/> I am working at KIT in the field of Innovation and Relations Management, especially Licensing of Intellectual Property Rights from KIT to Free Maket Economy. My particual focus is to Out-license Computerprograms to Third Parties and Industry.<br/> Thank you for your great efforts with your extensive and intersting article including the nicely written and ever-recurring story of Kim.<br/> Your FLOSS-based approach for sustainable software devlopment is holding immense savings potential. That’s great.<br/> Following are my comments to your article from the perspective of a TTO license manager, typically supporting RSEs in cases of proprietary licensing e.g. to spin-offs or industrial companies.<br/> Hence, most of my comments might go in a slightly different direction than the main focus of your article, but in my point of view these comments are complementary. Hence, I wonder what you think of them. Looking forward to your opinion and you feedback to my comments, if you like also gladly by telephone<br/> Best regards<br/> Dirk Feuchter<br/> <i>Thank you very much for your comments regarding distribution issues, IP compliance and commercialization. We have generally subsumed these issues under the more general term “availability”. We believe that this implies that software is legally available only if it is licensed, either under an open source license or proprietarily. This impacts possible modes of distribution, but does not concern actual (commercial) distribution processes. As a community of Research Software Engineers, we strongly believe that publicly funded research software should be F(L)OSS-licensed per default, although we recognize that this may not always be possible (and hence have weakened “must” to “should” as you suggested in the respective paragraph under “Why sustainable research software in the first place”). We have further avoided any changes to the text that would weaken this point, as we see no central obstacles to making publicly funded research software open source in general. Contrarily, we do not accept that IP should override public interest (both intellectually and fiscally) here, some corner cases excluded. Concurrently, we purposefully do not focus on marketability and commercialization of research software. Instead, we see commercial opportunities, e.g., in the provision of services for a research software product, whereas the product itself should remain free and open source.<br/> Thank you, also, for notifying us of some errors in the text itself and the figures, which we are fixing in the next version.<br/> Below, we address some of your concrete suggestions in more detail:</i><br/> <br/> Abstract:<br/> I would like to suggest to extend<br/> „Research software must be sustainable in order to<br/> understand, replicate, reproduce, and…“<br/> as follows<br/> „Research software must be sustainable in order to<br/> understand, replicate, reproduce, distribute and…“ <br/> <i>We thank you for the comment. The development of research software does not focus on distribution of software results, which is more of a business aspect. The availability, regardless of the actual distribution, is noted in the next sentence of the abstract.</i><br/> <br/> Abstract:<br/> I would like to suggest to extend<br/> „In other words, software must be available, discoverable, usable, and adaptable to new needs, both now and in the future.“<br/> as follows<br/> „In other words, software must be available, (IP-/FLOSS)compliant*, discoverable, usable, and adaptable to new needs, both now and in the future.“<br/> *By „(IP-/FLOSS)compliant“ I mean<br/> in compliance<br/> with intellectual property of third-party suppliers,<br/> with the terms of free/libre open source licenses and with the aim to protect own intellectual property from unintended disclosure <br/> <i>We thank you for the comment. As a community of RSEs we are aiming for FLOSS whenever feasible but acknowledge that there are scenarios in which non-FLOSS licenses need to be considered.</i><br/> <br/> Suggestion of a new penultimate paragraph in subchapter „Stakeholder motivations for research software sustainability“ as follows: RSEs, Research leaders and research performing organisations are interested in software sustainability also in the sense that their (research)software is sustainable concerning legal compliance.This is an important issue distributing (research)software for both acaedmic puposes as well as commerical purposes. For the latter RSEs and their research leaders (typically contacting their TTO) are interested to marktet (parts of) their research software and/or additional connectable closed software (which they prevent from unintended disclosure) to a spin-off or an industrial company using a proprietary or a dual licensing model. For both, acaedmic and commerical purposes RSEs and their research leaders and their research performing organisation are interested that their (research) software plus any connectable closed software is compliant with intellectual property of third-party suppliers and compliant with the terms of free/libre open source licenses. An important stakeholder motivation is therefore „software sustainbility with the aim of clarification of all software rights ownerships“. <br/> <i>We thank you for the suggestion. While we aim for reusable software in terms of licenses we as a community do not focus on commercial purposes.</i><br/> <br/> Please replace Subchapter heading„Challenges and clarifications Clarification of rights“by „Challenges and Clarification of rights“ <br/> <i>We thank you for pointing this out and corrected the structure (section and subsection headings, respectively).</i><br/> <br/> The term "research institution" appears in the glossary as a duplicate. Therefore, "research institution" should be deleted in the brackets.<br/> <i>Thank you, we have replaced it by “national labs”.</i></div> <span class="disclosures competing-interests-display padding-top"> <strong>Competing Interests:</strong> <span class=c-i-d>No competing interests were disclosed.</span> </span> <span class="hide-comment-container response-and-comment">Close</span> </span> <div class="report-concern-wrapper last-comment"> <a href="mailto:research@f1000.com?subject=Report a concern (23224) " class=report-a-concern-link>Report a concern</a> </div> </div> </div> </li> <li class="comment-wrapper not-sortable " data-commentID=5522> <div class="one-comment version-comments "> <a name=article-comment-5522></a> <div class="f1r-article-desk commenter-role"> Reader Comment 19 May 2020 </div> <div> <span class=name>Dirk Feuchter</span><span class=name>, </span><span class=commenter-affiliation>Karlsruhe Institute of Technology (KIT), Eggenstein-Leopoldshafen and Karlsruhe, Germany</span> </div> <div class="f1r-article-mobile the-comment-date"> 19 May 2020 </div> <div class=f1r-article-mobile> <div class="article-pubinfo-mobile comments-section"> <span class="version-info-box comments"> Reader Comment </span> </div> </div> <div class="comment-content article-comment-list"> <span class=truncated-comment> Subject: Feedback to your article „An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action…“.<br/> <br/>Dear Mr. Anzt, dear Mr. Loewe, dear Mr. ... <span class="continue-reading-container response-and-comment">Continue reading</span> </span> <span class="full-comment hidden"> <span id=full-comment-original-5522 class=hidden>Subject: Feedback to your article „An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action…“.<br/> <br/>Dear Mr. Anzt, dear Mr. Loewe, dear Mr. Bach, dear Mr. Seemann, dear Elke, and dear Sven, as well as dear authors as yet unknown to me,<br/>I am working at KIT in the field of Innovation and Relations Management, especially Licensing of Intellectual Property Rights from KIT to Free Maket Economy. My particual focus is to Out-license Computerprograms to Third Parties and Industry.<br/>Thank you for your great efforts with your extensive and intersting article including the nicely written and ever-recurring story of Kim.<br/>Your FLOSS-based approach for sustainable software devlopment is holding immense savings potential. That’s great.<br/>Following are my comments to your article from the perspective of a TTO license manager, typically supporting RSEs in cases of proprietary licensing e.g. to spin-offs or industrial companies.<br/>Hence, most of my comments might go in a slightly different direction than the main focus of your article, but in my point of view these comments are complementary. Hence, I wonder what you think of them. Looking forward to your opinion and you feedback to my comments, if you like also gladly by telephone<br/>Best regards<br/>Dirk Feuchter<br/> <br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-01:<br/>Abstract:<br/>I would like to suggest to extend<br/>„Research software must be sustainable in order to<br/>understand, replicate, reproduce, and…“<br/>as follows<br/>„Research software must be sustainable in order to<br/>understand, replicate, reproduce, <b>distribute</b> and…“ <br/> <br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-02:<br/>Abstract:<br/>I would like to suggest to extend<br/>„In other words, software must be available, discoverable, usable, and adaptable to new needs, both now and in the future.“<br/> <br/>as follows<br/> <br/>„In other words, software must be available, (IP-/FLOSS)<b>compliant*</b>, discoverable, usable, and adaptable to new needs, both now and in the future.“<br/> <br/>*By „(IP-/FLOSS)<b>compliant“</b><i> I mean</i><br/><i>in </i>compliance <ul> <li>with intellectual property of third-party suppliers,</li> <li>with the terms of free/libre open source licenses and with the aim to protect own intellectual property from unintended disclosure </li> </ul> # # # # # # # # # # # # # # # # # # # # # # #Comment-DF-03:Why sustainable research software in the first place =>I would like to suggest to extend„In order to support research, a sustainable software must be correct<a href="https://f1000research.com/articles/9-295/v1#ref-14"><sup>14</sup></a>, validatable, understandable, documented, publicly released,...“ as follows „In order to support research, a sustainable software must be correct<a href="https://f1000research.com/articles/9-295/v1#ref-14"><sup>14</sup></a>, (IP-/FLOSS)<b>compliant* </b>validatable, understandable, documented, publicly released,...“ *Concerning „(IP-/FLOSS)<b>compliant“</b> please see upon # # # # # # # # # # # # # # # # # # # # # # # <ul> <li>Comment-DF-04:Why sustainable research software in the first place =>„We also argue that truly sustainable research software must ideallybe published under a Free/Libre Open Source Software (FLOSS) license, and…“„We also argue that truly sustainable research software <b>should typically</b> be published under a Free/Libre Open Source Software (FLOSS) license, and…“</li> </ul> # # # # # # # # # # # # # # # # # # # # # # #<br/><br/>Comment-DF-05:Why sustainable research software in the first place =>„Therefore, if research software is publicly funded, it should be freely available under a FLOSS license.„Therefore, if research software is publicly funded, it should be <b>normally</b> freely available under a FLOSS license.<br/># # # # # # # # # # # # # # # # # # # # # # #<br/><br/>Comment-DF-06:Why sustainable research software in the first place =>I would like to suggest to extend„Legal issues: Many obstacles for research software pertain to legal issues, such as applicable licensing and compatibility of licenses<a href="https://f1000research.com/articles/9-295/v1#ref-45"><sup>45</sup></a>, and decisions about license types. as follows „Legal issues: Many obstacles for research software pertain to legal issues, such as <b>IT law, copyright law, copyright notices and author attributions,</b> applicable licensing and compatibility of licenses<a href="https://f1000research.com/articles/9-295/v1#ref-45"><sup>45</sup></a>, and decisions about license types.<br/><br/> # # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-07:Why sustainable research software in the first place =>„A subset of this <b>group</b> may be interested in …“.Which „group“ do you mean? => the „Taxpayers“? # # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-08:Suggestion of a new penultimate paragraph in subchapter „Stakeholder motivations for research software sustainability“ as follows: RSEs, Research leaders and research performing organisations are interested in software sustainability also in the sense that their (research)software is sustainable concerning legal compliance.This is an important issue distributing (research)software for both acaedmic puposes as well as commerical purposes. For the latter RSEs and their research leaders (typically contacting their TTO) are interested to marktet (parts of) their research software and/or additional connectable closed software (which they prevent from unintended disclosure) to a spin-off or an industrial company using a proprietary or a dual licensing model. For both, acaedmic and commerical purposes RSEs and their research leaders and their research performing organisation are interested that their (research) software plus any connectable closed software is compliant with intellectual property of third-party suppliers and compliant with the terms of free/libre open source licenses. An important stakeholder motivation is therefore „software sustainbility with the aim of clarification of all software rights ownerships“. <br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-09:Abstract:I would like to suggest to extend„Failing to do so will threaten the quality and validity of research..“ as follows „Failing to do so will threaten the quality, Xi and validity of research..“ // X1= marketability or X2= distribution // As TTO-license-manager I personally would prefer X1 but X2 is fine as well <br/><br/># # # # # # # # # # # # # # # # # # # # # # #Comment-DF-10:In the heading such as „Why sustainable research software in the first place?“, one might prefix a chapter number, just as announced at the end of the introductory background: „This all leads to unmet requirements and unsolved challenges that we want to highlight in this paper by elaborating on (1)…(2)…(3)…(4)…(5)…(6)…“Hence, instead of„Why sustainable research software in the first place?“use„(1) Why sustainable research software in the first place?“and so on:(2) How to decide which software to sustain?(3) Who sustains research software?(4) How can research software be sustainably funded?(5) Which infrastructure is needed to sustain research software?(6) Legal aspects.That's a matter of taste, of course. <br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/><br/>Comment-DF-11:Please replace Subchapter heading„Challenges and clarifications Clarification of rights“by „Challenges and Clarification of rights“ <br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-12:In both Fig. 1 and „Legal aspects/Challenges and clarifications of rights“instead of „Subject to directives“ {weisungsgebunden dt.} from UrhG 69 bI propose to write „in execution of his duties {in Wahrnehmung seiner Aufgaben dt.}“ from UrhG 69b, in no way to constrain scientists, RSEs and Research leaders, but in order to free up the scope for decision-making and thus open up opportunities.[Regardless of that, I would translate the German "weisungsgebunden" from „UrhG 69 b“ with "bound by instructions"(short) or with "following the instructions given by his employer"(long)]<br/><br/> # # # # # # # # # # # # # # # # # # # # # # # <ul> <li>Comment-DF-13:In Subchapter „Conclusions“I would not write the following sentence, or at most I would write it differently.„We encourage the research funding bodies to reflect the licensing models for academic software development, and to decide whether the “public money, public code” paradigm justifies the requirement that all publicly funded software has to be publicly available under a Free/Libre Open Source Software (FLOSS) license.“ I can understand your point of view and this sentence. But isn't there a lack of perspective regarding those research leaders or RSE-teams, who are considering a foundation based on a proprietary license model regarding their software development or RSE teams whose software developments are taken over and marketed by industrial companies in or outside of Germany, without return to the research performing organizations and the corresponding RSE teams.Hence, an alternative formulation might be as follows: „We encourage the research funding bodies to reflect the licensing models for academic software development, and to suggest research performing organisations and their research leaders in the sense of sustainability to make their software typically publicly available under a FLOSS-license but also to take into consideration revenue-oriented approaches such as FLOSS business models or proprietrary licensing if applicable.“ </li> </ul> In my opinion, the final decision <b>to license in and out as well as to transfer computer programs</b> under FLOSS licenses should be the responsibility of the (authorised for this purpose by the Presidium/Board of Directors of the research performing organisation) Research Leaders, with whom the computerprogram developing RSEs should therefore consult. Furthermore, the RSEs and their research leaders ideally should have the opportunity at their research institution to contact (preferably at an early stage) science-supporting specialist departments such as Legal, TTO, research software local task forces or even a SSI-like nationwide institution and seek advice.<br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-14:In Fig.3 it should read "Check (1)..." instead of "Check (2)..." in both the centre left and the top right.<br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-15: The term "research institution" appears in the glossary as a duplicate. Therefore, "research institution" should be deleted in the brackets.<br/># # # # # # # # # # # # # # # # # # # # # # # <ul> <li>Best regardsDirk Feuchter------------------------------------------------------------------Karlsruhe Institute of Technology (KIT)INNOVATION AND RELATIONS MANAGEMENT (IRM)Intellectual Property ManagementDr. Dirk Feuchter (Licenses)Hermann-von-Helmholtz-Platz 1D-76344 Eggenstein LeopoldshafenGermanyPhone: +49 721 608-2-3921E-Mail: dirk.feuchter at kit.eduWeb: <a href="https://www.irm.kit.edu/116_1500.php">https://www.irm.kit.edu/116_1500.php</a> <a href="https://www.irm.kit.edu/english/91.php">https://www.irm.kit.edu/english/91.php</a> <a href="http://www.irm.kit.edu/">www.irm.kit.edu/</a> </li> </ul></span> <div class=padding-bottom id=full-comment-5522>Subject: Feedback to your article „An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action…“.<br/> <br/>Dear Mr. Anzt, dear Mr. Loewe, dear Mr. Bach, dear Mr. Seemann, dear Elke, and dear Sven, as well as dear authors as yet unknown to me,<br/>I am working at KIT in the field of Innovation and Relations Management, especially Licensing of Intellectual Property Rights from KIT to Free Maket Economy. My particual focus is to Out-license Computerprograms to Third Parties and Industry.<br/>Thank you for your great efforts with your extensive and intersting article including the nicely written and ever-recurring story of Kim.<br/>Your FLOSS-based approach for sustainable software devlopment is holding immense savings potential. That’s great.<br/>Following are my comments to your article from the perspective of a TTO license manager, typically supporting RSEs in cases of proprietary licensing e.g. to spin-offs or industrial companies.<br/>Hence, most of my comments might go in a slightly different direction than the main focus of your article, but in my point of view these comments are complementary. Hence, I wonder what you think of them. Looking forward to your opinion and you feedback to my comments, if you like also gladly by telephone<br/>Best regards<br/>Dirk Feuchter<br/> <br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-01:<br/>Abstract:<br/>I would like to suggest to extend<br/>„Research software must be sustainable in order to<br/>understand, replicate, reproduce, and…“<br/>as follows<br/>„Research software must be sustainable in order to<br/>understand, replicate, reproduce, <b>distribute</b> and…“ <br/> <br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-02:<br/>Abstract:<br/>I would like to suggest to extend<br/>„In other words, software must be available, discoverable, usable, and adaptable to new needs, both now and in the future.“<br/> <br/>as follows<br/> <br/>„In other words, software must be available, (IP-/FLOSS)<b>compliant*</b>, discoverable, usable, and adaptable to new needs, both now and in the future.“<br/> <br/>*By „(IP-/FLOSS)<b>compliant“</b><i> I mean</i><br/><i>in </i>compliance <ul> <li>with intellectual property of third-party suppliers,</li> <li>with the terms of free/libre open source licenses and with the aim to protect own intellectual property from unintended disclosure </li> </ul> # # # # # # # # # # # # # # # # # # # # # # #Comment-DF-03:Why sustainable research software in the first place =>I would like to suggest to extend„In order to support research, a sustainable software must be correct<a href="https://f1000research.com/articles/9-295/v1#ref-14"><sup>14</sup></a>, validatable, understandable, documented, publicly released,...“ as follows „In order to support research, a sustainable software must be correct<a href="https://f1000research.com/articles/9-295/v1#ref-14"><sup>14</sup></a>, (IP-/FLOSS)<b>compliant* </b>validatable, understandable, documented, publicly released,...“ *Concerning „(IP-/FLOSS)<b>compliant“</b> please see upon # # # # # # # # # # # # # # # # # # # # # # # <ul> <li>Comment-DF-04:Why sustainable research software in the first place =>„We also argue that truly sustainable research software must ideallybe published under a Free/Libre Open Source Software (FLOSS) license, and…“„We also argue that truly sustainable research software <b>should typically</b> be published under a Free/Libre Open Source Software (FLOSS) license, and…“</li> </ul> # # # # # # # # # # # # # # # # # # # # # # #<br/><br/>Comment-DF-05:Why sustainable research software in the first place =>„Therefore, if research software is publicly funded, it should be freely available under a FLOSS license.„Therefore, if research software is publicly funded, it should be <b>normally</b> freely available under a FLOSS license.<br/># # # # # # # # # # # # # # # # # # # # # # #<br/><br/>Comment-DF-06:Why sustainable research software in the first place =>I would like to suggest to extend„Legal issues: Many obstacles for research software pertain to legal issues, such as applicable licensing and compatibility of licenses<a href="https://f1000research.com/articles/9-295/v1#ref-45"><sup>45</sup></a>, and decisions about license types. as follows „Legal issues: Many obstacles for research software pertain to legal issues, such as <b>IT law, copyright law, copyright notices and author attributions,</b> applicable licensing and compatibility of licenses<a href="https://f1000research.com/articles/9-295/v1#ref-45"><sup>45</sup></a>, and decisions about license types.<br/><br/> # # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-07:Why sustainable research software in the first place =>„A subset of this <b>group</b> may be interested in …“.Which „group“ do you mean? => the „Taxpayers“? # # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-08:Suggestion of a new penultimate paragraph in subchapter „Stakeholder motivations for research software sustainability“ as follows: RSEs, Research leaders and research performing organisations are interested in software sustainability also in the sense that their (research)software is sustainable concerning legal compliance.This is an important issue distributing (research)software for both acaedmic puposes as well as commerical purposes. For the latter RSEs and their research leaders (typically contacting their TTO) are interested to marktet (parts of) their research software and/or additional connectable closed software (which they prevent from unintended disclosure) to a spin-off or an industrial company using a proprietary or a dual licensing model. For both, acaedmic and commerical purposes RSEs and their research leaders and their research performing organisation are interested that their (research) software plus any connectable closed software is compliant with intellectual property of third-party suppliers and compliant with the terms of free/libre open source licenses. An important stakeholder motivation is therefore „software sustainbility with the aim of clarification of all software rights ownerships“. <br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-09:Abstract:I would like to suggest to extend„Failing to do so will threaten the quality and validity of research..“ as follows „Failing to do so will threaten the quality, Xi and validity of research..“ // X1= marketability or X2= distribution // As TTO-license-manager I personally would prefer X1 but X2 is fine as well <br/><br/># # # # # # # # # # # # # # # # # # # # # # #Comment-DF-10:In the heading such as „Why sustainable research software in the first place?“, one might prefix a chapter number, just as announced at the end of the introductory background: „This all leads to unmet requirements and unsolved challenges that we want to highlight in this paper by elaborating on (1)…(2)…(3)…(4)…(5)…(6)…“Hence, instead of„Why sustainable research software in the first place?“use„(1) Why sustainable research software in the first place?“and so on:(2) How to decide which software to sustain?(3) Who sustains research software?(4) How can research software be sustainably funded?(5) Which infrastructure is needed to sustain research software?(6) Legal aspects.That's a matter of taste, of course. <br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/><br/>Comment-DF-11:Please replace Subchapter heading„Challenges and clarifications Clarification of rights“by „Challenges and Clarification of rights“ <br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-12:In both Fig. 1 and „Legal aspects/Challenges and clarifications of rights“instead of „Subject to directives“ {weisungsgebunden dt.} from UrhG 69 bI propose to write „in execution of his duties {in Wahrnehmung seiner Aufgaben dt.}“ from UrhG 69b, in no way to constrain scientists, RSEs and Research leaders, but in order to free up the scope for decision-making and thus open up opportunities.[Regardless of that, I would translate the German "weisungsgebunden" from „UrhG 69 b“ with "bound by instructions"(short) or with "following the instructions given by his employer"(long)]<br/><br/> # # # # # # # # # # # # # # # # # # # # # # # <ul> <li>Comment-DF-13:In Subchapter „Conclusions“I would not write the following sentence, or at most I would write it differently.„We encourage the research funding bodies to reflect the licensing models for academic software development, and to decide whether the “public money, public code” paradigm justifies the requirement that all publicly funded software has to be publicly available under a Free/Libre Open Source Software (FLOSS) license.“ I can understand your point of view and this sentence. But isn't there a lack of perspective regarding those research leaders or RSE-teams, who are considering a foundation based on a proprietary license model regarding their software development or RSE teams whose software developments are taken over and marketed by industrial companies in or outside of Germany, without return to the research performing organizations and the corresponding RSE teams.Hence, an alternative formulation might be as follows: „We encourage the research funding bodies to reflect the licensing models for academic software development, and to suggest research performing organisations and their research leaders in the sense of sustainability to make their software typically publicly available under a FLOSS-license but also to take into consideration revenue-oriented approaches such as FLOSS business models or proprietrary licensing if applicable.“ </li> </ul> In my opinion, the final decision <b>to license in and out as well as to transfer computer programs</b> under FLOSS licenses should be the responsibility of the (authorised for this purpose by the Presidium/Board of Directors of the research performing organisation) Research Leaders, with whom the computerprogram developing RSEs should therefore consult. Furthermore, the RSEs and their research leaders ideally should have the opportunity at their research institution to contact (preferably at an early stage) science-supporting specialist departments such as Legal, TTO, research software local task forces or even a SSI-like nationwide institution and seek advice.<br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-14:In Fig.3 it should read "Check (1)..." instead of "Check (2)..." in both the centre left and the top right.<br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-15: The term "research institution" appears in the glossary as a duplicate. Therefore, "research institution" should be deleted in the brackets.<br/># # # # # # # # # # # # # # # # # # # # # # # <ul> <li>Best regardsDirk Feuchter------------------------------------------------------------------Karlsruhe Institute of Technology (KIT)INNOVATION AND RELATIONS MANAGEMENT (IRM)Intellectual Property ManagementDr. Dirk Feuchter (Licenses)Hermann-von-Helmholtz-Platz 1D-76344 Eggenstein LeopoldshafenGermanyPhone: +49 721 608-2-3921E-Mail: dirk.feuchter at kit.eduWeb: <a href="https://www.irm.kit.edu/116_1500.php">https://www.irm.kit.edu/116_1500.php</a> <a href="https://www.irm.kit.edu/english/91.php">https://www.irm.kit.edu/english/91.php</a> <a href="http://www.irm.kit.edu/">www.irm.kit.edu/</a> </li> </ul></div> <span class="disclosures competing-interests-display padding-top"> <strong>Competing Interests:</strong> <span class=c-i-d>No competing interests were disclosed.</span> </span> <span class="hide-comment-container response-and-comment">Close</span> </span> <div class="report-concern-wrapper last-comment"> <a href="mailto:research@f1000.com?subject=Report a concern (23224) " class=report-a-concern-link>Report a concern</a> </div> </div> </div> </li> <li class="comment-wrapper not-sortable f1r-article-desk"> <div class="bold padding-top padding-left-20"> Discussion is closed on this version, please comment on the latest version above. </div> </li> </ul> </div> </div> <div class="f1r-article-mobile margin-bottom-30"> <div class=contracted-details> <a href="#" class="contracted-details-label author-affiliations"><span class=contracted></span>Author details</a> <a href="#" class=section-title>Author details</a> <span class="f1r-icon icon-14_more_small section-control"></span> <span class="f1r-icon icon-10_less_small section-control"></span> <div class="expanded-details affiliations is-hidden"> <sup>1</sup> Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany<br/> <sup>2</sup> University of Tennessee, Knoxville, TN, USA<br/> <sup>3</sup> Friedrich Schiller University, Jena, Germany<br/> <sup>4</sup> German Aerospace Center (DLR), Berlin, Germany<br/> <sup>5</sup> Humboldt-Universität zu Berlin, Berlin, Germany<br/> <sup>6</sup> Louisiana State University, Baton Rouge, LA, USA<br/> <sup>7</sup> Hasso Plattner Institute, Potsdam, Germany<br/> <sup>8</sup> University Heart Centre Freiburg Bad Krozingen, Freiburg, Germany<br/> <sup>9</sup> Technische Universität München, München, Germany<br/> <sup>10</sup> Universität Duisburg-Essen, Duisburg, Germany<br/> <sup>11</sup> Leibniz Institute of Agricultural Development in Transition Economies (IAMO), Halle (Saale), Germany<br/> <sup>12</sup> Technische Universität Dresden, Dresden, Germany<br/> <sup>13</sup> Deutsches Krebsforschungszentrum, Heidelberg, Germany<br/> <sup>14</sup> Hochschule für Technik und Wirtschaft Berlin, Berlin, Germany<br/> <sup>15</sup> Saarland Informatics Campus, Saarbrücken, Germany<br/> <sup>16</sup> University of Stuttgart, Stuttgart, Germany<br/> <sup>17</sup> Berlin Institute of Health, Berlin, Germany<br/> <sup>18</sup> Alfred Wegener Institute, Bremerhaven, Germany<br/> <sup>19</sup> Max-Planck-Gesellschaft e.V., München, Germany<br/> <sup>20</sup> Universität Würzburg, Würzburg, Germany<br/> <sup>21</sup> Universität Bremen, Bremen, Germany<br/> <sup>22</sup> Max Planck Institute of Molecular Cell Biology and Genetics, Dresden, Germany<br/> <sup>23</sup> European Molecular Biology Laboratory, Heidelberg, Germany<br/> <sup>24</sup> Technische Universität Braunschweig, Braunschweig, Germany<br/> <sup>25</sup> Federal Institute for Materials Research and Testing, Berlin, Germany<br/> <sup>26</sup> Konrad-Zuse-Zentrum für Informationstechnik Berlin (ZIB), Berlin, Germany<br/> <sup>27</sup> FIZ Karlsruhe - Leibniz Institute for Information Infrastructure, Karlsruhe, Germany<br/> <sup>28</sup> University of Goettingen, Göttingen, Germany<br/> <sup>29</sup> University of Münster, Münster, Germany<br/> <sup>30</sup> Institute for Advanced Sustainability Studies, Potsdam, Germany<br/> <sup>31</sup> Ulm University, Ulm, Germany<br/> <sup>32</sup> Linköping University, Linköping, Sweden<br/> <sup>33</sup> Ludwig Maximilian University of Munich, München, Germany<br/> <sup>34</sup> Leibniz University Hannover, Hannover, Germany<br/> <sup>35</sup> Eindhoven University of Technology, Eindhoven, The Netherlands<br/> <sup>36</sup> Julius Kühn-Institut (JKI), Quedlinburg, Germany<br/> <p> <div class=margin-bottom> Hartwig Anzt <br/> <span>Roles: </span> Conceptualization, Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Felix Bach <br/> <span>Roles: </span> Conceptualization, Investigation, Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Stephan Druskat <br/> <span>Roles: </span> Conceptualization, Investigation, Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Frank Löffler <br/> <span>Roles: </span> Conceptualization, Investigation, Visualization, Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Axel Loewe <br/> <span>Roles: </span> Conceptualization, Funding Acquisition, Investigation, Project Administration, Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Bernhard Y. Renard <br/> <span>Roles: </span> Conceptualization, Investigation, Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Gunnar Seemann <br/> <span>Roles: </span> Conceptualization, Funding Acquisition, Investigation, Project Administration, Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Alexander Struck <br/> <span>Roles: </span> Conceptualization, Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Elke Achhammer <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Piush Aggarwal <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Franziska Appel <br/> <span>Roles: </span> Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Michael Bader <br/> <span>Roles: </span> Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Lutz Brusch <br/> <span>Roles: </span> Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Christian Busse <br/> <span>Roles: </span> Writing – Review & Editing </div> <div class=margin-bottom> Gerasimos Chourdakis <br/> <span>Roles: </span> Writing – Review & Editing </div> <div class=margin-bottom> Piotr Wojciech Dabrowski <br/> <span>Roles: </span> Writing – Review & Editing </div> <div class=margin-bottom> Peter Ebert <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Bernd Flemisch <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Sven Friedl <br/> <span>Roles: </span> Visualization, Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Bernadette Fritzsch <br/> <span>Roles: </span> Writing – Review & Editing </div> <div class=margin-bottom> Maximilian D. Funk <br/> <span>Roles: </span> Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Volker Gast <br/> <span>Roles: </span> Writing – Review & Editing </div> <div class=margin-bottom> Florian Goth <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Jean-Noël Grad <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Jan Hegewald <br/> <span>Roles: </span> Writing – Review & Editing </div> <div class=margin-bottom> Sibylle Hermann <br/> <span>Roles: </span> Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Florian Hohmann <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Stephan Janosch <br/> <span>Roles: </span> Writing – Review & Editing </div> <div class=margin-bottom> Dominik Kutra <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Jan Linxweiler <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Thilo Muth <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Wolfgang Peters-Kottig <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Fabian Rack <br/> <span>Roles: </span> Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Fabian H.C. Raters <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Stephan Rave <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Guido Reina <br/> <span>Roles: </span> Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Malte Reißig <br/> <span>Roles: </span> Writing – Review & Editing </div> <div class=margin-bottom> Timo Ropinski <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Joerg Schaarschmidt <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> <div class=margin-bottom> Heidi Seibold <br/> <span>Roles: </span> Writing – Review & Editing </div> <div class=margin-bottom> Jan P. Thiele <br/> <span>Roles: </span> Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Benjamin Uekermann <br/> <span>Roles: </span> Writing – Original Draft Preparation, Writing – Review & Editing </div> <div class=margin-bottom> Stefan Unger <br/> <span>Roles: </span> Visualization, Writing – Original Draft Preparation </div> <div class=margin-bottom> Rudolf Weeber <br/> <span>Roles: </span> Writing – Original Draft Preparation </div> </p> </div> </div> <div class=contracted-details> <a href="#" class=section-title>Competing interests</a> <span class="f1r-icon icon-14_more_small section-control"></span> <span class="f1r-icon icon-10_less_small section-control"></span> <div class="expanded-details grant-information article-page is-hidden"> No competing interests were disclosed. </div> </div> <div class=contracted-details> <a href="#" class=section-title>Grant information</a> <span class="f1r-icon icon-14_more_small section-control"></span> <span class="f1r-icon icon-10_less_small section-control"></span> <div class="expanded-details grant-information article-page is-hidden"> The authors thank the Deutsche Forschungsgemeinschaft (DFG) for funding a meeting (Rundgespräch, grants LO 2093/3-1 and SE 1758/6-1) during which the initial draft of this paper has been created. We are particularly grateful for the support from Dr. Matthias Katerbow (DFG). This work was additionally supported by Research Software Sustainability grants funded by the DFG: Aggarwal: 390886566; PI: Zesch. Appel: 391099391; PI: Balmann. Bach & Loewe & Seemann: 391128822; PIs: Loewe/Scholze/Seemann/Selzer/Streit/Upmeier.Bader: 391134334; PIs: Bader/Gabriel/Frank. Brusch: 391070520; PI: Brusch. Druskat & Gast: 391160252; PI: Gast/Lüdeling. Ebert: 391137747; PI: Marschall.Flemisch & Hermann: 391049448; PIs: Boehringer/Flemisch/Hermann.Hohmann: 391054082; PI: Hepp. Goth: 390966303; PI: Assaad. Grad & Weeber: 391126171; PI: Holm. Kutra: 391125810; PI: Kreshuk.Mehl & Uekermann: 391150578; PIs: Bungartz/Mehl/Uekermann. Peters-Kottig: 391087700; PIs: Gleixner/Peters-Kottig/Shinano/Sperber. Raters: 39099699; PI:Herwartz. Reina: 391302154; PIs: Ertl/Reina. Muth & Renard: 391179955; PIs Renard/Fuchs. Ropinski:391107954; PI: Ropinski.<br>Alexander Struck acknowledges the support of the Cluster of Excellence Matters of Activity. Image Space Material funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany´s Excellence Strategy – EXC 2025.<br>We acknowledge support by the KIT-Publication Fund of the Karlsruhe Institute of Technology. <br/> <i>The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.</i> </div> </div> <div class=contracted-details> <a href="#" class=section-title>Article Versions (2)</a> <span class="f1r-icon icon-14_more_small section-control"></span> <span class="f1r-icon icon-10_less_small section-control"></span> <div class="expanded-details grant-information article-page is-hidden"> <div> <a href="https://f1000research.com/articles/9-295/v2" title="Open version 2 of this article." class="" gahelper=2>version 2</a> <span class="article-pubinfo-mobile versions-section"> <div title="#springMessage("NewVersionReasonType.REVISED.tooltip")" class="version-info-box revised ">Revised</div> </span> </div> <div> Published: 26 Jan 2021, 9:295 </div> <div class=margin-bottom><a href="https://doi.org/10.12688/f1000research.23224.2">https://doi.org/10.12688/f1000research.23224.2</a></div> <div> <a href="https://f1000research.com/articles/9-295/v1" title="Open version 1 of this article." class="" gahelper=1>version 1</a> <span class="article-pubinfo-mobile versions-section"> </span> </div> <div> Published: 27 Apr 2020, 9:295 </div> <div class=""><a href="https://doi.org/10.12688/f1000research.23224.1">https://doi.org/10.12688/f1000research.23224.1</a></div> </div> </div> <div class=contracted-details> <a href="#" class=section-title> <span class="f1r-icon icon-100_open_access"></span> Copyright </a> <span class="f1r-icon icon-14_more_small section-control"></span> <span class="f1r-icon icon-10_less_small section-control"></span> <div class="expanded-details grant-information article-page is-hidden"> © 2021 Anzt H <em>et al</em>. This is an open access article distributed under the terms of the <a href="https://creativecommons.org/licenses/by/4.0/" target=_blank data-test-id=box-licence-link>Creative Commons Attribution License</a>, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. </div> </div> <div class="padding-top-30 research-layout"> <div class=article-toolbox-wrapper-mobile> <div class=article-tools-icon-mobile data-section=download> <span class="f1r-icon icon-76_download_file white"></span> </div> <div class="article-tools-icon-mobile mobile-metrics article-metrics-wrapper metrics-icon-wrapper" data-section=metrics data-version-id=31504 data-id=23224 data-downloads="" data-views="" data-scholar="10.12688/f1000research.23224.2" data-recommended="" data-f1r-ga-helper="Article Page Metrics (Mobile)"> <span class="f1r-icon icon-89_metrics white"></span> </div> <div class=article-tools-divider-mobile></div> <div class=article-tools-icon-mobile data-section=cite> <span class="f1r-icon icon-82_quote white"></span> </div> <div class="article-tools-icon-mobile " data-section=track> <span class="f1r-icon icon-90_track white"></span> </div> <div class=article-tools-divider-mobile></div> <div class=article-tools-icon-mobile data-section=share> <span class="f1r-icon icon-34_share white"></span> </div> <span class=article-toolbox-stretch></span> </div> <div class=article-toolbox-content-mobile> <div class="toolbox-section download"> <div class=toolbox-section-heading>Download</div> <div class=toolbox-section-content> <a href="https://f1000research.com/articles/9-295/v2/pdf?article_uuid=15417a87-a637-4761-bc45-e2c4e71a7858" title="Download PDF" class="no-decoration pdf-download-helper"> <span class="f1r-icon icon-102_download_pdf toolbox-section-icon"></span> </a> <div class=toolbox-section-option-divider> </div> <a id=mobile-download-xml class=no-decoration href="#" title="Download XML"> <span class="f1r-icon icon-103_download_xml toolbox-section-icon"></span> </a> </div> <div class=toolbox-section-divider></div> <div class=toolbox-section-heading>Export To</div> <div class=toolbox-section-content> <button class="primary no-fill orange-text-and-border mobile-export" data-etype=WORKSPACE>Sciwheel</button> <button class="primary no-fill orange-text-and-border mobile-export" data-etype=BIBTEX>Bibtex</button> <button class="primary no-fill orange-text-and-border mobile-export" data-etype=ENDNOTE>EndNote</button> <button class="primary no-fill orange-text-and-border mobile-export" data-etype=PROCITE>ProCite</button> <button class="primary no-fill orange-text-and-border mobile-export" data-etype=REF_MANAGER>Ref. Manager (RIS)</button> <button class="primary no-fill orange-text-and-border mobile-export" data-etype=SENTE>Sente</button> </div> </div> <div class="toolbox-section metrics"> <div class="toolbox-section-heading no-top-border">metrics</div> <div class="toolbox-section-divider toolbox-section-divider--no-height"></div> <div class=article-metrics-pageinfo> <div class=c-article-metrics-table> <table class=c-article-metrics-table__table> <thead> <tr> <th></th> <th><span class=c-article-metrics-table__heading>Views</span></th> <th><span class=c-article-metrics-table__heading>Downloads</span></th> </tr> </thead> <tbody> <tr> <th class=c-article-metrics-table__row-heading><span class="c-article-metrics-table__platform u-platform--" data-test-id=metrics_platform_name_mob>F1000Research</span></th> <td class="c-article-metrics-table__value js-article-views-count" data-test-id=metrics_platform_views_mob>-</td> <td class="c-article-metrics-table__value js-article-downloads-count" data-test-id=metrics_platform_downloads_mob>-</td> </tr> <tr> <th class=c-article-metrics-table__row-heading> <span class="u-ib u-middle c-article-metrics-table__pmc" data-test-id=metrics_pmc_name_mob>PubMed Central</span> <div class="c-block-tip c-block-tip--centered c-article-metrics-table__tooltip c-block-tip--md-padding c-block-tip--small-arrow u-ib u-middle"> <button type=button class="u-black--medium u-black--high@hover u-ib u-middle c-button--icon c-button--text c-block-tip__toggle"><i class="material-icons c-button--icon__icon">info_outline</i></button> <div class=c-block-tip__content>Data from PMC are received and updated monthly.</div> </div> </th> <td class="c-article-metrics-table__value js-pmc-views-count" data-test-id=metrics_pmc_views_mob>-</td> <td class="c-article-metrics-table__value js-pmc-downloads-count" data-test-id=metrics_pmc_downloads_mob>-</td> </tr> </tbody> </table> </div> </div> <span class=metrics-citations-container> <div class=toolbox-section-divider></div> <div class="toolbox-section-heading u-mb--1">Citations</div> <div> <div class=toolbox-section-colsplit> <div class=citations-scopus-logo> <a href="" target=_blank class="is-hidden metrics-citation-icon" title="View full citation details at www.scopus.com"><i class="material-icons scopus-icon">open_in_new</i></a> </div> <div class="toolbox-section-count scopus"> <a href='' class='scopus-citation-link is-hidden' target=_blank title='View full citation details at www.scopus.com'>0</a> </div> </div> <div class=toolbox-section-colsplit> <div class="citations-pubmed-logo f1000research"> <a href="" target=_blank class="is-hidden metrics-citation-icon f1000research" title="View full citation details"><i class="material-icons scopus-icon">open_in_new</i></a> </div> <div class="toolbox-section-count pubmed"> <a href='' class='pubmed-citation-link is-hidden' target=_blank title='View full citation details'>0</a> </div> </div> <div class=toolbox-section-divider></div> <div class=toolbox-section-content> <div class="citations-scholar-logo f1000research"> <a href="" target=_blank class="is-hidden metrics-citation-icon google-scholar f1000research" title="View full citation details" data-scholar="10.12688/f1000research.23224.2"><i class="material-icons scopus-icon">open_in_new</i></a> </div> </div> </div> </span> <span class=metrics-details-container> <div class=toolbox-section-divider></div> <div class="toolbox-section-content altmetric-section"> <div class=altmetrics-image></div> <div class=altmetrics-more-link> <a href="" target=_blank class=f1r-standard-link>SEE MORE DETAILS</a> </div> <div class=altmetric-mobile-column-counts></div> <div class=altmetric-mobile-column-readers></div> <div class=toolbox-section-divider></div> </div> </span> </div> <div class="toolbox-section cite"> <div class="toolbox-section-heading no-top-border">CITE</div> <div class=toolbox-section-divider></div> <div class=toolbox-section-heading>how to cite this article</div> <div id=citation-copy-mobile class="toolbox-section-content text-content heading9 small" data-test-id=mob_copy-citation_text> Anzt H, Bach F, Druskat S <em>et al.</em> An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action [version 2; peer review: 2 approved]</span>. <i>F1000Research</i> 2021, <b>9</b>:295 (<a href="https://doi.org/10.12688/f1000research.23224.2" target=_blank>https://doi.org/10.12688/f1000research.23224.2</a>) </div> <div class=toolbox-section-divider></div> <div class="toolbox-section-content text-content heading9 small"> NOTE: <em>If applicable, it is important to ensure the information in <b>square brackets after the title</b> is included in all citations of this article.</em> </div> <div class=toolbox-section-content> <button class="primary orange extra-padding copy-cite-article-mobile js-clipboard" data-clipboard-target="#citation-copy-mobile" title="Copy the current citation details." data-test-id=mob_copy-citation_button-mob>COPY CITATION DETAILS</button> </div> </div> <div class="toolbox-section track"> <div class="toolbox-section-heading no-top-border">track</div> <div class=toolbox-section-divider></div> <div class=toolbox-section-heading>receive updates on this article</div> <div class="toolbox-section-content padding-left-20 padding-right-20 heading9 small"> Track an article to receive email alerts on any updates to this article. </div> <div class=toolbox-section-content> <a data-article-id=23224 id=mobile-track-article-signin-23224 title="Receive updates on new activity such as publication of new versions, peer reviews or author responses." href="/login?originalPath=/trackArticle/23224?target=/articles/9-295/v2"> <button class="primary orange extra-padding"> TRACK THIS ARTICLE </button> </a> </div> </div> <div class="toolbox-section share"> <div class="toolbox-section-heading no-top-border">Share</div> <div class=toolbox-section-divider></div> <div class=toolbox-section-content> <a target=_blank class="f1r-shares-icon-square f1r-shares-twitter" title="Share on Twitter"></a> <a target=_blank class="f1r-shares-icon-square f1r-shares-facebook" title="Share on Facebook"></a> <a target=_blank class="f1r-shares-icon-square f1r-shares-linkedin" title="Share on LinkedIn"></a> <a target=_blank class="f1r-shares-icon-square f1r-shares-reddit" title="Share on Reddit"></a> <a target=_blank class="f1r-shares-icon-square f1r-shares-mendelay" title="Share on Mendeley"></a> </div> </div> </div> </div> <a name=article-reports></a> <div id=article-reports class="u-mt--3 reports-comments no-divider"> <div class="current-referee-status "> <h2 class=main-title id=current-referee-status> <span class="research-layout f1r-article-mobile-inline valign-middle"> <span class="f1r-icon icon-85_peer_review size30"></span> </span> Open Peer Review <span class="f1r-article-mobile-inline float-right"> <span class="f1r-icon icon-14_more_small"></span> <span class="f1r-icon icon-10_less_small"></span> </span> </h2> <a name=current-referee-status></a> <div class=current-referee-status__content name=add-new-report-comment id=add-new-report-comment> Current Reviewer Status: <span class="research-layout f1r-article-desk-inline"> <span class="f1r-icon icon-86_approved status-green smaller" title=Approved></span> </span> <span class="research-layout f1r-article-mobile float-right"> <span class="f1r-icon icon-86_approved status-green smaller" title=Approved></span> </span> <span class="research-layout f1r-article-desk-inline"> <span class="f1r-icon icon-86_approved status-green smaller" title=Approved></span> </span> <span class="research-layout f1r-article-mobile float-right"> <span class="f1r-icon icon-86_approved status-green smaller" title=Approved></span> </span> <span class="circle-icon-small to-right" data-window=about-referee-status title=Help>?</span> <span class="research-layout f1r-article-mobile"> <div class=mobile-ref-status-help> Key to Reviewer Statuses <span class=referee-status-pointer></span> <span class="view-control float-right">VIEW</span> <span class="view-control float-right is-hidden">HIDE</span> <div class=mobile-ref-status-help-content> <div class="cf margin-top"> <span class="f1r-icon icon-86_approved status-green smaller float-left margin-bottom-40 margin-right" title=Approved></span> <span class=title>Approved</span>The paper is scientifically sound in its current form and only minor, if any, improvements are suggested </div> <div class="cf margin-top"> <span class="f1r-icon icon-87_approved_reservations status-green smaller float-left margin-bottom-40 margin-right" title="Approved with Reservations"></span> <span class=title>Approved with reservations</span> A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit. </div> <div class="cf margin-top"> <span class="f1r-icon icon-88_not_approved status-red small float-left margin-bottom-30 margin-right" title="Not Approved"></span> <span class=title>Not approved</span>Fundamental flaws in the paper seriously undermine the findings and conclusions </div> </div> </div> </span> </div> </div> <div class="f1r-article-desk-inline referee-report-info-box referee-report-version-box">Version 2</div> <div class="f1r-article-mobile research-layout mobile-version-info padding-top-30 padding-bottom-30"> <div class="mversion ">VERSION 2</div> <span class=details>PUBLISHED 26 Jan 2021</span> <span class="article-pubinfo-mobile versions-section"> <div title="#springMessage("NewVersionReasonType.REVISED.tooltip")" class="version-info-box revised ">Revised</div> </span> </div> <div class="referee-response-container no-border last-report-for-version" data-reportID=78211> <div class="f1r-article-mobile research-layout"> <div class="version-border current"></div> </div> <a name=referee-response-78211></a> <div id=report-metadata-78211 class=is-hidden> <input type=hidden name=reportId value=78211 /> </div> <div class="f1r-article-mobile-inline research-layout float-right margin-right-20"> <span class="f1r-icon-circle-wrapper with-border white-bg big referee-reports"> <span class="f1r-icon icon-86_approved status-green smaller" title=Approved></span> </span> </div> <div class="article-count-box response-and-comment has-shadow"> <div class=comments-count-title>Views</div> <div class=comments-count-container>0</div> <div class="article-interaction-info article-page"> <div class="cite-article-popup-wrapper article-page-interaction-box"> <div class="article-interaction-button cite-article-button" title="Cite this report" data-windowref=cite-report-popup-78211-1> <span class="f1r-icon icon-82_quote"></span> <a href="#" class="button-link cite-article-popup-link response-and-comment" title="Cite Report" data-test-id=cite-report-1>Cite</a> </div> <div id=cite-report-popup-78211-1 class="popup-window-wrapper is-hidden"> <div class=cite-popup-background></div> <div class="popup-window cite-this-article-box research-layout"> <div class="popup-window-title small cite-title">How to cite this report:</div> <span id=cite-report-text-78211-1 data-test-id=mob_report_copy-citation_text> <span class="article-title-and-info in-popup">Hasselbring W. Reviewer Report For: An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action [version 2; peer review: 2 approved]</span>. <i>F1000Research</i> 2021, <b>9</b>:295 (<a class=new-orange href="https://doi.org/10.5256/f1000research.31504.r78211" target=_blank>https://doi.org/10.5256/f1000research.31504.r78211</a>) </span> <div class=padding-top data-test-id=report-url> The direct URL for this report is:<br> <a href="https://f1000research.com/articles/9-295/v2#referee-response-78211" class=new-orange target=_blank>https://f1000research.com/articles/9-295/v2#referee-response-78211</a> </div> <div class="popup-window-title small note margin-top-20 margin-bottom-20"> <strong>NOTE:</strong> it is important to ensure the information in square brackets after the title is included in this citation. </div> <div class=float-right> <button class="secondary no-fill orange-text-and-border margin-right-20 close-cite-popup uppercase">Close</button> <button class="secondary orange copy-cite-article-version uppercase js-clipboard" title="Copy the current citation details to the clipboard." data-clipboard-target="#cite-report-text-78211-1" data-test-id=mob_report_copy-citation_button>Copy Citation Details</button> </div> </div> </div> </div> </div> </div> <div class="referee-report-info-box referee-report-details-box"> <span class="bold mobile-report-text">Reviewer Report</span> <span class=mobile-report-info>28 Jan 2021</span> </div> <span class="info-separation padding-bottom padding-left"> <div> <span class=bold>Wilhelm Hasselbring<a href="http://orcid.org/0000-0001-6625-4335" target=_blank title="Open ORCID"><span class=orcid-logo-for-author-list></span></a></span><span class=f1r-article-desk-inline>,</span> Software Engineering Group, Kiel University, Kiel, Germany </div> </span> <div class="research-layout referee-report-status-line"> <span class="f1r-icon icon-86_approved status-green smaller" title=Approved></span> <span class=status-text>Approved</span> </div> <div class=f1r-article-mobile> <span class=article-pubinfo-mobile> <div class="version-info-box views">VIEWS 0</div> </span> </div> <div class=referee-report-content> <div class=hidden-text> <a href="https://doi.org/10.5256/f1000research.31504.r78211" target=_blank>https://doi.org/10.5256/f1000research.31504.r78211</a> </div> <span class=truncated-comment> Thanks for delivering this revised version of your opinion article. I highly appreciate that you addressed all ... <span class="f1r-article-desk-inline continue-reading-container response-and-comment increment-view-count" data-reportID=78211>Continue reading</span> <div class="f1r-article-mobile research-layout margin-top-30 is-centered"> <button class="primary orange-text white-bg bigger-text expand-button-mobile continue-reading-container show with-border"> READ ALL <span class="f1r-icon icon-14_more_small orange vmiddle big"></span> </button> </div> </span> <span class="hidden hidden-report-text"> <div class=hidden-report-text__comment>Thanks for delivering this revised version of your opinion article. I highly appreciate that you addressed all the concerns I had with the previous version, such that I can now fully approve your paper!<br/> </div> <p style="margin-top: 10px;"><b class=mobile-block-display>Competing Interests:</b> No competing interests were disclosed.</p> <p><b>Reviewer Expertise:</b> Software Engineering</p> <span class="info-separation padding-top padding-bottom comment-statement-mobile"> <div class=f1r-article-mobile> <div class="content-divider report-info"></div> </div> <strong> I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. </strong> </span> <div class=f1r-article-mobile> <div class="content-divider report-info"></div> </div> <span class="f1r-article-desk hide-comment-container response-and-comment">Close</span> <div class="f1r-article-mobile research-layout margin-top-30 is-centered"> <button class="primary orange-text white-bg bigger-text expand-button-mobile hide-comment-container hide with-border"> READ LESS <span class="f1r-icon icon-10_less_small orange vmiddle big"></span> </button> </div> </span> <div class="f1r-article-mobile cite-report-wrapper"> <a href="#" class=mobile-cite-report>CITE</a> </div> <div class="research-layout mobile-cite-report-popup-wrapper"> <div class=bgmask></div> <div class=mobile-cite-report-popup> <span class="f1r-icon icon-88_not_approved close-cite-report-popup"></span> <div class=cite-report-popup-title>CITE</div> <span class=cite-report-popup-top-content> HOW TO CITE THIS REPORT </span> <div class=cite-report-popup-content> <span class=copy-cite-report id=cite-report-text-mobile-78211-1> <span class=article-title-and-info>Hasselbring W. Reviewer Report For: An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action [version 2; peer review: 2 approved]</span>. <i>F1000Research</i> 2021, <b>9</b>:295 (<a class=new-orange href="https://doi.org/10.5256/f1000research.31504.r78211" target=_blank>https://doi.org/10.5256/f1000research.31504.r78211</a>) </span> </div> <div class=cite-report-popup-divider></div> <div class=cite-report-popup-content data-test-id=report-url> The direct URL for this report is:<br> <a href="https://f1000research.com/articles/9-295/v2#referee-response-78211" class=new-orangetarget=_blank>https://f1000research.com/articles/9-295/v2#referee-response-78211</a> </div> <div class=cite-report-popup-divider></div> <div class=cite-report-popup-content> NOTE: <em>it is important to ensure the information in <b>square brackets after the title</b> is included in all citations of this article.</em> </div> <div class=cite-report-popup-buttons> <button type=button class="primary orange extra-padding copy-cite-report-mobile js-clipboard" title="Copy the current citation details to the clipboard." data-clipboard-target="#cite-report-text-mobile-78211-1" data-test-id=mob_report_copy-citation_button>COPY CITATION DETAILS</button> </div> </div> </div> <div class=report-concern-wrapper> <a class=report-a-concern-link href="mailto:research@f1000.com?subject=Report a concern (23224) ">Report a concern</a> </div> </div> <div class=comments-list-container id=referee-comments-78211-1> <div class=f1r-article-desk> <ul class=all-comment-wrapper data-reportID=78211> <li class="comment-wrapper not-sortable"> <div class="one-comment respond-or-comment"> <a href="/login?originalPath=/articles/9-295/v2&scrollTo=referee-response-78211" class=register-report-comment-button data-test-id=add-response-comment><span class=contracted></span>Respond or Comment</a> </div> </li> </ul> </div> <div class="f1r-article-mobile research-layout"> <ul class=all-comment-wrapper data-reportID=78211> <li class="comment-wrapper not-sortable button-wrapper"> <div class=comment-button-wrapper> <a href="/login?originalPath=/articles/9-295/v2&scrollTo=referee-response-78211" class=register-report-comment-button data-test-id=add-report-comment> <button class="primary orange extra-padding"> COMMENT ON THIS REPORT </button> </a> </div> </li> </ul> </div> </div> </div> <div class="f1r-article-desk-inline referee-report-info-box referee-report-version-box">Version 1</div> <div class="f1r-article-mobile research-layout"> <div class="mobile-sections-divider versions-divider"></div> </div> <div class="f1r-article-mobile research-layout mobile-version-info padding-top-30 padding-bottom-30"> <div class="mversion previous-version">VERSION 1</div> <span class=details>PUBLISHED 27 Apr 2020</span> <span class="article-pubinfo-mobile versions-section"> </span> </div> <div class="referee-response-container no-border" data-reportID=62872> <div class="f1r-article-mobile research-layout"> <div class="version-border previous"></div> </div> <a name=referee-response-62872></a> <div id=report-metadata-62872 class=is-hidden> <input type=hidden name=reportId value=62872 /> </div> <div class="f1r-article-mobile-inline research-layout float-right margin-right-20"> <span class="f1r-icon-circle-wrapper with-border white-bg big referee-reports"> <span class="f1r-icon icon-86_approved status-green smaller" title=Approved></span> </span> </div> <div class="article-count-box response-and-comment has-shadow"> <div class=comments-count-title>Views</div> <div class=comments-count-container>0</div> <div class="article-interaction-info article-page"> <div class="cite-article-popup-wrapper article-page-interaction-box"> <div class="article-interaction-button cite-article-button" title="Cite this report" data-windowref=cite-report-popup-62872-1> <span class="f1r-icon icon-82_quote"></span> <a href="#" class="button-link cite-article-popup-link response-and-comment" title="Cite Report" data-test-id=cite-report-1>Cite</a> </div> <div id=cite-report-popup-62872-1 class="popup-window-wrapper is-hidden"> <div class=cite-popup-background></div> <div class="popup-window cite-this-article-box research-layout"> <div class="popup-window-title small cite-title">How to cite this report:</div> <span id=cite-report-text-62872-1 data-test-id=mob_report_copy-citation_text> <span class="article-title-and-info in-popup">Bast R. Reviewer Report For: An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action [version 2; peer review: 2 approved]</span>. <i>F1000Research</i> 2021, <b>9</b>:295 (<a class=new-orange href="https://doi.org/10.5256/f1000research.25640.r62872" target=_blank>https://doi.org/10.5256/f1000research.25640.r62872</a>) </span> <div class=padding-top data-test-id=report-url> The direct URL for this report is:<br> <a href="https://f1000research.com/articles/9-295/v1#referee-response-62872" class=new-orange target=_blank>https://f1000research.com/articles/9-295/v1#referee-response-62872</a> </div> <div class="popup-window-title small note margin-top-20 margin-bottom-20"> <strong>NOTE:</strong> it is important to ensure the information in square brackets after the title is included in this citation. </div> <div class=float-right> <button class="secondary no-fill orange-text-and-border margin-right-20 close-cite-popup uppercase">Close</button> <button class="secondary orange copy-cite-article-version uppercase js-clipboard" title="Copy the current citation details to the clipboard." data-clipboard-target="#cite-report-text-62872-1" data-test-id=mob_report_copy-citation_button>Copy Citation Details</button> </div> </div> </div> </div> </div> </div> <div class="referee-report-info-box referee-report-details-box"> <span class="bold mobile-report-text">Reviewer Report</span> <span class=mobile-report-info>01 Jun 2020</span> </div> <span class="info-separation padding-bottom padding-left"> <div> <span class=bold>Radovan Bast<a href="http://orcid.org/0000-0002-7658-1847" target=_blank title="Open ORCID"><span class=orcid-logo-for-author-list></span></a></span><span class=f1r-article-desk-inline>,</span> Department of Information Technology, UiT The Arctic University of Norway, Tromsø, Norway </div> </span> <div class="research-layout referee-report-status-line"> <span class="f1r-icon icon-86_approved status-green smaller" title=Approved></span> <span class=status-text>Approved</span> </div> <div class=f1r-article-mobile> <span class=article-pubinfo-mobile> <div class="version-info-box views">VIEWS 0</div> </span> </div> <div class=referee-report-content> <div class=hidden-text> <a href="https://doi.org/10.5256/f1000research.25640.r62872" target=_blank>https://doi.org/10.5256/f1000research.25640.r62872</a> </div> <span class=truncated-comment> In "An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action" the authors identify challenges for research software sustainability in Germany and beyond.<br/> <br/> They examine the current state of ... <span class="f1r-article-desk-inline continue-reading-container response-and-comment increment-view-count" data-reportID=62872>Continue reading</span> <div class="f1r-article-mobile research-layout margin-top-30 is-centered"> <button class="primary orange-text white-bg bigger-text expand-button-mobile continue-reading-container show with-border"> READ ALL <span class="f1r-icon icon-14_more_small orange vmiddle big"></span> </button> </div> </span> <span class="hidden hidden-report-text"> <div class=hidden-report-text__comment>In "An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action" the authors identify challenges for research software sustainability in Germany and beyond.<br/> <br/> They examine the current state of research software sustainability and challenges in motivating sustainable research software development, selection criteria for funding, personnel, funding, infrastructure, and legal aspects, and offer recommendations for addressing these challenges. These sections are accompanied and with a story using a fictional character Kim which helps to<br/> relate these aspects to typical career stages of a research software engineer.<br/> <br/> The article is thoroughly researched, well-written, and offers an excellent overview of the challenges when building an environment for sustainable research software. Most of the discussed challenges and recommendations carry beyond Germany and are relevant and transferable to other countries.<br/> <br/> Below I give few (minor) suggestions for consideration when improving the manuscript.<br/> <br/> Regarding the list of challenges under "Why sustainable research software in the first place?" (pages 5 and 6): <ul> <li>Infrastructure issues: One design choice that often limits the use or usability of local infrastructure resources is that they are often bound to institutional user accounts and thus limit collaboration possibilities with collaborators in other institutions and countries. On the other hand, pooling of infrastructure resources which could enable collaboration across organizations can be limited by lack of authentication and authorization infrastructure (AAI) or legal constraints. Later in the paper the authors indeed mention AAI (page 12) but this could already be pointed out and connected earlier.<br/> </li> <li>Legal issues: Not only licensing is an issue but legal constraints or uncertainty about legal boundaries and identity federation can also limit the deployment of infrastructure services. Often the deployment and operation of infrastructure services is given to technical teams who may lack the legal support or expertise to clarify legal and privacy terms for the storage of data and processing of data.<br/> </li> <li>Funding issues: The challenge is not only that funding is scarce but also that it does not align well with pricing models of cloud infrastructure providers. It can be easier for research groups to spend a larger chunk of the budget towards the end of a year for hardware compared to pay possibly relatively modest monthly fees for a cloud service, which however may not fit into the budget forms. These budget constraints may also limit the possibility of pooling resources and sharing them with other research groups. Software cloud infrastructure is often not considered at all in the proposal. There is also a resistance among some of my research colleagues to pay 20-50 USD/ month for an infrastructure service which is sometimes solved by reinventing the service locally "for free".<br/> </li> <li>Another mismatch between traditional funding models and support of software which "must continue to exist" to be sustainable (page 5), is the experience that it can take months or years until the software is picked up by other groups and contributions and questions start to roll in. But by that time the funding of the project stopped, the developer (team) may have already moved on to other positions and projects, and may not have the time to react and help, even though they still may have interest and the knowledge. Our traditional funding models consider the software to be "done" by the end of the project.</li> </ul> Selection criteria for "How to decide which software to sustain?" (page 9): <ul> <li>The authors mention "usage and impact", "software quality", as well as "maturity". But I would like to see also "openness and transparency" among these. The reason is that we can expect the research community to adapt to these or any metrics and we will over time observe what we measure. Any set of metrics could be criticized as to some extent being arbitrary but the advantage of including "openness and transparency" is that the community as whole would benefit from such a metric [Enrico Glerean, "Responsible conduct of research and questionable research practices", presentation, slide 47<a href="#rep-ref-62872-1"><sup>1</sup></a>].</li> </ul> Regarding "Who sustains research software?": <ul> <li>The authors discuss the lack of recognition and awareness, as well as lack of career opportunities. It is also about respect and I was happy to see the sentence: "Not all researchers that think of themselves as RSEs pursue a faculty position as their main career goal." I have experienced that RSEs are sometimes regarded as those who somehow "failed" to obtain a faculty position whereas many RSEs have chosen this position over a faculty position because it was a better fit for their career goals. This misunderstanding can lead to a lack of respect towards this position and this career choice and can lead to excellent personnel leaving the academic environment towards commercial employment, possibly not primarily for financial reasons but sometimes to be more respected and recognized.</li> </ul> Archiving and software preservation (page 13: <ul> <li>The authors mention Docker but also Singularity should be mentioned as a tool since it is getting traction in particular on many-user systems such as higher performance computing clusters.</li> </ul> Legal aspects (page 14): <ul> <li>Re-licensing is mentioned in the story box and the text starts by pointing out that licensing is often not considered early enough in the project. Indeed re-licensing later in the project can be not only legally, but also organizationally very tricky, in particular for projects which developed over many years and involved many contributors in different organizations. This could be pointed out in the text as additional motivation to consider these very early in the project.<br/> </li> <li>I very much like the recommendation of providing a legal help desk for research groups to avoid the problem that out of uncertainty and fear of making a legal mistake some research groups end up not choosing any license at all which may limit further reuse of the software.<br/> </li> <li>The manuscript presents a decision tree for contributors (Figure 1) and also discusses contributor license agreements. It could be useful to point out that without clear policies or legal help desks, individuals or organizations may be hesitant to contribute to a project because they may not feel confident having enough knowledge or authority to sign such agreements and too many legal steps and question can also raise the barrier to contribute, in particular for smaller projects. Also here clear guidelines and a support desk can help removing these barriers.</li> </ul></div> <div class=md> <div class=questionnaire> <ul class=c-questionnaire> <li class=c-questionnaire__item> <p class=c-questionnaire__question>Is the topic of the opinion article discussed accurately in the context of the current literature?</p> <p class=c-questionnaire__answer>Yes</p> </li> <li class=c-questionnaire__item> <p class=c-questionnaire__question>Are all factual statements correct and adequately supported by citations?</p> <p class=c-questionnaire__answer>Yes</p> </li> <li class=c-questionnaire__item> <p class=c-questionnaire__question>Are arguments sufficiently supported by evidence from the published literature?</p> <p class=c-questionnaire__answer>Yes</p> </li> <li class=c-questionnaire__item> <p class=c-questionnaire__question>Are the conclusions drawn balanced and justified on the basis of the presented arguments?</p> <p class=c-questionnaire__answer>Yes</p> </li> </ul> </div> </div> <p><b>References</b></p> <a name=rep-ref-62872-1></a>1. Enrico Glerean: Responsible conduct of research and questionable research practices. 2018. <a target=_blank href="https://doi.org/10.6084/m9.figshare.10303487.v1">Reference Source</a> <br> <p style="margin-top: 10px;"><b class=mobile-block-display>Competing Interests:</b> No competing interests were disclosed.</p> <p><b>Reviewer Expertise:</b> computational chemistry, research software engineering</p> <span class="info-separation padding-top padding-bottom comment-statement-mobile"> <div class=f1r-article-mobile> <div class="content-divider report-info"></div> </div> <strong> I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. </strong> </span> <div class=f1r-article-mobile> <div class="content-divider report-info"></div> </div> <span class="f1r-article-desk hide-comment-container response-and-comment">Close</span> <div class="f1r-article-mobile research-layout margin-top-30 is-centered"> <button class="primary orange-text white-bg bigger-text expand-button-mobile hide-comment-container hide with-border"> READ LESS <span class="f1r-icon icon-10_less_small orange vmiddle big"></span> </button> </div> </span> <div class="f1r-article-mobile cite-report-wrapper"> <a href="#" class=mobile-cite-report>CITE</a> </div> <div class="research-layout mobile-cite-report-popup-wrapper"> <div class=bgmask></div> <div class=mobile-cite-report-popup> <span class="f1r-icon icon-88_not_approved close-cite-report-popup"></span> <div class=cite-report-popup-title>CITE</div> <span class=cite-report-popup-top-content> HOW TO CITE THIS REPORT </span> <div class=cite-report-popup-content> <span class=copy-cite-report id=cite-report-text-mobile-62872-1> <span class=article-title-and-info>Bast R. Reviewer Report For: An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action [version 2; peer review: 2 approved]</span>. <i>F1000Research</i> 2021, <b>9</b>:295 (<a class=new-orange href="https://doi.org/10.5256/f1000research.25640.r62872" target=_blank>https://doi.org/10.5256/f1000research.25640.r62872</a>) </span> </div> <div class=cite-report-popup-divider></div> <div class=cite-report-popup-content data-test-id=report-url> The direct URL for this report is:<br> <a href="https://f1000research.com/articles/9-295/v1#referee-response-62872" class=new-orangetarget=_blank>https://f1000research.com/articles/9-295/v1#referee-response-62872</a> </div> <div class=cite-report-popup-divider></div> <div class=cite-report-popup-content> NOTE: <em>it is important to ensure the information in <b>square brackets after the title</b> is included in all citations of this article.</em> </div> <div class=cite-report-popup-buttons> <button type=button class="primary orange extra-padding copy-cite-report-mobile js-clipboard" title="Copy the current citation details to the clipboard." data-clipboard-target="#cite-report-text-mobile-62872-1" data-test-id=mob_report_copy-citation_button>COPY CITATION DETAILS</button> </div> </div> </div> <div class=report-concern-wrapper> <a class=report-a-concern-link href="mailto:research@f1000.com?subject=Report a concern (23224) ">Report a concern</a> </div> </div> <div class=comments-list-container id=referee-comments-62872-1> <div class=f1r-article-desk> <ul class=all-comment-wrapper data-reportID=62872> <li class="comment-wrapper not-sortable " data-commentID=6210> <div class="one-comment with-divider "> <a name=article-comment-6210></a> <div class="f1r-article-desk commenter-role"> Author Response 26 Jan 2021 </div> <div> <span class=name>Axel Loewe</span><span class=name>, </span><span class=commenter-affiliation>Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany</span> </div> <div class="f1r-article-mobile the-comment-date"> 26 Jan 2021 </div> <div class=f1r-article-mobile> <div class="article-pubinfo-mobile comments-section"> <span class="version-info-box comments"> Author Response </span> </div> </div> <div class="comment-content "> <span class=truncated-comment> <i>We thank you for the thorough review and constructive feedback regarding the manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> <b>Regarding</b> ... <span class="continue-reading-container response-and-comment">Continue reading</span> </span> <span class="full-comment hidden"> <span id=full-comment-original-6210 class=hidden><i>We thank you for the thorough review and constructive feedback regarding the manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> <b>Regarding the list of challenges under "Why sustainable research software in the first place?" (pages 5 and 6):</b><br/> Infrastructure issues: One design choice that often limits the use or usability of local infrastructure resources is that they are often bound to institutional user accounts and thus limit collaboration possibilities with collaborators in other institutions and countries. On the other hand, pooling of infrastructure resources which could enable collaboration across organizations can be limited by lack of authentication and authorization infrastructure (AAI) or legal constraints. Later in the paper the authors indeed mention AAI (page 12) but this could already be pointed out and connected earlier.<br/> <i>We thank the reviewer for their suggestion, and have included the mentioned issues in the respective list in the section “Why sustainable research software in the first place?”.</i><br/> <b> </b><br/> Legal issues: Not only licensing is an issue but legal constraints or uncertainty about legal boundaries and identity federation can also limit the deployment of infrastructure services. Often the deployment and operation of infrastructure services is given to technical teams who may lack the legal support or expertise to clarify legal and privacy terms for the storage of data and processing of data.<br/> <i>We thank the reviewer for their suggestion, and have included the mentioned issues in the respective list in the section “Why sustainable research software in the first place?”<br/> </i><br/> Funding issues: The challenge is not only that funding is scarce but also that it does not align well with pricing models of cloud infrastructure providers. It can be easier for research groups to spend a larger chunk of the budget towards the end of a year for hardware compared to pay possibly relatively modest monthly fees for a cloud service, which however may not fit into the budget forms. These budget constraints may also limit the possibility of pooling resources and sharing them with other research groups. Software cloud infrastructure is often not considered at all in the proposal. There is also a resistance among some of my research colleagues to pay 20-50 USD/ month for an infrastructure service which is sometimes solved by reinventing the service locally "for free".<br/> Another mismatch between traditional funding models and support of software which "must continue to exist" to be sustainable (page 5), is the experience that it can take months or years until the software is picked up by other groups and contributions and questions start to roll in. But by that time the funding of the project stopped, the developer (team) may have already moved on to other positions and projects, and may not have the time to react and help, even though they still may have interest and the knowledge. Our traditional funding models consider the software to be "done" by the end of the project.<br/> <i>We thank the reviewer for these two comments. To address them, we have extended the “Funding issues” list item with a discussion of these issues.</i><br/> <br/> <b>Selection criteria for "How to decide which software to sustain?" (page 9):</b><br/> The authors mention "usage and impact", "software quality", as well as "maturity". But I would like to see also "openness and transparency" among these. The reason is that we can expect the research community to adapt to these or any metrics and we will over time observe what we measure. Any set of metrics could be criticized as to some extent being arbitrary but the advantage of including "openness and transparency" is that the community as whole would benefit from such a metric [Enrico Glerean, "Responsible conduct of research and questionable research practices", presentation, slide 471].<br/> <i>We thank the reviewer for this suggestion and realized that indeed most aspects in this section (6-10, 12, 13) are actually related to openness and transparency. Therefore, we changed the title of this section to “Software transparency and quality”.</i><br/> <br/> <b>Regarding "Who sustains research software?":</b><br/> The authors discuss the lack of recognition and awareness, as well as lack of career opportunities. It is also about respect and I was happy to see the sentence: "Not all researchers that think of themselves as RSEs pursue a faculty position as their main career goal." I have experienced that RSEs are sometimes regarded as those who somehow "failed" to obtain a faculty position whereas many RSEs have chosen this position over a faculty position because it was a better fit for their career goals. This misunderstanding can lead to a lack of respect towards this position and this career choice and can lead to excellent personnel leaving the academic environment towards commercial employment, possibly not primarily for financial reasons but sometimes to be more respected and recognized.<br/> <i>We fully agree with the reviewer and thank them for the renewed confirmation that this is seen as problematic not only by the authors.</i><br/> <br/> <b>Archiving and software preservation (page 13):</b><br/> The authors mention Docker but also Singularity should be mentioned as a tool since it is getting traction in particular on many-user systems such as higher performance computing clusters.<br/> <i>We thank the reviewer for this suggestion and now also mention Singularity and GUIX. However, we are not aiming for an exhaustive list, as options change dynamically and might even be specific to certain research communities.</i><br/> <br/> <b>Legal aspects (page 14):</b><br/> Re-licensing is mentioned in the story box and the text starts by pointing out that licensing is often not considered early enough in the project. Indeed re-licensing later in the project can be not only legally, but also organizationally very tricky, in particular for projects which developed over many years and involved many contributors in different organizations. This could be pointed out in the text as additional motivation to consider these very early in the project.<br/> <i>We thank the reviewer to point this out and it also nicely fits into the message of increasing the awareness of legal aspects early on in the project. We have added the suggested sentence in the manuscript.</i><br/> <b> </b><br/> I very much like the recommendation of providing a legal help desk for research groups to avoid the problem that out of uncertainty and fear of making a legal mistake some research groups end up not choosing any license at all which may limit further reuse of the software.<br/> <i>Thank you for supporting this idea. We have further included your idea of avoiding any license out of a fear to make legal mistakes.</i><br/> <b> </b><br/> The manuscript presents a decision tree for contributors (Figure 1) and also discusses contributor license agreements. It could be useful to point out that without clear policies or legal help desks, individuals or organizations may be hesitant to contribute to a project because they may not feel confident having enough knowledge or authority to sign such agreements and too many legal steps and question can also raise the barrier to contribute, in particular for smaller projects. Also here clear guidelines and a support desk can help removing these barriers.<br/> <i>We have decided to take the decision trees out of the manuscript to strengthen our point of publishing software under a FLOSS license. Instead, we published the decision trees together with documentation templates under a Creative Commons license via Zenodo: <a href="https://zenodo.org/record/4327148#.X9n6ui337OQ">https://zenodo.org/record/4327148#.X9n6ui337OQ</a>. In order to strengthen the point you addressed, we added some more details related to infrastructural investment.</i></span> <div class=padding-bottom id=full-comment-6210><i>We thank you for the thorough review and constructive feedback regarding the manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> <b>Regarding the list of challenges under "Why sustainable research software in the first place?" (pages 5 and 6):</b><br/> Infrastructure issues: One design choice that often limits the use or usability of local infrastructure resources is that they are often bound to institutional user accounts and thus limit collaboration possibilities with collaborators in other institutions and countries. On the other hand, pooling of infrastructure resources which could enable collaboration across organizations can be limited by lack of authentication and authorization infrastructure (AAI) or legal constraints. Later in the paper the authors indeed mention AAI (page 12) but this could already be pointed out and connected earlier.<br/> <i>We thank the reviewer for their suggestion, and have included the mentioned issues in the respective list in the section “Why sustainable research software in the first place?”.</i><br/> <b> </b><br/> Legal issues: Not only licensing is an issue but legal constraints or uncertainty about legal boundaries and identity federation can also limit the deployment of infrastructure services. Often the deployment and operation of infrastructure services is given to technical teams who may lack the legal support or expertise to clarify legal and privacy terms for the storage of data and processing of data.<br/> <i>We thank the reviewer for their suggestion, and have included the mentioned issues in the respective list in the section “Why sustainable research software in the first place?”<br/> </i><br/> Funding issues: The challenge is not only that funding is scarce but also that it does not align well with pricing models of cloud infrastructure providers. It can be easier for research groups to spend a larger chunk of the budget towards the end of a year for hardware compared to pay possibly relatively modest monthly fees for a cloud service, which however may not fit into the budget forms. These budget constraints may also limit the possibility of pooling resources and sharing them with other research groups. Software cloud infrastructure is often not considered at all in the proposal. There is also a resistance among some of my research colleagues to pay 20-50 USD/ month for an infrastructure service which is sometimes solved by reinventing the service locally "for free".<br/> Another mismatch between traditional funding models and support of software which "must continue to exist" to be sustainable (page 5), is the experience that it can take months or years until the software is picked up by other groups and contributions and questions start to roll in. But by that time the funding of the project stopped, the developer (team) may have already moved on to other positions and projects, and may not have the time to react and help, even though they still may have interest and the knowledge. Our traditional funding models consider the software to be "done" by the end of the project.<br/> <i>We thank the reviewer for these two comments. To address them, we have extended the “Funding issues” list item with a discussion of these issues.</i><br/> <br/> <b>Selection criteria for "How to decide which software to sustain?" (page 9):</b><br/> The authors mention "usage and impact", "software quality", as well as "maturity". But I would like to see also "openness and transparency" among these. The reason is that we can expect the research community to adapt to these or any metrics and we will over time observe what we measure. Any set of metrics could be criticized as to some extent being arbitrary but the advantage of including "openness and transparency" is that the community as whole would benefit from such a metric [Enrico Glerean, "Responsible conduct of research and questionable research practices", presentation, slide 471].<br/> <i>We thank the reviewer for this suggestion and realized that indeed most aspects in this section (6-10, 12, 13) are actually related to openness and transparency. Therefore, we changed the title of this section to “Software transparency and quality”.</i><br/> <br/> <b>Regarding "Who sustains research software?":</b><br/> The authors discuss the lack of recognition and awareness, as well as lack of career opportunities. It is also about respect and I was happy to see the sentence: "Not all researchers that think of themselves as RSEs pursue a faculty position as their main career goal." I have experienced that RSEs are sometimes regarded as those who somehow "failed" to obtain a faculty position whereas many RSEs have chosen this position over a faculty position because it was a better fit for their career goals. This misunderstanding can lead to a lack of respect towards this position and this career choice and can lead to excellent personnel leaving the academic environment towards commercial employment, possibly not primarily for financial reasons but sometimes to be more respected and recognized.<br/> <i>We fully agree with the reviewer and thank them for the renewed confirmation that this is seen as problematic not only by the authors.</i><br/> <br/> <b>Archiving and software preservation (page 13):</b><br/> The authors mention Docker but also Singularity should be mentioned as a tool since it is getting traction in particular on many-user systems such as higher performance computing clusters.<br/> <i>We thank the reviewer for this suggestion and now also mention Singularity and GUIX. However, we are not aiming for an exhaustive list, as options change dynamically and might even be specific to certain research communities.</i><br/> <br/> <b>Legal aspects (page 14):</b><br/> Re-licensing is mentioned in the story box and the text starts by pointing out that licensing is often not considered early enough in the project. Indeed re-licensing later in the project can be not only legally, but also organizationally very tricky, in particular for projects which developed over many years and involved many contributors in different organizations. This could be pointed out in the text as additional motivation to consider these very early in the project.<br/> <i>We thank the reviewer to point this out and it also nicely fits into the message of increasing the awareness of legal aspects early on in the project. We have added the suggested sentence in the manuscript.</i><br/> <b> </b><br/> I very much like the recommendation of providing a legal help desk for research groups to avoid the problem that out of uncertainty and fear of making a legal mistake some research groups end up not choosing any license at all which may limit further reuse of the software.<br/> <i>Thank you for supporting this idea. We have further included your idea of avoiding any license out of a fear to make legal mistakes.</i><br/> <b> </b><br/> The manuscript presents a decision tree for contributors (Figure 1) and also discusses contributor license agreements. It could be useful to point out that without clear policies or legal help desks, individuals or organizations may be hesitant to contribute to a project because they may not feel confident having enough knowledge or authority to sign such agreements and too many legal steps and question can also raise the barrier to contribute, in particular for smaller projects. Also here clear guidelines and a support desk can help removing these barriers.<br/> <i>We have decided to take the decision trees out of the manuscript to strengthen our point of publishing software under a FLOSS license. Instead, we published the decision trees together with documentation templates under a Creative Commons license via Zenodo: <a href="https://zenodo.org/record/4327148#.X9n6ui337OQ">https://zenodo.org/record/4327148#.X9n6ui337OQ</a>. In order to strengthen the point you addressed, we added some more details related to infrastructural investment.</i></div> <span class="disclosures competing-interests-display padding-top"> <strong>Competing Interests:</strong> <span class=c-i-d>No competing interests were disclosed.</span> </span> <span class="hide-comment-container response-and-comment">Close</span> </span> <div class="report-concern-wrapper last-comment"> <a href="mailto:research@f1000.com?subject=Report a concern (23224) " class=report-a-concern-link>Report a concern</a> </div> </div> </div> </li> <li class="comment-wrapper not-sortable"> <div class="one-comment respond-or-comment"> <a href="/login?originalPath=/articles/9-295/v2&scrollTo=referee-response-62872" class=register-report-comment-button data-test-id=add-response-comment><span class=contracted></span>Respond or Comment</a> </div> </li> </ul> </div> <div class="f1r-article-mobile research-layout"> <div class=reader-comment-header>COMMENTS ON THIS REPORT</div> <ul class=all-comment-wrapper data-reportID=62872> <li class="comment-wrapper not-sortable response-on-mobile" data-commentID=6210> <div class="one-comment with-divider "> <a name=article-comment-6210></a> <div class="f1r-article-desk commenter-role"> Author Response 26 Jan 2021 </div> <div> <span class=name>Axel Loewe</span><span class=name>, </span><span class=commenter-affiliation>Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany</span> </div> <div class="f1r-article-mobile the-comment-date"> 26 Jan 2021 </div> <div class=f1r-article-mobile> <div class="article-pubinfo-mobile comments-section"> <span class="version-info-box comments"> Author Response </span> </div> </div> <div class="comment-content "> <span class=truncated-comment> <i>We thank you for the thorough review and constructive feedback regarding the manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> <b>Regarding</b> ... <span class="continue-reading-container response-and-comment">Continue reading</span> </span> <span class="full-comment hidden"> <span id=full-comment-original-6210 class=hidden><i>We thank you for the thorough review and constructive feedback regarding the manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> <b>Regarding the list of challenges under "Why sustainable research software in the first place?" (pages 5 and 6):</b><br/> Infrastructure issues: One design choice that often limits the use or usability of local infrastructure resources is that they are often bound to institutional user accounts and thus limit collaboration possibilities with collaborators in other institutions and countries. On the other hand, pooling of infrastructure resources which could enable collaboration across organizations can be limited by lack of authentication and authorization infrastructure (AAI) or legal constraints. Later in the paper the authors indeed mention AAI (page 12) but this could already be pointed out and connected earlier.<br/> <i>We thank the reviewer for their suggestion, and have included the mentioned issues in the respective list in the section “Why sustainable research software in the first place?”.</i><br/> <b> </b><br/> Legal issues: Not only licensing is an issue but legal constraints or uncertainty about legal boundaries and identity federation can also limit the deployment of infrastructure services. Often the deployment and operation of infrastructure services is given to technical teams who may lack the legal support or expertise to clarify legal and privacy terms for the storage of data and processing of data.<br/> <i>We thank the reviewer for their suggestion, and have included the mentioned issues in the respective list in the section “Why sustainable research software in the first place?”<br/> </i><br/> Funding issues: The challenge is not only that funding is scarce but also that it does not align well with pricing models of cloud infrastructure providers. It can be easier for research groups to spend a larger chunk of the budget towards the end of a year for hardware compared to pay possibly relatively modest monthly fees for a cloud service, which however may not fit into the budget forms. These budget constraints may also limit the possibility of pooling resources and sharing them with other research groups. Software cloud infrastructure is often not considered at all in the proposal. There is also a resistance among some of my research colleagues to pay 20-50 USD/ month for an infrastructure service which is sometimes solved by reinventing the service locally "for free".<br/> Another mismatch between traditional funding models and support of software which "must continue to exist" to be sustainable (page 5), is the experience that it can take months or years until the software is picked up by other groups and contributions and questions start to roll in. But by that time the funding of the project stopped, the developer (team) may have already moved on to other positions and projects, and may not have the time to react and help, even though they still may have interest and the knowledge. Our traditional funding models consider the software to be "done" by the end of the project.<br/> <i>We thank the reviewer for these two comments. To address them, we have extended the “Funding issues” list item with a discussion of these issues.</i><br/> <br/> <b>Selection criteria for "How to decide which software to sustain?" (page 9):</b><br/> The authors mention "usage and impact", "software quality", as well as "maturity". But I would like to see also "openness and transparency" among these. The reason is that we can expect the research community to adapt to these or any metrics and we will over time observe what we measure. Any set of metrics could be criticized as to some extent being arbitrary but the advantage of including "openness and transparency" is that the community as whole would benefit from such a metric [Enrico Glerean, "Responsible conduct of research and questionable research practices", presentation, slide 471].<br/> <i>We thank the reviewer for this suggestion and realized that indeed most aspects in this section (6-10, 12, 13) are actually related to openness and transparency. Therefore, we changed the title of this section to “Software transparency and quality”.</i><br/> <br/> <b>Regarding "Who sustains research software?":</b><br/> The authors discuss the lack of recognition and awareness, as well as lack of career opportunities. It is also about respect and I was happy to see the sentence: "Not all researchers that think of themselves as RSEs pursue a faculty position as their main career goal." I have experienced that RSEs are sometimes regarded as those who somehow "failed" to obtain a faculty position whereas many RSEs have chosen this position over a faculty position because it was a better fit for their career goals. This misunderstanding can lead to a lack of respect towards this position and this career choice and can lead to excellent personnel leaving the academic environment towards commercial employment, possibly not primarily for financial reasons but sometimes to be more respected and recognized.<br/> <i>We fully agree with the reviewer and thank them for the renewed confirmation that this is seen as problematic not only by the authors.</i><br/> <br/> <b>Archiving and software preservation (page 13):</b><br/> The authors mention Docker but also Singularity should be mentioned as a tool since it is getting traction in particular on many-user systems such as higher performance computing clusters.<br/> <i>We thank the reviewer for this suggestion and now also mention Singularity and GUIX. However, we are not aiming for an exhaustive list, as options change dynamically and might even be specific to certain research communities.</i><br/> <br/> <b>Legal aspects (page 14):</b><br/> Re-licensing is mentioned in the story box and the text starts by pointing out that licensing is often not considered early enough in the project. Indeed re-licensing later in the project can be not only legally, but also organizationally very tricky, in particular for projects which developed over many years and involved many contributors in different organizations. This could be pointed out in the text as additional motivation to consider these very early in the project.<br/> <i>We thank the reviewer to point this out and it also nicely fits into the message of increasing the awareness of legal aspects early on in the project. We have added the suggested sentence in the manuscript.</i><br/> <b> </b><br/> I very much like the recommendation of providing a legal help desk for research groups to avoid the problem that out of uncertainty and fear of making a legal mistake some research groups end up not choosing any license at all which may limit further reuse of the software.<br/> <i>Thank you for supporting this idea. We have further included your idea of avoiding any license out of a fear to make legal mistakes.</i><br/> <b> </b><br/> The manuscript presents a decision tree for contributors (Figure 1) and also discusses contributor license agreements. It could be useful to point out that without clear policies or legal help desks, individuals or organizations may be hesitant to contribute to a project because they may not feel confident having enough knowledge or authority to sign such agreements and too many legal steps and question can also raise the barrier to contribute, in particular for smaller projects. Also here clear guidelines and a support desk can help removing these barriers.<br/> <i>We have decided to take the decision trees out of the manuscript to strengthen our point of publishing software under a FLOSS license. Instead, we published the decision trees together with documentation templates under a Creative Commons license via Zenodo: <a href="https://zenodo.org/record/4327148#.X9n6ui337OQ">https://zenodo.org/record/4327148#.X9n6ui337OQ</a>. In order to strengthen the point you addressed, we added some more details related to infrastructural investment.</i></span> <div class=padding-bottom id=full-comment-6210><i>We thank you for the thorough review and constructive feedback regarding the manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> <b>Regarding the list of challenges under "Why sustainable research software in the first place?" (pages 5 and 6):</b><br/> Infrastructure issues: One design choice that often limits the use or usability of local infrastructure resources is that they are often bound to institutional user accounts and thus limit collaboration possibilities with collaborators in other institutions and countries. On the other hand, pooling of infrastructure resources which could enable collaboration across organizations can be limited by lack of authentication and authorization infrastructure (AAI) or legal constraints. Later in the paper the authors indeed mention AAI (page 12) but this could already be pointed out and connected earlier.<br/> <i>We thank the reviewer for their suggestion, and have included the mentioned issues in the respective list in the section “Why sustainable research software in the first place?”.</i><br/> <b> </b><br/> Legal issues: Not only licensing is an issue but legal constraints or uncertainty about legal boundaries and identity federation can also limit the deployment of infrastructure services. Often the deployment and operation of infrastructure services is given to technical teams who may lack the legal support or expertise to clarify legal and privacy terms for the storage of data and processing of data.<br/> <i>We thank the reviewer for their suggestion, and have included the mentioned issues in the respective list in the section “Why sustainable research software in the first place?”<br/> </i><br/> Funding issues: The challenge is not only that funding is scarce but also that it does not align well with pricing models of cloud infrastructure providers. It can be easier for research groups to spend a larger chunk of the budget towards the end of a year for hardware compared to pay possibly relatively modest monthly fees for a cloud service, which however may not fit into the budget forms. These budget constraints may also limit the possibility of pooling resources and sharing them with other research groups. Software cloud infrastructure is often not considered at all in the proposal. There is also a resistance among some of my research colleagues to pay 20-50 USD/ month for an infrastructure service which is sometimes solved by reinventing the service locally "for free".<br/> Another mismatch between traditional funding models and support of software which "must continue to exist" to be sustainable (page 5), is the experience that it can take months or years until the software is picked up by other groups and contributions and questions start to roll in. But by that time the funding of the project stopped, the developer (team) may have already moved on to other positions and projects, and may not have the time to react and help, even though they still may have interest and the knowledge. Our traditional funding models consider the software to be "done" by the end of the project.<br/> <i>We thank the reviewer for these two comments. To address them, we have extended the “Funding issues” list item with a discussion of these issues.</i><br/> <br/> <b>Selection criteria for "How to decide which software to sustain?" (page 9):</b><br/> The authors mention "usage and impact", "software quality", as well as "maturity". But I would like to see also "openness and transparency" among these. The reason is that we can expect the research community to adapt to these or any metrics and we will over time observe what we measure. Any set of metrics could be criticized as to some extent being arbitrary but the advantage of including "openness and transparency" is that the community as whole would benefit from such a metric [Enrico Glerean, "Responsible conduct of research and questionable research practices", presentation, slide 471].<br/> <i>We thank the reviewer for this suggestion and realized that indeed most aspects in this section (6-10, 12, 13) are actually related to openness and transparency. Therefore, we changed the title of this section to “Software transparency and quality”.</i><br/> <br/> <b>Regarding "Who sustains research software?":</b><br/> The authors discuss the lack of recognition and awareness, as well as lack of career opportunities. It is also about respect and I was happy to see the sentence: "Not all researchers that think of themselves as RSEs pursue a faculty position as their main career goal." I have experienced that RSEs are sometimes regarded as those who somehow "failed" to obtain a faculty position whereas many RSEs have chosen this position over a faculty position because it was a better fit for their career goals. This misunderstanding can lead to a lack of respect towards this position and this career choice and can lead to excellent personnel leaving the academic environment towards commercial employment, possibly not primarily for financial reasons but sometimes to be more respected and recognized.<br/> <i>We fully agree with the reviewer and thank them for the renewed confirmation that this is seen as problematic not only by the authors.</i><br/> <br/> <b>Archiving and software preservation (page 13):</b><br/> The authors mention Docker but also Singularity should be mentioned as a tool since it is getting traction in particular on many-user systems such as higher performance computing clusters.<br/> <i>We thank the reviewer for this suggestion and now also mention Singularity and GUIX. However, we are not aiming for an exhaustive list, as options change dynamically and might even be specific to certain research communities.</i><br/> <br/> <b>Legal aspects (page 14):</b><br/> Re-licensing is mentioned in the story box and the text starts by pointing out that licensing is often not considered early enough in the project. Indeed re-licensing later in the project can be not only legally, but also organizationally very tricky, in particular for projects which developed over many years and involved many contributors in different organizations. This could be pointed out in the text as additional motivation to consider these very early in the project.<br/> <i>We thank the reviewer to point this out and it also nicely fits into the message of increasing the awareness of legal aspects early on in the project. We have added the suggested sentence in the manuscript.</i><br/> <b> </b><br/> I very much like the recommendation of providing a legal help desk for research groups to avoid the problem that out of uncertainty and fear of making a legal mistake some research groups end up not choosing any license at all which may limit further reuse of the software.<br/> <i>Thank you for supporting this idea. We have further included your idea of avoiding any license out of a fear to make legal mistakes.</i><br/> <b> </b><br/> The manuscript presents a decision tree for contributors (Figure 1) and also discusses contributor license agreements. It could be useful to point out that without clear policies or legal help desks, individuals or organizations may be hesitant to contribute to a project because they may not feel confident having enough knowledge or authority to sign such agreements and too many legal steps and question can also raise the barrier to contribute, in particular for smaller projects. Also here clear guidelines and a support desk can help removing these barriers.<br/> <i>We have decided to take the decision trees out of the manuscript to strengthen our point of publishing software under a FLOSS license. Instead, we published the decision trees together with documentation templates under a Creative Commons license via Zenodo: <a href="https://zenodo.org/record/4327148#.X9n6ui337OQ">https://zenodo.org/record/4327148#.X9n6ui337OQ</a>. In order to strengthen the point you addressed, we added some more details related to infrastructural investment.</i></div> <span class="disclosures competing-interests-display padding-top"> <strong>Competing Interests:</strong> <span class=c-i-d>No competing interests were disclosed.</span> </span> <span class="hide-comment-container response-and-comment">Close</span> </span> <div class="report-concern-wrapper last-comment"> <a href="mailto:research@f1000.com?subject=Report a concern (23224) " class=report-a-concern-link>Report a concern</a> </div> </div> </div> </li> <li class="comment-wrapper not-sortable button-wrapper"> <div class=comment-button-wrapper> <a href="/login?originalPath=/articles/9-295/v2&scrollTo=referee-response-62872" class=register-report-comment-button data-test-id=add-report-comment> <button class="primary orange extra-padding"> COMMENT ON THIS REPORT </button> </a> </div> </li> </ul> </div> </div> </div> <div class="referee-response-container no-border last-report-for-version" data-reportID=62873> <div class="f1r-article-mobile research-layout"> <div class="version-border previous"></div> </div> <a name=referee-response-62873></a> <div id=report-metadata-62873 class=is-hidden> <input type=hidden name=reportId value=62873 /> </div> <div class="f1r-article-mobile-inline research-layout float-right margin-right-20"> <span class="f1r-icon-circle-wrapper with-border white-bg big referee-reports"> <span class="f1r-icon icon-87_approved_reservations status-green smaller" title="Approved with Reservations"></span> </span> </div> <div class="article-count-box response-and-comment has-shadow"> <div class=comments-count-title>Views</div> <div class=comments-count-container>0</div> <div class="article-interaction-info article-page"> <div class="cite-article-popup-wrapper article-page-interaction-box"> <div class="article-interaction-button cite-article-button" title="Cite this report" data-windowref=cite-report-popup-62873-2> <span class="f1r-icon icon-82_quote"></span> <a href="#" class="button-link cite-article-popup-link response-and-comment" title="Cite Report" data-test-id=cite-report-2>Cite</a> </div> <div id=cite-report-popup-62873-2 class="popup-window-wrapper is-hidden"> <div class=cite-popup-background></div> <div class="popup-window cite-this-article-box research-layout"> <div class="popup-window-title small cite-title">How to cite this report:</div> <span id=cite-report-text-62873-2 data-test-id=mob_report_copy-citation_text> <span class="article-title-and-info in-popup">Hasselbring W. Reviewer Report For: An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action [version 2; peer review: 2 approved]</span>. <i>F1000Research</i> 2021, <b>9</b>:295 (<a class=new-orange href="https://doi.org/10.5256/f1000research.25640.r62873" target=_blank>https://doi.org/10.5256/f1000research.25640.r62873</a>) </span> <div class=padding-top data-test-id=report-url> The direct URL for this report is:<br> <a href="https://f1000research.com/articles/9-295/v1#referee-response-62873" class=new-orange target=_blank>https://f1000research.com/articles/9-295/v1#referee-response-62873</a> </div> <div class="popup-window-title small note margin-top-20 margin-bottom-20"> <strong>NOTE:</strong> it is important to ensure the information in square brackets after the title is included in this citation. </div> <div class=float-right> <button class="secondary no-fill orange-text-and-border margin-right-20 close-cite-popup uppercase">Close</button> <button class="secondary orange copy-cite-article-version uppercase js-clipboard" title="Copy the current citation details to the clipboard." data-clipboard-target="#cite-report-text-62873-2" data-test-id=mob_report_copy-citation_button>Copy Citation Details</button> </div> </div> </div> </div> </div> </div> <div class="referee-report-info-box referee-report-details-box"> <span class="bold mobile-report-text">Reviewer Report</span> <span class=mobile-report-info>13 May 2020</span> </div> <span class="info-separation padding-bottom padding-left"> <div> <span class=bold>Wilhelm Hasselbring<a href="http://orcid.org/0000-0001-6625-4335" target=_blank title="Open ORCID"><span class=orcid-logo-for-author-list></span></a></span><span class=f1r-article-desk-inline>,</span> Software Engineering Group, Kiel University, Kiel, Germany </div> </span> <div class="research-layout referee-report-status-line"> <span class="f1r-icon icon-87_approved_reservations status-green smaller" title="Approved with Reservations"></span> <span class=status-text>Approved with Reservations</span> </div> <div class=f1r-article-mobile> <span class=article-pubinfo-mobile> <div class="version-info-box views">VIEWS 0</div> </span> </div> <div class=referee-report-content> <div class=hidden-text> <a href="https://doi.org/10.5256/f1000research.25640.r62873" target=_blank>https://doi.org/10.5256/f1000research.25640.r62873</a> </div> <span class=truncated-comment> The authors describe the state of the practice and current challenges for research software sustainability and suggest measures towards improvements that can solve these challenges. In particular, they propose to fund a German Software Sustainability Institute. The paper is the ... <span class="f1r-article-desk-inline continue-reading-container response-and-comment increment-view-count" data-reportID=62873>Continue reading</span> <div class="f1r-article-mobile research-layout margin-top-30 is-centered"> <button class="primary orange-text white-bg bigger-text expand-button-mobile continue-reading-container show with-border"> READ ALL <span class="f1r-icon icon-14_more_small orange vmiddle big"></span> </button> </div> </span> <span class="hidden hidden-report-text"> <div class=hidden-report-text__comment>The authors describe the state of the practice and current challenges for research software sustainability and suggest measures towards improvements that can solve these challenges. In particular, they propose to fund a German Software Sustainability Institute. The paper is the result of a community effort, with work undertaken during two workshops and subsequent collaborative work across the larger RSE community in Germany.<br/> <br/> The UK Software Sustainability Institute has already been established during a decade (<a href="https://www.software.ac.uk/blog/2020-05-05-impact-institute-10-years">https://www.software.ac.uk/blog/2020-05-05-impact-institute-10-years</a>). Thus, the idea of such an institute is not new, but it makes sense to take a specific look at the German situation. Besides universities, the German states (local and in particular federal) fund significant large-scale research associations (Helmholtz/DLR, Max-Planck, Leibniz). This is not the case for most other European states, at least not with a similar scale. Another specialty is the lack of long-term funding for research software engineers, as discussed by the authors.<br/> <br/> The paper is well-written and easy to read. I like the boxed story of Kim’s career path.<br/> <br/> However, I’ve some suggestions for improving the paper: <ul> <li>Concerning the statement “In order to support research, a sustainable software must be correct”, I suggest to include a short discussion of the test oracle problem for scientific software (see for instance https://doi.org/10.1109/SECSE.2013.6615099<a href="#rep-ref-62873-1"><sup>1</sup></a>).<br/> </li> <li>Concerning the discussion of “The list of criteria presented in this section could be the basis for a structured review process…” I suggest to include two additional initiatives for software review. The first is artifact evaluation in computer science conferences (the process is explained in <a href="https://doi.org/10.1515/itit-2019-0040">https://doi.org/10.1515/itit-2019-0040</a><a href="#rep-ref-62873-2"><sup>2</sup></a>). The second is the SPEC Research Group’s review process of tools for quantitative system evaluation and analysis (https://research.spec.org/tools/submission.html).</li> </ul> The authors write “We also argue that truly sustainable research software must ideally be published under a Free/Libre Open Source Software (FLOSS) license, and follow an open development model…” what I fully support (see for instance <a href="https://doi.org/10.1515/itit-2019-0040">https://doi.org/10.1515/itit-2019-0040</a><a href="#rep-ref-62873-2"><sup>2</sup></a>). However, later under the section heading “Legal aspects” this requirement is thwarted. I fully agree that legal aspects have to be considered, but the general bias of this section seems to be on commercial licensing of research software. For instance, the decision tree in Figure 1 starts with the question “Licensing planned?”. I assume that commercial licensing is meant, but this is not clear since the figures are not explained in the paper. Instead, the process should start with open sourcing the software. If licenses such as Apache or MIT are applied, the research institutions may later still commercialize the software if appropriate. Such open source licensing is also beneficial for start-ups, that intend to provide professional services for the software.<br/> <br/> My experience with technology transfer units of German universities and research institutes is that they do not understand the ideas of open source business models (see for instance <a href="https://doi.org/10.1109/MC.2019.2898163">https://doi.org/10.1109/MC.2019.2898163</a><a href="#rep-ref-62873-3"><sup>3</sup></a>). Their focus is on patents and commercializing licenses, sometimes also on start-ups. Conversely, in the software industry, one major motivation for open sourcing software is on improving the quality of software. I cite from <a href="https://doi.org/10.1109/ICSAW.2017.11">https://doi.org/10.1109/ICSAW.2017.11</a><a href="#rep-ref-62873-4"><sup>4</sup></a> : “the open-source approach has some psychological effects: Developers show a tendency to apply higher quality standards if they know that the code will be publicly available.” For sustainability, quality is an important property of software.<br/> <br/> The Figures 1-4 do more harm than good. They are daunting to researchers who intend to publish their code open source. These figures should be removed from the paper, they are useless without proper explanation.<br/> <br/> I suggest that the authors focus in the present paper on their main message (request for funding a German Software Sustainability Institute, which I fully support). Figures 1-4 could be moved to a separate paper, enriched with proper explanation.</div> <div class=md> <div class=questionnaire> <ul class=c-questionnaire> <li class=c-questionnaire__item> <p class=c-questionnaire__question>Is the topic of the opinion article discussed accurately in the context of the current literature?</p> <p class=c-questionnaire__answer>Yes</p> </li> <li class=c-questionnaire__item> <p class=c-questionnaire__question>Are all factual statements correct and adequately supported by citations?</p> <p class=c-questionnaire__answer>Yes</p> </li> <li class=c-questionnaire__item> <p class=c-questionnaire__question>Are arguments sufficiently supported by evidence from the published literature?</p> <p class=c-questionnaire__answer>Partly</p> </li> <li class=c-questionnaire__item> <p class=c-questionnaire__question>Are the conclusions drawn balanced and justified on the basis of the presented arguments?</p> <p class=c-questionnaire__answer>Partly</p> </li> </ul> </div> </div> <p><b>References</b></p> <a name=rep-ref-62873-1></a>1. Kanewala U, Bieman JM: Techniques for testing scientific programs without an oracle. <i>IEEE Xplore</i>. 2013. <a target=_blank href="https://doi.org/10.1109/SECSE.2013.6615099">Publisher Full Text</a> <br><a name=rep-ref-62873-2></a>2. Hasselbring W, Carr L, Hettrick S, Packer H, et al.: From FAIR research data toward FAIR and open research software. <i>it - Information Technology</i>. 2020; <b>62</b> (1): 39-47 <a target=_blank href="https://doi.org/10.1515/itit-2019-0040">Publisher Full Text</a> <br><a name=rep-ref-62873-3></a>3. Riehle D: The Innovations of Open Source. <i>Computer</i>. 2019; <b>52</b> (4): 59-63 <a target=_blank href="https://doi.org/10.1109/MC.2019.2898163">Publisher Full Text</a> <br><a name=rep-ref-62873-4></a>4. Hasselbring W, Steinacker G: Microservice Architectures for Scalability, Agility and Reliability in E-Commerce. <i>IEEE</i>. 2017. <a target=_blank href="https://doi.org/10.1109/ICSAW.2017.11">Publisher Full Text</a> <br><a name=rep-ref-62873-5></a>5. Tools Submission Portal. <i>SPEC Research Group</i>. <a target=_blank href="https://research.spec.org/tools/submission.html">Reference Source</a> <br> <p style="margin-top: 10px;"><b class=mobile-block-display>Competing Interests:</b> No competing interests were disclosed.</p> <p><b>Reviewer Expertise:</b> Software Engineering</p> <span class="info-separation padding-top padding-bottom comment-statement-mobile"> <div class=f1r-article-mobile> <div class="content-divider report-info"></div> </div> <strong> I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. </strong> </span> <div class=f1r-article-mobile> <div class="content-divider report-info"></div> </div> <span class="f1r-article-desk hide-comment-container response-and-comment">Close</span> <div class="f1r-article-mobile research-layout margin-top-30 is-centered"> <button class="primary orange-text white-bg bigger-text expand-button-mobile hide-comment-container hide with-border"> READ LESS <span class="f1r-icon icon-10_less_small orange vmiddle big"></span> </button> </div> </span> <div class="f1r-article-mobile cite-report-wrapper"> <a href="#" class=mobile-cite-report>CITE</a> </div> <div class="research-layout mobile-cite-report-popup-wrapper"> <div class=bgmask></div> <div class=mobile-cite-report-popup> <span class="f1r-icon icon-88_not_approved close-cite-report-popup"></span> <div class=cite-report-popup-title>CITE</div> <span class=cite-report-popup-top-content> HOW TO CITE THIS REPORT </span> <div class=cite-report-popup-content> <span class=copy-cite-report id=cite-report-text-mobile-62873-2> <span class=article-title-and-info>Hasselbring W. Reviewer Report For: An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action [version 2; peer review: 2 approved]</span>. <i>F1000Research</i> 2021, <b>9</b>:295 (<a class=new-orange href="https://doi.org/10.5256/f1000research.25640.r62873" target=_blank>https://doi.org/10.5256/f1000research.25640.r62873</a>) </span> </div> <div class=cite-report-popup-divider></div> <div class=cite-report-popup-content data-test-id=report-url> The direct URL for this report is:<br> <a href="https://f1000research.com/articles/9-295/v1#referee-response-62873" class=new-orangetarget=_blank>https://f1000research.com/articles/9-295/v1#referee-response-62873</a> </div> <div class=cite-report-popup-divider></div> <div class=cite-report-popup-content> NOTE: <em>it is important to ensure the information in <b>square brackets after the title</b> is included in all citations of this article.</em> </div> <div class=cite-report-popup-buttons> <button type=button class="primary orange extra-padding copy-cite-report-mobile js-clipboard" title="Copy the current citation details to the clipboard." data-clipboard-target="#cite-report-text-mobile-62873-2" data-test-id=mob_report_copy-citation_button>COPY CITATION DETAILS</button> </div> </div> </div> <div class=report-concern-wrapper> <a class=report-a-concern-link href="mailto:research@f1000.com?subject=Report a concern (23224) ">Report a concern</a> </div> </div> <div class=comments-list-container id=referee-comments-62873-2> <div class=f1r-article-desk> <ul class=all-comment-wrapper data-reportID=62873> <li class="comment-wrapper not-sortable " data-commentID=6209> <div class="one-comment with-divider "> <a name=article-comment-6209></a> <div class="f1r-article-desk commenter-role"> Author Response 26 Jan 2021 </div> <div> <span class=name>Axel Loewe</span><span class=name>, </span><span class=commenter-affiliation>Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany</span> </div> <div class="f1r-article-mobile the-comment-date"> 26 Jan 2021 </div> <div class=f1r-article-mobile> <div class="article-pubinfo-mobile comments-section"> <span class="version-info-box comments"> Author Response </span> </div> </div> <div class="comment-content "> <span class=truncated-comment> <i>Thank you for the thorough review and constructive feedback regarding our manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> Concerning the ... <span class="continue-reading-container response-and-comment">Continue reading</span> </span> <span class="full-comment hidden"> <span id=full-comment-original-6209 class=hidden><i>Thank you for the thorough review and constructive feedback regarding our manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> Concerning the statement “In order to support research, a sustainable software must be correct”, I suggest to include a short discussion of the test oracle problem for scientific software (see for instance <a href="https://doi.org/10.1109/SECSE.2013.66150991">https://doi.org/10.1109/SECSE.2013.66150991</a>).<br/> <i>We thank the reviewer for this suggestion, and included a brief discussion of the test oracle problem as suggested, and additionally of further challenges to verification and validation, such as large configuration spaces and heterogeneous data (as discussed in e.g. <a href="https://doi.org/10.1109/SE4Science.2019.00010">https://doi.org/10.1109/SE4Science.2019.00010</a>), and have suggested to implement the solutions mentioned in the literature.</i><br/> <br/> Concerning the discussion of “The list of criteria presented in this section could be the basis for a structured review process…” I suggest to include two additional initiatives for software review. The first is artifact evaluation in computer science conferences (the process is explained in https://doi.org/10.1515/itit-2019-00402). The second is the SPEC Research Group’s review process of tools for quantitative system evaluation and analysis (<a href="https://research.spec.org/tools/submission.html">https://research.spec.org/tools/submission.html</a>).<br/> <i>We thank the reviewer for this suggestion and included the artifact review approach in the introduction to the criteria section. The aspects of repeatability, reproducibility, and replicability are aimed more at the results of computational research rather than research software itself, we feel. Therefore, we didn’t include specific criteria in the list suggested to be used when evaluating research software for sustained funding.<br/> While the SPEC submission process is very clear, we could not find any concrete criteria applied during the review (except for requirements regarding the license).</i><br/> <br/> The authors write “We also argue that truly sustainable research software must ideally be published under a Free/Libre Open Source Software (FLOSS) license, and follow an open development model…” what I fully support (see for instance https://doi.org/10.1515/itit-2019-00402). However, later under the section heading “Legal aspects” this requirement is thwarted. I fully agree that legal aspects have to be considered, but the general bias of this section seems to be on commercial licensing of research software. For instance, the decision tree in Figure 1 starts with the question “Licensing planned?”. I assume that commercial licensing is meant, but this is not clear since the figures are not explained in the paper. Instead, the process should start with open sourcing the software. If licenses such as Apache or MIT are applied, the research institutions may later still commercialize the software if appropriate. Such open source licensing is also beneficial for start-ups that intend to provide professional services for the software. <br/> My experience with technology transfer units of German universities and research institutes is that they do not understand the ideas of open source business models (see for instance https://doi.org/10.1109/MC.2019.28981633). Their focus is on patents and commercializing licenses, sometimes also on start-ups. Conversely, in the software industry, one major motivation for open sourcing software is on improving the quality of software. I cite from https://doi.org/10.1109/ICSAW.2017.114 : “the open-source approach has some psychological effects: Developers show a tendency to apply higher quality standards if they know that the code will be publicly available.” For sustainability, quality is an important property of software.<br/> <i>As further detailed below, we have moved the decision trees out of this manuscript as we see the problems and agree to your arguments. We like your suggested aspect of commercialization of FLOSS licensed software and included this aspect in the manuscript.</i><br/> <br/> The Figures 1-4 do more harm than good. They are daunting to researchers who intend to publish their code open source. These figures should be removed from the paper, they are useless without proper explanation.<br/> I suggest that the authors focus in the present paper on their main message (request for funding a German Software Sustainability Institute, which I fully support). Figures 1-4 could be moved to a separate paper, enriched with proper explanation.<br/> <i>We thank the reviewer for this suggestion. Our initial thought was to place these decision trees in the supplemental material but did not realize that this is not the policy of f1000. The editorial team moved them into the main article. This is the reason why the decision trees appeared without additional information in the manuscript. We have now decided to take the Figures out and have published them together with documentation templates in a separate report via Zenodo unde a Creative Commons license, see <a href="https://zenodo.org/record/4327148#.X9n6ui337OQ">https://zenodo.org/record/4327148#.X9n6ui337OQ</a>. This report is now cited in the F1000 manuscript.</i></span> <div class=padding-bottom id=full-comment-6209><i>Thank you for the thorough review and constructive feedback regarding our manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> Concerning the statement “In order to support research, a sustainable software must be correct”, I suggest to include a short discussion of the test oracle problem for scientific software (see for instance <a href="https://doi.org/10.1109/SECSE.2013.66150991">https://doi.org/10.1109/SECSE.2013.66150991</a>).<br/> <i>We thank the reviewer for this suggestion, and included a brief discussion of the test oracle problem as suggested, and additionally of further challenges to verification and validation, such as large configuration spaces and heterogeneous data (as discussed in e.g. <a href="https://doi.org/10.1109/SE4Science.2019.00010">https://doi.org/10.1109/SE4Science.2019.00010</a>), and have suggested to implement the solutions mentioned in the literature.</i><br/> <br/> Concerning the discussion of “The list of criteria presented in this section could be the basis for a structured review process…” I suggest to include two additional initiatives for software review. The first is artifact evaluation in computer science conferences (the process is explained in https://doi.org/10.1515/itit-2019-00402). The second is the SPEC Research Group’s review process of tools for quantitative system evaluation and analysis (<a href="https://research.spec.org/tools/submission.html">https://research.spec.org/tools/submission.html</a>).<br/> <i>We thank the reviewer for this suggestion and included the artifact review approach in the introduction to the criteria section. The aspects of repeatability, reproducibility, and replicability are aimed more at the results of computational research rather than research software itself, we feel. Therefore, we didn’t include specific criteria in the list suggested to be used when evaluating research software for sustained funding.<br/> While the SPEC submission process is very clear, we could not find any concrete criteria applied during the review (except for requirements regarding the license).</i><br/> <br/> The authors write “We also argue that truly sustainable research software must ideally be published under a Free/Libre Open Source Software (FLOSS) license, and follow an open development model…” what I fully support (see for instance https://doi.org/10.1515/itit-2019-00402). However, later under the section heading “Legal aspects” this requirement is thwarted. I fully agree that legal aspects have to be considered, but the general bias of this section seems to be on commercial licensing of research software. For instance, the decision tree in Figure 1 starts with the question “Licensing planned?”. I assume that commercial licensing is meant, but this is not clear since the figures are not explained in the paper. Instead, the process should start with open sourcing the software. If licenses such as Apache or MIT are applied, the research institutions may later still commercialize the software if appropriate. Such open source licensing is also beneficial for start-ups that intend to provide professional services for the software. <br/> My experience with technology transfer units of German universities and research institutes is that they do not understand the ideas of open source business models (see for instance https://doi.org/10.1109/MC.2019.28981633). Their focus is on patents and commercializing licenses, sometimes also on start-ups. Conversely, in the software industry, one major motivation for open sourcing software is on improving the quality of software. I cite from https://doi.org/10.1109/ICSAW.2017.114 : “the open-source approach has some psychological effects: Developers show a tendency to apply higher quality standards if they know that the code will be publicly available.” For sustainability, quality is an important property of software.<br/> <i>As further detailed below, we have moved the decision trees out of this manuscript as we see the problems and agree to your arguments. We like your suggested aspect of commercialization of FLOSS licensed software and included this aspect in the manuscript.</i><br/> <br/> The Figures 1-4 do more harm than good. They are daunting to researchers who intend to publish their code open source. These figures should be removed from the paper, they are useless without proper explanation.<br/> I suggest that the authors focus in the present paper on their main message (request for funding a German Software Sustainability Institute, which I fully support). Figures 1-4 could be moved to a separate paper, enriched with proper explanation.<br/> <i>We thank the reviewer for this suggestion. Our initial thought was to place these decision trees in the supplemental material but did not realize that this is not the policy of f1000. The editorial team moved them into the main article. This is the reason why the decision trees appeared without additional information in the manuscript. We have now decided to take the Figures out and have published them together with documentation templates in a separate report via Zenodo unde a Creative Commons license, see <a href="https://zenodo.org/record/4327148#.X9n6ui337OQ">https://zenodo.org/record/4327148#.X9n6ui337OQ</a>. This report is now cited in the F1000 manuscript.</i></div> <span class="disclosures competing-interests-display padding-top"> <strong>Competing Interests:</strong> <span class=c-i-d>No competing interests were disclosed.</span> </span> <span class="hide-comment-container response-and-comment">Close</span> </span> <div class="report-concern-wrapper last-comment"> <a href="mailto:research@f1000.com?subject=Report a concern (23224) " class=report-a-concern-link>Report a concern</a> </div> </div> </div> </li> <li class="comment-wrapper not-sortable"> <div class="one-comment respond-or-comment"> <a href="/login?originalPath=/articles/9-295/v2&scrollTo=referee-response-62873" class=register-report-comment-button data-test-id=add-response-comment><span class=contracted></span>Respond or Comment</a> </div> </li> </ul> </div> <div class="f1r-article-mobile research-layout"> <div class=reader-comment-header>COMMENTS ON THIS REPORT</div> <ul class=all-comment-wrapper data-reportID=62873> <li class="comment-wrapper not-sortable response-on-mobile" data-commentID=6209> <div class="one-comment with-divider "> <a name=article-comment-6209></a> <div class="f1r-article-desk commenter-role"> Author Response 26 Jan 2021 </div> <div> <span class=name>Axel Loewe</span><span class=name>, </span><span class=commenter-affiliation>Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany</span> </div> <div class="f1r-article-mobile the-comment-date"> 26 Jan 2021 </div> <div class=f1r-article-mobile> <div class="article-pubinfo-mobile comments-section"> <span class="version-info-box comments"> Author Response </span> </div> </div> <div class="comment-content "> <span class=truncated-comment> <i>Thank you for the thorough review and constructive feedback regarding our manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> Concerning the ... <span class="continue-reading-container response-and-comment">Continue reading</span> </span> <span class="full-comment hidden"> <span id=full-comment-original-6209 class=hidden><i>Thank you for the thorough review and constructive feedback regarding our manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> Concerning the statement “In order to support research, a sustainable software must be correct”, I suggest to include a short discussion of the test oracle problem for scientific software (see for instance <a href="https://doi.org/10.1109/SECSE.2013.66150991">https://doi.org/10.1109/SECSE.2013.66150991</a>).<br/> <i>We thank the reviewer for this suggestion, and included a brief discussion of the test oracle problem as suggested, and additionally of further challenges to verification and validation, such as large configuration spaces and heterogeneous data (as discussed in e.g. <a href="https://doi.org/10.1109/SE4Science.2019.00010">https://doi.org/10.1109/SE4Science.2019.00010</a>), and have suggested to implement the solutions mentioned in the literature.</i><br/> <br/> Concerning the discussion of “The list of criteria presented in this section could be the basis for a structured review process…” I suggest to include two additional initiatives for software review. The first is artifact evaluation in computer science conferences (the process is explained in https://doi.org/10.1515/itit-2019-00402). The second is the SPEC Research Group’s review process of tools for quantitative system evaluation and analysis (<a href="https://research.spec.org/tools/submission.html">https://research.spec.org/tools/submission.html</a>).<br/> <i>We thank the reviewer for this suggestion and included the artifact review approach in the introduction to the criteria section. The aspects of repeatability, reproducibility, and replicability are aimed more at the results of computational research rather than research software itself, we feel. Therefore, we didn’t include specific criteria in the list suggested to be used when evaluating research software for sustained funding.<br/> While the SPEC submission process is very clear, we could not find any concrete criteria applied during the review (except for requirements regarding the license).</i><br/> <br/> The authors write “We also argue that truly sustainable research software must ideally be published under a Free/Libre Open Source Software (FLOSS) license, and follow an open development model…” what I fully support (see for instance https://doi.org/10.1515/itit-2019-00402). However, later under the section heading “Legal aspects” this requirement is thwarted. I fully agree that legal aspects have to be considered, but the general bias of this section seems to be on commercial licensing of research software. For instance, the decision tree in Figure 1 starts with the question “Licensing planned?”. I assume that commercial licensing is meant, but this is not clear since the figures are not explained in the paper. Instead, the process should start with open sourcing the software. If licenses such as Apache or MIT are applied, the research institutions may later still commercialize the software if appropriate. Such open source licensing is also beneficial for start-ups that intend to provide professional services for the software. <br/> My experience with technology transfer units of German universities and research institutes is that they do not understand the ideas of open source business models (see for instance https://doi.org/10.1109/MC.2019.28981633). Their focus is on patents and commercializing licenses, sometimes also on start-ups. Conversely, in the software industry, one major motivation for open sourcing software is on improving the quality of software. I cite from https://doi.org/10.1109/ICSAW.2017.114 : “the open-source approach has some psychological effects: Developers show a tendency to apply higher quality standards if they know that the code will be publicly available.” For sustainability, quality is an important property of software.<br/> <i>As further detailed below, we have moved the decision trees out of this manuscript as we see the problems and agree to your arguments. We like your suggested aspect of commercialization of FLOSS licensed software and included this aspect in the manuscript.</i><br/> <br/> The Figures 1-4 do more harm than good. They are daunting to researchers who intend to publish their code open source. These figures should be removed from the paper, they are useless without proper explanation.<br/> I suggest that the authors focus in the present paper on their main message (request for funding a German Software Sustainability Institute, which I fully support). Figures 1-4 could be moved to a separate paper, enriched with proper explanation.<br/> <i>We thank the reviewer for this suggestion. Our initial thought was to place these decision trees in the supplemental material but did not realize that this is not the policy of f1000. The editorial team moved them into the main article. This is the reason why the decision trees appeared without additional information in the manuscript. We have now decided to take the Figures out and have published them together with documentation templates in a separate report via Zenodo unde a Creative Commons license, see <a href="https://zenodo.org/record/4327148#.X9n6ui337OQ">https://zenodo.org/record/4327148#.X9n6ui337OQ</a>. This report is now cited in the F1000 manuscript.</i></span> <div class=padding-bottom id=full-comment-6209><i>Thank you for the thorough review and constructive feedback regarding our manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> Concerning the statement “In order to support research, a sustainable software must be correct”, I suggest to include a short discussion of the test oracle problem for scientific software (see for instance <a href="https://doi.org/10.1109/SECSE.2013.66150991">https://doi.org/10.1109/SECSE.2013.66150991</a>).<br/> <i>We thank the reviewer for this suggestion, and included a brief discussion of the test oracle problem as suggested, and additionally of further challenges to verification and validation, such as large configuration spaces and heterogeneous data (as discussed in e.g. <a href="https://doi.org/10.1109/SE4Science.2019.00010">https://doi.org/10.1109/SE4Science.2019.00010</a>), and have suggested to implement the solutions mentioned in the literature.</i><br/> <br/> Concerning the discussion of “The list of criteria presented in this section could be the basis for a structured review process…” I suggest to include two additional initiatives for software review. The first is artifact evaluation in computer science conferences (the process is explained in https://doi.org/10.1515/itit-2019-00402). The second is the SPEC Research Group’s review process of tools for quantitative system evaluation and analysis (<a href="https://research.spec.org/tools/submission.html">https://research.spec.org/tools/submission.html</a>).<br/> <i>We thank the reviewer for this suggestion and included the artifact review approach in the introduction to the criteria section. The aspects of repeatability, reproducibility, and replicability are aimed more at the results of computational research rather than research software itself, we feel. Therefore, we didn’t include specific criteria in the list suggested to be used when evaluating research software for sustained funding.<br/> While the SPEC submission process is very clear, we could not find any concrete criteria applied during the review (except for requirements regarding the license).</i><br/> <br/> The authors write “We also argue that truly sustainable research software must ideally be published under a Free/Libre Open Source Software (FLOSS) license, and follow an open development model…” what I fully support (see for instance https://doi.org/10.1515/itit-2019-00402). However, later under the section heading “Legal aspects” this requirement is thwarted. I fully agree that legal aspects have to be considered, but the general bias of this section seems to be on commercial licensing of research software. For instance, the decision tree in Figure 1 starts with the question “Licensing planned?”. I assume that commercial licensing is meant, but this is not clear since the figures are not explained in the paper. Instead, the process should start with open sourcing the software. If licenses such as Apache or MIT are applied, the research institutions may later still commercialize the software if appropriate. Such open source licensing is also beneficial for start-ups that intend to provide professional services for the software. <br/> My experience with technology transfer units of German universities and research institutes is that they do not understand the ideas of open source business models (see for instance https://doi.org/10.1109/MC.2019.28981633). Their focus is on patents and commercializing licenses, sometimes also on start-ups. Conversely, in the software industry, one major motivation for open sourcing software is on improving the quality of software. I cite from https://doi.org/10.1109/ICSAW.2017.114 : “the open-source approach has some psychological effects: Developers show a tendency to apply higher quality standards if they know that the code will be publicly available.” For sustainability, quality is an important property of software.<br/> <i>As further detailed below, we have moved the decision trees out of this manuscript as we see the problems and agree to your arguments. We like your suggested aspect of commercialization of FLOSS licensed software and included this aspect in the manuscript.</i><br/> <br/> The Figures 1-4 do more harm than good. They are daunting to researchers who intend to publish their code open source. These figures should be removed from the paper, they are useless without proper explanation.<br/> I suggest that the authors focus in the present paper on their main message (request for funding a German Software Sustainability Institute, which I fully support). Figures 1-4 could be moved to a separate paper, enriched with proper explanation.<br/> <i>We thank the reviewer for this suggestion. Our initial thought was to place these decision trees in the supplemental material but did not realize that this is not the policy of f1000. The editorial team moved them into the main article. This is the reason why the decision trees appeared without additional information in the manuscript. We have now decided to take the Figures out and have published them together with documentation templates in a separate report via Zenodo unde a Creative Commons license, see <a href="https://zenodo.org/record/4327148#.X9n6ui337OQ">https://zenodo.org/record/4327148#.X9n6ui337OQ</a>. This report is now cited in the F1000 manuscript.</i></div> <span class="disclosures competing-interests-display padding-top"> <strong>Competing Interests:</strong> <span class=c-i-d>No competing interests were disclosed.</span> </span> <span class="hide-comment-container response-and-comment">Close</span> </span> <div class="report-concern-wrapper last-comment"> <a href="mailto:research@f1000.com?subject=Report a concern (23224) " class=report-a-concern-link>Report a concern</a> </div> </div> </div> </li> <li class="comment-wrapper not-sortable button-wrapper"> <div class=comment-button-wrapper> <a href="/login?originalPath=/articles/9-295/v2&scrollTo=referee-response-62873" class=register-report-comment-button data-test-id=add-report-comment> <button class="primary orange extra-padding"> COMMENT ON THIS REPORT </button> </a> </div> </li> </ul> </div> </div> </div> </div> <div class="f1r-article-mobile research-layout"> <div class="mobile-sections-divider before-comments"></div> </div> <div id=article-comments class="article-comments padding-bottom-20"> <div class=current-article-comment-section> <h2 class=main-title name=add-new-comment id=add-new-comment> <span class="research-layout f1r-article-mobile-inline valign-middle"> <span class="f1r-icon icon-104_comments size30"></span> </span> <span class=f1r-article-desk-inline>Comments on this article</span> <span class=f1r-article-mobile-inline>Comments (3)</span> <span class="f1r-article-mobile-inline float-right"> <span class="f1r-icon icon-14_more_small"></span> <span class="f1r-icon icon-10_less_small"></span> </span> </h2> </div> <div class=comments-list-container> <div class="f1r-article-desk-inline referee-report-info-box referee-report-version-box"> Version 2 </div> <div class="f1r-article-mobile research-layout mobile-version-info padding-top-30"> <span class="mversion ">VERSION 2</span> <span class=details>PUBLISHED 26 Jan 2021</span> <span class="article-pubinfo-mobile versions-section"> <div title="#springMessage("NewVersionReasonType.REVISED.tooltip")" class="version-info-box revised ">Revised</div> </span> </div> <ul class=all-comment-wrapper data-reportID=NONE data-versionID=2> <li class="comment-wrapper not-sortable " data-commentID=6376> <div class="one-comment version-comments "> <a name=article-comment-6376></a> <div class="f1r-article-desk commenter-role"> Reader Comment 22 Feb 2021 </div> <div> <span class=name>Dirk Feuchter</span><span class=name>, </span><span class=commenter-affiliation>Karlsruhe Institute of Technology, Karlsruhe, Germany</span> </div> <div class="f1r-article-mobile the-comment-date"> 22 Feb 2021 </div> <div class=f1r-article-mobile> <div class="article-pubinfo-mobile comments-section"> <span class="version-info-box comments"> Reader Comment </span> </div> </div> <div class="comment-content article-comment-list"> <span class=truncated-comment> Dear Mr. Anzt, dear Mr. Loewe, dear Mr. Bach, dear Mr. Seemann, as well as dear other authors and reviewers,<br/> <br/> just discovered your feedback to my Comment. Thank you ... <span class="continue-reading-container response-and-comment">Continue reading</span> </span> <span class="full-comment hidden"> <span id=full-comment-original-6376 class=hidden>Dear Mr. Anzt, dear Mr. Loewe, dear Mr. Bach, dear Mr. Seemann, as well as dear other authors and reviewers,<br/> <br/> just discovered your feedback to my Comment. Thank you very much for that.<br/> <br/> <i>I can well understand that your community has in mind proprietary licensing at most in exceptional situations.</i><br/> <br/> <i>Please note and obviously you keep this already in mind, that even in the case of pure F(L)OSS licensing, i.e. completely without proprietary licensing intentions the topic of "F(L)OSS-right compliance" should be considered especially in the sense of sustainability of the software.</i><br/> <br/> <i>Hence,and only as a suggestion, the „OpenChain movement“ and the new ISO 5230 standard resulting from it at the end of 2020 should possibly be mentioned and cited in your paper; </i>possibly in „Legal Aspects -> Challenges and clarifications“. after your sentence „…The challenge in the use, distribution, and commercialization of software is to determine the chain of rights and to identify all right holders….“<br/> <br/> <i>Best regards</i><br/> <br/> <i>Dirk Feuchter</i></span> <div class=padding-bottom id=full-comment-6376>Dear Mr. Anzt, dear Mr. Loewe, dear Mr. Bach, dear Mr. Seemann, as well as dear other authors and reviewers,<br/> <br/> just discovered your feedback to my Comment. Thank you very much for that.<br/> <br/> <i>I can well understand that your community has in mind proprietary licensing at most in exceptional situations.</i><br/> <br/> <i>Please note and obviously you keep this already in mind, that even in the case of pure F(L)OSS licensing, i.e. completely without proprietary licensing intentions the topic of "F(L)OSS-right compliance" should be considered especially in the sense of sustainability of the software.</i><br/> <br/> <i>Hence,and only as a suggestion, the „OpenChain movement“ and the new ISO 5230 standard resulting from it at the end of 2020 should possibly be mentioned and cited in your paper; </i>possibly in „Legal Aspects -> Challenges and clarifications“. after your sentence „…The challenge in the use, distribution, and commercialization of software is to determine the chain of rights and to identify all right holders….“<br/> <br/> <i>Best regards</i><br/> <br/> <i>Dirk Feuchter</i></div> <span class="disclosures competing-interests-display padding-top"> <strong>Competing Interests:</strong> <span class=c-i-d>No competing interests were disclosed.</span> </span> <span class="hide-comment-container response-and-comment">Close</span> </span> <div class="report-concern-wrapper last-comment"> <a href="mailto:research@f1000.com?subject=Report a concern (23224) " class=report-a-concern-link>Report a concern</a> </div> </div> </div> </li> <li class="comment-wrapper not-sortable button-wrapper"> <div class="one-comment version-comments add-the-comment"> <a href="/login?originalPath=/articles/9-295/v2&scrollTo=add-new-comments" class=register-report-comment-button data-test-id=add-comment> <span class=f1r-article-desk><span class=contracted></span>Comment</span> <span class="f1r-article-mobile research-layout "> <button class="primary orange extra-padding comment-on-this-article">ADD YOUR COMMENT</button> </span> </a> </div> </li> </ul> <div class="f1r-article-mobile research-layout"> <div class="mobile-sections-divider versions-divider"></div> </div> <div class="f1r-article-desk-inline referee-report-info-box referee-report-version-box"> Version 1 </div> <div class="f1r-article-mobile research-layout mobile-version-info padding-top-30"> <span class="mversion previous-version">VERSION 1</span> <span class=details>PUBLISHED 27 Apr 2020</span> <span class="article-pubinfo-mobile versions-section"> </span> </div> <div class="comments-closed f1r-article-mobile"> Discussion is closed on this version, please comment on the latest version above. </div> <ul class=all-comment-wrapper data-reportID=NONE data-versionID=1> <li class="comment-wrapper not-sortable " data-commentID=6211> <div class="one-comment version-comments "> <a name=article-comment-6211></a> <div class="f1r-article-desk commenter-role"> Author Response 26 Jan 2021 </div> <div> <span class=name>Axel Loewe</span><span class=name>, </span><span class=commenter-affiliation>Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany</span> </div> <div class="f1r-article-mobile the-comment-date"> 26 Jan 2021 </div> <div class=f1r-article-mobile> <div class="article-pubinfo-mobile comments-section"> <span class="version-info-box comments"> Author Response </span> </div> </div> <div class="comment-content article-comment-list"> <span class=truncated-comment> <i>We thank you for your thorough review and constructive feedback regarding the manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> Dear ... <span class="continue-reading-container response-and-comment">Continue reading</span> </span> <span class="full-comment hidden"> <span id=full-comment-original-6211 class=hidden><i>We thank you for your thorough review and constructive feedback regarding the manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> Dear Mr. Anzt, dear Mr. Loewe, dear Mr. Bach, dear Mr. Seemann, dear Elke, and dear Sven, as well as dear authors as yet unknown to me,<br/> I am working at KIT in the field of Innovation and Relations Management, especially Licensing of Intellectual Property Rights from KIT to Free Maket Economy. My particual focus is to Out-license Computerprograms to Third Parties and Industry.<br/> Thank you for your great efforts with your extensive and intersting article including the nicely written and ever-recurring story of Kim.<br/> Your FLOSS-based approach for sustainable software devlopment is holding immense savings potential. That’s great.<br/> Following are my comments to your article from the perspective of a TTO license manager, typically supporting RSEs in cases of proprietary licensing e.g. to spin-offs or industrial companies.<br/> Hence, most of my comments might go in a slightly different direction than the main focus of your article, but in my point of view these comments are complementary. Hence, I wonder what you think of them. Looking forward to your opinion and you feedback to my comments, if you like also gladly by telephone<br/> Best regards<br/> Dirk Feuchter<br/> <i>Thank you very much for your comments regarding distribution issues, IP compliance and commercialization. We have generally subsumed these issues under the more general term “availability”. We believe that this implies that software is legally available only if it is licensed, either under an open source license or proprietarily. This impacts possible modes of distribution, but does not concern actual (commercial) distribution processes. As a community of Research Software Engineers, we strongly believe that publicly funded research software should be F(L)OSS-licensed per default, although we recognize that this may not always be possible (and hence have weakened “must” to “should” as you suggested in the respective paragraph under “Why sustainable research software in the first place”). We have further avoided any changes to the text that would weaken this point, as we see no central obstacles to making publicly funded research software open source in general. Contrarily, we do not accept that IP should override public interest (both intellectually and fiscally) here, some corner cases excluded. Concurrently, we purposefully do not focus on marketability and commercialization of research software. Instead, we see commercial opportunities, e.g., in the provision of services for a research software product, whereas the product itself should remain free and open source.<br/> Thank you, also, for notifying us of some errors in the text itself and the figures, which we are fixing in the next version.<br/> Below, we address some of your concrete suggestions in more detail:</i><br/> <br/> Abstract:<br/> I would like to suggest to extend<br/> „Research software must be sustainable in order to<br/> understand, replicate, reproduce, and…“<br/> as follows<br/> „Research software must be sustainable in order to<br/> understand, replicate, reproduce, distribute and…“ <br/> <i>We thank you for the comment. The development of research software does not focus on distribution of software results, which is more of a business aspect. The availability, regardless of the actual distribution, is noted in the next sentence of the abstract.</i><br/> <br/> Abstract:<br/> I would like to suggest to extend<br/> „In other words, software must be available, discoverable, usable, and adaptable to new needs, both now and in the future.“<br/> as follows<br/> „In other words, software must be available, (IP-/FLOSS)compliant*, discoverable, usable, and adaptable to new needs, both now and in the future.“<br/> *By „(IP-/FLOSS)compliant“ I mean<br/> in compliance<br/> with intellectual property of third-party suppliers,<br/> with the terms of free/libre open source licenses and with the aim to protect own intellectual property from unintended disclosure <br/> <i>We thank you for the comment. As a community of RSEs we are aiming for FLOSS whenever feasible but acknowledge that there are scenarios in which non-FLOSS licenses need to be considered.</i><br/> <br/> Suggestion of a new penultimate paragraph in subchapter „Stakeholder motivations for research software sustainability“ as follows: RSEs, Research leaders and research performing organisations are interested in software sustainability also in the sense that their (research)software is sustainable concerning legal compliance.This is an important issue distributing (research)software for both acaedmic puposes as well as commerical purposes. For the latter RSEs and their research leaders (typically contacting their TTO) are interested to marktet (parts of) their research software and/or additional connectable closed software (which they prevent from unintended disclosure) to a spin-off or an industrial company using a proprietary or a dual licensing model. For both, acaedmic and commerical purposes RSEs and their research leaders and their research performing organisation are interested that their (research) software plus any connectable closed software is compliant with intellectual property of third-party suppliers and compliant with the terms of free/libre open source licenses. An important stakeholder motivation is therefore „software sustainbility with the aim of clarification of all software rights ownerships“. <br/> <i>We thank you for the suggestion. While we aim for reusable software in terms of licenses we as a community do not focus on commercial purposes.</i><br/> <br/> Please replace Subchapter heading„Challenges and clarifications Clarification of rights“by „Challenges and Clarification of rights“ <br/> <i>We thank you for pointing this out and corrected the structure (section and subsection headings, respectively).</i><br/> <br/> The term "research institution" appears in the glossary as a duplicate. Therefore, "research institution" should be deleted in the brackets.<br/> <i>Thank you, we have replaced it by “national labs”.</i></span> <div class=padding-bottom id=full-comment-6211><i>We thank you for your thorough review and constructive feedback regarding the manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> Dear Mr. Anzt, dear Mr. Loewe, dear Mr. Bach, dear Mr. Seemann, dear Elke, and dear Sven, as well as dear authors as yet unknown to me,<br/> I am working at KIT in the field of Innovation and Relations Management, especially Licensing of Intellectual Property Rights from KIT to Free Maket Economy. My particual focus is to Out-license Computerprograms to Third Parties and Industry.<br/> Thank you for your great efforts with your extensive and intersting article including the nicely written and ever-recurring story of Kim.<br/> Your FLOSS-based approach for sustainable software devlopment is holding immense savings potential. That’s great.<br/> Following are my comments to your article from the perspective of a TTO license manager, typically supporting RSEs in cases of proprietary licensing e.g. to spin-offs or industrial companies.<br/> Hence, most of my comments might go in a slightly different direction than the main focus of your article, but in my point of view these comments are complementary. Hence, I wonder what you think of them. Looking forward to your opinion and you feedback to my comments, if you like also gladly by telephone<br/> Best regards<br/> Dirk Feuchter<br/> <i>Thank you very much for your comments regarding distribution issues, IP compliance and commercialization. We have generally subsumed these issues under the more general term “availability”. We believe that this implies that software is legally available only if it is licensed, either under an open source license or proprietarily. This impacts possible modes of distribution, but does not concern actual (commercial) distribution processes. As a community of Research Software Engineers, we strongly believe that publicly funded research software should be F(L)OSS-licensed per default, although we recognize that this may not always be possible (and hence have weakened “must” to “should” as you suggested in the respective paragraph under “Why sustainable research software in the first place”). We have further avoided any changes to the text that would weaken this point, as we see no central obstacles to making publicly funded research software open source in general. Contrarily, we do not accept that IP should override public interest (both intellectually and fiscally) here, some corner cases excluded. Concurrently, we purposefully do not focus on marketability and commercialization of research software. Instead, we see commercial opportunities, e.g., in the provision of services for a research software product, whereas the product itself should remain free and open source.<br/> Thank you, also, for notifying us of some errors in the text itself and the figures, which we are fixing in the next version.<br/> Below, we address some of your concrete suggestions in more detail:</i><br/> <br/> Abstract:<br/> I would like to suggest to extend<br/> „Research software must be sustainable in order to<br/> understand, replicate, reproduce, and…“<br/> as follows<br/> „Research software must be sustainable in order to<br/> understand, replicate, reproduce, distribute and…“ <br/> <i>We thank you for the comment. The development of research software does not focus on distribution of software results, which is more of a business aspect. The availability, regardless of the actual distribution, is noted in the next sentence of the abstract.</i><br/> <br/> Abstract:<br/> I would like to suggest to extend<br/> „In other words, software must be available, discoverable, usable, and adaptable to new needs, both now and in the future.“<br/> as follows<br/> „In other words, software must be available, (IP-/FLOSS)compliant*, discoverable, usable, and adaptable to new needs, both now and in the future.“<br/> *By „(IP-/FLOSS)compliant“ I mean<br/> in compliance<br/> with intellectual property of third-party suppliers,<br/> with the terms of free/libre open source licenses and with the aim to protect own intellectual property from unintended disclosure <br/> <i>We thank you for the comment. As a community of RSEs we are aiming for FLOSS whenever feasible but acknowledge that there are scenarios in which non-FLOSS licenses need to be considered.</i><br/> <br/> Suggestion of a new penultimate paragraph in subchapter „Stakeholder motivations for research software sustainability“ as follows: RSEs, Research leaders and research performing organisations are interested in software sustainability also in the sense that their (research)software is sustainable concerning legal compliance.This is an important issue distributing (research)software for both acaedmic puposes as well as commerical purposes. For the latter RSEs and their research leaders (typically contacting their TTO) are interested to marktet (parts of) their research software and/or additional connectable closed software (which they prevent from unintended disclosure) to a spin-off or an industrial company using a proprietary or a dual licensing model. For both, acaedmic and commerical purposes RSEs and their research leaders and their research performing organisation are interested that their (research) software plus any connectable closed software is compliant with intellectual property of third-party suppliers and compliant with the terms of free/libre open source licenses. An important stakeholder motivation is therefore „software sustainbility with the aim of clarification of all software rights ownerships“. <br/> <i>We thank you for the suggestion. While we aim for reusable software in terms of licenses we as a community do not focus on commercial purposes.</i><br/> <br/> Please replace Subchapter heading„Challenges and clarifications Clarification of rights“by „Challenges and Clarification of rights“ <br/> <i>We thank you for pointing this out and corrected the structure (section and subsection headings, respectively).</i><br/> <br/> The term "research institution" appears in the glossary as a duplicate. Therefore, "research institution" should be deleted in the brackets.<br/> <i>Thank you, we have replaced it by “national labs”.</i></div> <span class="disclosures competing-interests-display padding-top"> <strong>Competing Interests:</strong> <span class=c-i-d>No competing interests were disclosed.</span> </span> <span class="hide-comment-container response-and-comment">Close</span> </span> <div class="report-concern-wrapper last-comment"> <a href="mailto:research@f1000.com?subject=Report a concern (23224) " class=report-a-concern-link>Report a concern</a> </div> </div> </div> </li> <li class="comment-wrapper not-sortable " data-commentID=5522> <div class="one-comment version-comments "> <a name=article-comment-5522></a> <div class="f1r-article-desk commenter-role"> Reader Comment 19 May 2020 </div> <div> <span class=name>Dirk Feuchter</span><span class=name>, </span><span class=commenter-affiliation>Karlsruhe Institute of Technology (KIT), Eggenstein-Leopoldshafen and Karlsruhe, Germany</span> </div> <div class="f1r-article-mobile the-comment-date"> 19 May 2020 </div> <div class=f1r-article-mobile> <div class="article-pubinfo-mobile comments-section"> <span class="version-info-box comments"> Reader Comment </span> </div> </div> <div class="comment-content article-comment-list"> <span class=truncated-comment> Subject: Feedback to your article „An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action…“.<br/> <br/>Dear Mr. Anzt, dear Mr. Loewe, dear Mr. ... <span class="continue-reading-container response-and-comment">Continue reading</span> </span> <span class="full-comment hidden"> <span id=full-comment-original-5522 class=hidden>Subject: Feedback to your article „An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action…“.<br/> <br/>Dear Mr. Anzt, dear Mr. Loewe, dear Mr. Bach, dear Mr. Seemann, dear Elke, and dear Sven, as well as dear authors as yet unknown to me,<br/>I am working at KIT in the field of Innovation and Relations Management, especially Licensing of Intellectual Property Rights from KIT to Free Maket Economy. My particual focus is to Out-license Computerprograms to Third Parties and Industry.<br/>Thank you for your great efforts with your extensive and intersting article including the nicely written and ever-recurring story of Kim.<br/>Your FLOSS-based approach for sustainable software devlopment is holding immense savings potential. That’s great.<br/>Following are my comments to your article from the perspective of a TTO license manager, typically supporting RSEs in cases of proprietary licensing e.g. to spin-offs or industrial companies.<br/>Hence, most of my comments might go in a slightly different direction than the main focus of your article, but in my point of view these comments are complementary. Hence, I wonder what you think of them. Looking forward to your opinion and you feedback to my comments, if you like also gladly by telephone<br/>Best regards<br/>Dirk Feuchter<br/> <br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-01:<br/>Abstract:<br/>I would like to suggest to extend<br/>„Research software must be sustainable in order to<br/>understand, replicate, reproduce, and…“<br/>as follows<br/>„Research software must be sustainable in order to<br/>understand, replicate, reproduce, <b>distribute</b> and…“ <br/> <br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-02:<br/>Abstract:<br/>I would like to suggest to extend<br/>„In other words, software must be available, discoverable, usable, and adaptable to new needs, both now and in the future.“<br/> <br/>as follows<br/> <br/>„In other words, software must be available, (IP-/FLOSS)<b>compliant*</b>, discoverable, usable, and adaptable to new needs, both now and in the future.“<br/> <br/>*By „(IP-/FLOSS)<b>compliant“</b><i> I mean</i><br/><i>in </i>compliance <ul> <li>with intellectual property of third-party suppliers,</li> <li>with the terms of free/libre open source licenses and with the aim to protect own intellectual property from unintended disclosure </li> </ul> # # # # # # # # # # # # # # # # # # # # # # #Comment-DF-03:Why sustainable research software in the first place =>I would like to suggest to extend„In order to support research, a sustainable software must be correct<a href="https://f1000research.com/articles/9-295/v1#ref-14"><sup>14</sup></a>, validatable, understandable, documented, publicly released,...“ as follows „In order to support research, a sustainable software must be correct<a href="https://f1000research.com/articles/9-295/v1#ref-14"><sup>14</sup></a>, (IP-/FLOSS)<b>compliant* </b>validatable, understandable, documented, publicly released,...“ *Concerning „(IP-/FLOSS)<b>compliant“</b> please see upon # # # # # # # # # # # # # # # # # # # # # # # <ul> <li>Comment-DF-04:Why sustainable research software in the first place =>„We also argue that truly sustainable research software must ideallybe published under a Free/Libre Open Source Software (FLOSS) license, and…“„We also argue that truly sustainable research software <b>should typically</b> be published under a Free/Libre Open Source Software (FLOSS) license, and…“</li> </ul> # # # # # # # # # # # # # # # # # # # # # # #<br/><br/>Comment-DF-05:Why sustainable research software in the first place =>„Therefore, if research software is publicly funded, it should be freely available under a FLOSS license.„Therefore, if research software is publicly funded, it should be <b>normally</b> freely available under a FLOSS license.<br/># # # # # # # # # # # # # # # # # # # # # # #<br/><br/>Comment-DF-06:Why sustainable research software in the first place =>I would like to suggest to extend„Legal issues: Many obstacles for research software pertain to legal issues, such as applicable licensing and compatibility of licenses<a href="https://f1000research.com/articles/9-295/v1#ref-45"><sup>45</sup></a>, and decisions about license types. as follows „Legal issues: Many obstacles for research software pertain to legal issues, such as <b>IT law, copyright law, copyright notices and author attributions,</b> applicable licensing and compatibility of licenses<a href="https://f1000research.com/articles/9-295/v1#ref-45"><sup>45</sup></a>, and decisions about license types.<br/><br/> # # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-07:Why sustainable research software in the first place =>„A subset of this <b>group</b> may be interested in …“.Which „group“ do you mean? => the „Taxpayers“? # # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-08:Suggestion of a new penultimate paragraph in subchapter „Stakeholder motivations for research software sustainability“ as follows: RSEs, Research leaders and research performing organisations are interested in software sustainability also in the sense that their (research)software is sustainable concerning legal compliance.This is an important issue distributing (research)software for both acaedmic puposes as well as commerical purposes. For the latter RSEs and their research leaders (typically contacting their TTO) are interested to marktet (parts of) their research software and/or additional connectable closed software (which they prevent from unintended disclosure) to a spin-off or an industrial company using a proprietary or a dual licensing model. For both, acaedmic and commerical purposes RSEs and their research leaders and their research performing organisation are interested that their (research) software plus any connectable closed software is compliant with intellectual property of third-party suppliers and compliant with the terms of free/libre open source licenses. An important stakeholder motivation is therefore „software sustainbility with the aim of clarification of all software rights ownerships“. <br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-09:Abstract:I would like to suggest to extend„Failing to do so will threaten the quality and validity of research..“ as follows „Failing to do so will threaten the quality, Xi and validity of research..“ // X1= marketability or X2= distribution // As TTO-license-manager I personally would prefer X1 but X2 is fine as well <br/><br/># # # # # # # # # # # # # # # # # # # # # # #Comment-DF-10:In the heading such as „Why sustainable research software in the first place?“, one might prefix a chapter number, just as announced at the end of the introductory background: „This all leads to unmet requirements and unsolved challenges that we want to highlight in this paper by elaborating on (1)…(2)…(3)…(4)…(5)…(6)…“Hence, instead of„Why sustainable research software in the first place?“use„(1) Why sustainable research software in the first place?“and so on:(2) How to decide which software to sustain?(3) Who sustains research software?(4) How can research software be sustainably funded?(5) Which infrastructure is needed to sustain research software?(6) Legal aspects.That's a matter of taste, of course. <br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/><br/>Comment-DF-11:Please replace Subchapter heading„Challenges and clarifications Clarification of rights“by „Challenges and Clarification of rights“ <br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-12:In both Fig. 1 and „Legal aspects/Challenges and clarifications of rights“instead of „Subject to directives“ {weisungsgebunden dt.} from UrhG 69 bI propose to write „in execution of his duties {in Wahrnehmung seiner Aufgaben dt.}“ from UrhG 69b, in no way to constrain scientists, RSEs and Research leaders, but in order to free up the scope for decision-making and thus open up opportunities.[Regardless of that, I would translate the German "weisungsgebunden" from „UrhG 69 b“ with "bound by instructions"(short) or with "following the instructions given by his employer"(long)]<br/><br/> # # # # # # # # # # # # # # # # # # # # # # # <ul> <li>Comment-DF-13:In Subchapter „Conclusions“I would not write the following sentence, or at most I would write it differently.„We encourage the research funding bodies to reflect the licensing models for academic software development, and to decide whether the “public money, public code” paradigm justifies the requirement that all publicly funded software has to be publicly available under a Free/Libre Open Source Software (FLOSS) license.“ I can understand your point of view and this sentence. But isn't there a lack of perspective regarding those research leaders or RSE-teams, who are considering a foundation based on a proprietary license model regarding their software development or RSE teams whose software developments are taken over and marketed by industrial companies in or outside of Germany, without return to the research performing organizations and the corresponding RSE teams.Hence, an alternative formulation might be as follows: „We encourage the research funding bodies to reflect the licensing models for academic software development, and to suggest research performing organisations and their research leaders in the sense of sustainability to make their software typically publicly available under a FLOSS-license but also to take into consideration revenue-oriented approaches such as FLOSS business models or proprietrary licensing if applicable.“ </li> </ul> In my opinion, the final decision <b>to license in and out as well as to transfer computer programs</b> under FLOSS licenses should be the responsibility of the (authorised for this purpose by the Presidium/Board of Directors of the research performing organisation) Research Leaders, with whom the computerprogram developing RSEs should therefore consult. Furthermore, the RSEs and their research leaders ideally should have the opportunity at their research institution to contact (preferably at an early stage) science-supporting specialist departments such as Legal, TTO, research software local task forces or even a SSI-like nationwide institution and seek advice.<br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-14:In Fig.3 it should read "Check (1)..." instead of "Check (2)..." in both the centre left and the top right.<br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-15: The term "research institution" appears in the glossary as a duplicate. Therefore, "research institution" should be deleted in the brackets.<br/># # # # # # # # # # # # # # # # # # # # # # # <ul> <li>Best regardsDirk Feuchter------------------------------------------------------------------Karlsruhe Institute of Technology (KIT)INNOVATION AND RELATIONS MANAGEMENT (IRM)Intellectual Property ManagementDr. Dirk Feuchter (Licenses)Hermann-von-Helmholtz-Platz 1D-76344 Eggenstein LeopoldshafenGermanyPhone: +49 721 608-2-3921E-Mail: dirk.feuchter at kit.eduWeb: <a href="https://www.irm.kit.edu/116_1500.php">https://www.irm.kit.edu/116_1500.php</a> <a href="https://www.irm.kit.edu/english/91.php">https://www.irm.kit.edu/english/91.php</a> <a href="http://www.irm.kit.edu/">www.irm.kit.edu/</a> </li> </ul></span> <div class=padding-bottom id=full-comment-5522>Subject: Feedback to your article „An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action…“.<br/> <br/>Dear Mr. Anzt, dear Mr. Loewe, dear Mr. Bach, dear Mr. Seemann, dear Elke, and dear Sven, as well as dear authors as yet unknown to me,<br/>I am working at KIT in the field of Innovation and Relations Management, especially Licensing of Intellectual Property Rights from KIT to Free Maket Economy. My particual focus is to Out-license Computerprograms to Third Parties and Industry.<br/>Thank you for your great efforts with your extensive and intersting article including the nicely written and ever-recurring story of Kim.<br/>Your FLOSS-based approach for sustainable software devlopment is holding immense savings potential. That’s great.<br/>Following are my comments to your article from the perspective of a TTO license manager, typically supporting RSEs in cases of proprietary licensing e.g. to spin-offs or industrial companies.<br/>Hence, most of my comments might go in a slightly different direction than the main focus of your article, but in my point of view these comments are complementary. Hence, I wonder what you think of them. Looking forward to your opinion and you feedback to my comments, if you like also gladly by telephone<br/>Best regards<br/>Dirk Feuchter<br/> <br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-01:<br/>Abstract:<br/>I would like to suggest to extend<br/>„Research software must be sustainable in order to<br/>understand, replicate, reproduce, and…“<br/>as follows<br/>„Research software must be sustainable in order to<br/>understand, replicate, reproduce, <b>distribute</b> and…“ <br/> <br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-02:<br/>Abstract:<br/>I would like to suggest to extend<br/>„In other words, software must be available, discoverable, usable, and adaptable to new needs, both now and in the future.“<br/> <br/>as follows<br/> <br/>„In other words, software must be available, (IP-/FLOSS)<b>compliant*</b>, discoverable, usable, and adaptable to new needs, both now and in the future.“<br/> <br/>*By „(IP-/FLOSS)<b>compliant“</b><i> I mean</i><br/><i>in </i>compliance <ul> <li>with intellectual property of third-party suppliers,</li> <li>with the terms of free/libre open source licenses and with the aim to protect own intellectual property from unintended disclosure </li> </ul> # # # # # # # # # # # # # # # # # # # # # # #Comment-DF-03:Why sustainable research software in the first place =>I would like to suggest to extend„In order to support research, a sustainable software must be correct<a href="https://f1000research.com/articles/9-295/v1#ref-14"><sup>14</sup></a>, validatable, understandable, documented, publicly released,...“ as follows „In order to support research, a sustainable software must be correct<a href="https://f1000research.com/articles/9-295/v1#ref-14"><sup>14</sup></a>, (IP-/FLOSS)<b>compliant* </b>validatable, understandable, documented, publicly released,...“ *Concerning „(IP-/FLOSS)<b>compliant“</b> please see upon # # # # # # # # # # # # # # # # # # # # # # # <ul> <li>Comment-DF-04:Why sustainable research software in the first place =>„We also argue that truly sustainable research software must ideallybe published under a Free/Libre Open Source Software (FLOSS) license, and…“„We also argue that truly sustainable research software <b>should typically</b> be published under a Free/Libre Open Source Software (FLOSS) license, and…“</li> </ul> # # # # # # # # # # # # # # # # # # # # # # #<br/><br/>Comment-DF-05:Why sustainable research software in the first place =>„Therefore, if research software is publicly funded, it should be freely available under a FLOSS license.„Therefore, if research software is publicly funded, it should be <b>normally</b> freely available under a FLOSS license.<br/># # # # # # # # # # # # # # # # # # # # # # #<br/><br/>Comment-DF-06:Why sustainable research software in the first place =>I would like to suggest to extend„Legal issues: Many obstacles for research software pertain to legal issues, such as applicable licensing and compatibility of licenses<a href="https://f1000research.com/articles/9-295/v1#ref-45"><sup>45</sup></a>, and decisions about license types. as follows „Legal issues: Many obstacles for research software pertain to legal issues, such as <b>IT law, copyright law, copyright notices and author attributions,</b> applicable licensing and compatibility of licenses<a href="https://f1000research.com/articles/9-295/v1#ref-45"><sup>45</sup></a>, and decisions about license types.<br/><br/> # # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-07:Why sustainable research software in the first place =>„A subset of this <b>group</b> may be interested in …“.Which „group“ do you mean? => the „Taxpayers“? # # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-08:Suggestion of a new penultimate paragraph in subchapter „Stakeholder motivations for research software sustainability“ as follows: RSEs, Research leaders and research performing organisations are interested in software sustainability also in the sense that their (research)software is sustainable concerning legal compliance.This is an important issue distributing (research)software for both acaedmic puposes as well as commerical purposes. For the latter RSEs and their research leaders (typically contacting their TTO) are interested to marktet (parts of) their research software and/or additional connectable closed software (which they prevent from unintended disclosure) to a spin-off or an industrial company using a proprietary or a dual licensing model. For both, acaedmic and commerical purposes RSEs and their research leaders and their research performing organisation are interested that their (research) software plus any connectable closed software is compliant with intellectual property of third-party suppliers and compliant with the terms of free/libre open source licenses. An important stakeholder motivation is therefore „software sustainbility with the aim of clarification of all software rights ownerships“. <br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-09:Abstract:I would like to suggest to extend„Failing to do so will threaten the quality and validity of research..“ as follows „Failing to do so will threaten the quality, Xi and validity of research..“ // X1= marketability or X2= distribution // As TTO-license-manager I personally would prefer X1 but X2 is fine as well <br/><br/># # # # # # # # # # # # # # # # # # # # # # #Comment-DF-10:In the heading such as „Why sustainable research software in the first place?“, one might prefix a chapter number, just as announced at the end of the introductory background: „This all leads to unmet requirements and unsolved challenges that we want to highlight in this paper by elaborating on (1)…(2)…(3)…(4)…(5)…(6)…“Hence, instead of„Why sustainable research software in the first place?“use„(1) Why sustainable research software in the first place?“and so on:(2) How to decide which software to sustain?(3) Who sustains research software?(4) How can research software be sustainably funded?(5) Which infrastructure is needed to sustain research software?(6) Legal aspects.That's a matter of taste, of course. <br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/><br/>Comment-DF-11:Please replace Subchapter heading„Challenges and clarifications Clarification of rights“by „Challenges and Clarification of rights“ <br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-12:In both Fig. 1 and „Legal aspects/Challenges and clarifications of rights“instead of „Subject to directives“ {weisungsgebunden dt.} from UrhG 69 bI propose to write „in execution of his duties {in Wahrnehmung seiner Aufgaben dt.}“ from UrhG 69b, in no way to constrain scientists, RSEs and Research leaders, but in order to free up the scope for decision-making and thus open up opportunities.[Regardless of that, I would translate the German "weisungsgebunden" from „UrhG 69 b“ with "bound by instructions"(short) or with "following the instructions given by his employer"(long)]<br/><br/> # # # # # # # # # # # # # # # # # # # # # # # <ul> <li>Comment-DF-13:In Subchapter „Conclusions“I would not write the following sentence, or at most I would write it differently.„We encourage the research funding bodies to reflect the licensing models for academic software development, and to decide whether the “public money, public code” paradigm justifies the requirement that all publicly funded software has to be publicly available under a Free/Libre Open Source Software (FLOSS) license.“ I can understand your point of view and this sentence. But isn't there a lack of perspective regarding those research leaders or RSE-teams, who are considering a foundation based on a proprietary license model regarding their software development or RSE teams whose software developments are taken over and marketed by industrial companies in or outside of Germany, without return to the research performing organizations and the corresponding RSE teams.Hence, an alternative formulation might be as follows: „We encourage the research funding bodies to reflect the licensing models for academic software development, and to suggest research performing organisations and their research leaders in the sense of sustainability to make their software typically publicly available under a FLOSS-license but also to take into consideration revenue-oriented approaches such as FLOSS business models or proprietrary licensing if applicable.“ </li> </ul> In my opinion, the final decision <b>to license in and out as well as to transfer computer programs</b> under FLOSS licenses should be the responsibility of the (authorised for this purpose by the Presidium/Board of Directors of the research performing organisation) Research Leaders, with whom the computerprogram developing RSEs should therefore consult. Furthermore, the RSEs and their research leaders ideally should have the opportunity at their research institution to contact (preferably at an early stage) science-supporting specialist departments such as Legal, TTO, research software local task forces or even a SSI-like nationwide institution and seek advice.<br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-14:In Fig.3 it should read "Check (1)..." instead of "Check (2)..." in both the centre left and the top right.<br/><br/># # # # # # # # # # # # # # # # # # # # # # #<br/>Comment-DF-15: The term "research institution" appears in the glossary as a duplicate. Therefore, "research institution" should be deleted in the brackets.<br/># # # # # # # # # # # # # # # # # # # # # # # <ul> <li>Best regardsDirk Feuchter------------------------------------------------------------------Karlsruhe Institute of Technology (KIT)INNOVATION AND RELATIONS MANAGEMENT (IRM)Intellectual Property ManagementDr. Dirk Feuchter (Licenses)Hermann-von-Helmholtz-Platz 1D-76344 Eggenstein LeopoldshafenGermanyPhone: +49 721 608-2-3921E-Mail: dirk.feuchter at kit.eduWeb: <a href="https://www.irm.kit.edu/116_1500.php">https://www.irm.kit.edu/116_1500.php</a> <a href="https://www.irm.kit.edu/english/91.php">https://www.irm.kit.edu/english/91.php</a> <a href="http://www.irm.kit.edu/">www.irm.kit.edu/</a> </li> </ul></div> <span class="disclosures competing-interests-display padding-top"> <strong>Competing Interests:</strong> <span class=c-i-d>No competing interests were disclosed.</span> </span> <span class="hide-comment-container response-and-comment">Close</span> </span> <div class="report-concern-wrapper last-comment"> <a href="mailto:research@f1000.com?subject=Report a concern (23224) " class=report-a-concern-link>Report a concern</a> </div> </div> </div> </li> <li class="comment-wrapper not-sortable f1r-article-desk"> <div class="bold padding-top padding-left-20"> Discussion is closed on this version, please comment on the latest version above. </div> </li> </ul> </div> </div> </div> </div> <div id=article_main-column class="p-article__sidebar o-layout__item u-1/3 not-expanded js-article-sidebar"> <div class="o-tab p-article__column-toggle-container"> <button class="c-tab c-tab--left js-column-toggle p-article__column-toggle not-expanded " type=button data-target-main=article_main-column data-target-secondary=article_secondary-column><i class="c-tab__icon material-icons u-hide@expanded">keyboard_arrow_left</i><i class="c-tab__icon material-icons u-show@expanded">keyboard_arrow_right</i></button> </div> <div class=p-article__sidebar-content> <div class="p-article__sidebar-scroller js-article-sidebar-scroller"> <section class="p-article__sidebar-view js-article-sidebar-view js-article-sidebar-main u-pt u-pb--8" data-view=peer-review> <div class="o-layout o-layout--flush"> <div class="o-layout__item u-pl"> <h3 class="u-mt--0 u-mb--2 t-h3 u-weight--md u-pl" data-test-id=article_sidebar_heading>Open Peer Review</h3> </div> <div class=o-layout__item> <section class=""> <div class="p-article__sidebar-highlight u-mb--2 u-pr--1"> <div class="o-actions o-actions--middle"> <div class=o-actions__primary> <h4 class="u-mt--0 u-mb--0 u-ib u-middle t-h4 u-weight--md u-mr--1/2">Reviewer Status</h4> <div class="c-referee-status__icons u-ib u-middle"> <i class="c-icn--f1r-icon c-icn--approved " title=Approved #data-refInfo=66679-78211></i> <i class="c-icn--f1r-icon c-icn--approved " title=Approved #data-refInfo=66678-62872></i> </div> </div> <div class="o-actions__secondary _mdl-layout"> <div class="c-block-tip p-article__sidebar-tooltip c-block-tip--below c-block-tip--small-arrow c-block-tip--sm-padding"> <button type=button class="c-button c-button--icon c-button--text c-block-tip__toggle"><i class="material-icons c-button--icon__icon">info_outline</i></button> <div class=c-block-tip__content> <p class="t-body u-mt--0 u-mb--0"><em class=u-weight--md>Alongside their report, reviewers assign a status to the article:</em></p> <dl class=c-definitions> <dt class="c-definitions__term u-upper t-control u-weight--md u-black--high"><span class="u-ib u-middle"> <i class="c-icn--f1r-icon c-icn--approved " title=Approved #data-refInfo=""></i> </span> <span class="u-middle u-ib u-ml--1/2">Approved</span></dt> <dd class="c-definitions__description t-caption">The paper is scientifically sound in its current form and only minor, if any, improvements are suggested</dd> <dt class="c-definitions__term u-upper t-control u-weight--md u-black--high"><span class="u-ib u-middle"> <i class="c-icn--f1r-icon c-icn--reservations" title="Approved with Reservations" #data-refInfo=""></i> </span> <span class="u-middle u-ib u-ml--1/2">Approved with reservations</span></dt> <dd class="c-definitions__description t-caption"> A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit. </dd> <dt class="c-definitions__term u-upper t-control u-weight--md u-black--high"><span class="u-ib u-middle"> <i class="c-icn--f1r-icon c-icn--not-approved " title="Not Approved" #data-refInfo=""></i> </span> <span class="u-middle u-ib u-ml--1/2">Not approved</span></dt> <dd class="c-definitions__description t-caption">Fundamental flaws in the paper seriously undermine the findings and conclusions</dd> </dl> </div> </div> </div> </div> </div> <div class="o-layout__item u-mb--1"><h4 class="u-mt--0 u-mb--0 u-ib u-middle t-h4 u-weight--md">Reviewer Reports</h4></div> <table class="c-report-timeline u-mb--2"> <thead class=c-report-timeline__headings> <tr> <th></th> <th class="u-pb--1/2" colspan=2><em class="t-body u-weight--rg">Invited Reviewers</em></th> </tr> <tr class=c-report-timeline__headings-row> <th></th> <th class="c-report-timeline__headings-version p-article__color--dark">1</th> <th class="c-report-timeline__headings-version p-article__color--dark">2</th> </tr> </thead> <tbody> <tr class="c-report-timeline__row c-report-timeline__row--selected "> <th class=c-report-timeline__version> <a data-test-id=sidebar_timeline_v2_version href="https://f1000research.com/articles/9-295/v2">Version 2</a><br/> <span class=p-article__color--dark> <span class=c-report-timeline__update-type data-test-id=sidebar_timeline_v2_update-type>(revision)<br/></span> <span data-test-id=sidebar_timeline_v2_date class=c-report-timeline__date>26 Jan 21</span> </span> </th> <td class="c-report-timeline__report c-report-timeline__cell c-report-timeline__report--has-prev "> <a class="js-route c-report-timeline__report-link" title="28 Jan 2021" href="https://f1000research.com/articles/9-295/v2#referee-response-78211" data-path="/reports/78211" target-attr=referee-response-78211 data-reportID=78211 style="cursor: pointer;"> <i class="c-icn--f1r-icon c-icn--approved small" title=Approved #data-refInfo=""></i> <span class="u-block c-report-timeline__report-read u-mt--1">read</span> </a> </td> <td class="c-report-timeline__report--empty c-report-timeline__cell c-report-timeline__report--has-prev "></td> </tr> <tr class="c-report-timeline__row "> <th class=c-report-timeline__version> <a data-test-id=sidebar_timeline_v1_version href="https://f1000research.com/articles/9-295/v1">Version 1</a><br/> <span class=p-article__color--dark> <span data-test-id=sidebar_timeline_v1_date class=c-report-timeline__date>27 Apr 20</span> </span> </th> <td class="c-report-timeline__report c-report-timeline__cell c-report-timeline__report--has-next "> <a class="js-route c-report-timeline__report-link" title="13 May 2020" href="https://f1000research.com/articles/9-295/v1#referee-response-62873" data-path="/reports/62873" target-attr=referee-response-62873 data-reportID=62873 style="cursor: pointer;"> <i class="c-icn--f1r-icon c-icn--reservations" title="Approved with Reservations" #data-refInfo=""></i> <span class="u-block c-report-timeline__report-read u-mt--1">read</span> </a> </td> <td class="c-report-timeline__report c-report-timeline__cell "> <a class="js-route c-report-timeline__report-link" title="01 Jun 2020" href="https://f1000research.com/articles/9-295/v1#referee-response-62872" data-path="/reports/62872" target-attr=referee-response-62872 data-reportID=62872 style="cursor: pointer;"> <i class="c-icn--f1r-icon c-icn--approved small" title=Approved #data-refInfo=""></i> <span class="u-block c-report-timeline__report-read u-mt--1">read</span> </a> </td> </tr> </tbody> </table> <div class=o-layout__item> <hr class="c-hr c-hr--low u-mb--2"> <ol class=p-article__sidebar-referees-list> <li class="u-pl--1 u-mb--1/2 t-small"> <div data-test-id=sidebar_referee> <strong data-test-id=sidebar_referee_name>Wilhelm Hasselbring</strong><a href="http://orcid.org/0000-0001-6625-4335" target=_blank title="Open ORCID"><span class=orcid-logo-for-author-list></span></a>, Kiel University, Kiel, Germany </div> </li> <li class="u-pl--1 u-mb--1/2 t-small"> <div data-test-id=sidebar_referee> <strong data-test-id=sidebar_referee_name>Radovan Bast</strong><a href="http://orcid.org/0000-0002-7658-1847" target=_blank title="Open ORCID"><span class=orcid-logo-for-author-list></span></a>, UiT The Arctic University of Norway, Tromsø, Norway </div> </li> </ol> </div> </section> </div> <section class="o-layout__item u-pl"> <div class=u-pl> <hr class="c-hr c-hr--low c-hr--md u-mb--3"> <h4 class="t-h3 u-weight--md u-mt--0 u-mb--2">Comments on this article</h4> <div class=u-mb--4> <p class="u-mt--0 u-mb--1 t-h4"><a class=p-article__color--light href="#article-comments">All Comments</a><span class=" u-ib u-ml--1/2 p-article__color--light">(3)</span></p> <a class=t-h4 href="/login?originalPath=/articles/9-295/v2&scrollTo=add-new-comment" data-test-id=add-comment>Add a comment</a> </div> <hr class="c-hr c-hr--low c-hr--md u-mb--4"> <div class="research-layout f1r-article-desk"> <div class="heading6 c-ribbon-wrapper c-ribbon-wrapper--etoc f1000research "> <div class=c-ribbon-wrapper__body>Sign up for content alerts</div> </div> </div> <div class="research-layout sidebar-sign-up-form f1r-article-desk u-mb--4 "> <form class=js-email-alert-signup action="#" method=POST data-email=tocAlertWeekly> <input type=hidden name=isUserLoggedIn class=js-email-alert-signup-logged-in value=N /> <input type=hidden name=userId class=js-email-alert-signup-user-id value=""/> <input type=hidden name=frequency class=js-email-alert-signup-frequency value=WEEKLY /> <div class="o-actions o-actions--middle"> <div class=o-actions__primary> <input type=email name=emailAddress class="form-input-field js-email-alert-signup-address u-1/1 u-bb" required=required placeholder=Email /> </div> <div class=o-actions__secondary> <div class="_mdl-layout u-ml--1/2"> <button class="mdl-button mdl-js-button mdl-button--colored mdl-button--small mdl-button--filled js-email-alert-signup-submit">Sign Up</button> </div> </div> </div> </form> <div id=sidebar-sign-up-message class="section-text js-email-alert-signup-msg is-hidden">You are now signed up to receive this alert</div> </div> <section class=js-terms-container> <hr class="c-hr c-hr--low c-hr--md u-mb--3"> <h4 class="t-h3 u-weight--md u-mt--0 u-mb--1">Browse by related subjects</h4> <div class="article-subcontainer article-subcontainer--sidebar"> <ul class=js-terms-list></ul> </div> </section> </div> </section> </div> </section> <div class="p-article__sidebar-view u-pl u-hide js-article-sidebar-view" data-view=report-78211> <a class="u-block js-route js-article-sidebar-view-fixed-header c-link--basic p-article__sidebar-back" href="#" data-path="/"><i class="material-icons c-link__icon">keyboard_arrow_left</i> <span class=c-link__text>Back to all reports</span></a> <aside class="c-referee-report js-referee-report t-body u-pr--2 u-pb--8" data-reportid=78211 data-test-id=article_referee_report> <div class=o-layout> <header class="o-layout__item c-referee-report__header"> <div class="o-actions o-actions--middle u-mb--1"> <div class=o-actions__primary> <h2 class="t-h3 u-weight--md u-mb--0 u-mt--0">Reviewer Report</h2> </div> <div class="o-actions__secondary u-hide@expanded"> <span class="t-body u-weight--rg js-report-view-count u-ml--1 u-black--medium"><span class=js-referee-report-views>0</span> Views</span> </div> <div class="o-actions__secondary u-hide@expanded"> <div class="c-block-tip p-article__sidebar-tooltip c-block-tip--below c-block-tip--small-arrow c-block-tip--sm-padding"> <button type=button class="c-button c-button--icon c-button--text c-block-tip__toggle"><i class="material-icons c-button--icon__icon">copyright</i></button> <div class="c-block-tip__content u-fs--normal"> <span class=u-weight--bd>© 2021 Hasselbring W.</span> This is an open access peer review report distributed under the terms of the <a href="https://creativecommons.org/licenses/by/4.0/" target=_blank>Creative Commons Attribution License</a>, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. </div> </div> </div> </div> <p class="u-black--medium u-mt--0 u-mb--1"><em>28 Jan 2021 | for Version 2</em></p> <div class="u-mb--1/2"> <span class=bold>Wilhelm Hasselbring<a href="http://orcid.org/0000-0001-6625-4335" target=_blank title="Open ORCID"><span class=orcid-logo-for-author-list></span></a></span><span class=f1r-article-desk-inline>,</span> Software Engineering Group, Kiel University, Kiel, Germany </div> <hr class="c-hr c-hr--low u-mt--2 u-mb--2 u-hide@expanded"> </header> <div class="o-layout__item c-referee-report__actions u-mb--2"> <span class="t-body u-weight--rg js-report-view-count u-mt--0 u-mb--1 u-black--medium u-show@expanded"><span class=js-referee-report-views>0</span> Views</span> <div id=copyrightInfo class="c-block-tip p-article__sidebar-tooltip c-block-tip--below c-block-tip--small-arrow u-show@expanded u-fr"> <button type=button class="c-button c-button--icon c-button--text c-block-tip__toggle"><i class="material-icons c-button--icon__icon">copyright</i></button> <div class="c-block-tip__content u-fs--normal"> <span class=u-weight--bd>© 2021 Hasselbring W.</span> This is an open access peer review report distributed under the terms of the <a href="https://creativecommons.org/licenses/by/4.0/" target=_blank>Creative Commons Attribution License</a>, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. </div> </div> <a class="u-block c-link--basic u-mb--1 js-route" data-path="/reports/78211/cite" href="#" data-test-id=article_referee-report_cite-link><i class="material-icons c-link__icon">format_quote</i> <span class="u-ml--1 c-link__text">Cite this report</span></a> <a href="#report-responses-78211" class="u-block c-link--basic u-mb--1/2 js-route" data-path="/reports/78211/responses" href="#" data-test-id=article_referee-report_responses-link><i class="material-icons c-link__icon">speaker_notes</i> <span class="u-ml--1 u-mr--1/2 c-link__text u-ib">Responses</span><span class="u-black--medium u-ib u-middle">(0)</span></a> </div> <div class="u-mb--1 u-show@expanded"></div> </div> <hr class="c-hr c-hr--low u-mt--0 u-mb--2"> <div class="o-actions o-actions--middle u-mb--1"> <div class="o-actions__primary p-article__color--dark"> <i class="c-icn--f1r-icon c-icn--approved " title=Approved #data-refInfo=66678-62872></i> <span class="p-article__report-status t-caption u-upper-spacing u-upper u-ib u-middle u-weight--md">Approved</span> </div> <div class="o-actions__secondary _mdl-layout"> <div class="c-block-tip p-article__sidebar-tooltip c-block-tip--below c-block-tip--small-arrow c-block-tip--sm-padding"> <button type=button class="c-button c-button--icon c-button--text c-block-tip__toggle"><i class="material-icons c-button--icon__icon">info_outline</i></button> <div class=c-block-tip__content> <p class="t-body u-mt--0 u-mb--0"><em class=u-weight--md>Alongside their report, reviewers assign a status to the article:</em></p> <dl class=c-definitions> <dt class="c-definitions__term u-upper t-control u-weight--md u-black--high"><span class="u-ib u-middle"> <i class="c-icn--f1r-icon c-icn--approved " title=Approved #data-refInfo=""></i> </span> <span class="u-middle u-ib u-ml--1/2">Approved</span></dt> <dd class="c-definitions__description t-caption">The paper is scientifically sound in its current form and only minor, if any, improvements are suggested</dd> <dt class="c-definitions__term u-upper t-control u-weight--md u-black--high"><span class="u-ib u-middle"> <i class="c-icn--f1r-icon c-icn--reservations" title="Approved with Reservations" #data-refInfo=""></i> </span> <span class="u-middle u-ib u-ml--1/2">Approved with reservations</span></dt> <dd class="c-definitions__description t-caption"> A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit. </dd> <dt class="c-definitions__term u-upper t-control u-weight--md u-black--high"><span class="u-ib u-middle"> <i class="c-icn--f1r-icon c-icn--not-approved " title="Not Approved" #data-refInfo=""></i> </span> <span class="u-middle u-ib u-ml--1/2">Not approved</span></dt> <dd class="c-definitions__description t-caption">Fundamental flaws in the paper seriously undermine the findings and conclusions</dd> </dl> </div> </div> </div> </div> <div class="">Thanks for delivering this revised version of your opinion article. I highly appreciate that you addressed all the concerns I had with the previous version, such that I can now fully approve your paper!<br/> </div> <div class=u-mb--2> </div> <hr class="c-hr c-hr--low u-mb--2"> <h5 class="t-body u-weight--md u-mt--0 u-mb--1/2">Competing Interests</h5> <p class="u-mt--0 u-mb--2 u-italic">No competing interests were disclosed.</p> <hr class="c-hr c-hr--low u-mb--2"> <h5 class="t-body u-weight--md u-mt--0 u-mb--1/2">Reviewer Expertise</h5> <p class="u-mb--2 u-mt--0">Software Engineering</p> <hr class="c-hr c-hr--low u-mb--3"> <p class="u-mb--3 u-mt--0"> <strong> I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. </strong> </p> <div class="_mdl-layout u-mb--3"> <a href="/login?originalPath=/articles/9-295/v2&scrollTo=referee-response-78211" class="u-1/1 mdl-button mdl-js-button mdl-button--outline mdl-button--multi-line mdl-button--colored mdl-button--white-ish" data-test-id=article_referee-report_respond><i class="material-icons u-ib u-middle">reply</i><span class="p-article__sidebar-report-respond-text u-ib u-middle">Respond to this report</span></a> </div> <hr class="c-hr c-hr--low c-hr--md u-mb--3"> <section id=report-responses-78211> <h4 class="t-h3 u-weight--md u-mb--2 u-mt--0">Responses <span class="u-weight--rg u-black--medium t-body">(0)</span></h4> </section> <script type="text/template" data-template=refereeReportCitation-78211 id=refereeReportCitation-78211 class=u-hide> <div class="u-black--high"> <p id="referee-report-citation" data-test-id="report_copy-citation_text" >Hasselbring W. Peer Review Report For: An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action [version 2; peer review: 2 approved]</span>. <i>F1000Research</i> 2021, <b>9</b>:295 (<a class="new-orange" href="https://doi.org/10.5256/f1000research.31504.r78211" target="_blank">https://doi.org/10.5256/f1000research.31504.r78211)</a></p> <p class="c-modal__flush u-pl u-pr u-pt--3/2 u-pb--3/2 u-red u-bg--12"><strong>NOTE:</strong> it is important to ensure the information in square brackets after the title is included in this citation.</p> <p class="t-body">The direct URL for this report is: <br><a href="https://f1000research.com/articles/9-295/v2#referee-response-78211" target="_blank">https://f1000research.com/articles/9-295/v2#referee-response-78211</a></p> </div> </script> </aside> </div> <div class="p-article__sidebar-view u-pl u-hide js-article-sidebar-view" data-view=report-62872> <a class="u-block js-route js-article-sidebar-view-fixed-header c-link--basic p-article__sidebar-back" href="#" data-path="/"><i class="material-icons c-link__icon">keyboard_arrow_left</i> <span class=c-link__text>Back to all reports</span></a> <aside class="c-referee-report js-referee-report t-body u-pr--2 u-pb--8" data-reportid=62872 data-test-id=article_referee_report> <div class=o-layout> <header class="o-layout__item c-referee-report__header"> <div class="o-actions o-actions--middle u-mb--1"> <div class=o-actions__primary> <h2 class="t-h3 u-weight--md u-mb--0 u-mt--0">Reviewer Report</h2> </div> <div class="o-actions__secondary u-hide@expanded"> <span class="t-body u-weight--rg js-report-view-count u-ml--1 u-black--medium"><span class=js-referee-report-views>0</span> Views</span> </div> <div class="o-actions__secondary u-hide@expanded"> <div class="c-block-tip p-article__sidebar-tooltip c-block-tip--below c-block-tip--small-arrow c-block-tip--sm-padding"> <button type=button class="c-button c-button--icon c-button--text c-block-tip__toggle"><i class="material-icons c-button--icon__icon">copyright</i></button> <div class="c-block-tip__content u-fs--normal"> <span class=u-weight--bd>© 2020 Bast R.</span> This is an open access peer review report distributed under the terms of the <a href="https://creativecommons.org/licenses/by/4.0/" target=_blank>Creative Commons Attribution License</a>, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. </div> </div> </div> </div> <p class="u-black--medium u-mt--0 u-mb--1"><em>01 Jun 2020 | for Version 1</em></p> <div class="u-mb--1/2"> <span class=bold>Radovan Bast<a href="http://orcid.org/0000-0002-7658-1847" target=_blank title="Open ORCID"><span class=orcid-logo-for-author-list></span></a></span><span class=f1r-article-desk-inline>,</span> Department of Information Technology, UiT The Arctic University of Norway, Tromsø, Norway </div> <hr class="c-hr c-hr--low u-mt--2 u-mb--2 u-hide@expanded"> </header> <div class="o-layout__item c-referee-report__actions u-mb--2"> <span class="t-body u-weight--rg js-report-view-count u-mt--0 u-mb--1 u-black--medium u-show@expanded"><span class=js-referee-report-views>0</span> Views</span> <div id=copyrightInfo class="c-block-tip p-article__sidebar-tooltip c-block-tip--below c-block-tip--small-arrow u-show@expanded u-fr"> <button type=button class="c-button c-button--icon c-button--text c-block-tip__toggle"><i class="material-icons c-button--icon__icon">copyright</i></button> <div class="c-block-tip__content u-fs--normal"> <span class=u-weight--bd>© 2020 Bast R.</span> This is an open access peer review report distributed under the terms of the <a href="https://creativecommons.org/licenses/by/4.0/" target=_blank>Creative Commons Attribution License</a>, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. </div> </div> <a class="u-block c-link--basic u-mb--1 js-route" data-path="/reports/62872/cite" href="#" data-test-id=article_referee-report_cite-link><i class="material-icons c-link__icon">format_quote</i> <span class="u-ml--1 c-link__text">Cite this report</span></a> <a href="#report-responses-62872" class="u-block c-link--basic u-mb--1/2 js-route" data-path="/reports/62872/responses" href="#" data-test-id=article_referee-report_responses-link><i class="material-icons c-link__icon">speaker_notes</i> <span class="u-ml--1 u-mr--1/2 c-link__text u-ib">Responses</span><span class="u-black--medium u-ib u-middle">(1)</span></a> </div> <div class="u-mb--1 u-show@expanded"></div> </div> <hr class="c-hr c-hr--low u-mt--0 u-mb--2"> <div class="o-actions o-actions--middle u-mb--1"> <div class="o-actions__primary p-article__color--dark"> <i class="c-icn--f1r-icon c-icn--approved " title=Approved #data-refInfo=66678-62872></i> <span class="p-article__report-status t-caption u-upper-spacing u-upper u-ib u-middle u-weight--md">Approved</span> </div> <div class="o-actions__secondary _mdl-layout"> <div class="c-block-tip p-article__sidebar-tooltip c-block-tip--below c-block-tip--small-arrow c-block-tip--sm-padding"> <button type=button class="c-button c-button--icon c-button--text c-block-tip__toggle"><i class="material-icons c-button--icon__icon">info_outline</i></button> <div class=c-block-tip__content> <p class="t-body u-mt--0 u-mb--0"><em class=u-weight--md>Alongside their report, reviewers assign a status to the article:</em></p> <dl class=c-definitions> <dt class="c-definitions__term u-upper t-control u-weight--md u-black--high"><span class="u-ib u-middle"> <i class="c-icn--f1r-icon c-icn--approved " title=Approved #data-refInfo=""></i> </span> <span class="u-middle u-ib u-ml--1/2">Approved</span></dt> <dd class="c-definitions__description t-caption">The paper is scientifically sound in its current form and only minor, if any, improvements are suggested</dd> <dt class="c-definitions__term u-upper t-control u-weight--md u-black--high"><span class="u-ib u-middle"> <i class="c-icn--f1r-icon c-icn--reservations" title="Approved with Reservations" #data-refInfo=""></i> </span> <span class="u-middle u-ib u-ml--1/2">Approved with reservations</span></dt> <dd class="c-definitions__description t-caption"> A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit. </dd> <dt class="c-definitions__term u-upper t-control u-weight--md u-black--high"><span class="u-ib u-middle"> <i class="c-icn--f1r-icon c-icn--not-approved " title="Not Approved" #data-refInfo=""></i> </span> <span class="u-middle u-ib u-ml--1/2">Not approved</span></dt> <dd class="c-definitions__description t-caption">Fundamental flaws in the paper seriously undermine the findings and conclusions</dd> </dl> </div> </div> </div> </div> <div class="">In "An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action" the authors identify challenges for research software sustainability in Germany and beyond.<br/> <br/> They examine the current state of research software sustainability and challenges in motivating sustainable research software development, selection criteria for funding, personnel, funding, infrastructure, and legal aspects, and offer recommendations for addressing these challenges. These sections are accompanied and with a story using a fictional character Kim which helps to<br/> relate these aspects to typical career stages of a research software engineer.<br/> <br/> The article is thoroughly researched, well-written, and offers an excellent overview of the challenges when building an environment for sustainable research software. Most of the discussed challenges and recommendations carry beyond Germany and are relevant and transferable to other countries.<br/> <br/> Below I give few (minor) suggestions for consideration when improving the manuscript.<br/> <br/> Regarding the list of challenges under "Why sustainable research software in the first place?" (pages 5 and 6): <ul> <li>Infrastructure issues: One design choice that often limits the use or usability of local infrastructure resources is that they are often bound to institutional user accounts and thus limit collaboration possibilities with collaborators in other institutions and countries. On the other hand, pooling of infrastructure resources which could enable collaboration across organizations can be limited by lack of authentication and authorization infrastructure (AAI) or legal constraints. Later in the paper the authors indeed mention AAI (page 12) but this could already be pointed out and connected earlier.<br/> </li> <li>Legal issues: Not only licensing is an issue but legal constraints or uncertainty about legal boundaries and identity federation can also limit the deployment of infrastructure services. Often the deployment and operation of infrastructure services is given to technical teams who may lack the legal support or expertise to clarify legal and privacy terms for the storage of data and processing of data.<br/> </li> <li>Funding issues: The challenge is not only that funding is scarce but also that it does not align well with pricing models of cloud infrastructure providers. It can be easier for research groups to spend a larger chunk of the budget towards the end of a year for hardware compared to pay possibly relatively modest monthly fees for a cloud service, which however may not fit into the budget forms. These budget constraints may also limit the possibility of pooling resources and sharing them with other research groups. Software cloud infrastructure is often not considered at all in the proposal. There is also a resistance among some of my research colleagues to pay 20-50 USD/ month for an infrastructure service which is sometimes solved by reinventing the service locally "for free".<br/> </li> <li>Another mismatch between traditional funding models and support of software which "must continue to exist" to be sustainable (page 5), is the experience that it can take months or years until the software is picked up by other groups and contributions and questions start to roll in. But by that time the funding of the project stopped, the developer (team) may have already moved on to other positions and projects, and may not have the time to react and help, even though they still may have interest and the knowledge. Our traditional funding models consider the software to be "done" by the end of the project.</li> </ul> Selection criteria for "How to decide which software to sustain?" (page 9): <ul> <li>The authors mention "usage and impact", "software quality", as well as "maturity". But I would like to see also "openness and transparency" among these. The reason is that we can expect the research community to adapt to these or any metrics and we will over time observe what we measure. Any set of metrics could be criticized as to some extent being arbitrary but the advantage of including "openness and transparency" is that the community as whole would benefit from such a metric [Enrico Glerean, "Responsible conduct of research and questionable research practices", presentation, slide 47<a href="#rep-ref-62872-1"><sup>1</sup></a>].</li> </ul> Regarding "Who sustains research software?": <ul> <li>The authors discuss the lack of recognition and awareness, as well as lack of career opportunities. It is also about respect and I was happy to see the sentence: "Not all researchers that think of themselves as RSEs pursue a faculty position as their main career goal." I have experienced that RSEs are sometimes regarded as those who somehow "failed" to obtain a faculty position whereas many RSEs have chosen this position over a faculty position because it was a better fit for their career goals. This misunderstanding can lead to a lack of respect towards this position and this career choice and can lead to excellent personnel leaving the academic environment towards commercial employment, possibly not primarily for financial reasons but sometimes to be more respected and recognized.</li> </ul> Archiving and software preservation (page 13: <ul> <li>The authors mention Docker but also Singularity should be mentioned as a tool since it is getting traction in particular on many-user systems such as higher performance computing clusters.</li> </ul> Legal aspects (page 14): <ul> <li>Re-licensing is mentioned in the story box and the text starts by pointing out that licensing is often not considered early enough in the project. Indeed re-licensing later in the project can be not only legally, but also organizationally very tricky, in particular for projects which developed over many years and involved many contributors in different organizations. This could be pointed out in the text as additional motivation to consider these very early in the project.<br/> </li> <li>I very much like the recommendation of providing a legal help desk for research groups to avoid the problem that out of uncertainty and fear of making a legal mistake some research groups end up not choosing any license at all which may limit further reuse of the software.<br/> </li> <li>The manuscript presents a decision tree for contributors (Figure 1) and also discusses contributor license agreements. It could be useful to point out that without clear policies or legal help desks, individuals or organizations may be hesitant to contribute to a project because they may not feel confident having enough knowledge or authority to sign such agreements and too many legal steps and question can also raise the barrier to contribute, in particular for smaller projects. Also here clear guidelines and a support desk can help removing these barriers.</li> </ul></div> <div class=u-mb--2> <div class=md> <div class=questionnaire> <ul class=c-questionnaire> <li class=c-questionnaire__item> <p class=c-questionnaire__question>Is the topic of the opinion article discussed accurately in the context of the current literature?</p> <p class=c-questionnaire__answer>Yes</p> </li> <li class=c-questionnaire__item> <p class=c-questionnaire__question>Are all factual statements correct and adequately supported by citations?</p> <p class=c-questionnaire__answer>Yes</p> </li> <li class=c-questionnaire__item> <p class=c-questionnaire__question>Are arguments sufficiently supported by evidence from the published literature?</p> <p class=c-questionnaire__answer>Yes</p> </li> <li class=c-questionnaire__item> <p class=c-questionnaire__question>Are the conclusions drawn balanced and justified on the basis of the presented arguments?</p> <p class=c-questionnaire__answer>Yes</p> </li> </ul> </div> </div> </div> <hr class="c-hr c-hr--low u-mb--2"> <h5 class="t-body u-weight--md u-mt--0 u-mb--1/2"><b>References</b></h5> <div class=u-mb--2><a name=rep-ref-62872-1></a>1. Enrico Glerean: Responsible conduct of research and questionable research practices. 2018. <a target=_blank href="https://doi.org/10.6084/m9.figshare.10303487.v1">Reference Source</a> <br></div> <hr class="c-hr c-hr--low u-mb--2"> <h5 class="t-body u-weight--md u-mt--0 u-mb--1/2">Competing Interests</h5> <p class="u-mt--0 u-mb--2 u-italic">No competing interests were disclosed.</p> <hr class="c-hr c-hr--low u-mb--2"> <h5 class="t-body u-weight--md u-mt--0 u-mb--1/2">Reviewer Expertise</h5> <p class="u-mb--2 u-mt--0">computational chemistry, research software engineering</p> <hr class="c-hr c-hr--low u-mb--3"> <p class="u-mb--3 u-mt--0"> <strong> I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. </strong> </p> <div class="_mdl-layout u-mb--3"> <a href="/login?originalPath=/articles/9-295/v2&scrollTo=referee-response-62872" class="u-1/1 mdl-button mdl-js-button mdl-button--outline mdl-button--multi-line mdl-button--colored mdl-button--white-ish" data-test-id=article_referee-report_respond><i class="material-icons u-ib u-middle">reply</i><span class="p-article__sidebar-report-respond-text u-ib u-middle">Respond to this report</span></a> </div> <hr class="c-hr c-hr--low c-hr--md u-mb--3"> <section id=report-responses-62872> <h4 class="t-h3 u-weight--md u-mb--2 u-mt--0">Responses <span class="u-weight--rg u-black--medium t-body">(1)</span></h4> <span class="c-hr c-hr--low u-mb--3"></span> <article id=referee-report-comment-6210 class="c-comment js-referee-report-comment " data-commentid=6210 data-reportid=62872 data-test-id=article_referee_report_comment> <header> <div class="c-comment__role t-caption u-black--medium" data-test-id=article_referee-report-comment_roles> <span class="u-weight--md u-upper u-ib u-mr--1/2">Author Response</span> </div> <div class="c-comment__date u-black--medium t-caption">26 Jan 2021</div> <h5 class="c-comment__name t-body u-weight--rg u-mt--0 u-mb--1" data-test-id=article_referee-report-comment_name> <strong class=u-weight--md>Axel Loewe, </strong><span class="c-comment__affiliations u-weight--rg">Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany</span> </h5> <div class="c-comment__body u-mb--1"> <div class="c-read-more js-read-more t-body" data-lines=4> <div class="c-read-more__content js-read-more-content "> <div class=u-mb--1 data-test-id=article_referee-report-comment_text data-commentid=6210><i>We thank you for the thorough review and constructive feedback regarding the manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> <b>Regarding the list of challenges under "Why sustainable research software in the first place?" (pages 5 and 6):</b><br/> Infrastructure issues: One design choice that often limits the use or usability of local infrastructure resources is that they are often bound to institutional user accounts and thus limit collaboration possibilities with collaborators in other institutions and countries. On the other hand, pooling of infrastructure resources which could enable collaboration across organizations can be limited by lack of authentication and authorization infrastructure (AAI) or legal constraints. Later in the paper the authors indeed mention AAI (page 12) but this could already be pointed out and connected earlier.<br/> <i>We thank the reviewer for their suggestion, and have included the mentioned issues in the respective list in the section “Why sustainable research software in the first place?”.</i><br/> <b> </b><br/> Legal issues: Not only licensing is an issue but legal constraints or uncertainty about legal boundaries and identity federation can also limit the deployment of infrastructure services. Often the deployment and operation of infrastructure services is given to technical teams who may lack the legal support or expertise to clarify legal and privacy terms for the storage of data and processing of data.<br/> <i>We thank the reviewer for their suggestion, and have included the mentioned issues in the respective list in the section “Why sustainable research software in the first place?”<br/> </i><br/> Funding issues: The challenge is not only that funding is scarce but also that it does not align well with pricing models of cloud infrastructure providers. It can be easier for research groups to spend a larger chunk of the budget towards the end of a year for hardware compared to pay possibly relatively modest monthly fees for a cloud service, which however may not fit into the budget forms. These budget constraints may also limit the possibility of pooling resources and sharing them with other research groups. Software cloud infrastructure is often not considered at all in the proposal. There is also a resistance among some of my research colleagues to pay 20-50 USD/ month for an infrastructure service which is sometimes solved by reinventing the service locally "for free".<br/> Another mismatch between traditional funding models and support of software which "must continue to exist" to be sustainable (page 5), is the experience that it can take months or years until the software is picked up by other groups and contributions and questions start to roll in. But by that time the funding of the project stopped, the developer (team) may have already moved on to other positions and projects, and may not have the time to react and help, even though they still may have interest and the knowledge. Our traditional funding models consider the software to be "done" by the end of the project.<br/> <i>We thank the reviewer for these two comments. To address them, we have extended the “Funding issues” list item with a discussion of these issues.</i><br/> <br/> <b>Selection criteria for "How to decide which software to sustain?" (page 9):</b><br/> The authors mention "usage and impact", "software quality", as well as "maturity". But I would like to see also "openness and transparency" among these. The reason is that we can expect the research community to adapt to these or any metrics and we will over time observe what we measure. Any set of metrics could be criticized as to some extent being arbitrary but the advantage of including "openness and transparency" is that the community as whole would benefit from such a metric [Enrico Glerean, "Responsible conduct of research and questionable research practices", presentation, slide 471].<br/> <i>We thank the reviewer for this suggestion and realized that indeed most aspects in this section (6-10, 12, 13) are actually related to openness and transparency. Therefore, we changed the title of this section to “Software transparency and quality”.</i><br/> <br/> <b>Regarding "Who sustains research software?":</b><br/> The authors discuss the lack of recognition and awareness, as well as lack of career opportunities. It is also about respect and I was happy to see the sentence: "Not all researchers that think of themselves as RSEs pursue a faculty position as their main career goal." I have experienced that RSEs are sometimes regarded as those who somehow "failed" to obtain a faculty position whereas many RSEs have chosen this position over a faculty position because it was a better fit for their career goals. This misunderstanding can lead to a lack of respect towards this position and this career choice and can lead to excellent personnel leaving the academic environment towards commercial employment, possibly not primarily for financial reasons but sometimes to be more respected and recognized.<br/> <i>We fully agree with the reviewer and thank them for the renewed confirmation that this is seen as problematic not only by the authors.</i><br/> <br/> <b>Archiving and software preservation (page 13):</b><br/> The authors mention Docker but also Singularity should be mentioned as a tool since it is getting traction in particular on many-user systems such as higher performance computing clusters.<br/> <i>We thank the reviewer for this suggestion and now also mention Singularity and GUIX. However, we are not aiming for an exhaustive list, as options change dynamically and might even be specific to certain research communities.</i><br/> <br/> <b>Legal aspects (page 14):</b><br/> Re-licensing is mentioned in the story box and the text starts by pointing out that licensing is often not considered early enough in the project. Indeed re-licensing later in the project can be not only legally, but also organizationally very tricky, in particular for projects which developed over many years and involved many contributors in different organizations. This could be pointed out in the text as additional motivation to consider these very early in the project.<br/> <i>We thank the reviewer to point this out and it also nicely fits into the message of increasing the awareness of legal aspects early on in the project. We have added the suggested sentence in the manuscript.</i><br/> <b> </b><br/> I very much like the recommendation of providing a legal help desk for research groups to avoid the problem that out of uncertainty and fear of making a legal mistake some research groups end up not choosing any license at all which may limit further reuse of the software.<br/> <i>Thank you for supporting this idea. We have further included your idea of avoiding any license out of a fear to make legal mistakes.</i><br/> <b> </b><br/> The manuscript presents a decision tree for contributors (Figure 1) and also discusses contributor license agreements. It could be useful to point out that without clear policies or legal help desks, individuals or organizations may be hesitant to contribute to a project because they may not feel confident having enough knowledge or authority to sign such agreements and too many legal steps and question can also raise the barrier to contribute, in particular for smaller projects. Also here clear guidelines and a support desk can help removing these barriers.<br/> <i>We have decided to take the decision trees out of the manuscript to strengthen our point of publishing software under a FLOSS license. Instead, we published the decision trees together with documentation templates under a Creative Commons license via Zenodo: <a href="https://zenodo.org/record/4327148#.X9n6ui337OQ">https://zenodo.org/record/4327148#.X9n6ui337OQ</a>. In order to strengthen the point you addressed, we added some more details related to infrastructural investment.</i></div> </div> <a href="#" class="c-read-more__toggle js-read-more-toggle t-caption"> <span class=c-read-more__expand>View more</span> <span class=c-read-more__contract>View less</span> </a> </div> </div> <div class=c-comment__footer> <h5 class="t-body u-weight--md u-mt--0 u-mb--1/2">Competing Interests</h5> <p class="u-mt--0 u-mb--1 u-italic" data-test-id=article_referee-report-comment_competing-interests>No competing interests were disclosed.</p> </div> <footer> <div class="o-actions o-actions--middle"> <div class=o-actions__primary> <a href="/login?originalPath=/articles/9-295/v2&scrollTo=referee-response-62872" class="js-referee-report-comment c-link--basic" data-test-id=article_referee-report-comment_respond><i class="material-icons c-link__icon">reply</i> <span class=c-link__text>Respond</span></a> </div> <div class=o-actions__secondary> <a href="mailto:research@f1000.com?subject=Report a concern (23224) " data-test-id=article_referee-report-comment_concern class="js-report-concern t-caption c-link c-link--inline">Report a concern</a> </div> </div> </footer> </header> </article> </section> <script type="text/template" data-template=refereeReportCitation-62872 id=refereeReportCitation-62872 class=u-hide> <div class="u-black--high"> <p id="referee-report-citation" data-test-id="report_copy-citation_text" >Bast R. Peer Review Report For: An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action [version 2; peer review: 2 approved]</span>. <i>F1000Research</i> 2021, <b>9</b>:295 (<a class="new-orange" href="https://doi.org/10.5256/f1000research.25640.r62872" target="_blank">https://doi.org/10.5256/f1000research.25640.r62872)</a></p> <p class="c-modal__flush u-pl u-pr u-pt--3/2 u-pb--3/2 u-red u-bg--12"><strong>NOTE:</strong> it is important to ensure the information in square brackets after the title is included in this citation.</p> <p class="t-body">The direct URL for this report is: <br><a href="https://f1000research.com/articles/9-295/v1#referee-response-62872" target="_blank">https://f1000research.com/articles/9-295/v1#referee-response-62872</a></p> </div> </script> </aside> </div> <div class="p-article__sidebar-view u-pl u-hide js-article-sidebar-view" data-view=report-62873> <a class="u-block js-route js-article-sidebar-view-fixed-header c-link--basic p-article__sidebar-back" href="#" data-path="/"><i class="material-icons c-link__icon">keyboard_arrow_left</i> <span class=c-link__text>Back to all reports</span></a> <aside class="c-referee-report js-referee-report t-body u-pr--2 u-pb--8" data-reportid=62873 data-test-id=article_referee_report> <div class=o-layout> <header class="o-layout__item c-referee-report__header"> <div class="o-actions o-actions--middle u-mb--1"> <div class=o-actions__primary> <h2 class="t-h3 u-weight--md u-mb--0 u-mt--0">Reviewer Report</h2> </div> <div class="o-actions__secondary u-hide@expanded"> <span class="t-body u-weight--rg js-report-view-count u-ml--1 u-black--medium"><span class=js-referee-report-views>0</span> Views</span> </div> <div class="o-actions__secondary u-hide@expanded"> <div class="c-block-tip p-article__sidebar-tooltip c-block-tip--below c-block-tip--small-arrow c-block-tip--sm-padding"> <button type=button class="c-button c-button--icon c-button--text c-block-tip__toggle"><i class="material-icons c-button--icon__icon">copyright</i></button> <div class="c-block-tip__content u-fs--normal"> <span class=u-weight--bd>© 2020 Hasselbring W.</span> This is an open access peer review report distributed under the terms of the <a href="https://creativecommons.org/licenses/by/4.0/" target=_blank>Creative Commons Attribution License</a>, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. </div> </div> </div> </div> <p class="u-black--medium u-mt--0 u-mb--1"><em>13 May 2020 | for Version 1</em></p> <div class="u-mb--1/2"> <span class=bold>Wilhelm Hasselbring<a href="http://orcid.org/0000-0001-6625-4335" target=_blank title="Open ORCID"><span class=orcid-logo-for-author-list></span></a></span><span class=f1r-article-desk-inline>,</span> Software Engineering Group, Kiel University, Kiel, Germany </div> <hr class="c-hr c-hr--low u-mt--2 u-mb--2 u-hide@expanded"> </header> <div class="o-layout__item c-referee-report__actions u-mb--2"> <span class="t-body u-weight--rg js-report-view-count u-mt--0 u-mb--1 u-black--medium u-show@expanded"><span class=js-referee-report-views>0</span> Views</span> <div id=copyrightInfo class="c-block-tip p-article__sidebar-tooltip c-block-tip--below c-block-tip--small-arrow u-show@expanded u-fr"> <button type=button class="c-button c-button--icon c-button--text c-block-tip__toggle"><i class="material-icons c-button--icon__icon">copyright</i></button> <div class="c-block-tip__content u-fs--normal"> <span class=u-weight--bd>© 2020 Hasselbring W.</span> This is an open access peer review report distributed under the terms of the <a href="https://creativecommons.org/licenses/by/4.0/" target=_blank>Creative Commons Attribution License</a>, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. </div> </div> <a class="u-block c-link--basic u-mb--1 js-route" data-path="/reports/62873/cite" href="#" data-test-id=article_referee-report_cite-link><i class="material-icons c-link__icon">format_quote</i> <span class="u-ml--1 c-link__text">Cite this report</span></a> <a href="#report-responses-62873" class="u-block c-link--basic u-mb--1/2 js-route" data-path="/reports/62873/responses" href="#" data-test-id=article_referee-report_responses-link><i class="material-icons c-link__icon">speaker_notes</i> <span class="u-ml--1 u-mr--1/2 c-link__text u-ib">Responses</span><span class="u-black--medium u-ib u-middle">(1)</span></a> </div> <div class="u-mb--1 u-show@expanded"></div> </div> <hr class="c-hr c-hr--low u-mt--0 u-mb--2"> <div class="o-actions o-actions--middle u-mb--1"> <div class="o-actions__primary p-article__color--dark"> <span class="u-ib u-middle"> <i class="c-icn--f1r-icon c-icn--reservations" title="Approved with Reservations" #data-refInfo=66678-62872></i> </span> <span class="p-article__report-status t-caption u-upper-spacing u-upper u-ib u-middle u-weight--md">Approved With Reservations</span> </div> <div class="o-actions__secondary _mdl-layout"> <div class="c-block-tip p-article__sidebar-tooltip c-block-tip--below c-block-tip--small-arrow c-block-tip--sm-padding"> <button type=button class="c-button c-button--icon c-button--text c-block-tip__toggle"><i class="material-icons c-button--icon__icon">info_outline</i></button> <div class=c-block-tip__content> <p class="t-body u-mt--0 u-mb--0"><em class=u-weight--md>Alongside their report, reviewers assign a status to the article:</em></p> <dl class=c-definitions> <dt class="c-definitions__term u-upper t-control u-weight--md u-black--high"><span class="u-ib u-middle"> <i class="c-icn--f1r-icon c-icn--approved " title=Approved #data-refInfo=""></i> </span> <span class="u-middle u-ib u-ml--1/2">Approved</span></dt> <dd class="c-definitions__description t-caption">The paper is scientifically sound in its current form and only minor, if any, improvements are suggested</dd> <dt class="c-definitions__term u-upper t-control u-weight--md u-black--high"><span class="u-ib u-middle"> <i class="c-icn--f1r-icon c-icn--reservations" title="Approved with Reservations" #data-refInfo=""></i> </span> <span class="u-middle u-ib u-ml--1/2">Approved with reservations</span></dt> <dd class="c-definitions__description t-caption"> A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit. </dd> <dt class="c-definitions__term u-upper t-control u-weight--md u-black--high"><span class="u-ib u-middle"> <i class="c-icn--f1r-icon c-icn--not-approved " title="Not Approved" #data-refInfo=""></i> </span> <span class="u-middle u-ib u-ml--1/2">Not approved</span></dt> <dd class="c-definitions__description t-caption">Fundamental flaws in the paper seriously undermine the findings and conclusions</dd> </dl> </div> </div> </div> </div> <div class="">The authors describe the state of the practice and current challenges for research software sustainability and suggest measures towards improvements that can solve these challenges. In particular, they propose to fund a German Software Sustainability Institute. The paper is the result of a community effort, with work undertaken during two workshops and subsequent collaborative work across the larger RSE community in Germany.<br/> <br/> The UK Software Sustainability Institute has already been established during a decade (<a href="https://www.software.ac.uk/blog/2020-05-05-impact-institute-10-years">https://www.software.ac.uk/blog/2020-05-05-impact-institute-10-years</a>). Thus, the idea of such an institute is not new, but it makes sense to take a specific look at the German situation. Besides universities, the German states (local and in particular federal) fund significant large-scale research associations (Helmholtz/DLR, Max-Planck, Leibniz). This is not the case for most other European states, at least not with a similar scale. Another specialty is the lack of long-term funding for research software engineers, as discussed by the authors.<br/> <br/> The paper is well-written and easy to read. I like the boxed story of Kim’s career path.<br/> <br/> However, I’ve some suggestions for improving the paper: <ul> <li>Concerning the statement “In order to support research, a sustainable software must be correct”, I suggest to include a short discussion of the test oracle problem for scientific software (see for instance https://doi.org/10.1109/SECSE.2013.6615099<a href="#rep-ref-62873-1"><sup>1</sup></a>).<br/> </li> <li>Concerning the discussion of “The list of criteria presented in this section could be the basis for a structured review process…” I suggest to include two additional initiatives for software review. The first is artifact evaluation in computer science conferences (the process is explained in <a href="https://doi.org/10.1515/itit-2019-0040">https://doi.org/10.1515/itit-2019-0040</a><a href="#rep-ref-62873-2"><sup>2</sup></a>). The second is the SPEC Research Group’s review process of tools for quantitative system evaluation and analysis (https://research.spec.org/tools/submission.html).</li> </ul> The authors write “We also argue that truly sustainable research software must ideally be published under a Free/Libre Open Source Software (FLOSS) license, and follow an open development model…” what I fully support (see for instance <a href="https://doi.org/10.1515/itit-2019-0040">https://doi.org/10.1515/itit-2019-0040</a><a href="#rep-ref-62873-2"><sup>2</sup></a>). However, later under the section heading “Legal aspects” this requirement is thwarted. I fully agree that legal aspects have to be considered, but the general bias of this section seems to be on commercial licensing of research software. For instance, the decision tree in Figure 1 starts with the question “Licensing planned?”. I assume that commercial licensing is meant, but this is not clear since the figures are not explained in the paper. Instead, the process should start with open sourcing the software. If licenses such as Apache or MIT are applied, the research institutions may later still commercialize the software if appropriate. Such open source licensing is also beneficial for start-ups, that intend to provide professional services for the software.<br/> <br/> My experience with technology transfer units of German universities and research institutes is that they do not understand the ideas of open source business models (see for instance <a href="https://doi.org/10.1109/MC.2019.2898163">https://doi.org/10.1109/MC.2019.2898163</a><a href="#rep-ref-62873-3"><sup>3</sup></a>). Their focus is on patents and commercializing licenses, sometimes also on start-ups. Conversely, in the software industry, one major motivation for open sourcing software is on improving the quality of software. I cite from <a href="https://doi.org/10.1109/ICSAW.2017.11">https://doi.org/10.1109/ICSAW.2017.11</a><a href="#rep-ref-62873-4"><sup>4</sup></a> : “the open-source approach has some psychological effects: Developers show a tendency to apply higher quality standards if they know that the code will be publicly available.” For sustainability, quality is an important property of software.<br/> <br/> The Figures 1-4 do more harm than good. They are daunting to researchers who intend to publish their code open source. These figures should be removed from the paper, they are useless without proper explanation.<br/> <br/> I suggest that the authors focus in the present paper on their main message (request for funding a German Software Sustainability Institute, which I fully support). Figures 1-4 could be moved to a separate paper, enriched with proper explanation.</div> <div class=u-mb--2> <div class=md> <div class=questionnaire> <ul class=c-questionnaire> <li class=c-questionnaire__item> <p class=c-questionnaire__question>Is the topic of the opinion article discussed accurately in the context of the current literature?</p> <p class=c-questionnaire__answer>Yes</p> </li> <li class=c-questionnaire__item> <p class=c-questionnaire__question>Are all factual statements correct and adequately supported by citations?</p> <p class=c-questionnaire__answer>Yes</p> </li> <li class=c-questionnaire__item> <p class=c-questionnaire__question>Are arguments sufficiently supported by evidence from the published literature?</p> <p class=c-questionnaire__answer>Partly</p> </li> <li class=c-questionnaire__item> <p class=c-questionnaire__question>Are the conclusions drawn balanced and justified on the basis of the presented arguments?</p> <p class=c-questionnaire__answer>Partly</p> </li> </ul> </div> </div> </div> <hr class="c-hr c-hr--low u-mb--2"> <h5 class="t-body u-weight--md u-mt--0 u-mb--1/2"><b>References</b></h5> <div class=u-mb--2><a name=rep-ref-62873-1></a>1. Kanewala U, Bieman JM: Techniques for testing scientific programs without an oracle. <i>IEEE Xplore</i>. 2013. <a target=_blank href="https://doi.org/10.1109/SECSE.2013.6615099">Publisher Full Text</a> <br><a name=rep-ref-62873-2></a>2. Hasselbring W, Carr L, Hettrick S, Packer H, et al.: From FAIR research data toward FAIR and open research software. <i>it - Information Technology</i>. 2020; <b>62</b> (1): 39-47 <a target=_blank href="https://doi.org/10.1515/itit-2019-0040">Publisher Full Text</a> <br><a name=rep-ref-62873-3></a>3. Riehle D: The Innovations of Open Source. <i>Computer</i>. 2019; <b>52</b> (4): 59-63 <a target=_blank href="https://doi.org/10.1109/MC.2019.2898163">Publisher Full Text</a> <br><a name=rep-ref-62873-4></a>4. Hasselbring W, Steinacker G: Microservice Architectures for Scalability, Agility and Reliability in E-Commerce. <i>IEEE</i>. 2017. <a target=_blank href="https://doi.org/10.1109/ICSAW.2017.11">Publisher Full Text</a> <br><a name=rep-ref-62873-5></a>5. Tools Submission Portal. <i>SPEC Research Group</i>. <a target=_blank href="https://research.spec.org/tools/submission.html">Reference Source</a> <br></div> <hr class="c-hr c-hr--low u-mb--2"> <h5 class="t-body u-weight--md u-mt--0 u-mb--1/2">Competing Interests</h5> <p class="u-mt--0 u-mb--2 u-italic">No competing interests were disclosed.</p> <hr class="c-hr c-hr--low u-mb--2"> <h5 class="t-body u-weight--md u-mt--0 u-mb--1/2">Reviewer Expertise</h5> <p class="u-mb--2 u-mt--0">Software Engineering</p> <hr class="c-hr c-hr--low u-mb--3"> <p class="u-mb--3 u-mt--0"> <strong> I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. </strong> </p> <div class="_mdl-layout u-mb--3"> <a href="/login?originalPath=/articles/9-295/v2&scrollTo=referee-response-62873" class="u-1/1 mdl-button mdl-js-button mdl-button--outline mdl-button--multi-line mdl-button--colored mdl-button--white-ish" data-test-id=article_referee-report_respond><i class="material-icons u-ib u-middle">reply</i><span class="p-article__sidebar-report-respond-text u-ib u-middle">Respond to this report</span></a> </div> <hr class="c-hr c-hr--low c-hr--md u-mb--3"> <section id=report-responses-62873> <h4 class="t-h3 u-weight--md u-mb--2 u-mt--0">Responses <span class="u-weight--rg u-black--medium t-body">(1)</span></h4> <span class="c-hr c-hr--low u-mb--3"></span> <article id=referee-report-comment-6209 class="c-comment js-referee-report-comment " data-commentid=6209 data-reportid=62873 data-test-id=article_referee_report_comment> <header> <div class="c-comment__role t-caption u-black--medium" data-test-id=article_referee-report-comment_roles> <span class="u-weight--md u-upper u-ib u-mr--1/2">Author Response</span> </div> <div class="c-comment__date u-black--medium t-caption">26 Jan 2021</div> <h5 class="c-comment__name t-body u-weight--rg u-mt--0 u-mb--1" data-test-id=article_referee-report-comment_name> <strong class=u-weight--md>Axel Loewe, </strong><span class="c-comment__affiliations u-weight--rg">Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany</span> </h5> <div class="c-comment__body u-mb--1"> <div class="c-read-more js-read-more t-body" data-lines=4> <div class="c-read-more__content js-read-more-content "> <div class=u-mb--1 data-test-id=article_referee-report-comment_text data-commentid=6209><i>Thank you for the thorough review and constructive feedback regarding our manuscript. Below, we address the issues raised by you point-by-point. Our responses are set in italics.</i><br/> <br/> Concerning the statement “In order to support research, a sustainable software must be correct”, I suggest to include a short discussion of the test oracle problem for scientific software (see for instance <a href="https://doi.org/10.1109/SECSE.2013.66150991">https://doi.org/10.1109/SECSE.2013.66150991</a>).<br/> <i>We thank the reviewer for this suggestion, and included a brief discussion of the test oracle problem as suggested, and additionally of further challenges to verification and validation, such as large configuration spaces and heterogeneous data (as discussed in e.g. <a href="https://doi.org/10.1109/SE4Science.2019.00010">https://doi.org/10.1109/SE4Science.2019.00010</a>), and have suggested to implement the solutions mentioned in the literature.</i><br/> <br/> Concerning the discussion of “The list of criteria presented in this section could be the basis for a structured review process…” I suggest to include two additional initiatives for software review. The first is artifact evaluation in computer science conferences (the process is explained in https://doi.org/10.1515/itit-2019-00402). The second is the SPEC Research Group’s review process of tools for quantitative system evaluation and analysis (<a href="https://research.spec.org/tools/submission.html">https://research.spec.org/tools/submission.html</a>).<br/> <i>We thank the reviewer for this suggestion and included the artifact review approach in the introduction to the criteria section. The aspects of repeatability, reproducibility, and replicability are aimed more at the results of computational research rather than research software itself, we feel. Therefore, we didn’t include specific criteria in the list suggested to be used when evaluating research software for sustained funding.<br/> While the SPEC submission process is very clear, we could not find any concrete criteria applied during the review (except for requirements regarding the license).</i><br/> <br/> The authors write “We also argue that truly sustainable research software must ideally be published under a Free/Libre Open Source Software (FLOSS) license, and follow an open development model…” what I fully support (see for instance https://doi.org/10.1515/itit-2019-00402). However, later under the section heading “Legal aspects” this requirement is thwarted. I fully agree that legal aspects have to be considered, but the general bias of this section seems to be on commercial licensing of research software. For instance, the decision tree in Figure 1 starts with the question “Licensing planned?”. I assume that commercial licensing is meant, but this is not clear since the figures are not explained in the paper. Instead, the process should start with open sourcing the software. If licenses such as Apache or MIT are applied, the research institutions may later still commercialize the software if appropriate. Such open source licensing is also beneficial for start-ups that intend to provide professional services for the software. <br/> My experience with technology transfer units of German universities and research institutes is that they do not understand the ideas of open source business models (see for instance https://doi.org/10.1109/MC.2019.28981633). Their focus is on patents and commercializing licenses, sometimes also on start-ups. Conversely, in the software industry, one major motivation for open sourcing software is on improving the quality of software. I cite from https://doi.org/10.1109/ICSAW.2017.114 : “the open-source approach has some psychological effects: Developers show a tendency to apply higher quality standards if they know that the code will be publicly available.” For sustainability, quality is an important property of software.<br/> <i>As further detailed below, we have moved the decision trees out of this manuscript as we see the problems and agree to your arguments. We like your suggested aspect of commercialization of FLOSS licensed software and included this aspect in the manuscript.</i><br/> <br/> The Figures 1-4 do more harm than good. They are daunting to researchers who intend to publish their code open source. These figures should be removed from the paper, they are useless without proper explanation.<br/> I suggest that the authors focus in the present paper on their main message (request for funding a German Software Sustainability Institute, which I fully support). Figures 1-4 could be moved to a separate paper, enriched with proper explanation.<br/> <i>We thank the reviewer for this suggestion. Our initial thought was to place these decision trees in the supplemental material but did not realize that this is not the policy of f1000. The editorial team moved them into the main article. This is the reason why the decision trees appeared without additional information in the manuscript. We have now decided to take the Figures out and have published them together with documentation templates in a separate report via Zenodo unde a Creative Commons license, see <a href="https://zenodo.org/record/4327148#.X9n6ui337OQ">https://zenodo.org/record/4327148#.X9n6ui337OQ</a>. This report is now cited in the F1000 manuscript.</i></div> </div> <a href="#" class="c-read-more__toggle js-read-more-toggle t-caption"> <span class=c-read-more__expand>View more</span> <span class=c-read-more__contract>View less</span> </a> </div> </div> <div class=c-comment__footer> <h5 class="t-body u-weight--md u-mt--0 u-mb--1/2">Competing Interests</h5> <p class="u-mt--0 u-mb--1 u-italic" data-test-id=article_referee-report-comment_competing-interests>No competing interests were disclosed.</p> </div> <footer> <div class="o-actions o-actions--middle"> <div class=o-actions__primary> <a href="/login?originalPath=/articles/9-295/v2&scrollTo=referee-response-62873" class="js-referee-report-comment c-link--basic" data-test-id=article_referee-report-comment_respond><i class="material-icons c-link__icon">reply</i> <span class=c-link__text>Respond</span></a> </div> <div class=o-actions__secondary> <a href="mailto:research@f1000.com?subject=Report a concern (23224) " data-test-id=article_referee-report-comment_concern class="js-report-concern t-caption c-link c-link--inline">Report a concern</a> </div> </div> </footer> </header> </article> </section> <script type="text/template" data-template=refereeReportCitation-62873 id=refereeReportCitation-62873 class=u-hide> <div class="u-black--high"> <p id="referee-report-citation" data-test-id="report_copy-citation_text" >Hasselbring W. Peer Review Report For: An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action [version 2; peer review: 2 approved]</span>. <i>F1000Research</i> 2021, <b>9</b>:295 (<a class="new-orange" href="https://doi.org/10.5256/f1000research.25640.r62873" target="_blank">https://doi.org/10.5256/f1000research.25640.r62873)</a></p> <p class="c-modal__flush u-pl u-pr u-pt--3/2 u-pb--3/2 u-red u-bg--12"><strong>NOTE:</strong> it is important to ensure the information in square brackets after the title is included in this citation.</p> <p class="t-body">The direct URL for this report is: <br><a href="https://f1000research.com/articles/9-295/v1#referee-response-62873" target="_blank">https://f1000research.com/articles/9-295/v1#referee-response-62873</a></p> </div> </script> </aside> </div> </div> <script src="/js/shared_scripts/modal-dialogue.js"></script> <script src="/js/shared_scripts/read-more.js"></script> <script src="/js/article/article-router.js"></script> <script src="/js/article/article-sidebar.js"></script> <script src="/js/referee/new/referee_helpers.js"></script> <script src="/js/article/article-column-toggle.js"></script> </div> </div> </div> </main> <input type=hidden id=_articleVersionUrl value="https://f1000research.com/articles/9-295/v2/"> <div class=research-help id=about-referee-status> <div class="research-layout research-help-content about-referee-status"> <span class="close-research-help dark-cross" title=Close></span> Alongside their report, reviewers assign a status to the article: <div class="cf research-help-row"> <span class="f1r-icon icon-86_approved status-green smaller" title=Approved></span> <span class=research-help-text>Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested</span> </div> <div class="cf research-help-row"> <span class="f1r-icon icon-87_approved_reservations status-green smaller" title="Approved with Reservations"></span> <span class=research-help-text>Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit. </span> </div> <div class="cf research-help-row"> <span class="f1r-icon icon-88_not_approved status-red small" title="Not Approved"></span> <span class=research-help-text>Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions</span> </div> </div> </div> <div id=datasets-info class=is-hidden> </div> <div class=article-interactive-content-container style="display: none;"> <a name=f-template class=n-a></a> <div class="interactive-content-wrapper padding-20"> <img src="" class=interactive-content-image title="Open the interactive image display"> <div class=interactive-content-title></div> <div class=interactive-content-text></div> <div class="f1r-article-desk interactive-content-ribbon" data-interactive-content-type=R-Script> <div class=interactive-content-label>Adjust parameters to alter display</div> <div class=interactive-content-button></div> </div> <div class="f1r-article-mobile mobile-interactive-note"> View on desktop for interactive features <img src="/img/icon/interactive_content.png" class="float-right margin-right-40"/> </div> <div class=clearfix></div> </div> </div> <div id=article-interactive-omero-container class=article-interactive-omero-container style="display: none;"> <div class=interactive-content-wrapper> <div class="interactive-omero-button omero-content" title="Open the interactive content window." data-interactive-content-type=Omero></div> <div class=has-interactive-content-image> <span class=box-arrow></span> <span class=box-middle>Includes Interactive Elements</span> <span class=box-end></span> </div> <div class="fig panel clearfix" style="margin: 0; padding-bottom: 20px;"> <a name=templatelink class=n-a></a> <a target=_blank href="" class=link-for-omero-image> <img src="" class=interactive-omero-image title="Open the image display window."> </a> <div class=caption> <div class=interactive-content-title></div> <div class=interactive-content-text></div> </div> <div class="is-hidden omero-image-list"></div> </div> <div class="f1r-article-mobile mobile-interactive-note omero"> View on desktop for interactive features <img src="/img/icon/interactive_content.png" class="float-right margin-right-40"/> </div> <div class=clearfix></div> </div> </div> <div class="modal-window-wrapper is-hidden"> <div id=conflicts-interests class="modal-window padding-20"> <div class=modal-window__content> <h2 class=h2-title>Competing Interests Policy</h2> <p> Provide sufficient details of any financial or non-financial competing interests to enable users to assess whether your comments might lead a reasonable person to question your impartiality. Consider the following examples, but note that this is not an exhaustive list: </p> <div class=heading5>Examples of 'Non-Financial Competing Interests'</div> <ol class="numbered-list no-padding"> <li class=standard-padding>Within the past 4 years, you have held joint grants, published or collaborated with any of the authors of the selected paper.</li> <li class=standard-padding>You have a close personal relationship (e.g. parent, spouse, sibling, or domestic partner) with any of the authors.</li> <li class=standard-padding>You are a close professional associate of any of the authors (e.g. scientific mentor, recent student).</li> <li class=standard-padding>You work at the same institute as any of the authors.</li> <li class=standard-padding>You hope/expect to benefit (e.g. favour or employment) as a result of your submission.</li> <li class=standard-padding>You are an Editor for the journal in which the article is published.</li> </ol> <div class="heading5 padding-top">Examples of 'Financial Competing Interests'</div> <ol class="numbered-list no-padding"> <li class=standard-padding>You expect to receive, or in the past 4 years have received, any of the following from any commercial organisation that may gain financially from your submission: a salary, fees, funding, reimbursements.</li> <li class=standard-padding>You expect to receive, or in the past 4 years have received, shared grant support or other funding with any of the authors.</li> <li class=standard-padding>You hold, or are currently applying for, any patents or significant stocks/shares relating to the subject matter of the paper you are commenting on.</li> </ol> </div> <div class="f1r-article-mobile research-layout"> <span class=modal-window-close-button></span> </div> </div> </div> <div class="o-modal c-email-alert-popup js-email-alert-popup is-hidden"> <div class="o-modal__body js-modal-body c-email-alert-popup__inner"> <h3 class=c-email-alert-popup__title>Stay Updated</h3> <p class=c-email-alert-popup__sub>Sign up for content alerts and receive a weekly or monthly email with all newly published articles</p> <p><a href="/register?originalPath=" class="c-email-alert-popup__lnk c-email-alert-popup__button js-email-alert-popup-action">Register with F1000Research</a></p> <p>Already registered? <a href="/login?originalPath=" class="c-email-alert-popup__lnk js-email-alert-popup-action">Sign in</a></p> <p class=c-email-alert-popup__footer><a class="c-email-alert-popup__lnk js-email-alert-popup-cancel" href="#">Not now, thanks</a></p> </div> </div> <div id=addCommentModal role=dialog aria-labelledby=addCommentModal_title aria-describedby=addCommentModal_description> <div class="c-modal js-modal is-closed p-article__add-comment-modal c-modal--xlarge js-article-comment-modal c-modal--scroll " lang=""> <aside class="c-modal__content u-black--high o-box u-pb--0 u-bg--2 c-modal--scroll-always "> <div class=c-modal__close> <button type=button class="c-button c-button--icon c-button--medium c-button--full@hover js-modal__close"><i class="c-button--icon__icon material-icons">close</i></button> </div> <div id=addCommentModal_description class="c-modal__description js-modal__content t-h4 u-mt--0 u-mb--2 u-black--medium"> <div class="js-add-comment-container t-caption u-black--high"> <div class=u-weight--bd>PLEASE NOTE</div> <div class=u-mt--1> <span class="red u-weight--bd">If you are an AUTHOR of this article,</span> please check that you signed in with the account associated with this article otherwise we cannot automatically identify your role as an author and your comment will be labelled as a “User Comment”. </div> <div class=u-mt--1> <span class="red u-weight--bd">If you are a REVIEWER of this article,</span> please check that you have signed in with the account associated with this article and then go to your <a href="/my/referee">account</a> to submit your report, please do not post your review here. </div> <div class="u-mt--1 u-mb--1"> If you do not have access to your original account, please <a href="mailto:research@f1000.com">contact us</a>. </div> <p class=no-top-margin>All commenters must hold a formal affiliation as per our <a href="/about/policies#commentspolicy" target=_blank>Policies</a>. The information that you give us will be displayed next to your comment.</p> <p>User comments must be in English, comprehensible and relevant to the article under discussion. We reserve the right to remove any comments that we consider to be inappropriate, offensive or otherwise in breach of the <a href="/about/legal/additionaltermsandconditions#usercomment" target=_blank>User Comment Terms and Conditions</a>. Commenters must not use a comment for personal attacks. When criticisms of the article are based on unpublished data, the data should be made available.</p> <div class="comments-note margin-bottom" id=accept-user-comments> <input class=js-add-comment-accept-terms type=checkbox id=acceptedTermsAndConditions name=acceptedTermsAndConditions> I accept the <a href="/about/legal/additionaltermsandconditions#usercomment" target=_blank> User Comment Terms and Conditions</a> <span class=required> </span> </div> <div class="default-error margin-top is-hidden comment-accept-conditions utac">Please confirm that you accept the User Comment Terms and Conditions.</div> <form class="research-layout registration-form u-mb--2 comment-affiliations-form js-affiliations-form" id=comment_affiliation_1> <div class="u-mb--1 u-mt--2"> <strong>Affiliation</strong> </div> <input type=hidden class=js-affiliation-id value=""> <div class="form-field c-affiliation-clear-wrapper"> <input type=text id=comment_affiliation_1_institution name=institution class="form-input-field check-xss js-affiliation-institution" placeholder="Institution *" autocomplete=off /> <a class=c-affiliation-clear>✕</a> <div class="form-error-message-js space-above space-below"> <div class="o-flex o-flex--space-between o-flex--cross-center _mdl-layout"> <div class=o-flex__item> <span class=message></span> </div> <div class=o-flex__item> <button type=button class="mdl-button mdl-js-button mdl-js-ripple-effect mdl-button--icon mdl-button--mini-icon mdl-button--secondary" data-enabled-title="Click to retry search"> <i class=material-icons>refresh</i> </button> </div> </div> </div> <div class="default-error margin-top is-hidden comment-enter-institution institution">Please enter your institution.</div> <input type=hidden class=js-affiliation-institution-id value=""> </div> <div class="margin-top margin-bottom"> <i><b>Note:</b> To add your institution or organisation, start typing the name and then select the correct name from the list. Where applicable, the name will appear in both the original language and in English. Do not paste in the name. If the name does not appear in the drop-down list, we will display the information you have entered.</i> </div> <div class="form-field c-affiliation-clear-wrapper"> <input id=comment_affiliation_1_department type=text name=department class="form-input-field check-xss js-affiliation-department" placeholder=Department> <a class=c-affiliation-clear>✕</a> <div class="form-error-message-js space-above space-below"> <div class="o-flex o-flex--space-between o-flex--cross-center _mdl-layout"> <div class=o-flex__item> <span class=message></span> </div> <div class=o-flex__item> <button type=button class="mdl-button mdl-js-button mdl-js-ripple-effect mdl-button--icon mdl-button--mini-icon mdl-button--secondary" data-enabled-title="Click to retry search"> <i class=material-icons>refresh</i> </button> </div> </div> </div> <input type=hidden class=js-affiliation-department-id value=""> </div> <div class=form-field> <input type=text name=place class="form-input-field check-xss js-add-comment-place" placeholder="Town/City"> </div> <div class=form-field> <input type=text name=state class="form-input-field check-xss js-add-comment-state" placeholder=State> </div> <div class=form-field> <div class="form-input-wrapper hundred-percent-wide"> <div class="new-select-standard-wrapper inline-display heading10"> <select name=countryCode class="form-select-menu hundred-percent-wide js-add-comment-country"> <option value=-1>Country/Region *</option> <option value=US data-country-code=US>USA</option> <option value=GB data-country-code=GB>UK</option> <option value=CA data-country-code=CA>Canada</option> <option value=CN data-country-code=CN>China</option> <option value=FR data-country-code=FR>France</option> <option value=DE data-country-code=DE>Germany</option> <optgroup label=-------------------------></optgroup> <option value=AF data-country-code=AF>Afghanistan</option> <option value=AX data-country-code=AX>Aland Islands</option> <option value=AL data-country-code=AL>Albania</option> <option value=DZ data-country-code=DZ>Algeria</option> <option value=AS data-country-code=AS>American Samoa</option> <option value=AD data-country-code=AD>Andorra</option> <option value=AO data-country-code=AO>Angola</option> <option value=AI data-country-code=AI>Anguilla</option> <option value=AQ data-country-code=AQ>Antarctica</option> <option value=AG data-country-code=AG>Antigua and Barbuda</option> <option value=AR data-country-code=AR>Argentina</option> <option value=AM data-country-code=AM>Armenia</option> <option value=AW data-country-code=AW>Aruba</option> <option value=AU data-country-code=AU>Australia</option> <option value=AT data-country-code=AT>Austria</option> <option value=AZ data-country-code=AZ>Azerbaijan</option> <option value=BS data-country-code=BS>Bahamas</option> <option value=BH data-country-code=BH>Bahrain</option> <option value=BD data-country-code=BD>Bangladesh</option> <option value=BB data-country-code=BB>Barbados</option> <option value=BY data-country-code=BY>Belarus</option> <option value=BE data-country-code=BE>Belgium</option> <option value=BZ data-country-code=BZ>Belize</option> <option value=BJ data-country-code=BJ>Benin</option> <option value=BM data-country-code=BM>Bermuda</option> <option value=BT data-country-code=BT>Bhutan</option> <option value=BO data-country-code=BO>Bolivia</option> <option value=BA data-country-code=BA>Bosnia and Herzegovina</option> <option value=BW data-country-code=BW>Botswana</option> <option value=BV data-country-code=BV>Bouvet Island</option> <option value=BR data-country-code=BR>Brazil</option> <option value=IO data-country-code=IO>British Indian Ocean Territory</option> <option value=VG data-country-code=VG>British Virgin Islands</option> <option value=BN data-country-code=BN>Brunei</option> <option value=BG data-country-code=BG>Bulgaria</option> <option value=BF data-country-code=BF>Burkina Faso</option> <option value=BI data-country-code=BI>Burundi</option> <option value=KH data-country-code=KH>Cambodia</option> <option value=CM data-country-code=CM>Cameroon</option> <option value=CA data-country-code=CA>Canada</option> <option value=CV data-country-code=CV>Cape Verde</option> <option value=KY data-country-code=KY>Cayman Islands</option> <option value=CF data-country-code=CF>Central African Republic</option> <option value=TD data-country-code=TD>Chad</option> <option value=CL data-country-code=CL>Chile</option> <option value=CN data-country-code=CN>China</option> <option value=CX data-country-code=CX>Christmas Island</option> <option value=CC data-country-code=CC>Cocos (Keeling) Islands</option> <option value=CO data-country-code=CO>Colombia</option> <option value=KM data-country-code=KM>Comoros</option> <option value=CG data-country-code=CG>Congo</option> <option value=CK data-country-code=CK>Cook Islands</option> <option value=CR data-country-code=CR>Costa Rica</option> <option value=CI data-country-code=CI>Cote d'Ivoire</option> <option value=HR data-country-code=HR>Croatia</option> <option value=CU data-country-code=CU>Cuba</option> <option value=CY data-country-code=CY>Cyprus</option> <option value=CZ data-country-code=CZ>Czech Republic</option> <option value=CD data-country-code=CD>Democratic Republic of the Congo</option> <option value=DK data-country-code=DK>Denmark</option> <option value=DJ data-country-code=DJ>Djibouti</option> <option value=DM data-country-code=DM>Dominica</option> <option value=DO data-country-code=DO>Dominican Republic</option> <option value=EC data-country-code=EC>Ecuador</option> <option value=EG data-country-code=EG>Egypt</option> <option value=SV data-country-code=SV>El Salvador</option> <option value=GQ data-country-code=GQ>Equatorial Guinea</option> <option value=ER data-country-code=ER>Eritrea</option> <option value=EE data-country-code=EE>Estonia</option> <option value=ET data-country-code=ET>Ethiopia</option> <option value=FK data-country-code=FK>Falkland Islands</option> <option value=FO data-country-code=FO>Faroe Islands</option> <option value=FM data-country-code=FM>Federated States of Micronesia</option> <option value=FJ data-country-code=FJ>Fiji</option> <option value=FI data-country-code=FI>Finland</option> <option value=FR data-country-code=FR>France</option> <option value=GF data-country-code=GF>French Guiana</option> <option value=PF data-country-code=PF>French Polynesia</option> <option value=TF data-country-code=TF>French Southern Territories</option> <option value=GA data-country-code=GA>Gabon</option> <option value=GE data-country-code=GE>Georgia</option> <option value=DE data-country-code=DE>Germany</option> <option value=GH data-country-code=GH>Ghana</option> <option value=GI data-country-code=GI>Gibraltar</option> <option value=GR data-country-code=GR>Greece</option> <option value=GL data-country-code=GL>Greenland</option> <option value=GD data-country-code=GD>Grenada</option> <option value=GP data-country-code=GP>Guadeloupe</option> <option value=GU data-country-code=GU>Guam</option> <option value=GT data-country-code=GT>Guatemala</option> <option value=GG data-country-code=GG>Guernsey</option> <option value=GN data-country-code=GN>Guinea</option> <option value=GW data-country-code=GW>Guinea-Bissau</option> <option value=GY data-country-code=GY>Guyana</option> <option value=HT data-country-code=HT>Haiti</option> <option value=HM data-country-code=HM>Heard Island and Mcdonald Islands</option> <option value=VA data-country-code=VA>Holy See (Vatican City State)</option> <option value=HN data-country-code=HN>Honduras</option> <option value=HK data-country-code=HK>Hong Kong</option> <option value=HU data-country-code=HU>Hungary</option> <option value=IS data-country-code=IS>Iceland</option> <option value=IN data-country-code=IN>India</option> <option value=ID data-country-code=ID>Indonesia</option> <option value=IR data-country-code=IR>Iran</option> <option value=IQ data-country-code=IQ>Iraq</option> <option value=IE data-country-code=IE>Ireland</option> <option value=IL data-country-code=IL>Israel</option> <option value=IT data-country-code=IT>Italy</option> <option value=JM data-country-code=JM>Jamaica</option> <option value=JP data-country-code=JP>Japan</option> <option value=JE data-country-code=JE>Jersey</option> <option value=JO data-country-code=JO>Jordan</option> <option value=KZ data-country-code=KZ>Kazakhstan</option> <option value=KE data-country-code=KE>Kenya</option> <option value=KI data-country-code=KI>Kiribati</option> <option value=XK data-country-code=XK>Kosovo (Serbia and Montenegro)</option> <option value=KW data-country-code=KW>Kuwait</option> <option value=KG data-country-code=KG>Kyrgyzstan</option> <option value=LA data-country-code=LA>Lao People's Democratic Republic</option> <option value=LV data-country-code=LV>Latvia</option> <option value=LB data-country-code=LB>Lebanon</option> <option value=LS data-country-code=LS>Lesotho</option> <option value=LR data-country-code=LR>Liberia</option> <option value=LY data-country-code=LY>Libya</option> <option value=LI data-country-code=LI>Liechtenstein</option> <option value=LT data-country-code=LT>Lithuania</option> <option value=LU data-country-code=LU>Luxembourg</option> <option value=MO data-country-code=MO>Macao</option> <option value=MG data-country-code=MG>Madagascar</option> <option value=MW data-country-code=MW>Malawi</option> <option value=MY data-country-code=MY>Malaysia</option> <option value=MV data-country-code=MV>Maldives</option> <option value=ML data-country-code=ML>Mali</option> <option value=MT data-country-code=MT>Malta</option> <option value=MH data-country-code=MH>Marshall Islands</option> <option value=MQ data-country-code=MQ>Martinique</option> <option value=MR data-country-code=MR>Mauritania</option> <option value=MU data-country-code=MU>Mauritius</option> <option value=YT data-country-code=YT>Mayotte</option> <option value=MX data-country-code=MX>Mexico</option> <option value=UM data-country-code=UM>Minor Outlying Islands of the United States</option> <option value=MD data-country-code=MD>Moldova</option> <option value=MC data-country-code=MC>Monaco</option> <option value=MN data-country-code=MN>Mongolia</option> <option value=ME data-country-code=ME>Montenegro</option> <option value=MS data-country-code=MS>Montserrat</option> <option value=MA data-country-code=MA>Morocco</option> <option value=MZ data-country-code=MZ>Mozambique</option> <option value=MM data-country-code=MM>Myanmar</option> <option value=NA data-country-code=NA>Namibia</option> <option value=NR data-country-code=NR>Nauru</option> <option value=NP data-country-code=NP>Nepal</option> <option value=AN data-country-code=AN>Netherlands Antilles</option> <option value=NC data-country-code=NC>New Caledonia</option> <option value=NZ data-country-code=NZ>New Zealand</option> <option value=NI data-country-code=NI>Nicaragua</option> <option value=NE data-country-code=NE>Niger</option> <option value=NG data-country-code=NG>Nigeria</option> <option value=NU data-country-code=NU>Niue</option> <option value=NF data-country-code=NF>Norfolk Island</option> <option value=KP data-country-code=KP>North Korea</option> <option value=MK data-country-code=MK>North Macedonia</option> <option value=MP data-country-code=MP>Northern Mariana Islands</option> <option value=NO data-country-code=NO>Norway</option> <option value=OM data-country-code=OM>Oman</option> <option value=PK data-country-code=PK>Pakistan</option> <option value=PW data-country-code=PW>Palau</option> <option value=PS data-country-code=PS>Palestinian Territory</option> <option value=PA data-country-code=PA>Panama</option> <option value=PG data-country-code=PG>Papua New Guinea</option> <option value=PY data-country-code=PY>Paraguay</option> <option value=PE data-country-code=PE>Peru</option> <option value=PH data-country-code=PH>Philippines</option> <option value=PN data-country-code=PN>Pitcairn</option> <option value=PL data-country-code=PL>Poland</option> <option value=PT data-country-code=PT>Portugal</option> <option value=PR data-country-code=PR>Puerto Rico</option> <option value=QA data-country-code=QA>Qatar</option> <option value=RE data-country-code=RE>Reunion</option> <option value=RO data-country-code=RO>Romania</option> <option value=RU data-country-code=RU>Russian Federation</option> <option value=RW data-country-code=RW>Rwanda</option> <option value=SH data-country-code=SH>Saint Helena</option> <option value=KN data-country-code=KN>Saint Kitts and Nevis</option> <option value=LC data-country-code=LC>Saint Lucia</option> <option value=PM data-country-code=PM>Saint Pierre and Miquelon</option> <option value=VC data-country-code=VC>Saint Vincent and the Grenadines</option> <option value=WS data-country-code=WS>Samoa</option> <option value=SM data-country-code=SM>San Marino</option> <option value=ST data-country-code=ST>Sao Tome and Principe</option> <option value=SA data-country-code=SA>Saudi Arabia</option> <option value=SN data-country-code=SN>Senegal</option> <option value=RS data-country-code=RS>Serbia</option> <option value=SC data-country-code=SC>Seychelles</option> <option value=SL data-country-code=SL>Sierra Leone</option> <option value=SG data-country-code=SG>Singapore</option> <option value=SK data-country-code=SK>Slovakia</option> <option value=SI data-country-code=SI>Slovenia</option> <option value=SB data-country-code=SB>Solomon Islands</option> <option value=SO data-country-code=SO>Somalia</option> <option value=ZA data-country-code=ZA>South Africa</option> <option value=GS data-country-code=GS>South Georgia and the South Sandwich Is</option> <option value=KR data-country-code=KR>South Korea</option> <option value=SS data-country-code=SS>South Sudan</option> <option value=ES data-country-code=ES>Spain</option> <option value=LK data-country-code=LK>Sri Lanka</option> <option value=SD data-country-code=SD>Sudan</option> <option value=SR data-country-code=SR>Suriname</option> <option value=SJ data-country-code=SJ>Svalbard and Jan Mayen</option> <option value=SZ data-country-code=SZ>Swaziland</option> <option value=SE data-country-code=SE>Sweden</option> <option value=CH data-country-code=CH>Switzerland</option> <option value=SY data-country-code=SY>Syria</option> <option value=TW data-country-code=TW>Taiwan</option> <option value=TJ data-country-code=TJ>Tajikistan</option> <option value=TZ data-country-code=TZ>Tanzania</option> <option value=TH data-country-code=TH>Thailand</option> <option value=GM data-country-code=GM>The Gambia</option> <option value=NL data-country-code=NL>The Netherlands</option> <option value=TL data-country-code=TL>Timor-Leste</option> <option value=TG data-country-code=TG>Togo</option> <option value=TK data-country-code=TK>Tokelau</option> <option value=TO data-country-code=TO>Tonga</option> <option value=TT data-country-code=TT>Trinidad and Tobago</option> <option value=TN data-country-code=TN>Tunisia</option> <option value=TR data-country-code=TR>Turkey</option> <option value=TM data-country-code=TM>Turkmenistan</option> <option value=TC data-country-code=TC>Turks and Caicos Islands</option> <option value=TV data-country-code=TV>Tuvalu</option> <option value=GB data-country-code=GB>UK</option> <option value=US data-country-code=US>USA</option> <option value=UG data-country-code=UG>Uganda</option> <option value=UA data-country-code=UA>Ukraine</option> <option value=AE data-country-code=AE>United Arab Emirates</option> <option value=VI data-country-code=VI>United States Virgin Islands</option> <option value=UY data-country-code=UY>Uruguay</option> <option value=UZ data-country-code=UZ>Uzbekistan</option> <option value=VU data-country-code=VU>Vanuatu</option> <option value=VE data-country-code=VE>Venezuela</option> <option value=VN data-country-code=VN>Vietnam</option> <option value=WF data-country-code=WF>Wallis and Futuna</option> <option value=ZZ data-country-code=ZZ>West Bank and Gaza Strip</option> <option value=EH data-country-code=EH>Western Sahara</option> <option value=YE data-country-code=YE>Yemen</option> <option value=ZM data-country-code=ZM>Zambia</option> <option value=ZW data-country-code=ZW>Zimbabwe</option> </select> </div> </div> <div class="default-error margin-top is-hidden comment-enter-country country">Please select your country/region.</div> </div> </form> <textarea data-test-id=article_add-comment_comment id=new-comment-report name=new-comment class="js-add-comment-comment comment margin-bottom margin-top js-ckeditor" data-ckeditor-config=submissionComment></textarea> <div class="default-error margin-top comment-enter-text comment-error is-hidden ">You must enter a comment.</div> <label class="comments-note u-mt--2 u-mb--2" for=competingInterests_2> <div class="u-mb--1 u-mt--2"><strong data-test-id=article_report-add-comment_competing-interests-title>Competing Interests</strong></div> <p class="u-mb--2 u-mt--0" data-test-id=article_report-add-comment_competing-interests-description>Please disclose any <a href="#article-competing-intersts-policy" class=js-modal-competing-intersts-toggle>competing interests</a> that might be construed to influence your judgment of the article's or peer review report's validity or importance.</p> </label> <div id=article-competing-intersts-policy class=js-article-competing-interests-policy style="display: none;"> <h2 class="h2-title u-mt--0 u-pt--0">Competing Interests Policy</h2> <p> Provide sufficient details of any financial or non-financial competing interests to enable users to assess whether your comments might lead a reasonable person to question your impartiality. Consider the following examples, but note that this is not an exhaustive list: </p> <div class=heading5>Examples of 'Non-Financial Competing Interests'</div> <ol class="numbered-list no-padding"> <li class=standard-padding>Within the past 4 years, you have held joint grants, published or collaborated with any of the authors of the selected paper.</li> <li class=standard-padding>You have a close personal relationship (e.g. parent, spouse, sibling, or domestic partner) with any of the authors.</li> <li class=standard-padding>You are a close professional associate of any of the authors (e.g. scientific mentor, recent student).</li> <li class=standard-padding>You work at the same institute as any of the authors.</li> <li class=standard-padding>You hope/expect to benefit (e.g. favour or employment) as a result of your submission.</li> <li class=standard-padding>You are an Editor for the journal in which the article is published.</li> </ol> <div class="heading5 padding-top">Examples of 'Financial Competing Interests'</div> <ol class="numbered-list no-padding"> <li class=standard-padding>You expect to receive, or in the past 4 years have received, any of the following from any commercial organisation that may gain financially from your submission: a salary, fees, funding, reimbursements.</li> <li class=standard-padding>You expect to receive, or in the past 4 years have received, shared grant support or other funding with any of the authors.</li> <li class=standard-padding>You hold, or are currently applying for, any patents or significant stocks/shares relating to the subject matter of the paper you are commenting on.</li> </ol> </div> <div id=competingInterests_2 class="c-inline-editor js-inline-editor js-form_input u-bb--all u-mb--2 js-comment-competing-interests" data-name=competing-interests data-rows=3> <textarea id=competingInterests_2_input name=competing-interests placeholder=" " required=false class="u-hide-visually js-inline-editor_input" tabindex=-1></textarea> <div class="c-inline-editor__editor js-inline-editor_editor o-box c-box o-box--tiny u-bg--11" data-target=competingInterests_2_input role=textbox required=false data-placeholder=" " contenteditable=true></div> <span class="c-inline-editor__error js-inline-editor-message">Please state your competing interests</span> </div> <div data-test-id=article_add-comment_saved class="green-message comments is-hidden comment-is-saved">The comment has been saved.</div> <div data-test-id=article_add-comment_error class="default-error comments is-hidden comment-not-added">An error has occurred. Please try again.</div> <div class=clearfix></div> <div class=js-hook></div> </div> <div class="c-modal__extra-message js-modal__extra-message t-h4 u-black--medium"></div></div> <div class="c-modal__actions o-box__actions"> <a href="#" data-test-id=article_add-comment_cancel class="c-button c-button--full js-modal__close c-button--secondary">Cancel</a> <a href="#" data-test-id=article_add-comment_post class="c-button c-button--full js-modal__confirm c-button--primary">Post</a> </div> </aside> </div> </div> <script src="/js/article/alert-signup.js"></script> <script type='text/javascript'> var lTitle = "An environment for sustainable research software...".replace("'", ''); var linkedInUrl = "http://www.linkedin.com/shareArticle?url=https://f1000research.com/articles/9-295/v2" + "&title=" + encodeURIComponent(lTitle) + "&summary=" + encodeURIComponent('Read the article by '); var deliciousUrl = "https://del.icio.us/post?url=https://f1000research.com/articles/9-295/v2&title=" + encodeURIComponent(lTitle); var redditUrl = "http://reddit.com/submit?url=https://f1000research.com/articles/9-295/v2" + "&title=" + encodeURIComponent(lTitle); linkedInUrl += encodeURIComponent('Anzt H et al.'); var offsetTop = /chrome/i.test( navigator.userAgent ) ? 4 : -10; var addthis_config = { ui_offset_top: offsetTop, services_compact : "facebook,twitter,www.linkedin.com,www.mendeley.com,reddit.com", services_expanded : "facebook,twitter,www.linkedin.com,www.mendeley.com,reddit.com", services_custom : [ { name: "LinkedIn", url: linkedInUrl, icon:"/img/icon/at_linkedin.svg" }, { name: "Mendeley", url: "http://www.mendeley.com/import/?url=https://f1000research.com/articles/9-295/v2/mendeley", icon:"/img/icon/at_mendeley.svg" }, { name: "Reddit", url: redditUrl, icon:"/img/icon/at_reddit.svg" }, ] }; var addthis_share = { url: "https://f1000research.com/articles/9-295", templates : { twitter : "An environment for sustainable research software in Germany and.... Anzt H et al., published by " + "@F1000Research" + ", https://f1000research.com/articles/9-295/v2" } }; if (typeof(addthis) != "undefined"){ addthis.addEventListener('addthis.ready', checkCount); addthis.addEventListener('addthis.menu.share', checkCount); } $(".f1r-shares-twitter").attr("href", "https://twitter.com/intent/tweet?text=" + addthis_share.templates.twitter); $(".f1r-shares-facebook").attr("href", "https://www.facebook.com/sharer/sharer.php?u=" + addthis_share.url); $(".f1r-shares-linkedin").attr("href", addthis_config.services_custom[0].url); $(".f1r-shares-reddit").attr("href", addthis_config.services_custom[2].url); $(".f1r-shares-mendelay").attr("href", addthis_config.services_custom[1].url); function checkCount(){ setTimeout(function(){ $(".addthis_button_expanded").each(function(){ var count = $(this).text(); if (count !== "" && count != "0") $(this).removeClass("is-hidden"); else $(this).addClass("is-hidden"); }); }, 1000); } </script> <div id=citeReportModal role=dialog aria-labelledby=citeReportModal_title aria-describedby=citeReportModal_description> <div class="c-modal js-modal is-closed c-modal--large js-cite-report-modal " lang=""> <aside class="c-modal__content u-black--high o-box u-pb--0 u-black--high "> <div class=c-modal__close> <button type=button class="c-button c-button--icon c-button--medium c-button--full@hover js-modal__close"><i class="c-button--icon__icon material-icons">close</i></button> </div> <h1 id=citeReportModal_title class="c-modal__title t-h3 u-mt--0 u-mb--2 u-weight--md">How to cite this report</h1> <div id=citeReportModal_description class="c-modal__description js-modal__content t-h4 u-mt--0 u-mb--2 u-black--medium"> <div id="" class=js-report-citation-container>{{reportCitation}}</div> <div class="c-modal__extra-message js-modal__extra-message t-h4 u-black--medium"></div></div> <div class="c-modal__actions o-box__actions"> <a href="#" class="c-button c-button--full js-modal__close c-button--secondary">Cancel</a> <a href="#" title="Copy the current citation details to the clipboard." data-clipboard-target="#referee-report-citation" data-test-id=report_copy-citation_button class="c-button c-button--full js-modal__confirm c-button--primary js-clipboard c-mini-tooltip--above">Copy Citation Details</a> </div> </aside> </div> </div> <script src="/js/referee/new/referee_validators.js"></script> <script src="/js/referee/new/referee_helpers.js"></script> <script src="/js/referee/new/referee_checkbox-input.js"></script> <script src="/js/referee/new/referee_inline-editor.js"></script> <script> $(function(){R.ui.buttonDropdowns('.dropdown-for-downloads');}); $(function(){R.ui.toolbarDropdowns('.toolbar-dropdown-for-downloads');}); </script> <script src="/js/article/track_article.js" type="text/javascript"></script> <script type="text/javascript"> $.get("/articles/acj/23224/31504") </script> <script type="text/javascript" src="https://d1bxh8uas1mnw7.cloudfront.net/assets/embed.js"></script> <script type="text/javascript" src="/js/app/messenger.js"></script> <script type="text/javascript" src="/js/article/article_mobiles.js"></script> <script src="/js/vendor/clipboard.min.js"></script> <script src="/js/shared_scripts/modal-dialogue.js"></script> <script src="/js/shared_scripts/clipboard.js"></script> <script src="/js/article/thesaurus-terms-display.js"></script> <script> new F1000.Clipboard(); new F1000.ThesaurusTermsDisplay("articles", "article", "31504"); </script> <script> $(document).ready(function() { $( "#frame1" ).on('load', function() { var mydiv = $(this).contents().find("div"); var h = mydiv.height(); console.log(h) }); var tooltipLivingFigure = jQuery(".interactive-living-figure-label .icon-more-info"), titleLivingFigure = tooltipLivingFigure.attr("title"); tooltipLivingFigure.simpletip({ fixed: true, position: ["-115", "30"], baseClass: 'small-tooltip', content:titleLivingFigure + "<div class='tooltip-arrow'></div>" }); tooltipLivingFigure.removeAttr("title"); $("body").on("click", ".cite-living-figure", function(e) { e.preventDefault(); var ref = $(this).attr("data-ref"); $(this).closest(".living-figure-list-container").find("#" + ref).fadeIn(200); }); $("body").on("click", ".close-cite-living-figure", function(e) { e.preventDefault(); $(this).closest(".popup-window-wrapper").fadeOut(200); }); $(document).on("mouseup", function(e) { var metricsContainer = $(".article-metrics-popover-wrapper"); if (!metricsContainer.is(e.target) && metricsContainer.has(e.target).length === 0) { $(".article-metrics-close-button").click(); } }); var articleId = $('#articleId').val(); if($("#main-article-count-box").attachArticleMetrics) { $("#main-article-count-box").attachArticleMetrics(articleId, { articleMetricsView: true }); } }); var figshareWidget = $(".new_figshare_widget"); if (figshareWidget.length > 0) { window.figshare.load("f1000", function(Widget) { // Select a tag/tags defined in your page. In this tag we will place the widget. _.map(figshareWidget, function(el){ var widget = new Widget({ articleId: $(el).attr("figshare_articleId") //height:300 // this is the height of the viewer part. [Default: 550] }); widget.initialize(); // initialize the widget widget.mount(el); // mount it in a tag that's on your page // this will save the widget on the global scope for later use from // your JS scripts. This line is optional. //window.widget = widget; }); }); } </script> <template id=templateOverlay> <div class="o-overlay js-overlay is-hidden" hidden=hidden></div> </template><template id=templateModal> <div class="o-modal js-modal is-hidden" hidden=hidden> <div class="o-modal__close js-modal-close"></div> <div class="o-modal__body js-modal-body"></div> </div> </template><template id=templateConsole> <section class="c-console js-console"> <header class="c-console__hdr js-console-header"> <h1 class="c-console__ttl u-font-size--title js-console-title"></h1> </header> <div class="_mdl-layout c-console__fixed js-console-fixed"></div> <div class="_mdl-layout c-console__bdy js-console-body"></div> <div class="_mdl-layout c-console__message u-invisible js-console-message"> <div class="o-media o-media--center o-media--sm"> <i class="material-icons o-media__img js-message-img">close</i> <div class="mdl-typography--body-1 o-media__body js-message-body">Error</div> </div> </div> <footer class="_mdl-layout c-console__ftr o-flex o-flex--reverse js-console-footer"> <button type=button class="mdl-button mdl-js-button mdl-button--raised mdl-button--colored c-console__btn js-console-close">Close</button> <button type=button class="mdl-button mdl-js-button mdl-button--raised mdl-button--colored c-console__btn js-console-add">Add</button> <button type=button class="mdl-button mdl-js-button c-console__btn js-console-reset">Reset</button> </footer> </section> </template> <script src="/js/vendor/ckeditor/ckeditor.js"></script> <script src="/js/shared_scripts/extend.js"></script> <script src="/js/shared_scripts/helper.ckeditor.js"></script> <script> F1000.MICROSERVICES.AFFILIATION = ''; </script> <script src="/js/editorial_mgmt/utilities.js"></script> <script src="/js/shared_scripts/ajax.js"></script> <script src="/js/shared_scripts/overlay.js"></script> <script src="/js/shared_scripts/console.js"></script> <script src="/js/shared_scripts/modal.js"></script> <script src="/js/shared_scripts/confirm-action.js"></script> <script src="/js/shared_scripts/helper.security.js"></script> <script src="/js/shared_scripts/ringgold.js"></script> <script src="/js/shared_scripts/autocomplete.js"></script> <script src="/js/shared_scripts/affiliation-form.js"></script> <script> $(document).ready(function () { $('.js-affiliations-form').each((index, form) => { new AffiliationForm({ formId: form.id, institutionErrorSelector: '.comment-enter-institution', departmentErrorSelector: '.comment-enter-department', placeSelector: '.js-add-comment-place', stateSelector: '.js-add-comment-state', zipCodeSelector: '.js-add-comment-zipcode', countrySelector: '.js-add-comment-country', countryErrorSelector: '.comment-enter-country', }); }); }); </script> <script> $(document).ready(function () { var reportIds = { "62848": 0, "62849": 0, "78211": 32, "62850": 0, "62851": 0, "78212": 0, "62762": 0, "62763": 0, "62764": 0, "62765": 0, "62766": 0, "62872": 59, "62873": 101, }; $(".referee-response-container,.js-referee-report").each(function(index, el) { var reportId = $(el).attr("data-reportid"), reportCount = reportIds[reportId] || 0; $(el).find(".comments-count-container,.js-referee-report-views").html(reportCount); }); var uuidInput = $("#article_uuid"), oldUUId = uuidInput.val(), newUUId = "1cb1368a-9428-4397-b052-911f0a696bc0"; uuidInput.val(newUUId); $("a[href*='article_uuid=']").each(function(index, el) { var newHref = $(el).attr("href").replace(oldUUId, newUUId); $(el).attr("href", newHref); }); }); </script> </div> </div> <div class="o-page__footer sticky-email-wrapper"> <footer class="c-footer t-inverted"> <div class="o-wrapper"> <div class="o-layout"> <div class="o-layout__item u-mb--3"> <div class="c-branding c-branding--research"> <img src="/img/research/F1000Research_white.svg" alt="F1000Research"> </div> </div> <div class="o-layout__item u-1/3@md u-mb--3"> <span class="c-hr c-hr--thick c-hr--low u-mb--2"></span> <p class="t-h3 u-mt--0 u-mb--0">An innovative open access publishing platform offering rapid publication and open peer review, whilst supporting data deposition and sharing.</p> </div> <div class="o-layout__item u-2/3@md"> <span class="c-hr c-hr--thick c-hr--low u-mb--2"></span> <div class="o-layout"> <nav class="c-footer__nav"> <div class="o-layout__item u-3/5@sm u-mb--3"> <div class="o-columns o-columns--2"> <a href="/browse/articles" class="t-body c-footer__nav-item " >Browse</a> <a href="/gateways" class="t-body c-footer__nav-item " >Gateways</a> <a href="/collections" class="t-body c-footer__nav-item " >Collections</a> <a href="/about" class="t-body c-footer__nav-item " >How it Works</a> <a href="/contact" class="t-body c-footer__nav-item " >Contact</a> <a href="/developers" class="t-body c-footer__nav-item u-hide u-show@navbar" >For Developers</a> <a href="/about/legal/privacypolicy/cookies" class="t-body c-footer__nav-item " >Cookie Notice</a> <a href="/about/legal/privacypolicy" class="t-body c-footer__nav-item " >Privacy Notice</a> <a href="/published/rss" class="t-body c-footer__nav-item " title="RSS feed of published articles" >RSS</a> </div> </div> <div class="o-layout__item u-2/5@sm u-center u-right@sm u-mb--3"> <div class="u-hide u-show@lg"> <div class="_mdl-layout"> <a class="mdl-button mdl-js-button mdl-button--inverted mdl-button--no-shadow mdl-js-ripple-effect mdl-button--outline" href="/for-authors/publish-your-research" data-test-id="footer_submit_research" >Submit Your Research</a> </div> </div> </div> </nav> </div> </div> <div class="o-layout__item u-mb--2"> <div class="c-footer__share"> <div class="c-footer__share"> <span class="c-footer__share-icon" title="Open Access"> <span class="f1r-icon icon-100_open_access license-icon"></span> </span> <a class="c-footer__share-icon" href="//creativecommons.org/licenses" target="_blank" title="Creative Commons License CC-BY"> <span class="f1r-icon icon-116_cc license-icon license-icon-cc"></span> <span class="f1r-icon icon-117_ccby license-icon license-icon-cc"></span> </a> <a class="c-footer__share-icon" href="//creativecommons.org/about/cc0" target="_blank" title="Creative Commons License CC0"> <span class="f1r-icon icon-118_cco license-icon"></span> </a> </div> </div> </div> <div class="o-layout__item u-1/3@md u-mb--3"> <span class="c-hr c-hr--low u-mb--3"></span> <p class="c-footer__social u-mt--0 u-mb--0 u-white--low-med">Follow us <a href="https://www.facebook.com/F1000" target="_blank" class="c-footer__social-icon f1r-icon icon-55_footer_facebook"></a> <a href="https://twitter.com/#!/F1000Research" target="_blank" class="c-footer__social-icon f1r-icon icon-56_footer_twitter"></a> <a href="http://www.youtube.com/user/F1000research" target="_blank" class="c-footer__social-icon f1r-icon icon-57_footer_youtube"></a></p> </div> <div class="o-layout__item u-2/3@md u-right@md"> <span class="c-hr c-hr--low u-mb--3"></span> <p class="t-caption u-white--low-med">© 2012-2025 F1000 Research Ltd. ISSN 2046-1402 | <a href="/about/legal" class="copyrightLegal">Legal</a> | Partner of <a target="_blank" href="https://www.research4life.org/">Research4Life</a> • <a target="_blank" href="http://crossref.org/">CrossRef</a> • <a target="_blank" href="http://orcid.org/">ORCID</a> • <a target="_blank" href="http://www.fairsharing.org">FAIRSharing</a></p> </div> </div> </div> </footer> </div> </div> <script> R.templateTests.simpleTemplate = R.template('<p class="$variable.one">$text</p><p class="${variable.two}">$text</p><p class="$!variable.three">$text</p><p class="$!{variable.four}">$text</p><p class="${selector}.five">$text</p>'); R.templateTests.runTests(); var F1000platform = new F1000.Platform({ name: "f1000research", displayName: "F1000Research", hostName: "f1000research.com", id: "1", editorialEmail: "research@f1000.com", infoEmail: "info@f1000.com", usePmcStats: true }); $(function(){R.ui.dropdowns('.dropdown-for-authors, .dropdown-for-about, .dropdown-for-myresearch');}); // $(function(){R.ui.dropdowns('.dropdown-for-referees');}); $(document).ready(function () { if ($(".cookie-warning").is(":visible")) { $(".sticky").css("margin-bottom", "35px"); $(".devices").addClass("devices-and-cookie-warning"); } $(".cookie-warning .close-button").click(function (e) { $(".devices").removeClass("devices-and-cookie-warning"); $(".sticky").css("margin-bottom", "0"); }); $("#tweeter-feed .tweet-message").each(function (i, message) { var self = $(message); self.html(linkify(self.html())); }); $(".partner").on("mouseenter mouseleave", function() { $(this).find(".gray-scale, .colour").toggleClass("is-hidden"); }); }); </script> <div class="sign-in-popup"> <!-- <a href="#" class="sign-in shadow">Sign in <span class="sign-in-image-active"></span></a> --> <a href="#" class="sign-in ${locale}">Sign In <span class="arrow-closed sign-in-arrow-padding arrow-opened"></span></a> <div class="sign-in-form"> <form action="https://f1000research.com/j_spring_oauth_security_check" id="googleOAuth" method="post" target="_top" > <input class="target-field" type="hidden" name="target" value="/articles/9-295/v2"/> <input id="google-remember-me" name="_spring_security_oauth_remember_me" type="hidden" value="true"/> <input id="system-google" name="system" type="hidden" value="GOOGLE"/> </form> <form action="https://f1000research.com/j_spring_oauth_security_check" id="ECAuth" method="post" target="_top" > <input class="target-field" type="hidden" name="target" value="/articles/9-295/v2"/> <input id="ec-remember-me" name="_spring_security_oauth_remember_me" type="hidden" value="true"/> <input id="system-ec" name="system" type="hidden" value="EC"/> </form> <form action="https://f1000research.com/j_spring_oauth_security_check" id="facebookOAuth" method="post" target="_top" > <input class="target-field" type="hidden" name="target" value="/articles/9-295/v2"/> <input id="facebook-remember-me" name="_spring_security_oauth_remember_me" type="hidden" value="true"/> <input id="system-fb" name="system" type="hidden" value="FACEBOOK"/> </form> <form action="https://f1000research.com/j_spring_oauth_security_check" id="orcidOAuth" method="post" target="_top" > <input class="target-field" type="hidden" name="target" value="/articles/9-295/v2"/> <input id="orcid-remember-me" name="_spring_security_oauth_remember_me" type="hidden" value="true"/> <input id="system-orcid" name="system" type="hidden" value="ORCID"/> </form> <form id="sign-in-form" class="login-container" action="https://f1000research.com/login" method="post" name="f"> <div id="sign-in-form-gfb-popup"></div> <input class="target-field" type="hidden" name="target" value="/articles/9-295/v2"/> <input type="text" name="username" id="signin-email-box" class="sign-in-input" placeholder="Email address" autocomplete="email"> <input type="password" name="password" id="signin-password-box" class="sign-in-input" placeholder="Password" autocomplete="current-password"> <div class="sign-in-remember"> <div class="checkbox-wrapper"> <input type="checkbox" id="remember-me" name="remember_me" class="checkbox is-hidden"> </div> <span class="checkbox-label">Remember me</span> </div> <a href="#" class="sign-in-link" id="forgot-password-link">Forgotten your password?</a> <div class="sign-in-button-container margin-top margin-left-20 margin-bottom"> <button type="submit" id="sign-in-button" class="sign-in-buttons general-white-orange-button">Sign In</button> <button type="button" id="sign-in-cancel" class="sign-in-buttons sign-in-cancel-button margin-left">Cancel</button> <div class="clearfix"></div> </div> <div class="sign-in-error">Email or password not correct. Please try again</div> <div class="sign-in-loading">Please wait...</div> </form> <div class="forgot-password-container"> <script type="text/javascript"> $(function(){ // Note: All the setup needs to run against a name attribute and *not* the id due the clonish // nature of facebox... $("a[id=googleSignInButton]").click(function(event){ event.preventDefault(); $("input[id=oAuthSystem]").val("GOOGLE"); $("form[id=oAuthForm]").submit(); }); $("a[id=facebookSignInButton]").click(function(event){ event.preventDefault(); $("input[id=oAuthSystem]").val("FACEBOOK"); $("form[id=oAuthForm]").submit(); }); $("a[id=orcidSignInButton]").click(function(event){ event.preventDefault(); $("input[id=oAuthSystem]").val("ORCID"); $("form[id=oAuthForm]").submit(); }); }); </script> <span class="text first"> If you've forgotten your password, please enter your email address below and we'll send you instructions on how to reset your password. <p>The email address should be the one you originally registered with F1000.</p> </span> <input name="email" class="sign-in-input" id="email-forgot-password" type="text" placeholder="Email address"> <div class="forgot-password-email-error"> Email address not valid, please try again </div> <div class="forgot-password-google-email-error"> <p>You registered with F1000 via Google, so we cannot reset your password.</p> <p>To sign in, please click <a href="#" id="googleSignInButton">here</a>.</p> <p>If you still need help with your Google account password, please click <a href="https://www.google.com/accounts/recovery">here</a>.</p> </div> <div class="forgot-password-facebook-email-error"> <p>You registered with F1000 via Facebook, so we cannot reset your password.</p> <p>To sign in, please click <a href="#" id="facebookSignInButton">here</a>.</p> <p>If you still need help with your Facebook account password, please click <a href="https://www.facebook.com/recover/initiate">here</a>.</p> </div> <div class="clearfix"></div> <div class="forgot-password-captcha-error"> Code not correct, please try again </div> <div class="clearfix"></div> <div class="sign-in-button-container margin-left-20 margin-bottom"> <button type="button" id="sign-in-reset-password" class="sign-in-buttons general-white-orange-button">Reset password</button> <button type="button" id="forgot-password-cancel" class="sign-in-buttons sign-in-cancel-button margin-left">Cancel</button> <div class="clearfix"></div> </div> <span class="text last"> <a href="mailto:">Email us</a> for further assistance. </span> <form action="https://f1000research.com/j_spring_oauth_security_check" id="oAuthForm" method="post" target="_top"> <input class="target-field" type="hidden" name="target" value="/articles/9-295/v2"/> <input id="oAuthSystem" name="system" type="hidden"/> </form> <div class="forgot-password-server-error">Server error, please try again.</div> <div class="sign-in-success"> <p>If your email address is registered with us, we will email you instructions to reset your password.</p> <p>If you think you should have received this email but it has not arrived, please check your spam filters and/or contact <a href="mailto:"></a> for further assistance.</p> </div> <div class="sign-in-loading">Please wait...</div> </div> <div class="sign-in-form-register-section"> <div class="sign-in-button-container margin-left-20 margin-bottom"> <a href="/register" title="Register"><button type="button" id="sign-in-register-button" class="sign-in-buttons general-white-orange-button">Register</button></a> <div class="clearfix"></div> </div> </div> </div> </div> <script type="text/javascript"> $(document).ready(function () { signIn.createSignInAsRow($("#sign-in-form-gfb-popup")); $(".target-field").each(function () { var uris = $(this).val().split("/"); if (uris.pop() === "login") { $(this).val(uris.toString().replace(",","/")); } }); }); </script> <script type="text/javascript" src="/js/app/research.analytics.js"></script> <!-- Start of HubSpot Embed Code --> <script type="text/javascript" id="hs-script-loader" async defer src="//js.hs-scripts.com/4475190.js"></script> <!-- End of HubSpot Embed Code --> <script src="https://my.hellobar.com/4e0495c6f18cbd68731a1dc1978195a144e767ba.js" type="text/javascript" charset="utf-8" async="async"></script> </body> </html>