CINXE.COM
Infrastructure | Zenodo
<!doctype html> <html> <head> <meta charset="utf-8"> <meta property="og:image" content="https://blog.zenodo.org/static/img/logos/zenodo-gradient-1000.png" /> <meta property="og:title" content="Zenodo - Research. Shared." /> <meta property="og:type" content="website" /> <meta property="og:description" content="Zenodo is a free and open digital archive built by CERN and OpenAIRE, enabling researchers to share and preserve research output in any size, format and from all fields of research." /> <meta name="twitter:card" content="summary_large_image" /> <meta name="twitter:site" content="@zenodo_org" /> <meta name="twitter:creator" content="@zenodo_org" /> <meta name="twitter:title" content="Zenodo - Research. Shared." /> <meta name="twitter:description" content="Zenodo is a free and open digital archive built by CERN and OpenAIRE, enabling researchers to share and preserve research output in any size, format and from all fields of research." /> <meta name="twitter:image" content="https://blog.zenodo.org/static/img/logos/zenodo-gradient-1000.png" /> <link rel="stylesheet" href="../static/zenodo.css"> <link rel="stylesheet" href="../static/style.css"> <link rel="stylesheet" href="../static/font-awesome/css/font-awesome.min.css"> <link href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700,100,italic" rel="stylesheet"> <title>Infrastructure | Zenodo</title> <script src="https://code.jquery.com/jquery-3.1.1.min.js"></script> <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script> </head> <body> <header> <nav class="navbar navbar-default navbar-static-top"> <div class="container"> <div class="navbar-header"> <a href="https://zenodo.org"><img class="navbar-brand" src="/static/img/zenodo.svg" alt="Zenodo" /></a> <span class="logo-divider"></span> <span class="header-title" aria-label="header-title">About</span> </div> <div id="navbar" class="navbar-collapse collapse"> <ul class="nav navbar-nav navbar-right toplist"> <li><a href="https://about.zenodo.org">About</a></li> <li><a href="https://blog.zenodo.org">Blog</a></li> <li><a href="https://help.zenodo.org">Help</a></li> <li><a href="https://about.zenodo.org/projects">Projects</a></li> <li><a href="https://developers.zenodo.org">Developers</a></li> </ul> </div> </div> </nav> </header> <div class="communities"> <div class="header"> <div class="container"> <div class="row"> <div class="col-md-12 nav navbar-nav navbar subbar"> <li><a href="../">About</a></li> <li><a href="../privacy-policy/">Privacy policy</a></li> <li><a href="../cookie-policy/">Cookie policy</a></li> <li><a href="../terms/">Terms of use</a></li> <li><a href="../policies/">General policies</a></li> <li class="active"><a href="./">Infrastructure</a></li> <li><a href="../principles/">Principles</a></li> <li><a href="../roadmap/">Roadmap</a></li> <li><a href="../contact/">Contact</a></li> </div> </div> </div> </div> <div class="container body-container"> <div class="row"> <div class="col-md-12"> <h1>Infrastructure</h1> <p><hr /></p> <h2>Organisational</h2> <h4>Host institution</h4> <p>Zenodo is hosted by CERN which has existed since 1954 and currently has an experimental programme defined for the next 20+ years. CERN is a memory institution for High Energy Physics and renowned for its pioneering work in Open Access. Organisationally Zenodo is embedded in the IT Department, Collaboration Devices and Applications Group, Digital Repositories Section (IT-CDA-DR).</p> <p>Zenodo is offered by CERN as part of its mission to make available the results of its work (<a href="https://council.web.cern.ch/en/content/convention-establishment-european-organization-nuclear-research#2">CERN Convention, Article II, §1</a>).</p> <h4>Legal status</h4> <p>CERN is an intergovernmental organisation and has legal personality in the metropolitan territories of all CERN Member States (<a href="https://council.web.cern.ch/en/content/convention-establishment-european-organization-nuclear-research#9">CERN Convention, Article IX</a>) and enjoys the corresponding legal capacity under public international law.</p> <p>As an intergovernmental organization CERN enjoys certain privileges and immunities, including e.g. immunity from jurisdiction of the national courts to ensure our independence from individual Member States. This does not mean that CERN operate in some kind of legal vacuum as protocols requires that CERN settle its disputes by other means. Read more about CERN's legal status in the <a href="https://cds.cern.ch/record/1035135?ln=en">CERN Bulletin</a>.</p> <p>Legal documents:</p> <ul> <li><a href="https://council.web.cern.ch/en/content/convention-establishment-european-organization-nuclear-research">CERN Convention</a></li> <li><a href="https://cds.cern.ch/record/1035110/">Protocol on the privileges and immunities of the European Organization for Nuclear Research</a></li> </ul> <h4>Funding</h4> <p>Zenodo is funded by:</p> <ul> <li>European Commission via the <a href="https://www.openaire.eu">OpenAIRE</a> projects:<ul> <li>FP7: <a href="https://cordis.europa.eu/project/id/246686">OpenAIRE</a> (246686), <a href="https://cordis.europa.eu/project/id/283595">OpenAIREplus</a> (283595)</li> <li>Horizon 2020: <a href="https://doi.org/10.3030/643410">OpenAIRE2020</a> (643410), <a href="https://doi.org/10.3030/731011">OpenAIRE-Connect</a> (731011), <a href="https://doi.org/10.3030/777541">OpenAIRE-Advance</a> (777541), <a href="https://doi.org/10.3030/101017452">OpenAIRE-Nexus</a> (101017452), <a href="https://doi.org/10.3030/101007492">BICIKL</a> (101007492), <a href="https://doi.org/10.3030/863353">CS3MESH4EOSC</a>.</li> <li>Horizon Europe: <a href="https://doi.org/10.3030/101057264">FAIRCORE4EOSC</a> (101057264), <a href="https://doi.org/10.3030/101122956">HORIZON-ZEN</a> (101122956), <a href="https://cordis.europa.eu/project/id/101129744">EVERSE</a> (101129744)</li> </ul> </li> <li><a href="https://home.cern">CERN</a></li> <li>US National Institutes of Health (NIH):<ul> <li><a href="https://datascience.nih.gov/exploring-generalist-repository-nih-funded-data">Generalist Repository Ecosystem Initiative (GREI)</a></li> </ul> </li> <li><a href="https://www.arcadiafund.org.uk/">Arcadia Fund</a></li> <li><a href="https://sloan.org">Alfred P. Sloan Foundation</a></li> <li>Donations via <a href="https://giving.web.cern.ch/content/cern-society-foundation">CERN & Society Foundation</a></li> </ul> <p>Zenodo is developed and supported as a marginal activity, and hosted on top of existing infrastructure and services at CERN, in order to reduce operational costs and rely on existing efforts for High Energy Physics. CERN has some of the world’s top experts in running large scale research data infrastructures and digital repositories that we rely on in order to deliver a trusted digital repository.</p> <h4>Staff</h4> <p>Zenodo is currently operated by:</p> <ul> <li><strong>Steering board:</strong> Alexandros Ioannidis-Pantopikos, Jose Benito Gonzalez Lopez, Lars Holm Nielsen, Tim Smith</li> <li><strong>Service manager:</strong> Alexandros Ioannidis-Pantopikos</li> <li><strong>Developers and supporters:</strong> Carlin MacKenzie, Fatimah Zulfiqar, Manuel Alejandro De Oliveira da Costa, Pablo Tamarit, Yash Lamba</li> </ul> <p>We co-develop InvenioRDM (the underlying technical software platform) with CERN's Institutional Repositories team who builds and operates services such as <a href="https://cds.cern.ch">CERN Document Server</a> and <a href="http://opendata.cern.ch">CERN Open Data</a>. We rely heavily on CERN IT Department's teams and infrastructure such as database services, search services, platform-as-a-service, monitoring and logging services, storage services, compute and network services, project support services to mention a few. We further co-develop InvenioRDM with the wider InvenioRDM community consisting of 25+ institutional partners.</p> <h4>Memberships</h4> <p>CERN is an active member of the following organisations and international bodies (non-exhaustive):</p> <ul> <li><a href="https://datacite.org">DataCite</a></li> <li><a href="https://www.eosc.eu">EOSC</a></li> <li><a href="https://www.eudat.eu">EUDAT</a></li> <li><a href="https://www.openaire.eu">OpenAIRE</a></li> <li><a href="https://orcid.org">ORCID</a></li> <li><a href="https://www.rd-alliance.org/">Research Data Alliance (RDA)</a></li> <li><a href="https://scoap3.org/">SCOAP3</a></li> </ul> <hr /> <h2>Technical</h2> <p>Zenodo is powered by <a href="https://home.cern/science/computing/data-centre">CERN Data Centre</a> and the <a href="https://inveniordm.docs.cern.ch">InvenioRDM</a> and is fully run on open source products all the way through.</p> <p>Physically, Zenodo's entire technical infrastructure is located on CERN's premises which is subject to CERN's legal status (see above).</p> <h4>Server management</h4> <p>Zenodo servers are managed via <a href="https://docs.openshift.com">OpenShift</a> which itself runs on top of CERN's private cloud which is using <a href="https://openstack.org/">OpenStack</a> and <a href="https://puppet.com">Puppet</a> configuration management system. Servers are monitored via CERN’s monitoring infrastructure based on Logstash, OpenSearch, and Hadoop. Application errors are logged and aggregated in a local <a href="https://sentry.io/">Sentry</a> instance. Traffic to Zenodo frontend servers is load balanced via a combination of DNS load balancing and HAProxy load balancers.</p> <p>We are furthermore running three independent systems: one <strong>production</strong> system, one <strong>quality assurance</strong> system, and one <strong>development</strong> system. This ensures that all changes, whether at infrastructure level or source code level, can be tested and validated on our quality assurance system prior to being applied to our production system.</p> <h4>Frontend servers</h4> <p>Zenodo frontend servers are responsible for running the InvenioRDM repository platform application which is based on Python and the Flask web development framework. The frontend servers are running nginx HTTP server and uwsgi application server in front of the application and nginx is in addition in charge of serving static content.</p> <h4>Data storage</h4> <p>All files uploaded to Zenodo are stored in CERN’s <a href="https://eos-web.web.cern.ch/eos-web/">EOS service</a> in an 5 petabytes disk cluster. Each file copy has two replicas located on different disk servers. A daily incremental backup is performed of the EOS storage cluster into a <a href="https://docs.ceph.com/en/reef/">Ceph</a> storage cluster located in a different geographical location (~3.5 km apart). The backup retention policy keeps the last 7 daily backups, last 5 weekly backups and last 6 monthly backups.</p> <p>For each file we store two independent MD5 checksums. One checksum is stored by Invenio, and used to detect changes to files made from outside of Invenio. The other checksum is stored by EOS, and used for automatic detection and recovery of file corruption on disks.</p> <p>EOS is the primary low latency storage infrastructure for physics data from the Large Hadron Collider (LHC) and CERN currently operates multiple instances totalling 1+ exabyte of data.</p> <h4>Metadata storage</h4> <p>Metadata and persistent identifiers in Zenodo are stored in a PostgreSQL instance (with a master-slave setup) operated on CERN’s Database on Demand infrastructure with 24-hourly backup cycle with one backup sent to tape storage once a week. Metadata is in addition indexed in an OpenSearch cluster for fast and powerful searching. Metadata is stored in JSON format in PostgreSQL in a structure described by versioned JSONSchemas. All changes to metadata records on Zenodo are versioned, and happening inside database transactions.</p> <p>In addition to the metadata and data storage, Zenodo relies on Redis for caching and RabbitMQ and python Celery for distributed background jobs.</p> <h4>Additional infrastructure</h4> <p>Zenodo uses self-hosted versions of <a href="https://zammad.org">Zammad</a> for helpdesk management, <a href="https://listmonk.app">listmonk</a> for newsletter management, <a href="https://www.pgbouncer.org">PgBouncer</a> for database connection pooling, and <a href="https://iipimage.sourceforge.io">IIPServer</a> for our image zoom serving.</p> <p><hr /></p> <h2><a id="security"></a> Security</h2> <p>We take security very seriously and do our best to protect your data.</p> <ul> <li>CERN Data Centre: Our data centres is located on CERN premises and all physical access is restricted to a limited number of staff with appropriate training and who have been granted access in line with their professional duties (e.g. Zenodo staff do not have physical access to the CERN Data Centre) .</li> <li>Servers: Our servers are managed according to the CERN Security Baseline for Servers, meaning e.g. remote access to our servers are restricted to Zenodo staff with appropriate training, and the operating system and installed applications are kept updated with latest security patches via our automatic configuration management system Puppet.</li> <li>Network: CERN Security Team runs both host and network based intrusion detection systems and monitors the traffic flow, pattern and contents into and out of CERN networks in order to detect attacks. All access to zenodo.org happens over HTTPS, except for static documentation pages which are hosted on GitHub Pages.</li> <li>Data: Zenodo stores user passwords using strong cryptographic password hashing algorithms (currently PBKDF2+SHA512). Users’ access tokens to GitHub and ORCID are stored encrypted and can only be decrypted with the application’s secret key.</li> <li>Application: We are employing a suite of techniques to protect your session from being stolen by an attacker when you are logged in and run vulnerability scans against the application.</li> <li>Staff: CERN staff with access to user data operate under <a href="https://security.web.cern.ch/security/rules/en/OC5_english.pdf">CERN Operational Circular no. 5</a>, meaning among other things that<ul> <li>staff should not exchange among themselves information acquired unless it is expressly required for the execution of their duties.</li> <li>access to user data must always be consistent with the professional duties and only permitted for resolution of problems, detection of security issues, monitoring of resources and similar.</li> <li>staff are liable for damage resulting from any infringement and can have access withdrawn and/or be subject to disciplinary or legal proceedings depending on seriousness of the infringement.</li> </ul> </li> </ul> <p><strong>Special note on closed access data</strong></p> <p>Zenodo allows users to upload files under closed access. Closed access means that zenodo.org users will not be able to access the files you uploaded. The files are however stored unencrypted and <strong>may</strong> be viewed by Zenodo operational staff under specific conditions. This means that “closed access” on Zenodo <strong>is not</strong> suitable for secret or confidential data.</p> </div> </div> </div> </div> <footer class="footer"> <div class="menu-wrapper"> <div class="container"> <div class="row footer-menu"> <div class="col-xs-12 col-md-8"> <div class="row"> <div class="col-xs-2 col-md-2"> <h5>About</h5> <ul class="list-unstyled"> <li><a href="http://about.zenodo.org">About</a></li> <li><a href="http://about.zenodo.org/policies/">Policies</a></li> <li><a href="http://about.zenodo.org/infrastructure/">Infrastructure</a></li> <li><a href="http://about.zenodo.org/principles/">Principles</a></li> <li><a href="http://about.zenodo.org/roadmap/">Roadmap</a></li> <li><a href="http://about.zenodo.org/projects/">Projects</a></li> <li><a href="http://about.zenodo.org/contact/">Contact</a></li> </ul> </div> <div class="col-xs-2 col-md-2"> <h5>Blog</h5> <ul class="list-unstyled"> <li><a href="https://blog.zenodo.org">Blog</a></li> </ul> </div> <div class="col-xs-2 col-md-2"> <h5>Help</h5> <ul class="list-unstyled"> <li><a href="https://help.zenodo.org">Overview</a></li> <li><a href="https://help.zenodo.org/faq/">FAQ</a></li> <li><a href="https://help.zenodo.org/guides/">Guides</a></li> <li><a href="https://zenodo.org/support">Support</a></li> </ul> </div> <div class="col-xs-2 col-md-2"> <h5>Developers</h5> <ul class="list-unstyled"> <li><a href="https://developers.zenodo.org">REST API</a></li> <li><a href="https://developers.zenodo.org#oai-pmh">OAI-PMH</a></li> </ul> </div> <div class="col-xs-2 col-md-2"> <h5>Contribute</h5> <ul class="list-unstyled"> <li><a href="https://github.com/zenodo/zenodo-rdm"><i class="fa fa-external-link"></i> GitHub</a></li> <li><a href="https://zenodo.org/donate"><i class="fa fa-external-link"></i> Donate</a></li> </ul> </div> </div> </div> <div class="col-xs-12 col-md-4"> <div class="pull-right-md text-center-sm text-center-xs"> <h5>Funded by</h5> <ul class="list-inline"> <li><a href="https://home.cern"><img src="/static/img/cern.png" width="60" height="60" /></a></li> <li><a href="https://www.openaire.eu"><img src="/static/img/openaire.png" width="80"/></a></li> <li><a href="https://ec.europa.eu/programmes/horizon2020/"><img src="/static/img/eu.png" width="88" height="60" /></a></li> </ul> </div> </div> </div> </div> </div> <div class="container"> <div class="row"> <div class="col-xs-12 col-sm-6 col-sm-push-6"> <div class="pull-right-sm text-center-xs"> <ul class="list-inline"> <li><a href="https://stats.uptimerobot.com/vlYOVuWgM">Status</a></li> <li><a href="https://about.zenodo.org/privacy-policy">Privacy policy</a></li> <li><a href="https://about.zenodo.org/cookie-policy">Cookie policy</a></li> <li><a href="https://about.zenodo.org/terms">Terms of Use</a></li> <li><a href="https://about.zenodo.org/contact">Support</a></li> </ul> </div> </div> <div class="col-xs-12 col-sm-6 col-sm-pull-6 text-center-xs"> <p><a title="Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution 4.0 International License." rel="license" href="http://creativecommons.org/licenses/by/4.0/"><img alt="Creative Commons Licence" height="20" src="https://i.creativecommons.org/l/by/4.0/88x31.png" /></a> Powered by <a href="https://home.cern/science/computing/data-centre">CERN Data Centre</a> & <a href="http://inveniosoftware.org">Invenio</a>.</p> </div> </div> </div> </footer> <div class="cookie-banner hidden"> <i class="close icon"></i> <div> <p >This site uses cookies. Find out more on <a href="https://about.zenodo.org/cookie-policy">how we use cookies</a></p> </div> <div class="buttons"> <button class="small primary" id="cookies-all">Accept all cookies</button> <button class="small" id="cookies-essential">Accept only essential cookies</button> </div> </div> <script> var _paq = window._paq = window._paq || []; _paq.push(['requireCookieConsent']); (function() { var u="https://webanalytics.web.cern.ch/"; _paq.push(['setTrackerUrl', u+'matomo.php']); _paq.push(['setSiteId', '361']); var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0]; g.async=true; g.src=u+'matomo.js'; s.parentNode.insertBefore(g,s); })(); var cookieConsent = document.cookie .split("; ") .find((row) => row.startsWith("cookie_consent=")) ?.split("=")[1]; if (cookieConsent) { if (cookieConsent === "all") { matomo(); } } else { document.querySelector(".cookie-banner").classList.remove("hidden") _paq.push(['forgetConsentGiven']); } $('.cookie-banner .close') .on('click', function () { $(this) .closest('.cookie-banner') .fadeOut('fast'); setCookie("cookie_consent","essential"); }); $('#cookies-essential') .on('click', function () { $(this) .closest('.cookie-banner') .fadeOut('fast'); setCookie("cookie_consent","essential"); }); $('#cookies-all') .on('click', function () { $(this) .closest('.cookie-banner') .fadeOut('fast'); setCookie("cookie_consent","all"); _paq.push(['rememberCookieConsentGiven']); matomo(); }); function matomo() { /* tracker methods like "setCustomDimension" should be called before "trackPageView" */ _paq.push(['trackPageView']); _paq.push(['enableLinkTracking']); } function setCookie(cname, cvalue) { var d = new Date(); d.setTime(d.getTime() + (365 * 24 * 60 * 60 * 1000)); // one year var expires = "expires=" + d.toUTCString(); var cookie = cname + "=" + cvalue + ";" + expires + ";" cookie += "Domain=zenodo.org;Path=/;SameSite=None; Secure"; // so that it works across subdomains document.cookie = cookie; } </script> </body> </html>