CINXE.COM
UBLabs
<!DOCTYPE html> <html lang="en" itemscope itemtype="http://schema.org/Blog"> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <title>UBLabs</title> <meta name="renderer" content="webkit"> <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1"> <meta name="theme-color" content="#f8f5ec"> <meta name="msapplication-navbutton-color" content="#f8f5ec"> <meta name="apple-mobile-web-app-capable" content="yes"> <meta name="apple-mobile-web-app-status-bar-style" content="#f8f5ec"> <meta name="description" content="The University Library Johann Christian Senckenberg (UB JCS) in Frankfurt am Main hosts several "Specialised Information Services" (FID) who jointly maintain this blog. We, the FID IT team, are writing about technical challenges that we face in our daily work."><meta name="keywords" content="UB Frankfurt, UB JCS, FID, Fachinformationsdienste"> <meta name="generator" content="Hugo 0.139.0 with theme ublabs (based on even)"> <link rel="canonical" href="https://labs.ub.uni-frankfurt.de/"/> <link href="https://labs.ub.uni-frankfurt.de/index.xml" rel="alternate" type="application/rss+xml" title="UBLabs" /> <link href="https://labs.ub.uni-frankfurt.de/index.xml" rel="feed" type="application/rss+xml" title="UBLabs" /> <link rel="manifest" href="/manifest.json"> <link href="/sass/main.min.ad091aeab1eaf5b038e2a283815e42116fda7dd8fb0ac70714f158222c11147a.css" rel="stylesheet"> <link href="/lib/fancybox/jquery.fancybox-3.1.20.min.css" rel="stylesheet"> <meta name="DC.creator" content="Aaron Christiansson" /> <meta name="DC.creator" content="Adrian Pachzelt" /> <meta name="DC.creator" content="Julia Beck" /> <meta name="DC.creator" content="Jan Eberhardt" /> <meta name="DC.creator" content="Jan Miehle" /> <meta name="DC.creator" content="Thorsten Fritze" /> <meta name="DC.creator" content="Yannic J盲ckel" /> <meta name="DC.publisher" content="Universit盲tsbibliothek Johann Christian Senckenberg, Frankfurt am Main"> <meta name="publisher" content="Universit盲tsbibliothek Johann Christian Senckenberg, Frankfurt am Main"> <meta name="DC.subject" content="UB Frankfurt, UB JCS, FID, Fachinformationsdienste"> <meta property="og:url" content="https://labs.ub.uni-frankfurt.de/"> <meta property="og:site_name" content="UBLabs"> <meta property="og:title" content="UBLabs"> <meta property="og:description" content="The University Library Johann Christian Senckenberg (UB JCS) in Frankfurt am Main hosts several "Specialised Information Services" (FID) who jointly maintain this blog. We, the FID IT team, are writing about technical challenges that we face in our daily work."> <meta property="og:locale" content="en"> <meta property="og:type" content="website"> <meta itemprop="name" content="UBLabs"> <meta itemprop="description" content="The University Library Johann Christian Senckenberg (UB JCS) in Frankfurt am Main hosts several "Specialised Information Services" (FID) who jointly maintain this blog. We, the FID IT team, are writing about technical challenges that we face in our daily work."> <meta itemprop="datePublished" content="2024-11-01T00:00:00+00:00"> <meta itemprop="dateModified" content="2024-10-22T14:00:00+00:00"> <meta name="twitter:card" content="summary"> <meta name="twitter:title" content="UBLabs"> <meta name="twitter:description" content="The University Library Johann Christian Senckenberg (UB JCS) in Frankfurt am Main hosts several "Specialised Information Services" (FID) who jointly maintain this blog. We, the FID IT team, are writing about technical challenges that we face in our daily work."> </head> <body> <div id="mobile-navbar" class="mobile-navbar"> <div class="mobile-header-logo"> <a href="/" class="logo">UBLabs</a> </div> <div class="mobile-navbar-icon"> <span></span> <span></span> <span></span> </div> </div> <nav id="mobile-menu" class="mobile-menu slideout-menu"> <ul class="mobile-menu-list"> <li class="mobile-menu-item"> <a href="/">Home</a> </li><li class="mobile-menu-item"> <a href="/post/">Archives</a> </li><li class="mobile-menu-item"> <a href="/tags/">Tags</a> </li><li class="mobile-menu-item"> <a href="/projects/">Projects</a> </li><li class="mobile-menu-item"> <a href="/about/">About</a> </li> </ul> </nav> <div class="container" id="mobile-panel"> <header id="header" class="header"> <div class="logo-wrapper goethe-wrapper"> <a class="goethe-logo" href="http://www.uni-frankfurt.de"> <img src="/img/unilogo.svg" alt="Goethe University Logo"> </a> <h1 class="department"> <a href="http://www.ub.uni-frankfurt.de"> <span class="departmentTitle">Universitätsbibliothek</span> <br> <span class="departmentName">J.C. Senckenberg</span> </a> <span class="departmentNumber">Labs</span> </h1> </div> </header> <nav class="navbar"> <div class="navmain"> <ul id="menu" class="menu"> <li class="menu-item"> <a class="menu-item-link" href="/">Home</a> </li><li class="menu-item"> <a class="menu-item-link" href="/post/">Archives</a> </li><li class="menu-item"> <a class="menu-item-link" href="/tags/">Tags</a> </li><li class="menu-item"> <a class="menu-item-link" href="/projects/">Projects</a> </li><li class="menu-item"> <a class="menu-item-link" href="/about/">About</a> </li> <li class="social-links"> <a href="/index.xml" type="application/rss+xml" class="iconfont icon-rss" title="rss"></a> <a href="mailto:ublabs@ub.uni-frankfurt.de" class="iconfont icon-email" title="Email Contact"></a> </li> </ul> </div> </nav> <main id="main" class="main"> <div class="content-wrapper"> <div id="content" class="content"> <section id="posts" class="posts"> <article class="post"> <header class="post-header"> <h1 class="post-title"><a class="post-link" href="/post/single-sign-on-with-django-and-keycloak/">Single Sign-On with Django and Keycloak</a></h1> <div class="post-meta"> <span class="post-time"> 2024-07-17 </span> <div class="post-author"> <a href="/authors/afr/"> <img class="post-image" src="/img/icons/favicon_afr.png" alt="icon FID African Studies" title="FID African Studies"/> </a> <a href="/authors/avl/"> <img class="post-image" src="/img/icons/favicon_avl.png" alt="icon FID Comparative Literature Studies (avldigital)" title="FID Comparative Literature Studies (avldigital)"/> </a> </div> <span class="more-meta"> 1251 words </span> <span class="more-meta"> 6 mins read </span> </div> </header> <div class="post-content"> <div class="post-summary"> <h1 id="introduction">Introduction</h1> <p>User management is a fundamental aspect of web application development. While a local login is the simplest method to grant users access to resources, it comes with certain limitations. If users want to access multiple services using the same credentials, a centralized authentication system is necessary.</p> <p>For a diverse user base (spanning different contexts such as industry specialists, academia, and various organizations or locations) a <em>Single Sign-On</em> (SSO) solution is essential. Well-known <em>identity providers</em> (IdPs) like GitHub<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup> in the tech world, ORCID<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup> for the scientific community, and the lesser-known DNF-AAI<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup> in German academia enable users to reuse their credentials across services.</p> </div> <div class="read-more"> <a href="/post/single-sign-on-with-django-and-keycloak/" class="read-more-link">Read more...</a> </div> </div> </article> <article class="post"> <header class="post-header"> <h1 class="post-title"><a class="post-link" href="/post/data-engineering-with-luigi-lessons-learned/">Data Engineering with luigi - Lessons learned</a></h1> <div class="post-meta"> <span class="post-time"> 2023-08-17 </span> <div class="post-author"> <a href="/authors/bio/"> <img class="post-image" src="/img/icons/favicon_bio.ico" alt="icon FID Biodiversity Research (BIOfid)" title="FID Biodiversity Research (BIOfid)"/> </a> </div> <span class="more-meta"> 2013 words </span> <span class="more-meta"> 10 mins read </span> </div> </header> <div class="post-content"> <div class="post-summary"> <h1 id="introduction">Introduction</h1> <p>At the UB JCS, we make extensive usage of the Python <a href="https://luigi.readthedocs.io/en/stable/">luigi framework</a> for data engineering. The framework is capable of handling thousands of tasks, calculating non-circular task dependencies, and run over days. Additionally, it provides a convenient web control panel to see, e.g. the task dependencies in a tree diagram or start specific tasks.</p> <p>Although luigi itself supports the user already by enforcing a very specific structure, there are still some things to consider when designing a data pipeline with luigi (for a general introduction, see in <a href="https://labs.ub.uni-frankfurt.de/post/common-engineering-strategies-in-luigi/">a previous post</a>). In this post, I present ideas, that I learned while using luigi. Since luigi is a heavily object-oriented framework, some approaches in this post rely naturally on <a href="https://en.wikipedia.org/wiki/Architectural_pattern">Software architecture patterns</a>.</p> </div> <div class="read-more"> <a href="/post/data-engineering-with-luigi-lessons-learned/" class="read-more-link">Read more...</a> </div> </div> </article> <article class="post"> <header class="post-header"> <h1 class="post-title"><a class="post-link" href="/post/common-engineering-strategies-in-luigi/">Common engineering strategies in luigi</a></h1> <div class="post-meta"> <span class="post-time"> 2023-06-29 </span> <div class="post-author"> <a href="/authors/bio/"> <img class="post-image" src="/img/icons/favicon_bio.ico" alt="icon FID Biodiversity Research (BIOfid)" title="FID Biodiversity Research (BIOfid)"/> </a> <a href="/authors/lin/"> <img class="post-image" src="/img/icons/favicon_lin.ico" alt="icon FID Linguistics" title="FID Linguistics"/> </a> <a href="/authors/per/"> <img class="post-image" src="/img/icons/favicon_dk.ico" alt="icon FID Performing Arts" title="FID Performing Arts"/> </a> </div> <span class="more-meta"> 2097 words </span> <span class="more-meta"> 10 mins read </span> </div> </header> <div class="post-content"> <div class="post-summary"> <h1 id="introduction">Introduction</h1> <p>For many automated data processing tasks within the context of the Specialised Information Services (FID) at the University Library Frankfurt, we use the Python package <a href="https://luigi.readthedocs.io/en/stable/"><code>luigi</code></a>. This package proves especially useful when a task (e.g. the loading of data into a database) depends on the work of other tasks that have to run successfully, before the next task starts (e.g. first you need to download the data). <code>luigi</code> orchestrates all required tasks and their respective required task(s) and then processes everything for you. This approach makes the maintenance of tasks very easy, since you only have to add or remove required tasks from any task and <code>luigi</code> handles the rest for you, while you don’t have to worry about the <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph">computer science behind it</a> too much. But although <code>luigi</code> takes a lot of mental load off of you, it also requires strategies to handle common situations that you may find yourself in.</p> </div> <div class="read-more"> <a href="/post/common-engineering-strategies-in-luigi/" class="read-more-link">Read more...</a> </div> </div> </article> <article class="post"> <header class="post-header"> <h1 class="post-title"><a class="post-link" href="/post/using-another-metadata-standard-than-marc21-in-vufind-part-ii/">Using another metadata standard than MARC21 in VuFind, Part II</a></h1> <div class="post-meta"> <span class="post-time"> 2023-03-31 </span> <div class="post-author"> <a href="/authors/per/"> <img class="post-image" src="/img/icons/favicon_dk.ico" alt="icon FID Performing Arts" title="FID Performing Arts"/> </a> </div> <span class="more-meta"> 1518 words </span> <span class="more-meta"> 8 mins read </span> </div> </header> <div class="post-content"> <div class="post-summary"> <h1 id="introduction">Introduction</h1> <p>In the <a href="https://labs.ub.uni-frankfurt.de/post/using-another-metadata-standard-than-marc21-in-vufind-part-i/">first post of this series</a>, we covered the necessary steps to populate VuFind’s Solr cores with title and authority records. This second post describes changes to configuration files, as well as modifications that are necessary to interact, display and export records. All our customizations are based on existing VuFind code and, to ensure maintainability, stored in the <code>local/</code> folder and the custom module <code>Fiddk</code>. We want to remind the reader, that we assume basic VuFind knowledge, which can be acquired from the <a href="https://vufind.org/wiki/">documentation</a>.</p> </div> <div class="read-more"> <a href="/post/using-another-metadata-standard-than-marc21-in-vufind-part-ii/" class="read-more-link">Read more...</a> </div> </div> </article> <article class="post"> <header class="post-header"> <h1 class="post-title"><a class="post-link" href="/post/using-another-metadata-standard-than-marc21-in-vufind-part-i/">Using another metadata standard than MARC21 in VuFind, Part I</a></h1> <div class="post-meta"> <span class="post-time"> 2023-02-06 </span> <div class="post-author"> <a href="/authors/per/"> <img class="post-image" src="/img/icons/favicon_dk.ico" alt="icon FID Performing Arts" title="FID Performing Arts"/> </a> </div> <span class="more-meta"> 1682 words </span> <span class="more-meta"> 8 mins read </span> </div> </header> <div class="post-content"> <div class="post-summary"> <h1 id="introduction">Introduction</h1> <p>When you install the open source discovery system <a href="https://vufind.org">VuFind</a>, follow basic configuration steps and feed it with library records, it works well out-of-the-box and provides you with faceted search results and the possibility to browse through your data besides many other features. The easiest way to achieve this, is to load the standard interchange format for library records, i.e. <a href="https://www.loc.gov/marc/bibliographic/">MARC21</a>, into the included <a href="https://solr.apache.org/">Apache Solr</a>-based search index. The <a href="https://www.performing-arts.eu/">FID Performing Arts</a> uses VuFind since 2015, but as mentioned in an <a href="https://labs.ub.uni-frankfurt.de/post/tips-for-cleaner-faster-and-more-maintainable-xslt-code/">earlier post</a>, we receive a vast amount of metadata from performing arts museums and archives in standards other than MARC21, such as EAD, METS/MODS and LIDO as well as other individual data formats which result from database systems like MS Access or FAUST DB. In order to meet the special requirements of performing arts metadata and work with a consistent data set during data aggregation, we decided to map all received data into an extended version of the universal and flexible <a href="https://pro.europeana.eu/page/edm-documentation">Europeana Data Model</a> (EDM).</p> </div> <div class="read-more"> <a href="/post/using-another-metadata-standard-than-marc21-in-vufind-part-i/" class="read-more-link">Read more...</a> </div> </div> </article> </section> <nav class="pagination"> <a class="next" href="/page/2/"> <span class="next-text">Next</span> <i class="iconfont icon-right"></i> </a> </nav> </div> </div> </main> <footer id="footer" class="footer"> <div class="mainfooter"> <span class=""> 漏 2025 <span>Universit盲tsbibliothek Frankfurt am Main</span> </span> <span class="division">|</span> <span class="privacy"> <a href="http://www.ub.uni-frankfurt.de/benutzung/datenschutz.html" target="_blank">Privacy policy</a> </span> <span class="division">|</span> <span class="imprint"> <a href="http://www2.uni-frankfurt.de/impressum" target="_blank">Imprint</a> </span> </div> <div class="subfooter"> <span class="license"> All content is published under <a href="https://creativecommons.org/licenses/by/4.0/">CC-BY 4.0</a> </span> <span class="division">|</span> <span class="power-by"> Powered by <a class="hugo-link" href="https://gohugo.io">Hugo</a> </span> <span class="division">|</span> <span class="theme-info"> Theme Based on: <a class="theme-link" href="https://github.com/olOwOlo/hugo-theme-even">Even</a> </span> </div> </footer> <div class="back-to-top" id="back-to-top"> <i class="iconfont icon-up"></i> </div> </div> <script type="text/javascript" src="/lib/jquery/jquery-3.7.1.min.js"></script> <script type="text/javascript" src="/lib/slideout/slideout-1.0.1.min.js"></script> <script type="text/javascript" src="/lib/fancybox/jquery.fancybox-3.1.20.min.js"></script> <script type="text/javascript" src="/js/main.min.38c58a0299372513e2724f2a06fb322e7071174702e5774ed248acbfe2792847.js"></script> <script type="text/plain" data-type="application/javascript" data-name="matomo" src="/js/matomo.min.98802dad1c3d07520b23a33655e4894e2ce05b03f0cbb0bce6145f71c7e1a7d5.js"></script> <script defer type="application/javascript" src="/js/klaro-config.min.5fcb5bcb8f6d5fd559a161ebaab603991013c5088521829d736adff924975542.js"></script> <script defer data-config="klaroConfig" src="/lib/klaro/klaro-no-css.js"> </script> </body> </html>