<?xml version="1.0" encoding="utf-8"?> <rss version="2.0"><channel><title>David Kane</title><link>https://dkane.net/</link><description></description><lastBuildDate>Mon, 07 Oct 2024 11:08:47 +0100</lastBuildDate><item><title>Why PostgreSQL + Django is my data pipeline of choice</title><link>https://dkane.net/2024/why-postgresql-django-is-my-data-pipeline-of-choice/</link><description>&lt;p&gt;&lt;em&gt;This blog post is based partly on an interesting conversation at &lt;a href="https://www.odcamp.uk/"&gt;Open Data camp 9 in Manchester&lt;/a&gt; this year, where Giles from Open Innovations led a session called &lt;a href="https://docs.google.com/document/d/1zY1KZGlpac-rUnfW3pRwncXotmXe4e1FHzo2VECHQ6M/edit"&gt;DIY FTW! What’s in YOUR toolbox?&lt;/a&gt;. As well as showcasing a bunch of tools that I'd not come across, it also made me think about what my typical workflow looks like.&lt;/em&gt;&lt;/p&gt; &lt;p&gt;A large proportion of my work at the moment involves creating data pipelines. These data pipelines start with "inputs" - generally some public data source, like the Charity Commission Register of Charities. There is then some cleaning or processing that happens - merging data together, manually adding or amending data. And then finally some "outputs" happen - this might be reports on an individual charity, or a set of aggregate data around a particular group of charities.&lt;/p&gt; &lt;p&gt;There's a few particular requirements for these pipelines - I generally need them to be repeatable, including re-running with updated data. This means I need to carefully separate the data cleaning and additions so that they are not lost when new data comes in.&lt;/p&gt; &lt;p&gt;For a small scale data pipeline - maybe running a quick piece of analyis from data fetched in an SQL query, I'll generally turn to a python Notebook. I've also experimented with new tools like &lt;a href="https://duckdb.org/"&gt;DuckDB&lt;/a&gt; for fetching and manipulating data.&lt;/p&gt; &lt;p&gt;But for anything larger than an afternoon's work, I've found that I'm reaching for &lt;a href="https://www.djangoproject.com/"&gt;Django&lt;/a&gt;, the python web framework. This might seem like an unintuitive choice at first - Django is a big sprawling piece of software, and creating data pipelines is not what it is designed for. But I've found that its "batteries-included" approach means that it has all the pieces I need in place.&lt;/p&gt; &lt;div class="toc"&gt;&lt;span class="toctitle"&gt;Table of Contents&lt;/span&gt;&lt;ul&gt; &lt;li&gt;&lt;a href="#what-you-get-out-of-the-box"&gt;What you get out of the box&lt;/a&gt;&lt;ul&gt; &lt;li&gt;&lt;a href="#data-model"&gt;Data model&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="#admin-area"&gt;Admin area&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="#user-accessauthentication"&gt;User access/authentication&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="#management-commands"&gt;Management commands&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; &lt;/li&gt; &lt;li&gt;&lt;a href="#what-i-add-with-add-ons"&gt;What I add with add-ons&lt;/a&gt;&lt;ul&gt; &lt;li&gt;&lt;a href="#django-sql-dashboard"&gt;Django SQL Dashboard&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="#dokku-for-deployment"&gt;Dokku for deployment&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="#django-ninja-api"&gt;Django Ninja API&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; &lt;/li&gt; &lt;li&gt;&lt;a href="#whats-missing-what-are-the-issues"&gt;What's missing / what are the issues&lt;/a&gt;&lt;ul&gt; &lt;li&gt;&lt;a href="#no-easy-ways-of-getting-data-out"&gt;No easy ways of getting data out&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="#overheads-are-quite-high"&gt;Overheads are quite high&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="#dealing-with-large-datasets"&gt;Dealing with large datasets&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; &lt;/li&gt; &lt;/ul&gt; &lt;/div&gt; &lt;h2 id="what-you-get-out-of-the-box"&gt;What you get out of the box&lt;/h2&gt; &lt;p&gt;A big attraction for using Django is that it comes fully loaded - a lot of the components I might need in a data pipeline are included. Even if you don't need every feature for every project, there's a lot of value in having them at your fingertips anyway. I often find that I start a project thinking, for example, that I won't need user authentication - but then if the requirements change then it's easy to add.&lt;/p&gt; &lt;p&gt;The out of the box parts I find most useful are:&lt;/p&gt; &lt;h3 id="data-model"&gt;Data model&lt;/h3&gt; &lt;p&gt;A django app is structured around the &lt;a href="https://docs.djangoproject.com/en/5.1/topics/db/models/"&gt;database models&lt;/a&gt; you create. It's easy to define your models in code, and there are powerful features for managing relationships between them.&lt;/p&gt; &lt;p&gt;A key positive is the flexibility of &lt;a href="https://docs.djangoproject.com/en/5.1/topics/migrations/"&gt;django migrations&lt;/a&gt;. These mean that I can change the data model, test the effect these changes will have on the data, and then deploy these changes.&lt;/p&gt; &lt;p&gt;If the rigid structure of a data model ends up being too prescriptive for the data you're dealing with, there are options to have something more flexible. I've recently been using a pattern of storing data in JSON fields, and then &lt;a href="generated-field.md"&gt;using Generated Fields&lt;/a&gt; to access the data.&lt;/p&gt; &lt;h3 id="admin-area"&gt;Admin area&lt;/h3&gt; &lt;p&gt;After you've defined your database models, Django makes it quick and easy to add an &lt;a href="https://docs.djangoproject.com/en/5.1/ref/contrib/admin/"&gt;admin interface&lt;/a&gt; that can be accessed from the web to view the data. You can make some complex views with the built-in admin tools - adding filters, search, the ability to link records together. You can also add and edit data - though you do need to carefully think about how this works when overwriting a data import.&lt;/p&gt; &lt;h3 id="user-accessauthentication"&gt;User access/authentication&lt;/h3&gt; &lt;p&gt;When working with clients, it is invaluable to be able to &lt;a href="https://docs.djangoproject.com/en/5.1/topics/auth/"&gt;set up user accounts&lt;/a&gt; out of the box, in order to let them see the data. This might be about giving them access to the &lt;a href="#admin-area"&gt;admin area&lt;/a&gt; or setting up some views to show them what the outputs might look like, or filter and download the data themselves.&lt;/p&gt; &lt;p&gt;Adding user authentication, particularly when data can be edited in the admin area, means that I can also add an audit trail easily, to see who changed the data and why.&lt;/p&gt; &lt;h3 id="management-commands"&gt;Management commands&lt;/h3&gt; &lt;p&gt;Often a data pipeline has a set of reusable scripts that don't fit nicely with the website paradigm that Django works with. But this is where &lt;a href="https://docs.djangoproject.com/en/5.1/howto/custom-management-commands/"&gt;management commands&lt;/a&gt; come in - you can easily define python scripts that run on the data. These are useful for data management tasks like import, export or bulk cleaning processes.&lt;/p&gt; &lt;p&gt;I've started making use of &lt;a href="https://github.com/GaretJax/django-click"&gt;django-click&lt;/a&gt; which I slightly prefer over the native django commands.&lt;/p&gt; &lt;p&gt;My one gripe with management commands is that a bit of work is needed in order to define logging, transaction management, etc - but it is possible to do that.&lt;/p&gt; &lt;h2 id="what-i-add-with-add-ons"&gt;What I add with add-ons&lt;/h2&gt; &lt;p&gt;Although Django comes pretty fully-featured out of the box, there are a few add-ons and other tools that I typically install at the same time. These add specific functionality that isn't present in standard Django, but add particular features around data science.&lt;/p&gt; &lt;h3 id="django-sql-dashboard"&gt;Django SQL Dashboard&lt;/h3&gt; &lt;p&gt;&lt;a href="https://django-sql-dashboard.datasette.io/en/stable/"&gt;Django SQL Dashboard&lt;/a&gt; is a great add on created by &lt;a href="https://simonwillison.net/"&gt;Simon Willison&lt;/a&gt; the creator of &lt;a href="https://datasette.io/"&gt;Datasette&lt;/a&gt; (and one of the co-creators of Django itself!)&lt;/p&gt; &lt;p&gt;The plugin basically lets you write custom SQL queries against your database, which are then displayed as webpages. These are great for making always-updated dashboards showing the latest state of your data, without needing to wrangle the Django models into the right shape for a custom view (creating aggregate queries with Django models can be a bit fiddly).&lt;/p&gt; &lt;p&gt;I use these extensively with &lt;a href="https://findthatcharity.uk/"&gt;findthatcharity.uk&lt;/a&gt; - for example, &lt;a href="https://findthatcharity.uk/dashboard/data-completeness/"&gt;here's a dashboard showing how up-to-date the Charity register is&lt;/a&gt;.&lt;/p&gt; &lt;p&gt;I've actually created &lt;a href="https://github.com/drkane/django-sql-dashboard"&gt;my own version of the plugin&lt;/a&gt;, which adds a bit more customisability, for example by allowing titles and descriptions for individual queries within a dashboard.&lt;/p&gt; &lt;p&gt;There's a bit of overhead to setting this up - you need to be using a PostgreSQL database, you need to create a read-only SQL user, etc. And it does have potential security implications if not done correctly.&lt;/p&gt; &lt;h3 id="dokku-for-deployment"&gt;Dokku for deployment&lt;/h3&gt; &lt;p&gt;Obviously if you're running Django partly because it's a web server, you need to be able to deploy it to the web. I use &lt;a href="https://dokku.com/docs/getting-started/installation/"&gt;dokku&lt;/a&gt; for this - it's like running your own heroku instance.&lt;/p&gt; &lt;p&gt;Dokku makes it easy to add things like a database (I generally use PostgreSQL) or even a search engine like elasticsearch. I can easily test changes locally, then deploy simply using &lt;code&gt;git push&lt;/code&gt;.&lt;/p&gt; &lt;p&gt;I also use dokku's scheduling feature to run scripts periodically, for example setting up nightly data import.&lt;/p&gt; &lt;h3 id="django-ninja-api"&gt;Django Ninja API&lt;/h3&gt; &lt;p&gt;This is slightly more rare, but it can be useful to have an API to access the data. I really like &lt;a href="https://django-ninja.dev/"&gt;Django Ninja&lt;/a&gt; for defining APIs. It fits easily with the OpenAPI standard, which means you get nice documentation and testing out the box. I generally add some sort of API Key based authentication to aid with sharing with others.&lt;/p&gt; &lt;h2 id="whats-missing-what-are-the-issues"&gt;What's missing / what are the issues&lt;/h2&gt; &lt;p&gt;Although it's my tool of choice for data pipelines, Django isn't perfect for this purpose (which after all, it wasn't specifically designed for). There are a few gaps and downsides to this approach:&lt;/p&gt; &lt;h3 id="no-easy-ways-of-getting-data-out"&gt;No easy ways of getting data out&lt;/h3&gt; &lt;p&gt;Often the output of a data pipeline is a bulk output that needs to go somewhere else - a spreadsheet download, a set of tables for a report, etc. I usually end up producing some django views to output or download these, but it can quickly get complex or inflexible.&lt;/p&gt; &lt;p&gt;One approach that I've had some success with is adding functionality to export (and import) from admin views. There are some add-ons that do this, or it's relatively straightforward to create your own using custom admin templates.&lt;/p&gt; &lt;h3 id="overheads-are-quite-high"&gt;Overheads are quite high&lt;/h3&gt; &lt;p&gt;This approach does mean there are quite high setup costs, compared to something like a python Notebook. For a full deployment, you need to have a server, a domain name, a database. You need to manage SSL certificates, users, downtime etc. You need to be able to get command line access to run scripts, etc on the server. This is all doable, particularly with something like dokku, but it does require a bit of effort.&lt;/p&gt; &lt;h3 id="dealing-with-large-datasets"&gt;Dealing with large datasets&lt;/h3&gt; &lt;p&gt;Django, particularly in terms of data models, is designed around manually creating and editing individual instances of a model. I find with a lot of my work I am working on bulk data imported from a source. Sometimes these can run to hundreds of thousands or even millions of rows.&lt;/p&gt; &lt;p&gt;While this is all possible to handle, and you can bypass Django to a certain extent, it can be tricky when running bulk updates &amp;amp; imports. For example, a pattern I've not quite cracked is bulk upserts - where data is either created or updated depending on whether it already exists. This is possible to do, but can be a bit fiddly.&lt;/p&gt; &lt;hr&gt; &lt;p&gt;I hope I've successfully shown how Django can work well as a tool for managing data pipelines. I have a &lt;a href="https://github.com/kanedata/cookiecutter-django-kd"&gt;cookiecutter template&lt;/a&gt; that forms the basis for a new django project, which shows some of this approach in practice.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Mon, 07 Oct 2024 11:08:47 +0100</pubDate><guid isPermaLink="false">tag:dkane.net,2024-10-07:/2024/why-postgresql-django-is-my-data-pipeline-of-choice/</guid><category>Blog</category></item><item><title>Where to get boundary data on new parliamentary constituencies</title><link>https://dkane.net/2024/where-to-get-boundary-data-on-new-parliamentary-constituencies/</link><description>&lt;p&gt;The 2024 UK General Election happens on July 4th. This election will be fought on a new set of constituency boundaries, with the makeup of constituencies changing for the first time since 2010. You can find out more information about these changes &lt;a href="https://www.parliament.uk/about/how/elections-and-voting/constituencies/"&gt;on the parliament.uk website&lt;/a&gt;.&lt;/p&gt; &lt;p&gt;If you're analysing or visualising data by parliamentary constituency, then you probably want to use the new 2024 boundaries. I thought it might be useful to show how to get data on these new boundaries, particularly as lots of the &lt;a href="./where-to-find-boundary-data.md"&gt;main sources of boundary data&lt;/a&gt; haven't yet updated to take them into account.&lt;/p&gt; &lt;h2 id="source-data"&gt;Source data&lt;/h2&gt; &lt;p&gt;The source I tend to use for boundaries is the ONS Geoportal. They've &lt;a href="https://geoportal.statistics.gov.uk/search?q=Future_Parlicons&amp;amp;sort=Title%7Ctitle%7Casc"&gt;produced a category bringing together all the data on the new constituencies&lt;/a&gt;. Some of the useful files are:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;List of the &lt;a href="https://geoportal.statistics.gov.uk/maps/9a876e4777bc47e392e670a7b8bc3f5c"&gt;names and codes&lt;/a&gt; of the new constituencies&lt;/li&gt; &lt;li&gt;Boundary data can be downloaded as GeoJSON or in other formats at a variety of resolutions (the higher resolution files are larger): &lt;a href="https://geoportal.statistics.gov.uk/maps/0d698c9712de4afcac9377367d831c1a"&gt;Highest resolution 1&lt;/a&gt;, &lt;a href="https://geoportal.statistics.gov.uk/maps/e489d4e5fe9f4f9caa6af161da5442af"&gt;Highest resolution 2&lt;/a&gt;, &lt;a href="https://geoportal.statistics.gov.uk/maps/7f5c8a729c2f4ccbb8edc3a067845823"&gt;Lower resolution&lt;/a&gt; and &lt;a href="https://geoportal.statistics.gov.uk/maps/ef63f363ac824b79ae9670744fcc4307"&gt;lowest resolution&lt;/a&gt;.&lt;/li&gt; &lt;li&gt;&lt;a href="https://geoportal.statistics.gov.uk/datasets/6f2f35a9a0b94e7e949eeba7785911d4"&gt;Postcode lookup&lt;/a&gt; - mapping every UK postcode to the new constituencies&lt;/li&gt; &lt;li&gt;&lt;a href="https://geoportal.statistics.gov.uk/maps/c776b66c0e534b849cae5a5121b7a16a"&gt;Lookups from old constituencies to new ones&lt;/a&gt; - note that these aren't one-to-one, so where it'll show two constituencies that overlap to form a new one&lt;/li&gt; &lt;li&gt;Lookups from smaller geographies to the new consitutencies: &lt;a href="https://geoportal.statistics.gov.uk/maps/5968b5b2c0f14dd29ba277beaae6dec3"&gt;Output Area 2021&lt;/a&gt;, &lt;a href="https://geoportal.statistics.gov.uk/maps/4491820e4f274755a2553c42fc6a250c"&gt;Output Area 2011&lt;/a&gt;, &lt;a href="https://geoportal.statistics.gov.uk/maps/0d62b50d90d14b8d8b2ccf9f770b5751"&gt;LSOA 2021&lt;/a&gt;, &lt;a href="https://geoportal.statistics.gov.uk/maps/098360c460dd41beacbdfad83bc4fea2"&gt;MSOA 2021&lt;/a&gt;, &lt;a href="https://geoportal.statistics.gov.uk/maps/62eb9df29a2f4521b5076a419ff9a47e"&gt;Ward&lt;/a&gt; - note that these lookups are generally done through a "best fit" method, because the areas don't match exactly.&lt;/li&gt; &lt;/ul&gt; &lt;h2 id="find-that-postcode"&gt;Find that Postcode&lt;/h2&gt; &lt;p&gt;I've updated the &lt;a href="https://findthatpostcode.uk/"&gt;Find that Postcode&lt;/a&gt; tool to use parliamentary constituencies as the default. &lt;a href="https://findthatpostcode.uk/areas/E14001444.html"&gt;Here's an example constituency&lt;/a&gt;.&lt;/p&gt; &lt;p&gt;In particular, you can use the &lt;a href="https://findthatpostcode.uk/addtocsv/"&gt;Add fields to CSV tool&lt;/a&gt; if you have a list of postcodes in CSV format, to add data about parliamentary constituencies (or other areas) to that list.&lt;/p&gt; &lt;h2 id="other-resources"&gt;Other resources&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;Open Innovations have created a &lt;a href="https://open-innovations.org/projects/hexmaps/builder.html?https://raw.githubusercontent.com/odileeds/hexmaps/gh-pages/maps/uk-constituencies-2023.hexjson"&gt;hex map layout&lt;/a&gt; for the new constituencies.&lt;/li&gt; &lt;li&gt;Open Innovations have also then used this to create a &lt;a href="https://constituencies.open-innovations.org/"&gt;resource of maps using the new constituency data&lt;/a&gt;.&lt;/li&gt; &lt;li&gt;MySociety have a &lt;a href="https://pages.mysociety.org/2025-constituencies/datasets/parliament_con_2025/0_1_4"&gt;data resource with data on the new files&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="http://www.statsmapsnpix.com/2023/07/a-new-uk-constituency-hex-map.html"&gt;Alasdair Rae and Philip Brown have also produced a hex map&lt;/a&gt; along with some nifty 3-letter codes&lt;/li&gt; &lt;li&gt;The &lt;a href="https://commonslibrary.parliament.uk/constituency-boundary-review-data-for-new-constituencies/"&gt;House of Commons library&lt;/a&gt; has some data plus links to other resources&lt;/li&gt; &lt;li&gt;&lt;a href="https://www.ordnancesurvey.co.uk/blog/os-election-maps-boundary-lines"&gt;Ordnance Survey have also released boundary data&lt;/a&gt;&lt;/li&gt; &lt;li&gt;For more politics-focused data, the &lt;a href="https://www.instituteforgovernment.org.uk/explainer/2023-boundary-changes"&gt;Institute for Government&lt;/a&gt; have some data on the implications of the boundary changes.&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;&lt;strong&gt;This map shows an example of a boundary change, produced by the House of Commons Library&lt;/strong&gt;&lt;/p&gt; &lt;p&gt;&lt;img alt="A map produced by the House of Commons Library showing an example boundary change" src="/images/boundaries/ogmore-1536x1152.webp"&gt;&lt;/p&gt; &lt;p&gt;Source: &lt;a href="https://commonslibrary.parliament.uk/boundary-review-2023-which-seats-will-change/"&gt;House of Commons Library&lt;/a&gt;&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Fri, 07 Jun 2024 10:45:00 +0100</pubDate><guid isPermaLink="false">tag:dkane.net,2024-06-07:/2024/where-to-get-boundary-data-on-new-parliamentary-constituencies/</guid><category>Blog</category></item><item><title>Getting VueJS 3 to work nicely with Django</title><link>https://dkane.net/2024/getting-vuejs-3-to-work-nicely-with-django/</link><description>&lt;p&gt;Most of the tutorials for using Django and VueJS assume that you want to completely remove Django views and instead just use VueJS to serve the entire frontend.&lt;/p&gt; &lt;p&gt;My usecase are usually different - I mainly find that I want to use VueJS sparingly to provide some enhancement to a particular page, serving all the pages in Django through the usual way.&lt;/p&gt; &lt;p&gt;I've not found a good tutorial for doing this, so I thought I would write my own. This tutorial assumes:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;you know your way around Django and VueJS, and just want to connect them.&lt;/li&gt; &lt;li&gt;You've already got a working Django site that you want to add to.&lt;/li&gt; &lt;li&gt;You've got NPM installed to add VueJS.&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;There's also a couple of requirements that are specific to how I like to set up Django, but these are optional. In particular:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;This is based on VueJS 3 (specifically )&lt;/li&gt; &lt;li&gt;I prefer to use Jinja2 templates rather than Django's built in templating&lt;/li&gt; &lt;li&gt;I use Whitenoise to serve static files&lt;/li&gt; &lt;li&gt;I use Dokku for app deployment (the tips here may also work with Heroku)&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;&lt;strong&gt;Edit: 2024-06-19&lt;/strong&gt; I've changed some of the details below based on the experience of deploying to dokku&lt;/p&gt; &lt;h2 id="initial-setup"&gt;Initial setup&lt;/h2&gt; &lt;h3 id="1-install-django-vite"&gt;1. Install &lt;a href="https://github.com/MrBin99/django-vite"&gt;django-vite&lt;/a&gt;&lt;/h3&gt; &lt;p&gt;&lt;code&gt;django-vite&lt;/code&gt; provides some useful template tags to automatically find the scripts to add to a page.&lt;/p&gt; &lt;ol&gt; &lt;li&gt;Run &lt;code&gt;pip install django-vite&lt;/code&gt;&lt;/li&gt; &lt;li&gt;Add &lt;code&gt;django_vite&lt;/code&gt; to &lt;code&gt;INSTALLED_APPS&lt;/code&gt; in &lt;code&gt;settings.py&lt;/code&gt;&lt;/li&gt; &lt;/ol&gt; &lt;h3 id="2-create-the-vue-app-in-your-directory"&gt;2. Create the vue app in your directory&lt;/h3&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;npm&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;vite@latest&lt;span class="w"&gt; &lt;/span&gt;frontend&lt;span class="w"&gt; &lt;/span&gt;--&lt;span class="w"&gt; &lt;/span&gt;--template&lt;span class="w"&gt; &lt;/span&gt;vue &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;This creates a directory called &lt;code&gt;./frontend/&lt;/code&gt; in the root directory of your django project. If you want to call it something different you can change &lt;code&gt;frontend&lt;/code&gt; in the command.&lt;/p&gt; &lt;p&gt;Move the &lt;code&gt;package.json&lt;/code&gt; file from &lt;code&gt;./frontend/package.json&lt;/code&gt; to &lt;code&gt;./package.json&lt;/code&gt; (ie into the root directory of your project). Change the three scripts created in &lt;code&gt;package.json&lt;/code&gt; so they use the &lt;code&gt;frontend/vite.config.js&lt;/code&gt; config file. This will look like:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="err"&gt;...&lt;/span&gt; &lt;span class="nt"&gt;&amp;quot;scripts&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;dev&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vite -c frontend/vite.config.js&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;build&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vite build -c frontend/vite.config.js&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;preview&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vite preview -c frontend/vite.config.js&amp;quot;&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="err"&gt;...&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Delete the &lt;code&gt;./frontend/index.html&lt;/code&gt; file that's created here - we'll be inserting the scripts into our django templates so we don't need this file. You might also want to delete more of the sample files that are created at this stage.&lt;/p&gt; &lt;h3 id="3-update-settingspy-again"&gt;3. Update &lt;code&gt;settings.py&lt;/code&gt; again&lt;/h3&gt; &lt;p&gt;We need to tell django where to look for the static files. Add a value to your &lt;code&gt;STATICFILES_DIRS&lt;/code&gt; setting:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;STATICFILES_DIRS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt; &lt;span class="c1"&gt;# ...existing dirs (probably BASE_DIR/static)&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;BASE_DIR&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;frontend&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;dist&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="c1"&gt;# or&lt;/span&gt; &lt;span class="n"&gt;BASE_DIR&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;frontend&amp;quot;&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;dist&amp;quot;&lt;/span&gt; &lt;span class="p"&gt;]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Also add a setting called &lt;code&gt;DJANGO_VITE&lt;/code&gt;&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;DJANGO_VITE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;default&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;dev_mode&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;DEBUG&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;manifest_path&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;BASE_DIR&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;frontend&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;dist&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;manifest.json&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;This will use the existing &lt;code&gt;DEBUG&lt;/code&gt; setting to work out whether django vite operates in dev mode or production mode.&lt;/p&gt; &lt;p&gt;It also adds a path telling Django Vite where to find the manifest for production deployment. This will facilitate the building of the static assets in production.&lt;/p&gt; &lt;p&gt;In dev mode you must be running a static server using &lt;code&gt;npm run dev&lt;/code&gt;. This will run a server at &lt;code&gt;http://localhost:5173/static/&lt;/code&gt; which serves the files.&lt;/p&gt; &lt;p&gt;In production mode, you must have generated the files using &lt;code&gt;npm run build&lt;/code&gt; and then run &lt;code&gt;python manage.py collectstatic&lt;/code&gt; to find the files. This will make sure that the files are added to your &lt;code&gt;staticfiles&lt;/code&gt; directory.&lt;/p&gt; &lt;p&gt;If you're using Whitenoise to serve your static files, you may also need to add a new test that takes into account the URL format generated by Vite. This is described in the &lt;a href="https://github.com/MrBin99/django-vite?tab=readme-ov-file#whitenoise"&gt;django-vite documentation&lt;/a&gt;. I added the snippet to &lt;code&gt;settings.py&lt;/code&gt;.&lt;/p&gt; &lt;h3 id="4-update-frontendviteconfigjs"&gt;4. Update &lt;code&gt;frontend/vite.config.js&lt;/code&gt;&lt;/h3&gt; &lt;p&gt;The &lt;code&gt;vite.config.js&lt;/code&gt; file tells vite both which files to build and where to put the outputs. This should look like:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;defineConfig&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vite&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;resolve&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;path&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;vue&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;@vitejs/plugin-vue&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;export&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;default&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;defineConfig&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;base&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/static/&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;plugins&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;vue&lt;/span&gt;&lt;span class="p"&gt;()],&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;build&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;manifest&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;manifest.json&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;outDir&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;resolve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;./frontend/dist&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;rollupOptions&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;main&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;resolve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;./frontend/src/main.js&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;This sets the following options:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;code&gt;base: "/static/"&lt;/code&gt; - this makes sure that the vite development server serves files from &lt;code&gt;/static/&lt;/code&gt; url, as this is where they will be served in production.&lt;/li&gt; &lt;li&gt;&lt;code&gt;manifest: "manifest.json"&lt;/code&gt; - the manifest file shows where all the outputs can be found&lt;/li&gt; &lt;li&gt;&lt;code&gt;outDir: resolve("./frontend/dist")&lt;/code&gt; - this sets the directory where outputs are stored to &lt;code&gt;/frontend/dist/&lt;/code&gt;. This is the same as is defined in &lt;code&gt;settings.py&lt;/code&gt; above. I chose this directory because it's already excluded from git by the default .gitignore.&lt;/li&gt; &lt;li&gt;&lt;code&gt;main: resolve("./frontend/src/main.js")&lt;/code&gt; - this is an example of an entrypoint for an app. This will create an app called "main" using the script found at &lt;code&gt;/frontend/src/main.js&lt;/code&gt;. You can create one of these entries for each entrypoint within your app.&lt;/li&gt; &lt;/ul&gt; &lt;h3 id="4-optional-set-up-jinja-to-use-the-django_vite-template-tags"&gt;4. [optional] Set up Jinja to use the django_vite template tags&lt;/h3&gt; &lt;p&gt;I prefer to use Jinja over the default Django templates. This extra step is needed to make the django-vite template tags available to Jinja templates.&lt;/p&gt; &lt;p&gt;I've set up a Jinja2 environment &lt;a href="https://docs.djangoproject.com/en/5.0/topics/templates/#django.template.backends.jinja2.Jinja2"&gt;as described in the Django docs&lt;/a&gt; so I add the django-vite template tags to the global environment there.&lt;/p&gt; &lt;details&gt;&lt;summary&gt;Show how to add django_vite tags to jinja global environment&lt;/summary&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;django_vite.templatetags.django_vite&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;vite_asset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;vite_asset_url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;vite_hmr_client&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;vite_legacy_asset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;vite_legacy_polyfills&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;vite_preload_asset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;vite_react_refresh&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;jinja2&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Environment&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;environment&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;options&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;env&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Environment&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;options&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;globals&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="c1"&gt;# ... any other globals&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;vite_hmr_client&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;vite_hmr_client&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;vite_asset&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;vite_asset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;vite_preload_asset&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;vite_preload_asset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;vite_asset_url&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;vite_asset_url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;vite_legacy_polyfills&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;vite_legacy_polyfills&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;vite_legacy_asset&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;vite_legacy_asset&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;vite_react_refresh&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;vite_react_refresh&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;filters&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="c1"&gt;# ... any filters&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;env&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;/details&gt; &lt;h2 id="adding-a-frontend-app"&gt;Adding a frontend app&lt;/h2&gt; &lt;p&gt;This setup allows you to serve different VueJS apps from different pages in your app. To add a VueJS app to a page, you need to do the following:&lt;/p&gt; &lt;h3 id="1-create-a-vuejs-app-in-the-frontendsrc-directory"&gt;1. Create a VueJS app in the frontend/src directory&lt;/h3&gt; &lt;p&gt;This should look something like the &lt;code&gt;/frontend/src/main.js&lt;/code&gt; script, e.g. it should mount a VueJS component to a particular HTML element, usually identified by its ID. A calculator app saved in &lt;code&gt;calculator.js&lt;/code&gt; might look something like:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;createApp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vue&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;CalculatorApp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;./CalculatorApp.vue&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vite/modulepreload-polyfill&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nx"&gt;createApp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;CalculatorApp&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nx"&gt;mount&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;#calculator&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;h3 id="2-add-the-new-app-to-viteconfigjs"&gt;2. Add the new app to &lt;code&gt;vite.config.js&lt;/code&gt;&lt;/h3&gt; &lt;p&gt;Within your existing &lt;code&gt;vite.config.js&lt;/code&gt; file add another entry to &lt;code&gt;build.rollupOptions.input&lt;/code&gt;, alongside any existing entries:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;default&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;defineConfig&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;build&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;rollupOptions&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;main&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;resolve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;./src/main.js&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;calculator&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;resolve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;./src/calculator.js&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;h3 id="3-insert-the-template-tags-into-your-templates"&gt;3. Insert the template tags into your templates&lt;/h3&gt; &lt;p&gt;As I mentioned above, I use Jinja2 for compiling templates. My base templates tend to have a &lt;code&gt;headscripts&lt;/code&gt; block which is where &lt;code&gt;&amp;lt;script&amp;gt;&lt;/code&gt; tags go. So my template has a block like this:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cp"&gt;{%&lt;/span&gt; &lt;span class="k"&gt;block&lt;/span&gt; &lt;span class="nv"&gt;headscripts&lt;/span&gt; &lt;span class="cp"&gt;%}&lt;/span&gt; &lt;span class="cp"&gt;{{&lt;/span&gt; &lt;span class="nb"&gt;super&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="cp"&gt;}}&lt;/span&gt; &lt;span class="cp"&gt;{{&lt;/span&gt; &lt;span class="nv"&gt;vite_hmr_client&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="cp"&gt;}}&lt;/span&gt; &lt;span class="cp"&gt;{{&lt;/span&gt; &lt;span class="nv"&gt;vite_asset&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;frontend/src/calculator.js&amp;#39;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="cp"&gt;}}&lt;/span&gt; &lt;span class="cp"&gt;{%&lt;/span&gt; &lt;span class="k"&gt;endblock&lt;/span&gt; &lt;span class="cp"&gt;%}&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;The &lt;code&gt;vite_hmr_client()&lt;/code&gt; call creates script tags in development mode to load the development script. And the &lt;code&gt;vite_asset('frontend/src/calculator.js')&lt;/code&gt; call loads the actual calculator script.&lt;/p&gt; &lt;p&gt;In the django template language it would look more like:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cp"&gt;{%&lt;/span&gt; &lt;span class="k"&gt;load&lt;/span&gt; &lt;span class="nv"&gt;django_vite&lt;/span&gt; &lt;span class="cp"&gt;%}&lt;/span&gt; &lt;span class="cp"&gt;{%&lt;/span&gt; &lt;span class="k"&gt;block&lt;/span&gt; &lt;span class="nv"&gt;headscripts&lt;/span&gt; &lt;span class="cp"&gt;%}&lt;/span&gt; &lt;span class="cp"&gt;{{&lt;/span&gt; &lt;span class="nb"&gt;block&lt;/span&gt;&lt;span class="nv"&gt;.super&lt;/span&gt; &lt;span class="cp"&gt;}}&lt;/span&gt; &lt;span class="cp"&gt;{%&lt;/span&gt; &lt;span class="k"&gt;vite_hmr_client&lt;/span&gt; &lt;span class="cp"&gt;%}&lt;/span&gt; &lt;span class="cp"&gt;{%&lt;/span&gt; &lt;span class="k"&gt;vite_asset&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;frontend/src/calculator.js&amp;#39;&lt;/span&gt; &lt;span class="cp"&gt;%}&lt;/span&gt; &lt;span class="cp"&gt;{%&lt;/span&gt; &lt;span class="k"&gt;endblock&lt;/span&gt; &lt;span class="cp"&gt;%}&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;You'll also need to at this point add a &lt;code&gt;&amp;lt;div id="calculator"&amp;gt;&amp;lt;/div&amp;gt;&lt;/code&gt; tag to your template to provide somewhere for your app to live.&lt;/p&gt; &lt;h2 id="serving-in-development-mode"&gt;Serving in development mode&lt;/h2&gt; &lt;p&gt;When developing your site locally, you'll need to run two different servers.&lt;/p&gt; &lt;p&gt;I usually use django's built in development server by running &lt;code&gt;python manage.py runserver&lt;/code&gt;.&lt;/p&gt; &lt;p&gt;You'll also need to run the vite server by running &lt;code&gt;npm run dev&lt;/code&gt;.&lt;/p&gt; &lt;h2 id="serving-the-files-in-production-mode"&gt;Serving the files in production mode&lt;/h2&gt; &lt;p&gt;When your site is in production, the built files will need to be available for the static file storage engine to use. You can build the files by running &lt;code&gt;npm run build&lt;/code&gt;.&lt;/p&gt; &lt;p&gt;This will create the files in &lt;code&gt;/frontend/dist/&lt;/code&gt;. By default this folder won't be included in Git, so if you're using something like Heroku or Dokku to deploy, then these files won't make it into your staticfiles. There are two ways to achieve this:&lt;/p&gt; &lt;h3 id="1-use-a-nodejs-buildpack-recommended"&gt;1. Use a nodejs buildpack (recommended)&lt;/h3&gt; &lt;p&gt;You can add a nodejs buildpack, which will install the needed modules and build your files to the right place. To do this add a file called &lt;code&gt;.buildpacks&lt;/code&gt; in the root directory of your app. This file should contain two lines:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;https://github.com/heroku/heroku-buildpack-nodejs.git https://github.com/heroku/heroku-buildpack-python.git &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;When you deploy your app to dokku/heroku, it will then install the needed packages from &lt;code&gt;package.json&lt;/code&gt; and then automatically run the &lt;code&gt;npm run build&lt;/code&gt; command. These files will then be available for django's collectstatic process.&lt;/p&gt; &lt;h3 id="2-manually-build-files-every-time-you-deploy"&gt;2. Manually build files every time you deploy&lt;/h3&gt; &lt;p&gt;Or you can unignore the &lt;code&gt;/frontend/dist&lt;/code&gt; folder from git (or use a different folder to output) and make sure you build the files every time you deploy.&lt;/p&gt; &lt;h2 id="appendix-i-example-file-structure"&gt;Appendix I - example file structure&lt;/h2&gt; &lt;p&gt;This shows an indicative file structure of the key files that need to be in the right place.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;- config/ # django configuration directory - admin.py - apps.py - asgi.py - jinja2.py - settings.py # main django settings file - urls.py - wsgi.py - frontend/ - dist/ # created by vite build - should be git ignored - assets/ - main-1a2b3c4e.js # compiled main JS - manifest.json - src/ - components/ - Calculator.vue - ... any other files - calculator.js - vite.config.js # where vite configuration lives - my-app/ # normal django app - migrations/ - models.py - views.py - ...etc - static/ - ... other static files like CSS, images, etc - manage.py &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Wed, 05 Jun 2024 11:05:50 +0100</pubDate><guid isPermaLink="false">tag:dkane.net,2024-06-05:/2024/getting-vuejs-3-to-work-nicely-with-django/</guid><category>Code</category><category>Django</category></item><item><title>Using Django's GeneratedField</title><link>https://dkane.net/2024/using-djangos-generatedfield/</link><description>&lt;p&gt;Django (the python web framework) version 5.0, released in December 2023, introduced a new field type for models - the &lt;a href="https://docs.djangoproject.com/en/5.0/releases/5.0/#database-generated-model-field"&gt;&lt;code&gt;GeneratedField&lt;/code&gt;&lt;/a&gt;. This is potentially a very handy tool - it allows you to define fields that are based on other fields, but gives you the flexibility and speed that comes with a field belonging to the database, without the overhead of needing to store duplicate data.&lt;/p&gt; &lt;p&gt;I recently had an opportunity to use this field in a real life project, so I wanted to test how well it worked.&lt;/p&gt; &lt;h2 id="my-use-case"&gt;My use case&lt;/h2&gt; &lt;p&gt;I work a lot with financial data that comes from regulators, and that data is often wrong or needs cleaning in some way. However, I'm also consuming a "firehose" of data from those regulators, with updates coming every day.&lt;/p&gt; &lt;p&gt;What I need is a way of storing both the original value that comes from the regulator, along with the ability to overwrite it with a manually checked value. That way I can keep storing the original value from regulator data without needing to re-enter the manual value every time.&lt;/p&gt; &lt;p&gt;So if I'm gathering an organisation's turnover in a given year I end up with two fields in the database:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;code&gt;turnover_regulator&lt;/code&gt; - the value of Turnover from the regulator's data. This could be null if not provided, but would typically be available for most organisations. This field is updated by a nightly data fetching process - any data is this field is overwritten every night.&lt;/li&gt; &lt;li&gt;&lt;code&gt;turnover_manual&lt;/code&gt; - this is a field that is null by default but can be manually changed to a correct value if needed. This field isn't touched by the data fetching process so any changes will be kept.&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Both these fields are integer, float or decimal depending on the exact need, and they can both have null values. So in a django model they would look something like:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;django.db&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;models&lt;/span&gt; &lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;FinancialYear&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;models&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Model&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;organisation&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;...&lt;/span&gt; &lt;span class="n"&gt;financial_year_end&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;...&lt;/span&gt; &lt;span class="n"&gt;turnover_regulator&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;models&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DecimalField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;null&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;turnover_manual&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;models&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DecimalField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;null&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Once this data is stored, I need some way of finding the best available estimate of turnover for an organisation. This is found by checking whether &lt;code&gt;turnover_manual&lt;/code&gt; has a value in it - if it does then use that, if not then use the value in &lt;code&gt;turnover_regulator&lt;/code&gt;.&lt;/p&gt; &lt;h2 id="the-old-way"&gt;The old way&lt;/h2&gt; &lt;p&gt;With this setup I would have two options for calculating the final &lt;code&gt;turnover&lt;/code&gt; figure:&lt;/p&gt; &lt;h3 id="1-django-model-property"&gt;1. Django model property&lt;/h3&gt; &lt;p&gt;I could set up a property on the model which fetches both values from the database and then uses Python to choose which one to use. It might look something like:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;FinancialYear&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;models&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Model&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="o"&gt;...&lt;/span&gt; &lt;span class="nd"&gt;@property&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;turnover&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;turnover_manual&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;turnover_manual&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;turnover_regulator&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;This works fine for looking at a single record, and has the advantage of not needing unnecessary storage in the database. However, when dealing with thousands of rows it requires every row to be fetched from the database and then processed in Python, so you can't take advantage of the optimisations that come with calculating things in the database. For example, I wouldn't be able to create a sum of turnover for a set of organisations using the database.&lt;/p&gt; &lt;h3 id="2-updating-a-separate-field"&gt;2. Updating a separate field&lt;/h3&gt; &lt;p&gt;The other way to store this is to create a separate &lt;code&gt;turnover&lt;/code&gt; field on the model which you thne need to make sure is updated with the right value.&lt;/p&gt; &lt;p&gt;You could update it by patching into the models' &lt;code&gt;save()&lt;/code&gt; method, although this isn't always trigged when the model is updated in bulk. Or you could have some kind of bulk update process that runs periodically - although this would mean that there's times when the field is out of sync. Or you may be able to set up a trigger to update the data in the database - but this would decouple the database code from Django and could have performance concerns.&lt;/p&gt; &lt;h2 id="using-generatedfield-instead"&gt;Using &lt;code&gt;GeneratedField&lt;/code&gt; instead&lt;/h2&gt; &lt;p&gt;The new &lt;code&gt;GeneratedField&lt;/code&gt; gives us a third option. Instead, we can use &lt;code&gt;GeneratedField&lt;/code&gt; to define an SQL function, and then the database will make sure that the value is always available to use as if it was any normal field.&lt;/p&gt; &lt;p&gt;I did this using the following definition:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;FinancialYear&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;models&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Model&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="o"&gt;...&lt;/span&gt; &lt;span class="n"&gt;turnover&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;models&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;GeneratedField&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;expression&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;models&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;functions&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Coalesce&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;turnover_manual&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;turnover_regulator&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;output_field&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;models&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DecimalField&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;db_persist&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;What I'm doing here is defining a field called &lt;code&gt;turnover&lt;/code&gt; that uses the &lt;code&gt;Coalesce&lt;/code&gt; SQL function to choose the first non-null value from the two fields. It's a pretty simple use-case, but it is possible to do more complex expression there too.&lt;/p&gt; &lt;p&gt;I've also defined an &lt;code&gt;output_field&lt;/code&gt; which tells Django/the database what type of data will be returned, and &lt;code&gt;db_persist&lt;/code&gt; which sets whether it's permanently stored in the database or not (I only used this when it was set to &lt;code&gt;True&lt;/code&gt; so I haven't seen what the performance impact might be).&lt;/p&gt; &lt;p&gt;I found this worked really well - I could access the value in Django, for example adding it as a (read-only) field in admin pages, but it was also available in more complex SQL queries that I was creating to aggregate and filter data. I haven't done any proper benchmarking but I didn't notice any big impacts on performance when using it. And it seemed to work - Django and the databases take care of the details of keeping the field up-to-date, so I didn't need to worry about out-of-date data being used.&lt;/p&gt; &lt;h2 id="limitations"&gt;Limitations&lt;/h2&gt; &lt;p&gt;My experience was generally great, but there were a few limitations I found. I've also only tested it in a fairly narrow way - I only used it with Sqlite, and only with a narrow set of generated field expressions so there may be other issues.&lt;/p&gt; &lt;h3 id="bugs"&gt;Bugs&lt;/h3&gt; &lt;p&gt;This is quite a new feature, so I imagine there are a few bugs around that might cause issues when using it.&lt;/p&gt; &lt;p&gt;I particularly found &lt;a href="https://code.djangoproject.com/ticket/35350"&gt;an issue with creating new objects when the primary key was defined rather than generated&lt;/a&gt; - this happened to match my exact use-case! Fortunately, this bug has now been fixed in version 5.0.5 (released this week), but as with any new features there may be others out there.&lt;/p&gt; &lt;h3 id="generating-migrations"&gt;Generating migrations&lt;/h3&gt; &lt;p&gt;At the moment, &lt;code&gt;GeneratedField&lt;/code&gt; will work correctly when first created in a migration, but changing the &lt;code&gt;GeneratedField&lt;/code&gt; will result in a migration that doesn't work. There is a simple workaround for this - simply replace the &lt;code&gt;migrations.AlterField&lt;/code&gt; operation that is created with two new migrations - one that does &lt;code&gt;migrations.RemoveField&lt;/code&gt; and one that does &lt;code&gt;migrations.AddField&lt;/code&gt; (because the field is generated from other data you don't need to worry about losing data).&lt;/p&gt; &lt;p&gt;It isn't ideal to have to do this manually every time you change the &lt;code&gt;GeneratedField&lt;/code&gt;, but it's not a big problem.&lt;/p&gt; &lt;h3 id="using-with-relationships"&gt;Using with relationships&lt;/h3&gt; &lt;p&gt;As &lt;a href="https://docs.djangoproject.com/en/5.0/ref/models/fields/#generatedfield"&gt;the docs explain&lt;/a&gt;, &lt;code&gt;GeneratedField&lt;/code&gt;s can only access fields from the same model, so it's not possible to access fields in related tables. This does block of some of the potential use cases for a generated field in aggregating data from a related model. But again, not a major blocker.&lt;/p&gt; &lt;h2 id="further-reading"&gt;Further reading&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;&lt;a href="https://docs.djangoproject.com/en/5.0/releases/5.0/#database-generated-model-field"&gt;Django 5.0 Release notes announcing the feature&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="https://docs.djangoproject.com/en/5.0/ref/models/fields/#generatedfield"&gt;Django model Documentation&lt;/a&gt;&lt;/li&gt; &lt;li&gt;&lt;a href="https://modern-sql.com/caniuse/generated-always-as"&gt;sql &lt;code&gt;GENERATED ALWAYS&lt;/code&gt; support&lt;/a&gt; - this is what Django uses behind the scenes to store the value&lt;/li&gt; &lt;li&gt;&lt;a href="https://www.paulox.net/2023/11/07/database-generated-columns-part-1-django-and-sqlite/"&gt;Another blog post explaining the feature&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Wed, 08 May 2024 10:03:01 +0100</pubDate><guid isPermaLink="false">tag:dkane.net,2024-05-08:/2024/using-djangos-generatedfield/</guid><category>Code</category><category>Django</category></item><item><title>Blocking bots by UserAgent in dokku using nginx</title><link>https://dkane.net/2023/blocking-bots-by-useragent-in-dokku-using-nginx/</link><description>&lt;h2 id="the-problem"&gt;The problem&lt;/h2&gt; &lt;p&gt;A website I ran was being aggresively scraped by someone using a particular UserAgent string. It wasn't a normal search engine or similar bot (e.g. GoogleBot) so adding something to robots.txt wasn't going to work.&lt;/p&gt; &lt;p&gt;This site is using &lt;a href="https://dokku.com/"&gt;dokku&lt;/a&gt; to serve the web app, dokku uses &lt;a href="https://www.nginx.com/"&gt;nginx&lt;/a&gt; to route requests to the app.&lt;/p&gt; &lt;h2 id="the-solution"&gt;The solution&lt;/h2&gt; &lt;p&gt;You can use dokku's server configuration to add a rule to nginx telling it to return HTTP status 444 when given a particular User Agent string.&lt;/p&gt; &lt;p&gt;&lt;a href="https://dokku.com/docs/networking/proxies/nginx/#customizing-via-configuration-files-included-by-the-default-templates"&gt;Dokku loads any nginx conf files&lt;/a&gt; which are stored in the &lt;code&gt;/home/dokku/$APPNAME/nginx.conf.d/&lt;/code&gt; directory (replace &lt;code&gt;$APPNAME&lt;/code&gt; with the name of your app). These files are loaded into the server block of the main configuration for the app.&lt;/p&gt; &lt;p&gt;On your server, first make sure the directory exists by running:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;/home/dokku/&lt;span class="nv"&gt;$APPNAME&lt;/span&gt;/nginx.conf.d/ &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;(again, replace &lt;code&gt;$APPNAME&lt;/code&gt; with the app you're using it on).&lt;/p&gt; &lt;p&gt;Then create the file. I called mine &lt;code&gt;bots.conf&lt;/code&gt;, and used nano to edit it:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;nano&lt;span class="w"&gt; &lt;/span&gt;/home/dokku/&lt;span class="nv"&gt;$APPNAME&lt;/span&gt;/nginx.conf.d/bots.conf &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Add the following rule to the file, and then save it.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;if ($http_user_agent = &amp;quot;bot-name&amp;quot;){ return 444; } &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;&lt;a href="https://http.dev/444"&gt;444 is an unofficial status code&lt;/a&gt; that nginx uses. Instead of returning anything, nginx will simply close the connection without a response.&lt;/p&gt; &lt;p&gt;My example will close the connection if the user agent string exactly equals "bot-name". You could also use regular expressions here for more flexible matching (see the &lt;a href="https://nginx.org/en/docs/http/ngx_http_rewrite_module.html#if"&gt;documentation on &lt;code&gt;if&lt;/code&gt; for more details&lt;/a&gt;).&lt;/p&gt; &lt;p&gt;To exclude more than one bot, simply add multiple rules to the file.&lt;/p&gt; &lt;p&gt;Next make sure that dokku owns the file you've created&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chown&lt;span class="w"&gt; &lt;/span&gt;dokku:dokku&lt;span class="w"&gt; &lt;/span&gt;/home/dokku/&lt;span class="nv"&gt;$APPNAME&lt;/span&gt;/nginx.conf.d/bots.conf &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Finally reload nginx - the rule should then be applied.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;service&lt;span class="w"&gt; &lt;/span&gt;nginx&lt;span class="w"&gt; &lt;/span&gt;reload &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Wed, 25 Oct 2023 15:16:06 +0100</pubDate><guid isPermaLink="false">tag:dkane.net,2023-10-25:/2023/blocking-bots-by-useragent-in-dokku-using-nginx/</guid><category>Code</category></item><item><title>Where to find geographic boundaries for the UK</title><link>https://dkane.net/2022/where-to-find-geographic-boundaries-for-the-uk/</link><description>&lt;p&gt;Geographic boundary data is a requirement for any geographic analysis or map-making. But finding boundary data is &lt;a href="https://twitter.com/mapstack_io/status/1600142044986372097"&gt;often a nightmare&lt;/a&gt;.&lt;/p&gt; &lt;blockquote class="tweet-embed" id="tweet-embed-1600142044986372097" lang="" &gt; &lt;header class="tweet-embed-header" &gt; &lt;a href="https://twitter.com/mapstack_io" class="tweet-embed-user" &gt; &lt;img class="tweet-embed-avatar tweet-embed-avatar-circle" src="data:image/webp;base64,UklGRigFAABXRUJQVlA4WAoAAAAQAAAALwAALwAAQUxQSK4CAAABoITtnyFJ+iKipmdmbdu27d3RbW3btr1ne/dm27Z9sm11RsR36J2s7DrNMSImgGmEMaU0SCEBpDgwCebZbf+VIIWBSDD9kZ/oX1csBimMWoyw81vaNPrz2XNCCqMSImz1tOaiNevnh08PqbuQYM07tRT/t2Z9Z+ceIXYTEix5dWMttqxZn9wGUuwgwbzn/6a52r4UvWV16IW+mPHoL7Sxw6x/XbYo/QZ2eltztdusP54xntAmMUFzsfOa9RJiu91ssqNYhr21XQgzXtpYc1cl66OLtxu59p1aShe10Td3gkD70IMJz2iufWX95LAZSJF+IJB2eVdzbVOz/nLO3HS66oRAZLojPtVcp1Kqw1csCbD2lhDaJDb8tTy9NQTmOOdHzSNK0dtWB1j2hr/z0cR2+zusd64FsNRV/1pzLvrEFgDzXvKrjXe2C2HO27Rx+OqlANa5Q/XNHSAy0/FfW/W1ldqNnPyCZn87fz6ACY+/cvh4AmP2/kj148PHEWgfegzt8aHq50fPCDAEMOVlLf58zjzESD8AMx3zlUXf2gVShA3u18bhK5ek0412nA6Y58JfLPrkVrDCDY1Zb18DYKtte4Q2ic0a39mlByx1zb8WffjmX6361BYA69ynpxHb7e2/+tTWAOverdmRb+0IsNwN2X+9uV0IM16ezXrrWgBTnlP9+NBxwAKX/GHWp5cNrQiw3r1a/PuKxYCh3R5/6ZQ5gVmO/Vr1vV16BNqHCJOe1+oP58zBVMfs+Z5WPz9ieoj0HSNhj/e0+tnh05GGYMKzWvz53LkgBbpMMPPxX6q+s3OPte/S4j9XLgYp0HWCBS/506wvPtiYrbetDikwiiHB8jdUG8365DaQIqMcE6x3v+p7u/YIiQGMESbddNeBM0JiQGNgZAoMbEgxpMC0QVZQOCBUAgAAsAwAnQEqMAAwAD61TqNMJyQjIiq6qVjgFolsAMYtKpeeg78eeYB1L/MB57foe/93p/5Kb5KrRBWgyx9IEyru1fQl/apYX8JML4zmkseemiPlJ/OIbeZyx+A3WDkbdxwZA3EzL4IxH/KDQbg97oAA/v6TlR98S3CCnKSfXmPznkYecDUwoC8hiAxP8Z2y5A0a6w/+2x4/NzWScTeWhSl/t9XlLPy/g+LRBkyj99G3F+B8lwCClV9zr19AxV7U3YpxahGkHX//8Nn6YBf44zPIaBxPNOk0ZSwt67+xXYaUDlqSwpU5GYUEeHxlwV/msd5G/51nJrWGL0UZ3DObD4TnDL++fQKrVvY7HZbr7365hgWt50wXvsvgrEauuRN0Bt1re8Np+xAc1TIjpPVGnosKa+E2e8pBp5wFUuA0jn75k5dkeSNICCRWU0QjGnDDRRaxE7BhFikWzeNRG6P6ePXdhfOwN6nXRSZJo4F1uVSZfu4TmjCWyXN2dpDS6hI09SNqRG+le0xABVzow9HAcxUEXCicy6rc4xXhRMu1y0biFM/CG71chBAN7exv2gx39fugz9tDj3BKhoVj3RtUwLG4f+2WwVBWywBm8mVHPi4Ix3UdKn4vVaCxA8ECj1cuG+M+T6yX8+vXBYz3gndEWPVMl6WhRSGhIy+KVOnI4IzagL3rB1d1gNfx10x+PVkKaeX6LE2AIbiY+nsAOadxUJ0pUMTpRKIus9fj69vTw3DT4C0sVzr9SCb7r+VrNp6dIAfNWcZVZB/J+GPzmRBwDD+oFy+9wAA=" alt="" &gt; &lt;div class="tweet-embed-user-names"&gt; &lt;p class="tweet-embed-user-names-name" &gt;Mapstack &lt;/p&gt; @mapstack_io &lt;/div&gt; &lt;/a&gt; &lt;img class="tweet-embed-logo" alt="" src='data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciCmFyaWEtbGFiZWw9IlR3aXR0ZXIiIHJvbGU9ImltZyIKdmlld0JveD0iMCAwIDUxMiA1MTIiPjxwYXRoCmQ9Im0wIDBINTEyVjUxMkgwIgpmaWxsPSIjZmZmIi8+PHBhdGggZmlsbD0iIzFkOWJmMCIgZD0ibTQ1OCAxNDBxLTIzIDEwLTQ1IDEyIDI1LTE1IDM0LTQzLTI0IDE0LTUwIDE5YTc5IDc5IDAgMDAtMTM1IDcycS0xMDEtNy0xNjMtODNhODAgODAgMCAwMDI0IDEwNnEtMTcgMC0zNi0xMHMtMyA2MiA2NCA3OXEtMTkgNS0zNiAxczE1IDUzIDc0IDU1cS01MCA0MC0xMTcgMzNhMjI0IDIyNCAwIDAwMzQ2LTIwMHEyMy0xNiA0MC00MSIvPjwvc3ZnPg=='&gt; &lt;/header&gt; &lt;section class="tweet-embed-text" &gt; Just Googled for "UK county boundaries as GeoJSON". Results:&lt;br&gt;&lt;br&gt;1. &lt;a href='http://data.gov.uk'&gt;data.gov.uk&lt;/a&gt; no joy (see screenshot).&lt;br&gt;&lt;br&gt;2. &lt;a href='http://gis.stackexchange.com'&gt;gis.stackexchange.com&lt;/a&gt; post from 2015, suggesting I use ogr2ogr...&lt;br&gt;&lt;br&gt;3. A site that has data, unfortunately it's not UK counties or &lt;a href='https://twitter.com/hashtag/geojson'&gt;#geojson&lt;/a&gt;.&lt;br&gt;&lt;br&gt;&lt;a href='https://twitter.com/hashtag/opendata'&gt;#opendata&lt;/a&gt; &lt;a href='https://twitter.com/hashtag/toohard'&gt;#toohard&lt;/a&gt; &lt;a href='https://twitter.com/mapstack_io/status/1600142044986372097/photo/1'&gt;pic.x.com/bi5gvhd61d&lt;/a&gt; &lt;div class="tweet-embed-media-grid"&gt;&lt;a href="https://pbs.twimg.com/media/FjTYsd3WAAEa5yp.png" class="tweet-embed-media-link"&gt;&lt;img class="tweet-embed-media" alt="" src="data:image/webp;base64,UklGRtBaAABXRUJQVlA4IMRaAACQcQGdASqoAuoBPrVUpE2nJKOiJbVaaOAWiWNu/GPZbYGP6Au5M6gO3bwfGm/SzD/9Hpt9/8cKGz/geqz/9eot/9PT76TPMB5sXph/vnS9/93//+6B/Uv/J////18Ff8n/0vrietH/tvV7/ar1AP//wNnjL+t/j7/R/ix38fd/7l+1P9z9M/yH6P/Hf3f9w/XQ+kfGd6X/T/+v/N+qX8r+2H6T+2/4/9l/nB+3/6T8xP6V6R/H/+w/NT4CPyD+a/5v+4/uX/i/V//we2v3P/d/sb7Avsl9R/4X9//zn7aenv/sf6L1M/R/8j/2/8d8AH8u/s//Y9c/+p4c343/g/tl8Af9A/z//z/3Pu2f3f/y/1/5y+5X9N/1v/v/1/wH/z/+7f9v/Ndrn0hSmBBm0/47bnZ17G252dextudnXsbbnZz1cViExgCUSluOlcfWAOnx23Ka/KOhCCME26AzB+KeRzc4c/2PAVlXwaL/BedB88f4k7WHBEnXq2tFpAUS0WQVljDdZaMqTfuErHTtK8/1GnqLDVxdHGAEYtyGa4l8GdiruRViKCzrGW+fvbMeMH/VbQyijTOzr2Ntzs69jbc59lKacjs80nlw9KfEqt/6c5OF2FOXD2FHnmCa4+cnT4HSbbVzNf3+iWEy1DsqMQ9N4gkc51o73OunRUgd4z189ehYlUthBk/INzg7A/7x8I06W5i2jPYh12WwEamTNd/47bnZ17G2sciDq2pkahZNGD1GPQFdXiSgotTxJODTWym0vfp2hySSwwlNpe/c697hcVi3u3caDqqsTSDDunUE9ctZyYQgsmVxiIVksDyKRxHxWwIU3eLLsToU7ic4HN3yRFhZ6dd6oFW0GB1AJcCQHG72BLu2v8pJLOCCM1SkJAPV8KT2CPjZULINj6nrGrEcv9O4pHhfaoE4nM3eWbKl8jguYwGkN75WXcdrUEJEhmoQWo6JX96nlBSlD6sv/r13Rgxf0rgOET8Rms0+NtDqSAoc4tfpheAee/ilOSEPyIyQH3+5Z4aTa2UUKq0c3VfvmrtDue4dB9GvBNf7GnSAvFPSMlm9fhADcubkYZJJw1ZpLNGSM8jLnZfwletOSKzTATDmvi/S980PH1pQK+HXBDYgMZYMEe4PjB5tN4QrvaFqzP5d7/Gso/TNDIHMCBM0RK9M0MgcwGOTlVPndTWU1cx6fj9EZPLpm0U+d1NgJAdkSxH6IyeXTNopCZrv/Hbc7OvY23Ozr2Ntzs69jbc7OvY23Ozr2Ntzs69jbc7OvY23Ozr2Ntzs51WYIK3CPCrhi6j5ehcDtojeO252dextudnXsbbnZ17Gy1Nr/FlEAQ2AGe1c9QUR01hjkOiQz+eVSVHZ+Sv2i85LccdIyGzlGhfX2t+J6CgxltdUdG2U8dUieza9jbc7OvY23Ozr2Ntzs6OCtTIWt+Cjbx23Ozr2Ntzs69jbc7OvY23Hvs8TUrDqHw/WTCih627uGoqo1i9DIfsSV+WykV0mF0XoTVR6eXPcltXfn9i9Oyv1vSvyOI33FTqCtXXGAW/TSUSaaVCLB/r85WMHp711SilQy9+WdmP6UyIEd0CyPWG/vQT3ZrXkJ5Laf7+hpIq9GqWKtYrODMRJKogPksyepFdMVilaWQfrdK73/PncW7h4mjTl8vP02geULWN9RYbvK0m1xr7cG0B7lvg2p0ZUSTTYwazahu51UZSPa04iept1R1F+8SFfji1NJ9TSbWwxkEZR20OeT6Mtl4f2ez27SdWxfPIfEl+vuYB1LUfgjn13vyl1Nn/d/wQc5Qok+Jmh8ZZVZH+qnYw3//A9gAcY5z2+rGuuEpFRt0pGpJFbIepP1arhjTXMYGgWPSCcxOz3keqnHZ3RnDevCBVBt1HCX0Mm3ZwOCSxj621Q3fdb1rYyXSh+bb1mS7D6lpf7J3a0idtQrttEyZgYwy/B+heASAl5Lbym1O+WUt5lrDCmHnpyh/Evs85TYh2egVgoVL6XCvAkjoeDDJKx0PhmpC10CFi54u3On4sZAKddl/5zZTh/JOvv0Gy6xhxCg+Fh3P5WcVU1C0L3yrTZ54IC/sikqEJk1/rdSh3uUILHoUCE5ARqeAmj2f9sSk2KyUiSvvuJ1Iyg0LFnHABHeQ5ak9b7w0sXBhNJaK1U7ccCiqAfP7astZCrwdXoBmva1oQEHJh5AXcRh9AbwtuB6iYHtUB4BZgrxoZpYBaXQJgo416Zw5j+MhPwrxE/tjLGAUV60itSwjY86H2cTWH/IhW1ITytPpCiAfad2+zO/tUp82VQ0oFNZNUQBityDl1CNN1YtBg5oQhd++hkwy9ESL/ucqRdfUg97qZW9304ZHAWfAZsPBlydB+Ltb7aHMmslTo18VK56aUO3/sK58JVrPS2FYk4rEkBL2OfbL2FwAsKAEy4c/1Ipx/bh6a3P/Hr5LpdEQI+NxG5IX1/lDHkdV3uensH8pIpp4nJLpbUpnB8mtaFq/DEBsjYDaXm6aS0QtKmNsdQdJYCE+Cp/C3ZymnwDdWJDg82m6GdkLrj2qEhuWlICR4fvf6kNSqg3tkOYRIvnTrqo4nzMdrTaUWFf0TGlL+0kd7A7keo7D1PS+BAbv/G1XMVyrsiL+TUu1be3X3/Jct6369Q58hlWRGjQb4uWFDFfK3W6Fdy2iHiBjWz1m3Aut60lBvbNbXTKQ+XUwGFzCdQdYQY+8FGeIKBArejXKq/J1+SMg1inlVdzOmCNq5Zyf3pcOIduti4cnHATDzgVx+/D+1fK+QhYNIYJmkMXHpw35dVXBo5DR9nCGjgd1IrrlqYwGEbzmFzAhVWSyI8s6QuXSM5vcx+dMS6HzT+25MTRvHj9sEc8H56nwphon4T+aEpYnfBUAk24bdwfLk22EN5ZlZj+q74PvpdBtwdolL9zLmTMoUNiYwEEGyUmgvRR8s7VDShYNg5EH5riNdgvHNNctmDHZA2WBZJhG5pSyeozCUkcBKvXrORh/U3Sjk7peZyMedPZs0Fz3XDiFCmGJCseG/COIWSazFozZwgUyHPhsk3qV00LsmsNC8w3jLdaidrLgZfHFZRkjWxvmYW6OcJxWKotpE6LivGoTa0zvnuOsj6kL3MNLuIsnevOkesdXTFJP9RYghZAyE/vVecsfyJ5OhAG9MArT0m7crCL1alTpo2KZHoOxuP/ieYV9dsCIttbN5gdQ2G5RCQ123GAsnVQUKSkX1ScykYEcp/VusGi35wSxHqa7Cq5jFO6vViAChGVN22eTe5IEk9BFmIWUYyRKhnGs1EaMWnWC5Hi5NIwsCB7hyYStrf4HIQelNOozqyd/47bnZ2aDbc7OvY23Ozr2Nt8D2i1L1Qzaf8dtzs69jbc7OvY23Ozr2Ntzs69jbc7OvY23Ozr2Ntzs69jbc7OvY23Ozr2Ntzs69jbc7OvY23Ozr2Ntzl8us7xfptnKVfGzCY/19mefm05OQ/w6HkqjLjlDTC/6YZG51TzRUrAkFDiRaKkiY/1DEDV1dKuEVLXZV6eUpZojJ46WIgnIwrh7vbsocW1/RdbNrk1/0o6KPzSnMY16Jt1CxyHztIt9BivMCuFIRjUyGIDohgMw9dTYfffvnlB1926Xx2e4bEEopBZZ2wt9xxwxxVXbZ4G1GtdWbBAaXr5y99qzZjxM+P5XrD6OuHlbiZimAZYJN5IhbKDNktwkQRSoqB1V9L7HuzhaJaKteAkZ7TTinOLS5JlW8SExkVf4Zjc/gLV1/Euiy855PyW7VmkmE42Z3CrDhgoF4s2nE222PFdqo6aWr53Ef2NQwC6Syhh5mYcsqttANjJMdRMiq9CX39Bi62bXsVDI4QmQdVwjalA3TTHmm0/zhequEbUoG6aY802n+cL1VwgN3vL5ey+0/47bnZ17G252dextudnXsbbnZ17G252dextudnXsbbnZ17G252dextudnXsbbnZzAAAP7/F8gAAAW+A0/gLgGSLUzwT/O1oFkkQRT4BoLHUhNfdYPMRdugJOcs9TEgE1DGNbg2ZvzA20nUVRHNwuSIFuC5X23TTVAhFAVKvaVsGItGa93TVEbMRgtBI/WsHLSULKv6KgNwj2kvhTgTvLNepE4+RckX9gMsTGsqVzEagYXogwJ7p7yCQGArz+cWsCs4DMMUYrJ4W81bR7hR2/1dI9tc+H/VAH2TODCIjc3clxJYmMVjQE9VAt1JaOCvKW8wYY0+Ncek2cgZZtJ4AkEvB/Dgi3bAM86uLM45C0aj0HWmXWjXklMBXtN09liOXCNvT28hpzZEm1LUluVmMNJ1TqqJMf2VMlhH4Njdv/a8unh1ZWjSncI5gpmKC1eSt9Nz5+hmTttdwh0w1J/adM3ZOPXIS2MtKg/Ecu6agJuWgb+yuVcl2pJT6vkIfTFLwwJjRjAliMfPr9QzxH+9u4b18ATvLvqpsUul/558fMWvS615nAZVwTBC3d7u++oTPbn1meLZ8gEPBh6ehrFcLloCG/V7S8gb7iO5FVP79NzHnkYt8ndPI60VjgGBiP7vbvPNYx7s0VGKlE2hghgqFUej97rdZusAOe8mT0qes+5gI/cWBJFeLdBZsCTBUy5CTK68DKW5QuPxb22MPEZYH9U7CLT64SoUE8rkHYnDTK54CSJD8kjEf5m1PlnXMT2PUdxSLkjZ8Frck3i2p7TbNcAjt6V7Ook1I7VoSw8GRqPOL1VSZ2qXnYdy/LAou0blevx80nH0l54wP/yW3CdR8Zbye1KW1nBUxYbIC4VISDjVhT4Kn82qIrBdyUBh8Wh/bRpQrZnTGeZM0BGtsvcAkm+sk6hzTn8mpH+O7VcFyro4ordZn+28+iFt9aFrq6u7Wt93SzPyXhST4q3V6+gxBGDv1BUqheRR4oIsEIY9pBaIUlkrnxq/YOGVk9Om+5GNwHUGDod5R/P1gh09vCsun0NJQuUrZPo/NuUla0O+aHlRH4NYRwC4hModuceOGIDN794rpMTKio8+u+reUOdACMLpT6Tl1WRlQrhXunU5U0u3WIOdsDutn2G9JB2/FZPX+e7JFR3eVejMxFEpef9l0XmKmso1dzciUQmMIYavEOOBNuv6ACpC30O34U1YPwNeziHaUL2Do98g9ZWxEBWwqccn1eB7aafsormi1FlQdUvVitvooblwvOKA2JXwZLuRraODgsH79K8O8wwcb6oQhwPAAUZrGk+d0Wp889IJ8jPWiILURn9cIM46871bJs6PZu9JmDXCLmY3DFNIyVcYgclmg7EbTtniLKs2nu3Hwg/MolJ5KMb+I9IHg9b9bPTmnbHLpSBNHt/lYAky56Zjn2pQJjZt15S07e0UbI2bSdlmYZOEWc8e3kgNMF8uud8OIzocW8gSAcElA3tUW3ERTzRCIlmMwwz6sGm+Fk1ffNd99UabPHZgbgy95D+zZnGPd4kCaHIAqq29k0T0Xb/Au17ZdOZrjR/fsBm61fPwyI4kVDsk9pwZNYsBAWansTIFzSX80mgZJZOG0UFHAtxiV1lk1swU2gJIZCwv8ENeucMb4xiE06olF//O+gJ9z71M2dRTa7EPgae8tRLWx1GgqoAAMcC4rSxxE8EABl/3ifltIq3WUA/8oM/qW3jHEAETT2tc/7g4X6eXlQB2avhVxI/hZZVLC2QB8Uam2SXJUPxbrsHK8Pv+4y8MRepUWGmhhhBLnZf/LoxxRbN96MB1jd5uR93WyDqguZRhfJ9ddRysOl3vlq8Sdy0D5AGbDlh0sNzjsZB/d+jnMB3wZWOJXBZ1nZQY9bO23Ql51nW2x6ay8KFWJ2ruwvBr6dji9Xen0ODvjUv1M2wUC5sjBHPe/bJuwjj+jb2nWY0GFCysF02a1jKivVj15R9C9Fh9uiW0hwwjAhkINCHu2z78unZavLg0/ddSRIY0lceVDdx9V4kYikLO+gU7wGJxIVz0PJBifZ12Rs3m2rRD4XN/5hgN8ee7MNWyVTmzYSQaCjPj0pUwqvLaRRVMswN8lHvzE/uL/ykT9dgmBlhA3214Ob2sIw59E21B/Kc5VRk01STBs249hN3wA0PGa9Qie9r5JE9Z7b3v+nqDc2t03pzmaTw1oX0fmfhJjUnaTc9qeLD4vv0lglsKxewE903BUoIpT8NadNnP0aKcyVIAyjaf/kebCR7ENJLha8PT1rgveaFYuJ7aDjtyzRy55PhUTQwn4t0/ZPH6siSvV/WY2KHU6Z0oaoWYc7l35GlE/6nL5eEjUE7XKm5D3gmfYwgjAPt4783WWAcI47EqZJPhCqwwGCHx6CK/UgA0OVd/BUUmU09aBUOujCaefkETf6iGbwPBQhlvyZqdtfqs1wg/JwtHQLRKb7wA8FlNbB9QjHRO0ui0GHTgoqkksOcuYeLyMGxl4gs2E39cQm1XrenqEQV2qEQUB9H8CvK71EOjuYtXpHWHV3vwfqM9H2M4ejr64hd61H6CT3UR4ZPhJNEfADIUC++pv5QQ7cko8MmnLQQirwPrEE4lDsvYCP/lU+RzRbwgTf8cPsC9FrVCNBLadz/71RMgqm8t0BhjeHbEOgLKtCEoWPIJnKNIi2HFWEftPRr5rye6xxmAnz3FblVSCZsTRXo0PZB2REtSKllHugyByvwAAkP8tvJdcsq4AAAAAi/wDC7QAClOVvoutAbS4I9d5P/A1/O7WFnKL68lltfJknRoQmzF7OlMXeU/EiA8GVm5PKgCTEW12S6ClOx9hAJ/U2KaVYXozkwNT9om44JyWbGJUqhaJd3S/3jKvs6TfeKtRN67bOS5zWASccMXoXvhTcT62Io7CFJ53pEA38vG01ngcjAhYIyoB5YAB1TRLZO34hcdy0aUvEzMRHcjukbjUJsdGmSylFDf92IfbSKB7UmMavJN28niNhe1XINbPJVo1Rn1xNRrUS0TajT33woQJQI7AYhd57auI5ecb/7PzTHilbVPcUclIvEMXkOf16zTpSwN+XlRZ/Lm9oaObeReJRTpvzFS0oDhSELXPRTFJ7UZxJUsyRtarctgxYcDTsztglKs626C5/Zce/9JaWrrvmDl3l6evBQ7VeUZstwkaB4nYwJLdkbbH0eBLtmEq5eg9Rk+myhxoH/az4PXTP6gT15OFtuN1tRWRtEZqHUHazjrWfMcYbDPijZ6oAesNu3YgWid8nIbdYOu00K0iLL5pE8EIpI8nnKUkRvpfnLBiGernYZfXtGxKl88TzOlHgcfDyCnrhmsUv0Mf/vhRyH2BF5oCcGhsYF78efOUuPT6YzWZ7hsX/r9Wq63OXdEkXdw2fq55A8vfj8fP1o1LvqJQRt9o6pJ6+QJAGsw1IZo+XLZfbqkz+allnVgp5iRXSP/u9QGttylvn/s4Wa00JLjb1gbmbb0vAqCDu9mTr99fvPqShWnu4GnHDoGxLZMDXFih4HewLz2dBE550oHzsEW/2o6alpyEzqnt2TmMsDC7YBnHvNXoS1enIA3DT/KyZ5oB6WQ3wcgt8+EVG5T3UZKWTtiozFY7oaMtC9Uaosf/i6/ZMo5SvXmV+G2gBN5UlAArejnsu1kc/UUdtTOpKeEMOaq7TQOyx1IK/b1WGw8A7JwURTD3gyZK1Sny2emtJbHTpWJhZx0TCalb5BMMIDtDUtlA2vm6SoWx6n+BG0m3rLD/uk5b2iYtgPQiNF5Tyk5SR1BvhHymBztwbcDEkpFWLyLR/Oj6ymAFDNjaLSm/ICoIg46sxv2HynMe4dN/ZssQjZaqM86V5r9fNdUe2PDZgVoZf/5JDRYGuva7eI5LmbuY3WYZIu1EaedmmrUgScuZTKDyMJEC4vW1C/ETRyYmaEdsMBEnasEyIOvxhgnxHxTbnuUqcdzK9UVXfC88Ko3rx5yirUg8vmPYHXBvE8/03JIHVcVbCxpNqcFZuy1L+m6+qod7TNOm5s7txDiTlNvIr5AW6Nv09br5wYqwajfV9ks4XTuD6722rCDsAcu0CAjRemGwH7X5WlIWCaOpDQ/eZUK0qJirx/Emvm3u/lw7yEiAygGUMlkxdYlAJuHXhH4+ln1Rj+vLUjm9wecjV3z1qAa1U4EHRG722ln/BMAoVRvFsZbL4QeFw2OD0Cq8MIgNqS+t559UTTvaLm7UMtYATnir+F0EUYoeQByt8/bVRvKEzC8QlCn4mFKusKZsl9S8Q61MxtbhB/f6SCFB9XYv6z28zDYa4EydEutYHmV+u0QcvdJw1CbKdkF0s2yZyqcLXI8wvk8aYDh5ZHA4reJQWU97lm5Oo/PvrWy992zmfBMOlma2S/jNH05Wq8GtfbqJSCvHNfdSU6ZqehsOtNFTHz5g78vVikKnJjQY95fV5j2TR1MXf7VCno73k+kBJvHhav4aw7jYKCqJ97Yp/l5rw6hQk18StbTPSGkpaTyf2dgDKmZpcAssGj6OSUvK5uzeBS87jo24h6ORYaDnBWyOGMC2MwLfnPscewFo69MkxefZrRG8mEf2IbKbjbJsqZUhBO2kNn6pro4YMtyokogFV1776NTTTox0Wx/fFcsab9AWFmPwLcztkFe5QJnpD9CHh7vrwi3cIpCas2P1tVV4o09qsERJVXkyv4LEkxoZUYnaWREDfToyfpZYBHSi/kKocqKlYFOU1laa/vzXkfzvDT6+mJXhXJF8gt5+nQuKc+fzQSCzl4THvXStabGVRD/kWew4fRdxQoQJsFWnQTKKYCuHaB1AEfoXj2F5+zaGccC9yh4tk3hbFy2AkSJy0cqdz9CkbCsAxIV5a6I7CKh2GyTsYFNWH7VaXLKc5g3zjyyQWxORcQIrnYAKWmIfVocvsFrf/O/SkpScqxhumsieNJxqqOr/O7t4U/Rq8Eo8umKSuVi+xxtIuc+9ePTuzoViys/GmIh2bU2kE43nFk9v1e1Z9uoF3MGfaT/1k3NiMfL897KERN7Xg3Led50LTN5kvZMu0m6guv3JD9btAiqxKzLUOFEG6RsXxieumiv6JLnEJW/WiZWjol/VS2Ep0rMswmP6gybH28hhaKW4dd01UasFyuw6UwGxbKHhZWp3W3pt9+5SYd9HtkhPl3tZGCcjt99egb3HOGrZHddTDP9nNbuCJqvLW2C6hGxSCoIDbcPtNYKBrmd1p68xSjocoNcRX+IG7DhrGb6rSrTccusUAy8VKgT4/5CZ4G+0Cf5vRsyuyHlRP8ccCKlITKrG5Cw05c7gTnNCHGbrD1c/NLFycpnbORMHzKdkk72WNHkd/0gwcQ+Bkl9Skszpfk6+hxSgY++vJN1+JiLf3km/qSRb/PERpJPExl9KB8wQWVkyqTcVa0ALJiAqy+9jiIaGFyadagAB0eArbq6ScuribEwct780EDNVxAFFf8ZA7lk66nBw3uLQU3ssA6oXuR6+Srmi4dxIozVhuDz0UXscja3KMfX6JU7UGgLTIB79+yiMK1jW/mVO1HKXsOarxb4/rW5jqQKBpfHaEPJzi+aEWRZPR8uCD7LhbShQa8rKioITrFwXXGumu1J0t0i4Zhph6Pcoiq/J2HBK9jJ6vZvTTwYAWyXSfuZS4Z6dvEv6tL7dYwNyZtUPXdnyGaYY9PjsOYfpd/BU+6Fv5mfRxKRAD3uoc3a140cS9mxPQZdkqOfmAY8qO0PjYcNw1bFquWd9eNRxJQinPfSaTarKqeZZJ56R9I3rKo+9XKmC/wUKAJoojAAXak+cwFRJ8WSesuIYH5IDxRcZOfeEZOTKD4+BqMzd4Df7bFeYxvWRGfp8F2YEpfrBHpY2UNn/Wbz8QYuSgKOjrZvu7wsXSrLSeZw487Az+p6hLQLi66q8jTZvMtxog39hAy2fCQxbWMFjXa9+qAERq113ROr7O6nfbhKVQKUeFxn6Aj4Q2DnPrpguv/Wz1GwHdvJ8PIymYpK2ktLxBpaQNODVaDS7AC/M+NVHPwPlQAPAXeeRxpuAY7GvbeBMQsop8XAkh2C8CuF58NO0RYinsXJonXl5vGJUu8sdqBl08lL6gwu2MnG/U+Waf7AAJX+KpmN/JnV60e2MA8quMWdg2v4W9CRLpb/69ltbjIwuATYVW9ijezF4EM9cF6CDBdQynKE+w8GfvYZvVLaWhXzMBhRJhXG02cRpNZ6svu+/dIVk+jvQ6WcCkJ4FdFkZWzh+u+7rCrQbZW08vZ+xvbXR48+Vd7mD8Ha9mIlD6dkErCYaNLrn2KLszX453ZZHL6Xl/8K5e2SE2fznmppHqsJ2bDwsA6vx7y4RSK8Jh0/Jkt+kSwoK5C8UKDHodFHpvTnEtJs7ICU5ruILKvp5qZeebl293wA0l/j0f8YrYfPlncAE0f6jJNheREx2LGqHceigY9ZrP+84kn439RkoWyxz6uTQ1o2D/eJStnbSdQeiHoVrB4UVPYMhdl6MCMN3qzIJ3IJw1VNw1ecu4siGUxGvBLnW8m1EPzJcq6hukOM70xXpAVu0ObW0oYhrL93L/Y+gD3UDoQc3VltpwAAAAAAAAAAAAAAAAAAAAAAAARIKtNmoFC0KeuSCHhvfzN4gBdCFVgSCbeVZYM4fGfuUqJFEBguCasHdBqbCK6gVWhdltyb2YBg+7phJrmugZ8PjwZq1tNB9jLVjwFz6Bcck4hEMOP/waCNVmERnCxRR4Ohgom3K7FvvOXgjt89UJ+Q3x++6D5gS+xZWEN+O3ugR6pxr1nFkD/JN4uOlNaR7iRQZTWsK21XgAADi/LaccYnTVPpt34YP7im3w2lwizWkbscwALks75EntYunrGMdkm5FJe2bP2F5NU99XtIBmPLUiyTAkYIpAev6P7ymmB4UttzLpaaZj4Y9i2uXuAr+tXzV8YIV44XYWYvg1LFcn77UvtJ3yd3Y64hYUtUTQDMhnldF52qJ/JxRjfCJhTd3gOHYlqyHJ4Fdw93+PxCR5FVa1XSqh+xUZ/NKB9BEhUwFEaAYbptxXK0BAr+HIb43t+PULb52lFKkEf/QZtdKeJjoM5a8gJ6jT929ulwCgs0hIhWQVoWXJ5f1Lsj/d8RclU7YdJ1BNgFS6QG9jVyJ6M316z6G1uwEedcQ3CdNymGfapi069P/KI2wZ4iQjLN5BjgUbEmEvtqIP4GNIPqr1ESi4L30FrQqPAiXDSrUVPrXruHZNdfVDI4j5Tcev1orXZbrxKj9get2P6w/wSUaPifEWKSIckR7MhfUnISmZIM/8pEEfoFwc9Vy/GxObylY3kc2JpFDhXXa9ZZyH3OMSsin6bNJ3S/hWsITAOcz1CGQXvCoBhaXTVo1WohZiKYjizl4ivY1JXO4HI7PYu201YkhyQv58a+O7AZEktBSKnSfZEQq+O//BcqmB+fZTmWhxciYFzEli2G0fTzjg2kBqmmqV2/HP4lhnb1GKJEenJ3fmyBqdEAUVeOcJsvWAV72Ld6RGRePHRUj9sO2VMUYWa+nm0kJgF+3jD2pnBWTrEurr+gP+5jZRBEyPygBKIwP2IV6aggHQkQ2BB9T5jVh+TsOGEyD1U0kHLqbA0xEr48iTC8n7KImDTP7TJO4D+ujSqMgQOO1/N8bVLozvvj2VDBQHGOlRyazhLkZr4h0vZq5G03gUat4noVd5I6FBNXDT/27n4VgtOYEqJurjccaEAAAAAAAAAgmAsNuPcFixji1pf7JVRg7TU0oY2484nlwqFqrGPGxcWpyAoge9iFijLM4h3g0wbtaFDjVHQwGGDPgi8OSeIAaRe0UBxUwTC7FeqccSL1dzgY7uNdNnkjBaoyZCujyJWGMt0AYX5WYFPDyJzZpMM02uNdfyT+xWE4scDf+fj/DCyOWAwqXGNYmCTiZSlJF5xFmyKVm2tuTCRJhYxY9M1BDTqWHqfatv58+IXdbD7LyeCLNhFojH0iO1IjwKrSowZ19hr8RkepuEpcycnb76UFp9UaLP7eR1EeyUdbkpMOSfOVgciaV479FP2kfmF0+oS70i6tIjG7+9hTNnQ5u3mth0PwBVRFOi1pps74XColzVlsF9GJl3Yep7bqVw1kfqQikiEuD8EvpzUA4aiNvuMoGRMCB5SdeN9GGsTNaHKKJbYWjw4MpmSnMxIC400s67he441TPe9RyV3ivh+qfbASqPN8OzY2XCg45n+K2NbCb5YE+qFRPVk1wbUxnETWnqrlM/E0oTIZCePC9voP6yVr2S0szInPa76kFekv1eSkS6en3t0vw7aYCzR/lCr/VFYcWEXP/yMz/iSvMQDUYmDsNr6yyM1MZ1ioyu2v4UzBNgqcuqAICVy8+ykiwIchrTE0WBHvjMCmirDSG5+85IoyyMWXQMEwxbg23emRPzsUvhx0byG31sxElwqT6fZOnDE9GTBGzRqY/zr2Occg803h7muPJxkhqOw7OPOHSnX4UJh1cREogYYKrTFEuSKRlP0jRV59TUk0wWKbBH9H7G3zUmPMsHEH8CXA8iErCKfdkfTE3ujaOSo77oAneH8NHviX9wzPyxIlkz9s1S+MMWYzbg2Le3y3lDut9AV3kyCqJQgiN7cNgrf0hrTDh4KKWQnfuTZAlijg9Lt5mQcU+qQaXkJCz9aRpj5cjSfXZwmJrN6b7QhpiFbmNcKVJRYm+MtiYqBkiSHkDmYTmGp4HR72KSNI2paOpMPNayHWEKo8oi34EvNxfqXNVvJfgF5gKaDGj9WecX9mJ6/Yt2Ho9u5jOSVfso96gm9xcq9j05cFWaX0P8xlmDin1SDpGy1HLBwnvWYCIGZ9VzTtVUr0sHVwq66SVJz2a0nPqsEFlPL1IAhDv+iGEExH2qPIHFz4Ve/XxE18VSzEOl2zuPEC35bZSEahqxu3AUaOAjF1ZJrhFNW7Df80oJ+o/h2P8dIjKTuFwn0smuTjrOehSO4m7jdRoWxaGY+fJJlPBzAixpGvnYtgJPP5Cniq0tZw4Tl+k9UfqShE6Vam2k2zPGMQ1hyGYnzrqbO+NtA9WYIAdrKIZOITsp6a0BZ+CM/MpbDfU3R6PApiFBcFd4k2QKxBA8C1+ytO0TUxn2D7qaD0N1V7fmtFSHqrmbrZAKyOkeMGtBFsmpqmWjUGff3K+rdnuOuqzjjjUXx+bl+jDlQu8ELI7hYTTDPtXoe06RHhtCS9Hj4dyJVmhholPT0gdmsRhIr3lftj4tgGFIXoKLovHMvFpVjj8xwvLPdq+CUe/hjaOsXoLygPyq2JPuY3a+HpV+5SxbACNjdAughJkHyh5tWOyYNGAf5BTmvmtG2Prw0Z692bbbTmhvWiYrfztTbUHkFBm8Zo6lyq7pI5PQ7IHprZcvxrlHVPr0Lj93lFnwcUQ2wluywk7pce9WwPg+fiGJ5xe2lj9cEwx2kZ0hHqqZkXDOskMX4b5tSNcGs99gSxcpSZD//RuA5/Q4whyDRRG+4UB7bZsoyZ8hW083nHLjOwlucGO+MKFnQkwK+I/5s35P6mXqOO/dTXv4ie06zzDXpIW+WGMzLLUbIwVk3KtTvPzoms0mUUu6MG4dvd0tBJU0gR6gCwo8tEMcpzAYPzD0N+EnhKCqn0MeQzInTV0Yzy9ZH7Jz2S3Lf5vnD90lwuMDskw4CC/XNrnXaupuJhAwhqfp4H3L8LA/wG/gLiOl3xvGfGbgtyLOHOGKATrkLrHxFUaWDwl6dmTzZdw6MDIhautduMEZl5VpGQYYOHQCjA9Da0jLRzvJYCc8o0+6UScUUrpUTmObmRr2/bcropA60xT8PWJl8IIoHg7/YxZP6LA2xB/ukjxRB6Ie1C2tFZ9urW9Dsavyr4wI/1BEAvMBtdFNKFJM3C74hy7AymgGOforrTwZDV4AFAMTit+MPK4ma9F3SgiPldmr12vrQUy1o6zPsRdxjx1uNwKhWUBMc6kdHUimUGiZOzUltvqnBH3tBGk621uwIRNWSmO32g8hbjROXvoUC+/PffxtJEY2gnLagiGiTR4txwB/kLCSpS7jKShyR5BBS1dYawH6cc+A6r/mpxFe9tm9c0N1b64VjGLigmVdN1skCv+O3eLYsed7hW3ZJr700tBQF+h0o2uFmpbCE6TxT7xgXYUgOAgMqFuRy2YauPLC+2w4iS6xdYEotsJj0wue7Ecm9dOsTD9wRen6YGmN66NWf8LWeCpSoxg5c5gDnTDQQ87/ZwJvAIB+l7/CqfEb/ipiKYOUqoFxgdvm6off06PspFi12FoLWidWJGbVfn/2Eyi4S463BtzxMnYegZutxHhpS5gXHziyB3IRHPNAuLmRKJB+Egc+KsUEgDuVppVGXr8Wlvro76of7qVHsImkAussakEQxepujrZ6d57RU1UL5cb+/2IHDMj7W3lkKzG53ZWre64JFHeiuHqVAotLgFuj9w/w1m8IM6LmRRtqZyrHIT9kbZrFZQ3a0ahyozENYK9CeRx41Q+zLEud3GKOGALeJzWj7D2/eQYXNt4zg3iIOqW5vJWL47YRh8kEaGp6GLT9uN12TcpGFNiWPgMi2rj4YdzYSVyiasVgNhC4eMXdgqdst6cjzxfvlhU+mNeL5IioFX9Vc4THGguCItXrdQY31CNNsG65/523zqCVtJEwkQ0x3pz1hD6NfQo7dXF2P1RhjGk32FfCLHoOqi6kISWaPodmxnX6oXiAW2tFr59J923xKrRcBqsbAlv424PYFpdcWHT4pqAMaKSgeZMtbk89Us1t6RROgDAMZmdYa2i7CnVndfj/y+eFylpSp1tXEggs0j6qYlrH2ti5TGw6xhwC1piu81njIzjjG0AJNFPFF5wR0SySq3P7U4lLEURFk4Iiutl5qltc7LWbvhrYEvgcZuCGDNARzi83NaO7WkqqxejU4xxmCjNz+yg0vdNcCYv/6DeDhTVIJPzsbvHzfNHRN44/oxrLdI33lrXjmE/7d0MXe+SQFnOpfbOJDJGD/thD8+EJhYOln6Fr/XEqIss+FIGwbp79DwzDnBy/9s0/TMbn0OY9lyZlqWpt31ijBgW3BoOfXsjQiGRp9q5RHESgez3iUyArGrAI5yiwgK0Fuz1s7vuSZfoDH1Tm37hV4nVi2q0v3iuRNvLEVMRB8w4IUOsZBxmdvDhATd2I867rKmuirWtfnaBdG5oU/dO5X97gy+11xcK+E+MzZxaslQZHk+0ZD9GhvkQEGSeDQWjREUAU2BprOhrwpv2qmJMLQf/R6O8IIjfLanCn+x/WpAtRi1G9QKxI+unvOMPGNbZauQPOkLU2U5xKWV2FkVM27dtqgPtDK0gNF4wwRDik4U5gvLKBgZoJ/peY9qaka8rvTaf8Z73qyb/sxiqSn86kfAF46UwY9axQ7+6Px1Bkf9M/OcA7vbDEnDF3S/17vUY4gHgsoGdpgCCL78NtuGr8aNEo3aWS39rQaP8Fu1ZXhLih+PYJ85bg7GiOlqaYi/+G/khZUwKUilRul5SwBBDWpj66trhpxlpasaqXs9eYdttOYwG3BxSoiC+y9+5/Locq5W1W9HWlSQ1TFAEnMafr3VMMauP/zWi2MXf66j+W9Jmqh98L5aED2JiRdKqbGODFnAsScfCNn8OMHQmQqHW0xiaLuxM3SHinU2fO+TDEM+Ad4cumxKo5Qxky4e78yvSBFiL4Ee0OBJXRi+sTO9KG0dDEpf+tfwBaYEef1/LsR2YEY+557UWSrQewpwZzOOfW523Fa78etF4T2uZhSgs5G0ZlBGgNVjHbraVSrIrbuQFZBbNBbj7ii5mAEOqABRef10PZKgRUV51AuiAEheXlNqf5QFu1vEh2aKEnucJ4a/9wCwwF6CFak2rKj4LXPjAbBbn9EPJ/bGY9g4BLuVspzjDOnNiDOp4sq7EnaFMolOCMi9lvy/HlpuTHQmM117pkfR/QbFnI/KM1fFHW9VU14aD5UE3AsfCVjxVzZqX8uzCLFIT8wv7BAIg+vKrYMD2cy30X3I0LcXFo/bPA5X0zsp+6kzrzZF+tY/wHrC6DkOLKX33cYogKEnnJG6YELrKfxz/Ecr2OeH8gchlGE46XKSBvPJ103jq7Uco90tOqu6yUCbQGcc9i7O3srSrAl4lZloDg9Q+6LX5zEilAThaeYI2Y+foKszJ6YLx7aasqaR19j6nzSCnLGG/Dd5lJGpZsgPJQQC4KT1nTELutbIww9vQDNqkvRt1jJfDMHVTcbIqLoAtBxkdKSFbBqUsbedYJRMXNprwS95yGRl3UdXQ+xkTyOAkgaH8gHSgfM1WjF9TVFj5JwkMHA4T4e/0dU5XsqgKERSli+aXTRFhHKvIxoo2RFhkgiLQXYKcXBvPR7ii0Pkn7Oc+4CY84ObVMluK5kPwxoClxnABYeGiRaVZxnFnPDHGa3yYCXIbyR5su/wZ3cw3a93trHyLVK+HPaEscFm8v/NoVFbF7pDBwP0FaKjsZcSyYrPzGp5NrLHYKoNcN7EZaKgTY0E6ZXQQzXLWGC4EVtMuu38VtaYI5lsP6l3vdL4WcM+IH5GZlYST8bwizp2Bz8qozK2XS6qvjZ2qqiKX0wduW3TdgxiXz7WFBx9ioGvyDsC2KS4t4zHtxuaG/R9C1DfOo7DSEaAzkvYU3fSJ7y2gj78b44NCLw+Ksz5IRifz4FsN9f8q10qoEQ2MazDWDaPYhMt6HuwylqRKL5CsX+B9QCbewNQb+BjoJUCSgShqpgQIAsyf8w+oEm/UaWSIbuuokIiCTAR4FdnKp67/AVFJKHOFLl0AB3S3vHmKk82qKl4iI8cnSJrUGLXMzrwcH0baPmR701S1LFENl5V6KfdSrABD5WLBO6yRPN+Hnt4IxiYqEqNoe5j25dcRnGBgDe37d/PFQ4QNDJuPmNSBVdc9Ijifhn86uPNrbg9q0Pzjw9e9vNHxMre2i9XsK4gMPRf8qyxvV1oXBsZizV45WsETniTVSDG6OLmxH+avzoQaaay2whPLY4ya6xYrYxg2wO89aRzX2qLTWdRPj5OITQZnbvcKKDv8Pc0MLkQmeQhHsR8rtL7BZnxluA7Cr4x/M+RW59+BRljtAq21e9YJLTmKm1E6nXmPqc9yNmPBBY7cozfSTOb5/VGBvSk9StPLGiWToxwSafo5OC9OlfDtmyVQ19Rk3JIQlXOQlxYus8X0bfO2HwnWAgTDTYmq5hjUHsaRwWzffduVmjOxOro1OUSpRZYh7WtQICsInZYDkA/f7/fQOectLjW8GOO7pmci4bj0VPlgsioOdJ+fKxDKL+2dlMd245nN6HsIX35tjSgtze2P2ybRdDQ5gvq1hvcT0JOsFX/9NiXpc4vLX11TdZ5q/GOmyIchmm1xapXZcCEaIfZ3Fgm+cml1QwBh2swqziNwiyB5Q17vBOI7QcOg6ww02tN/Z0r6R6seoV4EncLBR3emVcMnwYhiwY9OTZjLiGrh1pCrtq6g21xtVq0weVgYj7QPv7voud3e4ZYndBolcedImtq4hKawOzBH8Yc/CLX9VUhPy8x4dxEBG0bJKpqv4ZacSvE3HQJhwyvBqGDcggGraC8ZyN+i+muXid43om7VMNRs21zJxm4/9lNo3BxDcdxM+8UVq4q34jKaLhgCFcqC2OmMJEF9IzTjRfOwfrDyVyZiZSKuKH44nsL0d5SpAeiLbRIHTnOxSnm2SEY+hLMMagEevzHCByqvSOaOQbGV00kupx1PEl12RoT6npUk0RB9L0Q3BMfQcY6QNRVjkSpDsSuPfh6a9/+e2vjwNv6ElkPwIfDoKRY24AgH61msKZJnZ3Go+hJE4KgyQIbk5d1p8jldDaTzA3risnRnTgFjHqb8wGuJNve4a6SoNCY5wbaGvxpqh0g587kMfHCnbB3aqlWSM3bGx8WoQaStdSfMbl2unbUymq6n1a1jRdkCz20TaXXinltPGPY9hYwzoERGcWCtUfyKRG/4SGMC3FtPnQD1kxNPMwLDMcEg1U0a3cmFliUJXWWG8RMWEyC03wu+7m7eABCUggJuZB2RKmOq6kF06pb9F4AcobkNJ+5KkpKS1cjGHuk3U2ONqOcoSXpTQaURuky7KS62Klq1Zwg+TpdteQ3WJQEADqlz4CPPO1m3cSwLmVBforb/UajmBZb65YHFCCNb93QRyw4WYPaxOI7u2uB/9tnZoD0w2oI4w3276c/j/4zjkvhvc6ZgQxvJlXbKCIQ7GxfRcTuUHiUapfklivbwV+tR9LFGdwkwzBss1gCf4T+JOGiCyyRTKM9BknXTOxSwwqdivvcPdYooN6bFCnKlidBwD/En6/Lwh1bq6L6l5L6e4e3mfeeEw5Np2Gfxtm9/dz+cS3w/y4MB2Xrsx8ZZSyQMfX52hFv1jultYNJdgIDSP+2dC9fsLS8wQuOreiqLO0NwdRfLGRuiUEPlkirRFCbhDqgnCiXssUT8AtYBfXwJClvwDOSDw74tXN+9G9z3VX2D9gx5vT0Tve4EZLGSbu5VRfmIjNhF43A7FXejW7R55c3dBZ0Zy5Ka9A+oT7Ag/E/V2s4KBNYtewy/tvp0/mO0W8TJeS2vWS1+B40xivJF74E7wpoNFRJBcbxeiJGmezedT3bkHJqIFb/eCXXT3nkrPWXF4Lb+uNeDDXClzn2KQKu5HYhNoJD5z5CvasY5SgDf9uGZLeJANbWrvVi4LAwjp0mZ1TSnKuRVEsIhLF1PYwdTFdhggnVyB/isqwUbX+euOxWddwKnbBlR42CBT4ZT88sDIA1ieMOFB1u/z6dyaKuyhWvMfX5ZK+mqNEQa/p0kT9Q2OyJJB1DL+Gsbome890i1lr5rJdCjcPOFFWY44m/q+onsNSq5zbLV8Pqo6t+3XNNeU5+0+rDT8bIq1ZAYnYvhWHXMuML2/55epeLOg5lqcjtA7tHPUZcFY+yzJCS+ge59M8Lj7p8cHsO11k7T5n3T3/IExrPIMu6moLZIPEAnyL+dwd24AUsKvBDILqAS3tq2kaxlLCzP2EeTozng++LwOLifTzV+PXt71M85/Lm7kP1jPUOdjGdNtj2mNzLjgjoVk+U/AwNtDAbRVvQpCZgopV1GIqwivqjNa8UZygb3uc5xnPB0NiEgtPFKW3aP9ubvoboewOGa2quSgjFddC33swnVh+aTYCFResJkoBdKtdFmKXOlKZ77aoNgXZcwwImYsO0ZD2U5hIkgYqMs3jmR/53ZUbcnHN/xAfaLEULXpdEI8AcPAGtKQ95boBmwzvwwjRqU7Ejn0kGbOpvPHKRtLbqrWBA+qqD2fJJGRkCiDIFTwuCh387ZF9QaFY14461ilGeaj8ZLvVaLXfKWlCphwJnHpgRhdgnPANwftcS3cybEhod6gK7wcluv8YGYOxjVvHBAax81EXRAd4HRw1mVcxHqfVRl4JEIgTP6Dens3L1X9NGCnODSeUdpb6rfUNCfNMw90IE4Yni4fTGuGbEnAyPpLD6p5SpZwrcKrfZmoi+/Q3hVZcLb38LV7b26+qoR3KIrrxZjgRDvIiOWfeOkJZRV43JTtzkjUbiCJ3wslcfAkNWNJ6vOe1s0xZmZaCyVqMxVO/0JmHPSNny1qpa/YJN2ccURj8H54aptqwweY08wWQ+vaQ55G0v1e5x1SXo39oyVROLj75XIqbKX6Ak2OW12uUutVBfrLl5l+1eOC69sdeADDMbTV60hWFxbOwaRjDXzGh64ZrSXTMb5nePniqWsppaYeBxy5A7qmyou4rS4CrdY5QBICXFoibb3hc/clhiRZC3gQIJxh03XqAO48sgzyh8R/Xhv8C3zdIPxODKqbuxnZtKTI3H1tIp5pmHv6Mgt0KvpMrgK8CUcz8el39z6x3KWvh/6IVIdDicacctqQ4k8Zid7DhkQwZNLZQulUs2GcQa/W/+ykCnZLkAIG4C0jgk7L3gdVj2+63FIFdFZRKXNaHyyqJFY0eH3hY8BPUfMqyrX2hFzgh25tv9tfgnciXPq0/M790tU4KSaNx9PXaKQzZlpnOjHdd9fY5r5WXr5Ipisxt6PLVvL4g8NfQHdUo049fpD7DMkccdbK/5i0I4Rr3y9B3B0cmDFO72nOpPg2C9oNJh83Nh+bGQY1+ASvSpv13F6/ci9TAy3t+hbuVzVyDu2wuM+SKRINRkoZ1rOV57i4Z6zjd2HbV1VvWIh9t+pr6V+JknmGv3tcMH8kINOi3kgnvKZAk7ZOPJWXp9E9mgWmvitr80KRDTeZnHGmviJ0ecZUxwa6Tem3YOjarvu+aYysuo3lqRrTmcYSNmH34HDLJlGIF860wK5lme4OhiSrqzU5iNlA+IH0mSepyWhn/7/2qwvDKqtcqwyEeT5zWcMZV45qbgA88Jmv26+ljSCjEEBtdRVYJUoYbu3a/Edc6FXkdnPF6bXN5hxNkp7cNbU1k1maYCSqEjeta3AQ+/f38nAPO/THrdqflI2LxexcEIkFQCLL+S7t7vpRJJklMekwith9mB3Xa7w7DiuaIOcBg/C0Z9DDw4kFWrw6LRVufXtV5hoYpbVp2moB1Rp2CEFOryS/sPULAuHYzXPX72IYbFh1dq/xQzl47NwaOsbYI8QF3VZYLOCGyq/c2+pS758nXAm2PJUnFNIpxfZdlBHcg7s7SaDzBee7NrR5NVdPAlv3opVK97YHvk1M7UA15OtK17Alxgq8uDY363WoEzQKVJH8MNTXv0uoXzRyHNcdadEZ42UqzlYupPrSaRh1cJaREuoFe9MAFozlkiYQSTgEW4PRDZw95t0Bvz4bYyWu0wx6eMZAYFDpHaa2P+V7NyGSgts278qct/9F31FV3AMCFIshPrm1/wpVha6uifPhQiCtwzDLfDy9TaqUKL993zwxwxMj9rQcvjzxcGXbIEvMhA4P56Qov63t5RLW8sDaWHNieLLLj5HHm0EE3fe1kz6M/E+5Hq0NKliIPM6rbU/Y+guWzVv/JqatemW9C2xawYesMgYUD+XwUIqSUwKBBOepN04FQZnHCS/4N17Oi8SemAgUM98DJ9gtubYOGLmlCyTiFp5wbwBLNhnJIMpDBUBjjPZivVwUZf0GjpABYOpZ8ilqmL2947AJXURtxyPOImwwA9C718IAjnYO/YDkBrKlg8ADCGLC6/eu4/neKQA8RnrX0//y+zEI0mUZtZuT36xbwUE0xqEOKzAvoX5wathSNVukrk0LSyMI/CnmTm6AEicgwwCiZ+dz8Y+o49eicSpC71H4jib18ZHtGxwxOwmOSnnAABplGxIbkEU+emsbZ4LJiVZAAEECwkUJBNBO/VCLbSxqA6evJM1m0e6xm+ne8ejU9r69HkqqCEXH7ZSDCjA3nS+2u/BcTg4sri731tDcn74IxcloTMjXi0T6cwp452A/FcjNbCP0r/Gdkq+NqbiMH19BshpJ1oFjFrJsBgSIePz6clgsQ02Rl8rGz0PUEFMCENZSBb15/5OPt0korq2X0GOReNJyx4migWywF+Hd2uZJpg6FO+ioL5Qw1mNMeu2QReZS/pQymTYPy/pgANVL0SzJJE69gtZo+AcT5aRj11HOrkyjIU7TvLVG3fRXBRn1ofKRVwQxCl6fNXswtvD/tG5P0GgZ+85wlHMCY8DmTz2TZcdffp3RA7n04dIQCN/xkr0aWmh78/wG36FQYAElLjLH78fArACoOj/NKTaRSYDbWtilILbJsvEAdKAKcf+08po3pZ1ANzRwT8rSS0HjNY4aaHZoMg9sDuxMSmKQxpgN37LtSBs27wqGz2CY6alLt1+JSqs2SjcHsRgLnt2vhKohkYUd1s9qMoCaQaZb9clMsGpYqyVJ6zMEBj1peftWYhqfZzjQ13u86TichcePKox1EXN9AN9ZKjqnVsYsVrf/7TaFhEXRpkZEcLb9M2zBjPrJCwsE9JzYtxoILOiwxIkZtvQlzdL8KbBnxqg0HYuF15spD+dF4ZggEzNNfErAeGZZrs3bx/5Hr56ChwUUsVkSUGfzPyZh96An3a/cjt6RN/b0xIHR0vmvvTZsSWdsmWI1H3Y2lqShrBDo1VE/fdme4QJnMJzvIQu3irXfAMnQeUpFXtAKyXk6mJr5N7qefeDS3NgMatdMu+kKeAlaUAZ2Z61+jj5bHVG1v0qwu/BjMFNCt5/2AWAFc4Gu7Ounw8Jmxxn1vYFSpaUjaGR+ve7/4TbMabEkDzYmDY3spq+Ijf4XtjTu540BRqfdCXqe9iCXbtQJ1JPnqLsYzUVqcWJwKs3n8/l7iv32i/KZuVFqznB8FB4+5eG0pZ9endN0TuhYdUOqnaziyiWZtv9QLeAbabPMhdmmdQMghnoGUGkxuBe80LsNLsqHwINXs+2njUHEafyKN3AkW4OYzxFCGoWxjmPy9Mec1jPGtLqUPv7o1BiQCLOXPqkcWm3R3SG/jbDSmx0Ov83sdYOGIXmDjlSSMfMwrsNKkbvpmVzEjgsPv1xkv8vj/oALgDcxpYQhqjsjbg3gknyt8cyKfD2PYqbxcpS0qnL7liXe9120vET92BSXAGb+f+EOQKj3A4M3hE1loa0Nqo71aDKA5dgkDXefmUXjjyskYqZzRjbpztv0mvdqW4ehHLSynU9dOO8Vhp6g8349tiz38Seb+oAONV1uge7r0I2Ew/PFARvSkvCATj1GODQK2KjyS3ispDEbe5pH1fM2xgcztWznTncSibkhNbxZX7RtKjRTYI59d1DNL9/w4oe6fLBZJfvALDf147Uz4JqeA6uI0Ou2nRQf9FTb+V8kEBD5zOaA2tdvfhjiyGwYNoKo/wbxTZdneio1RUTzEJIiW2mqgcv/h96oUlBosl6co56P2fDgv/SY+8xexHLLq1K1zdCWZO9kOCuy08whA9XV16jeSI9EeNH7YQ/5NCzJeRmpVzXrny8hDK9k1vOz3PAc8/CBVNRRDzQpyccypieRuHkTPzLfPP37oftopvZWWoU+Lay4hqd6pDJ9oIQoz/ZIHzn5Kti2WJqVVSGqkvb+xEh7Hm3M5eRhWhA1esqOg+j9t47TGp7n+VQ6xHB8JfehqZMff4viDaEid2Ek64vqg687EcBC7RpUDFKYeuzLPCVmHUBxi9KSQhiijT4i2H6/VDVhifppuhM+6O4+6uFaeDVql3ONEX9ICPWPHHIKAjZdluIl7Ixkt9sE4HabbzmcvgcOtE58M9616jihh175PhtkPqmTmE5NeZqhm1n+l11R64sc8/IcbHTkiKWWEvKk74SpJ+OCU2EFzcoNTesipA39zycXTK2GzM2w1Tj+RMl6fSXZjw0ruZokwAQ7KXPHswu5/CMfp0fxvg8Yat6LyEHxPLwQeq4T2PKcvxuZReAgWaVBdLX9LZ4gZHQ6EqGAN9Zo5xpZ9SP++wpspKoGXiW2CQkFiAJ98jTZVzQUdpfQMcaJtwvFl3CUOQznVPz0NLCX5XhlHsxrc2rPb9v/CM0oG3EbPTagP6sRpNB1F1BGjEbll9ryxBQwREa3a3yjaBte99w8yROZKpVj5GKX4pjprMFPtsGCRBEnYgiXISWOKx6EPg21Jjfs+AooIpFxS4tfiJR2swzFAFl1UE1T4aJlI+J9CKCabA2AJenA9oEmXwTPt84anEvEGO+IjW1pz825Jh3oJKoqIUMl1kPg9288Dmk26KowkGIN4ScKvpA+tmy6zr78kJAotXVgULwsOzqqgiYfocwLNBtIE51qpTnYxGdt/qPcW/sM9wIs22oboM2QGzNMaNG7AW7fd2BR4LYUPUhM837bJ+DiWnL5792+LTZ+XHeXZ62kzDkbp9fLCMrsY7eRA4RS3MoaVljZN0F6SZ5eAk80QGqPMHSRMNV/s/lT6suvEaaHzqPZ0tVLXoBBWgxVIku2eGlyI/dNmeYZKAoKOOCu7jQIoSZX+x/vcMGH2uEwWBpGrYQAvOL2TZriUiphe8l1VZblO97nLz67PBbv4EJIVnhzHz0dwFA4+tD6W65dM/ceD6P+gSahHKjSBroMDZblje6xCfw1iDvMbA1dtskOvpXVghUy0UF8ROByM8zOpY9ZNCGoTlY6HPTOCnX9dkPe0RJc/BjwqWcFunCsB4rjfI+QB8ptQUtz1jk+49WOcU4oazmcvNWfTD4P8Qweiju+4e2erN1CsRP5MhBDTpS/Z6NZCmhOn5qZ2sUC69Bd8oWpazlzQPY3xLT2yaVtZkThZykXgqPF+7Tuh0nWPXAv2aLpyQh3mclwPQAvyjsFk/JeHdlplvYGC3xptc3aczOrZ3mgv2hvblvpU6kddaFtQxdCpE7CxuUGYxYkb5yjHE8FY2f21kZzVhJeieGmE86QqQdxapI/pVkww7FffdhAUeB8tIV+ks7E01b8aWTwnPnDNLh18yXESTNp/V63obKiHmb8fMEX3/x5XVSZyFAOWQnz/HaZQdmpVwPkZycnQkAv1kK248ejP9R1rM96ShlXphfN7IRsuk1tUTaoFJeLFcE/c+5T8lQy4+UM2wPoBmGoUdLJmihum0b1UHGAqCS3f4Xear/FoNBwhCbnLhcvkht71DMVdwZOav8/n1sLKIp37RfjZ7W6c6YfPemZZG1YnKcQ7dbIJNwd2/cWbesISMI6wfKIMwPcwXGrYrmShfGhWK8/+/VsT5tHw78KelLI5omuBy9udoZtKn++t//xK/8qvt9VzP4V0liF2WDBKYUzdpxR5e8w36NsITlrVo8Dq/7+SEAWUEvnd+nn8vKJZLQly/zN7/46BGB5QESpakMT9CousMcnrGj6OWa1iRuNaJktZ+fbnq7aAnF22mOTeYP0mGwaVCG5elqMheobSZJX+Y9rpYW8OCUBipqpMGl7Hx1ZU96gJ3E2aMV5za6kRnJHZlvoVs1HWsnvrmNJgrO6otoUjvA3mFZCbSZS6SBScG3CgasJ6gcKQ1K3SRA9haW7bNDKy/5sr9VTEEsHeKtEOPP2d1JMUbtr+Nkjz6xBjcM/6sxc5hj2JK+t1oRqNuenflyyDKSksLrE/8cDWLDSM16n7xrVnVNEFpZsrPIGt7BfW/eHQ9k9TXVe1PydtmVkiuXWnL5sdVhAu8ty2nXHfIjF+FUtVrFZaTaOgnkS9z34PpOz5L+/WXH14DewQjR8zJAGrEVwhvX60ToGOdn5GyQR3yKEXjfWGDlG7V9pLUn8ezFXJ4m1AWEC/lVJvxnEUibk4w8e44piA+RinCTfTQ+m69hYQqmvoPDJNjmGrAF5dpaDIUrPqbHk2fED3y7BQSvszZo/+ofhH4tOPWu9d8alUxFSpLg+vOx6nrEtr72t24lq2PEZv4H+Igni0d0Z2nFKrQdvjIetaDH7zS8w+N+BhcizXqEHAVLXNhR1dLGGrd3u3PF8X2zj8qiGqWcdAFTTEpwi/bZuBnEOgC+4BritpBTgM2cSi/1hB2vU82pcUpj5fG2SgN1/DYCO2CQfSkSbVRMi36An/+mGFx3m5suCslnflhe52RBM9rbTw2C7cCM86vDD3EggCjszjbzeKB5GmohXhmLIOw8ks9SPK0dzxfco5o3jqHoUhw0AbZ2yYS7Ibn20qkAuXkdTRLJb0qhp/NzAFfdoJqzHkGPfEcdPK6rsV+zSxYmU375TPR5qqbqFqN9psxYudh1r1YmkbbEQu+0IkHaHlV32vflHkOFROc4+G2RmQASU8w4dTEb4YbJz6TjUx1IZmva1eNl6XaoQ7VmbwI742yUBluFQyVwoFNYrv9q9ihpmpHfO/+Pfl9LHk+sn8z07iTEht8xGGhk5V2sOz9vznyMME4KZ7pz7UFpDvECMm0DZ8mlmVwK4YZw4nx++VdzG/6j3MQRI8W6/B6mIT4K0SnxdJAsm5hvZQK5q/BHF53p9HB7Zd8y0/f2KCQdVZ63s9cD6nzixZ+udjNqiy8fMwWECFb9KANdOjPO/6NpYijEIhCOZnHQuvERNYrVc9vMfeDYVjdXezVkPvMXHKPrI3SslqSolDkciker/kdps4I2OyDN7E0K+8OKzVOTOTxYH1MuAuqzBkGLNxYoPXzyCKlO7ARXeblzVNNbZyIcYsTFgkEFaA0HyweXXWwfMIA5180M7fuS6sb5arCYriCLNUNeOuQa4ViE/8J0G68K+VJ1zWtzaRjXUZV3H0qH53pvwPdx2tnVEkY/AGN1v8xK8nyNzwEd4LeWKgvFW0QVF+gJ77gp3kOWebd0xE1qeLJm6vRH16HpDcKZdURfwDHIFyf1gtn4xz4EDUaiDD06/U4szGUSc3peyUZAlGDqnr9BztQ1qYW18G54v5czqA1tIQHSr5rmhX5Nq/Ck+T/+Yoo15e6HltUmAaaWJbtJXUxoAHWTAIJDDVOUdGAG6KeXZw/tS/od4A1EXiY8ABWtztgEoUX8pUSoup2hmTNvo5aekFHMk8LxpoBu3p231obgMqdZY7FDfHiEvO+o4FBUa+dzGOJOJs/rampAK8mxMUj2rBbhqnDUeIAwHLwnseY4CvtrkYtdeeXeqlSW1uzNFmlUwwie/S4mnd7F2NHa+xqK/ehU/qpAutn1nb4LEKmDjp5iS630RNhrRpc/Kk4bEq3F3amZbTQjp9JvQFPIbAG5p9+AiiBHWq33hMiTNWBMuG+gKZb7eAAFP1SZY7ExWm/in2HGgrUXjQAAAAAAB9yLT+cVMMlX7AYoYkV2anuB8nOwI7cxer+fXBbngalFQoepo8y9vbir1NaSK9/ayfMcDHxQ9289eDMlMBOn6jnwD21Wo4Ay3wzoRWAmhz9RLJAAloZ2o7YKyWcRUlwezTZ9romm9YfCIxvFvB5DbRQom5xbg0u/5yzLVw9vfFZsI7Xf2cETGAZh6cIIEGEv0Ca9fV0K5J4ZtQgEJt6BAW7dYeddvNxcy6/IsmtNLyCyVTMILj/kaPzLh2r6Tozte9VHPk80rJGFBtuglWphIAXoJA1WsqNDSotIZnhvXbEsjFs40zOFZzyrErDxpdg2sEsSp6AqyhfcR4PElgnCVMYyMHeRWTclBxXO6o1r+GXmq2Fx8WzOX0ynk0xOVPT4PZtDgh0XoHE4+evitHDpciR1Qsd5C6yIf7KvaThX14BExKGIiO1D14+AUlbhPxv6v6gxCL4qC2PaQvJ2Ur3GqFwyP/pSIAAAAAAAAAAAAAAAAAABRI+MGisX1V2eH3V0P1nkgNNkKxW8/6/QR5V5Sh4U+0/QAVRuYjPze/st/3YRCeSoYffV/m36PfyeF/JToYrpa65V5iXKQqhNwFC2u1Fn7waMZ4qXhwmTx86vx1JqqprrWvoC2awNt7YXYbRhLC2Vi4B8sDaUMtfpmfeS6Our6cuSXGprHDTC2F5Bnnq7K9BOsivd6kep96sgi2QqXWcYKNCGQmTi6Act2DxhqxL41b+SGuobjF5p4B0FaUHfsL2yH+OShqjsEvrn6dvaOKlLTnKtre3qddlg50ZY2KG2xI6m1S53gs2d9CMukMjJsjZbSmDUi21SauTPz3TPZhHdhSGXjZCRPClOOZSohryyU3gKr4ELTnSYiHMqLB0tR0huwXItOjeYcwH3IRCtjR+kUYI4WaPZF3z4pslLd5xzcAS+J8tKQrGUh/hEo6JuOVE7iKVX99tuV5Rnoyzupocwv5W0UJPL9N4NkhooFVfUICofZ7htfWXC94F/6vdsnzzMmpaqtM7LU7b70Sryxneqcq4WSosPiMG/eGmYx204Gtnb7yg9HgzzRV93Yj/oY7a7qT/rKOXNJSlVS91f3W++J4EvoJ4g136jsNs5DPUpQZy7zfXoU24FewYCjgIlgfuufH6vPvyqn1HEYRBnyr1biCDqxDqsdj6EZkNfJnJQeAI5BE2uvl+BGbJyrM0N0VVfzDdP1yt3Rm91SUcB+0/KFJx/ZFXkwvfMGHkaDME+w5vhhsWyoI2jlvASvURDGIo8wml0qNWe44VG18oYhxbbAfUAKkSgM2GUxVMWB/QwR33muLTGsm0GiKos2DLwEqIuQW+8ax8acEtqOME818AvJTVXDwCdtSkqYejXxFLuYDUN58teswvWUxTVep0khSdr7o3SmVKuMjw+d3xfDb2pEvOus3DG99gwetacU+l9KDCqd+q0+jiGkV5cmWCsy4H7aerXs/3BuC6VXiruIYNjSXd+FWL/xSKYUEJYZuFdUyAhe4dPDNhfQoWMWp+C2zmtGo9TSxsPjclvnN9cnbI/YeOcefGNdbXxFHXUaOTrzIqtYxMc8e5wmknMwcY8HGO3i+XFAfSfGbxwT6nYaXlzM7/C6fq4VKKZhn5y1I36s4+AR6loau3gs9xKSoHuZ9HDuEHEgfS6UgRmYe7PgsJyHMvti4ahXLTZupYXkcqIcG4nbNxUU44pnQ16jUQi9aH1J/QkBjkNAvqXtepPIvyCW8wCHfZkAkYBQLEoVhp6up5c7UzSzcM++ROlm4Gu05P8nzKXPezKTexKKsg/JxjcU+ufBVAaJmDwab7RPfNbgZvEiCjn8icv3w0pGxkQHAd5FCWbtzJiPcrFiVJ6UoZRlhPL0AHgIMZWpCwiPz1lcWQ6zHkYWP1kgCKcj6DiRKZwU461NBsPk3SUB/EYWuf59u/E6Qf1opS8y1K0b7f3K+L9WPQuyUlQk7P+ykoj85fjtRpDRnEboTnf+eOoQSZs64BH7H36rIiy7Id42skfP5SpxBFx5rX5SHhgnqQamt/mdUyYDSsdzacalBH2c7u06OsmF41xjgeMLl+T8t5gXpAiXCQOZ6RgtfsdbcB3Ere/qPcM0oIOWwph8lcCoHJJiG09lC1AyPr09xTnGQBlOhb1gVuq8Oh39+KYXLAHAk6p3/VRCcBGNeXvwmdGZtehLIjBkFblIUBQNEsOlPIiE3Qs9bZpcbT7/qW1ktC+k2VWCbUOiqiRcL3YwKFIlZXBd8zjbG4HlAwKuAyegI03SWhsKCStv6jtonsM2rPEiMRgZ5im4qxlOH9W79JOuEL+qN0TIGsz24E0LsAglNF3xkdRGbhn/W6tHLG39hw42bh+7Q3rQkgfmVShpID+AYHHiyS+/Gg+WaS881Q2BElRHY73jM4UCx3XmRzBGfeZkducK7uh8Rv0vucThdVrrEwDl6MQbpyb2Mp2z+J15JVHTBfFKaGJZaedr+Hlyb3HdRLwYZ4In/gyid5jAs8vm3wjwFVYS/I4hRbuEeV3j7TakOSNfgrA+37C7L0x/sBXJtklS2u8EBWi34Le/SIv3DZPLvoy/C32jLSQLti0/lTUMkqV6xIbmus+UM7xyNPnmApyCfEuXF6s5OjqwHJMj2yVa2EXt4DDbP+1StoKOZ13sDlpVgw9RSAcf4ZBF/c3oYQS8B32hTG5QlQbuGZ1fGews9LfC7luXz0v+gFM+GPlKRaLaKjdq3tjeKs3ZI94R2XC9nvQFAAPX2oE70iZQfno4tXhznirsws5VjxBrUMO/I/MkcopkIVPY/B8XDec0ARffw80KiV9rG53lQ0MUcKF0lOvFy00NZqGFuJvI2Klv375t4ftidzs6Zt7hxSFaZNyKf4BF1BpFIx4NEcb7nQTfTQlH/vBvOXJxJFIrQD1aXGlnNsiMShREvWqxX3htCPTdwJdK2XG12v5hsPcJa97Rvq0685YPfGBJJYTDFDwaUzXIa1YjaRlI8Uq2TTheEWwxFyuo8Zvejmxg9N9L+idd5LB0Cqpy2OuJjA6OtZvpnE06WzMXwtWXI2r5+h2JCnyKq2K35pJB+phmfLqkizAmxKW2PbueyWDSFith8KKsS3BFkgpgcqpclxUO1pd0S3ZqD8PiqVptaHxDO1VE+Ul3YggOtZt33EgS3aTQ0Uxa7TxtqklhsAKxbl/IH73C7ZsRLPtUP4YWCrMibvY+y/tQ7bfxoToJxP2lGAmCbJgmZqF7L1WEfXIzHpeBraUAGmnm73yDuFDB9uAwCq3pDzJipR4QHwZxIqVt05c/2JlrOmTjTcFlB9E5kFJoSOSIRrdqn2VMZxPf88FBBrX83wl6lRGr/6xwbDD4+a8HJwNrvOLAgOG+inrVcSTwjTM6pn6U725jWhNm6RngK/YBH3QzCFau+itAw6D6qSiATFEoQUDv2genBBfoE3Hn1Tr80o7sxFCS50AtkcXQNes5h6VuAv3tNdUpuul6tbswYW0ok+VWQQqhFmOfitlCcgcjpiixnhM8V0+tVUvmc935o3RP4WFe3iNsVn80vHBDWP3671w+hXEQBZ7TTeVQnfWzS/5pJ8vgtf5zQc+NRuotb4E7bVHE16JdXlb0CenXDE0SPdUnUX6HvueX6fqbz78q5L2UV/ELVizSb9exp12v9ltCkiqDKsWVEFlFNvK8LwssjUYvEeCzG1amxXqQrEwwspZzy0iAL7Dmh3Tjh8lmeFSwYSsShgCT9ICask2mbmIHv5Yb569RcJ+K40BYyWHhMiKvIbVbnlvxZtWa6mXO6b9K4be0T9fI9rJcqPnfvbO62Z2PZh61wa4KnCy7dutob+gle7bCsf1RBW5vdd+Yg9F/D/vNhj6LcnnhYXlErtyaH2jzLImrurFMQxiC+TxRe78AbRopBtrIjwgJaw+rJfBa8DMulm/Ue3JDf/9MSNGTFH4H4c/1NtlK/3KWQdpb0+2ffDFdH2MzZRqrR6AxAShuxoi97onjqmeBHDHCNC579BzYzG4Vk5jY7f3b1ISmjjKQW0tKxo+ChxbRjcxVubAo2/XvoNsrrI8Fk2BTxB3C0NgdhOiOUjB2b91dEWAb9l2zmNiyRFvYLRW09b43/Pyw8nXCfAmod6RTTx4XAL8A+2oWW+Xls9t9F72z2aP3hPkxpMBlm3ZcAAD7mEDrkK+v1qBzFwgPW/kuEvBVaOaxgliUuLHnaB7kejJlYj1dBCRqXwD5of9HfuMucUk3pkCr8s0ss1TLOXwFmI3iVdeuBEtNpTqID9NAXlsjjMKm4PIhD8G2lA+gKyWWpRsMgBB4RQxEbW+0o4vkSeNJl1/hJ+otEpdYAEHxoXtUD+gxfOrEBjv2vFnSQeBcpA6sPQ8FiILnqTTlZH5hozzTuf/QT/9+cO+vG6jra9T9d3g/x2l/0B4pxheAKi2G2Up+BM7lHa7cC34YMgUrKAALdiWxO3G+sGYZdpzEg3HcquVWCAgOHgAEsAJzsMPCxhfho9lyYmyJDH0rJREGPz070X8Y81VXTsF9/AAAAAAAAAAAAA"&gt;&lt;/a&gt;&lt;/div&gt; &lt;/section&gt; &lt;hr class="tweet-embed-hr"&gt; &lt;footer class="tweet-embed-footer"&gt; &lt;a href="" aria-label="4 likes" class="tweet-embed-meta"&gt;❤️ 4&lt;/a&gt; &lt;a href="" aria-label="4 replies" class="tweet-embed-meta"&gt;💬 4&lt;/a&gt; &lt;a href="" aria-label="0 retweets" class="tweet-embed-meta"&gt;🔁 0&lt;/a&gt; &lt;a href=""&gt;&lt;time datetime="" &gt;14:56 - Tue 06 December 2022&lt;/time&gt;&lt;/a&gt; &lt;/footer&gt;&lt;/blockquote&gt; &lt;p&gt;This is a problem I've faced too, so I thought it might be useful to outline some of the places to look for boundary data:&lt;/p&gt; &lt;h2 id="ons-open-geography-portal"&gt;&lt;a href="https://geoportal.statistics.gov.uk/"&gt;ONS Open Geography Portal&lt;/a&gt;&lt;/h2&gt; &lt;p&gt;This site - which leaves a lot to be desired in terms of usability &amp;amp; reliability&lt;sup id="fnref:1"&gt;&lt;a class="footnote-ref" href="#fn:1"&gt;1&lt;/a&gt;&lt;/sup&gt; - is the main place the ONS shares their boundary data. What it lacks in user experience it makes up for in breadth and depth - there are a range of boundaries across lots of different types of area, and with a variety of different variations to download.&lt;/p&gt; &lt;p&gt;For a given boundary, there are usually four or five different versions with different resolution. The lower resolution files are generally smaller, but will produce simplified versions of the boundaries rather than the exact boundaries. The more generalised versions are better for things like chloropleth maps where you're comparing lots of different areas, and don't need the boundaries to be exactly right. The exact versions are better for when you show just one area in a lot of detail.&lt;/p&gt; &lt;p&gt;The different versions included are:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Full resolution - extent of the realm&lt;/li&gt; &lt;li&gt;Full resolution - clipped to the coastline&lt;/li&gt; &lt;li&gt;Generalised (20m)&lt;/li&gt; &lt;li&gt;Super generalised (200m)&lt;/li&gt; &lt;li&gt;Ultra generalised (500m)&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;The "extent of the realm" files include some offshore islands and use the mean low water mark, all the other files use the mean high water mark as a boundary.&lt;/p&gt; &lt;p&gt;Most boundaries can be downloaded in GeoJSON format - you can also get KML, Shapefile or other formats.&lt;/p&gt; &lt;p&gt;One helpful thing the geoportal provides is historic versions - so if you've used local authority data from 2015 you can get a file that includes local authorities that were active at that time.&lt;/p&gt; &lt;p&gt;&lt;strong&gt;NOTE:&lt;/strong&gt; Stuart Lowe at Open Innovations has found an issue with some of the GeoJSON produced by the Geoportal, and has &lt;a href="https://open-innovations.github.io/ONS-GeoJSON-Fixer/"&gt;created a tool to fix the files&lt;/a&gt;. &lt;a href="https://open-innovations.org/blog/2024-08-22-fixing-ons-geoportal-geojson"&gt;There's alsoa blog post with more information about the issue&lt;/a&gt;.&lt;/p&gt; &lt;h2 id="ordance-survey-boundary-line"&gt;&lt;a href="https://www.ordnancesurvey.co.uk/business-government/products/boundaryline"&gt;Ordance Survey Boundary-Line&lt;/a&gt;&lt;/h2&gt; &lt;figure&gt; &lt;p&gt;&lt;img alt="A snippet of boundary-line from Newcaster" src="/images/boundaries/boundaryline-newcastle.jpg"&gt;&lt;/p&gt; &lt;figcaption&gt;Image &amp;copy; Ordnance Survey&lt;/figcaption&gt; &lt;/figure&gt; &lt;p&gt;Part of the OS's Open Data products, the Boundary-Line product contains a range of administrative boundaries across Great Britain. The boundaries it includes are:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Civil parishes&lt;/li&gt; &lt;li&gt;Wards&lt;/li&gt; &lt;li&gt;Communities&lt;/li&gt; &lt;li&gt;Districts&lt;/li&gt; &lt;li&gt;Counties&lt;/li&gt; &lt;li&gt;Metropolitan districts&lt;/li&gt; &lt;li&gt;Unitary authorities&lt;/li&gt; &lt;li&gt;Parliamentary constituencies&lt;/li&gt; &lt;li&gt;Electoral divisions&lt;/li&gt; &lt;li&gt;London boroughs&lt;/li&gt; &lt;li&gt;Greater London authorities&lt;/li&gt; &lt;li&gt;Greater London authority assembly constituencies&lt;/li&gt; &lt;li&gt;European electoral regions&lt;/li&gt; &lt;li&gt;Welsh assembly constituencies&lt;/li&gt; &lt;li&gt;Welsh assembly electoral regions&lt;/li&gt; &lt;li&gt;Scottish parliamentary constituencies&lt;/li&gt; &lt;li&gt;Scottish parliamentary electoral regions&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Note that it doesn't include ONS-specific statistic geography like Lower Super Output Areas and Middle Super Output Areas. There's lots of other boundary types that aren't included either - like health geography.&lt;/p&gt; &lt;p&gt;A direct link to &lt;a href="https://osdatahub.os.uk/downloads/open/BoundaryLine"&gt;download Boundary-Line can be found on the OS Data Hub&lt;/a&gt;. You can get it in a few different formats - GeoPackage is probably the most versatile. It's quite a big file - over 600Mb - and you'll probably need some specialist GIS software (like &lt;a href="https://qgis.org/"&gt;QGIS&lt;/a&gt;) to use it.&lt;/p&gt; &lt;p&gt;The data is available under the Open Government Licence, so you can use it for any purpose.&lt;/p&gt; &lt;h2 id="geolookups-by-david-kane"&gt;&lt;a href="https://github.com/drkane/geo-lookups/tree/master/boundaries"&gt;Geolookups by David Kane&lt;/a&gt;&lt;/h2&gt; &lt;figure&gt; &lt;p&gt;&lt;img alt="A map of the Lower Super Output Areas in Cambridgeshire and Peterborough" src="/images/boundaries/geolookup.png"&gt;&lt;/p&gt; &lt;/figure&gt; &lt;p&gt;This is a repository of boundaries that I've put together, using data from the ONS geoportal.&lt;/p&gt; &lt;p&gt;The aim of this is to produce 3 GeoJSON files for each Local Authority, Combined Authority or Region. Each file shows the boundaries of either Lower Super Output Areas (LSOA), Middle Super Output Areas (MSOA) or Wards within that area. They're meant to help people who want to map data in a small area. They use a fairly high resolution so you can pick out detail in the areas.&lt;/p&gt; &lt;p&gt;If you want to combile GeoJSON files together into one file, you can use my &lt;a href="https://findthatpostcode.uk/tools/merge-geojson"&gt;GeoJSON Merge Tool&lt;/a&gt;.&lt;/p&gt; &lt;h2 id="other-resources"&gt;Other Resources&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;&lt;a href="https://bothness.github.io/geo-draw/"&gt;&lt;strong&gt;Draw your own geography&lt;/strong&gt;&lt;/a&gt; - A nice site that allows you to download a set of output areas based on either drawing on the map or choosing an existing area.&lt;/li&gt; &lt;li&gt;&lt;a href="https://automaticknowledge.co.uk/resources/#UKLocal"&gt;&lt;strong&gt;UK local Authorities with insets&lt;/strong&gt;&lt;/a&gt; - a readymade UK map with insets of urban areas. It's worth looking at the rest of the resources on that page - Alasdair Rae's resources are very useful.&lt;/li&gt; &lt;li&gt;&lt;a href="https://automaticknowledge.org/gb/"&gt;&lt;strong&gt;Local authority GIS data&lt;/strong&gt;&lt;/a&gt; - a set of useful GIS data for Local Authorities in Great Britain. Includes boundaries but also includes roads, surface water, buildings, greenspace, etc - everything you need for making a map. More useful resources from Alasdair.&lt;/li&gt; &lt;li&gt;&lt;a href="https://flourish.studio/examples/"&gt;&lt;strong&gt;Flourish&lt;/strong&gt;&lt;/a&gt; - this is a great tool for making lots of different types of visualisations, and their mapping visualisation has lots of UK boundaries already available. You can also upload GeoJSON files from other sources.&lt;/li&gt; &lt;li&gt;&lt;a href="https://mapit.mysociety.org/"&gt;&lt;strong&gt;MapIt&lt;/strong&gt;&lt;/a&gt; by MySociety and &lt;a href="https://findthatpostcode.uk/"&gt;&lt;strong&gt;Find that Postcode&lt;/strong&gt;&lt;/a&gt; by me both allow you to lookup and individual area's GeoJSON boundary and access it through an API.&lt;/li&gt; &lt;li&gt;&lt;a href="https://github.com/open-innovations/geography-bits/tree/master/data"&gt;&lt;strong&gt;Open Innovations&lt;/strong&gt;&lt;/a&gt; maintain a collection of GeoJSON files for different areas. (Thanks to &lt;a href="https://mastodon.social/@slowe@mastodon.me.uk"&gt;Stuart from OI on mastodon.social&lt;/a&gt; for the suggestion).&lt;/li&gt; &lt;/ul&gt; &lt;h2 id="whats-missing"&gt;What's missing?&lt;/h2&gt; &lt;p&gt;I'd love to hear from you if there are other resources you find useful - I'm sure to have missed some! I'm particularly conscious that lots of the resources cover only either England and Wales or Great Britain, and are harder to find for Northern Ireland.&lt;/p&gt; &lt;div class="footnote"&gt; &lt;hr&gt; &lt;ol&gt; &lt;li id="fn:1"&gt; &lt;p&gt;The site wasn't working at all when I wrote this blog post!&amp;#160;&lt;a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text"&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt; &lt;/li&gt; &lt;/ol&gt; &lt;/div&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Tue, 06 Dec 2022 15:08:59 +0000</pubDate><guid isPermaLink="false">tag:dkane.net,2022-12-06:/2022/where-to-find-geographic-boundaries-for-the-uk/</guid><category>Blog</category></item><item><title>What powers charity websites?</title><link>https://dkane.net/2022/what-powers-charity-websites/</link><description>&lt;p&gt;I wanted to understand what technologies charities are using with their websites. What &lt;abbr title="Content Management Systems"&gt;CMSs&lt;/abbr&gt; do they use to create content and layout the sites? What web technologies are they using - to what extent have the embraced the latest tech on their sites? And how well are they complying with some of the regulatory requirements for their sites?&lt;/p&gt; &lt;p&gt;There's lots more detail below, but here's the headlines:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;strong&gt;Nearly half&lt;/strong&gt; of UK charities don't have a website, although larger charities are much more likely to have one&lt;/li&gt; &lt;li&gt;&lt;strong&gt;At least 10%&lt;/strong&gt; of charity websites don't work at a given moment&lt;/li&gt; &lt;li&gt;&lt;strong&gt;At least one third&lt;/strong&gt; of charities don't have their charity number on their website&lt;/li&gt; &lt;li&gt;&lt;strong&gt;WordPress&lt;/strong&gt; powers one third of charities' websites&lt;/li&gt; &lt;li&gt;&lt;strong&gt;jQuery&lt;/strong&gt; is present on 71% of charity websites&lt;/li&gt; &lt;li&gt;&lt;strong&gt;Nearly two thirds&lt;/strong&gt; of charities use Google Analytics&lt;/li&gt; &lt;/ul&gt; &lt;h2 id="which-charities-have-websites"&gt;Which charities have websites?&lt;/h2&gt; &lt;p&gt;I took a random sample of 1,137 charities from the UK, using data from the three UK charity regulators. The sample was based on those charities with a valid website address. I collected 200 random websites for each group of charities in 5 income bands, plus websites for all 137 charities with over £100m.&lt;/p&gt; &lt;p&gt;Only just over half (57%) of UK charities supply a website address to their regulator, although larger organsiations are much more likely to - 84% of charities with more than £100,000 annual income have a valid website address.&lt;/p&gt; &lt;div class="flourish-embed flourish-chart" data-src="visualisation/8655291"&gt;&lt;/div&gt; &lt;p&gt;I excluded those charities that use a facebook page as their website. 750 charities give a facebook.com address - less than 1% of charities with websites, mainly concentrated in the smaller charities.&lt;/p&gt; &lt;h2 id="do-the-websites-actually-work"&gt;Do the websites actually work?&lt;/h2&gt; &lt;p&gt;With my sample of charities, I could check which of these actually return a response. Valid responses were received for 82% of charity sites. Of the remainder, 6% returned a "403 Forbidden" response. This suggests that they are aiming to only respond to valid web browser requests, rather than attempts at web scraping (although some of these responses could be because they have given a site address that requires a login).&lt;/p&gt; &lt;p&gt;The remaining 11% response returned either no response at all, or gave an error of some kind. This was much higher for smaller charities - over 15% of websites for charities under £100,000 did not work at all.&lt;/p&gt; &lt;p&gt;Even for larger charities with income over £10m, a worrying proportion of websites just don't work, although some of this may be due to anti-scraping measures.&lt;/p&gt; &lt;div class="flourish-embed flourish-chart" data-src="visualisation/8655406"&gt;&lt;/div&gt; &lt;h2 id="do-charities-include-their-charity-number-on-their-website"&gt;Do charities include their charity number on their website?&lt;/h2&gt; &lt;p&gt;Charities should &lt;a href="https://www.wrighthassall.co.uk/knowledge-base/guide-to-advertising-charitable-status-on-all-written-materials"&gt;display their charity number&lt;/a&gt; on their website, so that members of the public can see that they are regulated as a charity.&lt;/p&gt; &lt;p&gt;Most charities complied with this, but a significant minority (40% of the working websites checked), don't have their registered charity number on their website. There are some caveats to this - I only checked the HTML of the website homepage, so if a site is entirely powered by javascript then I won't pick up the content of the page as a real user might see it. But I think it's a reasonable estimate.&lt;/p&gt; &lt;p&gt;And although larger charities do slightly better at having their charity number visible on the website, it's still nearly a quarter of the largest charities with incomes over £10m that don't.&lt;/p&gt; &lt;div class="flourish-embed flourish-chart" data-src="visualisation/8655836"&gt;&lt;/div&gt; &lt;h2 id="what-content-management-systems-do-charities-use"&gt;What Content Management Systems do charities use?&lt;/h2&gt; &lt;p&gt;I did some keyword searching of the website homepage to try and detect which content management system (CMS) the website uses. For example, if I found the text &lt;code&gt;wp-content&lt;/code&gt; in the HTML of the homepage, that's a good indication that they are using WordPress.&lt;/p&gt; &lt;p&gt;These keywords aren't perfect - I detected the CMS for about 58% of websites. Larger charities were harder to detect the CMS - they are more likely to use a custom CMS, or one that isn't easily revealed through the HTML.&lt;/p&gt; &lt;p&gt;WordPress was the most popular CMS, powering 36% of sites. This is followed by Drupal, which powered 9% of sites. WordPress is more commonly used by smaller charities, where Drupal is more popular with larger ones.&lt;/p&gt; &lt;div class="flourish-embed flourish-chart" data-src="visualisation/8606184"&gt;&lt;/div&gt; &lt;h2 id="which-web-technologies-do-charities-use"&gt;Which web technologies do charities use?&lt;/h2&gt; &lt;p&gt;I also searched the HTML text for a set of web technologies that could be present on pages. This was a fairly arbritrary list based on existing knowledge a little bit of checking of some of the responses - so any suggestions for what else to search for would be very useful.&lt;/p&gt; &lt;p&gt;The tools I looked for, and the proportion of sites that appeared to use the tool, were:&lt;/p&gt; &lt;h3 id="web-technologies-and-javascriptcss-frameworks"&gt;Web technologies and JavaScript/CSS frameworks&lt;/h3&gt; &lt;ul&gt; &lt;li&gt;site claims to be in HTML5 (&lt;code&gt;&amp;lt;!DOCTYPE html&amp;gt;&lt;/code&gt; - &lt;strong&gt;91%&lt;/strong&gt; of sites)&lt;/li&gt; &lt;li&gt;uses jQuery (&lt;strong&gt;71%&lt;/strong&gt;)&lt;/li&gt; &lt;li&gt;uses Google fonts (&lt;strong&gt;44%&lt;/strong&gt;)&lt;/li&gt; &lt;li&gt;uses Bootstrap CSS framework (&lt;strong&gt;13%&lt;/strong&gt;)&lt;/li&gt; &lt;li&gt;uses React JS framework (&lt;strong&gt;6%&lt;/strong&gt;)&lt;/li&gt; &lt;/ul&gt; &lt;h3 id="analytics-and-social-media"&gt;Analytics and social media&lt;/h3&gt; &lt;ul&gt; &lt;li&gt;Mentions cookies - potentially a proxy for whether a cookie popup is shown (&lt;strong&gt;72%&lt;/strong&gt; of sites)&lt;/li&gt; &lt;li&gt;Google Analytics is enabled (&lt;strong&gt;62%&lt;/strong&gt;)&lt;/li&gt; &lt;li&gt;Twitter cards or Open Graph metadata available (&lt;strong&gt;62%&lt;/strong&gt;)&lt;/li&gt; &lt;li&gt;Facebook tracking enabled (&lt;strong&gt;14%&lt;/strong&gt;)&lt;/li&gt; &lt;/ul&gt; &lt;h2 id="assistive-technology"&gt;Assistive technology&lt;/h2&gt; &lt;p&gt;Note that these are pretty poor proxies for how accessible a website is for a range of users&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Includes &lt;code&gt;alt=&lt;/code&gt; somewhere on the page (could indicate that alt-text is included for some images - &lt;strong&gt;93%&lt;/strong&gt;)&lt;/li&gt; &lt;li&gt;Includes a &lt;code&gt;aria&lt;/code&gt; property somewhere on the page (&lt;strong&gt;72%&lt;/strong&gt;)&lt;/li&gt; &lt;li&gt;Uses the browsealoud extension (&lt;strong&gt;1.2%&lt;/strong&gt;)&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;As shown below, use of these tools changes based on the size of the charity. Larger charities are more likely to use Google analytics &amp;amp; Facebook tracking, more likely to mention cookies and use Open Graph or Twitter Cards.&lt;/p&gt; &lt;div class="flourish-embed flourish-chart" data-src="visualisation/8606173"&gt;&lt;/div&gt; &lt;script src="https://public.flourish.studio/resources/embed.js"&gt;&lt;/script&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Wed, 09 Feb 2022 16:06:00 +0000</pubDate><guid isPermaLink="false">tag:dkane.net,2022-02-09:/2022/what-powers-charity-websites/</guid><category>Blog</category><category>Website</category></item><item><title>Better horizontal bar charts with plotly</title><link>https://dkane.net/2020/better-horizontal-bar-charts-with-plotly/</link><description>&lt;p&gt;I often find that horizontal bar charts are a great way of visualising comparisons between different categories. They're easy to understand and make, and provide a really simple way of displaying data.&lt;/p&gt; &lt;p&gt;But I've found the default way of labelling them often doesn't make sense. Labels for the bars are usually presented to the left of the axis, and often have to be crammed together, split across multiple lines or truncated to make them readable. &lt;/p&gt; &lt;p&gt;Some charting tools can produce a much easier to read version, where the labels for the bars appear on a separate line to the bars themselves. This gives the data itself more room to breathe and makes the labels less cramped.&lt;/p&gt; &lt;p&gt;Here's an example from &lt;a href="https://www.datawrapper.de/"&gt;DataWrapper&lt;/a&gt; using some of their sample data. First using their default settings:&lt;/p&gt; &lt;figure&gt; &lt;p&gt;&lt;img alt="Bar chart showing labels to the left of the vertical axis" src="/images/horizontal-bar-1.png"&gt;&lt;/p&gt; &lt;/figure&gt; &lt;p&gt;And then with the "labels on separate line" options selected:&lt;/p&gt; &lt;figure&gt; &lt;p&gt;&lt;img alt="Bar chart showing labels above each of the bars" src="/images/horizontal-bar-2.png"&gt;&lt;/p&gt; &lt;/figure&gt; &lt;h2 id="implementing-in-plotly"&gt;Implementing in plotly&lt;/h2&gt; &lt;p&gt;I've been using &lt;a href="https://plotly.com/python/"&gt;Plotly&lt;/a&gt; for charting a lot recently, particularly in conjunction with &lt;a href="https://dash.plotly.com/"&gt;Dash&lt;/a&gt; for making rich interactive dashboards. It's a very powerful combination.&lt;/p&gt; &lt;p&gt;I built &lt;a href=""&gt;360Insights&lt;/a&gt; using Dash, and plotly's charts powers the charts on &lt;a href="https://ngoexplorer.org/"&gt;NGO Explorer&lt;/a&gt;. I wanted to record how I created the charts with labels above the bars in NGO Explorer. You can see some examples of these on a &lt;a href="https://ngoexplorer.org/country/fra"&gt;country page&lt;/a&gt;.&lt;/p&gt; &lt;figure&gt; &lt;p&gt;&lt;img alt="Bar chart from NGO Explorer showing labels above each of the bars" src="/images/horizontal-bar-3.png"&gt;&lt;/p&gt; &lt;/figure&gt; &lt;h2 id="making-subplots"&gt;Making subplots&lt;/h2&gt; &lt;p&gt;The only way I could find to accomplish the same effect in plotly was to use &lt;a href="https://plotly.com/python/subplots/"&gt;subplots&lt;/a&gt;. These essentially split the figure into a number of different charts. What we're going to do is use a different subplot for each chart, using the &lt;code&gt;make_subplots&lt;/code&gt; function. Start by importing the &lt;code&gt;make_subplots&lt;/code&gt; function.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;plotly.subplots&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;make_subplots&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Add our data to the &lt;code&gt;categories&lt;/code&gt; variable.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;categories&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Musée du Louvre, Paris&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;value&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10200000&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;National Museum of China, Beijing&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;value&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;8610092&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Metropolitan Museum of Art, New York City&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;value&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;6953927&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Vatican Museums, Vatican City&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;value&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;6756186&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Tate Modern, London&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;value&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5868562&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;British Museum, London&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;value&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5820000&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;National Gallery, London&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;value&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5735831&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;National Gallery of Art, Washington D.C.&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;value&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;4404212&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;State Hermitage Museum, Saint Petersburg&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;value&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;4220000&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Victoria and Albert Museum, London&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;value&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3967566&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="p"&gt;]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Create as many subplots as there are categories. For each subplot we'll set the title to the name of the museum using the &lt;code&gt;subplot_titles&lt;/code&gt; parameter. We also set &lt;code&gt;shared_xaxes=True&lt;/code&gt; to ensure that all the bars are still in proportion to each other, and &lt;code&gt;print_grid=False&lt;/code&gt; to make it seamless between the plots (otherwise each would be in a box.&lt;/p&gt; &lt;p&gt;The &lt;code&gt;vertical_spacing&lt;/code&gt; parameter is important, but can be tricky to get right. I've found that 0.45 divided by the number of categories gives a good spacing between them, but you may need to play around with it.&lt;/p&gt; &lt;p&gt;I've also removed the background colour from the subplots, and set the width of the plots.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;subplots&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;make_subplots&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;categories&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;cols&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;subplot_titles&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;categories&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;shared_xaxes&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;print_grid&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;vertical_spacing&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.45&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;categories&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;subplots&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;layout&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;width&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;550&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;plot_bgcolor&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;#fff&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Next we go through each category and add it to the relevant subplot, using the &lt;code&gt;add_trace&lt;/code&gt; function.&lt;/p&gt; &lt;p&gt;This functions takes a dictionary with the values and settings for the bar (notice that the &lt;code&gt;y&lt;/code&gt; and &lt;code&gt;x&lt;/code&gt; values are wrapped in lists because plotly would normally expect a list of values here). &lt;/p&gt; &lt;p&gt;We'll set the marker so they're all the same colour, and add a text label for the amount to the inside of the bar.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;categories&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;subplots&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add_trace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;bar&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;orientation&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;h&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]],&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;value&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]],&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;{:,.0f}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;value&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;])],&lt;/span&gt; &lt;span class="n"&gt;hoverinfo&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;text&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;textposition&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;auto&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;marker&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;color&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;#7030a0&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;figure&gt; &lt;p&gt;&lt;img alt="After adding traces" src="/images/plotly-bar-stage-1.png"&gt;&lt;/p&gt; &lt;/figure&gt; &lt;p&gt;This gives us subplots that are much closer to what we're looking for, but they look a bit of a mess as there's still quite a bit of the layout we need to adjust.&lt;/p&gt; &lt;p&gt;First we remove the legend on the right hand side and make sure the titles (stored by plotly as annotations) are aligned to the left hand side, and have smaller font sizes).&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;subplots&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;layout&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;showlegend&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;subplots&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;layout&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;annotations&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;x&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;xanchor&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;left&amp;#39;&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;align&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;left&amp;#39;&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;font&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;figure&gt; &lt;p&gt;&lt;img alt="After removing annotations and images" src="/images/plotly-bar-stage-2.png"&gt;&lt;/p&gt; &lt;/figure&gt; &lt;p&gt;Next we remove all the axes to give a clean look to the chart. If you wanted, you could keep the x-axis to help see the values. &lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;axis&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;subplots&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;layout&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;axis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;yaxis&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;axis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;xaxis&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;subplots&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;layout&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="n"&gt;axis&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;visible&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Finally we remove the margins at the left hand side, and calculate the height so that the categories fit nicely.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;subplots&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;layout&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;margin&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;l&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;r&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;t&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;b&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="n"&gt;height_calc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;45&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;categories&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;height_calc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;height_calc&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;350&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="n"&gt;subplots&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;layout&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;height&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;height_calc&lt;/span&gt; &lt;span class="n"&gt;subplots&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;layout&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;width&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;height_calc&lt;/span&gt; &lt;span class="n"&gt;subplots&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;h2 id="the-finished-product"&gt;The finished product&lt;/h2&gt; &lt;script type="text/javascript"&gt;window.PlotlyConfig = {MathJaxConfig: 'local'};&lt;/script&gt; &lt;script src="https://cdn.plot.ly/plotly-latest.min.js"&gt;&lt;/script&gt; &lt;div id="6165b01c-fa55-4212-99f5-9b48ce3a75d5" class="plotly-graph-div" style="height:450px; width:450px;"&gt;&lt;/div&gt; &lt;script type="text/javascript"&gt; window.PLOTLYENV=window.PLOTLYENV || {}; if (document.getElementById("6165b01c-fa55-4212-99f5-9b48ce3a75d5")) { Plotly.newPlot( '6165b01c-fa55-4212-99f5-9b48ce3a75d5', [{"hoverinfo": "text", "marker": {"color": "#7030a0"}, "orientation": "h", "text": ["10,200,000"], "textposition": "auto", "type": "bar", "x": [10200000], "xaxis": "x", "y": ["Mus\u00e9e du Louvre, Paris"], "yaxis": "y"}, {"hoverinfo": "text", "marker": {"color": "#7030a0"}, "orientation": "h", "text": ["8,610,092"], "textposition": "auto", "type": "bar", "x": [8610092], "xaxis": "x2", "y": ["National Museum of China, Beijing"], "yaxis": "y2"}, {"hoverinfo": "text", "marker": {"color": "#7030a0"}, "orientation": "h", "text": ["6,953,927"], "textposition": "auto", "type": "bar", "x": [6953927], "xaxis": "x3", "y": ["Metropolitan Museum of Art, New York City"], "yaxis": "y3"}, {"hoverinfo": "text", "marker": {"color": "#7030a0"}, "orientation": "h", "text": ["6,756,186"], "textposition": "auto", "type": "bar", "x": [6756186], "xaxis": "x4", "y": ["Vatican Museums, Vatican City"], "yaxis": "y4"}, {"hoverinfo": "text", "marker": {"color": "#7030a0"}, "orientation": "h", "text": ["5,868,562"], "textposition": "auto", "type": "bar", "x": [5868562], "xaxis": "x5", "y": ["Tate Modern, London"], "yaxis": "y5"}, {"hoverinfo": "text", "marker": {"color": "#7030a0"}, "orientation": "h", "text": ["5,820,000"], "textposition": "auto", "type": "bar", "x": [5820000], "xaxis": "x6", "y": ["British Museum, London"], "yaxis": "y6"}, {"hoverinfo": "text", "marker": {"color": "#7030a0"}, "orientation": "h", "text": ["5,735,831"], "textposition": "auto", "type": "bar", "x": [5735831], "xaxis": "x7", "y": ["National Gallery, London"], "yaxis": "y7"}, {"hoverinfo": "text", "marker": {"color": "#7030a0"}, "orientation": "h", "text": ["4,404,212"], "textposition": "auto", "type": "bar", "x": [4404212], "xaxis": "x8", "y": ["National Gallery of Art, Washington D.C."], "yaxis": "y8"}, {"hoverinfo": "text", "marker": {"color": "#7030a0"}, "orientation": "h", "text": ["4,220,000"], "textposition": "auto", "type": "bar", "x": [4220000], "xaxis": "x9", "y": ["State Hermitage Museum, Saint Petersburg"], "yaxis": "y9"}, {"hoverinfo": "text", "marker": {"color": "#7030a0"}, "orientation": "h", "text": ["3,967,566"], "textposition": "auto", "type": "bar", "x": [3967566], "xaxis": "x10", "y": ["Victoria and Albert Museum, London"], "yaxis": "y10"}], {"annotations": [{"align": "left", "font": {"size": 12}, "showarrow": false, "text": "Mus\u00e9e du Louvre, Paris", "x": 0, "xanchor": "left", "xref": "paper", "y": 0.9999999999999999, "yanchor": "bottom", "yref": "paper"}, {"align": "left", "font": {"size": 12}, "showarrow": false, "text": "National Museum of China, Beijing", "x": 0, "xanchor": "left", "xref": "paper", "y": 0.8955, "yanchor": "bottom", "yref": "paper"}, {"align": "left", "font": {"size": 12}, "showarrow": false, "text": "Metropolitan Museum of Art, New York City", "x": 0, "xanchor": "left", "xref": "paper", "y": 0.791, "yanchor": "bottom", "yref": "paper"}, {"align": "left", "font": {"size": 12}, "showarrow": false, "text": "Vatican Museums, Vatican City", "x": 0, "xanchor": "left", "xref": "paper", "y": 0.6865, "yanchor": "bottom", "yref": "paper"}, {"align": "left", "font": {"size": 12}, "showarrow": false, "text": "Tate Modern, London", "x": 0, "xanchor": "left", "xref": "paper", "y": 0.582, "yanchor": "bottom", "yref": "paper"}, {"align": "left", "font": {"size": 12}, "showarrow": false, "text": "British Museum, London", "x": 0, "xanchor": "left", "xref": "paper", "y": 0.4775, "yanchor": "bottom", "yref": "paper"}, {"align": "left", "font": {"size": 12}, "showarrow": false, "text": "National Gallery, London", "x": 0, "xanchor": "left", "xref": "paper", "y": 0.373, "yanchor": "bottom", "yref": "paper"}, {"align": "left", "font": {"size": 12}, "showarrow": false, "text": "National Gallery of Art, Washington D.C.", "x": 0, "xanchor": "left", "xref": "paper", "y": 0.26849999999999996, "yanchor": "bottom", "yref": "paper"}, {"align": "left", "font": {"size": 12}, "showarrow": false, "text": "State Hermitage Museum, Saint Petersburg", "x": 0, "xanchor": "left", "xref": "paper", "y": 0.16399999999999998, "yanchor": "bottom", "yref": "paper"}, {"align": "left", "font": {"size": 12}, "showarrow": false, "text": "Victoria and Albert Museum, London", "x": 0, "xanchor": "left", "xref": "paper", "y": 0.0595, "yanchor": "bottom", "yref": "paper"}], "height": 450, "margin": {"b": 1, "l": 0, "r": 0, "t": 20}, "plot_bgcolor": "#fff", "showlegend": false, "template": {"data": {"bar": [{"error_x": {"color": "#2a3f5f"}, "error_y": {"color": "#2a3f5f"}, "marker": {"line": {"color": "#E5ECF6", "width": 0.5}}, "type": "bar"}], "barpolar": [{"marker": {"line": {"color": "#E5ECF6", "width": 0.5}}, "type": "barpolar"}], "carpet": [{"aaxis": {"endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f"}, "baxis": {"endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f"}, "type": "carpet"}], "choropleth": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "type": "choropleth"}], "contour": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "type": "contour"}], "contourcarpet": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "type": "contourcarpet"}], "heatmap": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "type": "heatmap"}], "heatmapgl": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "type": "heatmapgl"}], "histogram": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "histogram"}], "histogram2d": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "type": "histogram2d"}], "histogram2dcontour": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "type": "histogram2dcontour"}], "mesh3d": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "type": "mesh3d"}], "parcoords": [{"line": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "parcoords"}], "pie": [{"automargin": true, "type": "pie"}], "scatter": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scatter"}], "scatter3d": [{"line": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scatter3d"}], "scattercarpet": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scattercarpet"}], "scattergeo": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scattergeo"}], "scattergl": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scattergl"}], "scattermapbox": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scattermapbox"}], "scatterpolar": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scatterpolar"}], "scatterpolargl": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scatterpolargl"}], "scatterternary": [{"marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "type": "scatterternary"}], "surface": [{"colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "type": "surface"}], "table": [{"cells": {"fill": {"color": "#EBF0F8"}, "line": {"color": "white"}}, "header": {"fill": {"color": "#C8D4E3"}, "line": {"color": "white"}}, "type": "table"}]}, "layout": {"annotationdefaults": {"arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1}, "coloraxis": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "colorscale": {"diverging": [[0, "#8e0152"], [0.1, "#c51b7d"], [0.2, "#de77ae"], [0.3, "#f1b6da"], [0.4, "#fde0ef"], [0.5, "#f7f7f7"], [0.6, "#e6f5d0"], [0.7, "#b8e186"], [0.8, "#7fbc41"], [0.9, "#4d9221"], [1, "#276419"]], "sequential": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "sequentialminus": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]]}, "colorway": ["#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52"], "font": {"color": "#2a3f5f"}, "geo": {"bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white"}, "hoverlabel": {"align": "left"}, "hovermode": "closest", "mapbox": {"style": "light"}, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": {"angularaxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}, "bgcolor": "#E5ECF6", "radialaxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}}, "scene": {"xaxis": {"backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white"}, "yaxis": {"backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white"}, "zaxis": {"backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white"}}, "shapedefaults": {"line": {"color": "#2a3f5f"}}, "ternary": {"aaxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}, "baxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}, "bgcolor": "#E5ECF6", "caxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}}, "title": {"x": 0.05}, "xaxis": {"automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": {"standoff": 15}, "zerolinecolor": "white", "zerolinewidth": 2}, "yaxis": {"automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": {"standoff": 15}, "zerolinecolor": "white", "zerolinewidth": 2}}}, "width": 450, "xaxis": {"anchor": "y", "domain": [0.0, 1.0], "matches": "x10", "showticklabels": false, "visible": false}, "xaxis10": {"anchor": "y10", "domain": [0.0, 1.0], "visible": false}, "xaxis2": {"anchor": "y2", "domain": [0.0, 1.0], "matches": "x10", "showticklabels": false, "visible": false}, "xaxis3": {"anchor": "y3", "domain": [0.0, 1.0], "matches": "x10", "showticklabels": false, "visible": false}, "xaxis4": {"anchor": "y4", "domain": [0.0, 1.0], "matches": "x10", "showticklabels": false, "visible": false}, "xaxis5": {"anchor": "y5", "domain": [0.0, 1.0], "matches": "x10", "showticklabels": false, "visible": false}, "xaxis6": {"anchor": "y6", "domain": [0.0, 1.0], "matches": "x10", "showticklabels": false, "visible": false}, "xaxis7": {"anchor": "y7", "domain": [0.0, 1.0], "matches": "x10", "showticklabels": false, "visible": false}, "xaxis8": {"anchor": "y8", "domain": [0.0, 1.0], "matches": "x10", "showticklabels": false, "visible": false}, "xaxis9": {"anchor": "y9", "domain": [0.0, 1.0], "matches": "x10", "showticklabels": false, "visible": false}, "yaxis": {"anchor": "x", "domain": [0.9404999999999999, 0.9999999999999999], "visible": false}, "yaxis10": {"anchor": "x10", "domain": [0.0, 0.0595], "visible": false}, "yaxis2": {"anchor": "x2", "domain": [0.836, 0.8955], "visible": false}, "yaxis3": {"anchor": "x3", "domain": [0.7315, 0.791], "visible": false}, "yaxis4": {"anchor": "x4", "domain": [0.627, 0.6865], "visible": false}, "yaxis5": {"anchor": "x5", "domain": [0.5225, 0.582], "visible": false}, "yaxis6": {"anchor": "x6", "domain": [0.418, 0.4775], "visible": false}, "yaxis7": {"anchor": "x7", "domain": [0.3135, 0.373], "visible": false}, "yaxis8": {"anchor": "x8", "domain": [0.209, 0.26849999999999996], "visible": false}, "yaxis9": {"anchor": "x9", "domain": [0.1045, 0.16399999999999998], "visible": false}}, {"responsive": true} ) }; &lt;/script&gt; &lt;p&gt;I think the final result is a lot clearer than the default chart for the data. I've put together all of this together in a function to help making these types of charts easier.&lt;/p&gt; &lt;p&gt;You can play around with the settings, either as part of &lt;code&gt;make_subplots&lt;/code&gt;, the individual bars (&lt;code&gt;add_trace&lt;/code&gt;) or the layout (make changes to &lt;code&gt;subplots['layout']&lt;/code&gt;.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;plotly.subplots&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;make_subplots&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;horizontal_bar_labels&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;categories&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;subplots&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;make_subplots&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;categories&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;cols&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;subplot_titles&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;categories&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;shared_xaxes&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;print_grid&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;vertical_spacing&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.45&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;categories&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;subplots&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;layout&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;width&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;550&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;plot_bgcolor&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;#fff&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# add bars for the categories&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;categories&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;subplots&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add_trace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;bar&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;orientation&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;h&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]],&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;value&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]],&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;{:,.0f}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;value&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;])],&lt;/span&gt; &lt;span class="n"&gt;hoverinfo&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;text&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;textposition&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;auto&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;marker&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;color&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;#7030a0&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# update the layout&lt;/span&gt; &lt;span class="n"&gt;subplots&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;layout&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;showlegend&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;subplots&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;layout&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;annotations&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;x&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;xanchor&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;left&amp;#39;&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;align&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;left&amp;#39;&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;font&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# hide the axes&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;axis&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;subplots&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;layout&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;axis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;yaxis&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;axis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;xaxis&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;subplots&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;layout&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="n"&gt;axis&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;visible&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt; &lt;span class="c1"&gt;# update the margins and size&lt;/span&gt; &lt;span class="n"&gt;subplots&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;layout&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;margin&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;l&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;r&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;t&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;b&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="n"&gt;height_calc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;45&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;categories&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;height_calc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;height_calc&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;350&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="n"&gt;subplots&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;layout&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;height&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;height_calc&lt;/span&gt; &lt;span class="n"&gt;subplots&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;layout&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;width&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;height_calc&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;subplots&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;h2 id="a-note-on-log-axes"&gt;A note on log axes&lt;/h2&gt; &lt;p&gt;If you're using a log axis, as some of the charts on &lt;ngoexplorer.org&gt; do, then the implementation above may not work due to the different way that log ranges are calculated.&lt;/p&gt; &lt;p&gt;Adding something similar to this on the function above should help (where &lt;code&gt;log_axis&lt;/code&gt; is a boolean variable):&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;hb_plot&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;layout&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;xaxis&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;log_axis&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;hb_plot&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;layout&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;type&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;log&amp;quot;&lt;/span&gt; &lt;span class="n"&gt;hb_plot&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;layout&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;range&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;log10&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;max_value&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;hb_plot&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;layout&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;range&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_value&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;1.1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;h2 id="updated-2021-11-25"&gt;Updated 2021-11-25&lt;/h2&gt; &lt;p&gt;Thanks to Chris Caudill for pointing out a missing part of the function - this is now included. I've also added the section at the end on log axes.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Tue, 16 Jun 2020 15:46:51 +0100</pubDate><guid isPermaLink="false">tag:dkane.net,2020-06-16:/2020/better-horizontal-bar-charts-with-plotly/</guid><category>Code</category><category>Charts</category><category>Plotly</category></item><item><title>How many charity employees have been furloughed?</title><link>https://dkane.net/2020/how-many-charity-employees-have-been-furloughed/</link><description>&lt;p&gt;&lt;em&gt;This blog post is a write-up of a &lt;a href="https://twitter.com/kanedr/status/1271395238331011080"&gt;twitter thread&lt;/a&gt; exploring data available. These are very rough estimates, based on data that's available at the time of writing.&lt;/em&gt;&lt;/p&gt; &lt;p&gt;It's frustratingly difficult to find out how many charities have used the government's Coronavirus Job Retention Scheme, or how many employees have been furloughed through it.&lt;/p&gt; &lt;p&gt;For the rest of the economy, HMRC has &lt;a href="https://www.gov.uk/government/statistics/coronavirus-job-retention-scheme-statistics-june-2020"&gt;produced "experimental statistics"&lt;/a&gt; with estimates of the number of employers and employees that have used the scheme. These statistics are based on the data HMRC uses to administer the scheme. As well as overall totals, they've been able to produce estimates of how use of the scheme has varied across sectors of the economy (aka "industries", using SIC2007) and geography (local authorities and regions).&lt;/p&gt; &lt;p&gt;Unfortunately it's not possible to directly extract charities from this data - charity is a legal form, and charities work in lots of different industries as classified by SIC2007 - health, arts, social care, education, etc. So the industry breakdown doesn't allow charities to be identified.&lt;/p&gt; &lt;h2 id="estimating-using-sector-data"&gt;Estimating using sector data&lt;/h2&gt; &lt;p&gt;However, even though the industry data doesn't allow us to directly estimate the numbers of charity employers and employees using the scheme, we can use the industry breakdowns as a base for some rough estimates. &lt;/p&gt; &lt;p&gt;If we make the assumption that charities behave similarly to other organisations in their industry (which may not be true) then if we know the proportion of organisations in each industry that are charities, we can apply that proportion to the number of employers and employees to make and estimate of figures for charities.&lt;/p&gt; &lt;p&gt;But how to get an estimate of the proportion of companies that are charities? Here I've turned to Companies House and Charity Commission data. Using data extracts from these two regulators, it's possible to get a list of companies, their SIC codes (for their industry) and whether or not they are a registered charity. From this we can produce a list of industries and the proportion of organisations in each that are charities.&lt;/p&gt; &lt;p&gt;But is this a fair picture? Is the number of companies a proxy for the number of employees? Well possibly, but it's probably more complicated. We know that employees aren't evenly distributed in any sector of the economy - &lt;a href="https://data.ncvo.org.uk/workforce/#by-size"&gt;NCVO's data shows that 40-50% of employees in private &amp;amp; voluntary sector work in organisations with more than 50 employees&lt;/a&gt;.&lt;/p&gt; &lt;p&gt;There are lots of small charities without paid staff, and similarly there are hundreds of thousands of small registered companies that don't employ people, or are currently dormant. To strip these out, I used a field in the company data that describes the type of accounts the company last submitted - this is based on their size. But limiting it to accounts returned by active companies, we get a better sense of the proportion of "active" companies that are charities. &lt;/p&gt; &lt;h2 id="results-of-these-estimates"&gt;Results of these estimates&lt;/h2&gt; &lt;p&gt;The two methods - using all companies and then using larger "active" companies, give two different figures for the proportion of registered companies that are charities. For all companies charities represent 0.7% of companies, compared to 2% of "active" companies. The latter figure perhaps feels instinctively more correct - we know from NCVO that charity employees are around 3% of the workforce.&lt;/p&gt; &lt;p&gt;The figures give the proportions for industries. These proportions feel correct - charities are concentrated in 4 SIC industries: education (20% of companies), health and social work (12% of companies), arts &amp;amp; recreation (12% of companies), trade union, religious, political and repair (8% of companies). They're also absent where you would expect - mining, manufacturing and construction show negligible numbers of charities (the SIC classification system is definitely not perfect in every case).&lt;/p&gt; &lt;iframe title="% of companies that are charities, by industry" aria-label="Range Plot" id="datawrapper-chart-oCE6u" src="https://datawrapper.dwcdn.net/oCE6u/1/" scrolling="no" frameborder="0" style="border: none;" width="600" height="605"&gt;&lt;/iframe&gt; &lt;p&gt;And using these proportions we can make estimates of the number of charities that are using the furlough scheme, and the number of employees. Using the proportions for all companies gives a figure of 8,000 charities and 69,000 employees, but using just larger active companies gives &lt;strong&gt;22,000 charities&lt;/strong&gt; and &lt;strong&gt;188,000 employees&lt;/strong&gt;.&lt;/p&gt; &lt;p&gt;While there's a number of assumptions going into these figures, I think they feel like sensible estimates of the boundaries of the actual number, though the larger one is perhaps closer to the truth.&lt;/p&gt; &lt;iframe title="Estimated number of charity employees furloughed" aria-label="Range Plot" id="datawrapper-chart-oX8S0" src="https://datawrapper.dwcdn.net/oX8S0/1/" scrolling="no" frameborder="0" style="border: none;" width="600" height="586"&gt;&lt;/iframe&gt; &lt;h2 id="comparing-with-survey-results"&gt;Comparing with survey results&lt;/h2&gt; &lt;p&gt;When I shared these results on Twitter, there were some helpful pointers to attempts to answer the same questions using surveys.&lt;/p&gt; &lt;p&gt;Nick Temple of the Social Investment Business estimates that 13% of their customer base's employees are on furlough.&lt;/p&gt; &lt;blockquote class="twitter-tweet"&gt;&lt;p lang="en" dir="ltr"&gt;Thanks for doing this David; even a rough guess is helpful. Recent analysis of our customer base came up with around 13% of all employees on furlough. Though SIB customer base tends to be medium, rather than small/micro.&lt;/p&gt;&amp;mdash; Nick Temple (@nicktemple1) &lt;a href="https://twitter.com/nicktemple1/status/1271438133473705986?ref_src=twsrc%5Etfw"&gt;June 12, 2020&lt;/a&gt;&lt;/blockquote&gt; &lt;script async src="https://platform.twitter.com/widgets.js" charset="utf-8"&gt;&lt;/script&gt; &lt;p&gt;Charity Tax Group &amp;amp; Charity Retail suggest that many of the 26,000 charity shop employees have been furloughed - the unique nature of these employees (similar private businesses do not have retail employees) means that they are potentially missed from my estimate.&lt;/p&gt; &lt;blockquote class="twitter-tweet"&gt;&lt;p lang="en" dir="ltr"&gt;Based on recent discussions with &lt;a href="https://twitter.com/CharityRetail?ref_src=twsrc%5Etfw"&gt;@CharityRetail&lt;/a&gt; there are 26,000 FTE charity shop employees &amp;amp; many of them have been furloughed. &lt;a href="https://twitter.com/CivilSocietyUK?ref_src=twsrc%5Etfw"&gt;@CivilSocietyUK&lt;/a&gt; estimated 20,000 from the 20 largest charities some time back. Very difficult to be certain though &lt;a href="https://t.co/FZhj3jdC0e"&gt;https://t.co/FZhj3jdC0e&lt;/a&gt;&lt;/p&gt;&amp;mdash; Charity Tax Group (@CharityTaxGroup) &lt;a href="https://twitter.com/CharityTaxGroup/status/1271440561581498370?ref_src=twsrc%5Etfw"&gt;June 12, 2020&lt;/a&gt;&lt;/blockquote&gt; &lt;script async src="https://platform.twitter.com/widgets.js" charset="utf-8"&gt;&lt;/script&gt; &lt;p&gt;A Charity Tax Group survey showed 70% of respondents had made a claim.&lt;/p&gt; &lt;blockquote class="twitter-tweet"&gt;&lt;p lang="en" dir="ltr"&gt;This does not provide specific figures but a recent CTG snapshot poll showed high level of take-up organisationally (without specific responses on furloughed staff per charity) &lt;a href="https://t.co/6Z7gxsMeab"&gt;pic.twitter.com/6Z7gxsMeab&lt;/a&gt;&lt;/p&gt;&amp;mdash; Charity Tax Group (@CharityTaxGroup) &lt;a href="https://twitter.com/CharityTaxGroup/status/1271441766068166656?ref_src=twsrc%5Etfw"&gt;June 12, 2020&lt;/a&gt;&lt;/blockquote&gt; &lt;script async src="https://platform.twitter.com/widgets.js" charset="utf-8"&gt;&lt;/script&gt; &lt;p&gt;66% of respondents to &lt;a href="https://www.probonoeconomics.com/news/pbe-launches-new-survey-understand-impact-covid-19-charity-sector"&gt;Pro Bono Economics&lt;/a&gt; survey (9-10 June) said they had furloughed or used job retention scheme.&lt;/p&gt; &lt;h2 id="what-next"&gt;What next?&lt;/h2&gt; &lt;p&gt;These estimates are probably wrong. They're based on imperfect data and assumptions that are unlikely to be correct. However, given the absence of better data, I think they give an indication of the scale we would expect to see in figures.&lt;/p&gt; &lt;p&gt;ONS and HMRC have done a fantastic job during this crisis of producing statistics to understand the environment and measure responses. But charities can often fall through the cracks in official statistics - due to lack of data, or because they're not thought about when the measures are designed.&lt;/p&gt; &lt;p&gt;I'd like to see better official data on charities in this crisis, to help the charity sector measure the impact and respond. &lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Fri, 12 Jun 2020 16:06:00 +0100</pubDate><guid isPermaLink="false">tag:dkane.net,2020-06-12:/2020/how-many-charity-employees-have-been-furloughed/</guid><category>Blog</category><category>Covid19</category></item><item><title>Working with XBRL</title><link>https://dkane.net/2018/working-with-xbrl/</link><description>&lt;p&gt;Some notes on the XBRL data format, as I've been working with it this week. I'm working with Power to Change to help them extract data from accounts.&lt;/p&gt; &lt;h2 id="whats-xbrlixbrl"&gt;What's XBRL/iXBRL?&lt;/h2&gt; &lt;p&gt;XBRL standards for eXtensible Business Reporting Language. iXBRL is the same but "inline". They describe a data standard for financial reporting, based on XML. In the UK it's used by Companies House and HMRC for reporting - instead of submitting a PDF or paper copy of their accounts (which contains no machine readable version of the data within) companies can (and in some cases must) submit in XBRL format.&lt;/p&gt; &lt;p&gt;The iXBRL format differs slightly from XBRL in that the machine-readable financial data is contained within a human-readable HTML file. That way you get the best of both words - documents that can be read by people interested in that company, with data that can be extracted in a consistent way.&lt;/p&gt; &lt;p&gt;That's the theory, anyway.&lt;/p&gt; &lt;h2 id="whats-in-xbrl"&gt;What's in XBRL?&lt;/h2&gt; &lt;p&gt;In my mind, XBRL has two main parts:&lt;/p&gt; &lt;ol&gt; &lt;li&gt;The XBRL &lt;strong&gt;format&lt;/strong&gt;, describing how to represent various concepts within an XML or HTML document. The formats for XBRL and iXBRL are slightly different - you should be able to convert an iXBRL document to an XBRL one, but that wouldn't work the other way round.&lt;/li&gt; &lt;li&gt;&lt;strong&gt;Schemas&lt;/strong&gt; which are applied on top of the XBRL format to represent financial data in a specific framework - for example the FRS accounting framework or the Charity SORP. &lt;/li&gt; &lt;/ol&gt; &lt;p&gt;The format describes what the data should look like, while the schema describes what each individual data item refers to. &lt;/p&gt; &lt;h2 id="how-do-you-make-xbrl"&gt;How do you make XBRL?&lt;/h2&gt; &lt;p&gt;This isn't a particularly relevant question for me - for this exercise I'm more interested in how to get data out of the documents.&lt;/p&gt; &lt;p&gt;But I think most XBRL documents are produced by standard accounting packages. There are also specialist firms like &lt;a href="http://ixbrl-assist.co.uk/"&gt;Hypercube Consultancy&lt;/a&gt; who provide tagging services for accounts. Stewart from Hypercube was kind enough to get in touch with me and share his experience of creating and using XBRL and some of the tools and resources out there.&lt;/p&gt; &lt;h2 id="what-can-you-do-with-xbrl"&gt;What can you do with XBRL?&lt;/h2&gt; &lt;p&gt;This is what I'm interested in - how do you get data out of XBRL documents. My aim in this is to look for particular values (total assets, for example) and compare them across different organisations. &lt;/p&gt; &lt;p&gt;This is trickier that I had thought. I had hoped that there would be a nice suite of (open source) tools for parsing and extracting data from XBRL - but as far as I can tell there isn't. I did know from trying beforehand that the flexibility of XBRL and iXBRL means it is very complex to deal with - often the same value can be represented in different ways depending on the choices made.&lt;/p&gt; &lt;p&gt;&lt;strong&gt;Resources&lt;/strong&gt;&lt;/p&gt; &lt;p&gt;There are some resources out there though:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href="https://uk-taxonomies-tdp.corefiling.com/yeti/resources/yeti-gwt/Yeti.jsp"&gt;Yeti&lt;/a&gt; - a tool created by CoreFiling, who I think were one of the original authors of XBRL. One issue I've had looking at XBRL is navigating the schemas, and this tool makes it much easier to do so.&lt;/li&gt; &lt;li&gt;&lt;a href="https://github.com/XBRLInternational/ixbrl-samples"&gt;Sample iXBRL files&lt;/a&gt; - This seems to be the only github presence of XBRL International who look after the standard, but it's really useful for testing tools.&lt;/li&gt; &lt;li&gt;Companies House - There's a few ways you can get real-life iXBRL files for companies:&lt;ul&gt; &lt;li&gt;Via the &lt;a href="https://beta.companieshouse.gov.uk/"&gt;Companies House register&lt;/a&gt; if a company has filed iXBRL accounts they're available for download on the filing history page for that company. Not all companies file iXBRL accounts so if they're not available then PDF copies are instead.&lt;/li&gt; &lt;li&gt;You can also download accounts in bulk via the &lt;a href="http://download.companieshouse.gov.uk/en_accountsdata.html"&gt;Free Accounts Data Product&lt;/a&gt;. This has daily and monthly dumps of any XBRL or iXBRL accounts filed with Companies House. This is useful for getting accounts to test on.&lt;/li&gt; &lt;/ul&gt; &lt;/li&gt; &lt;li&gt;You can also view the official specifications for the format - I found these very difficult to read and get a sense of what the resulting data looks like:&lt;ul&gt; &lt;li&gt;&lt;a href="http://www.xbrl.org/Specification/XBRL-2.1/REC-2003-12-31/XBRL-2.1-REC-2003-12-31+corrected-errata-2013-02-20.html#_4.7"&gt;XBRL 2.1&lt;/a&gt; - released 2003, corrected in 2013&lt;/li&gt; &lt;li&gt;&lt;a href="https://www.xbrl.org/specification/inlinexbrl-part1/rec-2013-11-18/inlinexbrl-part1-rec-2013-11-18.html"&gt;iXBRL 1.1&lt;/a&gt; - released 2013&lt;/li&gt; &lt;li&gt;&lt;a href="http://www.xbrl.org/Specification/xbrl-csv/PWD-2017-05-02/xbrl-csv-PWD-2017-05-02.html"&gt;xBRL-CSV&lt;/a&gt; - a proposal for converting XBRL to CSV format. I've not seen whether this is implemented anywhere, but this would be very useful for my purposes.&lt;/li&gt; &lt;/ul&gt; &lt;/li&gt; &lt;/ul&gt; &lt;h2 id="what-have-i-done"&gt;What have I done?&lt;/h2&gt; &lt;p&gt;As these notes suggest, a lot of what I've done is trying to make sense of the landscape and explore the data. I did most of this through jupyter notebooks, looking through iXBRL files of a selection of community businesses. I've found the python &lt;a href="https://www.crummy.com/software/BeautifulSoup/"&gt;BeautifulSoup&lt;/a&gt; library helpful for parsing the XML.&lt;/p&gt; &lt;p&gt;Using what I'd learnt from exploring the iXBRL files I've started to put together a python module for parsing iXBRL and extracting the data. I'll be aiming to develop this further, and hopefully use it in anger: &lt;a href="https://github.com/drkane/ixbrl-parse"&gt;https://github.com/drkane/ixbrl-parse&lt;/a&gt;.&lt;/p&gt; &lt;h2 id="who-else-is-doing-things-with-xbrl"&gt;Who else is doing things with XBRL&lt;/h2&gt; &lt;p&gt;I did a search of GitHub to find XBRL repositories. There were a few people who'd tried to do similar things over the years. But most interestingly, the &lt;a href="https://github.com/ONSBigData/parsing_company_accounts"&gt;ONS Big Data Unit&lt;/a&gt; has an active repository with some code, approaching it in a very similar way to me. I've got in touch with them to see if they want to compare notes.&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Tue, 18 Dec 2018 11:06:07 +0000</pubDate><guid isPermaLink="false">tag:dkane.net,2018-12-18:/2018/working-with-xbrl/</guid><category>Blog</category><category>XBRL</category><category>iXBRL</category><category>accounts</category></item><item><title>Text analysis in the voluntary sector</title><link>https://dkane.net/2018/text-analysis-in-the-voluntary-sector/</link><description>&lt;p&gt;&lt;img alt="Chart from Google n-grams showing results for data and cheese since 1800. Data overtakes cheese in 1970ish" src="/images/data-cheese-chart.png"&gt; &lt;/p&gt; &lt;p&gt;Last week at the &lt;a href="https://www.data4goodconf.org.uk/"&gt;Data 4 Good event&lt;/a&gt; in Birmingham I gave a talk on the opportunities and challenges that text analysis offers to the voluntary sector. It was a short lightning talk so I was only able to give a quick overview of the possibilities, and it's a big area so there's lots more to explore.&lt;/p&gt; &lt;p&gt;You can find the slides on Google drive.&lt;/p&gt; &lt;iframe src="https://docs.google.com/presentation/d/e/2PACX-1vTvN-R9g5Dtcr72YKIe5RWSSBRqQlpUmaVpQfRDxd5EIPVzwPmzMPbVGCnJapJPjOMnIqhBCii7Eq30/embed?start=false&amp;loop=false&amp;delayms=3000" frameborder="0" allowfullscreen="true" mozallowfullscreen="true" webkitallowfullscreen="true" style="width:100%;height:389px;"&gt;&lt;/iframe&gt; &lt;p&gt;In the last slide I suggest some tools you can use to take your data further with text analysis. They are:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href="https://databasic.io/en/samediff/"&gt;&lt;strong&gt;DataBasic – SameDiff&lt;/strong&gt;&lt;/a&gt; - a simple tool for comparing two sets of text&lt;/li&gt; &lt;li&gt;&lt;a href="https://databasic.io/en/wordcounter/"&gt;&lt;strong&gt;DataBasic – word counter&lt;/strong&gt;&lt;/a&gt; - create word clouds and look in more detail at the frequency of words in a piece of text.&lt;/li&gt; &lt;li&gt;&lt;a href="http://www.opencalais.com/opencalais-demo/"&gt;&lt;strong&gt;Reuters OpenCalais&lt;/strong&gt;&lt;/a&gt; - a proprietary tool for getting started with entity recognition. I'd be interested to know if there are any similar open-source projects - one I've seen is &lt;a href="https://spacy.io/usage/spacy-101#annotations-ner"&gt;spacy&lt;/a&gt; but it requires some programming knowledge to use.&lt;/li&gt; &lt;li&gt;&lt;a href="http://openrefine.org/"&gt;&lt;strong&gt;OpenRefine&lt;/strong&gt;&lt;/a&gt; - a useful data wrangling tool with lots of useful features for working with text.&lt;/li&gt; &lt;/ul&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Mon, 19 Nov 2018 09:58:43 +0000</pubDate><guid isPermaLink="false">tag:dkane.net,2018-11-19:/2018/text-analysis-in-the-voluntary-sector/</guid><category>Blog</category><category>Text analysis</category></item><item><title>A map of World Cup stadia using wikidata</title><link>https://dkane.net/2018/a-map-of-world-cup-stadia-using-wikidata/</link><description>&lt;p&gt;&lt;a href="https://www.wikidata.org/wiki/Wikidata:Main_Page"&gt;Wikidata&lt;/a&gt; is an amazing project that aims to turn the unstructured text of Wikipedia into a database of facts and figures that allows you to go beyond just presenting a page about something to using data about it.&lt;/p&gt; &lt;p&gt;I've been wanting to try out using it, and "SPARQL", the language used to query it, so I decided to try and create a map of every stadium that has hosted a game at the Fifa World Cup finals - a topical query as the 2018 World Cup in Russia has just started.&lt;/p&gt; &lt;h2 id="step-1-querying-the-data"&gt;Step 1. Querying the data&lt;/h2&gt; &lt;p&gt;I used &lt;a href="https://query.wikidata.org/"&gt;query.wikidata.org&lt;/a&gt; to come up with a query that got me the data I was looking for. Having never used SPARQL before it took a bit of tweaking to get the query I needed - I found the interface helpful for finding the right entities and the included examples for how to structure it.&lt;/p&gt; &lt;p&gt;Here's the query I came up with. I'll go through what each part does below.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;wc_sparql&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;&amp;quot;&amp;quot;&lt;/span&gt; &lt;span class="s2"&gt;SELECT ?FIFA_World_CupLabel ?location ?locationLabel ?coord ?countryLabel WHERE {&lt;/span&gt; &lt;span class="s2"&gt; SERVICE wikibase:label { bd:serviceParam wikibase:language &amp;quot;[AUTO_LANGUAGE],en&amp;quot;. }&lt;/span&gt; &lt;span class="s2"&gt; ?FIFA_World_Cup wdt:P3450 wd:Q19317.&lt;/span&gt; &lt;span class="s2"&gt; ?FIFA_World_Cup wdt:P276 ?location.&lt;/span&gt; &lt;span class="s2"&gt; ?location wdt:P625 ?coord.&lt;/span&gt; &lt;span class="s2"&gt; ?location wdt:P17 ?country&lt;/span&gt; &lt;span class="s2"&gt;}&lt;/span&gt; &lt;span class="s2"&gt;ORDER BY ?FIFA_World_CupLabel&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;&amp;quot;&amp;quot;&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;The first part sets up the fields we want to return - the name of the World Cup, the location ID (a stadium), the name of the stadium, the latitude and longitude and the name of the country&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;SELECT ?FIFA_World_CupLabel ?location ?locationLabel ?coord ?countryLabel WHERE { &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;This next part allows you to fetch labels for each of the items, which is more helpful than the URI that gets returned.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;SERVICE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;wikibase&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;bd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;serviceParam&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;wikibase&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="k"&gt;language&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;&amp;quot;[AUTO_LANGUAGE],en&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;}&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Then we start off by adding a field called "FIFA_World_Cup" based on finding "sports season of league or competition" (&lt;code&gt;wdt:P3450&lt;/code&gt;) with the labels "FIFA World Cup" (&lt;code&gt;wd:Q19317&lt;/code&gt;)&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;?FIFA_World_Cup wdt:P3450 wd:Q19317. &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Then we look for the locations (&lt;code&gt;wdt:P276&lt;/code&gt;) attached to each of these competitions:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;?FIFA_World_Cup wdt:P276 ?location. &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;And for each location we want the co-ordinates (&lt;code&gt;wdt:P625&lt;/code&gt;) and country (&lt;code&gt;wdt:P17&lt;/code&gt;).&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;?location wdt:P625 ?coord. ?location wdt:P17 ?country &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;I then used a python library called &lt;a href="https://rdflib.github.io/sparqlwrapper/"&gt;SPARQLWrapper&lt;/a&gt; to send the query to the WikiData sparql endpoint, and get JSON data back.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;SPARQLWrapper&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SPARQLWrapper&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;JSON&lt;/span&gt; &lt;span class="n"&gt;sparql&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;SPARQLWrapper&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;https://query.wikidata.org/sparql&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;sparql&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;setQuery&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;wc_sparql&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;sparql&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;setReturnFormat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sparql&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;convert&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Here's an example of what one of the results looks like - a stadium used in the very first World Cup in Uruguay.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;results&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;bindings&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;FIFA_World_CupLabel&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;literal&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;value&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="mi"&gt;1930&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;FIFA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;World&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Cup&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;xml&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="nx"&gt;lang&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;en&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;coord&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;datatype&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;http&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="c1"&gt;//www.opengis.net/ont/geosparql#wktLiteral&amp;#39;,&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;literal&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;value&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;Point&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="m m-Double"&gt;56.152778&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="m m-Double"&gt;34.894444&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;countryLabel&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;literal&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;value&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;Uruguay&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;xml&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="nx"&gt;lang&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;en&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;location&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;uri&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;value&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;http&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="c1"&gt;//www.wikidata.org/entity/Q498245&amp;#39;},&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;locationLabel&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;literal&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;value&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;Estadio&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Centenario&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;xml&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="nx"&gt;lang&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;en&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;}}&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;h2 id="step-2-processing-the-results"&gt;Step 2: processing the results&lt;/h2&gt; &lt;p&gt;I then want to turn the results into nicely formatted data for plotting on a map. I'm looking for data that contains one record for each stadium, even if it has hosted games at more than one World Cup (e.g Mexico in 1970 and 1986). &lt;/p&gt; &lt;p&gt;The co-ordinates for each location come in WKT format, so I use a library called &lt;a href="https://pypi.org/project/Shapely/"&gt;Shapely&lt;/a&gt; to extract the latitude and longitude.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# for converting coordinates&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;shapely.wkt&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Then I go through each of the results and add to a python dictionary. If the stadium is already in the dictionary I just add the extra World Cup year to the dictionary, rather than adding a new record.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;stadia&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;results&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bindings&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="n"&gt;stadium_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;location&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;value&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="n"&gt;worldcup&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;FIFA_World_CupLabel&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;value&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot; FIFA World Cup&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;stadium_id&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;stadia&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;stadia&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;stadium_id&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;worldcups&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;worldcup&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;stadia&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;stadium_id&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;lat_lng&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;shapely&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;wkt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;coord&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;value&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;coords&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;worldcups&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;worldcup&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;stadium&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;locationLabel&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;value&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;country&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;countryLabel&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;value&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Here's what an entry in the processed data looks like. I've used the wikidata URI as an identifier for each stadium.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;stadia&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;http://www.wikidata.org/entity/Q498245&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;{&amp;#39;country&amp;#39;: &amp;#39;Uruguay&amp;#39;, &amp;#39;lat_lng&amp;#39;: (-56.152778, -34.894444), &amp;#39;stadium&amp;#39;: &amp;#39;Estadio Centenario&amp;#39;, &amp;#39;worldcups&amp;#39;: [&amp;#39;1930&amp;#39;]} &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;h2 id="step-3-mapping-the-results"&gt;Step 3: Mapping the results&lt;/h2&gt; &lt;p&gt;I really like &lt;a href="http://python-visualization.github.io/folium/index.html"&gt;folium&lt;/a&gt; for easily producing Leaflet-based maps in python. I'm going to also use the MarkerCluster plugin to cluster the markers to make it easier to view all the stadia on one map - with clusters based on countries.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;folium&lt;/span&gt; &lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;folium.plugins&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;MarkerCluster&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;html&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;First I initialise the map and zoom out so you can see the whole world.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;folium&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;location&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;zoom_start&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tiles&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Stamen Toner&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;attr&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&amp;#39;&amp;#39;&amp;lt;a id=&amp;quot;home-link&amp;quot; target=&amp;quot;_top&amp;quot; href=&amp;quot;../&amp;quot;&amp;gt;Map tiles&amp;lt;/a&amp;gt; by &lt;/span&gt; &lt;span class="s1"&gt; &amp;lt;a target=&amp;quot;_top&amp;quot; href=&amp;quot;http://stamen.com&amp;quot;&amp;gt;Stamen Design&amp;lt;/a&amp;gt;, &lt;/span&gt; &lt;span class="s1"&gt; under &amp;lt;a target=&amp;quot;_top&amp;quot; href=&amp;quot;http://creativecommons.org/licenses/by/3.0&amp;quot;&amp;gt;CC BY 3.0&amp;lt;/a&amp;gt;. &lt;/span&gt; &lt;span class="s1"&gt; Data by &amp;lt;a target=&amp;quot;_top&amp;quot; href=&amp;quot;http://openstreetmap.org&amp;quot;&amp;gt;OpenStreetMap&amp;lt;/a&amp;gt;, &lt;/span&gt; &lt;span class="s1"&gt; under &amp;lt;a target=&amp;quot;_top&amp;quot; href=&amp;quot;http://creativecommons.org/licenses/by-sa/3.0&amp;quot;&amp;gt;CC BY SA&amp;lt;/a&amp;gt;.&lt;/span&gt; &lt;span class="s1"&gt; | Locations powered by &amp;lt;a href=&amp;quot;https://query.wikidata.org/&amp;quot;&amp;gt;Wikidata&amp;lt;/a&amp;gt;.&amp;#39;&amp;#39;&amp;#39;&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Then we go through the stadia and add each one to a cluster based on its country. I've also added a little popup which tells you the stadium's name and which World Cups it hosted games at. I also set a football icon for the pins.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;clusters&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;stadium_id&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;stadia&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;stadia&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;stadium_id&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;country&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;clusters&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;clusters&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;country&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;MarkerCluster&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add_to&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;folium&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Marker&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;lat_lng&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;lat_lng&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]],&lt;/span&gt; &lt;span class="n"&gt;popup&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="si"&gt;{}&lt;/span&gt;&lt;span class="s1"&gt;, &lt;/span&gt;&lt;span class="si"&gt;{}&lt;/span&gt;&lt;span class="s1"&gt; - &amp;lt;i&amp;gt;&lt;/span&gt;&lt;span class="si"&gt;{}&lt;/span&gt;&lt;span class="s1"&gt;&amp;lt;/i&amp;gt;&amp;#39;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;escape&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;stadium&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]),&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;escape&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;country&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]),&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;escape&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;, &amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;worldcups&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]))&lt;/span&gt; &lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;icon&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;folium&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Icon&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;icon&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;soccer-ball-o&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;prefix&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;fa&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add_to&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;clusters&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;country&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]])&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Finally we show the resulting map, which can be zoomed and panned to look at particular countries.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;m&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div style="width:100%;"&gt;&lt;div style="position:relative;width:100%;height:0;padding-bottom:60%;"&gt;&lt;iframe src="/data/world-cup-stadia-map.html" style="position:absolute;width:100%;height:100%;left:0;top:0;border:none !important;" allowfullscreen webkitallowfullscreen mozallowfullscreen&gt;&lt;/iframe&gt;&lt;/div&gt;&lt;/div&gt; &lt;p&gt;As an extra I wanted to convert the data into GeoJSON format so it's easy to use elsewhere.&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href="https://dkane.net/data/world_cup_stadia.geojson"&gt;world_cup_stadia.geojson&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;geojson&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Feature&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Point&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;FeatureCollection&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;wc_geojson&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;FeatureCollection&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Feature&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;geometry&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;Point&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;stadia&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;lat_lng&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]),&lt;/span&gt; &lt;span class="n"&gt;properties&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;stadia&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;stadia&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;world_cup_stadia.geojson&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;w&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;geojson&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dump&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;wc_geojson&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;indent&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;h2 id="step-4-taking-it-further"&gt;Step 4: taking it further&lt;/h2&gt; &lt;p&gt;This was just a quick exercise to try and get data out of wikidata and then use it. There's a few things that could be done to take it further:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;add filters to the map to filter by country, World Cup, etc.&lt;/li&gt; &lt;li&gt;see if Wikidata has data on the matches that took place at each location and the teams that have played there, allowing you to filter by team or stage of the competition.&lt;/li&gt; &lt;li&gt;visualise the data by adding in details like the maximum attendance &lt;/li&gt; &lt;/ul&gt; &lt;h2 id="acknowledgements"&gt;Acknowledgements&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;The data is from Wikidata and is used under the &lt;a href="https://creativecommons.org/about/cc0"&gt;CC0 Public Domain licence&lt;/a&gt;. &lt;/li&gt; &lt;li&gt;&lt;a href="https://rdflib.github.io/sparqlwrapper/"&gt;sparqlwrapper&lt;/a&gt; for querying the data.&lt;/li&gt; &lt;li&gt;&lt;a href="http://python-visualization.github.io/folium/index.html"&gt;Folium&lt;/a&gt; mapping using &lt;a href="https://leafletjs.com/"&gt;Leaflet&lt;/a&gt; and &lt;a href="http://fontawesome.com/icons?d=gallery&amp;amp;c=sports&amp;amp;m=free"&gt;FontAwesome&lt;/a&gt; icons.&lt;/li&gt; &lt;li&gt;Map tiles by &lt;a href="http://stamen.com"&gt;Stamen Design&lt;/a&gt;, under &lt;a href="http://creativecommons.org/licenses/by/3.0"&gt;CC BY 3.0&lt;/a&gt;. Data by &lt;a href="http://openstreetmap.org"&gt;OpenStreetMap&lt;/a&gt;, under &lt;a href="http://creativecommons.org/licenses/by-sa/3.0"&gt;CC BY SA&lt;/a&gt;.&lt;/li&gt; &lt;li&gt;&lt;a href="https://pypi.org/project/Shapely/"&gt;Shapely&lt;/a&gt; and &lt;a href="https://pypi.org/project/geojson/"&gt;geojson&lt;/a&gt; python libraries for manipulating the geodata&lt;/li&gt; &lt;li&gt;&lt;a href="https://nbviewer.jupyter.org/urls/dkane.net/data/world_cup_stadia.ipynb"&gt;This blog post was written alongside the code in a Juypter notebook&lt;/a&gt;.&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;&lt;a title="By Charlie Kritschmar (WMDE) [CC0], from Wikimedia Commons" href="https://commons.wikimedia.org/wiki/File:Wikidata_stamp.png"&gt;&lt;img width="128" alt="Wikidata stamp" src="https://upload.wikimedia.org/wikipedia/commons/thumb/c/cd/Wikidata_stamp.png/512px-Wikidata_stamp.png"&gt;&lt;/a&gt;&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Sun, 17 Jun 2018 17:23:04 +0100</pubDate><guid isPermaLink="false">tag:dkane.net,2018-06-17:/2018/a-map-of-world-cup-stadia-using-wikidata/</guid><category>Code</category><category>Football</category><category>Wikidata</category></item><item><title>Grants in Kingston</title><link>https://dkane.net/2018/grants-in-kingston/</link><description>&lt;p&gt;On Tuesday I was lucky enough to take part in a Kingston "Data Day", organised by the fantastic &lt;a href="https://superhighways.org.uk/"&gt;Superhighways&lt;/a&gt; and &lt;a href="https://kva.org.uk/"&gt;Kingston Voluntary Action&lt;/a&gt;.&lt;/p&gt; &lt;p&gt;The aim of the day was to showcase some of the work that charities in Kingston are already doing to use data in their work, and see some of the tools and resources out there. We heard from charities like &lt;a href="https://www.leah.org.uk/"&gt;LEAH&lt;/a&gt;, &lt;a href="https://www.manandboy.org/"&gt;Man and Boy&lt;/a&gt; and &lt;a href="https://kingstoncarers.org.uk/young-carers/"&gt;Kingston Young Carers&lt;/a&gt; about how they were using data to measure outcomes and improve their services. We also heard from &lt;a href="https://data.kingston.gov.uk/"&gt;Kingston Borough Council&lt;/a&gt;, 360 Giving and others on resources out there that local charities could tap into.&lt;/p&gt; &lt;p&gt;I gave a quick presentation on what the 360 Giving data on grants tells us about voluntary sector funding in Kingston. Here's a few things I took away from the data:&lt;/p&gt; &lt;h2 id="1-grants-to-kingston-are-lower-than-other-areas"&gt;1. Grants to Kingston are lower than other areas&lt;/h2&gt; &lt;p&gt;I didn't actually produce any comparisons for Kingston and other London boroughs, but in Katherine from 360 Giving's presentation she showed that looking at the Grantnav 360 Giving data, Kingston comes right at the bottom of the table in terms of the number of grants made. &lt;/p&gt; &lt;p&gt;You can see this for yourself when looking at a &lt;a href="http://grantnav.threesixtygiving.org/search?page=1&amp;amp;json_query=%7B%22extra_context%22%3A+%7B%22amountAwardedFixed_facet_size%22%3A+3%2C+%22awardYear_facet_size%22%3A+3%7D%2C+%22sort%22%3A+%7B%22_score%22%3A+%7B%22order%22%3A+%22desc%22%7D%7D%2C+%22aggs%22%3A+%7B%22recipientOrganization%22%3A+%7B%22terms%22%3A+%7B%22field%22%3A+%22recipientOrganization.id_and_name%22%2C+%22size%22%3A+3%7D%7D%2C+%22fundingOrganization%22%3A+%7B%22terms%22%3A+%7B%22field%22%3A+%22fundingOrganization.id_and_name%22%2C+%22size%22%3A+3%7D%7D%2C+%22currency%22%3A+%7B%22terms%22%3A+%7B%22field%22%3A+%22currency%22%2C+%22size%22%3A+3%7D%7D%2C+%22recipientRegionName%22%3A+%7B%22terms%22%3A+%7B%22field%22%3A+%22recipientRegionName%22%2C+%22size%22%3A+3%7D%7D%2C+%22recipientDistrictName%22%3A+%7B%22terms%22%3A+%7B%22field%22%3A+%22recipientDistrictName%22%2C+%22size%22%3A+50%7D%7D%7D%2C+%22query%22%3A+%7B%22bool%22%3A+%7B%22filter%22%3A+%5B%7B%22bool%22%3A+%7B%22should%22%3A+%5B%5D%7D%7D%2C+%7B%22bool%22%3A+%7B%22should%22%3A+%5B%5D%7D%7D%2C+%7B%22bool%22%3A+%7B%22should%22%3A+%5B%5D%2C+%22must%22%3A+%7B%7D%7D%7D%2C+%7B%22bool%22%3A+%7B%22should%22%3A+%7B%22range%22%3A+%7B%22amountAwarded%22%3A+%7B%7D%7D%7D%2C+%22must%22%3A+%7B%7D%7D%7D%2C+%7B%22bool%22%3A+%7B%22should%22%3A+%5B%5D%7D%7D%2C+%7B%22bool%22%3A+%7B%22should%22%3A+%5B%5D%7D%7D%2C+%7B%22bool%22%3A+%7B%22should%22%3A+%5B%5D%7D%7D%2C+%7B%22bool%22%3A+%7B%22should%22%3A+%5B%5D%7D%7D%5D%2C+%22must%22%3A+%7B%22query_string%22%3A+%7B%22query%22%3A+%22London%22%2C+%22default_field%22%3A+%22recipientLocation%22%7D%7D%7D%7D%7D&amp;amp;page_y=684"&gt;list of all the grants to London&lt;/a&gt; (see the "Recipient District" filter on the left hand side for counts in each Borough).&lt;/p&gt; &lt;h2 id="2-the-big-lottery-fund-is-important-to-the-area"&gt;2. The Big Lottery Fund is important to the area&lt;/h2&gt; &lt;p&gt;The Big Lottery Fund is by far the largest funder that has published 360 Giving data, and this shows in Kingston. Half of the grant amount received by organisations in the borough comes from Big Lottery Fund.&lt;/p&gt; &lt;h2 id="3-but-other-funders-play-important-roles"&gt;3. But other funders play important roles&lt;/h2&gt; &lt;p&gt;Funders like the Co-operative Group, Sports England and Kingston Voluntary Action (through the &lt;a href="https://www.lovekingston.org.uk/"&gt;Love Kingston&lt;/a&gt; programme) also provide key funding, particularly to smaller groups.&lt;/p&gt; &lt;h2 id="4-grants-are-going-to-medium-sized-groups"&gt;4. Grants are going to medium-sized groups&lt;/h2&gt; &lt;p&gt;Most grants that we found are going to organisations with between £100,000 and £1m annual income. These organisations provide vital services in the local area, but there could be scope for increasing the funding available for smaller groups.&lt;/p&gt; &lt;hr&gt; &lt;p&gt;I've created a map of all the grants to Kingston that is embedded below:&lt;/p&gt; &lt;figure&gt; &lt;iframe src="/data/kingston-grants-map.html" width="100%" height="500" frameborder="0"&gt;&lt;/iframe&gt; &lt;/figure&gt; &lt;p&gt;You can also find my presentation on Google slides:&lt;/p&gt; &lt;figure&gt; &lt;iframe src="https://docs.google.com/presentation/d/e/2PACX-1vTaYI58fQlEc2pY66Y_3Y9MEUMpYUEw-cCTDHI6vYtIWhpt8hTO3dqXgjx7XY6urIzarSIrozhgVeDJ/embed?start=false&amp;loop=false&amp;delayms=3000" frameborder="0" width="100%" height="500" allowfullscreen="true" mozallowfullscreen="true" webkitallowfullscreen="true"&gt;&lt;/iframe&gt; &lt;/figure&gt; &lt;p&gt;Finally you can also download the full dataset to use yourself:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href="/data/all_kingston_grants.csv"&gt;Kingston grants data [CSV - 263Kb]&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;This data is released under the &lt;a href="https://creativecommons.org/licenses/by-sa/4.0/"&gt;Creative Commons Attribution Sharealike license (CC-BY-SA)&lt;/a&gt; in accordance with the terms under which the data was used from GrantNav.&lt;/p&gt; &lt;hr&gt; &lt;h3 id="acknowledgements-and-data-licensing"&gt;Acknowledgements and data licensing&lt;/h3&gt; &lt;p&gt;All data has been used under the license applied by the data publisher. &lt;a href="http://grantnav.threesixtygiving.org/datasets/#copyright"&gt;A list of the licenses used&lt;/a&gt; can be found on GrantNav.&lt;/p&gt; &lt;p&gt;Contains data from &lt;a href="http://grantnav.threesixtygiving.org/datasets/"&gt;GrantNav&lt;/a&gt; a &lt;a href="http://www.threesixtygiving.org/"&gt;360Giving&lt;/a&gt; application released under the terms of the &lt;a href="https://creativecommons.org/licenses/by-sa/4.0/"&gt;Creative Commons Attribution Sharealike license (CC-BY-SA)&lt;/a&gt;&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Thu, 14 Jun 2018 12:57:19 +0100</pubDate><guid isPermaLink="false">tag:dkane.net,2018-06-14:/2018/grants-in-kingston/</guid><category>Blog</category><category>Grants</category><category>Kingston</category></item><item><title>Adding charity details using findthatcharity - Part II</title><link>https://dkane.net/2018/adding-charity-details-using-findthatcharity-part-ii/</link><description>&lt;p&gt;&lt;a href="/2018/adding-charity-details-using-findthatcharityuk-part-i/"&gt;...continued from part 1&lt;/a&gt;&lt;/p&gt; &lt;h2 id="part-ii-add-charity-details"&gt;Part II - add charity details&lt;/h2&gt; &lt;p&gt;Once you've got a list of organisations in OpenRefine with charity numbers you can then add more details about the organisations using &lt;a href="https://findthatcharity.uk/"&gt;findthatcharity.uk&lt;/a&gt;. You can add:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;postcode&lt;/li&gt; &lt;li&gt;website&lt;/li&gt; &lt;li&gt;latest income&lt;/li&gt; &lt;li&gt;link to Charity Commission register&lt;/li&gt; &lt;li&gt;date registered/removed&lt;/li&gt; &lt;li&gt;Company Number&lt;/li&gt; &lt;/ul&gt; &lt;h3 id="1-fetch-json-data"&gt;1: Fetch JSON data&lt;/h3&gt; &lt;p&gt;First you fetch data from the findthatcharity page for that charity. We'll be getting the data in "JSON" format - this means we can get the fields we need out of the data.&lt;/p&gt; &lt;p&gt;To fetch the data, click the arrow menu next to the column with the charity numbers and select &lt;code&gt;Edit Column &amp;gt; Add column by fetching URLs...&lt;/code&gt;. &lt;/p&gt; &lt;figure&gt; &lt;p&gt;&lt;img alt="Screenshot of OpenRefine option for adding column based on an url" src="/images/reconcile/add-column-by-url.png"&gt;&lt;/p&gt; &lt;/figure&gt; &lt;p&gt;In the resulting dialog you need to add a formula which will create the URL that needs to be fetched. This means adding the charity number value into a template. The formula you need is:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&amp;quot;https://findthatcharity.uk/charity/&amp;quot; + value + &amp;quot;.json&amp;quot; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Make sure you include the quotation marks in the correct place. Add a new column name (something like "Charity Details"). The "Throttle delay" setting helps to lessen the load on the site being fetched from - the default value is 5000 milliseconds (5 seconds). Findthat charity can cope with more requests than that (within reason!) So I'll generally set it to 50-100 milliseconds, which works out as between 10 and 20 requests per second.&lt;/p&gt; &lt;p&gt;The resulting dialog should look something like the following. You'll see a preview of the url which should look sensible. You can test that the URL is correct by visiting one of the preview URLs. You'll also see that columns with a &lt;code&gt;null&lt;/code&gt; value also produce a &lt;code&gt;null&lt;/code&gt; URL so won't be fetched.&lt;/p&gt; &lt;figure&gt; &lt;p&gt;&lt;img alt="Preview of the Add column by fetching from URL dialog" src="/images/reconcile/add-column-dialog.png"&gt;&lt;/p&gt; &lt;/figure&gt; &lt;h3 id="2-extract-details-from-the-json-data"&gt;2. Extract details from the JSON data&lt;/h3&gt; &lt;p&gt;You should see a new column filled with JSON data - it looks very hard to read, but all the data we'll looking for. You'll need to get OpenRefine to look through this data and extract the fields you're looking for.&lt;/p&gt; &lt;p&gt;I'll demo this with the &lt;code&gt;latestincome&lt;/code&gt; and &lt;code&gt;company number&lt;/code&gt; fields, but the same principle can be used to extract anything from the data.&lt;/p&gt; &lt;p&gt;Use the arrow menu next to the new column, and then select &lt;code&gt;Edit Column &amp;gt; Add column based on this column...&lt;/code&gt; Again, we'll enter a formula to parse the JSON and find the right value within. We're using the &lt;code&gt;parseJson&lt;/code&gt; OpenRefine function. Enter the following formula:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;parseJson(value)[&amp;quot;latest_income&amp;quot;] &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Enter a name for the column and press OK to add it.&lt;/p&gt; &lt;figure&gt; &lt;p&gt;&lt;img alt="Dialog for adding a new column by parsing JSON" src="/images/reconcile/parse-json-1.png"&gt;&lt;/p&gt; &lt;/figure&gt; &lt;h4 id="company-number"&gt;Company number&lt;/h4&gt; &lt;p&gt;Adding a company number is slightly more complicated. This is because the &lt;code&gt;company_number&lt;/code&gt; field can contain multiple values (if a charity is linked with more than one company). So the formula we'll use in this instance is:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;parseJson(value)[&amp;quot;company_number&amp;quot;][0][&amp;quot;number&amp;quot;] &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;The &lt;code&gt;[0]&lt;/code&gt; part selects the first value within the list of values of &lt;code&gt;company_number&lt;/code&gt;, and then &lt;code&gt;["number"]&lt;/code&gt; selects the company number within that.&lt;/p&gt; &lt;h3 id="3-save-the-data"&gt;3. Save the data&lt;/h3&gt; &lt;p&gt;While OpenRefine is useful for this kind of data cleaning and linking, you'll want to export the data to another programme to perform analysis. You can do this using the &lt;code&gt;Export&lt;/code&gt; menu found in the top right hand corner. The default formats are pretty straightforward, and it's possible to create more sophisticated exports (including a JSON export) using the &lt;code&gt;Custom tabular exporter...&lt;/code&gt; option.&lt;/p&gt; &lt;p&gt;One thing to remember with export - it will use any active filters when exporting, so make sure you don't have restrictive filters that exclude data you actually need.&lt;/p&gt; &lt;h3 id="4-optional-using-charity-base"&gt;4. Optional - using Charity Base&lt;/h3&gt; &lt;p&gt;The data found on findthatcharity.uk is minimal as it is only included to support the site's main purpose of being a search engine. If there are other details you need then you may need to look for other data sources.&lt;/p&gt; &lt;p&gt;A good place to look is &lt;a href="https://charitybase.uk/"&gt;CharityBase&lt;/a&gt; which has more detailed data extracted from the Charity Commission.&lt;/p&gt; &lt;p&gt;A sample formula for getting a CharityBase URL for extracting data might look like:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="s"&gt;&amp;quot;https://charitybase.uk/api/v0.2.0/charities?subNumber=0&amp;amp;charityNumber=&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&amp;amp;fields=mainCharity.income,registration,beta.people.employees&amp;quot;&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;(Note that you need to specify the fields you want to return - here we're looking for the number of employees).&lt;/p&gt; &lt;p&gt;And the sample formula for parsing the resulting JSON data might look like:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;parseJson(value)[&amp;quot;charities&amp;quot;][0][&amp;quot;beta&amp;quot;][&amp;quot;people&amp;quot;][&amp;quot;employees&amp;quot;] &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Sat, 31 Mar 2018 13:25:00 +0100</pubDate><guid isPermaLink="false">tag:dkane.net,2018-03-31:/2018/adding-charity-details-using-findthatcharity-part-ii/</guid><category>Blog</category><category>Charities</category><category>findthatcharity</category></item><item><title>Adding charity details using findthatcharity.uk - Part I</title><link>https://dkane.net/2018/adding-charity-details-using-findthatcharityuk-part-i/</link><description>&lt;h2 id="part-i-add-charity-numbers"&gt;Part I - add charity numbers&lt;/h2&gt; &lt;p&gt;&lt;em&gt;&lt;a href="/2018/adding-charity-details-using-findthatcharity-part-ii/"&gt;See part II&lt;/a&gt;&lt;/em&gt;&lt;/p&gt; &lt;p&gt;&lt;a href="https://findthatcharity.uk/"&gt;findthatcharity.uk&lt;/a&gt; is a site I've created to help access data about charities. One of the ways you can use it is to add details about a list of charities if you only know their names.&lt;/p&gt; &lt;p&gt;This is often called "reconciliation" - turning a list of names into something that unambiguously identifies the charity. Charities are often known by different names, or have variations like "Ltd" rather than "Limited", so by adding charity numbers we can make sure we're talking about the same charity whether we say "Comic Relief" or "Charity Projects".&lt;/p&gt; &lt;p&gt;In this post I'm going to show you how to use &lt;a href="http://openrefine.org/"&gt;OpenRefine&lt;/a&gt; and &lt;a href="https://findthatcharity.uk/"&gt;findthatcharity.uk&lt;/a&gt; to turn charity names into numbers. In the next part, I'll show how to use charity numbers to fetch extra information about a charity - like their web address or latest income.&lt;/p&gt; &lt;h3 id="1-get-open-refine"&gt;1. Get Open Refine&lt;/h3&gt; &lt;p&gt;&lt;a href="http://openrefine.org/"&gt;OpenRefine&lt;/a&gt; is a great data cleaning/wrangling tool. It was originally started by Google (and so is sometimes still called Google Refine), but is now run by a team of volunteer developers.&lt;/p&gt; &lt;p&gt;You can find the download for your platform on the &lt;a href="http://openrefine.org/download.html"&gt;download page&lt;/a&gt;. Note that if you're a Windows user you won't "install" it in the same way you might for other programs - instead you extract the zip file into a directory and then run &lt;code&gt;refine.bat&lt;/code&gt; from that directory.&lt;/p&gt; &lt;p&gt;Again, OpenRefine might be a bit different from other programs you've seen in that it doesn't open in it's own window - instead it has a "server" that runs in the background, and you interact with it through a web browser. The address you visit (normally &lt;a href="http://localhost:3333"&gt;http://localhost:3333&lt;/a&gt;) is only accessible through your computer though - other parts of the internet can't see it.&lt;/p&gt; &lt;h3 id="2-load-your-data"&gt;2. Load your data&lt;/h3&gt; &lt;p&gt;When you first open OpenRefine you'll see a page that looks a bit like this:&lt;/p&gt; &lt;p&gt;&lt;img alt="Web page screenshot showing the OpenRefine screen for importing data" src="/images/reconcile/first-screen.png"&gt; &lt;/p&gt; &lt;p&gt;You can follow the instructions for importing your data. OpenRefine has some very flexible options for importing data. The important thing about your data is that it should be a "flat file" - a simple table with rows and columns. You might need to use another program (like Excel) to get your data in this format first.&lt;/p&gt; &lt;p&gt;I've chosen to use some data published by the &lt;a href="http://cheshirecommunityfoundation.org.uk/wp-content/uploads/2017/06/Grants-Awarded-Fiscal-Year-2016.xlsx"&gt;Cheshire Community Foundation [xlsx - 57Kb]&lt;/a&gt; on the grants they've published. This data is published to the &lt;a href="https://www.threesixtygiving.org/"&gt;360 Giving grant data standard&lt;/a&gt; and I found it in the &lt;a href="https://www.threesixtygiving.org/data/data-registry/"&gt;360 Giving data registry&lt;/a&gt;. It already includes charity numbers so we wouldn't need to do this reconciliation process, but it's useful to have them there to check how good the reconiliation process is.&lt;/p&gt; &lt;p&gt;I can point OpenRefine at this URL and it will download and load it itself. Here's what the data looks like once it's been imported. I've hidden the &lt;code&gt;description&lt;/code&gt; field to get more rows on the screen.&lt;/p&gt; &lt;p&gt;&lt;img alt="Data loaded into OpenRefine" src="/images/reconcile/data-ready.png"&gt;&lt;/p&gt; &lt;h3 id="3-reconcile-charity-names"&gt;3. Reconcile charity names&lt;/h3&gt; &lt;p&gt;We're going to be using the &lt;code&gt;Recipient Org:Name&lt;/code&gt; field to reconcile - this contains the name of the organisation that received the grant. Click on the arrow next to the column title to bring up the menu, and click &lt;code&gt;Reconcile &amp;gt; Start reconciling...&lt;/code&gt;.&lt;/p&gt; &lt;p&gt;&lt;img alt="Location of the reconcile menu" src="/images/reconcile/reconcile-menu.png"&gt;&lt;/p&gt; &lt;p&gt;Now you need to add the Find that Charity reconciliation service. This is a web service that will recieve data from OpenRefine (in this case the name of a charity) and tries to match it to an entity. It returns a list of possible matches, and allows us to extract the ID of each match (in this case the charity number).&lt;/p&gt; &lt;p&gt;Click &lt;code&gt;Add Standard Service...&lt;/code&gt; and then paste &lt;code&gt;https://findthatcharity.uk/reconcile&lt;/code&gt; in the textbox, and click &lt;code&gt;Add Service&lt;/code&gt;. This will then (possibly after a short delay) add a service called "charitysearch".&lt;/p&gt; &lt;p&gt;&lt;img alt="Dialog box for adding a reconciliation service" src="/images/reconcile/add-service.png"&gt;&lt;/p&gt; &lt;p&gt;You'll see a list with just "charity" on the left hand side (you might need to hide the sidebar). You can leave the default options (at the moment findthatcharity doesn't take any additional properties to help matching, but this might be something we add later).&lt;/p&gt; &lt;p&gt;Click &lt;code&gt;Start Reconciling&lt;/code&gt; to make the magic happen. This might take a while if you have lots of records. OpenRefine will batch up charities and send a few at a time to findthatcharity, but the server can only cope with so many requests at a time. If you need to do lots of requests (&amp;gt;5,000 in one go), speak to me directly.&lt;/p&gt; &lt;p&gt;&lt;img alt="Completed options for running reconcilation" src="/images/reconcile/start-reconciling.png"&gt;&lt;/p&gt; &lt;p&gt;You'll see a counter with the percentage complete at the top, and when finished each field will show the matches that have been made. &lt;/p&gt; &lt;p&gt;&lt;img alt="Reconciliation results, showing a series of checkboxes for each entry with possible matches" src="/images/reconcile/reconcile-results.png"&gt;&lt;/p&gt; &lt;h3 id="4-check-the-matches"&gt;4. Check the matches&lt;/h3&gt; &lt;p&gt;You now need to check the matches. Depending on your time and the number of records, there's a few options for this. &lt;/p&gt; &lt;h4 id="manually-check-every-record"&gt;Manually check every record&lt;/h4&gt; &lt;p&gt;You can manually check each record - just click on the single tick next to the correct match. You can also click the box with two ticks if you'd like every instance of this record to be matched to this entity.&lt;/p&gt; &lt;h4 id="match-all-records-to-the-best-result"&gt;Match all records to the best result&lt;/h4&gt; &lt;p&gt;If you trust the algorithm, or have done some manual checking and think they're all correct, you can just take the best match for each record. To do this, click the arrow menu next to the &lt;code&gt;Recipient Org:Name&lt;/code&gt; column heading, and select &lt;code&gt;Reconcile &amp;gt; Actions &amp;gt; Match each cell to its best candidate&lt;/code&gt;.&lt;/p&gt; &lt;h4 id="filter-and-match"&gt;Filter and match&lt;/h4&gt; &lt;p&gt;You can also combine these two approaches using filters. On the left hand side of the screen, OpenRefine has added two "facets" based on the results of the reconciliation. &lt;/p&gt; &lt;p&gt;The &lt;code&gt;judgement&lt;/code&gt; facet at the top shows the overall result - either &lt;code&gt;(blank)&lt;/code&gt; where no matches have been found, &lt;code&gt;none&lt;/code&gt; where there are some matches or &lt;code&gt;matched&lt;/code&gt; where a single match has been selected (some records are marked as &lt;code&gt;matched&lt;/code&gt; already if findthatcharity thinks they are an exact match).&lt;/p&gt; &lt;p&gt;The &lt;code&gt;best candidate's score&lt;/code&gt; facet shows the score of the best match candidate, and allows you to select different numbers by dragging the range sliders. You can also remove &lt;code&gt;Error&lt;/code&gt; values: these are the ones with no match.&lt;/p&gt; &lt;p&gt;You can drag the score fact to select only those with high matching scores - this will filter to show only those rows with the selected scores. The &lt;code&gt;Match each cell to its best candidate&lt;/code&gt; action also respects this filter, so you can use that to match all the records found by the filter, with the remainder remaining unmatched.&lt;/p&gt; &lt;p&gt;&lt;img alt="Results of combining the score filter with matching each cell to best candidate" src="/images/reconcile/filtered-matches.png"&gt;&lt;/p&gt; &lt;h3 id="5-add-charity-number"&gt;5. Add charity number&lt;/h3&gt; &lt;p&gt;Once you're happy with the matches you can add a new column with the matched charity number in. To do this, use the arrow menu next to the column heading, and click &lt;code&gt;Edit column &amp;gt; Add column based on this column...&lt;/code&gt;&lt;/p&gt; &lt;p&gt;The box that pops up allows you to specify lots of different columns. Add a name for your column at the top ("Charity Number" would make sense), and then add &lt;code&gt;cell.recon.match.id&lt;/code&gt; in the "Expression" box. This will look at the cell reconcilation object, and find the &lt;code&gt;id&lt;/code&gt; of the matched candidate.&lt;/p&gt; &lt;p&gt;&lt;img alt="Entering an expression for getting the matched cell id" src="/images/reconcile/add-charity-number.png"&gt;&lt;/p&gt; &lt;p&gt;Press &lt;code&gt;OK&lt;/code&gt; to add this column.&lt;/p&gt; &lt;hr&gt; &lt;p&gt;That's it for part one - hopefully you've now got a charity number for all the charities in your data. In &lt;a href="/2018/adding-charity-details-using-findthatcharity-part-ii/"&gt;the next part&lt;/a&gt; I'll take a look at using the charity number to add more details about the charity, like it's web address or latest income.&lt;/p&gt; &lt;p&gt;&lt;a href="/2018/adding-charity-details-using-findthatcharity-part-ii/"&gt;Part II continues here...&lt;/a&gt;&lt;/p&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Mon, 26 Mar 2018 13:25:00 +0100</pubDate><guid isPermaLink="false">tag:dkane.net,2018-03-26:/2018/adding-charity-details-using-findthatcharityuk-part-i/</guid><category>Blog</category><category>Charities</category><category>findthatcharity</category></item><item><title>Back in the GDPR</title><link>https://dkane.net/2018/back-in-the-gdpr/</link><description>&lt;p&gt;G-day (the 25 May) is fast approaching, when charities will need to be GDPR-compliant. While &lt;a href="https://www.thirdsector.co.uk/less-half-charities-heard-new-data-protection-rule/fundraising/article/1455490"&gt;half of charities apparently haven't heard of it&lt;/a&gt; there's no shortage of resources so I've put together &lt;a href="https://github.com/drkane/charity-gdpr-resources"&gt;a list of the resources I've seen produced&lt;/a&gt; on github.&lt;/p&gt; &lt;p&gt;If you see anything missing then please add an issue or make a pull request (or &lt;a href="https://twitter.com/kanedr"&gt;tweet me @kanedr&lt;/a&gt;) and I'll add it.&lt;/p&gt; &lt;p&gt;My view on GDPR is that, as a consumer, it's a welcome rebalancing of data rights from organisations to individuals, and I think organisations can embrace it positively to think better about how they handle data about their clients, staff, donors and beneficiaries.&lt;/p&gt; &lt;p&gt;With that in mind I'd highly recommend these resources by &lt;a href="https://projectsbyif.com/"&gt;IF&lt;/a&gt;. They show ways in which GDPR doesn't just have to be an annoyance that means you have to reword your forms, but instead something that can help you fundamentally rethink the way your users need you to handle their data.&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;a href="https://newdigitalrights.projectsbyif.com/"&gt;Designing for new Digital Rights&lt;/a&gt;&lt;/li&gt; &lt;/ul&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Thu, 01 Mar 2018 13:25:00 +0000</pubDate><guid isPermaLink="false">tag:dkane.net,2018-03-01:/2018/back-in-the-gdpr/</guid><category>Blog</category><category>GDPR</category><category>Data Protection</category></item><item><title>Acrostic football league tables</title><link>https://dkane.net/2018/acrostic-football-league-tables/</link><description>&lt;p&gt;Following from a &lt;a href="https://www.theguardian.com/football/2018/feb/14/footballers-joining-and-leaving-a-club-in-the-same-transfer-window-the-knowledge"&gt;question to the Guardian's "The Knowledge" column&lt;/a&gt; I've tried to find a longer acrostic than "TABLE".&lt;/p&gt; &lt;blockquote&gt; &lt;p&gt;“Has there ever been a longer acrostic spelled out in a table than the oft-recurring ‘TABLE’ from the Premier League this season – or ‘LAWNS’ from League Two?” asks Marco Jackson.&lt;/p&gt; &lt;/blockquote&gt; &lt;p&gt;To do this I'm going to fetch data on every match played in English leagues since 1994-95 and use it to generate league tables for the end of every day that a match has been played. I'll then check the first letter of every team in the league table against a dictionary of English words to find the longest.&lt;/p&gt; &lt;h2 id="import-needed-modules"&gt;Import needed modules&lt;/h2&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;sys&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;urllib.parse&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;os&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;csv&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;re&lt;/span&gt; &lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;bs4&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;BeautifulSoup&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;requests&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;pandas&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nn"&gt;pd&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;pd&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;options&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;display&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;float_format&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="si"&gt;{:,.0f}&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;h2 id="fetch-league-data"&gt;Fetch league data&lt;/h2&gt; &lt;p&gt;From &lt;a href="http://www.football-data.co.uk/englandm.php"&gt;http://www.football-data.co.uk/englandm.php&lt;/a&gt; - series of CSV tables with data on matches played in English leagues (Premier League, Division One/Championship, Division Two/League One, Division Three/League Two, Conference) since 1994/95.&lt;/p&gt; &lt;p&gt;I haven't checked the accuracy of this data against any other source. In particular, it doesn't include any data on points deductions which do happen and affect the league table.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# download the main page and extract the links to CSV files&lt;/span&gt; &lt;span class="n"&gt;data_url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;http://www.football-data.co.uk/englandm.php&amp;quot;&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data_url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;soup&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;BeautifulSoup&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;html.parser&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# go through all the links in the document&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;link&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;soup&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;find_all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;a&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;href&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;link&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;href&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# check for those that are CSV files&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;href&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;endswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;.csv&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;data/&amp;quot;&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;href&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;_&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;wb&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;fd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;data_file_url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parse&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;urljoin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;href&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;r_data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data_file_url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;stream&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# write the file to disk in the `/data` directory&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;chunk&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;r_data&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;iter_content&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chunk_size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;fd&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chunk&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Downloaded &lt;/span&gt;&lt;span class="si"&gt;{}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data_file_url&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;h2 id="fetch-dictionary-data"&gt;Fetch dictionary data&lt;/h2&gt; &lt;p&gt;From &lt;a href="https://github.com/dwyl/english-words/blob/master/words_alpha.txt"&gt;https://github.com/dwyl/english-words/blob/master/words_alpha.txt&lt;/a&gt;&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;dictionary&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;https://github.com/dwyl/english-words/blob/master/words_alpha.txt?raw=true&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;stream&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;data/dictionary.txt&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;wb&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;chunk&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;dictionary&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;iter_content&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chunk_size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;128&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chunk&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;h2 id="useful-functions"&gt;Useful functions&lt;/h2&gt; &lt;p&gt;&lt;code&gt;matches_to_table()&lt;/code&gt; takes a list of matches (as a pandas dataframe) and a list of teams and turns it into a league table. The optional list of teams ensures that any teams that haven't yet played are included in the table (if not provided it's worked out based on the matches. Tables are sorted according to the following criteria: points, then goal difference, then goals scored.&lt;/p&gt; &lt;p&gt;The matches dataframe requires the following columns for each match:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;code&gt;HomeTeam&lt;/code&gt; - team playing at home&lt;/li&gt; &lt;li&gt;&lt;code&gt;AwayTeam&lt;/code&gt; - team playing away&lt;/li&gt; &lt;li&gt;&lt;code&gt;FTR&lt;/code&gt; - full time result: "H" = Home win, "A" = Away win, "D" = Draw&lt;/li&gt; &lt;li&gt;&lt;code&gt;FTHG&lt;/code&gt; - goals scored by full time by the home team&lt;/li&gt; &lt;li&gt;&lt;code&gt;FTAG&lt;/code&gt; - goals scored by full time by the home team&lt;/li&gt; &lt;/ul&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;matches_to_table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;teams&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;points_for_win&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;points_for_draw&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;teams&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;teams&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;HomeTeam&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;unique&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;AwayTeam&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;unique&lt;/span&gt;&lt;span class="p"&gt;())))&lt;/span&gt; &lt;span class="n"&gt;fields&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;played&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;won&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;drawn&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;lost&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;for&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;against&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;gd&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;points&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;points_deduction&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pd&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DataFrame&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;teams&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;columns&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;fields&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fillna&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;won&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;won&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;FTR&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;H&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;HomeTeam&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value_counts&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;fill_value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# Home wins&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;lost&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;lost&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;FTR&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;A&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;HomeTeam&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value_counts&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;fill_value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# Home losses&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;drawn&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;drawn&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;FTR&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;D&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;HomeTeam&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value_counts&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;fill_value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# Home draws&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;won&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;won&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;FTR&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;A&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;AwayTeam&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value_counts&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;fill_value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# Away wins&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;lost&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;lost&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;FTR&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;H&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;AwayTeam&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value_counts&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;fill_value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# Away losses&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;drawn&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;drawn&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;FTR&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;D&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;AwayTeam&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value_counts&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;fill_value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# Away draws&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;for&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;for&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;groupby&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;HomeTeam&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;FTHG&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fill_value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# home for&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;for&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;for&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;groupby&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;AwayTeam&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;FTAG&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fill_value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# away for&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;against&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;against&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;groupby&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;HomeTeam&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;FTAG&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fill_value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# home against&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;against&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;against&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;matches&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;groupby&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;AwayTeam&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;FTHG&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fill_value&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# away against&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;points&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;won&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;points_for_win&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; \ &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;drawn&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;points_for_draw&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;points_deduction&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;gd&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;for&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;against&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;played&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;won&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;drawn&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;lost&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sort_values&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;points&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;gd&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;for&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;ascending&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;&lt;code&gt;table_to_string()&lt;/code&gt; takes the table output from &lt;code&gt;matches_to_table&lt;/code&gt; and turns it into a string consisting of the first letter of each of the teams in the table (in order)&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;table_to_string&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;join&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;&lt;code&gt;get_char_ngrams()&lt;/code&gt; takes a string and returns a list of ngrams found within it, with the length of the ngrams found controlled by &lt;code&gt;min_n&lt;/code&gt; and &lt;code&gt;max_n&lt;/code&gt;.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get_char_ngrams&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;input_string&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;min_n&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_n&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;ngrams&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;min_n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_n&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;ngrams&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;input_string&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;input_string&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;ngrams&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;&lt;code&gt;get_division_from_filename()&lt;/code&gt; gets the division and year based on the filename given&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get_division_from_filename&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;filename&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;filename&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;_|\.&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;filename&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;year&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;filename&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;year&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;90&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;year&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;year&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1900&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;year&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;year&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;2000&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;year&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;2004&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;division&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;E0&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Premier League&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;E1&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Division One&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;E2&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Division Two&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;E3&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Division Three&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;EC&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Conference&amp;quot;&lt;/span&gt; &lt;span class="p"&gt;}[&lt;/span&gt;&lt;span class="n"&gt;filename&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;division&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;E0&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Premier League&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;E1&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Championship&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;E2&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;League One&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;E3&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;League Two&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;EC&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Conference&amp;quot;&lt;/span&gt; &lt;span class="p"&gt;}[&lt;/span&gt;&lt;span class="n"&gt;filename&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt; &lt;span class="n"&gt;year&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;{}&lt;/span&gt;&lt;span class="s2"&gt;-&lt;/span&gt;&lt;span class="si"&gt;{}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;year&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;year&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;year&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;division&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;find_word&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;word&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;found_words&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;full&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;tuple&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;found_words&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="n"&gt;word&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;()][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;get_division_from_filename&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;],))&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tables&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reset_index&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rename&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;columns&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;index&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Team&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;full&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt; &lt;span class="n"&gt;word_location&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;join&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Team&lt;/span&gt;&lt;span class="p"&gt;)])&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;find&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;word&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;iloc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;word_location&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;word_location&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;word&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;h2 id="generate-tables"&gt;Generate tables&lt;/h2&gt; &lt;p&gt;Here I go through each file downloaded in the &lt;code&gt;/data&lt;/code&gt; directory, and carry out the following steps:&lt;/p&gt; &lt;ol&gt; &lt;li&gt;read file into a pandas dataframe&lt;/li&gt; &lt;li&gt;find unique dates on which matches were played&lt;/li&gt; &lt;li&gt;go through each date and extract the matches played by that date&lt;/li&gt; &lt;li&gt;create a table based on the matches played by that date&lt;/li&gt; &lt;li&gt;extract the string of the first letter of each team played by that date&lt;/li&gt; &lt;/ol&gt; &lt;p&gt;The strings and tables are added to arrays to make it easy to find them again.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;tstrings&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt; &lt;span class="n"&gt;files_checked&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt; &lt;span class="n"&gt;tables&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;dfile&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;listdir&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;data&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;dfile&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;files_checked&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;continue&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;dfile&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;endswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;.csv&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pd&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;read_csv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;data/&lt;/span&gt;&lt;span class="si"&gt;{}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dfile&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="c1"&gt;# step 1&lt;/span&gt; &lt;span class="c1"&gt;# print(dfile)&lt;/span&gt; &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;pd&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ParserError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dfile&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot; [SKIPPED]&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;continue&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Date&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pd&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to_datetime&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;format&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;%d&lt;/span&gt;&lt;span class="s2"&gt;/%m/%y&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dropna&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;inplace&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;how&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;all&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;teams&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;HomeTeam&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;unique&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;AwayTeam&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;unique&lt;/span&gt;&lt;span class="p"&gt;())))&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Date&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;unique&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt; &lt;span class="c1"&gt;# step 2&lt;/span&gt; &lt;span class="n"&gt;played_matches&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;df&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Date&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="c1"&gt;# step 3&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;matches_to_table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;played_matches&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;teams&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# step 4&lt;/span&gt; &lt;span class="n"&gt;tables&lt;/span&gt;&lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="n"&gt;dfile&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;])]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt; &lt;span class="n"&gt;tstring&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;table_to_string&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# step 5&lt;/span&gt; &lt;span class="n"&gt;tstrings&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;dfile&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;tstring&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="n"&gt;files_checked&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dfile&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# print(&amp;quot;{} - {} matches played by {}&amp;quot;.format(tstring, len(played_matches), i))&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# save the strings to a CSV file for later use&lt;/span&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;tablestrings.csv&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;w&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;newline&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;writer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;writer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;writer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;writerow&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;file&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;date&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;tstring&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;tstrings&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;writer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;writerow&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;{:,}&lt;/span&gt;&lt;span class="s2"&gt; files checked (each file contains matches for one year for one division)&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;files_checked&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="s1"&gt;&amp;#39;113 files checked (each file contains matches for one year for one division)&amp;#39;&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;{:,}&lt;/span&gt;&lt;span class="s2"&gt; table strings generated. Example string: &lt;/span&gt;&lt;span class="si"&gt;{}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tstrings&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;tstrings&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&amp;#39;10,268 table strings generated. Example string: CCMTLLSSDALNMABWICEM&amp;#39; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;h2 id="import-dictionary-data"&gt;Import dictionary data&lt;/h2&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;dictionary&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;data/dictionary.txt&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;r&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;dictionary&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;readlines&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="n"&gt;dictionary&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;dictionary&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;h2 id="find-long-words-in-the-table-strings"&gt;Find long words in the table strings&lt;/h2&gt; &lt;p&gt;Split each iteration of the league table into ngrams and check whether it is a word found in the dictionary.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;found_words&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tstrings&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;ngrams&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;get_char_ngrams&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;min_n&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;ngrams&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ngrams&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;found&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ngrams&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;intersection&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dictionary&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;found&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;found_words&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Iteration &lt;/span&gt;&lt;span class="si"&gt;{}&lt;/span&gt;&lt;span class="s2"&gt;, found &lt;/span&gt;&lt;span class="si"&gt;{}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found_words&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Iteration &lt;/span&gt;&lt;span class="si"&gt;{}&lt;/span&gt;&lt;span class="s2"&gt;, found &lt;/span&gt;&lt;span class="si"&gt;{}&lt;/span&gt;&lt;span class="s2"&gt; [COMPLETE]&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;found_words&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;Iteration&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;found&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="n"&gt;Iteration&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;found&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;73&lt;/span&gt; &lt;span class="n"&gt;Iteration&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;found&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;109&lt;/span&gt; &lt;span class="n"&gt;Iteration&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;found&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;146&lt;/span&gt; &lt;span class="n"&gt;Iteration&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;found&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;195&lt;/span&gt; &lt;span class="n"&gt;Iteration&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;found&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;242&lt;/span&gt; &lt;span class="n"&gt;Iteration&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;6000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;found&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;330&lt;/span&gt; &lt;span class="n"&gt;Iteration&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;7000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;found&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;358&lt;/span&gt; &lt;span class="n"&gt;Iteration&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;8000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;found&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;411&lt;/span&gt; &lt;span class="n"&gt;Iteration&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;9000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;found&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;450&lt;/span&gt; &lt;span class="n"&gt;Iteration&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;10000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;found&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;478&lt;/span&gt; &lt;span class="n"&gt;Iteration&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;10267&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;found&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;481&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;COMPLETE&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Words with six or more letters&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;get_division_from_filename&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]))&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;])]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;found_words&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[[&amp;#39;2000-01&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;2000-12-17&amp;#39;, &amp;#39;maills&amp;#39;, 6], [&amp;#39;2000-01&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;2000-12-18&amp;#39;, &amp;#39;maills&amp;#39;, 6], [&amp;#39;2000-01&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;2000-12-22&amp;#39;, &amp;#39;maills&amp;#39;, 6], [&amp;#39;2000-01&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;2001-01-30&amp;#39;, &amp;#39;maslin&amp;#39;, 6], [&amp;#39;2000-01&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;2001-04-02&amp;#39;, &amp;#39;maills&amp;#39;, 6], [&amp;#39;2000-01&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;2001-04-04&amp;#39;, &amp;#39;maills&amp;#39;, 6], [&amp;#39;2000-01&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;2001-04-10&amp;#39;, &amp;#39;maills&amp;#39;, 6], [&amp;#39;2000-01&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;2001-04-11&amp;#39;, &amp;#39;maills&amp;#39;, 6], [&amp;#39;2000-01&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;2001-04-30&amp;#39;, &amp;#39;scants&amp;#39;, 6], [&amp;#39;2000-01&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;2001-05-01&amp;#39;, &amp;#39;scants&amp;#39;, 6], [&amp;#39;2001-02&amp;#39;, &amp;#39;Division Three&amp;#39;, &amp;#39;2001-10-27&amp;#39;, &amp;#39;blokes&amp;#39;, 6], [&amp;#39;2001-02&amp;#39;, &amp;#39;Division Three&amp;#39;, &amp;#39;2001-10-28&amp;#39;, &amp;#39;blokes&amp;#39;, 6], [&amp;#39;2006-07&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;2006-08-22&amp;#39;, &amp;#39;rental&amp;#39;, 6], [&amp;#39;2006-07&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;2007-03-14&amp;#39;, &amp;#39;albert&amp;#39;, 6], [&amp;#39;2006-07&amp;#39;, &amp;#39;League One&amp;#39;, &amp;#39;2006-12-30&amp;#39;, &amp;#39;snobby&amp;#39;, 6], [&amp;#39;2008-09&amp;#39;, &amp;#39;League One&amp;#39;, &amp;#39;2009-01-27&amp;#39;, &amp;#39;smolts&amp;#39;, 6], [&amp;#39;2008-09&amp;#39;, &amp;#39;League Two&amp;#39;, &amp;#39;2009-01-03&amp;#39;, &amp;#39;begall&amp;#39;, 6], [&amp;#39;2010-11&amp;#39;, &amp;#39;League Two&amp;#39;, &amp;#39;2010-09-11&amp;#39;, &amp;#39;cobcab&amp;#39;, 6], [&amp;#39;2011-12&amp;#39;, &amp;#39;League Two&amp;#39;, &amp;#39;2011-08-20&amp;#39;, &amp;#39;monasa&amp;#39;, 6], [&amp;#39;2012-13&amp;#39;, &amp;#39;Conference&amp;#39;, &amp;#39;2013-04-09&amp;#39;, &amp;#39;chants&amp;#39;, 6], [&amp;#39;2013-14&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;2014-02-08&amp;#39;, &amp;#39;camlet&amp;#39;, 6], [&amp;#39;2013-14&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;2014-03-08&amp;#39;, &amp;#39;amtmen&amp;#39;, 6], [&amp;#39;2014-15&amp;#39;, &amp;#39;League Two&amp;#39;, &amp;#39;2015-02-24&amp;#39;, &amp;#39;peacod&amp;#39;, 6], [&amp;#39;2015-16&amp;#39;, &amp;#39;League Two&amp;#39;, &amp;#39;2015-09-12&amp;#39;, &amp;#39;cancha&amp;#39;, 6], [&amp;#39;2015-16&amp;#39;, &amp;#39;League Two&amp;#39;, &amp;#39;2015-10-21&amp;#39;, &amp;#39;blanch&amp;#39;, 6], [&amp;#39;1993-94&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;1994-05-07&amp;#39;, &amp;#39;tmesis&amp;#39;, 6], [&amp;#39;1993-94&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;1994-05-08&amp;#39;, &amp;#39;tmesis&amp;#39;, 6], [&amp;#39;1996-97&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;1997-02-15&amp;#39;, &amp;#39;wastel&amp;#39;, 6], [&amp;#39;1998-99&amp;#39;, &amp;#39;Division Three&amp;#39;, &amp;#39;1998-12-18&amp;#39;, &amp;#39;cherts&amp;#39;, 6], [&amp;#39;1999-00&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;2000-03-05&amp;#39;, &amp;#39;calesa&amp;#39;, 6]] &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Words with five letters&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;five_letters&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;get_division_from_filename&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]))&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;])]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;found_words&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;five_letters&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="c1"&gt;# five_letters&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="mf"&gt;451&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="s2"&gt;&amp;quot;, &amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;five_letters&lt;/span&gt;&lt;span class="p"&gt;])))&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;abama&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;abamp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;accel&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;alans&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;allan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;allis&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;amahs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ambos&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;amlet&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;amsel&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ancha&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ancle&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ankhs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;anlas&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;appal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;apter&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ascan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;astel&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;atlas&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;awacs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;baccy&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;baled&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;banal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;barat&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;barms&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;basan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bassa&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;beant&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;becco&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;belam&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;belch&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;betas&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bibbs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;blams&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;blanc&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;blens&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bless&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;blobs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bloke&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;blows&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bolty&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;brent&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;cabda&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;cable&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;caleb&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;camel&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;camla&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;canch&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;canis&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;canos&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;cants&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;capsa&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;casts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;catel&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;celts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;chant&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;chats&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;cheap&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;cheek&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;chert&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;chics&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;claes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;clans&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;clape&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;clast&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;cleam&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;clefs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;clomb&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;clyde&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;cobby&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;coman&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;comby&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;comdt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;compd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;copes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;copps&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;copra&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;cordy&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;corms&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;cotes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;crabs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;crost&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dampy&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dobra&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dwarf&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;egall&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;elamp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ental&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;etwas&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fable&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;facet&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fawns&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;feces&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;forbs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fpsps&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;gasts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;gecks&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ghast&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;gotra&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hants&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;herls&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hewgh&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hoers&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hyleg&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kohls&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;laban&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;laics&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lamas&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lames&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lanas&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lanch&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lansa&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lawns&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;leban&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lessn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;liwan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;llama&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;macan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;maced&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;maces&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;macle&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;madge&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;maill&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mails&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;malam&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;masts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mated&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mates&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;meach&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;melds&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;merak&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;metal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mewls&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;molts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;monas&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mosso&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;nabal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;nable&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;nambe&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;nants&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;nasab&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;nasch&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;neaps&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;nobby&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;norit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;orang&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;parma&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;plebs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;plote&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ponds&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;praam&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;prate&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proms&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pyche&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;regal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;resaw&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rests&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rotas&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rybat&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sabes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;samba&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sawed&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;scale&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;scams&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;scant&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;scase&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sceat&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;scobs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;scrob&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;segos&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sensa&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;shawn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sherd&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;slabs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;slaws&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;smash&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;smolt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;snows&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;snowy&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;stale&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;stops&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;stoss&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;swail&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;swash&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;swats&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;swing&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;swobs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;swots&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;syces&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;table&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;taces&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;taels&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tamas&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;teals&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;thaws&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tramp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;twaes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;twale&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;wants&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;wasco&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;wasnt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;waste&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;whang&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;wises&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;worms&lt;/span&gt;&lt;span class="p"&gt;&amp;#39;&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;h2 id="words-in-context"&gt;Words in context&lt;/h2&gt; &lt;p&gt;Display the league table centered around the teams making the words.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;find_word&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;blokes&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;found_words&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;(&amp;#39;2001-02&amp;#39;, &amp;#39;Division Three&amp;#39;, &amp;#39;2001-10-27&amp;#39;) &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div&gt; &lt;style&gt; .dataframe thead tr:only-child th { text-align: right; } .dataframe thead th { text-align: left; } .dataframe tbody tr th { vertical-align: top; } &lt;/style&gt; &lt;table border="1" class="dataframe"&gt; &lt;thead&gt; &lt;tr style="text-align: right;"&gt; &lt;th&gt;&lt;/th&gt; &lt;th&gt;Team&lt;/th&gt; &lt;th&gt;played&lt;/th&gt; &lt;th&gt;won&lt;/th&gt; &lt;th&gt;drawn&lt;/th&gt; &lt;th&gt;lost&lt;/th&gt; &lt;th&gt;for&lt;/th&gt; &lt;th&gt;against&lt;/th&gt; &lt;th&gt;gd&lt;/th&gt; &lt;th&gt;points&lt;/th&gt; &lt;th&gt;points_deduction&lt;/th&gt; &lt;/tr&gt; &lt;/thead&gt; &lt;tbody&gt; &lt;tr&gt; &lt;th&gt;14&lt;/th&gt; &lt;td&gt;Darlington&lt;/td&gt; &lt;td&gt;15&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;19&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;2&lt;/td&gt; &lt;td&gt;19&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;15&lt;/th&gt; &lt;td&gt;Bristol Rvs&lt;/td&gt; &lt;td&gt;14&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;15&lt;/td&gt; &lt;td&gt;-2&lt;/td&gt; &lt;td&gt;19&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;16&lt;/th&gt; &lt;td&gt;Lincoln&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;-1&lt;/td&gt; &lt;td&gt;18&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;17&lt;/th&gt; &lt;td&gt;Oxford&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;15&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;-1&lt;/td&gt; &lt;td&gt;18&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;18&lt;/th&gt; &lt;td&gt;Kidderminster&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;10&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;-7&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;19&lt;/th&gt; &lt;td&gt;Exeter&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;18&lt;/td&gt; &lt;td&gt;31&lt;/td&gt; &lt;td&gt;-13&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;20&lt;/th&gt; &lt;td&gt;Swansea&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;8&lt;/td&gt; &lt;td&gt;22&lt;/td&gt; &lt;td&gt;29&lt;/td&gt; &lt;td&gt;-7&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;21&lt;/th&gt; &lt;td&gt;Hartlepool&lt;/td&gt; &lt;td&gt;15&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;3&lt;/td&gt; &lt;td&gt;8&lt;/td&gt; &lt;td&gt;14&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;-2&lt;/td&gt; &lt;td&gt;15&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;find_word&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;chants&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;found_words&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;(&amp;#39;2012-13&amp;#39;, &amp;#39;Conference&amp;#39;, &amp;#39;2013-04-09&amp;#39;) &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div&gt; &lt;style&gt; .dataframe thead tr:only-child th { text-align: right; } .dataframe thead th { text-align: left; } .dataframe tbody tr th { vertical-align: top; } &lt;/style&gt; &lt;table border="1" class="dataframe"&gt; &lt;thead&gt; &lt;tr style="text-align: right;"&gt; &lt;th&gt;&lt;/th&gt; &lt;th&gt;Team&lt;/th&gt; &lt;th&gt;played&lt;/th&gt; &lt;th&gt;won&lt;/th&gt; &lt;th&gt;drawn&lt;/th&gt; &lt;th&gt;lost&lt;/th&gt; &lt;th&gt;for&lt;/th&gt; &lt;th&gt;against&lt;/th&gt; &lt;th&gt;gd&lt;/th&gt; &lt;th&gt;points&lt;/th&gt; &lt;th&gt;points_deduction&lt;/th&gt; &lt;/tr&gt; &lt;/thead&gt; &lt;tbody&gt; &lt;tr&gt; &lt;th&gt;12&lt;/th&gt; &lt;td&gt;Braintree Town&lt;/td&gt; &lt;td&gt;42&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;18&lt;/td&gt; &lt;td&gt;59&lt;/td&gt; &lt;td&gt;70&lt;/td&gt; &lt;td&gt;-11&lt;/td&gt; &lt;td&gt;58&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;13&lt;/th&gt; &lt;td&gt;Cambridge&lt;/td&gt; &lt;td&gt;44&lt;/td&gt; &lt;td&gt;14&lt;/td&gt; &lt;td&gt;14&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;65&lt;/td&gt; &lt;td&gt;66&lt;/td&gt; &lt;td&gt;-1&lt;/td&gt; &lt;td&gt;56&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;14&lt;/th&gt; &lt;td&gt;Hyde United&lt;/td&gt; &lt;td&gt;44&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;21&lt;/td&gt; &lt;td&gt;61&lt;/td&gt; &lt;td&gt;67&lt;/td&gt; &lt;td&gt;-6&lt;/td&gt; &lt;td&gt;55&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;15&lt;/th&gt; &lt;td&gt;Alfreton Town&lt;/td&gt; &lt;td&gt;43&lt;/td&gt; &lt;td&gt;14&lt;/td&gt; &lt;td&gt;12&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;64&lt;/td&gt; &lt;td&gt;71&lt;/td&gt; &lt;td&gt;-7&lt;/td&gt; &lt;td&gt;54&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;16&lt;/th&gt; &lt;td&gt;Nuneaton Town&lt;/td&gt; &lt;td&gt;44&lt;/td&gt; &lt;td&gt;12&lt;/td&gt; &lt;td&gt;15&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;51&lt;/td&gt; &lt;td&gt;62&lt;/td&gt; &lt;td&gt;-11&lt;/td&gt; &lt;td&gt;51&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;17&lt;/th&gt; &lt;td&gt;Tamworth&lt;/td&gt; &lt;td&gt;42&lt;/td&gt; &lt;td&gt;14&lt;/td&gt; &lt;td&gt;9&lt;/td&gt; &lt;td&gt;19&lt;/td&gt; &lt;td&gt;51&lt;/td&gt; &lt;td&gt;62&lt;/td&gt; &lt;td&gt;-11&lt;/td&gt; &lt;td&gt;51&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;18&lt;/th&gt; &lt;td&gt;Southport&lt;/td&gt; &lt;td&gt;44&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;12&lt;/td&gt; &lt;td&gt;19&lt;/td&gt; &lt;td&gt;70&lt;/td&gt; &lt;td&gt;83&lt;/td&gt; &lt;td&gt;-13&lt;/td&gt; &lt;td&gt;51&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;19&lt;/th&gt; &lt;td&gt;Gateshead&lt;/td&gt; &lt;td&gt;43&lt;/td&gt; &lt;td&gt;12&lt;/td&gt; &lt;td&gt;14&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;54&lt;/td&gt; &lt;td&gt;59&lt;/td&gt; &lt;td&gt;-5&lt;/td&gt; &lt;td&gt;50&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;find_word&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;snobby&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;found_words&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;(&amp;#39;2006-07&amp;#39;, &amp;#39;League One&amp;#39;, &amp;#39;2006-12-30&amp;#39;) &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div&gt; &lt;style&gt; .dataframe thead tr:only-child th { text-align: right; } .dataframe thead th { text-align: left; } .dataframe tbody tr th { vertical-align: top; } &lt;/style&gt; &lt;table border="1" class="dataframe"&gt; &lt;thead&gt; &lt;tr style="text-align: right;"&gt; &lt;th&gt;&lt;/th&gt; &lt;th&gt;Team&lt;/th&gt; &lt;th&gt;played&lt;/th&gt; &lt;th&gt;won&lt;/th&gt; &lt;th&gt;drawn&lt;/th&gt; &lt;th&gt;lost&lt;/th&gt; &lt;th&gt;for&lt;/th&gt; &lt;th&gt;against&lt;/th&gt; &lt;th&gt;gd&lt;/th&gt; &lt;th&gt;points&lt;/th&gt; &lt;th&gt;points_deduction&lt;/th&gt; &lt;/tr&gt; &lt;/thead&gt; &lt;tbody&gt; &lt;tr&gt; &lt;th&gt;1&lt;/th&gt; &lt;td&gt;Scunthorpe&lt;/td&gt; &lt;td&gt;25&lt;/td&gt; &lt;td&gt;14&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;38&lt;/td&gt; &lt;td&gt;20&lt;/td&gt; &lt;td&gt;18&lt;/td&gt; &lt;td&gt;48&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;2&lt;/th&gt; &lt;td&gt;Nott'm Forest&lt;/td&gt; &lt;td&gt;25&lt;/td&gt; &lt;td&gt;14&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;35&lt;/td&gt; &lt;td&gt;20&lt;/td&gt; &lt;td&gt;15&lt;/td&gt; &lt;td&gt;48&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;3&lt;/th&gt; &lt;td&gt;Oldham&lt;/td&gt; &lt;td&gt;25&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;40&lt;/td&gt; &lt;td&gt;21&lt;/td&gt; &lt;td&gt;19&lt;/td&gt; &lt;td&gt;45&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;4&lt;/th&gt; &lt;td&gt;Bristol City&lt;/td&gt; &lt;td&gt;25&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;34&lt;/td&gt; &lt;td&gt;24&lt;/td&gt; &lt;td&gt;10&lt;/td&gt; &lt;td&gt;45&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;5&lt;/th&gt; &lt;td&gt;Blackpool&lt;/td&gt; &lt;td&gt;25&lt;/td&gt; &lt;td&gt;11&lt;/td&gt; &lt;td&gt;8&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;40&lt;/td&gt; &lt;td&gt;27&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;41&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;6&lt;/th&gt; &lt;td&gt;Yeovil&lt;/td&gt; &lt;td&gt;24&lt;/td&gt; &lt;td&gt;11&lt;/td&gt; &lt;td&gt;8&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;31&lt;/td&gt; &lt;td&gt;22&lt;/td&gt; &lt;td&gt;9&lt;/td&gt; &lt;td&gt;41&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;7&lt;/th&gt; &lt;td&gt;Swansea&lt;/td&gt; &lt;td&gt;25&lt;/td&gt; &lt;td&gt;11&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;8&lt;/td&gt; &lt;td&gt;40&lt;/td&gt; &lt;td&gt;29&lt;/td&gt; &lt;td&gt;11&lt;/td&gt; &lt;td&gt;39&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;find_word&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;albert&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;found_words&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;(&amp;#39;2006-07&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;2007-03-14&amp;#39;) &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div&gt; &lt;style&gt; .dataframe thead tr:only-child th { text-align: right; } .dataframe thead th { text-align: left; } .dataframe tbody tr th { vertical-align: top; } &lt;/style&gt; &lt;table border="1" class="dataframe"&gt; &lt;thead&gt; &lt;tr style="text-align: right;"&gt; &lt;th&gt;&lt;/th&gt; &lt;th&gt;Team&lt;/th&gt; &lt;th&gt;played&lt;/th&gt; &lt;th&gt;won&lt;/th&gt; &lt;th&gt;drawn&lt;/th&gt; &lt;th&gt;lost&lt;/th&gt; &lt;th&gt;for&lt;/th&gt; &lt;th&gt;against&lt;/th&gt; &lt;th&gt;gd&lt;/th&gt; &lt;th&gt;points&lt;/th&gt; &lt;th&gt;points_deduction&lt;/th&gt; &lt;/tr&gt; &lt;/thead&gt; &lt;tbody&gt; &lt;tr&gt; &lt;th&gt;2&lt;/th&gt; &lt;td&gt;Chelsea&lt;/td&gt; &lt;td&gt;29&lt;/td&gt; &lt;td&gt;20&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;3&lt;/td&gt; &lt;td&gt;51&lt;/td&gt; &lt;td&gt;19&lt;/td&gt; &lt;td&gt;32&lt;/td&gt; &lt;td&gt;66&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;3&lt;/th&gt; &lt;td&gt;Arsenal&lt;/td&gt; &lt;td&gt;28&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;51&lt;/td&gt; &lt;td&gt;23&lt;/td&gt; &lt;td&gt;28&lt;/td&gt; &lt;td&gt;55&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;4&lt;/th&gt; &lt;td&gt;Liverpool&lt;/td&gt; &lt;td&gt;29&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;8&lt;/td&gt; &lt;td&gt;44&lt;/td&gt; &lt;td&gt;20&lt;/td&gt; &lt;td&gt;24&lt;/td&gt; &lt;td&gt;53&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;5&lt;/th&gt; &lt;td&gt;Bolton&lt;/td&gt; &lt;td&gt;29&lt;/td&gt; &lt;td&gt;14&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;10&lt;/td&gt; &lt;td&gt;34&lt;/td&gt; &lt;td&gt;34&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;td&gt;47&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;6&lt;/th&gt; &lt;td&gt;Everton&lt;/td&gt; &lt;td&gt;29&lt;/td&gt; &lt;td&gt;11&lt;/td&gt; &lt;td&gt;10&lt;/td&gt; &lt;td&gt;8&lt;/td&gt; &lt;td&gt;37&lt;/td&gt; &lt;td&gt;26&lt;/td&gt; &lt;td&gt;11&lt;/td&gt; &lt;td&gt;43&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;7&lt;/th&gt; &lt;td&gt;Reading&lt;/td&gt; &lt;td&gt;29&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;12&lt;/td&gt; &lt;td&gt;43&lt;/td&gt; &lt;td&gt;38&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;43&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;8&lt;/th&gt; &lt;td&gt;Tottenham&lt;/td&gt; &lt;td&gt;29&lt;/td&gt; &lt;td&gt;12&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;11&lt;/td&gt; &lt;td&gt;40&lt;/td&gt; &lt;td&gt;43&lt;/td&gt; &lt;td&gt;-3&lt;/td&gt; &lt;td&gt;42&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;9&lt;/th&gt; &lt;td&gt;Portsmouth&lt;/td&gt; &lt;td&gt;29&lt;/td&gt; &lt;td&gt;11&lt;/td&gt; &lt;td&gt;8&lt;/td&gt; &lt;td&gt;10&lt;/td&gt; &lt;td&gt;36&lt;/td&gt; &lt;td&gt;31&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;41&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;find_word&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;table&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;found_words&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;(&amp;#39;2017-18&amp;#39;, &amp;#39;Premier League&amp;#39;, &amp;#39;2017-12-23&amp;#39;) &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div&gt; &lt;style&gt; .dataframe thead tr:only-child th { text-align: right; } .dataframe thead th { text-align: left; } .dataframe tbody tr th { vertical-align: top; } &lt;/style&gt; &lt;table border="1" class="dataframe"&gt; &lt;thead&gt; &lt;tr style="text-align: right;"&gt; &lt;th&gt;&lt;/th&gt; &lt;th&gt;Team&lt;/th&gt; &lt;th&gt;played&lt;/th&gt; &lt;th&gt;won&lt;/th&gt; &lt;th&gt;drawn&lt;/th&gt; &lt;th&gt;lost&lt;/th&gt; &lt;th&gt;for&lt;/th&gt; &lt;th&gt;against&lt;/th&gt; &lt;th&gt;gd&lt;/th&gt; &lt;th&gt;points&lt;/th&gt; &lt;th&gt;points_deduction&lt;/th&gt; &lt;/tr&gt; &lt;/thead&gt; &lt;tbody&gt; &lt;tr&gt; &lt;th&gt;4&lt;/th&gt; &lt;td&gt;Liverpool&lt;/td&gt; &lt;td&gt;19&lt;/td&gt; &lt;td&gt;9&lt;/td&gt; &lt;td&gt;8&lt;/td&gt; &lt;td&gt;2&lt;/td&gt; &lt;td&gt;41&lt;/td&gt; &lt;td&gt;23&lt;/td&gt; &lt;td&gt;18&lt;/td&gt; &lt;td&gt;35&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;5&lt;/th&gt; &lt;td&gt;Tottenham&lt;/td&gt; &lt;td&gt;19&lt;/td&gt; &lt;td&gt;10&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;34&lt;/td&gt; &lt;td&gt;18&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;34&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;6&lt;/th&gt; &lt;td&gt;Arsenal&lt;/td&gt; &lt;td&gt;19&lt;/td&gt; &lt;td&gt;10&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;34&lt;/td&gt; &lt;td&gt;23&lt;/td&gt; &lt;td&gt;11&lt;/td&gt; &lt;td&gt;34&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;7&lt;/th&gt; &lt;td&gt;Burnley&lt;/td&gt; &lt;td&gt;19&lt;/td&gt; &lt;td&gt;9&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;15&lt;/td&gt; &lt;td&gt;1&lt;/td&gt; &lt;td&gt;32&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;8&lt;/th&gt; &lt;td&gt;Leicester&lt;/td&gt; &lt;td&gt;19&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;29&lt;/td&gt; &lt;td&gt;28&lt;/td&gt; &lt;td&gt;1&lt;/td&gt; &lt;td&gt;27&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;9&lt;/th&gt; &lt;td&gt;Everton&lt;/td&gt; &lt;td&gt;19&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;24&lt;/td&gt; &lt;td&gt;30&lt;/td&gt; &lt;td&gt;-6&lt;/td&gt; &lt;td&gt;26&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;10&lt;/th&gt; &lt;td&gt;Watford&lt;/td&gt; &lt;td&gt;19&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;9&lt;/td&gt; &lt;td&gt;27&lt;/td&gt; &lt;td&gt;34&lt;/td&gt; &lt;td&gt;-7&lt;/td&gt; &lt;td&gt;22&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;/div&gt; &lt;h2 id="teams-in-alphabetical-order"&gt;Teams in alphabetical order&lt;/h2&gt; &lt;p&gt;I also wanted to look for the table in which a run of teams were in alphabetical order. I've ignored tables in August and September when the league table is less developed and so a default alphabetical team list is likely to take place.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;alpha_tables&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;tables&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;division&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;get_division_from_filename&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="n"&gt;team_order&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tables&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;ngrams&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;get_char_ngrams&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;team_order&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;min_n&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;ngrams&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;reverse&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;alpha_tables&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;division&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;gt_seven_teams&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;alpha_tables&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;gt_nine_teams&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;alpha_tables&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="mi"&gt;9&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;gt_seven_teams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;gt_seven_teams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;gt_seven_teams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;gt_seven_teams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tables&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;gt_seven_teams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reset_index&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rename&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;columns&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;index&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Team&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="mf"&gt;2004&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;05&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;League&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;On&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;2004&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;11&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;09&lt;/span&gt; &lt;span class="err"&gt;[&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Bournemouth&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Bradford&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Brentford&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Bristol&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;City&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Chesterfield&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Hartlepool&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Huddersfield&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Swindon&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div&gt; &lt;style&gt; .dataframe thead tr:only-child th { text-align: right; } .dataframe thead th { text-align: left; } .dataframe tbody tr th { vertical-align: top; } &lt;/style&gt; &lt;table border="1" class="dataframe"&gt; &lt;thead&gt; &lt;tr style="text-align: right;"&gt; &lt;th&gt;&lt;/th&gt; &lt;th&gt;Team&lt;/th&gt; &lt;th&gt;played&lt;/th&gt; &lt;th&gt;won&lt;/th&gt; &lt;th&gt;drawn&lt;/th&gt; &lt;th&gt;lost&lt;/th&gt; &lt;th&gt;for&lt;/th&gt; &lt;th&gt;against&lt;/th&gt; &lt;th&gt;gd&lt;/th&gt; &lt;th&gt;points&lt;/th&gt; &lt;th&gt;points_deduction&lt;/th&gt; &lt;/tr&gt; &lt;/thead&gt; &lt;tbody&gt; &lt;tr&gt; &lt;th&gt;1&lt;/th&gt; &lt;td&gt;Luton&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;12&lt;/td&gt; &lt;td&gt;2&lt;/td&gt; &lt;td&gt;3&lt;/td&gt; &lt;td&gt;34&lt;/td&gt; &lt;td&gt;18&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;38&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;2&lt;/th&gt; &lt;td&gt;Hull&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;10&lt;/td&gt; &lt;td&gt;2&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;28&lt;/td&gt; &lt;td&gt;22&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;32&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;3&lt;/th&gt; &lt;td&gt;Tranmere&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;9&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;3&lt;/td&gt; &lt;td&gt;26&lt;/td&gt; &lt;td&gt;20&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;32&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;4&lt;/th&gt; &lt;td&gt;Bournemouth&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;9&lt;/td&gt; &lt;td&gt;3&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;31&lt;/td&gt; &lt;td&gt;21&lt;/td&gt; &lt;td&gt;10&lt;/td&gt; &lt;td&gt;30&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;5&lt;/th&gt; &lt;td&gt;Bradford&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;9&lt;/td&gt; &lt;td&gt;2&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;28&lt;/td&gt; &lt;td&gt;25&lt;/td&gt; &lt;td&gt;3&lt;/td&gt; &lt;td&gt;29&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;6&lt;/th&gt; &lt;td&gt;Brentford&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;8&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;21&lt;/td&gt; &lt;td&gt;22&lt;/td&gt; &lt;td&gt;-1&lt;/td&gt; &lt;td&gt;28&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;7&lt;/th&gt; &lt;td&gt;Bristol City&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;34&lt;/td&gt; &lt;td&gt;23&lt;/td&gt; &lt;td&gt;11&lt;/td&gt; &lt;td&gt;27&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;8&lt;/th&gt; &lt;td&gt;Chesterfield&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;23&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;26&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;9&lt;/th&gt; &lt;td&gt;Hartlepool&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;8&lt;/td&gt; &lt;td&gt;2&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;26&lt;/td&gt; &lt;td&gt;28&lt;/td&gt; &lt;td&gt;-2&lt;/td&gt; &lt;td&gt;26&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;10&lt;/th&gt; &lt;td&gt;Huddersfield&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;28&lt;/td&gt; &lt;td&gt;23&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;25&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;11&lt;/th&gt; &lt;td&gt;Swindon&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;28&lt;/td&gt; &lt;td&gt;25&lt;/td&gt; &lt;td&gt;3&lt;/td&gt; &lt;td&gt;25&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;12&lt;/th&gt; &lt;td&gt;Sheffield Weds&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;25&lt;/td&gt; &lt;td&gt;23&lt;/td&gt; &lt;td&gt;2&lt;/td&gt; &lt;td&gt;24&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;13&lt;/th&gt; &lt;td&gt;Port Vale&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;2&lt;/td&gt; &lt;td&gt;8&lt;/td&gt; &lt;td&gt;20&lt;/td&gt; &lt;td&gt;24&lt;/td&gt; &lt;td&gt;-4&lt;/td&gt; &lt;td&gt;23&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;14&lt;/th&gt; &lt;td&gt;Doncaster&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;21&lt;/td&gt; &lt;td&gt;26&lt;/td&gt; &lt;td&gt;-5&lt;/td&gt; &lt;td&gt;23&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;15&lt;/th&gt; &lt;td&gt;Colchester&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;26&lt;/td&gt; &lt;td&gt;22&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;22&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;16&lt;/th&gt; &lt;td&gt;Wrexham&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;18&lt;/td&gt; &lt;td&gt;23&lt;/td&gt; &lt;td&gt;-5&lt;/td&gt; &lt;td&gt;21&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;17&lt;/th&gt; &lt;td&gt;Barnsley&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;21&lt;/td&gt; &lt;td&gt;23&lt;/td&gt; &lt;td&gt;-2&lt;/td&gt; &lt;td&gt;18&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;18&lt;/th&gt; &lt;td&gt;Blackpool&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;21&lt;/td&gt; &lt;td&gt;25&lt;/td&gt; &lt;td&gt;-4&lt;/td&gt; &lt;td&gt;18&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;19&lt;/th&gt; &lt;td&gt;Walsall&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;24&lt;/td&gt; &lt;td&gt;30&lt;/td&gt; &lt;td&gt;-6&lt;/td&gt; &lt;td&gt;18&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;20&lt;/th&gt; &lt;td&gt;Peterboro&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;8&lt;/td&gt; &lt;td&gt;19&lt;/td&gt; &lt;td&gt;19&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;21&lt;/th&gt; &lt;td&gt;Oldham&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;9&lt;/td&gt; &lt;td&gt;21&lt;/td&gt; &lt;td&gt;27&lt;/td&gt; &lt;td&gt;-6&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;22&lt;/th&gt; &lt;td&gt;Torquay&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;3&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;20&lt;/td&gt; &lt;td&gt;27&lt;/td&gt; &lt;td&gt;-7&lt;/td&gt; &lt;td&gt;15&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;23&lt;/th&gt; &lt;td&gt;Milton Keynes Dons&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;3&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;9&lt;/td&gt; &lt;td&gt;20&lt;/td&gt; &lt;td&gt;31&lt;/td&gt; &lt;td&gt;-11&lt;/td&gt; &lt;td&gt;14&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;24&lt;/th&gt; &lt;td&gt;Stockport&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;2&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;11&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;35&lt;/td&gt; &lt;td&gt;-19&lt;/td&gt; &lt;td&gt;10&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;gt_seven_teams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;gt_seven_teams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;gt_seven_teams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;gt_seven_teams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tables&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;gt_seven_teams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reset_index&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rename&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;columns&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;index&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Team&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="n"&gt;oteams&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Team&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;isin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;gt_seven_teams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;iloc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;oteams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;oteams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;:]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="mf"&gt;2011&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;12&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Premier&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;League&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;2011&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;11&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;27&lt;/span&gt; &lt;span class="err"&gt;[&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Arsenal&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Aston&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Villa&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Everton&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Norwich&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;QPR&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Stoke&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Swansea&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;West&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Brom&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div&gt; &lt;style&gt; .dataframe thead tr:only-child th { text-align: right; } .dataframe thead th { text-align: left; } .dataframe tbody tr th { vertical-align: top; } &lt;/style&gt; &lt;table border="1" class="dataframe"&gt; &lt;thead&gt; &lt;tr style="text-align: right;"&gt; &lt;th&gt;&lt;/th&gt; &lt;th&gt;Team&lt;/th&gt; &lt;th&gt;played&lt;/th&gt; &lt;th&gt;won&lt;/th&gt; &lt;th&gt;drawn&lt;/th&gt; &lt;th&gt;lost&lt;/th&gt; &lt;th&gt;for&lt;/th&gt; &lt;th&gt;against&lt;/th&gt; &lt;th&gt;gd&lt;/th&gt; &lt;th&gt;points&lt;/th&gt; &lt;th&gt;points_deduction&lt;/th&gt; &lt;/tr&gt; &lt;/thead&gt; &lt;tbody&gt; &lt;tr&gt; &lt;th&gt;6&lt;/th&gt; &lt;td&gt;Liverpool&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;2&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;12&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;23&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;7&lt;/th&gt; &lt;td&gt;Arsenal&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;2&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;26&lt;/td&gt; &lt;td&gt;23&lt;/td&gt; &lt;td&gt;3&lt;/td&gt; &lt;td&gt;23&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;8&lt;/th&gt; &lt;td&gt;Aston Villa&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;3&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;3&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;-1&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;9&lt;/th&gt; &lt;td&gt;Everton&lt;/td&gt; &lt;td&gt;12&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;1&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;15&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;-1&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;10&lt;/th&gt; &lt;td&gt;Norwich&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;19&lt;/td&gt; &lt;td&gt;21&lt;/td&gt; &lt;td&gt;-2&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;11&lt;/th&gt; &lt;td&gt;QPR&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;3&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;14&lt;/td&gt; &lt;td&gt;24&lt;/td&gt; &lt;td&gt;-10&lt;/td&gt; &lt;td&gt;15&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;12&lt;/th&gt; &lt;td&gt;Stoke&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;3&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;23&lt;/td&gt; &lt;td&gt;-10&lt;/td&gt; &lt;td&gt;15&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;13&lt;/th&gt; &lt;td&gt;Swansea&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;3&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;12&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;-4&lt;/td&gt; &lt;td&gt;14&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;14&lt;/th&gt; &lt;td&gt;West Brom&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;4&lt;/td&gt; &lt;td&gt;2&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;12&lt;/td&gt; &lt;td&gt;20&lt;/td&gt; &lt;td&gt;-8&lt;/td&gt; &lt;td&gt;14&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;15&lt;/th&gt; &lt;td&gt;Fulham&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;2&lt;/td&gt; &lt;td&gt;6&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;15&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;-1&lt;/td&gt; &lt;td&gt;12&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;gt_nine_teams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;gt_nine_teams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;gt_nine_teams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;gt_nine_teams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tables&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;gt_nine_teams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reset_index&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rename&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;columns&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;index&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Team&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="n"&gt;oteams&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Team&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;isin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;gt_nine_teams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;index&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;iloc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;oteams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;oteams&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;:]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="mf"&gt;2011&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;12&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Conference&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;2012&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;03&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;31&lt;/span&gt; &lt;span class="err"&gt;[&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;York&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Southport&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Luton&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Kidderminster&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Grimsby&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Gateshead&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="kr"&gt;For&lt;/span&gt;&lt;span class="n"&gt;est&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Green&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Cambridge&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Braintree&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;To&lt;/span&gt;&lt;span class="n"&gt;wn&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="n"&gt;Barrow&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div&gt; &lt;style&gt; .dataframe thead tr:only-child th { text-align: right; } .dataframe thead th { text-align: left; } .dataframe tbody tr th { vertical-align: top; } &lt;/style&gt; &lt;table border="1" class="dataframe"&gt; &lt;thead&gt; &lt;tr style="text-align: right;"&gt; &lt;th&gt;&lt;/th&gt; &lt;th&gt;Team&lt;/th&gt; &lt;th&gt;played&lt;/th&gt; &lt;th&gt;won&lt;/th&gt; &lt;th&gt;drawn&lt;/th&gt; &lt;th&gt;lost&lt;/th&gt; &lt;th&gt;for&lt;/th&gt; &lt;th&gt;against&lt;/th&gt; &lt;th&gt;gd&lt;/th&gt; &lt;th&gt;points&lt;/th&gt; &lt;th&gt;points_deduction&lt;/th&gt; &lt;/tr&gt; &lt;/thead&gt; &lt;tbody&gt; &lt;tr&gt; &lt;th&gt;3&lt;/th&gt; &lt;td&gt;Mansfield&lt;/td&gt; &lt;td&gt;41&lt;/td&gt; &lt;td&gt;20&lt;/td&gt; &lt;td&gt;14&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;74&lt;/td&gt; &lt;td&gt;46&lt;/td&gt; &lt;td&gt;28&lt;/td&gt; &lt;td&gt;74&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;4&lt;/th&gt; &lt;td&gt;York&lt;/td&gt; &lt;td&gt;39&lt;/td&gt; &lt;td&gt;19&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;74&lt;/td&gt; &lt;td&gt;41&lt;/td&gt; &lt;td&gt;33&lt;/td&gt; &lt;td&gt;70&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;5&lt;/th&gt; &lt;td&gt;Southport&lt;/td&gt; &lt;td&gt;41&lt;/td&gt; &lt;td&gt;20&lt;/td&gt; &lt;td&gt;10&lt;/td&gt; &lt;td&gt;11&lt;/td&gt; &lt;td&gt;63&lt;/td&gt; &lt;td&gt;60&lt;/td&gt; &lt;td&gt;3&lt;/td&gt; &lt;td&gt;70&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;6&lt;/th&gt; &lt;td&gt;Luton&lt;/td&gt; &lt;td&gt;39&lt;/td&gt; &lt;td&gt;18&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;8&lt;/td&gt; &lt;td&gt;67&lt;/td&gt; &lt;td&gt;37&lt;/td&gt; &lt;td&gt;30&lt;/td&gt; &lt;td&gt;67&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;7&lt;/th&gt; &lt;td&gt;Kidderminster&lt;/td&gt; &lt;td&gt;41&lt;/td&gt; &lt;td&gt;19&lt;/td&gt; &lt;td&gt;10&lt;/td&gt; &lt;td&gt;12&lt;/td&gt; &lt;td&gt;71&lt;/td&gt; &lt;td&gt;55&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;67&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;8&lt;/th&gt; &lt;td&gt;Grimsby&lt;/td&gt; &lt;td&gt;41&lt;/td&gt; &lt;td&gt;18&lt;/td&gt; &lt;td&gt;10&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;75&lt;/td&gt; &lt;td&gt;57&lt;/td&gt; &lt;td&gt;18&lt;/td&gt; &lt;td&gt;64&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;9&lt;/th&gt; &lt;td&gt;Gateshead&lt;/td&gt; &lt;td&gt;40&lt;/td&gt; &lt;td&gt;18&lt;/td&gt; &lt;td&gt;10&lt;/td&gt; &lt;td&gt;12&lt;/td&gt; &lt;td&gt;61&lt;/td&gt; &lt;td&gt;56&lt;/td&gt; &lt;td&gt;5&lt;/td&gt; &lt;td&gt;64&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;10&lt;/th&gt; &lt;td&gt;Forest Green&lt;/td&gt; &lt;td&gt;41&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;12&lt;/td&gt; &lt;td&gt;59&lt;/td&gt; &lt;td&gt;42&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;61&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;11&lt;/th&gt; &lt;td&gt;Cambridge&lt;/td&gt; &lt;td&gt;39&lt;/td&gt; &lt;td&gt;15&lt;/td&gt; &lt;td&gt;12&lt;/td&gt; &lt;td&gt;12&lt;/td&gt; &lt;td&gt;47&lt;/td&gt; &lt;td&gt;37&lt;/td&gt; &lt;td&gt;10&lt;/td&gt; &lt;td&gt;57&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;12&lt;/th&gt; &lt;td&gt;Braintree Town&lt;/td&gt; &lt;td&gt;41&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;9&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;70&lt;/td&gt; &lt;td&gt;71&lt;/td&gt; &lt;td&gt;-1&lt;/td&gt; &lt;td&gt;57&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;13&lt;/th&gt; &lt;td&gt;Barrow&lt;/td&gt; &lt;td&gt;41&lt;/td&gt; &lt;td&gt;16&lt;/td&gt; &lt;td&gt;7&lt;/td&gt; &lt;td&gt;18&lt;/td&gt; &lt;td&gt;55&lt;/td&gt; &lt;td&gt;67&lt;/td&gt; &lt;td&gt;-12&lt;/td&gt; &lt;td&gt;55&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;14&lt;/th&gt; &lt;td&gt;Ebbsfleet&lt;/td&gt; &lt;td&gt;40&lt;/td&gt; &lt;td&gt;13&lt;/td&gt; &lt;td&gt;10&lt;/td&gt; &lt;td&gt;17&lt;/td&gt; &lt;td&gt;59&lt;/td&gt; &lt;td&gt;69&lt;/td&gt; &lt;td&gt;-10&lt;/td&gt; &lt;td&gt;49&lt;/td&gt; &lt;td&gt;0&lt;/td&gt; &lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;/div&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Tue, 20 Feb 2018 13:45:00 +0000</pubDate><guid isPermaLink="false">tag:dkane.net,2018-02-20:/2018/acrostic-football-league-tables/</guid><category>Code</category><category>Football</category></item><item><title>Names shared by genders</title><link>https://dkane.net/2018/names-shared-by-genders/</link><description>&lt;p&gt;Building the gender classifier I've &lt;a href="/a-name-gender-classifier/"&gt;written about here&lt;/a&gt; got me interested in ambiguous names - those that are shared by people of both genders.&lt;/p&gt; &lt;p&gt;I realised I could use the list of male and female charity trustee names I'd gathered to look into this in a bit more detail. Bearing in mind the limitations and bias of the source dataset, I think it can generate some insight.&lt;/p&gt; &lt;p&gt;This blogpost was written as a juypter notebook, so it can be run to recreate the research. &lt;a href="https://github.com/"&gt;You can find the notebook code on github&lt;/a&gt;.&lt;/p&gt; &lt;p&gt;We start by importing the libraries we need - two built-in python libraries (&lt;code&gt;csv&lt;/code&gt; and &lt;code&gt;collections&lt;/code&gt;), &lt;code&gt;pandas&lt;/code&gt; for analysing the data and &lt;code&gt;matplotlib&lt;/code&gt; for making charts.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="n"&gt;matplotlib&lt;/span&gt; &lt;span class="n"&gt;inline&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;csv&lt;/span&gt; &lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;collections&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Counter&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;pandas&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nn"&gt;pd&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;To start, we go through all the names in our source data, and use the names with titles to find first names of males and females. To do this I've just taken the first word from the string, excluding the title, providing it's more than one character. This is fairly crude, and will be wrong in some cases (eg someone who uses two first names). But it should be good enough for our purposes.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;names&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;female&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;male&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;extract_trustee.csv&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;reader&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reader&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;reader&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="c1"&gt;# ignore rows that aren&amp;#39;t two records long&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;continue&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot; &amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;# split the name by spaces&lt;/span&gt; &lt;span class="c1"&gt;# if there&amp;#39;s only one field then ignore the name&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;continue&lt;/span&gt; &lt;span class="c1"&gt;# if the second name string (usually surname) is less that 2 characters ignore the row&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;continue&lt;/span&gt; &lt;span class="c1"&gt;# if there are non-alpha characters (numbers, symbols) then ignore the row&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;isalpha&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt; &lt;span class="k"&gt;continue&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="c1"&gt;# get the first name (assuming name[0] is a title)&lt;/span&gt; &lt;span class="c1"&gt;# if the first word is one of these titles it&amp;#39;s a female name&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;miss&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;mrs&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;ms&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;dame&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="n"&gt;names&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;female&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="c1"&gt;# if the first word is one of these titles it&amp;#39;s a male name&lt;/span&gt; &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;mr&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;sir&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="n"&gt;names&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;male&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;Lets check the female and male names to check we're on the right track.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;names&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;female&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;names&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;male&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[&amp;#39;Felicity&amp;#39;, &amp;#39;Tessa&amp;#39;, &amp;#39;Elizabeth&amp;#39;, &amp;#39;Julie&amp;#39;, &amp;#39;Rosemary&amp;#39;, &amp;#39;Catherine&amp;#39;, &amp;#39;Eileen&amp;#39;, &amp;#39;Christa&amp;#39;, &amp;#39;Roberta&amp;#39;, &amp;#39;Beverley&amp;#39;]&lt;/span&gt; &lt;span class="k"&gt;[&amp;#39;Oliver&amp;#39;, &amp;#39;Kenneth&amp;#39;, &amp;#39;Neil&amp;#39;, &amp;#39;Keith&amp;#39;, &amp;#39;John&amp;#39;, &amp;#39;Herschel&amp;#39;, &amp;#39;Alex&amp;#39;, &amp;#39;David&amp;#39;, &amp;#39;Christopher&amp;#39;, &amp;#39;Daniel&amp;#39;]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;female&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;names&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;female&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="n"&gt;male&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;names&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;male&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="n"&gt;female_names&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;female&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;male_names&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;male&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;all_names&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;female_names&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;male_names&lt;/span&gt; &lt;span class="n"&gt;female_people&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;female&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;elements&lt;/span&gt;&lt;span class="p"&gt;()))&lt;/span&gt; &lt;span class="n"&gt;male_people&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;male&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;elements&lt;/span&gt;&lt;span class="p"&gt;()))&lt;/span&gt; &lt;span class="n"&gt;all_people&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;female_people&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;male_people&lt;/span&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Female: &lt;/span&gt;&lt;span class="si"&gt;{:,.0f}&lt;/span&gt;&lt;span class="s2"&gt; (from &lt;/span&gt;&lt;span class="si"&gt;{:,.0f}&lt;/span&gt;&lt;span class="s2"&gt; people)&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;female_names&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;female_people&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Male: &lt;/span&gt;&lt;span class="si"&gt;{:,.0f}&lt;/span&gt;&lt;span class="s2"&gt; (from &lt;/span&gt;&lt;span class="si"&gt;{:,.0f}&lt;/span&gt;&lt;span class="s2"&gt; people)&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;male_names&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;male_people&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;{:,.1f}&lt;/span&gt;&lt;span class="s2"&gt; &lt;/span&gt;&lt;span class="si"&gt;% o&lt;/span&gt;&lt;span class="s2"&gt;f names are female, &lt;/span&gt;&lt;span class="si"&gt;{:,.1f}&lt;/span&gt;&lt;span class="s2"&gt; of people in sample&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;female_names&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;all_names&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;female_people&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;all_people&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt; &lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="n"&gt;weighting&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;male_people&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mf"&gt;0.489&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;male_people&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;female_people&lt;/span&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Use a weighting of &lt;/span&gt;&lt;span class="si"&gt;{:,.3f}&lt;/span&gt;&lt;span class="s2"&gt; to bring female population to 51.1&lt;/span&gt;&lt;span class="si"&gt;% o&lt;/span&gt;&lt;span class="s2"&gt;f sample&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="n"&gt;weighting&lt;/span&gt; &lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="c1"&gt;# 100 = X + Y&lt;/span&gt; &lt;span class="c1"&gt;# X = male&lt;/span&gt; &lt;span class="c1"&gt;# Y = female&lt;/span&gt; &lt;span class="c1"&gt;# Z = female weighted&lt;/span&gt; &lt;span class="c1"&gt;# X / (X + Y) = 0.520&lt;/span&gt; &lt;span class="c1"&gt;# X / (X + Z) = 0.489&lt;/span&gt; &lt;span class="c1"&gt;# X = 0.489 * (X + Z)&lt;/span&gt; &lt;span class="c1"&gt;# X / 0.489 = X + Z&lt;/span&gt; &lt;span class="c1"&gt;# Z = (X / 0.489) - X&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;Female&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;15&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;367&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;335&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;237&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;people&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="n"&gt;Male&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;085&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;363&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;296&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;people&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="mf"&gt;48.9&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;names&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;are&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;female&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;48.0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;people&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sample&lt;/span&gt; &lt;span class="n"&gt;Use&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;weighting&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1.132&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bring&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;female&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;population&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;51.1&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sample&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;This gives us 15,367 female first names (from 335,000 people) and 16,085 male ones (from 363,296 people). The only issue is that the ratio is slightly off - 48.0% of our sample are female, compared to &lt;a href="https://www.ons.gov.uk/peoplepopulationandcommunity/populationandmigration/populationestimates/datasets/populationestimatesforukenglandandwalesscotlandandnorthernireland"&gt;51.1% of the 18+ population in England and Wales&lt;/a&gt;. So we can use a weighting of 1.132 to correct this when we use the counts.&lt;/p&gt; &lt;p&gt;Next we work out the count of how many times each name appears for men and women. To do this the data is put into panda dataframes.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;female_df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pd&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DataFrame&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;female&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;orient&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;index&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;female_df&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;columns&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;female&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="n"&gt;male_df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pd&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DataFrame&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;male&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;orient&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;index&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;male_df&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;columns&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;male&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;We then apply our weighting to the female figures to adjust for the lower number in our sample.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;female_df&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;female_df&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;weighting&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;These dataframes are merged together, joining on the name, to give us a list of first names found in the dataset alongside the number of females and males found with that name.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;both&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pd&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;concat&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;female_df&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;male_df&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;join&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;outer&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;axis&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fillna&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;both&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;total&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;both&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;female&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;both&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;male&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="n"&gt;both&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;female_pc&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;both&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;female&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;both&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;total&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="n"&gt;both&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;male_pc&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;both&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;male&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;both&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;[:,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;total&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="n"&gt;both&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sort_values&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;total&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ascending&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;inplace&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;We can see a list of the top 10 most common first names in the dataset (only two female names make the list - even after applying our weighting).&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;both&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div&gt; &lt;style&gt; .dataframe thead tr:only-child th { text-align: right; } .dataframe thead th { text-align: left; } .dataframe tbody tr th { vertical-align: top; } &lt;/style&gt; &lt;table border="1" class="dataframe"&gt; &lt;thead&gt; &lt;tr style="text-align: right;"&gt; &lt;th&gt;&lt;/th&gt; &lt;th&gt;female&lt;/th&gt; &lt;th&gt;male&lt;/th&gt; &lt;th&gt;total&lt;/th&gt; &lt;th&gt;female_pc&lt;/th&gt; &lt;th&gt;male_pc&lt;/th&gt; &lt;/tr&gt; &lt;/thead&gt; &lt;tbody&gt; &lt;tr&gt; &lt;th&gt;David&lt;/th&gt; &lt;td&gt;8.0&lt;/td&gt; &lt;td&gt;21480.0&lt;/td&gt; &lt;td&gt;21488.0&lt;/td&gt; &lt;td&gt;0.000372&lt;/td&gt; &lt;td&gt;0.999628&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;John&lt;/th&gt; &lt;td&gt;9.0&lt;/td&gt; &lt;td&gt;21155.0&lt;/td&gt; &lt;td&gt;21164.0&lt;/td&gt; &lt;td&gt;0.000425&lt;/td&gt; &lt;td&gt;0.999575&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Peter&lt;/th&gt; &lt;td&gt;6.0&lt;/td&gt; &lt;td&gt;12421.0&lt;/td&gt; &lt;td&gt;12427.0&lt;/td&gt; &lt;td&gt;0.000483&lt;/td&gt; &lt;td&gt;0.999517&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Michael&lt;/th&gt; &lt;td&gt;3.0&lt;/td&gt; &lt;td&gt;12021.0&lt;/td&gt; &lt;td&gt;12024.0&lt;/td&gt; &lt;td&gt;0.000250&lt;/td&gt; &lt;td&gt;0.999750&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Richard&lt;/th&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;9702.0&lt;/td&gt; &lt;td&gt;9703.0&lt;/td&gt; &lt;td&gt;0.000103&lt;/td&gt; &lt;td&gt;0.999897&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Susan&lt;/th&gt; &lt;td&gt;8872.0&lt;/td&gt; &lt;td&gt;3.0&lt;/td&gt; &lt;td&gt;8875.0&lt;/td&gt; &lt;td&gt;0.999662&lt;/td&gt; &lt;td&gt;0.000338&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Andrew&lt;/th&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;8746.0&lt;/td&gt; &lt;td&gt;8747.0&lt;/td&gt; &lt;td&gt;0.000114&lt;/td&gt; &lt;td&gt;0.999886&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Paul&lt;/th&gt; &lt;td&gt;3.0&lt;/td&gt; &lt;td&gt;8492.0&lt;/td&gt; &lt;td&gt;8495.0&lt;/td&gt; &lt;td&gt;0.000353&lt;/td&gt; &lt;td&gt;0.999647&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Robert&lt;/th&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;7879.0&lt;/td&gt; &lt;td&gt;7880.0&lt;/td&gt; &lt;td&gt;0.000127&lt;/td&gt; &lt;td&gt;0.999873&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Margaret&lt;/th&gt; &lt;td&gt;7866.0&lt;/td&gt; &lt;td&gt;5.0&lt;/td&gt; &lt;td&gt;7871.0&lt;/td&gt; &lt;td&gt;0.999365&lt;/td&gt; &lt;td&gt;0.000635&lt;/td&gt; &lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;/div&gt; &lt;p&gt;And a list of some of the least common - these names only appear once.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;both&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;:]&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div&gt; &lt;style&gt; .dataframe thead tr:only-child th { text-align: right; } .dataframe thead th { text-align: left; } .dataframe tbody tr th { vertical-align: top; } &lt;/style&gt; &lt;table border="1" class="dataframe"&gt; &lt;thead&gt; &lt;tr style="text-align: right;"&gt; &lt;th&gt;&lt;/th&gt; &lt;th&gt;female&lt;/th&gt; &lt;th&gt;male&lt;/th&gt; &lt;th&gt;total&lt;/th&gt; &lt;th&gt;female_pc&lt;/th&gt; &lt;th&gt;male_pc&lt;/th&gt; &lt;/tr&gt; &lt;/thead&gt; &lt;tbody&gt; &lt;tr&gt; &lt;th&gt;Jibola&lt;/th&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Jibi&lt;/th&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Jiba&lt;/th&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Jiaokun&lt;/th&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Jiann&lt;/th&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Jianmin&lt;/th&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Jiale&lt;/th&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Jia&lt;/th&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Jhumar&lt;/th&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Zyta&lt;/th&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;1.0&lt;/td&gt; &lt;td&gt;0.0&lt;/td&gt; &lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;/div&gt; &lt;h2 id="shared-names"&gt;Shared names&lt;/h2&gt; &lt;p&gt;We now move on to what we were trying to do - get a list of names that are commonly shared between people of different genders. You can see from the top 10s above that it's not a perfect dataset - some names that you might assume are unambiguously male or female have some counterparts - there are 8 female Davids, and 3 male Susans.&lt;/p&gt; &lt;p&gt;This could be a mistake in the way the algorithm was applied, a typo in the data, or people who have an unusual name. But to look only at names we would expect to be more common, I've filtered to only show only names where more than 30 instances of both male and female people have the name. This threshold should also ensure we have a decent sample of people for each name.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;shared&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;both&lt;/span&gt;&lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="n"&gt;both&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;male&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;both&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;female&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sort_values&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;female_pc&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ascending&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;shared&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="n"&gt;shared&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="mf"&gt;23&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div&gt; &lt;style&gt; .dataframe thead tr:only-child th { text-align: right; } .dataframe thead th { text-align: left; } .dataframe tbody tr th { vertical-align: top; } &lt;/style&gt; &lt;table border="1" class="dataframe"&gt; &lt;thead&gt; &lt;tr style="text-align: right;"&gt; &lt;th&gt;&lt;/th&gt; &lt;th&gt;female&lt;/th&gt; &lt;th&gt;male&lt;/th&gt; &lt;th&gt;total&lt;/th&gt; &lt;th&gt;female_pc&lt;/th&gt; &lt;th&gt;male_pc&lt;/th&gt; &lt;/tr&gt; &lt;/thead&gt; &lt;tbody&gt; &lt;tr&gt; &lt;th&gt;Jean&lt;/th&gt; &lt;td&gt;3787.0&lt;/td&gt; &lt;td&gt;50.0&lt;/td&gt; &lt;td&gt;3837.0&lt;/td&gt; &lt;td&gt;0.986969&lt;/td&gt; &lt;td&gt;0.013031&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Pat&lt;/th&gt; &lt;td&gt;1154.0&lt;/td&gt; &lt;td&gt;67.0&lt;/td&gt; &lt;td&gt;1221.0&lt;/td&gt; &lt;td&gt;0.945127&lt;/td&gt; &lt;td&gt;0.054873&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Kerry&lt;/th&gt; &lt;td&gt;530.0&lt;/td&gt; &lt;td&gt;45.0&lt;/td&gt; &lt;td&gt;575.0&lt;/td&gt; &lt;td&gt;0.921739&lt;/td&gt; &lt;td&gt;0.078261&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Lyn&lt;/th&gt; &lt;td&gt;337.0&lt;/td&gt; &lt;td&gt;36.0&lt;/td&gt; &lt;td&gt;373.0&lt;/td&gt; &lt;td&gt;0.903485&lt;/td&gt; &lt;td&gt;0.096515&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Kim&lt;/th&gt; &lt;td&gt;711.0&lt;/td&gt; &lt;td&gt;80.0&lt;/td&gt; &lt;td&gt;791.0&lt;/td&gt; &lt;td&gt;0.898862&lt;/td&gt; &lt;td&gt;0.101138&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Jan&lt;/th&gt; &lt;td&gt;802.0&lt;/td&gt; &lt;td&gt;108.0&lt;/td&gt; &lt;td&gt;910.0&lt;/td&gt; &lt;td&gt;0.881319&lt;/td&gt; &lt;td&gt;0.118681&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Lindsay&lt;/th&gt; &lt;td&gt;347.0&lt;/td&gt; &lt;td&gt;62.0&lt;/td&gt; &lt;td&gt;409.0&lt;/td&gt; &lt;td&gt;0.848411&lt;/td&gt; &lt;td&gt;0.151589&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Sandy&lt;/th&gt; &lt;td&gt;126.0&lt;/td&gt; &lt;td&gt;42.0&lt;/td&gt; &lt;td&gt;168.0&lt;/td&gt; &lt;td&gt;0.750000&lt;/td&gt; &lt;td&gt;0.250000&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Mel&lt;/th&gt; &lt;td&gt;62.0&lt;/td&gt; &lt;td&gt;54.0&lt;/td&gt; &lt;td&gt;116.0&lt;/td&gt; &lt;td&gt;0.534483&lt;/td&gt; &lt;td&gt;0.465517&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Vivian&lt;/th&gt; &lt;td&gt;87.0&lt;/td&gt; &lt;td&gt;79.0&lt;/td&gt; &lt;td&gt;166.0&lt;/td&gt; &lt;td&gt;0.524096&lt;/td&gt; &lt;td&gt;0.475904&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Leigh&lt;/th&gt; &lt;td&gt;88.0&lt;/td&gt; &lt;td&gt;92.0&lt;/td&gt; &lt;td&gt;180.0&lt;/td&gt; &lt;td&gt;0.488889&lt;/td&gt; &lt;td&gt;0.511111&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Jose&lt;/th&gt; &lt;td&gt;43.0&lt;/td&gt; &lt;td&gt;60.0&lt;/td&gt; &lt;td&gt;103.0&lt;/td&gt; &lt;td&gt;0.417476&lt;/td&gt; &lt;td&gt;0.582524&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Sam&lt;/th&gt; &lt;td&gt;215.0&lt;/td&gt; &lt;td&gt;340.0&lt;/td&gt; &lt;td&gt;555.0&lt;/td&gt; &lt;td&gt;0.387387&lt;/td&gt; &lt;td&gt;0.612613&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Laurie&lt;/th&gt; &lt;td&gt;32.0&lt;/td&gt; &lt;td&gt;71.0&lt;/td&gt; &lt;td&gt;103.0&lt;/td&gt; &lt;td&gt;0.310680&lt;/td&gt; &lt;td&gt;0.689320&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Alex&lt;/th&gt; &lt;td&gt;177.0&lt;/td&gt; &lt;td&gt;506.0&lt;/td&gt; &lt;td&gt;683.0&lt;/td&gt; &lt;td&gt;0.259151&lt;/td&gt; &lt;td&gt;0.740849&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Chris&lt;/th&gt; &lt;td&gt;419.0&lt;/td&gt; &lt;td&gt;1856.0&lt;/td&gt; &lt;td&gt;2275.0&lt;/td&gt; &lt;td&gt;0.184176&lt;/td&gt; &lt;td&gt;0.815824&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Ali&lt;/th&gt; &lt;td&gt;52.0&lt;/td&gt; &lt;td&gt;268.0&lt;/td&gt; &lt;td&gt;320.0&lt;/td&gt; &lt;td&gt;0.162500&lt;/td&gt; &lt;td&gt;0.837500&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Lee&lt;/th&gt; &lt;td&gt;92.0&lt;/td&gt; &lt;td&gt;524.0&lt;/td&gt; &lt;td&gt;616.0&lt;/td&gt; &lt;td&gt;0.149351&lt;/td&gt; &lt;td&gt;0.850649&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Ashley&lt;/th&gt; &lt;td&gt;43.0&lt;/td&gt; &lt;td&gt;257.0&lt;/td&gt; &lt;td&gt;300.0&lt;/td&gt; &lt;td&gt;0.143333&lt;/td&gt; &lt;td&gt;0.856667&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Leslie&lt;/th&gt; &lt;td&gt;78.0&lt;/td&gt; &lt;td&gt;729.0&lt;/td&gt; &lt;td&gt;807.0&lt;/td&gt; &lt;td&gt;0.096654&lt;/td&gt; &lt;td&gt;0.903346&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Francis&lt;/th&gt; &lt;td&gt;51.0&lt;/td&gt; &lt;td&gt;533.0&lt;/td&gt; &lt;td&gt;584.0&lt;/td&gt; &lt;td&gt;0.087329&lt;/td&gt; &lt;td&gt;0.912671&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Terry&lt;/th&gt; &lt;td&gt;54.0&lt;/td&gt; &lt;td&gt;973.0&lt;/td&gt; &lt;td&gt;1027.0&lt;/td&gt; &lt;td&gt;0.052580&lt;/td&gt; &lt;td&gt;0.947420&lt;/td&gt; &lt;/tr&gt; &lt;tr&gt; &lt;th&gt;Robin&lt;/th&gt; &lt;td&gt;36.0&lt;/td&gt; &lt;td&gt;1438.0&lt;/td&gt; &lt;td&gt;1474.0&lt;/td&gt; &lt;td&gt;0.024423&lt;/td&gt; &lt;td&gt;0.975577&lt;/td&gt; &lt;/tr&gt; &lt;/tbody&gt; &lt;/table&gt; &lt;/div&gt; &lt;p&gt;This identifies 23 names meeting our criteria. They range from Jean (97% female), to Robin (98% male). But the most interesting names come in the middle. The closest to a 50-50 split are Leigh (49% female), Vivian (52% female) and Mel (53% female). &lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;shared&lt;/span&gt;&lt;span class="p"&gt;[[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;female_pc&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;male_pc&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;multiply&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;iloc&lt;/span&gt;&lt;span class="p"&gt;[::&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;plot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;barh&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;stacked&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;figsize&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&amp;lt;matplotlib.axes._subplots.AxesSubplot at 0x23fe0924e80&amp;gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;&lt;img alt="png" src="/images/output_20_1.png"&gt;&lt;/p&gt; &lt;p&gt;Charting the data gives us a few clusters:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Names that are mostly female, but with some male use: &lt;strong&gt;Jean, Pat, Kerry, Lyn, Kim, Jan, Lindsay&lt;/strong&gt;&lt;/li&gt; &lt;li&gt;Names that are pretty close to 50-50: &lt;strong&gt;Mel, Vivian, Leigh, Jose, Sam&lt;/strong&gt;&lt;/li&gt; &lt;li&gt;Names that are mostly male, but with some female use: &lt;strong&gt;Alex, Chris, Ali, Lee, Ashley, Leslie, Terry, Robin&lt;/strong&gt;&lt;/li&gt; &lt;/ul&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Sat, 10 Feb 2018 21:32:35 +0100</pubDate><guid isPermaLink="false">tag:dkane.net,2018-02-10:/2018/names-shared-by-genders/</guid><category>Blog</category></item><item><title>A name gender classifier</title><link>https://dkane.net/2018/a-name-gender-classifier/</link><description>&lt;p&gt;Something I've needed to do a couple of times is take a long list of names and classify them into male and female. For example, I've looked at lists of people who attended events to see whether they were reaching more men or women - this then helps target future events.&lt;/p&gt; &lt;p&gt;&lt;em&gt;Note: there are lots of reasons why this is a terrible idea. For a start, gender is more complicated than male/female, and there are lots of reasons why a name is a bad guide to someone's gender (&lt;a href="http://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/"&gt;and other things programmers wrongly believe about names&lt;/a&gt;). But I've justified it here on the basis that I'm not interested in getting 100% of the genders exactly right, but instead to get a general idea of a split in a population.&lt;/em&gt;&lt;/p&gt; &lt;p&gt;This is a good place to use a naïve Bayesian classifier. The best example of this technique is a spam classifier - this will work out whether an email is spam, based on comparing it to a model generated from a training data set. The "naïve" part comes from the fact that it's simply based on words that appear in the training data set, without considering the meaning of or connections between those words.&lt;/p&gt; &lt;p&gt;To do this I'm going to use python (3.6 to be exact) and the &lt;a href="http://scikit-learn.org/stable/"&gt;scikit&lt;/a&gt; library which has a naïve Bayesian classifier built in. It's based on the tutorial found here.&lt;/p&gt; &lt;h2 id="step-1-get-some-training-data"&gt;Step 1 - get some training data&lt;/h2&gt; &lt;p&gt;The training data needs to be a list of names with their genders. I've found this a tricky thing to come by, but luckily my work with the &lt;a href="http://data.charitycommission.gov.uk/"&gt;Charity Commission Register of Charities&lt;/a&gt; throws up a good source.&lt;/p&gt; &lt;p&gt;The large data extract they publish, under an open license, contains a list of trustees for each charity. It's just a list of names - they hold information like address, date of birth, etc but don't publish it for data protection reasons - but it's possible to work out the gender of a proportion of the names.&lt;/p&gt; &lt;p&gt;Out of the X,000 names, X,000 (X%) include a title (Mrs, Miss, Ms, Mr). Using those four titles which are gender specific and cover X% of titles used. I wrote a python script to parse the name CSV file, look for names that start with Mr, Mrs, etc and put them into a male list or female list. This produces lists of X male names and X female names.&lt;/p&gt; &lt;p&gt;A couple of notes here. Like any training data set, this is not unbiased. Charity trustees represent a particular slice of the population - people from all walks of life are trustees, but the average trustee is older, whiter and richer than the average person. So the names we're using are going to reflect those demographics.&lt;/p&gt; &lt;p&gt;Note too that I'm not doing anything to separate out first names and last names - I'm just using the whole string. This means that last names will be included in the model - with the hope that they'll be equally distributed and so won't make a difference to the final total. But this isn't a guarantee.&lt;/p&gt; &lt;h2 id="step-2-training-the-model"&gt;Step 2: training the model&lt;/h2&gt; &lt;h2 id="step-3-testing-the-model"&gt;Step 3: testing the model&lt;/h2&gt; &lt;h2 id="step-4-using-the-model"&gt;Step 4: using the model&lt;/h2&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Sat, 10 Feb 2018 21:19:35 +0100</pubDate><guid isPermaLink="false">tag:dkane.net,2018-02-10:/2018/a-name-gender-classifier/</guid><category>Blog</category></item><item><title>Twitter bot – random charity</title><link>https://dkane.net/2016/twitter-bot-random-charity/</link><description>&lt;p&gt;I’ve been playing about with twitter bots, and following &lt;a href="https://videlais.com/2015/03/02/how-to-create-a-basic-twitterbot-in-python/"&gt;some instructions on using the twitter API in python&lt;/a&gt;, I’ve created a bot that tweets a link to a random charity every half an hour.&lt;/p&gt; &lt;p&gt;The &lt;a href="https://github.com/drkane/random-charity-bot"&gt;code and more details on the bot can be found on github&lt;/a&gt;. From the readme:&lt;/p&gt; &lt;blockquote&gt; &lt;p&gt;Twitter bot that tweets the name and website of a random charity. The data is based on the register held by the Charity Commission for England and Wales, &lt;a href="http://data.charitycommission.gov.uk/"&gt;released as open data&lt;/a&gt;.&lt;/p&gt; &lt;p&gt;The bot randomly selects an active charity from the register, and tweets it. If the website of the charity is known it tweets that too, otherwise it gives a link to the official register entry for that charity.&lt;/p&gt; &lt;p&gt;This bot isn’t affiliated with the Charity Commission, and gives no endorsement of the charities tweeted.&lt;/p&gt; &lt;/blockquote&gt; &lt;p&gt;Credit due to the following places:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Fetch charity data script adapted from &lt;a href="https://github.com/OpenDataServices/grantnav/blob/master/dataload/fetch_charity_data.py"&gt;https://github.com/OpenDataServices/grantnav/blob/master/dataload/fetch_charity_data.py&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Twitter API use from &lt;a href="https://videlais.com/2015/03/02/how-to-create-a-basic-twitterbot-in-python/"&gt;https://videlais.com/2015/03/02/how-to-create-a-basic-twitterbot-in-python/&lt;/a&gt;&lt;/li&gt; &lt;li&gt;Charity data from &lt;a href="http://data.charitycommission.gov.uk/"&gt;http://data.charitycommission.gov.uk/&lt;/a&gt; – used under Open Government Licence v3.0&lt;/li&gt; &lt;/ul&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Sun, 16 Oct 2016 10:20:00 +0100</pubDate><guid isPermaLink="false">tag:dkane.net,2016-10-16:/2016/twitter-bot-random-charity/</guid><category>Blog</category></item><item><title>Development version of WordPress on Windows 10</title><link>https://dkane.net/2016/development-version-of-wordpress-on-windows-10/</link><description>&lt;p&gt;&lt;strong&gt;The problem&lt;/strong&gt;: I need a local development version of WordPress to help test problems with the live site and also test updates. Previously I’ve used WAMP but that tends to get a bit messy, particularly on Windows 10.&lt;/p&gt; &lt;p&gt;&lt;strong&gt;The solution&lt;/strong&gt;: use docker to create a container with all the needed files that can be run when necessary. The steps described here create a copy of WordPress which allows you to access the WordPress installation files and database directly, allowing you to create an exact copy of the live site.&lt;/p&gt; &lt;h2 id="step-1-install-docker"&gt;Step 1 – install Docker&lt;/h2&gt; &lt;p&gt;Download and install the &lt;a href="https://www.docker.com/products/docker-toolbox"&gt;Docker Toolbox&lt;/a&gt;. We’ll be mostly using &lt;code&gt;docker-compose&lt;/code&gt;.&lt;/p&gt; &lt;h2 id="step-2-configure-the-container"&gt;Step 2 – configure the container&lt;/h2&gt; &lt;p&gt;This step was adapted from the &lt;a href="https://docs.docker.com/compose/wordpress/"&gt;instructions shown in the Docker WordPress docs&lt;/a&gt;. The main adaptations are to allow for access to the filesystem and direct database access.&lt;/p&gt; &lt;p&gt;Navigate to an empty directory and create a file called &lt;code&gt;docker-compose.yaml&lt;/code&gt;. Open and edit the file to include the following text:&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nt"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;2&amp;#39;&lt;/span&gt; &lt;span class="nt"&gt;services&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;db&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;mysql:5.7&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;volumes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;./.data/db:/var/lib/mysql&amp;quot;&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;restart&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;always&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;ports&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;3306:3306&amp;quot;&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;environment&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;MYSQL_ROOT_PASSWORD&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;wordpress&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;MYSQL_DATABASE&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;wordpress&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;MYSQL_USER&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;wordpress&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;MYSQL_PASSWORD&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;wordpress&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;wordpress&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;volumes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;./html:/var/www/html&amp;quot;&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;depends_on&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;db&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;wordpress:latest&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;links&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;db&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;ports&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;8000:80&amp;quot;&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;restart&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;always&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;environment&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;WORDPRESS_DB_HOST&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;db:3306&lt;/span&gt; &lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;WORDPRESS_DB_PASSWORD&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;wordpress&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;h2 id="step-3-copy-the-live-files"&gt;Step 3 – copy the live files&lt;/h2&gt; &lt;p&gt;You now need to make a copy of the files from your live site. I did this by downloading them via FTP from the server. Make sure you have a copy of all three main WordPress directorys (&lt;code&gt;wp-admin&lt;/code&gt;, &lt;code&gt;wp-content&lt;/code&gt;, &lt;code&gt;wp-includes&lt;/code&gt;) plus the &lt;code&gt;index.php&lt;/code&gt; file and any files starting &lt;code&gt;wp-*.php&lt;/code&gt; in the main directory.&lt;/p&gt; &lt;p&gt;Copy these files into a directory called &lt;code&gt;html&lt;/code&gt; in your project directory.&lt;/p&gt; &lt;h2 id="step-4-edit-wp-configphp"&gt;Step 4 – edit wp-config.php&lt;/h2&gt; &lt;p&gt;Because you’ve directly copied the files from the live site, your configuration settings won’t work. Open &lt;code&gt;wp-config.php&lt;/code&gt; from the &lt;code&gt;html&lt;/code&gt; directory in your text editor, and change the following configuration values to match the values specified in the docker-compose.yaml file.&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="x"&gt;/** The name of the database for WordPress */&lt;/span&gt; &lt;span class="x"&gt;define(&amp;#39;DB_NAME&amp;#39;, &amp;#39;wordpress&amp;#39;);&lt;/span&gt; &lt;span class="x"&gt;/** MySQL database username */&lt;/span&gt; &lt;span class="x"&gt;define(&amp;#39;DB_USER&amp;#39;, &amp;#39;wordpress&amp;#39;);&lt;/span&gt; &lt;span class="x"&gt;/** MySQL database password */&lt;/span&gt; &lt;span class="x"&gt;define(&amp;#39;DB_PASSWORD&amp;#39;, &amp;#39;wordpress&amp;#39;);&lt;/span&gt; &lt;span class="x"&gt;/** MySQL hostname */&lt;/span&gt; &lt;span class="x"&gt;define(&amp;#39;DB_HOST&amp;#39;, &amp;#39;db:3306&amp;#39;);&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;You’ll also need to add or edit variables that give the URL of your site so that it doesn’t redirect to the live one. Add the following lines (removing any existing definitions of &lt;code&gt;WP_SITEURL&lt;/code&gt; and &lt;code&gt;WP_HOME&lt;/code&gt;):&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="x"&gt;define(&amp;#39;WP_HOME&amp;#39;,&amp;#39;http://192.168.99.100:8000/&amp;#39;);&lt;/span&gt; &lt;span class="x"&gt;define(&amp;#39;WP_SITEURL&amp;#39;,&amp;#39;http://192.168.99.100:8000/&amp;#39;);&lt;/span&gt; &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;(Note: I think &lt;code&gt;192.168.99.100&lt;/code&gt; is the default IP address given by Docker to the container, but yours may be different. The &lt;code&gt;:8000&lt;/code&gt; port is configured in the &lt;code&gt;docker-compose.yaml&lt;/code&gt; file.)&lt;/p&gt; &lt;h2 id="step-5-run-the-container"&gt;Step 5 – run the container&lt;/h2&gt; &lt;p&gt;To run the container I ran the “Docker Quickstart Terminal” which has been installed to the start menu. This gives access to the Docker virtual machine command line so you can run commands.&lt;/p&gt; &lt;p&gt;Navigate to the directory where the &lt;code&gt;docker-compose.yaml&lt;/code&gt; you created above is located (using &lt;code&gt;cd xxxx&lt;/code&gt;) and then run&lt;/p&gt; &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;docker-compose up -d &lt;/code&gt;&lt;/pre&gt;&lt;/div&gt; &lt;p&gt;This will create the database and WordPress containers and start them. You can now access the installation in three ways:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;View the website (and access the admin backend) through your web browser at &lt;a href="http://192.168.99.100:8000/"&gt;http://192.168.99.100:8000/&lt;/a&gt; (nb don’t do this yet as we haven’t copied the database)&lt;/li&gt; &lt;li&gt;Edit theme and plugin files through your file system using the &lt;code&gt;html/wp-content/&lt;/code&gt; directory in your project&lt;/li&gt; &lt;li&gt;Directly access the database by connecting through host &lt;code&gt;192.168.99.100&lt;/code&gt; on port &lt;code&gt;3306&lt;/code&gt; using the user &lt;code&gt;wordpress&lt;/code&gt; with password &lt;code&gt;wordpress&lt;/code&gt;&lt;/li&gt; &lt;/ul&gt; &lt;h2 id="step-6-copy-the-database"&gt;Step 6 – copy the database&lt;/h2&gt; &lt;p&gt;The next stage is to take a copy of your live MySQL database and apply it to the local version. I did this by:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;Connect to the database using &lt;a href="http://www.heidisql.com/"&gt;HeidiSQL&lt;/a&gt; with the credentials given above&lt;/li&gt; &lt;li&gt;Dropping or emptying all the tables created when the container was started&lt;/li&gt; &lt;li&gt;Getting a sql dump of the live site as a single file (I used &lt;a href="https://wordpress.org/plugins/wordpress-backup-to-dropbox/"&gt;wpb2d&lt;/a&gt; for this)&lt;/li&gt; &lt;li&gt;Editing the SQL dump file so that the &lt;code&gt;USE database;&lt;/code&gt; statement at the top of the file was changed to &lt;code&gt;USE wordpress;&lt;/code&gt; – this makes sure the SQL is executed in the right database&lt;/li&gt; &lt;li&gt;Loading the SQL dump file in HeidiSQL and then running it.&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Depending on the size of your file this might take a little while.&lt;/p&gt; &lt;h2 id="step-7-use-the-site"&gt;Step 7 – use the site&lt;/h2&gt; &lt;p&gt;Now if you go to &lt;a href="http://192.168.99.100:8000/"&gt;http://192.168.99.100:8000/&lt;/a&gt; you should be able to access your site, and the admin backend should be available at &lt;a href="http://192.168.99.100:8000/wp-admin/"&gt;http://192.168.99.100:8000/wp-admin/&lt;/a&gt;. You should be able to log in using the same credentials you use on the live site.&lt;/p&gt; &lt;h2 id="notes"&gt;Notes&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;I found that if I made changes to the &lt;code&gt;docker-compose.yaml&lt;/code&gt; file when tweaking the setup I would then need to run the &lt;code&gt;docker-compose down&lt;/code&gt; command before re-running &lt;code&gt;docker-compose up -d&lt;/code&gt;. There may be a better way to do this (&lt;code&gt;docker-compose build&lt;/code&gt;?)&lt;/li&gt; &lt;li&gt;You may find that some URLs (particularly in pages and posts) are hardcoded to point to the live site – this can mean you switch to the live version without realising it, so be careful! You can use &lt;a href="https://interconnectit.com/products/search-and-replace-for-wordpress-databases/"&gt;something like this tool from interconnectit&lt;/a&gt; to change these values in the database.&lt;/li&gt; &lt;li&gt;Be careful about user data – any personal data held in the live site is now also held in the local site. You may want to replace or anonymise the data in the &lt;code&gt;wp_users&lt;/code&gt; table.&lt;/li&gt; &lt;li&gt;You can &lt;a href="http://blog.pavelsklenar.com/5-useful-docker-tip-and-tricks-on-windows/"&gt;add your docker IP address to your hosts file&lt;/a&gt; to make it into a nicer URL. If you do this don’t forget to change the &lt;code&gt;WP_SITEURL&lt;/code&gt; and &lt;code&gt;WP_HOME&lt;/code&gt; variables to reflect the change.&lt;/li&gt; &lt;/ul&gt;</description><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">David Kane</dc:creator><pubDate>Mon, 13 Jun 2016 11:19:35 +0100</pubDate><guid isPermaLink="false">tag:dkane.net,2016-06-13:/2016/development-version-of-wordpress-on-windows-10/</guid><category>Code</category></item></channel></rss>