CINXE.COM

<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:wfw="http://wellformedweb.org/CommentAPI/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:sy="http://purl.org/rss/1.0/modules/syndication/" xmlns:slash="http://purl.org/rss/1.0/modules/slash/" > <channel> <title>Launchpad blog</title> <atom:link href="https://blog.launchpad.net/feed" rel="self" type="application/rss+xml" /> <link>https://blog.launchpad.net</link> <description>Thoughts from the Launchpad team</description> <lastBuildDate>Fri, 01 Mar 2024 13:55:19 +0000</lastBuildDate> <language>en-US</language> <sy:updatePeriod> hourly </sy:updatePeriod> <sy:updateFrequency> 1 </sy:updateFrequency> <generator>https://wordpress.org/?v=5.9.3</generator> <item> <title>Launchpad’s new homepage</title> <link>https://blog.launchpad.net/general/launchpads-new-homepage</link> <comments>https://blog.launchpad.net/general/launchpads-new-homepage#respond</comments> <dc:creator><![CDATA[ines-almeida]]></dc:creator> <pubDate>Fri, 01 Mar 2024 13:55:19 +0000</pubDate> <category><![CDATA[General]]></category> <category><![CDATA[front-page]]></category> <category><![CDATA[launchpad]]></category> <guid isPermaLink="false">https://blog.launchpad.net/?p=4404</guid> <description><![CDATA[Launchpad’s new homepage Launchpad has been around for a while, and its frontpage has remained untouched for a few years now. If you go into launchpad.net, you’ll notice it looks quite different from what it has looked like for the past 10 years – it has been updated! The goal was to modernize it while […]]]></description> <content:encoded><![CDATA[ <h2>Launchpad’s new homepage</h2> Launchpad has been around for a while, and its frontpage has remained untouched for a few years now. If you go into launchpad.net, you’ll notice it looks quite different from what it has looked like for the past 10 years – it has been updated! The goal was to modernize it while trying to keep it looking like Launchpad. The contents have remained the same with only a few text additions, but there were a lot of styling changes. The most relevant change is that the frontpage now uses Vanilla components (<a href="https://vanillaframework.io/docs">https://vanillaframework.io/docs</a>). This alone, not only made the layout look more modern, but also made it better for a new curious user reaching the page from a mobile device. The accessibility score of the page – calculated with Google’s Lighthouse extension – increased from a 75 to an almost perfect 98! Given the frontpage is so often the first impression users get when they want to check out Launchpad, we started there. But in the future, we envision the rest of Launchpad looking more modern and having a more intuitive UX. As a final note, thank you to Peter Makowski for always giving a helping hand with frontend changes in Launchpad. If you have any feedback for us, don’t forget to reach out in any of our channels. For feature requests you can reach us as <a href="mailto:feedback@launchapd.net">feedback@launchpad.net</a> or open a report in <a href="https://bugs.launchpad.net/launchpad">https://bugs.launchpad.net/launchpad</a>. To conclude this post, here is what Launchpad looked like in 2006, yesterday and today. <figure class="wp-block-image is-resized"><img src="https://lh7-us.googleusercontent.com/0Qg10XhP6eab7Ot1JsR9Liv1q8wkAE853XhYbKx1znKP6L6IlBVeHTtHuyfFILcPs2ETtpY50yoLRNl9QShk1SCLN4k3-mRjk0thLNFvhckie8ZYnnrXaq4wOWtMNYWpk7OBYlFznvzW9XdDjLhNnTc" alt="" width="720" height="200"/><figcaption> Launchpad in 2006</figcaption></figure> <hr class="wp-block-separator"/> <figure class="wp-block-image is-resized"><img loading="lazy" src="https://lh7-us.googleusercontent.com/sP2jp-wNHRqhGlc7JDXqT4j4NnDMBZ3hzhv4enDbjt-_89KBHdCfDHmS1n9RxnFwtTC_QqLt3MAfRtyiiFKwKf7hZRSf9NC_y2nDHg7h-NXU26H_3cB5iht67jDhSU5C9JU4-zb6x3IrJyhy82a89rw" alt="" width="800" height="513"/><figcaption>Launchpad yesterday</figcaption></figure> <hr class="wp-block-separator"/> <figure class="wp-block-image is-resized"><img loading="lazy" src="https://lh7-us.googleusercontent.com/oWuZ5CGCk_EofuupnzeNnbjK2hvZa_fTBWpf7FsyqNFrRnAjRlbVyPO9QaflAJgjzSEH9vnhTMyWI7eTlhJBBPT6tkYbj0WOAcp6WctyjVzNgLauDOehMlXs3hdtuyp_ojBQu2WgyqGlom6f0W3_UhU" alt="" width="800" height="537"/><figcaption>Launchpad today</figcaption></figure> ]]></content:encoded> <wfw:commentRss>https://blog.launchpad.net/general/launchpads-new-homepage/feed</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item> <title>Launchpad-linked federated Matrix accounts</title> <link>https://blog.launchpad.net/general/launchpad-verified-federated-matrix-accounts</link> <comments>https://blog.launchpad.net/general/launchpad-verified-federated-matrix-accounts#respond</comments> <dc:creator><![CDATA[ines-almeida]]></dc:creator> <pubDate>Mon, 22 Jan 2024 16:30:05 +0000</pubDate> <category><![CDATA[General]]></category> <category><![CDATA[feature]]></category> <category><![CDATA[front-page]]></category> <category><![CDATA[launchpad]]></category> <guid isPermaLink="false">https://blog.launchpad.net/?p=4400</guid> <description><![CDATA[Users can now add their Matrix accounts to their profile in Launchpad, as requested by Canonical’s Community team. We also took the chance to slightly rework the frontend and how we display social accounts in the user profiles. Instead of having different sections in the profile for each social account , all social accounts are […]]]></description> <content:encoded><![CDATA[ Users can now add their Matrix accounts to their profile in Launchpad, as requested by Canonical’s Community team. We also took the chance to slightly rework the frontend and how we display social accounts in the user profiles. Instead of having different sections in the profile for each social account , all social accounts are now all under a “Social Accounts” section. Adding a new matrix account to your profile works similarly to how it has always worked for other accounts. Under the “Social Accounts” section in your user profile, you should now see a “No matrix accounts registered” and an edit button that will lead you to the Matrix accounts edit page. To edit, remove or add new ones, you will see an edit button in front of your newly added accounts in your profile. We also added new API endpoints <code>Person.social_accounts</code> and <code>Person.getSocialAccountsByPlatform()</code> that will list the social accounts for a user. For more information, see our <a href="https://launchpad.net/+apidoc/devel.html#social_account">API documentation</a>. Currently, only Matrix was added as a social platform. But during this process, we made it much easier for Launchpad developers to add new social platforms to Launchpad in the future. ]]></content:encoded> <wfw:commentRss>https://blog.launchpad.net/general/launchpad-verified-federated-matrix-accounts/feed</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item> <title>Self-service riscv64 builds</title> <link>https://blog.launchpad.net/ppa/self-service-riscv64-builds</link> <comments>https://blog.launchpad.net/ppa/self-service-riscv64-builds#comments</comments> <dc:creator><![CDATA[Colin Watson]]></dc:creator> <pubDate>Wed, 22 Nov 2023 14:00:43 +0000</pubDate> <category><![CDATA[PPA]]></category> <category><![CDATA[front-page]]></category> <category><![CDATA[soyuz]]></category> <guid isPermaLink="false">https://blog.launchpad.net/?p=4397</guid> <description><![CDATA[Launchpad has supported building for riscv64 for a while, since it was a requirement to get Ubuntu’s riscv64 port going. We don’t actually have riscv64 hardware in our datacentre, since we’d need server-class hardware with the hypervisor extension and that’s still in its infancy; instead, we do full-system emulation of riscv64 on beefy amd64 hardware […]]]></description> <content:encoded><![CDATA[ Launchpad has supported building for riscv64 for a while, since it was a requirement to get Ubuntu’s riscv64 port going. We don’t actually have riscv64 hardware in our datacentre, since we’d need server-class hardware with the hypervisor extension and that’s still in its infancy; instead, we do full-system emulation of riscv64 on beefy amd64 hardware using <code>qemu</code>. This has worked well enough for a while, although it isn’t exactly fast. The biggest problem with our setup wasn’t so much performance, though; it was that we were just using a bunch of manually-provisioned virtual machines, and they weren’t being reset to a clean state between builds. As a result, it would have been possible for a malicious build to compromise future builds on the same builder: it would only need a chroot or container escape. This violated our standard security model for builders, in which each build runs in an isolated ephemeral VM, and each VM is destroyed and restarted from a clean image at the end of every build. As a result, we had to limit the set of people who were allowed to have riscv64 builds on Launchpad, and we had to restrict things like snap recipes to only use very tightly-pinned parts from elsewhere on the internet (pinning is often a good idea anyway, but at an infrastructural level it isn’t something we need to require on other architectures). We’ve wanted to bring this onto the same footing as our other architectures for some time. In Canonical’s most recent product development cycle, we worked with the OpenStack team to get <a href="https://bugs.launchpad.net/bugs/2023211">riscv64 emulation support into nova</a>, and installed a backport of this on our newest internal cloud region. This almost took care of the problem. However, Launchpad builder images start out as <a href="https://cloud-images.ubuntu.com/">standard Ubuntu cloud images</a>, which on riscv64 are only available from Ubuntu 22.04 LTS onwards; in testing 22.04-based VMs on other relatively slow architectures we already knew that we were seeing some mysterious hangs in snap recipe builds. Figuring this out blocked us for some time, and involved some pretty intensive debugging of the “<code>strace</code> absolutely everything in sight and see if anything sensible falls out” variety. We eventually narrowed this down to a LXD bug and were at least able to provide a <a href="https://github.com/canonical/lxd/pull/12530">workaround</a>, at which point bringing up new builders was easy. As a result, you can now enable riscv64 builds for yourself in your PPAs or snap recipes. Visit the PPA and follow the “Change details” link, or visit the snap recipe and follow the “Edit snap package” link; you’ll see a list of checkboxes under “Processors”, and you can enable or disable any that aren’t greyed out, including riscv64. This now means that all Ubuntu architectures are fully virtualized and unrestricted in Launchpad, making it easier for developers to experiment. ]]></content:encoded> <wfw:commentRss>https://blog.launchpad.net/ppa/self-service-riscv64-builds/feed</wfw:commentRss> <slash:comments>1</slash:comments> </item> <item> <title>Introducing Project-Scoped Access Tokens</title> <link>https://blog.launchpad.net/general/introducing-project-scoped-access-tokens</link> <comments>https://blog.launchpad.net/general/introducing-project-scoped-access-tokens#respond</comments> <dc:creator><![CDATA[ines-almeida]]></dc:creator> <pubDate>Mon, 20 Nov 2023 09:31:11 +0000</pubDate> <category><![CDATA[General]]></category> <category><![CDATA[feature]]></category> <category><![CDATA[front-page]]></category> <category><![CDATA[launchpad]]></category> <guid isPermaLink="false">https://blog.launchpad.net/?p=4394</guid> <description><![CDATA[Access tokens can be used to access repositories on behalf of someone. They have scope limitations, optional expiry dates, and can be revoked at any time. They are a stricter and safer alternative to using real user authentication when needing to automate pushing and/or pulling from your git repositories. This is a concept that has […]]]></description> <content:encoded><![CDATA[ Access tokens can be used to access repositories on behalf of someone. They have scope limitations, optional expiry dates, and can be revoked at any time. They are a stricter and safer alternative to using real user authentication when needing to automate pushing and/or pulling from your git repositories. This is a concept that has existed in Launchpad for a while now. If you have the right permissions in a git repository, you might have seen a “Manage Access Tokens” button in your repository’s page in the past. These tokens can be extremely useful. But if you have multiple git repositories within a project, it can be a bit of a nuisance to create and manage access tokens for each repository. So what’s new? We’ve now introduced project-scoped access tokens. These tokens reduce the trouble for the creation and maintenance of tokens for larger projects. A project access token will work as authentication for any git repository within that project. Let’s say user A wants to run something in a remote server that requires pulling multiple git repositories from a project. User A can create a project access token, and restrict it to “repository pull” scope only. This token will then be valid authentication to pull from any repository within that project. And user A will be able to revoke that token once it’s no longer needed, keeping their real user authentication safe. The same token will be invalid for pushing, or for accessing repositories within other projects. Also note that this is used for ‘authentication’, not ‘authorization’ – if the user doesn’t have access to a given git repository, their access token will not grant them permissions. Anyone with permissions to edit a project will be able to create an access token, either through the UI or the API, using the same method as to create access tokens for git repositories. See <a href="https://help.launchpad.net/Code/Git#Generating_access_tokens">Generating Access Tokens</a> section in our documentation for instructions and other information. This feature was implemented on request by our colleagues from the ROS team. We would love to get some feedback whether this also covers your use case. Please <a href="https://help.launchpad.net/Feedback">let us know</a>. ]]></content:encoded> <wfw:commentRss>https://blog.launchpad.net/general/introducing-project-scoped-access-tokens/feed</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item> <title>New domain names for PPAs</title> <link>https://blog.launchpad.net/ppa/new-domain-names-for-ppas</link> <comments>https://blog.launchpad.net/ppa/new-domain-names-for-ppas#comments</comments> <dc:creator><![CDATA[Colin Watson]]></dc:creator> <pubDate>Wed, 16 Feb 2022 18:08:27 +0000</pubDate> <category><![CDATA[PPA]]></category> <category><![CDATA[announcement]]></category> <category><![CDATA[front-page]]></category> <guid isPermaLink="false">https://blog.launchpad.net/?p=4375</guid> <description><![CDATA[Since they were introduced in 2007, Launchpad’s Personal Package Archives (PPAs) have always been hosted on ppa.launchpad.net. This has generally worked well, but one significant snag became clear later on: it was difficult to add HTTPS support for PPAs due to the way that cookies work on the web. Launchpad uses a cookie for your […]]]></description> <content:encoded><![CDATA[ Since they were introduced in 2007, Launchpad’s Personal Package Archives (PPAs) have always been hosted on ppa.launchpad.net. This has generally worked well, but one significant snag became clear later on: it was difficult to add <a href="https://bugs.launchpad.net/launchpad/+bug/1473091" data-type="URL" data-id="https://bugs.launchpad.net/launchpad/+bug/1473091">HTTPS support for PPAs</a> due to the way that cookies work on the web. Launchpad uses a cookie for your login session, which is of course security-critical, and because we use multiple domain names for the main web application (bugs.launchpad.net, code.launchpad.net, and so on), the session cookie domain has to be set to allow subdomains of launchpad.net. We set the “Secure” flag on session cookies to ensure that browsers only ever send them over HTTPS, as well as the “HttpOnly” flag to prevent direct access to it from JavaScript; but there are still ways in which arbitrary JS on an HTTPS subdomain of launchpad.net might be able to exfiltrate or abuse users’ session cookies. As a result, we can never allow any HTTPS subdomain of launchpad.net to publish completely user-generated HTML that we don’t process first. We don’t currently know of a way to get ppa.launchpad.net to serve arbitrary HTML as <code>Content-Type: text/html</code>, but this is quite a brittle protection as there are certainly ways (used for things like installer uploads) to upload arbitrary files to ppa.launchpad.net under a user-controlled directory structure, and we don’t want the webapp’s security to depend on nobody figuring out how to convince a browser to interpret any of that as arbitrary HTML. The librarian is already on a separate launchpadlibrarian.net domain name for a similar reason. To resolve this dilemma, we’ve added a new ppa.launchpadcontent.net domain name which supports both HTTP and HTTPS (and similarly private-ppa.launchpadcontent.net for private PPAs, which as before is HTTPS-only). <code>add-apt-repository</code> in Ubuntu 22.04 will <a href="https://bugs.launchpad.net/ubuntu/+source/software-properties/+bug/1959015">use the new domain name by default</a>. The old names will carry on working indefinitely – we know they’re embedded in lots of configuration and scripts, and we have no inclination to break all of those – but we recommend moving to the new names where possible. ppa.launchpad.net will remain HTTP-only. Some systems may need to be updated to support the new domain name, particularly things like HTTP(S) proxy configuration files and <code>no_proxy</code> environment variables. ]]></content:encoded> <wfw:commentRss>https://blog.launchpad.net/ppa/new-domain-names-for-ppas/feed</wfw:commentRss> <slash:comments>5</slash:comments> </item> <item> <title>Comment editing is now possible</title> <link>https://blog.launchpad.net/general/comment-editing-is-now-possible</link> <comments>https://blog.launchpad.net/general/comment-editing-is-now-possible#comments</comments> <dc:creator><![CDATA[Thiago F Pappacena]]></dc:creator> <pubDate>Fri, 28 May 2021 18:26:30 +0000</pubDate> <category><![CDATA[General]]></category> <guid isPermaLink="false">https://blog.launchpad.net/?p=4366</guid> <description><![CDATA[It took a while, but now Launchpad finally allows users to edit their comments on questions, bug reports and merge proposal pages. The first request for this feature dates back from 2007. Since then, Launchpad increased a lot in terms of new features, and the other priorities took precedence over that request, but the request […]]]></description> <content:encoded><![CDATA[ It took a while, but now Launchpad finally allows users to edit their comments on questions, bug reports and merge proposal pages. The first request for this feature dates back from <a href="https://bugs.launchpad.net/launchpad/+bug/80895">2007</a>. Since then, Launchpad increased a lot in terms of new features, and the other priorities took precedence over that request, but the request was still more than valid. More recently, we managed to bump the priority of this feature, and now we have it: users are now allowed to edit their comments on Launchpad answers, bugs and merge proposals! This has been <a href="https://launchpad.net/+apidoc/devel.html#message">available in the API for a few days already</a>, but today we finally released the fresh new pencil icon in the top-right corner of your messages. Once you click it, the message is turned into a small form that allows you to edit your message content. <figure class="wp-block-image"><img src="https://lh4.googleusercontent.com/Vm7o5SK2Py-rT6djGvKbj_tRNsgySZs6EiBTdoAg0FvMD58NUQdwdOBO7jobQ7iJsZteKCHNp_1zGJe4WjNb8GXosP0N9HBr6QYx14YRM3H5k-ZiZTpIbqiOUPFiBAM2FgA701Cs" alt=""/></figure> For messages that were edited before, it is possible to see old versions of that edited message by clicking the “last edit …” link, also at the top of the message. <figure class="wp-block-image"><img src="https://lh4.googleusercontent.com/RJvzA0Z9vNbpAsuBVhjnzSEbMZCy6WzDDC4r7akd3sqVOqf-mYdutZXLzpB-_HoDhNd1b2iSqpKRZeAdtPxQwhrVWRDxQpowYprTBa9xcIUZvfX3UTf2dw8bS4GYoGpaR0LvDiwI" alt=""/></figure> In case you introduce sensitive information by mistake in your comment and need to remove it from the message history after editing it, you can always use the API to do so. We plan to add a remove button to the message’s revision history UI soon, to make this work easier. The Launchpad team is proud of this new feature, and we hope that it will be useful for everyone! Let us know if you have any <a href="https://help.launchpad.net/Feedback">feedback</a>! ]]></content:encoded> <wfw:commentRss>https://blog.launchpad.net/general/comment-editing-is-now-possible/feed</wfw:commentRss> <slash:comments>2</slash:comments> </item> <item> <title>Git Protocol v2 Available at Launchpad</title> <link>https://blog.launchpad.net/code/git-protocol-v2-available-at-launchpad</link> <comments>https://blog.launchpad.net/code/git-protocol-v2-available-at-launchpad#respond</comments> <dc:creator><![CDATA[Thiago F Pappacena]]></dc:creator> <pubDate>Mon, 28 Sep 2020 13:26:21 +0000</pubDate> <category><![CDATA[Code]]></category> <category><![CDATA[Performance]]></category> <category><![CDATA[feature]]></category> <category><![CDATA[git]]></category> <guid isPermaLink="false">https://blog.launchpad.net/?p=4351</guid> <description><![CDATA[After a few weeks of development and testing, we are proud to finally announce that Git protocol v2 is available at Launchpad! But what are the improvements in the protocol itself, and how can you benefit from that? The git v2 protocol was released a while ago, in May 2018, with the intent of simplifying […]]]></description> <content:encoded><![CDATA[ After a few weeks of development and testing, we are proud to finally announce that Git protocol v2 is available at Launchpad! But what are the improvements in the protocol itself, and how can you benefit from that? The git v2 protocol was released a while ago, in <a href="https://opensource.googleblog.com/2018/05/introducing-git-protocol-version-2.html">May 2018</a>, with the intent of simplifying git over HTTP transfer protocol, allowing extensibility of git capabilities, and reducing the network usage in some operations. For the end user, the main clear benefit is the bandwidth reduction: in the previous version of the protocol, when one does a “git pull origin master”, for example, even if you have no new commits to fetch from the remote origin, git server would first “advertise” to the client all refs (branches and tags) available. In big repositories with hundreds or thousands of refs, this simple handshake operation could consume a lot of bandwidth and time to communicate a bunch of data that would potentially be discarded by the client after. In the v2 protocol, this waste is no longer present: the client now has the ability to filter which refs it wants to know about before the server starts advertising it. The v2 protocol is not the default on git clients yet, but if you are using a git version higher than 2.19, you can use v2: simply run <code>git config --global protocol.version 2</code>, and you will be using the most recent protocol version when communicating with servers that support this version. Including Launchpad, of course. And even if you have repositories hosted in a server that is not yet compatible with v2, don’t worry: the git client is backward compatible. If the server does not support v2, git client should fall back gracefully to the previous version and everything should continue to work as expected. We hope you enjoy the new feature. And let us know if you have any <a href="https://help.launchpad.net/Feedback">feedback</a>! ]]></content:encoded> <wfw:commentRss>https://blog.launchpad.net/code/git-protocol-v2-available-at-launchpad/feed</wfw:commentRss> <slash:comments>0</slash:comments> </item> <item> <title>Login regression for users with non-ASCII names</title> <link>https://blog.launchpad.net/general/login-regression-for-users-with-non-ascii-names</link> <comments>https://blog.launchpad.net/general/login-regression-for-users-with-non-ascii-names#comments</comments> <dc:creator><![CDATA[Colin Watson]]></dc:creator> <pubDate>Thu, 20 Aug 2020 10:01:59 +0000</pubDate> <category><![CDATA[General]]></category> <category><![CDATA[bug-fixes]]></category> <guid isPermaLink="false">https://blog.launchpad.net/?p=4339</guid> <description><![CDATA[On 2020-08-13, we deployed an update that caused users whose full names contain non-ASCII characters (which is of course very common) to be unable to log into Launchpad. We heard about this serious regression from users on 2020-08-17, and rolled out a fix on 2020-08-18. We’re sorry about this; it doesn’t meet the standards of […]]]></description> <content:encoded><![CDATA[ On 2020-08-13, we deployed an update that caused users whose full names contain non-ASCII characters (which is of course <a href="https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/">very common</a>) to be unable to log into Launchpad. We heard about this serious regression from users on 2020-08-17, and rolled out a fix on 2020-08-18. We’re sorry about this; it doesn’t meet the standards of both inclusion and quality that we set for ourselves. This post aims to explain what happened, technical details of why it happened, and the steps we’ve taken to avoid it happening again. Launchpad still runs on Python 2. This is <a href="https://www.python.org/doc/sunset-python-2/">a problem</a>, and we’ve been gradually chipping away at it for the last couple of years. With about three-quarters of a million lines of Python code in the main tree and over 200 dependencies, it’s a big job – but we’re well underway! Some of those dependencies have been difficult problems in their own right. The one at issue here was <a href="https://pypi.org/project/python-openid/">python-openid</a>, which we use as part of our login workflow, but which hasn’t been actively maintained for over ten years. Fortunately, in this case we didn’t have to port it ourselves, because there were already a couple of forks featuring Python 3 support while preserving more or less the same interface: we chose <a href="https://pypi.org/project/python-openid2/">python-openid2</a> on the grounds that it had done a good job of maintaining both Python 2 and 3 support in the same codebase, which we needed in order to arrange a practical transition, and that it was in itself well-maintained. We worked with upstream to fix a couple of issues discovered by the Launchpad test suite that blocked us migrating to it (notably <a href="https://github.com/ziima/python-openid/pull/41">PR #41</a>, although that was fixed as <a href="https://github.com/ziima/python-openid/pull/43">PR #43</a> instead), and <a href="https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/387907">switched Launchpad over</a> once python-openid2 3.2 was released. So far, so good. One of the major reasons for much of the disruption in the Python 3 transition was to provide a clean separation between the concept of a sequence of bytes and a text string, which was often a problem for code that needed to handle Unicode: it’s all too common in Python 2 to have code that works on the ASCII domain (which can be represented either as <code>str</code> or <code>unicode</code>) but that fails on Unicode strings outside that subset. Launchpad is less prone to that than many Python 2 applications because the <a href="https://en.wikipedia.org/wiki/Object-relational_mapping">ORM</a> we use (<a href="https://pypi.org/project/storm/">Storm</a>) has always been relatively strict about the boundary between bytes and text; nevertheless, having a stricter data model here is a good thing for us in the long term. It might seem ironic that we ran into exactly such a bug as part of porting to Python 3; but then, we aren’t using the new interpreter yet. Launchpad uses the <a href="https://openid.net/specs/openid-simple-registration-extension-1_0.html">OpenID Simple Registration Extension</a> in its login workflow. It specifically requests the user’s full name from Canonical’s OpenID provider (<a href="https://login.ubuntu.com/">login.ubuntu.com</a>, which we generally call “SSO”): this means that if the user has an SSO account but not yet a Launchpad account, we can create a Launchpad account for them without them needing to enter their name again. That full name is encoded as a UTF-8 string, which in turn is URL-encoded using the usual %xx mechanism. This means that if, say, your name is <a href="https://en.wikipedia.org/wiki/Gr%C3%A1inne_N%C3%AD_Mh%C3%A1ille">Gráinne Ní Mháille</a>, it will show up in the OpenID response’s query string as <code>openid.sreg.fullname=Gr%C3%A1inne+N%C3%AD+Mh%C3%A1ille</code>. python-openid2 uses its <code>openid.urinorm</code> module to normalise parts of the response, decoding and re-encoding it to make sure comparisons work as expected; this is built on top of the URL handling code in Python’s standard library. Now, unlike Python 3, Python 2’s <a href="https://docs.python.org/2/library/urllib.html#urllib.urlencode">urlencode</a> has undocumented restrictions on values in the <code>query</code> argument: if the <code>doseq</code> argument is False (the default), then it converts values using <code>str(v)</code>, while if it’s True then it converts Unicode values using <code>v.encode("ASCII", "replace")</code> (potentially losing information!). In this case, <code>doseq</code> is False, and the input given to it is always text (<code>unicode</code> on Python 2): this works fine if the input is within the ASCII subset, but if it’s not: <pre class="wp-block-code"><code>>>> urlencode({u'openid.sreg.fullname': u'Gráinne Ní Mháille'}) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.7/urllib.py", line 1350, in urlencode v = quote_plus(str(v)) UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 2: ordinal not in range(128)</code></pre> The fix is that on Python 2 one must always pass values to <code>urlencode</code> as bytes rather than text: <pre class="wp-block-code"><code>>>> urlencode({u'openid.sreg.fullname': u'Gráinne Ní Mháille'.encode('UTF-8')}) 'openid.sreg.fullname=Gr%C3%A1inne+N%C3%AD+Mh%C3%A1ille'</code></pre> We’ve sent <a href="https://github.com/ziima/python-openid/pull/47">PR #47</a> to python-openid2 to implement this. We’ve also made a temporary local fork of python-openid2 containing this patch and deployed it to Launchpad production. One thing to be clear about here: though the root cause was a bug in python-openid2, it’s our responsibility to make sure it works correctly when integrated into Launchpad. We missed this bug because of a gap in testing: although we did test the full login workflow, we only did so with a test user whose full name was entirely ASCII. We’ve <a href="https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/389437">closed this gap</a> now, so we’ll catch it if a dependency regresses in future. ]]></content:encoded> <wfw:commentRss>https://blog.launchpad.net/general/login-regression-for-users-with-non-ascii-names/feed</wfw:commentRss> <slash:comments>1</slash:comments> </item> <item> <title>Bug emails now use the bug’s address in the From: header</title> <link>https://blog.launchpad.net/notifications/bug-emails-now-use-the-bugs-address-in-the-from-header</link> <comments>https://blog.launchpad.net/notifications/bug-emails-now-use-the-bugs-address-in-the-from-header#comments</comments> <dc:creator><![CDATA[Colin Watson]]></dc:creator> <pubDate>Wed, 20 May 2020 14:43:11 +0000</pubDate> <category><![CDATA[Notifications]]></category> <category><![CDATA[email]]></category> <category><![CDATA[front-page]]></category> <guid isPermaLink="false">http://blog.launchpad.net/?p=4318</guid> <description><![CDATA[The From: addresses used by Launchpad’s bug notifications have changed, to improve the chances of our messages being delivered over modern internet email. Launchpad sends a lot of email, most of which is the result of Launchpad users performing some kind of action. For example, when somebody adds a comment to a bug, Launchpad sends […]]]></description> <content:encoded><![CDATA[The <code>From:</code> addresses used by Launchpad’s bug notifications have changed, to improve the chances of our messages being delivered over modern internet email. Launchpad sends a lot of email, most of which is the result of Launchpad users performing some kind of action. For example, when somebody adds a comment to a bug, Launchpad sends that comment by email to everyone who’s subscribed to the bug. Most of Launchpad was designed in an earlier era of internet email. In that era, it was perfectly reasonable to take the attitude that we were sending email on behalf of the user – in effect, being a fancy mail user agent or perhaps a little like a mailing list – and so if we generated an email that’s a direct result of something that a user did and consisting mostly of text they wrote, it made sense to put their email address in the <code>From:</code> header. <code>Reply-To:</code> was set so that replies would normally go to the appropriate place (the bug, in the case of bug notifications), but if somebody wanted to go to a bit of effort to start a private side conversation then it was easy to do so; and if email clients had automatic address books then those wouldn’t get confused because the address being used was a legitimate address belonging to the user in question. Of course, some people always wanted to hide their addresses for obvious privacy reasons, so since 2006 Launchpad has had a “Hide my email address from other Launchpad users” switch (which you can set on your <a href="https://launchpad.net/~/+edit">Change your personal details</a> page), and since 2010 Launchpad has honoured this for bug notifications, so if you have that switch set then your bug comments will be sent out as something like “<code>From: Your Name <bug-id@bugs.launchpad.net></code>“. This compromise worked tolerably well for a while. But spammers and other bad actors ruin everything, and the internet email landscape has changed. It’s reasonably common now for operators of email domains to publish <a href="https://en.wikipedia.org/wiki/DMARC"><abbr title="Domain-based Message Authentication, Reporting and Conformance">DMARC</abbr></a> policies that require emails whose From: headers are within that domain to be authenticated in some way, and this is incompatible with the older approach. As a result, it’s been getting increasingly common for Launchpad bug notifications not to be delivered because they failed these authentication checks. Regardless of how justifiable our notification-sending practices were, we have to exist within the reality of internet email as it’s actually deployed. So, thanks to a <a href="https://code.launchpad.net/~teward/launchpad/+git/launchpad/+merge/383643">contribution from Thomas Ward</a>, Launchpad now sends all its bug notifications as if the user in question had the “Hide my email address from other Launchpad users” switch set: that is, they’ll all appear as something like “<code>From: Your Name <bug-id@bugs.launchpad.net></code>“. Over time we expect to extend this sort of approach to the other types of email that we send, possibly with different details depending on the situation. Please <a href="https://help.launchpad.net/Feedback">let us know</a> if this causes any strange behaviour in your email client. We may not be able to fix all of them, depending on how they interact with DMARC’s requirements, but we’d like to be aware of what’s going on. ]]></content:encoded> <wfw:commentRss>https://blog.launchpad.net/notifications/bug-emails-now-use-the-bugs-address-in-the-from-header/feed</wfw:commentRss> <slash:comments>1</slash:comments> </item> <item> <title>Launchpad news, March 2019 – July 2019</title> <link>https://blog.launchpad.net/general/launchpad-news-march-2019-july-2019</link> <comments>https://blog.launchpad.net/general/launchpad-news-march-2019-july-2019#comments</comments> <dc:creator><![CDATA[Colin Watson]]></dc:creator> <pubDate>Tue, 06 Aug 2019 18:16:20 +0000</pubDate> <category><![CDATA[General]]></category> <category><![CDATA[bug fixes]]></category> <category><![CDATA[front-page]]></category> <guid isPermaLink="false">http://blog.launchpad.net/?p=4294</guid> <description><![CDATA[Here’s a brief changelog of what we’ve been up to since our last general update. Bugs Add basic GitLab bug linking (#1603679) Expect the upstream bug ID in the “number” field of GitHub issue objects, not the “id” field (#1824728) Include metadata-only bug changes in Person:+commentedbugs Build farm Filter ASCII NUL characters out of build […]]]></description> <content:encoded><![CDATA[Here’s a brief changelog of what we’ve been up to since our last general update. <h2>Bugs</h2> <ul> <li>Add basic GitLab bug linking (<a href="https://launchpad.net/bugs/1603679">#1603679</a>)</li> <li>Expect the upstream bug ID in the “number” field of GitHub issue objects, not the “id” field (<a href="https://launchpad.net/bugs/1824728">#1824728</a>)</li> <li>Include metadata-only bug changes in <code>Person:+commentedbugs</code></li> </ul> <h2>Build farm</h2> <ul> <li>Filter ASCII NUL characters out of build logtails (<a href="https://launchpad.net/bugs/1831500">#1831500</a>)</li> <li>Encode non-bytes subprocess arguments on Python 2 to avoid crashing on non-ASCII file names under <code>LC_CTYPE=C</code> (<a href="https://launchpad.net/bugs/1832072">#1832072</a>)</li> </ul> <h2>Code</h2> <ul> <li>Don’t preload recipe data when deleting recipes associated with branches or repositories, and add some more job indexes (<a href="https://launchpad.net/bugs/1793266">#1793266</a>, <a href="https://launchpad.net/bugs/1828062">#1828062</a>)</li> <li>Fix crash if <code>checkRefPermissions</code> finds that the repository is nonexistent</li> <li>Add a rescan button to branch merge proposals for failed branch or repository scans</li> <li>Land parts of the work required for Git HTTPS push tokens, though this is not yet complete (<a href="https://launchpad.net/bugs/1824399">#1824399</a>)</li> <li>Refactor code import authorisation to be clearer and safer</li> <li>Set line-height on <code><pre></code> elements in Bazaar file views</li> <li>Work in progress to redeploy Launchpad’s Git backend on more scalable infrastructure</li> </ul> <h2>Infrastructure</h2> <ul> <li>Upgrade to PostgreSQL 10</li> <li>Fix make-lp-user, broken by the fix for <a href="https://launchpad.net/bugs/1576142">#1576142</a></li> <li>Use our own GPG key retrieval implementation when verifying signatures rather than relying on auto-key-retrieve</li> <li>Give urlfetch a default timeout, fixing a regression in process-mail (<a href="https://launchpad.net/bugs/1820552">#1820552</a>)</li> <li>Make test suite pass on Ubuntu 18.04</li> <li>Retry webhook deliveries that respond with 4xx for an hour rather than a day</li> <li>Merge up to a current version of Storm</li> <li>Upgrade to Celery 4.1.1</li> <li>Move development sites from <code>.dev</code> to <code>.test</code></li> <li>Upgrade to Twisted 19.2.1</li> <li>Upgrade to requests 2.22.0</li> <li>Use defusedxml to parse untrusted XML</li> <li>Improve caching of several delegated authorization checks (#<a href="https://launchpad.net/bugs/1834625">1834625</a>)</li> </ul> <h2>Registry</h2> <ul> <li>Fix redaction in pillar listings of projects for which the user only has <code>LimitedView</code> (<a href="https://launchpad.net/bugs/1650430">#1650430</a>)</li> <li>Tighten up the permitted pattern for newly-chosen usernames</li> </ul> <h2>Snappy</h2> <ul> <li>Landed parts of the work required to support private snap builds, though this is not yet complete (<a href="https://launchpad.net/bugs/1639975">#1639975</a>)</li> <li>Generalise snap channel handling slightly, allowing channel selection for core16 and core18</li> <li>Add <code>build-aux/snap/snapcraft.yaml</code> to the list of possible <code>snapcraft.yaml</code> paths (<a href="https://launchpad.net/bugs/1805219">#1805219</a>)</li> <li>Add <code>build-request-id</code> and <code>build-request-timestamp</code> to <code>SNAPCRAFT_IMAGE_INFO</code></li> <li>Allow selecting source snap channels when requesting manual snap builds (<a href="https://launchpad.net/bugs/1791265">#1791265</a>)</li> <li>Push build start timestamps to the store, and use release intents so that builds are more reliably released to channels in the proper sequence (<a href="https://launchpad.net/bugs/1684529">#1684529</a>)</li> <li>Try to manually resolve symlinks in remote Git repositories when fetching <code>snapcraft.yaml</code> (<a href="https://launchpad.net/bugs/1797366">#1797366</a>)</li> <li>Consistently commit transactions in <code>SnapStoreUploadJob</code> (<a href="https://launchpad.net/bugs/1833424">#1833424</a>)</li> <li>Use build request jobs for all snap build requests in the web UI</li> <li>Honour “<code>base: bare</code>” and “<code>build-base</code>” when requesting snap builds (<a href="https://launchpad.net/bugs/1819196">#1819196</a>)</li> </ul> <h2>Soyuz (package management)</h2> <ul> <li>Add command-not-found metadata in the archive to the Release file</li> <li>Check the <code>.deb</code> format using <code>dpkg-deb</code> rather than <code>ar</code></li> <li>Add s390x Secure Initial Program Load signing support (<a href="https://launchpad.net/bugs/1829749">#1829749</a>)</li> <li>Add u-boot Flat Image Tree signing support (<a href="https://launchpad.net/bugs/1831942">#1831942</a>)</li> <li>Use <a href="https://manpages.ubuntu.com/manpages/xenial/en/man1/timeout.1.html">timeout(1)</a> to limit <code>debdiff</code> rather than using <a href="https://manpages.ubuntu.com/manpages/xeni al/en/man3/alarm.3posix.html">alarm(3)</a> ourselves</li> <li>Allow configuring the binary file retention period of a <code>LiveFS</code> (<a href="https://launchpad.net/bugs/1832477">#1832477</a>)</li> <li>Import source packages from Debian bullseye</li> </ul> ]]></content:encoded> <wfw:commentRss>https://blog.launchpad.net/general/launchpad-news-march-2019-july-2019/feed</wfw:commentRss> <slash:comments>2</slash:comments> </item> </channel> </rss>