CINXE.COM
Bottomless S3-backed volumes - Fresh Produce - Fly.io
<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <title>Bottomless S3-backed volumes - Fresh Produce - Fly.io</title> <meta name="description" content="Fly Volumes provide persistent storage to your app by exposing a slice of a local NVMe drive. This works well in many cases, but it requires a “provision-first” approach: you need to know roughly what you’re going to use&hellip;"> <meta name="generator" content="Discourse 3.4.0.beta3-dev - https://github.com/discourse/discourse version 9c4c17ad3fc80f79e8046b522efa4c769c76130d"> <link rel="icon" type="image/png" href="https://global.discourse-cdn.com/fly1/optimized/1X/276bd78a68c1a9248075881ff96190aa81ee5804_2_32x32.png"> <link rel="apple-touch-icon" type="image/png" href="https://global.discourse-cdn.com/fly1/optimized/1X/6e8955d00cb218e186ebbe3a6ab06fefdce15290_2_180x180.svg"> <meta name="theme-color" media="all" content="#3c83e7"> <meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0, user-scalable=yes, viewport-fit=cover"> <link rel="canonical" href="https://community.fly.io/t/bottomless-s3-backed-volumes/15648" /> <link rel="search" type="application/opensearchdescription+xml" href="https://community.fly.io/opensearch.xml" title="Fly.io Search"> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/color_definitions_graceful_10_11_05d1e43bbdb1fe55a3904a42cf7c328266256175.css?__ws=community.fly.io" media="all" rel="stylesheet" class="light-scheme"/> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/desktop_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="desktop" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/checklist_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="checklist" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/discourse-ai_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="discourse-ai" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/discourse-akismet_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="discourse-akismet" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/discourse-cakeday_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="discourse-cakeday" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/discourse-chat-integration_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="discourse-chat-integration" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/discourse-details_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="discourse-details" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/discourse-lazy-videos_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="discourse-lazy-videos" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/discourse-local-dates_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="discourse-local-dates" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/discourse-policy_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="discourse-policy" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/discourse-presence_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="discourse-presence" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/discourse-solved_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="discourse-solved" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/discourse-templates_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="discourse-templates" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/discourse-topic-voting_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="discourse-topic-voting" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/footnote_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="footnote" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/hosted-site_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="hosted-site" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/poll_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="poll" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/spoiler-alert_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="spoiler-alert" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/discourse-ai_desktop_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="discourse-ai_desktop" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/discourse-topic-voting_desktop_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="discourse-topic-voting_desktop" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/poll_desktop_4c18115f55f7c6b0f3db50cefb171df31a9e515f.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="poll_desktop" /> <link href="https://sea1.discourse-cdn.com/fly1/stylesheets/desktop_theme_11_32689e130506ba5d5284a6e44e033e8dadba8149.css?__ws=community.fly.io" media="all" rel="stylesheet" data-target="desktop_theme" data-theme-id="11" data-theme-name="graceful"/> <script src="https://status.flyio.net/embed/script.js" defer="" nonce="rWNuyIImxcmS8hElzBp6BOwwy"></script> <script defer="" data-domain="community.fly.io" src="https://plausible.io/js/script.hash.js" nonce="rWNuyIImxcmS8hElzBp6BOwwy"></script> <link rel="alternate nofollow" type="application/rss+xml" title="RSS feed of 'Bottomless S3-backed volumes'" href="https://community.fly.io/t/bottomless-s3-backed-volumes/15648.rss" /> <meta property="og:site_name" content="Fly.io" /> <meta property="og:type" content="website" /> <meta name="twitter:card" content="summary_large_image" /> <meta name="twitter:image" content="https://global.discourse-cdn.com/fly1/original/3X/d/f/df182ca52467742d389d4af72b63e05802bf38d8.jpeg" /> <meta property="og:image" content="https://global.discourse-cdn.com/fly1/original/3X/6/e/6e8955d00cb218e186ebbe3a6ab06fefdce15290.svg" /> <meta property="og:url" content="https://community.fly.io/t/bottomless-s3-backed-volumes/15648" /> <meta name="twitter:url" content="https://community.fly.io/t/bottomless-s3-backed-volumes/15648" /> <meta property="og:title" content="Bottomless S3-backed volumes" /> <meta name="twitter:title" content="Bottomless S3-backed volumes" /> <meta property="og:description" content="Fly Volumes provide persistent storage to your app by exposing a slice of a local NVMe drive. This works well in many cases, but it requires a “provision-first” approach: you need to know roughly what you’re going to use in advance (yes, you can manually extend your disks, even without restarting your machine, but you cannot yet shrink them back). Also, there’s only so much space available to a single volume on a single server: right now, the cap is 500 GiB. If you have terabyte upon terabyte of..." /> <meta name="twitter:description" content="Fly Volumes provide persistent storage to your app by exposing a slice of a local NVMe drive. This works well in many cases, but it requires a “provision-first” approach: you need to know roughly what you’re going to use in advance (yes, you can manually extend your disks, even without restarting your machine, but you cannot yet shrink them back). Also, there’s only so much space available to a single volume on a single server: right now, the cap is 500 GiB. If you have terabyte upon terabyte of..." /> <meta property="og:article:section" content="Fresh Produce" /> <meta property="og:article:section:color" content="A461EF" /> <meta property="og:article:tag" content="storage" /> <meta property="og:article:tag" content="volumes" /> <meta name="twitter:label1" value="Reading time" /> <meta name="twitter:data1" value="9 mins 🕑" /> <meta name="twitter:label2" value="Likes" /> <meta name="twitter:data2" value="59 ❤" /> <meta property="article:published_time" content="2023-09-27T17:35:26+00:00" /> <meta property="og:ignore_canonical" content="true" /> <link rel="next" href="/t/bottomless-s3-backed-volumes/15648?page=2"> <script type="application/ld+json">{"@context":"http://schema.org","@type":"QAPage","name":"Bottomless S3-backed volumes","mainEntity":{"@type":"Question","name":"Bottomless S3-backed volumes","text":"<a href=\"https://fly.io/docs/reference/volumes/\">Fly Volumes<\/a> provide persistent storage to your app by exposing a slice of a local NVMe drive. This works well in many cases, but it requires a “provision-first” approach: you need to know roughly what you’re going to use in advance (yes, you can manually extend your disks, <a href=\"https://community.fly.io/t/restartless-volume-extension/14130\">even without restarting yo…<\/a>","upvoteCount":24,"answerCount":0,"datePublished":"2023-09-27T17:35:25.972Z","author":{"@type":"Person","name":"MatthewIngwersen","url":"https://community.fly.io/u/MatthewIngwersen"}}}</script> </head> <body class="crawler browser-update"> <div class="background-container"></div> <header> <a href="/"> Fly.io </a> </header> <div id="main-outlet" class="wrap" role="main"> <div id="topic-title"> <h1> <a href="/t/bottomless-s3-backed-volumes/15648">Bottomless S3-backed volumes</a> </h1> <div class="topic-category" itemscope itemtype="http://schema.org/BreadcrumbList"> <span itemprop="itemListElement" itemscope itemtype="http://schema.org/ListItem"> <a href="/c/fresh-produce/27" class="badge-wrapper bullet" itemprop="item"> <span class='badge-category-bg' style='background-color: #A461EF'></span> <span class='badge-category clear-badge'> <span class='category-name' itemprop='name'>Fresh Produce</span> </span> </a> <meta itemprop="position" content="1" /> </span> </div> <div class="topic-category"> <div class='discourse-tags list-tags'> <a href='https://community.fly.io/tag/storage' class='discourse-tag' rel="tag">storage</a>, <a href='https://community.fly.io/tag/volumes' class='discourse-tag' rel="tag">volumes</a> </div> </div> </div> <div itemscope itemtype='http://schema.org/DiscussionForumPosting'> <meta itemprop='headline' content='Bottomless S3-backed volumes'> <link itemprop='url' href='https://community.fly.io/t/bottomless-s3-backed-volumes/15648'> <meta itemprop='datePublished' content='2023-09-27T17:35:25Z'> <meta itemprop='articleSection' content='Fresh Produce'> <meta itemprop='keywords' content='storage, volumes'> <div itemprop='publisher' itemscope itemtype="http://schema.org/Organization"> <meta itemprop='name' content='Fly.io'> <div itemprop='logo' itemscope itemtype="http://schema.org/ImageObject"> <meta itemprop='url' content='https://global.discourse-cdn.com/fly1/original/3X/4/9/49b670bceec1e465f38edfcb9223f572edcbe900.svg'> </div> </div> <div id='post_1' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/MatthewIngwersen'><span itemprop='name'>MatthewIngwersen</span></a> </span> <link itemprop="mainEntityOfPage" href="https://community.fly.io/t/bottomless-s3-backed-volumes/15648"> <span class="crawler-post-infos"> <time datetime='2023-09-27T17:35:26Z' class='post-time'> September 27, 2023, 5:35pm </time> <meta itemprop='dateModified' content='2023-09-27T17:35:26Z'> <span itemprop='position'>1</span> </span> </div> <div class='post' itemprop='text'> <p><a href="https://fly.io/docs/reference/volumes/">Fly Volumes</a> provide persistent storage to your app by exposing a slice of a local NVMe drive. This works well in many cases, but it requires a “provision-first” approach: you need to know roughly what you’re going to use in advance (yes, you can manually extend your disks, <a href="https://community.fly.io/t/restartless-volume-extension/14130">even without restarting your machine</a>, but you cannot yet shrink them back). Also, there’s only so much space available to a single volume on a single server: right now, the cap is 500 GiB. If you have terabyte upon terabyte of data to store, especially if it’s not frequently accessed, then you’ll need to look for alternatives.</p> <p>One option might be an object storage service like S3. They can store a virtually limitless amount of data, they have great durability, and they’re cost-effective. There’s a good chance that you’re already using one. And if your app already supports object storage directly, great!</p> <p>But what if your app doesn’t — like, say, your Postgres database? Or perhaps it would just be easier to store your data as files on a disk, but with the capacity and durability a you can get from object storage.</p> <p>With this in mind, we wanted to know whether disk volumes backed by object storage were possible. So we read up on it and built an experimental version based on recent research into <a href="https://doi.org/10.1145/3492321.3524271" rel="noopener nofollow ugc">log-structured virtual disks (LSVD)</a>. (That’s why you’ll see a lot of <code>lsvd</code> commands below — we’re not committed to the name though <img src="https://emoji.discourse-cdn.com/twitter/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20">.) As a proof-of-concept, we even got a 100 GiB (and growing) Postgres database running on it—<a href="https://ct-log-viewer.fly.dev" rel="noopener nofollow ugc">check it out here</a>.</p> <p>Now it’s your turn to try it out—we want to know what <em>you</em> think!</p> <blockquote> <p><strong><img src="https://emoji.discourse-cdn.com/twitter/warning.png?v=12" title=":warning:" class="emoji" alt=":warning:" loading="lazy" width="20" height="20"> NB:</strong> Just a quick reminder before you proceed—<strong>this is an experimental feature</strong>. It’s not ready for production use, and it’s not officially supported. If you run into problems, please get in touch with us here on the community forum.</p> </blockquote> <h1><a name="how-can-i-try-it-1" class="anchor" href="#how-can-i-try-it-1"></a>How can I try it?</h1> <p>Before starting, make sure that you have the latest <code>flyctl</code> version (you’ll need features introduced in v0.1.103).</p> <h2><a name="create-a-bucket-2" class="anchor" href="#create-a-bucket-2"></a>Create a bucket</h2> <p>First, you’ll need to create a bucket on S3 or a compatible object storage service. You’ll also need credentials that can read, write, and list objects in that bucket. You can do this however you’d like, but for S3 itself, you can check out the open-source <code>s3-credentials</code> tool. Specifically, you can use the <code>--create-bucket</code> and <code>--bucket-region</code> flags with <code>s3-credentials create</code> to get a new bucket created along with credentials for a new AWS user that has access to it.</p> <blockquote> <p><strong><img src="https://emoji.discourse-cdn.com/twitter/warning.png?v=12" title=":warning:" class="emoji" alt=":warning:" loading="lazy" width="20" height="20"> NB:</strong> Create your new bucket in a region close to the <a href="http://Fly.io" rel="noopener nofollow ugc">Fly.io</a> region from which you plan to use it. This is important—it will keep the latency of I/O operations down! For S3, this <a href="https://rtt.fly.dev/to/s3" rel="noopener nofollow ugc">live RTT table</a> can help you choose.</p> </blockquote> <h2><a name="set-up-your-fly-app-3" class="anchor" href="#set-up-your-fly-app-3"></a>Set up your Fly app</h2> <p>Since this is an experimental feature, we strongly recommend creating a fresh app with <code>fly apps create</code> to test this, rather than using an existing one.</p> <p>Once you have your bucket and credentials, you can run</p> <pre><code class="lang-auto">fly volumes lsvd setup -a <your-app-name> </code></pre> <p>It’ll prompt you for the relevant information and set secrets on your app to configure it.</p> <h2><a name="create-an-lsvd-enabled-fly-machine-4" class="anchor" href="#create-an-lsvd-enabled-fly-machine-4"></a>Create an LSVD-enabled Fly Machine</h2> <p>Use the new <code>--lsvd</code> flag with <code>fly machines run</code> to create a new Machine in your app. This will inject and start a background daemon (called <code>lsvd</code>) in your Machine that actually provides the volume. It will be mounted at the path you specified in the previous (setup) step. (The raw volume will be exposed as <code>/dev/nbd0</code> for those of you who want to get real fancy/crazy with the raw block device.)</p> <p>Make sure to give your Machine enough memory (<code>--vm-memory</code>), because the <code>lsvd</code> daemon will need some of it. 512 MiB is a good baseline. Larger disks will require more memory: the <code>lsvd</code> process currently needs 2 MiB of memory per GiB of disk. (We realize that this is a lot of overhead; we hope to reduce it in the future!)</p> <blockquote> <p><strong><img src="https://emoji.discourse-cdn.com/twitter/warning.png?v=12" title=":warning:" class="emoji" alt=":warning:" loading="lazy" width="20" height="20"> NB:</strong> Before deploying your machine, you’ll need the CA certificates available in your Docker image, so that the <code>lsvd</code> daemon can connect to object storage over HTTPS. They’re usually easy to add. Here’s an example <code>Dockerfile</code> line for Debian-based images (with <code>apt-get</code>):</p> </blockquote> <pre><code class="lang-auto">RUN apt-get -y update \ && apt-get -y install ca-certificates \ && apt-get -y clean \ && rm -rf /var/lib/apt/lists/* </code></pre> <p>In summary, this is the full command to get you started with a local <code>Dockerfile</code>:</p> <pre><code class="lang-auto">fly machines run -a <app-name> --lsvd --vm-memory 512 . </code></pre> <blockquote> <p><strong><img src="https://emoji.discourse-cdn.com/twitter/warning.png?v=12" title=":warning:" class="emoji" alt=":warning:" loading="lazy" width="20" height="20"> NB:</strong> For now, don’t run more than one LSVD-enabled machine per app! They’ll conflict with each other and corrupt your volume.</p> </blockquote> <hr> <h1><a name="nerdy-stuff-below-for-all-of-you-nerds-5" class="anchor" href="#nerdy-stuff-below-for-all-of-you-nerds-5"></a>Nerdy stuff below <img src="https://emoji.discourse-cdn.com/twitter/arrow_down.png?v=12" title=":arrow_down:" class="emoji" alt=":arrow_down:" loading="lazy" width="20" height="20">, for all of you nerds <img src="https://emoji.discourse-cdn.com/twitter/nerd_face.png?v=12" title=":nerd_face:" class="emoji" alt=":nerd_face:" loading="lazy" width="20" height="20"></h1> <h2><a name="diving-deeper-what-are-all-these-objects-showing-up-in-s3-6" class="anchor" href="#diving-deeper-what-are-all-these-objects-showing-up-in-s3-6"></a>Diving deeper: what are all these objects showing up in S3?</h2> <p>Once you’ve run your shiny new S3-backed volume for a while, you’ll notice objects with funny hexadecimal names appearing in your bucket. Each object contains logs of the writes made to the disk. They’re numbered sequentially.</p> <p>Each log entry records both what part of the disk was written and the actual data that was written. To read back a sector of the disk, you can scan the logs to find the most recent write to that sector and pull the data from that entry.</p> <p>This basic idea can ultimately be optimized enough to make it practical (for more on this, check out the <a href="https://doi.org/10.1145/3492321.3524271" rel="noopener nofollow ugc">paper</a>!), and it has a really nice quality: <strong>snapshotting is “built-in” to the design</strong>. To restore the volume to a given point, all you need to do is ignore all the log entries that come after that point. You can try this yourself:</p> <ol> <li>Write a bunch of data to the disk. Shut the machine down and write down the name of the most recent log object.</li> <li>Start the machine again, write even more data, and then shut it down again.</li> <li>Delete all the log objects that come after one you found in step (1).</li> <li>Start the machine one last time. Observe that the disk state rolled back to how it was at the end of step (1).</li> </ol> <h2><a name="but-whats-the-performance-like-7" class="anchor" href="#but-whats-the-performance-like-7"></a>But what’s the performance like?</h2> <p>There’s no getting around the fact that accessing data from S3 (or a compatible service) will have a much higher latency (tens to hundreds of milliseconds) than a locally attached SSD (tens to hundreds of microseconds). However, like an SSD (and unlike a hard drive), S3 can handle many requests simultaneously. Accordingly, we’ll use up to 100 connections at once to access the backend. We also tune the Linux kernel in your Machine to read ahead aggressively during sequential reads to provide reasonable throughput even with high latency. From our testing, up to a few thousand IOPS and transfer rates on the order of 10 to 100 MiB/s are achievable.</p> <p>The tradeoffs won’t make sense for every app! However, if your application issues many I/O operations concurrently, deals with a large amount of cold data, and can work with the latency, we think that this is an option worth exploring.</p> <blockquote> <p><strong><img src="https://emoji.discourse-cdn.com/twitter/warning.png?v=12" title=":warning:" class="emoji" alt=":warning:" loading="lazy" width="20" height="20"> NB:</strong> For improving performance, here are some additional things to consider:</p> <ul> <li>If you’re running a database, then it may be possible to tune it. E.g., Postgres users might try setting <code>effective_io_concurrency</code> to 100 to match the number of concurrent I/O operations available.</li> <li>If you’re feeling intrepid, then you can experiment with locally caching hot data via the Linux kernel’s <a href="https://www.kernel.org/doc/Documentation/device-mapper/cache.txt" rel="noopener nofollow ugc">dm-cache</a> feature, which is built into our guest kernel. If interested, let us know below and we can share some additional information about this!</li> </ul> </blockquote> <h1><a name="acknowledgments-8" class="anchor" href="#acknowledgments-8"></a>Acknowledgments</h1> <p>This feature is based on research presented in the <a href="https://doi.org/10.1145/3492321.3524271" rel="noopener nofollow ugc">LSVD paper</a> and earlier work on <a href="https://dspace.cuni.cz/bitstream/handle/20.500.11956/148791/120397658.pdf" rel="noopener nofollow ugc">block devices in userspace</a>. You can find software published with the papers on GitHub: <a href="https://github.com/asch/dis" rel="noopener nofollow ugc">asch/dis</a>, <a href="https://github.com/asch/buse" rel="noopener nofollow ugc">asch/buse</a>, and <a href="https://github.com/asch/bs3" rel="noopener nofollow ugc">asch/bs3</a>.</p> <p>Furthermore, we built the <code>lsvd</code> program that gets added to your VM using a number of open-source libraries—you can see the full list by running <code>/.fly/lsvd licenses</code> in your LSVD-enabled Machine.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="24" /> <span class='post-likes'>24 Likes</span> </div> <div class='crawler-linkback-list' itemscope itemtype='http://schema.org/ItemList'> <div itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <a itemprop='url' href="https://community.fly.io/t/postgres-barman-backup-with-tigris/18454/3">Postgres Barman backup with Tigris</a> <meta itemprop='position' content='12'> </div> <div itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <a itemprop='url' href="https://community.fly.io/t/global-caching-object-storage-on-fly-io-in-private-beta/17744/15">Global caching object storage on Fly.io in private beta</a> <meta itemprop='position' content='13'> </div> <div itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <a itemprop='url' href="https://community.fly.io/t/auto-extend-volume-size-after-a-certain-threshold/16326/4">Auto extend volume size after a certain threshold!</a> <meta itemprop='position' content='14'> </div> <div itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <a itemprop='url' href="https://community.fly.io/t/tigris-as-local-volume/20043/2">Tigris as local volume</a> <meta itemprop='position' content='15'> </div> <div itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <a itemprop='url' href="https://community.fly.io/t/host-issue-notifications/19789/5">Host Issue notifications</a> <meta itemprop='position' content='16'> </div> <div itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <a itemprop='url' href="https://community.fly.io/t/tigris-backup-for-persistent-volumes-and-mysql-database/21519">Tigris backup for persistent volumes and MySQL database?</a> <meta itemprop='position' content='17'> </div> <div itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <a itemprop='url' href="https://community.fly.io/t/tigris-backed-volumes/20792">Tigris-backed volumes</a> <meta itemprop='position' content='18'> </div> </div> </div> <div id='post_2' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/gdub01'><span itemprop='name'>gdub01</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2023-09-28T14:57:19Z' class='post-time'> September 28, 2023, 2:57pm </time> <meta itemprop='dateModified' content='2023-09-28T14:57:19Z'> <span itemprop='position'>2</span> </span> </div> <div class='post' itemprop='text'> <p>Well this sounds amazing.</p> <p>Just wanted to make sure I’m understanding right…</p> <p>If I have an app that mounts a volume on fly with say a 100 gb limit, but then use this service to back it up to s3, what is that 100 gb used for? Would it just be better to have a 1 gb volume that is kind of a pass-through to s3 or does volume size still matter for some sort of temp file thing?</p> <p>If I have a mounted volume next to the app, it feels like I’m using the local filesystem. That would still be true even though the files are hosted on s3 right?</p> <p>If I had 2 servers, each with a volume backed by s3, I could have either volume backup to s3 and/or access files from s3? (I ask because I think fly apps can’t share volumes?)</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_3' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/MatthewIngwersen'><span itemprop='name'>MatthewIngwersen</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2023-09-28T18:38:27Z' class='post-time'> September 28, 2023, 6:38pm </time> <meta itemprop='dateModified' content='2023-09-28T18:38:27Z'> <span itemprop='position'>3</span> </span> </div> <div class='post' itemprop='text'> <p>Hi <a class="mention" href="/u/gdub01">@gdub01</a>! Thanks for asking.</p> <aside class="quote no-group" data-username="gdub01" data-post="2" data-topic="15648"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/gdub01/48/15432_2.png" class="avatar"> gdub01:</div> <blockquote> <p>If I have an app that mounts a volume on fly with say a 100 gb limit, but then use this service to back it up to s3, what is that 100 gb used for? Would it just be better to have a 1 gb volume that is kind of a pass-through to s3 or does volume size still matter for some sort of temp file thing?</p> </blockquote> </aside> <p>To clarify, for now these experimental S3-backed volumes are separate from the existing “Fly Volumes” product based on local NVMe drives. They do not back up or mirror a local NVMe Fly Volume to S3. Rather, when you set up LSVD, you get a virtual disk attached to your Machine whose data is stored and accessed <em>entirely</em> on S3, with no local Fly Volume involved.</p> <p>Since disks are expected to have a fixed size, you need to configure one for your S3-backed volume. However, only data that are written to the volume are sent to S3. If you create a 1 TiB S3-backed volume but write only 1 GiB of data to it, then your S3 bucket will have ~1 GiB of data in it. If you create a 1 TiB S3-backed volume and overwrite the entire disk twice, then your S3 bucket will have ~2 TiB of data in it. (A future iteration of this would be able to clean overwritten data from S3 and bring that back down toward 1 TiB.)</p> <aside class="quote no-group" data-username="gdub01" data-post="2" data-topic="15648"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/gdub01/48/15432_2.png" class="avatar"> gdub01:</div> <blockquote> <p>If I have a mounted volume next to the app, it feels like I’m using the local filesystem. That would still be true even though the files are hosted on s3 right?</p> </blockquote> </aside> <p>Exactly. From the perspective of your Machine, the S3-backed volume is just another disk drive. If you configure a mount point when you run <code>fly volumes lsvd setup</code>, then we’ll automatically format the drive with a filesystem and mount it for you, and you can use it like any other filesystem.</p> <aside class="quote no-group" data-username="gdub01" data-post="2" data-topic="15648"> <div class="title"> <div class="quote-controls"></div> <img loading="lazy" alt="" width="24" height="24" src="https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/gdub01/48/15432_2.png" class="avatar"> gdub01:</div> <blockquote> <p>If I had 2 servers, each with a volume backed by s3, I could have either volume backup to s3 and/or access files from s3? (I ask because I think fly apps can’t share volumes?)</p> </blockquote> </aside> <p>Admittedly, I’m not quite sure what you’re asking (sorry about that! <img src="https://emoji.discourse-cdn.com/twitter/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20">), but I hope that this might help:</p> <ul> <li> <p>Like existing Fly Volumes, these S3-backed volumes are designed to attach to a single Fly Machine. Making these work as shared disks isn’t something we’ve thought much about yet.</p> </li> <li> <p>However, since the data is stored remotely in S3, you can destroy your LSVD-enabled Fly Machine and create a new one, and your volume should continue to work on the new Machine—even if it’s on a different physical host, or even in a different region!</p> </li> <li> <p>The <code>fly volumes lsvd setup</code> tool configures your app to run one single LSVD-enabled Machine (and therefore one S3-backed volume). However, it’s possible to run <em>n</em> LSVD-enabled Machines (so <em>n</em> independent S3-backed volumes) by pointing each Machine at its own S3 bucket. If anyone here wants to try this, feel free to bug me for details here! (Or perhaps one of you will figure it out by reading the <a href="https://github.com/superfly/flyctl/blob/02946c0be5f1a9952ad3f00f31ff2a8f069c9390/internal/command/volumes/lsvd/setup.go" rel="noopener nofollow ugc"><code>flyctl</code> source code</a>. <img src="https://emoji.discourse-cdn.com/twitter/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20">)</p> </li> </ul> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="3" /> <span class='post-likes'>3 Likes</span> </div> </div> <div id='post_4' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/Hypermind'><span itemprop='name'>Hypermind</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2023-09-28T22:08:33Z' class='post-time'> September 28, 2023, 10:08pm </time> <meta itemprop='dateModified' content='2023-09-28T22:08:33Z'> <span itemprop='position'>4</span> </span> </div> <div class='post' itemprop='text'> <p>JFYI, Azure Blobs seem to be considerably faster than S3, like 3-5x better in terms of latencies. I know this because we use a similar system based on Rclone.</p> <p>Another useful feature for such kind of volumes is end-to-end encryption. We use that too (provided by Rclone).</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="2" /> <span class='post-likes'>2 Likes</span> </div> </div> <div id='post_5' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/lpil'><span itemprop='name'>lpil</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2023-09-29T15:14:05Z' class='post-time'> September 29, 2023, 3:14pm </time> <meta itemprop='dateModified' content='2023-09-29T15:14:05Z'> <span itemprop='position'>5</span> </span> </div> <div class='post' itemprop='text'> <p>Whoa! What a fab sounding feature!</p> <p>Are there clients we can use to read and write that data from outside of Fly? For example, for disaster recovery.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="3" /> <span class='post-likes'>3 Likes</span> </div> </div> <div id='post_6' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/MatthewIngwersen'><span itemprop='name'>MatthewIngwersen</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2023-10-02T18:20:25Z' class='post-time'> October 2, 2023, 6:20pm </time> <meta itemprop='dateModified' content='2023-10-02T18:20:25Z'> <span itemprop='position'>6</span> </span> </div> <div class='post' itemprop='text'> <p><a class="mention" href="/u/lpil">@lpil</a> we haven’t built an external client, but thanks for asking—it’s making me think about what kind of tooling a “full-scale” version of this might need! <img src="https://emoji.discourse-cdn.com/twitter/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p> <p>For now, I can at least share the (current) format of the S3 objects so that you know how your data is actually stored:</p> <p>Each object starts with a fixed-size (12 KiB) header. The header consists of 1,024 12-byte entries, each of which describes a single write to the volume. The entries have the following fields, in this order:</p> <ol> <li>An unsigned 64-bit big-endian integer giving the address of the write (in bytes), where 0 is the first byte of the volume</li> <li>An unsigned 32-bit big-endian integer giving the size of the write (in bytes)</li> </ol> <p>The implementation uses a sector size of 4 KiB, so in practice all of the addresses and sizes will be multiples of 4,096. An entry with a size field of 0 indicates that not all 1,024 of the header entries are used, and that it (and all subsequent header entries) should be ignored.</p> <p>After the 12 KiB header, you’ll find the actual data associated with each write, stored in the same order as the header entries.</p> <p><img src="https://emoji.discourse-cdn.com/twitter/warning.png?v=12" title=":warning:" class="emoji" alt=":warning:" loading="lazy" width="20" height="20"> <strong>NB:</strong> When generating the log objects, this implementation will merge/flatten/defragment overlapping and adjacent writes. For some workloads, this significantly reduces the amount of data stored in S3 and improves the fragmentation situation, but the trade-off is that the entries in the objects are no longer a literal log of the writes made to the disk. As a result, stopping in the middle of an object when replaying the log will <em>not</em> in general reproduce a disk state that ever actually existed. However, stopping <em>in between</em> objects will give you a true/valid snapshot.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="3" /> <span class='post-likes'>3 Likes</span> </div> </div> <div id='post_7' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/dholth'><span itemprop='name'>dholth</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2023-10-05T15:41:51Z' class='post-time'> October 5, 2023, 3:41pm </time> <meta itemprop='dateModified' content='2023-10-05T15:41:51Z'> <span itemprop='position'>7</span> </span> </div> <div class='post' itemprop='text'> <p>Is <a href="https://lwn.net/Articles/903855/" class="inline-onebox" rel="noopener nofollow ugc">An io_uring-based user-space block driver [LWN.net]</a> available in the fly machine?</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_8' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/DAlperin'><span itemprop='name'>DAlperin</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2023-10-06T20:08:52Z' class='post-time'> October 6, 2023, 8:08pm </time> <meta itemprop='dateModified' content='2023-10-06T20:08:52Z'> <span itemprop='position'>8</span> </span> </div> <div class='post' itemprop='text'> <p>Not quite yet! I’ve actually played around with this a bunch. It’s available in the 6-series kernel which we haven’t generally switched to yet unfortunately, but when we do upgrade, I plan to make it available.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="4" /> <span class='post-likes'>4 Likes</span> </div> </div> <div id='post_9' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/kot'><span itemprop='name'>kot</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2023-10-08T02:38:51Z' class='post-time'> October 8, 2023, 2:38am </time> <meta itemprop='dateModified' content='2023-10-11T04:37:29Z'> <span itemprop='position'>9</span> </span> </div> <div class='post' itemprop='text'> <p>Will/would Fly charge me for egress when writing to the filesystem?</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_10' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/pier'><span itemprop='name'>pier</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2023-10-08T18:50:14Z' class='post-time'> October 8, 2023, 6:50pm </time> <meta itemprop='dateModified' content='2023-10-08T18:50:14Z'> <span itemprop='position'>10</span> </span> </div> <div class='post' itemprop='text'> <p>This sounds great!</p> <p>Any info on pricing?</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_11' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/mileszim'><span itemprop='name'>mileszim</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2023-10-11T00:40:45Z' class='post-time'> October 11, 2023, 12:40am </time> <meta itemprop='dateModified' content='2023-10-11T08:42:24Z'> <span itemprop='position'>11</span> </span> </div> <div class='post' itemprop='text'> <p><strong>EDIT</strong></p> <p>I figured out what was needed–realized the mount is through an ext4 filesystem on top of the volume, so I ssh’d in and ran <code>resize2fs /dev/nbd0</code> and it worked!</p> <hr> <p><em>Original post:</em></p> <p>This is really cool! It’s perfect for a specific use case of mine: running ArchiveBox in a way that lets me capture large amounts of data, but without needing a huge volume allocated + attached all the time.</p> <p>Some feedback: Tried it out by loading up the drive until it was full, initially 3gb worth. Then went and tried to increase the size by updating the FLY_LSVD_DEVICE_SIZE secret, and now it’s having trouble booting. Here are the logs:</p> <pre><code class="lang-auto">2023-10-11T00:27:15.917 app[3d8d7669b1dd68] lax [info] [ 0.044189] Spectre V2 : WARNING: Unprivileged eBPF is enabled with eIBRS on, data leaks possible via Spectre v2 BHB attacks! 2023-10-11T00:27:15.975 app[3d8d7669b1dd68] lax [info] [ 0.069450] PCI: Fatal: No config space access function found 2023-10-11T00:27:16.173 app[3d8d7669b1dd68] lax [info] INFO Starting init (commit: 5d9c42f)... 2023-10-11T00:27:16.200 app[3d8d7669b1dd68] lax [info] INFO [fly api proxy] listening at /.fly/api 2023-10-11T00:27:16.203 app[3d8d7669b1dd68] lax [info] 2023/10/11 00:27:16 listening on [fdaa:1:53b:a7b:1e4:e173:d8c1:2]:22 (DNS: [fdaa::3]:53) 2023-10-11T00:27:16.208 app[3d8d7669b1dd68] lax [info] lsvd: Starting (commit 58f00b1). 2023-10-11T00:27:16.228 app[3d8d7669b1dd68] lax [info] lsvd: Starting recovery from the archive-zim-dev bucket. 2023-10-11T00:27:18.347 app[3d8d7669b1dd68] lax [info] lsvd: Recovery complete (processed 448 objects). 2023-10-11T00:27:18.449 app[3d8d7669b1dd68] lax [info] lsvd: Archived 0 objects after the recovery point. 2023-10-11T00:27:18.450 app[3d8d7669b1dd68] lax [info] lsvd: Connecting the kernel to the device. 2023-10-11T00:27:18.462 app[3d8d7669b1dd68] lax [info] lsvd: Device is now operational. 2023-10-11T00:27:18.462 app[3d8d7669b1dd68] lax [info] lsvd: Mounting /dev/nbd0 on /data. 2023-10-11T00:27:19.238 app[3d8d7669b1dd68] lax [info] INFO Preparing to run: `dumb-init -- /app/bin/docker_entrypoint.sh archivebox server --quick-init 0.0.0.0:8000` as root 2023-10-11T00:27:20.369 app[3d8d7669b1dd68] lax [info] find: ‘/home/archivebox/.config/chromium/Crash Reports/pending/’: No such file or directory 2023-10-11T00:27:21.470 app[3d8d7669b1dd68] lax [info] [i] [2023-10-11 00:27:21] ArchiveBox v0.6.3: archivebox server --quick-init 0.0.0.0:8000 2023-10-11T00:27:21.470 app[3d8d7669b1dd68] lax [info] > /data 2023-10-11T00:27:22.085 app[3d8d7669b1dd68] lax [info] find: ‘/home/archivebox/.config/chromium/Crash Reports/pending/’: No such file or directory 2023-10-11T00:27:22.327 app[3d8d7669b1dd68] lax [info] OSError: [Errno 28] No space left on device 2023-10-11T00:27:22.327 app[3d8d7669b1dd68] lax [info] During handling of the above exception, another exception occurred: 2023-10-11T00:27:22.327 app[3d8d7669b1dd68] lax [info] Traceback (most recent call last): 2023-10-11T00:27:22.327 app[3d8d7669b1dd68] lax [info] File "/usr/local/bin/archivebox", line 33, in <module> 2023-10-11T00:27:22.327 app[3d8d7669b1dd68] lax [info] sys.exit(load_entry_point('archivebox', 'console_scripts', 'archivebox')()) 2023-10-11T00:27:22.329 app[3d8d7669b1dd68] lax [info] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2023-10-11T00:27:22.329 app[3d8d7669b1dd68] lax [info] File "/app/archivebox/cli/__init__.py", line 140, in main 2023-10-11T00:27:22.329 app[3d8d7669b1dd68] lax [info] run_subcommand( 2023-10-11T00:27:22.329 app[3d8d7669b1dd68] lax [info] File "/app/archivebox/cli/__init__.py", line 74, in run_subcommand 2023-10-11T00:27:22.329 app[3d8d7669b1dd68] lax [info] setup_django(in_memory_db=subcommand in fake_db, check_db=cmd_requires_db and not init_pending) 2023-10-11T00:27:22.329 app[3d8d7669b1dd68] lax [info] File "/app/archivebox/config.py", line 1249, in setup_django 2023-10-11T00:27:22.329 app[3d8d7669b1dd68] lax [info] with open(settings.ERROR_LOG, "a", encoding='utf-8') as f: 2023-10-11T00:27:22.329 app[3d8d7669b1dd68] lax [info] OSError: [Errno 28] No space left on device 2023-10-11T00:27:23.245 app[3d8d7669b1dd68] lax [info] INFO Main child exited normally with code: 1 2023-10-11T00:27:23.245 app[3d8d7669b1dd68] lax [info] INFO Starting clean up. 2023-10-11T00:27:23.246 app[3d8d7669b1dd68] lax [info] lsvd: Received signal. Starting to shut down gracefully. 2023-10-11T00:27:23.246 app[3d8d7669b1dd68] lax [info] lsvd: Looking for mounted filesystems on the device. 2023-10-11T00:27:23.247 app[3d8d7669b1dd68] lax [info] lsvd: /dev/nbd0 is mounted on /data; unmounting. 2023-10-11T00:27:23.248 app[3d8d7669b1dd68] lax [info] WARN hallpass exited, pid: 305, status: signal: 15 (SIGTERM) 2023-10-11T00:27:23.249 app[3d8d7669b1dd68] lax [info] lsvd: Disconnecting the kernel from the device. 2023-10-11T00:27:23.253 app[3d8d7669b1dd68] lax [info] 2023/10/11 00:27:23 listening on [fdaa:1:53b:a7b:1e4:e173:d8c1:2]:22 (DNS: [fdaa::3]:53) 2023-10-11T00:27:23.262 app[3d8d7669b1dd68] lax [info] lsvd: Flushing remaining writes to object storage. 2023-10-11T00:27:23.779 app[3d8d7669b1dd68] lax [info] lsvd: Device has shut down. 2023-10-11T00:27:23.784 app[3d8d7669b1dd68] lax [info] WARN lsvd exited, pid: 304, status: exit status: 0 2023-10-11T00:27:24.786 app[3d8d7669b1dd68] lax [info] [ 8.877815] reboot: Restarting system 2023-10-11T00:27:25.012 runner[3d8d7669b1dd68] lax [info] machine restart policy set to 'no', not restarting </code></pre> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="1" /> <span class='post-likes'>1 Like</span> </div> </div> <div id='post_12' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/MatthewIngwersen'><span itemprop='name'>MatthewIngwersen</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2023-10-11T16:38:43Z' class='post-time'> October 11, 2023, 4:38pm </time> <meta itemprop='dateModified' content='2023-10-11T16:38:43Z'> <span itemprop='position'>12</span> </span> </div> <div class='post' itemprop='text'> <p><a class="mention" href="/u/kot">@kot</a> <a class="mention" href="/u/pier">@pier</a> at this (experimental) stage there’s no special pricing for this feature, so for billing purposes the <code>lsvd</code> process in your Machine is indistinguishable from your app itself. Therefore, outbound data transfer from the <code>lsvd</code> process is billed as usual. Likewise, if you increase your Machine’s CPU or RAM to accommodate the <code>lsvd</code> process, that will be billed as usual. There’s no cost for simply enabling the <code>lsvd</code> process though!</p> <p>It’s hard to say this early what might be different pricing-wise for future a GA version.</p> <p><a class="mention" href="/u/mileszim">@mileszim</a> it’s awesome to see you using this, and I’m glad that you figured it out! (Thanks for posting the solution too!)</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="4" /> <span class='post-likes'>4 Likes</span> </div> </div> <div id='post_13' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/smorimoto'><span itemprop='name'>smorimoto</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2023-11-10T23:57:35Z' class='post-time'> November 10, 2023, 11:57pm </time> <meta itemprop='dateModified' content='2023-11-10T23:57:35Z'> <span itemprop='position'>13</span> </span> </div> <div class='post' itemprop='text'> <p>I tried to use it with Cloudflare R2 and it just failed with this error. Maybe because it needs to use the region value “global”?</p> <pre><code class="lang-auto">thread 'main' panicked at 'removal index (is 0) should be < len (is 0)', init/src/main.rs:548:25 </code></pre> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_14' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/smorimoto'><span itemprop='name'>smorimoto</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2023-11-11T14:30:48Z' class='post-time'> November 11, 2023, 2:30pm </time> <meta itemprop='dateModified' content='2023-11-11T14:30:48Z'> <span itemprop='position'>14</span> </span> </div> <div class='post' itemprop='text'> <p>By the way, if anyone wants to use this feature as shared volume, I would recommend Cloudflare R2 to many people because it works like the earth region. Of course, it may depend on your technical limitations <img src="https://emoji.discourse-cdn.com/twitter/man_shrugging.png?v=12" title=":man_shrugging:" class="emoji" alt=":man_shrugging:" loading="lazy" width="20" height="20"></p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_15' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/MatthewIngwersen'><span itemprop='name'>MatthewIngwersen</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2023-11-13T21:16:26Z' class='post-time'> November 13, 2023, 9:16pm </time> <meta itemprop='dateModified' content='2023-11-13T21:16:26Z'> <span itemprop='position'>15</span> </span> </div> <div class='post' itemprop='text'> <p><a class="mention" href="/u/smorimoto">@smorimoto</a> I believe that that happens when no command is set for your machine to run (either explicitly through flyctl or the Machines API, or with a <code>CMD</code> or <code>ENTRYPOINT</code> in the <code>Dockerfile</code>). If you’re still encountering this, can you double-check that?</p> <p>(Also, when I get the chance, I’ll see about replacing that crash with a proper error message!)</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_16' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/smorimoto'><span itemprop='name'>smorimoto</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2023-11-14T15:26:30Z' class='post-time'> November 14, 2023, 3:26pm </time> <meta itemprop='dateModified' content='2023-11-14T15:26:30Z'> <span itemprop='position'>16</span> </span> </div> <div class='post' itemprop='text'> <p><a class="mention" href="/u/matthewingwersen">@MatthewIngwersen</a> Indeed, that fixed the thing! But I’m encountering a new one… I will come back here if it doesn’t get fixed with some more debugging.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_17' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/smorimoto'><span itemprop='name'>smorimoto</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2023-11-14T15:27:40Z' class='post-time'> November 14, 2023, 3:27pm </time> <meta itemprop='dateModified' content='2023-11-14T15:27:40Z'> <span itemprop='position'>17</span> </span> </div> <div class='post' itemprop='text'> <p>Anyway, this is really great and I can’t wait to see this released as a stable release soon!</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="0" /> <span class='post-likes'></span> </div> </div> <div id='post_18' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/Alex21'><span itemprop='name'>Alex21</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2024-03-03T23:19:51Z' class='post-time'> March 3, 2024, 11:19pm </time> <meta itemprop='dateModified' content='2024-03-03T23:19:51Z'> <span itemprop='position'>18</span> </span> </div> <div class='post' itemprop='text'> <p>Is there anything interesting in the works here now with Tigris being integrated on Fly? <a class="mention" href="/u/matthewingwersen">@MatthewIngwersen</a>: I’m guessing you already know about <a href="https://objectivefs.com/" rel="noopener nofollow ugc">https://objectivefs.com/</a> and <a href="https://cuno.io/" rel="noopener nofollow ugc">https://cuno.io/</a>? Those are proprietary software but perhaps can provide some inspiration for either architecture or performance benchmarks?</p> <p>I think the killer feature would be if you could provision a smallish volume per machine that could act as a cache that’s backed by Tigris, allowing each machine to see the same, globally (eventually?) consistent, bottomless filesystem, but with 0 latency access (and fewer billed Tigris requests) to frequently read files <img src="https://emoji.discourse-cdn.com/twitter/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="5" /> <span class='post-likes'>5 Likes</span> </div> </div> <div id='post_19' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/ovaistariq'><span itemprop='name'>ovaistariq</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2024-03-04T23:23:37Z' class='post-time'> March 4, 2024, 11:23pm </time> <meta itemprop='dateModified' content='2024-03-04T23:23:37Z'> <span itemprop='position'>19</span> </span> </div> <div class='post' itemprop='text'> <p>Bottomless storage backed into Fly machines would open up many interesting use cases for sure.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="2" /> <span class='post-likes'>2 Likes</span> </div> </div> <div id='post_20' itemprop='comment' itemscope itemtype='http://schema.org/Comment' class='topic-body crawler-post'> <div class='crawler-post-meta'> <span class="creator" itemprop="author" itemscope itemtype="http://schema.org/Person"> <a itemprop="url" href='https://community.fly.io/u/MatthewIngwersen'><span itemprop='name'>MatthewIngwersen</span></a> </span> <span class="crawler-post-infos"> <time itemprop='datePublished' datetime='2024-03-21T18:00:02Z' class='post-time'> March 21, 2024, 6:00pm </time> <meta itemprop='dateModified' content='2024-03-21T18:00:02Z'> <span itemprop='position'>20</span> </span> </div> <div class='post' itemprop='text'> <p>Sorry for the (very) late reply here! We haven’t done anything specific with S3-backed volumes + Tigris yet, but I’m hoping to make some time to test-drive the combination soon.</p> <p>Caching on smaller local volumes is something on our radar too. <a class="mention" href="/u/alex21">@Alex21</a>, if you’re willing, I’d love to hear if there are any specific use-cases or problems that made you think about this feature. (It helps us figure out what we should prioritize building!) And thanks for the links too!</p> <p>Also just wanted to note that I pushed out a very small update yesterday—the <code>lsvd</code> binary that runs in your Machines used to depend on the GNU C library, so it wouldn’t work with some images (particularly anything Alpine-based). This has been fixed. You should now be able to run S3-backed volumes on Alpine images, images built <code>FROM scratch</code>, even <code>registry.k8s.io/pause</code> if you are so inclined <img src="https://emoji.discourse-cdn.com/twitter/laughing.png?v=12" title=":laughing:" class="emoji" alt=":laughing:" loading="lazy" width="20" height="20">.</p> </div> <div itemprop="interactionStatistic" itemscope itemtype="http://schema.org/InteractionCounter"> <meta itemprop="interactionType" content="http://schema.org/LikeAction"/> <meta itemprop="userInteractionCount" content="2" /> <span class='post-likes'>2 Likes</span> </div> </div> </div> <div role='navigation' itemscope itemtype='http://schema.org/SiteNavigationElement' class="topic-body crawler-post"> <span itemprop='name'><b><a rel="next" itemprop="url" href="/t/bottomless-s3-backed-volumes/15648?page=2">next page →</a></b></span> </div> <div id="related-topics" class="more-topics__list " role="complementary" aria-labelledby="related-topics-title"> <h3 id="related-topics-title" class="more-topics__list-title"> Related topics </h3> <div class="topic-list-container" itemscope itemtype='http://schema.org/ItemList'> <meta itemprop='itemListOrder' content='http://schema.org/ItemListOrderDescending'> <table class='topic-list'> <thead> <tr> <th>Topic</th> <th></th> <th class="replies">Replies</th> <th class="views">Views</th> <th>Activity</th> </tr> </thead> <tbody> <tr class="topic-list-item" id="topic-list-item-20792"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='1'> <span class="link-top-line"> <a itemprop='url' href='https://community.fly.io/t/tigris-backed-volumes/20792' class='title raw-link raw-topic-link'>Tigris-backed volumes</a> </span> <div class="link-bottom-line"> <a href='/c/fresh-produce/27' class='badge-wrapper bullet'> <span class='badge-category-bg' style='background-color: #A461EF'></span> <span class='badge-category clear-badge'> <span class='category-name'>Fresh Produce</span> </span> </a> <div class="discourse-tags"> <a href='https://community.fly.io/tag/storage' class='discourse-tag'>storage</a> , <a href='https://community.fly.io/tag/volumes' class='discourse-tag'>volumes</a> , <a href='https://community.fly.io/tag/tigris' class='discourse-tag'>tigris</a> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>3</span> </td> <td class="views"> <span class='views' title='views'>896</span> </td> <td> November 28, 2024 </td> </tr> <tr class="topic-list-item" id="topic-list-item-332"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='2'> <span class="link-top-line"> <a itemprop='url' href='https://community.fly.io/t/preview-persistent-disks-for-fly-apps/332' class='title raw-link raw-topic-link'>Preview: persistent disks for Fly apps</a> </span> <div class="link-bottom-line"> <div class="discourse-tags"> <a href='https://community.fly.io/tag/volumes' class='discourse-tag'>volumes</a> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>29</span> </td> <td class="views"> <span class='views' title='views'>4807</span> </td> <td> August 8, 2022 </td> </tr> <tr class="topic-list-item" id="topic-list-item-8784"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='3'> <span class="link-top-line"> <a itemprop='url' href='https://community.fly.io/t/increase-disk-capacity-on-vm/8784' class='title raw-link raw-topic-link'>Increase disk capacity on VM</a> </span> <div class="link-bottom-line"> <a href='/c/questions-and-help/11' class='badge-wrapper bullet'> <span class='badge-category-bg' style='background-color: #0088CC'></span> <span class='badge-category clear-badge'> <span class='category-name'>Questions / Help</span> </span> </a> <div class="discourse-tags"> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>3</span> </td> <td class="views"> <span class='views' title='views'>1083</span> </td> <td> May 4, 2024 </td> </tr> <tr class="topic-list-item" id="topic-list-item-3226"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='4'> <span class="link-top-line"> <a itemprop='url' href='https://community.fly.io/t/how-do-i-upgrade-a-volume-to-a-larger-size-and-keep-its-data/3226' class='title raw-link raw-topic-link'>How do I upgrade a volume to a larger size and KEEP its data?</a> </span> <div class="link-bottom-line"> <div class="discourse-tags"> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>8</span> </td> <td class="views"> <span class='views' title='views'>1337</span> </td> <td> November 25, 2021 </td> </tr> <tr class="topic-list-item" id="topic-list-item-6170"> <td class="main-link" itemprop='itemListElement' itemscope itemtype='http://schema.org/ListItem'> <meta itemprop='position' content='5'> <span class="link-top-line"> <a itemprop='url' href='https://community.fly.io/t/volumes-larger-than-500-gb/6170' class='title raw-link raw-topic-link'>volumes larger than 500 gb</a> </span> <div class="link-bottom-line"> <a href='/c/questions-and-help/11' class='badge-wrapper bullet'> <span class='badge-category-bg' style='background-color: #0088CC'></span> <span class='badge-category clear-badge'> <span class='category-name'>Questions / Help</span> </span> </a> <div class="discourse-tags"> </div> </div> </td> <td class="replies"> <span class='posts' title='posts'>3</span> </td> <td class="views"> <span class='views' title='views'>524</span> </td> <td> February 28, 2024 </td> </tr> </tbody> </table> </div> </div> </div> <footer class="container wrap"> <nav class='crawler-nav'> <ul> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/' itemprop="url">Home </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/categories' itemprop="url">Categories </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/guidelines' itemprop="url">Guidelines </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/tos' itemprop="url">Terms of Service </a> </span> </li> <li itemscope itemtype='http://schema.org/SiteNavigationElement'> <span itemprop='name'> <a href='/privacy' itemprop="url">Privacy Policy </a> </span> </li> </ul> </nav> <p class='powered-by-link'>Powered by <a href="https://www.discourse.org">Discourse</a>, best viewed with JavaScript enabled</p> </footer> <div class="buorg"><div>Unfortunately, <a href="https://www.discourse.org/faq/#browser">your browser is unsupported</a>. Please <a href="https://browsehappy.com">switch to a supported browser</a> to view rich content, log in and reply.</div></div> </body> </html>