CINXE.COM

<!doctype html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <link rel="stylesheet" href="/css/layout.css?1732633144785"> <link rel="stylesheet" href="/css/nav.css?1732633144785"> <link rel="stylesheet" href="/css/style.css?1732633144785"> <link rel="stylesheet" href="/css/prismjs@1.24-themes-prism.css"> <title>IPLD ♦ How IPFS Web Gateways Work</title> </head> <body> <header> <div class="sidebar-button" onclick="document.getElementById('sidebar').classList.toggle('sidebar-open')"> <svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" role="img" viewBox="0 0 448 512" class="icon"> <path fill="currentColor" d="M436 124H12c-6.627 0-12-5.373-12-12V80c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12zm0 160H12c-6.627 0-12-5.373-12-12v-32c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12zm0 160H12c-6.627 0-12-5.373-12-12v-32c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12z"></path> </svg> </div> <a href="/" class="logo">IPLD</a> <aside id=breadcrumbs> <ul> <li><a href="/docs">docs</a></li> <li><a href="/docs/synthesis">synthesis</a></li> <li><a href="/docs/synthesis/how-ipfs-web-gateways-work/">how-ipfs-web-gateways-work</a></li> </ul> </aside> </header> <aside id=sidebar> <nav> <ul> <li> <a href="/docs/">Docs</a><ul> <li> <a href="/docs/intro/">Intro</a><ul> <li> <a href="/docs/intro/hello-world/">Hello, World</a></li> <li> <a href="/docs/intro/primer/">The Brief Primer</a></li> <li> <a href="/docs/intro/ecosystem/">InterPlanetary Ecosystem Overview</a></li> <li> <a href="/docs/intro/community/">Finding Community</a></li></ul></li> <li> <a href="/docs/motivation/">Motivation</a><ul> <li> <a href="/docs/motivation/benefits-of-content-addressing/">Benefits of Content Addressing</a></li> <li> <a href="/docs/motivation/data-to-data-structures/">From Data to Data Structures</a></li></ul></li> <li> <a href="/docs/codecs/">Codecs</a><ul> <li> <a href="/docs/codecs/known/">Known Codecs</a><ul> <li> <a href="/docs/codecs/known/dag-cbor/">DAG-CBOR</a></li> <li> <a href="/docs/codecs/known/dag-json/">DAG-JSON</a></li> <li> <a href="/docs/codecs/known/dag-pb/">DAG-PB</a></li></ul></li></ul></li> <li> <a href="/docs/data-model/">Data Model</a><ul> <li> <a href="/docs/data-model/node/">Nodes</a></li> <li> <a href="/docs/data-model/kinds/">Kinds</a></li> <li> <a href="/docs/data-model/pathing/">Pathing</a></li> <li> <a href="/docs/data-model/traversal/">Traversal</a></li></ul></li> <li> <a href="/docs/advanced-data-layouts/">Advanced Data Layouts</a><ul> <li> <a href="/docs/advanced-data-layouts/intro/">Intro to ADLs</a></li> <li> <a href="/docs/advanced-data-layouts/naming/">ADL Naming</a></li> <li> <a href="/docs/advanced-data-layouts/signalling/">Signalling ADLs</a></li> <li> <a href="/docs/advanced-data-layouts/dynamic-loading/">Dynamic Loading</a></li> <li> <a href="/docs/advanced-data-layouts/known/">Known ADLs</a></li></ul></li> <li> <a href="/docs/schemas/">Schemas</a><ul> <li> <a href="/docs/schemas/intro/">Introduction</a><ul> <li> <a href="/docs/schemas/intro/compare/">compare</a></li> <li> <a href="/docs/schemas/intro/goals/">Goals</a></li> <li> <a href="/docs/schemas/intro/feature-summary/">Feature Summary</a></li></ul></li> <li> <a href="/docs/schemas/features/">Features</a><ul> <li> <a href="/docs/schemas/features/typekinds/">Type Kinds</a></li> <li> <a href="/docs/schemas/features/representation-strategies/">Representation Strategies</a></li> <li> <a href="/docs/schemas/features/links/">Links</a></li> <li> <a href="/docs/schemas/features/indicating-adls/">Using ADLs in Schemas</a></li></ul></li> <li> <a href="/docs/schemas/using/">Using Wisely</a><ul> <li> <a href="/docs/schemas/using/authoring-guide/">Authoring Guide</a></li> <li> <a href="/docs/schemas/using/migrations/">Migrations</a></li></ul></li></ul></li> <li> <a href="/docs/synthesis/">Synthesis</a><ul> <li> <a href="/docs/synthesis/gtd/">Getting Things Done</a></li> <li> <a href="/docs/synthesis/building-in-alignment/">Building in Alignment</a></li> <li class="active-page"> <a href="/docs/synthesis/how-ipfs-web-gateways-work/">How IPFS Web Gateways Work</a></li> <li> <a href="/docs/synthesis/encryption/">Working With Encryption</a></li></ul></li></ul></li> <li> <a href="/specs/">Specs</a><ul> <li> <a href="/specs/about/">About the Specifications</a></li> <li> <a href="/specs/codecs/">Codecs</a><ul> <li> <a href="/specs/codecs/dag-cbor/">DAG-CBOR</a><ul> <li> <a href="/specs/codecs/dag-cbor/spec/">Spec</a></li> <li> <a href="/specs/codecs/dag-cbor/fixtures/">DAG-CBOR Test Fixtures</a><ul> <li> <a href="/specs/codecs/dag-cbor/fixtures/cross-codec/">cross-codec</a></li></ul></li></ul></li> <li> <a href="/specs/codecs/dag-cosmos/">DAG-COSMOS</a><ul> <li> <a href="/specs/codecs/dag-cosmos/basic_types/">basic_types</a></li> <li> <a href="/specs/codecs/dag-cosmos/cosmos_state/">cosmos_state</a></li> <li> <a href="/specs/codecs/dag-cosmos/crypto_types/">crypto_types</a></li> <li> <a href="/specs/codecs/dag-cosmos/tendermint_chain/">tendermint_chain</a></li> <li> <a href="/specs/codecs/dag-cosmos/typed_protobuf/">typed_protobuf</a></li></ul></li> <li> <a href="/specs/codecs/dag-eth/">DAG-ETH</a><ul> <li> <a href="/specs/codecs/dag-eth/basic_types/">basic_types</a></li> <li> <a href="/specs/codecs/dag-eth/chain/">chain</a></li> <li> <a href="/specs/codecs/dag-eth/convenience_types/">convenience_types</a></li> <li> <a href="/specs/codecs/dag-eth/state/">state</a></li></ul></li> <li> <a href="/specs/codecs/dag-jose/">DAG-JOSE</a><ul> <li> <a href="/specs/codecs/dag-jose/spec/">Spec</a></li> <li> <a href="/specs/codecs/dag-jose/fixtures/">fixtures</a></li></ul></li> <li> <a href="/specs/codecs/dag-json/">DAG-JSON</a><ul> <li> <a href="/specs/codecs/dag-json/spec/">Spec</a></li> <li> <a href="/specs/codecs/dag-json/fixtures/">DAG-JSON Test Fixtures</a><ul> <li> <a href="/specs/codecs/dag-json/fixtures/cross-codec/">cross-codec</a></li></ul></li></ul></li> <li> <a href="/specs/codecs/dag-pb/">DAG-PB</a><ul> <li> <a href="/specs/codecs/dag-pb/spec/">Spec</a></li> <li> <a href="/specs/codecs/dag-pb/fixtures/">DAG-PB Test Fixtures</a><ul> <li> <a href="/specs/codecs/dag-pb/fixtures/cross-codec/">cross-codec</a></li></ul></li></ul></li></ul></li> <li> <a href="/specs/advanced-data-layouts/">Advanced Data Layouts</a><ul> <li> <a href="/specs/advanced-data-layouts/fbl/">FBL ADL</a><ul> <li> <a href="/specs/advanced-data-layouts/fbl/spec/">spec</a></li></ul></li> <li> <a href="/specs/advanced-data-layouts/hamt/">HAMT ADL</a><ul> <li> <a href="/specs/advanced-data-layouts/hamt/spec/">spec</a></li> <li> <a href="/specs/advanced-data-layouts/hamt/fixture/">HashMap (HAMT) Test Fixtures</a><ul> <li> <a href="/specs/advanced-data-layouts/hamt/fixture/alice-words/">alice-words</a></li></ul></li></ul></li></ul></li> <li> <a href="/specs/schemas/">Schemas</a><ul> <li> <a href="/specs/schemas/prelude/">prelude</a></li></ul></li> <li> <a href="/specs/transport/">Transports</a><ul> <li> <a href="/specs/transport/car/">CAR</a><ul> <li> <a href="/specs/transport/car/carv1/">CARv1 Specification</a></li> <li> <a href="/specs/transport/car/carv2/">CARv2 Specification</a></li> <li> <a href="/specs/transport/car/fixture/">CAR Test Fixtures</a><ul> <li> <a href="/specs/transport/car/fixture/carv1-basic/">carv1-basic</a></li> <li> <a href="/specs/transport/car/fixture/carv2-basic/">carv2-basic</a></li></ul></li></ul></li> <li> <a href="/specs/transport/graphsync/">Graphsync</a><ul> <li> <a href="/specs/transport/graphsync/known_extensions/">known_extensions</a></li></ul></li> <li> <a href="/specs/transport/trustless-pathing/">Trustless Pathing</a><ul> <li> <a href="/specs/transport/trustless-pathing/fixtures/">Trustless Pathing Fixtures</a><ul> <li> <a href="/specs/transport/trustless-pathing/fixtures/unixfs_20m_variety/">unixfs_20m_variety</a></li></ul></li></ul></li></ul></li> <li> <a href="/specs/selectors/">Selectors</a><ul> <li> <a href="/specs/selectors/fixtures/">fixtures</a><ul> <li> <a href="/specs/selectors/fixtures/selector-fixtures-1/">selector-fixtures-1</a></li> <li> <a href="/specs/selectors/fixtures/selector-fixtures-adl/">selector-fixtures-adl</a></li> <li> <a href="/specs/selectors/fixtures/selector-fixtures-recursion/">selector-fixtures-recursion</a></li></ul></li></ul></li> <li> <a href="/specs/patch/">Patch</a><ul> <li> <a href="/specs/patch/fixtures/">IPLD Patch Test Fixtures</a><ul> <li> <a href="/specs/patch/fixtures/fixtures-1/">fixtures-1</a></li></ul></li></ul></li></ul></li> <li> <a href="/libraries/">Libraries</a><ul> <li> <a href="/libraries/golang/">Golang</a></li> <li> <a href="/libraries/javascript/">JavaScript</a></li> <li> <a href="/libraries/python/">Python</a></li> <li> <a href="/libraries/rust/">Rust</a></li></ul></li> <li> <a href="/design/">Design</a><ul> <li> <a href="/design/objectives/">Objectives</a></li> <li> <a href="/design/concepts/">Concepts</a><ul> <li> <a href="/design/concepts/type-theory-glossary/">type-theory-glossary</a></li></ul></li> <li> <a href="/design/libraries/">Libraries</a><ul> <li> <a href="/design/libraries/nodes-and-kinds/">nodes-and-kinds</a></li></ul></li> <li> <a href="/design/tricky-choices/">Tricky Choices</a><ul> <li> <a href="/design/tricky-choices/dag-pb-forms-impl-and-use/">dag-pb-forms-impl-and-use</a></li> <li> <a href="/design/tricky-choices/map-key-domain/">map-key-domain</a></li> <li> <a href="/design/tricky-choices/numeric-domain/">numeric-domain</a></li> <li> <a href="/design/tricky-choices/ordering/">ordering</a></li> <li> <a href="/design/tricky-choices/string-domain/">string-domain</a></li></ul></li> <li> <a href="/design/open-research/">Open Research</a><ul> <li> <a href="/design/open-research/ADL-autoexecution/">ADL autoexecution</a></li></ul></li></ul></li> <li> <a href="/tools/">Tools</a></li> <li> <a href="/glossary/">Glossary</a></li> <li> <a href="/media/">Media</a></li> <li> <a href="/FAQ/">FAQ</a></li></ul> </nav> </aside> <main> <div class=content> <h1>How IPFS Web Gateways Work</h1> <p>This document talks through how IPFS web gateways work; the various forms of abstraction used in accomplishing parts of that task; and how and why these designs are cool (and how you can reuse parts of them).</p> <div class="callout callout-info"> <blockquote> <p>Note! This is an <em>aspirational</em> document -- IPFS web gateways are a feature which has substantial history. They haven't always worked like this.</p> </blockquote> </div> <h2 id="what-are-the-goals" tabindex="-1"><a class="header-anchor" href="#what-are-the-goals">What are the goals?</a></h2> <h3 id="what-are-ipfs-web-gateways-anyway" tabindex="-1"><a class="header-anchor" href="#what-are-ipfs-web-gateways-anyway">What are IPFS Web Gateways, anyway?</a></h3> <p>The "web gateway" is a feature of IPFS that serves the contents of <em>files</em> over <em>HTTP</em>. It selects what content to serve by taking the path from the URL requested, and splitting it into two parts: the <em>CID</em> and the remainder of the <em>path</em>.</p> <p>The CID is used to pick some piece of data to start at. The remainder of the path will be used as instructions to traverse IPLD data, starting from that data which the CID identified. At the end of that traversal, we'll try to understand whatever is pointed to as a bunch of bytes (a "file" isn't much more than that)... and pour it out into an HTTP response.</p> <p>This is useful because it means we can store data in IPFS, and that data can become easily accessible as normal webpages.</p> <h2 id="how-do-we-do-it" tabindex="-1"><a class="header-anchor" href="#how-do-we-do-it">How do we do it?</a></h2> <p>There's actually a ton of moving parts to this. Let's break it down:</p> <h3 id="files-are-just-bytes" tabindex="-1"><a class="header-anchor" href="#files-are-just-bytes">files are just bytes</a></h3> <p>"Files" are really just "large byte sequences".</p> <p>... for the most part. Let's accept this simplification for a minute. (We'll add back in concepts about permissions and metadata later.)</p> <h3 id="large-bytes-are-provided-by-adls-fbl-specifically" tabindex="-1"><a class="header-anchor" href="#large-bytes-are-provided-by-adls-fbl-specifically">large bytes are provided by ADLs (FBL, specifically)</a></h3> <p>We can use the <a href="/specs/advanced-data-layouts/fbl/">"Flexible Byte Layout (FBL)" ADL</a> to describe file bodies. This ADL provides a generalized way to read sharded large byte sequences. It even supports efficient seeking!</p> <h3 id="directories-are-just-maps" tabindex="-1"><a class="header-anchor" href="#directories-are-just-maps">directories are just maps</a></h3> <p>"Directories" are really just "maps": a "directory" maps a filename to either another directory, or a file; that's it.</p> <p>... for the most part. Let's accept this simplification for a minute. (We'll add back in concepts about permissions and metadata later.)</p> <h3 id="directories-are-provided-by-adls-hamt-specifically" tabindex="-1"><a class="header-anchor" href="#directories-are-provided-by-adls-hamt-specifically">directories are provided by ADLs (HAMT, specifically)</a></h3> <p>We can use the <a href="/specs/advanced-data-layouts/hamt/">HAMT ADL</a> to describe directories. This ADL gives us sharded maps, which means we can store directories of arbitrary size.</p> <h3 id="navigating-the-path-is-defined-by-the-data-model" tabindex="-1"><a class="header-anchor" href="#navigating-the-path-is-defined-by-the-data-model">navigating the path is defined by the Data Model</a></h3> <p>... which is neat, because it means we don't need any special custom logic to describe how we're going to walk over this data. Because the path traversals are all going through ADLs, as we just described in the previous sections, that's... all there is to it.</p> <p>Because having an ADL definition means we have a way make the ADL's <a href="/glossary/#substrate">substrate</a> data legible as regular IPLD Data Model semantics, we can use standard library functions for pathing over this data. No special path handling logic is written for the Web Gateways (or other users of unixfsv2) at all!</p> <h3 id="knowing-where-to-use-adls-and-which-ones-is-defined-by-ipld-schemas" tabindex="-1"><a class="header-anchor" href="#knowing-where-to-use-adls-and-which-ones-is-defined-by-ipld-schemas">knowing where to use ADLs, and which ones, is defined by IPLD Schemas</a></h3> <p>... and more specifically, the Unixfsv2 schema, which is pretty common and used by much more than the IPFS web gateway system.</p> <p>The web gateways know about the schema for unixfsv2, and they assume that it applies to whatever data is being requested through the gateway system.</p> <p>If the data doesn't match, it won't be renderable through the web gateway (and we return errors about how and where the data ceased to match the schema that's expected for unixfsv2 data).</p> <p>If the data does match: then the schema also tells us where to expect ADLs to come into effect, and also tells us what Data Model semantics to expect from the data that is made available once the ADL logic is applied.</p> <h3 id="different-parts-of-the-system-can-use-different-views" tabindex="-1"><a class="header-anchor" href="#different-parts-of-the-system-can-use-different-views">different parts of the system can use different views</a></h3> <div class="callout callout-todo"> <ul> <li>Directories are actually a map pointing to metadata and to more filenodes.</li> <li>Sometimes you use the plain HAMT ADL and see all of that;</li> <li>Sometimes you use an ADL that steps past the HAMT <strong>and</strong> directly to the filenode.</li> <li>The former is necessary for writing new data.</li> <li>The latter provides the kind of pathing that an end-user wants when navigating and reading like a filesystem!</li> </ul> </div> <h3 id="if-you-dont-have-this-data-we-fetch-it" tabindex="-1"><a class="header-anchor" href="#if-you-dont-have-this-data-we-fetch-it">if you don't have this data: we fetch it</a></h3> <p>This part hits the limits of what is directly in the scope of IPLD, but in brief, what happens in IPFS is:</p> <ul> <li>a request for info about anyone who might have the missing content will be made to the DHT (provided by libp2p)</li> <li>contact with those other IPFS nodes will be made (again brokered by libp2p, which might be handling NAT traversal, etc)</li> <li>a protocol like bitswap or graphsync will be used to ferry raw blocks of IPLD data from those other IPFS nodes</li> <li>... and then the data is local: the process picks up again normally from there.</li> </ul> <h2 id="what-if-we-want-to-peek-behind-the-curtain" tabindex="-1"><a class="header-anchor" href="#what-if-we-want-to-peek-behind-the-curtain">What if we want to peek behind the curtain?</a></h2> <p>All of the structures above are serialized in CBOR. All the magic comes from IPLD semantic features like ADLs and Schemas.</p> <p>What's neat about this is it's also very easy to drop all the magic and look directly at the raw data.</p> <p>Want to see the raw CBOR objects that make up all these structures? No problem. They're hidden just barely under the hood, and we try to make it easy to lift that hood up and take a peek.</p> <p>In fact, you can inspect all the raw CBOR objects making up these structures simply by using IPLD libraries to start at the same CID, and then use regular Data Model traversal mechanisms to stroll around. If you provide the Unixfsv2 schema (and thereby its hints about where to apply ADLs), you'll get the high level view of "directories" (maps) and "files" (bytes); if you don't, you'll get to stroll around the raw CBOR objects instead.</p> <h2 id="whats-reusable" tabindex="-1"><a class="header-anchor" href="#whats-reusable">What's reusable?</a></h2> <p>Gosh. Dang near everything.</p> <ul> <li>The Unixfsv2 schema is reusable. If you want to describe filesystem-like data, it's probably got what you want.</li> <li>The Flexible Byte Layout (FBL) Advanced Data Layout is reusable for any of your big-sharded-byte-sequence needs! It's not unique to Unixfsv2 nor the IPFS web gateways; you can drop it in anywhere you want in any IPLD system.</li> <li>The HAMT ADL is similarly reusable for any of your big-sharded-map needs! It's not unique to Unixfsv2 nor the IPFS web gateways; you can drop it in anywhere you want in any IPLD system.</li> <li>... even if you don't want any of these things in particular, hopefully this is a good example of how you can use the various parts of the IPLD system to accomplish your own goals!</li> </ul> <p>The whole IPFS web gateway system was built in terms of standard IPLD components, assembled: schemas, ADLs, and plain basic pathing and traversal over the Data Model were all we needed to accomplish all this.</p> <h2 id="what-other-limits-does-this-have" tabindex="-1"><a class="header-anchor" href="#what-other-limits-does-this-have">What other limits does this have?</a></h2> <p>It's important to note that the web gateways do almost all of this work using IPLD libraries and not much else... but, ADLs also have some special caveats.</p> <p>Because ADLs need some logic to make their translations of data work, and because IPLD doesn't itself include an interpreter or scripting language runtime (and even if it did, we'd be worried about capping its resource budget)... IPLD libraries typically require that a list of allowed ADLs be defined explicitly.</p> <p>The IPFS web gateways specifically allow the FBL and HAMT ADLs.</p> <p>If you wanted to produce data that used other sharding algorithms, but was otherwise structurally matching the Unixfsv2 schema, and wanted those to be correctly displayed by some IPFS web gateway? You'd need to reconfigure that gateway to expand the list of allowed ADLs to include whatever sharding system it is your wanted to use.</p> <h2 id="historically-how-did-this-previously-work" tabindex="-1"><a class="header-anchor" href="#historically-how-did-this-previously-work">Historically: how did this previously work?</a></h2> <blockquote> <p>:closed_book: This section is historical notes -- it's here for readers who were familiar with the old system, and want to be refreshed on how it compares to the new designs documented above, or for readers who want to understand some of the learning experiences gathered from previous bouts at this work. However, you shouldn't really need to read it unless one of those two sources of curiosity appeals to you.</p> </blockquote> <p>The earliest implementations of IPFS used a codec called "dag-pb" to represent files and directories. ("dag-pb" is still around as an IPLD codec -- but nowadays, we consider it one of the more limited codecs, and a very esoteric one at that.)</p> <p>The "dag-pb" codec included many responsibilities that we now handle separately: it included the concept of directories (and sharding them) in the codec itself (whereas we now split this out into map ADLs); similarly, it include the concept of files (and sharding their component bytes) in the codec itself (whereas we now split this out into bytes ADLs).</p> <p>By baking these things into the codec itself, there were several major disadvantages compared to the present IPLD system design:</p> <ul> <li>by baking these sharding mechanisms into a single codec, it was not possible to reuse them with other codecs;</li> <li>because the sharding mechanisms were baked in the codec (and at the time, there wasn't really any coherent definition of a Data Model at all), there was no way to <em>switch</em> between high level and raw views of the data -- as a result, these systems were much harder to debug;</li> <li>because the sharding mechanisms were tightly bound into the codec, there was no way to introduce or even experiment with other sharding mechanisms!</li> </ul> <p>Looking at the other direction, "dag-pb", even with all its special features, was more limited than the modern IPLD system designs:</p> <ul> <li>by extracting the concept of ADLs, we've made it possible to shard other datastructures as well -- for example, we can now easily support sharded lists, a concept which simply didn't exist in dag-pb.</li> <li>dag-pb had very (very) rigid ideas about the topology of data. In modern terms, we would say it doesn't support the complete IPLD Data Model. The huge behavioral gaps and asymmetry in representability between dag-pb and other early IPLD codecs produced a great deal of confusion.</li> </ul> </div> </main> </body> </html>