CINXE.COM
IPLD ♦ Objectives
<!doctype html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <link rel="stylesheet" href="/css/layout.css?1718759055581"> <link rel="stylesheet" href="/css/nav.css?1718759055581"> <link rel="stylesheet" href="/css/style.css?1718759055581"> <link rel="stylesheet" href="/css/prismjs@1.24-themes-prism.css"> <title>IPLD ♦ Objectives</title> </head> <body> <header> <div class="sidebar-button" onclick="document.getElementById('sidebar').classList.toggle('sidebar-open')"> <svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" role="img" viewBox="0 0 448 512" class="icon"> <path fill="currentColor" d="M436 124H12c-6.627 0-12-5.373-12-12V80c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12zm0 160H12c-6.627 0-12-5.373-12-12v-32c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12zm0 160H12c-6.627 0-12-5.373-12-12v-32c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12z"></path> </svg> </div> <a href="/" class="logo">IPLD</a> <aside id=breadcrumbs> <ul> <li><a href="/design">design</a></li> <li><a href="/design/objectives/">objectives</a></li> </ul> </aside> </header> <aside id=sidebar> <nav> <ul> <li> <a href="/docs/">Docs</a><ul> <li> <a href="/docs/intro/">Intro</a><ul> <li> <a href="/docs/intro/hello-world/">Hello, World</a></li> <li> <a href="/docs/intro/primer/">The Brief Primer</a></li> <li> <a href="/docs/intro/ecosystem/">InterPlanetary Ecosystem Overview</a></li> <li> <a href="/docs/intro/community/">Finding Community</a></li></ul></li> <li> <a href="/docs/motivation/">Motivation</a><ul> <li> <a href="/docs/motivation/benefits-of-content-addressing/">Benefits of Content Addressing</a></li> <li> <a href="/docs/motivation/data-to-data-structures/">From Data to Data Structures</a></li></ul></li> <li> <a href="/docs/codecs/">Codecs</a><ul> <li> <a href="/docs/codecs/known/">Known Codecs</a><ul> <li> <a href="/docs/codecs/known/dag-cbor/">DAG-CBOR</a></li> <li> <a href="/docs/codecs/known/dag-json/">DAG-JSON</a></li> <li> <a href="/docs/codecs/known/dag-pb/">DAG-PB</a></li></ul></li></ul></li> <li> <a href="/docs/data-model/">Data Model</a><ul> <li> <a href="/docs/data-model/node/">Nodes</a></li> <li> <a href="/docs/data-model/kinds/">Kinds</a></li> <li> <a href="/docs/data-model/pathing/">Pathing</a></li> <li> <a href="/docs/data-model/traversal/">Traversal</a></li></ul></li> <li> <a href="/docs/advanced-data-layouts/">Advanced Data Layouts</a><ul> <li> <a href="/docs/advanced-data-layouts/intro/">Intro to ADLs</a></li> <li> <a href="/docs/advanced-data-layouts/naming/">ADL Naming</a></li> <li> <a href="/docs/advanced-data-layouts/signalling/">Signalling ADLs</a></li> <li> <a href="/docs/advanced-data-layouts/dynamic-loading/">Dynamic Loading</a></li> <li> <a href="/docs/advanced-data-layouts/known/">Known ADLs</a></li></ul></li> <li> <a href="/docs/schemas/">Schemas</a><ul> <li> <a href="/docs/schemas/intro/">Introduction</a><ul> <li> <a href="/docs/schemas/intro/compare/">compare</a></li> <li> <a href="/docs/schemas/intro/goals/">Goals</a></li> <li> <a href="/docs/schemas/intro/feature-summary/">Feature Summary</a></li></ul></li> <li> <a href="/docs/schemas/features/">Features</a><ul> <li> <a href="/docs/schemas/features/typekinds/">Type Kinds</a></li> <li> <a href="/docs/schemas/features/representation-strategies/">Representation Strategies</a></li> <li> <a href="/docs/schemas/features/links/">Links</a></li> <li> <a href="/docs/schemas/features/indicating-adls/">Using ADLs in Schemas</a></li></ul></li> <li> <a href="/docs/schemas/using/">Using Wisely</a><ul> <li> <a href="/docs/schemas/using/authoring-guide/">Authoring Guide</a></li> <li> <a href="/docs/schemas/using/migrations/">Migrations</a></li></ul></li></ul></li> <li> <a href="/docs/synthesis/">Synthesis</a><ul> <li> <a href="/docs/synthesis/gtd/">Getting Things Done</a></li> <li> <a href="/docs/synthesis/building-in-alignment/">Building in Alignment</a></li> <li> <a href="/docs/synthesis/how-ipfs-web-gateways-work/">How IPFS Web Gateways Work</a></li> <li> <a href="/docs/synthesis/encryption/">Working With Encryption</a></li></ul></li></ul></li> <li> <a href="/specs/">Specs</a><ul> <li> <a href="/specs/about/">About the Specifications</a></li> <li> <a href="/specs/codecs/">Codecs</a><ul> <li> <a href="/specs/codecs/dag-cbor/">DAG-CBOR</a><ul> <li> <a href="/specs/codecs/dag-cbor/fixtures/">DAG-CBOR Test Fixtures</a><ul> <li> <a href="/specs/codecs/dag-cbor/fixtures/cross-codec/">cross-codec</a></li></ul></li> <li> <a href="/specs/codecs/dag-cbor/spec/">Spec</a></li></ul></li> <li> <a href="/specs/codecs/dag-cosmos/">DAG-COSMOS</a><ul> <li> <a href="/specs/codecs/dag-cosmos/basic_types/">basic_types</a></li> <li> <a href="/specs/codecs/dag-cosmos/cosmos_state/">cosmos_state</a></li> <li> <a href="/specs/codecs/dag-cosmos/crypto_types/">crypto_types</a></li> <li> <a href="/specs/codecs/dag-cosmos/tendermint_chain/">tendermint_chain</a></li> <li> <a href="/specs/codecs/dag-cosmos/typed_protobuf/">typed_protobuf</a></li></ul></li> <li> <a href="/specs/codecs/dag-eth/">DAG-ETH</a><ul> <li> <a href="/specs/codecs/dag-eth/basic_types/">basic_types</a></li> <li> <a href="/specs/codecs/dag-eth/chain/">chain</a></li> <li> <a href="/specs/codecs/dag-eth/convenience_types/">convenience_types</a></li> <li> <a href="/specs/codecs/dag-eth/state/">state</a></li></ul></li> <li> <a href="/specs/codecs/dag-jose/">DAG-JOSE</a><ul> <li> <a href="/specs/codecs/dag-jose/fixtures/">fixtures</a></li> <li> <a href="/specs/codecs/dag-jose/spec/">Spec</a></li></ul></li> <li> <a href="/specs/codecs/dag-json/">DAG-JSON</a><ul> <li> <a href="/specs/codecs/dag-json/fixtures/">DAG-JSON Test Fixtures</a><ul> <li> <a href="/specs/codecs/dag-json/fixtures/cross-codec/">cross-codec</a></li></ul></li> <li> <a href="/specs/codecs/dag-json/spec/">Spec</a></li></ul></li> <li> <a href="/specs/codecs/dag-pb/">DAG-PB</a><ul> <li> <a href="/specs/codecs/dag-pb/fixtures/">DAG-PB Test Fixtures</a><ul> <li> <a href="/specs/codecs/dag-pb/fixtures/cross-codec/">cross-codec</a></li></ul></li> <li> <a href="/specs/codecs/dag-pb/spec/">Spec</a></li></ul></li></ul></li> <li> <a href="/specs/advanced-data-layouts/">Advanced Data Layouts</a><ul> <li> <a href="/specs/advanced-data-layouts/fbl/">FBL ADL</a><ul> <li> <a href="/specs/advanced-data-layouts/fbl/spec/">spec</a></li></ul></li> <li> <a href="/specs/advanced-data-layouts/hamt/">HAMT ADL</a><ul> <li> <a href="/specs/advanced-data-layouts/hamt/fixture/">HashMap (HAMT) Test Fixtures</a><ul> <li> <a href="/specs/advanced-data-layouts/hamt/fixture/alice-words/">alice-words</a></li></ul></li> <li> <a href="/specs/advanced-data-layouts/hamt/spec/">spec</a></li></ul></li></ul></li> <li> <a href="/specs/schemas/">Schemas</a><ul> <li> <a href="/specs/schemas/prelude/">prelude</a></li></ul></li> <li> <a href="/specs/transport/">Transports</a><ul> <li> <a href="/specs/transport/car/">CAR</a><ul> <li> <a href="/specs/transport/car/carv1/">CARv1 Specification</a></li> <li> <a href="/specs/transport/car/carv2/">CARv2 Specification</a></li> <li> <a href="/specs/transport/car/fixture/">CAR Test Fixtures</a><ul> <li> <a href="/specs/transport/car/fixture/carv1-basic/">carv1-basic</a></li> <li> <a href="/specs/transport/car/fixture/carv2-basic/">carv2-basic</a></li></ul></li></ul></li> <li> <a href="/specs/transport/graphsync/">Graphsync</a><ul> <li> <a href="/specs/transport/graphsync/known_extensions/">known_extensions</a></li></ul></li> <li> <a href="/specs/transport/trustless-pathing/">Trustless Pathing</a><ul> <li> <a href="/specs/transport/trustless-pathing/fixtures/">Trustless Pathing Fixtures</a><ul> <li> <a href="/specs/transport/trustless-pathing/fixtures/unixfs_20m_variety/">unixfs_20m_variety</a></li></ul></li></ul></li></ul></li> <li> <a href="/specs/selectors/">Selectors</a><ul> <li> <a href="/specs/selectors/fixtures/">fixtures</a><ul> <li> <a href="/specs/selectors/fixtures/selector-fixtures-1/">selector-fixtures-1</a></li> <li> <a href="/specs/selectors/fixtures/selector-fixtures-adl/">selector-fixtures-adl</a></li> <li> <a href="/specs/selectors/fixtures/selector-fixtures-recursion/">selector-fixtures-recursion</a></li></ul></li></ul></li> <li> <a href="/specs/patch/">Patch</a><ul> <li> <a href="/specs/patch/fixtures/">IPLD Patch Test Fixtures</a><ul> <li> <a href="/specs/patch/fixtures/fixtures-1/">fixtures-1</a></li></ul></li></ul></li></ul></li> <li> <a href="/libraries/">Libraries</a><ul> <li> <a href="/libraries/golang/">Golang</a></li> <li> <a href="/libraries/javascript/">JavaScript</a></li> <li> <a href="/libraries/python/">Python</a></li> <li> <a href="/libraries/rust/">Rust</a></li></ul></li> <li> <a href="/design/">Design</a><ul> <li class="active-page"> <a href="/design/objectives/">Objectives</a></li> <li> <a href="/design/concepts/">Concepts</a><ul> <li> <a href="/design/concepts/type-theory-glossary/">type-theory-glossary</a></li></ul></li> <li> <a href="/design/libraries/">Libraries</a><ul> <li> <a href="/design/libraries/nodes-and-kinds/">nodes-and-kinds</a></li></ul></li> <li> <a href="/design/tricky-choices/">Tricky Choices</a><ul> <li> <a href="/design/tricky-choices/dag-pb-forms-impl-and-use/">dag-pb-forms-impl-and-use</a></li> <li> <a href="/design/tricky-choices/map-key-domain/">map-key-domain</a></li> <li> <a href="/design/tricky-choices/numeric-domain/">numeric-domain</a></li> <li> <a href="/design/tricky-choices/ordering/">ordering</a></li> <li> <a href="/design/tricky-choices/string-domain/">string-domain</a></li></ul></li> <li> <a href="/design/open-research/">Open Research</a><ul> <li> <a href="/design/open-research/ADL-autoexecution/">ADL autoexecution</a></li></ul></li></ul></li> <li> <a href="/tools/">Tools</a></li> <li> <a href="/glossary/">Glossary</a></li> <li> <a href="/media/">Media</a></li> <li> <a href="/FAQ/">FAQ</a></li></ul> </nav> </aside> <main> <div class=content> <h1>Objectives and Scope of IPLD</h1> <h2 id="vision" tabindex="-1"><a class="header-anchor" href="#vision">Vision</a></h2> <p>We want to see more decentralized designs appearing in technology.</p> <p>We believe specifications for data exchange, design languages that work in decentralized contexts, and content-addressing (hash-linking) of data are all a key part of enabling that.</p> <p>IPLD is a body of work to move forward on all those fronts.</p> <h3 id="concretely" tabindex="-1"><a class="header-anchor" href="#concretely">Concretely</a></h3> <p>The goal of IPLD is to make it possible to build "the next git" in hours, instead of days. (Or weeks, instead of months: whatever the original timescale is, slide it down a whole order of magnitude.)</p> <p>We aim to do this by taking a bunch of the practical problems you need to solve to build a decentralized / content-addressed system, and turn them into non-issues: either by making solutions so good there's no need to argue with them (CIDs); or, by making it into a pluggable system so that you can make one choice now, and defer worrying about it again until much later, and have agility "for free" whenever it becomes necessary to change (multicodecs, multihashes, etc).</p> <p>On top of that, we start adding superpowers with more library layers: ADLs provide sharding and scaling; Schemas provide a design language and ways to talk about data migrations; and features like Selectors to describe graph walks unlock whole other flourishings of ecosystems, like graph-based data transfer systems, etc.</p> <h2 id="comparisons" tabindex="-1"><a class="header-anchor" href="#comparisons">Comparisons</a></h2> <h3 id="databases" tabindex="-1"><a class="header-anchor" href="#databases">Databases</a></h3> <p>We want IPLD to provide many of the useful attributes of an SQL-style database for purely local/offline applications: IPLD Schemas and IPLD Selectors provide features comparable to DDLs and SQL, and set the stage for building migration systems. ADLs provide features comparable to database indexing.</p> <p>At the same time, IPLD works on a concept of "blocks" which are much more comparable to messages: they're easy to send and receive over the network, and critically, can be regarded individually (there's no supposition of monolithicity, as SQL-style databases have). And the IPLD Data Model is fundamentally more similar to document stores than relational databases.</p> <p>Our aim is that IPLD should straddle this divide between databases and message systems, and operate on either side of it gracefully. The same libraries and concepts and tooling should work well regardless of if you're building an offline app or a massively distributed system with remote actors coming and going and sending messages all the time.</p> <!-- The below is a bunch of true stuff, and explains how we got where we did on a bunch of key design junctures, but it's also just awful as a wall of text. If it's necessary to present this, we should workshop it until getting to some better way to present it. Things IPLD is concerned with ----------------------------- In order to make building on decentralized data structures be excellent, we see the following major goals: - Some minimal legibility of data is required. - (This is what our work on the Data Model focuses around.) - It is awesome if our concept of data can be used across various encoding systems with a minimum of effort. - (This is why most IPLD libraries have a concept of various codecs; and we do a lot of work to specify codecs in terms of the Data Model.) - Whenever data is serialized, some standard indications of what codecs are necessary to use to transform that serial data back into a minimally legible form must be available, and these indicators should always be found very tightly bound to any stored data. - (This is what multicodecs are about, and why multicodec indicators are in CIDs.) - For data to be usable in decentralized/distributed systems, it's important for small pieces of data to link to other pieces of data; and this must be based on immutable references, because a user must be able to freely choose between fetching data either immediately or deferred at some later time: it must be possible to choose this without changing the correctness and coherency of their data by making their fetch actions at a later date. The most effective way to do this is content-addressing with the use of cryptographic hashes. - (This is what IPLD Linking and CIDs are all about.) - Given that we have identified that hashing data is a useful thing to do in the pursuit of building decentralized data structures, it would be excellent to standardize the process of feeding data to hashes, so that this can become a simple thing that happens without great special effort; and since there are various hashing systems in the world, it is important to identify which hashing algorithm was used to compute any hash we might store or reference. - (This is what multihashes are about, and why multihash indicators are in CIDs; and why IPLD libraries connect dots all the way through from Data Model to Codec to hashing.) - It's important to be able to traverse our minimally legible data, programmatically. We should be able to understand how to do this even without understanding any semantic details of this particular data, and we need this so that data remains usefully legible even in "deep time". - (This is again what the Data Model, and IPLD Linking, focuses on. IPLD Selectors then go further, providing a declarative way to describe traversals.) - It's useful to be able to define some topological expectations of the structure of data in a declarative way. - (This is what IPLD Schemas focus on.) - It's important that these topological expectations about the structure of data should be able to apply to any data -- even data that predates (or was otherwise causally independent of) the writing of the topological declarations. This is important because this kind of causal independence is frequent in decentralized and distributed development, and thus to be true to our vision and theory of change, we should support this kind of relationship. - (This figures deeply in the design of IPLD Schemas.) - It's desirable that these topological expectations about the structure of data should be something we can use to describe "version" detection and feature detection. Most concepts of "version" are brittle in decentralized systems (and especially, in decentralized development practices); we need ways to reason about data that continue to work when there's no central authority on version numbering, and mechanisms for describing data without resorting to explicit inbuilt versioning at all, so that it is possible to evolve the way we handle data separately from the data itself. - (This figures deeply in the design of IPLD Schemas.) - It is important to be able to collect large amounts of data, and make it accessible and navigable even if we split it up ("shard" it). - (We introduce an idea called Advanced Data Layouts as a way to describe such collections, and as part of this, describe how to make the sharded content directly legible as the IPLD Data Model.) Things not in the scope of IPLD ------------------------------- - Encryption. It's important, but it's not our battle. We hope that IPLD will be useful for structuring data, both in cleartext and ciphertext. We hope the the Advanced Data Layout extension system might be applicable to making systems simultaneously use encryption and are traversable as IPLD! But essentially, we're looking to _enable_ these developments -- it's not on our roadmap to drive them. - Identity systems. It's important, but it's not our battle. Decentralized identity is a very expansive topic; it's exceeding unlikely that there will even be "one" answer to the questions in this field: it blurs between cryptosystem design, reputation systems, statistical properties of graphs... there's truly a whole field here, if not _several_, and no one-size-fits-all answers. We will try to provide guidance on how IPLD relates to, and might be usefully composed with, other decentralized identity systems; but ultimately, how to do this will depend on the choices of application developers and the needs of their applications. - Networking. How to get data from one place to another is important, but IPLD should work with any mechanism you choose. IPLD should even work fine and provide value with *no* networking, or work fantastically with "sneakernet". (The libp2p project is also pursuing useful work in networking and focused on decentralized systems; it's not directly related to IPLD, but they should work well together, and you might want to check it out.) --> </div> </main> </body> </html>