CINXE.COM

<!doctype html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <link rel="stylesheet" href="/css/layout.css?1741648188062"> <link rel="stylesheet" href="/css/nav.css?1741648188062"> <link rel="stylesheet" href="/css/style.css?1741648188062"> <link rel="stylesheet" href="/css/prismjs@1.24-themes-prism.css"> <title>IPLD ♦ </title> </head> <body> <header> <div class="sidebar-button" onclick="document.getElementById('sidebar').classList.toggle('sidebar-open')"> <svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" role="img" viewBox="0 0 448 512" class="icon"> <path fill="currentColor" d="M436 124H12c-6.627 0-12-5.373-12-12V80c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12zm0 160H12c-6.627 0-12-5.373-12-12v-32c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12zm0 160H12c-6.627 0-12-5.373-12-12v-32c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12z"></path> </svg> </div> <a href="/" class="logo">IPLD</a> <aside id=breadcrumbs> <ul> <li><a href="/specs">specs</a></li> <li><a href="/specs/codecs">codecs</a></li> <li><a href="/specs/codecs/dag-cosmos">dag-cosmos</a></li> <li><a href="/specs/codecs/dag-cosmos/typed_protobuf/">typed_protobuf</a></li> </ul> </aside> </header> <aside id=sidebar> <nav> <ul> <li> <a href="/docs/">Docs</a><ul> <li> <a href="/docs/intro/">Intro</a><ul> <li> <a href="/docs/intro/hello-world/">Hello, World</a></li> <li> <a href="/docs/intro/primer/">The Brief Primer</a></li> <li> <a href="/docs/intro/ecosystem/">InterPlanetary Ecosystem Overview</a></li> <li> <a href="/docs/intro/community/">Finding Community</a></li></ul></li> <li> <a href="/docs/motivation/">Motivation</a><ul> <li> <a href="/docs/motivation/benefits-of-content-addressing/">Benefits of Content Addressing</a></li> <li> <a href="/docs/motivation/data-to-data-structures/">From Data to Data Structures</a></li></ul></li> <li> <a href="/docs/codecs/">Codecs</a><ul> <li> <a href="/docs/codecs/known/">Known Codecs</a><ul> <li> <a href="/docs/codecs/known/dag-cbor/">DAG-CBOR</a></li> <li> <a href="/docs/codecs/known/dag-json/">DAG-JSON</a></li> <li> <a href="/docs/codecs/known/dag-pb/">DAG-PB</a></li></ul></li></ul></li> <li> <a href="/docs/data-model/">Data Model</a><ul> <li> <a href="/docs/data-model/node/">Nodes</a></li> <li> <a href="/docs/data-model/kinds/">Kinds</a></li> <li> <a href="/docs/data-model/pathing/">Pathing</a></li> <li> <a href="/docs/data-model/traversal/">Traversal</a></li></ul></li> <li> <a href="/docs/advanced-data-layouts/">Advanced Data Layouts</a><ul> <li> <a href="/docs/advanced-data-layouts/intro/">Intro to ADLs</a></li> <li> <a href="/docs/advanced-data-layouts/naming/">ADL Naming</a></li> <li> <a href="/docs/advanced-data-layouts/signalling/">Signalling ADLs</a></li> <li> <a href="/docs/advanced-data-layouts/dynamic-loading/">Dynamic Loading</a></li> <li> <a href="/docs/advanced-data-layouts/known/">Known ADLs</a></li></ul></li> <li> <a href="/docs/schemas/">Schemas</a><ul> <li> <a href="/docs/schemas/intro/">Introduction</a><ul> <li> <a href="/docs/schemas/intro/compare/">compare</a></li> <li> <a href="/docs/schemas/intro/goals/">Goals</a></li> <li> <a href="/docs/schemas/intro/feature-summary/">Feature Summary</a></li></ul></li> <li> <a href="/docs/schemas/features/">Features</a><ul> <li> <a href="/docs/schemas/features/typekinds/">Type Kinds</a></li> <li> <a href="/docs/schemas/features/representation-strategies/">Representation Strategies</a></li> <li> <a href="/docs/schemas/features/links/">Links</a></li> <li> <a href="/docs/schemas/features/indicating-adls/">Using ADLs in Schemas</a></li></ul></li> <li> <a href="/docs/schemas/using/">Using Wisely</a><ul> <li> <a href="/docs/schemas/using/authoring-guide/">Authoring Guide</a></li> <li> <a href="/docs/schemas/using/migrations/">Migrations</a></li></ul></li></ul></li> <li> <a href="/docs/synthesis/">Synthesis</a><ul> <li> <a href="/docs/synthesis/gtd/">Getting Things Done</a></li> <li> <a href="/docs/synthesis/building-in-alignment/">Building in Alignment</a></li> <li> <a href="/docs/synthesis/how-ipfs-web-gateways-work/">How IPFS Web Gateways Work</a></li> <li> <a href="/docs/synthesis/encryption/">Working With Encryption</a></li></ul></li></ul></li> <li> <a href="/specs/">Specs</a><ul> <li> <a href="/specs/about/">About the Specifications</a></li> <li> <a href="/specs/codecs/">Codecs</a><ul> <li> <a href="/specs/codecs/dag-cbor/">DAG-CBOR</a><ul> <li> <a href="/specs/codecs/dag-cbor/spec/">Spec</a></li> <li> <a href="/specs/codecs/dag-cbor/fixtures/">DAG-CBOR Test Fixtures</a><ul> <li> <a href="/specs/codecs/dag-cbor/fixtures/cross-codec/">cross-codec</a></li></ul></li></ul></li> <li> <a href="/specs/codecs/dag-cosmos/">DAG-COSMOS</a><ul> <li> <a href="/specs/codecs/dag-cosmos/basic_types/">basic_types</a></li> <li> <a href="/specs/codecs/dag-cosmos/cosmos_state/">cosmos_state</a></li> <li> <a href="/specs/codecs/dag-cosmos/crypto_types/">crypto_types</a></li> <li> <a href="/specs/codecs/dag-cosmos/tendermint_chain/">tendermint_chain</a></li> <li class="active-page"> <a href="/specs/codecs/dag-cosmos/typed_protobuf/">typed_protobuf</a></li></ul></li> <li> <a href="/specs/codecs/dag-eth/">DAG-ETH</a><ul> <li> <a href="/specs/codecs/dag-eth/basic_types/">basic_types</a></li> <li> <a href="/specs/codecs/dag-eth/chain/">chain</a></li> <li> <a href="/specs/codecs/dag-eth/convenience_types/">convenience_types</a></li> <li> <a href="/specs/codecs/dag-eth/state/">state</a></li></ul></li> <li> <a href="/specs/codecs/dag-jose/">DAG-JOSE</a><ul> <li> <a href="/specs/codecs/dag-jose/spec/">Spec</a></li> <li> <a href="/specs/codecs/dag-jose/fixtures/">fixtures</a></li></ul></li> <li> <a href="/specs/codecs/dag-json/">DAG-JSON</a><ul> <li> <a href="/specs/codecs/dag-json/spec/">Spec</a></li> <li> <a href="/specs/codecs/dag-json/fixtures/">DAG-JSON Test Fixtures</a><ul> <li> <a href="/specs/codecs/dag-json/fixtures/cross-codec/">cross-codec</a></li></ul></li></ul></li> <li> <a href="/specs/codecs/dag-pb/">DAG-PB</a><ul> <li> <a href="/specs/codecs/dag-pb/spec/">Spec</a></li> <li> <a href="/specs/codecs/dag-pb/fixtures/">DAG-PB Test Fixtures</a><ul> <li> <a href="/specs/codecs/dag-pb/fixtures/cross-codec/">cross-codec</a></li></ul></li></ul></li></ul></li> <li> <a href="/specs/advanced-data-layouts/">Advanced Data Layouts</a><ul> <li> <a href="/specs/advanced-data-layouts/fbl/">FBL ADL</a><ul> <li> <a href="/specs/advanced-data-layouts/fbl/spec/">spec</a></li></ul></li> <li> <a href="/specs/advanced-data-layouts/hamt/">HAMT ADL</a><ul> <li> <a href="/specs/advanced-data-layouts/hamt/spec/">spec</a></li> <li> <a href="/specs/advanced-data-layouts/hamt/fixture/">HashMap (HAMT) Test Fixtures</a><ul> <li> <a href="/specs/advanced-data-layouts/hamt/fixture/alice-words/">alice-words</a></li></ul></li></ul></li></ul></li> <li> <a href="/specs/schemas/">Schemas</a><ul> <li> <a href="/specs/schemas/prelude/">prelude</a></li></ul></li> <li> <a href="/specs/transport/">Transports</a><ul> <li> <a href="/specs/transport/car/">CAR</a><ul> <li> <a href="/specs/transport/car/carv1/">CARv1 Specification</a></li> <li> <a href="/specs/transport/car/carv2/">CARv2 Specification</a></li> <li> <a href="/specs/transport/car/fixture/">CAR Test Fixtures</a><ul> <li> <a href="/specs/transport/car/fixture/carv1-basic/">carv1-basic</a></li> <li> <a href="/specs/transport/car/fixture/carv2-basic/">carv2-basic</a></li></ul></li></ul></li> <li> <a href="/specs/transport/graphsync/">Graphsync</a><ul> <li> <a href="/specs/transport/graphsync/known_extensions/">known_extensions</a></li></ul></li> <li> <a href="/specs/transport/trustless-pathing/">Trustless Pathing</a><ul> <li> <a href="/specs/transport/trustless-pathing/fixtures/">Trustless Pathing Fixtures</a><ul> <li> <a href="/specs/transport/trustless-pathing/fixtures/unixfs_20m_variety/">unixfs_20m_variety</a></li></ul></li></ul></li></ul></li> <li> <a href="/specs/selectors/">Selectors</a><ul> <li> <a href="/specs/selectors/fixtures/">fixtures</a><ul> <li> <a href="/specs/selectors/fixtures/selector-fixtures-1/">selector-fixtures-1</a></li> <li> <a href="/specs/selectors/fixtures/selector-fixtures-adl/">selector-fixtures-adl</a></li> <li> <a href="/specs/selectors/fixtures/selector-fixtures-recursion/">selector-fixtures-recursion</a></li></ul></li></ul></li> <li> <a href="/specs/patch/">Patch</a><ul> <li> <a href="/specs/patch/fixtures/">IPLD Patch Test Fixtures</a><ul> <li> <a href="/specs/patch/fixtures/fixtures-1/">fixtures-1</a></li></ul></li></ul></li></ul></li> <li> <a href="/libraries/">Libraries</a><ul> <li> <a href="/libraries/golang/">Golang</a></li> <li> <a href="/libraries/java/">Java</a></li> <li> <a href="/libraries/javascript/">JavaScript</a></li> <li> <a href="/libraries/python/">Python</a></li> <li> <a href="/libraries/rust/">Rust</a></li></ul></li> <li> <a href="/design/">Design</a><ul> <li> <a href="/design/objectives/">Objectives</a></li> <li> <a href="/design/concepts/">Concepts</a><ul> <li> <a href="/design/concepts/type-theory-glossary/">type-theory-glossary</a></li></ul></li> <li> <a href="/design/libraries/">Libraries</a><ul> <li> <a href="/design/libraries/nodes-and-kinds/">nodes-and-kinds</a></li></ul></li> <li> <a href="/design/tricky-choices/">Tricky Choices</a><ul> <li> <a href="/design/tricky-choices/dag-pb-forms-impl-and-use/">dag-pb-forms-impl-and-use</a></li> <li> <a href="/design/tricky-choices/map-key-domain/">map-key-domain</a></li> <li> <a href="/design/tricky-choices/numeric-domain/">numeric-domain</a></li> <li> <a href="/design/tricky-choices/ordering/">ordering</a></li> <li> <a href="/design/tricky-choices/string-domain/">string-domain</a></li></ul></li> <li> <a href="/design/open-research/">Open Research</a><ul> <li> <a href="/design/open-research/ADL-autoexecution/">ADL autoexecution</a></li></ul></li></ul></li> <li> <a href="/tools/">Tools</a></li> <li> <a href="/glossary/">Glossary</a></li> <li> <a href="/media/">Media</a></li> <li> <a href="/FAQ/">FAQ</a></li></ul> </nav> </aside> <main> <div class=content> <h1>Rich typing of protobuf objects in IPLD</h1> Any proto message type can be described using a set of <a href="https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/descriptor.proto#L62">FileDescriptorProto</a> messages which contains all information required to recreate the .proto message definition. In protobuf these are used for the <a href="https://github.com/grpc/grpc/blob/master/doc/server-reflection.md#grpc-server-reflection-protocol">gRPC server reflection protocol</a> as well as for creating <a href="https://developers.google.com/protocol-buffers/docs/techniques?authuser=2#self-description">self-describing messages</a>. Similarly, we propose to leverage <code>FileDescriptorProto</code> message sets in the context of IPLD to provide rich typing for arbitrary protobuf messages stored in on IPFS or Filecoin. To this end, we can prepend the block binary for a protobuf encoded object with a content hash reference to a set of <code>FileDescriptorProto</code>s that describe how to type that content in any programming language that has a .proto compiler. We avoid storing and referencing .proto files directly in order to avoid the additional dimensions of entropy that introduces, namely the variability in whitespace and ordering between and within message definitions. These differences can cause .proto files for the same type definitions to produce different, redundant block binaries. There exists a <a href="https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/descriptor.proto#L57">FileDescriptorSet</a>, but this type includes a list of full <code>FileDescriptorProto</code>s. In order to minimize duplicity of data in the graph, we introduce a new IPLD representation of a <code>FileDescriptorSet</code> which instead consists of a list of links to individual <code>FileDescriptorProto</code> objects. In this scheme we reserve three multicodec type byte prefixes: <ol> <li><code>FileDescriptorProto</code> protobuf serialization: multi-content type that stipulates that the referenced block binary is a protobuf encoded <code>FileDescriptorProto</code> message.</li> <li><code>FileDescriptorSet</code> IPLD object: multi-content type that stipulates that the referenced block binary is for an IPLD object which consists of a list of CIDs that reference <code>FileDescriptorProto</code> blocks.</li> <li><code>TypedProtobuf</code> IPLD object: multi-content type that stipulates the referenced binary is for an IPLD object whereby the first element is a CID for a <code>FileDescriptorSet</code> and the second element is the protobuf encoding of a value whose message type is described by the linked <code>FileDescriptorSet</code>.</li> </ol> <pre class="language-ipldsch"><code class="language-ipldsch"># FileDescriptorProto is the protobuf marshalled binary of a FileDescriptorProto message # FileDescriptorProtoCID is a CID link to a FileDescriptorProto block binary # This CID is composed of the SHA_256 multihash of the FileDescriptorProto block binary and the FileDescriptorProto codec (tbd) type FileDescriptorProtoCID &FileDescriptorProto # FileDescriptorSet is an IPLD object consisting of a list of CIDs that link to FileDescriptorProto message block binaries type FileDescriptorSet [FileDescriptorProtoCID] # FileDescriptorSetCID is a CID link to a FileDescriptorSet block binary # This CID is composed of the SHA_256 multihash of the FileDescriptorSet IPLD block binary and the FileDescriptorSet codec (tbd) type FileDescriptorSetCID &FileDescriptorSet # TypedProtobuf is an IPLD object consisting of a CID link to a FileDescriptorSet # and bytes for a protobuf encoded value whose type is described by the linked FileDescriptorSet # When calculating the multihash of this object, only the ProtoMessageBytes should be hashed type TypedProtobuf struct { DescriptorSetCID FileDescriptorSetCID ProtoMessageBytes Bytes } # TypedProtobufCID is a CID link to a TypedProtobuf block binary # This CID is composed of the SHA2-256-Ignore32BytePrefix multihash of the TypedProtobuf IPLD block binary and the TypedProtobuf codec (tbd) type TypedProtobufCID &TypedProtobuf</code></pre> In order to support this scheme, we also reserve a new multicodec type byte prefix for a multihash function that is a simple derivative of the existing <code>sha2-256</code> multihash function that ignores the first 32 bytes of the input. This is necessary in order to preserve the native <code>hash => content</code> mapping of these content-type prefixed protobuf objects. e.g. <pre class="language-ipldsch"><code class="language-ipldsch">SHA2-256-Ignore32BytePrefix</code></pre> The above scheme provides a mechanism to provide rich type information to arbitrary protobuf objects stored in IPFS/Filecoin without requiring the reservation and registration of multicodec type byte prefixes for each individual type. This is necessary to provide generic IPLD integration for the transactions and state in any Tendermint/Cosmos chain that abides by the below assumptions: <ol> <li>Protobuf encoding is used for the Tendermint transactions.</li> <li>IAVL or SMT are used as the state commitment data structure for the CosmosSDK app.</li> <li>Protobuf encoding is used for values stored in the IAVL or SMT state commitment data structures.</li> </ol> </div> </main> </body> </html>