CINXE.COM

<!doctype html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <link rel="stylesheet" href="/css/layout.css?1732633144785"> <link rel="stylesheet" href="/css/nav.css?1732633144785"> <link rel="stylesheet" href="/css/style.css?1732633144785"> <link rel="stylesheet" href="/css/prismjs@1.24-themes-prism.css"> <title>IPLD ♦ DAG-PB Specification</title> </head> <body> <header> <div class="sidebar-button" onclick="document.getElementById('sidebar').classList.toggle('sidebar-open')"> <svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" role="img" viewBox="0 0 448 512" class="icon"> <path fill="currentColor" d="M436 124H12c-6.627 0-12-5.373-12-12V80c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12zm0 160H12c-6.627 0-12-5.373-12-12v-32c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12zm0 160H12c-6.627 0-12-5.373-12-12v-32c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12z"></path> </svg> </div> <a href="/" class="logo">IPLD</a> <aside id=breadcrumbs> <ul> <li><a href="/specs">specs</a></li> <li><a href="/specs/codecs">codecs</a></li> <li><a href="/specs/codecs/dag-pb">dag-pb</a></li> <li><a href="/specs/codecs/dag-pb/spec/">spec</a></li> </ul> </aside> </header> <aside id=sidebar> <nav> <ul> <li> <a href="/docs/">Docs</a><ul> <li> <a href="/docs/intro/">Intro</a><ul> <li> <a href="/docs/intro/hello-world/">Hello, World</a></li> <li> <a href="/docs/intro/primer/">The Brief Primer</a></li> <li> <a href="/docs/intro/ecosystem/">InterPlanetary Ecosystem Overview</a></li> <li> <a href="/docs/intro/community/">Finding Community</a></li></ul></li> <li> <a href="/docs/motivation/">Motivation</a><ul> <li> <a href="/docs/motivation/benefits-of-content-addressing/">Benefits of Content Addressing</a></li> <li> <a href="/docs/motivation/data-to-data-structures/">From Data to Data Structures</a></li></ul></li> <li> <a href="/docs/codecs/">Codecs</a><ul> <li> <a href="/docs/codecs/known/">Known Codecs</a><ul> <li> <a href="/docs/codecs/known/dag-cbor/">DAG-CBOR</a></li> <li> <a href="/docs/codecs/known/dag-json/">DAG-JSON</a></li> <li> <a href="/docs/codecs/known/dag-pb/">DAG-PB</a></li></ul></li></ul></li> <li> <a href="/docs/data-model/">Data Model</a><ul> <li> <a href="/docs/data-model/node/">Nodes</a></li> <li> <a href="/docs/data-model/kinds/">Kinds</a></li> <li> <a href="/docs/data-model/pathing/">Pathing</a></li> <li> <a href="/docs/data-model/traversal/">Traversal</a></li></ul></li> <li> <a href="/docs/advanced-data-layouts/">Advanced Data Layouts</a><ul> <li> <a href="/docs/advanced-data-layouts/intro/">Intro to ADLs</a></li> <li> <a href="/docs/advanced-data-layouts/naming/">ADL Naming</a></li> <li> <a href="/docs/advanced-data-layouts/signalling/">Signalling ADLs</a></li> <li> <a href="/docs/advanced-data-layouts/dynamic-loading/">Dynamic Loading</a></li> <li> <a href="/docs/advanced-data-layouts/known/">Known ADLs</a></li></ul></li> <li> <a href="/docs/schemas/">Schemas</a><ul> <li> <a href="/docs/schemas/intro/">Introduction</a><ul> <li> <a href="/docs/schemas/intro/compare/">compare</a></li> <li> <a href="/docs/schemas/intro/goals/">Goals</a></li> <li> <a href="/docs/schemas/intro/feature-summary/">Feature Summary</a></li></ul></li> <li> <a href="/docs/schemas/features/">Features</a><ul> <li> <a href="/docs/schemas/features/typekinds/">Type Kinds</a></li> <li> <a href="/docs/schemas/features/representation-strategies/">Representation Strategies</a></li> <li> <a href="/docs/schemas/features/links/">Links</a></li> <li> <a href="/docs/schemas/features/indicating-adls/">Using ADLs in Schemas</a></li></ul></li> <li> <a href="/docs/schemas/using/">Using Wisely</a><ul> <li> <a href="/docs/schemas/using/authoring-guide/">Authoring Guide</a></li> <li> <a href="/docs/schemas/using/migrations/">Migrations</a></li></ul></li></ul></li> <li> <a href="/docs/synthesis/">Synthesis</a><ul> <li> <a href="/docs/synthesis/gtd/">Getting Things Done</a></li> <li> <a href="/docs/synthesis/building-in-alignment/">Building in Alignment</a></li> <li> <a href="/docs/synthesis/how-ipfs-web-gateways-work/">How IPFS Web Gateways Work</a></li> <li> <a href="/docs/synthesis/encryption/">Working With Encryption</a></li></ul></li></ul></li> <li> <a href="/specs/">Specs</a><ul> <li> <a href="/specs/about/">About the Specifications</a></li> <li> <a href="/specs/codecs/">Codecs</a><ul> <li> <a href="/specs/codecs/dag-cbor/">DAG-CBOR</a><ul> <li> <a href="/specs/codecs/dag-cbor/spec/">Spec</a></li> <li> <a href="/specs/codecs/dag-cbor/fixtures/">DAG-CBOR Test Fixtures</a><ul> <li> <a href="/specs/codecs/dag-cbor/fixtures/cross-codec/">cross-codec</a></li></ul></li></ul></li> <li> <a href="/specs/codecs/dag-cosmos/">DAG-COSMOS</a><ul> <li> <a href="/specs/codecs/dag-cosmos/basic_types/">basic_types</a></li> <li> <a href="/specs/codecs/dag-cosmos/cosmos_state/">cosmos_state</a></li> <li> <a href="/specs/codecs/dag-cosmos/crypto_types/">crypto_types</a></li> <li> <a href="/specs/codecs/dag-cosmos/tendermint_chain/">tendermint_chain</a></li> <li> <a href="/specs/codecs/dag-cosmos/typed_protobuf/">typed_protobuf</a></li></ul></li> <li> <a href="/specs/codecs/dag-eth/">DAG-ETH</a><ul> <li> <a href="/specs/codecs/dag-eth/basic_types/">basic_types</a></li> <li> <a href="/specs/codecs/dag-eth/chain/">chain</a></li> <li> <a href="/specs/codecs/dag-eth/convenience_types/">convenience_types</a></li> <li> <a href="/specs/codecs/dag-eth/state/">state</a></li></ul></li> <li> <a href="/specs/codecs/dag-jose/">DAG-JOSE</a><ul> <li> <a href="/specs/codecs/dag-jose/spec/">Spec</a></li> <li> <a href="/specs/codecs/dag-jose/fixtures/">fixtures</a></li></ul></li> <li> <a href="/specs/codecs/dag-json/">DAG-JSON</a><ul> <li> <a href="/specs/codecs/dag-json/spec/">Spec</a></li> <li> <a href="/specs/codecs/dag-json/fixtures/">DAG-JSON Test Fixtures</a><ul> <li> <a href="/specs/codecs/dag-json/fixtures/cross-codec/">cross-codec</a></li></ul></li></ul></li> <li> <a href="/specs/codecs/dag-pb/">DAG-PB</a><ul> <li class="active-page"> <a href="/specs/codecs/dag-pb/spec/">Spec</a></li> <li> <a href="/specs/codecs/dag-pb/fixtures/">DAG-PB Test Fixtures</a><ul> <li> <a href="/specs/codecs/dag-pb/fixtures/cross-codec/">cross-codec</a></li></ul></li></ul></li></ul></li> <li> <a href="/specs/advanced-data-layouts/">Advanced Data Layouts</a><ul> <li> <a href="/specs/advanced-data-layouts/fbl/">FBL ADL</a><ul> <li> <a href="/specs/advanced-data-layouts/fbl/spec/">spec</a></li></ul></li> <li> <a href="/specs/advanced-data-layouts/hamt/">HAMT ADL</a><ul> <li> <a href="/specs/advanced-data-layouts/hamt/spec/">spec</a></li> <li> <a href="/specs/advanced-data-layouts/hamt/fixture/">HashMap (HAMT) Test Fixtures</a><ul> <li> <a href="/specs/advanced-data-layouts/hamt/fixture/alice-words/">alice-words</a></li></ul></li></ul></li></ul></li> <li> <a href="/specs/schemas/">Schemas</a><ul> <li> <a href="/specs/schemas/prelude/">prelude</a></li></ul></li> <li> <a href="/specs/transport/">Transports</a><ul> <li> <a href="/specs/transport/car/">CAR</a><ul> <li> <a href="/specs/transport/car/carv1/">CARv1 Specification</a></li> <li> <a href="/specs/transport/car/carv2/">CARv2 Specification</a></li> <li> <a href="/specs/transport/car/fixture/">CAR Test Fixtures</a><ul> <li> <a href="/specs/transport/car/fixture/carv1-basic/">carv1-basic</a></li> <li> <a href="/specs/transport/car/fixture/carv2-basic/">carv2-basic</a></li></ul></li></ul></li> <li> <a href="/specs/transport/graphsync/">Graphsync</a><ul> <li> <a href="/specs/transport/graphsync/known_extensions/">known_extensions</a></li></ul></li> <li> <a href="/specs/transport/trustless-pathing/">Trustless Pathing</a><ul> <li> <a href="/specs/transport/trustless-pathing/fixtures/">Trustless Pathing Fixtures</a><ul> <li> <a href="/specs/transport/trustless-pathing/fixtures/unixfs_20m_variety/">unixfs_20m_variety</a></li></ul></li></ul></li></ul></li> <li> <a href="/specs/selectors/">Selectors</a><ul> <li> <a href="/specs/selectors/fixtures/">fixtures</a><ul> <li> <a href="/specs/selectors/fixtures/selector-fixtures-1/">selector-fixtures-1</a></li> <li> <a href="/specs/selectors/fixtures/selector-fixtures-adl/">selector-fixtures-adl</a></li> <li> <a href="/specs/selectors/fixtures/selector-fixtures-recursion/">selector-fixtures-recursion</a></li></ul></li></ul></li> <li> <a href="/specs/patch/">Patch</a><ul> <li> <a href="/specs/patch/fixtures/">IPLD Patch Test Fixtures</a><ul> <li> <a href="/specs/patch/fixtures/fixtures-1/">fixtures-1</a></li></ul></li></ul></li></ul></li> <li> <a href="/libraries/">Libraries</a><ul> <li> <a href="/libraries/golang/">Golang</a></li> <li> <a href="/libraries/javascript/">JavaScript</a></li> <li> <a href="/libraries/python/">Python</a></li> <li> <a href="/libraries/rust/">Rust</a></li></ul></li> <li> <a href="/design/">Design</a><ul> <li> <a href="/design/objectives/">Objectives</a></li> <li> <a href="/design/concepts/">Concepts</a><ul> <li> <a href="/design/concepts/type-theory-glossary/">type-theory-glossary</a></li></ul></li> <li> <a href="/design/libraries/">Libraries</a><ul> <li> <a href="/design/libraries/nodes-and-kinds/">nodes-and-kinds</a></li></ul></li> <li> <a href="/design/tricky-choices/">Tricky Choices</a><ul> <li> <a href="/design/tricky-choices/dag-pb-forms-impl-and-use/">dag-pb-forms-impl-and-use</a></li> <li> <a href="/design/tricky-choices/map-key-domain/">map-key-domain</a></li> <li> <a href="/design/tricky-choices/numeric-domain/">numeric-domain</a></li> <li> <a href="/design/tricky-choices/ordering/">ordering</a></li> <li> <a href="/design/tricky-choices/string-domain/">string-domain</a></li></ul></li> <li> <a href="/design/open-research/">Open Research</a><ul> <li> <a href="/design/open-research/ADL-autoexecution/">ADL autoexecution</a></li></ul></li></ul></li> <li> <a href="/tools/">Tools</a></li> <li> <a href="/glossary/">Glossary</a></li> <li> <a href="/media/">Media</a></li> <li> <a href="/FAQ/">FAQ</a></li></ul> </nav> </aside> <main> <div class=content> <h1>DAG-PB Spec</h1> Status: Descriptive - Final DAG-PB is an IPLD codec that uses <a href="https://developers.google.com/protocol-buffers/">Protocol Buffers</a> to describe a binary format that can encode a byte array and an associated list of links. It is the primary means of encoding structured file data for <a href="https://ipfs.io/">IPFS</a>, serving as the encoded data carrier for <a href="https://docs.ipfs.io/concepts/file-systems/#unix-file-system-unixfs">UnixFS</a>. DAG-PB does not support the full <a href="/docs/data-model/">IPLD Data Model</a>. <ul> <li><a href="#implementations">Implementations</a></li> <li><a href="#serial-format">Serial Format</a> <ul> <li><a href="#protobuf-strictness">Protobuf Strictness</a></li> </ul> </li> <li><a href="#logical-format">Logical Format</a> <ul> <li><a href="#constraints">Constraints</a></li> </ul> </li> <li><a href="#alternative-legacy-pathing">Alternative (Legacy) Pathing</a></li> <li><a href="#zero-length-blocks">Zero-length blocks</a></li> <li><a href="#link-sorting">Link sorting</a> <ul> <li><a href="#link-sorting-in-go-merkledag">Link sorting in go-merkledag</a></li> </ul> </li> </ul> <h2 id="implementations" tabindex="-1"><a class="header-anchor" href="#implementations">Implementations</a></h2> <ul> <li>JavaScript <ul> <li><a href="https://github.com/ipld/js-dag-pb">@ipld/dag-pb</a> - compatible with <a href="https://github.com/multiformats/js-multiformats">multiformats</a></li> <li><a href="https://github.com/ipld/js-ipld-dag-pb">ipld-dag-pb</a> - legacy implementation</li> </ul> </li> <li>Go <ul> <li><a href="https://github.com/ipld/go-codec-dagpb">go-codec-dagpb</a> - for use with <a href="https://github.com/ipld/go-ipld-prime">go-ipld-prime</a></li> <li><a href="https://github.com/ipfs/go-merkledag/tree/master/pb">go-merkledag/pb</a> - legacy implementation</li> <li><a href="https://github.com/ipld/go-ipld-prime-proto">go-ipld-prime-proto</a> - read-only interface for go-merkledag/pb through <a href="https://github.com/ipld/go-ipld-prime">go-ipld-prime</a></li> </ul> </li> </ul> <h2 id="serial-format" tabindex="-1"><a class="header-anchor" href="#serial-format">Serial Format</a></h2> The DAG-PB IPLD serial format is described with a single protobuf: <pre class="language-protobuf"><code class="language-protobuf">message PBLink { // binary CID (with no multibase prefix) of the target object optional bytes Hash = 1; // UTF-8 string name optional string Name = 2; // cumulative size of target object optional uint64 Tsize = 3; } message PBNode { // refs to other objects repeated PBLink Links = 2; // opaque user data optional bytes Data = 1; }</code></pre> <h3 id="protobuf-strictness" tabindex="-1"><a class="header-anchor" href="#protobuf-strictness">Protobuf Strictness</a></h3> DAG-PB aims to have a canonical form for any given set of data. Therefore, in addition to the standard Protobuf parsing rules, DAG-PB decoders should enforce additional constraints to ensure canonical forms (where possible): <ol> <li>Fields in the <code>PBLink</code> message must appear in the order as defined by the Protobuf schema above, following the field numbers. Blocks with out-of-order <code>PBLink</code> fields should be rejected. (Note that it is common for Protobuf decoders to accept out-of-order field entries, which means the DAG-PB spec is somewhat stricter than may be seen as typical for other Protobuf-based formats.)</li> <li>Fields in the <code>PBNode</code> message must be encoded in the order as defined by the Protobuf schema above. Note that this order does not follow the field numbers. The decoder should accept either order, as IPFS data exists in both forms.</li> <li>Duplicate entries in the binary form are invalid; blocks with duplicate field values should be rejected. (Note that it is common for Protobuf decoders to accept repeated field values in the binary data, and interpret them as updates to fields that have already been set; DAG-PB is stricter than this.)</li> <li>Fields and wire types other than those that appear in the Protobuf schema above are invalid and blocks containing these should be rejected. (Note that it is common for Protobuf decoders to skip data in each message type that does not match the fields in the schema.)</li> </ol> <h2 id="logical-format" tabindex="-1"><a class="header-anchor" href="#logical-format">Logical Format</a></h2> When we handle DAG-PB content at the Data Model level, we treat these objects as maps. This layout can be expressed with <a href="/docs/schemas/">IPLD Schemas</a> as: <pre class="language-ipldsch"><code class="language-ipldsch">type PBNode struct { Links [PBLink] Data optional Bytes } type PBLink struct { Hash Link Name optional String Tsize optional Int }</code></pre> <h3 id="constraints" tabindex="-1"><a class="header-anchor" href="#constraints">Constraints</a></h3> <ul> <li>The first node in a block of DAG-PB data will match the <code>PBNode</code> type.</li> <li><code>Data</code> may be omitted or a byte array with a length of zero or more.</li> <li><code>Links</code>: <ul> <li>must be present, even if empty; the binary form makes no distinction between an empty array and an omitted value, in the Data Model we always instantiate an array.</li> <li>when encoding, elements must be sorted in ascending order by their <code>Name</code> values, which are compared by bytes rather than as strings (also see <a href="#link-sorting">notes below regarding sorted links</a>). <ul> <li><code>Name</code>s must be unique or be omitted.</li> </ul> </li> </ul> </li> <li><code>Hash</code>: <ul> <li>even though <code>Hash</code> is <code>optional</code> in the Protobuf encoding, it should not be treated as optional when creating new blocks or decoding existing ones, an omitted <code>Hash</code> should be interpreted as a bad block</li> <li>the bytes in the encoding format is interpreted as the bytes of a CID, if the bytes cannot be converted to a CID then it should be treated as a bad block.</li> <li>the data is encoded in the binary form as a byte array, it is therefore possible for a decoder to read a correct binary form but fail to convert a <code>Hash</code> to a CID and therefore treat it as a bad block.</li> </ul> </li> <li>When creating data, you can create maps using the standard Data Model concepts, and as long as they have exactly these fields. If additional fields are present, the DAG-PB codec will error, because there is no way to encode them.</li> </ul> Both the most recent <a href="https://github.com/ipld/js-dag-pb">JavaScript</a> and <a href="https://github.com/ipld/go-codec-dagpb">Go</a> implementations strictly expose this logical format via the Data Model and do not support alternative means of resolving paths via named links as the legacy implementations do (see below). <h2 id="alternative-legacy-pathing" tabindex="-1"><a class="header-anchor" href="#alternative-legacy-pathing">Alternative (Legacy) Pathing</a></h2> While the <a href="#logical-format">logical format</a> implicitly describes a set of mechanisms for pathing over and through DAG-PB data in strict Data Model form, legacy implementations afford a means of resolving paths by privileging the <code>Name</code> in links. This alternative pathing is covered here as part of this descriptive spec, but was developed independently of the Data Model and is thus not well standardized. The alternative pathing mechanisms differ between implementations and has been removed from the newer implementations entirely. The legacy <a href="https://github.com/ipfs/go-merkledag/tree/master/pb">Go</a> and <a href="https://github.com/ipld/js-ipld-dag-pb">JavaScript</a> implementations both support pathing with link names: <code>/<name1>/<name2>/…</code>. In the legacy Go implementation, this is the only way, which implies that is is impossible to path through nodes that don't name their links. Also neither the Data section nor the Links section/metadata are accessible through paths. In the legacy JavaScript implementation, there is an additional way to path through the data. It's based purely on the structure of object, i.e. <code>/Links/<index>/Hash/…</code>. This way you have direct access to the <code>Data</code>, <code>Links</code>, and <code>size</code> fields, e.g. <code>/Links/<index>/Hash/Data</code>. These two ways of pathing can be combined, so you can access e.g. the <code>Data</code> field of a named link via <code>/<name/Data</code>. You can also use both approaches within a single path, e.g. <code>/<name1>/Links/0/Hash/Data</code> or <code>/Links/<index>/Hash/<name>/Data</code>. When using the DAG API in js-ipfs, then the pathing over the structure has precedence, so you won't be able to use named pathing on a named link called <code>Links</code>, you would need to use the index of the link instead. Both the most recent <a href="https://github.com/ipld/js-dag-pb">JavaScript</a> and <a href="https://github.com/ipld/go-codec-dagpb">Go</a> implementations do not expose novel pathing mechanisms but adhere strictly to the IPLD Data Model as described in the above <a href="#logical-format">Logical Format</a> schema. <h2 id="zero-length-blocks" tabindex="-1"><a class="header-anchor" href="#zero-length-blocks">Zero-length blocks</a></h2> The zero-length DAG-PB block is valid and will be decoded as having null <code>Data</code> and an empty <code>Links</code> array. With a SHA2-256 multihash, the CID of this block is: <ul> <li>CIDv1: <code>bafybeihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku</code></li> <li>CIDv0: <code>QmdfTbBqBPQ7VNxZEYEj14VmRuZBkqFbiwReogJgS1zR1n</code></li> </ul> <h2 id="link-sorting" tabindex="-1"><a class="header-anchor" href="#link-sorting">Link sorting</a></h2> The <code>Links</code> list in a DAG-PB encoded form must be sorted in ascending order by their <code>Name</code> values, which are compared by bytes rather than as strings (also see <a href="#link-sorting">notes below regarding sorted links</a>). Missing or empty <code>Name</code> values are treated as empty-strings. Sorting should be stable, leaving duplicate <code>Name</code>s (or multiple missing or empty <code>Name</code> values) in their original order. Sorting should not be applied on decode of a DAG-PB block. The order of links found within a DAG-PB block is the order in which they appear in their binary form, so traversal over those links follows that order. Any difference in decoded form links order can impact traversals where a stable order is required. <h3 id="link-sorting-in-go-merkledag" tabindex="-1"><a class="header-anchor" href="#link-sorting-in-go-merkledag">Link sorting in go-merkledag</a></h3> Versions of the legacy <a href="https://github.com/ipfs/go-merkledag">go-merkledag</a> interface to DAG-PB blocks prior to <a href="https://github.com/ipfs/go-merkledag/releases/tag/v0.4.0">v0.4.0</a> applied sorting on decoded blocks when read through the <a href="https://pkg.go.dev/github.com/ipfs/go-merkledag#DecodeProtobuf"><code>DecodeProtobuf()</code></a> and <a href="https://pkg.go.dev/github.com/ipfs/go-merkledag#DecodeProtobufBlock"><code>DecodeProtobufBlock()</code></a> APIs. Versions of go-merkledag from v0.4.0 to v0.7.0 will sort Links of deserialized blocks with unsorted Links when certain operations are performed (such as <code>Size()</code>, <code>RawData()</code> and some others). go-merkledag v0.7.0 and later keeps unsorted the Links of deserialized blocks with unsorted Links until the node is mutated in some way, at which point the Links are automatically sorted. See <a href="https://github.com/ipfs/go-merkledag/pull/87">this pull request</a> for further details. </div> </main> </body> </html>