CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="description" content="An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach."> <meta name="keywords" content="prometheus, monitoring, monitoring system, time series, time series database, alerting, metrics, telemetry"> <meta name="author" content="Prometheus"> <meta name="twitter:card" content="summary"> <meta property="og:description" content="An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach."> <meta property="og:image" content="https://prometheus.io/assets/favicons/android-chrome-192x192.png"> <link rel="alternate" type="application/atom+xml" title="Prometheus Blog » Feed" href="/blog/feed.xml"> <link rel="shortcut icon" href="/assets/favicons/favicon.ico"> <link rel="apple-touch-icon" sizes="57x57" href="/assets/favicons/apple-touch-icon-57x57.png"> <link rel="apple-touch-icon" sizes="60x60" href="/assets/favicons/apple-touch-icon-60x60.png"> <link rel="apple-touch-icon" sizes="72x72" href="/assets/favicons/apple-touch-icon-72x72.png"> <link rel="apple-touch-icon" sizes="76x76" href="/assets/favicons/apple-touch-icon-76x76.png"> <link rel="apple-touch-icon" sizes="114x114" href="/assets/favicons/apple-touch-icon-114x114.png"> <link rel="apple-touch-icon" sizes="120x120" href="/assets/favicons/apple-touch-icon-120x120.png"> <link rel="apple-touch-icon" sizes="144x144" href="/assets/favicons/apple-touch-icon-144x144.png"> <link rel="apple-touch-icon" sizes="152x152" href="/assets/favicons/apple-touch-icon-152x152.png"> <link rel="apple-touch-icon" sizes="180x180" href="/assets/favicons/apple-touch-icon-180x180.png"> <link rel="icon" type="image/png" href="/assets/favicons/favicon-32x32.png" sizes="32x32"> <link rel="icon" type="image/png" href="/assets/favicons/android-chrome-192x192.png" sizes="192x192"> <link rel="icon" type="image/png" href="/assets/favicons/favicon-96x96.png" sizes="96x96"> <link rel="icon" type="image/png" href="/assets/favicons/favicon-16x16.png" sizes="16x16"> <link rel="manifest" href="/assets/favicons/android-chrome-manifest.json">  <meta name="docsearch:prometheus-version" content="none" /> <meta name="docsearch:include" content="true" /> <meta name="msapplication-TileColor" content="#da532c"> <meta name="msapplication-TileImage" content="/assets/favicons/mstile-144x144.png"> <meta name="theme-color" content="#ffffff"> <meta property="og:title" content="Prometheus Remote-Write 2.0 [EXPERIMENTAL] | Prometheus"> <title>Prometheus Remote-Write 2.0 [EXPERIMENTAL] | Prometheus</title>  <link href="/assets/bootstrap-3.4.1/css/bootstrap.min.css" rel="stylesheet">  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.css" />  <link href="/css/docs.css" rel="stylesheet"> <link href="/css/routing-tree-editor.css" rel="stylesheet">  <link href="/assets/tabler-icons-3.19.0/webfont/tabler-icons.min.css" rel="stylesheet" type="text/css"> <link href='https://fonts.googleapis.com/css?family=Open+Sans:400,400italic,700' rel='stylesheet' type='text/css'> <link href='https://fonts.googleapis.com/css?family=Lato:300,300italic,400' rel='stylesheet' type='text/css'> <script async src="https://www.googletagmanager.com/gtag/js?id=G-80ZM8LGB96"></script> <script> window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'G-80ZM8LGB96'); </script> </head> <body> <div class=""> <nav class="navbar navbar-inverse navbar-static-top" role="navigation"> <div class="container"> <div class="navbar-header"> <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar"> <span class="sr-only">Toggle navigation</span> <span class="icon-bar"></span> <span class="icon-bar"></span> <span class="icon-bar"></span> </button> <a class="navbar-brand" href="/"><img src="/assets/prometheus_logo_grey.svg" alt="Prometheus logo"> Prometheus</a> </div> <div class="collapse navbar-collapse" id="navbar"> <ul class="nav navbar-nav navbar-right main-nav"> <li><a href="/docs/introduction/overview/">Docs</a></li> <li><a href="/download/">Download</a></li> <li><a href="/community/">Community</a></li> <li><a href="/support-training/">Support & Training</a></li> <li><a href="/blog/">Blog</a></li> <li><input class="searchbox form-control" type="search" placeholder="Search" aria-label="Search"></li> <li><a href="https://github.com/prometheus"><i class="ti ti-brand-github"></i></a></li> </ul> </div> </div> </nav> </div> <div class="container"> <div class="row"> <div class="col-md-3 side-nav-col"> <ul class="nav navbar-nav side-nav"> <li><span class="nav-header"><i class="ti ti-hand-finger-right"></i> <span>Introduction</span></span><ul class="nav "><li><a href="/docs/introduction/overview/">Overview</a></li><li><a href="/docs/introduction/first_steps/">First steps</a></li><li><a href="/docs/introduction/comparison/">Comparison to alternatives</a></li><li><a href="/docs/introduction/faq/">FAQ</a></li><li><a href="/docs/introduction/roadmap/">Roadmap</a></li><li><a href="/docs/introduction/design-doc/">Design Documents</a></li><li><a href="/docs/introduction/media/">Media</a></li><li><a href="/docs/introduction/glossary/">Glossary</a></li><li><a href="/docs/introduction/release-cycle/">Long-Term Support</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-flask"></i> <span>Concepts</span></span><ul class="nav "><li><a href="/docs/concepts/data_model/">Data model</a></li><li><a href="/docs/concepts/metric_types/">Metric types</a></li><li><a href="/docs/concepts/jobs_instances/">Jobs and instances</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-server"></i> <span>Prometheus Server</span></span><div class="">Version: <select><option value="/docs/prometheus/latest/getting_started/" selected="selected">latest (3.2)</option><option value="/docs/prometheus/3.2/getting_started/" >3.2</option><option value="/docs/prometheus/3.1/getting_started/" >3.1</option><option value="/docs/prometheus/3.0/getting_started/" >3.0</option><option value="/docs/prometheus/2.55/getting_started/" >2.55</option><option value="/docs/prometheus/2.54/getting_started/" >2.54</option><option value="/docs/prometheus/2.53/getting_started/" >2.53 (LTS)</option><option value="/docs/prometheus/2.52/getting_started/" >2.52</option><option value="/docs/prometheus/2.51/getting_started/" >2.51</option><option value="/docs/prometheus/2.50/getting_started/" >2.50</option><option value="/docs/prometheus/2.49/getting_started/" >2.49</option><option value="/docs/prometheus/1.8/getting_started/" >1.8</option></select></div><ul class="nav "><li><a href="/docs/prometheus/latest/getting_started/">Getting started</a></li><li><a href="/docs/prometheus/latest/installation/">Installation</a></li><li><span class="nav-subheader">Configuration</span><ul class="nav "><li><a href="/docs/prometheus/latest/configuration/configuration/">Configuration</a></li><li><a href="/docs/prometheus/latest/configuration/recording_rules/">Recording rules</a></li><li><a href="/docs/prometheus/latest/configuration/alerting_rules/">Alerting rules</a></li><li><a href="/docs/prometheus/latest/configuration/template_examples/">Template examples</a></li><li><a href="/docs/prometheus/latest/configuration/template_reference/">Template reference</a></li><li><a href="/docs/prometheus/latest/configuration/unit_testing_rules/">Unit Testing for Rules</a></li><li><a href="/docs/prometheus/latest/configuration/https/">HTTPS and authentication</a></li></ul></li><li><span class="nav-subheader">Querying</span><ul class="nav "><li><a href="/docs/prometheus/latest/querying/basics/">Basics</a></li><li><a href="/docs/prometheus/latest/querying/operators/">Operators</a></li><li><a href="/docs/prometheus/latest/querying/functions/">Functions</a></li><li><a href="/docs/prometheus/latest/querying/examples/">Examples</a></li><li><a href="/docs/prometheus/latest/querying/api/">HTTP API</a></li><li><a href="/docs/prometheus/latest/querying/remote_read_api/">Remote Read API</a></li></ul></li><li><a href="/docs/prometheus/latest/storage/">Storage</a></li><li><a href="/docs/prometheus/latest/federation/">Federation</a></li><li><a href="/docs/prometheus/latest/http_sd/">HTTP SD</a></li><li><a href="/docs/prometheus/latest/management_api/">Management API</a></li><li><span class="nav-subheader">Command Line</span><ul class="nav "><li><a href="/docs/prometheus/latest/command-line/prometheus/">prometheus</a></li><li><a href="/docs/prometheus/latest/command-line/promtool/">promtool</a></li></ul></li><li><a href="/docs/prometheus/latest/migration/">Migration</a></li><li><a href="/docs/prometheus/latest/stability/">API Stability</a></li><li><a href="/docs/prometheus/latest/feature_flags/">Feature flags</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-chart-line"></i> <span>Visualization</span></span><ul class="nav "><li><a href="/docs/visualization/browser/">Expression browser</a></li><li><a href="/docs/visualization/grafana/">Grafana</a></li><li><a href="/docs/visualization/consoles/">Console templates</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-code"></i> <span>Instrumenting</span></span><ul class="nav "><li><a href="/docs/instrumenting/clientlibs/">Client libraries</a></li><li><a href="/docs/instrumenting/writing_clientlibs/">Writing client libraries</a></li><li><a href="/docs/instrumenting/pushing/">Pushing metrics</a></li><li><a href="/docs/instrumenting/exporters/">Exporters and integrations</a></li><li><a href="/docs/instrumenting/writing_exporters/">Writing exporters</a></li><li><a href="/docs/instrumenting/exposition_formats/">Exposition formats</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-settings"></i> <span>Operating</span></span><ul class="nav "><li><a href="/docs/operating/security/">Security</a></li><li><a href="/docs/operating/integrations/">Integrations</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-bell"></i> <span>Alertmanager</span></span><div class="">Version: <select><option value="/docs/alerting/latest/overview/" selected="selected">latest (0.28)</option><option value="/docs/alerting/0.28/overview/" >0.28</option><option value="/docs/alerting/0.27/overview/" >0.27</option><option value="/docs/alerting/0.26/overview/" >0.26</option><option value="/docs/alerting/0.25/overview/" >0.25</option><option value="/docs/alerting/0.24/overview/" >0.24</option><option value="/docs/alerting/0.23/overview/" >0.23</option><option value="/docs/alerting/0.22/overview/" >0.22</option><option value="/docs/alerting/0.21/overview/" >0.21</option><option value="/docs/alerting/0.20/overview/" >0.20</option></select></div><ul class="nav "><li><a href="/docs/alerting/latest/overview/">Alerting overview</a></li><li><a href="/docs/alerting/latest/alertmanager/">Alertmanager</a></li><li><a href="/docs/alerting/latest/configuration/">Configuration</a></li><li><a href="/docs/alerting/latest/clients/">Clients</a></li><li><a href="/docs/alerting/latest/notifications/">Notification template reference</a></li><li><a href="/docs/alerting/latest/notification_examples/">Notification template examples</a></li><li><a href="/docs/alerting/latest/management_api/">Management API</a></li><li><a href="/docs/alerting/latest/https/">HTTPS and authentication</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-thumb-up"></i> <span>Best practices</span></span><ul class="nav "><li><a href="/docs/practices/naming/">Metric and label naming</a></li><li><a href="/docs/practices/consoles/">Consoles and dashboards</a></li><li><a href="/docs/practices/instrumentation/">Instrumentation</a></li><li><a href="/docs/practices/histograms/">Histograms and summaries</a></li><li><a href="/docs/practices/alerting/">Alerting</a></li><li><a href="/docs/practices/rules/">Recording rules</a></li><li><a href="/docs/practices/pushing/">When to use the Pushgateway</a></li><li><a href="/docs/practices/remote_write/">Remote write tuning</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-map"></i> <span>Guides</span></span><ul class="nav "><li><a href="/docs/guides/basic-auth/">Basic auth</a></li><li><a href="/docs/guides/cadvisor/">Monitoring Docker container metrics using cAdvisor</a></li><li><a href="/docs/guides/file-sd/">Use file-based service discovery to discover scrape targets</a></li><li><a href="/docs/guides/go-application/">Instrumenting a Go application</a></li><li><a href="/docs/guides/multi-target-exporter/">Understanding and using the multi-target exporter pattern</a></li><li><a href="/docs/guides/node-exporter/">Monitoring Linux host metrics with the Node Exporter</a></li><li><a href="/docs/guides/opentelemetry/">OpenTelemetry</a></li><li><a href="/docs/guides/utf8/">UTF-8 in Prometheus</a></li><li><a href="/docs/guides/dockerswarm/">Docker Swarm</a></li><li><a href="/docs/guides/query-log/">Query Log</a></li><li><a href="/docs/guides/tls-encryption/">TLS encryption</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-book"></i> <span>Tutorials</span></span><ul class="nav "><li><a href="/docs/tutorials/getting_started/">Getting Started with Prometheus</a></li><li><a href="/docs/tutorials/understanding_metric_types/">Understanding metric types</a></li><li><a href="/docs/tutorials/instrumenting_http_server_in_go/">Instrumenting HTTP server written in Go</a></li><li><a href="/docs/tutorials/visualizing_metrics_using_grafana/">Visualizing metrics using Grafana</a></li><li><a href="/docs/tutorials/alerting_based_on_metrics/">Alerting based on metrics.</a></li></ul></li> <li class="active"><span class="nav-header"><i class="ti ti-file-description"></i> <span>Specifications</span></span><ul class="nav active"><li class="active current"><a href="/docs/specs/remote_write_spec_2_0/">Prometheus Remote-Write 2.0 [EXPERIMENTAL]</a></li><li><a href="/docs/specs/remote_write_spec/">Prometheus Remote-Write 1.0</a></li><li><a href="/docs/specs/native_histograms/">Native Histograms [EXPERIMENTAL]</a></li></ul></li> </ul> </div> <div class="col-md-9 doc-content"> <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd"> <html><body> <h1 id="prometheus-remote-write-specification" class="page-header">Prometheus Remote-Write Specification<a class="header-anchor ti ti-link" href="#prometheus-remote-write-specification" name="prometheus-remote-write-specification"></a> </h1> <div class="toc toc-right"><ul> <li><a href="#introduction">Introduction </a></li> <ul> <li><a href="#background">Background </a></li> <li><a href="#glossary">Glossary </a></li> </ul> <li><a href="#definitions">Definitions </a></li> <ul> <li><a href="#protocol">Protocol </a></li> <li><a href="#response">Response </a></li> <li><a href="#retries-backoff">Retries & Backoff </a></li> <li><a href="#backward-and-forward-compatibility">Backward and Forward Compatibility </a></li> </ul> <li><a href="#protobuf-message">Protobuf Message </a></li> <ul><li><a href="#io-prometheus-write-v2-request"> <code>io.prometheus.write.v2.Request</code> </a></li></ul> <li><a href="#out-of-scope">Out of Scope </a></li> <li><a href="#future-plans">Future Plans </a></li> <li><a href="#related">Related </a></li> <ul><li><a href="#faq">FAQ </a></li></ul> </ul></div> <ul> <li>Version: 2.0-rc.3</li> <li>Status: <strong>Experimental</strong> </li> <li>Date: May 2024</li> </ul> <p>The Remote-Write specification, in general, is intended to document the standard for how Prometheus and Prometheus Remote-Write compatible senders send data to Prometheus or Prometheus Remote-Write compatible receivers.</p> <p>This document is intended to define a second version of the <a href="/docs/specs/remote_write_spec">Prometheus Remote-Write</a> API with minor changes to protocol and semantics. This second version adds a new Protobuf Message with new features enabling more use cases and wider adoption on top of performance and cost savings. The second version also deprecates the previous Protobuf Message from a <a href="/docs/specs/remote_write_spec/#protocol">1.0 Remote-Write specification</a> and adds mandatory <a href="#required-written-response-headers"><code>X-Prometheus-Remote-Write-*-Written</code> HTTP response headers</a>for reliability purposes. Finally, this spec outlines how to implement backwards-compatible senders and receivers (even under a single endpoint) using existing basic content negotiation request headers. More advanced, automatic content negotiation mechanisms might come in a future minor version if needed. For the rationales behind the 2.0 specification, see <a href="https://github.com/prometheus/proposals/pull/35">the formal proposal</a>.</p> <p>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in <a href="https://datatracker.ietf.org/doc/html/rfc2119">RFC 2119</a>.</p> <blockquote> <div class="admonition-wrapper note"><div class="admonition alert alert-info"> <strong>NOTE:</strong> This is a release candidate for Remote-Write 2.0 specification. This means that this specification is currently in an experimental state--no major changes are expected, but we reserve the right to break the compatibility if it's necessary, based on the early adopters' feedback. The potential feedback, questions and suggestions should be added as comments to the <a href="https://github.com/prometheus/proposals/pull/35">PR with the open proposal</a>.</div></div> </blockquote> <h2 id="introduction">Introduction<a class="header-anchor ti ti-link" href="#introduction" name="introduction"></a> </h2> <h3 id="background">Background<a class="header-anchor ti ti-link" href="#background" name="background"></a> </h3> <p>The Remote-Write protocol is designed to make it possible to reliably propagate samples in real-time from a sender to a receiver, without loss.</p>  <p>The Remote-Write protocol is designed to be stateless; there is strictly no inter-message communication. As such the protocol is not considered "streaming". To achieve a streaming effect multiple messages should be sent over the same connection using e.g. HTTP/1.1 or HTTP/2. "Fancy" technologies such as gRPC were considered, but at the time were not widely adopted, and it was challenging to expose gRPC services to the internet behind load balancers such as an AWS EC2 ELB.</p> <p>The Remote-Write protocol contains opportunities for batching, e.g. sending multiple samples for different series in a single request. It is not expected that multiple samples for the same series will be commonly sent in the same request, although there is support for this in the Protobuf Message.</p> <p>A test suite can be found at <a href="https://github.com/prometheus/compliance/tree/main/remote_write_sender">https://github.com/prometheus/compliance/tree/main/remote_write_sender</a>. The compliance tests for remote write 2.0 compatibility are still <a href="https://github.com/prometheus/compliance/issues/101">in progress</a>.</p> <h3 id="glossary">Glossary<a class="header-anchor ti ti-link" href="#glossary" name="glossary"></a> </h3> <p>In this document, the following definitions are followed:</p> <ul> <li> <code>Remote-Write</code> is the name of this Prometheus protocol.</li> <li>a <code>Protocol</code> is a communication specification that enables the client and server to transfer metrics.</li> <li>a <code>Protobuf Message</code> (or Proto Message) refers to the <a href="https://www.rfc-editor.org/rfc/rfc9110.html#name-content-type">content type</a> definition of the data structure for this Protocol. Since the specification uses <a href="https://protobuf.dev/">Google Protocol Buffers ("protobuf")</a> exclusively, the schema is defined in a <a href="https://protobuf.dev/programming-guides/proto3/">"proto" file</a> and represented by a single Protobuf <a href="https://protobuf.dev/programming-guides/proto3/#simple">"message"</a>.</li> <li>a <code>Wire Format</code> is the format of the data as it travels on the wire (i.e. in a network). In the case of Remote-Write, this is always the compressed binary protobuf format.</li> <li>a <code>Sender</code> is something that sends Remote-Write data.</li> <li>a <code>Receiver</code> is something that receives (writes) Remote-Write data. The meaning of <code>Written</code> is up to the Receiver e.g. usually it means storing received data in a database, but also just validating, splitting or enhancing it.</li> <li> <code>Written</code> refers to data the <code>Receiver</code> has received and is accepting. Whether or not it has ingested this data to persistent storage, written it to a WAL, etc. is up to the <code>Receiver</code>. The only distinction is that the <code>Receiver</code> has accepted this data rather than explicitly rejecting it with an error response.</li> <li>a <code>Sample</code> is a pair of (timestamp, value).</li> <li>a <code>Histogram</code> is a pair of (timestamp, <a href="https://github.com/prometheus/docs/blob/b9657b5f5b264b81add39f6db2f1df36faf03efe/content/docs/concepts/native_histograms.md">histogram value</a>).</li> <li>a <code>Label</code> is a pair of (key, value).</li> <li>a <code>Series</code> is a list of samples, identified by a unique set of labels.</li> </ul> <h2 id="definitions">Definitions<a class="header-anchor ti ti-link" href="#definitions" name="definitions"></a> </h2> <h3 id="protocol">Protocol<a class="header-anchor ti ti-link" href="#protocol" name="protocol"></a> </h3> <p>The Remote-Write Protocol MUST consist of RPCs with the request body serialized using a Google Protocol Buffers and then compressed.</p>  <p>The protobuf serialization MUST use either of the following Protobuf Messages:</p> <ul> <li>The <code>prometheus.WriteRequest</code> introduced in <a href="./remote_write_spec.md#protocol">the Remote-Write 1.0 specification</a>. As of 2.0, this message is deprecated. It SHOULD be used only for compatibility reasons. Senders and Receivers MAY NOT support the <code>prometheus.WriteRequest</code>.</li> <li>The <code>io.prometheus.write.v2.Request</code> introduced in this specification and defined <a href="#protobuf-message">below</a>. Senders and Receivers SHOULD use this message when possible. Senders and Receivers MUST support the <code>io.prometheus.write.v2.Request</code>.</li> </ul> <p>Protobuf Message MUST use binary Wire Format. Then, MUST be compressed with <a href="https://github.com/google/snappy">Google’s Snappy</a>. Snappy's <a href="https://github.com/google/snappy/blob/2c94e11145f0b7b184b831577c93e5a41c4c0346/format_description.txt">block format</a> MUST be used -- <a href="https://github.com/google/snappy/blob/2c94e11145f0b7b184b831577c93e5a41c4c0346/framing_format.txt">the framed format</a> MUST NOT be used.</p> <p>Senders MUST send a serialized and compressed Protobuf Message in the body of an HTTP POST request and send it to the Receiver via HTTP at the provided URL path. Receivers MAY specify any HTTP URL path to receive metrics.</p>  <p>Senders MUST send the following reserved headers with the HTTP request:</p> <ul> <li><code>Content-Encoding</code></li> <li><code>Content-Type</code></li> <li><code>X-Prometheus-Remote-Write-Version</code></li> <li><code>User-Agent</code></li> </ul> <p>Senders MAY allow users to add custom HTTP headers; they MUST NOT allow users to configure them in such a way as to send reserved headers.</p> <h4 id="content-encoding">Content-Encoding<a class="header-anchor ti ti-link" href="#content-encoding" name="content-encoding"></a> </h4> <pre><code>Content-Encoding: <compression> </code></pre>  <p>Content encoding request header MUST follow <a href="https://www.rfc-editor.org/rfc/rfc9110.html#name-content-encoding">the RFC 9110</a>. Senders MUST use the <code>snappy</code> value. Receivers MUST support <code>snappy</code> compression. New, optional compression algorithms might come in 2.x or beyond.</p> <h4 id="content-type">Content-Type<a class="header-anchor ti ti-link" href="#content-type" name="content-type"></a> </h4> <pre><code>Content-Type: application/x-protobuf Content-Type: application/x-protobuf;proto=<fully qualified name> </code></pre> <p>Content type request header MUST follow <a href="https://www.rfc-editor.org/rfc/rfc9110.html#name-content-type">the RFC 9110</a>. Senders MUST use <code>application/x-protobuf</code> as the only media type. Senders MAY add <code>;proto=</code> parameter to the header's value to indicate the fully qualified name of the Protobuf Message that was used, from the two mentioned above. As a result, Senders MUST send any of the three supported header values:</p> <p>For the deprecated message introduced in PRW 1.0, identified by <code>prometheus.WriteRequest</code>:</p> <ul> <li><code>Content-Type: application/x-protobuf</code></li> <li><code>Content-Type: application/x-protobuf;proto=prometheus.WriteRequest</code></li> </ul> <p>For the message introduced in PRW 2.0, identified by <code>io.prometheus.write.v2.Request</code>:</p> <ul> <li><code>Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request</code></li> </ul> <p>When talking to 1.x Receivers, Senders SHOULD use <code>Content-Type: application/x-protobuf</code> for backward compatibility. Otherwise, Senders SHOULD use <code>Content-Type: application/x-protobuf;proto=io.prometheus.write.v2.Request</code>. More Protobuf Messages might come in 2.x or beyond.</p> <p>Receivers MUST use the content type header to identify the Protobuf Message schema to use. Accidental wrong schema choices may result in non-deterministic behaviour (e.g. corruptions).</p> <blockquote> <div class="admonition-wrapper note"><div class="admonition alert alert-info"> <strong>NOTE:</strong> Thanks to reserved fields in <a href="#protobuf-message"><code>io.prometheus.write.v2.Request</code></a>, Receiver accidental use of wrong schema with <code>prometheus.WriteRequest</code> will result in empty message. This is generally for convenience to avoid surprising errors, but don't rely on it -- future Protobuf Messages might not have this feature.</div></div> </blockquote> <h4 id="x-prometheus-remote-write-version">X-Prometheus-Remote-Write-Version<a class="header-anchor ti ti-link" href="#x-prometheus-remote-write-version" name="x-prometheus-remote-write-version"></a> </h4> <pre><code>X-Prometheus-Remote-Write-Version: <Remote-Write spec major and minor version> </code></pre> <p>When talking to 1.x Receivers, Senders MUST use <code>X-Prometheus-Remote-Write-Version: 0.1.0</code> for backward compatibility. Otherwise, Senders SHOULD use the newest Remote-Write version it is compatible with e.g. <code>X-Prometheus-Remote-Write-Version: 2.0.0</code>.</p> <h4 id="user-agent">User-Agent<a class="header-anchor ti ti-link" href="#user-agent" name="user-agent"></a> </h4> <pre><code>User-Agent: <name & version of the Sender> </code></pre> <p>Senders MUST include a user agent header that SHOULD follow <a href="https://www.rfc-editor.org/rfc/rfc9110.html#name-user-agent">the RFC 9110 User-Agent header format</a>.</p> <h3 id="response">Response<a class="header-anchor ti ti-link" href="#response" name="response"></a> </h3> <p>Receivers that written all data successfully MUST return a <a href="https://www.rfc-editor.org/rfc/rfc9110.html#name-successful-2xx">success 2xx HTTP status code</a>. In such a successful case, the response body from the Receiver SHOULD be empty and the status code SHOULD be <a href="https://www.rfc-editor.org/rfc/rfc9110.html#name-204-no-content">204 HTTP No Content</a>; Senders MUST ignore the response body. The response body is RESERVED for future use.</p> <p>Receivers MUST NOT return a 2xx HTTP status code if any of the pieces of sent data known to the Receiver (e.g. Samples, Histograms, Exemplars) were NOT written successfully (both <a href="#partial-write">partial write</a> or full write rejection). In such a case, the Receiver MUST provide a human-readable error message in the response body. The Receiver's error SHOULD contain information about the amount of the samples being rejected and for what reasons. Senders MUST NOT try and interpret the error message and SHOULD log it as is.</p> <p>The following subsections specify Sender and Receiver semantics around headers and different write error cases.</p> <h4 id="required-written-response-headers">Required <code>Written</code> Response Headers<a class="header-anchor ti ti-link" href="#required-written-response-headers" name="required-written-response-headers"></a> </h4>  <p>Upon a successful content negotiation, Receivers process (write) the received batch of data. Once completed (with success or failure) for each important piece of data (currently Samples, Histograms and Exemplars) Receivers MUST send a dedicated HTTP <code>X-Prometheus-Remote-Write-*-Written</code> response header with the precise number of successfully written elements.</p> <p>Each header value MUST be a single 64-bit integer. The header names MUST be as follows:</p> <pre><code>X-Prometheus-Remote-Write-Samples-Written <count of all successfully written Samples> X-Prometheus-Remote-Write-Histograms-Written <count of all successfully written Histogram samples> X-Prometheus-Remote-Write-Exemplars-Written <count of all successfully written Exemplars> </code></pre> <p>Upon receiving a 2xx or a 4xx status code, Senders CAN assume that any missing <code>X-Prometheus-Remote-Write-*-Written</code> response header means no element from this category (e.g. Sample) was written by the Receiver (count of <code>0</code>). Senders MUST NOT assume the same when using the deprecated <code>prometheus.WriteRequest</code> Protobuf Message due to the risk of hitting 1.0 Receiver without this feature.</p> <p>Senders MAY use those headers to confirm which parts of data were successfully written by the Receiver. Common use cases:</p> <ul> <li>Better handling of the <a href="#partial-write">Partial Write</a> failure situations: Senders MAY use those headers for more accurate client instrumentation and error handling.</li> <li>Detecting broken 1.0 Receiver implementations: Senders SHOULD assume <a href="https://www.rfc-editor.org/rfc/rfc9110.html#name-415-unsupported-media-type">415 HTTP Unsupported Media Type</a> status code when sending the data using <code>io.prometheus.write.v2.Request</code> request and receiving 2xx HTTP status code, but none of the <code>X-Prometheus-Remote-Write-*-Written</code> response headers from the Receiver. This is a common issue for the 1.0 Receivers that do not check the <code>Content-Type</code> request header; accidental decoding of the <code>io.prometheus.write.v2.Request</code> payload with <code>prometheus.WriteRequest</code> schema results in empty result and no decoding errors.</li> <li>Detecting other broken implementations or issues: Senders MAY use those headers to detect broken Sender and Receiver implementations or other problems.</li> </ul> <p>Senders MUST NOT assume what Remote Write specification version the Receiver implements from the remote write response headers.</p> <p>More (optional) headers might come in the future, e.g. when more entities or fields are added and worth confirming.</p> <h4 id="partial-write">Partial Write<a class="header-anchor ti ti-link" href="#partial-write" name="partial-write"></a> </h4>  <p>Senders SHOULD use Remote-Write to send samples for multiple series in a single request. As a result, Receivers MAY write valid samples within a write request that also contains some invalid or otherwise unwritten samples, which represents a partial write case. In such a case, the Receiver MUST return non-2xx status code following the <a href="#invalid-samples">Invalid Samples</a> and <a href="#retries-on-partial-writes">Retry on Partial Writes</a> sections.</p> <h4 id="unsupported-request-content">Unsupported Request Content<a class="header-anchor ti ti-link" href="#unsupported-request-content" name="unsupported-request-content"></a> </h4> <p>Receivers MUST return <a href="https://www.rfc-editor.org/rfc/rfc9110.html#name-415-unsupported-media-type">415 HTTP Unsupported Media Type</a> status code if they don't support a given content type or encoding provided by Senders.</p> <p>Senders SHOULD expect <a href="https://www.rfc-editor.org/rfc/rfc9110.html#name-400-bad-request">400 HTTP Bad Request</a> for the above reasons from 1.x Receivers, for backwards compatibility.</p> <h4 id="invalid-samples">Invalid Samples<a class="header-anchor ti ti-link" href="#invalid-samples" name="invalid-samples"></a> </h4> <p>Receivers MAY NOT support certain metric types or samples (e.g. a Receiver might reject sample without metadata type specified or without created timestamp, while another Receiver might accept such sample.). It’s up to the Receiver what sample is invalid. Receivers MUST return a <a href="https://www.rfc-editor.org/rfc/rfc9110.html#name-400-bad-request">400 HTTP Bad Request</a> status code for write requests that contain any invalid samples unless the <a href="#retries-on-partial-writes">partial retriable write</a> occurs.</p> <p>Senders MUST NOT retry on a 4xx HTTP status codes (other than <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429">429</a>), which MUST be used by Receivers to indicate that the write operation will never be able to succeed and should not be retried. Senders MAY retry on the 415 HTTP status code with a different content type or encoding to see if the Receiver supports it.</p> <h3 id="retries-backoff">Retries & Backoff<a class="header-anchor ti ti-link" href="#retries-backoff" name="retries-backoff"></a> </h3> <p>Receivers MAY return a <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429">429 HTTP Too Many Requests</a> status code to indicate the overloaded server situation. Receivers MAY return <a href="https://www.rfc-editor.org/rfc/rfc9110.html#name-retry-after">the Retry-After</a> header to indicate the time for the next write attempt. Receivers MAY return a 5xx HTTP status code to represent internal server errors.</p> <p>Senders MAY retry on a 429 HTTP status code. Senders MUST retry write requests on 5xx HTTP. Senders MUST use a backoff algorithm to prevent overwhelming the server. Senders MAY handle <a href="https://www.rfc-editor.org/rfc/rfc9110.html#name-retry-after">the Retry-After response header</a> to estimate the next retry time.</p> <p>The difference between 429 vs 5xx handling is due to the potential situation of a Sender “falling behind” when the Receiver cannot keep up with the request volume, or the Receiver choosing to rate limit the Sender to protect its availability. As a result, Senders has the option to NOT retry on 429, which allows progress to be made when there are Sender side errors (e.g. too much traffic), while the data is not lost when there are Receiver side errors (5xx).</p> <h4 id="retries-on-partial-writes">Retries on Partial Writes<a class="header-anchor ti ti-link" href="#retries-on-partial-writes" name="retries-on-partial-writes"></a> </h4> <p>Receivers MAY return a 5xx HTTP or 429 HTTP status code on partial write or <a href="#partial-write">partial invalid sample cases</a> when it expects Senders to retry the whole request. In that case, the Receiver MUST support idempotency as Senders MAY retry with the same request.</p> <h3 id="backward-and-forward-compatibility">Backward and Forward Compatibility<a class="header-anchor ti ti-link" href="#backward-and-forward-compatibility" name="backward-and-forward-compatibility"></a> </h3> <p>The protocol follows <a href="https://semver.org/">semantic versioning 2.0</a>: any 2.x compatible Receiver MUST be able to read any 2.x compatible Senders and vice versa. Breaking or backwards incompatible changes will result in a 3.x version of the spec.</p> <p>The Protobuf Messages (in Wire Format) themselves are forward / backward compatible, in some respects:</p> <ul> <li>Removing fields from the Protobuf Message requires a major version bump.</li> <li>Adding (optional) fields can be done in a minor version bump.</li> </ul> <p>In other words, this means that future minor versions of 2.x MAY add new optional fields to <code>io.prometheus.write.v2.Request</code>, new compressions, Protobuf Messages and negotiation mechanisms, as long as they are backwards compatible (e.g. optional to both Receiver and Sender).</p> <h4 id="2-x-vs-1-x-compatibility">2.x vs 1.x Compatibility<a class="header-anchor ti ti-link" href="#2-x-vs-1-x-compatibility" name="2-x-vs-1-x-compatibility"></a> </h4> <p>The 2.x protocol is breaking compatibility with 1.x by introducing a new, mandatory <code>io.prometheus.write.v2.Request</code> Protobuf Message and deprecating the <code>prometheus.WriteRequest</code>.</p> <p>2.x Senders MAY support 1.x Receivers by allowing users to configure what content type Senders should use. 2.x Senders also MAY automatically fall back to different content types, if the Receiver returns 415 HTTP status code.</p> <h2 id="protobuf-message">Protobuf Message<a class="header-anchor ti ti-link" href="#protobuf-message" name="protobuf-message"></a> </h2> <h3 id="io-prometheus-write-v2-request"> <code>io.prometheus.write.v2.Request</code><a class="header-anchor ti ti-link" href="#io-prometheus-write-v2-request" name="io-prometheus-write-v2-request"></a> </h3> <p>The <code>io.prometheus.write.v2.Request</code> references the new Protobuf Message that's meant to replace and deprecate the Remote-Write 1.0's <code>prometheus.WriteRequest</code> message.</p>  <p>The full schema and source of the truth is in Prometheus repository in <a href="https://github.com/prometheus/prometheus/blob/remote-write-2.0/prompb/io/prometheus/write/v2/types.proto#L32"><code>prompb/io/prometheus/write/v2/types.proto</code></a>. The <code>gogo</code> dependency and options CAN be ignored (<a href="https://github.com/prometheus/prometheus/issues/11908">will be removed eventually</a>). They are not part of the specification as they don't impact the serialized format.</p> <p>The simplified version of the new <code>io.prometheus.write.v2.Request</code> is presented below.</p> <pre><code>// Request represents a request to write the given timeseries to a remote destination. message Request { // Since Request supersedes 1.0 spec's prometheus.WriteRequest, we reserve the top-down message // for the deterministic interop between those two. // Generally it's not needed, because Receivers must use the Content-Type header, but we want to // be sympathetic to adopters with mistaken implementations and have deterministic error (empty // message if you use the wrong proto schema). reserved 1 to 3; // symbols contains a de-duplicated array of string elements used for various // items in a Request message, like labels and metadata items. For the sender's convenience // around empty values for optional fields like unit_ref, symbols array MUST start with // empty string. // // To decode each of the symbolized strings, referenced, by "ref(s)" suffix, you // need to lookup the actual string by index from symbols array. The order of // strings is up to the sender. The receiver should not assume any particular encoding. repeated string symbols = 4; // timeseries represents an array of distinct series with 0 or more samples. repeated TimeSeries timeseries = 5; } // TimeSeries represents a single series. message TimeSeries { // labels_refs is a list of label name-value pair references, encoded // as indices to the Request.symbols array. This list's length is always // a multiple of two, and the underlying labels should be sorted lexicographically. // // Note that there might be multiple TimeSeries objects in the same // Requests with the same labels e.g. for different exemplars, metadata // or created timestamp. repeated uint32 labels_refs = 1; // Timeseries messages can either specify samples or (native) histogram samples // (histogram field), but not both. For a typical sender (real-time metric // streaming), in healthy cases, there will be only one sample or histogram. // // Samples and histograms are sorted by timestamp (older first). repeated Sample samples = 2; repeated Histogram histograms = 3; // exemplars represents an optional set of exemplars attached to this series' samples. repeated Exemplar exemplars = 4; // metadata represents the metadata associated with the given series' samples. Metadata metadata = 5; // created_timestamp represents an optional created timestamp associated with // this series' samples in ms format, typically for counter or histogram type // metrics. Created timestamp represents the time when the counter started // counting (sometimes referred to as start timestamp), which can increase // the accuracy of query results. // // Note that some receivers might require this and in return fail to // write such samples within the Request. // // For Go, see github.com/prometheus/prometheus/model/timestamp/timestamp.go // for conversion from/to time.Time to Prometheus timestamp. // // Note that the "optional" keyword is omitted due to // https://cloud.google.com/apis/design/design_patterns.md#optional_primitive_fields // Zero value means value not set. If you need to use exactly zero value for // the timestamp, use 1 millisecond before or after. int64 created_timestamp = 6; } // Exemplar represents additional information attached to some series' samples. message Exemplar { // labels_refs is an optional list of label name-value pair references, encoded // as indices to the Request.symbols array. This list's len is always // a multiple of 2, and the underlying labels should be sorted lexicographically. // If the exemplar references a trace it should use the `trace_id` label name, as a best practice. repeated uint32 labels_refs = 1; // value represents an exact example value. This can be useful when the exemplar // is attached to a histogram, which only gives an estimated value through buckets. double value = 2; // timestamp represents the timestamp of the exemplar in ms. // For Go, see github.com/prometheus/prometheus/model/timestamp/timestamp.go // for conversion from/to time.Time to Prometheus timestamp. int64 timestamp = 3; } // Sample represents series sample. message Sample { // value of the sample. double value = 1; // timestamp represents timestamp of the sample in ms. int64 timestamp = 2; } // Metadata represents the metadata associated with the given series' samples. message Metadata { enum MetricType { METRIC_TYPE_UNSPECIFIED = 0; METRIC_TYPE_COUNTER = 1; METRIC_TYPE_GAUGE = 2; METRIC_TYPE_HISTOGRAM = 3; METRIC_TYPE_GAUGEHISTOGRAM = 4; METRIC_TYPE_SUMMARY = 5; METRIC_TYPE_INFO = 6; METRIC_TYPE_STATESET = 7; } MetricType type = 1; // help_ref is a reference to the Request.symbols array representing help // text for the metric. Help is optional, reference should point to an empty string in // such a case. uint32 help_ref = 3; // unit_ref is a reference to the Request.symbols array representing a unit // for the metric. Unit is optional, reference should point to an empty string in // such a case. uint32 unit_ref = 4; } // A native histogram, also known as a sparse histogram. // See https://github.com/prometheus/prometheus/blob/remote-write-2.0/prompb/io/prometheus/write/v2/types.proto#L142 // for a full message that follows the native histogram spec for both sparse // and exponential, as well as, custom bucketing. message Histogram { ... } </code></pre> <p>All timestamps MUST be int64 counted as milliseconds since the Unix epoch. Sample's values MUST be float64.</p> <p>For every <code>TimeSeries</code> message:</p> <ul> <li> <code>labels_refs</code> MUST be provided.</li> </ul>  <ul> <li>At least one element in <code>samples</code> or in <code>histograms</code> MUST be provided. A <code>TimeSeries</code> MUST NOT include both <code>samples</code> and <code>histograms</code>. For series which (rarely) would mix float and histogram samples, a separate <code>TimeSeries</code> message MUST be used.</li> </ul>  <ul> <li> <code>metadata</code> sub-fields SHOULD be provided. Receivers MAY reject series with unspecified <code>Metadata.type</code>.</li> <li>Exemplars SHOULD be provided if they exist for a series.</li> <li> <code>created_timestamp</code> SHOULD be provided for metrics that follow counter semantics (e.g. counters and histograms). Receivers MAY reject those series without <code>created_timestamp</code> being set.</li> </ul> <p>The following subsections define some schema elements in detail.</p> <h4 id="symbols">Symbols<a class="header-anchor ti ti-link" href="#symbols" name="symbols"></a> </h4>  <p>The <code>io.prometheus.write.v2.Request</code> Protobuf Message is designed to <a href="https://en.wikipedia.org/wiki/String_interning">intern all strings</a> for the proven additional compression and memory efficiency gains on top of the standard compressions.</p> <p>The <code>symbols</code> table MUST be provided and it MUST contain deduplicated strings used in series, exemplar labels, and metadata strings. The first element of the <code>symbols</code> table MUST be an empty string, which is used to represent empty or unspecified values such as when <code>Metadata.unit_ref</code> or <code>Metadata.help_ref</code> are not provided. References MUST point to the existing index in the <code>symbols</code> string array.</p> <h4 id="series-labels">Series Labels<a class="header-anchor ti ti-link" href="#series-labels" name="series-labels"></a> </h4>  <p>The complete set of labels MUST be sent with each <code>Sample</code> or <code>Histogram</code> sample. Additionally, the label set associated with samples:</p> <ul> <li>SHOULD contain a <code>__name__</code> label.</li> <li>MUST NOT contain repeated label names.</li> <li>MUST have label names sorted in lexicographical order.</li> <li>MUST NOT contain any empty label names or values.</li> </ul> <p>Metric names, label names, and label values MUST be any sequence of UTF-8 characters.</p> <p>Metric names SHOULD adhere to the regex <code>[a-zA-Z_:]([a-zA-Z0-9_:])*</code>.</p> <p>Label names SHOULD adhere to the regex <code>[a-zA-Z_]([a-zA-Z0-9_])*</code>.</p> <p>Names that do not adhere to the above, might be harder to use for PromQL users (see <a href="https://github.com/prometheus/proposals/blob/main/proposals/2023-08-21-utf8.md">the UTF-8 proposal for more details</a>).</p> <p>Label names beginning with "__" are RESERVED for system usage and SHOULD NOT be used, see <a href="https://prometheus.io/docs/concepts/data_model/">Prometheus Data Model</a>.</p> <p>Receivers also MAY impose limits on the number and length of labels, but this is receiver-specific and is out of the scope of this document.</p> <h4 id="samples-and-histogram-samples">Samples and Histogram Samples<a class="header-anchor ti ti-link" href="#samples-and-histogram-samples" name="samples-and-histogram-samples"></a> </h4>  <p>Senders MUST send <code>samples</code> (or <code>histograms</code>) for any given <code>TimeSeries</code> in timestamp order. Senders MAY send multiple requests for different series in parallel.</p>  <p>Senders SHOULD send stale markers when a time series will no longer be appended to. Senders MUST send stale markers if the discontinuation of time series is possible to detect, for example:</p> <ul> <li>For series that were pulled (scraped), unless explicit timestamp was used.</li> <li>For series that is resulted by a recording rule evaluation.</li> </ul> <p>Generally, not sending stale markers for series that are discontinued can lead to the Receiver <a href="https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness">non-trivial query time alignment issues</a>.</p> <p>Stale markers MUST be signalled by the special NaN value <code>0x7ff0000000000002</code>. This value MUST NOT be used otherwise.</p> <p>Typically, Senders can detect when a time series will no longer be appended using the following techniques:</p> <ol> <li>Detecting, using service discovery, that the target exposing the series has gone away.</li> <li>Noticing the target is no longer exposing the time series between successive scrapes.</li> <li>Failing to scrape the target that originally exposed a time series.</li> <li>Tracking configuration and evaluation for recording and alerting rules.</li> <li>Tracking discontinuation of metrics for non-scrape source of metric (e.g. in k6 when the benchmark has finished for series per benchmark, it could emit a stale marker).</li> </ol> <h4 id="metadata">Metadata<a class="header-anchor ti ti-link" href="#metadata" name="metadata"></a> </h4> <p>Metadata SHOULD follow the official Prometheus guidelines for <a href="https://prometheus.io/docs/instrumenting/writing_exporters/#types">Type</a> and <a href="https://prometheus.io/docs/instrumenting/writing_exporters/#help-strings">Help</a>.</p> <p>Metadata MAY follow the official OpenMetrics guidelines for <a href="https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md#unit">Unit</a>.</p> <h4 id="exemplars">Exemplars<a class="header-anchor ti ti-link" href="#exemplars" name="exemplars"></a> </h4> <p>Each exemplar, if attached to a <code>TimeSeries</code>:</p> <ul> <li>MUST contain a value.</li> </ul>  <ul> <li>MAY contain labels e.g. referencing trace or request ID. If the exemplar references a trace it SHOULD use the <code>trace_id</code> label name, as a best practice.</li> <li>MUST contain a timestamp. While exemplar timestamps are optional in Prometheus/Open Metrics exposition formats, the assumption is that a timestamp is assigned at scrape time in the same way a timestamp is assigned to the scrape sample. Receivers require exemplar timestamps to reliably handle (e.g. deduplicate) incoming exemplars.</li> </ul> <h2 id="out-of-scope">Out of Scope<a class="header-anchor ti ti-link" href="#out-of-scope" name="out-of-scope"></a> </h2> <p>The same as in <a href="./remote_write_spec.md#out-of-scope">1.0</a>.</p> <h2 id="future-plans">Future Plans<a class="header-anchor ti ti-link" href="#future-plans" name="future-plans"></a> </h2> <p>This section contains speculative plans that are not considered part of protocol specification yet but are mentioned here for completeness. Note that 2.0 specification completed <a href="./remote_write_spec.md#future-plans">2 of 3 future plans in the 1.0</a>.</p> <ul> <li> <strong>Transactionality</strong> There is still no transactionality defined for 2.0 specification, mostly because it makes a scalable Sender implementation difficult. Prometheus Sender aims at being "transactional" - i.e. to never expose a partially scraped target to a query. We intend to do the same with Remote-Write -- for instance, in the future we would like to "align" Remote-Write with scrapes, perhaps such that all the samples, metadata and exemplars for a single scrape are sent in a single Remote-Write request.</li> </ul> <p>However, Remote-Write 2.0 specification solves an important transactionality problem for <a href="https://docs.google.com/document/d/1mpcSWH1B82q-BtJza-eJ8xMLlKt6EJ9oFGH325vtY1Q/edit#heading=h.ueg7q07wymku">the classic histogram buckets</a>. This is done thanks to the native histograms supporting custom bucket-ing possible with the <code>io.prometheus.write.v2.Request</code> wire format. Senders might translate all classic histograms to native histograms this way, but it's out of this specification to mandate this. However, for this reason, Receivers MAY ignore certain metric types (e.g. classic histograms).</p> <ul> <li><p><strong>Alternative wire formats</strong>. The OpenTelemetry community has shown the validity of Apache Arrow (and potentially other columnar formats) for over-wire data transfer with their OTLP protocol. We would like to do experiments to confirm the compatibility of a similar format with Prometheus’ data model and include benchmarks of any resource usage changes. We would potentially maintain both a protobuf and columnar format long term for compatibility reasons and use our content negotiation to add different Protobuf Messages for this purpose.</p></li> <li><p><strong>Global symbols</strong>. Pre-defined string dictionary for interning The protocol could pre-define a static dictionary of ref->symbol that includes strings that are considered common, e.g. “namespace”, “le”, “job”, “seconds”, “bytes”, etc. Senders could refer to these without the need to include them in the request’s symbols table. This dictionary could incrementally grow with minor version releases of this protocol.</p></li> </ul> <h2 id="related">Related<a class="header-anchor ti ti-link" href="#related" name="related"></a> </h2> <h3 id="faq">FAQ<a class="header-anchor ti ti-link" href="#faq" name="faq"></a> </h3> <p><strong>Why did you not use gRPC?</strong> Because the 1.0 protocol does not use gRPC, breaking it would increase friction in the adoption. See 1.0 <a href="./remote_write_spec.md#faq">reason</a>.</p> <p><strong>Why not stream protobuf messages?</strong> If you use persistent HTTP/1.1 connections, they are pretty close to streaming. Of course, headers have to be re-sent, but that is less expensive than a new TCP set up.</p> <p><strong>Why do we send samples in order?</strong> The in-order constraint comes from the encoding we use for time series data in Prometheus, the implementation of which is optimized for append-only workloads. However, this requirement is also shared across many other databases and vendors in the ecosystem. In fact, <a href="https://youtu.be/qYsycK3nTSQ?t=1321">Prometheus with OOO feature enabled</a>, allows out-of-order writes, but with the performance penalty, thus reserved for rare events. To sum up, Receivers may support out-of-order write, though it is not permitted by the specification. In the future e.g. 2.x spec versions, we could extend content type to negotiate the out-of-order writes, if needed.</p> <p><strong>How can we parallelise requests with the in-order constraint?</strong> Samples must be in-order <em>for a given series</em>. However, even if a Receiver does not support out-of-order write, the Remote-Write requests can be sent in parallel as long as they are for different series. Prometheus shards the samples by their labels into separate queues, and then writes happen sequentially in each queue. This guarantees samples for the same series are delivered in order, but samples for different series are sent in parallel - and potentially "out of order" between different series.</p> <p><strong>What are the differences between Remote-Write 2.0 and OpenTelemetry's OTLP protocol?</strong> <a href="https://github.com/open-telemetry/opentelemetry-proto/blob/a05597bff803d3d9405fcdd1e1fb1f42bed4eb7a/docs/specification.md">OpenTelemetry OTLP</a> is a protocol for transporting of telemetry data (such as metrics, logs, traces and profiles) between telemetry sources, intermediate nodes and telemetry backends. The recommended transport involves gRPC with protobuf, but HTTP with protobuf or JSON are also described. It was designed from scratch with the intent to support a variety of different observability signals, data types and extra information. For <a href="https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/metrics/v1/metrics.proto">metrics</a> that means additional non-identifying labels, flags, temporal aggregations types, resource or scoped metrics, schema URLs and more. OTLP also requires <a href="https://opentelemetry.io/docs/concepts/semantic-conventions/">the semantic convention</a> to be used.</p> <p>Remote-Write was designed for simplicity, efficiency and organic growth. The first version was officially released in 2023, when already <a href="./remote_write_spec.md#compatible-senders-and-receivers">dozens of battle-tested adopters in the CNCF ecosystem</a> had been using this protocol for years. Remote-Write 2.0 iterates on the previous protocol by adding a few new elements (metadata, exemplars, created timestamp and native histograms) and string interning. Remote-Write 2.0 is always stateless, focuses only on metrics and is opinionated; as such it is scoped down to elements that the Prometheus community considers enough to have a robust metric solution. The intention is to ensure the Remote-Write is a stable protocol that is cheaper and simpler to adopt and use than the alternatives in the observability ecosystem.</p> </body></html> <p class="open-source-notice"> <i class="fa fa-file"></i> This documentation is <a href="https://github.com/prometheus/docs#contributing-changes">open-source</a>. Please help improve it by filing issues or pull requests. </p> </div> </div> <hr> <footer> <p class="pull-left"> © Prometheus Authors 2014-2025 | Documentation Distributed under CC-BY-4.0 </p> <p class="pull-left"> © 2025 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our <a href="https://www.linuxfoundation.org/trademark-usage">Trademark Usage</a> page. </p> </footer> </div>   <script src="https://code.jquery.com/jquery-3.6.0.min.js" integrity="sha256-/xUj+3OJU5yExlq6GSYGSHk7tPXikynS7ogEvDej/m4=" crossorigin="anonymous"></script> <script src="/assets/bootstrap-3.4.1/js/bootstrap.min.js"></script> <script src="/assets/docs.js"></script>  <script src="/assets/ie10-viewport-bug-workaround.js"></script>  <script type="text/javascript" src="https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.js"></script> <script type="text/javascript"> docsearch({ apiKey: '48ac0b7924908a1fd40b1cb18b402ba1', indexName: 'prometheus', inputSelector: '.searchbox', // Search all pages that either are the latest or are not part of the versioned Prometheus docs subtrees. algoliaOptions: { 'filters': 'include:true' }, debug: false // Set debug to true if you want to inspect the dropdown }); </script> </body> </html>