CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="description" content="An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach."> <meta name="keywords" content="prometheus, monitoring, monitoring system, time series, time series database, alerting, metrics, telemetry"> <meta name="author" content="Prometheus"> <meta name="twitter:card" content="summary"> <meta property="og:description" content="An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach."> <meta property="og:image" content="https://prometheus.io/assets/favicons/android-chrome-192x192.png"> <link rel="alternate" type="application/atom+xml" title="Prometheus Blog » Feed" href="/blog/feed.xml"> <link rel="shortcut icon" href="/assets/favicons/favicon.ico"> <link rel="apple-touch-icon" sizes="57x57" href="/assets/favicons/apple-touch-icon-57x57.png"> <link rel="apple-touch-icon" sizes="60x60" href="/assets/favicons/apple-touch-icon-60x60.png"> <link rel="apple-touch-icon" sizes="72x72" href="/assets/favicons/apple-touch-icon-72x72.png"> <link rel="apple-touch-icon" sizes="76x76" href="/assets/favicons/apple-touch-icon-76x76.png"> <link rel="apple-touch-icon" sizes="114x114" href="/assets/favicons/apple-touch-icon-114x114.png"> <link rel="apple-touch-icon" sizes="120x120" href="/assets/favicons/apple-touch-icon-120x120.png"> <link rel="apple-touch-icon" sizes="144x144" href="/assets/favicons/apple-touch-icon-144x144.png"> <link rel="apple-touch-icon" sizes="152x152" href="/assets/favicons/apple-touch-icon-152x152.png"> <link rel="apple-touch-icon" sizes="180x180" href="/assets/favicons/apple-touch-icon-180x180.png"> <link rel="icon" type="image/png" href="/assets/favicons/favicon-32x32.png" sizes="32x32"> <link rel="icon" type="image/png" href="/assets/favicons/android-chrome-192x192.png" sizes="192x192"> <link rel="icon" type="image/png" href="/assets/favicons/favicon-96x96.png" sizes="96x96"> <link rel="icon" type="image/png" href="/assets/favicons/favicon-16x16.png" sizes="16x16"> <link rel="manifest" href="/assets/favicons/android-chrome-manifest.json">  <meta name="docsearch:prometheus-version" content="none" /> <meta name="docsearch:include" content="true" /> <meta name="msapplication-TileColor" content="#da532c"> <meta name="msapplication-TileImage" content="/assets/favicons/mstile-144x144.png"> <meta name="theme-color" content="#ffffff"> <meta property="og:title" content="Prometheus Remote-Write 1.0 | Prometheus"> <title>Prometheus Remote-Write 1.0 | Prometheus</title>  <link href="/assets/bootstrap-3.4.1/css/bootstrap.min.css" rel="stylesheet">  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.css" />  <link href="/css/docs.css" rel="stylesheet"> <link href="/css/routing-tree-editor.css" rel="stylesheet">  <link href="/assets/tabler-icons-3.19.0/webfont/tabler-icons.min.css" rel="stylesheet" type="text/css"> <link href='https://fonts.googleapis.com/css?family=Open+Sans:400,400italic,700' rel='stylesheet' type='text/css'> <link href='https://fonts.googleapis.com/css?family=Lato:300,300italic,400' rel='stylesheet' type='text/css'> <script async src="https://www.googletagmanager.com/gtag/js?id=G-80ZM8LGB96"></script> <script> window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'G-80ZM8LGB96'); </script> </head> <body> <div class=""> <nav class="navbar navbar-inverse navbar-static-top" role="navigation"> <div class="container"> <div class="navbar-header"> <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar"> <span class="sr-only">Toggle navigation</span> <span class="icon-bar"></span> <span class="icon-bar"></span> <span class="icon-bar"></span> </button> <a class="navbar-brand" href="/"><img src="/assets/prometheus_logo_grey.svg" alt="Prometheus logo"> Prometheus</a> </div> <div class="collapse navbar-collapse" id="navbar"> <ul class="nav navbar-nav navbar-right main-nav"> <li><a href="/docs/introduction/overview/">Docs</a></li> <li><a href="/download/">Download</a></li> <li><a href="/community/">Community</a></li> <li><a href="/support-training/">Support & Training</a></li> <li><a href="/blog/">Blog</a></li> <li><input class="searchbox form-control" type="search" placeholder="Search" aria-label="Search"></li> <li><a href="https://github.com/prometheus"><i class="ti ti-brand-github"></i></a></li> </ul> </div> </div> </nav> </div> <div class="container"> <div class="row"> <div class="col-md-3 side-nav-col"> <ul class="nav navbar-nav side-nav"> <li><span class="nav-header"><i class="ti ti-hand-finger-right"></i> <span>Introduction</span></span><ul class="nav "><li><a href="/docs/introduction/overview/">Overview</a></li><li><a href="/docs/introduction/first_steps/">First steps</a></li><li><a href="/docs/introduction/comparison/">Comparison to alternatives</a></li><li><a href="/docs/introduction/faq/">FAQ</a></li><li><a href="/docs/introduction/roadmap/">Roadmap</a></li><li><a href="/docs/introduction/design-doc/">Design Documents</a></li><li><a href="/docs/introduction/media/">Media</a></li><li><a href="/docs/introduction/glossary/">Glossary</a></li><li><a href="/docs/introduction/release-cycle/">Long-Term Support</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-flask"></i> <span>Concepts</span></span><ul class="nav "><li><a href="/docs/concepts/data_model/">Data model</a></li><li><a href="/docs/concepts/metric_types/">Metric types</a></li><li><a href="/docs/concepts/jobs_instances/">Jobs and instances</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-server"></i> <span>Prometheus Server</span></span><div class="">Version: <select><option value="/docs/prometheus/latest/getting_started/" selected="selected">latest (3.2)</option><option value="/docs/prometheus/3.2/getting_started/" >3.2</option><option value="/docs/prometheus/3.1/getting_started/" >3.1</option><option value="/docs/prometheus/3.0/getting_started/" >3.0</option><option value="/docs/prometheus/2.55/getting_started/" >2.55</option><option value="/docs/prometheus/2.54/getting_started/" >2.54</option><option value="/docs/prometheus/2.53/getting_started/" >2.53 (LTS)</option><option value="/docs/prometheus/2.52/getting_started/" >2.52</option><option value="/docs/prometheus/2.51/getting_started/" >2.51</option><option value="/docs/prometheus/2.50/getting_started/" >2.50</option><option value="/docs/prometheus/2.49/getting_started/" >2.49</option><option value="/docs/prometheus/1.8/getting_started/" >1.8</option></select></div><ul class="nav "><li><a href="/docs/prometheus/latest/getting_started/">Getting started</a></li><li><a href="/docs/prometheus/latest/installation/">Installation</a></li><li><span class="nav-subheader">Configuration</span><ul class="nav "><li><a href="/docs/prometheus/latest/configuration/configuration/">Configuration</a></li><li><a href="/docs/prometheus/latest/configuration/recording_rules/">Recording rules</a></li><li><a href="/docs/prometheus/latest/configuration/alerting_rules/">Alerting rules</a></li><li><a href="/docs/prometheus/latest/configuration/template_examples/">Template examples</a></li><li><a href="/docs/prometheus/latest/configuration/template_reference/">Template reference</a></li><li><a href="/docs/prometheus/latest/configuration/unit_testing_rules/">Unit Testing for Rules</a></li><li><a href="/docs/prometheus/latest/configuration/https/">HTTPS and authentication</a></li></ul></li><li><span class="nav-subheader">Querying</span><ul class="nav "><li><a href="/docs/prometheus/latest/querying/basics/">Basics</a></li><li><a href="/docs/prometheus/latest/querying/operators/">Operators</a></li><li><a href="/docs/prometheus/latest/querying/functions/">Functions</a></li><li><a href="/docs/prometheus/latest/querying/examples/">Examples</a></li><li><a href="/docs/prometheus/latest/querying/api/">HTTP API</a></li><li><a href="/docs/prometheus/latest/querying/remote_read_api/">Remote Read API</a></li></ul></li><li><a href="/docs/prometheus/latest/storage/">Storage</a></li><li><a href="/docs/prometheus/latest/federation/">Federation</a></li><li><a href="/docs/prometheus/latest/http_sd/">HTTP SD</a></li><li><a href="/docs/prometheus/latest/management_api/">Management API</a></li><li><span class="nav-subheader">Command Line</span><ul class="nav "><li><a href="/docs/prometheus/latest/command-line/prometheus/">prometheus</a></li><li><a href="/docs/prometheus/latest/command-line/promtool/">promtool</a></li></ul></li><li><a href="/docs/prometheus/latest/migration/">Migration</a></li><li><a href="/docs/prometheus/latest/stability/">API Stability</a></li><li><a href="/docs/prometheus/latest/feature_flags/">Feature flags</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-chart-line"></i> <span>Visualization</span></span><ul class="nav "><li><a href="/docs/visualization/browser/">Expression browser</a></li><li><a href="/docs/visualization/grafana/">Grafana</a></li><li><a href="/docs/visualization/consoles/">Console templates</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-code"></i> <span>Instrumenting</span></span><ul class="nav "><li><a href="/docs/instrumenting/clientlibs/">Client libraries</a></li><li><a href="/docs/instrumenting/writing_clientlibs/">Writing client libraries</a></li><li><a href="/docs/instrumenting/pushing/">Pushing metrics</a></li><li><a href="/docs/instrumenting/exporters/">Exporters and integrations</a></li><li><a href="/docs/instrumenting/writing_exporters/">Writing exporters</a></li><li><a href="/docs/instrumenting/exposition_formats/">Exposition formats</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-settings"></i> <span>Operating</span></span><ul class="nav "><li><a href="/docs/operating/security/">Security</a></li><li><a href="/docs/operating/integrations/">Integrations</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-bell"></i> <span>Alertmanager</span></span><div class="">Version: <select><option value="/docs/alerting/latest/overview/" selected="selected">latest (0.28)</option><option value="/docs/alerting/0.28/overview/" >0.28</option><option value="/docs/alerting/0.27/overview/" >0.27</option><option value="/docs/alerting/0.26/overview/" >0.26</option><option value="/docs/alerting/0.25/overview/" >0.25</option><option value="/docs/alerting/0.24/overview/" >0.24</option><option value="/docs/alerting/0.23/overview/" >0.23</option><option value="/docs/alerting/0.22/overview/" >0.22</option><option value="/docs/alerting/0.21/overview/" >0.21</option><option value="/docs/alerting/0.20/overview/" >0.20</option></select></div><ul class="nav "><li><a href="/docs/alerting/latest/overview/">Alerting overview</a></li><li><a href="/docs/alerting/latest/alertmanager/">Alertmanager</a></li><li><a href="/docs/alerting/latest/configuration/">Configuration</a></li><li><a href="/docs/alerting/latest/clients/">Clients</a></li><li><a href="/docs/alerting/latest/notifications/">Notification template reference</a></li><li><a href="/docs/alerting/latest/notification_examples/">Notification template examples</a></li><li><a href="/docs/alerting/latest/management_api/">Management API</a></li><li><a href="/docs/alerting/latest/https/">HTTPS and authentication</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-thumb-up"></i> <span>Best practices</span></span><ul class="nav "><li><a href="/docs/practices/naming/">Metric and label naming</a></li><li><a href="/docs/practices/consoles/">Consoles and dashboards</a></li><li><a href="/docs/practices/instrumentation/">Instrumentation</a></li><li><a href="/docs/practices/histograms/">Histograms and summaries</a></li><li><a href="/docs/practices/alerting/">Alerting</a></li><li><a href="/docs/practices/rules/">Recording rules</a></li><li><a href="/docs/practices/pushing/">When to use the Pushgateway</a></li><li><a href="/docs/practices/remote_write/">Remote write tuning</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-map"></i> <span>Guides</span></span><ul class="nav "><li><a href="/docs/guides/basic-auth/">Basic auth</a></li><li><a href="/docs/guides/cadvisor/">Monitoring Docker container metrics using cAdvisor</a></li><li><a href="/docs/guides/file-sd/">Use file-based service discovery to discover scrape targets</a></li><li><a href="/docs/guides/go-application/">Instrumenting a Go application</a></li><li><a href="/docs/guides/multi-target-exporter/">Understanding and using the multi-target exporter pattern</a></li><li><a href="/docs/guides/node-exporter/">Monitoring Linux host metrics with the Node Exporter</a></li><li><a href="/docs/guides/opentelemetry/">OpenTelemetry</a></li><li><a href="/docs/guides/utf8/">UTF-8 in Prometheus</a></li><li><a href="/docs/guides/dockerswarm/">Docker Swarm</a></li><li><a href="/docs/guides/query-log/">Query Log</a></li><li><a href="/docs/guides/tls-encryption/">TLS encryption</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-book"></i> <span>Tutorials</span></span><ul class="nav "><li><a href="/docs/tutorials/getting_started/">Getting Started with Prometheus</a></li><li><a href="/docs/tutorials/understanding_metric_types/">Understanding metric types</a></li><li><a href="/docs/tutorials/instrumenting_http_server_in_go/">Instrumenting HTTP server written in Go</a></li><li><a href="/docs/tutorials/visualizing_metrics_using_grafana/">Visualizing metrics using Grafana</a></li><li><a href="/docs/tutorials/alerting_based_on_metrics/">Alerting based on metrics.</a></li></ul></li> <li class="active"><span class="nav-header"><i class="ti ti-file-description"></i> <span>Specifications</span></span><ul class="nav active"><li><a href="/docs/specs/remote_write_spec_2_0/">Prometheus Remote-Write 2.0 [EXPERIMENTAL]</a></li><li class="active current"><a href="/docs/specs/remote_write_spec/">Prometheus Remote-Write 1.0</a></li><li><a href="/docs/specs/native_histograms/">Native Histograms [EXPERIMENTAL]</a></li></ul></li> </ul> </div> <div class="col-md-9 doc-content"> <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd"> <html><body> <h1 id="prometheus-remote-write-specification" class="page-header">Prometheus Remote-Write Specification<a class="header-anchor ti ti-link" href="#prometheus-remote-write-specification" name="prometheus-remote-write-specification"></a> </h1> <div class="toc toc-right"><ul> <li><a href="#introduction">Introduction </a></li> <ul> <li><a href="#background">Background </a></li> <li><a href="#glossary">Glossary </a></li> </ul> <li><a href="#definitions">Definitions </a></li> <ul> <li><a href="#protocol">Protocol </a></li> <li><a href="#backward-and-forward-compatibility">Backward and forward compatibility </a></li> <li><a href="#labels">Labels </a></li> <li><a href="#ordering">Ordering </a></li> <li><a href="#retries-backoff">Retries & Backoff </a></li> <li><a href="#stale-markers">Stale Markers </a></li> </ul> <li><a href="#out-of-scope">Out of Scope </a></li> <li><a href="#future-plans">Future Plans </a></li> <li><a href="#related">Related </a></li> <ul> <li><a href="#compatible-senders-and-receivers">Compatible Senders and Receivers </a></li> <li><a href="#faq">FAQ </a></li> </ul> </ul></div> <ul> <li>Version: 1.0</li> <li>Status: Published</li> <li>Date: April 2023</li> </ul> <p>This document is intended to define and standardise the API, wire format, protocol and semantics of the existing, widely and organically adopted protocol, and not to propose anything new.</p> <p>The remote write specification is intended to document the standard for how Prometheus and Prometheus remote-write-compatible agents send data to a Prometheus or Prometheus remote-write compatible receiver.</p> <p>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in <a href="https://datatracker.ietf.org/doc/html/rfc2119">RFC 2119</a>.</p> <blockquote> <div class="admonition-wrapper note"><div class="admonition alert alert-info"> <strong>NOTE:</strong> This specification has a 2.0 version available, see <a href="./remote_write_spec_2_0.md">here</a>.</div></div> </blockquote> <h2 id="introduction">Introduction<a class="header-anchor ti ti-link" href="#introduction" name="introduction"></a> </h2> <h3 id="background">Background<a class="header-anchor ti ti-link" href="#background" name="background"></a> </h3> <p>The remote write protocol is designed to make it possible to reliably propagate samples in real-time from a sender to a receiver, without loss.</p> <p>The Remote-Write protocol is designed to be stateless; there is strictly no inter-message communication. As such the protocol is not considered "streaming". To achieve a streaming effect multiple messages should be sent over the same connection using e.g. HTTP/1.1 or HTTP/2. "Fancy" technologies such as gRPC were considered, but at the time were not widely adopted, and it was challenging to expose gRPC services to the internet behind load balancers such as an AWS EC2 ELB.</p> <p>The remote write protocol contains opportunities for batching, e.g. sending multiple samples for different series in a single request. It is not expected that multiple samples for the same series will be commonly sent in the same request, although there is support for this in the protocol.</p> <p>The remote write protocol is not intended for use by applications to push metrics to Prometheus remote-write-compatible receivers. It is intended that a Prometheus remote-write-compatible sender scrapes instrumented applications or exporters and sends remote write messages to a server.</p> <p>A test suite can be found at <a href="https://github.com/prometheus/compliance/tree/main/remotewrite/sender">https://github.com/prometheus/compliance/tree/main/remotewrite/sender</a>.</p> <h3 id="glossary">Glossary<a class="header-anchor ti ti-link" href="#glossary" name="glossary"></a> </h3> <p>For the purposes of this document the following definitions MUST be followed:</p> <ul> <li>a "Sender" is something that sends Prometheus Remote Write data.</li> <li>a "Receiver" is something that receives Prometheus Remote Write data.</li> <li>a "Sample" is a pair of (timestamp, value).</li> <li>a "Label" is a pair of (key, value).</li> <li>a "Series" is a list of samples, identified by a unique set of labels.</li> </ul> <h2 id="definitions">Definitions<a class="header-anchor ti ti-link" href="#definitions" name="definitions"></a> </h2> <h3 id="protocol">Protocol<a class="header-anchor ti ti-link" href="#protocol" name="protocol"></a> </h3> <p>The Remote Write Protocol MUST consist of RPCs with the following signature:</p> <pre><code>func Send(WriteRequest) message WriteRequest { repeated TimeSeries timeseries = 1; // Cortex uses this field to determine the source of the write request. // We reserve it to avoid any compatibility issues. reserved 2; // Prometheus uses this field to send metadata, but this is // omitted from v1 of the spec as it is experimental. reserved 3; } message TimeSeries { repeated Label labels = 1; repeated Sample samples = 2; } message Label { string name = 1; string value = 2; } message Sample { double value = 1; int64 timestamp = 2; } </code></pre> <p>Remote write Senders MUST encode the Write Request in the body of a HTTP POST request and send it to the Receivers via HTTP at a provided URL path. The Receiver MAY specify any HTTP URL path to receive metrics.</p> <p>Timestamps MUST be int64 counted as milliseconds since the Unix epoch. Values MUST be float64.</p> <p>The following headers MUST be sent with the HTTP request:</p> <ul> <li><code>Content-Encoding: snappy</code></li> <li><code>Content-Type: application/x-protobuf</code></li> <li><code>User-Agent: <name & version of the sender></code></li> <li><code>X-Prometheus-Remote-Write-Version: 0.1.0</code></li> </ul> <p>Clients MAY allow users to send custom HTTP headers; they MUST NOT allow users to configure them in such a way as to send reserved headers. For more info see <a href="https://github.com/prometheus/prometheus/pull/8416">https://github.com/prometheus/prometheus/pull/8416</a>.</p> <p>The remote write request in the body of the HTTP POST MUST be compressed with <a href="https://github.com/google/snappy">Google’s Snappy</a>. The block format MUST be used - the framed format MUST NOT be used.</p> <p>The remote write request MUST be encoded using Google Protobuf 3, and MUST use the schema defined above. Note <a href="https://github.com/prometheus/prometheus/blob/v2.24.0/prompb/remote.proto">the Prometheus implementation</a> uses <a href="https://github.com/gogo/protobuf">gogoproto optimisations</a> - for receivers written in languages other than Golang the gogoproto types MAY be substituted for line-level equivalents.</p> <p>The response body from the remote write receiver SHOULD be empty; clients MUST ignore the response body. The response body is RESERVED for future use.</p> <h3 id="backward-and-forward-compatibility">Backward and forward compatibility<a class="header-anchor ti ti-link" href="#backward-and-forward-compatibility" name="backward-and-forward-compatibility"></a> </h3> <p>The protocol follows <a href="https://semver.org/">semantic versioning 2.0</a>: any 1.x compatible receivers MUST be able to read any 1.x compatible sender and so on. Breaking/backwards incompatible changes will result in a 2.x version of the spec.</p> <p>The proto format itself is forward / backward compatible, in some respects:</p> <ul> <li>Removing fields from the proto will mean a major version bump.</li> <li>Adding (optional) fields will be a minor version bump.</li> </ul> <p>Negotiation:</p> <ul> <li>Senders MUST send the version number in a headers.</li> <li>Receivers MAY return the highest version number they support in a response header ("X-Prometheus-Remote-Write-Version").</li> <li>Senders who wish to send in a format >1.x MUST start by sending an empty 1.x, and see if the response says the receiver supports something else. The Sender MAY use any supported version . If there is no version header in the response, senders MUST assume 1.x compatibility only.</li> </ul> <h3 id="labels">Labels<a class="header-anchor ti ti-link" href="#labels" name="labels"></a> </h3> <p>The complete set of labels MUST be sent with each sample. Whatsmore, the label set associated with samples:</p> <ul> <li>SHOULD contain a <code>__name__</code> label.</li> <li>MUST NOT contain repeated label names.</li> <li>MUST have label names sorted in lexicographical order.</li> <li>MUST NOT contain any empty label names or values.</li> </ul> <p>Senders MUST only send valid metric names, label names, and label values:</p> <ul> <li>Metric names MUST adhere to the regex <code>[a-zA-Z_:]([a-zA-Z0-9_:])*</code>.</li> <li>Label names MUST adhere to the regex <code>[a-zA-Z_]([a-zA-Z0-9_])*</code>.</li> <li>Label values MAY be any sequence of UTF-8 characters .</li> </ul> <p>Receivers MAY impose limits on the number and length of labels, but this will be receiver-specific and is out of scope for this document.</p> <p>Label names beginning with "__" are RESERVED for system usage and SHOULD NOT be used, see <a href="https://prometheus.io/docs/concepts/data_model/">Prometheus Data Model</a>.</p> <p>Remote write Receivers MAY ingest valid samples within a write request that otherwise contains invalid samples. Receivers MUST return a HTTP 400 status code ("Bad Request") for write requests that contain any invalid samples. Receivers SHOULD provide a human readable error message in the response body. Senders MUST NOT try and interpret the error message, and SHOULD log it as is.</p> <h3 id="ordering">Ordering<a class="header-anchor ti ti-link" href="#ordering" name="ordering"></a> </h3> <p>Prometheus Remote Write compatible senders MUST send samples for any given series in timestamp order. Prometheus Remote Write compatible Senders MAY send multiple requests for different series in parallel.</p> <h3 id="retries-backoff">Retries & Backoff<a class="header-anchor ti ti-link" href="#retries-backoff" name="retries-backoff"></a> </h3> <p>Prometheus Remote Write compatible senders MUST retry write requests on HTTP 5xx responses and MUST use a backoff algorithm to prevent overwhelming the server. They MUST NOT retry write requests on HTTP 2xx and 4xx responses other than 429. They MAY retry on HTTP 429 responses, which could result in senders "falling behind" if the server cannot keep up. This is done to ensure data is not lost when there are server side errors, and progress is made when there are client side errors.</p> <p>Prometheus remote Write compatible receivers MUST respond with a HTTP 2xx status code when the write is successful. They MUST respond with HTTP status code 5xx when the write fails and SHOULD be retried. They MUST respond with HTTP status code 4xx when the request is invalid, will never be able to succeed and should not be retried.</p> <h3 id="stale-markers">Stale Markers<a class="header-anchor ti ti-link" href="#stale-markers" name="stale-markers"></a> </h3> <p>Prometheus remote write compatible senders MUST send stale markers when a time series will no longer be appended to.</p> <p>Stale markers MUST be signalled by the special NaN value 0x7ff0000000000002. This value MUST NOT be used otherwise.</p> <p>Typically the sender can detect when a time series will no longer be appended to using the following techniques:</p> <ol> <li>Detecting, using service discovery, that the target exposing the series has gone away</li> <li>Noticing the target is no longer exposing the time series between successive scrapes</li> <li>Failing to scrape the target that originally exposed a time series</li> <li>Tracking configuration and evaluation for recording and alerting rules</li> </ol> <h2 id="out-of-scope">Out of Scope<a class="header-anchor ti ti-link" href="#out-of-scope" name="out-of-scope"></a> </h2> <p>This document does not intend to explain all the features required for a fully Prometheus-compatible monitoring system. In particular, the following areas are out of scope for the first version of the spec:</p> <p><strong>The "up" metric</strong> The definition and semantics of the "up" metric are beyond the scope of the remote write protocol and should be documented separately.</p> <p><strong>HTTP Path</strong> The path for HTTP handler can be anything - and MUST be provided by the sender. Generally we expect the whole URL to be specified in config.</p> <p><strong>Persistence</strong> It is recommended that Prometheus Remote Write compatible senders should persistently buffer sample data in the event of outages in the receiver. </p> <p><strong>Authentication & Encryption</strong> as remote write uses HTTP, we consider authentication & encryption to be a transport-layer problem. Senders and receivers should support all the usual suspects (Basic auth, TLS etc) and are free to add potentially custom authentication options. Support for custom authentication in the Prometheus remote write sender and eventual agent should not be assumed, but we will endeavour to support common and widely used auth protocols, where feasible.</p> <p><strong>Remote Read</strong> this is a separate interface that has already seen some iteration, and is less widely used.</p> <p><strong>Sharding</strong> the current sharding scheme in Prometheus for remote write parallelisation is very much an implementation detail, and isn’t part of the spec. When senders do implement parallelisation they MUST preserve per-series sample ordering.</p> <p><strong>Backfill</strong> The specification does not place a limit on how old series can be pushed, however server/implementation specific constraints may exist.</p> <p><strong>Limits</strong> Limits on the number and length of labels, batch sizes etc are beyond the scope of this document, however it is expected that implementation will impose reasonable limits.</p> <p><strong>Push-based Prometheus</strong> Applications pushing metrics to Prometheus Remote Write compatible receivers was not a design goal of this system, and should be explored in a separate doc.</p> <p><strong>Labels</strong> Every series MAY include a "job" and/or "instance" label, as these are typically added by service discovery in the Sender. These are not mandatory.</p> <h2 id="future-plans">Future Plans<a class="header-anchor ti ti-link" href="#future-plans" name="future-plans"></a> </h2> <p>This section contains speculative plans that are not considered part of protocol specification, but are mentioned here for completeness.</p> <p><strong>Transactionality</strong> Prometheus aims at being "transactional" - i.e. to never expose a partially scraped target to a query. We intend to do the same with remote write - for instance, in the future we would like to "align" remote write with scrapes, perhaps such that all the samples, metadata and exemplars for a single scrape are sent in a single remote write request. This is yet to be designed.</p> <p><strong>Metadata</strong> and Exemplars In line with above, we also send metadata (type information, help text) and exemplars along with the scraped samples. We plan to package this up in a single remote write request - future versions of the spec may insist on this. Prometheus currently has experimental support for sending metadata and exemplars.</p> <p><strong>Optimizations</strong> We would like to investigate various optimizations to reduce message size by eliminating repetition of label names and values.</p> <h2 id="related">Related<a class="header-anchor ti ti-link" href="#related" name="related"></a> </h2> <h3 id="compatible-senders-and-receivers">Compatible Senders and Receivers<a class="header-anchor ti ti-link" href="#compatible-senders-and-receivers" name="compatible-senders-and-receivers"></a> </h3> <p>The spec is intended to describe how the following components interact (as of April 2023):</p> <ul> <li> <a href="https://github.com/prometheus/prometheus/tree/master/storage/remote">Prometheus</a> (as both a "sender" and a "receiver")</li> <li> <a href="https://github.com/prometheus-community/avalanche">Avalanche</a> (as a "sender") - A Load Testing Tool Prometheus Metrics.</li> <li> <a href="https://github.com/cortexproject/cortex/blob/master/pkg/util/push/push.go#L20">Cortex</a> (as a "receiver")</li> <li> <a href="https://docs.elastic.co/integrations/prometheus#prometheus-server-remote-write">Elastic Agent</a> (as a "receiver")</li> <li> <a href="https://github.com/grafana/agent">Grafana Agent</a> (as both a "sender" and a "receiver")</li> <li> <a href="https://github.com/greptimeTeam/greptimedb">GreptimeDB</a> (as a <a href="https://docs.greptime.com/user-guide/ingest-data/for-observerbility/prometheus">"receiver"</a>)</li> <li>InfluxData’s Telegraf agent. (<a href="https://github.com/influxdata/telegraf/tree/master/plugins/serializers/prometheusremotewrite">as a sender</a>, and <a href="https://github.com/influxdata/telegraf/pull/8967">as a receiver</a>)</li> <li> <a href="https://m3db.io/docs/integrations/prometheus/#prometheus-configuration">M3</a> (as a "receiver")</li> <li> <a href="https://github.com/grafana/mimir">Mimir</a> (as a "receiver")</li> <li> <a href="https://github.com/open-telemetry/opentelemetry-collector-releases/">OpenTelemetry Collector</a> (as a <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusremotewriteexporter#readme">"sender"</a> and eventually as a "receiver")</li> <li> <a href="https://thanos.io/tip/components/receive.md/">Thanos</a> (as a "receiver")</li> <li>Vector (as a <a href="https://vector.dev/docs/reference/configuration/sinks/prometheus_remote_write/">"sender"</a> and a <a href="https://vector.dev/docs/reference/configuration/sources/prometheus_remote_write/">"receiver"</a>)</li> <li> <a href="https://github.com/VictoriaMetrics/VictoriaMetrics">VictoriaMetrics</a> (as a <a href="https://docs.victoriametrics.com/#prometheus-setup">"receiver"</a>)</li> </ul> <h3 id="faq">FAQ<a class="header-anchor ti ti-link" href="#faq" name="faq"></a> </h3> <p><strong>Why did you not use gRPC?</strong> Funnily enough we initially used gRPC, but switched to Protos atop HTTP as in 2016 it was hard to get them past ELBs: <a href="https://github.com/prometheus/prometheus/issues/1982">https://github.com/prometheus/prometheus/issues/1982</a></p> <p><strong>Why not streaming protobuf messages?</strong> If you use persistent HTTP/1.1 connections, they are pretty close to streaming… Of course headers have to be re-sent, but yes that is less expensive than a new TCP set up.</p> <p><strong>Why do we send samples in order?</strong> The in-order constraint comes from the encoding we use for time series data in Prometheus, the implementation of which is append only. It is possible to remove this constraint, for instance by buffering samples and reordering them before encoding. We can investigate this in future versions of the protocol.</p> <p><strong>How can we parallelise requests with the in-order constraint?</strong> Samples must be in-order <em>for a given series</em>. Remote write requests can be sent in parallel as long as they are for different series. In Prometheus, we shard the samples by their labels into separate queues, and then writes happen sequentially in each queue. This guarantees samples for the same series are delivered in order, but samples for different series are sent in parallel - and potentially "out of order" between different series.</p> <p>We believe this is necessary as, even if the receiver could support out-of-order samples, we can't have agents sending out of order as they would never be able to send to Prometheus, Cortex and Thanos. We’re doing this to ensure the integrity of the ecosystem and to prevent confusing/forking the community into "prometheus-agents-that-can-write-to-prometheus" and those that can’t.</p> </body></html> <p class="open-source-notice"> <i class="fa fa-file"></i> This documentation is <a href="https://github.com/prometheus/docs#contributing-changes">open-source</a>. Please help improve it by filing issues or pull requests. </p> </div> </div> <hr> <footer> <p class="pull-left"> © Prometheus Authors 2014-2025 | Documentation Distributed under CC-BY-4.0 </p> <p class="pull-left"> © 2025 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our <a href="https://www.linuxfoundation.org/trademark-usage">Trademark Usage</a> page. </p> </footer> </div>   <script src="https://code.jquery.com/jquery-3.6.0.min.js" integrity="sha256-/xUj+3OJU5yExlq6GSYGSHk7tPXikynS7ogEvDej/m4=" crossorigin="anonymous"></script> <script src="/assets/bootstrap-3.4.1/js/bootstrap.min.js"></script> <script src="/assets/docs.js"></script>  <script src="/assets/ie10-viewport-bug-workaround.js"></script>  <script type="text/javascript" src="https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.js"></script> <script type="text/javascript"> docsearch({ apiKey: '48ac0b7924908a1fd40b1cb18b402ba1', indexName: 'prometheus', inputSelector: '.searchbox', // Search all pages that either are the latest or are not part of the versioned Prometheus docs subtrees. algoliaOptions: { 'filters': 'include:true' }, debug: false // Set debug to true if you want to inspect the dropdown }); </script> </body> </html>