CINXE.COM

<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="description" content="An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach."> <meta name="keywords" content="prometheus, monitoring, monitoring system, time series, time series database, alerting, metrics, telemetry"> <meta name="author" content="Prometheus"> <meta name="twitter:card" content="summary"> <meta property="og:description" content="An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach."> <meta property="og:image" content="https://prometheus.io/assets/favicons/android-chrome-192x192.png"> <link rel="alternate" type="application/atom+xml" title="Prometheus Blog 禄 Feed" href="/blog/feed.xml"> <link rel="shortcut icon" href="/assets/favicons/favicon.ico"> <link rel="apple-touch-icon" sizes="57x57" href="/assets/favicons/apple-touch-icon-57x57.png"> <link rel="apple-touch-icon" sizes="60x60" href="/assets/favicons/apple-touch-icon-60x60.png"> <link rel="apple-touch-icon" sizes="72x72" href="/assets/favicons/apple-touch-icon-72x72.png"> <link rel="apple-touch-icon" sizes="76x76" href="/assets/favicons/apple-touch-icon-76x76.png"> <link rel="apple-touch-icon" sizes="114x114" href="/assets/favicons/apple-touch-icon-114x114.png"> <link rel="apple-touch-icon" sizes="120x120" href="/assets/favicons/apple-touch-icon-120x120.png"> <link rel="apple-touch-icon" sizes="144x144" href="/assets/favicons/apple-touch-icon-144x144.png"> <link rel="apple-touch-icon" sizes="152x152" href="/assets/favicons/apple-touch-icon-152x152.png"> <link rel="apple-touch-icon" sizes="180x180" href="/assets/favicons/apple-touch-icon-180x180.png"> <link rel="icon" type="image/png" href="/assets/favicons/favicon-32x32.png" sizes="32x32"> <link rel="icon" type="image/png" href="/assets/favicons/android-chrome-192x192.png" sizes="192x192"> <link rel="icon" type="image/png" href="/assets/favicons/favicon-96x96.png" sizes="96x96"> <link rel="icon" type="image/png" href="/assets/favicons/favicon-16x16.png" sizes="16x16"> <link rel="manifest" href="/assets/favicons/android-chrome-manifest.json">  <meta name="docsearch:prometheus-version" content="none" /> <meta name="docsearch:include" content="true" /> <meta name="msapplication-TileColor" content="#da532c"> <meta name="msapplication-TileImage" content="/assets/favicons/mstile-144x144.png"> <meta name="theme-color" content="#ffffff"> <meta property="og:title" content="Remote write tuning | Prometheus"> <title>Remote write tuning | Prometheus</title>  <link href="/assets/bootstrap-3.4.1/css/bootstrap.min.css" rel="stylesheet">  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.css" />  <link href="/css/docs.css" rel="stylesheet"> <link href="/css/routing-tree-editor.css" rel="stylesheet">  <link href="/assets/tabler-icons-3.19.0/webfont/tabler-icons.min.css" rel="stylesheet" type="text/css"> <link href='https://fonts.googleapis.com/css?family=Open+Sans:400,400italic,700' rel='stylesheet' type='text/css'> <link href='https://fonts.googleapis.com/css?family=Lato:300,300italic,400' rel='stylesheet' type='text/css'> <script async src="https://www.googletagmanager.com/gtag/js?id=G-80ZM8LGB96"></script> <script> window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'G-80ZM8LGB96'); </script> </head> <body> <div class=""> <nav class="navbar navbar-inverse navbar-static-top" role="navigation"> <div class="container"> <div class="navbar-header"> <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar"> <span class="sr-only">Toggle navigation</span> <span class="icon-bar"></span> <span class="icon-bar"></span> <span class="icon-bar"></span> </button> <a class="navbar-brand" href="/"><img src="/assets/prometheus_logo_grey.svg" alt="Prometheus logo"> Prometheus</a> </div> <div class="collapse navbar-collapse" id="navbar"> <ul class="nav navbar-nav navbar-right main-nav"> <li><a href="/docs/introduction/overview/">Docs</a></li> <li><a href="/download/">Download</a></li> <li><a href="/community/">Community</a></li> <li><a href="/support-training/">Support & Training</a></li> <li><a href="/blog/">Blog</a></li> <li><input class="searchbox form-control" type="search" placeholder="Search" aria-label="Search"></li> <li><a href="https://github.com/prometheus"><i class="ti ti-brand-github"></i></a></li> </ul> </div> </div> </nav> </div> <div class="container"> <div class="row"> <div class="col-md-3 side-nav-col"> <ul class="nav navbar-nav side-nav"> <li><span class="nav-header"><i class="ti ti-hand-finger-right"></i> <span>Introduction</span></span><ul class="nav "><li><a href="/docs/introduction/overview/">Overview</a></li><li><a href="/docs/introduction/first_steps/">First steps</a></li><li><a href="/docs/introduction/comparison/">Comparison to alternatives</a></li><li><a href="/docs/introduction/faq/">FAQ</a></li><li><a href="/docs/introduction/roadmap/">Roadmap</a></li><li><a href="/docs/introduction/design-doc/">Design Documents</a></li><li><a href="/docs/introduction/media/">Media</a></li><li><a href="/docs/introduction/glossary/">Glossary</a></li><li><a href="/docs/introduction/release-cycle/">Long-Term Support</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-flask"></i> <span>Concepts</span></span><ul class="nav "><li><a href="/docs/concepts/data_model/">Data model</a></li><li><a href="/docs/concepts/metric_types/">Metric types</a></li><li><a href="/docs/concepts/jobs_instances/">Jobs and instances</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-server"></i> <span>Prometheus Server</span></span><div class="">Version: <select><option value="/docs/prometheus/latest/getting_started/" selected="selected">latest (3.2)</option><option value="/docs/prometheus/3.2/getting_started/" >3.2</option><option value="/docs/prometheus/3.1/getting_started/" >3.1</option><option value="/docs/prometheus/3.0/getting_started/" >3.0</option><option value="/docs/prometheus/2.55/getting_started/" >2.55</option><option value="/docs/prometheus/2.54/getting_started/" >2.54</option><option value="/docs/prometheus/2.53/getting_started/" >2.53 (LTS)</option><option value="/docs/prometheus/2.52/getting_started/" >2.52</option><option value="/docs/prometheus/2.51/getting_started/" >2.51</option><option value="/docs/prometheus/2.50/getting_started/" >2.50</option><option value="/docs/prometheus/2.49/getting_started/" >2.49</option><option value="/docs/prometheus/1.8/getting_started/" >1.8</option></select></div><ul class="nav "><li><a href="/docs/prometheus/latest/getting_started/">Getting started</a></li><li><a href="/docs/prometheus/latest/installation/">Installation</a></li><li><span class="nav-subheader">Configuration</span><ul class="nav "><li><a href="/docs/prometheus/latest/configuration/configuration/">Configuration</a></li><li><a href="/docs/prometheus/latest/configuration/recording_rules/">Recording rules</a></li><li><a href="/docs/prometheus/latest/configuration/alerting_rules/">Alerting rules</a></li><li><a href="/docs/prometheus/latest/configuration/template_examples/">Template examples</a></li><li><a href="/docs/prometheus/latest/configuration/template_reference/">Template reference</a></li><li><a href="/docs/prometheus/latest/configuration/unit_testing_rules/">Unit Testing for Rules</a></li><li><a href="/docs/prometheus/latest/configuration/https/">HTTPS and authentication</a></li></ul></li><li><span class="nav-subheader">Querying</span><ul class="nav "><li><a href="/docs/prometheus/latest/querying/basics/">Basics</a></li><li><a href="/docs/prometheus/latest/querying/operators/">Operators</a></li><li><a href="/docs/prometheus/latest/querying/functions/">Functions</a></li><li><a href="/docs/prometheus/latest/querying/examples/">Examples</a></li><li><a href="/docs/prometheus/latest/querying/api/">HTTP API</a></li><li><a href="/docs/prometheus/latest/querying/remote_read_api/">Remote Read API</a></li></ul></li><li><a href="/docs/prometheus/latest/storage/">Storage</a></li><li><a href="/docs/prometheus/latest/federation/">Federation</a></li><li><a href="/docs/prometheus/latest/http_sd/">HTTP SD</a></li><li><a href="/docs/prometheus/latest/management_api/">Management API</a></li><li><span class="nav-subheader">Command Line</span><ul class="nav "><li><a href="/docs/prometheus/latest/command-line/prometheus/">prometheus</a></li><li><a href="/docs/prometheus/latest/command-line/promtool/">promtool</a></li></ul></li><li><a href="/docs/prometheus/latest/migration/">Migration</a></li><li><a href="/docs/prometheus/latest/stability/">API Stability</a></li><li><a href="/docs/prometheus/latest/feature_flags/">Feature flags</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-chart-line"></i> <span>Visualization</span></span><ul class="nav "><li><a href="/docs/visualization/browser/">Expression browser</a></li><li><a href="/docs/visualization/grafana/">Grafana</a></li><li><a href="/docs/visualization/consoles/">Console templates</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-code"></i> <span>Instrumenting</span></span><ul class="nav "><li><a href="/docs/instrumenting/clientlibs/">Client libraries</a></li><li><a href="/docs/instrumenting/writing_clientlibs/">Writing client libraries</a></li><li><a href="/docs/instrumenting/pushing/">Pushing metrics</a></li><li><a href="/docs/instrumenting/exporters/">Exporters and integrations</a></li><li><a href="/docs/instrumenting/writing_exporters/">Writing exporters</a></li><li><a href="/docs/instrumenting/exposition_formats/">Exposition formats</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-settings"></i> <span>Operating</span></span><ul class="nav "><li><a href="/docs/operating/security/">Security</a></li><li><a href="/docs/operating/integrations/">Integrations</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-bell"></i> <span>Alertmanager</span></span><div class="">Version: <select><option value="/docs/alerting/latest/overview/" selected="selected">latest (0.28)</option><option value="/docs/alerting/0.28/overview/" >0.28</option><option value="/docs/alerting/0.27/overview/" >0.27</option><option value="/docs/alerting/0.26/overview/" >0.26</option><option value="/docs/alerting/0.25/overview/" >0.25</option><option value="/docs/alerting/0.24/overview/" >0.24</option><option value="/docs/alerting/0.23/overview/" >0.23</option><option value="/docs/alerting/0.22/overview/" >0.22</option><option value="/docs/alerting/0.21/overview/" >0.21</option><option value="/docs/alerting/0.20/overview/" >0.20</option></select></div><ul class="nav "><li><a href="/docs/alerting/latest/overview/">Alerting overview</a></li><li><a href="/docs/alerting/latest/alertmanager/">Alertmanager</a></li><li><a href="/docs/alerting/latest/configuration/">Configuration</a></li><li><a href="/docs/alerting/latest/clients/">Clients</a></li><li><a href="/docs/alerting/latest/notifications/">Notification template reference</a></li><li><a href="/docs/alerting/latest/notification_examples/">Notification template examples</a></li><li><a href="/docs/alerting/latest/management_api/">Management API</a></li><li><a href="/docs/alerting/latest/https/">HTTPS and authentication</a></li></ul></li> <li class="active"><span class="nav-header"><i class="ti ti-thumb-up"></i> <span>Best practices</span></span><ul class="nav active"><li><a href="/docs/practices/naming/">Metric and label naming</a></li><li><a href="/docs/practices/consoles/">Consoles and dashboards</a></li><li><a href="/docs/practices/instrumentation/">Instrumentation</a></li><li><a href="/docs/practices/histograms/">Histograms and summaries</a></li><li><a href="/docs/practices/alerting/">Alerting</a></li><li><a href="/docs/practices/rules/">Recording rules</a></li><li><a href="/docs/practices/pushing/">When to use the Pushgateway</a></li><li class="active current"><a href="/docs/practices/remote_write/">Remote write tuning</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-map"></i> <span>Guides</span></span><ul class="nav "><li><a href="/docs/guides/basic-auth/">Basic auth</a></li><li><a href="/docs/guides/cadvisor/">Monitoring Docker container metrics using cAdvisor</a></li><li><a href="/docs/guides/file-sd/">Use file-based service discovery to discover scrape targets</a></li><li><a href="/docs/guides/go-application/">Instrumenting a Go application</a></li><li><a href="/docs/guides/multi-target-exporter/">Understanding and using the multi-target exporter pattern</a></li><li><a href="/docs/guides/node-exporter/">Monitoring Linux host metrics with the Node Exporter</a></li><li><a href="/docs/guides/opentelemetry/">OpenTelemetry</a></li><li><a href="/docs/guides/utf8/">UTF-8 in Prometheus</a></li><li><a href="/docs/guides/dockerswarm/">Docker Swarm</a></li><li><a href="/docs/guides/query-log/">Query Log</a></li><li><a href="/docs/guides/tls-encryption/">TLS encryption</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-book"></i> <span>Tutorials</span></span><ul class="nav "><li><a href="/docs/tutorials/getting_started/">Getting Started with Prometheus</a></li><li><a href="/docs/tutorials/understanding_metric_types/">Understanding metric types</a></li><li><a href="/docs/tutorials/instrumenting_http_server_in_go/">Instrumenting HTTP server written in Go</a></li><li><a href="/docs/tutorials/visualizing_metrics_using_grafana/">Visualizing metrics using Grafana</a></li><li><a href="/docs/tutorials/alerting_based_on_metrics/">Alerting based on metrics.</a></li></ul></li> <li><span class="nav-header"><i class="ti ti-file-description"></i> <span>Specifications</span></span><ul class="nav "><li><a href="/docs/specs/remote_write_spec_2_0/">Prometheus Remote-Write 2.0 [EXPERIMENTAL]</a></li><li><a href="/docs/specs/remote_write_spec/">Prometheus Remote-Write 1.0</a></li><li><a href="/docs/specs/native_histograms/">Native Histograms [EXPERIMENTAL]</a></li></ul></li> </ul> </div> <div class="col-md-9 doc-content"> <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd"> <html><body> <h1 id="remote-write-tuning" class="page-header">Remote write tuning<a class="header-anchor ti ti-link" href="#remote-write-tuning" name="remote-write-tuning"></a> </h1> <div class="toc toc-right"><ul> <li><a href="#remote-write-characteristics">Remote write characteristics </a></li> <ul><li><a href="#resource-usage">Resource usage </a></li></ul> <li><a href="#parameters">Parameters </a></li> <ul> <li><a href="#capacity"> <code>capacity</code> </a></li> <li><a href="#max_shards"> <code>max_shards</code> </a></li> <li><a href="#min_shards"> <code>min_shards</code> </a></li> <li><a href="#max_samples_per_send"> <code>max_samples_per_send</code> </a></li> <li><a href="#batch_send_deadline"> <code>batch_send_deadline</code> </a></li> <li><a href="#min_backoff"> <code>min_backoff</code> </a></li> <li><a href="#max_backoff"> <code>max_backoff</code> </a></li> </ul> </ul></div> <p>Prometheus implements sane defaults for remote write, but many users have different requirements and would like to optimize their remote settings.</p> <p>This page describes the tuning parameters available via the <a href="/docs/prometheus/latest/configuration/configuration/#remote_write">remote write configuration.</a></p> <h2 id="remote-write-characteristics">Remote write characteristics<a class="header-anchor ti ti-link" href="#remote-write-characteristics" name="remote-write-characteristics"></a> </h2> <p>Each remote write destination starts a queue which reads from the write-ahead log (WAL), writes the samples into an in memory queue owned by a shard, which then sends a request to the configured endpoint. The flow of data looks like:</p> <pre><code> |--> queue (shard_1) --> remote endpoint WAL --|--> queue (shard_...) --> remote endpoint |--> queue (shard_n) --> remote endpoint </code></pre> <p>When one shard backs up and fills its queue, Prometheus will block reading from the WAL into any shards. Failures will be retried without loss of data unless the remote endpoint remains down for more than 2 hours. After 2 hours, the WAL will be compacted and data that has not been sent will be lost.</p> <p>During operation, Prometheus will continuously calculate the optimal number of shards to use based on the incoming sample rate, number of outstanding samples not sent, and time taken to send each sample.</p> <h3 id="resource-usage">Resource usage<a class="header-anchor ti ti-link" href="#resource-usage" name="resource-usage"></a> </h3> <p>Using remote write increases the memory footprint of Prometheus. Most users report ~25% increased memory usage, but that number is dependent on the shape of the data. For each series in the WAL, the remote write code caches a mapping of series ID to label values, causing large amounts of series churn to significantly increase memory usage.</p> <p>In addition to the series cache, each shard and its queue increases memory usage. Shard memory is proportional to the <code>number of shards * (capacity + max_samples_per_send)</code>. When tuning, consider reducing <code>max_shards</code> alongside increases to <code>capacity</code> and <code>max_samples_per_send</code> to avoid inadvertently running out of memory. The default values for <code>capacity: 10000</code> and <code>max_samples_per_send: 2000</code> will constrain shard memory usage to less than 2 MB per shard.</p> <p>Remote write will also increase CPU and network usage. However, for the same reasons as above, it is difficult to predict by how much. It is generally a good practice to check for CPU and network saturation if your Prometheus server falls behind sending samples via remote write (<code>prometheus_remote_storage_samples_pending</code>).</p> <h2 id="parameters">Parameters<a class="header-anchor ti ti-link" href="#parameters" name="parameters"></a> </h2> <p>All the relevant parameters are found under the <code>queue_config</code> section of the remote write configuration.</p> <h3 id="capacity"> <code>capacity</code><a class="header-anchor ti ti-link" href="#capacity" name="capacity"></a> </h3> <p>Capacity controls how many samples are queued in memory per shard before blocking reading from the WAL. Once the WAL is blocked, samples cannot be appended to any shards and all throughput will cease.</p> <p>Capacity should be high enough to avoid blocking other shards in most cases, but too much capacity can cause excess memory consumption and longer times to clear queues during resharding. It is recommended to set capacity to 3-10 times <code>max_samples_per_send</code>.</p> <h3 id="max_shards"> <code>max_shards</code><a class="header-anchor ti ti-link" href="#max_shards" name="max_shards"></a> </h3> <p>Max shards configures the maximum number of shards, or parallelism, Prometheus will use for each remote write queue. Prometheus will try not to use too many shards, but if the queue falls behind the remote write component will increase the number of shards up to max shards to increase throughput. Unless remote writing to a very slow endpoint, it is unlikely that <code>max_shards</code> should be increased beyond the default. However, it may be necessary to reduce max shards if there is potential to overwhelm the remote endpoint, or to reduce memory usage when data is backed up.</p> <h3 id="min_shards"> <code>min_shards</code><a class="header-anchor ti ti-link" href="#min_shards" name="min_shards"></a> </h3> <p>Min shards configures the minimum number of shards used by Prometheus, and is the number of shards used when remote write starts. If remote write falls behind, Prometheus will automatically scale up the number of shards so most users do not have to adjust this parameter. However, increasing min shards will allow Prometheus to avoid falling behind at the beginning while calculating the required number of shards.</p> <h3 id="max_samples_per_send"> <code>max_samples_per_send</code><a class="header-anchor ti ti-link" href="#max_samples_per_send" name="max_samples_per_send"></a> </h3> <p>Max samples per send can be adjusted depending on the backend in use. Many systems work very well by sending more samples per batch without a significant increase in latency. Other backends will have issues if trying to send a large number of samples in each request. The default value is small enough to work for most systems.</p> <h3 id="batch_send_deadline"> <code>batch_send_deadline</code><a class="header-anchor ti ti-link" href="#batch_send_deadline" name="batch_send_deadline"></a> </h3> <p>Batch send deadline sets the maximum amount of time between sends for a single shard. Even if the queued shards has not reached <code>max_samples_per_send</code>, a request will be sent. Batch send deadline can be increased for low volume systems that are not latency sensitive in order to increase request efficiency.</p> <h3 id="min_backoff"> <code>min_backoff</code><a class="header-anchor ti ti-link" href="#min_backoff" name="min_backoff"></a> </h3> <p>Min backoff controls the minimum amount of time to wait before retrying a failed request. Increasing the backoff spreads out requests when a remote endpoint comes back online. The backoff interval is doubled for each failed requests up to <code>max_backoff</code>.</p> <h3 id="max_backoff"> <code>max_backoff</code><a class="header-anchor ti ti-link" href="#max_backoff" name="max_backoff"></a> </h3> <p>Max backoff controls the maximum amount of time to wait before retrying a failed request.</p> </body></html> <p class="open-source-notice"> <i class="fa fa-file"></i> This documentation is <a href="https://github.com/prometheus/docs#contributing-changes">open-source</a>. Please help improve it by filing issues or pull requests. </p> </div> </div> <hr> <footer> <p class="pull-left"> © Prometheus Authors 2014-2025 | Documentation Distributed under CC-BY-4.0 </p> <p class="pull-left"> © 2025 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our <a href="https://www.linuxfoundation.org/trademark-usage">Trademark Usage</a> page. </p> </footer> </div>   <script src="https://code.jquery.com/jquery-3.6.0.min.js" integrity="sha256-/xUj+3OJU5yExlq6GSYGSHk7tPXikynS7ogEvDej/m4=" crossorigin="anonymous"></script> <script src="/assets/bootstrap-3.4.1/js/bootstrap.min.js"></script> <script src="/assets/docs.js"></script>  <script src="/assets/ie10-viewport-bug-workaround.js"></script>  <script type="text/javascript" src="https://cdn.jsdelivr.net/npm/docsearch.js@2/dist/cdn/docsearch.min.js"></script> <script type="text/javascript"> docsearch({ apiKey: '48ac0b7924908a1fd40b1cb18b402ba1', indexName: 'prometheus', inputSelector: '.searchbox', // Search all pages that either are the latest or are not part of the versioned Prometheus docs subtrees. algoliaOptions: { 'filters': 'include:true' }, debug: false // Set debug to true if you want to inspect the dropdown }); </script> </body> </html>