prometheus If you are instrumenting your own code the general rules of how to library docs practices instrumentation should be followed When title Writing exporters instrument code with a Prometheus client sortrank 5 Writing exporters

Answer:
---
title: Writing exporters
sort_rank: 5
---

# Writing exporters

If you are instrumenting your own code, the [general rules of how to
instrument code with a Prometheus client
library](/docs/practices/instrumentation/) should be followed. When
taking metrics from another monitoring or instrumentation system, things
tend not to be so black and white.

This document contains things you should consider when writing an
exporter or custom collector. The theory covered will also be of
interest to those doing direct instrumentation.

If you are writing an exporter and are unclear on anything here, please
contact us on IRC (#prometheus on libera) or the [mailing
list](/community).

## Maintainability and purity

The main decision you need to make when writing an exporter is how much
work you’re willing to put in to get perfect metrics out of it.

If the system in question has only a handful of metrics that rarely
change, then getting everything perfect is an easy choice, a good
example of this is the [HAProxy
exporter](https://github.com/prometheus/haproxy_exporter).

On the other hand, if you try to get things perfect when the system has
hundreds of metrics that change frequently with new versions, then
you’ve signed yourself up for a lot of ongoing work. The [MySQL
exporter](https://github.com/prometheus/mysqld_exporter) is on this end
of the spectrum.

The [node exporter](https://github.com/prometheus/node_exporter) is a
mix of these, with complexity varying by module. For example, the
`mdadm` collector hand-parses a file and exposes metrics created
specifically for that collector, so we may as well get the metrics
right. For the `meminfo` collector the results vary across kernel
versions so we end up doing just enough of a transform to create valid
metrics.

## Configuration

When working with applications, you should aim for an exporter that
requires no custom configuration by the user beyond telling it where the
application is.  You may also need to offer the ability to filter out
certain metrics if they may be too granular and expensive on large
setups, for example the [HAProxy
exporter](https://github.com/prometheus/haproxy_exporter) allows
filtering of per-server stats. Similarly, there may be expensive metrics
that are disabled by default.

When working with other monitoring systems, frameworks and protocols you
will often need to provide additional configuration or customization to
generate metrics suitable for Prometheus. In the best case scenario, a
monitoring system has a similar enough data model to Prometheus that you
can automatically determine how to transform metrics. This is the case
for [Cloudwatch](https://github.com/prometheus/cloudwatch_exporter),
[SNMP](https://github.com/prometheus/snmp_exporter) and
[collectd](https://github.com/prometheus/collectd_exporter). At most, we
need the ability to let the user select which metrics they want to pull
out.

In other cases, metrics from the system are completely non-standard,
depending on the usage of the system and the underlying application.  In
that case the user has to tell us how to transform the metrics. The [JMX
exporter](https://github.com/prometheus/jmx_exporter) is the worst
offender here, with the
[Graphite](https://github.com/prometheus/graphite_exporter) and
[StatsD](https://github.com/prometheus/statsd_exporter) exporters also
requiring configuration to extract labels.

Ensuring the exporter works out of the box without configuration, and
providing a selection of example configurations for transformation if
required, is advised.

YAML is the standard Prometheus configuration format, all configuration
should use YAML by default.

## Metrics

### Naming

Follow the [best practices on metric naming](/docs/practices/naming).

Generally metric names should allow someone who is familiar with
Prometheus but not a particular system to make a good guess as to what a
metric means.  A metric named `http_requests_total` is not extremely
useful - are these being measured as they come in, in some filter or
when they get to the user’s code?  And `requests_total` is even worse,
what type of requests?

With direct instrumentation, a given metric should exist within exactly
one file. Accordingly, within exporters and collectors, a metric should
apply to exactly one subsystem and be named accordingly.

Metric names should never be procedurally generated, except when writing
a custom collector or exporter.

Metric names for applications should generally be prefixed by the
exporter name, e.g. `haproxy_up`.

Metrics must use base units (e.g. seconds, bytes) and leave converting
them to something more readable to graphing tools. No matter what units
you end up using, the units in the metric name must match the units in
use. Similarly, expose ratios, not percentages. Even better, specify a
counter for each of the two components of the ratio.

Metric names should not include the labels that they’re exported with,
e.g. `by_type`, as that won’t make sense if the label is aggregated
away.

The one exception is when you’re exporting the same data with different
labels via multiple metrics, in which case that’s usually the sanest way
to distinguish them. For direct instrumentation, this should only come
up when exporting a single metric with all the labels would have too
high a cardinality.

Prometheus metrics and label names are written in `snake_case`.
Converting `camelCase` to `snake_case` is desirable, though doing so
automatically doesn’t always produce nice results for things like
`myTCPExample` or `isNaN` so sometimes it’s best to leave them as-is.

Exposed metrics should not contain colons, these are reserved for user
defined recording rules to use when aggregating.

Only `[a-zA-Z0-9:_]` are valid in metric names.

The `_sum`, `_count`, `_bucket` and `_total` suffixes are used by
Summaries, Histograms and Counters. Unless you’re producing one of
those, avoid these suffixes.

`_total` is a convention for counters, you should use it if you’re using
the COUNTER type.

The `process_` and `scrape_` prefixes are reserved. It’s okay to add
your own prefix on to these if they follow matching semantics.
For example, Prometheus has `scrape_duration_seconds` for how long a
scrape took, it's good practice to also have an exporter-centric metric,
e.g. `jmx_scrape_duration_seconds`, saying how long the specific
exporter took to do its thing. For process stats where you have access
to the PID, both Go and Python offer collectors that’ll handle this for
you. A good example of this is the [HAProxy
exporter](https://github.com/prometheus/haproxy_exporter).

When you have a successful request count and a failed request count, the
best way to expose this is as one metric for total requests and another
metric for failed requests. This makes it easy to calculate the failure
ratio. Do not use one metric with a failed or success label. Similarly,
with hit or miss for caches, it’s better to have one metric for total and
another for hits.

Consider the likelihood that someone using monitoring will do a code or
web search for the metric name. If the names are very well-established
and unlikely to be used outside of the realm of people used to those
names, for example SNMP and network engineers, then leaving them as-is
may be a good idea. This logic doesn’t apply for all exporters, for
example the MySQL exporter metrics may be used by a variety of people,
not just DBAs. A `HELP` string with the original name can provide most
of the same benefits as using the original names.

### Labels

Read the [general
advice](/docs/practices/instrumentation/#things-to-watch-out-for) on
labels.

Avoid `type` as a label name, it’s too generic and often meaningless.
You should also try where possible to avoid names that are likely to
clash with target labels, such as `region`, `zone`, `cluster`,
`availability_zone`, `az`, `datacenter`, `dc`, `owner`, `customer`,
`stage`, `service`, `environment` and `env`. If, however, that’s what
the application calls some resource, it’s best not to cause confusion by
renaming it.

Avoid the temptation to put things into one metric just because they
share a prefix. Unless you’re sure something makes sense as one metric,
multiple metrics is safer.

The label `le` has special meaning for Histograms, and `quantile` for
Summaries. Avoid these labels generally.

Read/write and send/receive are best as separate metrics, rather than as
a label. This is usually because you care about only one of them at a
time, and it is easier to use them that way.

The rule of thumb is that one metric should make sense when summed or
averaged.  There is one other case that comes up with exporters, and
that’s where the data is fundamentally tabular and doing otherwise would
require users to do regexes on metric names to be usable. Consider the
voltage sensors on your motherboard, while doing math across them is
meaningless, it makes sense to have them in one metric rather than
having one metric per sensor. All values within a metric should
(almost) always have the same unit, for example consider if fan speeds
were mixed in with the voltages, and you had no way to automatically
separate them.

Don’t do this:

my_metric{label="a"} 1
my_metric{label="b"} 6
my_metric{label="total"} 7

or this:

my_metric{label="a"} 1
my_metric{label="b"} 6
my_metric{} 7

The former breaks for people who do a `sum()` over your metric, and the
latter breaks sum and is quite difficult to work with. Some client
libraries, for example Go, will actively try to stop you doing the
latter in a custom collector, and all client libraries should stop you
from doing the latter with direct instrumentation. Never do either of
these, rely on Prometheus aggregation instead.

If your monitoring exposes a total like this, drop the total. If you
have to keep it around for some reason, for example the total includes
things not counted individually, use different metric names.

Instrumentation labels should be minimal, every extra label is one more
that users need to consider when writing their PromQL. Accordingly,
avoid having instrumentation labels which could be removed without
affecting the uniqueness of the time series. Additional information
around a metric can be added via an info metric, for an example see
below how to handle version numbers.

However, there are cases where it is expected that virtually all users of
a metric will want the additional information. If so, adding a
non-unique label, rather than an info metric, is the right solution. For
example the
[mysqld_exporter](https://github.com/prometheus/mysqld_exporter)'s
`mysqld_perf_schema_events_statements_total`'s `digest` label is a hash
of the full query pattern and is sufficient for uniqueness. However, it
is of little use without the human readable `digest_text` label, which
for long queries will contain only the start of the query pattern and is
thus not unique. Thus we end up with both the `digest_text` label for
humans and the `digest` label for uniqueness.

### Target labels, not static scraped labels

If you ever find yourself wanting to apply the same label to all of your
metrics, stop.

There’s generally two cases where this comes up.

The first is for some label it would be useful to have on the metrics
such as the version number of the software. Instead, use the approach
described at
[https://www.robustperception.io/how-to-have-labels-for-machine-roles/](http://www.robustperception.io/how-to-have-labels-for-machine-roles/).

The second case is when a label is really a target label. These are
things like region, cluster names, and so on, that come from your
infrastructure setup rather than the application itself. It’s not for an
application to say where it fits in your label taxonomy, that’s for the
person running the Prometheus server to configure and different people
monitoring the same application may give it different names.

Accordingly, these labels belong up in the scrape configs of Prometheus
via whatever service discovery you’re using. It’s okay to apply the
concept of machine roles here as well, as it’s likely useful information
for at least some people scraping it.

### Types

You should try to match up the types of your metrics to Prometheus
types. This usually means counters and gauges. The `_count` and `_sum`
of summaries are also relatively common, and on occasion you’ll see
quantiles. Histograms are rare, if you come across one remember that the
exposition format exposes cumulative values.

Often it won’t be obvious what the type of metric is, especially if
you’re automatically processing a set of metrics. In general `UNTYPED`
is a safe default.

Counters can’t go down, so if you have a counter type coming from
another instrumentation system that can be decremented, for example
Dropwizard metrics then it's not a counter, it's a gauge. `UNTYPED` is
probably the best type to use there, as `GAUGE` would be misleading if
it were being used as a counter.

### Help strings

When you’re transforming metrics it’s useful for users to be able to
track back to what the original was, and what rules were in play that
caused that transformation. Putting in the name of the
collector or exporter, the ID of any rule that was applied and the
name and details of the original metric into the help string will greatly
aid users.

Prometheus doesn’t like one metric having different help strings. If
you’re making one metric from many others, choose one of them to put in
the help string.

For examples of this, the SNMP exporter uses the OID and the JMX
exporter puts in a sample mBean name. The [HAProxy
exporter](https://github.com/prometheus/haproxy_exporter) has
hand-written strings. The [node
exporter](https://github.com/prometheus/node_exporter) also has a wide
variety of examples.

### Drop less useful statistics

Some instrumentation systems expose 1m, 5m, 15m rates, average rates since
application start (these are called `mean` in Dropwizard metrics for
example) in addition to minimums, maximums and standard deviations.

These should all be dropped, as they’re not very useful and add clutter.
Prometheus can calculate rates itself, and usually more accurately as
the averages exposed are usually exponentially decaying. You don’t know
what time the min or max were calculated over, and the standard deviation
is statistically useless and you can always expose sum of squares,
`_sum` and `_count` if you ever need to calculate it.

Quantiles have related issues, you may choose to drop them or put them
in a Summary.

### Dotted strings

Many monitoring systems don’t have labels, instead doing things like
`my.class.path.mymetric.labelvalue1.labelvalue2.labelvalue3`.

The [Graphite](https://github.com/prometheus/graphite_exporter) and
[StatsD](https://github.com/prometheus/statsd_exporter) exporters share
a way of transforming these with a small configuration language. Other
exporters should implement the same. The transformation is currently
implemented only in Go, and would benefit from being factored out into a
separate library.

## Collectors

When implementing the collector for your exporter, you should never use
the usual direct instrumentation approach and then update the metrics on
each scrape.

Rather create new metrics each time. In Go this is done with
[MustNewConstMetric](https://godoc.org/github.com/prometheus/client_golang/prometheus#MustNewConstMetric)
in your `Collect()` method. For Python see
[https://github.com/prometheus/client_python#custom-collectors](https://prometheus.github.io/client_python/collector/custom/)
and for Java generate a `List<MetricFamilySamples>` in your collect
method, see
[StandardExports.java](https://github.com/prometheus/client_java/blob/master/simpleclient_hotspot/src/main/java/io/prometheus/client/hotspot/StandardExports.java)
for an example.

The reason for this is two-fold. Firstly, two scrapes could happen at
the same time, and direct instrumentation uses what are effectively
file-level global variables, so you’ll get race conditions. Secondly, if
a label value disappears, it’ll still be exported.

Instrumenting your exporter itself via direct instrumentation is fine,
e.g. total bytes transferred or calls performed by the exporter across
all scrapes.  For exporters such as the [blackbox
exporter](https://github.com/prometheus/blackbox_exporter) and [SNMP
exporter](https://github.com/prometheus/snmp_exporter), which aren’t
tied to a single target, these should only be exposed on a vanilla
`/metrics` call, not on a scrape of a particular target.

### Metrics about the scrape itself

Sometimes you’d like to export metrics that are about the scrape, like
how long it took or how many records you processed.

These should be exposed as gauges as they’re about an event, the scrape,
and the metric name prefixed by the exporter name, for example
`jmx_scrape_duration_seconds`. Usually the `_exporter` is excluded and
if the exporter also makes sense to use as just a collector, then
definitely exclude it.

Other scrape "meta" metrics should be avoided. For example, a counter for
the number of scrapes, or a histogram of the scrape duration. Having the
exporter track these metrics duplicate the [automatically generated
metrics](docs/concepts/jobs_instances/#automatically-generated-labels-and-time-series)
of Prometheus itself. This adds to the storage cost of every exporter instance.

### Machine and process metrics

Many systems, for example Elasticsearch, expose machine metrics such as
CPU, memory and filesystem information. As the [node
exporter](https://github.com/prometheus/node_exporter) provides these in
the Prometheus ecosystem, such metrics should be dropped.

In the Java world, many instrumentation frameworks expose process-level
and JVM-level stats such as CPU and GC. The Java client and JMX exporter
already include these in the preferred form via
[DefaultExports.java](https://github.com/prometheus/client_java/blob/master/simpleclient_hotspot/src/main/java/io/prometheus/client/hotspot/DefaultExports.java),
so these should also be dropped.

Similarly with other languages and frameworks.

## Deployment

Each exporter should monitor exactly one instance application,
preferably sitting right beside it on the same machine. That means for
every HAProxy you run, you run a `haproxy_exporter` process. For every
machine with a Mesos worker, you run the [Mesos
exporter](https://github.com/mesosphere/mesos_exporter) on it, and
another one for the master, if a machine has both.

The theory behind this is that for direct instrumentation this is what
you’d be doing, and we’re trying to get as close to that as we can in
other layouts.  This means that all service discovery is done in
Prometheus, not in exporters.  This also has the benefit that Prometheus
has the target information it needs to allow users probe your service
with the [blackbox
exporter](https://github.com/prometheus/blackbox_exporter).

There are two exceptions:

The first is where running beside the application you are monitoring is
completely nonsensical. The SNMP, blackbox and IPMI exporters are the
main examples of this. The IPMI and SNMP exporters as the devices are
often black boxes that it’s impossible to run code on (though if you
could run a node exporter on them instead that’d be better), and the
blackbox exporter where you’re monitoring something like a DNS name,
where there’s also nothing to run on. In this case, Prometheus should
still do service discovery, and pass on the target to be scraped. See
the blackbox and SNMP exporters for examples.

Note that it is only currently possible to write this type of exporter
with the Go, Python and Java client libraries.

The second exception is where you’re pulling some stats out of a random
instance of a system and don’t care which one you’re talking to.
Consider a set of MySQL replicas you wanted to run some business queries
against the data to then export. Having an exporter that uses your usual
load balancing approach to talk to one replica is the sanest approach.

This doesn’t apply when you’re monitoring a system with master-election,
in that case you should monitor each instance individually and deal with
the "masterness" in Prometheus. This is as there isn’t always exactly
one master, and changing what a target is underneath Prometheus’s feet
will cause oddities.

### Scheduling

Metrics should only be pulled from the application when Prometheus
scrapes them, exporters should not perform scrapes based on their own
timers. That is, all scrapes should be synchronous.

Accordingly, you should not set timestamps on the metrics you expose, let
Prometheus take care of that. If you think you need timestamps, then you
probably need the
[Pushgateway](https://prometheus.io/docs/instrumenting/pushing/)
instead.

If a metric is particularly expensive to retrieve, i.e. takes more than
a minute, it is acceptable to cache it. This should be noted in the
`HELP` string.

The default scrape timeout for Prometheus is 10 seconds. If your
exporter can be expected to exceed this, you should explicitly call this
out in your user documentation.

### Pushes

Some applications and monitoring systems only push metrics, for example
StatsD, Graphite and collectd.

There are two considerations here.

Firstly, when do you expire metrics? Collectd and things talking to
Graphite both export regularly, and when they stop we want to stop
exposing the metrics.  Collectd includes an expiry time so we use that,
Graphite doesn’t so it is a flag on the exporter.

StatsD is a bit different, as it is dealing with events rather than
metrics. The best model is to run one exporter beside each application
and restart them when the application restarts so that the state is
cleared.

Secondly, these sort of systems tend to allow your users to send either
deltas or raw counters. You should rely on the raw counters as far as
possible, as that’s the general Prometheus model.

For service-level metrics, e.g. service-level batch jobs, you should
have your exporter push into the Pushgateway and exit after the event
rather than handling the state yourself. For instance-level batch
metrics, there is no clear pattern yet. The options are either to abuse
the node exporter’s textfile collector, rely on in-memory state
(probably best if you don’t need to persist over a reboot) or implement
similar functionality to the textfile collector.

### Failed scrapes

There are currently two patterns for failed scrapes where the
application you’re talking to doesn’t respond or has other problems.

The first is to return a 5xx error.

The second is to have a `myexporter_up`, e.g. `haproxy_up`, variable
that has a value of 0 or 1 depending on whether the scrape worked.

The latter is better where there’s still some useful metrics you can get
even with a failed scrape, such as the HAProxy exporter providing
process stats. The former is a tad easier for users to deal with, as
[`up` works in the usual way](/docs/concepts/jobs_instances/#automatically-generated-labels-and-time-series), although you can’t distinguish between the
exporter being down and the application being down.

### Landing page

It’s nicer for users if visiting `http://yourexporter/` has a simple
HTML page with the name of the exporter, and a link to the `/metrics`
page.

### Port numbers

A user may have many exporters and Prometheus components on the same
machine, so to make that easier each has a unique port number.

[https://github.com/prometheus/prometheus/wiki/Default-port-allocations](https://github.com/prometheus/prometheus/wiki/Default-port-allocations)
is where we track them, this is publicly editable.

Feel free to grab the next free port number when developing your
exporter, preferably before publicly announcing it. If you’re not ready
to release yet, putting your username and WIP is fine.

This is a registry to make our users’ lives a little easier, not a
commitment to develop particular exporters. For exporters for internal
applications we recommend using ports outside of the range of default
port allocations.

## Announcing

Once you’re ready to announce your exporter to the world, email the
mailing list and send a PR to add it to [the list of available
exporters](https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exporters.md).

title  Writing exporters sort rank  5        Writing exporters  If you are instrumenting your own code  the  general rules of how to instrument code with a Prometheus client library   docs practices instrumentation   should be followed  When taking metrics from another monitoring or instrumentation system  things tend not to be so black and white   This document contains things you should consider when writing an exporter or custom collector  The theory covered will also be of interest to those doing direct instrumentation   If you are writing an exporter and are unclear on anything here  please contact us on IRC   prometheus on libera  or the  mailing list   community       Maintainability and purity  The main decision you need to make when writing an exporter is how much work you re willing to put in to get perfect metrics out of it   If the system in question has only a handful of metrics that rarely change  then getting everything perfect is an easy choice  a good example of this is the  HAProxy exporter  https   github com prometheus haproxy exporter    On the other hand  if you try to get things perfect when the system has hundreds of metrics that change frequently with new versions  then you ve signed yourself up for a lot of ongoing work  The  MySQL exporter  https   github com prometheus mysqld exporter  is on this end of the spectrum   The  node exporter  https   github com prometheus node exporter  is a mix of these  with complexity varying by module  For example  the  mdadm  collector hand parses a file and exposes metrics created specifically for that collector  so we may as well get the metrics right  For the  meminfo  collector the results vary across kernel versions so we end up doing just enough of a transform to create valid metrics      Configuration  When working with applications  you should aim for an exporter that requires no custom configuration by the user beyond telling it where the application is   You may also need to offer the ability to filter out certain metrics if they may be too granular and expensive on large setups  for example the  HAProxy exporter  https   github com prometheus haproxy exporter  allows filtering of per server stats  Similarly  there may be expensive metrics that are disabled by default   When working with other monitoring systems  frameworks and protocols you will often need to provide additional configuration or customization to generate metrics suitable for Prometheus  In the best case scenario  a monitoring system has a similar enough data model to Prometheus that you can automatically determine how to transform metrics  This is the case for  Cloudwatch  https   github com prometheus cloudwatch exporter    SNMP  https   github com prometheus snmp exporter  and  collectd  https   github com prometheus collectd exporter   At most  we need the ability to let the user select which metrics they want to pull out   In other cases  metrics from the system are completely non standard  depending on the usage of the system and the underlying application   In that case the user has to tell us how to transform the metrics  The  JMX exporter  https   github com prometheus jmx exporter  is the worst offender here  with the  Graphite  https   github com prometheus graphite exporter  and  StatsD  https   github com prometheus statsd exporter  exporters also requiring configuration to extract labels   Ensuring the exporter works out of the box without configuration  and providing a selection of example configurations for transformation if required  is advised   YAML is the standard Prometheus configuration format  all configuration should use YAML by default      Metrics      Naming  Follow the  best practices on metric naming   docs practices naming    Generally metric names should allow someone who is familiar with Prometheus but not a particular system to make a good guess as to what a metric means   A metric named  http requests total  is not extremely useful   are these being measured as they come in  in some filter or when they get to the user s code   And  requests total  is even worse  what type of requests   With direct instrumentation  a given metric should exist within exactly one file  Accordingly  within exporters and collectors  a metric should apply to exactly one subsystem and be named accordingly   Metric names should never be procedurally generated  except when writing a custom collector or exporter   Metric names for applications should generally be prefixed by the exporter name  e g   haproxy up    Metrics must use base units  e g  seconds  bytes  and leave converting them to something more readable to graphing tools  No matter what units you end up using  the units in the metric name must match the units in use  Similarly  expose ratios  not percentages  Even better  specify a counter for each of the two components of the ratio   Metric names should not include the labels that they re exported with  e g   by type   as that won t make sense if the label is aggregated away   The one exception is when you re exporting the same data with different labels via multiple metrics  in which case that s usually the sanest way to distinguish them  For direct instrumentation  this should only come up when exporting a single metric with all the labels would have too high a cardinality   Prometheus metrics and label names are written in  snake case   Converting  camelCase  to  snake case  is desirable  though doing so automatically doesn t always produce nice results for things like  myTCPExample  or  isNaN  so sometimes it s best to leave them as is   Exposed metrics should not contain colons  these are reserved for user defined recording rules to use when aggregating   Only   a zA Z0 9     are valid in metric names   The   sum     count     bucket  and   total  suffixes are used by Summaries  Histograms and Counters  Unless you re producing one of those  avoid these suffixes     total  is a convention for counters  you should use it if you re using the COUNTER type   The  process   and  scrape   prefixes are reserved  It s okay to add your own prefix on to these if they follow matching semantics  For example  Prometheus has  scrape duration seconds  for how long a scrape took  it s good practice to also have an exporter centric metric  e g   jmx scrape duration seconds   saying how long the specific exporter took to do its thing  For process stats where you have access to the PID  both Go and Python offer collectors that ll handle this for you  A good example of this is the  HAProxy exporter  https   github com prometheus haproxy exporter    When you have a successful request count and a failed request count  the best way to expose this is as one metric for total requests and another metric for failed requests  This makes it easy to calculate the failure ratio  Do not use one metric with a failed or success label  Similarly  with hit or miss for caches  it s better to have one metric for total and another for hits   Consider the likelihood that someone using monitoring will do a code or web search for the metric name  If the names are very well established and unlikely to be used outside of the realm of people used to those names  for example SNMP and network engineers  then leaving them as is may be a good idea  This logic doesn t apply for all exporters  for example the MySQL exporter metrics may be used by a variety of people  not just DBAs  A  HELP  string with the original name can provide most of the same benefits as using the original names       Labels  Read the  general advice   docs practices instrumentation  things to watch out for  on labels   Avoid  type  as a label name  it s too generic and often meaningless  You should also try where possible to avoid names that are likely to clash with target labels  such as  region    zone    cluster    availability zone    az    datacenter    dc    owner    customer    stage    service    environment  and  env   If  however  that s what the application calls some resource  it s best not to cause confusion by renaming it   Avoid the temptation to put things into one metric just because they share a prefix  Unless you re sure something makes sense as one metric  multiple metrics is safer   The label  le  has special meaning for Histograms  and  quantile  for Summaries  Avoid these labels generally   Read write and send receive are best as separate metrics  rather than as a label  This is usually because you care about only one of them at a time  and it is easier to use them that way   The rule of thumb is that one metric should make sense when summed or averaged   There is one other case that comes up with exporters  and that s where the data is fundamentally tabular and doing otherwise would require users to do regexes on metric names to be usable  Consider the voltage sensors on your motherboard  while doing math across them is meaningless  it makes sense to have them in one metric rather than having one metric per sensor  All values within a metric should  almost  always have the same unit  for example consider if fan speeds were mixed in with the voltages  and you had no way to automatically separate them   Don t do this    pre  my metric label  a   1 my metric label  b   6  b my metric label  total   7  b    pre   or this    pre  my metric label  a   1 my metric label  b   6  b my metric   7  b    pre   The former breaks for people who do a  sum    over your metric  and the latter breaks sum and is quite difficult to work with  Some client libraries  for example Go  will actively try to stop you doing the latter in a custom collector  and all client libraries should stop you from doing the latter with direct instrumentation  Never do either of these  rely on Prometheus aggregation instead   If your monitoring exposes a total like this  drop the total  If you have to keep it around for some reason  for example the total includes things not counted individually  use different metric names   Instrumentation labels should be minimal  every extra label is one more that users need to consider when writing their PromQL  Accordingly  avoid having instrumentation labels which could be removed without affecting the uniqueness of the time series  Additional information around a metric can be added via an info metric  for an example see below how to handle version numbers   However  there are cases where it is expected that virtually all users of a metric will want the additional information  If so  adding a non unique label  rather than an info metric  is the right solution  For example the  mysqld exporter  https   github com prometheus mysqld exporter  s  mysqld perf schema events statements total  s  digest  label is a hash of the full query pattern and is sufficient for uniqueness  However  it is of little use without the human readable  digest text  label  which for long queries will contain only the start of the query pattern and is thus not unique  Thus we end up with both the  digest text  label for humans and the  digest  label for uniqueness       Target labels  not static scraped labels  If you ever find yourself wanting to apply the same label to all of your metrics  stop   There s generally two cases where this comes up   The first is for some label it would be useful to have on the metrics such as the version number of the software  Instead  use the approach described at  https   www robustperception io how to have labels for machine roles   http   www robustperception io how to have labels for machine roles     The second case is when a label is really a target label  These are things like region  cluster names  and so on  that come from your infrastructure setup rather than the application itself  It s not for an application to say where it fits in your label taxonomy  that s for the person running the Prometheus server to configure and different people monitoring the same application may give it different names   Accordingly  these labels belong up in the scrape configs of Prometheus via whatever service discovery you re using  It s okay to apply the concept of machine roles here as well  as it s likely useful information for at least some people scraping it       Types  You should try to match up the types of your metrics to Prometheus types  This usually means counters and gauges  The   count  and   sum  of summaries are also relatively common  and on occasion you ll see quantiles  Histograms are rare  if you come across one remember that the exposition format exposes cumulative values   Often it won t be obvious what the type of metric is  especially if you re automatically processing a set of metrics  In general  UNTYPED  is a safe default   Counters can t go down  so if you have a counter type coming from another instrumentation system that can be decremented  for example Dropwizard metrics then it s not a counter  it s a gauge   UNTYPED  is probably the best type to use there  as  GAUGE  would be misleading if it were being used as a counter       Help strings  When you re transforming metrics it s useful for users to be able to track back to what the original was  and what rules were in play that caused that transformation  Putting in the name of the collector or exporter  the ID of any rule that was applied and the name and details of the original metric into the help string will greatly aid users   Prometheus doesn t like one metric having different help strings  If you re making one metric from many others  choose one of them to put in the help string   For examples of this  the SNMP exporter uses the OID and the JMX exporter puts in a sample mBean name  The  HAProxy exporter  https   github com prometheus haproxy exporter  has hand written strings  The  node exporter  https   github com prometheus node exporter  also has a wide variety of examples       Drop less useful statistics  Some instrumentation systems expose 1m  5m  15m rates  average rates since application start  these are called  mean  in Dropwizard metrics for example  in addition to minimums  maximums and standard deviations   These should all be dropped  as they re not very useful and add clutter  Prometheus can calculate rates itself  and usually more accurately as the averages exposed are usually exponentially decaying  You don t know what time the min or max were calculated over  and the standard deviation is statistically useless and you can always expose sum of squares    sum  and   count  if you ever need to calculate it   Quantiles have related issues  you may choose to drop them or put them in a Summary       Dotted strings  Many monitoring systems don t have labels  instead doing things like  my class path mymetric labelvalue1 labelvalue2 labelvalue3    The  Graphite  https   github com prometheus graphite exporter  and  StatsD  https   github com prometheus statsd exporter  exporters share a way of transforming these with a small configuration language  Other exporters should implement the same  The transformation is currently implemented only in Go  and would benefit from being factored out into a separate library      Collectors  When implementing the collector for your exporter  you should never use the usual direct instrumentation approach and then update the metrics on each scrape   Rather create new metrics each time  In Go this is done with  MustNewConstMetric  https   godoc org github com prometheus client golang prometheus MustNewConstMetric  in your  Collect    method  For Python see  https   github com prometheus client python custom collectors  https   prometheus github io client python collector custom   and for Java generate a  List MetricFamilySamples   in your collect method  see  StandardExports java  https   github com prometheus client java blob master simpleclient hotspot src main java io prometheus client hotspot StandardExports java  for an example   The reason for this is two fold  Firstly  two scrapes could happen at the same time  and direct instrumentation uses what are effectively file level global variables  so you ll get race conditions  Secondly  if a label value disappears  it ll still be exported   Instrumenting your exporter itself via direct instrumentation is fine  e g  total bytes transferred or calls performed by the exporter across all scrapes   For exporters such as the  blackbox exporter  https   github com prometheus blackbox exporter  and  SNMP exporter  https   github com prometheus snmp exporter   which aren t tied to a single target  these should only be exposed on a vanilla   metrics  call  not on a scrape of a particular target       Metrics about the scrape itself  Sometimes you d like to export metrics that are about the scrape  like how long it took or how many records you processed   These should be exposed as gauges as they re about an event  the scrape  and the metric name prefixed by the exporter name  for example  jmx scrape duration seconds   Usually the   exporter  is excluded and if the exporter also makes sense to use as just a collector  then definitely exclude it   Other scrape  meta  metrics should be avoided  For example  a counter for the number of scrapes  or a histogram of the scrape duration  Having the exporter track these metrics duplicate the  automatically generated metrics  docs concepts jobs instances  automatically generated labels and time series  of Prometheus itself  This adds to the storage cost of every exporter instance       Machine and process metrics  Many systems  for example Elasticsearch  expose machine metrics such as CPU  memory and filesystem information  As the  node exporter  https   github com prometheus node exporter  provides these in the Prometheus ecosystem  such metrics should be dropped   In the Java world  many instrumentation frameworks expose process level and JVM level stats such as CPU and GC  The Java client and JMX exporter already include these in the preferred form via  DefaultExports java  https   github com prometheus client java blob master simpleclient hotspot src main java io prometheus client hotspot DefaultExports java   so these should also be dropped   Similarly with other languages and frameworks      Deployment  Each exporter should monitor exactly one instance application  preferably sitting right beside it on the same machine  That means for every HAProxy you run  you run a  haproxy exporter  process  For every machine with a Mesos worker  you run the  Mesos exporter  https   github com mesosphere mesos exporter  on it  and another one for the master  if a machine has both   The theory behind this is that for direct instrumentation this is what you d be doing  and we re trying to get as close to that as we can in other layouts   This means that all service discovery is done in Prometheus  not in exporters   This also has the benefit that Prometheus has the target information it needs to allow users probe your service with the  blackbox exporter  https   github com prometheus blackbox exporter    There are two exceptions   The first is where running beside the application you are monitoring is completely nonsensical  The SNMP  blackbox and IPMI exporters are the main examples of this  The IPMI and SNMP exporters as the devices are often black boxes that it s impossible to run code on  though if you could run a node exporter on them instead that d be better   and the blackbox exporter where you re monitoring something like a DNS name  where there s also nothing to run on  In this case  Prometheus should still do service discovery  and pass on the target to be scraped  See the blackbox and SNMP exporters for examples   Note that it is only currently possible to write this type of exporter with the Go  Python and Java client libraries   The second exception is where you re pulling some stats out of a random instance of a system and don t care which one you re talking to  Consider a set of MySQL replicas you wanted to run some business queries against the data to then export  Having an exporter that uses your usual load balancing approach to talk to one replica is the sanest approach   This doesn t apply when you re monitoring a system with master election  in that case you should monitor each instance individually and deal with the  masterness  in Prometheus  This is as there isn t always exactly one master  and changing what a target is underneath Prometheus s feet will cause oddities       Scheduling  Metrics should only be pulled from the application when Prometheus scrapes them  exporters should not perform scrapes based on their own timers  That is  all scrapes should be synchronous   Accordingly  you should not set timestamps on the metrics you expose  let Prometheus take care of that  If you think you need timestamps  then you probably need the  Pushgateway  https   prometheus io docs instrumenting pushing   instead   If a metric is particularly expensive to retrieve  i e  takes more than a minute  it is acceptable to cache it  This should be noted in the  HELP  string   The default scrape timeout for Prometheus is 10 seconds  If your exporter can be expected to exceed this  you should explicitly call this out in your user documentation       Pushes  Some applications and monitoring systems only push metrics  for example StatsD  Graphite and collectd   There are two considerations here   Firstly  when do you expire metrics  Collectd and things talking to Graphite both export regularly  and when they stop we want to stop exposing the metrics   Collectd includes an expiry time so we use that  Graphite doesn t so it is a flag on the exporter   StatsD is a bit different  as it is dealing with events rather than metrics  The best model is to run one exporter beside each application and restart them when the application restarts so that the state is cleared   Secondly  these sort of systems tend to allow your users to send either deltas or raw counters  You should rely on the raw counters as far as possible  as that s the general Prometheus model   For service level metrics  e g  service level batch jobs  you should have your exporter push into the Pushgateway and exit after the event rather than handling the state yourself  For instance level batch metrics  there is no clear pattern yet  The options are either to abuse the node exporter s textfile collector  rely on in memory state  probably best if you don t need to persist over a reboot  or implement similar functionality to the textfile collector       Failed scrapes  There are currently two patterns for failed scrapes where the application you re talking to doesn t respond or has other problems   The first is to return a 5xx error   The second is to have a  myexporter up   e g   haproxy up   variable that has a value of 0 or 1 depending on whether the scrape worked   The latter is better where there s still some useful metrics you can get even with a failed scrape  such as the HAProxy exporter providing process stats  The former is a tad easier for users to deal with  as   up  works in the usual way   docs concepts jobs instances  automatically generated labels and time series   although you can t distinguish between the exporter being down and the application being down       Landing page  It s nicer for users if visiting  http   yourexporter   has a simple HTML page with the name of the exporter  and a link to the   metrics  page       Port numbers  A user may have many exporters and Prometheus components on the same machine  so to make that easier each has a unique port number    https   github com prometheus prometheus wiki Default port allocations  https   github com prometheus prometheus wiki Default port allocations  is where we track them  this is publicly editable   Feel free to grab the next free port number when developing your exporter  preferably before publicly announcing it  If you re not ready to release yet  putting your username and WIP is fine   This is a registry to make our users  lives a little easier  not a commitment to develop particular exporters  For exporters for internal applications we recommend using ports outside of the range of default port allocations      Announcing  Once you re ready to announce your exporter to the world  email the mailing list and send a PR to add it to  the list of available exporters  https   github com prometheus docs blob main content docs instrumenting exporters md


prometheus sortrank 6 title Exposition formats Exposition formats Metrics can be exposed to Prometheus using a simple that implement this format for you If your preferred language doesn t have a client exposition format There are various

Answer:
---
title: Exposition formats
sort_rank: 6
---

# Exposition formats

Metrics can be exposed to Prometheus using a simple [text-based](#text-based-format)
exposition format. There are various [client libraries](/docs/instrumenting/clientlibs/)
that implement this format for you. If your preferred language doesn't have a client
library you can [create your own](/docs/instrumenting/writing_clientlibs/).

## Text-based format

As of Prometheus version 2.0, all processes that expose metrics to Prometheus need to use
a text-based format. In this section you can find some [basic information](#basic-info)
about this format as well as a more [detailed breakdown](#text-format-details) of the
format.

### Basic info

| Aspect | Description |
|--------|-------------|
| **Inception** | April 2014  |
| **Supported in** |  Prometheus version `>=0.4.0` |
| **Transmission** | HTTP |
| **Encoding** | UTF-8, `\n` line endings |
| **HTTP `Content-Type`** | `text/plain; version=0.0.4` (A missing `version` value will lead to a fall-back to the most recent text format version.) |
| **Optional HTTP `Content-Encoding`** | `gzip` |
| **Advantages** |

Human-readable

Easy to assemble, especially for minimalistic cases (no nesting required)

Readable line by line (with the exception of type hints and docstrings)

 |
| **Limitations** |

Verbose

Types and docstrings not integral part of the syntax, meaning little-to-nonexistent metric contract validation

Parsing cost

|
| **Supported metric primitives** |

Counter

Gauge

Histogram

Summary

Untyped

 |

### Text format details

Prometheus' text-based format is line oriented. Lines are separated by a line
feed character (`\n`). The last line must end with a line feed character.
Empty lines are ignored.

#### Line format

Within a line, tokens can be separated by any number of blanks and/or tabs (and
must be separated by at least one if they would otherwise merge with the previous
token). Leading and trailing whitespace is ignored.

#### Comments, help text, and type information

Lines with a `#` as the first non-whitespace character are comments. They are
ignored unless the first token after `#` is either `HELP` or `TYPE`. Those
lines are treated as follows: If the token is `HELP`, at least one more token
is expected, which is the metric name. All remaining tokens are considered the
docstring for that metric name. `HELP` lines may contain any sequence of UTF-8
characters (after the metric name), but the backslash and the line feed
characters have to be escaped as `\\` and `\n`, respectively. Only one `HELP`
line may exist for any given metric name.

If the token is `TYPE`, exactly two more tokens are expected. The first is the
metric name, and the second is either `counter`, `gauge`, `histogram`,
`summary`, or `untyped`, defining the type for the metric of that name. Only
one `TYPE` line may exist for a given metric name. The `TYPE` line for a
metric name must appear before the first sample is reported for that metric
name. If there is no `TYPE` line for a metric name, the type is set to
`untyped`.

The remaining lines describe samples (one per line) using the following syntax
([EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form)):

```
metric_name [
  "{" label_name "=" `"` label_value `"` { "," label_name "=" `"` label_value `"` } [ "," ] "}"
] value [ timestamp ]
```

In the sample syntax:

*  `metric_name` and `label_name` carry the usual Prometheus expression language restrictions.
* `label_value` can be any sequence of UTF-8 characters, but the backslash (`\`), double-quote (`"`), and line feed (`\n`) characters have to be escaped as `\\`, `\"`, and `\n`, respectively.
* `value` is a float represented as required by Go's [`ParseFloat()`](https://golang.org/pkg/strconv/#ParseFloat) function. In addition to standard numerical values, `NaN`, `+Inf`, and `-Inf` are valid values representing not a number, positive infinity, and negative infinity, respectively.
* The `timestamp` is an `int64` (milliseconds since epoch, i.e. 1970-01-01 00:00:00 UTC, excluding leap seconds), represented as required by Go's [`ParseInt()`](https://golang.org/pkg/strconv/#ParseInt) function.

#### Grouping and sorting

All lines for a given metric must be provided as one single group, with
the optional `HELP` and `TYPE` lines first (in no particular order). Beyond
that, reproducible sorting in repeated expositions is preferred but not
required, i.e. do not sort if the computational cost is prohibitive.

Each line must have a unique combination of a metric name and labels. Otherwise,
the ingestion behavior is undefined.

#### Histograms and summaries

The `histogram` and `summary` types are difficult to represent in the text
format. The following conventions apply:

* The sample sum for a summary or histogram named `x` is given as a separate sample named `x_sum`.
* The sample count for a summary or histogram named `x` is given as a separate sample named `x_count`.
* Each quantile of a summary named `x` is given as a separate sample line with the same name `x` and a label `{quantile="y"}`.
* Each bucket count of a histogram named `x` is given as a separate sample line with the name `x_bucket` and a label `{le="y"}` (where `y` is the upper bound of the bucket).
* A histogram _must_ have a bucket with `{le="+Inf"}`. Its value _must_ be identical to the value of `x_count`.
* The buckets of a histogram and the quantiles of a summary must appear in increasing numerical order of their label values (for the `le` or the `quantile` label, respectively).

### Text format example

Below is an example of a full-fledged Prometheus metric exposition, including
comments, `HELP` and `TYPE` expressions, a histogram, a summary, character
escaping examples, and more.

```
# HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="post",code="200"} 1027 1395066363000
http_requests_total{method="post",code="400"}    3 1395066363000

# Escaping in label values:
msdos_file_access_time_seconds{path="C:\\DIR\\FILE.TXT",error="Cannot find file:\n\"FILE.TXT\""} 1.458255915e9

# Minimalistic line:
metric_without_timestamp_and_labels 12.47

# A weird metric from before the epoch:
something_weird{problem="division by zero"} +Inf -3982045

# A histogram, which has a pretty complex representation in the text format:
# HELP http_request_duration_seconds A histogram of the request duration.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.05"} 24054
http_request_duration_seconds_bucket{le="0.1"} 33444
http_request_duration_seconds_bucket{le="0.2"} 100392
http_request_duration_seconds_bucket{le="0.5"} 129389
http_request_duration_seconds_bucket{le="1"} 133988
http_request_duration_seconds_bucket{le="+Inf"} 144320
http_request_duration_seconds_sum 53423
http_request_duration_seconds_count 144320

# Finally a summary, which has a complex representation, too:
# HELP rpc_duration_seconds A summary of the RPC duration in seconds.
# TYPE rpc_duration_seconds summary
rpc_duration_seconds{quantile="0.01"} 3102
rpc_duration_seconds{quantile="0.05"} 3272
rpc_duration_seconds{quantile="0.5"} 4773
rpc_duration_seconds{quantile="0.9"} 9001
rpc_duration_seconds{quantile="0.99"} 76656
rpc_duration_seconds_sum 1.7560473e+07
rpc_duration_seconds_count 2693
```

## OpenMetrics Text Format

[OpenMetrics](https://github.com/OpenObservability/OpenMetrics) is the an effort to standardize metric wire formatting built off of Prometheus text format. It is possible to scrape targets
and it is also available to use for federating metrics since at least v2.23.0.

### Exemplars (Experimental)

Utilizing the OpenMetrics format allows for the exposition and querying of [Exemplars](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#exemplars).
Exemplars provide a point in time snapshot related to a metric set for an otherwise summarized MetricFamily. Additionally they may have a Trace ID attached to them which when used to together
with a tracing system can provide more detailed information related to the specific service.

To enable this experimental feature you must have at least version v2.26.0 and add `--enable-feature=exemplar-storage` to your arguments.

## Protobuf format

Earlier versions of Prometheus supported an exposition format based on [Protocol Buffers](https://developers.google.com/protocol-buffers/) (aka Protobuf) in addition to the current text-based format. With Prometheus 2.0, the Protobuf format was marked as deprecated and Prometheus stopped ingesting samples from said exposition format.

However, new experimental features were added to Prometheus where the Protobuf format was considered the most viable option. Making Prometheus accept Protocol Buffers once again.

Here is a list of experimental features that, once enabled, will configure Prometheus to favor the Protobuf exposition format:

| feature flag | version that introduced it |
|--------------|----------------------------|
| native-histograms | 2.40.0 |
| created-timestamp-zero-ingestion | 2.50.0 |

## Historical versions

For details on historical format versions, see the legacy
[Client Data Exposition Format](https://docs.google.com/document/d/1ZjyKiKxZV83VI9ZKAXRGKaUKK2BIWCT7oiGBKDBpjEY/edit?usp=sharing)
document.

The current version of the original Protobuf format (with the recent extensions
for native histograms) is maintained in the [prometheus/client_model
repository](https://github.com/prometheus/client_model).

title  Exposition formats sort rank  6        Exposition formats  Metrics can be exposed to Prometheus using a simple  text based   text based format  exposition format  There are various  client libraries   docs instrumenting clientlibs   that implement this format for you  If your preferred language doesn t have a client library you can  create your own   docs instrumenting writing clientlibs        Text based format  As of Prometheus version 2 0  all processes that expose metrics to Prometheus need to use a text based format  In this section you can find some  basic information   basic info  about this format as well as a more  detailed breakdown   text format details  of the format       Basic info    Aspect   Description                                Inception     April 2014        Supported in      Prometheus version    0 4 0        Transmission     HTTP       Encoding     UTF 8    n  line endings       HTTP  Content Type       text plain  version 0 0 4   A missing  version  value will lead to a fall back to the most recent text format version         Optional HTTP  Content Encoding       gzip        Advantages      ul  li Human readable  li  li Easy to assemble  especially for minimalistic cases  no nesting required   li  li Readable line by line  with the exception of type hints and docstrings   li   ul        Limitations      ul  li Verbose  li  li Types and docstrings not integral part of the syntax  meaning little to nonexistent metric contract validation  li  li Parsing cost  li   ul       Supported metric primitives      ul  li Counter  li  li Gauge  li  li Histogram  li  li Summary  li  li Untyped  li   ul         Text format details  Prometheus  text based format is line oriented  Lines are separated by a line feed character    n    The last line must end with a line feed character  Empty lines are ignored        Line format  Within a line  tokens can be separated by any number of blanks and or tabs  and must be separated by at least one if they would otherwise merge with the previous token   Leading and trailing whitespace is ignored        Comments  help text  and type information  Lines with a     as the first non whitespace character are comments  They are ignored unless the first token after     is either  HELP  or  TYPE   Those lines are treated as follows  If the token is  HELP   at least one more token is expected  which is the metric name  All remaining tokens are considered the docstring for that metric name   HELP  lines may contain any sequence of UTF 8 characters  after the metric name   but the backslash and the line feed characters have to be escaped as      and   n   respectively  Only one  HELP  line may exist for any given metric name   If the token is  TYPE   exactly two more tokens are expected  The first is the metric name  and the second is either  counter    gauge    histogram    summary   or  untyped   defining the type for the metric of that name  Only one  TYPE  line may exist for a given metric name  The  TYPE  line for a metric name must appear before the first sample is reported for that metric name  If there is no  TYPE  line for a metric name  the type is set to  untyped    The remaining lines describe samples  one per line  using the following syntax   EBNF  https   en wikipedia org wiki Extended Backus E2 80 93Naur form         metric name         label name         label value           label name         label value                     value   timestamp        In the sample syntax       metric name  and  label name  carry the usual Prometheus expression language restrictions     label value  can be any sequence of UTF 8 characters  but the backslash        double quote        and line feed    n   characters have to be escaped as             and   n   respectively     value  is a float represented as required by Go s   ParseFloat     https   golang org pkg strconv  ParseFloat  function  In addition to standard numerical values   NaN     Inf   and   Inf  are valid values representing not a number  positive infinity  and negative infinity  respectively    The  timestamp  is an  int64   milliseconds since epoch  i e  1970 01 01 00 00 00 UTC  excluding leap seconds   represented as required by Go s   ParseInt     https   golang org pkg strconv  ParseInt  function        Grouping and sorting  All lines for a given metric must be provided as one single group  with the optional  HELP  and  TYPE  lines first  in no particular order   Beyond that  reproducible sorting in repeated expositions is preferred but not required  i e  do not sort if the computational cost is prohibitive   Each line must have a unique combination of a metric name and labels  Otherwise  the ingestion behavior is undefined        Histograms and summaries  The  histogram  and  summary  types are difficult to represent in the text format  The following conventions apply     The sample sum for a summary or histogram named  x  is given as a separate sample named  x sum     The sample count for a summary or histogram named  x  is given as a separate sample named  x count     Each quantile of a summary named  x  is given as a separate sample line with the same name  x  and a label   quantile  y       Each bucket count of a histogram named  x  is given as a separate sample line with the name  x bucket  and a label   le  y     where  y  is the upper bound of the bucket     A histogram  must  have a bucket with   le   Inf     Its value  must  be identical to the value of  x count     The buckets of a histogram and the quantiles of a summary must appear in increasing numerical order of their label values  for the  le  or the  quantile  label  respectively        Text format example  Below is an example of a full fledged Prometheus metric exposition  including comments   HELP  and  TYPE  expressions  a histogram  a summary  character escaping examples  and more         HELP http requests total The total number of HTTP requests    TYPE http requests total counter http requests total method  post  code  200   1027 1395066363000 http requests total method  post  code  400      3 1395066363000    Escaping in label values  msdos file access time seconds path  C   DIR  FILE TXT  error  Cannot find file  n  FILE TXT     1 458255915e9    Minimalistic line  metric without timestamp and labels 12 47    A weird metric from before the epoch  something weird problem  division by zero    Inf  3982045    A histogram  which has a pretty complex representation in the text format    HELP http request duration seconds A histogram of the request duration    TYPE http request duration seconds histogram http request duration seconds bucket le  0 05   24054 http request duration seconds bucket le  0 1   33444 http request duration seconds bucket le  0 2   100392 http request duration seconds bucket le  0 5   129389 http request duration seconds bucket le  1   133988 http request duration seconds bucket le   Inf   144320 http request duration seconds sum 53423 http request duration seconds count 144320    Finally a summary  which has a complex representation  too    HELP rpc duration seconds A summary of the RPC duration in seconds    TYPE rpc duration seconds summary rpc duration seconds quantile  0 01   3102 rpc duration seconds quantile  0 05   3272 rpc duration seconds quantile  0 5   4773 rpc duration seconds quantile  0 9   9001 rpc duration seconds quantile  0 99   76656 rpc duration seconds sum 1 7560473e 07 rpc duration seconds count 2693         OpenMetrics Text Format   OpenMetrics  https   github com OpenObservability OpenMetrics  is the an effort to standardize metric wire formatting built off of Prometheus text format  It is possible to scrape targets and it is also available to use for federating metrics since at least v2 23 0       Exemplars  Experimental   Utilizing the OpenMetrics format allows for the exposition and querying of  Exemplars  https   github com OpenObservability OpenMetrics blob main specification OpenMetrics md exemplars   Exemplars provide a point in time snapshot related to a metric set for an otherwise summarized MetricFamily  Additionally they may have a Trace ID attached to them which when used to together with a tracing system can provide more detailed information related to the specific service   To enable this experimental feature you must have at least version v2 26 0 and add    enable feature exemplar storage  to your arguments      Protobuf format  Earlier versions of Prometheus supported an exposition format based on  Protocol Buffers  https   developers google com protocol buffers    aka Protobuf  in addition to the current text based format  With Prometheus 2 0  the Protobuf format was marked as deprecated and Prometheus stopped ingesting samples from said exposition format   However  new experimental features were added to Prometheus where the Protobuf format was considered the most viable option  Making Prometheus accept Protocol Buffers once again   Here is a list of experimental features that  once enabled  will configure Prometheus to favor the Protobuf exposition format     feature flag   version that introduced it                                                   native histograms   2 40 0     created timestamp zero ingestion   2 50 0       Historical versions  For details on historical format versions  see the legacy  Client Data Exposition Format  https   docs google com document d 1ZjyKiKxZV83VI9ZKAXRGKaUKK2BIWCT7oiGBKDBpjEY edit usp sharing  document   The current version of the original Protobuf format  with the recent extensions for native histograms  is maintained in the  prometheus client model repository  https   github com prometheus client model


prometheus Prometheus Remote Write Specification title Prometheus Remote Write 1 0 Version 1 0 sortrank 5 Status Published Date April 2023

Answer:
---
title: Prometheus Remote-Write 1.0
sort_rank: 5
---

# Prometheus Remote-Write Specification

- Version: 1.0
- Status: Published
- Date: April 2023

This document is intended to define and standardise the API, wire format, protocol and semantics of the existing, widely and organically adopted protocol, and not to propose anything new.

The remote write specification is intended to document the standard for how Prometheus and Prometheus remote-write-compatible agents send data to a Prometheus or Prometheus remote-write compatible receiver.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://datatracker.ietf.org/doc/html/rfc2119).

> NOTE: This specification has a 2.0 version available, see [here](./remote_write_spec_2_0.md).

## Introduction
### Background

The remote write protocol is designed to make it possible to reliably propagate samples in real-time from a sender to a receiver, without loss.

The Remote-Write protocol is designed to be stateless; there is strictly no inter-message communication. As such the protocol is not considered "streaming". To achieve a streaming effect multiple messages should be sent over the same connection using e.g. HTTP/1.1 or HTTP/2. "Fancy" technologies such as gRPC were considered, but at the time were not widely adopted, and it was challenging to expose gRPC services to the internet behind load balancers such as an AWS EC2 ELB.

The remote write protocol contains opportunities for batching, e.g. sending multiple samples for different series in a single request.  It is not expected that multiple samples for the same series will be commonly sent in the same request, although there is support for this in the protocol.

The remote write protocol is not intended for use by applications to push metrics to Prometheus remote-write-compatible receivers.  It is intended that a Prometheus remote-write-compatible sender scrapes instrumented applications or exporters and sends remote write messages to a server.

A test suite can be found at https://github.com/prometheus/compliance/tree/main/remotewrite/sender.

### Glossary

For the purposes of this document the following definitions MUST be followed:

- a "Sender" is something that sends Prometheus Remote Write data.
- a "Receiver" is something that receives Prometheus Remote Write data.
- a "Sample" is a pair of (timestamp, value).
- a "Label" is a pair of (key, value).
- a "Series" is a list of samples, identified by a unique set of labels.

## Definitions
### Protocol

The Remote Write Protocol MUST consist of RPCs with the following signature:

```
func Send(WriteRequest)

message WriteRequest {
  repeated TimeSeries timeseries = 1;
  // Cortex uses this field to determine the source of the write request.
  // We reserve it to avoid any compatibility issues.
  reserved  2;

  // Prometheus uses this field to send metadata, but this is
  // omitted from v1 of the spec as it is experimental.
  reserved  3;
}

message TimeSeries {
  repeated Label labels   = 1;
  repeated Sample samples = 2;
}

message Label {
  string name  = 1;
  string value = 2;
}

message Sample {
  double value    = 1;
  int64 timestamp = 2;
}
```

Remote write Senders MUST encode the Write Request in the body of a HTTP POST request and send it to the Receivers via HTTP at a provided URL path.  The Receiver MAY specify any HTTP URL path to receive metrics.

Timestamps MUST be int64 counted as milliseconds since the Unix epoch.  Values MUST be float64.

The following headers MUST be sent with the HTTP request:

- `Content-Encoding: snappy`
- `Content-Type: application/x-protobuf`
- `User-Agent: <name & version of the sender>`
- `X-Prometheus-Remote-Write-Version: 0.1.0`

Clients MAY allow users to send custom HTTP headers; they MUST NOT allow users to configure them in such a way as to send reserved headers.  For more info see https://github.com/prometheus/prometheus/pull/8416.

The remote write request in the body of the HTTP POST MUST be compressed with [Google’s Snappy](https://github.com/google/snappy).  The block format MUST be used - the framed format MUST NOT be used.

The remote write request MUST be encoded using Google Protobuf 3, and MUST use the schema defined above.  Note [the Prometheus implementation](https://github.com/prometheus/prometheus/blob/v2.24.0/prompb/remote.proto) uses [gogoproto optimisations](https://github.com/gogo/protobuf) - for receivers written in languages other than Golang the gogoproto types MAY be substituted for line-level equivalents.

The response body from the remote write receiver SHOULD be empty; clients MUST ignore the response body. The response body is RESERVED for future use.

### Backward and forward compatibility

The protocol follows [semantic versioning 2.0](https://semver.org/): any 1.x compatible receivers MUST be able to read any 1.x compatible sender and so on.  Breaking/backwards incompatible changes will result in a 2.x version of the spec.

The proto format itself is forward / backward compatible, in some respects:

- Removing fields from the proto will mean a major version bump.
- Adding (optional) fields will be a minor version bump.

Negotiation:

- Senders MUST send the version number in a headers.
- Receivers MAY return the highest version number they support in a response header ("X-Prometheus-Remote-Write-Version").
- Senders who wish to send in a format >1.x MUST start by sending an empty 1.x, and see if the response says the receiver supports something else.  The Sender MAY use any supported version .  If there is no version header in the response, senders MUST assume 1.x compatibility only.

### Labels

The complete set of labels MUST be sent with each sample. Whatsmore, the label set associated with samples:

- SHOULD contain a `__name__` label.
- MUST NOT contain repeated label names.
- MUST have label names sorted in lexicographical order.
- MUST NOT contain any empty label names or values.

Senders MUST only send valid metric names, label names, and label values:

- Metric names MUST adhere to the regex `[a-zA-Z_:]([a-zA-Z0-9_:])*`.
- Label names MUST adhere to the regex `[a-zA-Z_]([a-zA-Z0-9_])*`.
- Label values MAY be any sequence of UTF-8 characters .

Receivers MAY impose limits on the number and length of labels, but this will be receiver-specific and is out of scope for this document.

Label names beginning with "__" are RESERVED for system usage and SHOULD NOT be used, see [Prometheus Data Model](https://prometheus.io/docs/concepts/data_model/).

Remote write Receivers MAY ingest valid samples within a write request that otherwise contains invalid samples. Receivers MUST return a HTTP 400 status code ("Bad Request") for write requests that contain any invalid samples. Receivers SHOULD provide a human readable error message in the response body. Senders MUST NOT try and interpret the error message, and SHOULD log it as is.

### Ordering
Prometheus Remote Write compatible senders MUST send samples for any given series in timestamp order. Prometheus Remote Write compatible Senders MAY send multiple requests for different series in parallel.

### Retries & Backoff
Prometheus Remote Write compatible senders MUST retry write requests on HTTP 5xx responses and MUST use a backoff algorithm to prevent overwhelming the server. They MUST NOT retry write requests on HTTP 2xx and 4xx responses other than 429. They MAY retry on HTTP 429 responses, which could result in senders "falling behind" if the server cannot keep up. This is done to ensure data is not lost when there are server side errors, and progress is made when there are client side errors.

Prometheus remote Write compatible receivers MUST respond with a HTTP 2xx status code when the write is successful. They MUST respond with HTTP status code 5xx when the write fails and SHOULD be retried. They MUST respond with HTTP status code 4xx when the request is invalid, will never be able to succeed and should not be retried.

### Stale Markers
Prometheus remote write compatible senders MUST send stale markers when a time series will no longer be appended to.

Stale markers MUST be signalled by  the special NaN value 0x7ff0000000000002. This value MUST NOT be used otherwise.

Typically the sender can detect when a time series will no longer be appended to using the following techniques:

1. Detecting, using service discovery, that the target exposing the series has gone away
1. Noticing the target is no longer exposing the time series between successive scrapes
1. Failing to scrape the target that originally exposed a time series
1. Tracking configuration and evaluation for recording and alerting rules

## Out of Scope
This document does not intend to explain all the features required for a fully Prometheus-compatible monitoring system.  In particular, the following areas are out of scope for the first version of the spec:

**The "up" metric** The definition and semantics of the "up" metric are beyond the scope of the remote write protocol and should be documented separately.

**HTTP Path** The path for HTTP handler can be anything - and MUST be provided by the sender.  Generally we expect the whole URL to be specified in config.

**Persistence** It is recommended that Prometheus Remote Write compatible senders should persistently buffer sample data in the event of outages in the receiver.

**Authentication & Encryption** as remote write uses HTTP, we consider authentication & encryption to be a transport-layer problem.  Senders and receivers should support all the usual suspects (Basic auth, TLS etc) and are free to add potentially custom authentication options.  Support for custom authentication in the Prometheus remote write sender and eventual agent should not be assumed, but we will endeavour to support common and widely used auth protocols, where feasible.

**Remote Read** this is a separate interface that has already seen some iteration, and is less widely used.

**Sharding** the current sharding scheme in Prometheus for remote write parallelisation is very much an implementation detail, and isn’t part of the spec.  When senders do implement parallelisation they MUST preserve per-series sample ordering.

**Backfill** The specification does not place a limit on how old series can be pushed, however server/implementation specific constraints may exist.

**Limits** Limits on the number and length of labels, batch sizes etc are beyond the scope of this document, however it is expected that implementation will impose reasonable limits.

**Push-based Prometheus** Applications pushing metrics to Prometheus Remote Write compatible receivers was not a design goal of this system, and should be explored in a separate doc.

**Labels** Every series MAY include a "job" and/or "instance" label, as these are typically added by service discovery in the Sender. These are not mandatory.

## Future Plans
This section contains speculative plans that are not considered part of protocol specification, but are mentioned here for completeness.

**Transactionality** Prometheus aims at being "transactional" - i.e. to never expose a partially scraped target to a query. We intend to do the same with remote write -  for instance, in the future we would like to "align" remote write with scrapes, perhaps such that all the samples, metadata and exemplars for a single scrape are sent in a single remote write request.  This is yet to be designed.

**Metadata** and Exemplars In line with above, we also send metadata (type information, help text) and exemplars along with the scraped samples. We plan to package this up in a single remote write request - future versions of the spec may insist on this.  Prometheus currently has experimental support for sending metadata and exemplars.

**Optimizations** We would like to investigate various optimizations to reduce message size by eliminating repetition of label names and values.

## Related
### Compatible Senders and Receivers

The spec is intended to describe how the following components interact (as of April 2023):

- [Prometheus](https://github.com/prometheus/prometheus/tree/master/storage/remote) (as both a "sender" and a "receiver")
- [Avalanche](https://github.com/prometheus-community/avalanche) (as a "sender") - A Load Testing Tool Prometheus Metrics.
- [Cortex](https://github.com/cortexproject/cortex/blob/master/pkg/util/push/push.go#L20) (as a "receiver")
- [Elastic Agent](https://docs.elastic.co/integrations/prometheus#prometheus-server-remote-write) (as a "receiver")
- [Grafana Agent](https://github.com/grafana/agent) (as both a "sender" and a "receiver")
- [GreptimeDB](https://github.com/greptimeTeam/greptimedb) (as a ["receiver"](https://docs.greptime.com/user-guide/ingest-data/for-observerbility/prometheus))
- InfluxData’s Telegraf agent. ([as a sender](https://github.com/influxdata/telegraf/tree/master/plugins/serializers/prometheusremotewrite), and [as a receiver](https://github.com/influxdata/telegraf/pull/8967))
- [M3](https://m3db.io/docs/integrations/prometheus/#prometheus-configuration) (as a "receiver")
- [Mimir](https://github.com/grafana/mimir) (as a "receiver")
- [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector-releases/) (as a ["sender"](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusremotewriteexporter#readme) and eventually as a "receiver")
- [Thanos](https://thanos.io/tip/components/receive.md/) (as a "receiver")
- Vector (as a ["sender"](https://vector.dev/docs/reference/configuration/sinks/prometheus_remote_write/) and a ["receiver"](https://vector.dev/docs/reference/configuration/sources/prometheus_remote_write/))
- [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics) (as a ["receiver"](https://docs.victoriametrics.com/#prometheus-setup))

### FAQ

**Why did you not use gRPC?**
Funnily enough we initially used gRPC, but switched to Protos atop HTTP as in 2016 it was hard to get them past ELBs: https://github.com/prometheus/prometheus/issues/1982

**Why not streaming protobuf messages?**
If you use persistent HTTP/1.1 connections, they are pretty close to streaming…  Of course headers have to be re-sent, but yes that is less expensive than a new TCP set up.

**Why do we send samples in order?**
The in-order constraint comes from the encoding we use for time series data in Prometheus, the implementation of which is append only. It is possible to remove this constraint, for instance by buffering samples and reordering them before encoding.  We can investigate this in future versions of the protocol.

**How can we parallelise requests with the in-order constraint?**
Samples must be in-order _for a given series_.  Remote write requests can be sent in parallel as long as they are for different series. In Prometheus, we shard the samples by their labels into separate queues, and then writes happen sequentially in each queue.  This guarantees samples for the same series are delivered in order, but samples for different series are sent in parallel - and potentially "out of order" between different series.

We believe this is necessary as, even if the receiver could support out-of-order samples, we can't have agents sending out of order as they would never be able to send to Prometheus, Cortex and Thanos.  We’re doing this to ensure the integrity of the ecosystem and to prevent confusing/forking the community into "prometheus-agents-that-can-write-to-prometheus" and those that can’t.

title  Prometheus Remote Write 1 0 sort rank  5        Prometheus Remote Write Specification    Version  1 0   Status  Published   Date  April 2023  This document is intended to define and standardise the API  wire format  protocol and semantics of the existing  widely and organically adopted protocol  and not to propose anything new   The remote write specification is intended to document the standard for how Prometheus and Prometheus remote write compatible agents send data to a Prometheus or Prometheus remote write compatible receiver   The key words  MUST    MUST NOT    REQUIRED    SHALL    SHALL NOT    SHOULD    SHOULD NOT    RECOMMENDED     MAY   and  OPTIONAL  in this document are to be interpreted as described in  RFC 2119  https   datatracker ietf org doc html rfc2119      NOTE  This specification has a 2 0 version available  see  here    remote write spec 2 0 md       Introduction     Background  The remote write protocol is designed to make it possible to reliably propagate samples in real time from a sender to a receiver  without loss   The Remote Write protocol is designed to be stateless  there is strictly no inter message communication  As such the protocol is not considered  streaming   To achieve a streaming effect multiple messages should be sent over the same connection using e g  HTTP 1 1 or HTTP 2   Fancy  technologies such as gRPC were considered  but at the time were not widely adopted  and it was challenging to expose gRPC services to the internet behind load balancers such as an AWS EC2 ELB   The remote write protocol contains opportunities for batching  e g  sending multiple samples for different series in a single request   It is not expected that multiple samples for the same series will be commonly sent in the same request  although there is support for this in the protocol   The remote write protocol is not intended for use by applications to push metrics to Prometheus remote write compatible receivers   It is intended that a Prometheus remote write compatible sender scrapes instrumented applications or exporters and sends remote write messages to a server   A test suite can be found at https   github com prometheus compliance tree main remotewrite sender       Glossary  For the purposes of this document the following definitions MUST be followed     a  Sender  is something that sends Prometheus Remote Write data    a  Receiver  is something that receives Prometheus Remote Write data    a  Sample  is a pair of  timestamp  value     a  Label  is a pair of  key  value     a  Series  is a list of samples  identified by a unique set of labels      Definitions     Protocol  The Remote Write Protocol MUST consist of RPCs with the following signature       func Send WriteRequest   message WriteRequest     repeated TimeSeries timeseries   1       Cortex uses this field to determine the source of the write request       We reserve it to avoid any compatibility issues    reserved  2        Prometheus uses this field to send metadata  but this is      omitted from v1 of the spec as it is experimental    reserved  3     message TimeSeries     repeated Label labels     1    repeated Sample samples   2     message Label     string name    1    string value   2     message Sample     double value      1    int64 timestamp   2         Remote write Senders MUST encode the Write Request in the body of a HTTP POST request and send it to the Receivers via HTTP at a provided URL path   The Receiver MAY specify any HTTP URL path to receive metrics   Timestamps MUST be int64 counted as milliseconds since the Unix epoch   Values MUST be float64   The following headers MUST be sent with the HTTP request      Content Encoding  snappy     Content Type  application x protobuf     User Agent   name   version of the sender      X Prometheus Remote Write Version  0 1 0   Clients MAY allow users to send custom HTTP headers  they MUST NOT allow users to configure them in such a way as to send reserved headers   For more info see https   github com prometheus prometheus pull 8416   The remote write request in the body of the HTTP POST MUST be compressed with  Google s Snappy  https   github com google snappy    The block format MUST be used   the framed format MUST NOT be used   The remote write request MUST be encoded using Google Protobuf 3  and MUST use the schema defined above   Note  the Prometheus implementation  https   github com prometheus prometheus blob v2 24 0 prompb remote proto  uses  gogoproto optimisations  https   github com gogo protobuf    for receivers written in languages other than Golang the gogoproto types MAY be substituted for line level equivalents   The response body from the remote write receiver SHOULD be empty  clients MUST ignore the response body  The response body is RESERVED for future use       Backward and forward compatibility  The protocol follows  semantic versioning 2 0  https   semver org    any 1 x compatible receivers MUST be able to read any 1 x compatible sender and so on   Breaking backwards incompatible changes will result in a 2 x version of the spec   The proto format itself is forward   backward compatible  in some respects     Removing fields from the proto will mean a major version bump    Adding  optional  fields will be a minor version bump   Negotiation     Senders MUST send the version number in a headers    Receivers MAY return the highest version number they support in a response header   X Prometheus Remote Write Version      Senders who wish to send in a format  1 x MUST start by sending an empty 1 x  and see if the response says the receiver supports something else   The Sender MAY use any supported version    If there is no version header in the response  senders MUST assume 1 x compatibility only       Labels  The complete set of labels MUST be sent with each sample  Whatsmore  the label set associated with samples     SHOULD contain a    name    label    MUST NOT contain repeated label names    MUST have label names sorted in lexicographical order    MUST NOT contain any empty label names or values   Senders MUST only send valid metric names  label names  and label values     Metric names MUST adhere to the regex   a zA Z     a zA Z0 9          Label names MUST adhere to the regex   a zA Z    a zA Z0 9         Label values MAY be any sequence of UTF 8 characters    Receivers MAY impose limits on the number and length of labels  but this will be receiver specific and is out of scope for this document   Label names beginning with      are RESERVED for system usage and SHOULD NOT be used  see  Prometheus Data Model  https   prometheus io docs concepts data model     Remote write Receivers MAY ingest valid samples within a write request that otherwise contains invalid samples  Receivers MUST return a HTTP 400 status code   Bad Request   for write requests that contain any invalid samples  Receivers SHOULD provide a human readable error message in the response body  Senders MUST NOT try and interpret the error message  and SHOULD log it as is       Ordering Prometheus Remote Write compatible senders MUST send samples for any given series in timestamp order  Prometheus Remote Write compatible Senders MAY send multiple requests for different series in parallel       Retries   Backoff Prometheus Remote Write compatible senders MUST retry write requests on HTTP 5xx responses and MUST use a backoff algorithm to prevent overwhelming the server  They MUST NOT retry write requests on HTTP 2xx and 4xx responses other than 429  They MAY retry on HTTP 429 responses  which could result in senders  falling behind  if the server cannot keep up  This is done to ensure data is not lost when there are server side errors  and progress is made when there are client side errors   Prometheus remote Write compatible receivers MUST respond with a HTTP 2xx status code when the write is successful  They MUST respond with HTTP status code 5xx when the write fails and SHOULD be retried  They MUST respond with HTTP status code 4xx when the request is invalid  will never be able to succeed and should not be retried       Stale Markers Prometheus remote write compatible senders MUST send stale markers when a time series will no longer be appended to   Stale markers MUST be signalled by  the special NaN value 0x7ff0000000000002  This value MUST NOT be used otherwise   Typically the sender can detect when a time series will no longer be appended to using the following techniques   1  Detecting  using service discovery  that the target exposing the series has gone away 1  Noticing the target is no longer exposing the time series between successive scrapes 1  Failing to scrape the target that originally exposed a time series 1  Tracking configuration and evaluation for recording and alerting rules     Out of Scope This document does not intend to explain all the features required for a fully Prometheus compatible monitoring system   In particular  the following areas are out of scope for the first version of the spec     The  up  metric   The definition and semantics of the  up  metric are beyond the scope of the remote write protocol and should be documented separately     HTTP Path   The path for HTTP handler can be anything   and MUST be provided by the sender   Generally we expect the whole URL to be specified in config     Persistence   It is recommended that Prometheus Remote Write compatible senders should persistently buffer sample data in the event of outages in the receiver      Authentication   Encryption   as remote write uses HTTP  we consider authentication   encryption to be a transport layer problem   Senders and receivers should support all the usual suspects  Basic auth  TLS etc  and are free to add potentially custom authentication options   Support for custom authentication in the Prometheus remote write sender and eventual agent should not be assumed  but we will endeavour to support common and widely used auth protocols  where feasible     Remote Read   this is a separate interface that has already seen some iteration  and is less widely used     Sharding   the current sharding scheme in Prometheus for remote write parallelisation is very much an implementation detail  and isn t part of the spec   When senders do implement parallelisation they MUST preserve per series sample ordering     Backfill   The specification does not place a limit on how old series can be pushed  however server implementation specific constraints may exist     Limits   Limits on the number and length of labels  batch sizes etc are beyond the scope of this document  however it is expected that implementation will impose reasonable limits     Push based Prometheus   Applications pushing metrics to Prometheus Remote Write compatible receivers was not a design goal of this system  and should be explored in a separate doc     Labels   Every series MAY include a  job  and or  instance  label  as these are typically added by service discovery in the Sender  These are not mandatory      Future Plans This section contains speculative plans that are not considered part of protocol specification  but are mentioned here for completeness     Transactionality   Prometheus aims at being  transactional    i e  to never expose a partially scraped target to a query  We intend to do the same with remote write    for instance  in the future we would like to  align  remote write with scrapes  perhaps such that all the samples  metadata and exemplars for a single scrape are sent in a single remote write request   This is yet to be designed     Metadata   and Exemplars In line with above  we also send metadata  type information  help text  and exemplars along with the scraped samples  We plan to package this up in a single remote write request   future versions of the spec may insist on this   Prometheus currently has experimental support for sending metadata and exemplars     Optimizations   We would like to investigate various optimizations to reduce message size by eliminating repetition of label names and values      Related     Compatible Senders and Receivers  The spec is intended to describe how the following components interact  as of April 2023       Prometheus  https   github com prometheus prometheus tree master storage remote   as both a  sender  and a  receiver      Avalanche  https   github com prometheus community avalanche   as a  sender     A Load Testing Tool Prometheus Metrics     Cortex  https   github com cortexproject cortex blob master pkg util push push go L20   as a  receiver      Elastic Agent  https   docs elastic co integrations prometheus prometheus server remote write   as a  receiver      Grafana Agent  https   github com grafana agent   as both a  sender  and a  receiver      GreptimeDB  https   github com greptimeTeam greptimedb   as a   receiver   https   docs greptime com user guide ingest data for observerbility prometheus     InfluxData s Telegraf agent    as a sender  https   github com influxdata telegraf tree master plugins serializers prometheusremotewrite   and  as a receiver  https   github com influxdata telegraf pull 8967      M3  https   m3db io docs integrations prometheus  prometheus configuration   as a  receiver      Mimir  https   github com grafana mimir   as a  receiver      OpenTelemetry Collector  https   github com open telemetry opentelemetry collector releases    as a   sender   https   github com open telemetry opentelemetry collector contrib tree main exporter prometheusremotewriteexporter readme  and eventually as a  receiver      Thanos  https   thanos io tip components receive md    as a  receiver     Vector  as a   sender   https   vector dev docs reference configuration sinks prometheus remote write   and a   receiver   https   vector dev docs reference configuration sources prometheus remote write       VictoriaMetrics  https   github com VictoriaMetrics VictoriaMetrics   as a   receiver   https   docs victoriametrics com  prometheus setup        FAQ    Why did you not use gRPC    Funnily enough we initially used gRPC  but switched to Protos atop HTTP as in 2016 it was hard to get them past ELBs  https   github com prometheus prometheus issues 1982    Why not streaming protobuf messages    If you use persistent HTTP 1 1 connections  they are pretty close to streaming   Of course headers have to be re sent  but yes that is less expensive than a new TCP set up     Why do we send samples in order    The in order constraint comes from the encoding we use for time series data in Prometheus  the implementation of which is append only  It is possible to remove this constraint  for instance by buffering samples and reordering them before encoding   We can investigate this in future versions of the protocol     How can we parallelise requests with the in order constraint    Samples must be in order  for a given series    Remote write requests can be sent in parallel as long as they are for different series  In Prometheus  we shard the samples by their labels into separate queues  and then writes happen sequentially in each queue   This guarantees samples for the same series are delivered in order  but samples for different series are sent in parallel   and potentially  out of order  between different series   We believe this is necessary as  even if the receiver could support out of order samples  we can t have agents sending out of order as they would never be able to send to Prometheus  Cortex and Thanos   We re doing this to ensure the integrity of the ecosystem and to prevent confusing forking the community into  prometheus agents that can write to prometheus  and those that can t