prometheus In this guide we will Use file based service discovery to discover scrape targets title Use file based service discovery to discover scrape targets Prometheus offers a variety of for discovering scrape targets including and many others If you need to use a service discovery system that is not currently supported your use case may be best served by Prometheus mechanism which enables you to list scrape targets in a JSON file along with metadata about those targets

Answer:
---
title: Use file-based service discovery to discover scrape targets
---

# Use file-based service discovery to discover scrape targets

Prometheus offers a variety of [service discovery options](https://github.com/prometheus/prometheus/tree/main/discovery) for discovering scrape targets, including [Kubernetes](/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config), [Consul](/docs/prometheus/latest/configuration/configuration/#consul_sd_config), and many others. If you need to use a service discovery system that is not currently supported, your use case may be best served by Prometheus' [file-based service discovery](/docs/prometheus/latest/configuration/configuration/#file_sd_config) mechanism, which enables you to list scrape targets in a JSON file (along with metadata about those targets).

In this guide, we will:

* Install and run a Prometheus [Node Exporter](../node-exporter) locally
* Create a `targets.json` file specifying the host and port information for the Node Exporter
* Install and run a Prometheus instance that is configured to discover the Node Exporter using the `targets.json` file

## Installing and running the Node Exporter

See [this section](../node-exporter#installing-and-running-the-node-exporter) of the [Monitoring Linux host metrics with the Node Exporter](../node-exporter) guide. The Node Exporter runs on port 9100. To ensure that the Node Exporter is exposing metrics:

```bash
curl http://localhost:9100/metrics
```

The metrics output should look something like this:

```
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
...
```

## Installing, configuring, and running Prometheus

Like the Node Exporter, Prometheus is a single static binary that you can install via tarball. [Download the latest release](/download#prometheus) for your platform and untar it:

```bash
wget https://github.com/prometheus/prometheus/releases/download/v*/prometheus-*.*-amd64.tar.gz
tar xvf prometheus-*.*-amd64.tar.gz
cd prometheus-*.*
```

The untarred directory contains a `prometheus.yml` configuration file. Replace the current contents of that file with this:

```yaml
scrape_configs:
- job_name: 'node'
  file_sd_configs:
  - files:
    - 'targets.json'
```

This configuration specifies that there is a job called `node` (for the Node Exporter) that retrieves host and port information for Node Exporter instances from a `targets.json` file.

Now create that `targets.json` file and add this content to it:

```json
[
  {
    "labels": {
      "job": "node"
    },
    "targets": [
      "localhost:9100"
    ]
  }
]
```

NOTE: In this guide we'll work with JSON service discovery configurations manually for the sake of brevity. In general, however, we recommend that you use some kind of JSON-generating process or tool instead.

This configuration specifies that there is a `node` job with one target: `localhost:9100`.

Now you can start up Prometheus:

```bash
./prometheus
```

If Prometheus has started up successfully, you should see a line like this in the logs:

```
level=info ts=2018-08-13T20:39:24.905651509Z caller=main.go:500 msg="Server is ready to receive web requests."
```

## Exploring the discovered services' metrics

With Prometheus up and running, you can explore metrics exposed by the `node` service using the Prometheus [expression browser](/docs/visualization/browser). If you explore the [`up{job="node"}`](http://localhost:9090/graph?g0.range_input=1h&g0.expr=up%7Bjob%3D%22node%22%7D&g0.tab=1) metric, for example, you can see that the Node Exporter is being appropriately discovered.

## Changing the targets list dynamically

When using Prometheus' file-based service discovery mechanism, the Prometheus instance will listen for changes to the file and automatically update the scrape target list, without requiring an instance restart. To demonstrate this, start up a second Node Exporter instance on port 9200. First navigate to the directory containing the Node Exporter binary and run this command in a new terminal window:

```bash
./node_exporter --web.listen-address=":9200"
```

Now modify the config in `targets.json` by adding an entry for the new Node Exporter:

```json
[
  {
    "targets": [
      "localhost:9100"
    ],
    "labels": {
      "job": "node"
    }
  },
  {
    "targets": [
      "localhost:9200"
    ],
    "labels": {
      "job": "node"
    }
  }
]
```

When you save the changes, Prometheus will automatically be notified of the new list of targets. The [`up{job="node"}`](http://localhost:9090/graph?g0.range_input=1h&g0.expr=up%7Bjob%3D%22node%22%7D&g0.tab=1) metric should display two instances with `instance` labels `localhost:9100` and `localhost:9200`.

## Summary

In this guide, you installed and ran a Prometheus Node Exporter and configured Prometheus to discover and scrape metrics from the Node Exporter using file-based service discovery.

title  Use file based service discovery to discover scrape targets        Use file based service discovery to discover scrape targets  Prometheus offers a variety of  service discovery options  https   github com prometheus prometheus tree main discovery  for discovering scrape targets  including  Kubernetes   docs prometheus latest configuration configuration  kubernetes sd config    Consul   docs prometheus latest configuration configuration  consul sd config   and many others  If you need to use a service discovery system that is not currently supported  your use case may be best served by Prometheus   file based service discovery   docs prometheus latest configuration configuration  file sd config  mechanism  which enables you to list scrape targets in a JSON file  along with metadata about those targets    In this guide  we will     Install and run a Prometheus  Node Exporter     node exporter  locally   Create a  targets json  file specifying the host and port information for the Node Exporter   Install and run a Prometheus instance that is configured to discover the Node Exporter using the  targets json  file     Installing and running the Node Exporter  See  this section     node exporter installing and running the node exporter  of the  Monitoring Linux host metrics with the Node Exporter     node exporter  guide  The Node Exporter runs on port 9100  To ensure that the Node Exporter is exposing metrics      bash curl http   localhost 9100 metrics      The metrics output should look something like this         HELP go gc duration seconds A summary of the GC invocation durations    TYPE go gc duration seconds summary go gc duration seconds quantile  0   0 go gc duration seconds quantile  0 25   0 go gc duration seconds quantile  0 5   0             Installing  configuring  and running Prometheus  Like the Node Exporter  Prometheus is a single static binary that you can install via tarball   Download the latest release   download prometheus  for your platform and untar it      bash wget https   github com prometheus prometheus releases download v  prometheus     amd64 tar gz tar xvf prometheus     amd64 tar gz cd prometheus          The untarred directory contains a  prometheus yml  configuration file  Replace the current contents of that file with this      yaml scrape configs    job name   node    file sd configs      files         targets json       This configuration specifies that there is a job called  node   for the Node Exporter  that retrieves host and port information for Node Exporter instances from a  targets json  file   Now create that  targets json  file and add this content to it      json            labels            job    node              targets            localhost 9100                   NOTE  In this guide we ll work with JSON service discovery configurations manually for the sake of brevity  In general  however  we recommend that you use some kind of JSON generating process or tool instead   This configuration specifies that there is a  node  job with one target   localhost 9100    Now you can start up Prometheus      bash   prometheus      If Prometheus has started up successfully  you should see a line like this in the logs       level info ts 2018 08 13T20 39 24 905651509Z caller main go 500 msg  Server is ready to receive web requests           Exploring the discovered services  metrics  With Prometheus up and running  you can explore metrics exposed by the  node  service using the Prometheus  expression browser   docs visualization browser   If you explore the   up job  node     http   localhost 9090 graph g0 range input 1h g0 expr up 7Bjob 3D 22node 22 7D g0 tab 1  metric  for example  you can see that the Node Exporter is being appropriately discovered      Changing the targets list dynamically  When using Prometheus  file based service discovery mechanism  the Prometheus instance will listen for changes to the file and automatically update the scrape target list  without requiring an instance restart  To demonstrate this  start up a second Node Exporter instance on port 9200  First navigate to the directory containing the Node Exporter binary and run this command in a new terminal window      bash   node exporter   web listen address   9200       Now modify the config in  targets json  by adding an entry for the new Node Exporter      json            targets            localhost 9100              labels            job    node                      targets            localhost 9200              labels            job    node                   When you save the changes  Prometheus will automatically be notified of the new list of targets  The   up job  node     http   localhost 9090 graph g0 range input 1h g0 expr up 7Bjob 3D 22node 22 7D g0 tab 1  metric should display two instances with  instance  labels  localhost 9100  and  localhost 9200       Summary  In this guide  you installed and ran a Prometheus Node Exporter and configured Prometheus to discover and scrape metrics from the Node Exporter using file based service discovery


prometheus title OpenTelemetry Prometheus supports aka OpenTelemetry Protocol ingestion through Enable the OTLP receiver Using Prometheus as your OpenTelemetry backend

Answer:
---
title: OpenTelemetry
---

# Using Prometheus as your OpenTelemetry backend

Prometheus supports [OTLP](https://opentelemetry.io/docs/specs/otlp) (aka "OpenTelemetry Protocol") ingestion through [HTTP](https://opentelemetry.io/docs/specs/otlp/#otlphttp).

## Enable the OTLP receiver

By default, the OTLP receiver is disabled, similarly to the Remote Write receiver.
This is because Prometheus can work without any authentication, so it would not be
safe to accept incoming traffic unless explicitly configured.

To enable the receiver you need to toggle the CLI flag `--web.enable-otlp-receiver`.
This will cause Prometheus to serve OTLP metrics receiving on HTTP `/api/v1/otlp/v1/metrics` path.

```shell
$ prometheus --web.enable-otlp-receiver
```

## Send OpenTelemetry Metrics to the Prometheus Server

Generally you need to tell the source of the OTLP metrics traffic about Prometheus endpoint and the fact that the
[HTTP](https://opentelemetry.io/docs/specs/otlp/#otlphttp) mode of OTLP should be used (gRPC is usually a default).

OpenTelemetry SDKs and instrumentation libraries can be usually configured via [standard environment variables](https://opentelemetry.io/docs/languages/sdk-configuration/). The following are the OpenTelemetry variables needed to send OpenTelemetry metrics to a Prometheus server on localhost:

```shell
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://localhost:9090/api/v1/otlp/v1/metrics
```

Turn off traces and logs:

```shell
export OTEL_TRACES_EXPORTER=none
export OTEL_LOGS_EXPORTER=none
```

The default push interval for OpenTelemetry metrics is 60 seconds. The following will set a 15 second push interval:

```shell
export OTEL_METRIC_EXPORT_INTERVAL=15000
```

If your instrumentation library does not provide `service.name` and `service.instance.id` out-of-the-box, it is highly recommended to set them.

```shell
export OTEL_SERVICE_NAME="my-example-service"
export OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$(uuidgen)"
```

The above assumes that `uuidgen` command is available on your system. Make sure that `service.instance.id` is unique for each instance, and that a new `service.instance.id` is generated whenever a resource attribute chances. The [recommended](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/resource) way is to generate a new UUID on each startup of an instance.

## Configuring Prometheus

This section explains various recommended configuration aspects of Prometheus server to enable and tune your OpenTelemetry flow.

See the example Prometheus configuration [file](https://github.com/prometheus/prometheus/blob/main/documentation/examples/prometheus-otlp.yml)
we will use in the below section.

### Enable out-of-order ingestion

There are multiple reasons why you might want to enable out-of-order ingestion.

For example, the OpenTelemetry collector encourages batching and you could have multiple replicas of the collector sending data to Prometheus. Because there is no mechanism ordering those samples they could get out-of-order.

To enable out-of-order ingestion you need to extend the Prometheus configuration file with the following:

```shell
storage:
  tsdb:
    out_of_order_time_window: 30m
```

30 minutes of out-of-order have been enough for most cases but don't hesitate to adjust this value to your needs.

### Promoting resource attributes

Based on experience and conversations with our community, we've found that out of all the commonly seen resource attributes,
there are certain worth attaching to all your OTLP metrics.

By default, Prometheus won't be promoting any attributes. If you'd like to promote any
of them, you can do so in this section of the Prometheus configuration file. The following
snippet shares the best practise set of attributes to promote:

```yaml
otlp:
  # Recommended attributes to be promoted to labels.
  promote_resource_attributes:
    - service.instance.id
    - service.name
    - service.namespace
    - cloud.availability_zone
    - cloud.region
    - container.name
    - deployment.environment.name
    - k8s.cluster.name
    - k8s.container.name
    - k8s.cronjob.name
    - k8s.daemonset.name
    - k8s.deployment.name
    - k8s.job.name
    - k8s.namespace.name
    - k8s.pod.name
    - k8s.replicaset.name
    - k8s.statefulset.name
```

## Including resource attributes at query time

All non-promoted, more verbose or unique labels are attached to a special `target_info`.

You can use this metric to join some labels on query time.

An example of such a query can look like the following:

```promql
rate(http_server_request_duration_seconds_count[2m])
* on (job, instance) group_left (k8s_cluster_name)
target_info
```

What happens in this query is that the time series resulting from `rate(http_server_request_duration_seconds_count[2m])` are augmented with the `k8s_cluster_name` label from the `target_info` series that share the same `job` and `instance` labels.
In other words, the `job` and `instance` labels are shared between `http_server_request_duration_seconds_count` and `target_info`, akin to SQL foreign keys.
The `k8s_cluster_name` label, On the other hand, corresponds to the OTel resource attribute `k8s.cluster.name` (Prometheus converts dots to underscores).

So, what is the relation between the `target_info` metric and OTel resource attributes?
When Prometheus processes an OTLP write request, and provided that contained resources include the attributes `service.instance.id` and/or `service.name`, Prometheus generates the info metric `target_info` for every (OTel) resource.
It adds to each such `target_info` series the label `instance` with the value of the `service.instance.id` resource attribute, and the label `job` with the value of the `service.name` resource attribute.
If the resource attribute `service.namespace` exists, it's prefixed to the `job` label value (i.e., `<service.namespace>/<service.name>`).

By default `service.name`, `service.namespace` and `service.instance.id` themselves are not added to `target_info`, because they are converted into `job` and `instance`. However the following configuration parameter can be enabled to add them to `target_info` directly (going through normalization to replace dots with underscores, if `otlp.translation_strategy` is `UnderscoreEscapingWithSuffixes`) on top of the conversion into `job` and `instance`.

```
otlp:
  keep_identifying_resource_attributes: true
```

The rest of the resource attributes are also added as labels to the `target_info` series, names converted to Prometheus format (e.g. dots converted to underscores) if `otlp.translation_strategy` is `UnderscoreEscapingWithSuffixes`.
If a resource lacks both `service.instance.id` and `service.name` attributes, no corresponding `target_info` series is generated.

For each of a resource's OTel metrics, Prometheus converts it to a corresponding Prometheus time series, and (if `target_info` is generated) adds the right `instance` and `job` labels.

## UTF-8

From the 3.x version, Prometheus supports UTF-8 for metric names and labels, so [Prometheus normalization translator package from OpenTelemetry](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/translator/prometheus) can be omitted.

UTF-8 is enabled by default in Prometheus storage and UI, but you need to `translation_strategy` for OTLP metric receiver, which by default is set to old normalization `UnderscoreEscapingWithSuffixes`.

Setting it to `NoUTF8EscapingWithSuffixes`, which we recommend, will disable changing special characters to `_` which allows native use of OpenTelemetry metric format, especially with [the semantic conventions](https://opentelemetry.io/docs/specs/semconv/general/metrics/). Note that special suffixes like units and `_total` for counters will be attached. There is [ongoing work to have no suffix generation](https://github.com/prometheus/proposals/pull/39), stay tuned for that.

```
otlp:
  # Ingest OTLP data keeping UTF-8 characters in metric/label names.
  translation_strategy: NoUTF8EscapingWithSuffixes
```

> Currently there's a known limitation in the OTLP translation package where characters get removed from metric/label names if multiple UTF-8 characters are concatenated between words, e.g. `my___metric` becomes `my_metric`. Please see https://github.com/prometheus/prometheus/issues/15362 for more details.

## Delta Temporality

The [OpenTelemetry specification says](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality) that both Delta temporality and Cumulative temporality are supported.

While Delta temporality is common in systems like statsd and graphite, cumulative temporality is the default temporality for Prometheus.

Today Prometheus does not have support for delta temporality but we are learning from the OpenTelemetry community and we are considering adding support for it in the future.

If you are coming from a delta temporality system we recommend that you use the [delta to cumulative processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/deltatocumulativeprocessor) in your OTel pipeline.

title  OpenTelemetry        Using Prometheus as your OpenTelemetry backend  Prometheus supports  OTLP  https   opentelemetry io docs specs otlp   aka  OpenTelemetry Protocol   ingestion through  HTTP  https   opentelemetry io docs specs otlp  otlphttp       Enable the OTLP receiver  By default  the OTLP receiver is disabled  similarly to the Remote Write receiver  This is because Prometheus can work without any authentication  so it would not be safe to accept incoming traffic unless explicitly configured   To enable the receiver you need to toggle the CLI flag    web enable otlp receiver    This will cause Prometheus to serve OTLP metrics receiving on HTTP   api v1 otlp v1 metrics  path       shell   prometheus   web enable otlp receiver         Send OpenTelemetry Metrics to the Prometheus Server  Generally you need to tell the source of the OTLP metrics traffic about Prometheus endpoint and the fact that the  HTTP  https   opentelemetry io docs specs otlp  otlphttp  mode of OTLP should be used  gRPC is usually a default    OpenTelemetry SDKs and instrumentation libraries can be usually configured via  standard environment variables  https   opentelemetry io docs languages sdk configuration    The following are the OpenTelemetry variables needed to send OpenTelemetry metrics to a Prometheus server on localhost      shell export OTEL EXPORTER OTLP PROTOCOL http protobuf export OTEL EXPORTER OTLP METRICS ENDPOINT http   localhost 9090 api v1 otlp v1 metrics      Turn off traces and logs      shell export OTEL TRACES EXPORTER none export OTEL LOGS EXPORTER none      The default push interval for OpenTelemetry metrics is 60 seconds  The following will set a 15 second push interval      shell export OTEL METRIC EXPORT INTERVAL 15000      If your instrumentation library does not provide  service name  and  service instance id  out of the box  it is highly recommended to set them      shell export OTEL SERVICE NAME  my example service  export OTEL RESOURCE ATTRIBUTES  service instance id   uuidgen        The above assumes that  uuidgen  command is available on your system  Make sure that  service instance id  is unique for each instance  and that a new  service instance id  is generated whenever a resource attribute chances  The  recommended  https   github com open telemetry semantic conventions tree main docs resource  way is to generate a new UUID on each startup of an instance      Configuring Prometheus  This section explains various recommended configuration aspects of Prometheus server to enable and tune your OpenTelemetry flow   See the example Prometheus configuration  file  https   github com prometheus prometheus blob main documentation examples prometheus otlp yml  we will use in the below section       Enable out of order ingestion  There are multiple reasons why you might want to enable out of order ingestion   For example  the OpenTelemetry collector encourages batching and you could have multiple replicas of the collector sending data to Prometheus  Because there is no mechanism ordering those samples they could get out of order   To enable out of order ingestion you need to extend the Prometheus configuration file with the following      shell storage    tsdb      out of order time window  30m      30 minutes of out of order have been enough for most cases but don t hesitate to adjust this value to your needs       Promoting resource attributes  Based on experience and conversations with our community  we ve found that out of all the commonly seen resource attributes  there are certain worth attaching to all your OTLP metrics   By default  Prometheus won t be promoting any attributes  If you d like to promote any of them  you can do so in this section of the Prometheus configuration file  The following snippet shares the best practise set of attributes to promote      yaml otlp      Recommended attributes to be promoted to labels    promote resource attributes        service instance id       service name       service namespace       cloud availability zone       cloud region       container name       deployment environment name       k8s cluster name       k8s container name       k8s cronjob name       k8s daemonset name       k8s deployment name       k8s job name       k8s namespace name       k8s pod name       k8s replicaset name       k8s statefulset name         Including resource attributes at query time  All non promoted  more verbose or unique labels are attached to a special  target info    You can use this metric to join some labels on query time   An example of such a query can look like the following      promql rate http server request duration seconds count 2m     on  job  instance  group left  k8s cluster name  target info      What happens in this query is that the time series resulting from  rate http server request duration seconds count 2m    are augmented with the  k8s cluster name  label from the  target info  series that share the same  job  and  instance  labels  In other words  the  job  and  instance  labels are shared between  http server request duration seconds count  and  target info   akin to SQL foreign keys  The  k8s cluster name  label  On the other hand  corresponds to the OTel resource attribute  k8s cluster name   Prometheus converts dots to underscores    So  what is the relation between the  target info  metric and OTel resource attributes  When Prometheus processes an OTLP write request  and provided that contained resources include the attributes  service instance id  and or  service name   Prometheus generates the info metric  target info  for every  OTel  resource  It adds to each such  target info  series the label  instance  with the value of the  service instance id  resource attribute  and the label  job  with the value of the  service name  resource attribute  If the resource attribute  service namespace  exists  it s prefixed to the  job  label value  i e     service namespace   service name      By default  service name    service namespace  and  service instance id  themselves are not added to  target info   because they are converted into  job  and  instance   However the following configuration parameter can be enabled to add them to  target info  directly  going through normalization to replace dots with underscores  if  otlp translation strategy  is  UnderscoreEscapingWithSuffixes   on top of the conversion into  job  and  instance        otlp    keep identifying resource attributes  true      The rest of the resource attributes are also added as labels to the  target info  series  names converted to Prometheus format  e g  dots converted to underscores  if  otlp translation strategy  is  UnderscoreEscapingWithSuffixes   If a resource lacks both  service instance id  and  service name  attributes  no corresponding  target info  series is generated   For each of a resource s OTel metrics  Prometheus converts it to a corresponding Prometheus time series  and  if  target info  is generated  adds the right  instance  and  job  labels      UTF 8  From the 3 x version  Prometheus supports UTF 8 for metric names and labels  so  Prometheus normalization translator package from OpenTelemetry  https   github com open telemetry opentelemetry collector contrib tree main pkg translator prometheus  can be omitted   UTF 8 is enabled by default in Prometheus storage and UI  but you need to  translation strategy  for OTLP metric receiver  which by default is set to old normalization  UnderscoreEscapingWithSuffixes    Setting it to  NoUTF8EscapingWithSuffixes   which we recommend  will disable changing special characters to     which allows native use of OpenTelemetry metric format  especially with  the semantic conventions  https   opentelemetry io docs specs semconv general metrics    Note that special suffixes like units and   total  for counters will be attached  There is  ongoing work to have no suffix generation  https   github com prometheus proposals pull 39   stay tuned for that        otlp      Ingest OTLP data keeping UTF 8 characters in metric label names    translation strategy  NoUTF8EscapingWithSuffixes        Currently there s a known limitation in the OTLP translation package where characters get removed from metric label names if multiple UTF 8 characters are concatenated between words  e g   my   metric  becomes  my metric   Please see https   github com prometheus prometheus issues 15362 for more details      Delta Temporality  The  OpenTelemetry specification says  https   opentelemetry io docs specs otel metrics data model  temporality  that both Delta temporality and Cumulative temporality are supported   While Delta temporality is common in systems like statsd and graphite  cumulative temporality is the default temporality for Prometheus   Today Prometheus does not have support for delta temporality but we are learning from the OpenTelemetry community and we are considering adding support for it in the future   If you are coming from a delta temporality system we recommend that you use the  delta to cumulative processor  https   github com open telemetry opentelemetry collector contrib tree main processor deltatocumulativeprocessor  in your OTel pipeline


prometheus Versions of Prometheus before 3 0 required that metric and label names adhere to are valid names but there are some manual changes needed for other parts of the ecosystem to introduce names with any UTF 8 characters Introduction title UTF 8 in Prometheus a strict set of character requirements With Prometheus 3 0 all UTF 8 strings

Answer:
---
title: UTF-8 in Prometheus
---

# Introduction

Versions of Prometheus before 3.0 required that metric and label names adhere to
a strict set of character requirements. With Prometheus 3.0, all UTF-8 strings
are valid names, but there are some manual changes needed for other parts of the ecosystem to introduce names with any UTF-8 characters.

There may also be circumstances where users want to enforce the legacy character
set, perhaps for compatibility with an older system or one that does not yet
support UTF-8.

This document guides you through the UTF-8 transition details.

# Go Instrumentation

Currently, metrics created by the official Prometheus [client_golang library](github.com/prometheus/client_golang) will reject UTF-8 names
by default. It is necessary to change the default validation scheme to allow
UTF-8. The requirement to set this value will be removed in a future version of
the common library.

```golang
import "github.com/prometheus/common/model"

func init() {
	model.NameValidationScheme = model.UTF8Validation
}
```

If users want to enforce the legacy character set, they can set the validation
scheme to `LegacyValidation`.

Setting the validation scheme must be done before the instantiation of metrics
and can be set on the fly if desired.

## Instrumenting in other languages

Other client libraries may have similar requirements to set the validation
scheme. Check the documentation for the library you are using.

# Configuring Name Validation during Scraping

By default, Prometheus 3.0 accepts all UTF-8 strings as valid metric and label
names. It is possible to override this behavior for scraped targets and reject
names that do not conform to the legacy character set.

This option can be set in the Prometheus YAML file on a global basis:

```yaml
global:
  metric_name_validation_scheme: legacy
```

or on a per-scrape config basis:

```yaml
scrape_configs:
  - job_name: prometheus
    metric_name_validation_scheme: legacy
```

Scrape config settings override the global setting.

## Scrape Content Negotiation for UTF-8 escaping

At scrape time, the scraping system **must** pass `escaping=allow-utf-8` in the
Accept header in order to be served UTF-8 names. If a system being scraped does
not see this header, it will automatically convert UTF-8 names to
legacy-compatible using underscore replacement.

Scraping systems can also request a specfic escaping method if desired by
setting the `escaping` header to a different value.

* `underscores`: The default: convert legacy-invalid characters to underscores.
* `dots`: similar to UnderscoreEscaping, except that dots are converted to
  `_dot_` and pre-existing underscores are converted to `__`. This allows for
  round-tripping of simple metric names that also contain dots.
* `values`: This mode prepends the name with `U__` and replaces all invalid
  characters with the unicode value, surrounded by underscores. Single
  underscores are replaced with double underscores. This mode allows for full
  round-tripping of UTF-8 names with a legacy system.

## Remote Write 2.0

Remote Write 2.0 automatically accepts all UTF-8 names in Prometheus 3.0. There
is no way to enforce the legacy character set validation with Remote Write 2.0.

# OTLP Metrics

OTLP receiver in Prometheus 3.0 still normalizes all names to Prometheus format by default. You can change this in `otlp` section of the Prometheus configuration as follows:

    otlp:
      # Ingest OTLP data keeping UTF-8 characters in metric/label names.
      translation_strategy: NoUTF8EscapingWithSuffixes

See [OpenTelemetry guide](./opentelemetry) for more details.

# Querying

Querying for metrics with UTF-8 names will require a slightly different syntax
in PromQL.

The classic query syntax will still work for legacy-compatible names:

`my_metric{}`

But UTF-8 names must be quoted **and** moved into the braces:

`{"my.metric"}`

Label names must also be quoted if they contain legacy-incompatible characters:

`{"metric.name", "my.label.name"="bar"}`

The metric name can appear anywhere inside the braces, but style prefers that it
be the first term

title  UTF 8 in Prometheus        Introduction  Versions of Prometheus before 3 0 required that metric and label names adhere to a strict set of character requirements  With Prometheus 3 0  all UTF 8 strings are valid names  but there are some manual changes needed for other parts of the ecosystem to introduce names with any UTF 8 characters   There may also be circumstances where users want to enforce the legacy character set  perhaps for compatibility with an older system or one that does not yet support UTF 8   This document guides you through the UTF 8 transition details     Go Instrumentation   Currently  metrics created by the official Prometheus  client golang library  github com prometheus client golang  will reject UTF 8 names by default  It is necessary to change the default validation scheme to allow UTF 8  The requirement to set this value will be removed in a future version of the common library      golang import  github com prometheus common model   func init      model NameValidationScheme   model UTF8Validation        If users want to enforce the legacy character set  they can set the validation scheme to  LegacyValidation    Setting the validation scheme must be done before the instantiation of metrics and can be set on the fly if desired      Instrumenting in other languages  Other client libraries may have similar requirements to set the validation scheme  Check the documentation for the library you are using     Configuring Name Validation during Scraping  By default  Prometheus 3 0 accepts all UTF 8 strings as valid metric and label names  It is possible to override this behavior for scraped targets and reject names that do not conform to the legacy character set   This option can be set in the Prometheus YAML file on a global basis      yaml global    metric name validation scheme  legacy      or on a per scrape config basis      yaml scrape configs      job name  prometheus     metric name validation scheme  legacy      Scrape config settings override the global setting      Scrape Content Negotiation for UTF 8 escaping  At scrape time  the scraping system   must   pass  escaping allow utf 8  in the Accept header in order to be served UTF 8 names  If a system being scraped does not see this header  it will automatically convert UTF 8 names to legacy compatible using underscore replacement   Scraping systems can also request a specfic escaping method if desired by setting the  escaping  header to a different value      underscores   The default  convert legacy invalid characters to underscores     dots   similar to UnderscoreEscaping  except that dots are converted to     dot   and pre existing underscores are converted to       This allows for   round tripping of simple metric names that also contain dots     values   This mode prepends the name with  U    and replaces all invalid   characters with the unicode value  surrounded by underscores  Single   underscores are replaced with double underscores  This mode allows for full   round tripping of UTF 8 names with a legacy system      Remote Write 2 0  Remote Write 2 0 automatically accepts all UTF 8 names in Prometheus 3 0  There is no way to enforce the legacy character set validation with Remote Write 2 0     OTLP Metrics  OTLP receiver in Prometheus 3 0 still normalizes all names to Prometheus format by default  You can change this in  otlp  section of the Prometheus configuration as follows        otlp          Ingest OTLP data keeping UTF 8 characters in metric label names        translation strategy  NoUTF8EscapingWithSuffixes   See  OpenTelemetry guide    opentelemetry  for more details      Querying   Querying for metrics with UTF 8 names will require a slightly different syntax in PromQL   The classic query syntax will still work for legacy compatible names    my metric     But UTF 8 names must be quoted   and   moved into the braces      my metric     Label names must also be quoted if they contain legacy incompatible characters      metric name    my label name   bar     The metric name can appear anywhere inside the braces  but style prefers that it be the first term


prometheus to the usage of the specific types and in the wire protocol The Prometheus title Metric types currently only differentiated in the client libraries to enable APIs tailored sortrank 2 Metric types The Prometheus client libraries offer four core metric types These are

Answer:
---
title: Metric types
sort_rank: 2
---

# Metric types

The Prometheus client libraries offer four core metric types. These are
currently only differentiated in the client libraries (to enable APIs tailored
to the usage of the specific types) and in the wire protocol. The Prometheus
server does not yet make use of the type information and flattens all data into
untyped time series. This may change in the future.

## Counter

A _counter_ is a cumulative metric that represents a single [monotonically
increasing counter](https://en.wikipedia.org/wiki/Monotonic_function) whose
value can only increase or be reset to zero on restart. For example, you can
use a counter to represent the number of requests served, tasks completed, or
errors.

Do not use a counter to expose a value that can decrease. For example, do not
use a counter for the number of currently running processes; instead use a gauge.

Client library usage documentation for counters:

   * [Go](http://godoc.org/github.com/prometheus/client_golang/prometheus#Counter)
   * [Java](https://github.com/prometheus/client_java#counter)
   * [Python](https://prometheus.github.io/client_python/instrumenting/counter/)
   * [Ruby](https://github.com/prometheus/client_ruby#counter)
   * [.Net](https://github.com/prometheus-net/prometheus-net#counters)
   * [Rust](https://docs.rs/prometheus-client/latest/prometheus_client/metrics/counter/index.html)

## Gauge

A _gauge_ is a metric that represents a single numerical value that can
arbitrarily go up and down.

Gauges are typically used for measured values like temperatures or current
memory usage, but also "counts" that can go up and down, like the number of
concurrent requests.

Client library usage documentation for gauges:

   * [Go](http://godoc.org/github.com/prometheus/client_golang/prometheus#Gauge)
   * [Java](https://github.com/prometheus/client_java#gauge)
   * [Python](https://prometheus.github.io/client_python/instrumenting/gauge/)
   * [Ruby](https://github.com/prometheus/client_ruby#gauge)
   * [.Net](https://github.com/prometheus-net/prometheus-net#gauges)
   * [Rust](https://docs.rs/prometheus-client/latest/prometheus_client/metrics/gauge/index.html)

## Histogram

A _histogram_ samples observations (usually things like request durations or
response sizes) and counts them in configurable buckets. It also provides a sum
of all observed values.

A histogram with a base metric name of `<basename>` exposes multiple time series
during a scrape:

  * cumulative counters for the observation buckets, exposed as `<basename>_bucket{le="<upper inclusive bound>"}`
  * the **total sum** of all observed values, exposed as `<basename>_sum`
  * the **count** of events that have been observed, exposed as `<basename>_count` (identical to `<basename>_bucket{le="+Inf"}` above)

Use the
[`histogram_quantile()` function](/docs/prometheus/latest/querying/functions/#histogram_quantile)
to calculate quantiles from histograms or even aggregations of histograms. A
histogram is also suitable to calculate an
[Apdex score](http://en.wikipedia.org/wiki/Apdex). When operating on buckets,
remember that the histogram is
[cumulative](https://en.wikipedia.org/wiki/Histogram#Cumulative_histogram). See
[histograms and summaries](/docs/practices/histograms) for details of histogram
usage and differences to [summaries](#summary).

NOTE: Beginning with Prometheus v2.40, there is experimental support for native
histograms. A native histogram requires only one time series, which includes a
dynamic number of buckets in addition to the sum and count of
observations. Native histograms allow much higher resolution at a fraction of
the cost. Detailed documentation will follow once native histograms are closer
to becoming a stable feature.

Client library usage documentation for histograms:

   * [Go](http://godoc.org/github.com/prometheus/client_golang/prometheus#Histogram)
   * [Java](https://github.com/prometheus/client_java#histogram)
   * [Python](https://prometheus.github.io/client_python/instrumenting/histogram/)
   * [Ruby](https://github.com/prometheus/client_ruby#histogram)
   * [.Net](https://github.com/prometheus-net/prometheus-net#histogram)
   * [Rust](https://docs.rs/prometheus-client/latest/prometheus_client/metrics/histogram/index.html)

## Summary

Similar to a _histogram_, a _summary_ samples observations (usually things like
request durations and response sizes). While it also provides a total count of
observations and a sum of all observed values, it calculates configurable
quantiles over a sliding time window.

A summary with a base metric name of `<basename>` exposes multiple time series
during a scrape:

  * streaming **φ-quantiles** (0 ≤ φ ≤ 1) of observed events, exposed as `<basename>{quantile="<φ>"}`
  * the **total sum** of all observed values, exposed as `<basename>_sum`
  * the **count** of events that have been observed, exposed as `<basename>_count`

See [histograms and summaries](/docs/practices/histograms) for
detailed explanations of φ-quantiles, summary usage, and differences
to [histograms](#histogram).

Client library usage documentation for summaries:

   * [Go](http://godoc.org/github.com/prometheus/client_golang/prometheus#Summary)
   * [Java](https://github.com/prometheus/client_java#summary)
   * [Python](https://prometheus.github.io/client_python/instrumenting/summary/)
   * [Ruby](https://github.com/prometheus/client_ruby#summary)
   * [.Net](https://github.com/prometheus-net/prometheus-net#summary)

title  Metric types sort rank  2        Metric types  The Prometheus client libraries offer four core metric types  These are currently only differentiated in the client libraries  to enable APIs tailored to the usage of the specific types  and in the wire protocol  The Prometheus server does not yet make use of the type information and flattens all data into untyped time series  This may change in the future      Counter  A  counter  is a cumulative metric that represents a single  monotonically  increasing counter  https   en wikipedia org wiki Monotonic function  whose value can only increase or be reset to zero on restart  For example  you can use a counter to represent the number of requests served  tasks completed  or errors   Do not use a counter to expose a value that can decrease  For example  do not use a counter for the number of currently running processes  instead use a gauge   Client library usage documentation for counters         Go  http   godoc org github com prometheus client golang prometheus Counter        Java  https   github com prometheus client java counter        Python  https   prometheus github io client python instrumenting counter         Ruby  https   github com prometheus client ruby counter         Net  https   github com prometheus net prometheus net counters        Rust  https   docs rs prometheus client latest prometheus client metrics counter index html      Gauge  A  gauge  is a metric that represents a single numerical value that can arbitrarily go up and down   Gauges are typically used for measured values like temperatures or current memory usage  but also  counts  that can go up and down  like the number of concurrent requests   Client library usage documentation for gauges         Go  http   godoc org github com prometheus client golang prometheus Gauge        Java  https   github com prometheus client java gauge        Python  https   prometheus github io client python instrumenting gauge         Ruby  https   github com prometheus client ruby gauge         Net  https   github com prometheus net prometheus net gauges        Rust  https   docs rs prometheus client latest prometheus client metrics gauge index html      Histogram  A  histogram  samples observations  usually things like request durations or response sizes  and counts them in configurable buckets  It also provides a sum of all observed values   A histogram with a base metric name of   basename   exposes multiple time series during a scrape       cumulative counters for the observation buckets  exposed as   basename  bucket le   upper inclusive bound         the   total sum   of all observed values  exposed as   basename  sum      the   count   of events that have been observed  exposed as   basename  count   identical to   basename  bucket le   Inf    above   Use the   histogram quantile    function   docs prometheus latest querying functions  histogram quantile  to calculate quantiles from histograms or even aggregations of histograms  A histogram is also suitable to calculate an  Apdex score  http   en wikipedia org wiki Apdex   When operating on buckets  remember that the histogram is  cumulative  https   en wikipedia org wiki Histogram Cumulative histogram   See  histograms and summaries   docs practices histograms  for details of histogram usage and differences to  summaries   summary    NOTE  Beginning with Prometheus v2 40  there is experimental support for native histograms  A native histogram requires only one time series  which includes a dynamic number of buckets in addition to the sum and count of observations  Native histograms allow much higher resolution at a fraction of the cost  Detailed documentation will follow once native histograms are closer to becoming a stable feature   Client library usage documentation for histograms         Go  http   godoc org github com prometheus client golang prometheus Histogram        Java  https   github com prometheus client java histogram        Python  https   prometheus github io client python instrumenting histogram         Ruby  https   github com prometheus client ruby histogram         Net  https   github com prometheus net prometheus net histogram        Rust  https   docs rs prometheus client latest prometheus client metrics histogram index html      Summary  Similar to a  histogram   a  summary  samples observations  usually things like request durations and response sizes   While it also provides a total count of observations and a sum of all observed values  it calculates configurable quantiles over a sliding time window   A summary with a base metric name of   basename   exposes multiple time series during a scrape       streaming     quantiles    0       1  of observed events  exposed as   basename  quantile             the   total sum   of all observed values  exposed as   basename  sum      the   count   of events that have been observed  exposed as   basename  count   See  histograms and summaries   docs practices histograms  for detailed explanations of   quantiles  summary usage  and differences to  histograms   histogram    Client library usage documentation for summaries         Go  http   godoc org github com prometheus client golang prometheus Summary        Java  https   github com prometheus client java summary        Python  https   prometheus github io client python instrumenting summary         Ruby  https   github com prometheus client ruby summary         Net  https   github com prometheus net prometheus net summary


prometheus Gauge Prometheus supports four types of metrics which are sortrank 2 title Understanding metric types Counter Types of metrics

Answer:
---
title: Understanding metric types
sort_rank: 2
---

# Types of metrics.

Prometheus supports four types of metrics, which are
    - Counter
    - Gauge
    - Histogram
    - Summary

## Counter

Counter is a metric value that can only increase or reset i.e. the value cannot reduce than the previous value. It can be used for metrics like the number of requests, no of errors, etc.

Type the below query in the query bar and click execute.

go\_gc\_duration\_seconds\_count

[![Counter](/assets/tutorial/counter_example.png)](/assets/tutorial/counter_example.png)

The rate() function in PromQL takes the history of metrics over a time frame and calculates how fast the value is increasing per second. Rate is applicable on counter values only.

 rate(go\_gc\_duration\_seconds\_count[5m])
[![Rate Counter](/assets/tutorial/rate_example.png)](/assets/tutorial/rate_example.png)

## Gauge

Gauge is a number which can either go up or down. It can be used for metrics like the number of pods in a cluster, the number of events in a queue, etc.

 go\_memstats\_heap\_alloc\_bytes
[![Gauge](/assets/tutorial/gauge_example.png)](/assets/tutorial/gauge_example.png)

PromQL functions like `max_over_time`, `min_over_time` and `avg_over_time` can be used on gauge metrics

## Histogram

Histogram is a more complex metric type when compared to the previous two. Histogram can be used for any calculated value which is counted based on bucket values. Bucket boundaries can be configured by the developer. A common example would be the time it takes to reply to a request, called latency.

Example: Let's assume we want to observe the time taken to process API requests. Instead of storing the request time for each request, histograms allow us to store them in buckets. We define buckets for time taken, for example `lower or equal 0.3`, `le 0.5`, `le 0.7`, `le 1`, and `le 1.2`. So these are our buckets and once the time taken for a request is calculated it is added to the count of all the buckets whose bucket boundaries are higher than the measured value.

Let's say Request 1 for endpoint “/ping” takes 0.25 s. The count values for the buckets will be.

> /ping

| Bucket    | Count |
| --------- | ----- |
| 0 - 0.3   | 1     |
| 0 - 0.5   | 1     |
| 0 - 0.7   | 1     |
| 0 - 1     | 1     |
| 0 - 1.2   | 1     |
| 0 - +Inf  | 1     |

Note: +Inf bucket is added by default.

(Since the histogram is a cumulative frequency 1 is added to all the buckets that are greater than the value)

Request 2 for endpoint “/ping” takes 0.4s The count values for the buckets will be this.

> /ping

| Bucket    | Count |
| --------- | ----- |
| 0 - 0.3   | 1     |
| 0 - 0.5   | 2     |
| 0 - 0.7   | 2     |
| 0 - 1     | 2     |
| 0 - 1.2   | 2     |
| 0 - +Inf  | 2     |

Since 0.4 is below 0.5, all buckets up to that boundary increase their counts.

Let's explore a histogram metric from the Prometheus UI and apply a few functions.

prometheus\_http\_request\_duration\_seconds\_bucket{handler="/graph"}

[![Histogram](/assets/tutorial/histogram_example.png)](/assets/tutorial/histogram_example.png)

`histogram_quantile()` function can be used to calculate quantiles from a histogram

histogram\_quantile(0.9,prometheus\_http\_request\_duration\_seconds\_bucket{handler="/graph"})

[![Histogram Quantile](/assets/tutorial/histogram_quantile_example.png)](/assets/tutorial/histogram_quantile_example.png)

The graph shows that the 90th percentile is 0.09, To find the histogram_quantile over the last 5m you can use the rate() and time frame

histogram_quantile(0.9, rate(prometheus\_http\_request\_duration\_seconds\_bucket{handler="/graph"}[5m]))

[![Histogram Quantile Rate](/assets/tutorial/histogram_rate_example.png)](/assets/tutorial/histogram_rate_example.png)

## Summary

Summaries also measure events and are an alternative to histograms. They are cheaper but lose more data. They are calculated on the application level hence aggregation of metrics from multiple instances of the same process is not possible. They are used when the buckets of a metric are not known beforehand, but it is highly recommended to use histograms over summaries whenever possible.

In this tutorial, we covered the types of metrics in detail and a few PromQL operations like rate, histogram_quantile, etc.

title  Understanding metric types   sort rank  2        Types of metrics   Prometheus supports four types of metrics  which are       Counter       Gauge       Histogram       Summary      Counter  Counter is a metric value that can only increase or reset i e  the value cannot reduce than the previous value  It can be used for metrics like the number of requests  no of errors  etc   Type the below query in the query bar and click execute    code go  gc  duration  seconds  count  code       Counter   assets tutorial counter example png    assets tutorial counter example png   The rate   function in PromQL takes the history of metrics over a time frame and calculates how fast the value is increasing per second  Rate is applicable on counter values only    code  rate go  gc  duration  seconds  count 5m    code     Rate Counter   assets tutorial rate example png    assets tutorial rate example png      Gauge  Gauge is a number which can either go up or down  It can be used for metrics like the number of pods in a cluster  the number of events in a queue  etc    code  go  memstats  heap  alloc  bytes  code     Gauge   assets tutorial gauge example png    assets tutorial gauge example png   PromQL functions like  max over time    min over time  and  avg over time  can be used on gauge metrics     Histogram  Histogram is a more complex metric type when compared to the previous two  Histogram can be used for any calculated value which is counted based on bucket values  Bucket boundaries can be configured by the developer  A common example would be the time it takes to reply to a request  called latency   Example  Let s assume we want to observe the time taken to process API requests  Instead of storing the request time for each request  histograms allow us to store them in buckets  We define buckets for time taken  for example  lower or equal 0 3    le 0 5    le 0 7    le 1   and  le 1 2   So these are our buckets and once the time taken for a request is calculated it is added to the count of all the buckets whose bucket boundaries are higher than the measured value   Let s say Request 1 for endpoint   ping  takes 0 25 s  The count values for the buckets will be      ping    Bucket      Count                           0   0 3     1         0   0 5     1         0   0 7     1         0   1       1         0   1 2     1         0    Inf    1        Note   Inf bucket is added by default    Since the histogram is a cumulative frequency 1 is added to all the buckets that are greater than the value   Request 2 for endpoint   ping  takes 0 4s The count values for the buckets will be this      ping    Bucket      Count                           0   0 3     1         0   0 5     2         0   0 7     2         0   1       2         0   1 2     2         0    Inf    2        Since 0 4 is below 0 5  all buckets up to that boundary increase their counts   Let s explore a histogram metric from the Prometheus UI and apply a few functions    code prometheus  http  request  duration  seconds  bucket handler   graph    code      Histogram   assets tutorial histogram example png    assets tutorial histogram example png    histogram quantile    function can be used to calculate quantiles from a histogram   code histogram  quantile 0 9 prometheus  http  request  duration  seconds  bucket handler   graph     code      Histogram Quantile   assets tutorial histogram quantile example png    assets tutorial histogram quantile example png   The graph shows that the 90th percentile is 0 09  To find the histogram quantile over the last 5m you can use the rate   and time frame   code histogram quantile 0 9  rate prometheus  http  request  duration  seconds  bucket handler   graph   5m     code      Histogram Quantile Rate   assets tutorial histogram rate example png    assets tutorial histogram rate example png       Summary  Summaries also measure events and are an alternative to histograms  They are cheaper but lose more data  They are calculated on the application level hence aggregation of metrics from multiple instances of the same process is not possible  They are used when the buckets of a metric are not known beforehand  but it is highly recommended to use histograms over summaries whenever possible   In this tutorial  we covered the types of metrics in detail and a few PromQL operations like rate  histogram quantile  etc


prometheus What is Prometheus sortrank 1 Prometheus is a system monitoring and alerting system It was opensourced by SoundCloud in 2012 and is the second project both to join and to graduate within Cloud Native Computing Foundation after Kubernetes Prometheus stores all metrics data as time series i e metrics information is stored along with the timestamp at which it was recorded optional key value pairs called as labels can also be stored along with metrics title Getting Started with Prometheus What are metrics and why is it important

Answer:
---
title: Getting Started with Prometheus
sort_rank: 1
---

# What is Prometheus ?

Prometheus is a system monitoring and alerting system. It was opensourced by SoundCloud in 2012 and is the second project both to join and to graduate within Cloud Native Computing Foundation after Kubernetes. Prometheus stores all metrics data as time series, i.e metrics information is stored along with the timestamp at which it was recorded, optional key-value pairs called as labels can also be stored along with metrics.
# What are metrics and why is it important?

Metrics in layperson terms is a standard for measurement. What we want to measure depends from application to application. For a web server it can be request times, for a database it can be CPU usage or number of active connections etc.

Metrics play an important role in understanding why your application is working in a certain way. If you run a web application and someone comes up to you and says that the application is slow, you will need some information to find out what is happening with your application. For example the application can become slow when the number of requests are high. If you have the request count metric you can spot the reason and increase the number of servers to handle the heavy load. Whenever you are defining the metrics for your application you must put on your detective hat and ask this question **what all information will be important for me to debug if any issue occurs in my application?**

# Basic Architecture of Prometheus

The basic components of a Prometheus setup are:

- Prometheus Server (the server which scrapes and stores the metrics data).
- Targets to be scraped, for example an instrumented application that exposes its metrics, or an exporter that exposes metrics of another application.
- Alertmanager to raise alerts based on preset rules.

(Note: Apart from this Prometheus has push_gateway which is not covered here).

[![Architecture](/assets/tutorial/architecture.png)](/assets/tutorial/architecture.png)

Let's consider a web server as an example application and we want to extract a certain metric like the number of API calls processed by the web server. So we add certain instrumentation code using the Prometheus client library and expose the metrics information. Now that our web server exposes its metrics we can configure Prometheus to scrape it. Now Prometheus is configured to fetch the metrics from the web server which is listening on xyz IP address port 7500 at a specific time interval, say, every minute.

At 11:00:00 when I make the server public for consumption, the application calculates the request count and exposes it, Prometheus simultaneously scrapes the count metric and stores the value as 0.

By 11:01:00 one request is processed. The instrumentation logic in the server increments the count to 1. When Prometheus scrapes the metric the value of count is 1 now.

By 11:02:00 two more requests are processed and the request count is 1+2 = 3 now. Similarly metrics are scraped and stored.

The user can control the frequency at which metrics are scraped by Prometheus.

| Time Stamp | Request Count (metric) |
| ---------- | ---------------------- |
| 11:00:00   | 0                      |
| 11:01:00   | 1                      |
| 11:02:00   | 3                      |

(Note: This table is just a representation for understanding purposes. Prometheus doesn’t store the values in this exact format)

Prometheus also has an API which allows to query metrics which have been stored by scraping. This API is used to query the metrics, create dashboards/charts on it etc. PromQL is used to query these metrics.

A simple Line chart created on the Request Count metric will look like this

[![Graph](/assets/tutorial/sample_graph.png)](/assets/tutorial/sample_graph.png)

One can scrape multiple useful metrics to understand what is happening in the application and create multiple charts on them. Group the charts into a dashboard and use it to get an overview of the application.

# Show me how it is done

Let’s get our hands dirty and setup Prometheus. Prometheus is written using [Go](https://golang.org/) and all you need is the binary compiled for your operating system. Download the binary corresponding to your operating system from [here](https://prometheus.io/download/) and add the binary to your path.

Prometheus exposes its own metrics which can be consumed by itself or another Prometheus server.

Now that we have Prometheus installed, the next step is to run it. All that we need is just the binary and a configuration file. Prometheus uses yaml files for configuration.

```yaml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]
```

In the above configuration file we have mentioned the `scrape_interval`, i.e how frequently we want Prometheus to scrape the metrics. We have added `scrape_configs` which has a name and target to scrape the metrics from. Prometheus by default listens on port 9090. So add it to targets.

> prometheus --config.file=prometheus.yml

Now we have Prometheus up and running and scraping its own metrics every 15s. Prometheus has standard exporters available to export metrics. Next we will run a node exporter which is an exporter for machine metrics and scrape the same using Prometheus. ([Download node metrics exporter.](https://prometheus.io/download/#node_exporter))

Run the node exporter in a terminal.

./node_exporter

[![Node exporter](/assets/tutorial/node_exporter.png)](/assets/tutorial/node_exporter.png)

Next, add node exporter to the list of scrape_configs:

```yaml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: node_exporter
    static_configs:
      - targets: ["localhost:9100"]
```

In this tutorial we discussed what are metrics and why they are important, basic architecture of Prometheus and how to
run Prometheus.

title  Getting Started with Prometheus sort rank  1        What is Prometheus    Prometheus is a system monitoring and alerting system  It was opensourced by SoundCloud in 2012 and is the second project both to join and to graduate within Cloud Native Computing Foundation after Kubernetes  Prometheus stores all metrics data as time series  i e metrics information is stored along with the timestamp at which it was recorded  optional key value pairs called as labels can also be stored along with metrics    What are metrics and why is it important   Metrics in layperson terms is a standard for measurement  What we want to measure depends from application to application  For a web server it can be request times  for a database it can be CPU usage or number of active connections etc   Metrics play an important role in understanding why your application is working in a certain way  If you run a web application and someone comes up to you and says that the application is slow  you will need some information to find out what is happening with your application  For example the application can become slow when the number of requests are high  If you have the request count metric you can spot the reason and increase the number of servers to handle the heavy load  Whenever you are defining the metrics for your application you must put on your detective hat and ask this question   what all information will be important for me to debug if any issue occurs in my application       Basic Architecture of Prometheus  The basic components of a Prometheus setup are     Prometheus Server  the server which scrapes and stores the metrics data     Targets to be scraped  for example an instrumented application that exposes its metrics  or an exporter that exposes metrics of another application    Alertmanager to raise alerts based on preset rules    Note  Apart from this Prometheus has push gateway which is not covered here       Architecture   assets tutorial architecture png    assets tutorial architecture png   Let s consider a web server as an example application and we want to extract a certain metric like the number of API calls processed by the web server  So we add certain instrumentation code using the Prometheus client library and expose the metrics information  Now that our web server exposes its metrics we can configure Prometheus to scrape it  Now Prometheus is configured to fetch the metrics from the web server which is listening on xyz IP address port 7500 at a specific time interval  say  every minute   At 11 00 00 when I make the server public for consumption  the application calculates the request count and exposes it  Prometheus simultaneously scrapes the count metric and stores the value as 0   By 11 01 00 one request is processed  The instrumentation logic in the server increments the count to 1  When Prometheus scrapes the metric the value of count is 1 now   By 11 02 00 two more requests are processed and the request count is 1 2   3 now  Similarly metrics are scraped and stored   The user can control the frequency at which metrics are scraped by Prometheus     Time Stamp   Request Count  metric                                              11 00 00     0                          11 01 00     1                          11 02 00     3                          Note  This table is just a representation for understanding purposes  Prometheus doesn t store the values in this exact format   Prometheus also has an API which allows to query metrics which have been stored by scraping  This API is used to query the metrics  create dashboards charts on it etc  PromQL is used to query these metrics   A simple Line chart created on the Request Count metric will look like this     Graph   assets tutorial sample graph png    assets tutorial sample graph png   One can scrape multiple useful metrics to understand what is happening in the application and create multiple charts on them  Group the charts into a dashboard and use it to get an overview of the application     Show me how it is done  Let s get our hands dirty and setup Prometheus  Prometheus is written using  Go  https   golang org   and all you need is the binary compiled for your operating system  Download the binary corresponding to your operating system from  here  https   prometheus io download   and add the binary to your path   Prometheus exposes its own metrics which can be consumed by itself or another Prometheus server   Now that we have Prometheus installed  the next step is to run it  All that we need is just the binary and a configuration file  Prometheus uses yaml files for configuration       yaml global    scrape interval  15s  scrape configs      job name  prometheus     static configs          targets    localhost 9090        In the above configuration file we have mentioned the  scrape interval   i e how frequently we want Prometheus to scrape the metrics  We have added  scrape configs  which has a name and target to scrape the metrics from  Prometheus by default listens on port 9090  So add it to targets     prometheus   config file prometheus yml   iframe width  560  height  315  src  https   www youtube com embed ioa0eISf1Q0  frameborder  0  allowfullscreen   iframe   Now we have Prometheus up and running and scraping its own metrics every 15s  Prometheus has standard exporters available to export metrics  Next we will run a node exporter which is an exporter for machine metrics and scrape the same using Prometheus    Download node metrics exporter   https   prometheus io download  node exporter    Run the node exporter in a terminal    code   node exporter  code      Node exporter   assets tutorial node exporter png    assets tutorial node exporter png   Next  add node exporter to the list of scrape configs      yaml global    scrape interval  15s  scrape configs      job name  prometheus     static configs          targets    localhost 9090       job name  node exporter     static configs          targets    localhost 9100         iframe width  560  height  315  src  https   www youtube com embed hM5bp53C7Y8  frameborder  0  allowfullscreen   iframe   In this tutorial we discussed what are metrics and why they are important  basic architecture of Prometheus and how to run Prometheus


prometheus Here we have a simple HTTP server with endpoint which returns as response title Instrumenting HTTP server written in Go In this tutorial we will create a simple Go HTTP server and instrumentation it by adding a counter sortrank 3 metric to keep count of the total number of requests processed by the server

Answer:
---
title: Instrumenting HTTP server written in Go
sort_rank: 3
---

In this tutorial we will create a simple Go HTTP server and instrumentation it by adding a counter
metric to keep count of the total number of requests processed by the server.

Here we have a simple HTTP server with `/ping` endpoint which returns `pong` as response.

```go
package main

import (
   "fmt"
   "net/http"
)

func ping(w http.ResponseWriter, req *http.Request){
   fmt.Fprintf(w,"pong")
}

func main() {
   http.HandleFunc("/ping",ping)

   http.ListenAndServe(":8090", nil)
}
```

Compile and run the server

```bash
go build server.go
./server
```

Now open `http://localhost:8090/ping` in your browser and you must see `pong`.

[![Server](/assets/tutorial/server.png)](/assets/tutorial/server.png)

Now lets add a metric to the server which will instrument the number of requests made to the ping endpoint,the counter metric type is suitable for this as we know the request count doesn’t go down and only increases.

Create a Prometheus counter

```go
var pingCounter = prometheus.NewCounter(
   prometheus.CounterOpts{
       Name: "ping_request_count",
       Help: "No of request handled by Ping handler",
   },
)
```

Next lets update the ping Handler to increase the count of the counter using `pingCounter.Inc()`.

```go
func ping(w http.ResponseWriter, req *http.Request) {
   pingCounter.Inc()
   fmt.Fprintf(w, "pong")
}
```

Then register the counter to the Default Register and expose the metrics.

```go
func main() {
   prometheus.MustRegister(pingCounter)
   http.HandleFunc("/ping", ping)
   http.Handle("/metrics", promhttp.Handler())
   http.ListenAndServe(":8090", nil)
}
```

The `prometheus.MustRegister` function registers the pingCounter to the default Register.
To expose the metrics the Go Prometheus client library provides the promhttp package.
`promhttp.Handler()` provides a `http.Handler` which exposes the metrics registered in the Default Register.

The sample code depends on the

```go
package main

import (
   "fmt"
   "net/http"

   "github.com/prometheus/client_golang/prometheus"
   "github.com/prometheus/client_golang/prometheus/promhttp"
)

var pingCounter = prometheus.NewCounter(
   prometheus.CounterOpts{
       Name: "ping_request_count",
       Help: "No of request handled by Ping handler",
   },
)

func ping(w http.ResponseWriter, req *http.Request) {
   pingCounter.Inc()
   fmt.Fprintf(w, "pong")
}

func main() {
   prometheus.MustRegister(pingCounter)

   http.HandleFunc("/ping", ping)
   http.Handle("/metrics", promhttp.Handler())
   http.ListenAndServe(":8090", nil)
}
```

Run the example

```sh
go mod init prom_example
go mod tidy
go run server.go
```

Now hit the localhost:8090/ping endpoint a couple of times and sending a request to localhost:8090 will provide the metrics.

[![Ping Metric](/assets/tutorial/ping_metric.png)](/assets/tutorial/ping_metric.png)

Here the `ping_request_count` shows that `/ping` endpoint was called 3 times.

The Default Register comes with a collector for go runtime metrics and that is why we see other metrics like `go_threads`, `go_goroutines` etc.

We have built our first metric exporter. Let’s update our Prometheus config to scrape the metrics from our server.

```yaml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: simple_server
    static_configs:
      - targets: ["localhost:8090"]
```

prometheus --config.file=prometheus.yml

title  Instrumenting HTTP server written in Go   sort rank  3      In this tutorial we will create a simple Go HTTP server and instrumentation it by adding a counter metric to keep count of the total number of requests processed by the server   Here we have a simple HTTP server with   ping  endpoint which returns  pong  as response      go package main  import       fmt      net http     func ping w http ResponseWriter  req  http Request      fmt Fprintf w  pong      func main        http HandleFunc   ping  ping      http ListenAndServe   8090   nil         Compile and run the server     bash go build server go   server      Now open  http   localhost 8090 ping  in your browser and you must see  pong       Server   assets tutorial server png    assets tutorial server png    Now lets add a metric to the server which will instrument the number of requests made to the ping endpoint the counter metric type is suitable for this as we know the request count doesn t go down and only increases   Create a Prometheus counter     go var pingCounter   prometheus NewCounter     prometheus CounterOpts         Name   ping request count          Help   No of request handled by Ping handler                Next lets update the ping Handler to increase the count of the counter using  pingCounter Inc         go func ping w http ResponseWriter  req  http Request       pingCounter Inc      fmt Fprintf w   pong          Then register the counter to the Default Register and expose the metrics      go func main        prometheus MustRegister pingCounter     http HandleFunc   ping   ping     http Handle   metrics   promhttp Handler       http ListenAndServe   8090   nil         The  prometheus MustRegister  function registers the pingCounter to the default Register  To expose the metrics the Go Prometheus client library provides the promhttp package   promhttp Handler    provides a  http Handler  which exposes the metrics registered in the Default Register   The sample code depends on the       go package main  import       fmt      net http       github com prometheus client golang prometheus      github com prometheus client golang prometheus promhttp     var pingCounter   prometheus NewCounter     prometheus CounterOpts         Name   ping request count          Help   No of request handled by Ping handler            func ping w http ResponseWriter  req  http Request       pingCounter Inc      fmt Fprintf w   pong      func main        prometheus MustRegister pingCounter      http HandleFunc   ping   ping     http Handle   metrics   promhttp Handler       http ListenAndServe   8090   nil         Run the example     sh go mod init prom example go mod tidy go run server go      Now hit the localhost 8090 ping endpoint a couple of times and sending a request to localhost 8090 will provide the metrics      Ping Metric   assets tutorial ping metric png    assets tutorial ping metric png   Here the  ping request count  shows that   ping  endpoint was called 3 times   The Default Register comes with a collector for go runtime metrics and that is why we see other metrics like  go threads    go goroutines  etc   We have built our first metric exporter  Let s update our Prometheus config to scrape the metrics from our server      yaml global    scrape interval  15s  scrape configs      job name  prometheus     static configs          targets    localhost 9090       job name  simple server     static configs          targets    localhost 8090         code prometheus   config file prometheus yml  code    iframe width  560  height  315  src  https   www youtube com embed yQIWgZoiW0o  frameborder  0  allowfullscreen   iframe


prometheus title Remote write tuning Prometheus implements sane defaults for remote write but many users have different requirements and would like to optimize their remote settings Remote write tuning sortrank 8

Answer:
---
title: Remote write tuning
sort_rank: 8
---

# Remote write tuning

Prometheus implements sane defaults for remote write, but many users have
different requirements and would like to optimize their remote settings.

This page describes the tuning parameters available via the [remote write
configuration.](/docs/prometheus/latest/configuration/configuration/#remote_write)

## Remote write characteristics

Each remote write destination starts a queue which reads from the write-ahead
log (WAL), writes the samples into an in memory queue owned by a shard, which
then sends a request to the configured endpoint. The flow of data looks like:

```
      |-->  queue (shard_1)   --> remote endpoint
WAL --|-->  queue (shard_...) --> remote endpoint
      |-->  queue (shard_n)   --> remote endpoint
```

When one shard backs up and fills its queue, Prometheus will block reading from
the WAL into any shards. Failures will be retried without loss of data unless
the remote endpoint remains down for more than 2 hours. After 2 hours, the WAL
will be compacted and data that has not been sent will be lost.

During operation, Prometheus will continuously calculate the optimal number of
shards to use based on the incoming sample rate, number of outstanding samples
not sent, and time taken to send each sample.

### Resource usage

Using remote write increases the memory footprint of Prometheus. Most users
report ~25% increased memory usage, but that number is dependent on the shape
of the data. For each series in the WAL, the remote write code caches a mapping
of series ID to label values, causing large amounts of series churn to
significantly increase memory usage.

In addition to the series cache, each shard and its queue increases memory
usage. Shard memory is proportional to the `number of shards * (capacity +
max_samples_per_send)`. When tuning, consider reducing `max_shards` alongside
increases to `capacity` and `max_samples_per_send` to avoid inadvertently
running out of memory. The default values for `capacity: 10000` and
`max_samples_per_send: 2000` will constrain shard memory usage to less than 2
MB per shard.

Remote write will also increase CPU and network usage. However, for the same
reasons as above, it is difficult to predict by how much. It is generally a
good practice to check for CPU and network saturation if your Prometheus server
falls behind sending samples via remote write
(`prometheus_remote_storage_samples_pending`).

## Parameters

All the relevant parameters are found under the `queue_config` section of the
remote write configuration.

### `capacity`

Capacity controls how many samples are queued in memory per shard before
blocking reading from the WAL. Once the WAL is blocked, samples cannot be
appended to any shards and all throughput will cease.

Capacity should be high enough to avoid blocking other shards in most
cases, but too much capacity can cause excess memory consumption and longer
times to clear queues during resharding. It is recommended to set capacity
to 3-10 times `max_samples_per_send`.

### `max_shards`

Max shards configures the maximum number of shards, or parallelism, Prometheus
will use for each remote write queue. Prometheus will try not to use too many
shards, but if the queue falls behind the remote write component will increase
the number of shards up to max shards to increase throughput. Unless remote
writing to a very slow endpoint, it is unlikely that `max_shards` should be
increased beyond the default. However, it may be necessary to reduce max shards
if there is potential to overwhelm the remote endpoint, or to reduce memory
usage when data is backed up.

### `min_shards`

Min shards configures the minimum number of shards used by Prometheus, and is
the number of shards used when remote write starts. If remote write falls
behind, Prometheus will automatically scale up the number of shards so most
users do not have to adjust this parameter. However, increasing min shards will
allow Prometheus to avoid falling behind at the beginning while calculating the
required number of shards.

### `max_samples_per_send`

Max samples per send can be adjusted depending on the backend in use. Many
systems work very well by sending more samples per batch without a significant
increase in latency. Other backends will have issues if trying to send a large
number of samples in each request. The default value is small enough to work for
most systems.

### `batch_send_deadline`

Batch send deadline sets the maximum amount of time between sends for a single
shard. Even if the queued shards has not reached `max_samples_per_send`, a
request will be sent. Batch send deadline can be increased for low volume
systems that are not latency sensitive in order to increase request efficiency.

### `min_backoff`

Min backoff controls the minimum amount of time to wait before retrying a failed
request. Increasing the backoff spreads out requests when a remote endpoint
comes back online. The backoff interval is doubled for each failed requests up
to `max_backoff`.

### `max_backoff`

Max backoff controls the maximum amount of time to wait before retrying a failed
request.

title  Remote write tuning sort rank  8        Remote write tuning  Prometheus implements sane defaults for remote write  but many users have different requirements and would like to optimize their remote settings   This page describes the tuning parameters available via the  remote write configuration    docs prometheus latest configuration configuration  remote write      Remote write characteristics  Each remote write destination starts a queue which reads from the write ahead log  WAL   writes the samples into an in memory queue owned by a shard  which then sends a request to the configured endpoint  The flow of data looks like                   queue  shard 1        remote endpoint WAL         queue  shard          remote endpoint             queue  shard n        remote endpoint      When one shard backs up and fills its queue  Prometheus will block reading from the WAL into any shards  Failures will be retried without loss of data unless the remote endpoint remains down for more than 2 hours  After 2 hours  the WAL will be compacted and data that has not been sent will be lost   During operation  Prometheus will continuously calculate the optimal number of shards to use based on the incoming sample rate  number of outstanding samples not sent  and time taken to send each sample       Resource usage  Using remote write increases the memory footprint of Prometheus  Most users report  25  increased memory usage  but that number is dependent on the shape of the data  For each series in the WAL  the remote write code caches a mapping of series ID to label values  causing large amounts of series churn to significantly increase memory usage   In addition to the series cache  each shard and its queue increases memory usage  Shard memory is proportional to the  number of shards    capacity   max samples per send    When tuning  consider reducing  max shards  alongside increases to  capacity  and  max samples per send  to avoid inadvertently running out of memory  The default values for  capacity  10000  and  max samples per send  2000  will constrain shard memory usage to less than 2 MB per shard   Remote write will also increase CPU and network usage  However  for the same reasons as above  it is difficult to predict by how much  It is generally a good practice to check for CPU and network saturation if your Prometheus server falls behind sending samples via remote write   prometheus remote storage samples pending        Parameters  All the relevant parameters are found under the  queue config  section of the remote write configuration        capacity   Capacity controls how many samples are queued in memory per shard before blocking reading from the WAL  Once the WAL is blocked  samples cannot be appended to any shards and all throughput will cease   Capacity should be high enough to avoid blocking other shards in most cases  but too much capacity can cause excess memory consumption and longer times to clear queues during resharding  It is recommended to set capacity to 3 10 times  max samples per send         max shards   Max shards configures the maximum number of shards  or parallelism  Prometheus will use for each remote write queue  Prometheus will try not to use too many shards  but if the queue falls behind the remote write component will increase the number of shards up to max shards to increase throughput  Unless remote writing to a very slow endpoint  it is unlikely that  max shards  should be increased beyond the default  However  it may be necessary to reduce max shards if there is potential to overwhelm the remote endpoint  or to reduce memory usage when data is backed up        min shards   Min shards configures the minimum number of shards used by Prometheus  and is the number of shards used when remote write starts  If remote write falls behind  Prometheus will automatically scale up the number of shards so most users do not have to adjust this parameter  However  increasing min shards will allow Prometheus to avoid falling behind at the beginning while calculating the required number of shards        max samples per send   Max samples per send can be adjusted depending on the backend in use  Many systems work very well by sending more samples per batch without a significant increase in latency  Other backends will have issues if trying to send a large number of samples in each request  The default value is small enough to work for most systems        batch send deadline   Batch send deadline sets the maximum amount of time between sends for a single shard  Even if the queued shards has not reached  max samples per send   a request will be sent  Batch send deadline can be increased for low volume systems that are not latency sensitive in order to increase request efficiency        min backoff   Min backoff controls the minimum amount of time to wait before retrying a failed request  Increasing the backoff spreads out requests when a remote endpoint comes back online  The backoff interval is doubled for each failed requests up to  max backoff         max backoff   Max backoff controls the maximum amount of time to wait before retrying a failed request