vault Recommended patterns Help keep your Vault environments operating effectively by implementing the following best practice so you avoid common anti patterns page title Recommended patterns layout docs Follow these recommended patterns to effectively operate Vault

Answer:
---
layout: docs
page_title: Recommended patterns
description: Follow these recommended patterns to effectively operate Vault.
---

# Recommended patterns

Help keep your Vault environments operating effectively by implementing the following best practice so you avoid common anti-patterns.

| Description  	| Applicable Vault edition 	|
|---	|---	|
| [Adjust the default lease time](#adjust-the-default-lease-time) 	|  All 	|
| [Use identity entities for accurate client count](#use-identity-entities-for-accurate-client-count)  	| Enterprise, HCP	|
| [Increase IOPS](#increase-iops)  	| Enterprise, Community 	|
| [Enable disaster recovery](#enable-disaster-recovery)  	| Enterprise 	|
| [Test disaster recovery](#test-disaster-recovery)  	| Enterprise 	|
| [Improve upgrade cadence](#improve-upgrade-cadence)  	| Enterprise, Community |
| [Test before upgrades](#test-before-upgrades)  	| Enterprise, Community 	|
| [Rotate audit device logs](#rotate-audit-device-logs)  	| Enterprise, Community	|
| [Monitor metrics](#monitor-metrics)   	| Enterprise, Community 	|
| [Establish usage baseline](#establish-usage-baseline)  	| Enterprise, Community	|
| [Minimize root token use](#minimize-root-token-use)  	| All 	|
| [Rekey when necessary](#rekey-when-necessary)  	| All 	|

## Adjust the default lease time

The default lease time in Vault is 32 days or 768 hours. This time allows for some operations, such as re-authentication or renewal.
See [lease](/vault/docs/concepts/lease) documentation for more information.

**Recommended pattern:**

You should tune the lease TTL value for your needs. Vault holds leases in memory until the lease expires.
We recommend keeping TTLs as short as the use case will allow.
- [Auth tune](/vault/docs/commands/auth/tune)
- [Secrets tune](/vault/docs/commands/secrets/tune)

<Note>
Tuning or adjusting TTLs does not retroactively affect tokens that were issued. New tokens must be issued after tuning TTLs.
</Note>

**Anti-pattern issue:**

If you create leases without changing the default time-to-live (TTL), leases will live in Vault until the default lease time is up.
Depending on your infrastructure and available system memory, using the default or long TTL may cause performance issues as Vault stores
leases in memory.

## Use identity entities for accurate client count

Each Vault client may have multiple accounts with the auth methods enabled on the Vault server.

![Entity](/img/vault-entity-waf1.png)

**Recommended pattern:**

Since each token adds to the client count, and each unique authentication issues a token, you should use identity entities to create aliases that connect each login to a single identity.

  - [Client count](/vault/docs/concepts/client-count)
  - [Vault identity concepts](/vault/docs/concepts/identity)
  - [Vault Identity secrets engine](/vault/docs/secrets/identity)
  - [Identity: Entities and groups tutorial](/vault/tutorials/auth-methods/identity)

**Anti-pattern issue:**

When you do not use identity entities, each new client is counted as a separate identity when using another auth method not linked to the user's entity.

## Increase IOPS

IOPS (input/output operations per second) measures performance for Vault cluster members. Vault is bound by the IO limits of the storage backend rather than the compute requirements.

**Recommended pattern:**

Use the HashiCorp reference guidelines for Vault servers' hardware sizing and network considerations.

- [Vault with Integrated storage reference architecture](/vault/tutorials/day-one-raft/raft-reference-architecture#system-requirements)
- [Performance tuning](/vault/tutorials/operations/performance-tuning)
- [Transform secrets engine](/vault/docs/concepts/transform)

<Note>

Depending on the client count, the Transform (Enterprise) and Transit secret engines can be resource-intensive.

</Note>

**Anti-pattern issue:**

Limited IOPS can significantly degrade Vault’s performance.

## Enable disaster recovery

HashiCorp Vault's (HA) highly available [Integrated storage (Raft)](/vault/docs/concepts/integrated-storage)
backend provides intra-cluster data replication across cluster members. Integrated Storage provides Vault with
horizontal scalability and failure tolerance, but it does not provide backup for the entire cluster. Not utilizing
disaster recovery for your production environment will negatively impact your organization's Recovery Point
Objective (RPO) and Recovery Time Objective (RTO).

**Recommended pattern:**

For cluster-wide issues (i.e., network connectivity), Vault Enterprise Disaster Recovery (DR) replication
provides a warm standby cluster containing all primary cluster data. The DR cluster does not service reads
or writes but you can promote it to replace the primary cluster when needed.

- [Disaster recovery replication setup](/vault/tutorials/day-one-raft/disaster-recovery)
- [Disaster recovery (DR) replication](/vault/docs/enterprise/replication#disaster-recovery-dr-replication)
- [DR replication API documentation](/vault/api-docs/system/replication/replication-dr)

We also recommend periodically creating data snapshots to protect against data corruption.

- [Vault data backup standard procedure](/vault/tutorials/standard-procedures/sop-backup)
- [Automated integrated storage snapshots](/vault/docs/enterprise/automated-integrated-storage-snapshots)
- [/sys/storage/raft/snapshot-auto](/vault/api-docs/system/storage/raftautosnapshots)

**Anti-pattern issue:**

If you do not enable disaster recovery and catastrophic failure occurs, your use cases will encounter longer downtime duration and costs associated with not serving Vault clients in your environment.

## Test disaster recovery

Your disaster recovery (DR) solution is a key part of your overall disaster recovery plan.

Designing and configuring your Vault disaster recovery solution is only the first step. You also need to validate the DR solution, as not doing so can negatively impact your organization's Recovery Point Objective (RPO) and Recovery Time Objective (RTO).

**Recommended pattern:**

Vault's Disaster Recovery (DR) replication mode provides a warm standby for
failover if the primary cluster experiences catastrophic failure. You should
periodically test the disaster recovery replication cluster by completing the
failover and failback procedure.

- [Vault disaster recovery replication failover and failback tutorial](/vault/tutorials/enterprise/disaster-recovery-replication-failover)
- [Vault Enterprise replication](/vault/docs/enterprise/replication)
- [Monitoring Vault replication](/vault/tutorials/monitoring/monitor-replication)

You should establish standard operating procedures for restoring a Vault cluster from a snapshot. The restoration methods following a DR situation would be in response to data corruption or sabotage, which Disaster Recovery Replication might be unable to protect against.

- [Standard procedure for restoring a Vault cluster](/vault/tutorials/standard-procedures/sop-restore)

**Anti-pattern issue:**

If you don't test your disaster recovery solution, your key stakeholders will not feel confident they can effectively perform the disaster recovery plan. Testing the DR solution also helps your team to remove uncertainty around recovering the system during an outage.

## Improve upgrade cadence

While it might be easy to upgrade Vault whenever you have capacity, not having a frequent upgrade cadence can impact your Vault performance and security.

**Recommended pattern:**

We recommend upgrading to our latest version of Vault. Subscribe to the releases in [Vault's GitHub repository](https://github.com/hashicorp/vault), and notifications from [HashiCorp Vault discuss](https://discuss.hashicorp.com/c/release-notifications/57), will inform you when we release a new Vault version.

- [Vault upgrade guides](/vault/docs/upgrading)
- [Vault feature deprecation notice and plans](/vault/docs/deprecation)

**Anti-pattern issue:**

When you do not keep a regular upgrade cadence, your Vault environment could be missing key features or improvements.

- Missing patches for bugs or vulnerabilities as documented in the [CHANGELOG](https://github.com/hashicorp/vault/blob/main/CHANGELOG.md).
- New features to improve workflow.
- Must use version-specific rather than the latest documentation.
- Some educational resourcesrequire a specific minimum Vault version.
- Updates may require a stepped approach that uses an intermediate version before installing the latest binary.

## Test before upgrades

We recommend testing Vault in a sandbox environment before deploying to production.

Although it might be faster to upgrade immediately in production, testing will help identify any compatibility issues.

Be aware of the [CHANGELOG](https://github.com/hashicorp/vault/blob/main/CHANGELOG.md) and account for any new features, improvements, known issues and bug fixes in your testing.

**Recommended pattern:**

Test new Vault versions in sandbox environments before upgrading in production and follow our upgrading documentation.

We recommend adding a testing phase to your standard upgrade procedure.

- [Vault upgrade standard procedure](/vault/tutorials/standard-procedures/sop-upgrade)
- [Upgrading Vault](/vault/docs/upgrading)

**Anti-pattern issue:**

Without adequate testing before upgrading in production, you risk compatibility and performance issues.

<Warning>

This could lead to downtime or degradation in your production Vault environment.

</Warning>

## Rotate audit device logs

Audit devices in Vault maintain a detailed log of every client request and server response.

If you allow the logs for audit devices to run perpetually without rotating you may face a blocked audit device if the filesystem storage becomes exhausted.

**Recommended pattern:**

Inspect and rotate audit logs periodically.

- [Blocked audit devices tutorial](/vault/tutorials/monitoring/blocked-audit-devices)
- [blocked audit devices](/vault/docs/audit#blocked-audit-devices)

**Anti-pattern issue:**

Vault will not respond to requests when audit devices are not enabled to record them.

The audit device can exhaust the local storage if the audit device log is not maintained and rotated over time.

## Monitor metrics

Relying solely on Vault operational logs and data in Vault UI will give you a partial picture of the cluster's performance.

**Recommended pattern:**

Continuous monitoring will allow organizations to detect minor problems and promptly resolve them.
Migrating from reactive to proactive monitoring will help to prevent system failures. Vault has multiple outputs
that help monitor the cluster's activity: audit logs, operational logs, and telemetry data. This data can work
with a SIEM (security information and event management) tool for aggregation, inspection, and alerting capabilities.

- [Telemetry](/vault/docs/internals/telemetry#secrets-engines-metric)
- [Telemetry metrics reference](/vault/tutorials/monitoring/telemetry-metrics-reference)

Adding a monitoring solution:
- [Audit device logs and incident response with elasticsearch](/vault/tutorials/monitoring/audit-elastic-incident-response)
- [Monitor telemetry & audit device log data](/vault/tutorials/monitoring/monitor-telemetry-audit-splunk)
- [Monitor telemetry with Prometheus & Grafana](/vault/tutorials/monitoring/monitor-telemetry-grafana-prometheus)

<Note>

  Vault logs to standard output and standard error by default, automatically captured by the systemd journal. You can also instruct Vault to redirect operational log writes to a file.

</Note>

**Anti-pattern issue:**

Having partial insight into cluster activity can leave the business in a reactive state.

## Establish usage baseline

A baseline provides insight into current utilization and thresholds. Telemetry metrics are valuable, especially when monitored over time. You can use telemetry metrics to gather a baseline of cluster activity, while alerts inform you of abnormal activity.

**Recommended pattern:**

Telemetry information can also be streamed directly from Vault to a range of metrics aggregation solutions and
saved for aggregation and inspection.

- [Vault usage metrics](/vault/tutorials/monitoring/usage-metrics)
- [Diagnose server issues](/vault/tutorials/monitoring/diagnose-startup-issues)

**Anti-pattern issue:**

This issue closely relates to the recommended pattern for [monitor metrics](#monitor-metrics).
 Telemetry data is
only held in memory for a short period.

## Minimize root token use

Initializing a Vault server emits an initial root token that gives root-level access across all Vault features.

**Recommended pattern:**

We recommend that you revoke the root token after initializing Vault within your environment. If users require elevated access, create access control list policies that grant proper capabilities on the necessary paths in Vault. If your operations require the root token, keep it for the shortest possible time before revoking it.

- [Generate root tokens tutorial](/vault/tutorials/operations/generate-root)
- [Root tokens](/vault/docs/concepts/tokens#root-tokens)
- [Vault policies](/vault/docs/concepts/policies)

**Anti-pattern issue:**

A root token can perform all actions within Vault and never expire. Unrestricted access can give users higher privileges than necessary to all Vault operations and paths. Sharing and providing access to root tokens poses a security risk.

## Rekey when necessary

Vault distributes unsealed keys to stakeholders. A quorum of keys is needed to unlock Vault based on your initialization settings.

**Recommended pattern:**

Vault supports rekeying, and you should establish a workflow for rekeying when necessary.

- [Rekeying & rotating Vault](/vault/tutorials/operations/rekeying-and-rotating)
- [Operator rekey](/vault/docs/commands/operator/rekey)

**Anti-pattern issue:**

If several stakeholders leave the organization, you risk not having the required key shares to meet the unseal quorum, which could result in the loss of the ability to unseal Vault.

layout  docs page title  Recommended patterns description  Follow these recommended patterns to effectively operate Vault         Recommended patterns  Help keep your Vault environments operating effectively by implementing the following best practice so you avoid common anti patterns     Description     Applicable Vault edition                   Adjust the default lease time   adjust the default lease time      All       Use identity entities for accurate client count   use identity entities for accurate client count      Enterprise  HCP      Increase IOPS   increase iops      Enterprise  Community       Enable disaster recovery   enable disaster recovery      Enterprise       Test disaster recovery   test disaster recovery      Enterprise       Improve upgrade cadence   improve upgrade cadence      Enterprise  Community      Test before upgrades   test before upgrades      Enterprise  Community       Rotate audit device logs   rotate audit device logs      Enterprise  Community      Monitor metrics   monitor metrics       Enterprise  Community       Establish usage baseline   establish usage baseline      Enterprise  Community      Minimize root token use   minimize root token use      All       Rekey when necessary   rekey when necessary      All        Adjust the default lease time  The default lease time in Vault is 32 days or 768 hours  This time allows for some operations  such as re authentication or renewal  See  lease   vault docs concepts lease  documentation for more information     Recommended pattern     You should tune the lease TTL value for your needs  Vault holds leases in memory until the lease expires   We recommend keeping TTLs as short as the use case will allow      Auth tune   vault docs commands auth tune     Secrets tune   vault docs commands secrets tune    Note  Tuning or adjusting TTLs does not retroactively affect tokens that were issued  New tokens must be issued after tuning TTLs     Note     Anti pattern issue     If you create leases without changing the default time to live  TTL   leases will live in Vault until the default lease time is up   Depending on your infrastructure and available system memory  using the default or long TTL may cause performance issues as Vault stores leases in memory       Use identity entities for accurate client count  Each Vault client may have multiple accounts with the auth methods enabled on the Vault server     Entity   img vault entity waf1 png     Recommended pattern     Since each token adds to the client count  and each unique authentication issues a token  you should use identity entities to create aliases that connect each login to a single identity         Client count   vault docs concepts client count       Vault identity concepts   vault docs concepts identity       Vault Identity secrets engine   vault docs secrets identity       Identity  Entities and groups tutorial   vault tutorials auth methods identity     Anti pattern issue     When you do not use identity entities  each new client is counted as a separate identity when using another auth method not linked to the user s entity       Increase IOPS  IOPS  input output operations per second  measures performance for Vault cluster members  Vault is bound by the IO limits of the storage backend rather than the compute requirements      Recommended pattern     Use the HashiCorp reference guidelines for Vault servers  hardware sizing and network considerations      Vault with Integrated storage reference architecture   vault tutorials day one raft raft reference architecture system requirements     Performance tuning   vault tutorials operations performance tuning     Transform secrets engine   vault docs concepts transform    Note   Depending on the client count  the Transform  Enterprise  and Transit secret engines can be resource intensive     Note     Anti pattern issue     Limited IOPS can significantly degrade Vault s performance      Enable disaster recovery  HashiCorp Vault s  HA  highly available  Integrated storage  Raft    vault docs concepts integrated storage   backend provides intra cluster data replication across cluster members  Integrated Storage provides Vault with  horizontal scalability and failure tolerance  but it does not provide backup for the entire cluster  Not utilizing  disaster recovery for your production environment will negatively impact your organization s Recovery Point  Objective  RPO  and Recovery Time Objective  RTO       Recommended pattern      For cluster wide issues  i e   network connectivity   Vault Enterprise Disaster Recovery  DR  replication  provides a warm standby cluster containing all primary cluster data  The DR cluster does not service reads  or writes but you can promote it to replace the primary cluster when needed      Disaster recovery replication setup   vault tutorials day one raft disaster recovery     Disaster recovery  DR  replication   vault docs enterprise replication disaster recovery dr replication     DR replication API documentation   vault api docs system replication replication dr   We also recommend periodically creating data snapshots to protect against data corruption      Vault data backup standard procedure   vault tutorials standard procedures sop backup     Automated integrated storage snapshots   vault docs enterprise automated integrated storage snapshots      sys storage raft snapshot auto   vault api docs system storage raftautosnapshots     Anti pattern issue     If you do not enable disaster recovery and catastrophic failure occurs  your use cases will encounter longer downtime duration and costs associated with not serving Vault clients in your environment      Test disaster recovery  Your disaster recovery  DR  solution is a key part of your overall disaster recovery plan   Designing and configuring your Vault disaster recovery solution is only the first step  You also need to validate the DR solution  as not doing so can negatively impact your organization s Recovery Point Objective  RPO  and Recovery Time Objective  RTO      Recommended pattern     Vault s Disaster Recovery  DR  replication mode provides a warm standby for  failover if the primary cluster experiences catastrophic failure  You should  periodically test the disaster recovery replication cluster by completing the  failover and failback procedure       Vault disaster recovery replication failover and failback tutorial   vault tutorials enterprise disaster recovery replication failover     Vault Enterprise replication   vault docs enterprise replication     Monitoring Vault replication   vault tutorials monitoring monitor replication   You should establish standard operating procedures for restoring a Vault cluster from a snapshot  The restoration methods following a DR situation would be in response to data corruption or sabotage  which Disaster Recovery Replication might be unable to protect against      Standard procedure for restoring a Vault cluster   vault tutorials standard procedures sop restore     Anti pattern issue     If you don t test your disaster recovery solution  your key stakeholders will not feel confident they can effectively perform the disaster recovery plan  Testing the DR solution also helps your team to remove uncertainty around recovering the system during an outage      Improve upgrade cadence  While it might be easy to upgrade Vault whenever you have capacity  not having a frequent upgrade cadence can impact your Vault performance and security     Recommended pattern      We recommend upgrading to our latest version of Vault  Subscribe to the releases in  Vault s GitHub repository  https   github com hashicorp vault   and notifications from  HashiCorp Vault discuss  https   discuss hashicorp com c release notifications 57   will inform you when we release a new Vault version      Vault upgrade guides   vault docs upgrading     Vault feature deprecation notice and plans   vault docs deprecation     Anti pattern issue     When you do not keep a regular upgrade cadence  your Vault environment could be missing key features or improvements     Missing patches for bugs or vulnerabilities as documented in the  CHANGELOG  https   github com hashicorp vault blob main CHANGELOG md     New features to improve workflow    Must use version specific rather than the latest documentation    Some educational resourcesrequire a specific minimum Vault version    Updates may require a stepped approach that uses an intermediate version before installing the latest binary      Test before upgrades  We recommend testing Vault in a sandbox environment before deploying to production   Although it might be faster to upgrade immediately in production  testing will help identify any compatibility issues   Be aware of the  CHANGELOG  https   github com hashicorp vault blob main CHANGELOG md  and account for any new features  improvements  known issues and bug fixes in your testing     Recommended pattern      Test new Vault versions in sandbox environments before upgrading in production and follow our upgrading documentation   We recommend adding a testing phase to your standard upgrade procedure      Vault upgrade standard procedure   vault tutorials standard procedures sop upgrade     Upgrading Vault   vault docs upgrading     Anti pattern issue     Without adequate testing before upgrading in production  you risk compatibility and performance issues     Warning   This could lead to downtime or degradation in your production Vault environment     Warning      Rotate audit device logs  Audit devices in Vault maintain a detailed log of every client request and server response   If you allow the logs for audit devices to run perpetually without rotating you may face a blocked audit device if the filesystem storage becomes exhausted     Recommended pattern     Inspect and rotate audit logs periodically      Blocked audit devices tutorial   vault tutorials monitoring blocked audit devices     blocked audit devices   vault docs audit blocked audit devices     Anti pattern issue     Vault will not respond to requests when audit devices are not enabled to record them   The audit device can exhaust the local storage if the audit device log is not maintained and rotated over time      Monitor metrics  Relying solely on Vault operational logs and data in Vault UI will give you a partial picture of the cluster s performance      Recommended pattern     Continuous monitoring will allow organizations to detect minor problems and promptly resolve them   Migrating from reactive to proactive monitoring will help to prevent system failures  Vault has multiple outputs  that help monitor the cluster s activity  audit logs  operational logs  and telemetry data  This data can work  with a SIEM  security information and event management  tool for aggregation  inspection  and alerting capabilities      Telemetry   vault docs internals telemetry secrets engines metric     Telemetry metrics reference   vault tutorials monitoring telemetry metrics reference   Adding a monitoring solution     Audit device logs and incident response with elasticsearch   vault tutorials monitoring audit elastic incident response     Monitor telemetry   audit device log data   vault tutorials monitoring monitor telemetry audit splunk     Monitor telemetry with Prometheus   Grafana   vault tutorials monitoring monitor telemetry grafana prometheus     Note     Vault logs to standard output and standard error by default  automatically captured by the systemd journal  You can also instruct Vault to redirect operational log writes to a file      Note     Anti pattern issue      Having partial insight into cluster activity can leave the business in a reactive state      Establish usage baseline  A baseline provides insight into current utilization and thresholds  Telemetry metrics are valuable  especially when monitored over time  You can use telemetry metrics to gather a baseline of cluster activity  while alerts inform you of abnormal activity      Recommended pattern     Telemetry information can also be streamed directly from Vault to a range of metrics aggregation solutions and  saved for aggregation and inspection       Vault usage metrics   vault tutorials monitoring usage metrics     Diagnose server issues   vault tutorials monitoring diagnose startup issues     Anti pattern issue     This issue closely relates to the recommended pattern for  monitor metrics   monitor metrics    Telemetry data is  only held in memory for a short period      Minimize root token use  Initializing a Vault server emits an initial root token that gives root level access across all Vault features     Recommended pattern     We recommend that you revoke the root token after initializing Vault within your environment  If users require elevated access  create access control list policies that grant proper capabilities on the necessary paths in Vault  If your operations require the root token  keep it for the shortest possible time before revoking it      Generate root tokens tutorial   vault tutorials operations generate root     Root tokens   vault docs concepts tokens root tokens     Vault policies   vault docs concepts policies     Anti pattern issue     A root token can perform all actions within Vault and never expire  Unrestricted access can give users higher privileges than necessary to all Vault operations and paths  Sharing and providing access to root tokens poses a security risk      Rekey when necessary  Vault distributes unsealed keys to stakeholders  A quorum of keys is needed to unlock Vault based on your initialization settings      Recommended pattern      Vault supports rekeying  and you should establish a workflow for rekeying when necessary       Rekeying   rotating Vault   vault tutorials operations rekeying and rotating     Operator rekey   vault docs commands operator rekey     Anti pattern issue     If several stakeholders leave the organization  you risk not having the required key shares to meet the unseal quorum  which could result in the loss of the ability to unseal Vault


vault Learn about the key Vault metrics you should monitor with health checks Key metrics for common health checks layout docs This document covers common Vault monitoring patterns It is important to have operational and usage insight into a running Vault cluster to understand performance assist with proactive incident response and understand business workloads and use cases page title Key metrics for common health checks

Answer:
---
layout: docs
page_title: Key metrics for common health checks
description: >-
  Learn about the key Vault metrics you should monitor with health checks.
---

# Key metrics for common health checks

This document covers common Vault monitoring patterns. It is important to have operational and usage insight into a running Vault cluster to understand performance, assist with proactive incident response, and understand business workloads and use cases.

This document consists of five metrics sections: core, usage, storage backend, audit, and resource. Core metrics are fundamental internal metrics which you should monitor to ensure the health of your Vault cluster. The usage metrics section covers metrics which help count active and historical clients Vault. The storage backend section highlights the metrics to monitor so that you understand the storage infrastructure that your Vault cluster uses, allowing you to ensure your storage is functioning as intended. Audit metrics allow you to set up monitoring that helps you meet your compliance requirements. Resource metrics allow you to monitor metrics such as CPU, networking, and other resources Vault uses on its host. Replication covers metrics you can use to help ensure that Vault is replicating data as intended.

## Core metrics

### Servers assume the leader role

#### Metrics:

`Vault.core.leadership_lost`

`vault.core.leadership_setup_failed`

#### Background:

![Recommended architecture](/img/vault-integrated-storage-reference-architecture.svg)

The diagram illustrates a highly available Vault cluster with five nodes distributed between three availability zones. Vault's Integrated Storage uses a consensus protocol to provide consistency across the cluster nodes. The leader (active) node is responsible for ingesting new log entries, replicating them to the follower (standby) nodes, and managing when to commit an entry. Integrated Storage uses consensus protocol to provide consistency; therefore, if the leader is lost, the voting nodes will elect a new leader. Refer to the [Integrated storage](/vault/docs/internals/integrated-storage) documentation for more details.

<Tip>

When you operate Vault with Integrated Storage, it automatically provides [additional metrics for Raft leadership changes](/vault/docs/internals/integrated-storage#consensus-protocol).

</Tip>

#### Alerting:

The metric `vault.core.leadership_lost` measures the duration a server held the leader position before losing it. A consistently low value for this metric suggests a high leadership turnover, indicating potential instability within the cluster.

On the other hand, spikes in the `vault.core.leadership_setup_failed` metric indicate failures that standby servers cannot successfully assume the leader role when required. Investigate these failures promptly, and check for any issues related to acquiring the leader election lock. These metrics are important alerts and can signify security and reliability risks. For example, there might be a communication problem between Vault and its storage backend or a broader outage causing multiple Vault servers to fail. Monitoring and analyzing these metrics can help identify and address any underlying issues, ensuring the stability and security of your Vault cluster.

### Higher latency in your application

#### Metrics:

`vault.core.handle_login_request`

`vault.core.handle_request`

#### Background:

Vault can use trusted sources like Kubernetes, Active Directory, and Okta to verify the identity of clients (users or services) before granting them access to secrets. Clients must authenticate themselves by making a login request through the `vault login` command or the API. When the authentication is successful, Vault provides the client with a token, which is stored locally on the client's machine and is used to authorize future requests. As long as the client presents a valid token that has not expired, Vault recognizes the client as authenticated.

#### Alerting:

The metric `vault.core.handle_login_request`, when averaged, measures how fast Vault responds to client login requests. If you notice a significant increase in this metric but no significant increase in the number of tokens issued (`vault.token.creation`) it's crucial to investigate the cause of this issue immediately.

When a client sends a request to Vault, it typically needs to go through an authentication process to verify its identity and obtain the necessary permissions. This authentication process involves validating the client's credentials, such as username and password or API token, and ensuring the client has the appropriate access rights.

If the authentication process in Vault is slow, it takes longer for Vault to verify the client's credentials and authorize the request. This delay in authentication directly impacts the response time of Vault to the client's request.

You should also monitor the `vault.core.handle_request` metric, which measures server workload. This metric helps determine whether you need to scale up your cluster to accommodate increased traffic. On the other hand, a sudden drop in throughput may indicate connectivity problems between Vault and its clients, which you should investigate further.

### Difficulties with setting up auditing or problems with mounting a custom plugin backend

#### Metrics:

`vault.core.post_unseal`

#### Background:

Vault servers can be in one of two states: sealed or unsealed. To ensure security, Vault does not trust the storage backends and stores data in an encrypted form. After Vault is started, it must undergo an unsealing process to obtain the plaintext root key necessary to read the decryption key to decrypt the data. After unsealing, Vault performs various post-unseal operations to set up the server correctly before it can start responding to requests.

#### Alerting:

If you notice sudden increases in the `vault.core.post_unseal` metric, issues might affect your server's availability during the post-unseal process, such as errors with auditing or mounting a custom plugin backend. To diagnose the issues, refer to your Vault server's logs.

## Usage metrics

### Excessive token creations affecting Vault performance

#### Metrics:

`vault.token.creation`

##### Background:

All authenticated Vault requests require a valid client token. Tokens are linked to policies determining which capabilities  a client (user or system) has for a given path. Vault issues three types of tokens: service tokens, batch tokens, and recovery tokens.

Service tokens are what users will generally think of as "normal" Vault tokens. They support all features, such as renewal, revocation, creating child tokens, and more. They are correspondingly heavyweight to create and track.

Batch tokens are encrypted binary large objects (blobs) with just enough information about the client. While Vault does not persist the batch tokens, it persists the service tokens. The amount of space required to store the service token depends on the authentication method. Therefore, a large number of service tokens could contribute to an out-of-memory issue.

Recovery tokens are used exclusively for operating Vault in [recovery mode](/vault/docs/concepts/recovery-mode).

#### Alerting

By monitoring the number of tokens created (`vault.token.creation`) and the frequency of login requests (`vault.core.handle_login_request` counted as a total), you can gain insights into the overall workload of your system. If your scenario involves running numerous short-lived processes, such as serverless workloads, you may experience simultaneous creation and request of secrets from hundreds or thousands of functions. In such cases, you will observe correlated spikes in both metrics.

When dealing with transient workloads, you should utilize batch tokens to enhance the performance of your cluster. Vault creates a batch token, which encrypts all the client's information and provides it to the client. When the client employs this token, Vault decrypts the stored metadata and fulfills the request. Unlike service tokens, batch tokens do not retain client information or get replicated across clusters. This characteristic alleviates the burden on the storage backend and leads to improved cluster performance.

To learn more about batch tokens, refer to the [batch tokens](/vault/tutorials/tokens/batch-tokens) tutorial.

### Lease lifecycle introducing unexpected traffic spikes in Vault

#### Metrics:

`vault.expire.num_leases`

#### Background:

Vault creates a lease when it generates a dynamic secret or service token. This lease contains essential information like the secret or token’s time to live (TTL) value, and whether it can be extended or renewed. Vault stores the lease in the storage backend. If Vault doesn't renew the lease before reaching its TTL, it will expire and be invalidated, causing the associated secret or token to be revoked.

#### Alerting

Monitoring the number of active leases in your Vault server (`vault.expire.num_leases`) can provide valuable insights into the server's activity level. An increase in leases suggests a higher volume of traffic to your application. At the same time, a sudden decrease could indicate issues with accessing dynamic secrets quickly enough to serve incoming traffic.

We recommend setting the shortest possible TTL for leases to improve security and performance. There are two main reasons for this. Firstly, a shorter TTL reduces the impact of potential attacks. Secondly, it prevents leases from accumulating indefinitely and consuming excessive space in the storage backend. If you don't specify a TTL explicitly, leases default to 32 days. However, if there is a sudden surge in load and numerous leases are generated with this long default TTL, the storage backend can quickly reach its maximum capacity and crash, resulting in unavailability.

Depending on your specific use case, you may only require a token or secret for a few minutes or hours, rather than the full 32 days. By setting an appropriate TTL, you can free up storage space for storing new secrets and tokens. In the case of Vault Enterprise, you can set a [lease count quota](/vault/docs/enterprise/lease-count-quotas) to limit the number of leases generated below a certain threshold. When the threshold is reached, Vault will restrict the creation of new leases until an existing lease expires or is revoked. This helps manage the overall number of leases and prevents excessive resource usage.

Read the [Protecting Vault with resource quotas](/vault/tutorials/operations/resource-quotas) tutorial to learn how to set the lease count quotas.

Alternatively, you can leverage the [Vault agent caching](/vault/docs/agent-and-proxy/agent/caching) to delegate the lease lifecycle management to Vault Agent.

<Note>

The lifecycle of the leases are managed by the expiration manager, which handles the revocation of a lease when the time to live value associated with the lease is reached. Refer to the [Troubleshoot irrevocable leases](/vault/tutorials/monitoring/troubleshoot-irrevocable-leases) tutorial when you encounter irrevocable leases monitored by the `vault.expire.num_irrevocable_leases` metric.

</Note>

### Know the average time it takes to renew or revoke client tokens

#### Metrics:

`vault.expire.renew-token`

`vault.expire.revoke`

#### Background:

Vault automatically revokes access to secrets granted by a token when its time to live (TTL) expires. You can manually revoke a token if there are signs of a possible security breach. When a token is no longer valid (either expired or revoked), the client will lose access to the secrets managed by Vault. Therefore, the client must either renew the token before it expires, or request a new one.

#### Alerting

Monitoring the timely completion of revocation (`vault.expire.revoke`) and renewal (`vault.expire.renew-token`) operations is crucial for ensuring the validity and accessibility of secrets. Some long-running applications may require a token to be renewed instead of getting a new one. In such a case, the time it takes to renew a token can potentially affect the application from accessing secrets. Also, it is important to track the time it takes to complete the revoke operation helps to detect unauthorized access to secrets, as attackers who gain access can potentially infiltrate your system and cause harm. If you notice significant delays in the revocation process, you should investigate your server logs for any backend issues that might have hindered the revocation process.

## Storage backend metrics

### Detect any performance issues with your Vault's storage backend

#### Metrics:

`vault.<STORAGE>.get`

`vault.<STORAGE>.put`

`vault.<STORAGE>.list`

`vault.<STORAGE>.delete`

#### Background:

The performance of the storage backend affects the overall performance of the Vault; therefore, it is critical to  monitor the performance of your storage backend so that you can detect and react to any anomaly. Backend monitoring allows you to ensure that your storage infrastructure is functioning optimally. Tracking performance lets you gather detailed information and insights about the backend's operations and identify areas requiring improvement or optimization.

#### Alerting

If Vault takes longer to access the backend for operations like retrieving (`vault.<STORAGE>.get`), storing (`vault.<STORAGE>.put`), listing (`vault.<STORAGE>.list`), or deleting (`vault.<STORAGE>.delete`) items, the Vault clients may be experiencing delays caused by storage limitations. To address this issue, you can set up alerts that will notify your team automatically when Vault's access to the storage backend slows down. This will allow you to take action, such as upgrading to disks with better input/output (I/O) performance, before the increased latency negatively impacts your application users' experience.

If you are using Integrated Storage, the following resources provide additional guidance:
- [Inspect Data in Integrated Storage](/vault/tutorials/monitoring/inspect-data-integrated-storage)
- [Inspect Data in BoltDB](/vault/tutorials/monitoring/inspect-data-boltdb)

## Audit metrics

###  Blocked audit devices

#### Metrics:

`vault.audit.log_request_failure`

`vault.audit.log_response_failure`

#### Background:

Audit devices play a crucial role in meeting compliance requirements by recording a comprehensive audit log of requests and responses from Vault. For a production deployment, your Vault cluster should have at least one audit device enabled so that you can trace all incoming requests and outgoing responses associated with your cluster. If you rely on only one audit device and encounter problems (e.g., network connection loss or permission issues), Vault can become unresponsive and cease to handle requests. Enabling at least one  additional audit device is essential to ensure uninterrupted functionality and responsiveness from Vault.

#### Alerting

To ensure smooth operation, monitoring any unusual increases in audit log request failures (`vault.audit.log_request_failure`) and response failures (`vault.audit.log_response_failure`) is important. These failures could indicate a device blockage. If such issues arise, examining the audit logs can help identify the problematic device and provide additional clues about the underlying problem.

If Vault is unable to write audit logs to the syslog, the server will generate error logs similar to the following example:

```plaintext
2020-10-20T12:34:56.290Z [ERROR] audit: backend failed to log response: backend=syslog/ error="write unixgram @->/test/log: write: message too long"
2020-10-20T12:34:56.291Z [ERROR] core: failed to audit response: request_path=sys/mounts
 error="1 error occurred:
        * no audit backend succeeded in logging the response
```

You should expect to encounter a pair of errors from the audit and core modules for each failed log response. If you receive an error message containing "write: message too long," it suggests that the entries that Vault is trying to write to the syslog audit device exceed the size of the syslog host's socket send buffer. In such cases, it's necessary to investigate what is causing the generation of large log entries, such as an extensive list of Active Directory or LDAP groups.

Refer to the [Blocked audit devices](/vault/tutorials/monitoring/blocked-audit-devices) tutorial for additional guidance.

## Resource metrics

### Vault memory issues indicated by garbage collection

#### Metrics:

`vault.runtime.sys_bytes`

`vault.runtime.gc_pause_ns`

#### Background:

Garbage collection in the Go runtime temporarily pauses all operations. These pauses are usually brief, but garbage collection happens more often if memory usage is high. This increased frequency of garbage collection can cause delays in Vault's performance.

#### Alerting

Analyzing the relationship between Vault's memory usage (represented as a percentage of total available memory on the host) and garbage collection pause time (measured by `vault.runtime.gc_pause_ns`) can provide valuable insights into resource limitations and assist in effectively allocating compute resources.

To illustrate, when the `vault.runtime.sys_bytes` exceeds 90 percent of the available memory on the host, it is advisable to add more memory to prevent performance degradation. Additionally, you should set  up an alert that triggers if the GC pause time exceeds 5 seconds per minute. This alert will promptly notify you, enabling swift action to address the issue.

### CPU I/O wait time

#### Background:

Vault scales horizontally by adding more instances or nodes, but there are still practical limits to scalability. Excessive CPU wait time for I/O operations can indicate that the system is reaching its scalability limits or overusing specific resources. By tracking these metrics, administrators can assess the system's scalability and take appropriate actions, such as optimizing I/O operations or adding additional resources, to maintain performance as the system grows.

#### Alerting

We recommend keeping the I/O wait time below 10 percent to ensure optimal performance. If you notice excessively long wait times, it indicates that clients are experiencing delays while waiting for Vault to respond to their requests. This delay can negatively impact the performance of applications that rely on Vault. In such situations, evaluating if your resources are properly configured to handle your workload and if the requests are evenly distributed across all CPUs is necessary. These steps will help address potential performance issues and ensure the smooth operation of Vault and its dependent applications.

###  Keep your network throughput within the threshold

#### Background:

Monitoring the network throughput of your Vault clusters allows you to gauge their workload. A sudden decrease in traffic going in or out might indicate communication problems between Vault and its clients or dependencies. Conversely, if you observe an unexpected surge in network activity, it could be a sign of a denial of service (DoS) attack. Knowing these network patterns can provide valuable insights and help you identify potential issues or security threats.

#### Alerting

Starting from Vault 1.5, you can set rate limit quotas to ensure Vault's overall stability and health. When a server reaches this threshold, Vault will reject any new client requests and respond with an HTTP 429 error, specifically "Too Many Requests." These rejected requests will be recorded in your audit logs and display a message like this example: "error: request path kv/app/test: rate limit quota exceeded." Choosing an appropriate limit for the rate quota is important so that it doesn't block legitimate requests and cause slowdowns in your applications. To monitor the frequency of these breaches and adjust your limit accordingly, you can keep an eye on the metric called quota.rate_limit.violation, which increments with each violation of the rate limit quota.

Refer to the [Protecting Vault with resource quotas](/vault/tutorials/operations/resource-quotas) tutorial to learn how to set the rate limit quotas for your Vault.

## Replication metrics

### Free memory in the storage backend by monitoring Write-Ahead logs

#### Metrics:

`vault.wal_flushready`

`vault.wal.persistWALs`

#### Background:

To maintain high performance, Vault utilizes a garbage collector that periodically removes old Write-Ahead Logs (WALs) to free up memory on the storage backend. However, when there are unexpected surges in traffic, the accumulation of WALs can occur rapidly, leading to increased strain on the storage backend. These surges can negatively affect other processes in Vault that rely on the same storage backend. Therefore, it is important to assess the impact of replication on the performance of your storage backend. By doing so, you can better understand how the replication process influences your system's overall performance.

#### Alerting

We recommend you keep track of two metrics: `vault.wal_flushready` and `vault.wal.persistWALs`. The first metric measures the time it takes to flush a ready Write-Ahead Log (WAL) to the persist queue, while the second metric measures the time it takes to persist a WAL to the storage backend.

To ensure efficient performance, we advise you to set up alerts that will notify you when the `vault.wal_flushready` metric exceeds 500 milliseconds or when the `vault.wal.persistWALs` metric surpasses 1,000 milliseconds. These alerts serve as indicators that backpressure is slowing down your storage backend.

If either of these alerts is triggered, consider scaling your storage backend to accommodate the increased workload. Scaling can help alleviate the strain and maintain optimal performance.

###  Vault Enterprise Replication health check

#### Metrics:

`vault.replication.wal.last_wal`

#### Background:

Vault's Write-Ahead Log (WAL) is a durable data storage and recovery mechanism. WAL is a log file that records all changes made to the Vault data store before Vault persists them to the underlying storage backend. The WAL provides an extra layer of reliability and ensures data integrity in case of system failures or crashes.

#### Alerting

When you have Vault Enterprise deployments with Performance Replication and/or Disaster Recovery Replication configured, you want to monitor that the data gets replicated from the primary to the secondary clusters. To detect if your primary and secondary clusters are losing synchronization, you can compare the last Write-Ahead Log (WAL) index on both clusters. It's important to detect discrepancies between them because if the secondary clusters are significantly behind the primary and the primary cluster becomes unavailable, any requests made to Vault will yield outdated data. Therefore, if you notice missing values in the WAL, it's essential to investigate potential causes, which may include:
Network issues between the primary and secondary clusters: Problems with the network connection can hinder the proper replication of data between the clusters.
Resource limitations on the primary or secondary systems: If the primary or secondary clusters are experiencing resource constraints, it can affect their ability to replicate data effectively.

Issues with specific keys: Sometimes, the problem may relate to specific keys within the Vault. To identify such issues, examine the Vault's operational and storage logs, which will provide detailed information about the problematic keys causing the synchronization gaps.

Refer to the [Monitoring Vault replication](/vault/tutorials/monitoring/monitor-replication) tutorial to learn more.

## Additional references

- [Monitor telemetry with Prometheus & Grafana](/vault/tutorials/monitoring/monitor-telemetry-grafana-prometheus)
- [Monitor telemetry & Audit Device Log Data](/vault/tutorials/monitoring/monitor-telemetry-audit-splunk)
- [Vault usage metrics](/vault/tutorials/monitoring/usage-metrics)

layout  docs page title  Key metrics for common health checks description       Learn about the key Vault metrics you should monitor with health checks         Key metrics for common health checks  This document covers common Vault monitoring patterns  It is important to have operational and usage insight into a running Vault cluster to understand performance  assist with proactive incident response  and understand business workloads and use cases   This document consists of five metrics sections  core  usage  storage backend  audit  and resource  Core metrics are fundamental internal metrics which you should monitor to ensure the health of your Vault cluster  The usage metrics section covers metrics which help count active and historical clients Vault  The storage backend section highlights the metrics to monitor so that you understand the storage infrastructure that your Vault cluster uses  allowing you to ensure your storage is functioning as intended  Audit metrics allow you to set up monitoring that helps you meet your compliance requirements  Resource metrics allow you to monitor metrics such as CPU  networking  and other resources Vault uses on its host  Replication covers metrics you can use to help ensure that Vault is replicating data as intended      Core metrics      Servers assume the leader role       Metrics    Vault core leadership lost    vault core leadership setup failed        Background     Recommended architecture   img vault integrated storage reference architecture svg   The diagram illustrates a highly available Vault cluster with five nodes distributed between three availability zones  Vault s Integrated Storage uses a consensus protocol to provide consistency across the cluster nodes  The leader  active  node is responsible for ingesting new log entries  replicating them to the follower  standby  nodes  and managing when to commit an entry  Integrated Storage uses consensus protocol to provide consistency  therefore  if the leader is lost  the voting nodes will elect a new leader  Refer to the  Integrated storage   vault docs internals integrated storage  documentation for more details    Tip   When you operate Vault with Integrated Storage  it automatically provides  additional metrics for Raft leadership changes   vault docs internals integrated storage consensus protocol      Tip        Alerting   The metric  vault core leadership lost  measures the duration a server held the leader position before losing it  A consistently low value for this metric suggests a high leadership turnover  indicating potential instability within the cluster   On the other hand  spikes in the  vault core leadership setup failed  metric indicate failures that standby servers cannot successfully assume the leader role when required  Investigate these failures promptly  and check for any issues related to acquiring the leader election lock  These metrics are important alerts and can signify security and reliability risks  For example  there might be a communication problem between Vault and its storage backend or a broader outage causing multiple Vault servers to fail  Monitoring and analyzing these metrics can help identify and address any underlying issues  ensuring the stability and security of your Vault cluster       Higher latency in your application       Metrics    vault core handle login request    vault core handle request        Background   Vault can use trusted sources like Kubernetes  Active Directory  and Okta to verify the identity of clients  users or services  before granting them access to secrets  Clients must authenticate themselves by making a login request through the  vault login  command or the API  When the authentication is successful  Vault provides the client with a token  which is stored locally on the client s machine and is used to authorize future requests  As long as the client presents a valid token that has not expired  Vault recognizes the client as authenticated        Alerting   The metric  vault core handle login request   when averaged  measures how fast Vault responds to client login requests  If you notice a significant increase in this metric but no significant increase in the number of tokens issued   vault token creation   it s crucial to investigate the cause of this issue immediately   When a client sends a request to Vault  it typically needs to go through an authentication process to verify its identity and obtain the necessary permissions  This authentication process involves validating the client s credentials  such as username and password or API token  and ensuring the client has the appropriate access rights   If the authentication process in Vault is slow  it takes longer for Vault to verify the client s credentials and authorize the request  This delay in authentication directly impacts the response time of Vault to the client s request   You should also monitor the  vault core handle request  metric  which measures server workload  This metric helps determine whether you need to scale up your cluster to accommodate increased traffic  On the other hand  a sudden drop in throughput may indicate connectivity problems between Vault and its clients  which you should investigate further       Difficulties with setting up auditing or problems with mounting a custom plugin backend       Metrics    vault core post unseal        Background   Vault servers can be in one of two states  sealed or unsealed  To ensure security  Vault does not trust the storage backends and stores data in an encrypted form  After Vault is started  it must undergo an unsealing process to obtain the plaintext root key necessary to read the decryption key to decrypt the data  After unsealing  Vault performs various post unseal operations to set up the server correctly before it can start responding to requests        Alerting   If you notice sudden increases in the  vault core post unseal  metric  issues might affect your server s availability during the post unseal process  such as errors with auditing or mounting a custom plugin backend  To diagnose the issues  refer to your Vault server s logs      Usage metrics      Excessive token creations affecting Vault performance       Metrics    vault token creation         Background   All authenticated Vault requests require a valid client token  Tokens are linked to policies determining which capabilities  a client  user or system  has for a given path  Vault issues three types of tokens  service tokens  batch tokens  and recovery tokens   Service tokens are what users will generally think of as  normal  Vault tokens  They support all features  such as renewal  revocation  creating child tokens  and more  They are correspondingly heavyweight to create and track   Batch tokens are encrypted binary large objects  blobs  with just enough information about the client  While Vault does not persist the batch tokens  it persists the service tokens  The amount of space required to store the service token depends on the authentication method  Therefore  a large number of service tokens could contribute to an out of memory issue   Recovery tokens are used exclusively for operating Vault in  recovery mode   vault docs concepts recovery mode         Alerting  By monitoring the number of tokens created   vault token creation   and the frequency of login requests   vault core handle login request  counted as a total   you can gain insights into the overall workload of your system  If your scenario involves running numerous short lived processes  such as serverless workloads  you may experience simultaneous creation and request of secrets from hundreds or thousands of functions  In such cases  you will observe correlated spikes in both metrics   When dealing with transient workloads  you should utilize batch tokens to enhance the performance of your cluster  Vault creates a batch token  which encrypts all the client s information and provides it to the client  When the client employs this token  Vault decrypts the stored metadata and fulfills the request  Unlike service tokens  batch tokens do not retain client information or get replicated across clusters  This characteristic alleviates the burden on the storage backend and leads to improved cluster performance   To learn more about batch tokens  refer to the  batch tokens   vault tutorials tokens batch tokens  tutorial       Lease lifecycle introducing unexpected traffic spikes in Vault       Metrics    vault expire num leases        Background   Vault creates a lease when it generates a dynamic secret or service token  This lease contains essential information like the secret or token s time to live  TTL  value  and whether it can be extended or renewed  Vault stores the lease in the storage backend  If Vault doesn t renew the lease before reaching its TTL  it will expire and be invalidated  causing the associated secret or token to be revoked        Alerting  Monitoring the number of active leases in your Vault server   vault expire num leases   can provide valuable insights into the server s activity level  An increase in leases suggests a higher volume of traffic to your application  At the same time  a sudden decrease could indicate issues with accessing dynamic secrets quickly enough to serve incoming traffic   We recommend setting the shortest possible TTL for leases to improve security and performance  There are two main reasons for this  Firstly  a shorter TTL reduces the impact of potential attacks  Secondly  it prevents leases from accumulating indefinitely and consuming excessive space in the storage backend  If you don t specify a TTL explicitly  leases default to 32 days  However  if there is a sudden surge in load and numerous leases are generated with this long default TTL  the storage backend can quickly reach its maximum capacity and crash  resulting in unavailability   Depending on your specific use case  you may only require a token or secret for a few minutes or hours  rather than the full 32 days  By setting an appropriate TTL  you can free up storage space for storing new secrets and tokens  In the case of Vault Enterprise  you can set a  lease count quota   vault docs enterprise lease count quotas  to limit the number of leases generated below a certain threshold  When the threshold is reached  Vault will restrict the creation of new leases until an existing lease expires or is revoked  This helps manage the overall number of leases and prevents excessive resource usage   Read the  Protecting Vault with resource quotas   vault tutorials operations resource quotas  tutorial to learn how to set the lease count quotas   Alternatively  you can leverage the  Vault agent caching   vault docs agent and proxy agent caching  to delegate the lease lifecycle management to Vault Agent    Note   The lifecycle of the leases are managed by the expiration manager  which handles the revocation of a lease when the time to live value associated with the lease is reached  Refer to the  Troubleshoot irrevocable leases   vault tutorials monitoring troubleshoot irrevocable leases  tutorial when you encounter irrevocable leases monitored by the  vault expire num irrevocable leases  metric     Note       Know the average time it takes to renew or revoke client tokens       Metrics    vault expire renew token    vault expire revoke        Background   Vault automatically revokes access to secrets granted by a token when its time to live  TTL  expires  You can manually revoke a token if there are signs of a possible security breach  When a token is no longer valid  either expired or revoked   the client will lose access to the secrets managed by Vault  Therefore  the client must either renew the token before it expires  or request a new one        Alerting  Monitoring the timely completion of revocation   vault expire revoke   and renewal   vault expire renew token   operations is crucial for ensuring the validity and accessibility of secrets  Some long running applications may require a token to be renewed instead of getting a new one  In such a case  the time it takes to renew a token can potentially affect the application from accessing secrets  Also  it is important to track the time it takes to complete the revoke operation helps to detect unauthorized access to secrets  as attackers who gain access can potentially infiltrate your system and cause harm  If you notice significant delays in the revocation process  you should investigate your server logs for any backend issues that might have hindered the revocation process      Storage backend metrics      Detect any performance issues with your Vault s storage backend       Metrics    vault  STORAGE  get    vault  STORAGE  put    vault  STORAGE  list    vault  STORAGE  delete        Background   The performance of the storage backend affects the overall performance of the Vault  therefore  it is critical to  monitor the performance of your storage backend so that you can detect and react to any anomaly  Backend monitoring allows you to ensure that your storage infrastructure is functioning optimally  Tracking performance lets you gather detailed information and insights about the backend s operations and identify areas requiring improvement or optimization        Alerting  If Vault takes longer to access the backend for operations like retrieving   vault  STORAGE  get    storing   vault  STORAGE  put    listing   vault  STORAGE  list    or deleting   vault  STORAGE  delete   items  the Vault clients may be experiencing delays caused by storage limitations  To address this issue  you can set up alerts that will notify your team automatically when Vault s access to the storage backend slows down  This will allow you to take action  such as upgrading to disks with better input output  I O  performance  before the increased latency negatively impacts your application users  experience   If you are using Integrated Storage  the following resources provide additional guidance     Inspect Data in Integrated Storage   vault tutorials monitoring inspect data integrated storage     Inspect Data in BoltDB   vault tutorials monitoring inspect data boltdb      Audit metrics       Blocked audit devices       Metrics    vault audit log request failure    vault audit log response failure        Background   Audit devices play a crucial role in meeting compliance requirements by recording a comprehensive audit log of requests and responses from Vault  For a production deployment  your Vault cluster should have at least one audit device enabled so that you can trace all incoming requests and outgoing responses associated with your cluster  If you rely on only one audit device and encounter problems  e g   network connection loss or permission issues   Vault can become unresponsive and cease to handle requests  Enabling at least one  additional audit device is essential to ensure uninterrupted functionality and responsiveness from Vault        Alerting  To ensure smooth operation  monitoring any unusual increases in audit log request failures   vault audit log request failure   and response failures   vault audit log response failure   is important  These failures could indicate a device blockage  If such issues arise  examining the audit logs can help identify the problematic device and provide additional clues about the underlying problem   If Vault is unable to write audit logs to the syslog  the server will generate error logs similar to the following example      plaintext 2020 10 20T12 34 56 290Z  ERROR  audit  backend failed to log response  backend syslog  error  write unixgram     test log  write  message too long  2020 10 20T12 34 56 291Z  ERROR  core  failed to audit response  request path sys mounts  error  1 error occurred            no audit backend succeeded in logging the response       You should expect to encounter a pair of errors from the audit and core modules for each failed log response  If you receive an error message containing  write  message too long   it suggests that the entries that Vault is trying to write to the syslog audit device exceed the size of the syslog host s socket send buffer  In such cases  it s necessary to investigate what is causing the generation of large log entries  such as an extensive list of Active Directory or LDAP groups   Refer to the  Blocked audit devices   vault tutorials monitoring blocked audit devices  tutorial for additional guidance      Resource metrics      Vault memory issues indicated by garbage collection       Metrics    vault runtime sys bytes    vault runtime gc pause ns        Background   Garbage collection in the Go runtime temporarily pauses all operations  These pauses are usually brief  but garbage collection happens more often if memory usage is high  This increased frequency of garbage collection can cause delays in Vault s performance        Alerting  Analyzing the relationship between Vault s memory usage  represented as a percentage of total available memory on the host  and garbage collection pause time  measured by  vault runtime gc pause ns   can provide valuable insights into resource limitations and assist in effectively allocating compute resources   To illustrate  when the  vault runtime sys bytes  exceeds 90 percent of the available memory on the host  it is advisable to add more memory to prevent performance degradation  Additionally  you should set  up an alert that triggers if the GC pause time exceeds 5 seconds per minute  This alert will promptly notify you  enabling swift action to address the issue       CPU I O wait time       Background   Vault scales horizontally by adding more instances or nodes  but there are still practical limits to scalability  Excessive CPU wait time for I O operations can indicate that the system is reaching its scalability limits or overusing specific resources  By tracking these metrics  administrators can assess the system s scalability and take appropriate actions  such as optimizing I O operations or adding additional resources  to maintain performance as the system grows        Alerting  We recommend keeping the I O wait time below 10 percent to ensure optimal performance  If you notice excessively long wait times  it indicates that clients are experiencing delays while waiting for Vault to respond to their requests  This delay can negatively impact the performance of applications that rely on Vault  In such situations  evaluating if your resources are properly configured to handle your workload and if the requests are evenly distributed across all CPUs is necessary  These steps will help address potential performance issues and ensure the smooth operation of Vault and its dependent applications        Keep your network throughput within the threshold       Background   Monitoring the network throughput of your Vault clusters allows you to gauge their workload  A sudden decrease in traffic going in or out might indicate communication problems between Vault and its clients or dependencies  Conversely  if you observe an unexpected surge in network activity  it could be a sign of a denial of service  DoS  attack  Knowing these network patterns can provide valuable insights and help you identify potential issues or security threats        Alerting  Starting from Vault 1 5  you can set rate limit quotas to ensure Vault s overall stability and health  When a server reaches this threshold  Vault will reject any new client requests and respond with an HTTP 429 error  specifically  Too Many Requests   These rejected requests will be recorded in your audit logs and display a message like this example   error  request path kv app test  rate limit quota exceeded   Choosing an appropriate limit for the rate quota is important so that it doesn t block legitimate requests and cause slowdowns in your applications  To monitor the frequency of these breaches and adjust your limit accordingly  you can keep an eye on the metric called quota rate limit violation  which increments with each violation of the rate limit quota   Refer to the  Protecting Vault with resource quotas   vault tutorials operations resource quotas  tutorial to learn how to set the rate limit quotas for your Vault      Replication metrics      Free memory in the storage backend by monitoring Write Ahead logs       Metrics    vault wal flushready    vault wal persistWALs        Background   To maintain high performance  Vault utilizes a garbage collector that periodically removes old Write Ahead Logs  WALs  to free up memory on the storage backend  However  when there are unexpected surges in traffic  the accumulation of WALs can occur rapidly  leading to increased strain on the storage backend  These surges can negatively affect other processes in Vault that rely on the same storage backend  Therefore  it is important to assess the impact of replication on the performance of your storage backend  By doing so  you can better understand how the replication process influences your system s overall performance        Alerting  We recommend you keep track of two metrics   vault wal flushready  and  vault wal persistWALs   The first metric measures the time it takes to flush a ready Write Ahead Log  WAL  to the persist queue  while the second metric measures the time it takes to persist a WAL to the storage backend   To ensure efficient performance  we advise you to set up alerts that will notify you when the  vault wal flushready  metric exceeds 500 milliseconds or when the  vault wal persistWALs  metric surpasses 1 000 milliseconds  These alerts serve as indicators that backpressure is slowing down your storage backend   If either of these alerts is triggered  consider scaling your storage backend to accommodate the increased workload  Scaling can help alleviate the strain and maintain optimal performance        Vault Enterprise Replication health check       Metrics    vault replication wal last wal        Background   Vault s Write Ahead Log  WAL  is a durable data storage and recovery mechanism  WAL is a log file that records all changes made to the Vault data store before Vault persists them to the underlying storage backend  The WAL provides an extra layer of reliability and ensures data integrity in case of system failures or crashes        Alerting  When you have Vault Enterprise deployments with Performance Replication and or Disaster Recovery Replication configured  you want to monitor that the data gets replicated from the primary to the secondary clusters  To detect if your primary and secondary clusters are losing synchronization  you can compare the last Write Ahead Log  WAL  index on both clusters  It s important to detect discrepancies between them because if the secondary clusters are significantly behind the primary and the primary cluster becomes unavailable  any requests made to Vault will yield outdated data  Therefore  if you notice missing values in the WAL  it s essential to investigate potential causes  which may include  Network issues between the primary and secondary clusters  Problems with the network connection can hinder the proper replication of data between the clusters  Resource limitations on the primary or secondary systems  If the primary or secondary clusters are experiencing resource constraints  it can affect their ability to replicate data effectively   Issues with specific keys  Sometimes  the problem may relate to specific keys within the Vault  To identify such issues  examine the Vault s operational and storage logs  which will provide detailed information about the problematic keys causing the synchronization gaps   Refer to the  Monitoring Vault replication   vault tutorials monitoring monitor replication  tutorial to learn more      Additional references     Monitor telemetry with Prometheus   Grafana   vault tutorials monitoring monitor telemetry grafana prometheus     Monitor telemetry   Audit Device Log Data   vault tutorials monitoring monitor telemetry audit splunk     Vault usage metrics   vault tutorials monitoring usage metrics


vault page title Enable Vault telemetry Collect telemetry data from your Vault installation layout docs Step by step guide to enabling telemetry gathering with Vault Enable Vault telemetry gathering

Answer:
---
layout: docs
page_title: Enable Vault telemetry
description: >-
  Step-by-step guide to enabling telemetry gathering with Vault
---

# Enable Vault telemetry gathering

Collect telemetry data from your Vault installation.

## Before you start

- **You must have Vault 1.14 or later installed and running**.
- **You must have access to your [Vault configuration](/vault/docs/configuration) file**.

## Step 1: Choose an aggregation agent

@include 'telemetry/supported-aggregation-agents.mdx'

## Step 2: Enable at least one audit device

To include audit-related metrics, you must enable auditing on at least one device
with the `vault audit enable` command. For example, to enable auditing for the
`file` device and save the logs to `/var/log/vault_audit.log`:

```shell-session
$ vault audit enable file file_path=/var/log/vault_audit.log
```

By default, Enterprise installations replicate audit devices to the secondary
performance nodes in a cluster. To limit performance replication for an audit
device, use the `local` flag to mark the device as local to the current node:

```shell-session
$ vault audit enable file -local file_path=/var/log/vault_audit.log
```

## Step 3: Configure telemetry collection

To configure telemetry collection, update the telemetry stanza in your Vault
configuration with your collection preferences and aggregation agent details.

For example, the following `telemetry` stanza configures Vault with the standard
telemetry defaults and connects it to a Statsite agent running on the default
port within a company intranet at `mycompany.statsite`:

```hcl
telemetry {
  usage_gauge_period = "10m"
  maximum_gauge_cardinality = 500
  disable_hostname = false
  enable_hostname_label = false
  lease_metrics_epsilon = "1h"
  num_lease_metrics_buckets = 168
  add_lease_metrics_namespace_labels = false
  filter_default = true

  statsite_address = "mycompany.statsite:8125"
}
```

Many metrics solutions charge by the metric. You can set `filter_default` to
false and use the `prefix_filter` parameter to include and exclude specific
values based on metric name to avoid paying for irrelevant information.

For example, to limit your telemetry to the core token metrics plus the number
of leases set to expire:

```hcl
telemetry {
  filter_default = false
  prefix_filter = ["+vault.token", "-vault.expire", "+vault.expire.num_leases"]
}
```

## Step 4: Choose a reporting solution

You need to save or forward your telemetry data to a separate storage solution
for reporting, analysis, and alerting. Which solution you need depends on the
feature set provided by your aggregation agent and the protocol support of your
reporting platform.

Popular reporting solutions compatible with Vault:

- [Grafana](https://grafana.com/grafana)
- [Graphite](https://www.hostedgraphite.com)
- [InfluxData: Telegraf](https://www.influxdata.com/time-series-platform/telegraf)
- [InfluxData: InfluxDB](https://www.influxdata.com/products/influxdb-overview)
- [InfluxData: Chronograf](https://www.influxdata.com/time-series-platform/telegraf)
- [InfluxData: Kapacitor](https://www.influxdata.com/time-series-platform/kapacitor)
- [Splunk](https://www.splunk.com)

## Next steps

- Review the
  [Key metrics for common health checks](/well-architected-framework/reliability/reliability-vault-monitoring-key-metrics)
  guide to identify metrics you may want to start monitoring immediately.
- Review the full list of available
  [telemetry parameters](/vault/docs/configuration/telemetry#telemetry-parameters).
- Review the [Monitor telemetry and audit device log data](/vault/tutorials/monitoring/monitor-telemetry-audit-splunk)
  tutorial for general monitoring guidance and steps to configure your
  Vault telemetry for Splunk using Telegraf and Fluentd.
- Review the
  [Monitor telemetry with Prometheus and Grafana](/vault/tutorials/monitoring/monitor-telemetry-grafana-prometheus)
  tutorial to configure your Vault telemetry for Prometheus and Grafana.

layout  docs page title  Enable Vault telemetry description       Step by step guide to enabling telemetry gathering with Vault        Enable Vault telemetry gathering  Collect telemetry data from your Vault installation      Before you start      You must have Vault 1 14 or later installed and running        You must have access to your  Vault configuration   vault docs configuration  file         Step 1  Choose an aggregation agent   include  telemetry supported aggregation agents mdx      Step 2  Enable at least one audit device  To include audit related metrics  you must enable auditing on at least one device with the  vault audit enable  command  For example  to enable auditing for the  file  device and save the logs to   var log vault audit log       shell session   vault audit enable file file path  var log vault audit log      By default  Enterprise installations replicate audit devices to the secondary performance nodes in a cluster  To limit performance replication for an audit device  use the  local  flag to mark the device as local to the current node      shell session   vault audit enable file  local file path  var log vault audit log          Step 3  Configure telemetry collection  To configure telemetry collection  update the telemetry stanza in your Vault configuration with your collection preferences and aggregation agent details   For example  the following  telemetry  stanza configures Vault with the standard telemetry defaults and connects it to a Statsite agent running on the default port within a company intranet at  mycompany statsite       hcl telemetry     usage gauge period    10m    maximum gauge cardinality   500   disable hostname   false   enable hostname label   false   lease metrics epsilon    1h    num lease metrics buckets   168   add lease metrics namespace labels   false   filter default   true    statsite address    mycompany statsite 8125         Many metrics solutions charge by the metric  You can set  filter default  to false and use the  prefix filter  parameter to include and exclude specific values based on metric name to avoid paying for irrelevant information   For example  to limit your telemetry to the core token metrics plus the number of leases set to expire      hcl telemetry     filter default   false   prefix filter      vault token     vault expire     vault expire num leases             Step 4  Choose a reporting solution  You need to save or forward your telemetry data to a separate storage solution for reporting  analysis  and alerting  Which solution you need depends on the feature set provided by your aggregation agent and the protocol support of your reporting platform   Popular reporting solutions compatible with Vault      Grafana  https   grafana com grafana     Graphite  https   www hostedgraphite com     InfluxData  Telegraf  https   www influxdata com time series platform telegraf     InfluxData  InfluxDB  https   www influxdata com products influxdb overview     InfluxData  Chronograf  https   www influxdata com time series platform telegraf     InfluxData  Kapacitor  https   www influxdata com time series platform kapacitor     Splunk  https   www splunk com       Next steps    Review the    Key metrics for common health checks   well architected framework reliability reliability vault monitoring key metrics    guide to identify metrics you may want to start monitoring immediately    Review the full list of available    telemetry parameters   vault docs configuration telemetry telemetry parameters     Review the  Monitor telemetry and audit device log data   vault tutorials monitoring monitor telemetry audit splunk    tutorial for general monitoring guidance and steps to configure your   Vault telemetry for Splunk using Telegraf and Fluentd    Review the    Monitor telemetry with Prometheus and Grafana   vault tutorials monitoring monitor telemetry grafana prometheus    tutorial to configure your Vault telemetry for Prometheus and Grafana


vault Database telemetry page title Telemetry reference Database metrics Database telemetry provides general information about configured secrets engines layout docs Technical reference for database telemetry values

Answer:
---
layout: docs
page_title: "Telemetry reference: Database metrics"
description: >-
  Technical reference for database telemetry values.
---

# Database telemetry

Database telemetry provides general information about configured secrets engines
and databases.

## Secrets database metrics

@include 'telemetry-metrics/secretsdb-intro.mdx'

@include 'telemetry-metrics/database/close.mdx'

@include 'telemetry-metrics/database/close/error.mdx'

@include 'telemetry-metrics/database/createuser.mdx'

@include 'telemetry-metrics/database/createuser/error.mdx'

@include 'telemetry-metrics/database/initialize.mdx'

@include 'telemetry-metrics/database/initialize/error.mdx'

@include 'telemetry-metrics/database/name/close.mdx'

@include 'telemetry-metrics/database/name/close/error.mdx'

@include 'telemetry-metrics/database/name/createuser.mdx'

@include 'telemetry-metrics/database/name/createuser/error.mdx'

@include 'telemetry-metrics/database/name/initialize.mdx'

@include 'telemetry-metrics/database/name/initialize/error.mdx'

@include 'telemetry-metrics/database/name/renewuser.mdx'

@include 'telemetry-metrics/database/name/renewuser/error.mdx'

@include 'telemetry-metrics/database/name/revokeuser.mdx'

@include 'telemetry-metrics/database/name/revokeuser/error.mdx'

@include 'telemetry-metrics/database/renewuser.mdx'

@include 'telemetry-metrics/database/renewuser/error.mdx'

@include 'telemetry-metrics/database/revokeuser.mdx'

@include 'telemetry-metrics/database/revokeuser/error.mdx'

## Cockroach database

Metrics related to your Cockroach database **storage backend**.

@include 'telemetry-metrics/vault/cockroachdb/delete.mdx'

@include 'telemetry-metrics/vault/cockroachdb/get.mdx'

@include 'telemetry-metrics/vault/cockroachdb/list.mdx'

@include 'telemetry-metrics/vault/cockroachdb/put.mdx'

## Couch database

Metrics related to your Couch database **storage backend**.

@include 'telemetry-metrics/vault/couchdb/delete.mdx'

@include 'telemetry-metrics/vault/couchdb/get.mdx'

@include 'telemetry-metrics/vault/couchdb/list.mdx'

@include 'telemetry-metrics/vault/couchdb/put.mdx'

## Dynamo database

Metrics related to your Dynamo database **storage backend**.

@include 'telemetry-metrics/vault/dynamodb/delete.mdx'

@include 'telemetry-metrics/vault/dynamodb/get.mdx'

@include 'telemetry-metrics/vault/dynamodb/list.mdx'

@include 'telemetry-metrics/vault/dynamodb/put.mdx'

## Google Cloud - Spanner

Metrics related to your Spanner **storage backend**.

@include 'telemetry-metrics/vault/spanner/delete.mdx'

@include 'telemetry-metrics/vault/spanner/get.mdx'

@include 'telemetry-metrics/vault/spanner/list.mdx'

@include 'telemetry-metrics/vault/spanner/lock/lock.mdx'

@include 'telemetry-metrics/vault/spanner/lock/unlock.mdx'

@include 'telemetry-metrics/vault/spanner/lock/value.mdx'

@include 'telemetry-metrics/vault/spanner/put.mdx'

## Microsoft SQL Server (MSSQL)

Metrics related to your SQL Server **storage backend**.

@include 'telemetry-metrics/vault/mssql/delete.mdx'

@include 'telemetry-metrics/vault/mssql/get.mdx'

@include 'telemetry-metrics/vault/mssql/list.mdx'

@include 'telemetry-metrics/vault/mssql/put.mdx'

## MySQL

Metrics related to your MySQL **storage backend**.

@include 'telemetry-metrics/vault/mysql/delete.mdx'

@include 'telemetry-metrics/vault/mysql/get.mdx'

@include 'telemetry-metrics/vault/mysql/list.mdx'

@include 'telemetry-metrics/vault/mysql/put.mdx'

## PostgreSQL

Metrics related to your PostgreSQL **storage backend**.

@include 'telemetry-metrics/vault/postgres/delete.mdx'

@include 'telemetry-metrics/vault/postgres/get.mdx'

@include 'telemetry-metrics/vault/postgres/list.mdx'

@include 'telemetry-metrics/vault/postgres/put.mdx

layout  docs page title   Telemetry reference  Database metrics  description       Technical reference for database telemetry values         Database telemetry  Database telemetry provides general information about configured secrets engines and databases      Secrets database metrics   include  telemetry metrics secretsdb intro mdx    include  telemetry metrics database close mdx    include  telemetry metrics database close error mdx    include  telemetry metrics database createuser mdx    include  telemetry metrics database createuser error mdx    include  telemetry metrics database initialize mdx    include  telemetry metrics database initialize error mdx    include  telemetry metrics database name close mdx    include  telemetry metrics database name close error mdx    include  telemetry metrics database name createuser mdx    include  telemetry metrics database name createuser error mdx    include  telemetry metrics database name initialize mdx    include  telemetry metrics database name initialize error mdx    include  telemetry metrics database name renewuser mdx    include  telemetry metrics database name renewuser error mdx    include  telemetry metrics database name revokeuser mdx    include  telemetry metrics database name revokeuser error mdx    include  telemetry metrics database renewuser mdx    include  telemetry metrics database renewuser error mdx    include  telemetry metrics database revokeuser mdx    include  telemetry metrics database revokeuser error mdx      Cockroach database  Metrics related to your Cockroach database   storage backend      include  telemetry metrics vault cockroachdb delete mdx    include  telemetry metrics vault cockroachdb get mdx    include  telemetry metrics vault cockroachdb list mdx    include  telemetry metrics vault cockroachdb put mdx      Couch database  Metrics related to your Couch database   storage backend      include  telemetry metrics vault couchdb delete mdx    include  telemetry metrics vault couchdb get mdx    include  telemetry metrics vault couchdb list mdx    include  telemetry metrics vault couchdb put mdx      Dynamo database  Metrics related to your Dynamo database   storage backend      include  telemetry metrics vault dynamodb delete mdx    include  telemetry metrics vault dynamodb get mdx    include  telemetry metrics vault dynamodb list mdx    include  telemetry metrics vault dynamodb put mdx      Google Cloud   Spanner  Metrics related to your Spanner   storage backend      include  telemetry metrics vault spanner delete mdx    include  telemetry metrics vault spanner get mdx    include  telemetry metrics vault spanner list mdx    include  telemetry metrics vault spanner lock lock mdx    include  telemetry metrics vault spanner lock unlock mdx    include  telemetry metrics vault spanner lock value mdx    include  telemetry metrics vault spanner put mdx      Microsoft SQL Server  MSSQL   Metrics related to your SQL Server   storage backend      include  telemetry metrics vault mssql delete mdx    include  telemetry metrics vault mssql get mdx    include  telemetry metrics vault mssql list mdx    include  telemetry metrics vault mssql put mdx      MySQL  Metrics related to your MySQL   storage backend      include  telemetry metrics vault mysql delete mdx    include  telemetry metrics vault mysql get mdx    include  telemetry metrics vault mysql list mdx    include  telemetry metrics vault mysql put mdx      PostgreSQL  Metrics related to your PostgreSQL   storage backend      include  telemetry metrics vault postgres delete mdx    include  telemetry metrics vault postgres get mdx    include  telemetry metrics vault postgres list mdx    include  telemetry metrics vault postgres put mdx