vault page title Integrated Storage Learn about the integrated raft storage in Vault information As of Vault 1 4 an Integrated Storage option is offered This Integrated storage Vault supports a number of storage options for the durable storage of Vault s layout docs

Answer:
---
layout: docs
page_title: Integrated Storage
description: Learn about the integrated raft storage in Vault.
---

# Integrated storage

Vault supports a number of storage options for the durable storage of Vault's
information. As of Vault 1.4 an Integrated Storage option is offered. This
storage backend does not rely on any third party systems, implements high
availability semantics, supports Enterprise Replication features, and provides
backup/restore workflows.

The option stores Vault's data on the server's filesystem and
uses a consensus protocol to replicate data to each server in the cluster. More
information on the internals of Integrated Storage can be found in the
[Integrated Storage internals
documentation](/vault/docs/internals/integrated-storage/). Additionally, the
[Configuration](/vault/docs/configuration/storage/raft/) docs can help in configuring
Vault to use Integrated Storage.

The sections below go into various details on how to operate Vault with
Integrated Storage.

## Server-to-Server communication

Once nodes are joined to one another they begin to communicate using mTLS over
Vault's cluster port. The cluster port defaults to `8201`. The TLS information
is exchanged at join time and is rotated on a cadence.

A requirement for Integrated Storage is that the
[`cluster_addr`](/vault/docs/concepts/ha#per-node-cluster-address) configuration option
is set. This allows Vault to assign an address to the node ID at join time.

## Cluster membership

This section will outline how to bootstrap and manage a cluster of Vault nodes
running Integrated Storage.

Integrated Storage is bootstrapped during the [initialization
process](/vault/tutorials/getting-started/getting-started-deploy#initializing-the-vault),
and results in a cluster of size 1. Depending on the [desired deployment
size](/vault/docs/internals/integrated-storage/#deployment-table), nodes can be joined
to the active Vault node.

### Joining nodes

Joining is the process of taking an uninitialized Vault node and making it a
member of an existing cluster. In order to authenticate the new node to the
cluster it must use the same seal mechanism. If using a Auto Unseal the node
must be configured to use the same KMS provider and Key as the cluster it's
attempting to join. If using a Shamir seal the unseal keys must be provided to
the new node before the join process can complete. Once a node has successfully
joined, data from the active node can begin to replicate to it. Once a node has
been joined it cannot be re-joined to a different cluster.

You can either join the node automatically via the config file or manually through the
API (both methods described below). When joining a node, the API address of the leader node must be used. We
recommend setting the [`api_addr`](/vault/docs/concepts/ha#direct-access) configuration
option on all nodes to make joining simpler.

Always join nodes to a cluster one at a time and wait for the node to become
healthy and (if applicable) a voter before continuing to add more nodes. The
status of a node can be verified by performing a [`list-peers`](/vault/docs/commands/operator/raft#list-peers)
command or by checking the [`autopilot state`](/vault/docs/commands/operator/raft#autopilot-state).

#### `retry_join` configuration

This method enables setting one, or more, target leader nodes in the config file.
When an uninitialized Vault server starts up it will attempt to join each potential
leader that is defined, retrying until successful. When one of the specified
leaders become active this node will successfully join. When using Shamir seal,
the joined nodes will still need to be unsealed manually. When using Auto Unseal
the node will be able to join and unseal automatically.

An example [`retry_join`](/vault/docs/configuration/storage/raft#retry_join-stanza)
config can be seen below:

```hcl
storage "raft" {
  path    = "/var/raft/"
  node_id = "node3"

  retry_join {
    leader_api_addr = "https://node1.vault.local:8200"
  }
  retry_join {
    leader_api_addr = "https://node2.vault.local:8200"
  }
}
```

Note, in each [`retry_join`](/vault/docs/configuration/storage/raft#retry_join-stanza)
stanza, you may provide a single
[`leader_api_addr`](/vault/docs/configuration/storage/raft#leader_api_addr) or
[`auto_join`](/vault/docs/configuration/storage/raft#auto_join) value. When a cloud
[`auto_join`](/vault/docs/configuration/storage/raft#auto_join) configuration value is
provided, Vault will use [go-discover](https://github.com/hashicorp/go-discover)
to automatically attempt to discover and resolve potential Raft leader
addresses.

Check the go-discover
[README](https://github.com/hashicorp/go-discover/blob/master/README.md) for
details on the format of the [`auto_join`](/vault/docs/configuration/storage/raft#auto_join)
value per cloud provider.

```hcl
storage "raft" {
  path    = "/var/raft/"
  node_id = "node3"

  retry_join {
    auto_join = "provider=aws region=eu-west-1 tag_key=vault tag_value=... access_key_id=... secret_access_key=..."
  }
}
```

By default, Vault will attempt to reach discovered peers using HTTPS and port 8200. Operators may override these through the
[`auto_join_scheme`](/vault/docs/configuration/storage/raft#auto_join_scheme) and
[`auto_join_port`](/vault/docs/configuration/storage/raft#auto_join_port) fields
respectively.

```hcl
storage "raft" {
  path    = "/var/raft/"
  node_id = "node3"

  retry_join {
    auto_join = "provider=aws region=eu-west-1 tag_key=vault tag_value=... access_key_id=... secret_access_key=..."
    auto_join_scheme = "http"
    auto_join_port = 8201
  }
}
```

#### Join from the CLI

Alternatively you can use the [`join` CLI
command](/vault/docs/commands/operator/raft/#join) or the API to join a node. The
active node's API address will need to be specified:

```shell-session
$ vault operator raft join https://node1.vault.local:8200
```

#### Non-Voting nodes (Enterprise only)

Nodes that are joined to a cluster can be specified as non-voters. A non-voting
node has all of Vault's data replicated to it, but does not contribute to the
quorum count. This can be used in conjunction with [Performance
Standby](/vault/docs/enterprise/performance-standby/) nodes to add read scalability to
a cluster in cases where a high volume of reads to servers are needed.

```shell-session
$ vault operator raft join -non-voter https://node1.vault.local:8200
```

### Removing peers

Removing a peer node is a necessary step when you no longer want the node in the
cluster. This could happen if the node is rotated for a new one, the hostname
permanently changes and can no longer be accessed, you're attempting to shrink
the size of the cluster, or for many other reasons. Removing the peer will
ensure the cluster stays at the desired size, and that quorum is maintained.

To remove the peer you can issue a
[`remove-peer`](/vault/docs/commands/operator/raft#remove-peer) command and provide the
node ID you wish to remove:

```shell-session
$ vault operator raft remove-peer node1
Peer removed successfully!
```

#### Re-joining after removal

If you have used `remove-peer` to remove a node from the Raft cluster, but you
later want to have this same node re-join the cluster, you will need to delete
any existing Raft data on the removed node before adding it back to the cluster.
This will involve stopping the Vault process, deleting the data directory containing
Raft data, and then restarting the Vault process.

### Listing peers

To see the current peer set for the cluster you can issue a
[`list-peers`](/vault/docs/commands/operator/raft#list-peers) command. All the voting
nodes that are listed here contribute to the quorum and a majority must be alive
for Integrated Storage to continue to operate.

```shell-session
$ vault operator raft list-peers
Node     Address                   State       Voter
----     -------                   -----       -----
node1    node1.vault.local:8201    follower    true
node2    node2.vault.local:8201    follower    true
node3    node3.vault.local:8201    leader      true
```

## Integrated storage and TLS

We've glossed over some details in the above sections on bootstrapping clusters.
The instructions are sufficient for most cases, but some users have run into
problems when using auto-join and TLS in conjunction with things like auto-scaling.
The issue is that [go-discover](https://github.com/hashicorp/go-discover) on
most platforms returns IPs (not hostnames), and because the IPs aren't knowable
in advance, the TLS certificates used to secure the Vault API port don't contain
these IPs in their IP SANs.

### Vault networking recap

Before we explore solutions to this problem, let's recapitulate how Vault nodes
speak to one another.

Vault exposes two TCP ports: [the API port](/vault/docs/configuration#api_addr) and
[the cluster port](/vault/docs/configuration#cluster_addr).

The API port is where clients send their Vault HTTP requests.

For a single-node Vault cluster you don't worry about a cluster port as it won't be used.

When you have multiple nodes, you also need a cluster port. This is used by Vault
nodes to issue RPCs to one another, e.g. to forward requests from a standby node
to the active node, or when Raft is in use, to handle leader election and
replication of stored data.

The cluster port is secured using a TLS certificate that the Vault active node
generates internally. It's clear how this can work when not using integrated
storage: every node has at least read access to storage, so once the active
node has persisted the certificate, the standby nodes can fetch it, and all
agree on how cluster traffic should be encrypted.

It's less clear how this works with Integrated Storage, as there is a chicken
and egg problem. Nodes don't have a shared view of storage until the raft
cluster has been formed, but we're trying to form the raft cluster! To solve
this problem, a Vault node must speak to another Vault node using the API port
instead of the cluster port. This is currently the only situation in which
Vault Community Edition does this (Vault Enterprise also does something similar when setting
up replication.)

- `node2` wants to join the cluster, so issues challenge API request to existing member `node1`
- `node1` replies to challenge request with (1) an encrypted random UUID and (2) seal config
- `node2` must decrypt UUID using seal; if using auto-unseal can do it directly, if using Shamir must wait for user to provide enough unseal keys to perform decryption
- `node2` sends decrypted UUID back to `node1` using answer API
- `node1` sees `node2` can be trusted (since it has seal access) and replies with a bootstrap package which includes the cluster TLS certificate and private key
- `node2` gets sent a raft snapshot over the cluster port

After this procedure the new node will never again send traffic to the API port.
All subsequent inter-node communication will use the cluster port.

![Raft Join Process](/img/raft-join-detailed.png)

### Assisted raft join techniques

The simplest option is to do it by hand: issue [`raft join`](/vault/docs/commands/operator/raft#join) commands specifying the explicit names
or IPs of the nodes to join to. In this section we look at other TLS-compatible
options that lend themselves more to automation.

#### Autojoin with TLS servername

As of Vault 1.6.2, the simplest option might be to specify a
[`leader_tls_servername`](/vault/docs/configuration/storage/raft#leader_tls_servername)
in the [`retry_join`](/vault/docs/configuration/storage/raft#retry_join-stanza) stanza
which matches a [DNS
SAN](https://en.wikipedia.org/wiki/Subject_Alternative_Name) in the certificate.

Note that names in a certificate's DNS SAN don't actually have to be registered
in a DNS server. Your nodes may have no names found in DNS, while still
using certificate(s) that contain this shared `servername` in their DNS SANs.

#### Autojoin but constrain CIDR, list all possible IPs in certificate

If all the vault node IPs are assigned from a small subnet, e.g. a `/28`, it
becomes practical to put all the IPs that exist in that subnet into the IP SANs
of the TLS certificate the nodes will share.

The drawback here is that the cluster may someday outgrow the CIDR and changing
it may be a pain. For similar reasons this solution may be impractical when
using non-voting nodes and dynamically scaling clusters.

#### Load balancer instead of autojoin

Most Vault instances are going to have a load balancer (LB) between clients and
the Vault nodes. In that case, the LB knows how to route traffic to working
Vault nodes, and there's no need for auto-join: we can just use
[`retry_join`](/vault/docs/configuration/storage/raft#retry_join-stanza) with the LB
address as the target.

One potential issue here: some users want a public facing LB for clients to
connect to Vault, but aren't comfortable with Vault internal traffic
egressing from the internal network it normally runs on.

## Outage recovery

### Quorum maintained

This section outlines the steps to take when a single server or multiple servers
are in a failed state but quorum is still maintained. This means the remaining
alive servers are still operational, can elect a leader, and are able to process
write requests.

If the failed server is recoverable, the best option is to bring it back online
and have it reconnect to the cluster with the same host address. This will return
the cluster to a fully healthy state.

If this is impractical, you need to remove the failed server. Usually, you can
issue a [`remove-peer`](/vault/docs/commands/operator/raft#remove-peer) command to
remove the failed server if it's still a member of the cluster.

If the [`remove-peer`](/vault/docs/commands/operator/raft#remove-peer) command isn't
possible or you'd rather manually re-write the cluster membership a
[`raft/peers.json`](#manual-recovery-using-peers-json) file can be written to
the configured data directory.

### Quorum lost

In the event that multiple servers are lost, causing a loss of quorum and a
complete outage, partial recovery is still possible.

If the failed servers are recoverable, the best option is to bring them back
online and have them reconnect to the cluster using the same host addresses.
This will return the cluster to a fully healthy state.

If the failed servers are not recoverable, partial recovery is possible using
data on the remaining servers in the cluster. There may be data loss in this
situation because multiple servers were lost, so information about what's
committed could be incomplete. The recovery process implicitly commits all
outstanding Raft log entries, so it's also possible to commit data that was
uncommitted before the failure.

See the section below on manual recovery using
[`peers.json`](#manual-recovery-using-peers-json) for details of the recovery
procedure. You include only the remaining servers in the
[`peers.json`](#manual-recovery-using-peers-json) recovery file. The
cluster should be able to elect a leader once the remaining servers are all
restarted with an identical
[`peers.json`](#manual-recovery-using-peers-json) configuration.

Any servers you introduce later can be fresh with totally clean data
directories and joined using Vault's join command.

In extreme cases, it should be possible to recover with just a single remaining
server by starting that single server with itself as the only peer in the
[`peers.json`](#manual-recovery-using-peers-json) recovery file.

### Manual recovery using peers.json

Using `raft/peers.json` for recovery can cause uncommitted Raft log entries to be
implicitly committed, so this should only be used after an outage where no other
option is available to recover a lost server. Make sure you don't have any
automated processes that will put the peers file in place on a periodic basis.

To begin, stop all remaining servers.

The next step is to go to the [configured data
path](/vault/docs/configuration/storage/raft/#path) of each Vault server. Inside that
directory, there will be a `raft/` sub-directory. We need to create a
`raft/peers.json` file. The file should be formatted as a JSON array containing
the node ID, `address:port`, and suffrage information of each Vault server you
wish to be in the cluster:

```json
[
  {
    "id": "node1",
    "address": "node1.vault.local:8201",
    "non_voter": false
  },
  {
    "id": "node2",
    "address": "node2.vault.local:8201",
    "non_voter": false
  },
  {
    "id": "node3",
    "address": "node3.vault.local:8201",
    "non_voter": false
  }
]
```

- `id` `(string: <required>)` - Specifies the node ID of the server. This can be
  found in the config file, or inside the `node-id` file in the server's data
  directory if it was auto-generated.
- `address` `(string: <required>)` - Specifies the host and port of the server. The
  port is the server's cluster port.
- `non_voter` `(bool: <false>)` - This controls whether the server is a non-voter.
  If omitted, it will default to false, which is typical for most clusters. This
  is an enterprise only feature.

Create entries for all servers. You must confirm that servers you do not
include here have indeed failed and will not later rejoin the cluster. Ensure
that this file is the same across all remaining server nodes.

At this point, you can restart all the remaining servers. The cluster should be
in an operable state again. One of the nodes should claim leadership and become
active.

### Other recovery methods

For other, non-quorum related recovery [Vault's
recovery](/vault/docs/concepts/recovery-mode/) mode can be used.

layout  docs page title  Integrated Storage description  Learn about the integrated raft storage in Vault         Integrated storage  Vault supports a number of storage options for the durable storage of Vault s information  As of Vault 1 4 an Integrated Storage option is offered  This storage backend does not rely on any third party systems  implements high availability semantics  supports Enterprise Replication features  and provides backup restore workflows   The option stores Vault s data on the server s filesystem and uses a consensus protocol to replicate data to each server in the cluster  More information on the internals of Integrated Storage can be found in the  Integrated Storage internals documentation   vault docs internals integrated storage    Additionally  the  Configuration   vault docs configuration storage raft   docs can help in configuring Vault to use Integrated Storage   The sections below go into various details on how to operate Vault with Integrated Storage      Server to Server communication  Once nodes are joined to one another they begin to communicate using mTLS over Vault s cluster port  The cluster port defaults to  8201   The TLS information is exchanged at join time and is rotated on a cadence   A requirement for Integrated Storage is that the   cluster addr    vault docs concepts ha per node cluster address  configuration option is set  This allows Vault to assign an address to the node ID at join time      Cluster membership  This section will outline how to bootstrap and manage a cluster of Vault nodes running Integrated Storage   Integrated Storage is bootstrapped during the  initialization process   vault tutorials getting started getting started deploy initializing the vault   and results in a cluster of size 1  Depending on the  desired deployment size   vault docs internals integrated storage  deployment table   nodes can be joined to the active Vault node       Joining nodes  Joining is the process of taking an uninitialized Vault node and making it a member of an existing cluster  In order to authenticate the new node to the cluster it must use the same seal mechanism  If using a Auto Unseal the node must be configured to use the same KMS provider and Key as the cluster it s attempting to join  If using a Shamir seal the unseal keys must be provided to the new node before the join process can complete  Once a node has successfully joined  data from the active node can begin to replicate to it  Once a node has been joined it cannot be re joined to a different cluster   You can either join the node automatically via the config file or manually through the API  both methods described below   When joining a node  the API address of the leader node must be used  We recommend setting the   api addr    vault docs concepts ha direct access  configuration option on all nodes to make joining simpler   Always join nodes to a cluster one at a time and wait for the node to become healthy and  if applicable  a voter before continuing to add more nodes  The status of a node can be verified by performing a   list peers    vault docs commands operator raft list peers  command or by checking the   autopilot state    vault docs commands operator raft autopilot state          retry join  configuration  This method enables setting one  or more  target leader nodes in the config file  When an uninitialized Vault server starts up it will attempt to join each potential leader that is defined  retrying until successful  When one of the specified leaders become active this node will successfully join  When using Shamir seal  the joined nodes will still need to be unsealed manually  When using Auto Unseal the node will be able to join and unseal automatically   An example   retry join    vault docs configuration storage raft retry join stanza  config can be seen below      hcl storage  raft      path        var raft     node id    node3     retry join       leader api addr    https   node1 vault local 8200        retry join       leader api addr    https   node2 vault local 8200             Note  in each   retry join    vault docs configuration storage raft retry join stanza  stanza  you may provide a single   leader api addr    vault docs configuration storage raft leader api addr  or   auto join    vault docs configuration storage raft auto join  value  When a cloud   auto join    vault docs configuration storage raft auto join  configuration value is provided  Vault will use  go discover  https   github com hashicorp go discover  to automatically attempt to discover and resolve potential Raft leader addresses   Check the go discover  README  https   github com hashicorp go discover blob master README md  for details on the format of the   auto join    vault docs configuration storage raft auto join  value per cloud provider      hcl storage  raft      path        var raft     node id    node3     retry join       auto join    provider aws region eu west 1 tag key vault tag value     access key id     secret access key                 By default  Vault will attempt to reach discovered peers using HTTPS and port 8200  Operators may override these through the   auto join scheme    vault docs configuration storage raft auto join scheme  and   auto join port    vault docs configuration storage raft auto join port  fields respectively      hcl storage  raft      path        var raft     node id    node3     retry join       auto join    provider aws region eu west 1 tag key vault tag value     access key id     secret access key          auto join scheme    http      auto join port   8201                 Join from the CLI  Alternatively you can use the   join  CLI command   vault docs commands operator raft  join  or the API to join a node  The active node s API address will need to be specified      shell session   vault operator raft join https   node1 vault local 8200           Non Voting nodes  Enterprise only   Nodes that are joined to a cluster can be specified as non voters  A non voting node has all of Vault s data replicated to it  but does not contribute to the quorum count  This can be used in conjunction with  Performance Standby   vault docs enterprise performance standby   nodes to add read scalability to a cluster in cases where a high volume of reads to servers are needed      shell session   vault operator raft join  non voter https   node1 vault local 8200          Removing peers  Removing a peer node is a necessary step when you no longer want the node in the cluster  This could happen if the node is rotated for a new one  the hostname permanently changes and can no longer be accessed  you re attempting to shrink the size of the cluster  or for many other reasons  Removing the peer will ensure the cluster stays at the desired size  and that quorum is maintained   To remove the peer you can issue a   remove peer    vault docs commands operator raft remove peer  command and provide the node ID you wish to remove      shell session   vault operator raft remove peer node1 Peer removed successfully            Re joining after removal  If you have used  remove peer  to remove a node from the Raft cluster  but you later want to have this same node re join the cluster  you will need to delete any existing Raft data on the removed node before adding it back to the cluster  This will involve stopping the Vault process  deleting the data directory containing Raft data  and then restarting the Vault process       Listing peers  To see the current peer set for the cluster you can issue a   list peers    vault docs commands operator raft list peers  command  All the voting nodes that are listed here contribute to the quorum and a majority must be alive for Integrated Storage to continue to operate      shell session   vault operator raft list peers Node     Address                   State       Voter                                                      node1    node1 vault local 8201    follower    true node2    node2 vault local 8201    follower    true node3    node3 vault local 8201    leader      true         Integrated storage and TLS  We ve glossed over some details in the above sections on bootstrapping clusters  The instructions are sufficient for most cases  but some users have run into problems when using auto join and TLS in conjunction with things like auto scaling  The issue is that  go discover  https   github com hashicorp go discover  on most platforms returns IPs  not hostnames   and because the IPs aren t knowable in advance  the TLS certificates used to secure the Vault API port don t contain these IPs in their IP SANs       Vault networking recap  Before we explore solutions to this problem  let s recapitulate how Vault nodes speak to one another   Vault exposes two TCP ports   the API port   vault docs configuration api addr  and  the cluster port   vault docs configuration cluster addr    The API port is where clients send their Vault HTTP requests   For a single node Vault cluster you don t worry about a cluster port as it won t be used   When you have multiple nodes  you also need a cluster port  This is used by Vault nodes to issue RPCs to one another  e g  to forward requests from a standby node to the active node  or when Raft is in use  to handle leader election and replication of stored data   The cluster port is secured using a TLS certificate that the Vault active node generates internally  It s clear how this can work when not using integrated storage  every node has at least read access to storage  so once the active node has persisted the certificate  the standby nodes can fetch it  and all agree on how cluster traffic should be encrypted   It s less clear how this works with Integrated Storage  as there is a chicken and egg problem  Nodes don t have a shared view of storage until the raft cluster has been formed  but we re trying to form the raft cluster  To solve this problem  a Vault node must speak to another Vault node using the API port instead of the cluster port  This is currently the only situation in which Vault Community Edition does this  Vault Enterprise also does something similar when setting up replication       node2  wants to join the cluster  so issues challenge API request to existing member  node1     node1  replies to challenge request with  1  an encrypted random UUID and  2  seal config    node2  must decrypt UUID using seal  if using auto unseal can do it directly  if using Shamir must wait for user to provide enough unseal keys to perform decryption    node2  sends decrypted UUID back to  node1  using answer API    node1  sees  node2  can be trusted  since it has seal access  and replies with a bootstrap package which includes the cluster TLS certificate and private key    node2  gets sent a raft snapshot over the cluster port  After this procedure the new node will never again send traffic to the API port  All subsequent inter node communication will use the cluster port     Raft Join Process   img raft join detailed png       Assisted raft join techniques  The simplest option is to do it by hand  issue   raft join    vault docs commands operator raft join  commands specifying the explicit names or IPs of the nodes to join to  In this section we look at other TLS compatible options that lend themselves more to automation        Autojoin with TLS servername  As of Vault 1 6 2  the simplest option might be to specify a   leader tls servername    vault docs configuration storage raft leader tls servername  in the   retry join    vault docs configuration storage raft retry join stanza  stanza which matches a  DNS SAN  https   en wikipedia org wiki Subject Alternative Name  in the certificate   Note that names in a certificate s DNS SAN don t actually have to be registered in a DNS server  Your nodes may have no names found in DNS  while still using certificate s  that contain this shared  servername  in their DNS SANs        Autojoin but constrain CIDR  list all possible IPs in certificate  If all the vault node IPs are assigned from a small subnet  e g  a   28   it becomes practical to put all the IPs that exist in that subnet into the IP SANs of the TLS certificate the nodes will share   The drawback here is that the cluster may someday outgrow the CIDR and changing it may be a pain  For similar reasons this solution may be impractical when using non voting nodes and dynamically scaling clusters        Load balancer instead of autojoin  Most Vault instances are going to have a load balancer  LB  between clients and the Vault nodes  In that case  the LB knows how to route traffic to working Vault nodes  and there s no need for auto join  we can just use   retry join    vault docs configuration storage raft retry join stanza  with the LB address as the target   One potential issue here  some users want a public facing LB for clients to connect to Vault  but aren t comfortable with Vault internal traffic egressing from the internal network it normally runs on      Outage recovery      Quorum maintained  This section outlines the steps to take when a single server or multiple servers are in a failed state but quorum is still maintained  This means the remaining alive servers are still operational  can elect a leader  and are able to process write requests   If the failed server is recoverable  the best option is to bring it back online and have it reconnect to the cluster with the same host address  This will return the cluster to a fully healthy state   If this is impractical  you need to remove the failed server  Usually  you can issue a   remove peer    vault docs commands operator raft remove peer  command to remove the failed server if it s still a member of the cluster   If the   remove peer    vault docs commands operator raft remove peer  command isn t possible or you d rather manually re write the cluster membership a   raft peers json    manual recovery using peers json  file can be written to the configured data directory       Quorum lost  In the event that multiple servers are lost  causing a loss of quorum and a complete outage  partial recovery is still possible   If the failed servers are recoverable  the best option is to bring them back online and have them reconnect to the cluster using the same host addresses  This will return the cluster to a fully healthy state   If the failed servers are not recoverable  partial recovery is possible using data on the remaining servers in the cluster  There may be data loss in this situation because multiple servers were lost  so information about what s committed could be incomplete  The recovery process implicitly commits all outstanding Raft log entries  so it s also possible to commit data that was uncommitted before the failure   See the section below on manual recovery using   peers json    manual recovery using peers json  for details of the recovery procedure  You include only the remaining servers in the   peers json    manual recovery using peers json  recovery file  The cluster should be able to elect a leader once the remaining servers are all restarted with an identical   peers json    manual recovery using peers json  configuration   Any servers you introduce later can be fresh with totally clean data directories and joined using Vault s join command   In extreme cases  it should be possible to recover with just a single remaining server by starting that single server with itself as the only peer in the   peers json    manual recovery using peers json  recovery file       Manual recovery using peers json  Using  raft peers json  for recovery can cause uncommitted Raft log entries to be implicitly committed  so this should only be used after an outage where no other option is available to recover a lost server  Make sure you don t have any automated processes that will put the peers file in place on a periodic basis   To begin  stop all remaining servers   The next step is to go to the  configured data path   vault docs configuration storage raft  path  of each Vault server  Inside that directory  there will be a  raft   sub directory  We need to create a  raft peers json  file  The file should be formatted as a JSON array containing the node ID   address port   and suffrage information of each Vault server you wish to be in the cluster      json            id    node1        address    node1 vault local 8201        non voter   false               id    node2        address    node2 vault local 8201        non voter   false               id    node3        address    node3 vault local 8201        non voter   false               id    string   required      Specifies the node ID of the server  This can be   found in the config file  or inside the  node id  file in the server s data   directory if it was auto generated     address    string   required      Specifies the host and port of the server  The   port is the server s cluster port     non voter    bool   false      This controls whether the server is a non voter    If omitted  it will default to false  which is typical for most clusters  This   is an enterprise only feature   Create entries for all servers  You must confirm that servers you do not include here have indeed failed and will not later rejoin the cluster  Ensure that this file is the same across all remaining server nodes   At this point  you can restart all the remaining servers  The cluster should be in an operable state again  One of the nodes should claim leadership and become active       Other recovery methods  For other  non quorum related recovery  Vault s recovery   vault docs concepts recovery mode   mode can be used


vault page title Migration checklist Migration checklist Tip title This is a decision making checklist layout docs Use this checklist for decision making related to migrating your Vault deployment to Integrated Storage

Answer:
---
layout: docs
page_title: Migration checklist
description: Use this checklist for decision making related to migrating your Vault deployment to Integrated Storage.
---

# Migration checklist

<Tip title="This is a decision-making checklist">

The purpose of this checklist is not to walk you through the storage
migration steps. This content provides a quick self-check whether it is your
best interest to migrate your Vault storage from an external system to
Integrated Storage.

</Tip>

## Who should use this checklist?

Integrated Storage is a recommended storage option, made available in
Vault 1.4. Vault continues to also support other storage solutions
like Consul.

You should use this checklist if you are operating a Vault deployment backed
by external storage like Consul, and you are considering migration to
Integrated Storage.

## Understand architectural differences

It is important that you understand the differences between operating Vault
with external storage and operating with Integrated Storage. The following
sections detail key differences in architecture between Vault with Consul
storage, and Vault with Integrated Storage to help inform your decision.

### Reference architecture with Consul

The recommended number of Vault instances is **3** in a cluster which connects
to a Consul cluster which may have **5** or more nodes as shown in the diagram.

A total of 8 virtual machines hosts this Vault highly available architecture.

<ImageConfig hideBorder>

![Reference Diagram](/img/diagram-vault-ra-3-az.png)

</ImageConfig>

The processing requirements depend on the encryption and messaging workloads.
Memory requirements are dependant on the total size of secrets stored in
memory. The Vault server itself has minimal storage requirements, but
the Consul nodes should have a high-performance physical storage system.

### Reference architecture with Integrated Storage

The recommended number of Vault instances is **5** in a cluster. In a single HA
cluster, all Vault nodes share the data while an active node holds the lock;
therefore, only the active node has write access. To achieve n-2 redundancy,
(meaning that the cluster can still function after losing 2 nodes),
an ideal size for a Vault HA cluster is 5 nodes.

<Tip title="More deployment details in the documentation">

Refer to the [Integrated
Storage](/vault/docs/internals/integrated-storage#deployment-table)
documentation for more deployment details.

</Tip>

<ImageConfig hideBorder>

![Reference Diagram Details](/img/diagram-vault-integrated-ra-3_az.png)

</ImageConfig>

Because the data gets persisted on the same host, the Vault server should be
hosted on a relatively high-performance hard disk system.

## Consul vs. Integrated Storage

The Integrated Storage eliminates the need for external storage; therefore,
Vault is the only software you need to stand up a cluster. This indicates that
the host machine must have disk capacity in an amount equal or
greater to that of the existing external storage backend.

### System requirements comparison

The fundamental difference between Vault's Integrated Storage and Consul is
that the Integrated Storage stores everything on disk while [Consul
KV](/consul/docs/dynamic-app-config/kv) stores everything in its memory
which impacts the host's RAM.

#### Machine sizes for Vault - Consul as its storage backend

It is recommended to avoid hosting Consul on an instance with burstable CPU.

| Size  | CPU      | Memory       | Disk  | Typical Cloud Instance Types              |
| ----- | -------- | ------------ | ----- | ----------------------------------------- |
| Small | 2 core   | 4-8 GB RAM   | 25 GB | **AWS:** m5.large                         |
|       |          |              |       | **Azure:** Standard_D2_v3                 |
|       |          |              |       | **GCE:** n1-standard-2, n1-standard-4     |
| Large | 4-8 core | 16-32 GB RAM | 50 GB | **AWS:** m5.xlarge, m5.2xlarge            |
|       |          |              |       | **Azure:** Standard_D4_v3, Standard_D8_v3 |
|       |          |              |       | **GCE:** n1-standard-8, n1-standard-16    |

#### Machine sizes for Vault with Integrated Storage

| Size  | CPU      | Memory       | Disk   | Typical Cloud Instance Types               |
| ----- | -------- | ------------ | ------ | ------------------------------------------ |
| Small | 2 core   | 8-16 GB RAM  | 100 GB | **AWS:** m5.large, m5.xlarge               |
|       |          |              |        | **Azure:** Standard_D2_v3, Standard_D4_v3  |
|       |          |              |        | **GCE:** n2-standard-2, n2-standard-4      |
| Large | 4-8 core | 32-64 GB RAM | 200 GB | **AWS:** m5.2xlarge, m5.4xlarge            |
|       |          |              |        | **Azure:** Standard_D8_v3, Standard_D16_v3 |
|       |          |              |        | **GCE:** n2-standard-8, n2-standard-16     |

If many secrets are being generated or rotated frequently, this information will
need to be flushed to the disk often. Therefore, the infrastructure should have
a relatively high-performance hard disk system when using the integrated
storage.

<Note title="A note about the importance of IOPS">

 Vault's Integrated Storage is disk-bound; therefore, care should be taken when planning storage volume size and performance. For cloud providers, IOPS can be dependent on volume size and/or provisioned IOPS. It is recommended to provision IOPS and avoid burstable IOPS. Monitoring of IOPS performance should be implemented in order to tune the storage volume to the IOPS load.

</Note>

### Performance considerations

Because Consul KV is memory-bound, it is necessary to take a snapshot frequently.
However, Vault's Integrated Storage persists everything on the disk which eliminates
the need for such frequent snapshot operations. Take snapshots to back up the data
so that you can restore them in case of data loss. This reduces the performance cost
introduced by the frequent snapshot operations.

In considering disk performance, since Vault data changes are immediately written to disk,
rather than in batched snapshots as Consul does, it is important to monitor IOPS as well
as disk queues to limit storage bottlenecks.

### Inspect Vault data

Inspection of Vault data differs considerably from the `consul kv` commands used
to inspect Consul's KV store.
Consult the [Inspect Data in Integrated Storage](/vault/tutorials/monitoring/inspect-data-integrated-storage)
tutorial to learn more about querying Integrated Storage data.

### Summary

The table below highlights the differences between Consul and integrated
storage.

| Consideration       | Consul as storage backend                                                  | Vault Integrated Storage                                                                     |
| ------------------- | -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| System requirement  | Memory optimized machine                                                   | Storage optimized high IOPS machine                                                          |
| Data snapshot       | Frequent snapshots                                                         | Normal data backup strategy                                                                  |
| Snapshot automation | Snapshot agent (**Consul Enterprise only**)                                | Automatic snapshot (**Vault Enterprise v1.6.0 and later**)                                   |
| Data inspection     | [Online, use `consul kv` command](/vault/tutorials/monitoring/inspecting-data-consul) | [Offline, requires using recovery mode](/vault/tutorials/monitoring/inspect-data-integrated-storage) |
| Autopilot           | Supported                                                                  | Supported (**Vault 1.7.0 and later**)                                  |

## Self-check questions

- [ ] Where is the product expertise?
  - [ ] Do you already have Consul expertise?
  - [ ] Are you concerned about lack of Consul knowledge?
- [ ] Do you experience any technical issues with Consul?
- [ ] What motivates the data migration from the current storage to Integrated Storage?
  - [ ] Reduce the operational overhead?
  - [ ] Reduce the number of machines to run?
  - [ ] Reduce the cloud infrastructure cost?
- [ ] Do you have a staging environment where you can run production loads and verify that everything works as you expect?
- [ ] Have you thought through the storage backup process or workflow after migrating to the Integrated Storage?
- [ ] Do you currently rely heavily on using Consul to inspect Vault data?

## Tutorials

If you are ready to migrate the current storage backend to Integrated Storage,
refer to the [Storage Migration Tutorial - Consul to Integrated Storage](/vault/tutorials/raft/raft-migration).

To deploy a new cluster with Integrated Storage, refer to the [Vault HA Cluster
with Integrated Storage](/vault/tutorials/raft/raft-storage) tutorial.

layout  docs page title  Migration checklist description  Use this checklist for decision making related to migrating your Vault deployment to Integrated Storage         Migration checklist   Tip title  This is a decision making checklist    The purpose of this checklist is not to walk you through the storage migration steps  This content provides a quick self check whether it is your best interest to migrate your Vault storage from an external system to Integrated Storage     Tip      Who should use this checklist   Integrated Storage is a recommended storage option  made available in Vault 1 4  Vault continues to also support other storage solutions like Consul   You should use this checklist if you are operating a Vault deployment backed by external storage like Consul  and you are considering migration to Integrated Storage      Understand architectural differences  It is important that you understand the differences between operating Vault with external storage and operating with Integrated Storage  The following sections detail key differences in architecture between Vault with Consul storage  and Vault with Integrated Storage to help inform your decision       Reference architecture with Consul  The recommended number of Vault instances is   3   in a cluster which connects to a Consul cluster which may have   5   or more nodes as shown in the diagram   A total of 8 virtual machines hosts this Vault highly available architecture    ImageConfig hideBorder     Reference Diagram   img diagram vault ra 3 az png     ImageConfig   The processing requirements depend on the encryption and messaging workloads  Memory requirements are dependant on the total size of secrets stored in memory  The Vault server itself has minimal storage requirements  but  the Consul nodes should have a high performance physical storage system       Reference architecture with Integrated Storage  The recommended number of Vault instances is   5   in a cluster  In a single HA cluster  all Vault nodes share the data while an active node holds the lock  therefore  only the active node has write access  To achieve n 2 redundancy   meaning that the cluster can still function after losing 2 nodes   an ideal size for a Vault HA cluster is 5 nodes    Tip title  More deployment details in the documentation    Refer to the  Integrated Storage   vault docs internals integrated storage deployment table  documentation for more deployment details     Tip    ImageConfig hideBorder     Reference Diagram Details   img diagram vault integrated ra 3 az png     ImageConfig   Because the data gets persisted on the same host  the Vault server should be hosted on a relatively high performance hard disk system      Consul vs  Integrated Storage  The Integrated Storage eliminates the need for external storage  therefore  Vault is the only software you need to stand up a cluster  This indicates that the host machine must have disk capacity in an amount equal or greater to that of the existing external storage backend       System requirements comparison  The fundamental difference between Vault s Integrated Storage and Consul is that the Integrated Storage stores everything on disk while  Consul KV   consul docs dynamic app config kv  stores everything in its memory which impacts the host s RAM        Machine sizes for Vault   Consul as its storage backend  It is recommended to avoid hosting Consul on an instance with burstable CPU     Size    CPU        Memory         Disk    Typical Cloud Instance Types                                                                                Small   2 core     4 8 GB RAM     25 GB     AWS    m5 large                                                                         Azure    Standard D2 v3                                                                 GCE    n1 standard 2  n1 standard 4         Large   4 8 core   16 32 GB RAM   50 GB     AWS    m5 xlarge  m5 2xlarge                                                            Azure    Standard D4 v3  Standard D8 v3                                                 GCE    n1 standard 8  n1 standard 16            Machine sizes for Vault with Integrated Storage    Size    CPU        Memory         Disk     Typical Cloud Instance Types                                                                                Small   2 core     8 16 GB RAM    100 GB     AWS    m5 large  m5 xlarge                                                                Azure    Standard D2 v3  Standard D4 v3                                                   GCE    n2 standard 2  n2 standard 4          Large   4 8 core   32 64 GB RAM   200 GB     AWS    m5 2xlarge  m5 4xlarge                                                             Azure    Standard D8 v3  Standard D16 v3                                                  GCE    n2 standard 8  n2 standard 16        If many secrets are being generated or rotated frequently  this information will need to be flushed to the disk often  Therefore  the infrastructure should have a relatively high performance hard disk system when using the integrated storage    Note title  A note about the importance of IOPS     Vault s Integrated Storage is disk bound  therefore  care should be taken when planning storage volume size and performance  For cloud providers  IOPS can be dependent on volume size and or provisioned IOPS  It is recommended to provision IOPS and avoid burstable IOPS  Monitoring of IOPS performance should be implemented in order to tune the storage volume to the IOPS load     Note       Performance considerations  Because Consul KV is memory bound  it is necessary to take a snapshot frequently  However  Vault s Integrated Storage persists everything on the disk which eliminates the need for such frequent snapshot operations  Take snapshots to back up the data so that you can restore them in case of data loss  This reduces the performance cost introduced by the frequent snapshot operations   In considering disk performance  since Vault data changes are immediately written to disk  rather than in batched snapshots as Consul does  it is important to monitor IOPS as well as disk queues to limit storage bottlenecks        Inspect Vault data  Inspection of Vault data differs considerably from the  consul kv  commands used to inspect Consul s KV store  Consult the  Inspect Data in Integrated Storage   vault tutorials monitoring inspect data integrated storage  tutorial to learn more about querying Integrated Storage data       Summary  The table below highlights the differences between Consul and integrated storage     Consideration         Consul as storage backend                                                    Vault Integrated Storage                                                                                System requirement    Memory optimized machine                                                     Storage optimized high IOPS machine                                                              Data snapshot         Frequent snapshots                                                           Normal data backup strategy                                                                      Snapshot automation   Snapshot agent    Consul Enterprise only                                     Automatic snapshot    Vault Enterprise v1 6 0 and later                                          Data inspection        Online  use  consul kv  command   vault tutorials monitoring inspecting data consul     Offline  requires using recovery mode   vault tutorials monitoring inspect data integrated storage      Autopilot             Supported                                                                    Supported    Vault 1 7 0 and later                                           Self check questions        Where is the product expertise          Do you already have Consul expertise          Are you concerned about lack of Consul knowledge        Do you experience any technical issues with Consul        What motivates the data migration from the current storage to Integrated Storage          Reduce the operational overhead          Reduce the number of machines to run          Reduce the cloud infrastructure cost        Do you have a staging environment where you can run production loads and verify that everything works as you expect        Have you thought through the storage backup process or workflow after migrating to the Integrated Storage        Do you currently rely heavily on using Consul to inspect Vault data      Tutorials  If you are ready to migrate the current storage backend to Integrated Storage  refer to the  Storage Migration Tutorial   Consul to Integrated Storage   vault tutorials raft raft migration    To deploy a new cluster with Integrated Storage  refer to the  Vault HA Cluster with Integrated Storage   vault tutorials raft raft storage  tutorial


vault Client count calculation Technical overview of client count calculations in Vault Vault provides usage telemetry for the number of clients based on the number of layout docs page title Client count calculation

Answer:
---
layout: docs
page_title: Client count calculation
description: |-
  Technical overview of client count calculations in Vault
---

# Client count calculation

Vault provides usage telemetry for the number of clients based on the number of
unique entity assignments within a Vault cluster over a given billing period:

- Standard entity assignments based on authentication method for active entities.
- Constructed entity assignments for active non-entity tokens, including batch
  tokens created by performance standby nodes.
- Certificate entity assignments for ACME connections.
- Secrets being synced to at least one sync destination.

```markdown
CLIENT_COUNT_PER_CLUSTER = UNIQUE_STANDARD_ENTITIES +
                           UNIQUE_CONSTRUCTED_ENTITIES +
                           UNIQUE_CERTIFICATE_ENTITIES +
                           UNIQUE_SYNCED_SECRETS
```

Vault does not aggregate or de-duplicate clients across clusters, but all logs
and precomputed reports are included in DR replication.

## How Vault tracks clients

Each time a client authenticates, Vault checks whether the corresponding entity
ID has already been recorded in the client log as active for the current month:

- **If no record exists**, Vault adds an entry for the entity ID.
- If a record exists but the entity was last active **prior to the current month**,
  Vault adds a new entry to the client record for the entity ID.
- If a record exists and the entity was last active **within the current month**,
  Vault does not add a new entry to the client record for the entity ID.

For example:

- Two non-entity tokens under the same namespace, with the same alias name and
  policy assignment receive the same entity assignment and are only counted
  **once**.
- Two authentication requests from a single ACME client for the same certificate
  identifiers from different mounts receive the same entity assignments and
  are counted **once**.
- An application authenticating with AppRole receive the same entity assignment
  every time and only counted **once**.

At the **end of each month**, Vault pre-computes reports for each cluster on the
number of active entities, per namespace, for each time period within the
configured retention period. By de-duplicating records from the current month
against records for the previous month, Vault ensures entities that remain
active within every calendar month are only counted once for the year.

The deduplication process has two additional consequences:

1. Detailed reporting lags by 1 month at the start of the billing period.
1. Billing period reports that include the current month must use an
   approximation for the number of new clients in the current month.

## How Vault approximates current-month client count

Vault approximates client count for the current month using a
[hyperloglog algorithm](https://en.wikipedia.org/wiki/HyperLogLog) that looks
at the difference between the cardinalities of:

- the number of clients across the **entire** billing period, and
- the number of clients across the billing period **excluding** clients from the current month.

The approximation algorithm uses the
[axiomhq](https://github.com/axiomhq/hyperloglog) library with fourteen
registers and sparse representations (when applicable). The multiset for the
calculation is the total number of clients within a billing period, and the
accuracy estimate for the approximation decreases as the difference between the
number of clients in the current month and the number of clients in the billing
period increases.

### Testing verification for client count approximations

Given `CM` as the number of clients for the current month and `BP` as the number
of clients in the billing period, we found that the approximation becomes
increasingly imprecise as:

- the difference between `BC` and `CM` increases
- the value of `CM` approaches zero.
- the number of months in the billing period increase.

The maximum observed error rate
(`ER = (FOUND_NEW_CLIENTS / EXPECTED_NEW_CLIENTS)`) was 30% for 10,000 clients
or less, with an error rate of 5 – 10% in the average case.

For the purposes of predictive analysis, the following tables list a random
sample the values we found during testing for `CM`, `BP`, and `ER`.

<Tabs>

<Tab heading="Single-month tests">

| Current month (`CM`) | Billing period (`BP`) | Error rate (`ER`) |
| :-----------------: | :------------------: | :---------------: |
| 7                   | 10                   | 0%                |
| 20                  | 600                  | 0%                |
| 20                  | 1000                 | 0%                |
| 20                  | 6000                 | 10%               |
| 20                  | 10000                | 10%               |
| 200                 | 600                  | 0%                |
| 200                 | 10000                | 7%                |
| 400                 | 6000                 | 5%                |
| 2000                | 10000                | 4%                |

</Tab>

<Tab heading="Multi-month / multi-segment tests">

| Current month (`CM`) | Billing period (`BP`) | Error rate (`ER`) |
| :-----------------: | :------------------: | :---------------: |
| 20                  | 15                   |      0%           |
| 20                  | 100                  |      0%           |
| 20                  | 1000                 |      0%           |
| 20                  | 10000                |      30%          |
| 200                 | 10000                |      6%           |
| 2000                | 10000                |      2%           |

</Tab>

</Tabs>

## Resource costs for client computation

In addition to the storage used for storing the pre-computed reports, each
active entity in the client log consumes a few bytes of storage. As a safety
measure against runaway storage growth, Vault limits the number of entity
records to 656,000 per month, but typical storage costs are much less.

On average, 1000 monthly active entities requires 3.0 MiB of storage capacity
over the default 48-month retention period.

@include "content-footer-title.mdx"

<Tabs>

<Tab heading="Related concepts">

    Clients and entities

    Client count FAQ

</Tab>

<Tab heading="Related API docs">

    Client Count API

    Internal counters API

</Tab>

<Tab heading="Related tutorials">

      Vault Usage Metrics in Vault UI

    KMIP Client metrics

</Tab>

<Tab heading="Other resources">

    Accuracy estimates for the axiomhq hyperloglog library

    Blog post:
      Onboarding Applications to Vault Using Terraform: A Practical Guide

</Tab>

</Tabs>

layout  docs page title  Client count calculation description       Technical overview of client count calculations in Vault        Client count calculation  Vault provides usage telemetry for the number of clients based on the number of unique entity assignments within a Vault cluster over a given billing period     Standard entity assignments based on authentication method for active entities    Constructed entity assignments for active non entity tokens  including batch   tokens created by performance standby nodes    Certificate entity assignments for ACME connections    Secrets being synced to at least one sync destination      markdown CLIENT COUNT PER CLUSTER   UNIQUE STANDARD ENTITIES                              UNIQUE CONSTRUCTED ENTITIES                              UNIQUE CERTIFICATE ENTITIES                              UNIQUE SYNCED SECRETS      Vault does not aggregate or de duplicate clients across clusters  but all logs and precomputed reports are included in DR replication      How Vault tracks clients  Each time a client authenticates  Vault checks whether the corresponding entity ID has already been recorded in the client log as active for the current month       If no record exists    Vault adds an entry for the entity ID    If a record exists but the entity was last active   prior to the current month      Vault adds a new entry to the client record for the entity ID    If a record exists and the entity was last active   within the current month      Vault does not add a new entry to the client record for the entity ID   For example     Two non entity tokens under the same namespace  with the same alias name and   policy assignment receive the same entity assignment and are only counted     once      Two authentication requests from a single ACME client for the same certificate   identifiers from different mounts receive the same entity assignments and   are counted   once      An application authenticating with AppRole receive the same entity assignment   every time and only counted   once     At the   end of each month    Vault pre computes reports for each cluster on the number of active entities  per namespace  for each time period within the configured retention period  By de duplicating records from the current month against records for the previous month  Vault ensures entities that remain active within every calendar month are only counted once for the year   The deduplication process has two additional consequences   1  Detailed reporting lags by 1 month at the start of the billing period  1  Billing period reports that include the current month must use an    approximation for the number of new clients in the current month      How Vault approximates current month client count  Vault approximates client count for the current month using a  hyperloglog algorithm  https   en wikipedia org wiki HyperLogLog  that looks at the difference between the cardinalities of     the number of clients across the   entire   billing period  and   the number of clients across the billing period   excluding   clients from the current month   The approximation algorithm uses the  axiomhq  https   github com axiomhq hyperloglog  library with fourteen registers and sparse representations  when applicable   The multiset for the calculation is the total number of clients within a billing period  and the accuracy estimate for the approximation decreases as the difference between the number of clients in the current month and the number of clients in the billing period increases       Testing verification for client count approximations  Given  CM  as the number of clients for the current month and  BP  as the number of clients in the billing period  we found that the approximation becomes increasingly imprecise as     the difference between  BC  and  CM  increases   the value of  CM  approaches zero    the number of months in the billing period increase   The maximum observed error rate   ER    FOUND NEW CLIENTS   EXPECTED NEW CLIENTS    was 30  for 10 000 clients or less  with an error rate of 5  ndash  10  in the average case   For the purposes of predictive analysis  the following tables list a random sample the values we found during testing for  CM    BP   and  ER     Tabs    Tab heading  Single month tests      Current month   CM     Billing period   BP     Error rate   ER                                                                          7                     10                     0                     20                    600                    0                     20                    1000                   0                     20                    6000                   10                    20                    10000                  10                    200                   600                    0                     200                   10000                  7                     400                   6000                   5                     2000                  10000                  4                      Tab    Tab heading  Multi month   multi segment tests      Current month   CM     Billing period   BP     Error rate   ER                                                                          20                    15                          0                20                    100                         0                20                    1000                        0                20                    10000                       30               200                   10000                       6                2000                  10000                       2                 Tab     Tabs      Resource costs for client computation  In addition to the storage used for storing the pre computed reports  each active entity in the client log consumes a few bytes of storage  As a safety measure against runaway storage growth  Vault limits the number of entity records to 656 000 per month  but typical storage costs are much less   On average  1000 monthly active entities requires 3 0 MiB of storage capacity over the default 48 month retention period    include  content footer title mdx    Tabs    Tab heading  Related concepts    ul     li       a href   vault docs concepts client count   Clients and entities  a      li     li       a href   vault docs concepts client count faq  Client count FAQ  a      li    ul    Tab    Tab heading  Related API docs    ul     li       a href   vault api docs system internal counters client count  Client Count API  a      li     li       a href   vault api docs system internal counters  Internal counters API  a      li    ul    Tab    Tab heading  Related tutorials    ul     li       a href   vault tutorials monitoring usage metrics         Vault Usage Metrics in Vault UI       a      li     li       a href   vault tutorials monitoring usage metrics  KMIP Client metrics  a      li    ul    Tab    Tab heading  Other resources    ul     li       a href  https   github com axiomhq hyperloglog readme  Accuracy estimates for the axiomhq hyperloglog library  a      li     li      Blog post   a href  https   www hashicorp com blog onboarding applications to vault using terraform a practical guide         Onboarding Applications to Vault Using Terraform  A Practical Guide       a      li    ul    Tab     Tabs


vault in Vault page title Clients and entities Clients and entities layout docs Technical overview covering the concept of clients entities and entity IDs

Answer:
---
layout: docs
page_title: Clients and entities
description: |-
  Technical overview covering the concept of clients, entities, and entity IDs
  in Vault
---

# Clients and entities

Anything that connects and authenticates to Vault to accomplish a task is a
**client**. For example, a user logging into a cluster to manage policies or a
machine-based system (application or cloud service) requesting a database token
are both considered clients.

![Vault Client Workflows](https://www.datocms-assets.com/2885/1617325020-valult-client-workflows.png)

While there are many different potential clients, the most common are:

1. **Human users** interacting directly with Vault.
1. **Applications and microservices**.
1. **Servers and platforms** like VMs, Docker containers, or Kubernetes pods.
1. **Orchestrators** like Nomad, Terraform, Ansible, ACME, and other continuous
   integration / continuous delivery (CI/CD) pipelines.
1. **Vault agents and proxies** that act on behalf of an application or
   microservice.

## Identity and entity assignment

Authorized clients can connect to Vault with a variety of authentication methods.

Authorization source       | AuthN method
-------------------------- | ---------------------------------
Externally managed or SSO  | Active Directory, LDAP, OIDC, JWT, GitHub, username+password
Platform- or server-based  | Kubernetes, AWS, GCP, Azure, Cert, Cloud Foundry
Self                       | AppRole, tokens with no associated authN path or role

![Vault client types](https://www.datocms-assets.com/2885/1617325030-vault-clients.png)

When a client authenticates, Vault assigns a unique identifier
(**client entity**) in the [Vault identity system](/vault/docs/secrets/identity)
based on the authentication method used or a previously assigned alias.
**Entity aliases** let clients authenticate with multiple methods but still be
associated with a single policy, share resources, and count as the same entity,
regardless of the authentication method used for a particular session.

## Standard entity assignments

@include "authn-names.mdx"

Each authentication method has a unique ID string that corresponds to a client
entity used for telemetry. For example, a microservice authenticating with
AppRole takes the associated role ID as the entity. If you are running at scale
and have multiple copies of the microservices using the same role id, the full
set of instances will share the same identifier.

As a result, it is critical that you configure different clients
(microservices, humans, applications, services, platforms, servers, or pipelines)
in a way that results in distinct clients having unique identifiers. For example,
the role IDs should be different **between** two microservices, MicroserviceA and
MicroServiceB, even if the **specific instances** of MicroServiceA and
MicroServiceB share a common role ID.

## Entity assignment with ACME

Vault treats all ACME connections that authenticate under the same certificate
identifier (domain) as the same **certificate entity** for client count
calculations.

For example:

- ACME client requests (from the same server or separate servers) for the same
  certificate identifier (a unique combination of CN, DNS, SANS and IP SANS)
  are treated as the same entity.
- If an ACME client makes a request for `a.test.com`, and subsequently makes a new
  request for `b.test.com` and `*.test.com` then two distinct entities will be created,
  one for `a.test.com` and another for the combination of `b.test.com` and `*.test.com`.
- Overlap of certificate identifiers from different ACME clients will be treated
  as the same entity e.g. if client 1 requests `a.test.com` and client 2 requests
  `a.test.com` a single entity is created for both requests.

## Secret sync clients
Vault can automatically update secrets in external destinations with [secret sync](/vault/docs/sync).
A secret that gets synced to one or more destinations is considered a **secret
sync client** for client count calculations.

Note that:

- Each synced secret is counted distinctly based on the path and namespace of
  the secret. If you have secrets at path `kv1/secret` and `kv2/secret`
  which are both synced, then two distinct secret syncs will be counted.
- A secret can be synced to multiple different destinations, and it will still
  only be counted as one secret sync. If `kv/secret` is synced to both Azure Key
  Vault and AWS Secret Manager, this will be counted as only one secret sync
  client.
- Secret sync clients are only created after you create an association between a
  secret and a store. If you create `kv/secret` and do not associate this secret
  with any destinations, it will not be counted as a secret sync client.
- Secret syncs clients are registered in Vault's client counting system so long
  as the sync is active. If you create `kv/secret` and associate it with a
  destination in January, update the secret in May, and then delete the secret
  in September, Vault will consider this client as having been seen throughout
  the entire period of January through September.

## Entity assignment with namespaces

A namespace represents a isolated, logical space within a single Vault
cluster and is typically used for administrative purposes.

When a client authenticates **within a given namespace**, Vault assigns the same
client entity to activities within any child namespaces because the namespaces
exist within the same larger scope.

When a client authenticates **across namespace boundaries**, Vault treats the
single client as two distinct entities because the client is operating
across different scopes with different policy assignments and resources.

For example:

- Different requests under parent and child namespaces from a single client
  authenticated under the **parent** namespace are assigned **the same entity
  ID**. All the client activities occur **within** the boundaries of the
  namespace referenced in the original authentication request.
- Different requests under parent and child namespaces from a single client
  authenticated under the **child** namespace are assigned **different entity
  IDs**. Some of the client activities occur **outside** the boundaries of the
  namespace referenced in the original authentication request.
- Requests by the same client to two different namespaces, NAMESPACEA
  and NAMESPACEB are assigned **different entity IDs**.

## Entity assignment with non-entity tokens

Vault uses tokens as the core method for authentication. You can use tokens to
authenticate directly, or use token [auth methods](/vault/docs/concepts/auth)
to dynamically generate tokens based on external identities.

When clients authenticate with the [token auth method](/vault/docs/auth/token)
**without** a client identity, the result is a **non-entity token**. For example,
a service might use the token authentication method to create a token for a user
whose explicit identity is unknown.

Ultimately, non-entity tokens trace back to a particular client or purpose so
Vault assigns unique entity IDs to non-entity tokens based on a combination of
the:

- assigned entity alias name (if present),
- associated policies, and
- namespace under which the token was created.

In **rare** cases, tokens may be created outside of the Vault identity system
**without** an associated entity or identity. Vault treats every unaffiliated
token as a unique client for production usage. We strongly discourage the use of
unaffiliated tokens and recommend that you always associate a token with an
entity alias and token role.

<Note title="Behavior change in Vault 1.9+">
  As of Vault 1.9, all non-entity tokens with the same namespace and policy
  assignments are treated as the same client entity. Prior to Vault 1.9, every
  non-entity token was treated as a unique client entity, which drastically
  inflated telemetry around client count.

  If you are using Vault 1.8 or earlier, and need to address client count
  inflation without upgrading, we recommend creating a
  [token role](/vault/api-docs/auth/token#create-update-token-role) with
  allowable entity aliases and assigning all tokens to an appropriate
  [role and entity alias name](/vault/api-docs/auth/token#create-token) before
  using them.
</Note>

@include "content-footer-title.mdx"

<Tabs>

<Tab heading="Related concepts">

    Client count calculation

    Client count FAQ

</Tab>

<Tab heading="Related tutorials">

    Identity: Entities and Groups

    Secure Multi-Tenancy with Namespaces

</Tab>

<Tab heading="Other resources">

    Article:
      Identity-based Security and Low-trust Networks

</Tab>

</Tabs>

layout  docs page title  Clients and entities description       Technical overview covering the concept of clients  entities  and entity IDs   in Vault        Clients and entities  Anything that connects and authenticates to Vault to accomplish a task is a   client    For example  a user logging into a cluster to manage policies or a machine based system  application or cloud service  requesting a database token are both considered clients     Vault Client Workflows  https   www datocms assets com 2885 1617325020 valult client workflows png   While there are many different potential clients  the most common are   1    Human users   interacting directly with Vault  1    Applications and microservices    1    Servers and platforms   like VMs  Docker containers  or Kubernetes pods  1    Orchestrators   like Nomad  Terraform  Ansible  ACME  and other continuous    integration   continuous delivery  CI CD  pipelines  1    Vault agents and proxies   that act on behalf of an application or    microservice      Identity and entity assignment  Authorized clients can connect to Vault with a variety of authentication methods   Authorization source         AuthN method                                                                Externally managed or SSO    Active Directory  LDAP  OIDC  JWT  GitHub  username password Platform  or server based    Kubernetes  AWS  GCP  Azure  Cert  Cloud Foundry Self                         AppRole  tokens with no associated authN path or role    Vault client types  https   www datocms assets com 2885 1617325030 vault clients png   When a client authenticates  Vault assigns a unique identifier    client entity    in the  Vault identity system   vault docs secrets identity  based on the authentication method used or a previously assigned alias    Entity aliases   let clients authenticate with multiple methods but still be associated with a single policy  share resources  and count as the same entity  regardless of the authentication method used for a particular session      Standard entity assignments   include  authn names mdx   Each authentication method has a unique ID string that corresponds to a client entity used for telemetry  For example  a microservice authenticating with AppRole takes the associated role ID as the entity  If you are running at scale and have multiple copies of the microservices using the same role id  the full set of instances will share the same identifier   As a result  it is critical that you configure different clients  microservices  humans  applications  services  platforms  servers  or pipelines  in a way that results in distinct clients having unique identifiers  For example  the role IDs should be different   between   two microservices  MicroserviceA and MicroServiceB  even if the   specific instances   of MicroServiceA and MicroServiceB share a common role ID      Entity assignment with ACME  Vault treats all ACME connections that authenticate under the same certificate identifier  domain  as the same   certificate entity   for client count calculations   For example     ACME client requests  from the same server or separate servers  for the same   certificate identifier  a unique combination of CN  DNS  SANS and IP SANS    are treated as the same entity    If an ACME client makes a request for  a test com   and subsequently makes a new   request for  b test com  and    test com  then two distinct entities will be created    one for  a test com  and another for the combination of  b test com  and    test com     Overlap of certificate identifiers from different ACME clients will be treated   as the same entity e g  if client 1 requests  a test com  and client 2 requests    a test com  a single entity is created for both requests      Secret sync clients Vault can automatically update secrets in external destinations with  secret sync   vault docs sync   A secret that gets synced to one or more destinations is considered a   secret sync client   for client count calculations   Note that     Each synced secret is counted distinctly based on the path and namespace of   the secret  If you have secrets at path  kv1 secret  and  kv2 secret    which are both synced  then two distinct secret syncs will be counted    A secret can be synced to multiple different destinations  and it will still   only be counted as one secret sync  If  kv secret  is synced to both Azure Key   Vault and AWS Secret Manager  this will be counted as only one secret sync   client    Secret sync clients are only created after you create an association between a   secret and a store  If you create  kv secret  and do not associate this secret   with any destinations  it will not be counted as a secret sync client    Secret syncs clients are registered in Vault s client counting system so long   as the sync is active  If you create  kv secret  and associate it with a   destination in January  update the secret in May  and then delete the secret   in September  Vault will consider this client as having been seen throughout   the entire period of January through September      Entity assignment with namespaces  A namespace represents a isolated  logical space within a single Vault cluster and is typically used for administrative purposes   When a client authenticates   within a given namespace    Vault assigns the same client entity to activities within any child namespaces because the namespaces exist within the same larger scope   When a client authenticates   across namespace boundaries    Vault treats the single client as two distinct entities because the client is operating across different scopes with different policy assignments and resources   For example     Different requests under parent and child namespaces from a single client   authenticated under the   parent   namespace are assigned   the same entity   ID    All the client activities occur   within   the boundaries of the   namespace referenced in the original authentication request    Different requests under parent and child namespaces from a single client   authenticated under the   child   namespace are assigned   different entity   IDs    Some of the client activities occur   outside   the boundaries of the   namespace referenced in the original authentication request    Requests by the same client to two different namespaces  NAMESPACE sub A  sub    and NAMESPACE sub B  sub  are assigned   different entity IDs        Entity assignment with non entity tokens  Vault uses tokens as the core method for authentication  You can use tokens to authenticate directly  or use token  auth methods   vault docs concepts auth  to dynamically generate tokens based on external identities   When clients authenticate with the  token auth method   vault docs auth token    without   a client identity  the result is a   non entity token    For example  a service might use the token authentication method to create a token for a user whose explicit identity is unknown   Ultimately  non entity tokens trace back to a particular client or purpose so Vault assigns unique entity IDs to non entity tokens based on a combination of the     assigned entity alias name  if present     associated policies  and   namespace under which the token was created   In   rare   cases  tokens may be created outside of the Vault identity system   without   an associated entity or identity  Vault treats every unaffiliated token as a unique client for production usage  We strongly discourage the use of unaffiliated tokens and recommend that you always associate a token with an entity alias and token role    Note title  Behavior change in Vault 1 9      As of Vault 1 9  all non entity tokens with the same namespace and policy   assignments are treated as the same client entity  Prior to Vault 1 9  every   non entity token was treated as a unique client entity  which drastically   inflated telemetry around client count     If you are using Vault 1 8 or earlier  and need to address client count   inflation without upgrading  we recommend creating a    token role   vault api docs auth token create update token role  with   allowable entity aliases and assigning all tokens to an appropriate    role and entity alias name   vault api docs auth token create token  before   using them    Note    include  content footer title mdx    Tabs    Tab heading  Related concepts    ul     li       a href   vault docs concepts client count counting  Client count calculation  a      li     li       a href   vault docs concepts client count faq  Client count FAQ  a      li    ul    Tab    Tab heading  Related tutorials    ul     li       a href   vault tutorials auth methods identity  Identity  Entities and Groups  a      li     li       a href   vault tutorials enterprise namespaces  Secure Multi Tenancy with Namespaces  a      li    ul    Tab    Tab heading  Other resources    ul     li      Article   a href  https   www hashicorp com identity based security and low trust networks         Identity based Security and Low trust Networks       a      li    ul    Tab     Tabs