vault Due to the nature of Vault and the confidentiality of data it manages page title Security Model Security model the Vault security model is very critical The overall goal of Vault s security layout docs Learn about the security model of Vault

Answer:
---
layout: docs
page_title: Security Model
description: Learn about the security model of Vault.
---

# Security model

Due to the nature of Vault and the confidentiality of data it manages,
the Vault security model is very critical. The overall goal of Vault's security
model is to provide [confidentiality, integrity, availability, accountability,
authentication](https://en.wikipedia.org/wiki/Information_security).

This means that data at rest and in transit must be secure from eavesdropping
or tampering. Clients must be appropriately authenticated and authorized
to access data or modify policies. All interactions must be auditable and traced
uniquely back to the origin entity, and the system must be robust against intentional
attempts to bypass any of its access controls.

# Threat model

The following are the various parts of the Vault threat model:

- Eavesdropping on any Vault communication. Client communication with Vault
  should be secure from eavesdropping as well as communication from Vault to
  its storage backend or between Vault cluster nodes.

- Tampering with data at rest or in transit. Any tampering should be detectable
  and cause Vault to abort processing of the transaction.

- Access to data or controls without authentication or authorization. All requests
  must be proceeded by the applicable security policies.

- Access to data or controls without accountability. If audit logging
  is enabled, requests and responses must be logged before the client receives
  any secret material.

- Confidentiality of stored secrets. Any data that leaves Vault to rest in the
  storage backend must be safe from eavesdropping. In practice, this means all
  data at rest must be encrypted.

- Availability of secret material in the face of failure. Vault supports
  running in a highly available configuration to avoid loss of availability.

The following are not considered part of the Vault threat model:

- Protecting against arbitrary control of the storage backend. An attacker
  that can perform arbitrary operations against the storage backend can
  undermine security in any number of ways that are difficult or impossible to protect
  against. As an example, an attacker could delete or corrupt all the contents
  of the storage backend causing total data loss for Vault. The ability to control
  reads would allow an attacker to snapshot in a well-known state and rollback state
  changes if that would be beneficial to them.

- Protecting against the leakage of the existence of secret material. An attacker
  that can read from the storage backend may observe that secret material exists
  and is stored, even if it is kept confidential.

- Protecting against memory analysis of a running Vault. If an attacker is able
  to inspect the memory state of a running Vault instance, then the confidentiality
  of data may be compromised.

- Protecting against flaws in external systems or services used by Vault.
  Some authentication methods or secrets engines delegate sensitive operations to
  systems external to Vault. If an attacker can compromise credentials or otherwise
  exploit a vulnerability in these external systems, then the confidentiality or
  integrity of data may be compromised.

- Protecting against malicious plugins or code execution on the underlying host.
  If an attacker can gain code execution or write privileges to the underlying host,
  then the confidentiality or the integrity of data may be compromised.

- Protecting against flaws in clients or systems that access Vault. If an attacker
  can compromise a Vault client (e.g., system, browser) and obtain this client’s Vault
  credentials, they can access Vault with the level of privilege associated with this
  client.

- Protecting against Vault administrators supplying vulnerable or malicious configuration
  data. Any data provided as configuration values to Vault's administrative endpoints
  (e.g. [secret engines](/vault/docs/secrets) configurations), or Vault's
  configuration files should be validated. If an attacker can write to Vault's
  configuration, then the confidentiality or integrity of data can be compromised.

# External threat overview

Vault architecture compromises of three distinct systems:

- Client: Speaks to Vault over an API.
- Server: Provides an API and serves requests.
- Storage backend: Utilized by the server to read and write data.

There is no mutual trust between the Vault client and server. Clients use
[TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security) to verify the
identity of the server and to establish a secure communication channel. Servers
require that a client provides a client token for every request which is used
to identify the client. A client that does not provide their token is only
permitted to make login requests.

All server-to-server traffic between Vault instances within a cluster (i.e,
high availability, enterprise replication or integrated storage) uses
mutually-authenticated TLS to ensure the confidentiality and integrity of data
in transit. Nodes are authenticated prior to joining the cluster by an
[unseal challenge](/vault/docs/concepts/integrated-storage#vault-networking-recap) or
a [one-time-use activation token](/vault/docs/enterprise/replication#security-model).

The storage backends used by Vault are also untrusted by design. Vault uses a
security barrier for all requests made to the backend. The security barrier
automatically encrypts all data leaving Vault using a 256-bit [Advanced
Encryption Standard
(AES)](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) cipher in
the [Galois Counter Mode
(GCM)](https://en.wikipedia.org/wiki/Galois/Counter_Mode) with 96-bit nonces.
The nonce is randomly generated for every encrypted object. When data is read
from the security barrier, the GCM authentication tag is verified during the
decryption process to detect any tampering.

Depending on the backend used, Vault may communicate with the backend over TLS
to provide an added layer of security. In some cases, such as a file backend,
this is not applicable. Because storage backends are untrusted, an eavesdropper
would only gain access to encrypted data even if communication with the backend
was intercepted.

# Internal threat overview

Within the Vault system, a critical security concern is an attacker attempting
to gain access to secret material they are not authorized to. This is an internal
threat if the attacker is already permitted to some level of access to Vault, and is
able to authenticate.

When a client first authenticates with Vault, an auth method is used to verify
the identity of the client and to return a list of associated ACL policies.
This association is configured by operators of Vault ahead of time. For
example, GitHub users in the "engineering" team may be mapped to the
"engineering" and "ops" Vault policies. Vault then generates a client token
which is a randomly generated, serialized value and maps it to the policy list.
This client token is then returned to the client.

On each request, a client provides this token. Vault then uses it to check that
the token is valid and has not been revoked or expired, and generates an ACL
based on the associated policies. Vault uses a strict default deny
enforcement strategy. This means unless an associated policy allows for a given action,
it will be denied. Each policy specifies a level of access granted to a path in
Vault. When the policies are merged (if multiple policies are associated with a
client), the highest access level permitted is used. For example, if the
"engineering" policy permits read/update access to the "eng/" path, and the
"ops" policy permits read access to the "ops/" path, then the user gets the
union of those. Policy is matched using the most specific defined policy, which
may be an exact match or the longest-prefix match glob pattern. See
[Policy Syntax](/vault/docs/concepts/policies#policy-syntax) for more details.

Certain operations are only permitted by "root" users, which is a distinguished
policy built into Vault. This is similar to the concept of a root user on a
Unix system or an administrator on Windows. In cases where clients are provided
with root tokens or associated with the root policy, Vault supports the
notion of "sudo" privilege. As part of a policy, users may be granted "sudo"
privileges to certain paths, so that they can still perform security sensitive
operations without being granted global root access to Vault.

Lastly, Vault supports using a [Two-person
rule](https://en.wikipedia.org/wiki/Two-person_rule) for unsealing using [Shamir's
Secret Sharing
technique](https://en.wikipedia.org/wiki/Shamir's_Secret_Sharing). When Vault
is started, it starts in a _sealed_ state. This means that the encryption key
needed to read and write from the storage backend is not yet known. The process
of unsealing requires providing the root key so that the encryption key can
be retrieved. The risk of distributing the root key is that a single
malicious attacker with access to it can decrypt the entire Vault. Instead,
Shamir's technique allows us to split the root key into multiple shares or
parts. The number of shares and the threshold needed is configurable, but by
default Vault generates 5 shares, any 3 of which must be provided to
reconstruct the root key.

By using a secret sharing technique, we avoid the need to place absolute trust
in the holder of the root key, and avoid storing the root key at all. The
root key is only retrievable by reconstructing the shares. The shares are not
useful for making any requests to Vault, and can only be used for unsealing.
Once unsealed the standard ACL mechanisms are used for all requests.

To make an analogy, a bank puts security deposit boxes inside of a vault. Each
security deposit box has a key, while the vault door has both a combination and
a key. The vault is encased in steel and concrete so that the door is the only
practical entrance. The analogy to Vault is that the cryptosystem is the
steel and concrete protecting the data. While you could tunnel through the
concrete or brute force the encryption keys, it would be prohibitively time
consuming. Opening the bank vault requires two-factors: the key and the
combination. Similarly, Vault requires multiple shares be provided to
reconstruct the root key. Once unsealed, each security deposit boxes still
requires that the owner provide a key, and similarly the Vault ACL system protects
all the secrets stored.

layout  docs page title  Security Model description  Learn about the security model of Vault         Security model  Due to the nature of Vault and the confidentiality of data it manages  the Vault security model is very critical  The overall goal of Vault s security model is to provide  confidentiality  integrity  availability  accountability  authentication  https   en wikipedia org wiki Information security    This means that data at rest and in transit must be secure from eavesdropping or tampering  Clients must be appropriately authenticated and authorized to access data or modify policies  All interactions must be auditable and traced uniquely back to the origin entity  and the system must be robust against intentional attempts to bypass any of its access controls     Threat model  The following are the various parts of the Vault threat model     Eavesdropping on any Vault communication  Client communication with Vault   should be secure from eavesdropping as well as communication from Vault to   its storage backend or between Vault cluster nodes     Tampering with data at rest or in transit  Any tampering should be detectable   and cause Vault to abort processing of the transaction     Access to data or controls without authentication or authorization  All requests   must be proceeded by the applicable security policies     Access to data or controls without accountability  If audit logging   is enabled  requests and responses must be logged before the client receives   any secret material     Confidentiality of stored secrets  Any data that leaves Vault to rest in the   storage backend must be safe from eavesdropping  In practice  this means all   data at rest must be encrypted     Availability of secret material in the face of failure  Vault supports   running in a highly available configuration to avoid loss of availability   The following are not considered part of the Vault threat model     Protecting against arbitrary control of the storage backend  An attacker   that can perform arbitrary operations against the storage backend can   undermine security in any number of ways that are difficult or impossible to protect   against  As an example  an attacker could delete or corrupt all the contents   of the storage backend causing total data loss for Vault  The ability to control   reads would allow an attacker to snapshot in a well known state and rollback state   changes if that would be beneficial to them     Protecting against the leakage of the existence of secret material  An attacker   that can read from the storage backend may observe that secret material exists   and is stored  even if it is kept confidential     Protecting against memory analysis of a running Vault  If an attacker is able   to inspect the memory state of a running Vault instance  then the confidentiality   of data may be compromised     Protecting against flaws in external systems or services used by Vault    Some authentication methods or secrets engines delegate sensitive operations to   systems external to Vault  If an attacker can compromise credentials or otherwise   exploit a vulnerability in these external systems  then the confidentiality or   integrity of data may be compromised     Protecting against malicious plugins or code execution on the underlying host    If an attacker can gain code execution or write privileges to the underlying host    then the confidentiality or the integrity of data may be compromised     Protecting against flaws in clients or systems that access Vault  If an attacker   can compromise a Vault client  e g   system  browser  and obtain this client s Vault   credentials  they can access Vault with the level of privilege associated with this   client     Protecting against Vault administrators supplying vulnerable or malicious configuration   data  Any data provided as configuration values to Vault s administrative endpoints    e g   secret engines   vault docs secrets  configurations   or Vault s   configuration files should be validated  If an attacker can write to Vault s   configuration  then the confidentiality or integrity of data can be compromised     External threat overview  Vault architecture compromises of three distinct systems     Client  Speaks to Vault over an API    Server  Provides an API and serves requests    Storage backend  Utilized by the server to read and write data    There is no mutual trust between the Vault client and server  Clients use  TLS  https   en wikipedia org wiki Transport Layer Security  to verify the identity of the server and to establish a secure communication channel  Servers require that a client provides a client token for every request which is used to identify the client  A client that does not provide their token is only permitted to make login requests   All server to server traffic between Vault instances within a cluster  i e  high availability  enterprise replication or integrated storage  uses mutually authenticated TLS to ensure the confidentiality and integrity of data in transit  Nodes are authenticated prior to joining the cluster by an  unseal challenge   vault docs concepts integrated storage vault networking recap  or a  one time use activation token   vault docs enterprise replication security model    The storage backends used by Vault are also untrusted by design  Vault uses a security barrier for all requests made to the backend  The security barrier automatically encrypts all data leaving Vault using a 256 bit  Advanced Encryption Standard  AES   https   en wikipedia org wiki Advanced Encryption Standard  cipher in the  Galois Counter Mode  GCM   https   en wikipedia org wiki Galois Counter Mode  with 96 bit nonces  The nonce is randomly generated for every encrypted object  When data is read from the security barrier  the GCM authentication tag is verified during the decryption process to detect any tampering   Depending on the backend used  Vault may communicate with the backend over TLS to provide an added layer of security  In some cases  such as a file backend  this is not applicable  Because storage backends are untrusted  an eavesdropper would only gain access to encrypted data even if communication with the backend was intercepted     Internal threat overview  Within the Vault system  a critical security concern is an attacker attempting to gain access to secret material they are not authorized to  This is an internal threat if the attacker is already permitted to some level of access to Vault  and is able to authenticate   When a client first authenticates with Vault  an auth method is used to verify the identity of the client and to return a list of associated ACL policies  This association is configured by operators of Vault ahead of time  For example  GitHub users in the  engineering  team may be mapped to the  engineering  and  ops  Vault policies  Vault then generates a client token which is a randomly generated  serialized value and maps it to the policy list  This client token is then returned to the client   On each request  a client provides this token  Vault then uses it to check that the token is valid and has not been revoked or expired  and generates an ACL based on the associated policies  Vault uses a strict default deny enforcement strategy  This means unless an associated policy allows for a given action  it will be denied  Each policy specifies a level of access granted to a path in Vault  When the policies are merged  if multiple policies are associated with a client   the highest access level permitted is used  For example  if the  engineering  policy permits read update access to the  eng   path  and the  ops  policy permits read access to the  ops   path  then the user gets the union of those  Policy is matched using the most specific defined policy  which may be an exact match or the longest prefix match glob pattern  See  Policy Syntax   vault docs concepts policies policy syntax  for more details   Certain operations are only permitted by  root  users  which is a distinguished policy built into Vault  This is similar to the concept of a root user on a Unix system or an administrator on Windows  In cases where clients are provided with root tokens or associated with the root policy  Vault supports the notion of  sudo  privilege  As part of a policy  users may be granted  sudo  privileges to certain paths  so that they can still perform security sensitive operations without being granted global root access to Vault   Lastly  Vault supports using a  Two person rule  https   en wikipedia org wiki Two person rule  for unsealing using  Shamir s Secret Sharing technique  https   en wikipedia org wiki Shamir s Secret Sharing   When Vault is started  it starts in a  sealed  state  This means that the encryption key needed to read and write from the storage backend is not yet known  The process of unsealing requires providing the root key so that the encryption key can be retrieved  The risk of distributing the root key is that a single malicious attacker with access to it can decrypt the entire Vault  Instead  Shamir s technique allows us to split the root key into multiple shares or parts  The number of shares and the threshold needed is configurable  but by default Vault generates 5 shares  any 3 of which must be provided to reconstruct the root key   By using a secret sharing technique  we avoid the need to place absolute trust in the holder of the root key  and avoid storing the root key at all  The root key is only retrievable by reconstructing the shares  The shares are not useful for making any requests to Vault  and can only be used for unsealing  Once unsealed the standard ACL mechanisms are used for all requests   To make an analogy  a bank puts security deposit boxes inside of a vault  Each security deposit box has a key  while the vault door has both a combination and a key  The vault is encased in steel and concrete so that the door is the only practical entrance  The analogy to Vault is that the cryptosystem is the steel and concrete protecting the data  While you could tunnel through the concrete or brute force the encryption keys  it would be prohibitively time consuming  Opening the bank vault requires two factors  the key and the combination  Similarly  Vault requires multiple shares be provided to reconstruct the root key  Once unsealed  each security deposit boxes still requires that the owner provide a key  and similarly the Vault ACL system protects all the secrets stored


vault using this feature it is useful to understand the intended use cases design page title Replication layout docs Vault Enterprise 0 7 adds support for multi datacenter replication Before Replication Vault enterprise Learn about the details of multi datacenter replication within Vault

Answer:
---
layout: docs
page_title: Replication
description: Learn about the details of multi-datacenter replication within Vault.
---

# Replication (Vault enterprise)

Vault Enterprise 0.7 adds support for multi-datacenter replication. Before
using this feature, it is useful to understand the intended use cases, design
goals, and high level architecture.

Replication is based on a primary/secondary (1:N) model with asynchronous
replication, focusing on high availability for global deployments. The
trade-offs made in the design and implementation of replication reflect these
high level goals.

# Use cases

Vault replication is based on a number of common use cases:

- **Multi-Datacenter Deployments**: A common challenge is providing Vault to
  applications across many datacenters in a highly-available manner. Running a
  single Vault cluster imposes high latency of access for remote clients,
  availability loss or outages during connectivity failures, and limits
  scalability.

- **Backup Sites**: Implementing a robust business continuity plan around the
  loss of a primary datacenter requires the ability to quickly and easily fail
  to a hot backup site.

- **Scaling Throughput**: Applications that use Vault for
  Encryption-as-a-Service or cryptographic offload may generate a very high
  volume of requests for Vault. Replicating keys between multiple clusters
  allows load to be distributed across additional servers to scale request
  throughput.

# Design goals

Based on the use cases for Vault Replication, we had a number of design goals
for the implementation:

- **Availability**: Global deployments of Vault require high levels of
  availability, and can tolerate reduced consistency. During full connectivity,
  replication is nearly real-time between the primary and secondary clusters.
  Degraded connectivity between a primary and secondary does not impact the
  primary's ability to service requests, and the secondary will continue to
  service reads on last-known data.

- **Conflict Free**: Certain replication techniques allow for potential write
  conflicts to take place. Particularly, any active/active configuration where
  writes are allowed to multiple sites require a conflict resolution strategy.
  This varies from techniques that allow for data loss like last-write-wins, or
  techniques that require manual operator resolution like allowing multiple
  values per key. We avoid the possibility of conflicts to ensure there is no
  data loss or manual intervention required.

- **Transparent to Clients**: Vault replication should be transparent to
  clients of Vault, so that existing thin clients work unmodified. The Vault
  servers handle the logic of request forwarding to the primary when necessary,
  and multi-hop routing is performed internally to ensure requests are
  processed.

- **Simple to Operate**: Operating a replicated cluster should be simple to
  avoid administrative overhead and potentially introducing security gaps.
  Setup of replication is very simple, and secondaries can handle being
  arbitrarily behind the primary, avoiding the need for operator intervention
  to copy data or snapshot the primary.

# Architecture

The architecture of Vault replication is based on the design goals, focusing on
the intended use cases. When replication is enabled, a cluster is set as either
a _primary_ or _secondary_. The primary cluster is authoritative, and is the
only cluster allowed to perform actions that write to the underlying data
storage, such as modifying policies or secrets. Secondary clusters can service
all other operations, such as reading secrets or sending data through
`transit`, and forward any writes to the primary cluster. Disallowing multiple
primaries ensures the cluster is conflict free and has an authoritative state.

The primary cluster uses log shipping to replicate changes to all of the
secondaries. This ensures writes are visible globally in near real-time when
there is full network connectivity. If a secondary is down or unable to
communicate with the primary, writes are not blocked on the primary and reads
are still serviced on the secondary. This ensures the availability of Vault.
When the secondary is initialized or recovers from degraded connectivity it
will automatically reconcile with the primary.

Lastly, clients can speak to any Vault server without a thick client. If a
client is communicating with a standby instance, the request is automatically
forwarded to an active instance. Secondary clusters will service reads locally
and forward any write requests to the primary cluster. The primary cluster is
able to service all request types.

An important optimization Vault makes is to avoid replication of tokens or
leases between clusters. Policies and secrets are the minority of data managed
by Vault and tend to be relatively stable. Tokens and leases are much more
dynamic, as they are created and expire rapidly. Keeping tokens and leases
locally reduces the amount of data that needs to be replicated, and distributes
the work of TTL management between the clusters. The caveat is that clients
will need to re-authenticate if they switch the Vault cluster they are
communicating with.

# Implementation details

It is important to understand the high-level architecture of replication to
ensure the trade-offs are appropriate for your use case. The implementation
details may be useful for those who are curious or want to understand more
about the performance characteristics or failure scenarios.

Using replication requires a storage backend that supports transactional
updates, such as Consul. This allows multiple key/value updates to be
performed atomically. Replication uses this to maintain a
[Write-Ahead-Log][wal] (WAL) of all updates, so that the key update happens
atomically with the WAL entry creation. The WALs are then used to perform log
shipping between the Vault clusters. When a secondary is closely synchronized
with a primary, Vault directly streams new WALs to be applied, providing near
real-time replication. A bounded set of WALs are maintained for the
secondaries, and older WALs are garbage collected automatically.

When a secondary is initialized or is too far behind the primary there may not
be enough WALs to synchronize. To handle this scenario, Vault maintains a
[merkle index][merkle] of the encrypted keys. Any time a key is updated or
deleted, the merkle index is updated to reflect the change. When a secondary
needs to reconcile with a primary, they compare their merkle indexes to
determine which keys are out of sync. The structure of the index allows this to
be done very efficiently, usually requiring only two round trips and a small
amount of data. The secondary uses this information to reconcile and then
switches back into WAL streaming mode.

Performance is an important concern for Vault, so WAL entries are batched and
the merkle index is not flushed to disk with every operation. Instead, the
index is updated in memory for every operation and asynchronously flushed to
disk. As a result, a crash or power loss may cause the merkle index to become
out of sync with the underlying keys. Vault uses the [ARIES][aries] recovery
algorithm to ensure the consistency of the index under those failure
conditions.

Log shipping traditionally requires the WAL stream to be synchronized, which
can introduce additional complexity when a new primary cluster is promoted.
Vault uses the merkle index as the source of truth, allowing the WAL streams to
be completely distinct and unsynchronized. This simplifies administration of
Vault Replication for operators.

## Addressing

### Cluster addresses on the primary

When a cluster is enabled as replication primary, it persists a cluster
definition to storage, under `core/cluster/replicated/info` or
`core/cluster/replicated-dr/info`.  An optional field of the cluster definition
is `primary_cluster_addr`, which may be provided in the enable request.

Performance standbys regularly issue heartbeat RPC requests to the active node, and
one of the arguments to the RPC is the local node's `cluster_addr`.  The
primary active node retains these cluster addresses received from its peers in
an in-memory cache named `clusterPeerClusterAddrsCache` with a 15s expiry time.

### Cluster addresses on the secondary

When a secondary is enabled, its replication activation token (obtained from
the primary) includes a `primary_cluster_addr` field.  This is taken from the
persisted cluster definition created when the primary was enabled, or if no
`primary_cluster_addr` was provided at that time, the token contains the
`cluster_addr` of the current active node in the primary at the time the
activation token is created.

The secondary persists its own version of the cluster definition to storage,
again under `core/cluster/replicated/info` or
`core/cluster/replicated-dr/info`.  Here the `primary_cluster_addr` field is
the one obtained from the activation token.

The secondary active node regularly issues heartbeat RPC requests to the
primary active node.  In response to these, the primary returns a response
which includes a `ClusterAddrs` field, comprising the contents of its
`clusterPeerClusterAddrsCache` plus the current active node's `cluster_addr`.
The secondary uses the response to both to update its in-memory record of
`known_primary_cluster_addrs`, and in addition it persists them to storage
under `core/primary-addrs/dr` or `core/primary-addrs/perf`.  When this happens
it logs the line `"replication: successful heartbeat"`, which includes the
`ClusterAddrs` value obtained in the response.

### Secondary RPC address resolution

gRPC is given a list of target addresses for it to use in performing RPC
requests.  gRPC will discover that performance standbys can't service most RPCs, and
will quickly weed out all but the active node cluster address.  If the primary
active node changes, gRPC will learn that its address is no longer viable and
will automatically fail over to the new active node, assuming it's one of the
known target addresses.

The secondary runs a background resolver goroutine that, every few seconds,
builds the gRPC target list of addresses for the primary.  Its output is logged
at Trace level as `"loaded addresses"`.

To build the primary cluster address list, the resolver goroutine normally
simply concatenates `known_primary_cluster_addrs` with the
`primary_cluster_addr` in the cluster definition in storage.

There are two exceptions to that normal behaviour: the first time the goroutine
goes through its loop, and when gRPC asks for a forced ResolveNow, which
happens when it's unable to perform RPCs on any of its target addresses.  In
both these cases, the resolver goroutine issues a special RemoteResolve RPC to
the primary.  This RPC is special because unlike all the other replication
RPCs, it can be serviced by performance standbys as well as the active node.  In
either case the node will return the `primary_cluster_addr` stored in the
primary's cluster definition, if any, or failing that the current active node's
cluster address.  The result of the RemoteResolve call gets included in the
list of target addresses the resolver gives to gRPC for regular RPCs to the
primary.

# Caveats

~> **Mismatched Cluster Versions**: It is not safe to replicate from a newer
version of Vault to an older version. When upgrading replicated clusters,
ensure that upstream clusters are always on older version of Vault than
downstream clusters. See
[Upgrading Vault](/vault/docs/upgrading#replication-installations) for an example.

- **Read-After-Write Consistency**: All write requests are forwarded from
  secondaries to the primary cluster in order to avoid potential conflicts.
  While replication is near real-time, it is not instantaneous, meaning there
  is a potential for a client to write to a secondary and a subsequent read to
  return an old value. Secondaries attempt to mask this from an individual
  client making subsequent requests by stalling write requests until the write
  is replicated or a timeout is reached (2 seconds). If the timeout is reached,
  the client will receive a warning. Clients can also take steps to protect
  against this, see [Consistency](/vault/docs/enterprise/consistency#mitigations).

- **Stale Reads**: Secondary clusters service reads based on their
  locally-replicated data. During normal operation updates from a primary are
  received in near real-time by secondaries. However, during an outage or
  network service disruption, replication may stall and secondaries may have
  stale data. The cluster will automatically recover and reconcile any stale
  data once the outage has recovered, but reads in the intervening period may
  receive stale data.

[wal]: https://en.wikipedia.org/wiki/Write-ahead_logging
[merkle]: https://en.wikipedia.org/wiki/Merkle_tree
[aries]: https://en.wikipedia.org/wiki/Algorithms_for_Recovery_and_Isolation_Exploiting_Semantics

layout  docs page title  Replication description  Learn about the details of multi datacenter replication within Vault         Replication  Vault enterprise   Vault Enterprise 0 7 adds support for multi datacenter replication  Before using this feature  it is useful to understand the intended use cases  design goals  and high level architecture   Replication is based on a primary secondary  1 N  model with asynchronous replication  focusing on high availability for global deployments  The trade offs made in the design and implementation of replication reflect these high level goals     Use cases  Vault replication is based on a number of common use cases       Multi Datacenter Deployments    A common challenge is providing Vault to   applications across many datacenters in a highly available manner  Running a   single Vault cluster imposes high latency of access for remote clients    availability loss or outages during connectivity failures  and limits   scalability       Backup Sites    Implementing a robust business continuity plan around the   loss of a primary datacenter requires the ability to quickly and easily fail   to a hot backup site       Scaling Throughput    Applications that use Vault for   Encryption as a Service or cryptographic offload may generate a very high   volume of requests for Vault  Replicating keys between multiple clusters   allows load to be distributed across additional servers to scale request   throughput     Design goals  Based on the use cases for Vault Replication  we had a number of design goals for the implementation       Availability    Global deployments of Vault require high levels of   availability  and can tolerate reduced consistency  During full connectivity    replication is nearly real time between the primary and secondary clusters    Degraded connectivity between a primary and secondary does not impact the   primary s ability to service requests  and the secondary will continue to   service reads on last known data       Conflict Free    Certain replication techniques allow for potential write   conflicts to take place  Particularly  any active active configuration where   writes are allowed to multiple sites require a conflict resolution strategy    This varies from techniques that allow for data loss like last write wins  or   techniques that require manual operator resolution like allowing multiple   values per key  We avoid the possibility of conflicts to ensure there is no   data loss or manual intervention required       Transparent to Clients    Vault replication should be transparent to   clients of Vault  so that existing thin clients work unmodified  The Vault   servers handle the logic of request forwarding to the primary when necessary    and multi hop routing is performed internally to ensure requests are   processed       Simple to Operate    Operating a replicated cluster should be simple to   avoid administrative overhead and potentially introducing security gaps    Setup of replication is very simple  and secondaries can handle being   arbitrarily behind the primary  avoiding the need for operator intervention   to copy data or snapshot the primary     Architecture  The architecture of Vault replication is based on the design goals  focusing on the intended use cases  When replication is enabled  a cluster is set as either a  primary  or  secondary   The primary cluster is authoritative  and is the only cluster allowed to perform actions that write to the underlying data storage  such as modifying policies or secrets  Secondary clusters can service all other operations  such as reading secrets or sending data through  transit   and forward any writes to the primary cluster  Disallowing multiple primaries ensures the cluster is conflict free and has an authoritative state   The primary cluster uses log shipping to replicate changes to all of the secondaries  This ensures writes are visible globally in near real time when there is full network connectivity  If a secondary is down or unable to communicate with the primary  writes are not blocked on the primary and reads are still serviced on the secondary  This ensures the availability of Vault  When the secondary is initialized or recovers from degraded connectivity it will automatically reconcile with the primary   Lastly  clients can speak to any Vault server without a thick client  If a client is communicating with a standby instance  the request is automatically forwarded to an active instance  Secondary clusters will service reads locally and forward any write requests to the primary cluster  The primary cluster is able to service all request types   An important optimization Vault makes is to avoid replication of tokens or leases between clusters  Policies and secrets are the minority of data managed by Vault and tend to be relatively stable  Tokens and leases are much more dynamic  as they are created and expire rapidly  Keeping tokens and leases locally reduces the amount of data that needs to be replicated  and distributes the work of TTL management between the clusters  The caveat is that clients will need to re authenticate if they switch the Vault cluster they are communicating with     Implementation details  It is important to understand the high level architecture of replication to ensure the trade offs are appropriate for your use case  The implementation details may be useful for those who are curious or want to understand more about the performance characteristics or failure scenarios   Using replication requires a storage backend that supports transactional updates  such as Consul  This allows multiple key value updates to be performed atomically  Replication uses this to maintain a  Write Ahead Log  wal   WAL  of all updates  so that the key update happens atomically with the WAL entry creation  The WALs are then used to perform log shipping between the Vault clusters  When a secondary is closely synchronized with a primary  Vault directly streams new WALs to be applied  providing near real time replication  A bounded set of WALs are maintained for the secondaries  and older WALs are garbage collected automatically   When a secondary is initialized or is too far behind the primary there may not be enough WALs to synchronize  To handle this scenario  Vault maintains a  merkle index  merkle  of the encrypted keys  Any time a key is updated or deleted  the merkle index is updated to reflect the change  When a secondary needs to reconcile with a primary  they compare their merkle indexes to determine which keys are out of sync  The structure of the index allows this to be done very efficiently  usually requiring only two round trips and a small amount of data  The secondary uses this information to reconcile and then switches back into WAL streaming mode   Performance is an important concern for Vault  so WAL entries are batched and the merkle index is not flushed to disk with every operation  Instead  the index is updated in memory for every operation and asynchronously flushed to disk  As a result  a crash or power loss may cause the merkle index to become out of sync with the underlying keys  Vault uses the  ARIES  aries  recovery algorithm to ensure the consistency of the index under those failure conditions   Log shipping traditionally requires the WAL stream to be synchronized  which can introduce additional complexity when a new primary cluster is promoted  Vault uses the merkle index as the source of truth  allowing the WAL streams to be completely distinct and unsynchronized  This simplifies administration of Vault Replication for operators      Addressing      Cluster addresses on the primary  When a cluster is enabled as replication primary  it persists a cluster definition to storage  under  core cluster replicated info  or  core cluster replicated dr info    An optional field of the cluster definition is  primary cluster addr   which may be provided in the enable request   Performance standbys regularly issue heartbeat RPC requests to the active node  and one of the arguments to the RPC is the local node s  cluster addr    The primary active node retains these cluster addresses received from its peers in an in memory cache named  clusterPeerClusterAddrsCache  with a 15s expiry time       Cluster addresses on the secondary  When a secondary is enabled  its replication activation token  obtained from the primary  includes a  primary cluster addr  field   This is taken from the persisted cluster definition created when the primary was enabled  or if no  primary cluster addr  was provided at that time  the token contains the  cluster addr  of the current active node in the primary at the time the activation token is created   The secondary persists its own version of the cluster definition to storage  again under  core cluster replicated info  or  core cluster replicated dr info    Here the  primary cluster addr  field is the one obtained from the activation token   The secondary active node regularly issues heartbeat RPC requests to the primary active node   In response to these  the primary returns a response which includes a  ClusterAddrs  field  comprising the contents of its  clusterPeerClusterAddrsCache  plus the current active node s  cluster addr   The secondary uses the response to both to update its in memory record of  known primary cluster addrs   and in addition it persists them to storage under  core primary addrs dr  or  core primary addrs perf    When this happens it logs the line   replication  successful heartbeat    which includes the  ClusterAddrs  value obtained in the response       Secondary RPC address resolution  gRPC is given a list of target addresses for it to use in performing RPC requests   gRPC will discover that performance standbys can t service most RPCs  and will quickly weed out all but the active node cluster address   If the primary active node changes  gRPC will learn that its address is no longer viable and will automatically fail over to the new active node  assuming it s one of the known target addresses   The secondary runs a background resolver goroutine that  every few seconds  builds the gRPC target list of addresses for the primary   Its output is logged at Trace level as   loaded addresses     To build the primary cluster address list  the resolver goroutine normally simply concatenates  known primary cluster addrs  with the  primary cluster addr  in the cluster definition in storage   There are two exceptions to that normal behaviour  the first time the goroutine goes through its loop  and when gRPC asks for a forced ResolveNow  which happens when it s unable to perform RPCs on any of its target addresses   In both these cases  the resolver goroutine issues a special RemoteResolve RPC to the primary   This RPC is special because unlike all the other replication RPCs  it can be serviced by performance standbys as well as the active node   In either case the node will return the  primary cluster addr  stored in the primary s cluster definition  if any  or failing that the current active node s cluster address   The result of the RemoteResolve call gets included in the list of target addresses the resolver gives to gRPC for regular RPCs to the primary     Caveats       Mismatched Cluster Versions    It is not safe to replicate from a newer version of Vault to an older version  When upgrading replicated clusters  ensure that upstream clusters are always on older version of Vault than downstream clusters  See  Upgrading Vault   vault docs upgrading replication installations  for an example       Read After Write Consistency    All write requests are forwarded from   secondaries to the primary cluster in order to avoid potential conflicts    While replication is near real time  it is not instantaneous  meaning there   is a potential for a client to write to a secondary and a subsequent read to   return an old value  Secondaries attempt to mask this from an individual   client making subsequent requests by stalling write requests until the write   is replicated or a timeout is reached  2 seconds   If the timeout is reached    the client will receive a warning  Clients can also take steps to protect   against this  see  Consistency   vault docs enterprise consistency mitigations        Stale Reads    Secondary clusters service reads based on their   locally replicated data  During normal operation updates from a primary are   received in near real time by secondaries  However  during an outage or   network service disruption  replication may stall and secondaries may have   stale data  The cluster will automatically recover and reconcile any stale   data once the outage has recovered  but reads in the intervening period may   receive stale data    wal   https   en wikipedia org wiki Write ahead logging  merkle   https   en wikipedia org wiki Merkle tree  aries   https   en wikipedia org wiki Algorithms for Recovery and Isolation Exploiting Semantics


vault Vault imposes fixed upper limits on the size of certain fields and Vault limits and maximums page title Limits and Maximums layout docs Learn about the maximum number of objects within Vault objects and configurable limits on others Vault also has upper

Answer:
---
layout: docs
page_title: Limits and Maximums
description: Learn about the maximum number of objects within Vault.
---

# Vault limits and maximums

Vault imposes fixed upper limits on the size of certain fields and
objects, and configurable limits on others. Vault also has upper
bounds that are a consequence of its underlying storage. This page
attempts to collect these limits, to assist in planning Vault
deployments.

In some cases, the system will show performance problems in advance of
the absolute limits being reached.

## Storage-Related limits

### Storage entry size

@include 'storage-entry-size.mdx'

Many of the other limits within Vault derive from the maximum size of
a storage entry, as described in the next sections. It is possible to
recover from an error where a storage entry has reached its maximum
size by reconfiguring Vault or Consul to a larger maximum storage
entry.

### Mount point limits

All secret engine mount points, and all auth mount points, must each fit
within a single storage entry. Each JSON object describing a mount
takes about 500 bytes, but is stored in compressed form at a typical cost of
about 75 bytes. Each of (1) auth mounts, (2) secret engine mount points,
(3) local-only auth methods, and (4) local-only secret engine mounts are
stored separately, so the limit applies to each independently.

|                                              | Consul default (512 KiB) | Integrated storage default (1 MiB) |
| -------------------------------------------- | ------------------------ | ---------------------------------- |
| Maximum number of secret engine mount points | ~7000                    | ~14000                             |
| Maximum number of enabled auth methods       | ~7000                    | ~14000                             |
| Maximum mount point length                   | no enforced limit        | no enforced limit                  |

Specifying distinct per-mount options, or using long mount point paths, can
increase the space required per mount.

The number of mount points can be monitored by reading the
[`sys/auth`](/vault/api-docs/system/auth) and
[`sys/mounts`](/vault/api-docs/system/mounts) endpoints from the root namespace and
similar sub-paths for namespaces respectively, like: `namespace1/sys/auth`,
`namespace1/sys/mounts`, etc.

Alternatively, use the
[`vault.core.mount_table.num_entries`](/vault/docs/internals/telemetry/metrics/core-system#vault-core-mount_table-num_entries)
and
[`vault.core.mount_table.size`](/vault/docs/internals/telemetry/metrics/core-system#vault-core-mount_table-size)
telemetry metrics to monitor the number of mount points and size of each mount table.

### Namespace limits

@include 'namespace-limits.mdx'

### Entity and group limits

The metadata that may be attached to an identity entity or an entity group
has the following constraints:

|                                       | Limit     |
| ------------------------------------- | --------- |
| Number of key-value pairs in metadata | 64        |
| Metadata key size                     | 128 bytes |
| Metadata value size                   | 512 bytes |

Vault shards the entities across 256 storage entries. This creates a
hard limit of 128MiB storage space used for entities on Consul, or
256MiB on integrated storage with its default settings. Entity aliases
are stored inline in the Entity objects and so consume the same pool
of storage. Entity definitions are compressed within each storage
entry, and the pre-compression size varies with the number of entity
aliases and the amount of metadata. Minimally-populated entities
about 200 bytes after compression.

Group definitions are stored separately, in their own pool of 256
storage entries. The size of each group object depends on the number
of members and the amount of metadata. Group aliases and group
membership information is stored inline in each Group object. A group
with no metadata, holding 10 entities, will use about 500 bytes per
group. A group holding 100 entities would instead consume about 4,000
bytes.

The following table shows a best-case estimate and a more conservative
estimate for entities and groups. The number is slightly less than the
amount that fits in one shard, to reflect the fact that the first
shard to fill up will start inducing failures. This maximum will
decrease if each entity has a large amount of metadata, or if each
group has a large number of members.

|                                                                                | Consul default (512 KiB) | Integrated storage default (1 MiB) |
| ---------------------------------------------------------------------------------------- | ------------------------ | ---------------------------------- |
| Maximum number of identity entities (best case, 200 bytes per entity)                    | ~610,000                 | ~1,250,000                         |
| Maximum number of identity entities (conservative case, 500 bytes per entity)            | ~250,000                 | ~480,000                           |
| Maximum number of identity entities (maximum permitted metadata, 41160 bytes per entity) | 670                      | 2,400                              |
| Maximum number of groups (10 entities per group)                                         | ~250,000                 | ~480,000                           |
| Maximum number of groups (100 entities per group)                                        | ~22,000                  | ~50,000                            |
| Maximum number of members in a group                                                     | ~11,500                  | ~23,000                            |

The number of entities can be monitored using Vault's [telemetry](/vault/docs/internals/telemetry#token-identity-and-lease-metrics); see `vault.identity.num_entities` (total) or `vault.identity.entities.count` (by namespace).

The cost of entity and group updates grows as the number of objects in
each shard increases. This cost can be monitored via the
`vault.identity.upsert_entity_txn` and
the `vault.identity.upsert_group_txn` metrics.

Very large internal groups should be avoided (more than 1000 members),
because the membership list in a group must reside in a single storage entry.
Instead, consider using [external groups](/vault/docs/concepts/identity#external-vs-internal-groups) or split the group up into multiple sub-groups.

### Token limits

One storage entry is used per token; there is thus no
upper bound on the number of active tokens. There are no restrictions on
the token metadata field, other than the entire token must fit into one
storage entry:

|                                       | Limit    |
| ------------------------------------- | -------- |
| Number of key-value pairs in metadata | no limit |
| Metadata key size                     | no limit |
| Metadata value size                   | no limit |
| Total size of token metadata          | 512 KiB  |

### Policy limits

The maximum size of a policy is limited by the storage
entry size. Policy lists that appear in tokens or entities must fit
within a single storage entry.

|                                                | Consul default (512 KiB) | Integrated storage default (1 MiB) |
| ---------------------------------------------- | ------------------------ | ---------------------------------- |
| Maximum policy size                            | 512 KiB                  | 1 MiB                              |
| Maximum number of policies per namespace       | no limit                 | no limit                           |
| Maximum number of policies per token           | ~14,000                  | ~28,000                            |
| Maximum number of policies per entity or group | ~14,000                  | ~28,000                            |

Each time a token is used, Vault must assemble the collection of
policies attached to that token, to the entity, to any groups that the
entity belongs to, and recursively to any groups that contain those groups.
Very large numbers of policies are possible, but can cause Vault’s
response time to increase. You can monitor the
[`vault.core.fetch_acl_and_token`](/vault/docs/internals/telemetry#core-metrics)
metric to determine if the time required to assemble an access control list
is becoming excessive.

### Versioned key-value store (kv-v2 secret engine)

|                                                          | Limit                                                      |
| -------------------------------------------------------- | ---------------------------------------------------------- |
| Number of secrets                                        | no limit, up to available storage capacity                 |
| Maximum size of one version of a secret                  | slightly less than one storage entry (512 KiB or 1024 KiB) |
| Number of versions of a secret                           | default 10; configurable per-secret or per-mount           |
| Maximum number of versions (not checked when configured) | at least 24,000                                            |

Each version of a secret must fit in a single storage entry; the
key-value pairs are converted to JSON before storage.

Version metadata consumes 21 bytes per version and must fit in a
single storage entry, separate from the stored data.

Each secret also has version-agnostic metadata. This data can contain a `custom_metadata` field of
user-provided key-value pairs. Vault imposes the following custom metadata limits:

|                                           | Limit     |
| ----------------------------------------- | --------- |
| Number of custom metadata key-value pairs | 64        |
| Custom metadata key size                  | 128 bytes |
| Custom metadata value size                | 512 bytes |

### Transit secret engine

The maximum size of a Transit ciphertext or plaintext is limited by Vault's
maximum request size, as described [below](#request-size).

All archived versions of a single key must fit in a single storage entry.
This limit depends on the key size.

| Key length           | Consul default (512 KiB) | Integrated storage default (1 MiB) |
| -------------------- | ------------------------ | ---------------------------------- |
| aes128-gcm96 keys    | 2008                     | 4017                               |
| aes256-gcm96 keys    | 1865                     | 3731                               |
| chacha-poly1305 keys | 1865                     | 3731                               |
| ed25519 keys         | 1420                     | 2841                               |
| ecdsa-p256 keys      | 817                      | 1635                               |
| ecdsa-p384 keys      | 659                      | 1318                               |
| ecdsa-p523 keys      | 539                      | 1078                               |
| 1024-bit RSA keys    | 169                      | 333                                |
| 2048-bit RSA keys    | 116                      | 233                                |
| 4096-bit RSA keys    | 89                       | 178                                |

## Other limits

### Request size

The maximum size of an HTTP request sent to Vault is limited by
the `max_request_size` option in the [listener stanza](/vault/docs/configuration/listener/tcp). It defaults to 32 MiB. This value, minus the overhead of
the HTTP request itself, places an upper bound on any Transit operation,
and on the maximum size of any key-value secrets.

### Request duration

The maximum duration of a Vault operation is
[`max_request_duration`](/vault/docs//configuration/listener/tcp), which defaults to
90 seconds. If a particular secret engine takes longer than this to perform an
operation on a remote service, the Vault client will see a failure.

The environment variable [`VAULT_CLIENT_TIMEOUT`](/vault/docs/commands#vault_client_timeout) sets a client-side maximum duration as well,
which is 60 seconds by default.

### Cluster and replication limits

There are no implementation limits on the maximum size of a cluster,
or the maximum number of replicas associated with a primary. However,
each replica or performance standby adds considerable overhead to the
active node, as each write must be duplicated to all standbys. The overhead of
resyncing multiple replicas at once is also high.

Monitor the active Vault node's CPU and network utilization, as well as
the lag between the last WAL and replica WAL, to determine if the
maximum number of replicas has been exceeded.

|                                        | Limit                                  |
| -------------------------------------- | -------------------------------------- |
| Maximum cluster size                   | no limit, up to active node capability |
| Maximum number of DR replicas          | no limit, up to active node capability |
| Maximum number of performance replicas | no limit, up to active node capability |

### Lease limits

A systemwide [maximum TTL](/vault/docs/configuration#max_lease_ttl), and a
[maximum TTL per mount point](/vault/api-docs/system/mounts#max_lease_ttl-1) can be
configured.

Although no technical maximum exists, high lease counts can cause
degradation in system performance. We recommend short default
time-to-live values on tokens and leases to avoid a large backlog of
unexpired leases, or a large number of simultaneous expirations.

|                                    | Limit                     |
| ---------------------------------- | ------------------------- |
| Maximum number of leases           | advisory limit at 256,000 |
| Maximum duration of lease or token | 768 hours by default      |

The current number of unexpired leases can be monitored via the
[`vault.expire.num_leases`](/vault/docs/internals/telemetry#token-identity-and-lease-metrics) metric.

### Transform limits

The Transform secret engine obeys the [FF3-1 minimum and maximum sizes
on the length of an input](/vault/docs/secrets/transform#input-limits), which
are a function of the alphabet size.

### External plugin limits

The [plugin system](/vault/docs/plugins) launches a separate process
initiated by Vault that communicates over RPC. For each secret engine and auth
method that's enabled as an external plugin, Vault will spawn a process on the
host system. For the Database Secrets Engines, external database plugins will
spawn a process for every configured connection.

Regardless of plugin type, each of these processes will incur resource overhead
on the system, including but not limited to resources such as CPU, memory,
networking, and file descriptors. There's no specific limit on the number
secrets engines, auth methods, or database configured connections that can be
enabled. This ultimately depends on the particular plugin resource utilization,
the extent to which that plugin is being called, and the available resources on
the system. For plugins of the same type, each additional process will incur a
roughly linear increase in resource utilization. This assumes the usage of each
plugin of the same type is similar.

layout  docs page title  Limits and Maximums description  Learn about the maximum number of objects within Vault         Vault limits and maximums  Vault imposes fixed upper limits on the size of certain fields and objects  and configurable limits on others  Vault also has upper bounds that are a consequence of its underlying storage  This page attempts to collect these limits  to assist in planning Vault deployments   In some cases  the system will show performance problems in advance of the absolute limits being reached      Storage Related limits      Storage entry size   include  storage entry size mdx   Many of the other limits within Vault derive from the maximum size of a storage entry  as described in the next sections  It is possible to recover from an error where a storage entry has reached its maximum size by reconfiguring Vault or Consul to a larger maximum storage entry       Mount point limits  All secret engine mount points  and all auth mount points  must each fit within a single storage entry  Each JSON object describing a mount takes about 500 bytes  but is stored in compressed form at a typical cost of about 75 bytes  Each of  1  auth mounts   2  secret engine mount points   3  local only auth methods  and  4  local only secret engine mounts are stored separately  so the limit applies to each independently                                                    Consul default  512 KiB    Integrated storage default  1 MiB                                                                                Maximum number of secret engine mount points    7000                       14000                                 Maximum number of enabled auth methods          7000                       14000                                 Maximum mount point length                     no enforced limit          no enforced limit                     Specifying distinct per mount options  or using long mount point paths  can increase the space required per mount   The number of mount points can be monitored by reading the   sys auth    vault api docs system auth  and   sys mounts    vault api docs system mounts  endpoints from the root namespace and similar sub paths for namespaces respectively  like   namespace1 sys auth    namespace1 sys mounts   etc   Alternatively  use the   vault core mount table num entries    vault docs internals telemetry metrics core system vault core mount table num entries  and   vault core mount table size    vault docs internals telemetry metrics core system vault core mount table size  telemetry metrics to monitor the number of mount points and size of each mount table       Namespace limits   include  namespace limits mdx       Entity and group limits  The metadata that may be attached to an identity entity or an entity group has the following constraints                                             Limit                                                               Number of key value pairs in metadata   64            Metadata key size                       128 bytes     Metadata value size                     512 bytes    Vault shards the entities across 256 storage entries  This creates a hard limit of 128MiB storage space used for entities on Consul  or 256MiB on integrated storage with its default settings  Entity aliases are stored inline in the Entity objects and so consume the same pool of storage  Entity definitions are compressed within each storage entry  and the pre compression size varies with the number of entity aliases and the amount of metadata  Minimally populated entities about 200 bytes after compression   Group definitions are stored separately  in their own pool of 256 storage entries  The size of each group object depends on the number of members and the amount of metadata  Group aliases and group membership information is stored inline in each Group object  A group with no metadata  holding 10 entities  will use about 500 bytes per group  A group holding 100 entities would instead consume about 4 000 bytes   The following table shows a best case estimate and a more conservative estimate for entities and groups  The number is slightly less than the amount that fits in one shard  to reflect the fact that the first shard to fill up will start inducing failures  This maximum will decrease if each entity has a large amount of metadata  or if each group has a large number of members                                                                                Consul default  512 KiB    Integrated storage default  1 MiB                                                                                Maximum number of identity entities  best case  200 bytes per entity                        610 000                    1 250 000                             Maximum number of identity entities  conservative case  500 bytes per entity                250 000                    480 000                               Maximum number of identity entities  maximum permitted metadata  41160 bytes per entity    670                        2 400                                  Maximum number of groups  10 entities per group                                             250 000                    480 000                               Maximum number of groups  100 entities per group                                            22 000                     50 000                                Maximum number of members in a group                                                        11 500                     23 000                               The number of entities can be monitored using Vault s  telemetry   vault docs internals telemetry token identity and lease metrics   see  vault identity num entities   total  or  vault identity entities count   by namespace    The cost of entity and group updates grows as the number of objects in each shard increases  This cost can be monitored via the  vault identity upsert entity txn  and the  vault identity upsert group txn  metrics   Very large internal groups should be avoided  more than 1000 members   because the membership list in a group must reside in a single storage entry  Instead  consider using  external groups   vault docs concepts identity external vs internal groups  or split the group up into multiple sub groups       Token limits  One storage entry is used per token  there is thus no upper bound on the number of active tokens  There are no restrictions on the token metadata field  other than the entire token must fit into one storage entry                                             Limit                                                             Number of key value pairs in metadata   no limit     Metadata key size                       no limit     Metadata value size                     no limit     Total size of token metadata            512 KiB         Policy limits  The maximum size of a policy is limited by the storage entry size  Policy lists that appear in tokens or entities must fit within a single storage entry                                                      Consul default  512 KiB    Integrated storage default  1 MiB                                                                                Maximum policy size                              512 KiB                    1 MiB                                  Maximum number of policies per namespace         no limit                   no limit                               Maximum number of policies per token              14 000                     28 000                                Maximum number of policies per entity or group    14 000                     28 000                               Each time a token is used  Vault must assemble the collection of policies attached to that token  to the entity  to any groups that the entity belongs to  and recursively to any groups that contain those groups  Very large numbers of policies are possible  but can cause Vault s response time to increase  You can monitor the   vault core fetch acl and token    vault docs internals telemetry core metrics  metric to determine if the time required to assemble an access control list is becoming excessive       Versioned key value store  kv v2 secret engine                                                                Limit                                                                                Number of secrets                                          no limit  up to available storage capacity                     Maximum size of one version of a secret                    slightly less than one storage entry  512 KiB or 1024 KiB      Number of versions of a secret                             default 10  configurable per secret or per mount               Maximum number of versions  not checked when configured    at least 24 000                                               Each version of a secret must fit in a single storage entry  the key value pairs are converted to JSON before storage   Version metadata consumes 21 bytes per version and must fit in a single storage entry  separate from the stored data   Each secret also has version agnostic metadata  This data can contain a  custom metadata  field of user provided key value pairs  Vault imposes the following custom metadata limits                                                 Limit                                                                   Number of custom metadata key value pairs   64            Custom metadata key size                    128 bytes     Custom metadata value size                  512 bytes        Transit secret engine  The maximum size of a Transit ciphertext or plaintext is limited by Vault s maximum request size  as described  below   request size    All archived versions of a single key must fit in a single storage entry  This limit depends on the key size     Key length             Consul default  512 KiB    Integrated storage default  1 MiB                                                                                aes128 gcm96 keys      2008                       4017                                   aes256 gcm96 keys      1865                       3731                                   chacha poly1305 keys   1865                       3731                                   ed25519 keys           1420                       2841                                   ecdsa p256 keys        817                        1635                                   ecdsa p384 keys        659                        1318                                   ecdsa p523 keys        539                        1078                                   1024 bit RSA keys      169                        333                                    2048 bit RSA keys      116                        233                                    4096 bit RSA keys      89                         178                                      Other limits      Request size  The maximum size of an HTTP request sent to Vault is limited by the  max request size  option in the  listener stanza   vault docs configuration listener tcp   It defaults to 32 MiB  This value  minus the overhead of the HTTP request itself  places an upper bound on any Transit operation  and on the maximum size of any key value secrets       Request duration  The maximum duration of a Vault operation is   max request duration    vault docs  configuration listener tcp   which defaults to 90 seconds  If a particular secret engine takes longer than this to perform an operation on a remote service  the Vault client will see a failure   The environment variable   VAULT CLIENT TIMEOUT    vault docs commands vault client timeout  sets a client side maximum duration as well  which is 60 seconds by default       Cluster and replication limits  There are no implementation limits on the maximum size of a cluster  or the maximum number of replicas associated with a primary  However  each replica or performance standby adds considerable overhead to the active node  as each write must be duplicated to all standbys  The overhead of resyncing multiple replicas at once is also high   Monitor the active Vault node s CPU and network utilization  as well as the lag between the last WAL and replica WAL  to determine if the maximum number of replicas has been exceeded                                              Limit                                                                                Maximum cluster size                     no limit  up to active node capability     Maximum number of DR replicas            no limit  up to active node capability     Maximum number of performance replicas   no limit  up to active node capability        Lease limits  A systemwide  maximum TTL   vault docs configuration max lease ttl   and a  maximum TTL per mount point   vault api docs system mounts max lease ttl 1  can be configured   Although no technical maximum exists  high lease counts can cause degradation in system performance  We recommend short default time to live values on tokens and leases to avoid a large backlog of unexpired leases  or a large number of simultaneous expirations                                          Limit                                                                                Maximum number of leases             advisory limit at 256 000     Maximum duration of lease or token   768 hours by default         The current number of unexpired leases can be monitored via the   vault expire num leases    vault docs internals telemetry token identity and lease metrics  metric       Transform limits  The Transform secret engine obeys the  FF3 1 minimum and maximum sizes on the length of an input   vault docs secrets transform input limits   which are a function of the alphabet size       External plugin limits  The  plugin system   vault docs plugins  launches a separate process initiated by Vault that communicates over RPC  For each secret engine and auth method that s enabled as an external plugin  Vault will spawn a process on the host system  For the Database Secrets Engines  external database plugins will spawn a process for every configured connection   Regardless of plugin type  each of these processes will incur resource overhead on the system  including but not limited to resources such as CPU  memory  networking  and file descriptors  There s no specific limit on the number secrets engines  auth methods  or database configured connections that can be enabled  This ultimately depends on the particular plugin resource utilization  the extent to which that plugin is being called  and the available resources on the system  For plugins of the same type  each additional process will incur a roughly linear increase in resource utilization  This assumes the usage of each plugin of the same type is similar