vault Identity page title Identity This document contains conceptual information about Identity along with an Vault provides an identity management solution to maintain clients who are recognized by Vault layout docs

Answer:
---
layout: docs
page_title: 'Identity'
description: >-
  Vault provides an identity management solution to maintain clients who are recognized by Vault.
---

# Identity

This document contains conceptual information about **Identity** along with an
overview of the various terminologies and their concepts. The idea of Identity
is to maintain the clients who are recognized by Vault. As such, Vault provides
an identity management solution through the **Identity secrets engine**. For
more information about the Identity secrets engine and how it is used, refer to
the [Identity Secrets Engine](/vault/docs/secrets/identity) documentation.

## Entities and aliases

Each user may have multiple accounts with various identity providers, and Vault
supports many of those providers to authenticate with Vault. Vault Identity can
tie authentications from various auth methods to a single representation. This representation of a consolidated identity is called an **Entity** and their
corresponding accounts with authentication providers can be mapped as
**Aliases**. In essence, each entity is made up of zero or more aliases. An entity cannot have more than one alias for
a particular authentication backend.

For example, a user with accounts in both GitHub and LDAP can be mapped to a
single entity in Vault with two aliases, one of type GitHub and one of type
LDAP.

![Entity  overview](/img/vault-identity-doc-1.png)

However, if both aliases are created on the same auth mount, such as
a Github mount, both aliases cannot be mapped to the same entity. The aliases can
have the same auth type, as long as the auth mounts are different, and
still be associated to the same entity. The diagrams below illustrate both valid
and invalid scenarios.

![Valid Alias Mapping](/img/vault-identity-doc-4.png)
![Invalid Alias Mapping](/img/vault-identity-doc-5.png)

When a client authenticates via any credential backend (except the Token
backend), Vault creates a new entity. It attaches a new alias to it if a
corresponding entity does not already exist. The entity identifier will be tied
to the authenticated token. When such tokens are used, their entity identifiers
are audit logged, marking a trail of actions performed by specific users.

~> Vault Entity is used to count the number of Vault clients. To learn more
about client count, refer to the [Client Count](/vault/docs/concepts/client-count)
documentation.

## Entity management

Entities in Vault **do not** automatically pull identity information from
anywhere. It needs to be explicitly managed by operators. This way, it is
flexible in terms of administratively controlling the number of entities to be
synced against Vault. In some sense, Vault will serve as a _cache_ of
identities and not as a _source_ of identities.

## Entity policies

Vault policies can be assigned to entities which will grant _additional_
permissions to the token on top of the existing policies on the token. If the
token presented on the API request contains an identifier for the entity and if
that entity has a set of policies on it, then the token will be capable of
performing actions allowed by the policies on the entity as well.

![Entity policies](/img/vault-identity-doc-2.png)

This is a paradigm shift in terms of _when_ the policies of the token get
evaluated. Before identity, the policy names on the token were immutable (not
the contents of those policies though). But with entity policies, along with
the immutable set of policy names on the token, the evaluation of policies
applicable to the token through its identity will happen at request time. This
also adds enormous flexibility to control the behavior of already issued
tokens.

It is important to note that the policies on the entity are only a means to grant
_additional_ capabilities and not a replacement for the policies on the token.
To know the full set of capabilities of the token with an associated entity
identifier, the policies on the token should be taken into account.

~> **NOTE:** Be careful in granting permissions to non-readonly identity endpoints.
If a user can modify an entity, they can grant it additional privileges through
policies. If a user can modify an alias they can login with, they can bind it to
an entity with higher privileges. If a user can modify group membership, they
can add their entity to a group with higher privileges.

## Mount bound aliases

Vault supports multiple authentication backends and also allows enabling the
same type of authentication backend on different mount paths. The alias name of
the user will be unique within the backend's mount. But identity store needs to
uniquely distinguish between conflicting alias names across different mounts of
these identity providers. Hence, the alias name in combination with the
authentication backend mount's accessor, serve as the unique identifier of an
alias.

The table below shows what information each of the supported auth methods uses
to form the alias name. This is the identifying information that is used to match or create
an entity. If no entities are explicitly created or merged, then one [entity will be implicitly created](#implicit-entities)
for each object on the right-hand side of the table, when it is used to authenticate on
a particular auth mount point.

| Auth method         | Name reported by auth method                                                                        |
| ------------------- | --------------------------------------------------------------------------------------------------- |
| AliCloud            | Principal ID                                                                                |
| AppRole             | Role ID                                                                                |
| AWS IAM             | Configurable via `iam_alias` to one of: Role ID (default), IAM unique ID, Canonical ARN, Full ARN   |
| AWS EC2             | Configurable via `ec2_alias` to one of: Role ID (default), EC2 instance ID, AMI ID                  |
| Azure               | Subject (from JWT claim)                                                                            |
| Cloud Foundry       | App ID                                                                                |
| GitHub              | User login name associated with token                                                               |
| Google Cloud        | Configurable via `iam_alias` to one of: Role ID (default), Service account unique ID                |
| JWT/OIDC            | Configurable via `user_claim` to one of the presented claims (no default value)                     |
| Kerberos            | Username                                                                                |
| Kubernetes          | Configurable via `alias_name_source` to one of: Service account UID (default), Service account name |
| LDAP                | Username                                                                                |
| OCI                 | Role name                                                                                |
| Okta                | Username                                                                                |
| RADIUS              | Username                                                                                |
| TLS Certificate     | Subject CommonName                                                                                |
| Token               | `entity_alias`, if provided                                                                         |
| Username (userpass) | Username                                                                                |

## Local auth methods

**Vault Enterprise:** All the auth methods will generate an entity by default
when a token is being issued, with the exception of token store. This is applicable
for both mounts that are shared between clusters and cluster local auth mounts (using `local=true`)
when Vault replication is in use.
If the goal of marking an auth method as `local` was to comply to GDPR guidelines,
then care must be taken to not set the data pertaining to local auth mount or local auth
mount aliases in the metadata of the associated entity.

## Implicit entities

Operators can create entities for all the users of an auth mount beforehand and
assign policies to them, so that when users login, the desired capabilities to
the tokens via entities are already assigned. But if that's not done, upon a
successful user login from any of the authentication backends, Vault will
create a new entity and assign an alias against the login that was successful.

Note that the tokens created using the token authentication backend will not
normally have any associated identity information. An existing or new implicit
entity can be assigned by using the `entity_alias` parameter, when creating a
token using a token role with a configured list of `allowed_entity_aliases`.

## Identity auditing

If the token used to make API calls has an associated entity identifier, it
will be audit logged as well. This leaves a trail of actions performed by
specific users.

## Identity groups

Vault identity has support for **groups**. A group can contain multiple entities
as its members. A group can also have subgroups. Policies set on the group are
granted to all members of the group. During request time, when the token's
entity ID is being evaluated for the policies that it has access to, policies
that are inherited due to group memberships are granted along with the policies
on the entity itself.

![Identity overview](/img/vault-identity-doc-3.png)

## Group hierarchical permissions

Entities can be direct members of groups, in which case they inherit the
policies of the groups they belong to. Entities can also be indirect members of
groups. For example, if a GroupA has GroupB as subgroup, then members of GroupB
are indirect members of GroupA. Hence, the members of GroupB will have access
to policies on both GroupA and GroupB.

## External vs internal groups

By default, the groups created in identity store are called **internal groups**.
The membership management of these groups should be carried out
manually.

A group can also be created as an **external group**. In this case, the
entity membership in the group is managed semi-automatically. An external group
serves as a mapping to a group that is outside of the identity store. External
groups can have one (and only one) alias. This alias should map to a notion of
a group that is outside of the identity store.

For example, groups in LDAP and teams in GitHub.
A username in LDAP belonging to a group in LDAP can get its
entity ID added as a member of a group in Vault automatically during _logins_
and _token renewals_. This works only if the group in Vault is an external
group and has an alias that maps to the group in LDAP.

~> **NOTE:** If the user is removed from the group in LDAP, the user will
not immediately be removed from the external group in Vault. The group
membership change will be reflected in Vault only upon the
subsequent **login** or **renewal** operation.

For information about Identity Secrets Engine, refer to [Identity Secrets Engine](/vault/docs/secrets/identity).

## Tutorial

Refer to the [Identity: Entities and
Groups](/vault/tutorials/auth-methods/identity) tutorial to learn how Vault supports mutliple authentication methods and enables the same authentication method to be used with different mount paths.

layout  docs page title   Identity  description       Vault provides an identity management solution to maintain clients who are recognized by Vault         Identity  This document contains conceptual information about   Identity   along with an overview of the various terminologies and their concepts  The idea of Identity is to maintain the clients who are recognized by Vault  As such  Vault provides an identity management solution through the   Identity secrets engine    For more information about the Identity secrets engine and how it is used  refer to the  Identity Secrets Engine   vault docs secrets identity  documentation      Entities and aliases  Each user may have multiple accounts with various identity providers  and Vault supports many of those providers to authenticate with Vault  Vault Identity can tie authentications from various auth methods to a single representation  This representation of a consolidated identity is called an   Entity   and their corresponding accounts with authentication providers can be mapped as   Aliases    In essence  each entity is made up of zero or more aliases  An entity cannot have more than one alias for a particular authentication backend   For example  a user with accounts in both GitHub and LDAP can be mapped to a single entity in Vault with two aliases  one of type GitHub and one of type LDAP     Entity  overview   img vault identity doc 1 png   However  if both aliases are created on the same auth mount  such as a Github mount  both aliases cannot be mapped to the same entity  The aliases can have the same auth type  as long as the auth mounts are different  and still be associated to the same entity  The diagrams below illustrate both valid and invalid scenarios     Valid Alias Mapping   img vault identity doc 4 png    Invalid Alias Mapping   img vault identity doc 5 png   When a client authenticates via any credential backend  except the Token backend   Vault creates a new entity  It attaches a new alias to it if a corresponding entity does not already exist  The entity identifier will be tied to the authenticated token  When such tokens are used  their entity identifiers are audit logged  marking a trail of actions performed by specific users      Vault Entity is used to count the number of Vault clients  To learn more about client count  refer to the  Client Count   vault docs concepts client count  documentation      Entity management  Entities in Vault   do not   automatically pull identity information from anywhere  It needs to be explicitly managed by operators  This way  it is flexible in terms of administratively controlling the number of entities to be synced against Vault  In some sense  Vault will serve as a  cache  of identities and not as a  source  of identities      Entity policies  Vault policies can be assigned to entities which will grant  additional  permissions to the token on top of the existing policies on the token  If the token presented on the API request contains an identifier for the entity and if that entity has a set of policies on it  then the token will be capable of performing actions allowed by the policies on the entity as well     Entity policies   img vault identity doc 2 png   This is a paradigm shift in terms of  when  the policies of the token get evaluated  Before identity  the policy names on the token were immutable  not the contents of those policies though   But with entity policies  along with the immutable set of policy names on the token  the evaluation of policies applicable to the token through its identity will happen at request time  This also adds enormous flexibility to control the behavior of already issued tokens   It is important to note that the policies on the entity are only a means to grant  additional  capabilities and not a replacement for the policies on the token  To know the full set of capabilities of the token with an associated entity identifier  the policies on the token should be taken into account        NOTE    Be careful in granting permissions to non readonly identity endpoints  If a user can modify an entity  they can grant it additional privileges through policies  If a user can modify an alias they can login with  they can bind it to an entity with higher privileges  If a user can modify group membership  they can add their entity to a group with higher privileges      Mount bound aliases  Vault supports multiple authentication backends and also allows enabling the same type of authentication backend on different mount paths  The alias name of the user will be unique within the backend s mount  But identity store needs to uniquely distinguish between conflicting alias names across different mounts of these identity providers  Hence  the alias name in combination with the authentication backend mount s accessor  serve as the unique identifier of an alias   The table below shows what information each of the supported auth methods uses to form the alias name  This is the identifying information that is used to match or create an entity  If no entities are explicitly created or merged  then one  entity will be implicitly created   implicit entities  for each object on the right hand side of the table  when it is used to authenticate on a particular auth mount point     Auth method           Name reported by auth method                                                                                AliCloud              Principal ID                                                                                AppRole               Role ID                                                                                AWS IAM               Configurable via  iam alias  to one of  Role ID  default   IAM unique ID  Canonical ARN  Full ARN       AWS EC2               Configurable via  ec2 alias  to one of  Role ID  default   EC2 instance ID  AMI ID                      Azure                 Subject  from JWT claim                                                                                Cloud Foundry         App ID                                                                                GitHub                User login name associated with token                                                                   Google Cloud          Configurable via  iam alias  to one of  Role ID  default   Service account unique ID                    JWT OIDC              Configurable via  user claim  to one of the presented claims  no default value                          Kerberos              Username                                                                                Kubernetes            Configurable via  alias name source  to one of  Service account UID  default   Service account name     LDAP                  Username                                                                                OCI                   Role name                                                                                Okta                  Username                                                                                RADIUS                Username                                                                                TLS Certificate       Subject CommonName                                                                                Token                  entity alias   if provided                                                                             Username  userpass    Username                                                                                Local auth methods    Vault Enterprise    All the auth methods will generate an entity by default when a token is being issued  with the exception of token store  This is applicable for both mounts that are shared between clusters and cluster local auth mounts  using  local true   when Vault replication is in use  If the goal of marking an auth method as  local  was to comply to GDPR guidelines  then care must be taken to not set the data pertaining to local auth mount or local auth mount aliases in the metadata of the associated entity      Implicit entities  Operators can create entities for all the users of an auth mount beforehand and assign policies to them  so that when users login  the desired capabilities to the tokens via entities are already assigned  But if that s not done  upon a successful user login from any of the authentication backends  Vault will create a new entity and assign an alias against the login that was successful   Note that the tokens created using the token authentication backend will not normally have any associated identity information  An existing or new implicit entity can be assigned by using the  entity alias  parameter  when creating a token using a token role with a configured list of  allowed entity aliases       Identity auditing  If the token used to make API calls has an associated entity identifier  it will be audit logged as well  This leaves a trail of actions performed by specific users      Identity groups  Vault identity has support for   groups    A group can contain multiple entities as its members  A group can also have subgroups  Policies set on the group are granted to all members of the group  During request time  when the token s entity ID is being evaluated for the policies that it has access to  policies that are inherited due to group memberships are granted along with the policies on the entity itself     Identity overview   img vault identity doc 3 png      Group hierarchical permissions  Entities can be direct members of groups  in which case they inherit the policies of the groups they belong to  Entities can also be indirect members of groups  For example  if a GroupA has GroupB as subgroup  then members of GroupB are indirect members of GroupA  Hence  the members of GroupB will have access to policies on both GroupA and GroupB      External vs internal groups  By default  the groups created in identity store are called   internal groups    The membership management of these groups should be carried out manually   A group can also be created as an   external group    In this case  the entity membership in the group is managed semi automatically  An external group serves as a mapping to a group that is outside of the identity store  External groups can have one  and only one  alias  This alias should map to a notion of a group that is outside of the identity store   For example  groups in LDAP and teams in GitHub  A username in LDAP belonging to a group in LDAP can get its entity ID added as a member of a group in Vault automatically during  logins  and  token renewals   This works only if the group in Vault is an external group and has an alias that maps to the group in LDAP        NOTE    If the user is removed from the group in LDAP  the user will not immediately be removed from the external group in Vault  The group membership change will be reflected in Vault only upon the subsequent   login   or   renewal   operation   For information about Identity Secrets Engine  refer to  Identity Secrets Engine   vault docs secrets identity       Tutorial  Refer to the  Identity  Entities and Groups   vault tutorials auth methods identity  tutorial to learn how Vault supports mutliple authentication methods and enables the same authentication method to be used with different mount paths


vault sidebar title Storage layout docs Storage Vault relies on external storage to save its durable information page title Storage

Answer:
---
layout: docs
page_title: Storage
sidebar_title: Storage
description: >-
  Vault relies on external storage to save its durable information.
---

# Storage

As described on our [Architecture](/vault/docs/internals/architecture) page, Vault's
storage backend is untrusted storage used purely to keep encrypted information.

## Supported storage backends

@include 'ent-supported-storage.mdx'

Many other options for storage are available with community support for Vault - see our
[Storage Configuration](/vault/docs/configuration/storage) section for more
information.

-> **Choosing a storage backend:** Refer to the [integrated storage vs. external
storage](/vault/docs/configuration/storage#integrated-storage-vs-external-storage)
section of the storage configuration page to help make a decision about which
storage backend to use.

## Backups

Due to the highly flexible nature of Vault's potential storage configurations,
providing exact guidance on backing up Vault is challenging.

When backing up Vault, there are two pieces to consider:

1. Vault's encrypted data in the storage backend
2. Configuration files and management scripts for running the Vault server

There's also a big question - what is the error case you're trying to guard
against by saving a backup?

### The big question - why take backups?

It's important to consider the question of "why take a backup" while developing
your ongoing backup and disaster recovery strategy.

Taking a backup is recommended prior to upgrades, as downgrading Vault storage
is not always possible. Generally, a backup is recommended any time a major
change is planned for a cluster.

More specifically, we recommend taking backups **before**, but not during, write
operations to the `/sys` API (excluding the `/sys/leases`, `/sys/namespaces`,
`/sys/tools`, `/sys/wrapping`, `/sys/policies`, and `/sys/pprof` endpoints).
Some examples of workflows that write to the `/sys` API are upgrades and rekeys.
In the future, this guidance may change for the Integrated Storage backend.

Backups _can_ also help with accidental data deletions or modifications. In
this case, the story can get a little tricky. If you simply recover a backup
from 5AM with the correct data, but the current time is 10AM, you will lose data
written between 5 and 10AM. Lucy Davinhart gave a HashiConf talk that serves as
an interesting [case
study](https://www.hashicorp.com/resources/oh-no-i-deleted-my-vault-secret).

We do not recommend backups as protection against the failure of an individual
machine. Vault servers can run in clusters, so to protect against server
failure, we recommend running Vault in [HA
mode](/vault/docs/internals/high-availability). With community features, a
Vault cluster can extend across multiple availability zones within a region.

Vault Enterprise supports replicated clusters and disaster recovery for data
center failure. When using Vault Community Edition in [HA
Mode](/vault/docs/internals/high-availability), a backup can help guard against the
failure of a data center.

Ultimately, backups are not a replacement for running in HA, or for using
replication with Vault Enterprise. As you develop a plan for recovering from or
guarding against failure, you should consider both backups and HA as critical
components of that plan.

### Backing up vault's persisted data

Backups and restores are ideally performed while Vault is offline. If offline
backups are not feasible, we recommend using a storage backend that supports
atomic snapshots (such as
[Consul](/consul/commands/snapshot) or [Integrated
Storage](/vault/docs/commands/operator/raft#snapshot)).

~> If your storage backend does not support atomic snapshots, we recommend only
taking offline backups.

To perform a backup or restore of Vault's encrypted data when using a
HashiCorp-supported storage backend, see the instructions linked below. For
other storage backends, follow the documentation of that backend for taking and
restoring backups.

- Integrated Storage [snapshots](/vault/docs/commands/operator/raft#snapshot)
- Consul [snapshots](/consul/commands/snapshot)

#### Backing up multiple clusters

If you are using Vault Enterprise [Performance
Replication](/vault/docs/enterprise/replication#performance-replication-and-disaster-recovery-dr-replication),
you should plan to take backups of the active node on each of your clusters.

### Configuration

In addition to backing up Vault's encrypted data via the storage backend, you
may also wish to save the server configuration files, any scripts for managing
the Vault service, and ensure you can reinstall any user-installed plugins. The
location of these files will be specific to your installation of Vault.

~> **NOTE**: Although a backup or snapshot of Vault's data from the storage
backend is encrypted, some of your configuration may be sensitive (a Vault token
for Transit Autounseal or a TLS private key in your configuration, for example).
The presence of this information in your backups will mean that they may need
to be carefully protected.

layout  docs page title  Storage sidebar title  Storage description       Vault relies on external storage to save its durable information         Storage  As described on our  Architecture   vault docs internals architecture  page  Vault s storage backend is untrusted storage used purely to keep encrypted information      Supported storage backends   include  ent supported storage mdx   Many other options for storage are available with community support for Vault   see our  Storage Configuration   vault docs configuration storage  section for more information        Choosing a storage backend    Refer to the  integrated storage vs  external storage   vault docs configuration storage integrated storage vs external storage  section of the storage configuration page to help make a decision about which storage backend to use      Backups  Due to the highly flexible nature of Vault s potential storage configurations  providing exact guidance on backing up Vault is challenging   When backing up Vault  there are two pieces to consider   1  Vault s encrypted data in the storage backend 2  Configuration files and management scripts for running the Vault server  There s also a big question   what is the error case you re trying to guard against by saving a backup       The big question   why take backups   It s important to consider the question of  why take a backup  while developing your ongoing backup and disaster recovery strategy   Taking a backup is recommended prior to upgrades  as downgrading Vault storage is not always possible  Generally  a backup is recommended any time a major change is planned for a cluster   More specifically  we recommend taking backups   before    but not during  write operations to the   sys  API  excluding the   sys leases     sys namespaces     sys tools     sys wrapping     sys policies   and   sys pprof  endpoints   Some examples of workflows that write to the   sys  API are upgrades and rekeys  In the future  this guidance may change for the Integrated Storage backend   Backups  can  also help with accidental data deletions or modifications  In this case  the story can get a little tricky  If you simply recover a backup from 5AM with the correct data  but the current time is 10AM  you will lose data written between 5 and 10AM  Lucy Davinhart gave a HashiConf talk that serves as an interesting  case study  https   www hashicorp com resources oh no i deleted my vault secret    We do not recommend backups as protection against the failure of an individual machine  Vault servers can run in clusters  so to protect against server failure  we recommend running Vault in  HA mode   vault docs internals high availability   With community features  a Vault cluster can extend across multiple availability zones within a region   Vault Enterprise supports replicated clusters and disaster recovery for data center failure  When using Vault Community Edition in  HA Mode   vault docs internals high availability   a backup can help guard against the failure of a data center   Ultimately  backups are not a replacement for running in HA  or for using replication with Vault Enterprise  As you develop a plan for recovering from or guarding against failure  you should consider both backups and HA as critical components of that plan       Backing up vault s persisted data  Backups and restores are ideally performed while Vault is offline  If offline backups are not feasible  we recommend using a storage backend that supports atomic snapshots  such as  Consul   consul commands snapshot  or  Integrated Storage   vault docs commands operator raft snapshot        If your storage backend does not support atomic snapshots  we recommend only taking offline backups   To perform a backup or restore of Vault s encrypted data when using a HashiCorp supported storage backend  see the instructions linked below  For other storage backends  follow the documentation of that backend for taking and restoring backups     Integrated Storage  snapshots   vault docs commands operator raft snapshot    Consul  snapshots   consul commands snapshot        Backing up multiple clusters  If you are using Vault Enterprise  Performance Replication   vault docs enterprise replication performance replication and disaster recovery dr replication   you should plan to take backups of the active node on each of your clusters       Configuration  In addition to backing up Vault s encrypted data via the storage backend  you may also wish to save the server configuration files  any scripts for managing the Vault service  and ensure you can reinstall any user installed plugins  The location of these files will be specific to your installation of Vault        NOTE    Although a backup or snapshot of Vault s data from the storage backend is encrypted  some of your configuration may be sensitive  a Vault token for Transit Autounseal or a TLS private key in your configuration  for example   The presence of this information in your backups will mean that they may need to be carefully protected


vault High availability mode HA against outages layout docs page title High Availability Vault can be highly available allowing you to run multiple Vaults to protect

Answer:
---
layout: docs
page_title: High Availability
description: >-
  Vault can be highly available, allowing you to run multiple Vaults to protect
  against outages.
---

# High availability mode (HA)

Vault supports a multi-server mode for high availability. This mode protects
against outages by running multiple Vault servers. High availability mode
is automatically enabled when using a data store that supports it.

You can tell if a data store supports high availability mode ("HA") by starting
the server and seeing if "(HA available)" is output next to the data store
information. If it is, then Vault will automatically use HA mode. This
information is also available on the
[Configuration](/vault/docs/configuration) page.

To be highly available, one of the Vault server nodes grabs a lock within the
data store. The successful server node then becomes the active node; all other
nodes become standby nodes. At this point, if the standby nodes receive a
request, they will either [forward the request](#request-forwarding) or
[redirect the client](#client-redirection) depending on the current
configuration and state of the cluster -- see the sections below for details.
Due to this architecture, HA does not enable increased scalability. In general,
the bottleneck of Vault is the data store itself, not Vault core. For example:
to increase the scalability of Vault with Consul, you would generally scale
Consul instead of Vault.

Certain storage backends can support high availability mode, which enable them
to store both Vault's information in addition to the HA lock. However, Vault
also supports split data/HA mode, whereby the lock value and the rest of the
data live separately. This can be done by specifying both the
[`storage`](/vault/docs/configuration#storage) and
[`ha_storage`](/vault/docs/configuration#ha_storage) stanzas in the configuration file
with different backends. For instance, a Vault cluster can be set up to use
Consul as the [`ha_storage`](/vault/docs/configuration#ha_storage) to manage the lock,
and use Amazon S3 as the [`storage`](/vault/docs/configuration#storage) for all other
persisted data.

The sections below explain the server communication patterns and each type of
request handling in more detail. At a minimum, the requirements for redirection
mode must be met for an HA cluster to work successfully.

## Server-to-Server communication

Both methods of request handling rely on the active node advertising
information about itself to the other nodes. Rather than over the network, this
communication takes place within Vault's encrypted storage; the active node
writes this information and unsealed standby Vault nodes can read it.

For the client redirection method, this is the extent of server-to-server
communication -- no direct communication with only encrypted entries in the
data store used to transfer state.

For the request forwarding method, the servers need direct communication with
each other. In order to perform this securely, the active node also advertises,
via the encrypted data store entry, a newly-generated private key (ECDSA-P521)
and a newly-generated self-signed certificate designated for client and server
authentication. Each standby uses the private key and certificate to open a
mutually-authenticated TLS 1.2 connection to the active node via the advertised
cluster address. When client requests come in, the requests are serialized,
sent over this TLS-protected communication channel, and acted upon by the
active node. The active node then returns a response to the standby, which
sends the response back to the requesting client.

## Request forwarding

If request forwarding is enabled (turned on by default in 0.6.2), clients can
still force the older/fallback redirection behavior (see below) if desired by
setting the `X-Vault-No-Request-Forwarding` header to any non-empty value.

Successful cluster setup requires a few configuration parameters, although some
can be automatically determined.

## Client redirection

If `X-Vault-No-Request-Forwarding` header in the request is set to a non-empty
value, the standby nodes will redirect the client using a `307` status code to
the _active node's_ redirect address.

This is also the fallback method used when request forwarding is turned off or
there is an error performing the forwarding. As such, a redirect address is
always required for all HA setups.

Some HA data store drivers can autodetect the redirect address, but it is often
necessary to configure it manually via a top-level value in the configuration
file. The key for this value is [`api_addr`](/vault/docs/configuration#api_addr) and
the value can also be specified by the `VAULT_API_ADDR` environment variable,
which takes precedence.

What the [`api_addr`](/vault/docs/configuration#api_addr) value should be set to
depends on how Vault is set up. There are two common scenarios: Vault servers
accessed directly by clients, and Vault servers accessed via a load balancer.

In both cases, the [`api_addr`](/vault/docs/configuration#api_addr) should be a full
URL including scheme (`http`/`https`), not simply an IP address and port.

### Direct access

When clients are able to access Vault directly, the
[`api_addr`](/vault/docs/configuration#api_addr) for each node should be that node's
address. For instance, if there are two Vault nodes:

- `A`, accessed via `https://a.vault.mycompany.com:8200`
- `B`, accessed via `https://b.vault.mycompany.com:8200`

Then node `A` would set its
[`api_addr`](/vault/docs/configuration#api_addr) to
`https://a.vault.mycompany.com:8200` and node `B` would set its
[`api_addr`](/vault/docs/configuration#api_addr) to
`https://b.vault.mycompany.com:8200`.

This way, when `A` is the active node, any requests received by node `B` will
cause it to redirect the client to node `A`'s
[`api_addr`](/vault/docs/configuration#api_addr) at `https://a.vault.mycompany.com`,
and vice-versa.

### Behind load balancers

Sometimes clients use load balancers as an initial method to access one of the
Vault servers, but actually have direct access to each Vault node. In this
case, the Vault servers should actually be set up as described in the above
section, since for redirection purposes the clients have direct access.

However, if the only access to the Vault servers is via the load balancer, the
[`api_addr`](/vault/docs/configuration#api_addr) on each node should be the same: the
address of the load balancer. Clients that reach a standby node will be
redirected back to the load balancer; at that point hopefully the load
balancer's configuration will have been updated to know the address of the
current leader. This can cause a redirect loop and as such is not a recommended
setup when it can be avoided.

### Per-Node cluster listener addresses

Each [`listener`](/vault/docs/configuration/listener) block in Vault's configuration
file contains an [`address`](/vault/docs/configuration/listener/tcp#address) value on
which Vault listens for requests. Similarly, each
[`listener`](/vault/docs/configuration/listener) block can contain a
[`cluster_address`](/vault/docs/configuration/listener/tcp#cluster_address) on which
Vault listens for server-to-server cluster requests. If this value is not set,
its IP address will be automatically set to same as the
[`address`](/vault/docs/configuration/listener/tcp#address) value, and its port will
be automatically set to the same as the
[`address`](/vault/docs/configuration/listener/tcp#address) value plus one (so by
default, port `8201`).

Note that _only_ active nodes have active listeners. When a node becomes active
it will start cluster listeners, and when it becomes standby it will stop them.

### Per-Node cluster address

Similar to the [`api_addr`](/vault/docs/configuration#api_addr),
[`cluster_addr`](/vault/docs/configuration#cluster_addr) is the value that each node,
if active, should advertise to the standbys to use for server-to-server
communications, and lives as a top-level value in the configuration file. On
each node, this should be set to a host name or IP address that a standby can
use to reach one of that node's
[`cluster_address`](/vault/docs/configuration#cluster_address) values set in the
[`listener`](/vault/docs/configuration/listener) blocks, including port. (Note that
this will always be forced to `https` since only TLS connections are used
between servers.)

This value can also be specified by the `VAULT_CLUSTER_ADDR` environment
variable, which takes precedence.

## Storage support

Currently there are several storage backends that support high availability
mode, including [Consul](/vault/docs/configuration/storage/consul),
[ZooKeeper](/vault/docs/configuration/storage/zookeeper) and [etcd](/vault/docs/configuration/storage/etcd). These may
change over time, and the [configuration page](/vault/docs/configuration) should be
referenced.

HashiCorp recommends [Vault Integrated Storage](/vault/docs/configuration/storage/raft) as the default HA backend for new deployments of Vault. [Consul Storage Backend](/vault/docs/configuration/storage/consul) is also a supported option and used by many production deployments. See the [comparison chart](/vault/docs/configuration/storage#integrated-storage-vs-consul-as-vault-storage) for help deciding which option is best for you.

If you're interested in implementing another backend or adding HA support to
another backend, we'd love your contributions. Adding HA support requires
implementing the
[`physical.HABackend`](https://pkg.go.dev/github.com/hashicorp/vault/sdk/physical#HABackend)
interface for the storage backend.

layout  docs page title  High Availability description       Vault can be highly available  allowing you to run multiple Vaults to protect   against outages         High availability mode  HA   Vault supports a multi server mode for high availability  This mode protects against outages by running multiple Vault servers  High availability mode is automatically enabled when using a data store that supports it   You can tell if a data store supports high availability mode   HA   by starting the server and seeing if   HA available   is output next to the data store information  If it is  then Vault will automatically use HA mode  This information is also available on the  Configuration   vault docs configuration  page   To be highly available  one of the Vault server nodes grabs a lock within the data store  The successful server node then becomes the active node  all other nodes become standby nodes  At this point  if the standby nodes receive a request  they will either  forward the request   request forwarding  or  redirect the client   client redirection  depending on the current configuration and state of the cluster    see the sections below for details  Due to this architecture  HA does not enable increased scalability  In general  the bottleneck of Vault is the data store itself  not Vault core  For example  to increase the scalability of Vault with Consul  you would generally scale Consul instead of Vault   Certain storage backends can support high availability mode  which enable them to store both Vault s information in addition to the HA lock  However  Vault also supports split data HA mode  whereby the lock value and the rest of the data live separately  This can be done by specifying both the   storage    vault docs configuration storage  and   ha storage    vault docs configuration ha storage  stanzas in the configuration file with different backends  For instance  a Vault cluster can be set up to use Consul as the   ha storage    vault docs configuration ha storage  to manage the lock  and use Amazon S3 as the   storage    vault docs configuration storage  for all other persisted data   The sections below explain the server communication patterns and each type of request handling in more detail  At a minimum  the requirements for redirection mode must be met for an HA cluster to work successfully      Server to Server communication  Both methods of request handling rely on the active node advertising information about itself to the other nodes  Rather than over the network  this communication takes place within Vault s encrypted storage  the active node writes this information and unsealed standby Vault nodes can read it   For the client redirection method  this is the extent of server to server communication    no direct communication with only encrypted entries in the data store used to transfer state   For the request forwarding method  the servers need direct communication with each other  In order to perform this securely  the active node also advertises  via the encrypted data store entry  a newly generated private key  ECDSA P521  and a newly generated self signed certificate designated for client and server authentication  Each standby uses the private key and certificate to open a mutually authenticated TLS 1 2 connection to the active node via the advertised cluster address  When client requests come in  the requests are serialized  sent over this TLS protected communication channel  and acted upon by the active node  The active node then returns a response to the standby  which sends the response back to the requesting client      Request forwarding  If request forwarding is enabled  turned on by default in 0 6 2   clients can still force the older fallback redirection behavior  see below  if desired by setting the  X Vault No Request Forwarding  header to any non empty value   Successful cluster setup requires a few configuration parameters  although some can be automatically determined      Client redirection  If  X Vault No Request Forwarding  header in the request is set to a non empty value  the standby nodes will redirect the client using a  307  status code to the  active node s  redirect address   This is also the fallback method used when request forwarding is turned off or there is an error performing the forwarding  As such  a redirect address is always required for all HA setups   Some HA data store drivers can autodetect the redirect address  but it is often necessary to configure it manually via a top level value in the configuration file  The key for this value is   api addr    vault docs configuration api addr  and the value can also be specified by the  VAULT API ADDR  environment variable  which takes precedence   What the   api addr    vault docs configuration api addr  value should be set to depends on how Vault is set up  There are two common scenarios  Vault servers accessed directly by clients  and Vault servers accessed via a load balancer   In both cases  the   api addr    vault docs configuration api addr  should be a full URL including scheme   http   https    not simply an IP address and port       Direct access  When clients are able to access Vault directly  the   api addr    vault docs configuration api addr  for each node should be that node s address  For instance  if there are two Vault nodes      A   accessed via  https   a vault mycompany com 8200     B   accessed via  https   b vault mycompany com 8200   Then node  A  would set its   api addr    vault docs configuration api addr  to  https   a vault mycompany com 8200  and node  B  would set its   api addr    vault docs configuration api addr  to  https   b vault mycompany com 8200    This way  when  A  is the active node  any requests received by node  B  will cause it to redirect the client to node  A  s   api addr    vault docs configuration api addr  at  https   a vault mycompany com   and vice versa       Behind load balancers  Sometimes clients use load balancers as an initial method to access one of the Vault servers  but actually have direct access to each Vault node  In this case  the Vault servers should actually be set up as described in the above section  since for redirection purposes the clients have direct access   However  if the only access to the Vault servers is via the load balancer  the   api addr    vault docs configuration api addr  on each node should be the same  the address of the load balancer  Clients that reach a standby node will be redirected back to the load balancer  at that point hopefully the load balancer s configuration will have been updated to know the address of the current leader  This can cause a redirect loop and as such is not a recommended setup when it can be avoided       Per Node cluster listener addresses  Each   listener    vault docs configuration listener  block in Vault s configuration file contains an   address    vault docs configuration listener tcp address  value on which Vault listens for requests  Similarly  each   listener    vault docs configuration listener  block can contain a   cluster address    vault docs configuration listener tcp cluster address  on which Vault listens for server to server cluster requests  If this value is not set  its IP address will be automatically set to same as the   address    vault docs configuration listener tcp address  value  and its port will be automatically set to the same as the   address    vault docs configuration listener tcp address  value plus one  so by default  port  8201     Note that  only  active nodes have active listeners  When a node becomes active it will start cluster listeners  and when it becomes standby it will stop them       Per Node cluster address  Similar to the   api addr    vault docs configuration api addr     cluster addr    vault docs configuration cluster addr  is the value that each node  if active  should advertise to the standbys to use for server to server communications  and lives as a top level value in the configuration file  On each node  this should be set to a host name or IP address that a standby can use to reach one of that node s   cluster address    vault docs configuration cluster address  values set in the   listener    vault docs configuration listener  blocks  including port   Note that this will always be forced to  https  since only TLS connections are used between servers    This value can also be specified by the  VAULT CLUSTER ADDR  environment variable  which takes precedence      Storage support  Currently there are several storage backends that support high availability mode  including  Consul   vault docs configuration storage consul    ZooKeeper   vault docs configuration storage zookeeper  and  etcd   vault docs configuration storage etcd   These may change over time  and the  configuration page   vault docs configuration  should be referenced   HashiCorp recommends  Vault Integrated Storage   vault docs configuration storage raft  as the default HA backend for new deployments of Vault   Consul Storage Backend   vault docs configuration storage consul  is also a supported option and used by many production deployments  See the  comparison chart   vault docs configuration storage integrated storage vs consul as vault storage  for help deciding which option is best for you   If you re interested in implementing another backend or adding HA support to another backend  we d love your contributions  Adding HA support requires implementing the   physical HABackend   https   pkg go dev github com hashicorp vault sdk physical HABackend  interface for the storage backend


vault introduced in Vault 0 8 and may not be available in earlier releases Response wrapping layout docs page title Response Wrapping Wrapping responses in cubbyholes for secure distribution Note Some of this information relies on features of response wrapping tokens

Answer:
---
layout: docs
page_title: Response Wrapping
description: Wrapping responses in cubbyholes for secure distribution.
---

# Response wrapping

_Note_: Some of this information relies on features of response-wrapping tokens
introduced in Vault 0.8 and may not be available in earlier releases.

## Overview

In many Vault deployments, clients can access Vault directly and consume
returned secrets. In other situations, it may make sense to or be desired to
separate privileges such that one trusted entity is responsible for interacting
with most of the Vault API and passing secrets to the end consumer.

However, the more relays a secret travels through, the more possibilities for
accidental disclosure, especially if the secret is being transmitted in
plaintext. For instance, you may wish to get a TLS private key to a machine
that has been cold-booted, but since you do not want to store a decryption key
in persistent storage, you cannot encrypt this key in transit.

To help address this problem, Vault includes a feature called _response
wrapping_. When requested, Vault can take the response it would have sent to an
HTTP client and instead insert it into the
[`cubbyhole`](/vault/docs/secrets/cubbyhole) of a single-use token,
returning that single-use token instead.

Logically speaking, the response is
wrapped by the token, and retrieving it requires an unwrap operation against
this token. Functionally speaking, the token provides authorization to use
an encryption key from Vault's keyring to decrypt the data.

This provides a powerful mechanism for information sharing in many
environments. In the types of scenarios, described above, often the best
practical option is to provide _cover_ for the secret information, be able to
_detect malfeasance_ (interception, tampering), and limit _lifetime_ of the
secret's exposure. Response wrapping performs all three of these duties:

- It provides _cover_ by ensuring that the value being transmitted across the
  wire is not the actual secret but a reference to such a secret, namely the
  response-wrapping token. Information stored in logs or captured along the
  way do not directly see the sensitive information.
- It provides _malfeasance detection_ by ensuring that only a single party can
  ever unwrap the token and see what's inside. A client receiving a token that
  cannot be unwrapped can trigger an immediate security incident. In addition,
  a client can inspect a given token before unwrapping to ensure that its
  origin is from the expected location in Vault.
- It _limits the lifetime_ of secret exposure because the response-wrapping
  token has a lifetime that is separate from the wrapped secret (and often can
  be much shorter), so if a client fails to come up and unwrap the token, the
  token can expire very quickly.

## Response-Wrapping tokens

When a response is wrapped, the normal API response from Vault does not contain
the original secret, but rather contains a set of information related to the
response-wrapping token:

- TTL: The TTL of the response-wrapping token itself
- Token: The actual token value
- Creation Time: The time that the response-wrapping token was created
- Creation Path: The API path that was called in the original request
- Wrapped Accessor: If the wrapped response is an authentication response
  containing a Vault token, this is the value of the wrapped token's accessor.
  This is useful for orchestration systems (such as Nomad) to be able to control
  the lifetime of secrets based on their knowledge of the lifetime of jobs,
  without having to actually unwrap the response-wrapping token or gain
  knowledge of the token ID inside.

Vault currently does not provide signed response-wrapping tokens, as it
provides little extra protection. If you are being pointed to the correct Vault
server, token validation is performed by interacting with the server itself; a
signed token does not remove the need to validate the token with the server,
since the token is not carrying data but merely an access mechanism and the
server will not release data without validating it. If you are being attacked
and pointed to the wrong Vault server, the same attacker could trivially give
you the wrong signing public key that corresponds to the wrong Vault server.
You could cache a previously valid key, but could also cache a previously valid
address (and in most cases the Vault address will not change or will be set via
a service discovery mechanism). As such, we rely on the fact that the token
itself is not carrying authoritative data and do not sign it.

## Response-Wrapping token operations

Via the `sys/wrapping` path, several operations can be run against wrapping
tokens:

- Lookup (`sys/wrapping/lookup`): This allows fetching the response-wrapping
  token's creation time, creation path, and TTL. This path is unauthenticated
  and available to response-wrapping tokens themselves. In other words, a
  response-wrapping token holder wishing to perform validation is always
  allowed to look up the properties of the token.
- Unwrap (`sys/wrapping/unwrap`): Unwrap the token, returning the response
  inside. The response that is returned will be the original wire-format
  response; it can be used directly with API clients.
- Rewrap (`sys/wrapping/rewrap`): Allows migrating the wrapped data to a new
  response-wrapping token. This can be useful for long-lived secrets. For
  example, an organization may wish (or be required in a compliance scenario)
  to have the `pki` backend's root CA key be returned in a long-lived
  response-wrapping token to ensure that nobody has seen the key (easily
  verified by performing lookups on the response-wrapping token) but available
  for signing CRLs in case they ever accidentally change or lose the `pki`
  mount. Often, compliance schemes require periodic rotation of secrets, so
  this helps achieve that compliance goal without actually exposing what's
  inside.
- Wrap (`sys/wrapping/wrap`): A helper endpoint that echoes back the data sent
  to it in a response-wrapping token. Note that blocking access to this
  endpoint does not remove the ability for arbitrary data to be wrapped, as it
  can be done elsewhere in Vault.

## Response-Wrapping token creation

Response wrapping is per-request and is triggered by providing to Vault the
desired TTL for a response-wrapping token for that request. This is set by the
client using the `X-Vault-Wrap-TTL` header and can be either an integer number
of seconds or a string duration of seconds (`15s`), minutes (`20m`), or hours
(`25h`). When using the Vault CLI, you can set this via the `-wrap-ttl`
parameter. When using the Go API, wrapping is triggered by [setting a helper
function](https://godoc.org/github.com/hashicorp/vault/api#Client.SetWrappingLookupFunc)
that tells the API the conditions under which to request wrapping, by mapping
an operation and path to a desired TTL.

If a client requests wrapping:

1. The original HTTP response is serialized
2. A new single-use token is generated with the TTL supplied by the client
3. Internally, the original serialized response is stored in the single-use
   token's cubbyhole
4. A new response is generated, with the token ID, TTL, and path stored in the
   new response's wrap information object
5. The new response is returned to the caller

Note that policies can control minimum/maximum wrapping TTLs; see the [policies
concepts page](/vault/docs/concepts/policies) for
more information.

## Response-Wrapping token validation

Proper validation of response-wrapping tokens is essential to ensure that any
malfeasance is detected. It's also pretty straightforward.

Validation is best performed by the following steps:

1. If a client has been expecting delivery of a response-wrapping token and
   none arrives, this may be due to an attacker intercepting the token and then
   preventing it from traveling further. This should cause an alert to trigger
   an immediate investigation.
2. Perform a lookup on the response-wrapping token. This immediately tells you
   if the token has already been unwrapped or is expired (or otherwise
   revoked). If the lookup indicates that a token is invalid, it does not
   necessarily mean that the data was intercepted (for instance, perhaps the
   client took a long time to start up and the TTL expired) but should trigger
   an alert for immediate investigation, likely with the assistance of Vault's
   audit logs to see if the token really was unwrapped.
3. With the token information in hand, validate that the creation path matches
   expectations. If you expect to find a TLS key/certificate inside, chances
   are the path should be something like `pki/issue/...`. If the path is not
   what you expect, it is possible that the data contained inside was read and
   then put into a new response-wrapping token. (This is especially likely if
   the path starts with `cubbyhole` or `sys/wrapping/wrap`.) Particular care
   should be taken with `kv` secrets engine: exact matches on the path are best
   there. For example, if you expect a secret to come from `secret/foo` and
   the interceptor provides a token with `secret/bar` as the path, simply
   checking for a prefix of `secret/` is not enough.
4. After prefix validation, unwrap the token. If the unwrap fails, the response
   is similar to if the initial lookup fails: trigger an alert for immediate
   investigation.

Following those steps provides very strong assurance that the data contained
within the response-wrapping token has never been seen by anyone other than the
intended client and that any interception or tampering has resulted in a
security alert.

layout  docs page title  Response Wrapping description  Wrapping responses in cubbyholes for secure distribution         Response wrapping   Note   Some of this information relies on features of response wrapping tokens introduced in Vault 0 8 and may not be available in earlier releases      Overview  In many Vault deployments  clients can access Vault directly and consume returned secrets  In other situations  it may make sense to or be desired to separate privileges such that one trusted entity is responsible for interacting with most of the Vault API and passing secrets to the end consumer   However  the more relays a secret travels through  the more possibilities for accidental disclosure  especially if the secret is being transmitted in plaintext  For instance  you may wish to get a TLS private key to a machine that has been cold booted  but since you do not want to store a decryption key in persistent storage  you cannot encrypt this key in transit   To help address this problem  Vault includes a feature called  response wrapping   When requested  Vault can take the response it would have sent to an HTTP client and instead insert it into the   cubbyhole    vault docs secrets cubbyhole  of a single use token  returning that single use token instead   Logically speaking  the response is wrapped by the token  and retrieving it requires an unwrap operation against this token  Functionally speaking  the token provides authorization to use an encryption key from Vault s keyring to decrypt the data   This provides a powerful mechanism for information sharing in many environments  In the types of scenarios  described above  often the best practical option is to provide  cover  for the secret information  be able to  detect malfeasance   interception  tampering   and limit  lifetime  of the secret s exposure  Response wrapping performs all three of these duties     It provides  cover  by ensuring that the value being transmitted across the   wire is not the actual secret but a reference to such a secret  namely the   response wrapping token  Information stored in logs or captured along the   way do not directly see the sensitive information    It provides  malfeasance detection  by ensuring that only a single party can   ever unwrap the token and see what s inside  A client receiving a token that   cannot be unwrapped can trigger an immediate security incident  In addition    a client can inspect a given token before unwrapping to ensure that its   origin is from the expected location in Vault    It  limits the lifetime  of secret exposure because the response wrapping   token has a lifetime that is separate from the wrapped secret  and often can   be much shorter   so if a client fails to come up and unwrap the token  the   token can expire very quickly      Response Wrapping tokens  When a response is wrapped  the normal API response from Vault does not contain the original secret  but rather contains a set of information related to the response wrapping token     TTL  The TTL of the response wrapping token itself   Token  The actual token value   Creation Time  The time that the response wrapping token was created   Creation Path  The API path that was called in the original request   Wrapped Accessor  If the wrapped response is an authentication response   containing a Vault token  this is the value of the wrapped token s accessor    This is useful for orchestration systems  such as Nomad  to be able to control   the lifetime of secrets based on their knowledge of the lifetime of jobs    without having to actually unwrap the response wrapping token or gain   knowledge of the token ID inside   Vault currently does not provide signed response wrapping tokens  as it provides little extra protection  If you are being pointed to the correct Vault server  token validation is performed by interacting with the server itself  a signed token does not remove the need to validate the token with the server  since the token is not carrying data but merely an access mechanism and the server will not release data without validating it  If you are being attacked and pointed to the wrong Vault server  the same attacker could trivially give you the wrong signing public key that corresponds to the wrong Vault server  You could cache a previously valid key  but could also cache a previously valid address  and in most cases the Vault address will not change or will be set via a service discovery mechanism   As such  we rely on the fact that the token itself is not carrying authoritative data and do not sign it      Response Wrapping token operations  Via the  sys wrapping  path  several operations can be run against wrapping tokens     Lookup   sys wrapping lookup    This allows fetching the response wrapping   token s creation time  creation path  and TTL  This path is unauthenticated   and available to response wrapping tokens themselves  In other words  a   response wrapping token holder wishing to perform validation is always   allowed to look up the properties of the token    Unwrap   sys wrapping unwrap    Unwrap the token  returning the response   inside  The response that is returned will be the original wire format   response  it can be used directly with API clients    Rewrap   sys wrapping rewrap    Allows migrating the wrapped data to a new   response wrapping token  This can be useful for long lived secrets  For   example  an organization may wish  or be required in a compliance scenario    to have the  pki  backend s root CA key be returned in a long lived   response wrapping token to ensure that nobody has seen the key  easily   verified by performing lookups on the response wrapping token  but available   for signing CRLs in case they ever accidentally change or lose the  pki    mount  Often  compliance schemes require periodic rotation of secrets  so   this helps achieve that compliance goal without actually exposing what s   inside    Wrap   sys wrapping wrap    A helper endpoint that echoes back the data sent   to it in a response wrapping token  Note that blocking access to this   endpoint does not remove the ability for arbitrary data to be wrapped  as it   can be done elsewhere in Vault      Response Wrapping token creation  Response wrapping is per request and is triggered by providing to Vault the desired TTL for a response wrapping token for that request  This is set by the client using the  X Vault Wrap TTL  header and can be either an integer number of seconds or a string duration of seconds   15s    minutes   20m    or hours   25h    When using the Vault CLI  you can set this via the   wrap ttl  parameter  When using the Go API  wrapping is triggered by  setting a helper function  https   godoc org github com hashicorp vault api Client SetWrappingLookupFunc  that tells the API the conditions under which to request wrapping  by mapping an operation and path to a desired TTL   If a client requests wrapping   1  The original HTTP response is serialized 2  A new single use token is generated with the TTL supplied by the client 3  Internally  the original serialized response is stored in the single use    token s cubbyhole 4  A new response is generated  with the token ID  TTL  and path stored in the    new response s wrap information object 5  The new response is returned to the caller  Note that policies can control minimum maximum wrapping TTLs  see the  policies concepts page   vault docs concepts policies  for more information      Response Wrapping token validation  Proper validation of response wrapping tokens is essential to ensure that any malfeasance is detected  It s also pretty straightforward   Validation is best performed by the following steps   1  If a client has been expecting delivery of a response wrapping token and    none arrives  this may be due to an attacker intercepting the token and then    preventing it from traveling further  This should cause an alert to trigger    an immediate investigation  2  Perform a lookup on the response wrapping token  This immediately tells you    if the token has already been unwrapped or is expired  or otherwise    revoked   If the lookup indicates that a token is invalid  it does not    necessarily mean that the data was intercepted  for instance  perhaps the    client took a long time to start up and the TTL expired  but should trigger    an alert for immediate investigation  likely with the assistance of Vault s    audit logs to see if the token really was unwrapped  3  With the token information in hand  validate that the creation path matches    expectations  If you expect to find a TLS key certificate inside  chances    are the path should be something like  pki issue       If the path is not    what you expect  it is possible that the data contained inside was read and    then put into a new response wrapping token   This is especially likely if    the path starts with  cubbyhole  or  sys wrapping wrap    Particular care    should be taken with  kv  secrets engine  exact matches on the path are best    there  For example  if you expect a secret to come from  secret foo  and    the interceptor provides a token with  secret bar  as the path  simply    checking for a prefix of  secret   is not enough  4  After prefix validation  unwrap the token  If the unwrap fails  the response    is similar to if the initial lookup fails  trigger an alert for immediate    investigation   Following those steps provides very strong assurance that the data contained within the response wrapping token has never been seen by anyone other than the intended client and that any interception or tampering has resulted in a security alert