vault include alerts enterprise only mdx Learn the details about long term support for Vault Enterprise layout docs Long term support for Vault page title Long term support for Vault

Answer:
---
layout: docs
page_title: Long-term support for Vault
description: >-
  Learn the details about long-term support for Vault Enterprise.
---

# Long-term support for Vault

@include 'alerts/enterprise-only.mdx'

Long-term support (LTS) eases upgrade requirements for installations that cannot
upgrade frequently, quickly, or easily.

## LTS summary

      Question
      Answer

      Who should consider long-term support?

      Enterprise customers using Vault for sensitive or critical workflows.

      What is long-term support?

      Extended maintenance for select, major Vault Enterprise versions.
      By default, HashiCorp maintains Vault Enterprise versions for one year,
      which includes feature updates and critical patches. LTS extends
      maintenance for an additional year with critical patches.

      Where do I enable long-term support?

      You do not need to download a separate binary or set a flag for long-term
      support. As long as you select an LTS Vault Enterprise version when
      you install or upgrade your
      Vault instance, LTS is included.

      When are LTS versions released?

      As of Vault Enterprise 1.16, the first major release of a calendar year includes
      long-term support.

      Why is there a risk to updating to a non-LTS Vault Enterprise version?

      If you upgrade to a non-LTS Vault Enterprise version, your Vault instance
      will stop receiving critical updates when that version leaves the default
      maintenance window.

      How do I update my LTS Vault Enterprise installation?

      Follow your existing Vault upgrade process, but allow extra time for the
      possibility of transitional upgrades across multiple Vault versions.

## Who should consider long-term support?

Vault upgrades are challenging, especially for sensitive or critical workflows,
extensive integrations, and large-scale deployments. Strict upgrade policies
also require significant planning, testing, and employee hours to execute
successfully.

Customers who need assurances that their current installation will receive
critical bug fixes and security patches with minimal service disruptions should
consider moving to a Vault Enterprise version with long-term support.

## What is long-term support?

Long-term support offers extended maintenance through minor releases for select,
major Vault Enterprise versions.

The standard [support period and end of life policy](https://go.hashi.co/vault-support-policy)
covers "N−2" versions, which means, at any given time, HashiCorp maintains
the current version ("N") and the two previous versions ("N−2").

Vault versions typically update 3 times per calendar year (CY), which means that
**standard maintenance** for a given Vault version lasts approximately 1 year.
After the first year, LTS Vault versions move from standard maintenance to
**extended maintenance** for three additional major version releases (approximately one additional year)
with patches for bugs that may cause outages and critical vulnerabilities and exposures (CVEs).

Maintenance updates               | Standard maintenance | Extended maintenance
--------------------------------- | -------------------- | --------------------
Performance improvements          | YES                  | NO
Bug fixes                         | YES                  | OUTAGE-RISK ONLY
Security patches                  | YES                  | HIGH-RISK ONLY
CVE patches                       | YES                  | YES

## Where do I enable long-term support?

You do not need to download a separate binary or set a flag for long-term
support. As long as you select an LTS Vault Enterprise version
(e.g., 1.16, 1.19) when you [install](/vault/install) or [upgrade](/vault/docs/upgrading) your
Vault instance, LTS is included.

## When are LTS versions released?

As of Vault Enterprise 1.16, the first release of a calendar year includes
long-term support.

LTS versions overlap by one year with the previous LTS version entering its
extended maintenance window when the new LTS version begins its standard
maintenance window.

## Why is there a risk to updating to a non-LTS Vault Enterprise version?

Long-term support is intended for Enterprise customers who cannot upgrade
frequently enough to stay within the standard maintenance timeline of one year.
The goal is to establish a predictable upgrade path with a longer timeline
rather than extending the lifetime for every Vault version.

Long-term support ensures your Vault Enterprise version continues to receive
critical patches for an additional three major version releases (approximately one additional year).
If you upgrade to a non-LTS version,you are moving your Vault instance to a version
that lacks extended support. Non-LTS versions stop receiving updates once they leave
the standard maintenance window.

@include 'assets/lts-upgrade-path.mdx'

Version | Expected release | Standard maintenance ends  | Extended maintenance ends
------- | ---------------- | -------------------------- | ---------------------
1.19    | CY25 Q1          | CY26 Q1 (1.22 release)     | CY27 Q1 (1.25 release)
1.18    | CY24 Q3          | CY25 Q3 (1.21 release)     | Not provided
1.17    | CY24 Q2          | CY25 Q2 (1.20 release)     | Not provided
1.16    | CY24 Q1          | CY25 Q1 (1.19 release)     | CY26 Q1 (1.22 release)

If a newer version of Vault Enterprise includes features you want to take
advantage of, you have two options:

1. Wait for the next available LTS release to maintain long-term support.
1. Upgrade immediately, then upgrade to an LTS release before the standard
   maintenance window expires.

## How do I upgrade my Vault Enterprise LTS installation?

You should follow your existing upgrade process for major version upgrades but
allow additional time. Upgrading from version LTS to LTS+1 translates to jumping
3 major Vault Enterprise versions, which **may** require transitional upgrades
to move through the intermediate Vault versions.

layout  docs page title  Long term support for Vault description       Learn the details about long term support for Vault Enterprise         Long term support for Vault   include  alerts enterprise only mdx   Long term support  LTS  eases upgrade requirements for installations that cannot upgrade frequently  quickly  or easily      LTS summary   table      thead       tr         th style  Question  th         th style  Answer  th        tr      thead     tbody     tr       td style          a href   who  Who should consider long term support   a        td       td style         Enterprise customers using Vault for sensitive or critical workflows        td      tr     tr       td style          a href   what  What is long term support   a        td       td style         Extended maintenance for select  major Vault Enterprise versions        By default  HashiCorp maintains Vault Enterprise versions for one year        which includes feature updates and critical patches  LTS extends       maintenance for an additional year with critical patches        td      tr     tr       td style          a href   where  Where do I enable long term support   a        td       td style         You do not need to download a separate binary or set a flag for long term       support  As long as you select an LTS Vault Enterprise version when       you  a href   vault install  install  a  or  a href   vault docs upgrading  upgrade  a  your       Vault instance  LTS is included        td      tr     tr       td style          a href   when  When are LTS versions released   a        td       td style         As of Vault Enterprise 1 16  the first major release of a calendar year includes       long term support        td      tr     tr       td style          a href   why  Why is there a risk to updating to a non LTS Vault Enterprise version   a        td       td style         If you upgrade to a non LTS Vault Enterprise version  your Vault instance       will stop receiving critical updates when that version leaves the default       maintenance window        td      tr     tr       td style          a href   how  How do I update my LTS Vault Enterprise installation   a        td       td style         Follow your existing Vault upgrade process  but allow extra time for the       possibility of transitional upgrades across multiple Vault versions        td      tr    tbody    table    a id  who         Who should consider long term support   Vault upgrades are challenging  especially for sensitive or critical workflows  extensive integrations  and large scale deployments  Strict upgrade policies also require significant planning  testing  and employee hours to execute successfully   Customers who need assurances that their current installation will receive critical bug fixes and security patches with minimal service disruptions should consider moving to a Vault Enterprise version with long term support    a id  what         What is long term support   Long term support offers extended maintenance through minor releases for select  major Vault Enterprise versions   The standard  support period and end of life policy  https   go hashi co vault support policy  covers  N minus 2  versions  which means  at any given time  HashiCorp maintains the current version   N   and the two previous versions   N minus 2     Vault versions typically update 3 times per calendar year  CY   which means that   standard maintenance   for a given Vault version lasts approximately 1 year  After the first year  LTS Vault versions move from standard maintenance to   extended maintenance   for three additional major version releases  approximately one additional year  with patches for bugs that may cause outages and critical vulnerabilities and exposures  CVEs    Maintenance updates                 Standard maintenance   Extended maintenance                                                                                Performance improvements            YES                    NO Bug fixes                           YES                    OUTAGE RISK ONLY Security patches                    YES                    HIGH RISK ONLY CVE patches                         YES                    YES   a id  where         Where do I enable long term support   You do not need to download a separate binary or set a flag for long term support  As long as you select an LTS Vault Enterprise version  e g   1 16  1 19  when you  install   vault install  or  upgrade   vault docs upgrading  your Vault instance  LTS is included    a id  when         When are LTS versions released   As of Vault Enterprise 1 16  the first release of a calendar year includes long term support   LTS versions overlap by one year with the previous LTS version entering its extended maintenance window when the new LTS version begins its standard maintenance window    a id  why         Why is there a risk to updating to a non LTS Vault Enterprise version   Long term support is intended for Enterprise customers who cannot upgrade frequently enough to stay within the standard maintenance timeline of one year  The goal is to establish a predictable upgrade path with a longer timeline rather than extending the lifetime for every Vault version   Long term support ensures your Vault Enterprise version continues to receive critical patches for an additional three major version releases  approximately one additional year   If you upgrade to a non LTS version you are moving your Vault instance to a version that lacks extended support  Non LTS versions stop receiving updates once they leave the standard maintenance window    include  assets lts upgrade path mdx   Version   Expected release   Standard maintenance ends    Extended maintenance ends                                                                                1 19      CY25 Q1            CY26 Q1  1 22 release        CY27 Q1  1 25 release  1 18      CY24 Q3            CY25 Q3  1 21 release        Not provided 1 17      CY24 Q2            CY25 Q2  1 20 release        Not provided 1 16      CY24 Q1            CY25 Q1  1 19 release        CY26 Q1  1 22 release   If a newer version of Vault Enterprise includes features you want to take advantage of  you have two options   1  Wait for the next available LTS release to maintain long term support  1  Upgrade immediately  then upgrade to an LTS release before the standard    maintenance window expires    a id  how         How do I upgrade my Vault Enterprise LTS installation   You should follow your existing upgrade process for major version upgrades but allow additional time  Upgrading from version LTS to LTS 1 translates to jumping 3 major Vault Enterprise versions  which   may   require transitional upgrades to move through the intermediate Vault versions


vault page title Vault Enterprise Control Groups layout docs include alerts enterprise and hcp mdx Vault Enterprise has support for Control Group Authorization Vault Enterprise control groups

Answer:
---
layout: docs
page_title: Vault Enterprise Control Groups
description: Vault Enterprise has support for Control Group Authorization.
---

# Vault Enterprise control groups

@include 'alerts/enterprise-and-hcp.mdx'

Vault Enterprise has support for Control Group Authorization. Control Groups
add additional authorization factors to be required before satisfying a request.

When a Control Group is required for a request, a limited duration response
wrapping token is returned to the user instead of the requested data. The
accessor of the response wrapping token can be passed to the authorizers
required by the control group policy. Once all authorizations are satisfied,
the wrapping token can be used to unwrap and process the original request.

## Control group factors

Control Groups can verify the following factors:

- `Identity Groups` - Require an authorizer to be in a specific set of identity
  groups.

### Controlled capabilities

Control group factors can be configured to trigger the control group workflow
on specific capabilities. This is done with the `controlled_capabilities` field.
Not specifying the `controlled_capabilities` field will necessitate the factor to be
checked for all operations to the specified policy path. The `controlled_capabilities`
field can differ per factor, so that different factors can be required for different
operations.

Finally, the capabilities in the `controlled_capabilities` stanza must be a subset of the
`capabilities` specified in the policy itself. For example, a policy giving only `read` access to
the path `secret/foo` cannot specify a control group factor with `list` as a controlled capability.

Please see the following section for examples using ACL policies.

## Control groups in ACL policies

Control Group requirements on paths are specified as `control_group` along
with other ACL parameters.

### Sample ACL policies

```
path "secret/foo" {
    capabilities = ["read"]
    control_group = {
        factor "ops_manager" {
            identity {
                group_names = ["managers"]
                approvals = 1
            }
        }
    }
}
```

The above policy grants `read` access to `secret/foo` only after one member of
the "managers" group authorizes the request.

```
path "secret/foo" {
    capabilities = ["create", "update"]
    control_group = {
        ttl = "4h"
        factor "tech leads" {
            identity {
                group_names = ["managers", "leads"]
                approvals = 2
            }
        }
        factor "super users" {
            identity {
                group_names = ["superusers"]
                approvals = 1
            }
        }
    }
}
```

The above policy grants `create` and `update` access to `secret/foo` only after
two (2) members of the "managers" or "leads" group and one member of the "superusers"
group authorizes the request. If an authorizer is a member of both the
"managers" and "superusers" group, one authorization for both factors will be
satisfied.

```
path "secret/foo" {
    capabilities = ["write","read"]
    control_group = {
        factor "admin" {
            controlled_capabilities = ["write"]
            identity {
                group_names = ["admin"]
                approvals = 1
            }
        }
    }
}
```

The above policy grants `read` access to `secret/foo` for anyone that has a vault token
with this policy. It grants `write` access to `secret/foo` only after one member from the
admin group authorizes the request.

```
path "kv/*" {
    capabilities = ["create", "update","delete","list","sudo"]
    control_group = {
        factor "admin" {
            controlled_capabilities = ["delete","list","sudo"]
            identity {
                group_names = ["admin"]
                approvals = 1
            }
        }
    }
}
path "kv/*" {
    capabilities = ["create"]
    control_group = {
        factor "superuser" {
            identity {
                group_names = ["superuser"]
                approvals = 2
            }
        }
    }
}

```

Because the second path stanza has a control group factor with no `controlled_capabilities` field,
any token with this policy will be required to get two (2) approvals from the "superuser" group before executing
any operation against `kv/*`. In addition, by virtue of the `controlled_capabilities` field in the first
path stanza, `delete`,`list`, and `sudo` operations will require an additional approval from the "admin" group.

```
path "kv/*" {
    capabilities = ["read", "list", "create"]
    control_group = {
        controlled_capabilities = ["read"]
        factor "admin" {
            identity {
                group_names = ["admin"]
                approvals = 1
            }
        }
        factor "superuser" {
            controlled_capabilities = ["create"]
            identity {
                group_names = ["superuser"]
                approvals = 1
            }
        }
    }
}
```

In this case, `read` will require one admin approval and `create` will require
one superuser approval and one admin approval. `List` will require no extra approvals
from any of the control group factors, and a token with this policy will not be required
to go through the control group workflow in order to execute a read operation against `kv/*`.

## Control groups in Sentinel

Control Groups are also supported in Sentinel policies using the `controlgroup`
import. See [Sentinel Documentation](/vault/docs/enterprise/sentinel) for more
details on available properties.

### Sample Sentinel policy

```
import "time"
import "controlgroup"

control_group = func() {
    numAuthzs = 0
    for controlgroup.authorizations as authz {
		if "managers" in authz.groups.by_name {
			if time.load(authz.time).unix > time.now.unix - 3600 {
				numAuthzs = numAuthzs + 1
			}
		}
    }
    if numAuthzs >= 2 {
        return true
    }
    return false
}

main = rule {
    control_group()
}
```

The above policy will reject the request unless two members of the "managers"
group have authorized the request. Additionally it verifies the authorizations
happened in the last hour.

## Tutorial

Refer to the [Control Groups](/vault/tutorials/enterprise/control-groups)
tutorial to learn how to implement dual controller authorization within your policies.

## API

Control Groups can be managed over the HTTP API. Please see
[Control Groups API](/vault/api-docs/system/control-group) for more details.

layout  docs page title  Vault Enterprise Control Groups description  Vault Enterprise has support for Control Group Authorization         Vault Enterprise control groups   include  alerts enterprise and hcp mdx   Vault Enterprise has support for Control Group Authorization  Control Groups add additional authorization factors to be required before satisfying a request   When a Control Group is required for a request  a limited duration response wrapping token is returned to the user instead of the requested data  The accessor of the response wrapping token can be passed to the authorizers required by the control group policy  Once all authorizations are satisfied  the wrapping token can be used to unwrap and process the original request      Control group factors  Control Groups can verify the following factors      Identity Groups    Require an authorizer to be in a specific set of identity   groups       Controlled capabilities  Control group factors can be configured to trigger the control group workflow on specific capabilities  This is done with the  controlled capabilities  field  Not specifying the  controlled capabilities  field will necessitate the factor to be checked for all operations to the specified policy path  The  controlled capabilities  field can differ per factor  so that different factors can be required for different operations   Finally  the capabilities in the  controlled capabilities  stanza must be a subset of the  capabilities  specified in the policy itself  For example  a policy giving only  read  access to the path  secret foo  cannot specify a control group factor with  list  as a controlled capability   Please see the following section for examples using ACL policies      Control groups in ACL policies  Control Group requirements on paths are specified as  control group  along with other ACL parameters       Sample ACL policies      path  secret foo        capabilities     read       control group             factor  ops manager                identity                   group names     managers                   approvals   1                                      The above policy grants  read  access to  secret foo  only after one member of the  managers  group authorizes the request       path  secret foo        capabilities     create    update       control group             ttl    4h          factor  tech leads                identity                   group names     managers    leads                   approvals   2                                 factor  super users                identity                   group names     superusers                   approvals   1                                      The above policy grants  create  and  update  access to  secret foo  only after two  2  members of the  managers  or  leads  group and one member of the  superusers  group authorizes the request  If an authorizer is a member of both the  managers  and  superusers  group  one authorization for both factors will be satisfied       path  secret foo        capabilities     write   read       control group             factor  admin                controlled capabilities     write               identity                   group names     admin                   approvals   1                                      The above policy grants  read  access to  secret foo  for anyone that has a vault token with this policy  It grants  write  access to  secret foo  only after one member from the admin group authorizes the request       path  kv          capabilities     create    update   delete   list   sudo       control group             factor  admin                controlled capabilities     delete   list   sudo               identity                   group names     admin                   approvals   1                                 path  kv          capabilities     create       control group             factor  superuser                identity                   group names     superuser                   approvals   2                                       Because the second path stanza has a control group factor with no  controlled capabilities  field  any token with this policy will be required to get two  2  approvals from the  superuser  group before executing any operation against  kv     In addition  by virtue of the  controlled capabilities  field in the first path stanza   delete   list   and  sudo  operations will require an additional approval from the  admin  group       path  kv          capabilities     read    list    create       control group             controlled capabilities     read           factor  admin                identity                   group names     admin                   approvals   1                                 factor  superuser                controlled capabilities     create               identity                   group names     superuser                   approvals   1                                      In this case   read  will require one admin approval and  create  will require one superuser approval and one admin approval   List  will require no extra approvals from any of the control group factors  and a token with this policy will not be required to go through the control group workflow in order to execute a read operation against  kv         Control groups in Sentinel  Control Groups are also supported in Sentinel policies using the  controlgroup  import  See  Sentinel Documentation   vault docs enterprise sentinel  for more details on available properties       Sample Sentinel policy      import  time  import  controlgroup   control group   func         numAuthzs   0     for controlgroup authorizations as authz     if  managers  in authz groups by name      if time load authz time  unix   time now unix   3600       numAuthzs   numAuthzs   1                    if numAuthzs    2           return true           return false    main   rule       control group          The above policy will reject the request unless two members of the  managers  group have authorized the request  Additionally it verifies the authorizations happened in the last hour      Tutorial  Refer to the  Control Groups   vault tutorials enterprise control groups  tutorial to learn how to implement dual controller authorization within your policies      API  Control Groups can be managed over the HTTP API  Please see  Control Groups API   vault api docs system control group  for more details


vault Vault Enterprise Consistency Model page title Vault Enterprise Eventual Consistency layout docs include alerts enterprise and hcp mdx Vault eventual consistency

Answer:
---
layout: docs
page_title: Vault Enterprise Eventual Consistency
description: Vault Enterprise Consistency Model
---

# Vault eventual consistency

@include 'alerts/enterprise-and-hcp.mdx'

When running in a cluster, Vault has an eventual consistency model.
Only one node (the leader) can write to Vault's storage.
Users generally expect read-after-write consistency: in other
words, after writing foo=1, a subsequent read of foo should return 1. Depending
on the Vault configuration this isn't always the case. When using performance
standbys with Integrated Storage, or when using performance replication,
there are some sequences of operations that don't always yield read-after-write
consistency.

## Performance standby nodes

When using the Integrated Storage backend without performance standbys, only
a single Vault node (the active node) handles requests. Requests sent to
regular standbys are handled by forwarding them to the active node. This Vault configuration
gives Vault the same behavior as the default Consul consistency model.

When using the Integrated Storage backend with performance standbys, both the
active node and performance standbys can handle requests. If a performance standby
handles a login request, or a request that generates a dynamic secret, the
performance standby will issue a remote procedure call (RPC) to the active node to store the token
and/or lease. If the performance standby handles any other request that
results in a storage write, it will forward that request to the active node
in the same way a regular standby forwards all requests.

With Integrated Storage, all writes occur on the active node, which then issues
RPCs to update the local storage on every other node. Between when the active
node writes the data to its local disk, and when those RPCs are handled on the
other nodes to write the data to their local disks, those nodes present a stale
view of the data.

As a result, even if you're always talking to the same performance standby,
you may not get read-after-write semantics. The write gets sent to the active
node, and if the subsequent read request occurs before the new data gets sent
to the node handling the read request, the read request won't be able to take
the write into account because the new data isn't present on that node yet.

## Performance replication

A similar phenomenon occurs when using performance replication. One example
of how this manifests is when using shared mounts. If a KV secrets engine
is mounted on the primary with `local=false`, it will exist on the secondary
cluster as well. The secondary cluster can handle requests to that mount,
though as with performance standbys, write requests must be forwarded - in
this case to the primary active node. Once data is written to the primary cluster,
it won't be visible on the secondary cluster until the data has been replicated
from the primary. Therefore, on the secondary cluster, it initially appears as if
the data write hasn't happened.

If the secondary cluster is using Integrated Storage, and the read request is
being handled on one of its performance standbys, the problem is exacerbated because it
has to be sent first from the primary active node to the secondary active node,
and then from there to the secondary performance standby, each of which can
introduce their own form of lag.

Even without shared secret engines, stale reads can still happen with performance
replication. The Identity subsystem aims to provide a view on entities and
groups which span across clusters. As such, when logging in to a secondary cluster
using a shared mount, Vault tries to generate an entity and alias if they don't
already exist, and these must be stored on the primary using an RPC. Something
similar happens with groups.

## Clock skew and replication lag

As seen above, both performance standbys and replication secondaries can lag
behind the active node or the primary.  As of Vault 1.17, it's possible to get
some insight into that lag using sys/health, sys/ha-status, and the replication
status endpoints.

Secondaries and standbys regularly issue an "echo" heartbeat RPC to their upstream
source.  This heartbeat serves many purposes, one of them being to get a rough
idea of whether the clocks of the client and server are in sync.  The server
response to the heartbeat RPC includes the server's local clock time, and the
client takes the delta in milliseconds between that time and the client's local
clock time to compute the clock_skew_ms field.  No effort is made to factor into
that field the time it took to actually perform the RPC, though that information
is made available as the last_heartbeat_duration_ms field.  In other words, the
reported clock skew has an uncertainty of up to last_heartbeat_duration_ms.

Vault assumes that clocks are synced across all nodes in a cluster, and if they
aren't, problems may arise, e.g. one node may think that a lease has expired and
another node won't yet.  Some community-supported storage backends may have further
problems relating to HA mode.

There are fewer problems expected when clock skew exists between a replication primary
and secondary.  However, one known issue is that the replication lag canary discussed
next will produce surprising values if clocks aren't synced between the clusters.

Non-secondary active nodes periodically write a small record to storage containing the
local clock time for that node.  Replication secondaries read that record and compare
it to their local clock time, calling the delta the replication_primary_canary_age_ms,
which is exposed in the replication status endpoints.  Performance standbys do the same
computation, exposing replication_primary_canary_age_ms in the sys/health and
sys/ha-status endpoints.  Performance standbys and replication secondaries include
their current replication_primary_canary_age_ms as part of their payload for the
aforementioned "echo" heartbeat RPCs they issue, allowing the active node or primary
cluster to report on the lag seen by their downstream clients.

## Mitigations

There has long been a partial mitigation for the above problems. When writing
data via RPC, e.g. when a performance standby registers tokens and leases on the
active node after a login or generating a dynamic secret, part of the response
includes a number known as the "WAL index", aka Write-Ahead Log index.

A full explanation of this is outside the scope of this document, but the short
version is that both performance replication and performance standbys use log
shipping to stay in sync with the upstream source of writes. The mitigation
historically used by nodes doing writes via RPC is to look at the WAL index in
the response and wait up to 2 seconds to see if that WAL index appear in the
logs being shipped from upstream. Once the WAL index is seen, the Vault node
handling the request that resulted in RPCs can return its own response to the
client: it knows that any subsequent reads will be able to see the value that
was just written. If the WAL index isn't seen within those 2 seconds, the Vault
node completes the request anyway, returning a warning in the response.

This mitigation option still exists in Vault 1.7, though now there is a
configuration option to adjust the wait time:
[best_effort_wal_wait_duration](/vault/docs/configuration/replication).

## Vault 1.7 mitigations

There are now a variety of other mitigations available:

- per-request option to always forward the request to the active node
- per-request option to conditionally forward the request to the active node
  if it would otherwise result in a stale read
- per-request option to fail requests if they might result in a stale read
- Vault Proxy configuration to do the above for proxied requests

The remainder of this document describes the tradeoffs of these mitigations and
how to use them.

Note that any headers requesting forwarding are disabled by default, and must
be enabled using [allow_forwarding_via_header](/vault/docs/configuration/replication).

### Unconditional forwarding (Performance standbys only)

The simplest solution to never experience stale reads from a performance standby
is to provide the following HTTP header in the request:

```
X-Vault-Forward: active-node
```

The drawback here is that if all your requests are forwarded to the active node,
you might as well not be using performance standbys. So this mitigation only
makes sense to use selectively.

This mitigation will not help with stale reads relating to performance replication.

### Conditional forwarding (Performance standbys only)

As of Vault Enterprise 1.7, all requests that modify storage now return a new
HTTP response header:

```
X-Vault-Index: <base64 value>
```

To ensure that the state resulting from that write request is visible to a
subsequent request, add these headers to that second request:

```
X-Vault-Index: <base64 value taken from previous response>
X-Vault-Inconsistent: forward-active-node
```

The effect will be that the node handling the request will look at the state
it has locally, and if it doesn't contain the state described by the X-Vault-Index
header, the node will forward the request to the active node.

The drawback here is that when requests are forwarded to the active node,
performance standbys provide less value. If this happens often enough
the active node can become a bottleneck, limiting the horizontal read scalability
performance standbys are intended to provide.

### Retry stale requests

As of Vault Enterprise 1.7, all requests that modify storage now return a new
HTTP response header:

```
X-Vault-Index: <base64 value>
```

To ensure that the state resulting from that write request is visible to a
subsequent request, add this headers to that second request:

```
X-Vault-Index: <base64 value taken from previous response>
```

When the desired state isn't present, Vault will return a failure response with
HTTP status code 412. This tells the client that it should retry the request.
The advantage over the Conditional Forwarding solution above is twofold:
first, there's no additional load on the active node. Second, this solution
is applicable to performance replication as well as performance standbys.

The Vault Go API will now automatically retry 412s, and provides convenience
methods for propagating the X-Vault-Index response header into the request
header of subsequent requests. Those not using the Vault Go API will want
to build equivalent functionality into their client library.

### Vault proxy and consistency headers

When configured, the [Vault API Proxy](/vault/docs/agent-and-proxy/proxy/apiproxy) will proxy incoming requests to Vault. There is
Proxy configuration available in the `api_proxy` stanza that allows making use
of some of the above mitigations without modifying clients.

By setting `enforce_consistency="always"`, Proxy will always provide
the `X-Vault-Index` consistency header. The value it uses for the header
will be based on the responses that have passed through the Proxy previously.

The option `when_inconsistent` controls how stale reads are prevented:

- `"fail"` means that when a `412` response is seen, it is returned to the client
- `"retry"` means that `412` responses will be retried automatically by Proxy,
  so the client doesn't have to deal with them
- `"forward"` makes Proxy provide the
  `X-Vault-Inconsistent: forward-active-node` header as described above under
  Conditional Forwarding

## Vault 1.10 mitigations

In Vault 1.10, the token format has changed, where service tokens now employ server side consistency.
This means that by default, requests made
to nodes which cannot support read-after-write consistency due to
not having the necessary WAL index to check Vault tokens locally will output
a 412 status code. The Vault Go API automatically retries when receiving 412s, so
unless there is a considerable replication delay, users will experience
read-after-write consistency.

The replication option [allow_forwarding_via_token](/vault/docs/configuration/replication)
can be used to enforce requests that would have returned 412s in the
aforementioned way will be forwarded instead to the active node.

Refer to the [Server Side Consistent Token FAQ](/vault/docs/faq/ssct) for details.

## Client API helpers

There are some new helpers in the `api` package to work with the new headers.
`WithRequestCallbacks` and `WithResponseCallbacks` create a shallow clone of
the client and populate it with the given callbacks. `RecordState` and
`RequireState` are used to store the response header from one request and
provide it in a subsequent request. For example:

```go
client := api.NewClient(api.DefaultConfig)
var state string
_, err := client.WithResponseCallbacks(api.RecordState(&state)).Write(path, data)
secret, err := client.WithRequestCallbacks(api.RequireState(state)).Read(path)
```

This will retry the `Read` until the data stored by the `Write` is present.
There are also callbacks to use forwarding: `ForwardInconsistent` and
`ForwardAlways`.

layout  docs page title  Vault Enterprise Eventual Consistency description  Vault Enterprise Consistency Model        Vault eventual consistency   include  alerts enterprise and hcp mdx   When running in a cluster  Vault has an eventual consistency model  Only one node  the leader  can write to Vault s storage  Users generally expect read after write consistency  in other words  after writing foo 1  a subsequent read of foo should return 1  Depending on the Vault configuration this isn t always the case  When using performance standbys with Integrated Storage  or when using performance replication  there are some sequences of operations that don t always yield read after write consistency      Performance standby nodes  When using the Integrated Storage backend without performance standbys  only a single Vault node  the active node  handles requests  Requests sent to regular standbys are handled by forwarding them to the active node  This Vault configuration gives Vault the same behavior as the default Consul consistency model   When using the Integrated Storage backend with performance standbys  both the active node and performance standbys can handle requests  If a performance standby handles a login request  or a request that generates a dynamic secret  the performance standby will issue a remote procedure call  RPC  to the active node to store the token and or lease  If the performance standby handles any other request that results in a storage write  it will forward that request to the active node in the same way a regular standby forwards all requests   With Integrated Storage  all writes occur on the active node  which then issues RPCs to update the local storage on every other node  Between when the active node writes the data to its local disk  and when those RPCs are handled on the other nodes to write the data to their local disks  those nodes present a stale view of the data   As a result  even if you re always talking to the same performance standby  you may not get read after write semantics  The write gets sent to the active node  and if the subsequent read request occurs before the new data gets sent to the node handling the read request  the read request won t be able to take the write into account because the new data isn t present on that node yet      Performance replication  A similar phenomenon occurs when using performance replication  One example of how this manifests is when using shared mounts  If a KV secrets engine is mounted on the primary with  local false   it will exist on the secondary cluster as well  The secondary cluster can handle requests to that mount  though as with performance standbys  write requests must be forwarded   in this case to the primary active node  Once data is written to the primary cluster  it won t be visible on the secondary cluster until the data has been replicated from the primary  Therefore  on the secondary cluster  it initially appears as if the data write hasn t happened   If the secondary cluster is using Integrated Storage  and the read request is being handled on one of its performance standbys  the problem is exacerbated because it has to be sent first from the primary active node to the secondary active node  and then from there to the secondary performance standby  each of which can introduce their own form of lag   Even without shared secret engines  stale reads can still happen with performance replication  The Identity subsystem aims to provide a view on entities and groups which span across clusters  As such  when logging in to a secondary cluster using a shared mount  Vault tries to generate an entity and alias if they don t already exist  and these must be stored on the primary using an RPC  Something similar happens with groups      Clock skew and replication lag  As seen above  both performance standbys and replication secondaries can lag behind the active node or the primary   As of Vault 1 17  it s possible to get some insight into that lag using sys health  sys ha status  and the replication status endpoints   Secondaries and standbys regularly issue an  echo  heartbeat RPC to their upstream source   This heartbeat serves many purposes  one of them being to get a rough idea of whether the clocks of the client and server are in sync   The server response to the heartbeat RPC includes the server s local clock time  and the client takes the delta in milliseconds between that time and the client s local clock time to compute the clock skew ms field   No effort is made to factor into that field the time it took to actually perform the RPC  though that information is made available as the last heartbeat duration ms field   In other words  the reported clock skew has an uncertainty of up to last heartbeat duration ms   Vault assumes that clocks are synced across all nodes in a cluster  and if they aren t  problems may arise  e g  one node may think that a lease has expired and another node won t yet   Some community supported storage backends may have further problems relating to HA mode   There are fewer problems expected when clock skew exists between a replication primary and secondary   However  one known issue is that the replication lag canary discussed next will produce surprising values if clocks aren t synced between the clusters   Non secondary active nodes periodically write a small record to storage containing the local clock time for that node   Replication secondaries read that record and compare it to their local clock time  calling the delta the replication primary canary age ms  which is exposed in the replication status endpoints   Performance standbys do the same computation  exposing replication primary canary age ms in the sys health and sys ha status endpoints   Performance standbys and replication secondaries include their current replication primary canary age ms as part of their payload for the aforementioned  echo  heartbeat RPCs they issue  allowing the active node or primary cluster to report on the lag seen by their downstream clients      Mitigations  There has long been a partial mitigation for the above problems  When writing data via RPC  e g  when a performance standby registers tokens and leases on the active node after a login or generating a dynamic secret  part of the response includes a number known as the  WAL index   aka Write Ahead Log index   A full explanation of this is outside the scope of this document  but the short version is that both performance replication and performance standbys use log shipping to stay in sync with the upstream source of writes  The mitigation historically used by nodes doing writes via RPC is to look at the WAL index in the response and wait up to 2 seconds to see if that WAL index appear in the logs being shipped from upstream  Once the WAL index is seen  the Vault node handling the request that resulted in RPCs can return its own response to the client  it knows that any subsequent reads will be able to see the value that was just written  If the WAL index isn t seen within those 2 seconds  the Vault node completes the request anyway  returning a warning in the response   This mitigation option still exists in Vault 1 7  though now there is a configuration option to adjust the wait time   best effort wal wait duration   vault docs configuration replication       Vault 1 7 mitigations  There are now a variety of other mitigations available     per request option to always forward the request to the active node   per request option to conditionally forward the request to the active node   if it would otherwise result in a stale read   per request option to fail requests if they might result in a stale read   Vault Proxy configuration to do the above for proxied requests  The remainder of this document describes the tradeoffs of these mitigations and how to use them   Note that any headers requesting forwarding are disabled by default  and must be enabled using  allow forwarding via header   vault docs configuration replication        Unconditional forwarding  Performance standbys only   The simplest solution to never experience stale reads from a performance standby is to provide the following HTTP header in the request       X Vault Forward  active node      The drawback here is that if all your requests are forwarded to the active node  you might as well not be using performance standbys  So this mitigation only makes sense to use selectively   This mitigation will not help with stale reads relating to performance replication       Conditional forwarding  Performance standbys only   As of Vault Enterprise 1 7  all requests that modify storage now return a new HTTP response header       X Vault Index   base64 value       To ensure that the state resulting from that write request is visible to a subsequent request  add these headers to that second request       X Vault Index   base64 value taken from previous response  X Vault Inconsistent  forward active node      The effect will be that the node handling the request will look at the state it has locally  and if it doesn t contain the state described by the X Vault Index header  the node will forward the request to the active node   The drawback here is that when requests are forwarded to the active node  performance standbys provide less value  If this happens often enough the active node can become a bottleneck  limiting the horizontal read scalability performance standbys are intended to provide       Retry stale requests  As of Vault Enterprise 1 7  all requests that modify storage now return a new HTTP response header       X Vault Index   base64 value       To ensure that the state resulting from that write request is visible to a subsequent request  add this headers to that second request       X Vault Index   base64 value taken from previous response       When the desired state isn t present  Vault will return a failure response with HTTP status code 412  This tells the client that it should retry the request  The advantage over the Conditional Forwarding solution above is twofold  first  there s no additional load on the active node  Second  this solution is applicable to performance replication as well as performance standbys   The Vault Go API will now automatically retry 412s  and provides convenience methods for propagating the X Vault Index response header into the request header of subsequent requests  Those not using the Vault Go API will want to build equivalent functionality into their client library       Vault proxy and consistency headers  When configured  the  Vault API Proxy   vault docs agent and proxy proxy apiproxy  will proxy incoming requests to Vault  There is Proxy configuration available in the  api proxy  stanza that allows making use of some of the above mitigations without modifying clients   By setting  enforce consistency  always    Proxy will always provide the  X Vault Index  consistency header  The value it uses for the header will be based on the responses that have passed through the Proxy previously   The option  when inconsistent  controls how stale reads are prevented       fail   means that when a  412  response is seen  it is returned to the client     retry   means that  412  responses will be retried automatically by Proxy    so the client doesn t have to deal with them     forward   makes Proxy provide the    X Vault Inconsistent  forward active node  header as described above under   Conditional Forwarding     Vault 1 10 mitigations  In Vault 1 10  the token format has changed  where service tokens now employ server side consistency  This means that by default  requests made to nodes which cannot support read after write consistency due to not having the necessary WAL index to check Vault tokens locally will output a 412 status code  The Vault Go API automatically retries when receiving 412s  so unless there is a considerable replication delay  users will experience read after write consistency   The replication option  allow forwarding via token   vault docs configuration replication  can be used to enforce requests that would have returned 412s in the aforementioned way will be forwarded instead to the active node   Refer to the  Server Side Consistent Token FAQ   vault docs faq ssct  for details      Client API helpers  There are some new helpers in the  api  package to work with the new headers   WithRequestCallbacks  and  WithResponseCallbacks  create a shallow clone of the client and populate it with the given callbacks   RecordState  and  RequireState  are used to store the response header from one request and provide it in a subsequent request  For example      go client    api NewClient api DefaultConfig  var state string    err    client WithResponseCallbacks api RecordState  state   Write path  data  secret  err    client WithRequestCallbacks api RequireState state   Read path       This will retry the  Read  until the data stored by the  Write  is present  There are also callbacks to use forwarding   ForwardInconsistent  and  ForwardAlways


vault encryption for supporting seals Seal wrap layout docs page title Vault Enterprise Seal Wrap Vault Enterprise features a mechanism to wrap values with an extra layer of

Answer:
---
layout: docs
page_title: Vault Enterprise Seal Wrap
description: |-
  Vault Enterprise features a mechanism to wrap values with an extra layer of
  encryption for supporting seals.
---

# Seal wrap

@include 'alerts/enterprise-and-hcp.mdx'

Vault Enterprise features a mechanism to wrap values with an extra layer of
encryption for supporting [seals](/vault/docs/configuration/seal). This adds an
extra layer of protection and is useful in some compliance and regulatory
environments, including FIPS 140-2 environments.

To use this feature, you must have an active or trial license for Vault
Enterprise Plus (HSMs). To start a trial, contact [HashiCorp
sales](mailto:sales@hashicorp.com).

## Seal Wrap benefits

Your Vault deployments can gain the following benefits by enabling seal wrapping:

- Conformance with FIPS 140-2 directives on Key Storage and Key Transport as [certified by Leidos](/vault/docs/enterprise/sealwrap#fips-140-2-compliance)
- Supports FIPS level of security equal to HSM
  - For example, if you use Level 3 hardware encryption on an HSM, Vault will be
    using FIPS 140-2 Level 3 cryptography
- Enables Vault deployments in high security [GRC](https://en.wikipedia.org/wiki/Governance,_risk_management,_and_compliance)
  environments (e.g. PCI-DSS, HIPAA) where FIPS guidelines important for external audits
- Pathway to use Vault for managing Department of Defense (DOD) or North
  Atlantic Treaty Organization (NATO) military secrets

## Enabling/Disabling

Seal Wrap is enabled by default on supporting seals. This implies that the seal
must be available throughout Vault's runtime. Most cloud-based seals should be
quite reliable, but, for instance, if using an HSM in a non-HA setup a
connection interruption to the HSM will result in issues with Vault
functionality.

<Tip>

Having Vault generate its own key is the easiest way to get up and running, but for security, Vault marks the key as non-exportable. If your HSM key backup strategy requires the key to be exportable, you should generate the key yourself. Refer to the [key generation attributes](/vault/docs/configuration/seal/pkcs11#vault-key-generation-attributes).

</Tip>

To disable seal wrapping, set `disable_sealwrap = true` in Vault's
[configuration file][configuration]. This will not affect auto-unsealing functionality; Vault's
root key will still be protected by the seal wrapping mechanism. It will
simply prevent other storage entries within Vault from being seal wrapped.

_N.B._: This is a lazy downgrade; as keys are accessed or written their seal
wrapping status will change. Similarly, if the flag is removed, it will be a
lazy upgrade (which is the case when initially upgrading to a seal
wrap-supporting version of Vault).

## Activating seal wrapping

For some values, seal wrapping is always enabled with a supporting seal. This
includes the recovery key, any stored key shares, the root key, the keyring,
and more; essentially, any Critical Security Parameter (CSP) within Vault's
core. If upgrading from a version of Vault that did not support seal wrapping,
the next time these values are read they will be seal-wrapped and stored.

Backend mounts within Vault can also take advantage of seal wrapping. Seal
wrapping can be activated at mount time for a given mount by mounting the
backend with the `seal_wrap` configuration value set to `true`. (This value
cannot currently be changed later.)

A given backend's author can specify which values should be seal-wrapped by
identifying where CSPs are stored.  They may also choose to seal wrap all or none
of their values.

Note that it is often an order of magnitude or two slower to write to and read
from HSMs or remote seals. However, values will be cached in memory
un-seal-wrapped (but still encrypted by Vault's built-in cryptographic barrier)
in Vault, which will mitigate this for read-heavy workloads.

## Seal wrap and replication

Seal wrapping takes place below the replication logic. As a result, it is
transparent to replication. Replication will convey which values should be
seal-wrapped, but it is up to the seal on the local cluster to implement it.
In practice, this means that seal wrapping can be used without needing to have
the replicated keys on both ends of the connection; each cluster can have
distinct keys in an HSM or in KMS.

In addition, it is possible to replicate from a Shamir-protected primary
cluster to clusters that use HSMs when seal wrapping is required in downstream
datacenters but not in the primary.

## Wrapped parameters

Each plugin (whether secret or auth) maintains control over parameters it
feels are best to Seal Wrap. These are usually just a few core values as
Seal Wrapping does incur some performance overhead.

Some examples of places where seal wrapping is used include:

- The [LDAP](/vault/docs/auth/ldap), [RADIUS](/vault/docs/auth/radius),
  [Okta](/vault/docs/auth/okta), and [AWS](/vault/docs/auth/aws) auth methods,
  for storing their config.
- [PKI](/vault/docs/secrets/pki) for storing the issuers and their keys,
- [SSH](/vault/docs/secrets/ssh) for storing the CA's keys,
- [KMIP](/vault/docs/secrets/kmip) for storing managed objects (externally-provided
  keys) and its CA keys.
- [Transit](/vault/docs/secrets/transit) for storing keys and their policy.

## FIPS status

See the [FIPS-specific Seal Wrap documentation](/vault/docs/enterprise/fips/sealwrap)
for more information about using Seal Wrapping to achieve FIPS 140-2 compliance.
Note that there are additional [FIPS considerations](/vault/docs/enterprise/sealwrap#seal-wrap-and-replication)
regarding Seal Wrap usage and Vault Replication.

[configuration]: /vault/docs/configuration

layout  docs page title  Vault Enterprise Seal Wrap description       Vault Enterprise features a mechanism to wrap values with an extra layer of   encryption for supporting seals         Seal wrap   include  alerts enterprise and hcp mdx   Vault Enterprise features a mechanism to wrap values with an extra layer of encryption for supporting  seals   vault docs configuration seal   This adds an extra layer of protection and is useful in some compliance and regulatory environments  including FIPS 140 2 environments   To use this feature  you must have an active or trial license for Vault Enterprise Plus  HSMs   To start a trial  contact  HashiCorp sales  mailto sales hashicorp com       Seal Wrap benefits  Your Vault deployments can gain the following benefits by enabling seal wrapping     Conformance with FIPS 140 2 directives on Key Storage and Key Transport as  certified by Leidos   vault docs enterprise sealwrap fips 140 2 compliance    Supports FIPS level of security equal to HSM     For example  if you use Level 3 hardware encryption on an HSM  Vault will be     using FIPS 140 2 Level 3 cryptography   Enables Vault deployments in high security  GRC  https   en wikipedia org wiki Governance  risk management  and compliance    environments  e g  PCI DSS  HIPAA  where FIPS guidelines important for external audits   Pathway to use Vault for managing Department of Defense  DOD  or North   Atlantic Treaty Organization  NATO  military secrets     Enabling Disabling  Seal Wrap is enabled by default on supporting seals  This implies that the seal must be available throughout Vault s runtime  Most cloud based seals should be quite reliable  but  for instance  if using an HSM in a non HA setup a connection interruption to the HSM will result in issues with Vault functionality    Tip   Having Vault generate its own key is the easiest way to get up and running  but for security  Vault marks the key as non exportable  If your HSM key backup strategy requires the key to be exportable  you should generate the key yourself  Refer to the  key generation attributes   vault docs configuration seal pkcs11 vault key generation attributes      Tip   To disable seal wrapping  set  disable sealwrap   true  in Vault s  configuration file  configuration   This will not affect auto unsealing functionality  Vault s root key will still be protected by the seal wrapping mechanism  It will simply prevent other storage entries within Vault from being seal wrapped    N B    This is a lazy downgrade  as keys are accessed or written their seal wrapping status will change  Similarly  if the flag is removed  it will be a lazy upgrade  which is the case when initially upgrading to a seal wrap supporting version of Vault       Activating seal wrapping  For some values  seal wrapping is always enabled with a supporting seal  This includes the recovery key  any stored key shares  the root key  the keyring  and more  essentially  any Critical Security Parameter  CSP  within Vault s core  If upgrading from a version of Vault that did not support seal wrapping  the next time these values are read they will be seal wrapped and stored   Backend mounts within Vault can also take advantage of seal wrapping  Seal wrapping can be activated at mount time for a given mount by mounting the backend with the  seal wrap  configuration value set to  true    This value cannot currently be changed later    A given backend s author can specify which values should be seal wrapped by identifying where CSPs are stored   They may also choose to seal wrap all or none of their values   Note that it is often an order of magnitude or two slower to write to and read from HSMs or remote seals  However  values will be cached in memory un seal wrapped  but still encrypted by Vault s built in cryptographic barrier  in Vault  which will mitigate this for read heavy workloads      Seal wrap and replication  Seal wrapping takes place below the replication logic  As a result  it is transparent to replication  Replication will convey which values should be seal wrapped  but it is up to the seal on the local cluster to implement it  In practice  this means that seal wrapping can be used without needing to have the replicated keys on both ends of the connection  each cluster can have distinct keys in an HSM or in KMS   In addition  it is possible to replicate from a Shamir protected primary cluster to clusters that use HSMs when seal wrapping is required in downstream datacenters but not in the primary      Wrapped parameters  Each plugin  whether secret or auth  maintains control over parameters it feels are best to Seal Wrap  These are usually just a few core values as Seal Wrapping does incur some performance overhead   Some examples of places where seal wrapping is used include     The  LDAP   vault docs auth ldap    RADIUS   vault docs auth radius      Okta   vault docs auth okta   and  AWS   vault docs auth aws  auth methods    for storing their config     PKI   vault docs secrets pki  for storing the issuers and their keys     SSH   vault docs secrets ssh  for storing the CA s keys     KMIP   vault docs secrets kmip  for storing managed objects  externally provided   keys  and its CA keys     Transit   vault docs secrets transit  for storing keys and their policy      FIPS status  See the  FIPS specific Seal Wrap documentation   vault docs enterprise fips sealwrap  for more information about using Seal Wrapping to achieve FIPS 140 2 compliance  Note that there are additional  FIPS considerations   vault docs enterprise sealwrap seal wrap and replication  regarding Seal Wrap usage and Vault Replication    configuration    vault docs configuration


vault include alerts enterprise only mdx Learn what data HashiCorp collects to meter Enterprise license utilization Enable or disable reporting Review sample payloads and logs Automated license utilization reporting layout docs page title Automated license utilization reporting

Answer:
---
layout: docs
page_title: Automated license utilization reporting
description: >-
  Learn what data HashiCorp collects to meter Enterprise license utilization. Enable or disable reporting. Review sample payloads and logs.
---

# Automated license utilization reporting

@include 'alerts/enterprise-only.mdx'

Automated license utilization reporting sends license utilization data to
HashiCorp without requiring you to manually collect and report them. It also
lets you review your license usage with the monitoring solution you already use
(for example Splunk, Datadog, or others) so you can optimize and manage your
deployments. Use these reports to understand how much more you can deploy under
your current contract, protect against overutilization, and budget for predicted
consumption.

Automated reporting shares the minimum data required to validate license
utilization as defined in our contracts. They consist of mostly computed metrics
and will never contain Personal Identifiable Information (PII) or other
sensitive information. Automated reporting shares the data with HashiCorp using
a secure, unidirectional HTTPS API and makes an auditable record in the product
logs each time it submits a report. The reporting process is GDPR compliant and submits
reports roughly once every 24 hours.

## Enable automated reporting

To enable automated reporting, you need to make sure that outbound network
traffic is configured correctly and upgrade your enterprise product to a version
that supports it. If your installation is air-gapped or network settings are not
in place, automated reporting will not work.

### 1. Allow outbound HTTPS traffic on port 443

Make sure that your network allows HTTPS egress on port 443 to
https://reporting.hashicorp.services by allow-listing the following IP
addresses:

- 100.20.70.12
- 35.166.5.222
- 23.95.85.111
- 44.215.244.1

### 2. Upgrade

Upgrade to a release that supports license utilization reporting. These
releases include:

- [Vault Enterprise 1.14.0](https://releases.hashicorp.com/vault/) and later
- [Vault Enterprise 1.13.4](https://releases.hashicorp.com/vault/) and later 1.13.x versions
- [Vault Enterprise 1.12.8](https://releases.hashicorp.com/vault/) and later 1.12.x versions
- [Vault Enterprise 1.11.12](https://releases.hashicorp.com/vault/)

### 3. Check logs

Automatic license utilization reporting will start sending data within roughly 24 hours.
Check the server logs for records that the data sent successfully.

You will find log entries similar to the following:

<CodeBlockConfig hideClipboard>

```
[DEBUG] core.reporting: beginning snapshot export
[DEBUG] core.reporting: creating payload
[DEBUG] core.reporting: marshalling payload to json
[DEBUG] core.reporting: generating authentication headers
[DEBUG] core.reporting: creating request
[DEBUG] core.reporting: sending request
[DEBUG] core.reporting: performing request: method=POST url=https://reporting.hashicorp.services
[DEBUG] core.reporting: recording audit record
[INFO]  core.reporting: Report sent: auditRecord="{\"payload\":{\"payload_version\":\"1\",\"license_id\":\"97afe7b4-b9c8-bf19-bf35-b89b5cc0efea\",\"product\":\"vault\",\"product_version\":\"1.14.0-rc1+ent\",\"export_timestamp\":\"2023-06-01T09:34:44.215133-04:00\",\"snapshots\":[{\"snapshot_version\":1,\"snapshot_id\":\"0001J7H7KMEDRXKM5C1QJGBXV3\",\"process_id\":\"01H1T45CZK2GN9WR22863W2K32\",\"timestamp\":\"2023-06-01T09:34:44.215001-04:00\",\"schema_version\":\"1.0.0\",\"service\":\"vault\",\"metrics\":{\"clientcount.current_month_estimate\":{\"key\":\"clientcount.current_month_estimate\",\"kind\":\"sum\",\"mode\":\"write\",\"labels\":{\"type\":{\"entity\":20,\"nonentity\":11}}},\"clientcount.previous_month_complete\":{\"key\":\"clientcount.previous_month_complete\",\"kind\":\"sum\",\"mode\":\"write\",\"labels\":{\"type\":{\"entity\":10,\"nonentity\":11}}}}}],\"metadata\":{\"vault\":{\"billing_start\":\"2023-03-01T00:00:00Z\",\"cluster_id\":\"a8d95acc-ec0a-6087-d7f6-4f054ab2e7fd\"}}}}"
[DEBUG] core.reporting: completed recording audit record
[DEBUG] core.reporting: export finished successfully
```

</CodeBlockConfig>

If your installation is air-gapped or your network doesn’t allow the correct
egress, logs will show an error.

<CodeBlockConfig hideClipboard>

```
[DEBUG] core.reporting: beginning snapshot export
[DEBUG] core.reporting: creating payload
[DEBUG] core.reporting: marshalling payload to json
[DEBUG] core.reporting: generating authentication headers
[DEBUG] core.reporting: creating request
[DEBUG] core.reporting: sending request
[DEBUG] core.reporting: performing request: method=POST url=https://reporting.hashicorp.services
[DEBUG] core.reporting: error status code received: statusCode=403
```

</CodeBlockConfig>

In this case, reconfigure your network to allow egress and check back in 24
hours.

## Opt out

If your installation is air-gapped or you want to manually collect and report on
the same license utilization metrics, you can opt-out of automated reporting.

Manually reporting these metrics can be time consuming. Opting out of automated
reporting does not mean that you also opt out from sending license utilization
metrics. Customers who opt out of automated reporting will still be required to
manually collect and send license utilization metrics to HashiCorp.

If you are considering opting out because you’re worried about the data, we
strongly recommend that you review the [example payloads](#example-payloads)
before opting out. If you have concerns with any of the automatically-reported
data please bring them to your account manager.

You have two options to opt out of automated reporting:

- HCL configuration (recommended)
- Environment variable (requires restart)

#### HCL configuration

Opting out in your product’s configuration file doesn’t require a system
restart, and is the method we recommend. Add the following block to your server
configuration file (e.g. `vault-config.hcl`).

```hcl
reporting {
	license {
		enabled = false
   }
}
```

<Warning>

When you have a cluster, each node must have the reporting stanza in its
configuration to be consistent. In the event of leadership change, nodes will
use its server configuration to determine whether or not to opt-out the
automated reporting. Inconsistent configuration between nodes will change the
reporting status upon active unseal.

</Warning>

You will find the following entries in the server log.

<CodeBlockConfig hideClipboard>

```
[DEBUG] core: reloading automated reporting
[INFO]  core: opting out of automated reporting
[DEBUG] activity: there is no reporting agent configured, skipping counts reporting
```

</CodeBlockConfig>

#### Environment variable

If you need to, you can also opt out using an environment variable, which will
provide a startup message confirming that you have disabled automated reporting.
This option requires a system restart.

<Note>

If the reporting stanza exists in the configuration file, the
`OPTOUT_LICENSE_REPORTING` value overrides the configuration.

</Note>

Set the following environment variable.

```shell-session
$ export OPTOUT_LICENSE_REPORTING=true
```

Now, restart your [Vault servers](/vault/docs/commands/server) from the shell
where you set the environment variable.

You will find the following entries in the server log.

<CodeBlockConfig hideClipboard>

```
[INFO]  core: automated reporting disabled via environment variable: env=OPTOUT_LICENSE_REPORTING
[INFO]  core: opting out of automated reporting
[DEBUG] activity: there is no reporting agent configured, skipping counts reporting
```

</CodeBlockConfig>

Check your product logs roughly 24 hours after opting out to make sure that the system
isn’t trying to send reports.

If your configuration file and environment variable differ, the environment
variable setting will take precedence.

## Example payloads

HashiCorp collects the following utilization data as JSON payloads:

- `payload_version` - The version of this payload schema
- `license_id` - The license ID for this product
- `product` - The product that this contribution is for
- `product_version` - The product version this contribution is for
- `export_timestamp`- The date and time for this contribution
- `snapshots` - An array of snapshot details. A snapshot is a structure that
  represents a single data collection
   - `snapshot_version` - The version of the snapshot package that produced this
     snapshot
   - `snapshot_id` - A unique identifier for this particular snapshot
   - `process_id` - An identifier for the system that produced this snapshot
   - `timestamp` - The date and time for this snapshot
   - `schema_version` - The version of the schema associated with this snapshot
   - `service` - The service that produced this snapshot (likely to be product
     name)
   - `metrics` - A map of representations of snapshot metrics contained within
     this snapshot
      - `key` - The key name associated with this metric
      - `kind` - The kind of metric (feature, counter, sum, or mean)
      - `mode` - The mode of operation associated with this metric (write or
        collect)
      - `labels` - The labels associated with each collected metric
         - `entity` - The sum of tokens generated for a unique client identifier
         - `nonentity` - The sum of tokens without an entity attached
- `metadata` - Optional product-specific metadata
   - `billing_start` - The billing start date associated with the reporting cluster (license start date if not configured).

   <Note title="Important change to supported versions">

      As of 1.16.7, 1.17.3 and later,
      the billing start date automatically
      rolls over to the latest billing year at the end of the last cycle.

      For more information, refer to the upgrade guide for your Vault version:

      [Vault v1.16.x](/vault/docs/upgrading/upgrade-to-1.16.x#auto-rolled-billing-start-date),
      [Vault v1.17.x](/vault/docs/upgrading/upgrade-to-1.17.x#auto-rolled-billing-start-date)

    </Note>

   - `cluster_id` - The cluster UUID as shown by `vault status` on the reporting
     cluster

<CodeBlockConfig hideClipboard>

```json
{
  "payload_version": "1",
  "license_id": "97afe7b4-b9c8-bf19-bf35-b89b5cc0efea",
  "product": "vault",
  "product_version": "1.14.0-rc1+ent",
  "export_timestamp": "2023-06-01T11:39:00.76643-04:00",
  "snapshots": [
    {
      "snapshot_version": 1,
      "snapshot_id": "0001J7HEWM1PEHPMF5YZT8EV65",
      "process_id": "01H1VSQMNYAP77R566F1Y03GE6",
      "timestamp": "2023-06-01T11:39:00.766099-04:00",
      "schema_version": "1.0.0",
      "service": "vault",
      "metrics": {
        "clientcount.current_month_estimate": {
          "key": "clientcount.current_month_estimate",
          "kind": "sum",
          "mode": "write",
          "labels": {
            "type": {
              "entity": 20,
              "nonentity": 11
            }
          }
        },
        "clientcount.previous_month_complete": {
          "key": "clientcount.previous_month_complete",
          "kind": "sum",
          "mode": "write",
          "labels": {
            "type": {
              "entity": 10,
              "nonentity": 11
            }
          }
        }
      }
    }
  ],
  "metadata": {
    "vault": {
      "billing_start": "2023-03-01T00:00:00Z",
      "cluster_id": "a8d95acc-ec0a-6087-d7f6-4f054ab2e7fd"
    }
  }
}
```

</CodeBlockConfig>

## Pre-1.9 counts

When upgrading Vault from 1.8 (or earlier) to 1.9 (or later), utilization reporting will only include the [non-entity tokens](/vault/docs/concepts/client-count#non-entity-tokens) that are used after the upgrade.

Starting in Vault 1.9, the activity log records and de-duplicates non-entity tokens by using the namespace and token's policies to generate a unique identifier. Because Vault did not create identifiers for these tokens before 1.9, the activity log cannot know whether this token has been seen pre-1.9. To prevent inaccurate and inflated counts, the activity log will ignore any counts of non-entity tokens that were created before the upgrade and only the non-entity tokens from versions 1.9 and later will be counted.

See the client count [overview](/vault/docs/concepts/client-count) and [FAQ](/vault/docs/concepts/client-count/faq) for more information.

layout  docs page title  Automated license utilization reporting description       Learn what data HashiCorp collects to meter Enterprise license utilization  Enable or disable reporting  Review sample payloads and logs         Automated license utilization reporting   include  alerts enterprise only mdx   Automated license utilization reporting sends license utilization data to HashiCorp without requiring you to manually collect and report them  It also lets you review your license usage with the monitoring solution you already use  for example Splunk  Datadog  or others  so you can optimize and manage your deployments  Use these reports to understand how much more you can deploy under your current contract  protect against overutilization  and budget for predicted consumption   Automated reporting shares the minimum data required to validate license utilization as defined in our contracts  They consist of mostly computed metrics and will never contain Personal Identifiable Information  PII  or other sensitive information  Automated reporting shares the data with HashiCorp using a secure  unidirectional HTTPS API and makes an auditable record in the product logs each time it submits a report  The reporting process is GDPR compliant and submits reports roughly once every 24 hours      Enable automated reporting  To enable automated reporting  you need to make sure that outbound network traffic is configured correctly and upgrade your enterprise product to a version that supports it  If your installation is air gapped or network settings are not in place  automated reporting will not work       1  Allow outbound HTTPS traffic on port 443  Make sure that your network allows HTTPS egress on port 443 to https   reporting hashicorp services by allow listing the following IP addresses     100 20 70 12   35 166 5 222   23 95 85 111   44 215 244 1      2  Upgrade  Upgrade to a release that supports license utilization reporting  These releases include      Vault Enterprise 1 14 0  https   releases hashicorp com vault   and later    Vault Enterprise 1 13 4  https   releases hashicorp com vault   and later 1 13 x versions    Vault Enterprise 1 12 8  https   releases hashicorp com vault   and later 1 12 x versions    Vault Enterprise 1 11 12  https   releases hashicorp com vault        3  Check logs  Automatic license utilization reporting will start sending data within roughly 24 hours  Check the server logs for records that the data sent successfully   You will find log entries similar to the following    CodeBlockConfig hideClipboard        DEBUG  core reporting  beginning snapshot export  DEBUG  core reporting  creating payload  DEBUG  core reporting  marshalling payload to json  DEBUG  core reporting  generating authentication headers  DEBUG  core reporting  creating request  DEBUG  core reporting  sending request  DEBUG  core reporting  performing request  method POST url https   reporting hashicorp services  DEBUG  core reporting  recording audit record  INFO   core reporting  Report sent  auditRecord     payload      payload version     1     license id     97afe7b4 b9c8 bf19 bf35 b89b5cc0efea     product     vault     product version     1 14 0 rc1 ent     export timestamp     2023 06 01T09 34 44 215133 04 00     snapshots       snapshot version   1   snapshot id     0001J7H7KMEDRXKM5C1QJGBXV3     process id     01H1T45CZK2GN9WR22863W2K32     timestamp     2023 06 01T09 34 44 215001 04 00     schema version     1 0 0     service     vault     metrics      clientcount current month estimate      key     clientcount current month estimate     kind     sum     mode     write     labels      type      entity   20   nonentity   11      clientcount previous month complete      key     clientcount previous month complete     kind     sum     mode     write     labels      type      entity   10   nonentity   11         metadata      vault      billing start     2023 03 01T00 00 00Z     cluster id     a8d95acc ec0a 6087 d7f6 4f054ab2e7fd         DEBUG  core reporting  completed recording audit record  DEBUG  core reporting  export finished successfully        CodeBlockConfig   If your installation is air gapped or your network doesn t allow the correct egress  logs will show an error    CodeBlockConfig hideClipboard        DEBUG  core reporting  beginning snapshot export  DEBUG  core reporting  creating payload  DEBUG  core reporting  marshalling payload to json  DEBUG  core reporting  generating authentication headers  DEBUG  core reporting  creating request  DEBUG  core reporting  sending request  DEBUG  core reporting  performing request  method POST url https   reporting hashicorp services  DEBUG  core reporting  error status code received  statusCode 403        CodeBlockConfig   In this case  reconfigure your network to allow egress and check back in 24 hours      Opt out  If your installation is air gapped or you want to manually collect and report on the same license utilization metrics  you can opt out of automated reporting   Manually reporting these metrics can be time consuming  Opting out of automated reporting does not mean that you also opt out from sending license utilization metrics  Customers who opt out of automated reporting will still be required to manually collect and send license utilization metrics to HashiCorp   If you are considering opting out because you re worried about the data  we strongly recommend that you review the  example payloads   example payloads  before opting out  If you have concerns with any of the automatically reported data please bring them to your account manager   You have two options to opt out of automated reporting     HCL configuration  recommended    Environment variable  requires restart         HCL configuration  Opting out in your product s configuration file doesn t require a system restart  and is the method we recommend  Add the following block to your server configuration file  e g   vault config hcl        hcl reporting    license     enabled   false              Warning   When you have a cluster  each node must have the reporting stanza in its configuration to be consistent  In the event of leadership change  nodes will use its server configuration to determine whether or not to opt out the automated reporting  Inconsistent configuration between nodes will change the reporting status upon active unseal     Warning    You will find the following entries in the server log    CodeBlockConfig hideClipboard        DEBUG  core  reloading automated reporting  INFO   core  opting out of automated reporting  DEBUG  activity  there is no reporting agent configured  skipping counts reporting        CodeBlockConfig        Environment variable  If you need to  you can also opt out using an environment variable  which will provide a startup message confirming that you have disabled automated reporting  This option requires a system restart    Note   If the reporting stanza exists in the configuration file  the  OPTOUT LICENSE REPORTING  value overrides the configuration     Note   Set the following environment variable      shell session   export OPTOUT LICENSE REPORTING true      Now  restart your  Vault servers   vault docs commands server  from the shell where you set the environment variable   You will find the following entries in the server log    CodeBlockConfig hideClipboard        INFO   core  automated reporting disabled via environment variable  env OPTOUT LICENSE REPORTING  INFO   core  opting out of automated reporting  DEBUG  activity  there is no reporting agent configured  skipping counts reporting        CodeBlockConfig    Check your product logs roughly 24 hours after opting out to make sure that the system isn t trying to send reports   If your configuration file and environment variable differ  the environment variable setting will take precedence      Example payloads  HashiCorp collects the following utilization data as JSON payloads      payload version    The version of this payload schema    license id    The license ID for this product    product    The product that this contribution is for    product version    The product version this contribution is for    export timestamp   The date and time for this contribution    snapshots    An array of snapshot details  A snapshot is a structure that   represents a single data collection       snapshot version    The version of the snapshot package that produced this      snapshot       snapshot id    A unique identifier for this particular snapshot       process id    An identifier for the system that produced this snapshot       timestamp    The date and time for this snapshot       schema version    The version of the schema associated with this snapshot       service    The service that produced this snapshot  likely to be product      name        metrics    A map of representations of snapshot metrics contained within      this snapshot          key    The key name associated with this metric          kind    The kind of metric  feature  counter  sum  or mean           mode    The mode of operation associated with this metric  write or         collect           labels    The labels associated with each collected metric             entity    The sum of tokens generated for a unique client identifier             nonentity    The sum of tokens without an entity attached    metadata    Optional product specific metadata       billing start    The billing start date associated with the reporting cluster  license start date if not configured           Note title  Important change to supported versions                As of 1 16 7  1 17 3 and later         the  a href   vault docs concepts billing start date  billing start date  a  automatically        rolls over to the latest billing year at the end of the last cycle                 For more information  refer to the upgrade guide for your Vault version          Vault v1 16 x   vault docs upgrading upgrade to 1 16 x auto rolled billing start date          Vault v1 17 x   vault docs upgrading upgrade to 1 17 x auto rolled billing start date         Note              cluster id    The cluster UUID as shown by  vault status  on the reporting      cluster   CodeBlockConfig hideClipboard      json      payload version    1      license id    97afe7b4 b9c8 bf19 bf35 b89b5cc0efea      product    vault      product version    1 14 0 rc1 ent      export timestamp    2023 06 01T11 39 00 76643 04 00      snapshots                  snapshot version   1         snapshot id    0001J7HEWM1PEHPMF5YZT8EV65          process id    01H1VSQMNYAP77R566F1Y03GE6          timestamp    2023 06 01T11 39 00 766099 04 00          schema version    1 0 0          service    vault          metrics              clientcount current month estimate                key    clientcount current month estimate              kind    sum              mode    write              labels                  type                    entity   20                 nonentity   11                                               clientcount previous month complete                key    clientcount previous month complete              kind    sum              mode    write              labels                  type                    entity   10                 nonentity   11                                                           metadata          vault            billing start    2023 03 01T00 00 00Z          cluster id    a8d95acc ec0a 6087 d7f6 4f054ab2e7fd                     CodeBlockConfig      Pre 1 9 counts  When upgrading Vault from 1 8  or earlier  to 1 9  or later   utilization reporting will only include the  non entity tokens   vault docs concepts client count non entity tokens  that are used after the upgrade   Starting in Vault 1 9  the activity log records and de duplicates non entity tokens by using the namespace and token s policies to generate a unique identifier  Because Vault did not create identifiers for these tokens before 1 9  the activity log cannot know whether this token has been seen pre 1 9  To prevent inaccurate and inflated counts  the activity log will ignore any counts of non entity tokens that were created before the upgrade and only the non entity tokens from versions 1 9 and later will be counted   See the client count  overview   vault docs concepts client count  and  FAQ   vault docs concepts client count faq  for more information