vault page title 1 15 0 release notes Vault 1 15 0 release notes layout docs GA date 2023 09 27 Key updates for Vault 1 15 0

Answer:
---
layout: docs
page_title: "1.15.0 release notes"
description: |-
  Key updates for  Vault 1.15.0
---

# Vault 1.15.0 release notes

**GA date:** 2023-09-27

@include 'release-notes/intro.mdx'

## Known issues and breaking changes

| Version          | Issue                                                                                |
|------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1.15.0+          | [Vault no longer reports rollback metrics by mountpoint](/vault/docs/upgrading/upgrade-to-1.15.x#rollback-metrics)                                                                                |
| 1.15.0           | [Panic in AWS auth method during IAM-based login](/vault/docs/upgrading/upgrade-to-1.15.x#panic-in-aws-auth-method-during-iam-based-login)                                                                                |
| 1.15.0+          | [UI Collapsed navbar does not allow certain click events](/vault/docs/upgrading/upgrade-to-1.15.x#ui-collapsed-navbar)                                                                                |
| 1.15             | [Vault file audit devices do not honor SIGHUP signal to reload](/vault/docs/upgrading/upgrade-to-1.15.x#file-audit-devices-do-not-honor-sighup-signal-to-reload)                                                                  |
| 1.15.0 - 1.15.1  | [Vault storing references to ephemeral sub-loggers leading to unbounded memory consumption](/vault/docs/upgrading/upgrade-to-1.15.x#vault-is-storing-references-to-ephemeral-sub-loggers-leading-to-unbounded-memory-consumption) |
| 1.15.0 - 1.15.1  | [Internal error when vault policy in namespace does not exist](/vault/docs/upgrading/upgrade-to-1.15.x#internal-error-when-vault-policy-in-namespace-does-not-exist)                                                              |
| 1.15.0+          | [Sublogger levels not adjusted on reload](/vault/docs/upgrading/upgrade-to-1.15.x#sublogger-levels-unchanged-on-reload)                                                                                |
| 1.15.0+          | [URL change for KV v2 plugin](/vault/docs/upgrading/upgrade-to-1.15.x#kv2-url-change)                                                                                |
| 1.15.1           | [Fatal error during expiration metrics gathering causing Vault crash](/vault/docs/upgrading/upgrade-to-1.15.x#fatal-error-during-expiration-metrics-gathering-causing-vault-crash)                                                |
| 1.15.0 - 1.15.4  | [Audit devices could log raw data despite configuration](/vault/docs/upgrading/upgrade-to-1.15.x#audit-devices-could-log-raw-data-despite-configuration)                                                                          |
| 1.15.5           | [Unable to rotate LDAP credentials](/vault/docs/upgrading/upgrade-to-1.15.x#unable-to-rotate-ldap-credentials)                                                                                |
| 1.15.0 - 1.15.5  | [Deadlock can occur on performance secondary clusters with many mounts](/vault/docs/upgrading/upgrade-to-1.15.x#deadlock-can-occur-on-performance-secondary-clusters-with-many-mounts)                                            |
| 1.15.0 - 1.15.5  | [Audit fails to recover from panics when formatting audit entries](/vault/docs/upgrading/upgrade-to-1.15.x#audit-fails-to-recover-from-panics-when-formatting-audit-entries)                                                      |
| 1.15.0 - 1.15.7  | [Vault Enterprise performance standby nodes audit all request headers regardless of settings](/vault/docs/upgrading/upgrade-to-1.15.x#vault-enterprise-performance-standby-nodes-audit-all-request-headers)                       |
| 1.15.3 - 1.15.9  | [New nodes added by autopilot upgrades provisioned with the wrong version](/vault/docs/upgrading/upgrade-to-1.15.x#new-nodes-added-by-autopilot-upgrades-provisioned-with-the-wrong-version)                                      |
| 1.15.8 - 1.15.9  | [Autopilot upgrade for Vault Enterprise fails](/vault/docs/upgrading/upgrade-to-1.15.x#autopilot)                                                                                |
| 1.15.0 - 1.15.11 | [Listener stops listening on untrusted upstream connection with particular config settings](/vault/docs/upgrading/upgrade-to-1.15.x#listener-proxy-protocol-config)                                                               |
| 0.7.0+           | [Duplicate identity groups created](/vault/docs/upgrading/upgrade-to-1.15.x#duplicate-identity-groups-created-when-concurrent-requests-sent-to-the-primary-and-pr-secondary-cluster)                                              |                                                                                |
| Known Issue (0.7.0+) | [Manual entity merges fail](/vault/docs/upgrading/upgrade-to-1.15.x#manual-entity-merges-sent-to-a-pr-secondary-cluster-are-not-persisted-to-storage)

## Vault companion updates

Companion updates are Vault updates that live outside the main Vault binary.

      Release
      Update
      Description

      Vault Secrets Operator

    GA

      Run the Vault Secrets Operator (v0.3.0) on Red Hat OpenShift.

      Learn more: Vault Secrets Operator

## Core updates

Follow the learn more links for more information, or browse the list of
[Vault tutorials updated to highlight changes for the most recent GA release](/vault/tutorials/new-release).

      Release
      Update
      Description

      Vault Agent

    ENHANCED

      Updated to use the latest Azure SDK version and Workload Identity
      Federation (WIF).

      Learn more:
      What is Vault Agent?

    GA

      Fetch secrets directly into your application as environment variables.

      Learn more: Process Supervisor Mode

      External plugins

    BETA

      Run external plugins in their own container with native container platform
      controls.

      Learn more: Containerize Vault plugins

      Eventing

    BETA

      Subscribe to notifications for various events in Vault. Includes support
      for filtering, permissions, and cluster configurations with K-V secrets.

      Learn more: Events

    Vault GUI

    GA

      New LDAP secrets engine GUI.

      Learn more: Vault UI guide

    ENHANCED

      • New landing page dashboard.

      • View secrets you have read access to under your directory.

      • View diffs between previous and new secret versions.

      • Copy and paste secret paths from the GUI to the Vault CLI or API.

      Learn more: Vault UI guide

      Secrets management

    GA

      Connect to Google Cloud Platform (GCP) Cloud SQL instances using native
      IAM credentials.

      Learn more:
      Google Cloud Platform Secret Manager

    ENHANCED

      Improved TTL management for database credentials with configurable
      credential rotation.

      Learn more: Secrets engines

## Enterprise updates

      Release
      Update
      Description

      Secrets syncing

    BETA

      Sync Key Value (KV) v2 data between Vault and secrets managers from AWS,
      Azure, Google Cloud Platform (GCP), GitHub, and Vercel.

      Learn more: Secrets Sync

      Public Key Infrastructure (PKI)

    GA

      Control Vault PKI issued certificates with the Certificate Issuance
      External Policy Service (CIEPS) to ensure consistency and compliance to
      enterprise standards.

      Learn more: Certificate Issuance External Policy Service (CIEPS)

      Replication

    ENHANCED

      Holistic improvements to cluster replication including problem detection
      and remediation.

      Learn more: Vault Enterprise replication

      Seal High Availability

    BETA

      Enables Vault administrators to configure multiple KMS for seal keys to
      ensure Vault availability in the event a single KMS becomes unavailable.

      Learn more: Seal wrap

      Authentication

    GA

      Authenticate to Vault with your SAML identity provider.

      Learn more: SAML auth method

## Feature deprecations and EOL

Deprecated in 1.15 | Retired in 1.15
------------------ | ---------------
None | None

@include 'release-notes/deprecation-note.mdx'

layout  docs page title   1 15 0 release notes  description       Key updates for  Vault 1 15 0        Vault 1 15 0 release notes    GA date    2023 09 27   include  release notes intro mdx      Known issues and breaking changes    Version            Issue                                                                                1 15 0              Vault no longer reports rollback metrics by mountpoint   vault docs upgrading upgrade to 1 15 x rollback metrics                                                                                1 15 0              Panic in AWS auth method during IAM based login   vault docs upgrading upgrade to 1 15 x panic in aws auth method during iam based login                                                                                1 15 0              UI Collapsed navbar does not allow certain click events   vault docs upgrading upgrade to 1 15 x ui collapsed navbar                                                                                1 15                Vault file audit devices do not honor SIGHUP signal to reload   vault docs upgrading upgrade to 1 15 x file audit devices do not honor sighup signal to reload                                                                       1 15 0   1 15 1     Vault storing references to ephemeral sub loggers leading to unbounded memory consumption   vault docs upgrading upgrade to 1 15 x vault is storing references to ephemeral sub loggers leading to unbounded memory consumption      1 15 0   1 15 1     Internal error when vault policy in namespace does not exist   vault docs upgrading upgrade to 1 15 x internal error when vault policy in namespace does not exist                                                                   1 15 0              Sublogger levels not adjusted on reload   vault docs upgrading upgrade to 1 15 x sublogger levels unchanged on reload                                                                                1 15 0              URL change for KV v2 plugin   vault docs upgrading upgrade to 1 15 x kv2 url change                                                                                1 15 1              Fatal error during expiration metrics gathering causing Vault crash   vault docs upgrading upgrade to 1 15 x fatal error during expiration metrics gathering causing vault crash                                                     1 15 0   1 15 4     Audit devices could log raw data despite configuration   vault docs upgrading upgrade to 1 15 x audit devices could log raw data despite configuration                                                                               1 15 5              Unable to rotate LDAP credentials   vault docs upgrading upgrade to 1 15 x unable to rotate ldap credentials                                                                                1 15 0   1 15 5     Deadlock can occur on performance secondary clusters with many mounts   vault docs upgrading upgrade to 1 15 x deadlock can occur on performance secondary clusters with many mounts                                                 1 15 0   1 15 5     Audit fails to recover from panics when formatting audit entries   vault docs upgrading upgrade to 1 15 x audit fails to recover from panics when formatting audit entries                                                           1 15 0   1 15 7     Vault Enterprise performance standby nodes audit all request headers regardless of settings   vault docs upgrading upgrade to 1 15 x vault enterprise performance standby nodes audit all request headers                            1 15 3   1 15 9     New nodes added by autopilot upgrades provisioned with the wrong version   vault docs upgrading upgrade to 1 15 x new nodes added by autopilot upgrades provisioned with the wrong version                                           1 15 8   1 15 9     Autopilot upgrade for Vault Enterprise fails   vault docs upgrading upgrade to 1 15 x autopilot                                                                                1 15 0   1 15 11    Listener stops listening on untrusted upstream connection with particular config settings   vault docs upgrading upgrade to 1 15 x listener proxy protocol config                                                                    0 7 0               Duplicate identity groups created   vault docs upgrading upgrade to 1 15 x duplicate identity groups created when concurrent requests sent to the primary and pr secondary cluster                                                                                Known Issue  0 7 0      Manual entity merges fail   vault docs upgrading upgrade to 1 15 x manual entity merges sent to a pr secondary cluster are not persisted to storage       Vault companion updates  Companion updates are Vault updates that live outside the main Vault binary    table     thead       tr         th style  Release  th         th style  Update  th         th style  Description  th        tr      thead     tbody      tr       td style         Vault Secrets Operator       td       td style  GA  td       td style         Run the Vault Secrets Operator  v0 3 0  on Red Hat OpenShift         br    br          Learn more   a href   vault docs platform k8s vso openshift  Vault Secrets Operator  a        td      tr      tbody    table      Core updates  Follow the learn more links for more information  or browse the list of  Vault tutorials updated to highlight changes for the most recent GA release   vault tutorials new release     table     thead       tr         th style  Release  th         th style  Update  th         th style  Description  th        tr      thead     tbody      tr       td rowSpan  2  style         Vault Agent       td       td style  ENHANCED  td       td style         Updated to use the latest Azure SDK version and Workload Identity       Federation  WIF          br    br          Learn more  nbsp         a href   vault docs agent and proxy agent  What is Vault Agent   a        td      tr     tr       td style  GA  td       td style         Fetch secrets directly into your application as environment variables         br    br          Learn more   a href   vault docs agent and proxy agent process supervisor  Process Supervisor Mode  a        td      tr      tr       td style         External plugins       td       td style  BETA  td       td style         Run external plugins in their own container with native container platform       controls         br    br          Learn more   a href   vault docs plugins containerized plugins  Containerize Vault plugins  a        td      tr      tr       td style         Eventing       td       td style  BETA  td       td style         Subscribe to notifications for various events in Vault  Includes support       for filtering  permissions  and cluster configurations with K V secrets         br    br          Learn more   a href   vault docs concepts events  Events  a        td      tr      tr       td rowSpan  2  style       Vault GUI       td       td style  GA  td       td style         New LDAP secrets engine GUI         br    br          Learn more   a href   vault docs configuration ui  Vault UI guide  a        td      tr     tr       td style  ENHANCED  td       td style          bull  New landing page dashboard  br           bull  View secrets you have read access to under your directory  br           bull  View diffs between previous and new secret versions  br           bull  Copy and paste secret paths from the GUI to the Vault CLI or API         br    br          Learn more   a href   vault docs configuration ui  Vault UI guide  a        td      tr      tr       td rowSpan  2  style         Secrets management       td       td style  GA  td       td style         Connect to Google Cloud Platform  GCP  Cloud SQL instances using native       IAM credentials         br    br          Learn more  nbsp         a href   vault docs sync gcpsm  Google Cloud Platform Secret Manager  a        td      tr     tr       td style  ENHANCED  td       td style         Improved TTL management for database credentials with configurable       credential rotation         br    br          Learn more   a href   vault api docs secret  Secrets engines  a        td      tr       tbody    table      Enterprise updates   table     thead       tr         th style  Release  th         th style  Update  th         th style  Description  th        tr      thead     tbody      tr       td style         Secrets syncing       td       td style  BETA  td       td style         Sync Key Value  KV  v2 data between Vault and secrets managers from AWS        Azure  Google Cloud Platform  GCP   GitHub  and Vercel         br    br          Learn more   a href   vault docs sync  Secrets Sync  a        td      tr      tr       td style         Public Key Infrastructure  PKI        td       td style  GA  td       td style         Control Vault PKI issued certificates with the Certificate Issuance       External Policy Service  CIEPS  to ensure consistency and compliance to       enterprise standards         br    br          Learn more   a href   vault docs secrets pki cieps  Certificate Issuance External Policy Service  CIEPS   a        td      tr      tr       td style         Replication       td       td style  ENHANCED  td       td style         Holistic improvements to cluster replication including problem detection       and remediation         br    br          Learn more   a href   vault docs enterprise replication  Vault Enterprise replication  a        td      tr      tr       td style         Seal High Availability       td       td style  BETA  td       td style         Enables Vault administrators to configure multiple KMS for seal keys to       ensure Vault availability in the event a single KMS becomes unavailable         br    br          Learn more   a href   vault docs configuration seal seal ha  Seal wrap  a        td      tr      tr       td style         Authentication       td       td style  GA  td       td style         Authenticate to Vault with your SAML identity provider         br    br          Learn more   a href   vault docs auth saml  SAML auth method  a        td      tr       tbody    table      Feature deprecations and EOL  Deprecated in 1 15   Retired in 1 15                                      None   None   include  release notes deprecation note mdx


vault page title 1 17 0 release notes Key updates for Vault 1 17 0 layout docs Vault 1 17 0 release notes GA date 2024 06 12

Answer:
---
layout: docs
page_title: "1.17.0 release notes"
description: |-
  Key updates for  Vault 1.17.0
---

# Vault 1.17.0 release notes

**GA date:** 2024-06-12

@include 'release-notes/intro.mdx'

## Important changes

| Change                                         | Description                                                                                |
|------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| New default (1.17)                             | [Allowed audit headers now have unremovable defaults](/vault/docs/upgrading/upgrade-to-1.17.x#audit-headers)                                                                                |
| Opt out feature (1.17)                         | [PKI sign-intermediate now truncates `notAfter` field to signing issuer](/vault/docs/upgrading/upgrade-to-1.17.x#pki-truncate)                                                                   |
| Beta feature deprecated (1.17)                 | [Request limiter deprecated](/vault/docs/upgrading/upgrade-to-1.17.x#request-limiter)                                                                                |
| Known issue (1.17.0+)                          | [PKI OCSP GET requests can return HTTP redirect responses](/vault/docs/upgrading/upgrade-to-1.17.x#pki-ocsp)                                                                                |
| Known issue (1.17.0)                           | [Vault Agent and Vault Proxy consume excessive amounts of CPU](/vault/docs/upgrading/upgrade-to-1.17.x#agent-proxy-cpu-1-17)                                                                     |
| Known issue (1.15.8 - 1.15.9, 1.16.0 - 1.16.3) | [Autopilot upgrade for Vault Enterprise fails](/vault/docs/upgrading/upgrade-to-1.16.x#new-nodes-added-by-autopilot-upgrades-provisioned-with-the-wrong-version)                                 |
| Known issue (1.17.0 - 1.17.2)                  | [Vault standby nodes not deleting removed entity-aliases from in-memory database](/vault/docs/upgrading/upgrade-to-1.17.x#dangling-entity-alias-in-memory)                                       |
| Known issue (1.17.0 - 1.17.3)                  | [AWS Auth AssumeRole requires an external ID even if none is set](/vault/docs/upgrading/upgrade-to-1.17.x#aws-auth-role-configuration-requires-an-external_id)                                   |
| Known Issue (0.7.0+)                           | [Duplicate identity groups created](/vault/docs/upgrading/upgrade-to-1.17.x#duplicate-identity-groups-created-when-concurrent-requests-sent-to-the-primary-and-pr-secondary-cluster)             |
| Known Issue (0.7.0+)                           | [Manual entity merges fail](/vault/docs/upgrading/upgrade-to-1.17.x#manual-entity-merges-sent-to-a-pr-secondary-cluster-are-not-persisted-to-storage)                                            |
| Known Issue (1.17.3-1.17.4)                    | [Some values in the audit logs not hmac'd properly](/vault/docs/upgrading/upgrade-to-1.17.x#client-tokens-and-token-accessors-audited-in-plaintext)                                              |
| Known Issue (1.17.0-1.17.5)                    | [Cached activation flags for secrets sync on follower nodes are not updated](/vault/docs/upgrading/upgrade-to-1.17.x#cached-activation-flags-for-secrets-sync-on-follower-nodes-are-not-updated) |
| New default (1.17.9)                           | [Vault product usage metrics reporting](/vault/docs/upgrading/upgrade-to-1.17.x#product-usage-reporting)                                                                                |
| Deprecation (1.17.9)                           | [`default_report_months` is deprecated for the `sys/internal/counters` API](/vault/docs/upgrading/upgrade-to-1.17.x#activity-log-changes)                                                        |

## Vault companion updates

Companion updates are Vault updates that live outside the main Vault binary.

**None**.

## Core updates

Follow the learn more links for more information, or browse the list of
[Vault tutorials updated to highlight changes for the most recent GA release](/vault/tutorials/new-release).

      Release
      Update
      Description

      Security patches

    ENHANCED

      Various security improvements to remediate varying severity and
      informational findings from a 3rd party security audit.

      Vault Agent and Vault Proxy self-healing tokens

    ENHANCED

      Auto-authentication avoids agent/proxy restarts and config changes by
      automatically re-authenticating authN tokens to Vault.

      Learn more: Vault Agent and Vault Proxy auto-auth

## Enterprise updates

      Release
      Update
      Description

      Adaptive overload protection

    BETA

      Prevent client requests from overwhelming a variety of server resources
      that could lead to poor server availability.

      Learn more: Adaptive overload protection overview

      ACME Client Count

    ENHANCED

      To improve clarity around client counts, Vault now separates ACME clients
      from non-entity clients.

      Public Key Infrastructure (PKI)

    GA

      Automate certificate lifecycle management for IoT/EST enabled devices with
      native EST protocol support.

      Learn more: Enrollment over Secure Transport (EST) overview

    GA

      Submit custom metadata with certificate requests and store the additional
      information in Vault for further analysis.

      Learn more: PKI secrets engine API

      Resource management

    ENHANCED

      Vault now supports a greater number of namespaces and mounts for
      large-scale Vault installations.

    GA

      Use hierarchical mount paths to organize, manage, and control access to
      secret engine objects.

    GA

      Safely override the max entry size to set different limits for specific
      storage entries that contain mount tables, auth tables and namespace
      configuration data.

      Learn more: max_mount_and_namespace_table_entry_size parameter

      Transit

    GA

      Use cipher-based message authentication code (CMAC) with AES symmetric
      keys in the Vault Transit plugin.

      Learn more: CMAC support

      Plugin identity tokens

    GA

      Enable AWS, Azure, and GCP authentication flows with workload identity
      federation (WIF) tokens from the associated secrets plugins without
      explicitly configuring sensitive security credentials.

      Learn more: Plugin WIF overview

      LDAP Secrets Engine

    GA

      Use hierarchical paths with roles and set names to define policies that
      map 1-1 to LDAP secrets engine roles.

      Learn more: Hierarchical paths overview

      Clock skew and lag detection

    GA

      Use the sys/health and sys/ha-status endpoints
      to display lags in performance secondaries and performance standby nodes.

      Learn more: Clock skew and replication lag overview

## Feature deprecations and EOL

Deprecated in 1.17 | Retired in 1.17
------------------ | ---------------
None               | Centrify Auth plugin

@include 'release-notes/deprecation-note.mdx'

layout  docs page title   1 17 0 release notes  description       Key updates for  Vault 1 17 0        Vault 1 17 0 release notes    GA date    2024 06 12   include  release notes intro mdx      Important changes    Change                                           Description                                                                                New default  1 17                                 Allowed audit headers now have unremovable defaults   vault docs upgrading upgrade to 1 17 x audit headers                                                                                Opt out feature  1 17                             PKI sign intermediate now truncates  notAfter  field to signing issuer   vault docs upgrading upgrade to 1 17 x pki truncate                                                                        Beta feature deprecated  1 17                     Request limiter deprecated   vault docs upgrading upgrade to 1 17 x request limiter                                                                                Known issue  1 17 0                               PKI OCSP GET requests can return HTTP redirect responses   vault docs upgrading upgrade to 1 17 x pki ocsp                                                                                Known issue  1 17 0                               Vault Agent and Vault Proxy consume excessive amounts of CPU   vault docs upgrading upgrade to 1 17 x agent proxy cpu 1 17                                                                          Known issue  1 15 8   1 15 9  1 16 0   1 16 3     Autopilot upgrade for Vault Enterprise fails   vault docs upgrading upgrade to 1 16 x new nodes added by autopilot upgrades provisioned with the wrong version                                      Known issue  1 17 0   1 17 2                      Vault standby nodes not deleting removed entity aliases from in memory database   vault docs upgrading upgrade to 1 17 x dangling entity alias in memory                                            Known issue  1 17 0   1 17 3                      AWS Auth AssumeRole requires an external ID even if none is set   vault docs upgrading upgrade to 1 17 x aws auth role configuration requires an external id                                        Known Issue  0 7 0                                Duplicate identity groups created   vault docs upgrading upgrade to 1 17 x duplicate identity groups created when concurrent requests sent to the primary and pr secondary cluster                  Known Issue  0 7 0                                Manual entity merges fail   vault docs upgrading upgrade to 1 17 x manual entity merges sent to a pr secondary cluster are not persisted to storage                                                 Known Issue  1 17 3 1 17 4                        Some values in the audit logs not hmac d properly   vault docs upgrading upgrade to 1 17 x client tokens and token accessors audited in plaintext                                                   Known Issue  1 17 0 1 17 5                        Cached activation flags for secrets sync on follower nodes are not updated   vault docs upgrading upgrade to 1 17 x cached activation flags for secrets sync on follower nodes are not updated      New default  1 17 9                               Vault product usage metrics reporting   vault docs upgrading upgrade to 1 17 x product usage reporting                                                                                Deprecation  1 17 9                                default report months  is deprecated for the  sys internal counters  API   vault docs upgrading upgrade to 1 17 x activity log changes                                                               Vault companion updates  Companion updates are Vault updates that live outside the main Vault binary     None        Core updates  Follow the learn more links for more information  or browse the list of  Vault tutorials updated to highlight changes for the most recent GA release   vault tutorials new release     table     thead       tr         th style  Release  th         th style  Update  th         th style  Description  th        tr      thead     tbody      tr       td style         Security patches       td       td style  ENHANCED  td       td style         Various security improvements to remediate varying severity and       informational findings from a 3rd party security audit        td      tr      tr       td style         Vault Agent and Vault Proxy self healing tokens       td       td style  ENHANCED  td       td style         Auto authentication avoids agent proxy restarts and config changes by       automatically re authenticating authN tokens to Vault         br    br          Learn more   a href   vault docs agent and proxy autoauth  Vault Agent and Vault Proxy auto auth  a        td      tr       tbody    table      Enterprise updates   table     thead       tr         th style  Release  th         th style  Update  th         th style  Description  th        tr      thead     tbody      tr       td style         Adaptive overload protection       td       td style  BETA  td       td style         Prevent client requests from overwhelming a variety of server resources       that could lead to poor server availability         br    br          Learn more   a href   vault docs concepts adaptive overload protection  Adaptive overload protection overview  a        td      tr      tr       td style         ACME Client Count       td       td style  ENHANCED  td       td style         To improve clarity around client counts  Vault now separates ACME clients       from non entity clients        td      tr      tr       td rowSpan  2  style         Public Key Infrastructure  PKI        td       td style  GA  td       td style         Automate certificate lifecycle management for IoT EST enabled devices with       native EST protocol support         br    br          Learn more   a href   vault docs secrets pki est  Enrollment over Secure Transport  EST   a  overview       td      tr       tr       td style  GA  td       td style         Submit custom metadata with certificate requests and store the additional       information in Vault for further analysis         br    br          Learn more   a href   vault api docs secret pki metadata  PKI secrets engine API  a        td      tr     tr       td rowSpan  3  style         Resource management       td       td style  ENHANCED  td       td style         Vault now supports a greater number of namespaces and mounts for       large scale Vault installations        td      tr      tr       td style  GA  td       td style         Use hierarchical mount paths to organize  manage  and control access to       secret engine objects        td      tr      tr       td style  GA  td       td style         Safely override the max entry size to set different limits for specific       storage entries that contain mount tables  auth tables and namespace       configuration data         br    br          Learn more   a href   vault docs configuration storage raft max mount and namespace table entry size   code max mount and namespace table entry size  code  parameter  a        td      tr      tr       td style         Transit       td       td style  GA  td       td style         Use cipher based message authentication code  CMAC  with AES symmetric       keys in the Vault Transit plugin         br    br          Learn more   a href   docs secrets transit aes256 cmac  CMAC support  a        td      tr      tr       td style         Plugin identity tokens       td       td style  GA  td       td style         Enable AWS  Azure  and GCP authentication flows with workload identity       federation  WIF  tokens from the associated secrets plugins without       explicitly configuring sensitive security credentials         br    br          Learn more   a href   vault docs secrets aws plugin workload identity federation wif  Plugin WIF overview  a        td      tr      tr       td style         LDAP Secrets Engine       td       td style  GA  td       td style         Use hierarchical paths with roles and set names to define policies that       map 1 1 to LDAP secrets engine roles         br    br          Learn more   a href   vault docs secrets ldap hierarchical paths  Hierarchical paths  a  overview       td      tr      tr       td style         Clock skew and lag detection       td       td style  GA  td       td style         Use the  code sys health  code  and  code sys ha status  code  endpoints       to display lags in performance secondaries and performance standby nodes         br    br          Learn more   a href   vault docs enterprise consistency clock skew and replication lag  Clock skew and replication lag  a  overview       td      tr       tbody    table      Feature deprecations and EOL  Deprecated in 1 17   Retired in 1 17                                      None                 Centrify Auth plugin   include  release notes deprecation note mdx


vault GA date 2024 10 09 page title 1 18 0 release notes Key updates for Vault 1 18 0 layout docs Vault 1 18 0 release notes

Answer:
---
layout: docs
page_title: "1.18.0 release notes"
description: |-
  Key updates for  Vault 1.18.0
---

# Vault 1.18.0 release notes

**GA date:** 2024-10-09

@include 'release-notes/intro.mdx'

## Important changes

| Change                      | Description                                                                                |
|-----------------------------|----------------------------------------------------------------------------------------------------------------------|
| New default (1.18.0)        | [Default activity log querying period](/vault/docs/upgrading/upgrade-to-1.18.x#default-activity-log-querying-period) |
| New default (1.18.0)        | [Docker image no longer contains curl](/vault/docs/upgrading/upgrade-to-1.18.x#docker-image-no-longer-contains-curl) |
| Beta feature removed (1.18) | [Request limiter removed](/vault/docs/upgrading/upgrade-to-1.18.x#request-limiter-configuration-removal)             |
| New default (1.18.2)        | [Vault product usage metrics reporting](/vault/docs/upgrading/upgrade-to-1.18.x#product-usage-reporting)             |

## Vault companion updates

Companion updates are Vault updates that live outside the main Vault binary.

**None**.

## Community updates

Follow the learn more links for more information, or browse the list of
[Vault tutorials updated to highlight changes for the most recent GA release](/vault/tutorials/new-release).

**None**.

## Enterprise updates

      Release
      Update
      Description

      Adaptive overload protection

    GA

      Prevent client requests from overwhelming a variety of server resources
      that could lead to poor server availability.

      Learn more: Adaptive overload protection overview

      Autopilot

    ENHANCED

      Overall stability improvements.

      Learn more: Autopilot overview

      Client count

    ENHANCED

      Improved clarity around metering and billing attribution.

      Learn more: Client count calculations

      PKI CMPv2

    GA

      Enable PKI support for automated certificate enrollment with CMPv2
      protocols for 5G networks per 3G PP standards.

      Learn more: CMPv2 in the Vault PKI plugin

      Vault UI

    GA

      Use the Vault UI to configure AWS WIF plugins.

      Learn more: AWS WIF

      PostgreSQL plugin

    GA

      Use rootless rotation for PostgreSQL static roles so individual database
      accounts can rotate their own passwords.

      Learn more: PostgreSQL plugin overview

      KV Patch and Subkey support in Vault’s GUI

    GA

      Configure GUI access to key names in the KV plugin for users without
      granting read access to the values.

      Vault Enterprise with HSM for ARM architecture

    GA

      Run Vault Enterprise on ARM machines with Hardware Security Modules.

      Vault releases: releases.hashicorp.com/vault

## Feature deprecations and EOL

Deprecated in 1.18.x | Retired in 1.18.x
-------------------- | ---------------
None                 | None

@include 'release-notes/deprecation-note.mdx'

layout  docs page title   1 18 0 release notes  description       Key updates for  Vault 1 18 0        Vault 1 18 0 release notes    GA date    2024 10 09   include  release notes intro mdx      Important changes    Change                        Description                                                                                New default  1 18 0            Default activity log querying period   vault docs upgrading upgrade to 1 18 x default activity log querying period      New default  1 18 0            Docker image no longer contains curl   vault docs upgrading upgrade to 1 18 x docker image no longer contains curl      Beta feature removed  1 18     Request limiter removed   vault docs upgrading upgrade to 1 18 x request limiter configuration removal                  New default  1 18 2            Vault product usage metrics reporting   vault docs upgrading upgrade to 1 18 x product usage reporting                    Vault companion updates  Companion updates are Vault updates that live outside the main Vault binary     None        Community updates  Follow the learn more links for more information  or browse the list of  Vault tutorials updated to highlight changes for the most recent GA release   vault tutorials new release      None        Enterprise updates   table     thead       tr         th style  Release  th         th style  Update  th         th style  Description  th        tr      thead     tbody      tr       td style         Adaptive overload protection       td       td style  GA  td       td style         Prevent client requests from overwhelming a variety of server resources       that could lead to poor server availability         br    br          Learn more   a href   vault docs concepts adaptive overload protection  Adaptive overload protection overview  a        td      tr      tr       td style         Autopilot       td       td style  ENHANCED  td       td style         Overall stability improvements         br    br          Learn more   a href   vault docs concepts integrated storage autopilot  Autopilot overview  a        td      tr      tr       td style         Client count       td       td style  ENHANCED  td       td style         Improved clarity around metering and billing attribution         br    br          Learn more   a href   vault docs concepts client count counting  Client count calculations  a        td      tr      tr       td style         PKI CMPv2       td       td style  GA  td       td style         Enable PKI support for automated certificate enrollment with CMPv2       protocols for 5G networks per 3G PP standards         br    br          Learn more   a href   vault docs secrets pki cmpv2  CMPv2 in the Vault PKI plugin  a        td      tr       tr       td style         Vault UI       td       td style  GA  td       td style         Use the Vault UI to configure AWS WIF plugins         br    br          Learn more   a href   vault docs secrets aws plugin workload identity federation wif  AWS WIF  a        td      tr      tr       td style         PostgreSQL plugin       td       td style  GA  td       td style         Use rootless rotation for PostgreSQL static roles so individual database       accounts can rotate their own passwords         br    br          Learn more   a href   vault docs secrets databases postgresql  PostgreSQL plugin overview  a        td      tr      tr       td style         KV Patch and Subkey support in Vault s GUI       td       td style  GA  td       td style         Configure GUI access to key names in the KV plugin for users without       granting read access to the values        td      tr      tr       td style         Vault Enterprise with HSM for ARM architecture       td       td style  GA  td       td style         Run Vault Enterprise on ARM machines with Hardware Security Modules         br    br          Vault releases   a href  https   releases hashicorp com vault   releases hashicorp com vault  a        td      tr       tbody    table      Feature deprecations and EOL  Deprecated in 1 18 x   Retired in 1 18 x                                        None                   None   include  release notes deprecation note mdx


vault Key rotation Vault stores different encryption keys for different purposes Vault uses key layout docs page title Key Rotation rotation to periodically change the keys according to a configured limit or in Learn about the details of key rotation within Vault

Answer:
---
layout: docs
page_title: Key Rotation
description: Learn about the details of key rotation within Vault.
---

# Key rotation

Vault stores different encryption keys for different purposes. Vault uses key
rotation to periodically change the keys according to a configured limit or in
response to a potential leak or compromised service.

## Relevant key definitions

There are four keys involved in key rotation:

- **internal encryption key** - Encrypts and protects data written to the
  storage backend.
- **root key** - "Master" key that seals Vault and protects the internal
  encryption key.
- **unseal key** - A portion (share) of the root key used to reconstruct the
  root key. By default, Vault uses the
  [Shamir's secret sharing algorithm](https://en.wikipedia.org/wiki/Shamir's_Secret_Sharing)
  to split the root key into 5 shares.
- **upgrade key** - A short-lived copy of the internal encryption key created
  during key rotation in high-availability deployments. Vault encrypts upgrade
  keys using the previous internal encryption key.

## How key rotation works

Vault supports online **rekey** and **rotate** operations to update the root
key, unseal keys, and backend encryption key even for high-availability
deployments. In replicated deployments, the active node performs the operations
and standby nodes use an upgrade key to update their keys without requiring a
manual unseal operation.

1. Rekeying begins with a configured split and threshold for unseal keys:
    1. Vault receives the configured threshold of unseal keys.
    1. Vault generates and splits the new root key.
    1. Vault re-encrypts the internal encryption key with the new root key.
    1. Vault returns the new unseal keys.
1. Rotation begins:
    1. Vault generates a new internal encryption key.
    1. Vault adds the new encryption key to an internal keyring.
    1. Vault creates a temporary **upgrade key** (if needed).

![Key Rotate](/img/vault-key-rotate.png)

Once the rotation completes, Vault can encrypt new writes to the storage backend
using the new key, but still decrypt entries written under the previous key.

<Tip title="Related API endpoints">

  ConfigureKeyRotation - [`POST:/sys/rotate/config`](/vault/api-docs/system/rotate-config)

</Tip>

## NIST rotation guidance

The National Institute of Standards and Technology (NIST) recommends
periodically rotating encryption keys, even without a leak or compromise event.

Due to the nature of AES-256-GCM encryption,
[NIST publication 800-38D](https://csrc.nist.gov/pubs/sp/800/38/d/final)
recommends rotating keys **before** performing ~232 encryptions. By
default, Vault monitors the `vault.barrier.estimated_encryptions` metric and
automatically rotates the backend encryption key before reaching 232
encryption operations.

You can approximate the `vault.barrier.estimated_encryptions` metric with the
following sum:

<CodeBlockConfig hideClipboard>

```text
ESTIMATED_OPS = PUT_EVENTS + CREATE_EVENTS + MERKLE_FLUSH_EVENTS + WAL_INDEX
```

</CodeBlockConfig>

where:

- **`PUT_EVENTS`** is the `vault.barrier.put` telemetry metric.
- **`CREATION_EVENTS`** is the `vault.token.creation` metric where `token_type`
  is `batch`.
- **`MERKLE_FLUSH_EVENTS`** is the `merkle.flushDirty.num_pages` telemetry metric.
- **`WAL_INDEX`** is the current write-ahead-log index.

<Tip>

  Vault periodically persists the number of encryptions to support rotation. The
  save operation has a 1 second timeout to limit performance impact when Vault is
  under heavy load. If you use seal wrap, persisting encryptions involves the seal
  backend, which means that some seals, like HSMs, may routinely take longer than
  1 second to respond. You can override the save timeout by setting the
  `VAULT_ENCRYPTION_COUNT_PERSIST_TIMEOUT` environment variable on your Vault
  server to a larger value, such as "5s".

</Tip>

layout  docs page title  Key Rotation description  Learn about the details of key rotation within Vault         Key rotation  Vault stores different encryption keys for different purposes  Vault uses key rotation to periodically change the keys according to a configured limit or in response to a potential leak or compromised service      Relevant key definitions  There are four keys involved in key rotation       internal encryption key     Encrypts and protects data written to the   storage backend      root key      Master  key that seals Vault and protects the internal   encryption key      unseal key     A portion  share  of the root key used to reconstruct the   root key  By default  Vault uses the    Shamir s secret sharing algorithm  https   en wikipedia org wiki Shamir s Secret Sharing    to split the root key into 5 shares      upgrade key     A short lived copy of the internal encryption key created   during key rotation in high availability deployments  Vault encrypts upgrade   keys using the previous internal encryption key      How key rotation works  Vault supports online   rekey   and   rotate   operations to update the root key  unseal keys  and backend encryption key even for high availability deployments  In replicated deployments  the active node performs the operations and standby nodes use an upgrade key to update their keys without requiring a manual unseal operation   1  Rekeying begins with a configured split and threshold for unseal keys      1  Vault receives the configured threshold of unseal keys      1  Vault generates and splits the new root key      1  Vault re encrypts the internal encryption key with the new root key      1  Vault returns the new unseal keys  1  Rotation begins      1  Vault generates a new internal encryption key      1  Vault adds the new encryption key to an internal keyring      1  Vault creates a temporary   upgrade key    if needed      Key Rotate   img vault key rotate png   Once the rotation completes  Vault can encrypt new writes to the storage backend using the new key  but still decrypt entries written under the previous key    Tip title  Related API endpoints      ConfigureKeyRotation     POST  sys rotate config    vault api docs system rotate config     Tip       NIST rotation guidance  The National Institute of Standards and Technology  NIST  recommends periodically rotating encryption keys  even without a leak or compromise event   Due to the nature of AES 256 GCM encryption   NIST publication 800 38D  https   csrc nist gov pubs sp 800 38 d final  recommends rotating keys   before   performing  2 sup 32  sup  encryptions  By default  Vault monitors the  vault barrier estimated encryptions  metric and automatically rotates the backend encryption key before reaching 2 sup 32  sup  encryption operations   You can approximate the  vault barrier estimated encryptions  metric with the following sum    CodeBlockConfig hideClipboard      text ESTIMATED OPS   PUT EVENTS   CREATE EVENTS   MERKLE FLUSH EVENTS   WAL INDEX        CodeBlockConfig   where        PUT EVENTS    is the  vault barrier put  telemetry metric       CREATION EVENTS    is the  vault token creation  metric where  token type    is  batch        MERKLE FLUSH EVENTS    is the  merkle flushDirty num pages  telemetry metric       WAL INDEX    is the current write ahead log index    Tip     Vault periodically persists the number of encryptions to support rotation  The   save operation has a 1 second timeout to limit performance impact when Vault is   under heavy load  If you use seal wrap  persisting encryptions involves the seal   backend  which means that some seals  like HSMs  may routinely take longer than   1 second to respond  You can override the save timeout by setting the     VAULT ENCRYPTION COUNT PERSIST TIMEOUT  environment variable on your Vault   server to a larger value  such as  5s      Tip


vault Learn about integrated Raft storage in Vault Vault supports several options for durable information storage Each backend Integrated Raft storage page title Raft integrated storage layout docs offers pros cons advantages and trade offs For example some backends

Answer:
---
layout: docs
page_title: Raft integrated storage
description: Learn about integrated Raft storage in Vault.
---

# Integrated Raft storage

Vault supports several options for durable information storage. Each backend
offers pros, cons, advantages, and trade-offs. For example, some backends
support high availability while others provide a more robust backup and
restoration process. Integrated storage is a "built-in" storage option that
supports backup/restore workflows, high availability, and Enterprise replication
features without relying on third-party systems.

## Raft protocol overview

<Highlight>

  [The Secret Lives of Data] has a nice visual explanation of Raft storage.

</Highlight>

Raft storage uses a [consensus protocol] based on [Paxos] and the work in
["Raft: In search of an Understandable Consensus Algorithm"] to provide CAP
[consistency].

Raft performance is bound by disk I/O and network latency, and
comparable to Paxos. With stable leadership, committing a log entry requires a
single round trip to half of the peer set.

Compared to Paxos, Raft is designed to have fewer states and a simpler, more
understandable algorithm that depends on the following elements:

- **Log** - An ordered sequence of entries (replicated log) that tracks cluster
  changes. For example, writing data is a new event, which creates a
  corresponding log entry.

- **Peer set** - The set of all members participating in log replication. All
  server nodes are in the peer set of the local cluster.

- **Leader** - At any given time, the peer set elects a single node to be the
  leader. Leaders ingest new log entries, replicate the log to followers, and
  manage when an entry should be committed. Leaders manage log replication and
  inconsistencies within replicated log entries may indicate an issue with the
  leader.

- **Quorum** - A majority of members from a peer set. For a peer set of size `N`,
  quorum requires at least `ceil( (N + 1) / 2 )` members. For example, quorum in
  a peer set of 5 members requires 3 nodes. If a cluster cannot achieve quorum,
  **the cluster becomes unavailable** and cannot commit new logs.

- **Committed entry** - A log entry that is replicated to a quorum of nodes. Log
  entries are only applied once they are committed.

- **Deterministic finite-state machine ([DFSM])** - A collection of known states
  with predictable transitions between the states. In Raft, the DFSM transitions
  between states whenever new logs are applied. By DFSM rules, multiple
  applications of the same sequence of logs must always result in the same final
  state.

### Node states

Raft nodes are always in one of following states:

- **follower** - All nodes start as a follower. Followers accept log entries
  from a leader and cast votes for leader selection.
- **candidate** - A node self-promotes to the candidate state whenever it goes
  without receiving log entries for a given period of time. During
  self-promotion, candidates request votes from the rest of their peer set.
- **leader** - Nodes become leaders once they receive a quorum of votes as a
  candidate.

### Writing logs

With Raft, a log entry is an opaque binary blob. Once the peer set elects a
leader, the peer set can accept new log entries. When clients ask the set to
append a new log entry, the leader writes the entry to durable storage and tries
to replicate the data to a quorum of followers. Once the log entry is
**committed**, the leader **applies** the log entry to a deterministic finite
state machine to maintain the cluster state.

<Note title="Raft in Vault">

  Vault uses [BoltDB](https://github.com/etcd-io/bbolt) or WAL Raft as the
  deterministic finite state machine and blocks writes until they are both
  committed **and** applied.

</Note>

### Compacting logs

To avoid unbounded growth in the replicated logs, Raft saves the current state
to snapshots then compacts the associated logs. Because the finite-state machine
is deterministic, restoring a snapshot of the DFSM always results in the same
state as replaying the sequence of logs associated with the snapshot. Taking
snapshots lets Raft capture the DFSM state at any point in time and then remove
the logs used to reach that state, thereby compacting the log data.

<Note title="Raft in Vault">

  Vault compacts logs automatically to prevent unbounded disk usage while also
  minimizing the time spent replaying logs. Using BoltDB as the DFSM also keeps
  the Vault snapshots lightweight because the Vault data is already persisted to
  disk in BoltDB, the snapshot process just needs to truncate the Raft logs.

</Note>

### Quorum

Raft consensus is fault-tolerant when a peer set has quorum. However, when a
quorum of nodes is **not** available, the peer set cannot process log entries,
elect leaders, or manage peer membership.

For example, suppose there are only 2 peers: A and B. To have quorum, both nodes
must participate, so the quorum size is 2. As a result, both nodes must agree
before they can commit a log entry. If one of the nodes fails, the remaining
node cannot reach quorum; the peer set can no longer add or remove nodes or
commit additional log entries. When the peer set can no longer take action, it
becomes **unavailable**. Once a peer set becomes unavailable, it can only be
recovered manually by removing the failing node and restarting the remaining
node in bootstrap mode so it self-elects as leader.

## Raft leadership in Vault

When a single Vault server (node)
[initializes](/vault/docs/commands/operator/init/#operator-init), it establishes
a cluster (peer set) of size 1 and self-elects itself as leader. Once the
cluster has a leader, additional servers can join the cluster using an
encrypted challenge/answer workflow. For the join process to work, all nodes
in a single Raft cluster must share the same seal configuration. If the cluster
is configured to use auto-unseal, the join process automatically decrypts the
challenge and responds with the answer using the configured seal. For other seal
options, like a Shamir seal, nodes must have access to the unseal keys before
joining so they can decrypt the challenge and respond with the decrypted answer.

In a [high availability](/vault/docs/internals/high-availability#design-overview)
configuration, the active Vault node is the leader node and all standby nodes
are followers.

## Leadership elections

Nodes become the Raft leader through Raft leadership elections.

All nodes in a Raft cluster start as **followers**. Followers monitor leader
health through a **leader heartbeat**. If a follower does not receive a heartbeat
within the  configured **heartbeat timeout**, the node becomes a **candidate**.
Candidates watch for election notices from other nodes in the cluster. If the
**election timeout** period expires, the candidate starts an election for
leader. If the candidate gets responses from a quorum of other nodes in the
cluster, the candidate becomes the new leader node.

Raft leaders may step down voluntarily if the node cannot connect to a quorum
of nodes with the **leader lease timeout** period.

The relevant timeout periods (heartbeat timeout, election timeout, leader lease
timeout) scale according to the [`performance_multiplier`](/vault/docs/configuration/storage/raft#performance-multiplier) setting in your Vault configuration. By default,
the `performance_multiplier` is 5, which translates to the following timeout
values:

Timeout              | Default duration
-------------------- | ----------------
Heartbeat timeout    | 5 seconds
Election timeout     | 5 seconds
Leader lease timeout | 2.5 seconds

We recommend using the default multiplier unless one of the following is true:

- Platform telemetry strongly indicates the default behavior is insufficient.
- The reliability of your platform or network requires different behavior.

## BoltDB Raft logs

BoltDB is a single file database, which means BoltDB cannot shrink the file on
disk to recover space when you delete data. Instead, BoltDB notes the places
where the deleted data was stored on a "freelist". On subsequent writes, BoltDB
consults the freelist to reuse old pages before allocating new space to persist
the data.

<Warning title="BoltDB requires careful tuning">

1. On Vault clusters with high churn, the BoltDB freelist can become quite large
   and the database file can become highly fragmented. Large freelists and
   fragmented database files can slow BoltDB transaction and directly impact the
   performance of your Vault cluster.
1. On busy Vault clusters, where new followers struggle to sync Raft snapshots
   before receiving subsequent snapshots from the leader, the BoltDB file is
   susceptible to sudden bursts of writes. Not only will new followers potentially
   fail to join quorum, Vault installations that do not provide for spiky file
   growth or over-allocate and waste disk space will likely see poor performance.

</Warning>

## Write-ahead Raft logs

@include 'alerts/experimental.mdx'

By default, Vault uses the `raft-boltdb` library for BoltDB to store Raft logs,
but you can also configure Vault to use the
[`raft-wal`](https://github.com/hashicorp/raft-wal) library for write-ahead Raft
logs.

Library       | Filename(s)                                                | Storage directory
------------- |------------------------------------------------------------| ----------------
`raft-boltdb` | `raft.db`                                                  | `raft`
`raft-wal`    | `wal-meta.db`, `XXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXXXXXX.wal` | `raft/wal`

The `raft-wal` library is designed specifically for storing Raft logs. Rather
than using a freelist like `raft-boltdb`, `raft-wal` maintains a directory of
files as its data store and compacts data over time to free up space when a
given file is no longer needed.

Storing data as files in a directory also means that the `raft-wal` library can
easily increase or decrease the number of logs retained by leaders before
truncating and compacting without risking poor performance from spiky writes.

## Quorum management in Vault

### With autopilot

With the [autopilot](/vault/docs/concepts/integrated-storage/autopilot) feature,
Vault uses a configurable set of parameters to confirm a node is healthy before
considering it an eligible voter in the quorum list.

Autopilot is enabled by default and includes stabilization logic for nodes
joining the cluster:

- A node joins the cluster as a non-voter.
- The joined node syncs with the current Raft index.
- Once the configured stability threshold is met, the node becomes a full voting
  member of the cluster.

<Warning title="Verify your stability threshold is appropriate">

  Setting the stability threshold too low can lead to cluster instability because
  nodes may begin voting before they are fully in sync with the Raft index.

</Warning>

Autopilot also includes a dead server cleanup feature. When you enable dead
server cleanup with the
[Autopilot API](/vault/api-docs/system/storage/raftautopilot), Vault
automatically removes unhealthy nodes from the Raft cluster without manual
operator intervention.

### Without autopilot

Without autopilot, when a node joins a Raft cluster, the node tries to catch up
with the peer set just by replicating data received from the leader. While the
node is in the initial synchronization state, it cannot vote, but **is** counted for
the purposes of quorum. If multiple nodes join the cluster simultaneously (or
within a small enough window) the cluster may exceed the expected failure
tolerance, quorum may be lost, and the cluster can fail.

For example, consider a 3-node cluster with a large amount of data and a failure
tolerance of 1. If 3 nodes join the cluster at the same time, the cluster size
becomes 6 with an expected failure tolerance of 2. But 3 of the nodes are still
synchronizing and cannot vote, which means the cluster loses quorum.

If you are not using autopilot, we strongly recommend that you ensure all new
nodes have Raft indexes that are in sync (or very close to in sync) with the
leader before adding additional nodes. You can check the status of current Raft
indexes with the `vault status` CLI command.

## Quorum size and failure tolerance

The table below compares quorum size and failure tolerance for various
cluster sizes.

Servers | Quorum size | Failure tolerance
:-----: | :---------: | :---------------:
1       | 1           | 0
2       | 2           | 0
3       | 2           | 1
4       | 3           | 1
**5**   | **3**       | **2**
6       | 4           | 2
7       | 4           | 3

<Highlight title="Best practice">

  For best performance, we recommended at least 5 servers for a standard
  production deployment to maintained a minimum failure tolerance of 2. We also
  recommend maintaining a cluster with an odd number of nodes to avoid voting
  stalemates.

  We **strongly discourage** single server deployments for production use due to
  the high risk of data loss during failure scenarios.

</Highlight>

To maintain failure tolerance during maintenance and other changes, we recommend
sequentially scaling and reverting your cluster, 2 nodes at a time.

For example, if you start with a 5-node cluster:

1. Scale the cluster to 7 nodes.
1. Confirm the new nodes are joined and in sync with the rest of the peer set.
1. Stop or destroy 2 of the older nodes.
1. Repeat this process 2 more times to cycle out the rest of the pre-existing nodes.

You should always maintain quorum to limit the impact on failure tolerance when
changing or scaling your Vault instance.

### Redundancy Zones

If you are using autopilot with [redundancy zones](/vault/docs/enterprise/redundancy-zones),
the total number of servers will be different from the above, and is dependent
on how many redundancy zones and servers per redundancy zone that you choose.

@include 'autopilot/redundancy-zones.mdx'

<Highlight title="Best practice">

  If you choose to use redundancy zones, we **strongly recommend** using at least 3
  zones to ensure failure tolerance.

</Highlight>

Redundancy zones | Servers per zone | Quorum size | Failure tolerance | Optimistic failure tolerance
:--------------: | :--------------: | :---------: | :---------------: | :--------------------------:
2                | 2                | 2           | 0                 | 2
3                | 2                | 2           | 1                 | 3
3                | 3                | 2           | 1                 | 5
5                | 2                | 3           | 2                 | 6

[consensus protocol]: https://en.wikipedia.org/wiki/Consensus_(computer_science)
[consistency]: https://en.wikipedia.org/wiki/CAP_theorem
["Raft: In search of an Understandable Consensus Algorithm"]: https://raft.github.io/raft.pdf
[paxos]: https://en.wikipedia.org/wiki/Paxos_%28computer_science%29
[The Secret Lives of Data]: http://thesecretlivesofdata.com/raft
[FSM]: https://en.wikipedia.org/wiki/Finite-state_machine

layout  docs page title  Raft integrated storage description  Learn about integrated Raft storage in Vault         Integrated Raft storage  Vault supports several options for durable information storage  Each backend offers pros  cons  advantages  and trade offs  For example  some backends support high availability while others provide a more robust backup and restoration process  Integrated storage is a  built in  storage option that supports backup restore workflows  high availability  and Enterprise replication features without relying on third party systems      Raft protocol overview   Highlight      The Secret Lives of Data  has a nice visual explanation of Raft storage     Highlight   Raft storage uses a  consensus protocol  based on  Paxos  and the work in   Raft  In search of an Understandable Consensus Algorithm   to provide CAP  consistency    Raft performance is bound by disk I O and network latency  and comparable to Paxos  With stable leadership  committing a log entry requires a single round trip to half of the peer set   Compared to Paxos  Raft is designed to have fewer states and a simpler  more understandable algorithm that depends on the following elements       Log     An ordered sequence of entries  replicated log  that tracks cluster   changes  For example  writing data is a new event  which creates a   corresponding log entry       Peer set     The set of all members participating in log replication  All   server nodes are in the peer set of the local cluster       Leader     At any given time  the peer set elects a single node to be the   leader  Leaders ingest new log entries  replicate the log to followers  and   manage when an entry should be committed  Leaders manage log replication and   inconsistencies within replicated log entries may indicate an issue with the   leader       Quorum     A majority of members from a peer set  For a peer set of size  N     quorum requires at least  ceil   N   1    2    members  For example  quorum in   a peer set of 5 members requires 3 nodes  If a cluster cannot achieve quorum      the cluster becomes unavailable   and cannot commit new logs       Committed entry     A log entry that is replicated to a quorum of nodes  Log   entries are only applied once they are committed       Deterministic finite state machine   DFSM       A collection of known states   with predictable transitions between the states  In Raft  the DFSM transitions   between states whenever new logs are applied  By DFSM rules  multiple   applications of the same sequence of logs must always result in the same final   state       Node states  Raft nodes are always in one of following states       follower     All nodes start as a follower  Followers accept log entries   from a leader and cast votes for leader selection      candidate     A node self promotes to the candidate state whenever it goes   without receiving log entries for a given period of time  During   self promotion  candidates request votes from the rest of their peer set      leader     Nodes become leaders once they receive a quorum of votes as a   candidate       Writing logs  With Raft  a log entry is an opaque binary blob  Once the peer set elects a leader  the peer set can accept new log entries  When clients ask the set to append a new log entry  the leader writes the entry to durable storage and tries to replicate the data to a quorum of followers  Once the log entry is   committed    the leader   applies   the log entry to a deterministic finite state machine to maintain the cluster state    Note title  Raft in Vault      Vault uses  BoltDB  https   github com etcd io bbolt  or WAL Raft as the   deterministic finite state machine and blocks writes until they are both   committed   and   applied     Note       Compacting logs  To avoid unbounded growth in the replicated logs  Raft saves the current state to snapshots then compacts the associated logs  Because the finite state machine is deterministic  restoring a snapshot of the DFSM always results in the same state as replaying the sequence of logs associated with the snapshot  Taking snapshots lets Raft capture the DFSM state at any point in time and then remove the logs used to reach that state  thereby compacting the log data    Note title  Raft in Vault      Vault compacts logs automatically to prevent unbounded disk usage while also   minimizing the time spent replaying logs  Using BoltDB as the DFSM also keeps   the Vault snapshots lightweight because the Vault data is already persisted to   disk in BoltDB  the snapshot process just needs to truncate the Raft logs     Note       Quorum  Raft consensus is fault tolerant when a peer set has quorum  However  when a quorum of nodes is   not   available  the peer set cannot process log entries  elect leaders  or manage peer membership   For example  suppose there are only 2 peers  A and B  To have quorum  both nodes must participate  so the quorum size is 2  As a result  both nodes must agree before they can commit a log entry  If one of the nodes fails  the remaining node cannot reach quorum  the peer set can no longer add or remove nodes or commit additional log entries  When the peer set can no longer take action  it becomes   unavailable    Once a peer set becomes unavailable  it can only be recovered manually by removing the failing node and restarting the remaining node in bootstrap mode so it self elects as leader      Raft leadership in Vault  When a single Vault server  node   initializes   vault docs commands operator init  operator init   it establishes a cluster  peer set  of size 1 and self elects itself as leader  Once the cluster has a leader  additional servers can join the cluster using an encrypted challenge answer workflow  For the join process to work  all nodes in a single Raft cluster must share the same seal configuration  If the cluster is configured to use auto unseal  the join process automatically decrypts the challenge and responds with the answer using the configured seal  For other seal options  like a Shamir seal  nodes must have access to the unseal keys before joining so they can decrypt the challenge and respond with the decrypted answer   In a  high availability   vault docs internals high availability design overview  configuration  the active Vault node is the leader node and all standby nodes are followers      Leadership elections  Nodes become the Raft leader through Raft leadership elections   All nodes in a Raft cluster start as   followers    Followers monitor leader health through a   leader heartbeat    If a follower does not receive a heartbeat within the  configured   heartbeat timeout    the node becomes a   candidate    Candidates watch for election notices from other nodes in the cluster  If the   election timeout   period expires  the candidate starts an election for leader  If the candidate gets responses from a quorum of other nodes in the cluster  the candidate becomes the new leader node   Raft leaders may step down voluntarily if the node cannot connect to a quorum of nodes with the   leader lease timeout   period   The relevant timeout periods  heartbeat timeout  election timeout  leader lease timeout  scale according to the   performance multiplier    vault docs configuration storage raft performance multiplier  setting in your Vault configuration  By default  the  performance multiplier  is 5  which translates to the following timeout values   Timeout                Default duration                                         Heartbeat timeout      5 seconds Election timeout       5 seconds Leader lease timeout   2 5 seconds  We recommend using the default multiplier unless one of the following is true     Platform telemetry strongly indicates the default behavior is insufficient    The reliability of your platform or network requires different behavior      BoltDB Raft logs  BoltDB is a single file database  which means BoltDB cannot shrink the file on disk to recover space when you delete data  Instead  BoltDB notes the places where the deleted data was stored on a  freelist   On subsequent writes  BoltDB consults the freelist to reuse old pages before allocating new space to persist the data    Warning title  BoltDB requires careful tuning    1  On Vault clusters with high churn  the BoltDB freelist can become quite large    and the database file can become highly fragmented  Large freelists and    fragmented database files can slow BoltDB transaction and directly impact the    performance of your Vault cluster  1  On busy Vault clusters  where new followers struggle to sync Raft snapshots    before receiving subsequent snapshots from the leader  the BoltDB file is    susceptible to sudden bursts of writes  Not only will new followers potentially    fail to join quorum  Vault installations that do not provide for spiky file    growth or over allocate and waste disk space will likely see poor performance     Warning      Write ahead Raft logs   include  alerts experimental mdx   By default  Vault uses the  raft boltdb  library for BoltDB to store Raft logs  but you can also configure Vault to use the   raft wal   https   github com hashicorp raft wal  library for write ahead Raft logs   Library         Filename s                                                   Storage directory                                                                                raft boltdb     raft db                                                      raft   raft wal        wal meta db    XXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXX wal     raft wal   The  raft wal  library is designed specifically for storing Raft logs  Rather than using a freelist like  raft boltdb    raft wal  maintains a directory of files as its data store and compacts data over time to free up space when a given file is no longer needed   Storing data as files in a directory also means that the  raft wal  library can easily increase or decrease the number of logs retained by leaders before truncating and compacting without risking poor performance from spiky writes      Quorum management in Vault      With autopilot  With the  autopilot   vault docs concepts integrated storage autopilot  feature  Vault uses a configurable set of parameters to confirm a node is healthy before considering it an eligible voter in the quorum list   Autopilot is enabled by default and includes stabilization logic for nodes joining the cluster     A node joins the cluster as a non voter    The joined node syncs with the current Raft index    Once the configured stability threshold is met  the node becomes a full voting   member of the cluster    Warning title  Verify your stability threshold is appropriate      Setting the stability threshold too low can lead to cluster instability because   nodes may begin voting before they are fully in sync with the Raft index     Warning   Autopilot also includes a dead server cleanup feature  When you enable dead server cleanup with the  Autopilot API   vault api docs system storage raftautopilot   Vault automatically removes unhealthy nodes from the Raft cluster without manual operator intervention       Without autopilot  Without autopilot  when a node joins a Raft cluster  the node tries to catch up with the peer set just by replicating data received from the leader  While the node is in the initial synchronization state  it cannot vote  but   is   counted for the purposes of quorum  If multiple nodes join the cluster simultaneously  or within a small enough window  the cluster may exceed the expected failure tolerance  quorum may be lost  and the cluster can fail   For example  consider a 3 node cluster with a large amount of data and a failure tolerance of 1  If 3 nodes join the cluster at the same time  the cluster size becomes 6 with an expected failure tolerance of 2  But 3 of the nodes are still synchronizing and cannot vote  which means the cluster loses quorum   If you are not using autopilot  we strongly recommend that you ensure all new nodes have Raft indexes that are in sync  or very close to in sync  with the leader before adding additional nodes  You can check the status of current Raft indexes with the  vault status  CLI command      Quorum size and failure tolerance  The table below compares quorum size and failure tolerance for various cluster sizes   Servers   Quorum size   Failure tolerance                                           1         1             0 2         2             0 3         2             1 4         3             1   5         3             2   6         4             2 7         4             3   Highlight title  Best practice      For best performance  we recommended at least 5 servers for a standard   production deployment to maintained a minimum failure tolerance of 2  We also   recommend maintaining a cluster with an odd number of nodes to avoid voting   stalemates     We   strongly discourage   single server deployments for production use due to   the high risk of data loss during failure scenarios     Highlight   To maintain failure tolerance during maintenance and other changes  we recommend sequentially scaling and reverting your cluster  2 nodes at a time   For example  if you start with a 5 node cluster   1  Scale the cluster to 7 nodes  1  Confirm the new nodes are joined and in sync with the rest of the peer set  1  Stop or destroy 2 of the older nodes  1  Repeat this process 2 more times to cycle out the rest of the pre existing nodes   You should always maintain quorum to limit the impact on failure tolerance when changing or scaling your Vault instance       Redundancy Zones  If you are using autopilot with  redundancy zones   vault docs enterprise redundancy zones   the total number of servers will be different from the above  and is dependent on how many redundancy zones and servers per redundancy zone that you choose    include  autopilot redundancy zones mdx    Highlight title  Best practice      If you choose to use redundancy zones  we   strongly recommend   using at least 3   zones to ensure failure tolerance     Highlight   Redundancy zones   Servers per zone   Quorum size   Failure tolerance   Optimistic failure tolerance                                                                                2                  2                  2             0                   2 3                  2                  2             1                   3 3                  3                  2             1                   5 5                  2                  3             2                   6   consensus protocol   https   en wikipedia org wiki Consensus  computer science   consistency   https   en wikipedia org wiki CAP theorem   Raft  In search of an Understandable Consensus Algorithm    https   raft github io raft pdf  paxos   https   en wikipedia org wiki Paxos  28computer science 29  The Secret Lives of Data   http   thesecretlivesofdata com raft  FSM   https   en wikipedia org wiki Finite state machine