vault page title Check for Merkle tree corruption Check for Merkle tree corruption include alerts enterprise only mdx Learn how to check your Vault Enterprise cluster data for corruption in the Merkle trees used for replication layout docs

Answer:
---
layout: docs
page_title: Check for Merkle tree corruption
description: >-
  Learn how to check your Vault Enterprise cluster data for corruption in the Merkle trees used for replication.
---

# Check for Merkle tree corruption

@include 'alerts/enterprise-only.mdx'

Vault Enterprise replication uses Merkle trees to keep the cluster state, and rolls cluster state into a Merkle root hash. When data is updated or removed in a cluster, the Merkle tree is also updated. In certain circumstances detailed later in this document, the Merkle tree can become corrupted.

## Types of corruption

Merkle tree corruption can occur at different points in the tree:

- Composite root corruption
- Subtree root corruption
- Page and subpage corruption

## Diagnose Merkle tree corruption

If you run Vault Enterprise versions 1.15.0+, 1.14.3+ or 1.13.7+, you can use the [/sys/replication/merkle-check API](/vault/api-docs/system/replication#sys-replication-merkle-check) endpoint to help determine if your cluster is encountering Merkle tree corruption. In the following sections, you'll learn about some of the details of symptoms and corruption causes which the merkle-check endpoint can detect.

<Note>

Keep in mind that the merkle-check endpoint cannot detect every way in which a Merkle tree could be corrupted.

</Note>

You'll also learn how to query the merkle-check endpoint and interpret its output. Finally, you'll learn about some Vault CLI commands which can help you diagnose corruption.

## Consecutive Merkle difference and synchronization loop

One indication of potential Merkle tree corruption occurs when Vault logs display consecutive Merkle difference and synchronization (merkle-diff and merkle-sync) operations without lasting resolution to streaming write-ahead logs (WALs).

A known cause for this symptom is the occurrence of a split brain situation within a High Availability (HA) Vault cluster. In this case, the leader loses leadership during which it writes data to the storage. Meanwhile, a new leader is elected and reads or writes data that the old leader is mutating. During leadership transfer, an old leader can write data which becomes lost and results in an inconsistent Merkle tree state.

The two most common symptoms which can potentially indicate a split brain issue are detailed in the following sections.

## Merkle difference results in no delta

A merkle-diff operation resulting in no delta indicates conflicting Merkle tree pages. Despite the two clusters holding exactly the same data in both trees, their root hashes do not match.

The following example log shows entries from a performance replication secondary indicating the issue resulting from a corrupted tree, and no writes on the primary since the previous merkle-sync operation.

<CodeBlockConfig hideClipboard>

```plaintext
vault [INFO]  .perf-sec.core0.core: non-matching guard, exiting
vault [TRACE] .perf-sec.core0.core: finished client WAL streaming
vault [INFO]  .perf-sec.core0.replication: no matching WALs available
vault [DEBUG] .perf-sec.core0.replication: starting merkle diff
vault [TRACE] .perf-sec.core0.core: wal context done
vault [TRACE] .perf-sec.core0.core: checking conflicting pages
vault [TRACE] .perf-pri.core0.core: serving conflicting pages
vault [DEBUG] .perf-pri.core0.replication.index.perf: creating merkle state snapshot: generation=4
vault [DEBUG] .perf-pri.core0.replication.index.perf: removing state snapshot from cache: generation=4
vault [INFO]  .perf-sec.core0.replication: requesting WAL stream: guard=8acf94ac
vault [TRACE] .perf-sec.core0.core: starting client WAL streaming
vault [TRACE] .perf-sec.core0.core: receiving WALs
vault [TRACE] .perf-pri.core0.core: starting serving WALs: clientID=e16930a6-7d24-6924-41fe-aa8beb90b1b2
vault [TRACE] .perf-pri.core0.core: streaming from log shipper done: clientID=e16930a6-7d24-6924-41fe-aa8beb90b1b2
vault [TRACE] .perf-pri.core0.core: internal wal stream stop channel fired: clientID=e16930a6-7d24-6924-41fe-aa8beb90b1b2
vault [TRACE] .perf-pri.core0.core: stopping serving WALs: clientID=e16930a6-7d24-6924-41fe-aa8beb90b1b2
vault [INFO]  .perf-sec.core0.core: non-matching guard, exiting
vault [TRACE] .perf-sec.core0.core: finished client WAL streaming
vault [INFO]  .perf-sec.core0.replication: no matching WALs available
vault [TRACE] .perf-sec.core0.core: wal context done
vault [DEBUG] .perf-sec.core0.replication: starting merkle diff
vault [TRACE] .perf-sec.core0.core: checking conflicting pages
vault [TRACE] .perf-pri.core0.core: serving conflicting pages
```

</CodeBlockConfig>

In the example log output, the performance secondary cluster's finite state machine (FSM) is entering merkle-diff mode, in which it tries to fetch conflicting pages from the primary cluster. The diff result is empty, indicated by an immediate switch to the stream-wals mode and **skipping the merkle-sync operation**.

Further into the log, the performance secondary cluster immediately goes into the merkle-diff mode again trying to reconcile the discrepancies of its Merkle tree with the primary cluster. This loop goes on without resolution due to the Merkle tree corruption.

### Non resolving merkle-sync

When a diff operation reveals conflicting data and the sync operation fetches them, but Vault still cannot enter lasting streaming WALs mode afterwards, this indicates a non-matching Merkle roots condition.

In the following server log snippet, merkle-diff returns a non-empty list of page conflicts and merkle-sync fetches those keys. The FSM then transitions to the stream-wals state. Immediately after this transition, the FSM transitions to the merkle-diff again, and returns a **non-matching guard** error.

<CodeBlockConfig hideClipboard>

```plaintext
vault [INFO]  perf-sec.core0.core: non-matching guard, exiting
vault [TRACE] perf-sec.core0.core: finished client WAL streaming
vault [INFO]  perf-sec.core0.replication: no matching WALs available
vault [TRACE] perf-sec.core0.core: wal context done
vault [DEBUG] perf-sec.core0.replication: transitioning state: state=merkle-diff
vault [DEBUG] perf-sec.core0.replication: starting merkle diff
vault [TRACE] perf-sec.core0.core: checking conflicting pages
vault [TRACE] perf-pri.core0.core: serving conflicting pages
vault [DEBUG] perf-pri.core0.replication.index.perf: creating merkle state snapshot: generation=3
vault [TRACE] perf-sec.core0.core: fetching subpage hashes
vault [TRACE] perf-pri.core0.core: serving subpage hashes
vault [DEBUG] perf-pri.core0.replication.index.perf: removing state snapshot from cache: generation=3
vault [DEBUG] perf-sec.core0.replication: transitioning state: state=merkle-sync
vault [DEBUG] perf-sec.core0.replication: waiting for operations to complete before merkle sync
vault [DEBUG] perf-sec.core0.replication: starting merkle sync: num_conflict_keys=4
vault [DEBUG] perf-sec.core0.replication: merkle sync debug info: local_keys=[] remote_keys=[] conflicting_keys=["logical/67bf7b33-734e-f909-86e5-a7e69af0979f/junk9", "logical/67bf7b33-734e-f909-86e5-a7e69af0979f/junk7", "logical/67bf7b33-734e-f909-86e5-a7e69af0979f/junk8", "logical/67bf7b33-734e-f909-86e5-a7e69af0979f/junk6"]
vault [DEBUG] perf-sec.core0.replication: transitioning state: state=stream-wals
vault [INFO]  perf-sec.core0.replication: requesting WAL stream: guard=0c556858
vault [TRACE] perf-sec.core0.core: starting client WAL streaming
vault [TRACE] perf-sec.core0.core: receiving WALs
vault [TRACE] perf-pri.core0.core: starting serving WALs: clientID=6afbce30-67c5-bb15-6eda-001140d33275
vault [TRACE] perf-pri.core0.core: streaming from log shipper done: clientID=6afbce30-67c5-bb15-6eda-001140d33275
vault [TRACE] perf-pri.core0.core: internal wal stream stop channel fired: clientID=6afbce30-67c5-bb15-6eda-001140d33275
vault [TRACE] perf-pri.core0.core: stopping serving WALs: clientID=6afbce30-67c5-bb15-6eda-001140d33275
vault [INFO]  perf-sec.core0.core: non-matching guard, exiting
vault [TRACE] perf-sec.core0.core: finished client WAL streaming
vault [INFO]  perf-sec.core0.replication: no matching WALs available
vault [DEBUG] perf-sec.core0.replication: transitioning state: state=merkle-diff
vault [DEBUG] perf-sec.core0.replication: starting merkle diff
vault [TRACE] perf-sec.core0.core: wal context done
vault [TRACE] perf-sec.core0.core: checking conflicting pages
vault [TRACE] perf-pri.core0.core: serving conflicting pages
```

</CodeBlockConfig>

## Use the merkle-check endpoint

The following examples show how you can use curl to query the merkle-check endpoint.

<Note>

The merkle-check endpoint is authenticated. You need a Vault token with the capabilities detailed in the [endpoint documentation](/vault/api-docs/system/replication#sys-replication-merkle-check) to query the endpoint.

</Note>

### Check the primary cluster

```shell-session
$ curl $VAULT_ADDR/v1/sys/replication/merkle-check
```

Example output:

<CodeBlockConfig hideClipboard>

```json
{
  "request_id": "d4b2ad1a-6e5f-7f9e-edfe-558eb89a40e6",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "merkle_corruption_report": {
      "corrupted_root": false,
      "corrupted_tree_map": {
        "1": {
          "corrupted_index_tuples_map": {
            "5": {
              "corrupted": false,
              "subpages": [
                28
              ]
            }
          },
          "corrupted_subtree_root": false,
          "root_hash": "DyGc6rQTV9XgyNSff3zimhi3FJM=",
          "tree_type": "replicated"
        },
        "2": {
          "corrupted_index_tuples_map": null,
          "corrupted_subtree_root": false,
          "root_hash": "EXmRTdfYCZTm5i9wLef9RQqyLCw=",
          "tree_type": "local"
        }
      },
      "last_corruption_check_epoch": "2023-09-11T11:25:59.44956-07:00"
    }
  }
}

```

</CodeBlockConfig>

The `merkle_corruption_report` stanza provides information about Merkle tree corruption.

When the composite tree or subtree root hashes are corrupted, Vault sets the `corrupted_root` and `corrupted_subtree_root` field values to **true**. Vault sets the field values to true when it detects corruption in the both root hashes of the composite tree, and the subtree.

The `corrupted_tree_map` field identifies any corruption in the subtrees, including replicated and local subtrees. The replicated tree is indexed by number 1 in the map and the local tree is indexed by number `2`. The `tree_type` sub-field also shows which tree contains a corrupted page. The replicated subtree stores the information that is replicated to either a disaster recovery or a performance replication secondary cluster.

It contains replicated items like vault configuration, secrets engine and auth method configuration, and KV secrets. The local subtree stores information relevant to the local cluster such as local mounts, leases, and tokens.

In the event of corruption within a page or a subpage of a tree, the `corrupted_index_tuples_map` includes the page number along with a list of corrupted subpage numbers. If the page hash is corrupted, the `corrupted` field is set to true, otherwise, it sets to false.

<CodeBlockConfig hideClipboard>

```json
{
  "request_id": "d4b2ad1a-6e5f-7f9e-edfe-558eb89a40e6",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": {
    "merkle_corruption_report": {
      "corrupted_root": false,
      "corrupted_tree_map": {
        "1": {
          "corrupted_index_tuples_map": {
            "5": {
              "corrupted": false,
              "subpages": [
                28
              ]
            }
          },
          "corrupted_subtree_root": false,
          "root_hash": "DyGc6rQTV9XgyNSff3zimhi3FJM=",
          "tree_type": "replicated"
        },
        "2": {
          "corrupted_index_tuples_map": null,
          "corrupted_subtree_root": false,
          "root_hash": "EXmRTdfYCZTm5i9wLef9RQqyLCw=",
          "tree_type": "local"
        }
      },
      "last_corruption_check_epoch": "2023-09-11T11:25:59.44956-07:00"
    }
  }
}
```

</CodeBlockConfig>

In the example output, the replicated tree is corrupted on subpage 28 of page 5 only. This means that the page hash, the replicated tree hash, and the composite tree hash are all correct.

### Check a secondary cluster

The Merkle check endpoint prints information about the corruption status of the Merkle tree on a disaster recovery (DR) secondary cluster. You need a [DR operation token](/vault/tutorials/enterprise/disaster-recovery#dr-operation-token-strategy) to access this endpoint. Here is an example `curl` command that demonstrates querying the endpoint.

```shell-session
$ curl $VAULT_ADDR/v1/sys/replication/dr/secondary/merkle-check
```

## CLI commands for diagnosing corruption

You need to check four layers for potential Merkle tree corruption: composite root, subtree roots, page hash, and subpage hash. You can use the following example Vault CLI commands in combination with the [jq](https://jqlang.github.io/jq/) tool to diagnose corruption.

### Composite root corruption

Use a Vault command like the following to check if the composite tree root is corrupted:

```shell-session
$ vault write sys/replication/merkle-check \
    -format=json | jq -r '.data.merkle_corruption_report.corrupted_root'
```

If the response is true, the Merkle tree is corrupted at the composite root.

### Subtree root corruption

Use a Vault command like the following to check if the subtree root is corrupted:

```shell-session
$ vault write sys/replication/merkle-check -format=json \
    | jq -r '.data.merkle_corruption_report.corrupted_tree_map[] | select(.corrupted_subtree_root==true)'
```

If the response is true, the sub-tree is corrupted.

### Page or subpage corruption

Use a Vault command like the following to check if page or subpage is corrupted:

```shell-session
$ vault write sys/replication/merkle-check -format=json \
    | jq -r '.data.merkle_corruption_report.corrupted_tree_map[] | select(.corrupted_index_tuples_map!=null)'
```

If the response is a non-empty map, then at least one page or subpage is corrupted.

The following is an example of a non-empty map in the command output:

<CodeBlockConfig hideClipboard>

```json
{
  "corrupted_index_tuples_map": {
    "23": {
      "corrupted": false,
      "subpages": [
        234
      ]
    }
  },
  "corrupted_subtree_root": false,
  "root_hash": "A5uW54VXDM4jUryDkxN8Vauk8kE=",
  "tree_type": "replicated"
}
```

</CodeBlockConfig>

In this example, page number 23 is returned, however, as the `corrupted` field is false, which means that the page is not corrupted; however, subpage 234 is listed as a corrupted subpage.

To locate a corrupted subpage, Vault needs to also note its parent page, as each page contains 256 subpages and these indexes are repeated for every page. It's possible that only the full page is corrupted without having any corrupted subpages. In that case, the `corrupted` field in the page map is true, and no subpages are listed.

## Caveats and considerations

We generally recommend that you consult the merkle-check endpoint before reindexing to ensure the process will be useful as reindexing can be time-consuming and lead to downtime.

layout  docs page title  Check for Merkle tree corruption description       Learn how to check your Vault Enterprise cluster data for corruption in the Merkle trees used for replication         Check for Merkle tree corruption   include  alerts enterprise only mdx   Vault Enterprise replication uses Merkle trees to keep the cluster state  and rolls cluster state into a Merkle root hash  When data is updated or removed in a cluster  the Merkle tree is also updated  In certain circumstances detailed later in this document  the Merkle tree can become corrupted      Types of corruption  Merkle tree corruption can occur at different points in the tree     Composite root corruption   Subtree root corruption   Page and subpage corruption     Diagnose Merkle tree corruption  If you run Vault Enterprise versions 1 15 0   1 14 3  or 1 13 7   you can use the   sys replication merkle check API   vault api docs system replication sys replication merkle check  endpoint to help determine if your cluster is encountering Merkle tree corruption  In the following sections  you ll learn about some of the details of symptoms and corruption causes which the merkle check endpoint can detect     Note   Keep in mind that the merkle check endpoint cannot detect every way in which a Merkle tree could be corrupted     Note   You ll also learn how to query the merkle check endpoint and interpret its output  Finally  you ll learn about some Vault CLI commands which can help you diagnose corruption      Consecutive Merkle difference and synchronization loop  One indication of potential Merkle tree corruption occurs when Vault logs display consecutive Merkle difference and synchronization  merkle diff and merkle sync  operations without lasting resolution to streaming write ahead logs  WALs     A known cause for this symptom is the occurrence of a split brain situation within a High Availability  HA  Vault cluster  In this case  the leader loses leadership during which it writes data to the storage  Meanwhile  a new leader is elected and reads or writes data that the old leader is mutating  During leadership transfer  an old leader can write data which becomes lost and results in an inconsistent Merkle tree state   The two most common symptoms which can potentially indicate a split brain issue are detailed in the following sections      Merkle difference results in no delta   A merkle diff operation resulting in no delta indicates conflicting Merkle tree pages  Despite the two clusters holding exactly the same data in both trees  their root hashes do not match   The following example log shows entries from a performance replication secondary indicating the issue resulting from a corrupted tree  and no writes on the primary since the previous merkle sync operation    CodeBlockConfig hideClipboard      plaintext vault  INFO    perf sec core0 core  non matching guard  exiting vault  TRACE   perf sec core0 core  finished client WAL streaming vault  INFO    perf sec core0 replication  no matching WALs available vault  DEBUG   perf sec core0 replication  starting merkle diff vault  TRACE   perf sec core0 core  wal context done vault  TRACE   perf sec core0 core  checking conflicting pages vault  TRACE   perf pri core0 core  serving conflicting pages vault  DEBUG   perf pri core0 replication index perf  creating merkle state snapshot  generation 4 vault  DEBUG   perf pri core0 replication index perf  removing state snapshot from cache  generation 4 vault  INFO    perf sec core0 replication  requesting WAL stream  guard 8acf94ac vault  TRACE   perf sec core0 core  starting client WAL streaming vault  TRACE   perf sec core0 core  receiving WALs vault  TRACE   perf pri core0 core  starting serving WALs  clientID e16930a6 7d24 6924 41fe aa8beb90b1b2 vault  TRACE   perf pri core0 core  streaming from log shipper done  clientID e16930a6 7d24 6924 41fe aa8beb90b1b2 vault  TRACE   perf pri core0 core  internal wal stream stop channel fired  clientID e16930a6 7d24 6924 41fe aa8beb90b1b2 vault  TRACE   perf pri core0 core  stopping serving WALs  clientID e16930a6 7d24 6924 41fe aa8beb90b1b2 vault  INFO    perf sec core0 core  non matching guard  exiting vault  TRACE   perf sec core0 core  finished client WAL streaming vault  INFO    perf sec core0 replication  no matching WALs available vault  TRACE   perf sec core0 core  wal context done vault  DEBUG   perf sec core0 replication  starting merkle diff vault  TRACE   perf sec core0 core  checking conflicting pages vault  TRACE   perf pri core0 core  serving conflicting pages        CodeBlockConfig   In the example log output  the performance secondary cluster s finite state machine  FSM  is entering merkle diff mode  in which it tries to fetch conflicting pages from the primary cluster  The diff result is empty  indicated by an immediate switch to the stream wals mode and   skipping the merkle sync operation      Further into the log  the performance secondary cluster immediately goes into the merkle diff mode again trying to reconcile the discrepancies of its Merkle tree with the primary cluster  This loop goes on without resolution due to the Merkle tree corruption       Non resolving merkle sync  When a diff operation reveals conflicting data and the sync operation fetches them  but Vault still cannot enter lasting streaming WALs mode afterwards  this indicates a non matching Merkle roots condition   In the following server log snippet  merkle diff returns a non empty list of page conflicts and merkle sync fetches those keys  The FSM then transitions to the stream wals state  Immediately after this transition  the FSM transitions to the merkle diff again  and returns a   non matching guard   error    CodeBlockConfig hideClipboard      plaintext vault  INFO   perf sec core0 core  non matching guard  exiting vault  TRACE  perf sec core0 core  finished client WAL streaming vault  INFO   perf sec core0 replication  no matching WALs available vault  TRACE  perf sec core0 core  wal context done vault  DEBUG  perf sec core0 replication  transitioning state  state merkle diff vault  DEBUG  perf sec core0 replication  starting merkle diff vault  TRACE  perf sec core0 core  checking conflicting pages vault  TRACE  perf pri core0 core  serving conflicting pages vault  DEBUG  perf pri core0 replication index perf  creating merkle state snapshot  generation 3 vault  TRACE  perf sec core0 core  fetching subpage hashes vault  TRACE  perf pri core0 core  serving subpage hashes vault  DEBUG  perf pri core0 replication index perf  removing state snapshot from cache  generation 3 vault  DEBUG  perf sec core0 replication  transitioning state  state merkle sync vault  DEBUG  perf sec core0 replication  waiting for operations to complete before merkle sync vault  DEBUG  perf sec core0 replication  starting merkle sync  num conflict keys 4 vault  DEBUG  perf sec core0 replication  merkle sync debug info  local keys    remote keys    conflicting keys   logical 67bf7b33 734e f909 86e5 a7e69af0979f junk9    logical 67bf7b33 734e f909 86e5 a7e69af0979f junk7    logical 67bf7b33 734e f909 86e5 a7e69af0979f junk8    logical 67bf7b33 734e f909 86e5 a7e69af0979f junk6   vault  DEBUG  perf sec core0 replication  transitioning state  state stream wals vault  INFO   perf sec core0 replication  requesting WAL stream  guard 0c556858 vault  TRACE  perf sec core0 core  starting client WAL streaming vault  TRACE  perf sec core0 core  receiving WALs vault  TRACE  perf pri core0 core  starting serving WALs  clientID 6afbce30 67c5 bb15 6eda 001140d33275 vault  TRACE  perf pri core0 core  streaming from log shipper done  clientID 6afbce30 67c5 bb15 6eda 001140d33275 vault  TRACE  perf pri core0 core  internal wal stream stop channel fired  clientID 6afbce30 67c5 bb15 6eda 001140d33275 vault  TRACE  perf pri core0 core  stopping serving WALs  clientID 6afbce30 67c5 bb15 6eda 001140d33275 vault  INFO   perf sec core0 core  non matching guard  exiting vault  TRACE  perf sec core0 core  finished client WAL streaming vault  INFO   perf sec core0 replication  no matching WALs available vault  DEBUG  perf sec core0 replication  transitioning state  state merkle diff vault  DEBUG  perf sec core0 replication  starting merkle diff vault  TRACE  perf sec core0 core  wal context done vault  TRACE  perf sec core0 core  checking conflicting pages vault  TRACE  perf pri core0 core  serving conflicting pages        CodeBlockConfig      Use the merkle check endpoint  The following examples show how you can use curl to query the merkle check endpoint    Note   The merkle check endpoint is authenticated  You need a Vault token with the capabilities detailed in the  endpoint documentation   vault api docs system replication sys replication merkle check  to query the endpoint     Note       Check the primary cluster     shell session   curl  VAULT ADDR v1 sys replication merkle check      Example output    CodeBlockConfig hideClipboard      json      request id    d4b2ad1a 6e5f 7f9e edfe 558eb89a40e6      lease id          lease duration   0     renewable   false     data          merkle corruption report            corrupted root   false         corrupted tree map              1                corrupted index tuples map                  5                    corrupted   false                 subpages                     28                                                       corrupted subtree root   false             root hash    DyGc6rQTV9XgyNSff3zimhi3FJM               tree type    replicated                      2                corrupted index tuples map   null             corrupted subtree root   false             root hash    EXmRTdfYCZTm5i9wLef9RQqyLCw               tree type    local                            last corruption check epoch    2023 09 11T11 25 59 44956 07 00                      CodeBlockConfig   The  merkle corruption report  stanza provides information about Merkle tree corruption   When the composite tree or subtree root hashes are corrupted  Vault sets the  corrupted root  and  corrupted subtree root  field values to   true    Vault sets the field values to true when it detects corruption in the both root hashes of the composite tree  and the subtree   The  corrupted tree map  field identifies any corruption in the subtrees  including replicated and local subtrees  The replicated tree is indexed by number 1 in the map and the local tree is indexed by number  2   The  tree type  sub field also shows which tree contains a corrupted page  The replicated subtree stores the information that is replicated to either a disaster recovery or a performance replication secondary cluster    It contains replicated items like vault configuration  secrets engine and auth method configuration  and KV secrets  The local subtree stores information relevant to the local cluster such as local mounts  leases  and tokens    In the event of corruption within a page or a subpage of a tree  the  corrupted index tuples map  includes the page number along with a list of corrupted subpage numbers  If the page hash is corrupted  the  corrupted  field is set to true  otherwise  it sets to false    CodeBlockConfig hideClipboard      json      request id    d4b2ad1a 6e5f 7f9e edfe 558eb89a40e6      lease id          lease duration   0     renewable   false     data          merkle corruption report            corrupted root   false         corrupted tree map              1                corrupted index tuples map                  5                    corrupted   false                 subpages                     28                                                       corrupted subtree root   false             root hash    DyGc6rQTV9XgyNSff3zimhi3FJM               tree type    replicated                      2                corrupted index tuples map   null             corrupted subtree root   false             root hash    EXmRTdfYCZTm5i9wLef9RQqyLCw               tree type    local                            last corruption check epoch    2023 09 11T11 25 59 44956 07 00                     CodeBlockConfig   In the example output  the replicated tree is corrupted on subpage 28 of page 5 only  This means that the page hash  the replicated tree hash  and the composite tree hash are all correct       Check a secondary cluster  The Merkle check endpoint prints information about the corruption status of the Merkle tree on a disaster recovery  DR  secondary cluster  You need a  DR operation token   vault tutorials enterprise disaster recovery dr operation token strategy  to access this endpoint  Here is an example  curl  command that demonstrates querying the endpoint      shell session   curl  VAULT ADDR v1 sys replication dr secondary merkle check         CLI commands for diagnosing corruption  You need to check four layers for potential Merkle tree corruption  composite root  subtree roots  page hash  and subpage hash  You can use the following example Vault CLI commands in combination with the  jq  https   jqlang github io jq   tool to diagnose corruption       Composite root corruption  Use a Vault command like the following to check if the composite tree root is corrupted      shell session   vault write sys replication merkle check        format json   jq  r   data merkle corruption report corrupted root       If the response is true  the Merkle tree is corrupted at the composite root       Subtree root corruption  Use a Vault command like the following to check if the subtree root is corrupted      shell session   vault write sys replication merkle check  format json         jq  r   data merkle corruption report corrupted tree map     select  corrupted subtree root  true        If the response is true  the sub tree is corrupted       Page or subpage corruption  Use a Vault command like the following to check if page or subpage is corrupted      shell session   vault write sys replication merkle check  format json         jq  r   data merkle corruption report corrupted tree map     select  corrupted index tuples map  null        If the response is a non empty map  then at least one page or subpage is corrupted   The following is an example of a non empty map in the command output    CodeBlockConfig hideClipboard      json      corrupted index tuples map          23            corrupted   false         subpages             234                       corrupted subtree root   false     root hash    A5uW54VXDM4jUryDkxN8Vauk8kE       tree type    replicated           CodeBlockConfig   In this example  page number 23 is returned  however  as the  corrupted  field is false  which means that the page is not corrupted  however  subpage 234 is listed as a corrupted subpage    To locate a corrupted subpage  Vault needs to also note its parent page  as each page contains 256 subpages and these indexes are repeated for every page  It s possible that only the full page is corrupted without having any corrupted subpages  In that case  the  corrupted  field in the page map is true  and no subpages are listed      Caveats and considerations  We generally recommend that you consult the merkle check endpoint before reindexing to ensure the process will be useful as reindexing can be time consuming and lead to downtime


vault replicated across clusters to support horizontally scaling and disaster Vault Enterprise has support for Replication allowing critical data to be recovery workloads page title Replication Vault Enterprise Vault Enterprise replication layout docs

Answer:
---
layout: docs
page_title: Replication - Vault Enterprise
description: >-
  Vault Enterprise has support for Replication, allowing critical data to be
  replicated across clusters to support horizontally scaling and disaster
  recovery workloads.
---

# Vault Enterprise replication

@include 'alerts/enterprise-and-hcp.mdx'

## Overview

Many organizations have infrastructure that spans multiple datacenters. Vault
provides the critical services of identity management, secrets storage, and
policy management. This functionality is expected to be highly available and
to scale as the number of clients and their functional needs increase; at the
same time, operators would like to ensure that a common set of policies are
enforced globally, and a consistent set of secrets and keys are exposed to
applications that need to interoperate.

Vault replication addresses both of these needs in providing consistency,
scalability, and highly-available disaster recovery.

<Note title="Storage backend requirement">

Using replication requires a storage backend that supports transactional
updates, such as [Integrated Storage](/vault/docs/concepts/integrated-storage)
or Consul.

</Note>

## Architecture

The core unit of Vault replication is a **cluster**, which is comprised of a
collection of Vault nodes (an active and its corresponding HA nodes). Multiple Vault
clusters communicate in a one-to-many near real-time flow.

Replication operates on a leader/follower model, wherein a leader cluster (known as a
**primary**) is linked to a series of follower **secondary** clusters. The primary
cluster acts as the system of record and asynchronously replicates most Vault data.

All communication between primaries and secondaries is end-to-end encrypted
with mutually-authenticated TLS sessions, setup via replication tokens which are
exchanged during bootstrapping.

## Replicated data

What data is replicated between the primary and secondary depends on the type of
replication that is configured between the primary and secondary. These types
of relationships are either **disaster recovery** or
**performance replication** relationships.

![](/img/replication-overview.png)

The following table shows a capability comparison between Disaster Recovery
and Performance Replication.

| Capability                                                                                | Disaster Recovery | Performance Replication                                                                                |
| -------------------------------------------------------------------------------------------------------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Mirrors the configuration of a primary cluster                                                                       | Yes               | Yes                                                                                |
| Mirrors the configuration of a primary cluster’s backends (i.e., auth methods, secrets engines, audit devices, etc.) | Yes               | Yes                                                                                |
| Mirrors the tokens and leases for applications and users interacting with the primary cluster                        | Yes               | No. Secondaries keep track of their own tokens and leases. When the secondary is promoted, applications must reauthenticate and obtain new leases from the newly-promoted primary. |
| Allows the secondary cluster to handle client requests                                                               | No                | Yes                                            |

Everything written to storage is classified into one of three categories:
* `replicated` (or "shared"): all downstream clusters receive it
* `local`: only downstream disaster recovery clusters receive it
* `ignored`: not replicated to downstream clusters at all

When mounting a secret engine or auth method, you can choose whether to make it a
local mount or a shared mount.

Shared mounts (the default) usually replicate all their data to performance secondaries, but
they can choose to designate specific storage paths as local.  For example, PKI mounts
store certificates locally. If you query the roles on a shared PKI mount, you'll see
the same result for that mount when you send the query to either the performance primary or
secondary, but if you list stored certs, you'll see different values.

Local mounts replicate their data only to disaster recovery secondaries.  A local mount
created on a performance primary isn't visible at all to its performance secondaries.
Local mounts can also be created on performance secondaries, in which case they aren't visible
to the performance primary.

There exist other storage entries that aren't mount specific.  For example, the Integrated
Storage Autopilot configuration is an `ignored` storage entry, which allows for disaster recovery
secondaries to have a different configuration than their primary.  Tokens and leases are written to
`local` storage entries.

## Performance replication

@include 'alerts/enterprise-and-hcp.mdx'

In Performance Replication, secondaries keep track of their own tokens and leases
but share the underlying configuration, policies, and supporting secrets (KV values,
encryption keys for `transit`, etc).

If a user action would modify underlying shared state, the secondary forwards the request
to the primary to be handled; this is transparent to the client. In practice, most
high-volume workloads (reads in the `kv` backend, encryption/decryption operations
in `transit`, etc.) can be satisfied by the local secondary, allowing Vault to scale
relatively horizontally with the number of secondaries rather than vertically as
in the past.

<Note title="Performance replication does not include automated integrated storage snapshots">

Vault does not replicate automated integrated storage snapshots as a part of
performance replication.

You must explicitly configure each of the primary and secondary performance
clusters to create individual automated snapshots replicas.

</Note>

### Paths filter

The primary cluster's mount configuration gets replicated across its secondary
clusters when you enable Performance Replication. In some cases, you may not
want all data to be replicated. For example, your primary cluster is in the EU
region, and you have a secondary cluster outside of the EU region. [General Data
Protection Regulation (GDPR)](https://gdpr.eu/) requires that personally
identifiable data not be physically transferred to locations outside the
European Union unless the region or country has an equal rigor of data
protection regulation as the EU.

To comply with GDPR, leverage Vault's **paths filter**  feature to abide by
data movements and sovereignty regulations while ensuring performance access
across geographically distributed regions.

You can set filters based on the mount path of the secrets engines and
namespaces.

![Performance Replication - primary](/img/vault-mount-filter-7.png)

In the above example, the `EU_GDPR_data/` path and `office_FR` namespace will
not be replicated and remain only on the primary cluster.

On a similar note, if you want to avoid secondary cluster's data to be
replicated, you can mark those secrets engine and/or auth methods **local**.
Local secrets engines and auth methods are not replicated or removed by
replication.

**Example:** When you enable a secrets engine on a secondary cluster, use the
`-local` flag.

```shell-session
$ vault secrets enable -local -path=us_west_data kv-v2
```

<Highlight title="Tutorials">

Refer to the _manage replicated mounts_ section in the [Set up performance
replication](/vault/tutorials/enterprise/performance-replication#manage-replicated-mounts)
tutorial to learn how to specify the mounts to allow or deny data replication.

</Highlight>

## Disaster recovery (DR) replication

In disaster recovery (or DR) replication, secondaries share the same underlying configuration,
policy, and supporting secrets (KV values, encryption keys for `transit`, etc) infrastructure
as the primary. They also share the same token and lease infrastructure as the primary, as
they are designed to allow for continuous operations with applications connecting to the
original primary on the election of the DR secondary.

DR is designed to be a mechanism to protect against catastrophic failure of entire clusters.
They do not forward service read or write requests until they are elected and become a new primary.

-> **Note**: Unlike with Performance Replication, local secret engines, auth methods and audit devices are replicated to a DR secondary.

For more information on the capabilities of performance and disaster recovery replication, see the Vault Replication [API Documentation](/vault/api-docs/system/replication).

## Primary and secondary cluster compatibility

### Storage engines

There is no requirement that both clusters use the same storage engine.

### Seals

There is no requirement that both clusters use the same seal type, but see
[sealwrap](/vault/docs/enterprise/sealwrap#seal-wrap-and-replication) for the full
details.

Also note that enabling replication will modify the secondary seal.
If the secondary uses an auto seal, its recovery configuration and keys
will be replaced; if it uses shamir, its seal configuration and unseal
keys will be replaced. Here seal/recovery configuration means the number of
seal/recovery key fragments and the required threshold of those
fragments.

| Primary Seal | Secondary Seal (before) | Secondary Seal (after)                 | Secondary Recovery Key (after) | Impact on Secondary of Enabling Replication                                 |
|--------------|-------------------------|----------------------------------------|--------------------------------|-----------------------------------------------------------------------------|
| Shamir       | Shamir                  | Primary's shamir config & unseal keys  | N/A                            | Seal config and unseal keys replaced with primary's                         |
| Shamir       | Auto                    | Unchanged                              | Receives primary seal          | Seal recovery config and keys replaced with primary's seal config and keys  |
| Auto         | Auto                    | Unchanged                              | Receives primary recovery      | Seal recovery config and recovery keys replaced with primary's              |
| Auto         | Shamir                  | Receives primary recovery              | N/A                            | Seal config and keys replaced with primary's recovery seal config and keys  |

<Note>

Vault clusters configured with
[auto-unseal](/vault/docs/concepts/seal#auto-unseal) have recovery keys instead
of unseal keys.

</Note>

### Vault versions

Vault changes are designed and tested to ensure that the
[upgrade instructions](/vault/docs/upgrading#replication-installations) are viable, i.e.
that a secondary can run a newer Vault version than its primary.

That said, we do not recommend running replicated Vault clusters with different
versions any longer than necessary to perform the upgrade.

## Internals

Details on the internal design of the replication feature can be found in the
[replication internals](/vault/docs/internals/replication) document.

## Security model

Vault is trusted all over the world to keep secrets safe. As such, we have put
extreme focus to detail to our replication model as well.

### Primary/Secondary communication

When a cluster is marked as the primary it generates a self-signed CA
certificate. On request, and given a user-specified identifier, the primary
uses this CA certificate to generate a private key and certificate and packages
these, along with some other information, into a replication bootstrapping
bundle, a.k.a. a secondary activation token. The certificate is used to perform
TLS mutual authentication between the primary and that secondary.

This CA certificate is never shared with secondaries, and no secondary ever has
access to any other secondary’s certificate. In practice this means that
revoking a secondary’s access to the primary does not allow it continue
replication with any other machine; it also means that if a primary goes down,
there is full administrative control over which cluster becomes primary. An
attacker cannot spoof a secondary into believing that a cluster the attacker
controls is the new primary without also being able to administratively direct
the secondary to connect by giving it a new bootstrap package (which is an
ACL-protected call).

Vault makes use of Application Layer Protocol Negotiation on its cluster port.
This allows the same port to handle both request forwarding and replication,
even while keeping the certificate root of trust and feature set different.

### Secondary activation tokens

A secondary activation token is an extremely sensitive item and as such is
protected via response wrapping. Experienced Vault users will note that the
wrapping format for replication bootstrap packages is different from normal
response wrapping tokens: it is a signed JWT. This allows the replication token
to carry the redirect address of the primary cluster as part of the token. In
most cases this means that simply providing the token to a new secondary is
enough to activate replication, although this can also be overridden when the
token is provided to the secondary.

Secondary activation tokens should be treated like Vault root tokens. If
disclosed to a bad actor, that actor can gain access to all Vault data. It
should therefore be treated with utmost sensitivity. Like all
response-wrapping tokens, once the token is used successfully (in this case, to
activate a secondary) it is useless, so it is only necessary to safeguard it
from one machine to the next. Like with root tokens, HashiCorp recommends that
when a secondary activation token is live, there are multiple eyes on it from
generation until it is used.

Once a secondary is activated, its cluster information is stored safely behind
its encrypted barrier.

## Mutual TLS and load balancers

Vault generates its own certificates for cluster members. All replication traffic
uses the cluster port using these Vault-generated certificates after initial
bootstrapping. Because of this, the cluster traffic can NOT be terminated at the
cluster port at a load balancer level.

## Tutorial

Refer to the following tutorials replication setup and best practices:

- [Set up Performance Replication](/vault/tutorials/enterprise/performance-replication)
- [Disaster Recovery Replication Setup](/vault/tutorials/enterprise/disaster-recovery)
- [Monitoring Vault Replication](/vault/tutorials/monitoring/monitor-replication)

## API

The Vault replication component has a full HTTP API. Refer to the
[Vault Replication API](/vault/api-docs/system/replication) for more
details.

layout  docs page title  Replication   Vault Enterprise description       Vault Enterprise has support for Replication  allowing critical data to be   replicated across clusters to support horizontally scaling and disaster   recovery workloads         Vault Enterprise replication   include  alerts enterprise and hcp mdx      Overview  Many organizations have infrastructure that spans multiple datacenters  Vault provides the critical services of identity management  secrets storage  and policy management  This functionality is expected to be highly available and to scale as the number of clients and their functional needs increase  at the same time  operators would like to ensure that a common set of policies are enforced globally  and a consistent set of secrets and keys are exposed to applications that need to interoperate   Vault replication addresses both of these needs in providing consistency  scalability  and highly available disaster recovery    Note title  Storage backend requirement    Using replication requires a storage backend that supports transactional updates  such as  Integrated Storage   vault docs concepts integrated storage  or Consul     Note      Architecture  The core unit of Vault replication is a   cluster    which is comprised of a collection of Vault nodes  an active and its corresponding HA nodes   Multiple Vault clusters communicate in a one to many near real time flow   Replication operates on a leader follower model  wherein a leader cluster  known as a   primary    is linked to a series of follower   secondary   clusters  The primary cluster acts as the system of record and asynchronously replicates most Vault data   All communication between primaries and secondaries is end to end encrypted with mutually authenticated TLS sessions  setup via replication tokens which are exchanged during bootstrapping      Replicated data  What data is replicated between the primary and secondary depends on the type of replication that is configured between the primary and secondary  These types of relationships are either   disaster recovery   or   performance replication   relationships        img replication overview png   The following table shows a capability comparison between Disaster Recovery and Performance Replication     Capability                                                                                Disaster Recovery   Performance Replication                                                                                Mirrors the configuration of a primary cluster                                                                         Yes                 Yes                                                                                Mirrors the configuration of a primary cluster s backends  i e   auth methods  secrets engines  audit devices  etc     Yes                 Yes                                                                                Mirrors the tokens and leases for applications and users interacting with the primary cluster                          Yes                 No  Secondaries keep track of their own tokens and leases  When the secondary is promoted  applications must reauthenticate and obtain new leases from the newly promoted primary      Allows the secondary cluster to handle client requests                                                                 No                  Yes                                               Everything written to storage is classified into one of three categories     replicated   or  shared    all downstream clusters receive it    local   only downstream disaster recovery clusters receive it    ignored   not replicated to downstream clusters at all  When mounting a secret engine or auth method  you can choose whether to make it a local mount or a shared mount   Shared mounts  the default  usually replicate all their data to performance secondaries  but they can choose to designate specific storage paths as local   For example  PKI mounts store certificates locally  If you query the roles on a shared PKI mount  you ll see the same result for that mount when you send the query to either the performance primary or secondary  but if you list stored certs  you ll see different values   Local mounts replicate their data only to disaster recovery secondaries   A local mount created on a performance primary isn t visible at all to its performance secondaries  Local mounts can also be created on performance secondaries  in which case they aren t visible to the performance primary   There exist other storage entries that aren t mount specific   For example  the Integrated Storage Autopilot configuration is an  ignored  storage entry  which allows for disaster recovery secondaries to have a different configuration than their primary   Tokens and leases are written to  local  storage entries      Performance replication   include  alerts enterprise and hcp mdx   In Performance Replication  secondaries keep track of their own tokens and leases but share the underlying configuration  policies  and supporting secrets  KV values  encryption keys for  transit   etc    If a user action would modify underlying shared state  the secondary forwards the request to the primary to be handled  this is transparent to the client  In practice  most high volume workloads  reads in the  kv  backend  encryption decryption operations in  transit   etc   can be satisfied by the local secondary  allowing Vault to scale relatively horizontally with the number of secondaries rather than vertically as in the past    Note title  Performance replication does not include automated integrated storage snapshots    Vault does not replicate automated integrated storage snapshots as a part of performance replication   You must explicitly configure each of the primary and secondary performance clusters to create individual automated snapshots replicas     Note       Paths filter  The primary cluster s mount configuration gets replicated across its secondary clusters when you enable Performance Replication  In some cases  you may not want all data to be replicated  For example  your primary cluster is in the EU region  and you have a secondary cluster outside of the EU region   General Data Protection Regulation  GDPR   https   gdpr eu   requires that personally identifiable data not be physically transferred to locations outside the European Union unless the region or country has an equal rigor of data protection regulation as the EU   To comply with GDPR  leverage Vault s   paths filter    feature to abide by data movements and sovereignty regulations while ensuring performance access across geographically distributed regions   You can set filters based on the mount path of the secrets engines and namespaces     Performance Replication   primary   img vault mount filter 7 png   In the above example  the  EU GDPR data   path and  office FR  namespace will not be replicated and remain only on the primary cluster   On a similar note  if you want to avoid secondary cluster s data to be replicated  you can mark those secrets engine and or auth methods   local    Local secrets engines and auth methods are not replicated or removed by replication     Example    When you enable a secrets engine on a secondary cluster  use the   local  flag      shell session   vault secrets enable  local  path us west data kv v2       Highlight title  Tutorials    Refer to the  manage replicated mounts  section in the  Set up performance replication   vault tutorials enterprise performance replication manage replicated mounts  tutorial to learn how to specify the mounts to allow or deny data replication     Highlight      Disaster recovery  DR  replication  In disaster recovery  or DR  replication  secondaries share the same underlying configuration  policy  and supporting secrets  KV values  encryption keys for  transit   etc  infrastructure as the primary  They also share the same token and lease infrastructure as the primary  as they are designed to allow for continuous operations with applications connecting to the original primary on the election of the DR secondary   DR is designed to be a mechanism to protect against catastrophic failure of entire clusters  They do not forward service read or write requests until they are elected and become a new primary        Note    Unlike with Performance Replication  local secret engines  auth methods and audit devices are replicated to a DR secondary    For more information on the capabilities of performance and disaster recovery replication  see the Vault Replication  API Documentation   vault api docs system replication       Primary and secondary cluster compatibility      Storage engines  There is no requirement that both clusters use the same storage engine       Seals  There is no requirement that both clusters use the same seal type  but see  sealwrap   vault docs enterprise sealwrap seal wrap and replication  for the full details   Also note that enabling replication will modify the secondary seal  If the secondary uses an auto seal  its recovery configuration and keys will be replaced  if it uses shamir  its seal configuration and unseal keys will be replaced  Here seal recovery configuration means the number of seal recovery key fragments and the required threshold of those fragments     Primary Seal   Secondary Seal  before    Secondary Seal  after                    Secondary Recovery Key  after    Impact on Secondary of Enabling Replication                                                                                Shamir         Shamir                    Primary s shamir config   unseal keys    N A                              Seal config and unseal keys replaced with primary s                             Shamir         Auto                      Unchanged                                Receives primary seal            Seal recovery config and keys replaced with primary s seal config and keys      Auto           Auto                      Unchanged                                Receives primary recovery        Seal recovery config and recovery keys replaced with primary s                  Auto           Shamir                    Receives primary recovery                N A                              Seal config and keys replaced with primary s recovery seal config and keys      Note   Vault clusters configured with  auto unseal   vault docs concepts seal auto unseal  have recovery keys instead of unseal keys     Note       Vault versions  Vault changes are designed and tested to ensure that the  upgrade instructions   vault docs upgrading replication installations  are viable  i e  that a secondary can run a newer Vault version than its primary   That said  we do not recommend running replicated Vault clusters with different versions any longer than necessary to perform the upgrade      Internals  Details on the internal design of the replication feature can be found in the  replication internals   vault docs internals replication  document      Security model  Vault is trusted all over the world to keep secrets safe  As such  we have put extreme focus to detail to our replication model as well       Primary Secondary communication  When a cluster is marked as the primary it generates a self signed CA certificate  On request  and given a user specified identifier  the primary uses this CA certificate to generate a private key and certificate and packages these  along with some other information  into a replication bootstrapping bundle  a k a  a secondary activation token  The certificate is used to perform TLS mutual authentication between the primary and that secondary   This CA certificate is never shared with secondaries  and no secondary ever has access to any other secondary s certificate  In practice this means that revoking a secondary s access to the primary does not allow it continue replication with any other machine  it also means that if a primary goes down  there is full administrative control over which cluster becomes primary  An attacker cannot spoof a secondary into believing that a cluster the attacker controls is the new primary without also being able to administratively direct the secondary to connect by giving it a new bootstrap package  which is an ACL protected call    Vault makes use of Application Layer Protocol Negotiation on its cluster port  This allows the same port to handle both request forwarding and replication  even while keeping the certificate root of trust and feature set different       Secondary activation tokens  A secondary activation token is an extremely sensitive item and as such is protected via response wrapping  Experienced Vault users will note that the wrapping format for replication bootstrap packages is different from normal response wrapping tokens  it is a signed JWT  This allows the replication token to carry the redirect address of the primary cluster as part of the token  In most cases this means that simply providing the token to a new secondary is enough to activate replication  although this can also be overridden when the token is provided to the secondary   Secondary activation tokens should be treated like Vault root tokens  If disclosed to a bad actor  that actor can gain access to all Vault data  It should therefore be treated with utmost sensitivity  Like all response wrapping tokens  once the token is used successfully  in this case  to activate a secondary  it is useless  so it is only necessary to safeguard it from one machine to the next  Like with root tokens  HashiCorp recommends that when a secondary activation token is live  there are multiple eyes on it from generation until it is used   Once a secondary is activated  its cluster information is stored safely behind its encrypted barrier      Mutual TLS and load balancers  Vault generates its own certificates for cluster members  All replication traffic uses the cluster port using these Vault generated certificates after initial bootstrapping  Because of this  the cluster traffic can NOT be terminated at the cluster port at a load balancer level      Tutorial  Refer to the following tutorials replication setup and best practices      Set up Performance Replication   vault tutorials enterprise performance replication     Disaster Recovery Replication Setup   vault tutorials enterprise disaster recovery     Monitoring Vault Replication   vault tutorials monitoring monitor replication      API  The Vault replication component has a full HTTP API  Refer to the  Vault Replication API   vault api docs system replication  for more details