terraform Learn the authorization model potential security threats and our recommendations for securely using HCP Terraform HCP Terraform security model page title Security Model HCP Terraform tfc only true

Answer:
---
page_title: Security Model - HCP Terraform
description: >-
  Learn the authorization model, potential security threats, and our
  recommendations for securely using HCP Terraform.
tfc_only: true
---

# HCP Terraform security model

## Purpose of this document

This document explains the security model of HCP Terraform and the security controls available to end users. Additionally, it provides best practices for securely managing your infrastructure with HCP Terraform.

## Important concepts

### Projects, workspaces, and teams
HCP Terraform organizes infrastructure with workspaces. Workspaces represent a logical security boundary within the organization. Variables, state, SSH keys, and log output are local to a workspace. You can grant teams [read, plan, write, admin, or a customized set of permissions](/terraform/cloud-docs/users-teams-organizations/permissions) within a workspace.

Projects let you group related workspaces in your organization. You can use projects to assign [read, write, maintain, admin, or a customized set of permissions](/terraform/cloud-docs/users-teams-organizations/permissions#project-permissions) to a particular team which grants specific permissions to all workspaces in the project.

### Terraform runs - plans and applies

HCP Terraform will provision infrastructure according to your Terraform configuration which you can upload through the VCS-driven, API-driven, or CLI-driven workflows. You can read more about the different workflows [here](/terraform/cloud-docs/run/remote-operations#starting-runs). It’s important to note that HCP Terraform performs all Terraform operations within the same privilege context. Both the plan and apply operations have access to the full workspace variables, state versions, and Terraform configuration.

### Terraform state file

HCP Terraform retains the current and all historical [state](/terraform/language/state) versions for each workspace. Depending on the resources that are used in your Terraform configuration, these state versions may contain sensitive data such as database passwords, resource IDs, etc.

## Personas

### Organization owners

Members of the [owners team](/terraform/cloud-docs/users-teams-organizations/teams#the-owners-team) have administrator-level privileges within an organization. Members of this team will have access to workspaces, projects, and settings within the organization. This role is intended for users who will perform administrative tasks in your organization.

### Workspace and project team members
Teams let you group users within an organization. You can grant teams [read, plan, write, admin, or a customized set of permissions](/terraform/cloud-docs/users-teams-organizations/permissions), each of which allow them to perform various functions within the workspace. You can also grant teams [read, write, maintain, admin, or a customized set of permissions for a project](/terraform/cloud-docs/users-teams-organizations/permissions#project-permissions), which grants specific permissions to any workspaces in that project. At a higher level, you can use [organization-level privileges](/terraform/cloud-docs/users-teams-organizations/permissions#organization-permissions), which apply to projects and workspaces across the organization.

### Contributors to connected VCS repositories

HCP Terraform executes Terraform configuration from connected VCS repositories. Depending on the configuration, HCP Terraform may automatically trigger Terraform operations when the connected repositories receive new contributions.

## Authorization model

[![HCP Terraform authorization model diagram](/img/docs/terraform-cloud-authorization.png)](/img/docs/terraform-cloud-authorization.png)
_Click on the diagram for a larger view\._

~> **Note:** This diagram displays a useful subset of HCP Terraform's authorization model, but is not comprehensive. Some details were omitted for the sake of clarity. More information is available in our [Permissions documentation](/terraform/cloud-docs/users-teams-organizations/permissions).

Workspaces provide a logical security boundary within the organization. Environment variables and Terraform configurations are isolated within a workspace, and access to a workspace is granted on a per-team basis.

All organizations in HCP Terraform contain an “owners” team, which grants admin-level access to the organization and all its workspaces.

~> **Note:** Teams are not available to free-tier users on HCP Terraform. Organizations at the free-level will only have an owners team.

All workspaces in an organization belong to a project. You can grant teams [read, write, maintain, admin, or a customized set of permissions for the project](/terraform/cloud-docs/users-teams-organizations/permissions#project-permissions), which grants specific permissions on on all workspaces within the project. You can also grant teams [read, plan, write, admin, or a customized set of permissions](/terraform/cloud-docs/users-teams-organizations/permissions#workspace-permissions) for a specific workspace. It’s important to note that, from a security perspective, the plan permission is equivalent to the write permission. The plan permission is provided to protect against accidental Terraform runs but is not intended to stop malicious actors from accessing sensitive data within a workspace. Terraform `plan` and `apply` operations can execute arbitrary code within the ephemeral build environment. Both of these operations happen in the same security context with access to the full set of workspace variables, Terraform configuration, and Terraform state.

By default, Teams with read privileges within a workspace can view the workspace's state. You can remove this access by using [customized workspace permissions](/terraform/cloud-docs/users-teams-organizations/permissions#custom-workspace-permissions); however, this will only apply to state file access through the API or UI. Terraform must access the state file in order to perform plan and apply operations, so any user with the ability to upload Terraform configurations and initiate runs will transitively have access to the workspaces' state.

State may be shared across workspaces via the [remote state access workspace setting](/terraform/cloud-docs/workspaces/state#accessing-state-from-other-workspaces).

Terraform configuration files in connected VCS repositories are inherently trusted. Commits to connected repositories will automatically queue a plan within the corresponding workspace. Pull requests to connected repositories will initiate a speculative plan, though this behavior may be disabled via the [speculative plan setting](/terraform/cloud-docs/workspaces/settings/vcs#automatic-speculative-plans) on the workspace settings page. HCP Terraform has no knowledge of your VCS's authorization controls and does not associate HCP Terraform user accounts with VCS user accounts — the two should be considered separate identities.

## Threat model

HCP Terraform is designed to execute Terraform operations and manage the state file to ensure that infrastructure is reliably created, updated, and destroyed by multiple users of an organization.

The following are part of the HCP Terraform threat model:

### Confidentiality and integrity of communication between Terraform clients and HCP Terraform

All communication between clients and HCP Terraform is encrypted end-to-end using TLS. HCP Terraform currently supports TLS version 1.2. HCP Terraform communicates with linked VCS repositories using the Oauth2 authorization protocol. HCP Terraform can also be configured to fetch Terraform modules from private repositories using the SSH protocol with a customer-provided private key.

### Confidentiality of state versions, Terraform configurations, and stored variables

As a user, you will entrust HCP Terraform with information that is very sensitive to your organization such as API tokens, your Terraform configurations, and your Terraform state file. HCP Terraform is designed to ensure the confidentiality of this information, it relies on [Vault Transit](/vault/docs/secrets/transit) for encrypting workspace variables. Terraform configurations and state are encrypted at rest with uniquely derived encryption keys backed by Vault. You can view how all customer data is encrypted and stored on our [data security page](/terraform/cloud-docs/architectural-details/data-security).

### Enforcement of authentication and authorization policies for data access and actions taken through the UI or API

HCP Terraform enforces authorization checks for all actions taken within the API or through the UI. More information about HCP Terraform workspace-level and organization level permission are available [here](/terraform/cloud-docs/users-teams-organizations/permissions).

### Isolation of Terraform executions

Each Terraform operation (plan and apply) happens in an ephemeral environment that is created immediately before the run and destroyed after it is completed. The build environment is designed to provide isolation between Terraform executions and between HCP Terraform tenants.

### Reliability and availability of HCP Terraform

HCP Terraform is spread across multiple availability zones for reliability, we perform regular backups of our production data stores and have a process for recovering in case of a major outage.

## What isn’t part of the threat model

### Malicious contributions to Terraform configuration in VCS repositories

Commits and pull requests to connected VCS repositories will trigger a plan operation within the workspace. HCP Terraform does not perform any authentication or authorization checks against commits in linked VCS repositories, and cannot prevent malicious Terraform configuration from exfiltrating sensitive data during plan operations. For this reason, it is important to restrict access to connected VCS repositories. Speculative plans for pull requests may be disabled on the [workspace settings page](/terraform/cloud-docs/workspaces/settings/vcs#automatic-speculative-plans).

-> **Note:** HCP Terraform will not automatically trigger plans for pull requests from forked repositories.

### Malicious Terraform providers or modules

Terraform providers and modules used in your Terraform configuration will have full access to the variables and Terraform state within a workspace. HCP Terraform cannot prevent malicious providers and modules from exfiltrating this sensitive data. We recommend only using trusted modules and providers within your Terraform configuration.

### Malicious bypasses of Terraform policies
The policy-as-code frameworks used by the Terraform [Policy Enforcement](/terraform/cloud-docs/policy-enforcement) feature are embedded within HCP Terraform and can be used to ensure the infrastructure provisioned using Terraform complies with defined organizational policies. The goal of this feature is to enforce compliance with organizational policies and best practices when provisioning infrastructure using Terraform.

It is important to note that the policy-as-code integration in HCP Terraform should be viewed as a guide or set of guardrails, not a security boundary. It is not designed to prevent malicious actors from executing malicious Terraform configurations or modifying infrastructure.

### Malicious or insecure third-party run tasks
Terraform [Run Tasks](/terraform/cloud-docs/integrations/run-tasks) are provided with access to all Terraform configuration and plan data. HCP Terraform does not have the capability to prevent malicious Run Tasks from potentially exfiltrating sensitive data that may be present in either the Terraform configuration or plan.

In order to minimize potential security risks, it is highly recommended to only utilize trusted technology partners for Run Tasks within your Terraform organization and limit the number of users who have been assigned the [Manage Run Tasks](/terraform/cloud-docs/users-teams-organizations/permissions#manage-run-tasks) permission.

### Access to sensitive variables or state from Terraform operations

Marking a variable as “sensitive” will prevent it from being displayed in the UI, but will not prevent it from being read by Terraform during plan or apply operations. Similarly, customized workspace permissions allow you to restrict access to workspace state via the UI and API, but will not prevent it from being read during Terraform operations.

### Redaction of sensitive variables in Terraform logs

The logs from a Terraform plan or apply operation are visible to any user with at least “read” level access in the associated workspace. While Terraform tries to avoid writing sensitive information to logs, redactions are best-effort. This feature should not be treated as a security boundary, but instead as a mechanism to mitigate accidental exposure. Additionally, HCP Terraform is unable to protect against malicious users who attempt to use Terraform logs to exfiltrate sensitive data.

## Recommendations for securely using HCP Terraform

### Enforce strong authentication

HCP Terraform supports [two factor authentication](/terraform/cloud-docs/users-teams-organizations/2fa) via SMS or TOTP. Organizations can configure mandatory 2FA for all members in the [organization settings](/terraform/cloud-docs/users-teams-organizations/organizations#authentication). Organizations may choose to configure [SSO for their organization](/terraform/cloud-docs/users-teams-organizations/single-sign-on).

### Minimize the number of users in the owners team

Users of the [owners team](/terraform/cloud-docs/users-teams-organizations/teams#the-owners-team) will have full access to all workspaces within the organization. If SSO is enabled, members of the “Owners” team will still be able to authenticate with their username and password. This group should be reserved for only a small number of administrators, and membership should be audited periodically.

### Apply the principle of least privilege to workspace membership

[Teams](/terraform/cloud-docs/users-teams-organizations/teams) allow you to group users and assign them various privileges within workspaces. We recommend applying the [principle of least privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege) when creating teams and assigning permissions so that each user within your organization has the minimum required privileges.

### Protect API keys

HCP Terraform allows you to create [user, team, and organization API tokens](/terraform/cloud-docs/api-docs#authentication). You should take care to store these tokens securely, and rotate them periodically.
Vault users can leverage the [Terraform Cloud secret backend](/vault/docs/secrets/terraform), which allows you to generate ephemeral tokens.

### Control access to source code

By default, commits and pull requests to connected VCS repositories will automatically trigger a plan operation in an HCP Terraform workspace. HCP Terraform cannot protect against malicious code in linked repositories, so you should take care to only grant trusted operators access to these repositories.
Workspaces may be configured to [enable or disable speculative plans for pull requests](/terraform/cloud-docs/workspaces/settings/vcs#automatic-speculative-plans) to linked repositories. You should disable this setting if you allow untrusted users to open pull requests in connected VCS repositories.

-> **Note:** HCP Terraform will not automatically trigger plans for pull requests from forked repositories.

### Restrict access to workspace state

Workspaces may be configured to share their state with other workspaces within the organization or globally with the entire organization via the [remote state setting](/terraform/cloud-docs/workspaces/state#accessing-state-from-other-workspaces). Because workspace state may contain sensitive information, we recommend that you follow the principle of least privilege and only enable state access between workspaces that specifically need information from each other.

### Use separate agent pools for sensitive workspaces

You can share [HCP Terraform Agents](/terraform/cloud-docs/agents) across all workspaces within an organization or [scope them to specific workspaces](/terraform/cloud-docs/agents#scope-an-agent-pool-to-specific-workspaces). If multiple workspaces share agent pools, a malicious actor in one of those workspaces could exfiltrate the agent’s API token, access private resources from the perspective of the agent, or modify the agent’s environment, potentially impacting other workspaces. For this reason, we recommend creating separate agent pools for sensitive workspaces and using the agent scoping setting to restrict which workspaces can target each agent pool.

### Treat Archivist URLs as secrets

HCP Terraform uses a blob storage service called Archivist for storing various pieces of customer data. Archivist URLs have the origin `https://archivist.terraform.io` and are returned by various HCP Terraform APIs, such as the [state versions API](/terraform/cloud-docs/api-docs/state-versions#fetch-the-current-state-version-for-a-workspace). You do not need to submit a bearer token with each request to call the Archivist API. Instead, Archivist URLs contain a short-term signed authorization token that performs authorization checks. The expiry time depends on the API endpoints you used to generate the Archivist link. As a result, you must treat Archivist URLs as secrets and avoid logging or sharing them.

### Use dynamic credentials

Storing static credentials in HCP Terraform increases the inherent risk of a malicious user or a compromised plan or apply operation exposing your credentials. Because static credentials are usually long-lived and exposed in many locations, they are troublesome to revoke and replace.

Using [dynamic provider credentials](/terraform/cloud-docs/workspaces/dynamic-provider-credentials/) eliminates the need to store static credentials in HCP Terraform, reducing the risk of exposure. Dynamic provider credentials generate new temporary credentials for each operation and expire after that operation completes.

page title  Security Model   HCP Terraform description       Learn the authorization model  potential security threats  and our   recommendations for securely using HCP Terraform  tfc only  true        HCP Terraform security model     Purpose of this document  This document explains the security model of HCP Terraform and the security controls available to end users  Additionally  it provides best practices for securely managing your infrastructure with HCP Terraform      Important concepts      Projects  workspaces  and teams HCP Terraform organizes infrastructure with workspaces  Workspaces represent a logical security boundary within the organization  Variables  state  SSH keys  and log output are local to a workspace  You can grant teams  read  plan  write  admin  or a customized set of permissions   terraform cloud docs users teams organizations permissions  within a workspace   Projects let you group related workspaces in your organization  You can use projects to assign  read  write  maintain  admin  or a customized set of permissions   terraform cloud docs users teams organizations permissions project permissions  to a particular team which grants specific permissions to all workspaces in the project       Terraform runs   plans and applies  HCP Terraform will provision infrastructure according to your Terraform configuration which you can upload through the VCS driven  API driven  or CLI driven workflows  You can read more about the different workflows  here   terraform cloud docs run remote operations starting runs   It s important to note that HCP Terraform performs all Terraform operations within the same privilege context  Both the plan and apply operations have access to the full workspace variables  state versions  and Terraform configuration       Terraform state file  HCP Terraform retains the current and all historical  state   terraform language state  versions for each workspace  Depending on the resources that are used in your Terraform configuration  these state versions may contain sensitive data such as database passwords  resource IDs  etc      Personas      Organization owners  Members of the  owners team   terraform cloud docs users teams organizations teams the owners team  have administrator level privileges within an organization  Members of this team will have access to workspaces  projects  and settings within the organization  This role is intended for users who will perform administrative tasks in your organization       Workspace and project team members Teams let you group users within an organization  You can grant teams  read  plan  write  admin  or a customized set of permissions   terraform cloud docs users teams organizations permissions   each of which allow them to perform various functions within the workspace  You can also grant teams  read  write  maintain  admin  or a customized set of permissions for a project   terraform cloud docs users teams organizations permissions project permissions   which grants specific permissions to any workspaces in that project  At a higher level  you can use  organization level privileges   terraform cloud docs users teams organizations permissions organization permissions   which apply to projects and workspaces across the organization       Contributors to connected VCS repositories  HCP Terraform executes Terraform configuration from connected VCS repositories  Depending on the configuration  HCP Terraform may automatically trigger Terraform operations when the connected repositories receive new contributions      Authorization model     HCP Terraform authorization model diagram   img docs terraform cloud authorization png    img docs terraform cloud authorization png   Click on the diagram for a larger view          Note    This diagram displays a useful subset of HCP Terraform s authorization model  but is not comprehensive  Some details were omitted for the sake of clarity  More information is available in our  Permissions documentation   terraform cloud docs users teams organizations permissions    Workspaces provide a logical security boundary within the organization  Environment variables and Terraform configurations are isolated within a workspace  and access to a workspace is granted on a per team basis   All organizations in HCP Terraform contain an  owners  team  which grants admin level access to the organization and all its workspaces        Note    Teams are not available to free tier users on HCP Terraform  Organizations at the free level will only have an owners team   All workspaces in an organization belong to a project  You can grant teams  read  write  maintain  admin  or a customized set of permissions for the project   terraform cloud docs users teams organizations permissions project permissions   which grants specific permissions on on all workspaces within the project  You can also grant teams  read  plan  write  admin  or a customized set of permissions   terraform cloud docs users teams organizations permissions workspace permissions  for a specific workspace  It s important to note that  from a security perspective  the plan permission is equivalent to the write permission  The plan permission is provided to protect against accidental Terraform runs but is not intended to stop malicious actors from accessing sensitive data within a workspace  Terraform  plan  and  apply  operations can execute arbitrary code within the ephemeral build environment  Both of these operations happen in the same security context with access to the full set of workspace variables  Terraform configuration  and Terraform state   By default  Teams with read privileges within a workspace can view the workspace s state  You can remove this access by using  customized workspace permissions   terraform cloud docs users teams organizations permissions custom workspace permissions   however  this will only apply to state file access through the API or UI  Terraform must access the state file in order to perform plan and apply operations  so any user with the ability to upload Terraform configurations and initiate runs will transitively have access to the workspaces  state   State may be shared across workspaces via the  remote state access workspace setting   terraform cloud docs workspaces state accessing state from other workspaces    Terraform configuration files in connected VCS repositories are inherently trusted  Commits to connected repositories will automatically queue a plan within the corresponding workspace  Pull requests to connected repositories will initiate a speculative plan  though this behavior may be disabled via the  speculative plan setting   terraform cloud docs workspaces settings vcs automatic speculative plans  on the workspace settings page  HCP Terraform has no knowledge of your VCS s authorization controls and does not associate HCP Terraform user accounts with VCS user accounts   the two should be considered separate identities      Threat model  HCP Terraform is designed to execute Terraform operations and manage the state file to ensure that infrastructure is reliably created  updated  and destroyed by multiple users of an organization   The following are part of the HCP Terraform threat model       Confidentiality and integrity of communication between Terraform clients and HCP Terraform  All communication between clients and HCP Terraform is encrypted end to end using TLS  HCP Terraform currently supports TLS version 1 2  HCP Terraform communicates with linked VCS repositories using the Oauth2 authorization protocol  HCP Terraform can also be configured to fetch Terraform modules from private repositories using the SSH protocol with a customer provided private key       Confidentiality of state versions  Terraform configurations  and stored variables  As a user  you will entrust HCP Terraform with information that is very sensitive to your organization such as API tokens  your Terraform configurations  and your Terraform state file  HCP Terraform is designed to ensure the confidentiality of this information  it relies on  Vault Transit   vault docs secrets transit  for encrypting workspace variables  Terraform configurations and state are encrypted at rest with uniquely derived encryption keys backed by Vault  You can view how all customer data is encrypted and stored on our  data security page   terraform cloud docs architectural details data security        Enforcement of authentication and authorization policies for data access and actions taken through the UI or API  HCP Terraform enforces authorization checks for all actions taken within the API or through the UI  More information about HCP Terraform workspace level and organization level permission are available  here   terraform cloud docs users teams organizations permissions        Isolation of Terraform executions  Each Terraform operation  plan and apply  happens in an ephemeral environment that is created immediately before the run and destroyed after it is completed  The build environment is designed to provide isolation between Terraform executions and between HCP Terraform tenants       Reliability and availability of HCP Terraform  HCP Terraform is spread across multiple availability zones for reliability  we perform regular backups of our production data stores and have a process for recovering in case of a major outage      What isn t part of the threat model      Malicious contributions to Terraform configuration in VCS repositories  Commits and pull requests to connected VCS repositories will trigger a plan operation within the workspace  HCP Terraform does not perform any authentication or authorization checks against commits in linked VCS repositories  and cannot prevent malicious Terraform configuration from exfiltrating sensitive data during plan operations  For this reason  it is important to restrict access to connected VCS repositories  Speculative plans for pull requests may be disabled on the  workspace settings page   terraform cloud docs workspaces settings vcs automatic speculative plans         Note    HCP Terraform will not automatically trigger plans for pull requests from forked repositories       Malicious Terraform providers or modules  Terraform providers and modules used in your Terraform configuration will have full access to the variables and Terraform state within a workspace  HCP Terraform cannot prevent malicious providers and modules from exfiltrating this sensitive data  We recommend only using trusted modules and providers within your Terraform configuration       Malicious bypasses of Terraform policies The policy as code frameworks used by the Terraform  Policy Enforcement   terraform cloud docs policy enforcement  feature are embedded within HCP Terraform and can be used to ensure the infrastructure provisioned using Terraform complies with defined organizational policies  The goal of this feature is to enforce compliance with organizational policies and best practices when provisioning infrastructure using Terraform   It is important to note that the policy as code integration in HCP Terraform should be viewed as a guide or set of guardrails  not a security boundary  It is not designed to prevent malicious actors from executing malicious Terraform configurations or modifying infrastructure       Malicious or insecure third party run tasks Terraform  Run Tasks   terraform cloud docs integrations run tasks  are provided with access to all Terraform configuration and plan data  HCP Terraform does not have the capability to prevent malicious Run Tasks from potentially exfiltrating sensitive data that may be present in either the Terraform configuration or plan   In order to minimize potential security risks  it is highly recommended to only utilize trusted technology partners for Run Tasks within your Terraform organization and limit the number of users who have been assigned the  Manage Run Tasks   terraform cloud docs users teams organizations permissions manage run tasks  permission       Access to sensitive variables or state from Terraform operations  Marking a variable as  sensitive  will prevent it from being displayed in the UI  but will not prevent it from being read by Terraform during plan or apply operations  Similarly  customized workspace permissions allow you to restrict access to workspace state via the UI and API  but will not prevent it from being read during Terraform operations       Redaction of sensitive variables in Terraform logs  The logs from a Terraform plan or apply operation are visible to any user with at least  read  level access in the associated workspace  While Terraform tries to avoid writing sensitive information to logs  redactions are best effort  This feature should not be treated as a security boundary  but instead as a mechanism to mitigate accidental exposure  Additionally  HCP Terraform is unable to protect against malicious users who attempt to use Terraform logs to exfiltrate sensitive data      Recommendations for securely using HCP Terraform      Enforce strong authentication  HCP Terraform supports  two factor authentication   terraform cloud docs users teams organizations 2fa  via SMS or TOTP  Organizations can configure mandatory 2FA for all members in the  organization settings   terraform cloud docs users teams organizations organizations authentication   Organizations may choose to configure  SSO for their organization   terraform cloud docs users teams organizations single sign on        Minimize the number of users in the owners team  Users of the  owners team   terraform cloud docs users teams organizations teams the owners team  will have full access to all workspaces within the organization  If SSO is enabled  members of the  Owners  team will still be able to authenticate with their username and password  This group should be reserved for only a small number of administrators  and membership should be audited periodically       Apply the principle of least privilege to workspace membership   Teams   terraform cloud docs users teams organizations teams  allow you to group users and assign them various privileges within workspaces  We recommend applying the  principle of least privilege  https   en wikipedia org wiki Principle of least privilege  when creating teams and assigning permissions so that each user within your organization has the minimum required privileges       Protect API keys  HCP Terraform allows you to create  user  team  and organization API tokens   terraform cloud docs api docs authentication   You should take care to store these tokens securely  and rotate them periodically  Vault users can leverage the  Terraform Cloud secret backend   vault docs secrets terraform   which allows you to generate ephemeral tokens       Control access to source code  By default  commits and pull requests to connected VCS repositories will automatically trigger a plan operation in an HCP Terraform workspace  HCP Terraform cannot protect against malicious code in linked repositories  so you should take care to only grant trusted operators access to these repositories  Workspaces may be configured to  enable or disable speculative plans for pull requests   terraform cloud docs workspaces settings vcs automatic speculative plans  to linked repositories  You should disable this setting if you allow untrusted users to open pull requests in connected VCS repositories        Note    HCP Terraform will not automatically trigger plans for pull requests from forked repositories       Restrict access to workspace state  Workspaces may be configured to share their state with other workspaces within the organization or globally with the entire organization via the  remote state setting   terraform cloud docs workspaces state accessing state from other workspaces   Because workspace state may contain sensitive information  we recommend that you follow the principle of least privilege and only enable state access between workspaces that specifically need information from each other       Use separate agent pools for sensitive workspaces  You can share  HCP Terraform Agents   terraform cloud docs agents  across all workspaces within an organization or  scope them to specific workspaces   terraform cloud docs agents scope an agent pool to specific workspaces   If multiple workspaces share agent pools  a malicious actor in one of those workspaces could exfiltrate the agent s API token  access private resources from the perspective of the agent  or modify the agent s environment  potentially impacting other workspaces  For this reason  we recommend creating separate agent pools for sensitive workspaces and using the agent scoping setting to restrict which workspaces can target each agent pool       Treat Archivist URLs as secrets  HCP Terraform uses a blob storage service called Archivist for storing various pieces of customer data  Archivist URLs have the origin  https   archivist terraform io  and are returned by various HCP Terraform APIs  such as the  state versions API   terraform cloud docs api docs state versions fetch the current state version for a workspace   You do not need to submit a bearer token with each request to call the Archivist API  Instead  Archivist URLs contain a short term signed authorization token that performs authorization checks  The expiry time depends on the API endpoints you used to generate the Archivist link  As a result  you must treat Archivist URLs as secrets and avoid logging or sharing them       Use dynamic credentials  Storing static credentials in HCP Terraform increases the inherent risk of a malicious user or a compromised plan or apply operation exposing your credentials  Because static credentials are usually long lived and exposed in many locations  they are troublesome to revoke and replace   Using  dynamic provider credentials   terraform cloud docs workspaces dynamic provider credentials   eliminates the need to store static credentials in HCP Terraform  reducing the risk of exposure  Dynamic provider credentials generate new temporary credentials for each operation and expire after that operation completes


consul to config There may be a few other special cases not included but this covers off as you go you can mark them as done by replace with so github renders them as checked Then please include the completed lists you worked This is a checklist of all the places you need to update when adding a new field We suggest you copy the raw markdown into a gist or local file and check them through in your PR description the majority of configs Adding a Consul Config Field

Answer:
# Adding a Consul Config Field

This is a checklist of all the places you need to update when adding a new field
to config. There may be a few other special cases not included but this covers
the majority of configs.

We suggest you copy the raw markdown into a gist or local file and check them
off as you go (you can mark them as done by replace `[ ]` with `[x]` so github
renders them as checked). Then **please include the completed lists you worked
through in your PR description**.

Examples of special cases this doesn't cover are:
 - If the config needs special treatment like a different default in `-dev` mode
   or differences between CE and Enterprise.
 - If custom logic is needed to support backwards compatibility when changing
   syntax or semantics of anything

There are four specific cases covered with increasing complexity:
 1. adding a simple config field only used by client agents
 1. adding a CLI flag to mirror that config field
 1. adding a config field that needs to be used in Consul servers
 1. adding a field to the Service Definition

## Adding a Simple Config Field for Client Agents

 - [ ] Add the field to the Config struct (or an appropriate sub-struct) in
   `agent/config/config.go`.
 - [ ] Add the field to the actual RuntimeConfig struct in
   `agent/config/runtime.go`.
 - [ ] Add an appropriate parser/setter in `agent/config/builder.go` to
   translate.
 - [ ] Add the new field with a random value to both the JSON and HCL files in
   `agent/config/testdata/full-config.*`, which should cause the test to fail.
   Then update the expected value in `TestLoad_FullConfig` in
   `agent/config/runtime_test.go` to make the test pass again.
 - [ ] Run `go test -run TestRuntimeConfig_Sanitize ./agent/config -update` to update
   the expected value for `TestRuntimeConfig_Sanitize`. Look at `git diff` to
   make sure the value changed as you expect.
 - [ ] **If** your new config field needed some validation as it's only valid in
   some cases or with some values (often true).
      - [ ] Add validation to Validate in `agent/config/builder.go`.
      - [ ] Add a test case to the table test `TestLoad_IntegrationWithFlags` in
        `agent/config/runtime_test.go`.
 - [ ] **If** your new config field needs a non-zero-value default.
      - [ ] Add that to `DefaultSource` in `agent/config/defaults.go`.
      - [ ] Add a test case to the table test `TestLoad_IntegrationWithFlags` in
        `agent/config/runtime_test.go`.
      - [ ] If the config needs to be defaulted for the test server used in unit tests,
            also add it to `DefaultConfig()` in `agent/consul/config.go`.
 - [ ] **If** your config should take effect on a reload/HUP.
      - [ ] Add necessary code to to trigger a safe (locked or atomic) update to
        any state the feature needs changing. This needs to be added to one or
        more of the following places:
         - `ReloadConfig` in `agent/agent.go` if it needs to affect the local
           client state or another client agent component.
         - `ReloadConfig` in `agent/consul/client.go` if it needs to affect
           state for client agent's RPC client.
      - [ ] Add a test to `agent/agent_test.go` similar to others with prefix
        `TestAgent_reloadConfig*`.
 - [ ] Add documentation to `website/content/docs/agent/config/config-files.mdx`.

Done! You can now use your new field in a client agent by accessing
`s.agent.Config.<FieldName>`.

If you need a CLI flag, access to the variable in a Server context, or touched
the Service Definition, make sure you continue on to follow the appropriate
checklists below.

## Adding a CLI Flag Corresponding to the new Field
If the config field also needs a CLI flag, then follow these steps.

 - [ ] Do all of the steps in [Adding a Simple Config
   Field For Client Agents](#adding-a-simple-config-field-for-client-agents).
 - [ ] Add the new flag to `agent/config/flags.go`.
 - [ ] Add a test case to TestParseFlags in `agent/config/flag_test.go`.
 - [ ] Add a test case (or extend one if appropriate) to the table test
   `TestLoad_IntegrationWithFlags` in `agent/config/runtime_test.go` to ensure setting the
   flag works.
 - [ ] Add flag (as well as config file) documentation to
   `website/source/docs/agent/config/config-files.mdx` and `website/source/docs/agent/config/cli-flags.mdx`.

## Adding a Simple Config Field for Servers
Consul servers have a separate Config struct for reasons. Note that Consul
server agents are actually also client agents, so in some cases config that is
only destined for servers doesn't need to follow this checklist provided it's
only needed during the bootstrapping of the server (which is done in code shared
by both server and client components in `agent.go`). For example WAN Gossip
configs are only valid on server agents but since WAN Gossip is setup in
`agent.go` they don't need to follow this checklist. The simplest (and mostly
accurate) rule is:

> If you need to access the config field from code in  `agent/consul` (e.g. RPC
> endpoints), then you need to follow this. If it's only in `agent` (e.g. HTTP
> endpoints or agent startup) you don't.

A final word of warning - **you should never need to pass config into the FSM
(`agent/consul/fsm`) or state store (`agent/consul/state`)**. Doing so is **_very
dangerous_** and can violate consistency guarantees and corrupt databases. If
you think you need this then please discuss the design with the Consul team
before writing code!

Consul's server components for historical reasons don't use the `RuntimeConfig`
struct they have their own struct called `Config` in `agent/consul/config.go`.

 - [ ] Do all of the steps in [Adding a Simple Config
   Field For Client Agents](#adding-a-simple-config-field-for-client-agents).
 - [ ] Add the new field to Config struct in `agent/consul/config.go`
 - [ ] Add code to set the values from the `RuntimeConfig` in `newConsulConfig` method in `agent/agent.go`
 - [ ] **If needed**, add a test to `agent_test.go` if there is some non-trivial
   behavior in the code you added in the previous step. We tend not to test
   simple assignments from one to the other since these are typically caught by
   higher-level tests of the actual functionality that matters but some examples
   can be found prefixed with `TestAgent_consulConfig*`
 - [ ] **If** your config should take effect on a reload/HUP
      - [ ] Add necessary code to `ReloadConfig` in `agent/consul/server.go` this
        needs to be adequately synchronized with any readers of the state being
        updated.
       - [ ] Add a new test or a new assertion to `TestServer_ReloadConfig`

You can now access that field from `s.srv.config.<FieldName>` inside an RPC
handler.

## Adding a New Field to Service Definition
The [Service Definition](https://www.consul.io/docs/agent/services.html) syntax
appears both in Consul config files but also in the `/v1/agent/service/register`
API.

For wonderful historical reasons, our config files have always used `snake_case`
attribute names in both JSON and HCL (even before we supported HCL!!) while our
API uses `CamelCase`.

Because we want documentation examples to work in both config files and API
bodies to avoid needless confusion, we have to accept both snake case and camel
case field names for the service definition.

Finally, adding a field to the service definition implies adding the field to
several internal structs and to all API outputs that display services from the
catalog. That explains the multiple layers needed below.

This list assumes a new field in the base service definition struct. Adding new
fields to health checks is similar but mostly needs `HealthCheck` structs and
methods updating instead. Adding fields to embedded structs like `ProxyConfig`
is largely the same pattern but may need different test methods etc. updating.

 - [ ] Do all of the steps in [Adding a Simple Config
   Field For Client Agents](#adding-a-simple-config-field-for-client-agents).
 - [ ] `agent/structs` package
      - [ ] Add the field to `ServiceDefinition` (`service_definition.go`)
      - [ ] Add the field to `NodeService` (`structs.go`)
      - [ ] Add the field to `ServiceNode` (`structs.go`)
      - [ ] Update `ServiceDefinition.ToNodeService` to translate the field
      - [ ] Update `NodeService.ToServiceNode` to translate the field
      - [ ] Update `ServiceNode.ToNodeService` to translate the field
      - [ ] Update `TestStructs_ServiceNode_Conversions`
      - [ ] Update `ServiceNode.PartialClone`
      - [ ] Update `TestStructs_ServiceNode_PartialClone` (`structs_test.go`)
      - [ ] If needed, update `NodeService.Validate` to ensure the field value is
        reasonable
      - [ ] Add test like `TestStructs_NodeService_Validate*` in
        `structs_test.go`
      - [ ] Add comparison in `NodeService.IsSame`
      - [ ] Update `TestStructs_NodeService_IsSame`
      - [ ] Add comparison in `ServiceNode.IsSameService`
      - [ ] Update `TestStructs_ServiceNode_IsSameService`
      - [ ] **If** your field name has MultipleWords,
          - [ ] Add it to the `aux` inline struct in
            `ServiceDefinition.UnmarshalJSON` (`service_defintion.go`).
            - Note: if the field is embedded higher up in a nested struct,
              follow the chain and update the necessary struct's `UnmarshalJSON`
              method - you may need to add one if there are no other case
              transformations being done, copy and existing example.
            - Note: the tests that exercise this are in agent endpoint for
              historical reasons (this is where the translation used to happen).
 - [ ] `agent` package
      - [ ] Update `testAgent_RegisterService` and/or add a new test to ensure
        your fields register correctly via API (`agent_endpoint_test.go`)
      - [ ] **If** your field name has MultipleWords,
          - [ ] Update `testAgent_RegisterService_TranslateKeys` to include
            examples with it set in `snake_case` and ensure it is parsed
            correctly. Run this via `TestAgent_RegisterService_TranslateKeys`
            (agent_endpoint_test.go).
 - [ ] `api` package
      - [ ] Add the field to `AgentService` (`agent.go`)
      - [ ] Add/update an appropriate test in `agent_test.go`
        - (Note you need to use `make test` or ensure the `consul` binary on
          your `$PATH` is a build with your new field - usually `make dev`
          ensures this unless you're path is funky or you have a consul binary
          even further up the shell's `$PATH`).
 - [ ] Docs
      - [ ] Update docs in `website/source/docs/agent/services.html.md`
      - [ ] Consider if it's worth adding examples to feature docs or API docs
        that show the new field's usage.

Note that although the new field will show up in the API output of
`/agent/services` , `/catalog/services` and `/health/services`, those tests
right now don't exercise anything that's super useful unless custom logic is
required since they don't even encode the response object as JSON and just
assert on the structs you already modified. If custom presentation logic is
needed, tests for these endpoints might be warranted too. It's usual to use
`omit-empty` for new fields that will typically not be used by existing
registrations although we don't currently test for that systematically.

Adding a Consul Config Field  This is a checklist of all the places you need to update when adding a new field to config  There may be a few other special cases not included but this covers the majority of configs   We suggest you copy the raw markdown into a gist or local file and check them off as you go  you can mark them as done by replace       with   x   so github renders them as checked   Then   please include the completed lists you worked through in your PR description     Examples of special cases this doesn t cover are     If the config needs special treatment like a different default in   dev  mode    or differences between CE and Enterprise     If custom logic is needed to support backwards compatibility when changing    syntax or semantics of anything  There are four specific cases covered with increasing complexity   1  adding a simple config field only used by client agents  1  adding a CLI flag to mirror that config field  1  adding a config field that needs to be used in Consul servers  1  adding a field to the Service Definition     Adding a Simple Config Field for Client Agents         Add the field to the Config struct  or an appropriate sub struct  in     agent config config go          Add the field to the actual RuntimeConfig struct in     agent config runtime go          Add an appropriate parser setter in  agent config builder go  to    translate         Add the new field with a random value to both the JSON and HCL files in     agent config testdata full config     which should cause the test to fail     Then update the expected value in  TestLoad FullConfig  in     agent config runtime test go  to make the test pass again         Run  go test  run TestRuntimeConfig Sanitize   agent config  update  to update    the expected value for  TestRuntimeConfig Sanitize   Look at  git diff  to    make sure the value changed as you expect           If   your new config field needed some validation as it s only valid in    some cases or with some values  often true               Add validation to Validate in  agent config builder go               Add a test case to the table test  TestLoad IntegrationWithFlags  in          agent config runtime test go            If   your new config field needs a non zero value default              Add that to  DefaultSource  in  agent config defaults go               Add a test case to the table test  TestLoad IntegrationWithFlags  in          agent config runtime test go               If the config needs to be defaulted for the test server used in unit tests              also add it to  DefaultConfig    in  agent consul config go            If   your config should take effect on a reload HUP              Add necessary code to to trigger a safe  locked or atomic  update to         any state the feature needs changing  This needs to be added to one or         more of the following places              ReloadConfig  in  agent agent go  if it needs to affect the local            client state or another client agent component              ReloadConfig  in  agent consul client go  if it needs to affect            state for client agent s RPC client              Add a test to  agent agent test go  similar to others with prefix          TestAgent reloadConfig           Add documentation to  website content docs agent config config files mdx    Done  You can now use your new field in a client agent by accessing  s agent Config  FieldName     If you need a CLI flag  access to the variable in a Server context  or touched the Service Definition  make sure you continue on to follow the appropriate checklists below      Adding a CLI Flag Corresponding to the new Field If the config field also needs a CLI flag  then follow these steps          Do all of the steps in  Adding a Simple Config    Field For Client Agents   adding a simple config field for client agents          Add the new flag to  agent config flags go          Add a test case to TestParseFlags in  agent config flag test go          Add a test case  or extend one if appropriate  to the table test     TestLoad IntegrationWithFlags  in  agent config runtime test go  to ensure setting the    flag works         Add flag  as well as config file  documentation to     website source docs agent config config files mdx  and  website source docs agent config cli flags mdx       Adding a Simple Config Field for Servers Consul servers have a separate Config struct for reasons  Note that Consul server agents are actually also client agents  so in some cases config that is only destined for servers doesn t need to follow this checklist provided it s only needed during the bootstrapping of the server  which is done in code shared by both server and client components in  agent go    For example WAN Gossip configs are only valid on server agents but since WAN Gossip is setup in  agent go  they don t need to follow this checklist  The simplest  and mostly accurate  rule is     If you need to access the config field from code in   agent consul   e g  RPC   endpoints   then you need to follow this  If it s only in  agent   e g  HTTP   endpoints or agent startup  you don t   A final word of warning     you should never need to pass config into the FSM   agent consul fsm   or state store   agent consul state      Doing so is    very dangerous    and can violate consistency guarantees and corrupt databases  If you think you need this then please discuss the design with the Consul team before writing code   Consul s server components for historical reasons don t use the  RuntimeConfig  struct they have their own struct called  Config  in  agent consul config go           Do all of the steps in  Adding a Simple Config    Field For Client Agents   adding a simple config field for client agents          Add the new field to Config struct in  agent consul config go         Add code to set the values from the  RuntimeConfig  in  newConsulConfig  method in  agent agent go           If needed    add a test to  agent test go  if there is some non trivial    behavior in the code you added in the previous step  We tend not to test    simple assignments from one to the other since these are typically caught by    higher level tests of the actual functionality that matters but some examples    can be found prefixed with  TestAgent consulConfig            If   your config should take effect on a reload HUP             Add necessary code to  ReloadConfig  in  agent consul server go  this         needs to be adequately synchronized with any readers of the state being         updated               Add a new test or a new assertion to  TestServer ReloadConfig   You can now access that field from  s srv config  FieldName   inside an RPC handler      Adding a New Field to Service Definition The  Service Definition  https   www consul io docs agent services html  syntax appears both in Consul config files but also in the   v1 agent service register  API   For wonderful historical reasons  our config files have always used  snake case  attribute names in both JSON and HCL  even before we supported HCL    while our API uses  CamelCase    Because we want documentation examples to work in both config files and API bodies to avoid needless confusion  we have to accept both snake case and camel case field names for the service definition   Finally  adding a field to the service definition implies adding the field to several internal structs and to all API outputs that display services from the catalog  That explains the multiple layers needed below   This list assumes a new field in the base service definition struct  Adding new fields to health checks is similar but mostly needs  HealthCheck  structs and methods updating instead  Adding fields to embedded structs like  ProxyConfig  is largely the same pattern but may need different test methods etc  updating          Do all of the steps in  Adding a Simple Config    Field For Client Agents   adding a simple config field for client agents           agent structs  package             Add the field to  ServiceDefinition    service definition go               Add the field to  NodeService    structs go               Add the field to  ServiceNode    structs go               Update  ServiceDefinition ToNodeService  to translate the field             Update  NodeService ToServiceNode  to translate the field             Update  ServiceNode ToNodeService  to translate the field             Update  TestStructs ServiceNode Conversions              Update  ServiceNode PartialClone              Update  TestStructs ServiceNode PartialClone    structs test go               If needed  update  NodeService Validate  to ensure the field value is         reasonable             Add test like  TestStructs NodeService Validate   in          structs test go              Add comparison in  NodeService IsSame              Update  TestStructs NodeService IsSame              Add comparison in  ServiceNode IsSameService              Update  TestStructs ServiceNode IsSameService                If   your field name has MultipleWords                  Add it to the  aux  inline struct in              ServiceDefinition UnmarshalJSON    service defintion go                   Note  if the field is embedded higher up in a nested struct                follow the chain and update the necessary struct s  UnmarshalJSON                method   you may need to add one if there are no other case               transformations being done  copy and existing example                 Note  the tests that exercise this are in agent endpoint for               historical reasons  this is where the translation used to happen           agent  package             Update  testAgent RegisterService  and or add a new test to ensure         your fields register correctly via API   agent endpoint test go                 If   your field name has MultipleWords                  Update  testAgent RegisterService TranslateKeys  to include             examples with it set in  snake case  and ensure it is parsed             correctly  Run this via  TestAgent RegisterService TranslateKeys               agent endpoint test go           api  package             Add the field to  AgentService    agent go               Add update an appropriate test in  agent test go             Note you need to use  make test  or ensure the  consul  binary on           your   PATH  is a build with your new field   usually  make dev            ensures this unless you re path is funky or you have a consul binary           even further up the shell s   PATH           Docs             Update docs in  website source docs agent services html md              Consider if it s worth adding examples to feature docs or API docs         that show the new field s usage   Note that although the new field will show up in the API output of   agent services      catalog services  and   health services   those tests right now don t exercise anything that s super useful unless custom logic is required since they don t even encode the response object as JSON and just assert on the structs you already modified  If custom presentation logic is needed  tests for these endpoints might be warranted too  It s usual to use  omit empty  for new fields that will typically not be used by existing registrations although we don t currently test for that systematically


consul write requests from the RPC subsystem See the Consul Architecture Guide for an introduction to the Consul deployment architecture and the Consensus Protocol used by generic resource oriented storage layer introduced in Consul 1 16 Please see Note While the content of this document is still accurate it doesn t cover the new for more information Cluster Persistence The cluser persistence subsystem runs entirely in Server Agents It handles both read and

Answer:
# Cluster Persistence

> **Note**
> While the content of this document is still accurate, it doesn't cover the new
> generic resource-oriented storage layer introduced in Consul 1.16. Please see
> [Resources](../v2-architecture/controller-architecture) for more information.

The cluser persistence subsystem runs entirely in Server Agents. It handles both read and
write requests from the [RPC] subsystem. See the [Consul Architecture Guide] for an
introduction to the Consul deployment architecture and the [Consensus Protocol] used by
the cluster persistence subsystem.

[RPC]: ../rpc
[Consul Architecture Guide]: https://www.consul.io/docs/architecture
[Consensus Protocol]: https://www.consul.io/docs/architecture/consensus

![Overview](./overview.svg)

<sup>[source](./overview.mmd)</sup>

## Raft and FSM

[hashicorp/raft] is at the core of cluster persistence. Raft requires an [FSM], a
finite-state machine implementation, to persist state changes. The Consul FSM is
implemented in [agent/consul/fsm] as a set of commands.

[FSM]: https://pkg.go.dev/github.com/hashicorp/raft#FSM
[hashicorp/raft]: https://github.com/hashicorp/raft
[agent/consul/fsm]: https://github.com/hashicorp/consul/tree/main/agent/consul/fsm

Raft also requires a [LogStore] to persist logs to disk. Consul uses [hashicorp/raft-boltdb]
which implements [LogStore] using [boltdb]. In the near future we should be updating to
use [bbolt].

[LogStore]: https://pkg.go.dev/github.com/hashicorp/raft#LogStore
[hashicorp/raft-boltdb]: https://github.com/hashicorp/raft-boltdb
[boltdb]: https://github.com/boltdb/bolt
[bbolt]: https://github.com/etcd-io/bbolt

See [diagrams](#diagrams) below for more details on the interaction.

## State Store

Consul stores the full state of the cluster in memory using the state store. The state store is
implemented in [agent/consul/state] and uses [hashicorp/go-memdb] to maintain indexes of
data stored in a set of tables. The main entrypoint to the state store is [NewStateStore].

[agent/consul/state]: https://github.com/hashicorp/consul/tree/main/agent/consul/state
[hashicorp/go-memdb]: https://github.com/hashicorp/go-memdb
[NewStateStore]: https://github.com/hashicorp/consul/blob/main/agent/consul/state/state_store.go

### Tables, Schemas, and Indexes

The state store is organized as a set of tables, and each table has a set of indexes.
`newDBSchema` in [schema.go] shows the full list of tables, and each schema function shows
the full list of indexes.

[schema.go]: https://github.com/hashicorp/consul/blob/main/agent/consul/state/schema.go

There are two styles for defining table indexes. The original style uses generic indexer
implementations from [hashicorp/go-memdb] (ex: `StringFieldIndex`). These indexes use
[reflect] to find values for an index. These generic indexers work well when the index
value is a single value available directly from the struct field, and there are no
ce/enterprise differences.

The second style of indexers are custom indexers implemented using only functions and
based on the types defined in [indexer.go]. This style of index works well when the index
value is a value derived from one or multiple fields, or when there are ce/enterprise
differences between the indexes.

[reflect]: https://golang.org/pkg/reflect/
[indexer.go]: https://github.com/hashicorp/consul/blob/main/agent/consul/state/indexer.go

## Snapshot and Restore

Snapshots are the primary mechanism used to backup the data stored by cluster persistence.
If all Consul servers fail, a snapshot can be used to restore the cluster back
to its previous state.

Note that there are two different snapshot and restore concepts that exist at different
layers. First there is the `Snapshot` and `Restore` methods on the raft [FSM] interface,
that Consul must implement. These methods are implemented as mostly passthrough to the
state store. These methods may be called internally by raft to perform log compaction
(snapshot) or to bootstrap a new follower (restore). Consul implements snapshot and
restore using the `Snapshot` and `Restore` types in [agent/consul/state].

Snapshot and restore also exist as actions that a user may perform. There are [CLI]
commands, [HTTP API] endpoints, and [RPC] endpoints that allow a user to capture an
archive which contains a snapshot of the state, and restore that state to a running
cluster. The [consul/snapshot] package provides some of the logic for creating and reading
the snapshot archives for users. See [commands/snapshot] for a reference to these user
facing operations.

[CLI]: ../cli
[HTTP API]: ../http-api
[commands/snapshot]: https://www.consul.io/commands/snapshot
[consul/snapshot]: https://github.com/hashicorp/consul/tree/main/snapshot

Finally, there is also a [snapshot agent] (enterprise only) that uses the snapshot API
endpoints to periodically capture a snapshot, and optionally send it somewhere for
storage.

[snapshot agent]: https://www.consul.io/commands/snapshot/agent

## Raft Autopilot

[hashicorp/raft-autopilot] is used by Consul to automate some parts of the upgrade process.

[hashicorp/raft-autopilot]: https://github.com/hashicorp/raft-autopilot

## Diagrams
### High-level life of a write
![Overview](./write-overview.png)

### Deep-dive into write through Raft
![Deep dive](./write-deep-dive.png

Cluster Persistence      Note       While the content of this document is still accurate  it doesn t cover the new   generic resource oriented storage layer introduced in Consul 1 16  Please see    Resources     v2 architecture controller architecture  for more information   The cluser persistence subsystem runs entirely in Server Agents  It handles both read and write requests from the  RPC  subsystem  See the  Consul Architecture Guide  for an introduction to the Consul deployment architecture and the  Consensus Protocol  used by the cluster persistence subsystem    RPC      rpc  Consul Architecture Guide   https   www consul io docs architecture  Consensus Protocol   https   www consul io docs architecture consensus     Overview    overview svg    sup  source    overview mmd   sup       Raft and FSM   hashicorp raft  is at the core of cluster persistence  Raft requires an  FSM   a finite state machine implementation  to persist state changes  The Consul FSM is implemented in  agent consul fsm  as a set of commands    FSM   https   pkg go dev github com hashicorp raft FSM  hashicorp raft   https   github com hashicorp raft  agent consul fsm   https   github com hashicorp consul tree main agent consul fsm  Raft also requires a  LogStore  to persist logs to disk  Consul uses  hashicorp raft boltdb  which implements  LogStore  using  boltdb   In the near future we should be updating to use  bbolt      LogStore   https   pkg go dev github com hashicorp raft LogStore  hashicorp raft boltdb   https   github com hashicorp raft boltdb  boltdb   https   github com boltdb bolt  bbolt   https   github com etcd io bbolt  See  diagrams   diagrams  below for more details on the interaction      State Store  Consul stores the full state of the cluster in memory using the state store  The state store is implemented in  agent consul state  and uses  hashicorp go memdb  to maintain indexes of data stored in a set of tables  The main entrypoint to the state store is  NewStateStore     agent consul state   https   github com hashicorp consul tree main agent consul state  hashicorp go memdb   https   github com hashicorp go memdb  NewStateStore   https   github com hashicorp consul blob main agent consul state state store go      Tables  Schemas  and Indexes  The state store is organized as a set of tables  and each table has a set of indexes   newDBSchema  in  schema go  shows the full list of tables  and each schema function shows the full list of indexes    schema go   https   github com hashicorp consul blob main agent consul state schema go  There are two styles for defining table indexes  The original style uses generic indexer implementations from  hashicorp go memdb   ex   StringFieldIndex    These indexes use  reflect  to find values for an index  These generic indexers work well when the index value is a single value available directly from the struct field  and there are no ce enterprise differences   The second style of indexers are custom indexers implemented using only functions and based on the types defined in  indexer go   This style of index works well when the index value is a value derived from one or multiple fields  or when there are ce enterprise differences between the indexes    reflect   https   golang org pkg reflect   indexer go   https   github com hashicorp consul blob main agent consul state indexer go      Snapshot and Restore  Snapshots are the primary mechanism used to backup the data stored by cluster persistence  If all Consul servers fail  a snapshot can be used to restore the cluster back to its previous state   Note that there are two different snapshot and restore concepts that exist at different layers  First there is the  Snapshot  and  Restore  methods on the raft  FSM  interface  that Consul must implement  These methods are implemented as mostly passthrough to the state store  These methods may be called internally by raft to perform log compaction  snapshot  or to bootstrap a new follower  restore   Consul implements snapshot and restore using the  Snapshot  and  Restore  types in  agent consul state    Snapshot and restore also exist as actions that a user may perform  There are  CLI  commands   HTTP API  endpoints  and  RPC  endpoints that allow a user to capture an archive which contains a snapshot of the state  and restore that state to a running cluster  The  consul snapshot  package provides some of the logic for creating and reading the snapshot archives for users  See  commands snapshot  for a reference to these user facing operations    CLI      cli  HTTP API      http api  commands snapshot   https   www consul io commands snapshot  consul snapshot   https   github com hashicorp consul tree main snapshot  Finally  there is also a  snapshot agent   enterprise only  that uses the snapshot API endpoints to periodically capture a snapshot  and optionally send it somewhere for storage     snapshot agent   https   www consul io commands snapshot agent     Raft Autopilot   hashicorp raft autopilot  is used by Consul to automate some parts of the upgrade process     hashicorp raft autopilot   https   github com hashicorp raft autopilot     Diagrams     High level life of a write   Overview    write overview png       Deep dive into write through Raft   Deep dive    write deep dive png


consul At the time of writing only the service health endpoint uses streaming but more endpoints sends events as they occur and the client maintains a materialized view of the events Event Streaming will be added in the future Event streaming is a new asynchronous RPC mechanism that is being added to Consul Instead of synchronous blocking RPC calls long polling to fetch data when it changes streaming

Answer:
# Event Streaming

Event streaming is a new asynchronous RPC mechanism that is being added to Consul. Instead
of synchronous blocking RPC calls (long polling) to fetch data when it changes, streaming
sends events as they occur, and the client maintains a materialized view of the events.

At the time of writing only the service health endpoint uses streaming, but more endpoints
will be added in the future.

See [adding a topic](./adding-a-topic.md) for a guide on adding new topics to streaming.

## Overview

The diagram below shows the components that are used in streaming, and how they fit into
the rest of Consul.

![Streaming Overview](./overview.svg)

[source](./overview.mmd)

Read requests are received either from the HTTP API or from a DNS request. They use
[rpcclient/health.Health]
to query the cache. The [StreamingHealthServices cache-type] uses a [materialized view]
to manage subscriptions and store the aggregated events. On the server, the
[SubscribeEndpoint] subscribes and receives events from [EventPublisher].

Writes will likely enter the system through the client as well, but to make the diagram
less complicated the write flow starts when it is received by the RPC endpoint. The
endpoint calls raft.Apply, which if successful will save the new data in the state.Store.
When the [state.Store commits] it produces an event which is managed by the [EventPublisher]
and sent to any active subscriptions.

[rpcclient/health.Health]: https://github.com/hashicorp/consul/blob/main/agent/rpcclient/health/health.go
[StreamingHealthServices cache-type]: https://github.com/hashicorp/consul/blob/main/agent/cache-types/streaming_health_services.go
[materialized view]: https://github.com/hashicorp/consul/blob/main/agent/submatview/materializer.go
[SubscribeEndpoint]: https://github.com/hashicorp/consul/blob/main/agent/grpc-internal/services/subscribe/subscribe.go
[EventPublisher]: https://github.com/hashicorp/consul/blob/main/agent/consul/stream/event_publisher.go
[state.Store commits]: https://github.com/hashicorp/consul/blob/main/agent/consul/state/memdb.go

## Event Publisher

The [EventPublisher] is at the core of streaming. It receives published events, and
subscription requests, and forwards events to the appropriate subscriptions. The diagram
below illustrates how events are stored by the [EventPublisher].

![Event Publisher layout](./event-publisher-layout.svg)

[source](./event-publisher-layout.mmd)

When a new subscription is created it will create a snapshot of the events required to
reflect the current state. This snapshot is cached by the [EventPublisher] so that other
subscriptions can re-use the snapshot without having to recreate it.

The snapshot always points at the first item in the linked list of events. A subscription
will initially point at the first item, but the pointer advances each time
`Subscribe.Next` is called. The topic buffers in the EventPublisher always point at the
latest item in the linked list, so that new events can be appended to the buffer.

When a snapshot cache TTL expires, the snapshot is removed. If there are no other
subscriptions holding a reference to those items, the items will be garbage collected by
the Go runtime. This setup allows EventPublisher to keep some events around for a short
period of time, without any hard coded limit on the number of events to cache.

## Subscription events

A subscription provides a stream of events on a single topic. Most of the events contain
data for a change in state, but there are a few special "framing" events that are used to
communicate something to the client. The diagram below helps illustrate the logic in
`EventPublisher.Subscribe` and the [materialized view].

![Framing events](./framing-events.svg)

[source](./framing-events.mmd)

Events in the `Snapshot` contain the same data as those in the `EventStream`, the only
difference is that events in the `Snapshot` indicate the current state not a change in
state.

`NewSnapshotToFollow` is a framing event that indicates to the client that their existing
view is out of date. They must reset their view and prepare to receive a new snapshot.

`EndOfSnapshot` indicates to the client that the snapshot is complete. Any future events
will be changes in state.

## Event filtering

As events pass through the system from the `state.Store` to the client they are grouped
and filtered along the way. The diagram below helps illustrate where each of the grouping
and filtering happens.

![event filtering](./event-filtering.svg)

[source](./event-filtering.mmd)

Event Streaming  Event streaming is a new asynchronous RPC mechanism that is being added to Consul  Instead of synchronous blocking RPC calls  long polling  to fetch data when it changes  streaming sends events as they occur  and the client maintains a materialized view of the events   At the time of writing only the service health endpoint uses streaming  but more endpoints will be added in the future   See  adding a topic    adding a topic md  for a guide on adding new topics to streaming      Overview  The diagram below shows the components that are used in streaming  and how they fit into the rest of Consul     Streaming Overview    overview svg    sup  source    overview mmd   sup   Read requests are received either from the HTTP API or from a DNS request  They use  rpcclient health Health  to query the cache  The  StreamingHealthServices cache type  uses a  materialized view  to manage subscriptions and store the aggregated events  On the server  the  SubscribeEndpoint  subscribes and receives events from  EventPublisher    Writes will likely enter the system through the client as well  but to make the diagram less complicated the write flow starts when it is received by the RPC endpoint  The endpoint calls raft Apply  which if successful will save the new data in the state Store  When the  state Store commits  it produces an event which is managed by the  EventPublisher  and sent to any active subscriptions    rpcclient health Health   https   github com hashicorp consul blob main agent rpcclient health health go  StreamingHealthServices cache type   https   github com hashicorp consul blob main agent cache types streaming health services go  materialized view   https   github com hashicorp consul blob main agent submatview materializer go  SubscribeEndpoint   https   github com hashicorp consul blob main agent grpc internal services subscribe subscribe go  EventPublisher   https   github com hashicorp consul blob main agent consul stream event publisher go  state Store commits   https   github com hashicorp consul blob main agent consul state memdb go      Event Publisher  The  EventPublisher  is at the core of streaming  It receives published events  and subscription requests  and forwards events to the appropriate subscriptions  The diagram below illustrates how events are stored by the  EventPublisher      Event Publisher layout    event publisher layout svg    sup  source    event publisher layout mmd   sup   When a new subscription is created it will create a snapshot of the events required to reflect the current state  This snapshot is cached by the  EventPublisher  so that other subscriptions can re use the snapshot without having to recreate it   The snapshot always points at the first item in the linked list of events  A subscription will initially point at the first item  but the pointer advances each time  Subscribe Next  is called  The topic buffers in the EventPublisher always point at the latest item in the linked list  so that new events can be appended to the buffer   When a snapshot cache TTL expires  the snapshot is removed  If there are no other subscriptions holding a reference to those items  the items will be garbage collected by the Go runtime  This setup allows EventPublisher to keep some events around for a short period of time  without any hard coded limit on the number of events to cache       Subscription events  A subscription provides a stream of events on a single topic  Most of the events contain data for a change in state  but there are a few special  framing  events that are used to communicate something to the client  The diagram below helps illustrate the logic in  EventPublisher Subscribe  and the  materialized view       Framing events    framing events svg    sup  source    framing events mmd   sup    Events in the  Snapshot  contain the same data as those in the  EventStream   the only difference is that events in the  Snapshot  indicate the current state not a change in state    NewSnapshotToFollow  is a framing event that indicates to the client that their existing view is out of date  They must reset their view and prepare to receive a new snapshot    EndOfSnapshot  indicates to the client that the snapshot is complete  Any future events will be changes in state       Event filtering  As events pass through the system from the  state Store  to the client they are grouped and filtered along the way  The diagram below helps illustrate where each of the grouping and filtering happens      event filtering    event filtering svg    sup  source    event filtering mmd   sup


consul Certificate Authority Connect CA The code for the Certificate Authority is in the following packages 2 the providers are in agent connect ca 3 the RPC interface is in agent consul connectcaendpoint go 1 most of the core logic is in agent consul leaderconnectca go services and client agents via auto encrypt and auto config The Certificate Authority Subsystem manages a CA trust chain for issuing certificates to

Answer:
# Certificate Authority (Connect CA)

The Certificate Authority Subsystem manages a CA trust chain for issuing certificates to
services and client agents (via auto-encrypt and auto-config).

The code for the Certificate Authority is in the following packages:
1. most of the core logic is in [agent/consul/leader_connect_ca.go]
2. the providers are in [agent/connect/ca]
3. the RPC interface is in [agent/consul/connect_ca_endpoint.go]

[agent/consul/leader_connect_ca.go]: https://github.com/hashicorp/consul/blob/main/agent/consul/leader_connect_ca.go
[agent/connect/ca]: https://github.com/hashicorp/consul/blob/main/agent/connect/ca/
[agent/consul/connect_ca_endpoint.go]: https://github.com/hashicorp/consul/blob/main/agent/consul/connect_ca_endpoint.go

## Architecture

### High level overview

In Consul the leader is responsible for handling the CA management.
When a leader election happen, and the elected leader do not have any root CA available it will start a process of creating a set of CA certificate.
Those certificates will be used to authenticate/encrypt communication between services (service mesh) or between `Consul client agent` (auto-encrypt/auto-config). This process is described in the following diagram:

![CA creation](./hl-ca-overview.svg)

[source](./hl-ca-overview.mmd)

The features that benefit from Consul CA management are:
- [service Mesh/Connect](https://www.consul.io/docs/connect)
- [auto encrypt](https://www.consul.io/docs/agent/options#auto_encrypt)

### CA and Certificate relationship

This diagram shows the relationship between the CA certificates in Consul primary and
secondary.

![CA relationship](./cert-relationship.svg)

[source](./cert-relationship.mmd)

In most cases there is an external root CA that provides an intermediate CA that Consul
uses as the Primary Root CA. The only except to this is when the Consul CA Provider is
used without specifying a `RootCert`. In this one case Consul will generate the Root CA
from the provided primary key, and it will be used in the primary as the top of the chain
of trust.

In the primary datacenter, the Consul and AWS providers use the Primary Root CA to sign
leaf certificates. The Vault provider uses an intermediate CA to sign leaf certificates.

Leaf certificates are created for two purposes:
1. the Leaf Cert Service is used by envoy proxies in the mesh to perform mTLS with other
   services.
2. the Leaf Cert Client Agent is created by auto-encrypt and auto-config. It is used by
   client agents for HTTP API TLS, and for mTLS for RPC requests to servers.

Any secondary datacenters receive an intermediate certificate, signed by the Primary Root
CA, which is used as the CA certificate to sign leaf certificates in the secondary
datacenter.

## Operations

When trying to learn the CA subsystem it can be helpful to understand the operations that
it can perform. The sections below are the complete set of read, write, and periodic
operations that provide the full behaviour of the CA subsystem.

### Periodic Operations

Periodic (or background) opeartions are started automatically by the Consul leader. They run at some interval (often 1 hour).

- `CAManager.InitializeCA` - attempts to initialize the CA when a leader is ellected. If the synchronous InitializeCA fails, `CAManager.backgroundCAInitialization` runs `InitializeCA` periodically in a goroutine until it succeeds.
- `CAManager.RenewIntermediate` - (called by `CAManager.intermediateCertRenewalWatch`) runs in the primary if the provider uses a separate signing cert (the Vault provider). The operation always runs in the secondary. Renews the signing cert once half its lifetime has passed.
- `CAManager.secondaryCARootWatch` - runs in secondary only. Performs a blocking query to the primary to retrieve any updates to the CA roots and stores them locally.
- `Server.runCARootPruning` - removes non-active and expired roots from state.CARoots

### Read Operations

- `RPC.ConnectCA.ConfigurationGet` - returns the CA provider configuration. Only called by user, not by any internal subsystems.
- `RPC.ConnectCA.Roots` - returns all the roots, the trust domain ID, and the ID of the active root. Each "root" also includes the signing key/cert, and any intermediate certs in the chain. It is used (via the cache) by all the connect proxy types.

### Write Operations

- `CAManager.UpdateConfiguration` - (via `RPC.ConnectCA.ConfigurationSet`) called by a user when they want to change the provider or provider configuration (ex: rotate root CA).
- `CAManager.Provider.SignIntermediate` - (via `RPC.ConnectCA.SignIntermediate`) called from the secondary DC:
    1. by `CAManager.RenewIntermediate` to sign the new intermediate when the old intermediate is about to expire
    2. by `CAMananger.initializeSecondary` when setting up a new secondary, when the provider is changed in the secondary
   by a user action, or when the primary roots changed and the secondary needs to generate a new intermediate for the new
   primary roots.
- `CAMananger.SignCertificate` - is used by:
    1. (via `RPC.ConnectCA.Sign`) - called by client agents to sign a leaf cert for a connect proxy (via `agent/cache-types/connect_ca_leaf.go`)
    2. (via in-process call to `RPC.ConnectCA.Sign`) - called by auto-encrypt to sign a leaf cert for a client agent
    3. called by Auto-Config to sign a leaf cert for a client agent

## detailed call flow
![CA Leader Sequence](./ca-leader-sequence.svg)

[source](./ca-leader-sequence.mmd)

####TODO:
- sequence diagram for leaf signing
- sequence diagram for CA cert rotation

## CAManager states

This section is a work in progress

TODO: style the diagram to match the others, and add some narative text to describe the
diagram.

![CA Mananger states](./state-machine.svg)

Certificate Authority  Connect CA   The Certificate Authority Subsystem manages a CA trust chain for issuing certificates to services and client agents  via auto encrypt and auto config    The code for the Certificate Authority is in the following packages  1  most of the core logic is in  agent consul leader connect ca go  2  the providers are in  agent connect ca  3  the RPC interface is in  agent consul connect ca endpoint go     agent consul leader connect ca go   https   github com hashicorp consul blob main agent consul leader connect ca go  agent connect ca   https   github com hashicorp consul blob main agent connect ca   agent consul connect ca endpoint go   https   github com hashicorp consul blob main agent consul connect ca endpoint go      Architecture      High level overview  In Consul the leader is responsible for handling the CA management   When a leader election happen  and the elected leader do not have any root CA available it will start a process of creating a set of CA certificate  Those certificates will be used to authenticate encrypt communication between services  service mesh  or between  Consul client agent   auto encrypt auto config   This process is described in the following diagram     CA creation    hl ca overview svg    sup  source    hl ca overview mmd   sup   The features that benefit from Consul CA management are     service Mesh Connect  https   www consul io docs connect     auto encrypt  https   www consul io docs agent options auto encrypt        CA and Certificate relationship  This diagram shows the relationship between the CA certificates in Consul primary and secondary     CA relationship    cert relationship svg    sup  source    cert relationship mmd   sup    In most cases there is an external root CA that provides an intermediate CA that Consul uses as the Primary Root CA  The only except to this is when the Consul CA Provider is used without specifying a  RootCert   In this one case Consul will generate the Root CA from the provided primary key  and it will be used in the primary as the top of the chain of trust   In the primary datacenter  the Consul and AWS providers use the Primary Root CA to sign leaf certificates  The Vault provider uses an intermediate CA to sign leaf certificates   Leaf certificates are created for two purposes  1  the Leaf Cert Service is used by envoy proxies in the mesh to perform mTLS with other    services  2  the Leaf Cert Client Agent is created by auto encrypt and auto config  It is used by    client agents for HTTP API TLS  and for mTLS for RPC requests to servers   Any secondary datacenters receive an intermediate certificate  signed by the Primary Root CA  which is used as the CA certificate to sign leaf certificates in the secondary datacenter      Operations  When trying to learn the CA subsystem it can be helpful to understand the operations that it can perform  The sections below are the complete set of read  write  and periodic operations that provide the full behaviour of the CA subsystem       Periodic Operations  Periodic  or background  opeartions are started automatically by the Consul leader  They run at some interval  often 1 hour       CAManager InitializeCA    attempts to initialize the CA when a leader is ellected  If the synchronous InitializeCA fails   CAManager backgroundCAInitialization  runs  InitializeCA  periodically in a goroutine until it succeeds     CAManager RenewIntermediate     called by  CAManager intermediateCertRenewalWatch   runs in the primary if the provider uses a separate signing cert  the Vault provider   The operation always runs in the secondary  Renews the signing cert once half its lifetime has passed     CAManager secondaryCARootWatch    runs in secondary only  Performs a blocking query to the primary to retrieve any updates to the CA roots and stores them locally     Server runCARootPruning    removes non active and expired roots from state CARoots      Read Operations     RPC ConnectCA ConfigurationGet    returns the CA provider configuration  Only called by user  not by any internal subsystems     RPC ConnectCA Roots    returns all the roots  the trust domain ID  and the ID of the active root  Each  root  also includes the signing key cert  and any intermediate certs in the chain  It is used  via the cache  by all the connect proxy types       Write Operations     CAManager UpdateConfiguration     via  RPC ConnectCA ConfigurationSet   called by a user when they want to change the provider or provider configuration  ex  rotate root CA      CAManager Provider SignIntermediate     via  RPC ConnectCA SignIntermediate   called from the secondary DC      1  by  CAManager RenewIntermediate  to sign the new intermediate when the old intermediate is about to expire     2  by  CAMananger initializeSecondary  when setting up a new secondary  when the provider is changed in the secondary    by a user action  or when the primary roots changed and the secondary needs to generate a new intermediate for the new    primary roots     CAMananger SignCertificate    is used by      1   via  RPC ConnectCA Sign     called by client agents to sign a leaf cert for a connect proxy  via  agent cache types connect ca leaf go       2   via in process call to  RPC ConnectCA Sign     called by auto encrypt to sign a leaf cert for a client agent     3  called by Auto Config to sign a leaf cert for a client agent     detailed call flow   CA Leader Sequence    ca leader sequence svg    sup  source    ca leader sequence mmd   sup       TODO    sequence diagram for leaf signing    sequence diagram for CA cert rotation     CAManager states  This section is a work in progress  TODO  style the diagram to match the others  and add some narative text to describe the diagram     CA Mananger states    state machine svg