vault Plugin management External Plugins are mountable backends that are implemented using Vault s page title Plugin Management plugin system layout docs

Answer:
---
layout: docs
page_title: Plugin Management
description: >-
  External Plugins are mountable backends that are implemented using Vault's
  plugin system.
---

# Plugin management

External plugins are the components in Vault that can be implemented separately
from Vault's built-in plugins. These plugins can be either authentication or
secrets engines.

The [`api_addr`][api_addr] must be set in order for the plugin process to
establish communication with the Vault server during mount time. If the storage
backend has HA enabled and supports automatic host address detection (e.g.
Consul), Vault will automatically attempt to determine the `api_addr` as well.

Detailed information regarding the plugin system can be found in the
[internals documentation](/vault/docs/plugins).

## Registering external plugins

Before an external plugin can be mounted, it needs to be
[registered](/vault/docs/plugins/plugin-architecture#plugin-registration) in the
plugin catalog to ensure the plugin invoked by Vault is authentic and maintains
integrity:

```shell-session
$ vault plugin register -sha256=<SHA256 Hex value of the plugin binary> \
    secret \                  # type
    passthrough-plugin

Success! Registered plugin: passthrough-plugin
```

## Enabling/Disabling external plugins

After the plugin is registered, it can be mounted by specifying the registered
plugin name:

```shell-session
$ vault secrets enable -path=my-secrets passthrough-plugin
Success! Enabled the passthrough-plugin secrets engine at: my-secrets/
```

Listing secrets engines will display secrets engines that are mounted as
plugins:

```shell-session
$ vault secrets list
Path         Type       Accessor            Plugin              Default TTL  Max TTL  Force No Cache  Replication Behavior  Description
my-secrets/  plugin     plugin_deb84140     passthrough-plugin  system       system   false           replicated
```

Disabling an external plugins is identical to disabling a built-in plugin:

```shell-session
$ vault secrets disable my-secrets
```

## Upgrading plugins

Upgrade instructions can be found in the [Upgrading Plugins - Guides][upgrading_plugins]
page.

[api_addr]: /vault/docs/configuration#api_addr
[upgrading_plugins]: /vault/docs/upgrading/plugins

## Plugin environment variables

An advantage for external plugins over builtin plugins is they can specify
additional environment variables because they are run in their own process.

-> Vault 1.16.0 changed the precedence given to plugin-specific environment
variables so they take priority over Vault's environment. See full details in
the [upgrade notes](/vault/docs/upgrading/upgrade-to-1.16.x).

Use the `-env` flag once per environment variable that a plugin should be
started with:

```shell-session
$ vault plugin register -sha256=<SHA256 Hex value of the plugin binary> \
    -env REGION=eu \
    -env TOKEN_FILE=/var/run/token \
    secret \                  # type
    passthrough-plugin

Success! Registered plugin: passthrough-plugin
```

### Plugin-specific HTTP proxy settings

Many tools and libraries automatically consume `HTTP_PROXY`, `HTTPS_PROXY`, and
`NO_PROXY` environment variables to configure HTTP proxy settings, including the
Go standard library's default HTTP client. You can use these environment
variables to configure different network proxies for different plugins:

-> You must be using an external plugin to take advantage of custom environment
variables. If you are using a builtin plugin, you can still download and register
an external version of it in order to use this workflow. Check the
[releases](https://releases.hashicorp.com/) page for the latest prebuilt plugin
binaries.

```shell-session
$ vault plugin register -sha256=<SHA256 Hex value of the plugin binary> \
    -env HTTP_PROXY=eu.example.com \
    auth \
    jwt-eu

Success! Registered plugin: jwt-eu

$ vault plugin register -sha256=<SHA256 Hex value of the plugin binary> \
    -env HTTP_PROXY=us.example.com \
    auth \
    jwt-us

Success! Registered plugin: jwt-us
```

You can then enable each plugin on its own path, and configure clients that
should be associated with one or the other appropriately:

```shell-session
$ vault auth enable jwt-eu
Success! Enabled the jwt-eu auth method at: auth/jwt-eu/

$ vault auth enable jwt-us
Success! Enabled the jwt-us auth method at: auth/jwt-us/
```

layout  docs page title  Plugin Management description       External Plugins are mountable backends that are implemented using Vault s   plugin system         Plugin management  External plugins are the components in Vault that can be implemented separately from Vault s built in plugins  These plugins can be either authentication or secrets engines   The   api addr   api addr  must be set in order for the plugin process to establish communication with the Vault server during mount time  If the storage backend has HA enabled and supports automatic host address detection  e g  Consul   Vault will automatically attempt to determine the  api addr  as well   Detailed information regarding the plugin system can be found in the  internals documentation   vault docs plugins       Registering external plugins  Before an external plugin can be mounted  it needs to be  registered   vault docs plugins plugin architecture plugin registration  in the plugin catalog to ensure the plugin invoked by Vault is authentic and maintains integrity      shell session   vault plugin register  sha256  SHA256 Hex value of the plugin binary        secret                      type     passthrough plugin  Success  Registered plugin  passthrough plugin         Enabling Disabling external plugins  After the plugin is registered  it can be mounted by specifying the registered plugin name      shell session   vault secrets enable  path my secrets passthrough plugin Success  Enabled the passthrough plugin secrets engine at  my secrets       Listing secrets engines will display secrets engines that are mounted as plugins      shell session   vault secrets list Path         Type       Accessor            Plugin              Default TTL  Max TTL  Force No Cache  Replication Behavior  Description my secrets   plugin     plugin deb84140     passthrough plugin  system       system   false           replicated      Disabling an external plugins is identical to disabling a built in plugin      shell session   vault secrets disable my secrets         Upgrading plugins  Upgrade instructions can be found in the  Upgrading Plugins   Guides  upgrading plugins  page    api addr    vault docs configuration api addr  upgrading plugins    vault docs upgrading plugins     Plugin environment variables  An advantage for external plugins over builtin plugins is they can specify additional environment variables because they are run in their own process      Vault 1 16 0 changed the precedence given to plugin specific environment variables so they take priority over Vault s environment  See full details in the  upgrade notes   vault docs upgrading upgrade to 1 16 x    Use the   env  flag once per environment variable that a plugin should be started with      shell session   vault plugin register  sha256  SHA256 Hex value of the plugin binary         env REGION eu        env TOKEN FILE  var run token       secret                      type     passthrough plugin  Success  Registered plugin  passthrough plugin          Plugin specific HTTP proxy settings  Many tools and libraries automatically consume  HTTP PROXY    HTTPS PROXY   and  NO PROXY  environment variables to configure HTTP proxy settings  including the Go standard library s default HTTP client  You can use these environment variables to configure different network proxies for different plugins      You must be using an external plugin to take advantage of custom environment variables  If you are using a builtin plugin  you can still download and register an external version of it in order to use this workflow  Check the  releases  https   releases hashicorp com   page for the latest prebuilt plugin binaries      shell session   vault plugin register  sha256  SHA256 Hex value of the plugin binary         env HTTP PROXY eu example com       auth       jwt eu  Success  Registered plugin  jwt eu    vault plugin register  sha256  SHA256 Hex value of the plugin binary         env HTTP PROXY us example com       auth       jwt us  Success  Registered plugin  jwt us      You can then enable each plugin on its own path  and configure clients that should be associated with one or the other appropriately      shell session   vault auth enable jwt eu Success  Enabled the jwt eu auth method at  auth jwt eu     vault auth enable jwt us Success  Enabled the jwt us auth method at  auth jwt us


vault Advanced topic Plugin development is a highly advanced topic in Vault and is not required knowledge for day to day usage If you don t plan on writing any Learn about Vault plugin development page title Plugin Development layout docs Plugin development

Answer:
---
layout: docs
page_title: Plugin Development
description: Learn about Vault plugin development.
---

# Plugin development

~> Advanced topic! Plugin development is a highly advanced topic in Vault, and
is not required knowledge for day-to-day usage. If you don't plan on writing any
plugins, we recommend not reading this section of the documentation.

Because Vault communicates to plugins over a RPC interface, you can build and
distribute a plugin for Vault without having to rebuild Vault itself. This makes
it easy for you to build a Vault plugin for your organization's internal use,
for a proprietary API that you don't want to open source, or to prototype
something before contributing it back to the main project.

In theory, because the plugin interface is HTTP, you could even develop a plugin
using a completely different programming language! (Disclaimer, you would also
have to re-implement the plugin API which is not a trivial amount of work.)

Developing a plugin is simple. The only knowledge necessary to write
a plugin is basic command-line skills and basic knowledge of the
[Go programming language](http://golang.org).

Your plugin implementation needs to satisfy the interface for the plugin
type you want to build. You can find these definitions in the docs for the
backend running the plugin.

~> Note: Plugins should be prepared to handle multiple concurrent requests
from Vault.

## Serving a plugin

### Serving a plugin with multiplexing

~> Plugin multiplexing requires `github.com/hashicorp/vault/sdk v0.5.4` or above.

The following code exhibits an example main package for a Vault plugin using
the Vault SDK for a secrets engine or auth method:

```go
package main

import (
	"os"

	myPlugin "your/plugin/import/path"
	"github.com/hashicorp/vault/api"
	"github.com/hashicorp/vault/sdk/plugin"
)

func main() {
	apiClientMeta := &api.PluginAPIClientMeta{}
	flags := apiClientMeta.FlagSet()
	flags.Parse(os.Args[1:])

	tlsConfig := apiClientMeta.GetTLSConfig()
	tlsProviderFunc := api.VaultPluginTLSProvider(tlsConfig)

	err := plugin.ServeMultiplex(&plugin.ServeOpts{
		BackendFactoryFunc: myPlugin.Factory,
		TLSProviderFunc:    tlsProviderFunc,
	})
	if err != nil {
		logger := hclog.New(&hclog.LoggerOptions{})

		logger.Error("plugin shutting down", "error", err)
		os.Exit(1)
	}
}
```

And that's basically it! You would just need to change `myPlugin` to your actual
plugin.

### Plugin backwards compatibility with Vault

Let's take a closer look at a snippet from the above main package.

```go
	err := plugin.ServeMultiplex(&plugin.ServeOpts{
		BackendFactoryFunc: myPlugin.Factory,
		TLSProviderFunc:    tlsProviderFunc,
	})
```

The call to `plugin.ServeMultiplex` ensures that the plugin will use
Vault's [plugin
multiplexing](/vault/docs/plugins/plugin-architecture#plugin-multiplexing) feature.
However, this plugin will not be multiplexed if it is run by a version of Vault
that does not support multiplexing. Vault will simply fall back to a plugin
version that it can run. Additionally, we set the `TLSProviderFunc` to ensure
that our plugin is backwards compatible with versions of Vault that do not
support automatic mutual TLS for secure [plugin
communication](/vault/docs/plugins/plugin-architecture#plugin-communication). If you
are certain your plugin does not need backwards compatibility, this field can
be omitted.

[api_addr]: /vault/docs/configuration#api_addr

## Leveraging plugin versioning

@include 'plugin-versioning.mdx'

Auth and secrets plugins based on `framework.Backend` from the SDK should set the
[`RunningVersion`](https://github.com/hashicorp/vault/blob/sdk/v0.6.0/sdk/framework/backend.go#L95-L96)
variable, and the framework will implement the version interface.

Database plugins have a smaller API than `framework.Backend` exposes, and should
instead implement the
[`PluginVersioner`](https://github.com/hashicorp/vault/blob/sdk/v0.6.0/sdk/logical/logical.go#L150-L154)
interface directly.

## Plugin logging

Auth and secrets plugins based on `framework.Backend` from the SDK can take
advantage of the SDK's [default logger](https://github.com/hashicorp/vault/blob/fe55cbbf05586ec4c0cd9bdf865ec6f741a8933c/sdk/framework/backend.go#L437).
No additional setup is required. The logger can be used like the following:

```go
func (b *backend) example() {
    b.Logger().Trace("Trace level log")
    b.Logger().Debug("Debug level log")
    b.Logger().Info("Info level log")
    b.Logger().Warn("Warn level log")
    b.Logger().Error("Error level log")
}
```

See the source code of [vault-auth-plugin-example](https://github.com/hashicorp/vault-auth-plugin-example)
for a more complete example of a plugin using logging.

## Building a plugin from source

To build a plugin from source, first navigate to the location holding the
desired plugin version. Next, run `go build` to obtain a new binary for the
plugin. Finally,
[register](/vault/docs/plugins/plugin-architecture#plugin-registration) the
plugin and enable it.

## Plugin development - resources

For more information on how to register and enable your plugin, refer to the
[Building Plugin Backends](/vault/tutorials/app-integration/plugin-backends)
tutorial.

Other HashiCorp plugin development resources:

* [vault-auth-plugin-example](https://github.com/hashicorp/vault-auth-plugin-example)
* [Custom Secrets Engines](/vault/tutorials/custom-secrets-engine)

### Plugin development - resources - community

See the [Vault Integrations](/vault/integrations) page to find Community
plugin examples/guides developed by community members. HashiCorp does not
validate these for correctness.

layout  docs page title  Plugin Development description  Learn about Vault plugin development         Plugin development     Advanced topic  Plugin development is a highly advanced topic in Vault  and is not required knowledge for day to day usage  If you don t plan on writing any plugins  we recommend not reading this section of the documentation   Because Vault communicates to plugins over a RPC interface  you can build and distribute a plugin for Vault without having to rebuild Vault itself  This makes it easy for you to build a Vault plugin for your organization s internal use  for a proprietary API that you don t want to open source  or to prototype something before contributing it back to the main project   In theory  because the plugin interface is HTTP  you could even develop a plugin using a completely different programming language   Disclaimer  you would also have to re implement the plugin API which is not a trivial amount of work    Developing a plugin is simple  The only knowledge necessary to write a plugin is basic command line skills and basic knowledge of the  Go programming language  http   golang org    Your plugin implementation needs to satisfy the interface for the plugin type you want to build  You can find these definitions in the docs for the backend running the plugin      Note  Plugins should be prepared to handle multiple concurrent requests from Vault      Serving a plugin      Serving a plugin with multiplexing     Plugin multiplexing requires  github com hashicorp vault sdk v0 5 4  or above   The following code exhibits an example main package for a Vault plugin using the Vault SDK for a secrets engine or auth method      go package main  import     os    myPlugin  your plugin import path    github com hashicorp vault api    github com hashicorp vault sdk plugin     func main      apiClientMeta     api PluginAPIClientMeta    flags    apiClientMeta FlagSet    flags Parse os Args 1      tlsConfig    apiClientMeta GetTLSConfig    tlsProviderFunc    api VaultPluginTLSProvider tlsConfig    err    plugin ServeMultiplex  plugin ServeOpts    BackendFactoryFunc  myPlugin Factory    TLSProviderFunc     tlsProviderFunc       if err    nil     logger    hclog New  hclog LoggerOptions       logger Error  plugin shutting down    error   err    os Exit 1            And that s basically it  You would just need to change  myPlugin  to your actual plugin       Plugin backwards compatibility with Vault  Let s take a closer look at a snippet from the above main package      go  err    plugin ServeMultiplex  plugin ServeOpts    BackendFactoryFunc  myPlugin Factory    TLSProviderFunc     tlsProviderFunc           The call to  plugin ServeMultiplex  ensures that the plugin will use Vault s  plugin multiplexing   vault docs plugins plugin architecture plugin multiplexing  feature  However  this plugin will not be multiplexed if it is run by a version of Vault that does not support multiplexing  Vault will simply fall back to a plugin version that it can run  Additionally  we set the  TLSProviderFunc  to ensure that our plugin is backwards compatible with versions of Vault that do not support automatic mutual TLS for secure  plugin communication   vault docs plugins plugin architecture plugin communication   If you are certain your plugin does not need backwards compatibility  this field can be omitted    api addr    vault docs configuration api addr     Leveraging plugin versioning   include  plugin versioning mdx   Auth and secrets plugins based on  framework Backend  from the SDK should set the   RunningVersion   https   github com hashicorp vault blob sdk v0 6 0 sdk framework backend go L95 L96  variable  and the framework will implement the version interface   Database plugins have a smaller API than  framework Backend  exposes  and should instead implement the   PluginVersioner   https   github com hashicorp vault blob sdk v0 6 0 sdk logical logical go L150 L154  interface directly      Plugin logging  Auth and secrets plugins based on  framework Backend  from the SDK can take advantage of the SDK s  default logger  https   github com hashicorp vault blob fe55cbbf05586ec4c0cd9bdf865ec6f741a8933c sdk framework backend go L437   No additional setup is required  The logger can be used like the following      go func  b  backend  example         b Logger   Trace  Trace level log       b Logger   Debug  Debug level log       b Logger   Info  Info level log       b Logger   Warn  Warn level log       b Logger   Error  Error level log          See the source code of  vault auth plugin example  https   github com hashicorp vault auth plugin example  for a more complete example of a plugin using logging      Building a plugin from source  To build a plugin from source  first navigate to the location holding the desired plugin version  Next  run  go build  to obtain a new binary for the plugin  Finally   register   vault docs plugins plugin architecture plugin registration  the plugin and enable it      Plugin development   resources  For more information on how to register and enable your plugin  refer to the  Building Plugin Backends   vault tutorials app integration plugin backends  tutorial   Other HashiCorp plugin development resources      vault auth plugin example  https   github com hashicorp vault auth plugin example     Custom Secrets Engines   vault tutorials custom secrets engine       Plugin development   resources   community  See the  Vault Integrations   vault integrations  page to find Community plugin examples guides developed by community members  HashiCorp does not validate these for correctness


vault page title Add a containerized secrets plugin Add a containerized secrets plugin to Vault layout docs Run your external secrets plugins in containers to increases the isolation Add a containerized secrets plugin to your Vault instance

Answer:
---
layout: docs
page_title: Add a containerized secrets plugin
description: >-
  Add a containerized secrets plugin to your Vault instance.
---

# Add a containerized secrets plugin to Vault

Run your external secrets plugins in containers to increases the isolation
between the plugin and Vault.

## Before you start

- **Your Vault instance must be running on Linux**.
- **Your Vault instance must have local access to the Docker Engine API**.
  Vault uses the [Docker SDK](https://pkg.go.dev/github.com/docker/docker) to
  manage containerized plugins.
- **You must have [gVisor](https://gvisor.dev/docs/user_guide/install/)
  installed**. Vault uses `runsc` as the entrypoint to your container runtime.
- **If you are using a container runtime other than gVisor, you must have a
  `runsc`-compatible container runtime installed**.

## Step 1: Install your container engine

Install one of the supported container engines:

  - [Docker](https://docs.docker.com/engine/install/)
  - [Rootless Docker](https://docs.docker.com/engine/security/rootless/)

## Step 2: Configure your container runtime

Update your container engine to use `runsc` for Unix sockets between the host
and plugin binary.

<Tabs>

<Tab heading="Docker">

  1. Add `runsc` to your
     [Docker daemon configuration](https://docs.docker.com/config/daemon):

      ```shell-session
      $ sudo tee PATH_TO_DOCKER_DAEMON_CONFIG_FILE <<EOF
      {
        "runtimes": {
          "runsc": {
            "path": "PATH_TO_RUNSC_INSTALLATION",
            "runtimeArgs": [
              "--host-uds=all"
            ]
          }
        }
      }
      EOF
      ```

  1. Restart Docker:

      ```shell-session
      $ sudo systemctl reload docker
      ```

</Tab>

<Tab heading="Rootless Docker">

  1. Create a configuration directory if it does not exist already:

    ```shell-session
    $ mkdir -p ~/.config/docker
    ```

  1. Add `runsc` to your Docker configuration:

    ```shell-session
    $ tee ~/.config/docker/daemon.json  <<EOF
    {
      "runtimes": {
        "runsc": {
          "path": "PATH_TO_RUNSC_INSTALLATION",
          "runtimeArgs": [
            "--host-uds=all"
            "--ignore-cgroups"
          ]
        }
      }
    }
    EOF
    ```

  1. Restart Docker:

    ```shell-session
    $ systemctl --user restart docker
    ```

</Tab>

</Tabs>

## Step 3: Update the HashiCorp `go-plugin` library

You must build your plugin locally with v1.5.0+ of the HashiCorp
[`go-plugin`](https://github.com/hashicorp/go-plugin) library to ensure the
finished binary is compatible with containerization.

Use `go install` to pull the latest version of the plugin library from the
`hashicorp/go-plugin` repo on GitHub:

```shell-session
$ go install github.com/hashicorp/go-plugin@latest
```

<Tip title="The Vault SDK includes go-plugin">

  If you build with the Vault SDK, you can update `go-plugin` with `go install`
  by pulling the latest SDK version from the `hashicorp/vault` repo:

  `go install github.com/hashicorp/vault/sdk@latest`

</Tip>

## Step 4: Build the plugin container

Containerized plugins must run as a binary in the finished container and
behave the same whether run in a container or as a standalone application:

1. Build your plugin binary so it runs on Linux.
1. Create a container file for your plugin with the compiled binary as the
   entry-point.
1. Build the image with a unique tag.

For example, to build a containerized version of the built-in key-value (KV)
secrets plugin for Docker:

1. Clone the latest version of the KV secrets plugin from
   `hashicorp/vault-plugin-secrets-kv`.
    ```shell-session
    $ git clone https://github.com/hashicorp/vault-plugin-secrets-kv.git
    ```
1. Build the Go binary for Linux.
    ```shell-session
    $ cd vault-plugin-secrets-kv ; CGO_ENABLED=0 GOOS=linux \
    go build -o kv cmd/vault-plugin-secrets-kv/main.go
    ```
1. Create an empty Dockerfile.
    ```shell-session
    $ touch Dockerfile
    ```
1. Update the empty `Dockerfile` with your infrastructure build details and the
   compiled binary as the entry-point.
   ```Dockerfile
   FROM gcr.io/distroless/static-debian12
   COPY kv /bin/kv
   ENTRYPOINT [ "/bin/kv" ]
   ```
1. Build the container image and assign an identifiable tag.
   ```shell-session
   $ docker build -t hashicorp/vault-plugin-secrets-kv:mycontainer .
   ```

## Step 5: Register the plugin

Registering a containerized plugin with Vault is similar to registering any
other external plugin that is available locally to Vault.

1. Store the SHA256 of the plugin image:
   ```shell-session
   $ export SHA256=$(docker images \
       --no-trunc \
       --format="" \
       YOUR_PLUGIN_IMAGE_TAG | cut -d: -f2)
   ```
   For example:

   <CodeBlockConfig hideClipboard>

   ```shell-session
   $ export SHA256=$(docker images \
       --no-trunc \
       --format="" \
       hashicorp/vault-plugin-secrets-kv:mycontainer | cut -d: -f2)
   ```

   </CodeBlockConfig>

1. Register the plugin with `vault plugin register` and specify your plugin
   image with the `oci_image` flag:
   ```shell-session
   $ vault plugin register \
       -sha256="${SHA256}" \
       -oci_image=YOUR_PLUGIN_IMAGE_TAG \
       NEW_PLUGIN_TYPE NEW_PLUGIN_ID
   ```
   For example:

   <CodeBlockConfig hideClipboard>

   ```shell-session
   $ vault plugin register \
       -sha256="${SHA256}" \
       -oci_image=hashicorp/vault-plugin-secrets-kv:mycontainer \
       secret my-kv-container
   ```

   </CodeBlockConfig>

1. Enable the new plugin for your Vault instance with `vault secrets enable` and
   the new plugin ID:
   ```shell-session
   $ vault secrets enable NEW_PLUGIN_ID
   ```
   For example:

   <CodeBlockConfig hideClipboard>

   ```shell-session
   $ vault secrets enable my-kv-container
   ```

   </CodeBlockConfig>

<Tip title="Customize container behavior with registration flags">

  You can provide additional information about the image entrypoint, command,
  and environment with the `-command`, `-args`, and `-env` flags for
  `vault plugin register`.

</Tip>

## Step 6: Test your plugin

Now that the container is registered with Vault, you should be able to interact
with it like any other plugin. Try writing then fetching a new secret with your
new plugin.

1. Use `vault write` to store a secret with your containerized plugin:
   ```shell-session
   $ vault write NEW_PLUGIN_ID/SECRET_PATH SECRET_KEY=SECRET_VALUE
   ```
   For example:

   <CodeBlockConfig hideClipboard>

   ```shell-session
   $ vault write my-kv-container/testing subject=containers
   Success! Data written to: my-kv-container/testing
   ```

   </CodeBlockConfig>

1. Fetch the secret you just wrote:
   ```shell-session
   $ vault read NEW_PLUGIN_ID/SECRET_PATH
   ```
   For example:

   <CodeBlockConfig hideClipboard>

   ```shell-session
   $ vault read my-kv-container/testing
   ===== Data =====
   Key        Value
   ---        -----
   subject    containers
   ```

   </CodeBlockConfig>

## Use alternative runtimes ((#alt-runtimes))

You can force Vault to use alternative runtimes provided the runtime is
installed locally.

To use an alternative runtime:

1. Register and name the runtime with `vault plugin runtime register`. For
   example, to register the default Docker runtime (`runc`) as `docker-rt`:
   ```shell-session
   $ vault plugin runtime register  \
      -oci_runtime=runc             \
      -type=container docker-rt
   ```

1. Use the `--runtime` flag during plugin registration to tell Vault what
   runtime to use:
   ```shell-session
   $ vault plugin register  \
      -runtime=RUNTIME_NAME \
      -sha256="${SHA256}"   \
      -oci_image=YOUR_PLUGIN_IMAGE_TAG \
      NEW_PLUGIN_TYPE NEW_PLUGIN_ID
   ```
   For example:

   <CodeBlockConfig hideClipboard>

   ```shell-session
   $ vault plugin register \
      -runtime=docker-rt   \
      -sha256="${SHA256}"  \
      -oci_image=hashicorp/vault-plugin-secrets-kv:mycontainer \
      secret my-kv-container
   ```

   </CodeBlockConfig>

## Troubleshooting

### Invalid backend version error

If you run into the following error while registering your plugin:

<CodeBlockConfig hideClipboard>

```plaintext
invalid backend version error: 2 errors occurred:
        * error creating container: Error response from daemon: error while looking up the specified runtime path: exec: " /usr/bin/runsc": stat  /usr/bin/runsc: no such file or directory
        * error creating container: Error response from daemon: error while looking up the specified runtime path: exec: " /usr/bin/runsc": stat  /usr/bin/runsc: no such file or directory
```

</CodeBlockConfig>

it means that Vault cannot find the executable for `runsc`. Confirm the
following is true before trying again:

1. You have gVisor installed locally to Vault.
1. The path to `runsc` is correct in you your Docker configuration.
1. Vault has permission to run the `runsc` executable.

If you still get errors when registering a plugin, the recommended workaround is
to use the default Docker runtime (`runc`) as an
[alternative runtime](#alt-runtimes)

layout  docs page title  Add a containerized secrets plugin description       Add a containerized secrets plugin to your Vault instance         Add a containerized secrets plugin to Vault  Run your external secrets plugins in containers to increases the isolation between the plugin and Vault      Before you start      Your Vault instance must be running on Linux        Your Vault instance must have local access to the Docker Engine API      Vault uses the  Docker SDK  https   pkg go dev github com docker docker  to   manage containerized plugins      You must have  gVisor  https   gvisor dev docs user guide install     installed    Vault uses  runsc  as the entrypoint to your container runtime      If you are using a container runtime other than gVisor  you must have a    runsc  compatible container runtime installed        Step 1  Install your container engine  Install one of the supported container engines        Docker  https   docs docker com engine install        Rootless Docker  https   docs docker com engine security rootless       Step 2  Configure your container runtime   Update your container engine to use  runsc  for Unix sockets between the host and plugin binary    Tabs    Tab heading  Docker      1  Add  runsc  to your       Docker daemon configuration  https   docs docker com config daemon             shell session         sudo tee PATH TO DOCKER DAEMON CONFIG FILE   EOF                  runtimes                runsc                  path    PATH TO RUNSC INSTALLATION                runtimeArgs                      host uds all                                                    EOF              1  Restart Docker            shell session         sudo systemctl reload docker              Tab    Tab heading  Rootless Docker      1  Create a configuration directory if it does not exist already          shell session       mkdir  p    config docker            1  Add  runsc  to your Docker configuration          shell session       tee    config docker daemon json    EOF              runtimes              runsc                path    PATH TO RUNSC INSTALLATION              runtimeArgs                    host uds all                 ignore cgroups                                          EOF            1  Restart Docker          shell session       systemctl   user restart docker            Tab     Tabs      Step 3  Update the HashiCorp  go plugin  library  You must build your plugin locally with v1 5 0  of the HashiCorp   go plugin   https   github com hashicorp go plugin  library to ensure the finished binary is compatible with containerization   Use  go install  to pull the latest version of the plugin library from the  hashicorp go plugin  repo on GitHub      shell session   go install github com hashicorp go plugin latest       Tip title  The Vault SDK includes go plugin      If you build with the Vault SDK  you can update  go plugin  with  go install    by pulling the latest SDK version from the  hashicorp vault  repo      go install github com hashicorp vault sdk latest     Tip       Step 4  Build the plugin container  Containerized plugins must run as a binary in the finished container and behave the same whether run in a container or as a standalone application   1  Build your plugin binary so it runs on Linux  1  Create a container file for your plugin with the compiled binary as the    entry point  1  Build the image with a unique tag   For example  to build a containerized version of the built in key value  KV  secrets plugin for Docker   1  Clone the latest version of the KV secrets plugin from     hashicorp vault plugin secrets kv          shell session       git clone https   github com hashicorp vault plugin secrets kv git         1  Build the Go binary for Linux         shell session       cd vault plugin secrets kv   CGO ENABLED 0 GOOS linux       go build  o kv cmd vault plugin secrets kv main go         1  Create an empty Dockerfile         shell session       touch Dockerfile         1  Update the empty  Dockerfile  with your infrastructure build details and the    compiled binary as the entry point        Dockerfile    FROM gcr io distroless static debian12    COPY kv  bin kv    ENTRYPOINT     bin kv           1  Build the container image and assign an identifiable tag        shell session      docker build  t hashicorp vault plugin secrets kv mycontainer              Step 5  Register the plugin  Registering a containerized plugin with Vault is similar to registering any other external plugin that is available locally to Vault   1  Store the SHA256 of the plugin image        shell session      export SHA256   docker images            no trunc            format             YOUR PLUGIN IMAGE TAG   cut  d   f2            For example          CodeBlockConfig hideClipboard         shell session      export SHA256   docker images            no trunc            format             hashicorp vault plugin secrets kv mycontainer   cut  d   f2                  CodeBlockConfig   1  Register the plugin with  vault plugin register  and specify your plugin    image with the  oci image  flag        shell session      vault plugin register           sha256    SHA256             oci image YOUR PLUGIN IMAGE TAG          NEW PLUGIN TYPE NEW PLUGIN ID           For example          CodeBlockConfig hideClipboard            shell session      vault plugin register           sha256    SHA256             oci image hashicorp vault plugin secrets kv mycontainer          secret my kv container                 CodeBlockConfig   1  Enable the new plugin for your Vault instance with  vault secrets enable  and    the new plugin ID        shell session      vault secrets enable NEW PLUGIN ID           For example          CodeBlockConfig hideClipboard            shell session      vault secrets enable my kv container                      CodeBlockConfig     Tip title  Customize container behavior with registration flags      You can provide additional information about the image entrypoint  command    and environment with the   command     args   and   env  flags for    vault plugin register      Tip      Step 6  Test your plugin  Now that the container is registered with Vault  you should be able to interact with it like any other plugin  Try writing then fetching a new secret with your new plugin    1  Use  vault write  to store a secret with your containerized plugin        shell session      vault write NEW PLUGIN ID SECRET PATH SECRET KEY SECRET VALUE           For example          CodeBlockConfig hideClipboard            shell session      vault write my kv container testing subject containers    Success  Data written to  my kv container testing                  CodeBlockConfig   1  Fetch the secret you just wrote        shell session      vault read NEW PLUGIN ID SECRET PATH           For example          CodeBlockConfig hideClipboard            shell session      vault read my kv container testing          Data          Key        Value                        subject    containers                 CodeBlockConfig      Use alternative runtimes    alt runtimes    You can force Vault to use alternative runtimes provided the runtime is installed locally   To use an alternative runtime   1  Register and name the runtime with  vault plugin runtime register   For    example  to register the default Docker runtime   runc   as  docker rt         shell session      vault plugin runtime register           oci runtime runc                      type container docker rt         1  Use the    runtime  flag during plugin registration to tell Vault what    runtime to use        shell session      vault plugin register           runtime RUNTIME NAME          sha256    SHA256              oci image YOUR PLUGIN IMAGE TAG         NEW PLUGIN TYPE NEW PLUGIN ID           For example          CodeBlockConfig hideClipboard            shell session      vault plugin register          runtime docker rt            sha256    SHA256             oci image hashicorp vault plugin secrets kv mycontainer         secret my kv container              CodeBlockConfig      Troubleshooting      Invalid backend version error  If you run into the following error while registering your plugin    CodeBlockConfig hideClipboard      plaintext invalid backend version error  2 errors occurred            error creating container  Error response from daemon  error while looking up the specified runtime path  exec     usr bin runsc   stat   usr bin runsc  no such file or directory           error creating container  Error response from daemon  error while looking up the specified runtime path  exec     usr bin runsc   stat   usr bin runsc  no such file or directory        CodeBlockConfig   it means that Vault cannot find the executable for  runsc   Confirm the following is true before trying again   1  You have gVisor installed locally to Vault  1  The path to  runsc  is correct in you your Docker configuration  1  Vault has permission to run the  runsc  executable   If you still get errors when registering a plugin  the recommended workaround is to use the default Docker runtime   runc   as an  alternative runtime   alt runtimes


vault Learn about running external Vault plugins in containers page title Containerized plugins overview Note title Limited OS support Containerized plugins overview layout docs

Answer:
---
layout: docs
page_title: Containerized plugins overview
description: Learn about running external Vault plugins in containers.
---

# Containerized plugins overview

<Note title="Limited OS support">

  Support for the `container` runtime is currently limited to Linux.

</Note>

Vault has a wide selection of builtin plugins to support integrating with other
systems. For example, you can use plugins to exchange app identity information
with an authentication service to receive a Vault token, or manage database
credentials. You can also register **external** plugins with your Vault instance
to extend the capabilities of your Vault server.

By default, external plugins run as subprocesses that share the user and
environment variables of your Vault instance. Administrators managing Vault
instances on Linux can choose to run external plugins in containers. Running
plugins in containers increases the isolation between individual plugins and
between the plugins and Vault.

## System requirements

- **Your Vault instance must be running on Linux**.

- **Your environment must provide Vault local access to the Docker Engine API**.
  Vault uses the [Docker SDK](https://pkg.go.dev/github.com/docker/docker) to
  manage containerized plugins.

- **You must have a valid container runtime installed**. We recommend
  [installing gVisor](https://gvisor.dev/docs/user_guide/install/) for your
  container runtime as Vault specifies the `runsc` runtime by default.

- **You must have all your plugin container images pulled and available locally**.
  Vault does not currently support pulling images as part of the plugin
  registration process.

## Plugin requirements

All plugins have the following basic requirements to be containerized:

- **Your plugin must be built with at least v1.6.0 of the HashiCorp
  [`go-plugin`](https://github.com/hashicorp/go-plugin) library**.

- **The image entrypoint should run the plugin binary**.

Some configurations have additional requirements for the container image, listed
in [supported configurations](#supported-configurations).

## Supported configurations

Vault's containerized plugins are compatible with a variety of configurations.
In particular, it has been tested with the following:

- Default and [rootless](https://docs.docker.com/engine/security/rootless/) Docker.
- OCI-compatible runtimes `runsc` and `runc`.
- Plugin container images running as root and non-root users.
- [Mlock](/vault/docs/configuration#disable_mlock) disabled or enabled.

Not all combinations work and some have additional requirements, listed below.
If you use a configuration that matches multiple headings, you should combine
the requirements from each matching heading.

### `runsc` runtime

- You must pass an additional `--host-uds=create` flag to the `runsc` runtime.

### Rootless Docker with `runsc` runtime

- You must pass an additional `--ignore-cgroups` flag to the `runsc` runtime.
  - Cgroup limits are not currently supported for this configuration.

### Rootless Docker with non-root container user

- You must use a container plugin runtime with
  [`rootless`](/vault/docs/commands/plugin/runtime/register#rootless) enabled.
- Your filesystem must have Posix 1e ACL support, available by default in most
  modern Linux file systems.
- Only supported for gVisor's `runsc` runtime.

### Rootless Docker with mlock enabled

- Only supported for gVisor's `runsc` runtime.

### Non-root container user with mlock enabled

- You must set the `IPC_LOCK` capability on the plugin binary.

## Container lifecycle and metadata

Like any other external plugin, Vault will automatically manage the lifecycle
of plugin containers. If they are killed out of band, Vault will restart them
before servicing any requests that need to be handled by them. Vault will also
[multiplex](/vault/docs/plugins/plugin-architecture#plugin-multiplexing) multiple
mounts to be serviced by the same container if the plugin supports multiplexing.

Vault labels each plugin container with a standard set of metadata to help
identify the owner of the container, including the cluster ID, Vault's own
process ID, and the plugin's name, type, and version.

## Plugin runtimes

Users who require more control over plugin containers can use the "plugin
runtime" APIs for finer grained settings. See the CLI documentation for
[`vault plugin runtime`](/vault/docs/commands/plugin/runtime) for more details.

layout  docs page title  Containerized plugins overview description  Learn about running external Vault plugins in containers         Containerized plugins overview   Note title  Limited OS support      Support for the  container  runtime is currently limited to Linux     Note   Vault has a wide selection of builtin plugins to support integrating with other systems  For example  you can use plugins to exchange app identity information with an authentication service to receive a Vault token  or manage database credentials  You can also register   external   plugins with your Vault instance to extend the capabilities of your Vault server   By default  external plugins run as subprocesses that share the user and environment variables of your Vault instance  Administrators managing Vault instances on Linux can choose to run external plugins in containers  Running plugins in containers increases the isolation between individual plugins and between the plugins and Vault      System requirements      Your Vault instance must be running on Linux         Your environment must provide Vault local access to the Docker Engine API      Vault uses the  Docker SDK  https   pkg go dev github com docker docker  to   manage containerized plugins       You must have a valid container runtime installed    We recommend    installing gVisor  https   gvisor dev docs user guide install   for your   container runtime as Vault specifies the  runsc  runtime by default       You must have all your plugin container images pulled and available locally      Vault does not currently support pulling images as part of the plugin   registration process      Plugin requirements  All plugins have the following basic requirements to be containerized       Your plugin must be built with at least v1 6 0 of the HashiCorp     go plugin   https   github com hashicorp go plugin  library         The image entrypoint should run the plugin binary     Some configurations have additional requirements for the container image  listed in  supported configurations   supported configurations       Supported configurations  Vault s containerized plugins are compatible with a variety of configurations  In particular  it has been tested with the following     Default and  rootless  https   docs docker com engine security rootless   Docker    OCI compatible runtimes  runsc  and  runc     Plugin container images running as root and non root users     Mlock   vault docs configuration disable mlock  disabled or enabled   Not all combinations work and some have additional requirements  listed below  If you use a configuration that matches multiple headings  you should combine the requirements from each matching heading        runsc  runtime    You must pass an additional    host uds create  flag to the  runsc  runtime       Rootless Docker with  runsc  runtime    You must pass an additional    ignore cgroups  flag to the  runsc  runtime      Cgroup limits are not currently supported for this configuration       Rootless Docker with non root container user    You must use a container plugin runtime with     rootless    vault docs commands plugin runtime register rootless  enabled    Your filesystem must have Posix 1e ACL support  available by default in most   modern Linux file systems    Only supported for gVisor s  runsc  runtime       Rootless Docker with mlock enabled    Only supported for gVisor s  runsc  runtime       Non root container user with mlock enabled    You must set the  IPC LOCK  capability on the plugin binary      Container lifecycle and metadata  Like any other external plugin  Vault will automatically manage the lifecycle of plugin containers  If they are killed out of band  Vault will restart them before servicing any requests that need to be handled by them  Vault will also  multiplex   vault docs plugins plugin architecture plugin multiplexing  multiple mounts to be serviced by the same container if the plugin supports multiplexing   Vault labels each plugin container with a standard set of metadata to help identify the owner of the container  including the cluster ID  Vault s own process ID  and the plugin s name  type  and version      Plugin runtimes  Users who require more control over plugin containers can use the  plugin runtime  APIs for finer grained settings  See the CLI documentation for   vault plugin runtime    vault docs commands plugin runtime  for more details


vault page title Server Side Consistent Token FAQ Server side consistent token FAQ This FAQ section contains frequently asked questions about the Server Side Consistent Token feature layout docs An list of frequently asked questions about server side consistent tokens

Answer:
---
layout: docs
page_title: Server Side Consistent Token FAQ
description: An list of frequently asked questions about server side consistent tokens
---

# Server side consistent token FAQ

This FAQ section contains frequently asked questions about the Server Side Consistent Token feature.

- [Q: What is the Server Side Consistent Token feature?](#q-what-is-the-server-side-consistent-token-feature)
- [Q: I have Vault Community Edition. How does this feature impact me?](#q-i-have-vault-community-edition-how-does-this-feature-impact-me)
- [Q: What token changes does the Server Side Consistent Tokens feature introduce?](#q-what-token-changes-does-the-server-side-consistent-tokens-feature-introduce)
- [Q: Why are we changing the token?](#q-why-are-we-changing-the-token)
- [Q: What type of tokens are impacted by this feature?](#q-what-type-of-tokens-are-impacted-by-this-feature)
- [Q: Is there a new configuration that this feature introduces?](#q-is-there-a-new-configuration-that-this-feature-introduces)
- [Q: Is there anything else I need to consider to achieve consistency, besides upgrading to Vault 1.10?](#q-is-there-anything-else-i-need-to-consider-to-achieve-consistency-besides-upgrading-to-vault-1-10)
- [Q: What do I need to be paying attention to if I rely on tokens for some of my workflows?](#q-what-do-i-need-to-be-paying-attention-to-if-i-rely-on-tokens-for-some-of-my-workflows)
- [Q: What are the main mitigation options that Vault offers to achieve consistency, and what are the differences between them?](#q-what-are-the-main-mitigation-options-that-vault-offers-to-achieve-consistency-and-what-are-the-differences-between-them)
- [Q: Is this feature something I need with Consul Storage?](#q-is-this-feature-something-i-need-with-consul-storage)

### Q: What is the server side consistent token feature?

~> **Note**: This features requires Vault Enterprise.

Vault has an [eventual consistency](/vault/docs/enterprise/consistency) model where only the leader can write to Vault's storage. When using performance standbys with Integrated Storage, there are sequences of operations that don't always yield read-after-write consistency, which may pose a challenge for some use cases.

Several client-based mitigations were added in Vault version 1.7, which depended on some modifications to clients (provide the appropriate response header per request) so they can specify state. This may not be possible to do in some environments.

To help with such cases, we’ve now added support for the Server Side Consistent Tokens feature in Vault version 1.10. See [Replication](/vault/docs/configuration/replication), [Vault Eventual Consistency](/vault/docs/enterprise/consistency), and [Upgrade to 1.10](/vault/docs/upgrading/upgrade-to-1.10.x).
This feature provides a way for Service tokens, returned from logins (or token create requests), to have the relevant minimum WAL state information embedded within the token itself. Clients can then use this token to authenticate subsequent requests. Thus, clients can obtain read-after-write consistency for the token without typically having to make changes to their code or architecture.
If a performance standby does not have the state required to authenticate the token, it returns a 412 error to allow the client to retry. If client retry is not possible, there is a server config to allow for the consistency.

### Q: I have Vault Community Edition. How does this feature impact me?

For the sake of standardization between Community and Enterprise, and due to the value of adding token prefixes in Vault for token scanning use cases, the token formats are changed across all Vault versions starting Vault 1.10. However, since there are no performance standbys or replication in Vault Community Edition, the new Vault token will always show the local index of the WAL as 0 to indicate there is nothing to wait for.

### Q: What token changes does the Server Side Consistent Tokens feature introduce?

Server Side Consistent Tokens introduces the following key changes:

- Token length : Server side consistent tokens are longer, being 95+ characters as opposed to 27+ characters. Since the token can be subject to change (see [Token](/vault/docs/concepts/tokens)), we recommend that you plan for a maximum length of 255 bytes to future proof yourself if you have workflows that rely on the token size.
- By default, Vault 1.10 will use the new token prefixes and new token format.
- Tokens don't visibly have a ".namespaceID" suffix
- Token prefix: Token prefixes are being changed as follows:

| Token Type      | Old Prefix | New Prefix |
| --------------- | ---------- | ---------- |
| Service Tokens  | s.         | hvs.       |
| Batch Tokens    | b.         | hvb.       |
| Recovery Tokens | r.         | hvr.       |

### Q: Why are we changing the token?

To help with use cases that need read-after-write consistency, the Server Side Consistent Tokens feature provides a way for Service tokens, returned from logins (or token create requests), to embed the relevant information for Vault servers using Integrated Storage to know the minimum WAL index that includes the storage write for the token. This entails changes to the service token format.

The token prefix is being updated to make it easier for static-analysis code scanning tools to scan for Vault tokens, for example, to identify Vault tokens that are accidentally stored in a version control system.

### Q: What type of tokens are impacted by this feature?

With the exception of the prefix changes detailed above that apply to all token types, only Service tokens are impacted by the changes that are introduced by this feature. Other token types such as batch tokens, recovery tokens, or root service tokens are not impacted.

### Q: Is there a new configuration that this feature introduces?

There is a new configuration in the replication section as follows:

```
replication {
  allow_forwarding_via_token = "new_token"
}
```

This configuration allows Vault clusters to be configured so that requests made to performance standbys that don’t yet have the most up-to-date WAL index are forwarded to the active node. Please note that there will be extra load on the active node with this type of configuration.

### Q: Is there anything else I need to consider to achieve consistency, besides upgrading to Vault 1.10?

Yes, there are several considerations to keep in mind, and possibly things that may require you to take action, depending on your use case.

- As stated earlier, if a performance standby does not have the state required to authenticate the token, it returns a 412 error allowing the client to retry.
- Ensure that your clients can retry for the best experience.
- Starting with Go api version [1.1.0](https://pkg.go.dev/github.com/hashicorp/vault/api@v1.1.0), the Go client library enables automatic retries for 412 errors. By default, retries=2, or use client method [SetMaxRetries](https://pkg.go.dev/github.com/hashicorp/vault/api#Client.SetMaxRetries). Or, you can use the Vault environment variable [VAULT_MAX_RETRIES](/vault/docs/commands#vault_max_retries) to achieve the same result.
- If you use a client library other than Go, you may still need to ensure that your client can handle 412 retries in order to achieve consistency.
- If your client cannot retry, you can use the Vault server replication configuration `allow_forwarding_via_token`  to allow for consistency. As stated earlier, this will incur extra load on the server due to forwarding of requests that don't have the up-to-date WAL-state to the server:

```
replication {
  allow_forwarding_via_token = "new_token"
}
```

~> **Note:** If you are generating root tokens or recovery tokens without using the Vault CLI, you will need to modify the OTP length used. refer [here](/vault/docs/upgrading/upgrade-to-1.10.x) for details.

### Q: What do I need to be paying attention to if I rely on tokens for some of my workflows?

Our documentation on [tokens](/vault/docs/concepts/tokens) clearly identifies that the token body itself is subject to change between versions and should not be relied on. We strongly recommend that you consider this while architecting your environment.

However, if you use scripting and tooling to help in the authentication process for Vault-dependent applications, it is important that you take time to understand the changes (see [Replication](/vault/docs/configuration/replication), [Vault Eventual Consistency](/vault/docs/enterprise/consistency), and [Upgrade to 1.10](/vault/docs/upgrading/upgrade-to-1.10.x)), and test these changes in your specific dev environments before deploying this in production.

If your workflow used the embedded NamespaceID suffix, you will need to perform a [token lookup](/vault/docs/commands/token/lookup) because this is currently absent in the new tokens.

### Q: What are the main mitigation options that Vault offers to achieve consistency, and what are the differences between them?

Vault offers the following options to achieve consistency:

- [Client based mitigations](/vault/docs/enterprise/consistency#vault-1-7-mitigations), which was added in Vault Release 1.7, depend on some modifications to clients to include per request header options to ‘always forward the request to the active node’ OR to ‘conditionally forward the request to the active node’ if it would otherwise result in a stale read OR to ‘fail requests’ with error 412 if they might result in a stale read.
- The Vault Agent can also be leveraged for proxied requests to achieve consistency via the above mitigations without client modifications
- Server Side Consistent Tokens, added in Vault version 1.10, provide a more implicit way to achieve consistency, but only addresses consistency for new tokens.

The following table outlines the main differences:

| Client Controlled Consistency                                                                     | Agent with client controlled consistency settings                                                           | Server Side Consistent Tokens                                                                                |
| ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Needs Client side modifications for consistency (per request header options need to be included). | Vault Agent can also be leveraged to achieve consistency without client modifications for proxied requests. | Implicit way for consistency where the relevant minimum WAL state information is embedded within the token itself. |
| Works across clusters too (Performance standbys and Performance Replication)                      | Single cluster only (Performance Standby)                                                                   | Single cluster only (Performance Standby)                                                                          |
| Applies to any Vault operation                                                                    | Applies to any Vault operation                                                                              | Applies to login / token create requests only                                                                      |
| May have performance implications via enforcing too much consistency                              | May have performance implications, via enforcing too much consistency for proxied requests                  | May have performance implications if server side configuration to forward requests to active nodes is leveraged.   |

~> **Note:** Client controlled consistency headers , if configured, will take precedence over the server configuration.

Finally, when speaking of performance implications above, there are two kinds that you should keep in mind while selecting the best option for your use case:

- Using forwarding will impact horizontal scalability by placing additional load on the active node
- No using forwarding will impact latency of client requests due to retrying until the state is consistency

### Q: Is this feature something I need with Consul Storage?

Consul has a [default consistency model](/consul/api-docs/features/consistency) and this feature is not relevant with Consul storage.

layout  docs page title  Server Side Consistent Token FAQ description  An list of frequently asked questions about server side consistent tokens        Server side consistent token FAQ  This FAQ section contains frequently asked questions about the Server Side Consistent Token feature      Q  What is the Server Side Consistent Token feature    q what is the server side consistent token feature     Q  I have Vault Community Edition  How does this feature impact me    q i have vault community edition how does this feature impact me     Q  What token changes does the Server Side Consistent Tokens feature introduce    q what token changes does the server side consistent tokens feature introduce     Q  Why are we changing the token    q why are we changing the token     Q  What type of tokens are impacted by this feature    q what type of tokens are impacted by this feature     Q  Is there a new configuration that this feature introduces    q is there a new configuration that this feature introduces     Q  Is there anything else I need to consider to achieve consistency  besides upgrading to Vault 1 10    q is there anything else i need to consider to achieve consistency besides upgrading to vault 1 10     Q  What do I need to be paying attention to if I rely on tokens for some of my workflows    q what do i need to be paying attention to if i rely on tokens for some of my workflows     Q  What are the main mitigation options that Vault offers to achieve consistency  and what are the differences between them    q what are the main mitigation options that vault offers to achieve consistency and what are the differences between them     Q  Is this feature something I need with Consul Storage    q is this feature something i need with consul storage       Q  What is the server side consistent token feature        Note    This features requires Vault Enterprise   Vault has an  eventual consistency   vault docs enterprise consistency  model where only the leader can write to Vault s storage  When using performance standbys with Integrated Storage  there are sequences of operations that don t always yield read after write consistency  which may pose a challenge for some use cases   Several client based mitigations were added in Vault version 1 7  which depended on some modifications to clients  provide the appropriate response header per request  so they can specify state  This may not be possible to do in some environments   To help with such cases  we ve now added support for the Server Side Consistent Tokens feature in Vault version 1 10  See  Replication   vault docs configuration replication    Vault Eventual Consistency   vault docs enterprise consistency   and  Upgrade to 1 10   vault docs upgrading upgrade to 1 10 x   This feature provides a way for Service tokens  returned from logins  or token create requests   to have the relevant minimum WAL state information embedded within the token itself  Clients can then use this token to authenticate subsequent requests  Thus  clients can obtain read after write consistency for the token without typically having to make changes to their code or architecture  If a performance standby does not have the state required to authenticate the token  it returns a 412 error to allow the client to retry  If client retry is not possible  there is a server config to allow for the consistency       Q  I have Vault Community Edition  How does this feature impact me   For the sake of standardization between Community and Enterprise  and due to the value of adding token prefixes in Vault for token scanning use cases  the token formats are changed across all Vault versions starting Vault 1 10  However  since there are no performance standbys or replication in Vault Community Edition  the new Vault token will always show the local index of the WAL as 0 to indicate there is nothing to wait for       Q  What token changes does the Server Side Consistent Tokens feature introduce   Server Side Consistent Tokens introduces the following key changes     Token length   Server side consistent tokens are longer  being 95  characters as opposed to 27  characters  Since the token can be subject to change  see  Token   vault docs concepts tokens    we recommend that you plan for a maximum length of 255 bytes to future proof yourself if you have workflows that rely on the token size    By default  Vault 1 10 will use the new token prefixes and new token format    Tokens don t visibly have a   namespaceID  suffix   Token prefix  Token prefixes are being changed as follows     Token Type        Old Prefix   New Prefix                                                   Service Tokens    s            hvs            Batch Tokens      b            hvb            Recovery Tokens   r            hvr               Q  Why are we changing the token   To help with use cases that need read after write consistency  the Server Side Consistent Tokens feature provides a way for Service tokens  returned from logins  or token create requests   to embed the relevant information for Vault servers using Integrated Storage to know the minimum WAL index that includes the storage write for the token  This entails changes to the service token format   The token prefix is being updated to make it easier for static analysis code scanning tools to scan for Vault tokens  for example  to identify Vault tokens that are accidentally stored in a version control system       Q  What type of tokens are impacted by this feature   With the exception of the prefix changes detailed above that apply to all token types  only Service tokens are impacted by the changes that are introduced by this feature  Other token types such as batch tokens  recovery tokens  or root service tokens are not impacted       Q  Is there a new configuration that this feature introduces   There is a new configuration in the replication section as follows       replication     allow forwarding via token    new token         This configuration allows Vault clusters to be configured so that requests made to performance standbys that don t yet have the most up to date WAL index are forwarded to the active node  Please note that there will be extra load on the active node with this type of configuration       Q  Is there anything else I need to consider to achieve consistency  besides upgrading to Vault 1 10   Yes  there are several considerations to keep in mind  and possibly things that may require you to take action  depending on your use case     As stated earlier  if a performance standby does not have the state required to authenticate the token  it returns a 412 error allowing the client to retry    Ensure that your clients can retry for the best experience    Starting with Go api version  1 1 0  https   pkg go dev github com hashicorp vault api v1 1 0   the Go client library enables automatic retries for 412 errors  By default  retries 2  or use client method  SetMaxRetries  https   pkg go dev github com hashicorp vault api Client SetMaxRetries   Or  you can use the Vault environment variable  VAULT MAX RETRIES   vault docs commands vault max retries  to achieve the same result    If you use a client library other than Go  you may still need to ensure that your client can handle 412 retries in order to achieve consistency    If your client cannot retry  you can use the Vault server replication configuration  allow forwarding via token   to allow for consistency  As stated earlier  this will incur extra load on the server due to forwarding of requests that don t have the up to date WAL state to the server       replication     allow forwarding via token    new token              Note    If you are generating root tokens or recovery tokens without using the Vault CLI  you will need to modify the OTP length used  refer  here   vault docs upgrading upgrade to 1 10 x  for details       Q  What do I need to be paying attention to if I rely on tokens for some of my workflows   Our documentation on  tokens   vault docs concepts tokens  clearly identifies that the token body itself is subject to change between versions and should not be relied on  We strongly recommend that you consider this while architecting your environment   However  if you use scripting and tooling to help in the authentication process for Vault dependent applications  it is important that you take time to understand the changes  see  Replication   vault docs configuration replication    Vault Eventual Consistency   vault docs enterprise consistency   and  Upgrade to 1 10   vault docs upgrading upgrade to 1 10 x    and test these changes in your specific dev environments before deploying this in production   If your workflow used the embedded NamespaceID suffix  you will need to perform a  token lookup   vault docs commands token lookup  because this is currently absent in the new tokens       Q  What are the main mitigation options that Vault offers to achieve consistency  and what are the differences between them   Vault offers the following options to achieve consistency      Client based mitigations   vault docs enterprise consistency vault 1 7 mitigations   which was added in Vault Release 1 7  depend on some modifications to clients to include per request header options to  always forward the request to the active node  OR to  conditionally forward the request to the active node  if it would otherwise result in a stale read OR to  fail requests  with error 412 if they might result in a stale read    The Vault Agent can also be leveraged for proxied requests to achieve consistency via the above mitigations without client modifications   Server Side Consistent Tokens  added in Vault version 1 10  provide a more implicit way to achieve consistency  but only addresses consistency for new tokens   The following table outlines the main differences     Client Controlled Consistency                                                                       Agent with client controlled consistency settings                                                             Server Side Consistent Tokens                                                                                Needs Client side modifications for consistency  per request header options need to be included     Vault Agent can also be leveraged to achieve consistency without client modifications for proxied requests    Implicit way for consistency where the relevant minimum WAL state information is embedded within the token itself      Works across clusters too  Performance standbys and Performance Replication                         Single cluster only  Performance Standby                                                                      Single cluster only  Performance Standby                                                                               Applies to any Vault operation                                                                      Applies to any Vault operation                                                                                Applies to login   token create requests only                                                                          May have performance implications via enforcing too much consistency                                May have performance implications  via enforcing too much consistency for proxied requests                    May have performance implications if server side configuration to forward requests to active nodes is leveraged            Note    Client controlled consistency headers   if configured  will take precedence over the server configuration   Finally  when speaking of performance implications above  there are two kinds that you should keep in mind while selecting the best option for your use case     Using forwarding will impact horizontal scalability by placing additional load on the active node   No using forwarding will impact latency of client requests due to retrying until the state is consistency      Q  Is this feature something I need with Consul Storage   Consul has a  default consistency model   consul api docs features consistency  and this feature is not relevant with Consul storage


vault The Venafi integrated secrets engine for Vault Venafi secrets engine for HashiCorp Vault layout docs The Venafi Machine Identity Secrets Engine provides applications with the page title Venafi Secrets Engines ability to dynamically generate SSL TLS certificates that serve as machine

layout  docs page title  Venafi   Secrets Engines description  The Venafi integrated secrets engine for Vault         Venafi secrets engine for HashiCorp Vault  The Venafi Machine Identity Secrets Engine provides applications with the ability to dynamically generate SSL TLS certificates that serve as machine identities  Using  Venafi Trust Protection Platform  https   www venafi com platform trust protection platform  or  Venafi Cloud  https   www venafi com venaficloud  assures compliance with enterprise policy and consistency with industry standard trust protection  Designed for high performance with the same interface as the built in PKI secrets engine  services can get certificates without manually generating a private key and CSR  submitting to a certificate authority  and waiting for a verification and signing process to complete  Venafi s certificate authority integrations and policy controls  combined with Vault s built in authentication and authorization mechanisms  provide the verification functionality   Like the built in PKI secrets engine  short lived certificates for ephemeral workloads are the primary focus of the Venafi secrets engine  As such  revocation is not currently supported   The Venafi secrets engine makes use of HashiCorp Vault s  plugin system   vault docs plugins  and Venafi s  VCert Client SDK  https   github com Venafi vcert   If you have questions about the Venafi secrets engine  have an issue to report  or have developed improvements that you want to contribute  visit the  GitHub  https   github com Venafi vault pki backend venafi  repository      Considerations  To successfully deploy this secrets engine  there are some important considerations  Before using Venafi secrets engine  you should read every consideration       Venafi trust protection platform requirements  Your certificate authority  CA  must be able to issue a certificate in under one minute  Microsoft Active Directory Certificate Services  ADCS  is a popular choice  Other CA choices may have slightly different requirements   Within Trust Protection Platform  configure these settings  For more information see the  Venafi Administration Guide      A user account that has an authentication token for the  Venafi Secrets   Engine for HashiCorp Vault   ID  hashicorp vault by venafi   API Application   as of 20 1  or scope  certificate manage  for 19 2 through 19 4  or has been   granted WebSDK Access  deprecated    A Policy folder where the user has the following permissions  View  Read    Write  Create    Enterprise compliant policies applied to the folder including       Subject DN values for Organizational Unit  OU   Organization  O       City Locality  L   State Province  ST  and Country  C       CA Template that Trust Protection Platform will use to enroll general     certificate requests      Management Type not locked or locked to  Enrollment       Certificate Signing Request  CSR  Generation unlocked or not locked to      Service Generated CSR       Generate Key CSR on Application not locked or locked to  No        Recommended  Disable Automatic Renewal set to  Yes        Recommended  Key Bit Strength set to 2048 or higher       Recommended  Domain Whitelisting policy appropriately assigned       NOTE    If you are using Microsoft ACDS  the CRL distribution point and   Authority Information Access  AIA  URIs must start with an HTTP URI    non default configuration   If an LDAP URI appears first in the X509v3   extensions  some applications will fail  such as NGINX ingress controllers    These applications aren t able to retrieve CRL and OCSP information        Trust between Vault and trust protection platform  The Trust Protection Platform REST API  WebSDK  must be secured with a certificate  Generally  the certificate is issued by a CA that is not publicly trusted so establishing trust is a critical part of your setup   Two methods can be used to establish trust  Both require the trust anchor  root CA certificate  of the WebSDK certificate  If you have administrative access  you can import the root certificate into the trust store for your operating system  If you don t have administrative access  or prefer not to make changes to your system configuration  save the root certificate to a file in PEM format  e g   opt venafi bundle pem  and reference it using the  trust bundle file  parameter whenever you create or update a PKI role in your Vault       Venafi Cloud requirements  If you are using Venafi Cloud  be sure to set up an issuing template  project  and any other dependencies that appear in the Venafi Cloud documentation     Set up an issuing template to link Venafi Cloud to your CA  To learn more    search for  Issuing Templates  in the    Venafi Cloud Help system  https   docs venafi cloud help Default htm     Create a project and zone that identifies the template and other information    To learn more  search for  Projects  in the    Venafi Cloud Help system  https   docs venafi cloud help Default htm       Setup   Tabs   Tab heading  Vault  group  vault    Before certificates can be issued  you must complete these steps to configure the Venafi secrets engine   1  Create the  directory   vault docs plugins plugin architecture plugin directory     where your Vault server will look for plugins  e g   etc vault vault plugins      The directory must not be a symbolic link  On macOS  for example   etc is a    link to  private etc  To avoid errors  choose an alternative directory such    as  private etc vault vault plugins   1  Download the latest  vault pki backend venafi      release package  https   github com Venafi vault pki backend venafi releases latest     for your operating system  Unzip the binary to the plugin directory  Note    that the URL for the zip file  referenced below  changes as new versions of the    plugin are released  Replace the version  0 12 0  of the release in the command below to    download the desired version         shell session      wget https   github com Venafi vault pki backend venafi releases download v0 12 0 venafi pki backend v0 12 0 darwin zip      unzip venafi pki backend v0 12 0 darwin zip      mv venafi pki backend  etc vault vault plugins         1  Update the Vault  server configuration   vault docs configuration      to specify the plugin directory         shell session      plugin directory     etc vault vault plugins          1  Start your Vault using the  server command   vault docs commands server    1  Get the SHA 256 checksum of the  venafi pki backend  plugin binary         shell session      SHA256   sha256sum  etc vault vault plugins venafi pki backend  cut  d     f1          1  Register the  venafi pki backend  plugin in the Vault     system catalog   vault docs plugins plugin architecture plugin catalog          shell session      vault write sys plugins catalog secret venafi pki backend      sha 256    SHA256   command  venafi pki backend          1  Enable the Venafi secrets engine         shell session      vault secrets enable  path venafi pki  plugin name venafi pki backend plugin         1  Configure a Venafi secret that maps a name in Vault to connection and authentication    settings for enrolling certificates using Venafi  The zone is a policy folder for Trust    Protection Platform or a DevOps project zone for Venafi Cloud      Obtain the  access token  and  refresh token  for Trust Protection Platform using the     VCert CLI  https   github com Venafi vcert blob master README CLI PLATFORM md obtaining an authorization token       getcred  action with    client id  hashicorp vault by venafi   and       scope  certificate manage    or the Platform s Authorize REST API method      To see all options available for venafi secrets  use     vault path help venafi pki venafi  name  after creating the secret        Trust Protection Platform           shell session      vault write venafi pki venafi tpp          url  https   tpp venafi example           access token  tn1PwE1QTZorXmvnTowSyA             refresh token  MGxV7DzNnclQi9CkJMCXCg             zone  DevOps  HashiCorp Vault           trust bundle file   path to bundle pem               Venafi Cloud           shell session      vault write venafi pki venafi cloud          apikey  xxxxxxxx xxxx xxxx xxxx xxxxxxxxxxxx           zone  zzzzzzzz zzzz zzzz zzzz zzzzzzzzzzzz          1  Lastly  configure a  role   vault docs secrets pki     that maps a name in Vault to a Venafi secret for enrollment  To see all    options available for roles  including  ttl    max ttl  and  issuer hint      for validity   use  vault path help venafi pki roles  name  after    creating the role        Trust Protection Platform           shell session      vault write venafi pki roles tpp          venafi secret tpp          store by serial store pkey true          allowed domains example com          allow subdomains true              Venafi Cloud           shell session      vault write venafi pki roles cloud          venafi secret cloud          store by serial store pkey true          allowed domains example com          allow subdomains true           Tab   Tab heading  HCP Vault Dedicated  group  hcp       The Venafi Secrets Engine on HCP Vault Dedicated currently supports Venafi Cloud or Trust Protection Platform instances secured using a certificate from a publicly trusted CA  Support for uploading a certificate signed by a private CA using trust bundle file parameter is not available on HCP Vault Dedicated and requires running a self managed Vault to use   Before certificates can be issued  you must complete these steps to configure the Venafi secrets engine   1  Navigate to your HCP Vault Dedicated cluster s  Integrations   hcp docs vault integrations hashicorp partner plugins  page within the HCP portal to add the Venafi secrets engine to your cluster   1  After the Venafi plugin has been successfully added to your cluster  you can use the Vault CLI to configure the Venafi secrets engine  for use   1  Enable the Venafi secrets engine         shell session      vault secrets enable  path venafi pki  plugin name venafi pki backend plugin            Configure a Venafi secret that maps a name in Vault to connection and authentication    settings for enrolling certificates using Venafi  The zone is a DevOps project zone for Venafi Cloud      To see all options available for venafi secrets  use     vault path help venafi pki venafi  name  after creating the secret        Venafi Cloud           shell session      vault write venafi pki venafi cloud          apikey  xxxxxxxx xxxx xxxx xxxx xxxxxxxxxxxx           zone  zzzzzzzz zzzz zzzz zzzz zzzzzzzzzzzz          1  Lastly  configure a  role   vault docs secrets pki     that maps a name in Vault to a Venafi secret for enrollment  To see all    options available for roles  including  ttl    max ttl  and  issuer hint      for validity   use  vault path help venafi pki roles  name  after    creating the role        Venafi Cloud           shell session      vault write venafi pki roles cloud          venafi secret cloud          store by serial store pkey true          allowed domains example com          allow subdomains true           Tab    Tabs      Usage  After the Venafi secrets engine is configured and a user machine has a Vault token with the proper permission  it can enroll certificates using Venafi  To see all of the options available when requesting a certificate  including  ttl   for validity    key password   and  custom fields   use  vault path help venafi pki issue  role name  and  vault path help venafi pki sign  role name    1  Generate a certificate by writing to the   issue  endpoint with the name of    the role        Trust Protection Platform           shell session      vault write venafi pki issue tpp common name  common name example com           alt names  dns san 1 example com dns san 2 example com              Example output            text    Key                  Value                                  lease id             venafi pki issue tpp oLih42SCFzyjntxGc00vqmWH    lease duration       719h49m55s    lease renewable      false    certificate               BEGIN CERTIFICATE         certificate chain         BEGIN CERTIFICATE         common name          common name example com    private key               BEGIN RSA PRIVATE KEY         serial number        1d bc a8 3c 00 00 00 05 5c e8              Venafi Cloud           shell session      vault write venafi pki issue cloud common name  common name example com           alt names  dns san 1 example com dns san 2 example com               Example output            text    Key                  Value                                  lease id             venafi pki issue cloud 1WCNvXKiwboWfRRfjzlPAwEi    lease duration       167h59m58s    lease renewable      false    certificate               BEGIN CERTIFICATE         certificate chain         BEGIN CERTIFICATE         common name          common name example com    private key               BEGIN RSA PRIVATE KEY         serial number        17 47 8b 13 90 b8 3d 87 b0 dc b6 9e 00 2b 87 02 c9 d3 1e 8a         1  Or sign a CSR from a file by writing to the   sign  endpoint with the name of    the role        Trust Protection Platform           shell session      vault write venafi pki sign tpp csr  example req              Example output            text    Key                  Value                                  lease id             venafi pki sign tpp tQq3QNY45e4sJMqTTI9DXEGK    lease duration       719h49m57s    lease renewable      false    certificate               BEGIN CERTIFICATE         certificate chain         BEGIN CERTIFICATE         common name          common name example com    serial number        1d c4 07 9a 00 00 00 05 5c ea              Venafi Cloud           shell session      vault write venafi pki sign cloud csr  example req              Example output            text    Key                  Value                                  lease id             venafi pki sign cloud fF44FdMAjuCdC29w3Ff81hes    lease duration       167h59m58s    lease renewable      false    certificate               BEGIN CERTIFICATE         certificate chain         BEGIN CERTIFICATE         common name          common name example com    serial number        76 55 e2 14 de c8 3f e1 64 4a fa 37 d4 6e f5 ef 5e 4c 16 5b            API  Venafi Machine Identity Secrets Engine uses the same  Vault API   vault api docs secret pki  as the built in PKI secrets engine  Some methods  such as those for managing certificate authorities  do not apply