istio docs ops troubleshooting istioctl test no Istio includes a supplemental tool that provides debugging and diagnosis for Istio service mesh deployments title Using the Istioctl Command line Tool aliases owner istio wg user experience maintainers weight 10 keywords istioctl bash zsh shell command line help ops component debugging

Answer:
---
title: Using the Istioctl Command-line Tool
description: Istio includes a supplemental tool that provides debugging and diagnosis for Istio service mesh deployments.
weight: 10
keywords: [istioctl,bash,zsh,shell,command-line]
aliases:
  - /help/ops/component-debugging
  - /docs/ops/troubleshooting/istioctl
owner: istio/wg-user-experience-maintainers
test: no
---

You can gain insights into what individual components are doing by inspecting their
[logs](/docs/ops/diagnostic-tools/component-logging/) or peering inside via
[introspection](/docs/ops/diagnostic-tools/controlz/). If that's insufficient,
the steps below explain how to get under the hood.

The [`istioctl`](/docs/reference/commands/istioctl) tool is a configuration command line utility
that allows service operators to debug and diagnose their Istio service mesh deployments.
The Istio project also includes two helpful scripts for `istioctl` that enable auto-completion
for Bash and Zsh. Both of these scripts provide support for the currently available `istioctl` commands.

`istioctl` only has auto-completion enabled for non-deprecated commands.

## Before you begin

We recommend you use an `istioctl` version that is the same version as your Istio control plane.
Using matching versions helps avoid unforeseen issues.

If you have already [downloaded the Istio release](/docs/setup/additional-setup/download-istio-release/), you should
already have `istioctl` and do not need to install it again.

## Install

Install the `istioctl` binary with `curl`:

1. Download the latest release with the command:

    $ curl -sL https://istio.io/downloadIstioctl | sh -

1. Add the `istioctl` client to your path, on a macOS or Linux system:

    $ export PATH=$HOME/.istioctl/bin:$PATH

1. You can optionally enable the [auto-completion option](#enabling-auto-completion) when working with a bash or Zsh console.

## Get an overview of your mesh

You can get an overview of your mesh using the `proxy-status` or `ps` command:

$ istioctl proxy-status

If a proxy is missing from the output list it means that it is not currently connected to an istiod instance and so it
will not receive any configuration. Additionally, if it is marked stale, it likely means there are networking issues or
istiod needs to be scaled.

## Get proxy configuration

[`istioctl`](/docs/reference/commands/istioctl) allows you to retrieve information
about proxy configuration using the `proxy-config` or `pc` command.

For example, to retrieve information about cluster configuration for the Envoy instance in a specific pod:

$ istioctl proxy-config cluster <pod-name> [flags]

To retrieve information about bootstrap configuration for the Envoy instance in a specific pod:

$ istioctl proxy-config bootstrap <pod-name> [flags]

To retrieve information about listener configuration for the Envoy instance in a specific pod:

$ istioctl proxy-config listener <pod-name> [flags]

To retrieve information about route configuration for the Envoy instance in a specific pod:

$ istioctl proxy-config route <pod-name> [flags]

To retrieve information about endpoint configuration for the Envoy instance in a specific pod:

$ istioctl proxy-config endpoints <pod-name> [flags]

See [Debugging Envoy and Istiod](/docs/ops/diagnostic-tools/proxy-cmd/) for more advice on interpreting this information.

## `istioctl` auto-completion

If you are using the macOS operating system with the Zsh terminal shell, make sure that
the `zsh-completions` package is installed. With the [brew](https://brew.sh) package manager
for macOS, you can check to see if the `zsh-completions` package is installed with the following command:

$ brew list zsh-completions
/usr/local/Cellar/zsh-completions/0.34.0/share/zsh-completions/ (147 files)

If you receive `Error: No such keg: /usr/local/Cellar/zsh-completion`,
proceed with installing the `zsh-completions` package with the following command:

$ brew install zsh-completions

Once the `zsh-completions package` has been installed on your macOS system, add the following to your `~/.zshrc` file:

    if type brew &>/dev/null; then
      FPATH=$(brew --prefix)/share/zsh-completions:$FPATH

      autoload -Uz compinit
      compinit
    fi

You may also need to force rebuild `zcompdump`:

$ rm -f ~/.zcompdump; compinit

Additionally, if you receive `Zsh compinit: insecure directories` warnings
when attempting to load these completions, you may need to run this:

$ chmod -R go-w "$(brew --prefix)/share"

If you are using a Linux-based operating system, you can install the Bash completion package
with the `apt-get install bash-completion` command for Debian-based Linux distributions or
`yum install bash-completion` for RPM-based Linux distributions, the two most common occurrences.

Once the `bash-completion` package has been installed on your Linux system,
add the following line to your `~/.bash_profile` file:

[[ -r "/usr/local/etc/profile.d/bash_completion.sh" ]] && . "/usr/local/etc/profile.d/bash_completion.sh"

### Enabling auto-completion

To enable `istioctl` completion on your system, follow the steps for your preferred shell:

You will need to download the full Istio release containing the auto-completion files (in the `/tools` directory).
If you haven't already done so, [download the full release](/docs/setup/additional-setup/download-istio-release/) now.

Installing the bash auto-completion file

If you are using bash, the `istioctl` auto-completion file is located in the `tools` directory.
To use it, copy the `istioctl.bash` file to your home directory, then add the following line to
source the `istioctl` tab completion file from your `.bashrc` file:

$ source ~/istioctl.bash

Installing the Zsh auto-completion file

For Zsh users, the `istioctl` auto-completion file is located in the `tools` directory.
Copy the `_istioctl` file to your home directory, or any directory of your choosing
(update directory in script snippet below), and source the `istioctl` auto-completion file
in your `.zshrc` file as follows:

source ~/_istioctl

You may also add the `_istioctl` file to a directory listed in the `fpath` variable.
To achieve this, place the `_istioctl` file in an existing directory in the `fpath`,
or create a new directory and add it to the `fpath` variable in your `~/.zshrc` file.

If you get an error like `complete:13: command not found: compdef`,
then add the following to the beginning of your `~/.zshrc` file:

$ autoload -Uz compinit
$ compinit

If your auto-completion is not working, try again after restarting your terminal.
If auto-completion still does not work, try resetting the completion cache using
the above commands in your terminal.

### Using auto-completion

If the `istioctl` completion file has been installed correctly, press the Tab key
while writing an `istioctl` command, and it should return a set of command suggestions
for you to choose from:

$ istioctl proxy-<TAB>
proxy-config proxy-status

title  Using the Istioctl Command line Tool description  Istio includes a supplemental tool that provides debugging and diagnosis for Istio service mesh deployments  weight  10 keywords   istioctl bash zsh shell command line  aliases       help ops component debugging      docs ops troubleshooting istioctl owner  istio wg user experience maintainers test  no      You can gain insights into what individual components are doing by inspecting their  logs   docs ops diagnostic tools component logging   or peering inside via  introspection   docs ops diagnostic tools controlz    If that s insufficient  the steps below explain how to get under the hood   The   istioctl    docs reference commands istioctl  tool is a configuration command line utility that allows service operators to debug and diagnose their Istio service mesh deployments  The Istio project also includes two helpful scripts for  istioctl  that enable auto completion for Bash and Zsh  Both of these scripts provide support for the currently available  istioctl  commands     istioctl  only has auto completion enabled for non deprecated commands       Before you begin  We recommend you use an  istioctl  version that is the same version as your Istio control plane  Using matching versions helps avoid unforeseen issues    If you have already  downloaded the Istio release   docs setup additional setup download istio release    you should already have  istioctl  and do not need to install it again       Install   Install the  istioctl  binary with  curl    1  Download the latest release with the command              curl  sL https   istio io downloadIstioctl   sh         1  Add the  istioctl  client to your path  on a macOS or Linux system              export PATH  HOME  istioctl bin  PATH       1  You can optionally enable the  auto completion option   enabling auto completion  when working with a bash or Zsh console      Get an overview of your mesh  You can get an overview of your mesh using the  proxy status  or  ps  command      istioctl proxy status   If a proxy is missing from the output list it means that it is not currently connected to an istiod instance and so it will not receive any configuration  Additionally  if it is marked stale  it likely means there are networking issues or istiod needs to be scaled      Get proxy configuration    istioctl    docs reference commands istioctl  allows you to retrieve information about proxy configuration using the  proxy config  or  pc  command   For example  to retrieve information about cluster configuration for the Envoy instance in a specific pod      istioctl proxy config cluster  pod name   flags    To retrieve information about bootstrap configuration for the Envoy instance in a specific pod      istioctl proxy config bootstrap  pod name   flags    To retrieve information about listener configuration for the Envoy instance in a specific pod      istioctl proxy config listener  pod name   flags    To retrieve information about route configuration for the Envoy instance in a specific pod      istioctl proxy config route  pod name   flags    To retrieve information about endpoint configuration for the Envoy instance in a specific pod      istioctl proxy config endpoints  pod name   flags    See  Debugging Envoy and Istiod   docs ops diagnostic tools proxy cmd   for more advice on interpreting this information       istioctl  auto completion      If you are using the macOS operating system with the Zsh terminal shell  make sure that the  zsh completions  package is installed  With the  brew  https   brew sh  package manager for macOS  you can check to see if the  zsh completions  package is installed with the following command      brew list zsh completions  usr local Cellar zsh completions 0 34 0 share zsh completions   147 files    If you receive  Error  No such keg   usr local Cellar zsh completion   proceed with installing the  zsh completions  package with the following command      brew install zsh completions   Once the  zsh completions package  has been installed on your macOS system  add the following to your     zshrc  file        if type brew    dev null  then       FPATH   brew   prefix  share zsh completions  FPATH        autoload  Uz compinit       compinit     fi   You may also need to force rebuild  zcompdump       rm  f    zcompdump  compinit   Additionally  if you receive  Zsh compinit  insecure directories  warnings when attempting to load these completions  you may need to run this      chmod  R go w    brew   prefix  share        If you are using a Linux based operating system  you can install the Bash completion package with the  apt get install bash completion  command for Debian based Linux distributions or  yum install bash completion  for RPM based Linux distributions  the two most common occurrences   Once the  bash completion  package has been installed on your Linux system  add the following line to your     bash profile  file        r   usr local etc profile d bash completion sh            usr local etc profile d bash completion sh            Enabling auto completion  To enable  istioctl  completion on your system  follow the steps for your preferred shell    You will need to download the full Istio release containing the auto completion files  in the   tools  directory   If you haven t already done so   download the full release   docs setup additional setup download istio release   now        Installing the bash auto completion file  If you are using bash  the  istioctl  auto completion file is located in the  tools  directory  To use it  copy the  istioctl bash  file to your home directory  then add the following line to source the  istioctl  tab completion file from your   bashrc  file      source   istioctl bash       Installing the Zsh auto completion file  For Zsh users  the  istioctl  auto completion file is located in the  tools  directory  Copy the   istioctl  file to your home directory  or any directory of your choosing  update directory in script snippet below   and source the  istioctl  auto completion file in your   zshrc  file as follows    source    istioctl   You may also add the   istioctl  file to a directory listed in the  fpath  variable  To achieve this  place the   istioctl  file in an existing directory in the  fpath   or create a new directory and add it to the  fpath  variable in your     zshrc  file     If you get an error like  complete 13  command not found  compdef   then add the following to the beginning of your     zshrc  file      autoload  Uz compinit   compinit   If your auto completion is not working  try again after restarting your terminal  If auto completion still does not work  try resetting the completion cache using the above commands in your terminal             Using auto completion  If the  istioctl  completion file has been installed correctly  press the Tab key while writing an  istioctl  command  and it should return a set of command suggestions for you to choose from      istioctl proxy  TAB  proxy config proxy status


istio title Troubleshooting Multicluster test no keywords debug multicluster multi network envoy weight 90 This page describes how to troubleshoot issues with Istio deployed to multiple clusters and or networks owner istio wg environments maintainers Describes tools and techniques to diagnose issues with multicluster and multi network installations

Answer:
---
title: Troubleshooting Multicluster
description: Describes tools and techniques to diagnose issues with multicluster and multi-network installations.
weight: 90
keywords: [debug,multicluster,multi-network,envoy]
owner: istio/wg-environments-maintainers
test: no
---

This page describes how to troubleshoot issues with Istio deployed to multiple clusters and/or networks.
Before reading this, you should take the steps in [Multicluster Installation](/docs/setup/install/multicluster/)
and read the [Deployment Models](/docs/ops/deployment/deployment-models/) guide.

## Cross-Cluster Load Balancing

The most common, but also broad problem with multi-network installations is that cross-cluster load balancing doesn’t work. Usually this manifests itself as only seeing responses from the cluster-local instance of a Service:

$ for i in $(seq 10); do kubectl --context=$CTX_CLUSTER1 -n sample exec curl-dd98b5f48-djwdw -c curl -- curl -s helloworld:5000/hello; done
Hello version: v1, instance: helloworld-v1-578dd69f69-j69pf
Hello version: v1, instance: helloworld-v1-578dd69f69-j69pf
Hello version: v1, instance: helloworld-v1-578dd69f69-j69pf
...

When following the guide to [verify multicluster installation](/docs/setup/install/multicluster/verify/)
we would expect both `v1` and `v2` responses, indicating traffic is going to both clusters.

There are many possible causes to the problem:

### Connectivity and firewall issues

In some environments it may not be apparent that a firewall is blocking traffic between your clusters. It's possible
that `ICMP` (ping) traffic may succeed, but HTTP and other types of traffic do not. This can appear as a timeout, or
in some cases a more confusing error such as:

upstream connect error or disconnect/reset before headers. reset reason: local reset, transport failure reason: TLS error: 268435612:SSL routines:OPENSSL_internal:HTTP_REQUEST

While Istio provides service discovery capabilities to make it easier, cross-cluster traffic should still succeed
if pods in each cluster are on a single network without Istio. To rule out issues with TLS/mTLS, you can do a manual
traffic test using pods without Istio sidecars.

In each cluster, create a new namespace for this test. Do _not_ enable sidecar injection:

$ kubectl create --context="${CTX_CLUSTER1}" namespace uninjected-sample
$ kubectl create --context="${CTX_CLUSTER2}" namespace uninjected-sample

Then deploy the same apps used in [verify multicluster installation](/docs/setup/install/multicluster/verify/):

$ kubectl apply --context="${CTX_CLUSTER1}" \
    -f samples/helloworld/helloworld.yaml \
    -l service=helloworld -n uninjected-sample
$ kubectl apply --context="${CTX_CLUSTER2}" \
    -f samples/helloworld/helloworld.yaml \
    -l service=helloworld -n uninjected-sample
$ kubectl apply --context="${CTX_CLUSTER1}" \
    -f samples/helloworld/helloworld.yaml \
    -l version=v1 -n uninjected-sample
$ kubectl apply --context="${CTX_CLUSTER2}" \
    -f samples/helloworld/helloworld.yaml \
    -l version=v2 -n uninjected-sample
$ kubectl apply --context="${CTX_CLUSTER1}" \
    -f samples/curl/curl.yaml -n uninjected-sample
$ kubectl apply --context="${CTX_CLUSTER2}" \
    -f samples/curl/curl.yaml -n uninjected-sample

Verify that there is a helloworld pod running in `cluster2`, using the `-o wide` flag, so we can get the Pod IP:

$ kubectl --context="${CTX_CLUSTER2}" -n uninjected-sample get pod -o wide
NAME                             READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
curl-557747455f-jdsd8            1/1     Running   0          41s   10.100.0.2   node-2   <none>           <none>
helloworld-v2-54df5f84b-z28p5    1/1     Running   0          43s   10.100.0.1   node-1   <none>           <none>

Take note of the `IP` column for `helloworld`. In this case, it is `10.100.0.1`:

$ REMOTE_POD_IP=10.100.0.1

Next, attempt to send traffic from the `curl` pod in `cluster1` directly to this Pod IP:

$ kubectl exec --context="${CTX_CLUSTER1}" -n uninjected-sample -c curl \
    "$(kubectl get pod --context="${CTX_CLUSTER1}" -n uninjected-sample -l \
    app=curl -o jsonpath='{.items[0].metadata.name}')" \
    -- curl -sS $REMOTE_POD_IP:5000/hello
Hello version: v2, instance: helloworld-v2-54df5f84b-z28p5

If successful, there should be responses only from `helloworld-v2`. Repeat the steps, but send traffic from `cluster2`
to `cluster1`.

If this succeeds, you can rule out connectivity issues. If it does not, the cause of the problem may lie outside your
Istio configuration.

### Locality Load Balancing

[Locality load balancing](/docs/tasks/traffic-management/locality-load-balancing/failover/#configure-locality-failover)
can be used to make clients prefer that traffic go to the nearest destination. If the clusters
are in different localities (region/zone), locality load balancing will prefer the local-cluster and is working as
intended. If locality load balancing is disabled, or the clusters are in the same locality, there may be another issue.

### Trust Configuration

Cross-cluster traffic, as with intra-cluster traffic, relies on a common root of trust between the proxies. The default
Istio installation will use their own individually generated root certificate-authorities. For multi-cluster, we
must manually configure a shared root of trust. Follow Plug-in Certs below or read [Identity and Trust Models](/docs/ops/deployment/deployment-models/#identity-and-trust-models)
to learn more.

**Plug-in Certs:**

To verify certs are configured correctly, you can compare the root-cert in each cluster:

$ diff \
   <(kubectl --context="${CTX_CLUSTER1}" -n istio-system get secret cacerts -ojsonpath='{.data.root-cert\.pem}') \
   <(kubectl --context="${CTX_CLUSTER2}" -n istio-system get secret cacerts -ojsonpath='{.data.root-cert\.pem}')

If the root-certs do not match or the secret does not exist at all, you can follow the [Plugin CA Certs](/docs/tasks/security/cert-management/plugin-ca-cert/)
guide, ensuring to run the steps for every cluster.

### Step-by-step Diagnosis

If you've gone through the sections above and are still having issues, then it's time to dig a little deeper.

The following steps assume you're following the [HelloWorld verification](/docs/setup/install/multicluster/verify/).
Before continuing, make sure both `helloworld` and `curl` are deployed in each cluster.

From each cluster, find the endpoints the `curl` service has for `helloworld`:

$ istioctl --context $CTX_CLUSTER1 proxy-config endpoint curl-dd98b5f48-djwdw.sample | grep helloworld

Troubleshooting information differs based on the cluster that is the source of traffic:

$ istioctl --context $CTX_CLUSTER1 proxy-config endpoint curl-dd98b5f48-djwdw.sample | grep helloworld
10.0.0.11:5000                   HEALTHY     OK                outbound|5000||helloworld.sample.svc.cluster.local

Only one endpoint is shown, indicating the control plane cannot read endpoints from the remote cluster.
Verify that remote secrets are configured properly.

$ kubectl get secrets --context=$CTX_CLUSTER1 -n istio-system -l "istio/multiCluster=true"

* If the secret is missing, create it.
* If the secret is present:
    * Look at the config in the secret. Make sure the cluster name is used as the data key for the remote `kubeconfig`.
    * If the secret looks correct, check the logs of `istiod` for connectivity or permissions issues reaching the
     remote Kubernetes API server. Log messages may include `Failed to add remote cluster from secret` along with an
     error reason.

$ istioctl --context $CTX_CLUSTER2 proxy-config endpoint curl-dd98b5f48-djwdw.sample | grep helloworld
10.0.1.11:5000                   HEALTHY     OK                outbound|5000||helloworld.sample.svc.cluster.local

Only one endpoint is shown, indicating the control plane cannot read endpoints from the remote cluster.
Verify that remote secrets are configured properly.

$ kubectl get secrets --context=$CTX_CLUSTER1 -n istio-system -l "istio/multiCluster=true"

* If the secret is missing, create it.
* If the secret is present and the endpoint is a Pod in the **primary** cluster:
    * Look at the config in the secret. Make sure the cluster name is used as the data key for the remote `kubeconfig`.
    * If the secret looks correct, check the logs of `istiod` for connectivity or permissions issues reaching the
     remote Kubernetes API server. Log messages may include `Failed to add remote cluster from secret` along with an
     error reason.
* If the secret is present and the endpoint is a Pod in the **remote** cluster:
    * The proxy is reading configuration from an istiod inside the remote cluster. When a remote cluster has an in
     -cluster istiod,  it is only meant for sidecar injection and CA. You can verify this is the problem by looking
     for a Service named `istiod-remote` in the `istio-system` namespace. If it's missing, reinstall making sure
     `values.global.remotePilotAddress` is set.

The steps for Primary and Remote clusters still apply for multi-network, although multi-network has an additional case:

$ istioctl --context $CTX_CLUSTER1 proxy-config endpoint curl-dd98b5f48-djwdw.sample | grep helloworld
10.0.5.11:5000                   HEALTHY     OK                outbound|5000||helloworld.sample.svc.cluster.local
10.0.6.13:5000                   HEALTHY     OK                outbound|5000||helloworld.sample.svc.cluster.local

In multi-network, we expect one of the endpoint IPs to match the remote cluster's east-west gateway public IP. Seeing
multiple Pod IPs indicates one of two things:

* The address of the gateway for the remote network cannot be determined.
* The network of either the client or server pod cannot be determined.

**The address of the gateway for the remote network cannot be determined:**

In the remote cluster that cannot be reached, check that the Service has an External IP:

$ kubectl -n istio-system get service -l "istio=eastwestgateway"
NAME                      TYPE           CLUSTER-IP    EXTERNAL-IP      PORT(S)                                                           AGE
istio-eastwestgateway    LoadBalancer   10.8.17.119   <PENDING>        15021:31781/TCP,15443:30498/TCP,15012:30879/TCP,15017:30336/TCP   76m

If the `EXTERNAL-IP` is stuck in `<PENDING>`, the environment may not support `LoadBalancer` services. In this case, it
may be necessary to customize the `spec.externalIPs` section of the Service to manually give the Gateway an IP reachable
from outside the cluster.

If the external IP is present, check that the Service includes a `topology.istio.io/network` label with the correct
value. If that is incorrect, reinstall the gateway and make sure to set the --network flag on the generation script.

**The network of either the client or server cannot be determined.**

On the source pod, check the proxy metadata.

$ kubectl get pod $CURL_POD_NAME \
  -o jsonpath="{.spec.containers[*].env[?(@.name=='ISTIO_META_NETWORK')].value}"

$ kubectl get pod $HELLOWORLD_POD_NAME \
  -o jsonpath="{.metadata.labels.topology\.istio\.io/network}"

If either of these values aren't set, or have the wrong value, istiod may treat the source and client proxies as being on the same network and send network-local endpoints.
When these aren't set, check that `values.global.network` was set properly during install, or that the injection webhook is configured correctly.

Istio determines the network of a Pod using the `topology.istio.io/network` label which is set during injection. For
non-injected Pods, Istio relies on the `topology.istio.io/network` label set on the system namespace in the cluster.

In each cluster, check the network:

$ kubectl --context="${CTX_CLUSTER1}" get ns istio-system -ojsonpath='{.metadata.labels.topology\.istio\.io/network}'

If the above command doesn't output the expected network name, set the label:

$ kubectl --context="${CTX_CLUSTER1}" label namespace istio-system topology.istio.io/network=network1

title  Troubleshooting Multicluster description  Describes tools and techniques to diagnose issues with multicluster and multi network installations  weight  90 keywords   debug multicluster multi network envoy  owner  istio wg environments maintainers test  no      This page describes how to troubleshoot issues with Istio deployed to multiple clusters and or networks  Before reading this  you should take the steps in  Multicluster Installation   docs setup install multicluster   and read the  Deployment Models   docs ops deployment deployment models   guide      Cross Cluster Load Balancing  The most common  but also broad problem with multi network installations is that cross cluster load balancing doesn t work  Usually this manifests itself as only seeing responses from the cluster local instance of a Service      for i in   seq 10   do kubectl   context  CTX CLUSTER1  n sample exec curl dd98b5f48 djwdw  c curl    curl  s helloworld 5000 hello  done Hello version  v1  instance  helloworld v1 578dd69f69 j69pf Hello version  v1  instance  helloworld v1 578dd69f69 j69pf Hello version  v1  instance  helloworld v1 578dd69f69 j69pf       When following the guide to  verify multicluster installation   docs setup install multicluster verify   we would expect both  v1  and  v2  responses  indicating traffic is going to both clusters   There are many possible causes to the problem       Connectivity and firewall issues  In some environments it may not be apparent that a firewall is blocking traffic between your clusters  It s possible that  ICMP   ping  traffic may succeed  but HTTP and other types of traffic do not  This can appear as a timeout  or in some cases a more confusing error such as    upstream connect error or disconnect reset before headers  reset reason  local reset  transport failure reason  TLS error  268435612 SSL routines OPENSSL internal HTTP REQUEST   While Istio provides service discovery capabilities to make it easier  cross cluster traffic should still succeed if pods in each cluster are on a single network without Istio  To rule out issues with TLS mTLS  you can do a manual traffic test using pods without Istio sidecars   In each cluster  create a new namespace for this test  Do  not  enable sidecar injection      kubectl create   context    CTX CLUSTER1   namespace uninjected sample   kubectl create   context    CTX CLUSTER2   namespace uninjected sample   Then deploy the same apps used in  verify multicluster installation   docs setup install multicluster verify        kubectl apply   context    CTX CLUSTER1          f samples helloworld helloworld yaml        l service helloworld  n uninjected sample   kubectl apply   context    CTX CLUSTER2          f samples helloworld helloworld yaml        l service helloworld  n uninjected sample   kubectl apply   context    CTX CLUSTER1          f samples helloworld helloworld yaml        l version v1  n uninjected sample   kubectl apply   context    CTX CLUSTER2          f samples helloworld helloworld yaml        l version v2  n uninjected sample   kubectl apply   context    CTX CLUSTER1          f samples curl curl yaml  n uninjected sample   kubectl apply   context    CTX CLUSTER2          f samples curl curl yaml  n uninjected sample   Verify that there is a helloworld pod running in  cluster2   using the   o wide  flag  so we can get the Pod IP      kubectl   context    CTX CLUSTER2    n uninjected sample get pod  o wide NAME                             READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES curl 557747455f jdsd8            1 1     Running   0          41s   10 100 0 2   node 2    none             none  helloworld v2 54df5f84b z28p5    1 1     Running   0          43s   10 100 0 1   node 1    none             none    Take note of the  IP  column for  helloworld   In this case  it is  10 100 0 1       REMOTE POD IP 10 100 0 1   Next  attempt to send traffic from the  curl  pod in  cluster1  directly to this Pod IP      kubectl exec   context    CTX CLUSTER1    n uninjected sample  c curl          kubectl get pod   context    CTX CLUSTER1    n uninjected sample  l       app curl  o jsonpath    items 0  metadata name              curl  sS  REMOTE POD IP 5000 hello Hello version  v2  instance  helloworld v2 54df5f84b z28p5   If successful  there should be responses only from  helloworld v2   Repeat the steps  but send traffic from  cluster2  to  cluster1    If this succeeds  you can rule out connectivity issues  If it does not  the cause of the problem may lie outside your Istio configuration       Locality Load Balancing   Locality load balancing   docs tasks traffic management locality load balancing failover  configure locality failover  can be used to make clients prefer that traffic go to the nearest destination  If the clusters are in different localities  region zone   locality load balancing will prefer the local cluster and is working as intended  If locality load balancing is disabled  or the clusters are in the same locality  there may be another issue       Trust Configuration  Cross cluster traffic  as with intra cluster traffic  relies on a common root of trust between the proxies  The default Istio installation will use their own individually generated root certificate authorities  For multi cluster  we must manually configure a shared root of trust  Follow Plug in Certs below or read  Identity and Trust Models   docs ops deployment deployment models  identity and trust models  to learn more     Plug in Certs     To verify certs are configured correctly  you can compare the root cert in each cluster      diff        kubectl   context    CTX CLUSTER1    n istio system get secret cacerts  ojsonpath    data root cert  pem           kubectl   context    CTX CLUSTER2    n istio system get secret cacerts  ojsonpath    data root cert  pem      If the root certs do not match or the secret does not exist at all  you can follow the  Plugin CA Certs   docs tasks security cert management plugin ca cert   guide  ensuring to run the steps for every cluster       Step by step Diagnosis  If you ve gone through the sections above and are still having issues  then it s time to dig a little deeper   The following steps assume you re following the  HelloWorld verification   docs setup install multicluster verify    Before continuing  make sure both  helloworld  and  curl  are deployed in each cluster   From each cluster  find the endpoints the  curl  service has for  helloworld       istioctl   context  CTX CLUSTER1 proxy config endpoint curl dd98b5f48 djwdw sample   grep helloworld   Troubleshooting information differs based on the cluster that is the source of traffic          istioctl   context  CTX CLUSTER1 proxy config endpoint curl dd98b5f48 djwdw sample   grep helloworld 10 0 0 11 5000                   HEALTHY     OK                outbound 5000  helloworld sample svc cluster local   Only one endpoint is shown  indicating the control plane cannot read endpoints from the remote cluster  Verify that remote secrets are configured properly      kubectl get secrets   context  CTX CLUSTER1  n istio system  l  istio multiCluster true      If the secret is missing  create it    If the secret is present        Look at the config in the secret  Make sure the cluster name is used as the data key for the remote  kubeconfig         If the secret looks correct  check the logs of  istiod  for connectivity or permissions issues reaching the      remote Kubernetes API server  Log messages may include  Failed to add remote cluster from secret  along with an      error reason          istioctl   context  CTX CLUSTER2 proxy config endpoint curl dd98b5f48 djwdw sample   grep helloworld 10 0 1 11 5000                   HEALTHY     OK                outbound 5000  helloworld sample svc cluster local   Only one endpoint is shown  indicating the control plane cannot read endpoints from the remote cluster  Verify that remote secrets are configured properly      kubectl get secrets   context  CTX CLUSTER1  n istio system  l  istio multiCluster true      If the secret is missing  create it    If the secret is present and the endpoint is a Pod in the   primary   cluster        Look at the config in the secret  Make sure the cluster name is used as the data key for the remote  kubeconfig         If the secret looks correct  check the logs of  istiod  for connectivity or permissions issues reaching the      remote Kubernetes API server  Log messages may include  Failed to add remote cluster from secret  along with an      error reason    If the secret is present and the endpoint is a Pod in the   remote   cluster        The proxy is reading configuration from an istiod inside the remote cluster  When a remote cluster has an in       cluster istiod   it is only meant for sidecar injection and CA  You can verify this is the problem by looking      for a Service named  istiod remote  in the  istio system  namespace  If it s missing  reinstall making sure       values global remotePilotAddress  is set       The steps for Primary and Remote clusters still apply for multi network  although multi network has an additional case      istioctl   context  CTX CLUSTER1 proxy config endpoint curl dd98b5f48 djwdw sample   grep helloworld 10 0 5 11 5000                   HEALTHY     OK                outbound 5000  helloworld sample svc cluster local 10 0 6 13 5000                   HEALTHY     OK                outbound 5000  helloworld sample svc cluster local   In multi network  we expect one of the endpoint IPs to match the remote cluster s east west gateway public IP  Seeing multiple Pod IPs indicates one of two things     The address of the gateway for the remote network cannot be determined    The network of either the client or server pod cannot be determined     The address of the gateway for the remote network cannot be determined     In the remote cluster that cannot be reached  check that the Service has an External IP      kubectl  n istio system get service  l  istio eastwestgateway  NAME                      TYPE           CLUSTER IP    EXTERNAL IP      PORT S                                                            AGE istio eastwestgateway    LoadBalancer   10 8 17 119    PENDING         15021 31781 TCP 15443 30498 TCP 15012 30879 TCP 15017 30336 TCP   76m   If the  EXTERNAL IP  is stuck in   PENDING    the environment may not support  LoadBalancer  services  In this case  it may be necessary to customize the  spec externalIPs  section of the Service to manually give the Gateway an IP reachable from outside the cluster   If the external IP is present  check that the Service includes a  topology istio io network  label with the correct value  If that is incorrect  reinstall the gateway and make sure to set the   network flag on the generation script     The network of either the client or server cannot be determined     On the source pod  check the proxy metadata      kubectl get pod  CURL POD NAME      o jsonpath    spec containers    env     name   ISTIO META NETWORK    value        kubectl get pod  HELLOWORLD POD NAME      o jsonpath    metadata labels topology  istio  io network     If either of these values aren t set  or have the wrong value  istiod may treat the source and client proxies as being on the same network and send network local endpoints  When these aren t set  check that  values global network  was set properly during install  or that the injection webhook is configured correctly   Istio determines the network of a Pod using the  topology istio io network  label which is set during injection  For non injected Pods  Istio relies on the  topology istio io network  label set on the system namespace in the cluster   In each cluster  check the network      kubectl   context    CTX CLUSTER1   get ns istio system  ojsonpath    metadata labels topology  istio  io network     If the above command doesn t output the expected network name  set the label      kubectl   context    CTX CLUSTER1   label namespace istio system topology istio io network network1


istio keywords debug virtual machines envoy weight 80 This page describes how to troubleshoot issues with Istio deployed to Virtual Machines title Debugging Virtual Machines Describes tools and techniques to diagnose issues with Virtual Machines owner istio wg environments maintainers test n a

Answer:
---
title: Debugging Virtual Machines
description: Describes tools and techniques to diagnose issues with Virtual Machines.
weight: 80
keywords: [debug,virtual-machines,envoy]
owner: istio/wg-environments-maintainers
test: n/a
---

This page describes how to troubleshoot issues with Istio deployed to Virtual Machines.
Before reading this, you should take the steps in [Virtual Machine Installation](/docs/setup/install/virtual-machine/).
Additionally, [Virtual Machine Architecture](/docs/ops/deployment/vm-architecture/) can help you understand how the components interact.

Troubleshooting an Istio Virtual Machine installation is similar to troubleshooting issues with proxies running inside Kubernetes, but there are some key differences to be aware of.

While much of the same information is available on both platforms, accessing this information differs.

## Monitoring health

The Istio sidecar is typically run as a `systemd` unit. To ensure its running properly, you can check that status:

$ systemctl status istio

Additionally, the sidecar health can be programmatically check at its health endpoint:

$ curl localhost:15021/healthz/ready -I

## Logs

Logs for the Istio proxy can be found in a few places.

To access the `systemd` logs, which has details about the initialization of the proxy:

$ journalctl -f -u istio -n 1000

The proxy will redirect `stderr` and `stdout` to `/var/log/istio/istio.err.log` and  `/var/log/istio/istio.log`, respectively.
To view these in a format similar to `kubectl`:

$ tail /var/log/istio/istio.err.log /var/log/istio/istio.log -Fq -n 100

Log levels can be modified by changing the `cluster.env` configuration file. Make sure to restart `istio` if it is already running:

$ echo "ISTIO_AGENT_FLAGS=\"--log_output_level=dns:debug --proxyLogLevel=debug\"" >> /var/lib/istio/envoy/cluster.env
$ systemctl restart istio

## Iptables

To ensure `iptables` rules have been successfully applied:

$ sudo iptables-save
...
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT

## Istioctl

Most `istioctl` commands will function properly with virtual machines. For example, `istioctl proxy-status` can be used to view all connected proxies:

$ istioctl proxy-status
NAME           CDS        LDS        EDS        RDS      ISTIOD                    VERSION
vm-1.default   SYNCED     SYNCED     SYNCED     SYNCED   istiod-789ffff8-f2fkt

However, `istioctl proxy-config` relies on functionality in Kubernetes to connect to a proxy, which will not work for virtual machines.
Instead, a file containing the configuration dump from Envoy can be passed. For example:

$ curl -s localhost:15000/config_dump | istioctl proxy-config clusters --file -
SERVICE FQDN                            PORT      SUBSET  DIRECTION     TYPE
istiod.istio-system.svc.cluster.local   443       -       outbound      EDS
istiod.istio-system.svc.cluster.local   15010     -       outbound      EDS
istiod.istio-system.svc.cluster.local   15012     -       outbound      EDS
istiod.istio-system.svc.cluster.local   15014     -       outbound      EDS

## Automatic registration

When a virtual machine connects to Istiod, a `WorkloadEntry` will automatically be created. This enables
the virtual machine to become a part of a `Service`, similar to an `Endpoint` in Kubernetes.

To check these are created correctly:

$ kubectl get workloadentries
NAME             AGE   ADDRESS
vm-10.128.0.50   14m   10.128.0.50

## Certificates

Virtual machines handle certificates differently than Kubernetes Pods, which use a Kubernetes-provided service account token
to authenticate and renew mTLS certificates. Instead, existing mTLS credentials are used to authenticate with the certificate authority and
renew certificates.

The status of these certificates can be viewed in the same way as in Kubernetes:

$ curl -s localhost:15000/config_dump | ./istioctl proxy-config secret --file -
RESOURCE NAME     TYPE           STATUS     VALID CERT     SERIAL NUMBER                               NOT AFTER                NOT BEFORE
default           Cert Chain     ACTIVE     true           251932493344649542420616421203546836446     2021-01-29T18:07:21Z     2021-01-28T18:07:21Z
ROOTCA            CA             ACTIVE     true           81663936513052336343895977765039160718      2031-01-26T17:54:44Z     2021-01-28T17:54:44Z

Additionally, these are persisted to disk to ensure downtime or restarts do not lose state.

$ ls /etc/certs
cert-chain.pem  key.pem  root-cert.pem

title  Debugging Virtual Machines description  Describes tools and techniques to diagnose issues with Virtual Machines  weight  80 keywords   debug virtual machines envoy  owner  istio wg environments maintainers test  n a      This page describes how to troubleshoot issues with Istio deployed to Virtual Machines  Before reading this  you should take the steps in  Virtual Machine Installation   docs setup install virtual machine    Additionally   Virtual Machine Architecture   docs ops deployment vm architecture   can help you understand how the components interact   Troubleshooting an Istio Virtual Machine installation is similar to troubleshooting issues with proxies running inside Kubernetes  but there are some key differences to be aware of   While much of the same information is available on both platforms  accessing this information differs      Monitoring health  The Istio sidecar is typically run as a  systemd  unit  To ensure its running properly  you can check that status      systemctl status istio   Additionally  the sidecar health can be programmatically check at its health endpoint      curl localhost 15021 healthz ready  I      Logs  Logs for the Istio proxy can be found in a few places   To access the  systemd  logs  which has details about the initialization of the proxy      journalctl  f  u istio  n 1000   The proxy will redirect  stderr  and  stdout  to   var log istio istio err log  and    var log istio istio log   respectively  To view these in a format similar to  kubectl       tail  var log istio istio err log  var log istio istio log  Fq  n 100   Log levels can be modified by changing the  cluster env  configuration file  Make sure to restart  istio  if it is already running      echo  ISTIO AGENT FLAGS     log output level dns debug   proxyLogLevel debug        var lib istio envoy cluster env   systemctl restart istio      Iptables  To ensure  iptables  rules have been successfully applied      sudo iptables save      A ISTIO OUTPUT  d 127 0 0 1 32  j RETURN  A ISTIO OUTPUT  j ISTIO REDIRECT      Istioctl  Most  istioctl  commands will function properly with virtual machines  For example   istioctl proxy status  can be used to view all connected proxies      istioctl proxy status NAME           CDS        LDS        EDS        RDS      ISTIOD                    VERSION vm 1 default   SYNCED     SYNCED     SYNCED     SYNCED   istiod 789ffff8 f2fkt        However   istioctl proxy config  relies on functionality in Kubernetes to connect to a proxy  which will not work for virtual machines  Instead  a file containing the configuration dump from Envoy can be passed  For example      curl  s localhost 15000 config dump   istioctl proxy config clusters   file   SERVICE FQDN                            PORT      SUBSET  DIRECTION     TYPE istiod istio system svc cluster local   443               outbound      EDS istiod istio system svc cluster local   15010             outbound      EDS istiod istio system svc cluster local   15012             outbound      EDS istiod istio system svc cluster local   15014             outbound      EDS      Automatic registration  When a virtual machine connects to Istiod  a  WorkloadEntry  will automatically be created  This enables the virtual machine to become a part of a  Service   similar to an  Endpoint  in Kubernetes   To check these are created correctly      kubectl get workloadentries NAME             AGE   ADDRESS vm 10 128 0 50   14m   10 128 0 50      Certificates  Virtual machines handle certificates differently than Kubernetes Pods  which use a Kubernetes provided service account token to authenticate and renew mTLS certificates  Instead  existing mTLS credentials are used to authenticate with the certificate authority and renew certificates   The status of these certificates can be viewed in the same way as in Kubernetes      curl  s localhost 15000 config dump     istioctl proxy config secret   file   RESOURCE NAME     TYPE           STATUS     VALID CERT     SERIAL NUMBER                               NOT AFTER                NOT BEFORE default           Cert Chain     ACTIVE     true           251932493344649542420616421203546836446     2021 01 29T18 07 21Z     2021 01 28T18 07 21Z ROOTCA            CA             ACTIVE     true           81663936513052336343895977765039160718      2031 01 26T17 54 44Z     2021 01 28T17 54 44Z   Additionally  these are persisted to disk to ensure downtime or restarts do not lose state      ls  etc certs cert chain pem  key pem  root cert pem


istio weight 31 How to configure Istio to integrate with SPIRE to get cryptographic identities through Envoy s SDS API title SPIRE owner istio wg networking maintainers aliases test yes keywords kubernetes spiffe spire

Answer:
---
title: SPIRE
description: How to configure Istio to integrate with SPIRE to get cryptographic identities through Envoy's SDS API.
weight: 31
keywords: [kubernetes,spiffe,spire]
aliases:
owner: istio/wg-networking-maintainers
test: yes
---

[SPIRE](https://spiffe.io/docs/latest/spire-about/spire-concepts/) is a production-ready implementation of the SPIFFE specification that performs node and workload attestation in order to securely
issue cryptographic identities to workloads running in heterogeneous environments. SPIRE can be configured as a source of cryptographic identities for Istio workloads through an integration with
[Envoy's SDS API](https://www.envoyproxy.io/docs/envoy/latest/configuration/security/secret). Istio can detect the existence of a UNIX Domain Socket that implements the Envoy SDS API on a defined
socket path, allowing Envoy to communicate and fetch identities directly from it.

This integration with SPIRE provides flexible attestation options not available with the default Istio identity management while harnessing Istio's powerful service management.
For example, SPIRE's plugin architecture enables diverse workload attestation options beyond the Kubernetes namespace and service account attestation offered by Istio.
SPIRE's node attestation extends attestation to the physical or virtual hardware on which workloads run.

For a quick demo of how this SPIRE integration with Istio works, see [Integrating SPIRE as a CA through Envoy's SDS API](/samples/security/spire).

## Install SPIRE

We recommend you follow SPIRE's installation instructions and best practices for installing SPIRE, and for deploying SPIRE in production environments.

For the examples in this guide, the [SPIRE Helm charts](https://artifacthub.io/packages/helm/spiffe/spire) will be used with upstream defaults, to focus on just the configuration necessary to integrate SPIRE and Istio.

$ helm upgrade --install -n spire-server spire-crds spire-crds --repo https://spiffe.github.io/helm-charts-hardened/ --create-namespace

$ helm upgrade --install -n spire-server spire spire --repo https://spiffe.github.io/helm-charts-hardened/ --wait --set global.spire.trustDomain="example.org"

See the [SPIRE Helm chart](https://artifacthub.io/packages/helm/spiffe/spire) documentation for other values you can configure for your installation.

It is important that SPIRE and Istio are configured with the exact same trust domain, to prevent authentication and authorization errors, and that the [SPIFFE CSI driver](https://github.com/spiffe/spiffe-csi) is enabled and installed.

By default, the above will also install:

- The [SPIFFE CSI driver](https://github.com/spiffe/spiffe-csi), which is used to mount an Envoy-compatible SDS socket into proxies. Using the SPIFFE CSI driver to mount SDS sockets is strongly recommended by both Istio and SPIRE, as `hostMounts` are a larger security risk and introduce operational hurdles. This guide assumes the use of the SPIFFE CSI driver.

- The [SPIRE Controller Manager](https://github.com/spiffe/spire-controller-manager), which eases the creation of SPIFFE registrations for workloads.

## Register workloads

By design, SPIRE only grants identities to workloads that have been registered with the SPIRE server; this includes user workloads, as well as Istio components. Istio sidecars and gateways, once configured for SPIRE integration, cannot get identities, and therefore cannot reach READY status, unless there is a preexisting, matching SPIRE registration created for them ahead of time.

See the [SPIRE docs on registering workloads](https://spiffe.io/docs/latest/deploying/registering/) for more information on using multiple selectors to strengthen attestation criteria, and the selectors available.

This section describes the options available for registering Istio workloads in a SPIRE Server and provides some example workload registrations.

Istio currently requires a specific SPIFFE ID format for workloads. All registrations must follow the Istio SPIFFE ID pattern: `spiffe://<trust.domain>/ns/<namespace>/sa/<service-account>`

### Option 1: Auto-registration using the SPIRE Controller Manager

New entries will be automatically registered for each new pod that matches the selector defined in a [ClusterSPIFFEID](https://github.com/spiffe/spire-controller-manager/blob/main/docs/clusterspiffeid-crd.md) custom resource.

Both Istio sidecars and Istio gateways need to be registered with SPIRE, so that they can request identities.

#### Istio Gateway `ClusterSPIFFEID`

The following will create a `ClusterSPIFFEID`, which will auto-register any Istio Ingress gateway pod with SPIRE if it is scheduled into the `istio-system` namespace, and has a service account named `istio-ingressgateway-service-account`. These selectors are used as a simple example; consult the [SPIRE Controller Manager documentation](https://github.com/spiffe/spire-controller-manager/blob/main/docs/clusterspiffeid-crd.md) for more details.

$ kubectl apply -f - <<EOF
apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterSPIFFEID
metadata:
  name: istio-ingressgateway-reg
spec:
  spiffeIDTemplate: "spiffe:///ns//sa/"
  workloadSelectorTemplates:
    - "k8s:ns:istio-system"
    - "k8s:sa:istio-ingressgateway-service-account"
EOF

#### Istio Sidecar `ClusterSPIFFEID`

The following will create a `ClusterSPIFFEID` which will auto-register any pod with the `spiffe.io/spire-managed-identity: true` label that is deployed into the `default` namespace with SPIRE. These selectors are used as a simple example; consult the [SPIRE Controller Manager documentation](https://github.com/spiffe/spire-controller-manager/blob/main/docs/clusterspiffeid-crd.md) for more details.

$ kubectl apply -f - <<EOF
apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterSPIFFEID
metadata:
  name: istio-sidecar-reg
spec:
  spiffeIDTemplate: "spiffe:///ns//sa/"
  podSelector:
    matchLabels:
      spiffe.io/spire-managed-identity: "true"
  workloadSelectorTemplates:
    - "k8s:ns:default"
EOF

### Option 2: Manual Registration

If you wish to manually create your SPIRE registrations, rather than use the SPIRE Controller Manager mentioned in [the recommended option](#option-1-auto-registration-using-the-spire-controller-manager), refer to the [SPIRE documentation on manual registration](https://spiffe.io/docs/latest/deploying/registering/).

Below are the equivalent manual registrations based off the automatic registrations in [Option 1](#option-1-auto-registration-using-the-spire-controller-manager). The following steps assume you have [already followed the SPIRE documentation to manually register your SPIRE agent and node attestation](https://spiffe.io/docs/latest/deploying/registering/#1-defining-the-spiffe-id-of-the-agent) and that your SPIRE agent was registered with the SPIFFE identity `spiffe://example.org/ns/spire/sa/spire-agent`.

1. Get the `spire-server` pod:

    $ SPIRE_SERVER_POD=$(kubectl get pod -l statefulset.kubernetes.io/pod-name=spire-server-0 -n spire-server -o jsonpath="{.items[0].metadata.name}")

1. Register an entry for the Istio Ingress gateway pod:

    $ kubectl exec -n spire "$SPIRE_SERVER_POD" -- \
    /opt/spire/bin/spire-server entry create \
        -spiffeID spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account \
        -parentID spiffe://example.org/ns/spire/sa/spire-agent \
        -selector k8s:sa:istio-ingressgateway-service-account \
        -selector k8s:ns:istio-system \
        -socketPath /run/spire/sockets/server.sock

    Entry ID         : 6f2fe370-5261-4361-ac36-10aae8d91ff7
    SPIFFE ID        : spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account
    Parent ID        : spiffe://example.org/ns/spire/sa/spire-agent
    Revision         : 0
    TTL              : default
    Selector         : k8s:ns:istio-system
    Selector         : k8s:sa:istio-ingressgateway-service-account

1. Register an entry for workloads injected with an Istio sidecar:

    $ kubectl exec -n spire "$SPIRE_SERVER_POD" -- \
    /opt/spire/bin/spire-server entry create \
        -spiffeID spiffe://example.org/ns/default/sa/curl \
        -parentID spiffe://example.org/ns/spire/sa/spire-agent \
        -selector k8s:ns:default \
        -selector k8s:pod-label:spiffe.io/spire-managed-identity:true \
        -socketPath /run/spire/sockets/server.sock

## Install Istio

1. [Download the Istio release](/docs/setup/additional-setup/download-istio-release/).

1. Create the Istio configuration with custom patches for the Ingress Gateway and `istio-proxy`. The Ingress Gateway component includes the `spiffe.io/spire-managed-identity: "true"` label.

    $ cat <<EOF > ./istio.yaml
    apiVersion: install.istio.io/v1alpha1
    kind: IstioOperator
    metadata:
      namespace: istio-system
    spec:
      profile: default
      meshConfig:
        trustDomain: example.org
      values:
        # This is used to customize the sidecar template.
        # It adds both the label to indicate that SPIRE should manage the
        # identity of this pod, as well as the CSI driver mounts.
        sidecarInjectorWebhook:
          templates:
            spire: |
              labels:
                spiffe.io/spire-managed-identity: "true"
              spec:
                containers:
                - name: istio-proxy
                  volumeMounts:
                  - name: workload-socket
                    mountPath: /run/secrets/workload-spiffe-uds
                    readOnly: true
                volumes:
                  - name: workload-socket
                    csi:
                      driver: "csi.spiffe.io"
                      readOnly: true
      components:
        ingressGateways:
          - name: istio-ingressgateway
            enabled: true
            label:
              istio: ingressgateway
            k8s:
              overlays:
                # This is used to customize the ingress gateway template.
                # It adds the CSI driver mounts, as well as an init container
                # to stall gateway startup until the CSI driver mounts the socket.
                - apiVersion: apps/v1
                  kind: Deployment
                  name: istio-ingressgateway
                  patches:
                    - path: spec.template.spec.volumes.[name:workload-socket]
                      value:
                        name: workload-socket
                        csi:
                          driver: "csi.spiffe.io"
                          readOnly: true
                    - path: spec.template.spec.containers.[name:istio-proxy].volumeMounts.[name:workload-socket]
                      value:
                        name: workload-socket
                        mountPath: "/run/secrets/workload-spiffe-uds"
                        readOnly: true
                    - path: spec.template.spec.initContainers
                      value:
                        - name: wait-for-spire-socket
                          image: busybox:1.36
                          volumeMounts:
                            - name: workload-socket
                              mountPath: /run/secrets/workload-spiffe-uds
                              readOnly: true
                          env:
                            - name: CHECK_FILE
                              value: /run/secrets/workload-spiffe-uds/socket
                          command:
                            - sh
                            - "-c"
                            - |-
                              echo "$(date -Iseconds)" Waiting for: ${CHECK_FILE}
                              while [[ ! -e ${CHECK_FILE} ]] ; do
                                echo "$(date -Iseconds)" File does not exist: ${CHECK_FILE}
                                sleep 15
                              done
                              ls -l ${CHECK_FILE}
    EOF

1. Apply the configuration:

    $ istioctl install --skip-confirmation -f ./istio.yaml

1. Check Ingress Gateway pod state:

    $ kubectl get pods -n istio-system
    NAME                                    READY   STATUS    RESTARTS   AGE
    istio-ingressgateway-5b45864fd4-lgrxs   1/1     Running   0          17s
    istiod-989f54d9c-sg7sn                  1/1     Running   0          23s

    The Ingress Gateway pod is `Ready` since the corresponding registration entry is automatically created for it on the SPIRE Server. Envoy is able to fetch cryptographic identities from SPIRE.

    This configuration also adds an `initContainer` to the gateway that will wait for SPIRE to create the UNIX Domain Socket before starting the `istio-proxy`. If the SPIRE agent is not ready, or has not been properly configured with the same socket path, the Ingress Gateway `initContainer` will wait forever.

1. Deploy an example workload:

    $ istioctl kube-inject --filename @samples/security/spire/curl-spire.yaml@ | kubectl apply -f -

    In addition to needing `spiffe.io/spire-managed-identity` label, the workload will need the SPIFFE CSI Driver volume to access the SPIRE Agent socket. To accomplish this,
    you can leverage the `spire` pod annotation template from the [Install Istio](#install-istio) section or add the CSI volume to
    the deployment spec of your workload. Both of these alternatives are highlighted on the example snippet below:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: curl
    spec:
      replicas: 1
      selector:
          matchLabels:
            app: curl
      template:
          metadata:
            labels:
              app: curl
            # Injects custom sidecar template
            annotations:
                inject.istio.io/templates: "sidecar,spire"
          spec:
            terminationGracePeriodSeconds: 0
            serviceAccountName: curl
            containers:
            - name: curl
              image: curlimages/curl
              command: ["/bin/sleep", "3650d"]
              imagePullPolicy: IfNotPresent
              volumeMounts:
                - name: tmp
                  mountPath: /tmp
              securityContext:
                runAsUser: 1000
            volumes:
              - name: tmp
                emptyDir: {}
              # CSI volume
              - name: workload-socket
                csi:
                  driver: "csi.spiffe.io"
                  readOnly: true

The Istio configuration shares the `spiffe-csi-driver` with the Ingress Gateway and the sidecars that are going to be injected on workload pods, granting them access to the SPIRE Agent's UNIX Domain Socket.

See [Verifying that identities were created for workloads](#verifying-that-identities-were-created-for-workloads)
to check issued identities.

## Verifying that identities were created for workloads

Use the following command to confirm that identities were created for the workloads:

$ kubectl exec -t "$SPIRE_SERVER_POD" -n spire-server -c spire-server -- ./bin/spire-server entry show
Found 2 entries
Entry ID         : c8dfccdc-9762-4762-80d3-5434e5388ae7
SPIFFE ID        : spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account
Parent ID        : spiffe://example.org/spire/agent/k8s_psat/demo-cluster/bea19580-ae04-4679-a22e-472e18ca4687
Revision         : 0
X509-SVID TTL    : default
JWT-SVID TTL     : default
Selector         : k8s:pod-uid:88b71387-4641-4d9c-9a89-989c88f7509d

Entry ID         : af7b53dc-4cc9-40d3-aaeb-08abbddd8e54
SPIFFE ID        : spiffe://example.org/ns/default/sa/curl
Parent ID        : spiffe://example.org/spire/agent/k8s_psat/demo-cluster/bea19580-ae04-4679-a22e-472e18ca4687
Revision         : 0
X509-SVID TTL    : default
JWT-SVID TTL     : default
Selector         : k8s:pod-uid:ee490447-e502-46bd-8532-5a746b0871d6

Check the Ingress-gateway pod state:

$ kubectl get pods -n istio-system
NAME                                    READY   STATUS    RESTARTS   AGE
istio-ingressgateway-5b45864fd4-lgrxs   1/1     Running   0          60s
istiod-989f54d9c-sg7sn                  1/1     Running   0          45s

After registering an entry for the Ingress-gateway pod, Envoy receives the identity issued by SPIRE and uses it for all TLS and mTLS communications.

### Check that the workload identity was issued by SPIRE

1. Get pod information:

    $ CURL_POD=$(kubectl get pod -l app=curl -o jsonpath="{.items[0].metadata.name}")

1. Retrieve curl's SVID identity document using the istioctl proxy-config secret command:

    $ istioctl proxy-config secret "$CURL_POD" -o json | jq -r \
    '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | base64 --decode > chain.pem

1. Inspect the certificate and verify that SPIRE was the issuer:

    $ openssl x509 -in chain.pem -text | grep SPIRE
        Subject: C = US, O = SPIRE, CN = curl-5f4d47c948-njvpk

## SPIFFE federation

SPIRE Servers are able to authenticate SPIFFE identities originating from different trust domains. This is known as SPIFFE federation.

SPIRE Agent can be configured to push federated bundles to Envoy through the Envoy SDS API, allowing Envoy to use [validation context](https://spiffe.io/docs/latest/microservices/envoy/#validation-context)
to verify peer certificates and trust a workload from another trust domain.
To enable Istio to federate SPIFFE identities through SPIRE integration, consult [SPIRE Agent SDS configuration](https://github.com/spiffe/spire/blob/main/doc/spire_agent.md#sds-configuration) and set the following
SDS configuration values for your SPIRE Agent configuration file.

| Configuration              | Description                                                                                | Resource Name |
|----------------------------|--------------------------------------------------------------------------------------------------|---------------|
| `default_svid_name`        | The TLS Certificate resource name to use for the default `X509-SVID` with Envoy SDS              | default       |
| `default_bundle_name`      | The Validation Context resource name to use for the default X.509 bundle with Envoy SDS          | null          |
| `default_all_bundles_name` | The Validation Context resource name to use for all bundles (including federated) with Envoy SDS | ROOTCA        |

This will allow Envoy to get federated bundles directly from SPIRE.

### Create federated registration entries

- If using the SPIRE Controller Manager, create federated entries for workloads by setting the `federatesWith` field of the [ClusterSPIFFEID CR](https://github.com/spiffe/spire-controller-manager/blob/main/docs/clusterspiffeid-crd.md) to the trust domains you want the pod to federate with:

    apiVersion: spire.spiffe.io/v1alpha1
    kind: ClusterSPIFFEID
    metadata:
      name: federation
    spec:
      spiffeIDTemplate: "spiffe:///ns//sa/"
      podSelector:
        matchLabels:
          spiffe.io/spire-managed-identity: "true"
      federatesWith: ["example.io", "example.ai"]

- For manual registration see [Create Registration Entries for Federation](https://spiffe.io/docs/latest/architecture/federation/readme/#create-registration-entries-for-federation).

## Cleanup SPIRE

Remove SPIRE by uninstalling its Helm charts:

$ helm delete -n spire-server spire

$ helm delete -n spire-server spire-crds

title  SPIRE description  How to configure Istio to integrate with SPIRE to get cryptographic identities through Envoy s SDS API  weight  31 keywords   kubernetes spiffe spire  aliases  owner  istio wg networking maintainers test  yes       SPIRE  https   spiffe io docs latest spire about spire concepts   is a production ready implementation of the SPIFFE specification that performs node and workload attestation in order to securely issue cryptographic identities to workloads running in heterogeneous environments  SPIRE can be configured as a source of cryptographic identities for Istio workloads through an integration with  Envoy s SDS API  https   www envoyproxy io docs envoy latest configuration security secret   Istio can detect the existence of a UNIX Domain Socket that implements the Envoy SDS API on a defined socket path  allowing Envoy to communicate and fetch identities directly from it   This integration with SPIRE provides flexible attestation options not available with the default Istio identity management while harnessing Istio s powerful service management  For example  SPIRE s plugin architecture enables diverse workload attestation options beyond the Kubernetes namespace and service account attestation offered by Istio  SPIRE s node attestation extends attestation to the physical or virtual hardware on which workloads run   For a quick demo of how this SPIRE integration with Istio works  see  Integrating SPIRE as a CA through Envoy s SDS API   samples security spire       Install SPIRE  We recommend you follow SPIRE s installation instructions and best practices for installing SPIRE  and for deploying SPIRE in production environments   For the examples in this guide  the  SPIRE Helm charts  https   artifacthub io packages helm spiffe spire  will be used with upstream defaults  to focus on just the configuration necessary to integrate SPIRE and Istio      helm upgrade   install  n spire server spire crds spire crds   repo https   spiffe github io helm charts hardened    create namespace      helm upgrade   install  n spire server spire spire   repo https   spiffe github io helm charts hardened    wait   set global spire trustDomain  example org     See the  SPIRE Helm chart  https   artifacthub io packages helm spiffe spire  documentation for other values you can configure for your installation   It is important that SPIRE and Istio are configured with the exact same trust domain  to prevent authentication and authorization errors  and that the  SPIFFE CSI driver  https   github com spiffe spiffe csi  is enabled and installed    By default  the above will also install     The  SPIFFE CSI driver  https   github com spiffe spiffe csi   which is used to mount an Envoy compatible SDS socket into proxies  Using the SPIFFE CSI driver to mount SDS sockets is strongly recommended by both Istio and SPIRE  as  hostMounts  are a larger security risk and introduce operational hurdles  This guide assumes the use of the SPIFFE CSI driver     The  SPIRE Controller Manager  https   github com spiffe spire controller manager   which eases the creation of SPIFFE registrations for workloads      Register workloads  By design  SPIRE only grants identities to workloads that have been registered with the SPIRE server  this includes user workloads  as well as Istio components  Istio sidecars and gateways  once configured for SPIRE integration  cannot get identities  and therefore cannot reach READY status  unless there is a preexisting  matching SPIRE registration created for them ahead of time   See the  SPIRE docs on registering workloads  https   spiffe io docs latest deploying registering   for more information on using multiple selectors to strengthen attestation criteria  and the selectors available   This section describes the options available for registering Istio workloads in a SPIRE Server and provides some example workload registrations    Istio currently requires a specific SPIFFE ID format for workloads  All registrations must follow the Istio SPIFFE ID pattern   spiffe    trust domain  ns  namespace  sa  service account         Option 1  Auto registration using the SPIRE Controller Manager  New entries will be automatically registered for each new pod that matches the selector defined in a  ClusterSPIFFEID  https   github com spiffe spire controller manager blob main docs clusterspiffeid crd md  custom resource   Both Istio sidecars and Istio gateways need to be registered with SPIRE  so that they can request identities        Istio Gateway  ClusterSPIFFEID   The following will create a  ClusterSPIFFEID   which will auto register any Istio Ingress gateway pod with SPIRE if it is scheduled into the  istio system  namespace  and has a service account named  istio ingressgateway service account   These selectors are used as a simple example  consult the  SPIRE Controller Manager documentation  https   github com spiffe spire controller manager blob main docs clusterspiffeid crd md  for more details      kubectl apply  f     EOF apiVersion  spire spiffe io v1alpha1 kind  ClusterSPIFFEID metadata    name  istio ingressgateway reg spec    spiffeIDTemplate   spiffe    ns  sa     workloadSelectorTemplates         k8s ns istio system         k8s sa istio ingressgateway service account  EOF        Istio Sidecar  ClusterSPIFFEID   The following will create a  ClusterSPIFFEID  which will auto register any pod with the  spiffe io spire managed identity  true  label that is deployed into the  default  namespace with SPIRE  These selectors are used as a simple example  consult the  SPIRE Controller Manager documentation  https   github com spiffe spire controller manager blob main docs clusterspiffeid crd md  for more details      kubectl apply  f     EOF apiVersion  spire spiffe io v1alpha1 kind  ClusterSPIFFEID metadata    name  istio sidecar reg spec    spiffeIDTemplate   spiffe    ns  sa     podSelector      matchLabels        spiffe io spire managed identity   true    workloadSelectorTemplates         k8s ns default  EOF       Option 2  Manual Registration  If you wish to manually create your SPIRE registrations  rather than use the SPIRE Controller Manager mentioned in  the recommended option   option 1 auto registration using the spire controller manager   refer to the  SPIRE documentation on manual registration  https   spiffe io docs latest deploying registering     Below are the equivalent manual registrations based off the automatic registrations in  Option 1   option 1 auto registration using the spire controller manager   The following steps assume you have  already followed the SPIRE documentation to manually register your SPIRE agent and node attestation  https   spiffe io docs latest deploying registering  1 defining the spiffe id of the agent  and that your SPIRE agent was registered with the SPIFFE identity  spiffe   example org ns spire sa spire agent    1  Get the  spire server  pod              SPIRE SERVER POD   kubectl get pod  l statefulset kubernetes io pod name spire server 0  n spire server  o jsonpath    items 0  metadata name          1  Register an entry for the Istio Ingress gateway pod              kubectl exec  n spire   SPIRE SERVER POD            opt spire bin spire server entry create            spiffeID spiffe   example org ns istio system sa istio ingressgateway service account            parentID spiffe   example org ns spire sa spire agent            selector k8s sa istio ingressgateway service account            selector k8s ns istio system            socketPath  run spire sockets server sock      Entry ID           6f2fe370 5261 4361 ac36 10aae8d91ff7     SPIFFE ID          spiffe   example org ns istio system sa istio ingressgateway service account     Parent ID          spiffe   example org ns spire sa spire agent     Revision           0     TTL                default     Selector           k8s ns istio system     Selector           k8s sa istio ingressgateway service account       1  Register an entry for workloads injected with an Istio sidecar              kubectl exec  n spire   SPIRE SERVER POD            opt spire bin spire server entry create            spiffeID spiffe   example org ns default sa curl            parentID spiffe   example org ns spire sa spire agent            selector k8s ns default            selector k8s pod label spiffe io spire managed identity true            socketPath  run spire sockets server sock          Install Istio  1   Download the Istio release   docs setup additional setup download istio release     1  Create the Istio configuration with custom patches for the Ingress Gateway and  istio proxy   The Ingress Gateway component includes the  spiffe io spire managed identity   true   label              cat   EOF     istio yaml     apiVersion  install istio io v1alpha1     kind  IstioOperator     metadata        namespace  istio system     spec        profile  default       meshConfig          trustDomain  example org       values            This is used to customize the sidecar template            It adds both the label to indicate that SPIRE should manage the           identity of this pod  as well as the CSI driver mounts          sidecarInjectorWebhook            templates              spire                  labels                  spiffe io spire managed identity   true                spec                  containers                    name  istio proxy                   volumeMounts                      name  workload socket                     mountPath   run secrets workload spiffe uds                     readOnly  true                 volumes                      name  workload socket                     csi                        driver   csi spiffe io                        readOnly  true       components          ingressGateways              name  istio ingressgateway             enabled  true             label                istio  ingressgateway             k8s                overlays                    This is used to customize the ingress gateway template                    It adds the CSI driver mounts  as well as an init container                   to stall gateway startup until the CSI driver mounts the socket                    apiVersion  apps v1                   kind  Deployment                   name  istio ingressgateway                   patches                        path  spec template spec volumes  name workload socket                        value                          name  workload socket                         csi                            driver   csi spiffe io                            readOnly  true                       path  spec template spec containers  name istio proxy  volumeMounts  name workload socket                        value                          name  workload socket                         mountPath    run secrets workload spiffe uds                          readOnly  true                       path  spec template spec initContainers                       value                            name  wait for spire socket                           image  busybox 1 36                           volumeMounts                                name  workload socket                               mountPath   run secrets workload spiffe uds                               readOnly  true                           env                                name  CHECK FILE                               value   run secrets workload spiffe uds socket                           command                                sh                                 c                                                                 echo    date  Iseconds   Waiting for    CHECK FILE                                while       e   CHECK FILE       do                                 echo    date  Iseconds   File does not exist    CHECK FILE                                  sleep 15                               done                               ls  l   CHECK FILE      EOF       1  Apply the configuration              istioctl install   skip confirmation  f   istio yaml       1  Check Ingress Gateway pod state              kubectl get pods  n istio system     NAME                                    READY   STATUS    RESTARTS   AGE     istio ingressgateway 5b45864fd4 lgrxs   1 1     Running   0          17s     istiod 989f54d9c sg7sn                  1 1     Running   0          23s           The Ingress Gateway pod is  Ready  since the corresponding registration entry is automatically created for it on the SPIRE Server  Envoy is able to fetch cryptographic identities from SPIRE       This configuration also adds an  initContainer  to the gateway that will wait for SPIRE to create the UNIX Domain Socket before starting the  istio proxy   If the SPIRE agent is not ready  or has not been properly configured with the same socket path  the Ingress Gateway  initContainer  will wait forever   1  Deploy an example workload              istioctl kube inject   filename  samples security spire curl spire yaml    kubectl apply  f             In addition to needing  spiffe io spire managed identity  label  the workload will need the SPIFFE CSI Driver volume to access the SPIRE Agent socket  To accomplish this      you can leverage the  spire  pod annotation template from the  Install Istio   install istio  section or add the CSI volume to     the deployment spec of your workload  Both of these alternatives are highlighted on the example snippet below            apiVersion  apps v1     kind  Deployment     metadata        name  curl     spec        replicas  1       selector            matchLabels              app  curl       template            metadata              labels                app  curl               Injects custom sidecar template             annotations                  inject istio io templates   sidecar spire            spec              terminationGracePeriodSeconds  0             serviceAccountName  curl             containers                name  curl               image  curlimages curl               command     bin sleep    3650d                 imagePullPolicy  IfNotPresent               volumeMounts                    name  tmp                   mountPath   tmp               securityContext                  runAsUser  1000             volumes                  name  tmp                 emptyDir                     CSI volume                 name  workload socket                 csi                    driver   csi spiffe io                    readOnly  true       The Istio configuration shares the  spiffe csi driver  with the Ingress Gateway and the sidecars that are going to be injected on workload pods  granting them access to the SPIRE Agent s UNIX Domain Socket   See  Verifying that identities were created for workloads   verifying that identities were created for workloads  to check issued identities      Verifying that identities were created for workloads  Use the following command to confirm that identities were created for the workloads      kubectl exec  t   SPIRE SERVER POD   n spire server  c spire server      bin spire server entry show Found 2 entries Entry ID           c8dfccdc 9762 4762 80d3 5434e5388ae7 SPIFFE ID          spiffe   example org ns istio system sa istio ingressgateway service account Parent ID          spiffe   example org spire agent k8s psat demo cluster bea19580 ae04 4679 a22e 472e18ca4687 Revision           0 X509 SVID TTL      default JWT SVID TTL       default Selector           k8s pod uid 88b71387 4641 4d9c 9a89 989c88f7509d  Entry ID           af7b53dc 4cc9 40d3 aaeb 08abbddd8e54 SPIFFE ID          spiffe   example org ns default sa curl Parent ID          spiffe   example org spire agent k8s psat demo cluster bea19580 ae04 4679 a22e 472e18ca4687 Revision           0 X509 SVID TTL      default JWT SVID TTL       default Selector           k8s pod uid ee490447 e502 46bd 8532 5a746b0871d6   Check the Ingress gateway pod state      kubectl get pods  n istio system NAME                                    READY   STATUS    RESTARTS   AGE istio ingressgateway 5b45864fd4 lgrxs   1 1     Running   0          60s istiod 989f54d9c sg7sn                  1 1     Running   0          45s   After registering an entry for the Ingress gateway pod  Envoy receives the identity issued by SPIRE and uses it for all TLS and mTLS communications       Check that the workload identity was issued by SPIRE  1  Get pod information              CURL POD   kubectl get pod  l app curl  o jsonpath    items 0  metadata name          1  Retrieve curl s SVID identity document using the istioctl proxy config secret command              istioctl proxy config secret   CURL POD   o json   jq  r         dynamicActiveSecrets 0  secret tlsCertificate certificateChain inlineBytes    base64   decode   chain pem       1  Inspect the certificate and verify that SPIRE was the issuer              openssl x509  in chain pem  text   grep SPIRE         Subject  C   US  O   SPIRE  CN   curl 5f4d47c948 njvpk          SPIFFE federation  SPIRE Servers are able to authenticate SPIFFE identities originating from different trust domains  This is known as SPIFFE federation   SPIRE Agent can be configured to push federated bundles to Envoy through the Envoy SDS API  allowing Envoy to use  validation context  https   spiffe io docs latest microservices envoy  validation context  to verify peer certificates and trust a workload from another trust domain  To enable Istio to federate SPIFFE identities through SPIRE integration  consult  SPIRE Agent SDS configuration  https   github com spiffe spire blob main doc spire agent md sds configuration  and set the following SDS configuration values for your SPIRE Agent configuration file     Configuration                Description                                                                                Resource Name                                                                                default svid name           The TLS Certificate resource name to use for the default  X509 SVID  with Envoy SDS                default            default bundle name         The Validation Context resource name to use for the default X 509 bundle with Envoy SDS            null               default all bundles name    The Validation Context resource name to use for all bundles  including federated  with Envoy SDS   ROOTCA           This will allow Envoy to get federated bundles directly from SPIRE       Create federated registration entries    If using the SPIRE Controller Manager  create federated entries for workloads by setting the  federatesWith  field of the  ClusterSPIFFEID CR  https   github com spiffe spire controller manager blob main docs clusterspiffeid crd md  to the trust domains you want the pod to federate with            apiVersion  spire spiffe io v1alpha1     kind  ClusterSPIFFEID     metadata        name  federation     spec        spiffeIDTemplate   spiffe    ns  sa         podSelector          matchLabels            spiffe io spire managed identity   true        federatesWith    example io    example ai           For manual registration see  Create Registration Entries for Federation  https   spiffe io docs latest architecture federation readme  create registration entries for federation       Cleanup SPIRE  Remove SPIRE by uninstalling its Helm charts      helm delete  n spire server spire      helm delete  n spire server spire crds


istio test no docs examples advanced gateways ingress certmgr weight 26 title cert manager aliases keywords integration cert manager owner istio wg environments maintainers docs tasks traffic management ingress ingress certmgr Information on how to integrate with cert manager

Answer:
---
title: cert-manager
description: Information on how to integrate with cert-manager.
weight: 26
keywords: [integration,cert-manager]
aliases:
  - /docs/tasks/traffic-management/ingress/ingress-certmgr/
  - /docs/examples/advanced-gateways/ingress-certmgr/
owner: istio/wg-environments-maintainers
test: no
---

[cert-manager](https://cert-manager.io/) is a tool that automates certificate management.
This can be integrated with Istio gateways to manage TLS certificates.

## Configuration

Consult the [cert-manager installation documentation](https://cert-manager.io/docs/installation/kubernetes/)
to get started. No special changes are needed to work with Istio.

## Usage

### Istio Gateway

cert-manager can be used to write a secret to Kubernetes, which can then be referenced by a Gateway.

1. To get started, configure an `Issuer` resource, following the [cert-manager issuer documentation](https://cert-manager.io/docs/configuration/). `Issuer`s are Kubernetes resources that represent certificate authorities (CAs) that are able to generate signed certificates by honoring certificate signing requests. For example: an `Issuer` may look like:

    apiVersion: cert-manager.io/v1
    kind: Issuer
    metadata:
      name: ca-issuer
      namespace: istio-system
    spec:
      ca:
        secretName: ca-key-pair

    For a common Issuer type, ACME, a pod and service are created to respond to challenge requests in order to verify the client owns the domain. To respond to those challenges, an endpoint at `http://<YOUR_DOMAIN>/.well-known/acme-challenge/<TOKEN>` will need to be reachable. That configuration may be implementation specific.

1. Next, configure a `Certificate` resource, following the
[cert-manager documentation](https://cert-manager.io/docs/usage/certificate/).
The `Certificate` should be created in the same namespace as the `istio-ingressgateway` deployment.
For example, a `Certificate` may look like:

    apiVersion: cert-manager.io/v1
    kind: Certificate
    metadata:
      name: ingress-cert
      namespace: istio-system
    spec:
      secretName: ingress-cert
      commonName: my.example.com
      dnsNames:
      - my.example.com
      ...

1. Once we have the certificate created, we should see the secret created in the `istio-system` namespace.
  This can then be referenced in the `tls` config for a Gateway under `credentialName`:

    apiVersion: networking.istio.io/v1
    kind: Gateway
    metadata:
      name: gateway
    spec:
      selector:
        istio: ingressgateway
      servers:
      - port:
          number: 443
          name: https
          protocol: HTTPS
        tls:
          mode: SIMPLE
          credentialName: ingress-cert # This should match the Certificate secretName
        hosts:
        - my.example.com # This should match a DNS name in the Certificate

### Kubernetes Ingress

cert-manager provides direct integration with Kubernetes Ingress by configuring an
[annotation on the Ingress object](https://cert-manager.io/docs/usage/ingress/).
If this method is used, the Ingress must reside in the same namespace as the
`istio-ingressgateway` deployment, as secrets will only be read within the same namespace.

Alternatively, a `Certificate` can be created as described in [Istio Gateway](#istio-gateway),
then referenced in the `Ingress` object:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.class: istio
spec:
  rules:
  - host: my.example.com
    http: ...
  tls:
  - hosts:
    - my.example.com # This should match a DNS name in the Certificate
    secretName: ingress-cert # This should match the Certificate secretName

title  cert manager description  Information on how to integrate with cert manager  weight  26 keywords   integration cert manager  aliases       docs tasks traffic management ingress ingress certmgr       docs examples advanced gateways ingress certmgr  owner  istio wg environments maintainers test  no       cert manager  https   cert manager io   is a tool that automates certificate management  This can be integrated with Istio gateways to manage TLS certificates      Configuration  Consult the  cert manager installation documentation  https   cert manager io docs installation kubernetes   to get started  No special changes are needed to work with Istio      Usage      Istio Gateway  cert manager can be used to write a secret to Kubernetes  which can then be referenced by a Gateway   1  To get started  configure an  Issuer  resource  following the  cert manager issuer documentation  https   cert manager io docs configuration     Issuer s are Kubernetes resources that represent certificate authorities  CAs  that are able to generate signed certificates by honoring certificate signing requests  For example  an  Issuer  may look like            apiVersion  cert manager io v1     kind  Issuer     metadata        name  ca issuer       namespace  istio system     spec        ca          secretName  ca key pair                For a common Issuer type  ACME  a pod and service are created to respond to challenge requests in order to verify the client owns the domain  To respond to those challenges  an endpoint at  http    YOUR DOMAIN   well known acme challenge  TOKEN   will need to be reachable  That configuration may be implementation specific        1  Next  configure a  Certificate  resource  following the  cert manager documentation  https   cert manager io docs usage certificate    The  Certificate  should be created in the same namespace as the  istio ingressgateway  deployment  For example  a  Certificate  may look like            apiVersion  cert manager io v1     kind  Certificate     metadata        name  ingress cert       namespace  istio system     spec        secretName  ingress cert       commonName  my example com       dnsNames          my example com                 1  Once we have the certificate created  we should see the secret created in the  istio system  namespace    This can then be referenced in the  tls  config for a Gateway under  credentialName             apiVersion  networking istio io v1     kind  Gateway     metadata        name  gateway     spec        selector          istio  ingressgateway       servers          port            number  443           name  https           protocol  HTTPS         tls            mode  SIMPLE           credentialName  ingress cert   This should match the Certificate secretName         hosts            my example com   This should match a DNS name in the Certificate           Kubernetes Ingress  cert manager provides direct integration with Kubernetes Ingress by configuring an  annotation on the Ingress object  https   cert manager io docs usage ingress    If this method is used  the Ingress must reside in the same namespace as the  istio ingressgateway  deployment  as secrets will only be read within the same namespace   Alternatively  a  Certificate  can be created as described in  Istio Gateway   istio gateway   then referenced in the  Ingress  object    apiVersion  networking k8s io v1 kind  Ingress metadata    name  ingress   annotations      kubernetes io ingress class  istio spec    rules      host  my example com     http        tls      hosts        my example com   This should match a DNS name in the Certificate     secretName  ingress cert   This should match the Certificate secretName