istio docs ops telemetry monitoring multicluster prometheus owner istio wg policies and telemetry maintainers test no aliases help ops telemetry monitoring multicluster prometheus title Monitoring Multicluster Istio with Prometheus weight 10 Configure Prometheus to monitor multicluster Istio

Answer:
---
title: Monitoring Multicluster Istio with Prometheus
description: Configure Prometheus to monitor multicluster Istio.
weight: 10
aliases:
  - /help/ops/telemetry/monitoring-multicluster-prometheus
  - /docs/ops/telemetry/monitoring-multicluster-prometheus
owner: istio/wg-policies-and-telemetry-maintainers
test: no
---

## Overview

This guide is meant to provide operational guidance on how to configure monitoring of Istio meshes comprised of two
or more individual Kubernetes clusters. It is not meant to establish the *only* possible path forward, but rather
to demonstrate a workable approach to multicluster telemetry with Prometheus.

Our recommendation for multicluster monitoring of Istio with Prometheus is built upon the foundation of Prometheus
[hierarchical federation](https://prometheus.io/docs/prometheus/latest/federation/#hierarchical-federation).
Prometheus instances that are deployed locally to each cluster by Istio act as initial collectors that then federate up
to a production mesh-wide Prometheus instance. That mesh-wide Prometheus can either live outside of the mesh (external), or in one
of the clusters within the mesh.

## Multicluster Istio setup

Follow the [multicluster installation](/docs/setup/install/multicluster/) section to set up your Istio clusters in one of the
supported [multicluster deployment models](/docs/ops/deployment/deployment-models/#multiple-clusters). For the purpose of
this guide, any of those approaches will work, with the following caveat:

**Ensure that a cluster-local Istio Prometheus instance is installed in each cluster.**

Individual Istio deployment of Prometheus in each cluster is required to form the basis of cross-cluster monitoring by
way of federation to a production-ready instance of Prometheus that runs externally or in one of the clusters.

Validate that you have an instance of Prometheus running in each cluster:

$ kubectl -n istio-system get services prometheus
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
prometheus   ClusterIP   10.8.4.109   <none>        9090/TCP   20h

## Configure Prometheus federation

### External production Prometheus

There are several reasons why you may want to have a Prometheus instance running outside of your Istio deployment.
Perhaps you want long-term monitoring disjoint from the cluster being monitored. Perhaps you want to monitor multiple
separate meshes in a single place. Or maybe you have other motivations. Whatever your reason is, you’ll need some special
configurations to make it all work.

This guide demonstrates connectivity to cluster-local Prometheus instances, but does not address security considerations.
For production use, secure access to each Prometheus endpoint with HTTPS. In addition, take precautions, such as using an
internal load-balancer instead of a public endpoint and the appropriate configuration of firewall rules.

Istio provides a way to expose cluster services externally via [Gateways](/docs/reference/config/networking/gateway/).
You can configure an ingress gateway for the cluster-local Prometheus, providing external connectivity to the in-cluster
Prometheus endpoint.

For each cluster, follow the appropriate instructions from the [Remotely Accessing Telemetry Addons](/docs/tasks/observability/gateways/#option-1-secure-access-https) task.
Also note that you **SHOULD** establish secure (HTTPS) access.

Next, configure your external Prometheus instance to access the cluster-local Prometheus instances using a configuration
like the following (replacing the ingress domain and cluster name):

scrape_configs:
- job_name: 'federate-'
  scrape_interval: 15s

  honor_labels: true
  metrics_path: '/federate'

  params:
    'match[]':
      - '{job="kubernetes-pods"}'

  static_configs:
    - targets:
      - 'prometheus.'
      labels:
        cluster: ''

Notes:

* `CLUSTER_NAME` should be set to the same value that you used to create the cluster (set via `values.global.multiCluster.clusterName`).

* No authentication to the Prometheus endpoint(s) is provided. This means that anyone can query your
cluster-local Prometheus instances. This may not be desirable.

* Without proper HTTPS configuration of the gateway, everything is being transported via plaintext. This may not be
desirable.

### Production Prometheus on an in-mesh cluster

If you prefer to run the production Prometheus in one of the clusters, you need to establish connectivity from it to
the other cluster-local Prometheus instances in the mesh.

This is really just a variation of the configuration for external federation. In this case the configuration on the
cluster running the production Prometheus is different from the configuration for remote cluster Prometheus scraping.

Configure your production Prometheus to access both of the *local* and *remote* Prometheus instances.

First execute the following command:

$ kubectl -n istio-system edit cm prometheus -o yaml

Then add configurations for the *remote* clusters (replacing the ingress domain and cluster name for each cluster) and
add one configuration for the *local* cluster:

scrape_configs:
- job_name: 'federate-'
  scrape_interval: 15s

  honor_labels: true
  metrics_path: '/federate'

  params:
    'match[]':
      - '{job="kubernetes-pods"}'

  static_configs:
    - targets:
      - 'prometheus.'
      labels:
        cluster: ''

- job_name: 'federate-local'

  honor_labels: true
  metrics_path: '/federate'

  metric_relabel_configs:
  - replacement: ''
    target_label: cluster

  kubernetes_sd_configs:
  - role: pod
    namespaces:
      names: ['istio-system']
  params:
    'match[]':
    - '{__name__=~"istio_(.*)"}'
    - '{__name__=~"pilot(.*)"}'

title  Monitoring Multicluster Istio with Prometheus description  Configure Prometheus to monitor multicluster Istio  weight  10 aliases       help ops telemetry monitoring multicluster prometheus      docs ops telemetry monitoring multicluster prometheus owner  istio wg policies and telemetry maintainers test  no         Overview  This guide is meant to provide operational guidance on how to configure monitoring of Istio meshes comprised of two or more individual Kubernetes clusters  It is not meant to establish the  only  possible path forward  but rather to demonstrate a workable approach to multicluster telemetry with Prometheus   Our recommendation for multicluster monitoring of Istio with Prometheus is built upon the foundation of Prometheus  hierarchical federation  https   prometheus io docs prometheus latest federation  hierarchical federation   Prometheus instances that are deployed locally to each cluster by Istio act as initial collectors that then federate up to a production mesh wide Prometheus instance  That mesh wide Prometheus can either live outside of the mesh  external   or in one of the clusters within the mesh      Multicluster Istio setup  Follow the  multicluster installation   docs setup install multicluster   section to set up your Istio clusters in one of the supported  multicluster deployment models   docs ops deployment deployment models  multiple clusters   For the purpose of this guide  any of those approaches will work  with the following caveat     Ensure that a cluster local Istio Prometheus instance is installed in each cluster     Individual Istio deployment of Prometheus in each cluster is required to form the basis of cross cluster monitoring by way of federation to a production ready instance of Prometheus that runs externally or in one of the clusters   Validate that you have an instance of Prometheus running in each cluster      kubectl  n istio system get services prometheus NAME         TYPE        CLUSTER IP   EXTERNAL IP   PORT S     AGE prometheus   ClusterIP   10 8 4 109    none         9090 TCP   20h      Configure Prometheus federation      External production Prometheus  There are several reasons why you may want to have a Prometheus instance running outside of your Istio deployment  Perhaps you want long term monitoring disjoint from the cluster being monitored  Perhaps you want to monitor multiple separate meshes in a single place  Or maybe you have other motivations  Whatever your reason is  you ll need some special configurations to make it all work      This guide demonstrates connectivity to cluster local Prometheus instances  but does not address security considerations  For production use  secure access to each Prometheus endpoint with HTTPS  In addition  take precautions  such as using an internal load balancer instead of a public endpoint and the appropriate configuration of firewall rules    Istio provides a way to expose cluster services externally via  Gateways   docs reference config networking gateway    You can configure an ingress gateway for the cluster local Prometheus  providing external connectivity to the in cluster Prometheus endpoint   For each cluster  follow the appropriate instructions from the  Remotely Accessing Telemetry Addons   docs tasks observability gateways  option 1 secure access https  task  Also note that you   SHOULD   establish secure  HTTPS  access   Next  configure your external Prometheus instance to access the cluster local Prometheus instances using a configuration like the following  replacing the ingress domain and cluster name     scrape configs    job name   federate     scrape interval  15s    honor labels  true   metrics path    federate     params       match               job  kubernetes pods       static configs        targets           prometheus         labels          cluster       Notes      CLUSTER NAME  should be set to the same value that you used to create the cluster  set via  values global multiCluster clusterName       No authentication to the Prometheus endpoint s  is provided  This means that anyone can query your cluster local Prometheus instances  This may not be desirable     Without proper HTTPS configuration of the gateway  everything is being transported via plaintext  This may not be desirable       Production Prometheus on an in mesh cluster  If you prefer to run the production Prometheus in one of the clusters  you need to establish connectivity from it to the other cluster local Prometheus instances in the mesh   This is really just a variation of the configuration for external federation  In this case the configuration on the cluster running the production Prometheus is different from the configuration for remote cluster Prometheus scraping     Configure your production Prometheus to access both of the  local  and  remote  Prometheus instances   First execute the following command      kubectl  n istio system edit cm prometheus  o yaml   Then add configurations for the  remote  clusters  replacing the ingress domain and cluster name for each cluster  and add one configuration for the  local  cluster    scrape configs    job name   federate     scrape interval  15s    honor labels  true   metrics path    federate     params       match               job  kubernetes pods       static configs        targets           prometheus         labels          cluster        job name   federate local     honor labels  true   metrics path    federate     metric relabel configs      replacement         target label  cluster    kubernetes sd configs      role  pod     namespaces        names    istio system     params       match               name     istio                   name     pilot


istio owner istio wg networking maintainers weight 30 linktitle TLS Configuration keywords traffic management proxy How to configure TLS settings to secure network traffic test n a title Understanding TLS Configuration

Answer:
---
title: Understanding TLS Configuration
linktitle: TLS Configuration
description: How to configure TLS settings to secure network traffic.
weight: 30
keywords: [traffic-management,proxy]
owner: istio/wg-networking-maintainers
test: n/a
---

One of Istio's most important features is the ability to lock down and secure network traffic to, from,
and within the mesh. However, configuring TLS settings can be confusing and a common source of misconfiguration.
This document attempts to explain the various connections involved when sending requests in Istio and how
their associated TLS settings are configured.
Refer to [TLS configuration mistakes](/docs/ops/common-problems/network-issues/#tls-configuration-mistakes)
for a summary of some the most common TLS configuration problems.

## Sidecars

Sidecar traffic has a variety of associated connections. Let's break them down one at a time.

1. **External inbound traffic**
    This is traffic coming from an outside client that is captured by the sidecar.
    If the client is inside the mesh, this traffic may be encrypted with Istio mutual TLS.
    By default, the sidecar will be configured to accept both mTLS and non-mTLS traffic, known as `PERMISSIVE` mode.
    The mode can alternatively be configured to `STRICT`, where traffic must be mTLS, or `DISABLE`, where traffic must be plaintext.
    The mTLS mode is configured using a [`PeerAuthentication` resource](/docs/reference/config/security/peer_authentication/).

1. **Local inbound traffic**
    This is traffic going to your application service, from the sidecar. This traffic will always be forwarded as-is.
    Note that this does not mean it's always plaintext; the sidecar may pass a TLS connection through.
    It just means that a new TLS connection will never be originated from the sidecar.

1. **Local outbound traffic**
    This is outgoing traffic from your application service that is intercepted by the sidecar.
    Your application may be sending plaintext or TLS traffic.
    If [automatic protocol selection](/docs/ops/configuration/traffic-management/protocol-selection/#automatic-protocol-selection)
    is enabled, Istio will automatically detect the protocol. Otherwise you should use the port name in the destination service to
    [manually specify the protocol](/docs/ops/configuration/traffic-management/protocol-selection/#explicit-protocol-selection).

1. **External outbound traffic**
    This is traffic leaving the sidecar to some external destination. Traffic can be forwarded as is, or a TLS connection can
    be initiated (mTLS or standard TLS). This is controlled using the TLS mode setting in the `trafficPolicy` of a
    [`DestinationRule` resource](/docs/reference/config/networking/destination-rule/).
    A mode setting of `DISABLE` will send plaintext, while `SIMPLE`, `MUTUAL`, and `ISTIO_MUTUAL` will originate a TLS connection.

The key takeaways are:

- `PeerAuthentication` is used to configure what type of mTLS traffic the sidecar will accept.
- `DestinationRule` is used to configure what type of TLS traffic the sidecar will send.
- Port names, or automatic protocol selection, determines which protocol the sidecar will parse traffic as.

## Auto mTLS

As described above, a `DestinationRule` controls whether outgoing traffic uses mTLS or not.
However, configuring this for every workload can be tedious. Typically, you want Istio to always use mTLS
wherever possible, and only send plaintext to workloads that are not part of the mesh (i.e., ones without sidecars).

Istio makes this easy with a feature called "Auto mTLS". Auto mTLS works by doing exactly that. If TLS settings are
not explicitly configured in a `DestinationRule`, the sidecar will automatically determine if
[Istio mutual TLS](/about/faq/#difference-between-mutual-and-istio-mutual) should be sent.
This means that without any configuration, all inter-mesh traffic will be mTLS encrypted.

## Gateways

Any given request to a gateway will have two connections.

1. The inbound request, initiated by some client such as `curl` or a web browser. This is often called the "downstream" connection.

1. The outbound request, initiated by the gateway to some backend. This is often called the "upstream" connection.

Both of these connections have independent TLS configurations.

Note that the configuration of ingress and egress gateways are identical.
The `istio-ingress-gateway` and `istio-egress-gateway` are just two specialized gateway deployments.
The difference is that the client of an ingress gateway is running outside of the mesh while in the case of an egress gateway,
the destination is outside of the mesh.

### Inbound

As part of the inbound request, the gateway must decode the traffic in order to apply routing rules.
This is done based on the server configuration in a [`Gateway` resource](/docs/reference/config/networking/gateway/).
For example, if an inbound connection is plaintext HTTP, the port protocol is configured as `HTTP`:

apiVersion: networking.istio.io/v1
kind: Gateway
...
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP

Similarly, for raw TCP traffic, the protocol would be set to `TCP`.

For TLS connections, there are a few more options:

1. What protocol is encapsulated?
    If the connection is HTTPS, the server protocol should be configured as `HTTPS`.
    Otherwise, for a raw TCP connection encapsulated with TLS, the protocol should be set to `TLS`.

1. Is the TLS connection terminated or passed through?
    For passthrough traffic, configure the TLS mode field to `PASSTHROUGH`:

    apiVersion: networking.istio.io/v1
    kind: Gateway
    ...
      servers:
      - port:
          number: 443
          name: https
          protocol: HTTPS
        tls:
          mode: PASSTHROUGH

    In this mode, Istio will route based on SNI information and forward the connection as-is to the destination.

1. Should mutual TLS be used?
    Mutual TLS can be configured through the TLS mode `MUTUAL`. When this is configured, a client certificate will be
    requested and verified against the configured `caCertificates` or `credentialName`:

    apiVersion: networking.istio.io/v1
    kind: Gateway
    ...
      servers:
      - port:
          number: 443
          name: https
          protocol: HTTPS
        tls:
          mode: MUTUAL
          caCertificates: ...

### Outbound

While the inbound side configures what type of traffic to expect and how to process it, the outbound configuration controls
what type of traffic the gateway will send. This is configured by the TLS settings in a `DestinationRule`,
just like external outbound traffic from [sidecars](#sidecars), or [auto mTLS](#auto-mtls) by default.

The only difference is that you should be careful to consider the `Gateway` settings when configuring this.
For example, if the `Gateway` is configured with TLS `PASSTHROUGH` while the `DestinationRule` configures TLS origination,
you will end up with [double encryption](/docs/ops/common-problems/network-issues/#double-tls).
This works, but is often not the desired behavior.

A `VirtualService` bound to the gateway needs care as well to
[ensure it is consistent](/docs/ops/common-problems/network-issues/#gateway-mismatch)
with the `Gateway` definition.

title  Understanding TLS Configuration linktitle  TLS Configuration description  How to configure TLS settings to secure network traffic  weight  30 keywords   traffic management proxy  owner  istio wg networking maintainers test  n a      One of Istio s most important features is the ability to lock down and secure network traffic to  from  and within the mesh  However  configuring TLS settings can be confusing and a common source of misconfiguration  This document attempts to explain the various connections involved when sending requests in Istio and how their associated TLS settings are configured  Refer to  TLS configuration mistakes   docs ops common problems network issues  tls configuration mistakes  for a summary of some the most common TLS configuration problems      Sidecars  Sidecar traffic has a variety of associated connections  Let s break them down one at a time     1    External inbound traffic       This is traffic coming from an outside client that is captured by the sidecar      If the client is inside the mesh  this traffic may be encrypted with Istio mutual TLS      By default  the sidecar will be configured to accept both mTLS and non mTLS traffic  known as  PERMISSIVE  mode      The mode can alternatively be configured to  STRICT   where traffic must be mTLS  or  DISABLE   where traffic must be plaintext      The mTLS mode is configured using a   PeerAuthentication  resource   docs reference config security peer authentication     1    Local inbound traffic       This is traffic going to your application service  from the sidecar  This traffic will always be forwarded as is      Note that this does not mean it s always plaintext  the sidecar may pass a TLS connection through      It just means that a new TLS connection will never be originated from the sidecar   1    Local outbound traffic       This is outgoing traffic from your application service that is intercepted by the sidecar      Your application may be sending plaintext or TLS traffic      If  automatic protocol selection   docs ops configuration traffic management protocol selection  automatic protocol selection      is enabled  Istio will automatically detect the protocol  Otherwise you should use the port name in the destination service to      manually specify the protocol   docs ops configuration traffic management protocol selection  explicit protocol selection    1    External outbound traffic       This is traffic leaving the sidecar to some external destination  Traffic can be forwarded as is  or a TLS connection can     be initiated  mTLS or standard TLS   This is controlled using the TLS mode setting in the  trafficPolicy  of a       DestinationRule  resource   docs reference config networking destination rule        A mode setting of  DISABLE  will send plaintext  while  SIMPLE    MUTUAL   and  ISTIO MUTUAL  will originate a TLS connection   The key takeaways are      PeerAuthentication  is used to configure what type of mTLS traffic the sidecar will accept     DestinationRule  is used to configure what type of TLS traffic the sidecar will send    Port names  or automatic protocol selection  determines which protocol the sidecar will parse traffic as      Auto mTLS  As described above  a  DestinationRule  controls whether outgoing traffic uses mTLS or not  However  configuring this for every workload can be tedious  Typically  you want Istio to always use mTLS wherever possible  and only send plaintext to workloads that are not part of the mesh  i e   ones without sidecars    Istio makes this easy with a feature called  Auto mTLS   Auto mTLS works by doing exactly that  If TLS settings are not explicitly configured in a  DestinationRule   the sidecar will automatically determine if  Istio mutual TLS   about faq  difference between mutual and istio mutual  should be sent  This means that without any configuration  all inter mesh traffic will be mTLS encrypted      Gateways  Any given request to a gateway will have two connections     1  The inbound request  initiated by some client such as  curl  or a web browser  This is often called the  downstream  connection   1  The outbound request  initiated by the gateway to some backend  This is often called the  upstream  connection   Both of these connections have independent TLS configurations   Note that the configuration of ingress and egress gateways are identical  The  istio ingress gateway  and  istio egress gateway  are just two specialized gateway deployments  The difference is that the client of an ingress gateway is running outside of the mesh while in the case of an egress gateway  the destination is outside of the mesh       Inbound  As part of the inbound request  the gateway must decode the traffic in order to apply routing rules  This is done based on the server configuration in a   Gateway  resource   docs reference config networking gateway    For example  if an inbound connection is plaintext HTTP  the port protocol is configured as  HTTP     apiVersion  networking istio io v1 kind  Gateway       servers      port        number  80       name  http       protocol  HTTP   Similarly  for raw TCP traffic  the protocol would be set to  TCP    For TLS connections  there are a few more options   1  What protocol is encapsulated      If the connection is HTTPS  the server protocol should be configured as  HTTPS       Otherwise  for a raw TCP connection encapsulated with TLS  the protocol should be set to  TLS    1  Is the TLS connection terminated or passed through      For passthrough traffic  configure the TLS mode field to  PASSTHROUGH             apiVersion  networking istio io v1     kind  Gateway               servers          port            number  443           name  https           protocol  HTTPS         tls            mode  PASSTHROUGH           In this mode  Istio will route based on SNI information and forward the connection as is to the destination   1  Should mutual TLS be used      Mutual TLS can be configured through the TLS mode  MUTUAL   When this is configured  a client certificate will be     requested and verified against the configured  caCertificates  or  credentialName             apiVersion  networking istio io v1     kind  Gateway               servers          port            number  443           name  https           protocol  HTTPS         tls            mode  MUTUAL           caCertificates                Outbound  While the inbound side configures what type of traffic to expect and how to process it  the outbound configuration controls what type of traffic the gateway will send  This is configured by the TLS settings in a  DestinationRule   just like external outbound traffic from  sidecars   sidecars   or  auto mTLS   auto mtls  by default   The only difference is that you should be careful to consider the  Gateway  settings when configuring this  For example  if the  Gateway  is configured with TLS  PASSTHROUGH  while the  DestinationRule  configures TLS origination  you will end up with  double encryption   docs ops common problems network issues  double tls   This works  but is often not the desired behavior   A  VirtualService  bound to the gateway needs care as well to  ensure it is consistent   docs ops common problems network issues  gateway mismatch  with the  Gateway  definition


istio owner istio wg networking maintainers How to configure gateway network topology status Alpha keywords traffic management ingress gateway test yes weight 60 title Configuring Gateway Network Topology

Answer:
---
title: Configuring Gateway Network Topology
description: How to configure gateway network topology.
weight: 60
keywords: [traffic-management,ingress,gateway]
owner: istio/wg-networking-maintainers
test: yes
status: Alpha
---

## Forwarding external client attributes (IP address, certificate info) to destination workloads

Many applications require knowing the client IP address and certificate information of the originating request to behave
properly. Notable cases include logging and audit tools that require the client IP be populated and security tools,
such as Web Application Firewalls (WAF), that need this information to apply rule sets properly. The ability to
provide client attributes to services has long been a staple of reverse proxies. To forward these client
attributes to destination workloads, proxies use the `X-Forwarded-For` (XFF) and `X-Forwarded-Client-Cert` (XFCC) headers.

Today's networks vary widely in nature, but support for these attributes is a requirement no matter what the network topology is.
This information should be preserved
and forwarded whether the network uses cloud-based Load Balancers, on-premise Load Balancers, gateways that are
exposed directly to the internet, gateways that serve many intermediate proxies, and other deployment topologies not
specified.

While Istio provides an [ingress gateway](/docs/tasks/traffic-management/ingress/ingress-control/), given the varieties
of architectures mentioned above, reasonable defaults are not able to be shipped that support the proper forwarding of
client attributes to the destination workloads.
This becomes ever more vital as Istio multicluster deployment models become more common.

For more information on `X-Forwarded-For`, see the IETF's [RFC](https://tools.ietf.org/html/rfc7239).

## Configuring network topologies

Configuration of XFF and XFCC headers can be set globally for all gateway workloads via `MeshConfig` or per gateway using
a pod annotation. For example, to configure globally during install or upgrade when using an `IstioOperator` custom resource:

spec:
  meshConfig:
    defaultConfig:
      gatewayTopology:
        numTrustedProxies: <VALUE>
        forwardClientCertDetails: <ENUM_VALUE>

You can also configure both of these settings by adding the `proxy.istio.io/config` annotation to the Pod spec
of your Istio ingress gateway.

...
  metadata:
    annotations:
      "proxy.istio.io/config": '{"gatewayTopology" : { "numTrustedProxies": <VALUE>, "forwardClientCertDetails": <ENUM_VALUE> } }'

### Configuring X-Forwarded-For Headers

Applications rely on reverse proxies to forward client attributes in a request, such as `X-Forward-For` header. However, due to the variety of network
topologies that Istio can be deployed in, you must set the `numTrustedProxies` to the number of trusted proxies deployed in front
of the Istio gateway proxy, so that the client address can be extracted correctly.
This controls the value populated by the ingress gateway in the `X-Envoy-External-Address` header
which can be reliably used by the upstream services to access client's original IP address.

For example, if you have a cloud based Load Balancer and a reverse proxy in front of your Istio gateway, set `numTrustedProxies` to `2`.

Note that all proxies in front of the Istio gateway proxy must parse HTTP traffic and append to the `X-Forwarded-For`
header at each hop. If the number of entries in the `X-Forwarded-For` header is less than the number of
trusted hops configured, Envoy falls back to using the immediate downstream address as the trusted
client address. Please refer to the [Envoy documentation](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-forwarded-for)
to understand how `X-Forwarded-For` headers and trusted client addresses are determined.

#### Example using X-Forwarded-For capability with httpbin

1. Run the following command to create a file named `topology.yaml` with `numTrustedProxies` set to `2` and install Istio:

    $ cat <<EOF > topology.yaml
    apiVersion: install.istio.io/v1alpha1
    kind: IstioOperator
    spec:
      meshConfig:
        defaultConfig:
          gatewayTopology:
            numTrustedProxies: 2
    EOF
    $ istioctl install -f topology.yaml

    If you previously installed an Istio ingress gateway, restart all ingress gateway pods after step 1.

1. Create an `httpbin` namespace:

    $ kubectl create namespace httpbin
    namespace/httpbin created

1. Set the `istio-injection` label to `enabled` for sidecar injection:

    $ kubectl label --overwrite namespace httpbin istio-injection=enabled
    namespace/httpbin labeled

1. Deploy `httpbin` in the `httpbin` namespace:

    $ kubectl apply -n httpbin -f @samples/httpbin/httpbin.yaml@

1. Deploy a gateway associated with `httpbin`:

$ kubectl apply -n httpbin -f @samples/httpbin/httpbin-gateway.yaml@

$ kubectl apply -n httpbin -f @samples/httpbin/gateway-api/httpbin-gateway.yaml@
$ kubectl wait --for=condition=programmed gtw -n httpbin httpbin-gateway

6) Set a local `GATEWAY_URL` environmental variable based on your Istio ingress gateway's IP address:

$ export GATEWAY_URL=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

$ export GATEWAY_URL=$(kubectl get gateways.gateway.networking.k8s.io httpbin-gateway -n httpbin -ojsonpath='{.status.addresses[0].value}')

7) Run the following `curl` command to simulate a request with proxy addresses in the `X-Forwarded-For` header:

    $ curl -s -H 'X-Forwarded-For: 56.5.6.7, 72.9.5.6, 98.1.2.3' "$GATEWAY_URL/get?show_env=true" | jq '.headers["X-Forwarded-For"][0]'
      "56.5.6.7, 72.9.5.6, 98.1.2.3,10.244.0.1"

In the above example `$GATEWAY_URL` resolved to 10.244.0.1. This will not be the case in your environment.

The above output shows the request headers that the `httpbin` workload received. When the Istio gateway received this
request, it set the `X-Envoy-External-Address` header to the second to last (`numTrustedProxies: 2`) address in the
`X-Forwarded-For` header from your curl command. Additionally, the gateway appends its own IP to the `X-Forwarded-For`
header before forwarding it to the httpbin workload.

### Configuring X-Forwarded-Client-Cert Headers

From [Envoy's documentation](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-forwarded-client-cert)
regarding XFCC:

x-forwarded-client-cert (XFCC) is a proxy header which indicates certificate information of part or all of the clients
or proxies that a request has flowed through, on its way from the client to the server. A proxy may choose to
sanitize/append/forward the XFCC header before proxying the request.

To configure how XFCC headers are handled, set `forwardClientCertDetails` in your `IstioOperator`

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    defaultConfig:
      gatewayTopology:
        forwardClientCertDetails: <ENUM_VALUE>

where `ENUM_VALUE` can be of the following type.

| `ENUM_VALUE`          |                                                                                |
|-----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `UNDEFINED`           | Field is not set.                                                                                |
| `SANITIZE`            | Do not send the XFCC header to the next hop.                                                                                |
| `FORWARD_ONLY`        | When the client connection is mTLS (Mutual TLS), forward the XFCC header in the request.                                                                                |
| `APPEND_FORWARD`      | When the client connection is mTLS, append the client certificate information to the request’s XFCC header and forward it.                                              |
| `SANITIZE_SET`        | When the client connection is mTLS, reset the XFCC header with the client certificate information and send it to the next hop. This is the default value for a gateway. |
| `ALWAYS_FORWARD_ONLY` | Always forward the XFCC header in the request, regardless of whether the client connection is mTLS.                                                                     |

See the [Envoy documentation](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-forwarded-client-cert)
for examples of using this capability.

## PROXY Protocol

The [PROXY protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) allows for exchanging and preservation of client attributes between TCP proxies,
without relying on L7 protocols such as HTTP and the `X-Forwarded-For` and `X-Envoy-External-Address` headers. It is intended for scenarios where an external TCP load balancer needs to proxy TCP traffic through an Istio gateway to a backend TCP service and still expose client attributes such as source IP to upstream TCP service endpoints. PROXY protocol can be enabled via `EnvoyFilter`.

PROXY protocol is only supported for TCP traffic forwarding by Envoy. See the [Envoy documentation](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/other_features/ip_transparency#proxy-protocol) for more details, along with some important performance caveats.

PROXY protocol should not be used for L7 traffic, or for Istio gateways behind L7 load balancers.

If your external TCP load balancer is configured to forward TCP traffic and use the PROXY protocol, the Istio Gateway TCP listener must also be configured to accept the PROXY protocol.
To enable PROXY protocol on all TCP listeners on the gateways, set `proxyProtocol` in your `IstioOperator`. For example:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    defaultConfig:
      gatewayTopology:
        proxyProtocol: {}

Alternatively, deploy a gateway with the following pod annotation:

metadata:
  annotations:
    "proxy.istio.io/config": '{"gatewayTopology" : { "proxyProtocol": {} }}'

The client IP is retrieved from the PROXY protocol by the gateway and set (or appended) in the `X-Forwarded-For` and `X-Envoy-External-Address` header. Note that the PROXY protocol is mutually exclusive with L7 headers like `X-Forwarded-For` and `X-Envoy-External-Address`. When PROXY protocol is used in conjunction with the `gatewayTopology` configuration, the `numTrustedProxies` and the received `X-Forwarded-For` header takes precedence in determining the trusted client addresses, and PROXY protocol client information will be ignored.

Note that the above example only configures the Gateway to accept incoming PROXY protocol TCP traffic - See the [Envoy documentation](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/other_features/ip_transparency#proxy-protocol) for examples of how to configure Envoy itself to communicate with upstream services using PROXY protocol.

title  Configuring Gateway Network Topology description  How to configure gateway network topology  weight  60 keywords   traffic management ingress gateway  owner  istio wg networking maintainers test  yes status  Alpha             Forwarding external client attributes  IP address  certificate info  to destination workloads  Many applications require knowing the client IP address and certificate information of the originating request to behave properly  Notable cases include logging and audit tools that require the client IP be populated and security tools  such as Web Application Firewalls  WAF   that need this information to apply rule sets properly  The ability to provide client attributes to services has long been a staple of reverse proxies  To forward these client attributes to destination workloads  proxies use the  X Forwarded For   XFF  and  X Forwarded Client Cert   XFCC  headers   Today s networks vary widely in nature  but support for these attributes is a requirement no matter what the network topology is  This information should be preserved and forwarded whether the network uses cloud based Load Balancers  on premise Load Balancers  gateways that are exposed directly to the internet  gateways that serve many intermediate proxies  and other deployment topologies not specified   While Istio provides an  ingress gateway   docs tasks traffic management ingress ingress control    given the varieties of architectures mentioned above  reasonable defaults are not able to be shipped that support the proper forwarding of client attributes to the destination workloads  This becomes ever more vital as Istio multicluster deployment models become more common   For more information on  X Forwarded For   see the IETF s  RFC  https   tools ietf org html rfc7239       Configuring network topologies  Configuration of XFF and XFCC headers can be set globally for all gateway workloads via  MeshConfig  or per gateway using a pod annotation  For example  to configure globally during install or upgrade when using an  IstioOperator  custom resource    spec    meshConfig      defaultConfig        gatewayTopology          numTrustedProxies   VALUE          forwardClientCertDetails   ENUM VALUE    You can also configure both of these settings by adding the  proxy istio io config  annotation to the Pod spec of your Istio ingress gateway          metadata      annotations         proxy istio io config      gatewayTopology       numTrustedProxies    VALUE    forwardClientCertDetails    ENUM VALUE             Configuring X Forwarded For Headers  Applications rely on reverse proxies to forward client attributes in a request  such as  X Forward For  header  However  due to the variety of network topologies that Istio can be deployed in  you must set the  numTrustedProxies  to the number of trusted proxies deployed in front of the Istio gateway proxy  so that the client address can be extracted correctly  This controls the value populated by the ingress gateway in the  X Envoy External Address  header which can be reliably used by the upstream services to access client s original IP address   For example  if you have a cloud based Load Balancer and a reverse proxy in front of your Istio gateway  set  numTrustedProxies  to  2     Note that all proxies in front of the Istio gateway proxy must parse HTTP traffic and append to the  X Forwarded For  header at each hop  If the number of entries in the  X Forwarded For  header is less than the number of trusted hops configured  Envoy falls back to using the immediate downstream address as the trusted client address  Please refer to the  Envoy documentation  https   www envoyproxy io docs envoy latest configuration http http conn man headers x forwarded for  to understand how  X Forwarded For  headers and trusted client addresses are determined         Example using X Forwarded For capability with httpbin  1  Run the following command to create a file named  topology yaml  with  numTrustedProxies  set to  2  and install Istio              cat   EOF   topology yaml     apiVersion  install istio io v1alpha1     kind  IstioOperator     spec        meshConfig          defaultConfig            gatewayTopology              numTrustedProxies  2     EOF       istioctl install  f topology yaml                If you previously installed an Istio ingress gateway  restart all ingress gateway pods after step 1        1  Create an  httpbin  namespace              kubectl create namespace httpbin     namespace httpbin created       1  Set the  istio injection  label to  enabled  for sidecar injection              kubectl label   overwrite namespace httpbin istio injection enabled     namespace httpbin labeled       1  Deploy  httpbin  in the  httpbin  namespace              kubectl apply  n httpbin  f  samples httpbin httpbin yaml        1  Deploy a gateway associated with  httpbin           kubectl apply  n httpbin  f  samples httpbin httpbin gateway yaml           kubectl apply  n httpbin  f  samples httpbin gateway api httpbin gateway yaml    kubectl wait   for condition programmed gtw  n httpbin httpbin gateway       6  Set a local  GATEWAY URL  environmental variable based on your Istio ingress gateway s IP address          export GATEWAY URL   kubectl  n istio system get service istio ingressgateway  o jsonpath    status loadBalancer ingress 0  ip             export GATEWAY URL   kubectl get gateways gateway networking k8s io httpbin gateway  n httpbin  ojsonpath    status addresses 0  value          7  Run the following  curl  command to simulate a request with proxy addresses in the  X Forwarded For  header              curl  s  H  X Forwarded For  56 5 6 7  72 9 5 6  98 1 2 3    GATEWAY URL get show env true    jq   headers  X Forwarded For   0          56 5 6 7  72 9 5 6  98 1 2 3 10 244 0 1         In the above example   GATEWAY URL  resolved to 10 244 0 1  This will not be the case in your environment    The above output shows the request headers that the  httpbin  workload received  When the Istio gateway received this request  it set the  X Envoy External Address  header to the second to last   numTrustedProxies  2   address in the  X Forwarded For  header from your curl command  Additionally  the gateway appends its own IP to the  X Forwarded For  header before forwarding it to the httpbin workload       Configuring X Forwarded Client Cert Headers  From  Envoy s documentation  https   www envoyproxy io docs envoy latest configuration http http conn man headers x forwarded client cert  regarding XFCC    x forwarded client cert  XFCC  is a proxy header which indicates certificate information of part or all of the clients or proxies that a request has flowed through  on its way from the client to the server  A proxy may choose to sanitize append forward the XFCC header before proxying the request    To configure how XFCC headers are handled  set  forwardClientCertDetails  in your  IstioOperator    apiVersion  install istio io v1alpha1 kind  IstioOperator spec    meshConfig      defaultConfig        gatewayTopology          forwardClientCertDetails   ENUM VALUE    where  ENUM VALUE  can be of the following type      ENUM VALUE                                                                                UNDEFINED              Field is not set                                                                                SANITIZE               Do not send the XFCC header to the next hop                                                                                FORWARD ONLY           When the client connection is mTLS  Mutual TLS   forward the XFCC header in the request                                                                                APPEND FORWARD         When the client connection is mTLS  append the client certificate information to the request s XFCC header and forward it                                                    SANITIZE SET           When the client connection is mTLS  reset the XFCC header with the client certificate information and send it to the next hop  This is the default value for a gateway       ALWAYS FORWARD ONLY    Always forward the XFCC header in the request  regardless of whether the client connection is mTLS                                                                         See the  Envoy documentation  https   www envoyproxy io docs envoy latest configuration http http conn man headers x forwarded client cert  for examples of using this capability      PROXY Protocol  The  PROXY protocol  https   www haproxy org download 1 8 doc proxy protocol txt  allows for exchanging and preservation of client attributes between TCP proxies  without relying on L7 protocols such as HTTP and the  X Forwarded For  and  X Envoy External Address  headers  It is intended for scenarios where an external TCP load balancer needs to proxy TCP traffic through an Istio gateway to a backend TCP service and still expose client attributes such as source IP to upstream TCP service endpoints  PROXY protocol can be enabled via  EnvoyFilter     PROXY protocol is only supported for TCP traffic forwarding by Envoy  See the  Envoy documentation  https   www envoyproxy io docs envoy latest intro arch overview other features ip transparency proxy protocol  for more details  along with some important performance caveats   PROXY protocol should not be used for L7 traffic  or for Istio gateways behind L7 load balancers    If your external TCP load balancer is configured to forward TCP traffic and use the PROXY protocol  the Istio Gateway TCP listener must also be configured to accept the PROXY protocol  To enable PROXY protocol on all TCP listeners on the gateways  set  proxyProtocol  in your  IstioOperator   For example    apiVersion  install istio io v1alpha1 kind  IstioOperator spec    meshConfig      defaultConfig        gatewayTopology          proxyProtocol       Alternatively  deploy a gateway with the following pod annotation    metadata    annotations       proxy istio io config      gatewayTopology       proxyProtocol            The client IP is retrieved from the PROXY protocol by the gateway and set  or appended  in the  X Forwarded For  and  X Envoy External Address  header  Note that the PROXY protocol is mutually exclusive with L7 headers like  X Forwarded For  and  X Envoy External Address   When PROXY protocol is used in conjunction with the  gatewayTopology  configuration  the  numTrustedProxies  and the received  X Forwarded For  header takes precedence in determining the trusted client addresses  and PROXY protocol client information will be ignored   Note that the above example only configures the Gateway to accept incoming PROXY protocol TCP traffic   See the  Envoy documentation  https   www envoyproxy io docs envoy latest intro arch overview other features ip transparency proxy protocol  for examples of how to configure Envoy itself to communicate with upstream services using PROXY protocol


istio weight 60 In addition to capturing application traffic Istio can also capture DNS requests to improve the performance and usability of your mesh title DNS Proxying owner istio wg networking maintainers How to configure DNS proxying test yes keywords traffic management dns virtual machine

Answer:
---
title: DNS Proxying
description: How to configure DNS proxying.
weight: 60
keywords: [traffic-management,dns,virtual-machine]
owner: istio/wg-networking-maintainers
test: yes
---

In addition to capturing application traffic, Istio can also capture DNS requests to improve the performance and usability of your mesh.
When proxying DNS, all DNS requests from an application will be redirected to the sidecar, which stores a local mapping of domain names to IP addresses. If the request can be handled by the sidecar, it will directly return a response to the application, avoiding a roundtrip to the upstream DNS server. Otherwise, the request is forwarded upstream following the standard `/etc/resolv.conf` DNS configuration.

While Kubernetes provides DNS resolution for Kubernetes `Service`s out of the box, any custom `ServiceEntry`s will not be recognized. With this feature, `ServiceEntry` addresses can be resolved without requiring custom configuration of a DNS server. For Kubernetes `Service`s, the DNS response will be the same, but with reduced load on `kube-dns` and increased performance.

This functionality is also available for services running outside of Kubernetes. This means that all internal services can be resolved without clunky workarounds to expose Kubernetes DNS entries outside of the cluster.

## Getting started

This feature is not currently enabled by default. To enable it, install Istio with the following settings:

$ cat <<EOF | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    defaultConfig:
      proxyMetadata:
        # Enable basic DNS proxying
        ISTIO_META_DNS_CAPTURE: "true"
EOF

This can also be enabled on a per-pod basis with the [`proxy.istio.io/config` annotation](/docs/reference/config/annotations/):

kind: Deployment
metadata:
  name: curl
spec:
...
  template:
    metadata:
      annotations:
        proxy.istio.io/config: |
          proxyMetadata:
            ISTIO_META_DNS_CAPTURE: "true"
...

When deploying to a VM using [`istioctl workload entry configure`](/docs/setup/install/virtual-machine/), basic DNS proxying will be enabled by default.

## DNS capture In action

To try out the DNS capture, first setup a `ServiceEntry` for some external service:

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-address
spec:
  addresses:
  - 198.51.100.1
  hosts:
  - address.internal
  ports:
  - name: http
    number: 80
    protocol: HTTP
EOF

Bring up a client application to initiate the DNS request:

$ kubectl label namespace default istio-injection=enabled --overwrite
$ kubectl apply -f @samples/curl/curl.yaml@

Without the DNS capture, a request to `address.internal` would likely fail to resolve. Once this is enabled, you should instead get a response back based on the configured `address`:

$ kubectl exec deploy/curl -- curl -sS -v address.internal
*   Trying 198.51.100.1:80...

## Address auto allocation

In the above example, you had a predefined IP address for the service to which you sent the request. However, it's common to access external services that do not have stable addresses, and instead rely on DNS. In this case, the DNS proxy will not have enough information to return a response, and will need to forward DNS requests upstream.

This is especially problematic with TCP traffic. Unlike HTTP requests, which are routed based on `Host` headers, TCP carries much less information; you can only route on the destination IP and port number. Because you don't have a stable IP for the backend, you cannot route based on that either, leaving only port number, which leads to conflicts when multiple `ServiceEntry`s for TCP services share the same port. Refer
to [the following section](#external-tcp-services-without-vips) for more details.

To work around these issues, the DNS proxy additionally supports automatically allocating addresses for `ServiceEntry`s that do not explicitly define one. The DNS response will include a distinct and automatically assigned address for each `ServiceEntry`. The proxy is then configured to match requests to this IP address, and forward the request to the corresponding `ServiceEntry`. Istio will automatically allocate non-routable VIPs (from the Class E subnet) to such services as long as they do not use a wildcard host. The Istio agent on the sidecar will use the VIPs as responses to the DNS lookup queries from the application. Envoy can now clearly distinguish traffic bound for each external TCP service and forward it to the right target.

Because this feature modifies DNS responses, it may not be compatible with all applications.

To try this out, configure another `ServiceEntry`:

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-auto
spec:
  hosts:
  - auto.internal
  ports:
  - name: http
    number: 80
    protocol: HTTP
  resolution: DNS
EOF

Now, send a request:

$ kubectl exec deploy/curl -- curl -sS -v auto.internal
*   Trying 240.240.0.1:80...

As you can see, the request is sent to an automatically allocated address, `240.240.0.1`. These addresses will be picked from the `240.240.0.0/16` reserved IP address range to avoid conflicting with real services.

Users also have the flexibility for more granular configuration by adding the label `networking.istio.io/enable-autoallocate-ip="true/false"` to their `ServiceEntry`. This label configures whether a `ServiceEntry` without any `spec.addresses` set should get an IP address automatically allocated for it.

To try this out, update the existing `ServiceEntry` with the opt-out label:

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-auto
  labels:
    networking.istio.io/enable-autoallocate-ip: "false"
spec:
  hosts:
  - auto.internal
  ports:
  - name: http
    number: 80
    protocol: HTTP
  resolution: DNS
EOF

Now, send a request and verify that the auto allocation is no longer happening:

$ kubectl exec deploy/curl -- curl -sS -v auto.internal
* Could not resolve host: auto.internal
* shutting down connection #0

## External TCP services without VIPs

By default, Istio has a limitation when routing external TCP traffic because it is unable to distinguish between multiple TCP services on the same port. This limitation is particularly apparent when using third party databases such as AWS Relational Database Service or any database setup with geographical redundancy. Similar, but different external TCP services, cannot be handled separately by default. For the sidecar to distinguish traffic between two different TCP services that are outside of the mesh, the services must be on different ports or they need to have globally unique VIPs.

For example, if you have two external database services, `mysql-instance1` and `mysql-instance2`, and you create service entries for both, client sidecars will still have a single listener on `0.0.0.0:{port}` that looks up the IP address of only `mysql-instance1`, from public DNS servers, and forwards traffic to it. It cannot route traffic to `mysql-instance2` because it has no way of distinguishing whether traffic arriving at `0.0.0.0:{port}` is bound for `mysql-instance1` or `mysql-instance2`.

The following example shows how DNS proxying can be used to solve this problem.
A virtual IP address will be assigned to every service entry so that client sidecars can clearly distinguish traffic bound for each external TCP service.

1.  Update the Istio configuration specified in the [Getting Started](#getting-started) section to also configure `discoverySelectors` that restrict the mesh to namespaces with `istio-injection` enabled. This will let us use any other namespaces in the cluster to run TCP services outside of the mesh.

    $ cat <<EOF | istioctl install -y -f -
    apiVersion: install.istio.io/v1alpha1
    kind: IstioOperator
    spec:
      meshConfig:
        defaultConfig:
          proxyMetadata:
            # Enable basic DNS proxying
            ISTIO_META_DNS_CAPTURE: "true"
        # discoverySelectors configuration below is just used for simulating the external service TCP scenario,
        # so that we do not have to use an external site for testing.
        discoverySelectors:
        - matchLabels:
            istio-injection: enabled
    EOF

1.  Deploy the first external sample TCP application:

    $ kubectl create ns external-1
    $ kubectl -n external-1 apply -f samples/tcp-echo/tcp-echo.yaml

1.  Deploy the second external sample TCP application:

    $ kubectl create ns external-2
    $ kubectl -n external-2 apply -f samples/tcp-echo/tcp-echo.yaml

1.  Configure `ServiceEntry` to reach external services:

    $ kubectl apply -f - <<EOF
    apiVersion: networking.istio.io/v1
    kind: ServiceEntry
    metadata:
      name: external-svc-1
    spec:
      hosts:
      - tcp-echo.external-1.svc.cluster.local
      ports:
      - name: external-svc-1
        number: 9000
        protocol: TCP
      resolution: DNS
    ---
    apiVersion: networking.istio.io/v1
    kind: ServiceEntry
    metadata:
      name: external-svc-2
    spec:
      hosts:
      - tcp-echo.external-2.svc.cluster.local
      ports:
      - name: external-svc-2
        number: 9000
        protocol: TCP
      resolution: DNS
    EOF

1.  Verify listeners are configured separately for each service at the client side:

    $ istioctl pc listener deploy/curl | grep tcp-echo | awk '{printf "ADDRESS=%s, DESTINATION=%s %s\n", $1, $4, $5}'
    ADDRESS=240.240.105.94, DESTINATION=Cluster: outbound|9000||tcp-echo.external-2.svc.cluster.local
    ADDRESS=240.240.69.138, DESTINATION=Cluster: outbound|9000||tcp-echo.external-1.svc.cluster.local

## Cleanup

$ kubectl -n external-1 delete -f @samples/tcp-echo/tcp-echo.yaml@
$ kubectl -n external-2 delete -f @samples/tcp-echo/tcp-echo.yaml@
$ kubectl delete -f @samples/curl/curl.yaml@
$ istioctl uninstall --purge -y
$ kubectl delete ns istio-system external-1 external-2
$ kubectl label namespace default istio-injection-

title  DNS Proxying description  How to configure DNS proxying  weight  60 keywords   traffic management dns virtual machine  owner  istio wg networking maintainers test  yes      In addition to capturing application traffic  Istio can also capture DNS requests to improve the performance and usability of your mesh  When proxying DNS  all DNS requests from an application will be redirected to the sidecar  which stores a local mapping of domain names to IP addresses  If the request can be handled by the sidecar  it will directly return a response to the application  avoiding a roundtrip to the upstream DNS server  Otherwise  the request is forwarded upstream following the standard   etc resolv conf  DNS configuration   While Kubernetes provides DNS resolution for Kubernetes  Service s out of the box  any custom  ServiceEntry s will not be recognized  With this feature   ServiceEntry  addresses can be resolved without requiring custom configuration of a DNS server  For Kubernetes  Service s  the DNS response will be the same  but with reduced load on  kube dns  and increased performance   This functionality is also available for services running outside of Kubernetes  This means that all internal services can be resolved without clunky workarounds to expose Kubernetes DNS entries outside of the cluster      Getting started  This feature is not currently enabled by default  To enable it  install Istio with the following settings      cat   EOF   istioctl install  y  f   apiVersion  install istio io v1alpha1 kind  IstioOperator spec    meshConfig      defaultConfig        proxyMetadata            Enable basic DNS proxying         ISTIO META DNS CAPTURE   true  EOF   This can also be enabled on a per pod basis with the   proxy istio io config  annotation   docs reference config annotations      kind  Deployment metadata    name  curl spec        template      metadata        annotations          proxy istio io config              proxyMetadata              ISTIO META DNS CAPTURE   true         When deploying to a VM using   istioctl workload entry configure    docs setup install virtual machine    basic DNS proxying will be enabled by default       DNS capture In action  To try out the DNS capture  first setup a  ServiceEntry  for some external service      kubectl apply  f     EOF apiVersion  networking istio io v1 kind  ServiceEntry metadata    name  external address spec    addresses      198 51 100 1   hosts      address internal   ports      name  http     number  80     protocol  HTTP EOF   Bring up a client application to initiate the DNS request      kubectl label namespace default istio injection enabled   overwrite   kubectl apply  f  samples curl curl yaml    Without the DNS capture  a request to  address internal  would likely fail to resolve  Once this is enabled  you should instead get a response back based on the configured  address       kubectl exec deploy curl    curl  sS  v address internal     Trying 198 51 100 1 80         Address auto allocation  In the above example  you had a predefined IP address for the service to which you sent the request  However  it s common to access external services that do not have stable addresses  and instead rely on DNS  In this case  the DNS proxy will not have enough information to return a response  and will need to forward DNS requests upstream   This is especially problematic with TCP traffic  Unlike HTTP requests  which are routed based on  Host  headers  TCP carries much less information  you can only route on the destination IP and port number  Because you don t have a stable IP for the backend  you cannot route based on that either  leaving only port number  which leads to conflicts when multiple  ServiceEntry s for TCP services share the same port  Refer to  the following section   external tcp services without vips  for more details   To work around these issues  the DNS proxy additionally supports automatically allocating addresses for  ServiceEntry s that do not explicitly define one  The DNS response will include a distinct and automatically assigned address for each  ServiceEntry   The proxy is then configured to match requests to this IP address  and forward the request to the corresponding  ServiceEntry   Istio will automatically allocate non routable VIPs  from the Class E subnet  to such services as long as they do not use a wildcard host  The Istio agent on the sidecar will use the VIPs as responses to the DNS lookup queries from the application  Envoy can now clearly distinguish traffic bound for each external TCP service and forward it to the right target    Because this feature modifies DNS responses  it may not be compatible with all applications    To try this out  configure another  ServiceEntry       kubectl apply  f     EOF apiVersion  networking istio io v1 kind  ServiceEntry metadata    name  external auto spec    hosts      auto internal   ports      name  http     number  80     protocol  HTTP   resolution  DNS EOF   Now  send a request      kubectl exec deploy curl    curl  sS  v auto internal     Trying 240 240 0 1 80      As you can see  the request is sent to an automatically allocated address   240 240 0 1   These addresses will be picked from the  240 240 0 0 16  reserved IP address range to avoid conflicting with real services   Users also have the flexibility for more granular configuration by adding the label  networking istio io enable autoallocate ip  true false   to their  ServiceEntry   This label configures whether a  ServiceEntry  without any  spec addresses  set should get an IP address automatically allocated for it   To try this out  update the existing  ServiceEntry  with the opt out label      kubectl apply  f     EOF apiVersion  networking istio io v1 kind  ServiceEntry metadata    name  external auto   labels      networking istio io enable autoallocate ip   false  spec    hosts      auto internal   ports      name  http     number  80     protocol  HTTP   resolution  DNS EOF   Now  send a request and verify that the auto allocation is no longer happening      kubectl exec deploy curl    curl  sS  v auto internal   Could not resolve host  auto internal   shutting down connection  0      External TCP services without VIPs  By default  Istio has a limitation when routing external TCP traffic because it is unable to distinguish between multiple TCP services on the same port  This limitation is particularly apparent when using third party databases such as AWS Relational Database Service or any database setup with geographical redundancy  Similar  but different external TCP services  cannot be handled separately by default  For the sidecar to distinguish traffic between two different TCP services that are outside of the mesh  the services must be on different ports or they need to have globally unique VIPs   For example  if you have two external database services   mysql instance1  and  mysql instance2   and you create service entries for both  client sidecars will still have a single listener on  0 0 0 0  port   that looks up the IP address of only  mysql instance1   from public DNS servers  and forwards traffic to it  It cannot route traffic to  mysql instance2  because it has no way of distinguishing whether traffic arriving at  0 0 0 0  port   is bound for  mysql instance1  or  mysql instance2    The following example shows how DNS proxying can be used to solve this problem  A virtual IP address will be assigned to every service entry so that client sidecars can clearly distinguish traffic bound for each external TCP service   1   Update the Istio configuration specified in the  Getting Started   getting started  section to also configure  discoverySelectors  that restrict the mesh to namespaces with  istio injection  enabled  This will let us use any other namespaces in the cluster to run TCP services outside of the mesh              cat   EOF   istioctl install  y  f       apiVersion  install istio io v1alpha1     kind  IstioOperator     spec        meshConfig          defaultConfig            proxyMetadata                Enable basic DNS proxying             ISTIO META DNS CAPTURE   true            discoverySelectors configuration below is just used for simulating the external service TCP scenario            so that we do not have to use an external site for testing          discoverySelectors            matchLabels              istio injection  enabled     EOF       1   Deploy the first external sample TCP application              kubectl create ns external 1       kubectl  n external 1 apply  f samples tcp echo tcp echo yaml       1   Deploy the second external sample TCP application              kubectl create ns external 2       kubectl  n external 2 apply  f samples tcp echo tcp echo yaml       1   Configure  ServiceEntry  to reach external services              kubectl apply  f     EOF     apiVersion  networking istio io v1     kind  ServiceEntry     metadata        name  external svc 1     spec        hosts          tcp echo external 1 svc cluster local       ports          name  external svc 1         number  9000         protocol  TCP       resolution  DNS             apiVersion  networking istio io v1     kind  ServiceEntry     metadata        name  external svc 2     spec        hosts          tcp echo external 2 svc cluster local       ports          name  external svc 2         number  9000         protocol  TCP       resolution  DNS     EOF       1   Verify listeners are configured separately for each service at the client side              istioctl pc listener deploy curl   grep tcp echo   awk   printf  ADDRESS  s  DESTINATION  s  s n    1   4   5       ADDRESS 240 240 105 94  DESTINATION Cluster  outbound 9000  tcp echo external 2 svc cluster local     ADDRESS 240 240 69 138  DESTINATION Cluster  outbound 9000  tcp echo external 1 svc cluster local          Cleanup     kubectl  n external 1 delete  f  samples tcp echo tcp echo yaml    kubectl  n external 2 delete  f  samples tcp echo tcp echo yaml    kubectl delete  f  samples curl curl yaml    istioctl uninstall   purge  y   kubectl delete ns istio system external 1 external 2   kubectl label namespace default istio injection


istio weight 70 test no owner istio wg networking maintainers Within a multicluster mesh traffic rules specific to the cluster topology may be desirable This document describes How to configure how traffic is distributed among clusters in the mesh keywords traffic management multicluster title Multi cluster Traffic Management

Answer:
---
title: Multi-cluster Traffic Management
description: How to configure how traffic is distributed among clusters in the mesh.
weight: 70
keywords: [traffic-management,multicluster]
owner: istio/wg-networking-maintainers
test: no
---

Within a multicluster mesh, traffic rules specific to the cluster topology may be desirable. This document describes
a few ways to manage traffic in a multicluster mesh. Before reading this guide:

1. Read [Deployment Models](/docs/ops/deployment/deployment-models/#multiple-clusters)
1. Make sure your deployed services follow the concept of namespace sameness.

## Keeping traffic in-cluster

In some cases the default cross-cluster load balancing behavior is not desirable. To keep traffic "cluster-local" (i.e.
traffic sent from `cluster-a` will only reach destinations in `cluster-a`), mark hostnames or wildcards as `clusterLocal`
using [`MeshConfig.serviceSettings`](/docs/reference/config/istio.mesh.v1alpha1/#MeshConfig-ServiceSettings-Settings).

For example, you can enforce cluster-local traffic for an individual service, all services in a particular namespace, or globally for all services in the mesh, as follows:

serviceSettings:
- settings:
    clusterLocal: true
  hosts:
  - "mysvc.myns.svc.cluster.local"

serviceSettings:
- settings:
    clusterLocal: true
  hosts:
  - "*.myns.svc.cluster.local"

serviceSettings:
- settings:
    clusterLocal: true
  hosts:
  - "*"

## Partitioning Services {#partitioning-services}

[`DestinationRule.subsets`](/docs/reference/config/networking/destination-rule/#Subset) allows partitioning a service
by selecting labels. These labels can be the labels from Kubernetes metadata, or from [built-in labels](/docs/reference/config/labels/).
One of these built-in labels, `topology.istio.io/cluster`, in the subset selector for a `DestinationRule` allows
creating per-cluster subsets.

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: mysvc-per-cluster-dr
spec:
  host: mysvc.myns.svc.cluster.local
  subsets:
  - name: cluster-1
    labels:
      topology.istio.io/cluster: cluster-1
  - name: cluster-2
    labels:
      topology.istio.io/cluster: cluster-2

Using these subsets you can create various routing rules based on the cluster such as [mirroring](/docs/tasks/traffic-management/mirroring/)
or [shifting](/docs/tasks/traffic-management/traffic-shifting/).

This provides another option to create cluster-local traffic rules by restricting the destination subset in a `VirtualService`:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: mysvc-cluster-local-vs
spec:
  hosts:
  - mysvc.myns.svc.cluster.local
  http:
  - name: "cluster-1-local"
    match:
    - sourceLabels:
        topology.istio.io/cluster: "cluster-1"
    route:
    - destination:
        host: mysvc.myns.svc.cluster.local
        subset: cluster-1
  - name: "cluster-2-local"
    match:
    - sourceLabels:
        topology.istio.io/cluster: "cluster-2"
    route:
    - destination:
        host: mysvc.myns.svc.cluster.local
        subset: cluster-2

Using subset-based routing this way to control cluster-local traffic, as opposed to
[`MeshConfig.serviceSettings`](/docs/reference/config/istio.mesh.v1alpha1/#MeshConfig-ServiceSettings-Settings),
has the downside of mixing service-level policy with topology-level policy.
For example, a rule that sends 10% of traffic to `v2` of a service will need twice the
number of subsets (e.g., `cluster-1-v2`, `cluster-2-v2`).
This approach is best limited to situations where more granular control of cluster-based routing is needed.

title  Multi cluster Traffic Management description  How to configure how traffic is distributed among clusters in the mesh  weight  70 keywords   traffic management multicluster  owner  istio wg networking maintainers test  no      Within a multicluster mesh  traffic rules specific to the cluster topology may be desirable  This document describes a few ways to manage traffic in a multicluster mesh  Before reading this guide   1  Read  Deployment Models   docs ops deployment deployment models  multiple clusters  1  Make sure your deployed services follow the concept of namespace sameness      Keeping traffic in cluster  In some cases the default cross cluster load balancing behavior is not desirable  To keep traffic  cluster local   i e  traffic sent from  cluster a  will only reach destinations in  cluster a    mark hostnames or wildcards as  clusterLocal  using   MeshConfig serviceSettings    docs reference config istio mesh v1alpha1  MeshConfig ServiceSettings Settings    For example  you can enforce cluster local traffic for an individual service  all services in a particular namespace  or globally for all services in the mesh  as follows        serviceSettings    settings      clusterLocal  true   hosts       mysvc myns svc cluster local         serviceSettings    settings      clusterLocal  true   hosts         myns svc cluster local         serviceSettings    settings      clusterLocal  true   hosts                   Partitioning Services   partitioning services     DestinationRule subsets    docs reference config networking destination rule  Subset  allows partitioning a service by selecting labels  These labels can be the labels from Kubernetes metadata  or from  built in labels   docs reference config labels    One of these built in labels   topology istio io cluster   in the subset selector for a  DestinationRule  allows creating per cluster subsets    apiVersion  networking istio io v1 kind  DestinationRule metadata    name  mysvc per cluster dr spec    host  mysvc myns svc cluster local   subsets      name  cluster 1     labels        topology istio io cluster  cluster 1     name  cluster 2     labels        topology istio io cluster  cluster 2   Using these subsets you can create various routing rules based on the cluster such as  mirroring   docs tasks traffic management mirroring   or  shifting   docs tasks traffic management traffic shifting     This provides another option to create cluster local traffic rules by restricting the destination subset in a  VirtualService     apiVersion  networking istio io v1 kind  VirtualService metadata    name  mysvc cluster local vs spec    hosts      mysvc myns svc cluster local   http      name   cluster 1 local      match        sourceLabels          topology istio io cluster   cluster 1      route        destination          host  mysvc myns svc cluster local         subset  cluster 1     name   cluster 2 local      match        sourceLabels          topology istio io cluster   cluster 2      route        destination          host  mysvc myns svc cluster local         subset  cluster 2   Using subset based routing this way to control cluster local traffic  as opposed to   MeshConfig serviceSettings    docs reference config istio mesh v1alpha1  MeshConfig ServiceSettings Settings   has the downside of mixing service level policy with topology level policy  For example  a rule that sends 10  of traffic to  v2  of a service will need twice the number of subsets  e g    cluster 1 v2    cluster 2 v2    This approach is best limited to situations where more granular control of cluster based routing is needed


istio linktitle Traffic Routing owner istio wg networking maintainers weight 30 How Istio routes traffic through the mesh keywords traffic management proxy title Understanding Traffic Routing test n a

Answer:
---
title: Understanding Traffic Routing
linktitle: Traffic Routing
description: How Istio routes traffic through the mesh.
weight: 30
keywords: [traffic-management,proxy]
owner: istio/wg-networking-maintainers
test: n/a
---

One of the goals of Istio is to act as a "transparent proxy" which can be dropped into an existing cluster, allowing traffic to continue to flow as before.
However, there are powerful ways Istio can manage traffic differently than a typical Kubernetes cluster because of the additional features such as request load balancing.
To understand what is happening in your mesh, it is important to understand how Istio routes traffic.

This document describes low level implementation details. For a higher level overview, check out the traffic management [Concepts](/docs/concepts/traffic-management/) or [Tasks](/docs/tasks/traffic-management/).

## Frontends and backends

In traffic routing in Istio, there are two primary phases:

* The "frontend" refers to how we match the type of traffic we are handling.
  This is necessary to identify which backend to route traffic to, and which policies to apply.
  For example, we may read the `Host` header of `http.ns.svc.cluster.local` and identify the request is intended for the `http` Service.
  More information on how this matching works can be found below.
* The "backend" refers to where we send traffic once we have matched it.
  Using the example above, after identifying the request as targeting the `http` Service, we would send it to an endpoint in that Service.
  However, this selection is not always so simple; Istio allows customization of this logic, through `VirtualService` routing rules.

Standard Kubernetes networking has these same concepts, too, but they are much simpler and generally hidden.
When a `Service` is created, there is typically an associated frontend -- the automatically created DNS name (such as `http.ns.svc.cluster.local`),
and an automatically created IP address to represent the service (the `ClusterIP`).
Similarly, a backend is also created - the `Endpoints` or `EndpointSlice` - which represents all of the pods selected by the service.

## Protocols

Unlike Kubernetes, Istio has the ability to process application level protocols such as HTTP and TLS.
This allows for different types of [frontend](#frontends-and-backends) matching than is available in Kubernetes.

In general, there are three classes of protocols Istio understands:

* HTTP, which includes HTTP/1.1, HTTP/2, and gRPC. Note that this does not include TLS encrypted traffic (HTTPS).
* TLS, which includes HTTPS.
* Raw TCP bytes.

The [protocol selection](/docs/ops/configuration/traffic-management/protocol-selection/) document describes how Istio decides which protocol is used.

The use of "TCP" can be confusing, as in other contexts it is used to distinguish between other L4 protocols, such as UDP.
When referring to the TCP protocol in Istio, this typically means we are treating it as a raw stream of bytes,
and not parsing application level protocols such as TLS or HTTP.

## Traffic Routing

When an Envoy proxy receives a request, it must decide where, if anywhere, to forward it to.
By default, this will be to the original service that was requested, unless [customized](/docs/tasks/traffic-management/traffic-shifting/).
How this works depends on the protocol used.

### TCP

When processing TCP traffic, Istio has a very small amount of useful information to route the connection - only the destination IP and Port.
These attributes are used to determine the intended Service; the proxy is configured to listen on each service IP (`<Kubernetes ClusterIP>:<Port>`) pair and forward traffic to the upstream service.

For customizations, a TCP `VirtualService` can be configured, which allows [matching on specific IPs and ports](/docs/reference/config/networking/virtual-service/#L4MatchAttributes) and routing it to different upstream services than requested.

### TLS

When processing TLS traffic, Istio has slightly more information available than raw TCP: we can inspect the [SNI](https://en.wikipedia.org/wiki/Server_Name_Indication) field presented during the TLS handshake.

For standard Services, the same IP:Port matching is used as for raw TCP.
However, for services that do not have a Service IP defined, such as [ExternalName services](#externalname-services), the SNI field will be used for routing.

Additionally, custom routing can be configured with a TLS `VirtualService` to [match on SNI](/docs/reference/config/networking/virtual-service/#TLSMatchAttributes) and route requests to custom destinations.

### HTTP

HTTP allows much richer routing than TCP and TLS. With HTTP, you can route individual HTTP requests, rather than just connections.
In addition, a [number of rich attributes](/docs/reference/config/networking/virtual-service/#HTTPMatchRequest) are available, such as host, path, headers, query parameters, etc.

While TCP and TLS traffic generally behave the same with or without Istio (assuming no configuration has been applied to customize the routing), HTTP has significant differences.

* Istio will load balance individual requests. In general, this is highly desirable, especially in scenarios with long-lived connections such as gRPC and HTTP/2, where connection level load balancing is ineffective.
* Requests are routed based on the port and *`Host` header*, rather than port and IP. This means the destination IP address is effectively ignored. For example, `curl 8.8.8.8 -H "Host: productpage.default.svc.cluster.local"`, would be routed to the `productpage` Service.

## Unmatched traffic

If traffic cannot be matched using one of the methods described above, it is treated as [passthrough traffic](/docs/tasks/traffic-management/egress/egress-control/#envoy-passthrough-to-external-services).
By default, these requests will be forwarded as-is, which ensures that traffic to services that Istio is not aware of (such as external services that do not have `ServiceEntry`s created) continues to function.
Note that when these requests are forwarded, mutual TLS will not be used and telemetry collection is limited.

## Service types

Along with standard `ClusterIP` Services, Istio supports the full range of Kubernetes Services, with some caveats.

### `LoadBalancer` and `NodePort` Services

These Services are supersets of `ClusterIP` Services, and are mostly concerned with allowing access from external clients.
These service types are supported and behave exactly like standard `ClusterIP` Services.

### Headless Services

A [headless Service](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services) is a Service that does not have a `ClusterIP` assigned.
Instead, the DNS response will contain the IP addresses of each endpoint (i.e. the Pod IP) that is a part of the Service.

In general, Istio does not configure listeners for each Pod IP, as it works at the Service level.
However, to support headless services, listeners are set up for each IP:Port pair in the headless service.
An exception to this is for protocols declared as HTTP, which will match traffic by the `Host` header.

Without Istio, the `ports` field of a headless service is not strictly required because requests go directly to pod IPs, which can accept traffic on all ports.
However, with Istio the port must be declared in the Service, or it will [not be matched](/docs/ops/configuration/traffic-management/traffic-routing/#unmatched-traffic).

### ExternalName Services

An [ExternalName Service](https://kubernetes.io/docs/concepts/services-networking/service/#externalname) is essentially just a DNS alias.

To make things more concrete, consider the following example:

apiVersion: v1
kind: Service
metadata:
  name: alias
spec:
  type: ExternalName
  externalName: concrete.example.com

Because there is no `ClusterIP` nor pod IPs to match on, for TCP traffic there are no changes at all to traffic matching in Istio.
When Istio receives the request, they will see the IP for `concrete.example.com`.
If this is a service Istio knows about, it will be routed as described [above](#tcp).
If not, it will be handled as [unmatched traffic](#unmatched-traffic)

For HTTP and TLS, which match on hostname, things are a bit different.
If the target service (`concrete.example.com`) is a service Istio knows about, then the alias hostname (`alias.default.svc.cluster.local`) will be added
as an _additional_ match to the [TLS](#tls) or [HTTP](#http) matching.
If not, there will be no changes, so it will be handled as [unmatched traffic](#unmatched-traffic).

An `ExternalName` service can never be a [backend](#frontends-and-backends) on its own.
Instead, it is only ever used as additional [frontend](#frontends-and-backends) matches to existing Services.
If one is explicitly used as a backend, such as in a `VirtualService` destination, the same aliasing applies.
That is, if `alias.default.svc.cluster.local` is set as the destination, then requests will go to the `concrete.example.com`.
If that hostname is not known to Istio, the requests will fail; in this case, a `ServiceEntry` for `concrete.example.com` would make this configuration work.

### ServiceEntry

In addition to Kubernetes Services, [Service Entries](/docs/reference/config/networking/service-entry/#ServiceEntry) can be created to extend the set of services known to Istio.
This can be useful to ensure that traffic to external services, such as `example.com`, get the functionality of Istio.

A ServiceEntry with `addresses` set will perform routing just like a `ClusterIP` Service.

However, for Service Entries without any `addresses`, all IPs on the port will be matched.
This may prevent [unmatched traffic](#unmatched-traffic) on the same port from being forwarded correctly.
As such, it is best to avoid these where possible, or use dedicated ports when needed.
HTTP and TLS do not share this constraint, as routing is done based on the hostname/SNI.

The `addresses` field and `endpoints` field are often confused.
`addresses` refers to IPs that will be matched against, while endpoints refer to the set of IPs we will send traffic to.

For example, the Service entry below would match traffic for `1.1.1.1`, and send the request to `2.2.2.2` and `3.3.3.3` following the configured load balancing policy:

addresses: [1.1.1.1]
resolution: STATIC
endpoints:
- address: 2.2.2.2
- address: 3.3.3.3

title  Understanding Traffic Routing linktitle  Traffic Routing description  How Istio routes traffic through the mesh  weight  30 keywords   traffic management proxy  owner  istio wg networking maintainers test  n a      One of the goals of Istio is to act as a  transparent proxy  which can be dropped into an existing cluster  allowing traffic to continue to flow as before  However  there are powerful ways Istio can manage traffic differently than a typical Kubernetes cluster because of the additional features such as request load balancing  To understand what is happening in your mesh  it is important to understand how Istio routes traffic    This document describes low level implementation details  For a higher level overview  check out the traffic management  Concepts   docs concepts traffic management   or  Tasks   docs tasks traffic management         Frontends and backends  In traffic routing in Istio  there are two primary phases     The  frontend  refers to how we match the type of traffic we are handling    This is necessary to identify which backend to route traffic to  and which policies to apply    For example  we may read the  Host  header of  http ns svc cluster local  and identify the request is intended for the  http  Service    More information on how this matching works can be found below    The  backend  refers to where we send traffic once we have matched it    Using the example above  after identifying the request as targeting the  http  Service  we would send it to an endpoint in that Service    However  this selection is not always so simple  Istio allows customization of this logic  through  VirtualService  routing rules   Standard Kubernetes networking has these same concepts  too  but they are much simpler and generally hidden  When a  Service  is created  there is typically an associated frontend    the automatically created DNS name  such as  http ns svc cluster local    and an automatically created IP address to represent the service  the  ClusterIP    Similarly  a backend is also created   the  Endpoints  or  EndpointSlice    which represents all of the pods selected by the service      Protocols  Unlike Kubernetes  Istio has the ability to process application level protocols such as HTTP and TLS  This allows for different types of  frontend   frontends and backends  matching than is available in Kubernetes   In general  there are three classes of protocols Istio understands     HTTP  which includes HTTP 1 1  HTTP 2  and gRPC  Note that this does not include TLS encrypted traffic  HTTPS     TLS  which includes HTTPS    Raw TCP bytes   The  protocol selection   docs ops configuration traffic management protocol selection   document describes how Istio decides which protocol is used   The use of  TCP  can be confusing  as in other contexts it is used to distinguish between other L4 protocols  such as UDP  When referring to the TCP protocol in Istio  this typically means we are treating it as a raw stream of bytes  and not parsing application level protocols such as TLS or HTTP      Traffic Routing  When an Envoy proxy receives a request  it must decide where  if anywhere  to forward it to  By default  this will be to the original service that was requested  unless  customized   docs tasks traffic management traffic shifting    How this works depends on the protocol used       TCP  When processing TCP traffic  Istio has a very small amount of useful information to route the connection   only the destination IP and Port  These attributes are used to determine the intended Service  the proxy is configured to listen on each service IP    Kubernetes ClusterIP   Port    pair and forward traffic to the upstream service   For customizations  a TCP  VirtualService  can be configured  which allows  matching on specific IPs and ports   docs reference config networking virtual service  L4MatchAttributes  and routing it to different upstream services than requested       TLS  When processing TLS traffic  Istio has slightly more information available than raw TCP  we can inspect the  SNI  https   en wikipedia org wiki Server Name Indication  field presented during the TLS handshake   For standard Services  the same IP Port matching is used as for raw TCP  However  for services that do not have a Service IP defined  such as  ExternalName services   externalname services   the SNI field will be used for routing   Additionally  custom routing can be configured with a TLS  VirtualService  to  match on SNI   docs reference config networking virtual service  TLSMatchAttributes  and route requests to custom destinations       HTTP  HTTP allows much richer routing than TCP and TLS  With HTTP  you can route individual HTTP requests  rather than just connections  In addition  a  number of rich attributes   docs reference config networking virtual service  HTTPMatchRequest  are available  such as host  path  headers  query parameters  etc   While TCP and TLS traffic generally behave the same with or without Istio  assuming no configuration has been applied to customize the routing   HTTP has significant differences     Istio will load balance individual requests  In general  this is highly desirable  especially in scenarios with long lived connections such as gRPC and HTTP 2  where connection level load balancing is ineffective    Requests are routed based on the port and   Host  header   rather than port and IP  This means the destination IP address is effectively ignored  For example   curl 8 8 8 8  H  Host  productpage default svc cluster local    would be routed to the  productpage  Service      Unmatched traffic  If traffic cannot be matched using one of the methods described above  it is treated as  passthrough traffic   docs tasks traffic management egress egress control  envoy passthrough to external services   By default  these requests will be forwarded as is  which ensures that traffic to services that Istio is not aware of  such as external services that do not have  ServiceEntry s created  continues to function  Note that when these requests are forwarded  mutual TLS will not be used and telemetry collection is limited      Service types  Along with standard  ClusterIP  Services  Istio supports the full range of Kubernetes Services  with some caveats        LoadBalancer  and  NodePort  Services  These Services are supersets of  ClusterIP  Services  and are mostly concerned with allowing access from external clients  These service types are supported and behave exactly like standard  ClusterIP  Services       Headless Services  A  headless Service  https   kubernetes io docs concepts services networking service  headless services  is a Service that does not have a  ClusterIP  assigned  Instead  the DNS response will contain the IP addresses of each endpoint  i e  the Pod IP  that is a part of the Service   In general  Istio does not configure listeners for each Pod IP  as it works at the Service level  However  to support headless services  listeners are set up for each IP Port pair in the headless service  An exception to this is for protocols declared as HTTP  which will match traffic by the  Host  header    Without Istio  the  ports  field of a headless service is not strictly required because requests go directly to pod IPs  which can accept traffic on all ports  However  with Istio the port must be declared in the Service  or it will  not be matched   docs ops configuration traffic management traffic routing  unmatched traffic         ExternalName Services  An  ExternalName Service  https   kubernetes io docs concepts services networking service  externalname  is essentially just a DNS alias   To make things more concrete  consider the following example    apiVersion  v1 kind  Service metadata    name  alias spec    type  ExternalName   externalName  concrete example com   Because there is no  ClusterIP  nor pod IPs to match on  for TCP traffic there are no changes at all to traffic matching in Istio  When Istio receives the request  they will see the IP for  concrete example com   If this is a service Istio knows about  it will be routed as described  above   tcp   If not  it will be handled as  unmatched traffic   unmatched traffic   For HTTP and TLS  which match on hostname  things are a bit different  If the target service   concrete example com   is a service Istio knows about  then the alias hostname   alias default svc cluster local   will be added as an  additional  match to the  TLS   tls  or  HTTP   http  matching  If not  there will be no changes  so it will be handled as  unmatched traffic   unmatched traffic    An  ExternalName  service can never be a  backend   frontends and backends  on its own  Instead  it is only ever used as additional  frontend   frontends and backends  matches to existing Services  If one is explicitly used as a backend  such as in a  VirtualService  destination  the same aliasing applies  That is  if  alias default svc cluster local  is set as the destination  then requests will go to the  concrete example com   If that hostname is not known to Istio  the requests will fail  in this case  a  ServiceEntry  for  concrete example com  would make this configuration work       ServiceEntry  In addition to Kubernetes Services   Service Entries   docs reference config networking service entry  ServiceEntry  can be created to extend the set of services known to Istio  This can be useful to ensure that traffic to external services  such as  example com   get the functionality of Istio   A ServiceEntry with  addresses  set will perform routing just like a  ClusterIP  Service   However  for Service Entries without any  addresses   all IPs on the port will be matched  This may prevent  unmatched traffic   unmatched traffic  on the same port from being forwarded correctly  As such  it is best to avoid these where possible  or use dedicated ports when needed  HTTP and TLS do not share this constraint  as routing is done based on the hostname SNI    The  addresses  field and  endpoints  field are often confused   addresses  refers to IPs that will be matched against  while endpoints refer to the set of IPs we will send traffic to   For example  the Service entry below would match traffic for  1 1 1 1   and send the request to  2 2 2 2  and  3 3 3 3  following the configured load balancing policy    addresses   1 1 1 1  resolution  STATIC endpoints    address  2 2 2 2   address  3 3 3 3