istio help ops traffic management troubleshooting owner istio wg networking maintainers aliases weight 10 docs ops troubleshooting network issues title Traffic Management Problems forceinlinetoc true help ops troubleshooting network issues Techniques to address common Istio traffic management and network problems

Answer:
---
title: Traffic Management Problems
description: Techniques to address common Istio traffic management and network problems.
force_inline_toc: true
weight: 10
aliases:
  - /help/ops/traffic-management/troubleshooting
  - /help/ops/troubleshooting/network-issues
  - /docs/ops/troubleshooting/network-issues
owner: istio/wg-networking-maintainers
test: n/a
---

## Requests are rejected by Envoy

Requests may be rejected for various reasons. The best way to understand why requests are being rejected is
by inspecting Envoy's access logs. By default, access logs are output to the standard output of the container.
Run the following command to see the log:

$ kubectl logs PODNAME -c istio-proxy -n NAMESPACE

In the default access log format, Envoy response flags are located after the response code,
if you are using a custom log format, make sure to include `%RESPONSE_FLAGS%`.

Refer to the [Envoy response flags](https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#config-access-log-format-response-flags)
for details of response flags.

Common response flags are:

- `NR`: No route configured, check your `DestinationRule` or `VirtualService`.
- `UO`: Upstream overflow with circuit breaking, check your circuit breaker configuration in `DestinationRule`.
- `UF`: Failed to connect to upstream, if you're using Istio authentication, check for a
[mutual TLS configuration conflict](#503-errors-after-setting-destination-rule).

## Route rules don't seem to affect traffic flow

With the current Envoy sidecar implementation, up to 100 requests may be required for weighted
version distribution to be observed.

If route rules are working perfectly for the [Bookinfo](/docs/examples/bookinfo/) sample,
but similar version routing rules have no effect on your own application, it may be that
your Kubernetes services need to be changed slightly.
Kubernetes services must adhere to certain restrictions in order to take advantage of
Istio's L7 routing features.
Refer to the [Requirements for Pods and Services](/docs/ops/deployment/application-requirements/)
for details.

Another potential issue is that the route rules may simply be slow to take effect.
The Istio implementation on Kubernetes utilizes an eventually consistent
algorithm to ensure all Envoy sidecars have the correct configuration
including all route rules. A configuration change will take some time
to propagate to all the sidecars.  With large deployments the
propagation will take longer and there may be a lag time on the
order of seconds.

## 503 errors after setting destination rule

You should only see this error if you disabled [automatic mutual TLS](/docs/tasks/security/authentication/authn-policy/#auto-mutual-tls) during install.

If requests to a service immediately start generating HTTP 503 errors after you applied a `DestinationRule`
and the errors continue until you remove or revert the `DestinationRule`, then the `DestinationRule` is probably
causing a TLS conflict for the service.

For example, if you configure mutual TLS in the cluster globally, the `DestinationRule` must include the following `trafficPolicy`:

trafficPolicy:
  tls:
    mode: ISTIO_MUTUAL

Otherwise, the mode defaults to `DISABLE` causing client proxy sidecars to make plain HTTP requests
instead of TLS encrypted requests. Thus, the requests conflict with the server proxy because the server proxy expects
encrypted requests.

Whenever you apply a `DestinationRule`, ensure the `trafficPolicy` TLS mode matches the global server configuration.

## Route rules have no effect on ingress gateway requests

Let's assume you are using an ingress `Gateway` and corresponding `VirtualService` to access an internal service.
For example, your `VirtualService` looks something like this:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: myapp
spec:
  hosts:
  - "myapp.com" # or maybe "*" if you are testing without DNS using the ingress-gateway IP (e.g., http://1.2.3.4/hello)
  gateways:
  - myapp-gateway
  http:
  - match:
    - uri:
        prefix: /hello
    route:
    - destination:
        host: helloworld.default.svc.cluster.local
  - match:
    ...

You also have a `VirtualService` which routes traffic for the helloworld service to a particular subset:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: helloworld
spec:
  hosts:
  - helloworld.default.svc.cluster.local
  http:
  - route:
    - destination:
        host: helloworld.default.svc.cluster.local
        subset: v1

In this situation you will notice that requests to the helloworld service via the ingress gateway will
not be directed to subset v1 but instead will continue to use default round-robin routing.

The ingress requests are using the gateway host (e.g., `myapp.com`)
which will activate the rules in the myapp `VirtualService` that routes to any endpoint of the helloworld service.
Only internal requests with the host `helloworld.default.svc.cluster.local` will use the
helloworld `VirtualService` which directs traffic exclusively to subset v1.

To control the traffic from the gateway, you need to also include the subset rule in the myapp `VirtualService`:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: myapp
spec:
  hosts:
  - "myapp.com" # or maybe "*" if you are testing without DNS using the ingress-gateway IP (e.g., http://1.2.3.4/hello)
  gateways:
  - myapp-gateway
  http:
  - match:
    - uri:
        prefix: /hello
    route:
    - destination:
        host: helloworld.default.svc.cluster.local
        subset: v1
  - match:
    ...

Alternatively, you can combine both `VirtualServices` into one unit if possible:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: myapp
spec:
  hosts:
  - myapp.com # cannot use "*" here since this is being combined with the mesh services
  - helloworld.default.svc.cluster.local
  gateways:
  - mesh # applies internally as well as externally
  - myapp-gateway
  http:
  - match:
    - uri:
        prefix: /hello
      gateways:
      - myapp-gateway #restricts this rule to apply only to ingress gateway
    route:
    - destination:
        host: helloworld.default.svc.cluster.local
        subset: v1
  - match:
    - gateways:
      - mesh # applies to all services inside the mesh
    route:
    - destination:
        host: helloworld.default.svc.cluster.local
        subset: v1

## Envoy is crashing under load

Check your `ulimit -a`. Many systems have a 1024 open file descriptor limit by default which will cause Envoy to assert and crash with:

[2017-05-17 03:00:52.735][14236][critical][assert] assert failure: fd_ != -1: external/envoy/source/common/network/connection_impl.cc:58

Make sure to raise your ulimit. Example: `ulimit -n 16384`

## Envoy won't connect to my HTTP/1.0 service

Envoy requires `HTTP/1.1` or `HTTP/2` traffic for upstream services. For example, when using [NGINX](https://www.nginx.com/) for serving traffic behind Envoy, you
will need to set the [proxy_http_version](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_http_version) directive in your NGINX configuration to be "1.1", since the NGINX default is 1.0.

Example configuration:

upstream http_backend {
    server 127.0.0.1:8080;

    keepalive 16;
}

server {
    ...

    location /http/ {
        proxy_pass http://http_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        ...
    }
}

## 503 error while accessing headless services

Assume Istio is installed with the following configuration:

- `mTLS mode` set to `STRICT` within the mesh
- `meshConfig.outboundTrafficPolicy.mode` set to `ALLOW_ANY`

Consider `nginx` is deployed as a `StatefulSet` in the default namespace and a corresponding `Headless Service` is defined as shown below:

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: http-web  # Explicitly defining an http port
  clusterIP: None   # Creates a Headless Service
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx
  serviceName: "nginx"
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: registry.k8s.io/nginx-slim:0.8
        ports:
        - containerPort: 80
          name: web

The port name `http-web` in the Service definition explicitly specifies the http protocol for that port.

Let us assume we have a [curl](/samples/curl) pod `Deployment` as well in the default namespace.
When `nginx` is accessed from this `curl` pod using its Pod IP (this is one of the common ways to access a headless service), the request goes via the `PassthroughCluster` to the server-side, but the sidecar proxy on the server-side fails to find the route entry to `nginx` and fails with `HTTP 503 UC`.

$ export SOURCE_POD=$(kubectl get pod -l app=curl -o jsonpath='{.items..metadata.name}')
$ kubectl exec -it $SOURCE_POD -c curl -- curl 10.1.1.171 -s -o /dev/null -w "%{http_code}"
  503

`10.1.1.171` is the Pod IP of one of the replicas of `nginx` and the service is accessed on `containerPort` 80.

Here are some of the ways to avoid this 503 error:

1. Specify the correct Host header:

    The Host header in the curl request above will be the Pod IP by default. Specifying the Host header as `nginx.default` in our request to `nginx` successfully returns `HTTP 200 OK`.

    $ export SOURCE_POD=$(kubectl get pod -l app=curl -o jsonpath='{.items..metadata.name}')
    $ kubectl exec -it $SOURCE_POD -c curl -- curl -H "Host: nginx.default" 10.1.1.171 -s -o /dev/null -w "%{http_code}"
      200

1. Set port name to `tcp` or `tcp-web` or `tcp-<custom_name>`:

    Here the protocol is explicitly specified as `tcp`. In this case, only the `TCP Proxy` network filter on the sidecar proxy is used both on the client-side and server-side. HTTP Connection Manager is not used at all and therefore, any kind of header is not expected in the request.

    A request to `nginx` with or without explicitly setting the Host header successfully returns `HTTP 200 OK`.

    This is useful in certain scenarios where a client may not be able to include header information in the request.

    $ export SOURCE_POD=$(kubectl get pod -l app=curl -o jsonpath='{.items..metadata.name}')
    $ kubectl exec -it $SOURCE_POD -c curl -- curl 10.1.1.171 -s -o /dev/null -w "%{http_code}"
      200

    $ kubectl exec -it $SOURCE_POD -c curl -- curl -H "Host: nginx.default" 10.1.1.171 -s -o /dev/null -w "%{http_code}"
      200

1. Use domain name instead of Pod IP:

    A specific instance of a headless service can also be accessed using just the domain name.

    $ export SOURCE_POD=$(kubectl get pod -l app=curl -o jsonpath='{.items..metadata.name}')
    $ kubectl exec -it $SOURCE_POD -c curl -- curl web-0.nginx.default -s -o /dev/null -w "%{http_code}"
      200

    Here `web-0` is the pod name of one of the 3 replicas of `nginx`.

Refer to this [traffic routing](/docs/ops/configuration/traffic-management/traffic-routing/) page for some additional information on headless services and traffic routing behavior for different protocols.

## TLS configuration mistakes

Many traffic management problems
are caused by incorrect [TLS configuration](/docs/ops/configuration/traffic-management/tls-configuration/).
The following sections describe some of the most common misconfigurations.

### Sending HTTPS to an HTTP port

If your application sends an HTTPS request to a service declared to be HTTP,
the Envoy sidecar will attempt to parse the request as HTTP while forwarding the request,
which will fail because the HTTP is unexpectedly encrypted.

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: httpbin
spec:
  hosts:
  - httpbin.org
  ports:
  - number: 443
    name: http
    protocol: HTTP
  resolution: DNS

Although the above configuration may be correct if you are intentionally sending plaintext on port 443 (e.g., `curl http://httpbin.org:443`),
generally port 443 is dedicated for HTTPS traffic.

Sending an HTTPS request like `curl https://httpbin.org`, which defaults to port 443, will result in an error like
`curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number`.
The access logs may also show an error like `400 DPE`.

To fix this, you should change the port protocol to HTTPS:

spec:
  ports:
  - number: 443
    name: https
    protocol: HTTPS

### Gateway to virtual service TLS mismatch {#gateway-mismatch}

There are two common TLS mismatches that can occur when binding a virtual service to a gateway.

1. The gateway terminates TLS while the virtual service configures TLS routing.
1. The gateway does TLS passthrough while the virtual service configures HTTP routing.

#### Gateway with TLS termination

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
      - "*"
    tls:
      mode: SIMPLE
      credentialName: sds-credential
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
  - "*.example.com"
  gateways:
  - istio-system/gateway
  tls:
  - match:
    - sniHosts:
      - "*.example.com"
    route:
    - destination:
        host: httpbin.org

In this example, the gateway is terminating TLS (the `tls.mode` configuration of the gateway is `SIMPLE`,
not `PASSTHROUGH`) while the virtual service is using TLS-based routing. Evaluating routing rules
occurs after the gateway terminates TLS, so the TLS rule will have no effect because the
request is then HTTP rather than HTTPS.

With this misconfiguration, you will end up getting 404 responses because the requests will be
sent to HTTP routing but there are no HTTP routes configured.
You can confirm this using the `istioctl proxy-config routes` command.

To fix this problem, you should switch the virtual service to specify `http` routing, instead of `tls`:

spec:
  ...
  http:
  - match:
    - headers:
        ":authority":
          regex: "*.example.com"

#### Gateway with TLS passthrough

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - "*"
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: PASSTHROUGH
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: virtual-service
spec:
  gateways:
  - gateway
  hosts:
  - httpbin.example.com
  http:
  - route:
    - destination:
        host: httpbin.org

In this configuration, the virtual service is attempting to match HTTP traffic against TLS traffic passed through the gateway.
This will result in the virtual service configuration having no effect. You can observe that the HTTP route is not applied using
the `istioctl proxy-config listener` and `istioctl proxy-config route` commands.

To fix this, you should switch the virtual service to configure `tls` routing:

spec:
  tls:
  - match:
    - sniHosts: ["httpbin.example.com"]
    route:
    - destination:
        host: httpbin.org

Alternatively, you could terminate TLS, rather than passing it through, by switching the `tls` configuration in the gateway:

spec:
  ...
    tls:
      credentialName: sds-credential
      mode: SIMPLE

### Double TLS (TLS origination for a TLS request) {#double-tls}

When configuring Istio to perform TLS origination, you need to make sure
that the application sends plaintext requests to the sidecar, which will then originate the TLS.

The following `DestinationRule` originates TLS for requests to the `httpbin.org` service,
but the corresponding `ServiceEntry` defines the protocol as HTTPS on port 443.

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: httpbin
spec:
  hosts:
  - httpbin.org
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: originate-tls
spec:
  host: httpbin.org
  trafficPolicy:
    tls:
      mode: SIMPLE

With this configuration, the sidecar expects the application to send TLS traffic on port 443
(e.g., `curl https://httpbin.org`), but it will also perform TLS origination before forwarding requests.
This will cause the requests to be double encrypted.

For example, sending a request like `curl https://httpbin.org` will result in an error:
`(35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number`.

You can fix this example by changing the port protocol in the `ServiceEntry` to HTTP:

spec:
  hosts:
  - httpbin.org
  ports:
  - number: 443
    name: http
    protocol: HTTP

Note that with this configuration your application will need to send plaintext requests to port 443,
like `curl http://httpbin.org:443`, because TLS origination does not change the port.
However, starting in Istio 1.8, you can expose HTTP port 80 to the application (e.g., `curl http://httpbin.org`)
and then redirect requests to `targetPort` 443 for the TLS origination:

spec:
  hosts:
  - httpbin.org
  ports:
  - number: 80
    name: http
    protocol: HTTP
    targetPort: 443

### 404 errors occur when multiple gateways configured with same TLS certificate

Configuring more than one gateway using the same TLS certificate will cause browsers
that leverage [HTTP/2 connection reuse](https://httpwg.org/specs/rfc7540.html#reuse)
(i.e., most browsers) to produce 404 errors when accessing a second host after a
connection to another host has already been established.

For example, let's say you have 2 hosts that share the same TLS certificate like this:

- Wildcard certificate `*.test.com` installed in `istio-ingressgateway`
- `Gateway` configuration `gw1` with host `service1.test.com`, selector `istio: ingressgateway`, and TLS using gateway's mounted (wildcard) certificate
- `Gateway` configuration `gw2` with host `service2.test.com`, selector `istio: ingressgateway`, and TLS using gateway's mounted (wildcard) certificate
- `VirtualService` configuration `vs1` with host `service1.test.com` and gateway `gw1`
- `VirtualService` configuration `vs2` with host `service2.test.com` and gateway `gw2`

Since both gateways are served by the same workload (i.e., selector `istio: ingressgateway`) requests to both services
(`service1.test.com` and `service2.test.com`) will resolve to the same IP. If `service1.test.com` is accessed first, it
will return the wildcard certificate (`*.test.com`) indicating that connections to `service2.test.com` can use the same certificate.
Browsers like Chrome and Firefox will consequently reuse the existing connection for requests to `service2.test.com`.
Since the gateway (`gw1`) has no route for `service2.test.com`, it will then return a 404 (Not Found) response.

You can avoid this problem by configuring a single wildcard `Gateway`, instead of two (`gw1` and `gw2`).
Then, simply bind both `VirtualServices` to it like this:

- `Gateway` configuration `gw` with host `*.test.com`, selector `istio: ingressgateway`, and TLS using gateway's mounted (wildcard) certificate
- `VirtualService` configuration `vs1` with host `service1.test.com` and gateway `gw`
- `VirtualService` configuration `vs2` with host `service2.test.com` and gateway `gw`

### Configuring SNI routing when not sending SNI

An HTTPS `Gateway` that specifies the `hosts` field will perform an [SNI](https://en.wikipedia.org/wiki/Server_Name_Indication) match on incoming requests.
For example, the following configuration would only allow requests that match `*.example.com` in the SNI:

servers:
- port:
    number: 443
    name: https
    protocol: HTTPS
  hosts:
  - "*.example.com"

This may cause certain requests to fail.

For example, if you do not have DNS set up and are instead directly setting the host header, such as `curl 1.2.3.4 -H "Host: app.example.com"`, no SNI will be set, causing the request to fail.
Instead, you can set up DNS or use the `--resolve` flag of `curl`. See the [Secure Gateways](/docs/tasks/traffic-management/ingress/secure-ingress/) task for more information.

Another common issue is load balancers in front of Istio.
Most cloud load balancers will not forward the SNI, so if you are terminating TLS in your cloud load balancer you may need to do one of the following:

- Configure the cloud load balancer to instead passthrough the TLS connection
- Disable SNI matching in the `Gateway` by setting the hosts field to `*`

A common symptom of this is for the load balancer health checks to succeed while real traffic fails.

## Unchanged Envoy filter configuration suddenly stops working

An `EnvoyFilter` configuration that specifies an insert position relative to another filter can be very
fragile because, by default, the order of evaluation is based on the creation time of the filters.
Consider a filter with the following specification:

spec:
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: SIDECAR_OUTBOUND
      listener:
        portNumber: 443
        filterChain:
          filter:
            name: istio.stats
    patch:
      operation: INSERT_BEFORE
      value:
        ...

To work properly, this filter configuration depends on the `istio.stats` filter having an older creation time
than it. Otherwise, the `INSERT_BEFORE` operation will be silently ignored. There will be nothing in the
error log to indicate that this filter has not been added to the chain.

This is particularly problematic when matching filters, like `istio.stats`, that are version
specific (i.e., that include the `proxyVersion` field in their match criteria). Such filters may be removed
or replaced by newer ones when upgrading Istio. As a result, an `EnvoyFilter` like the one above may initially
be working perfectly but after upgrading Istio to a newer version it will no longer be included in the network
filter chain of the sidecars.

To avoid this issue, you can either change the operation to one that does not depend on the presence of
another filter (e.g., `INSERT_FIRST`), or set an explicit priority in the `EnvoyFilter` to override the
default creation time-based ordering. For example, adding `priority: 10` to the above filter will ensure
that it is processed after the `istio.stats` filter which has a default priority of 0.

## Virtual service with fault injection and retry/timeout policies not working as expected

Currently, Istio does not support configuring fault injections and retry or timeout policies on the
same `VirtualService`. Consider the following configuration:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: helloworld
spec:
  hosts:
    - "*"
  gateways:
  - helloworld-gateway
  http:
  - match:
    - uri:
        exact: /hello
    fault:
      abort:
        httpStatus: 500
        percentage:
          value: 50
    retries:
      attempts: 5
      retryOn: 5xx
    route:
    - destination:
        host: helloworld
        port:
          number: 5000

You would expect that given the configured five retry attempts, the user would almost never see any
errors when calling the `helloworld` service. However since both fault and retries are configured on
the same `VirtualService`, the retry configuration does not take effect, resulting in a 50% failure
rate. To work around this issue, you may remove the fault config from your `VirtualService` and
inject the fault to the upstream Envoy proxy using `EnvoyFilter` instead:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: hello-world-filter
spec:
  workloadSelector:
    labels:
      app: helloworld
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND # will match outbound listeners in all sidecars
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.fault
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
          abort:
            http_status: 500
            percentage:
              numerator: 50
              denominator: HUNDRED

This works because this way the retry policy is configured for the client proxy while the fault
injection is configured for the upstream proxy.

title  Traffic Management Problems description  Techniques to address common Istio traffic management and network problems  force inline toc  true weight  10 aliases       help ops traffic management troubleshooting      help ops troubleshooting network issues      docs ops troubleshooting network issues owner  istio wg networking maintainers test  n a         Requests are rejected by Envoy  Requests may be rejected for various reasons  The best way to understand why requests are being rejected is by inspecting Envoy s access logs  By default  access logs are output to the standard output of the container  Run the following command to see the log      kubectl logs PODNAME  c istio proxy  n NAMESPACE   In the default access log format  Envoy response flags are located after the response code  if you are using a custom log format  make sure to include   RESPONSE FLAGS     Refer to the  Envoy response flags  https   www envoyproxy io docs envoy latest configuration observability access log usage config access log format response flags  for details of response flags   Common response flags are      NR   No route configured  check your  DestinationRule  or  VirtualService      UO   Upstream overflow with circuit breaking  check your circuit breaker configuration in  DestinationRule      UF   Failed to connect to upstream  if you re using Istio authentication  check for a  mutual TLS configuration conflict   503 errors after setting destination rule       Route rules don t seem to affect traffic flow  With the current Envoy sidecar implementation  up to 100 requests may be required for weighted version distribution to be observed   If route rules are working perfectly for the  Bookinfo   docs examples bookinfo   sample  but similar version routing rules have no effect on your own application  it may be that your Kubernetes services need to be changed slightly  Kubernetes services must adhere to certain restrictions in order to take advantage of Istio s L7 routing features  Refer to the  Requirements for Pods and Services   docs ops deployment application requirements   for details   Another potential issue is that the route rules may simply be slow to take effect  The Istio implementation on Kubernetes utilizes an eventually consistent algorithm to ensure all Envoy sidecars have the correct configuration including all route rules  A configuration change will take some time to propagate to all the sidecars   With large deployments the propagation will take longer and there may be a lag time on the order of seconds      503 errors after setting destination rule   You should only see this error if you disabled  automatic mutual TLS   docs tasks security authentication authn policy  auto mutual tls  during install    If requests to a service immediately start generating HTTP 503 errors after you applied a  DestinationRule  and the errors continue until you remove or revert the  DestinationRule   then the  DestinationRule  is probably causing a TLS conflict for the service   For example  if you configure mutual TLS in the cluster globally  the  DestinationRule  must include the following  trafficPolicy     trafficPolicy    tls      mode  ISTIO MUTUAL   Otherwise  the mode defaults to  DISABLE  causing client proxy sidecars to make plain HTTP requests instead of TLS encrypted requests  Thus  the requests conflict with the server proxy because the server proxy expects encrypted requests   Whenever you apply a  DestinationRule   ensure the  trafficPolicy  TLS mode matches the global server configuration      Route rules have no effect on ingress gateway requests  Let s assume you are using an ingress  Gateway  and corresponding  VirtualService  to access an internal service  For example  your  VirtualService  looks something like this    apiVersion  networking istio io v1 kind  VirtualService metadata    name  myapp spec    hosts       myapp com    or maybe     if you are testing without DNS using the ingress gateway IP  e g   http   1 2 3 4 hello    gateways      myapp gateway   http      match        uri          prefix   hello     route        destination          host  helloworld default svc cluster local     match            You also have a  VirtualService  which routes traffic for the helloworld service to a particular subset    apiVersion  networking istio io v1 kind  VirtualService metadata    name  helloworld spec    hosts      helloworld default svc cluster local   http      route        destination          host  helloworld default svc cluster local         subset  v1   In this situation you will notice that requests to the helloworld service via the ingress gateway will not be directed to subset v1 but instead will continue to use default round robin routing   The ingress requests are using the gateway host  e g    myapp com   which will activate the rules in the myapp  VirtualService  that routes to any endpoint of the helloworld service  Only internal requests with the host  helloworld default svc cluster local  will use the helloworld  VirtualService  which directs traffic exclusively to subset v1   To control the traffic from the gateway  you need to also include the subset rule in the myapp  VirtualService     apiVersion  networking istio io v1 kind  VirtualService metadata    name  myapp spec    hosts       myapp com    or maybe     if you are testing without DNS using the ingress gateway IP  e g   http   1 2 3 4 hello    gateways      myapp gateway   http      match        uri          prefix   hello     route        destination          host  helloworld default svc cluster local         subset  v1     match            Alternatively  you can combine both  VirtualServices  into one unit if possible    apiVersion  networking istio io v1 kind  VirtualService metadata    name  myapp spec    hosts      myapp com   cannot use     here since this is being combined with the mesh services     helloworld default svc cluster local   gateways      mesh   applies internally as well as externally     myapp gateway   http      match        uri          prefix   hello       gateways          myapp gateway  restricts this rule to apply only to ingress gateway     route        destination          host  helloworld default svc cluster local         subset  v1     match        gateways          mesh   applies to all services inside the mesh     route        destination          host  helloworld default svc cluster local         subset  v1      Envoy is crashing under load  Check your  ulimit  a   Many systems have a 1024 open file descriptor limit by default which will cause Envoy to assert and crash with     2017 05 17 03 00 52 735  14236  critical  assert  assert failure  fd      1  external envoy source common network connection impl cc 58   Make sure to raise your ulimit  Example   ulimit  n 16384      Envoy won t connect to my HTTP 1 0 service  Envoy requires  HTTP 1 1  or  HTTP 2  traffic for upstream services  For example  when using  NGINX  https   www nginx com   for serving traffic behind Envoy  you will need to set the  proxy http version  https   nginx org en docs http ngx http proxy module html proxy http version  directive in your NGINX configuration to be  1 1   since the NGINX default is 1 0   Example configuration    upstream http backend       server 127 0 0 1 8080       keepalive 16     server                location  http            proxy pass http   http backend          proxy http version 1 1          proxy set header Connection                              503 error while accessing headless services  Assume Istio is installed with the following configuration      mTLS mode  set to  STRICT  within the mesh    meshConfig outboundTrafficPolicy mode  set to  ALLOW ANY   Consider  nginx  is deployed as a  StatefulSet  in the default namespace and a corresponding  Headless Service  is defined as shown below    apiVersion  v1 kind  Service metadata    name  nginx   labels      app  nginx spec    ports      port  80     name  http web    Explicitly defining an http port   clusterIP  None     Creates a Headless Service   selector      app  nginx     apiVersion  apps v1 kind  StatefulSet metadata    name  web spec    selector      matchLabels        app  nginx   serviceName   nginx    replicas  3   template      metadata        labels          app  nginx     spec        containers          name  nginx         image  registry k8s io nginx slim 0 8         ports            containerPort  80           name  web   The port name  http web  in the Service definition explicitly specifies the http protocol for that port   Let us assume we have a  curl   samples curl  pod  Deployment  as well in the default namespace  When  nginx  is accessed from this  curl  pod using its Pod IP  this is one of the common ways to access a headless service   the request goes via the  PassthroughCluster  to the server side  but the sidecar proxy on the server side fails to find the route entry to  nginx  and fails with  HTTP 503 UC       export SOURCE POD   kubectl get pod  l app curl  o jsonpath    items  metadata name      kubectl exec  it  SOURCE POD  c curl    curl 10 1 1 171  s  o  dev null  w    http code     503    10 1 1 171  is the Pod IP of one of the replicas of  nginx  and the service is accessed on  containerPort  80   Here are some of the ways to avoid this 503 error   1  Specify the correct Host header       The Host header in the curl request above will be the Pod IP by default  Specifying the Host header as  nginx default  in our request to  nginx  successfully returns  HTTP 200 OK               export SOURCE POD   kubectl get pod  l app curl  o jsonpath    items  metadata name          kubectl exec  it  SOURCE POD  c curl    curl  H  Host  nginx default  10 1 1 171  s  o  dev null  w    http code         200       1  Set port name to  tcp  or  tcp web  or  tcp  custom name         Here the protocol is explicitly specified as  tcp   In this case  only the  TCP Proxy  network filter on the sidecar proxy is used both on the client side and server side  HTTP Connection Manager is not used at all and therefore  any kind of header is not expected in the request       A request to  nginx  with or without explicitly setting the Host header successfully returns  HTTP 200 OK        This is useful in certain scenarios where a client may not be able to include header information in the request              export SOURCE POD   kubectl get pod  l app curl  o jsonpath    items  metadata name          kubectl exec  it  SOURCE POD  c curl    curl 10 1 1 171  s  o  dev null  w    http code         200                  kubectl exec  it  SOURCE POD  c curl    curl  H  Host  nginx default  10 1 1 171  s  o  dev null  w    http code         200       1  Use domain name instead of Pod IP       A specific instance of a headless service can also be accessed using just the domain name              export SOURCE POD   kubectl get pod  l app curl  o jsonpath    items  metadata name          kubectl exec  it  SOURCE POD  c curl    curl web 0 nginx default  s  o  dev null  w    http code         200           Here  web 0  is the pod name of one of the 3 replicas of  nginx    Refer to this  traffic routing   docs ops configuration traffic management traffic routing   page for some additional information on headless services and traffic routing behavior for different protocols      TLS configuration mistakes  Many traffic management problems are caused by incorrect  TLS configuration   docs ops configuration traffic management tls configuration    The following sections describe some of the most common misconfigurations       Sending HTTPS to an HTTP port  If your application sends an HTTPS request to a service declared to be HTTP  the Envoy sidecar will attempt to parse the request as HTTP while forwarding the request  which will fail because the HTTP is unexpectedly encrypted    apiVersion  networking istio io v1 kind  ServiceEntry metadata    name  httpbin spec    hosts      httpbin org   ports      number  443     name  http     protocol  HTTP   resolution  DNS   Although the above configuration may be correct if you are intentionally sending plaintext on port 443  e g    curl http   httpbin org 443    generally port 443 is dedicated for HTTPS traffic   Sending an HTTPS request like  curl https   httpbin org   which defaults to port 443  will result in an error like  curl   35  error 1408F10B SSL routines ssl3 get record wrong version number   The access logs may also show an error like  400 DPE    To fix this  you should change the port protocol to HTTPS    spec    ports      number  443     name  https     protocol  HTTPS       Gateway to virtual service TLS mismatch   gateway mismatch   There are two common TLS mismatches that can occur when binding a virtual service to a gateway   1  The gateway terminates TLS while the virtual service configures TLS routing  1  The gateway does TLS passthrough while the virtual service configures HTTP routing        Gateway with TLS termination   apiVersion  networking istio io v1 kind  Gateway metadata    name  gateway   namespace  istio system spec    selector      istio  ingressgateway   servers      port        number  443       name  https       protocol  HTTPS     hosts                  tls        mode  SIMPLE       credentialName  sds credential     apiVersion  networking istio io v1 kind  VirtualService metadata    name  httpbin spec    hosts         example com    gateways      istio system gateway   tls      match        sniHosts             example com      route        destination          host  httpbin org   In this example  the gateway is terminating TLS  the  tls mode  configuration of the gateway is  SIMPLE   not  PASSTHROUGH   while the virtual service is using TLS based routing  Evaluating routing rules occurs after the gateway terminates TLS  so the TLS rule will have no effect because the request is then HTTP rather than HTTPS   With this misconfiguration  you will end up getting 404 responses because the requests will be sent to HTTP routing but there are no HTTP routes configured  You can confirm this using the  istioctl proxy config routes  command   To fix this problem  you should switch the virtual service to specify  http  routing  instead of  tls     spec          http      match        headers            authority             regex     example com         Gateway with TLS passthrough   apiVersion  networking istio io v1 kind  Gateway metadata    name  gateway spec    selector      istio  ingressgateway   servers      hosts                port        name  https       number  443       protocol  HTTPS     tls        mode  PASSTHROUGH     apiVersion  networking istio io v1 kind  VirtualService metadata    name  virtual service spec    gateways      gateway   hosts      httpbin example com   http      route        destination          host  httpbin org   In this configuration  the virtual service is attempting to match HTTP traffic against TLS traffic passed through the gateway  This will result in the virtual service configuration having no effect  You can observe that the HTTP route is not applied using the  istioctl proxy config listener  and  istioctl proxy config route  commands   To fix this  you should switch the virtual service to configure  tls  routing    spec    tls      match        sniHosts    httpbin example com       route        destination          host  httpbin org   Alternatively  you could terminate TLS  rather than passing it through  by switching the  tls  configuration in the gateway    spec            tls        credentialName  sds credential       mode  SIMPLE       Double TLS  TLS origination for a TLS request    double tls   When configuring Istio to perform TLS origination  you need to make sure that the application sends plaintext requests to the sidecar  which will then originate the TLS   The following  DestinationRule  originates TLS for requests to the  httpbin org  service  but the corresponding  ServiceEntry  defines the protocol as HTTPS on port 443    apiVersion  networking istio io v1 kind  ServiceEntry metadata    name  httpbin spec    hosts      httpbin org   ports      number  443     name  https     protocol  HTTPS   resolution  DNS     apiVersion  networking istio io v1 kind  DestinationRule metadata    name  originate tls spec    host  httpbin org   trafficPolicy      tls        mode  SIMPLE   With this configuration  the sidecar expects the application to send TLS traffic on port 443  e g    curl https   httpbin org    but it will also perform TLS origination before forwarding requests  This will cause the requests to be double encrypted   For example  sending a request like  curl https   httpbin org  will result in an error    35  error 1408F10B SSL routines ssl3 get record wrong version number    You can fix this example by changing the port protocol in the  ServiceEntry  to HTTP    spec    hosts      httpbin org   ports      number  443     name  http     protocol  HTTP   Note that with this configuration your application will need to send plaintext requests to port 443  like  curl http   httpbin org 443   because TLS origination does not change the port  However  starting in Istio 1 8  you can expose HTTP port 80 to the application  e g    curl http   httpbin org   and then redirect requests to  targetPort  443 for the TLS origination    spec    hosts      httpbin org   ports      number  80     name  http     protocol  HTTP     targetPort  443       404 errors occur when multiple gateways configured with same TLS certificate  Configuring more than one gateway using the same TLS certificate will cause browsers that leverage  HTTP 2 connection reuse  https   httpwg org specs rfc7540 html reuse   i e   most browsers  to produce 404 errors when accessing a second host after a connection to another host has already been established   For example  let s say you have 2 hosts that share the same TLS certificate like this     Wildcard certificate    test com  installed in  istio ingressgateway     Gateway  configuration  gw1  with host  service1 test com   selector  istio  ingressgateway   and TLS using gateway s mounted  wildcard  certificate    Gateway  configuration  gw2  with host  service2 test com   selector  istio  ingressgateway   and TLS using gateway s mounted  wildcard  certificate    VirtualService  configuration  vs1  with host  service1 test com  and gateway  gw1     VirtualService  configuration  vs2  with host  service2 test com  and gateway  gw2   Since both gateways are served by the same workload  i e   selector  istio  ingressgateway   requests to both services   service1 test com  and  service2 test com   will resolve to the same IP  If  service1 test com  is accessed first  it will return the wildcard certificate     test com   indicating that connections to  service2 test com  can use the same certificate  Browsers like Chrome and Firefox will consequently reuse the existing connection for requests to  service2 test com   Since the gateway   gw1   has no route for  service2 test com   it will then return a 404  Not Found  response   You can avoid this problem by configuring a single wildcard  Gateway   instead of two   gw1  and  gw2    Then  simply bind both  VirtualServices  to it like this      Gateway  configuration  gw  with host    test com   selector  istio  ingressgateway   and TLS using gateway s mounted  wildcard  certificate    VirtualService  configuration  vs1  with host  service1 test com  and gateway  gw     VirtualService  configuration  vs2  with host  service2 test com  and gateway  gw       Configuring SNI routing when not sending SNI  An HTTPS  Gateway  that specifies the  hosts  field will perform an  SNI  https   en wikipedia org wiki Server Name Indication  match on incoming requests  For example  the following configuration would only allow requests that match    example com  in the SNI    servers    port      number  443     name  https     protocol  HTTPS   hosts         example com    This may cause certain requests to fail   For example  if you do not have DNS set up and are instead directly setting the host header  such as  curl 1 2 3 4  H  Host  app example com    no SNI will be set  causing the request to fail  Instead  you can set up DNS or use the    resolve  flag of  curl   See the  Secure Gateways   docs tasks traffic management ingress secure ingress   task for more information   Another common issue is load balancers in front of Istio  Most cloud load balancers will not forward the SNI  so if you are terminating TLS in your cloud load balancer you may need to do one of the following     Configure the cloud load balancer to instead passthrough the TLS connection   Disable SNI matching in the  Gateway  by setting the hosts field to      A common symptom of this is for the load balancer health checks to succeed while real traffic fails      Unchanged Envoy filter configuration suddenly stops working  An  EnvoyFilter  configuration that specifies an insert position relative to another filter can be very fragile because  by default  the order of evaluation is based on the creation time of the filters  Consider a filter with the following specification    spec    configPatches      applyTo  NETWORK FILTER     match        context  SIDECAR OUTBOUND       listener          portNumber  443         filterChain            filter              name  istio stats     patch        operation  INSERT BEFORE       value                To work properly  this filter configuration depends on the  istio stats  filter having an older creation time than it  Otherwise  the  INSERT BEFORE  operation will be silently ignored  There will be nothing in the error log to indicate that this filter has not been added to the chain   This is particularly problematic when matching filters  like  istio stats   that are version specific  i e   that include the  proxyVersion  field in their match criteria   Such filters may be removed or replaced by newer ones when upgrading Istio  As a result  an  EnvoyFilter  like the one above may initially be working perfectly but after upgrading Istio to a newer version it will no longer be included in the network filter chain of the sidecars   To avoid this issue  you can either change the operation to one that does not depend on the presence of another filter  e g    INSERT FIRST    or set an explicit priority in the  EnvoyFilter  to override the default creation time based ordering  For example  adding  priority  10  to the above filter will ensure that it is processed after the  istio stats  filter which has a default priority of 0      Virtual service with fault injection and retry timeout policies not working as expected  Currently  Istio does not support configuring fault injections and retry or timeout policies on the same  VirtualService   Consider the following configuration    apiVersion  networking istio io v1 kind  VirtualService metadata    name  helloworld spec    hosts              gateways      helloworld gateway   http      match        uri          exact   hello     fault        abort          httpStatus  500         percentage            value  50     retries        attempts  5       retryOn  5xx     route        destination          host  helloworld         port            number  5000   You would expect that given the configured five retry attempts  the user would almost never see any errors when calling the  helloworld  service  However since both fault and retries are configured on the same  VirtualService   the retry configuration does not take effect  resulting in a 50  failure rate  To work around this issue  you may remove the fault config from your  VirtualService  and inject the fault to the upstream Envoy proxy using  EnvoyFilter  instead    apiVersion  networking istio io v1alpha3 kind  EnvoyFilter metadata    name  hello world filter spec    workloadSelector      labels        app  helloworld   configPatches      applyTo  HTTP FILTER     match        context  SIDECAR INBOUND   will match outbound listeners in all sidecars       listener          filterChain            filter              name   envoy filters network http connection manager      patch        operation  INSERT BEFORE       value          name  envoy fault         typed config              type    type googleapis com envoy extensions filters http fault v3 HTTPFault            abort              http status  500             percentage                numerator  50               denominator  HUNDRED   This works because this way the retry policy is configured for the client proxy while the fault injection is configured for the upstream proxy


istio aliases owner istio wg user experience maintainers weight 40 test n a forceinlinetoc true docs ops troubleshooting injection title Sidecar Injection Problems Resolve common problems with Istio s use of Kubernetes webhooks for automatic sidecar injection

Answer:
---
title: Sidecar Injection Problems
description: Resolve common problems with Istio's use of Kubernetes webhooks for automatic sidecar injection.
force_inline_toc: true
weight: 40
aliases:
  - /docs/ops/troubleshooting/injection
owner: istio/wg-user-experience-maintainers
test: n/a
---

## The result of sidecar injection was not what I expected

This includes an injected sidecar when it wasn't expected and a lack
of injected sidecar when it was.

1. Ensure your pod is not in the `kube-system` or `kube-public` namespace.
   Automatic sidecar injection will be ignored for pods in these namespaces.

1. Ensure your pod does not have `hostNetwork: true` in its pod spec.
   Automatic sidecar injection will be ignored for pods that are on the host network.

    The sidecar model assumes that the iptables changes required for Envoy to intercept
    traffic are within the pod. For pods on the host network this assumption is violated,
    and this can lead to routing failures at the host level.

1. Check the webhook's `namespaceSelector` to determine whether the
   webhook is scoped to opt-in or opt-out for the target namespace.

    The `namespaceSelector` for opt-in will look like the following:

    $ kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml | grep "namespaceSelector:" -A5
      namespaceSelector:
        matchLabels:
          istio-injection: enabled
      rules:
      - apiGroups:
        - ""

    The injection webhook will be invoked for pods created
    in namespaces with the `istio-injection=enabled` label.

    $ kubectl get namespace -L istio-injection
    NAME           STATUS    AGE       ISTIO-INJECTION
    default        Active    18d       enabled
    istio-system   Active    3d
    kube-public    Active    18d
    kube-system    Active    18d

    The `namespaceSelector` for opt-out will look like the following:

    $ kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml | grep "namespaceSelector:" -A5
      namespaceSelector:
        matchExpressions:
        - key: istio-injection
          operator: NotIn
          values:
          - disabled
      rules:
      - apiGroups:
        - ""

    The injection webhook will be invoked for pods created in namespaces
    without the `istio-injection=disabled` label.

    $ kubectl get namespace -L istio-injection
    NAME           STATUS    AGE       ISTIO-INJECTION
    default        Active    18d
    istio-system   Active    3d        disabled
    kube-public    Active    18d       disabled
    kube-system    Active    18d       disabled

    Verify the application pod's namespace is labeled properly and (re) label accordingly, e.g.

    $ kubectl label namespace istio-system istio-injection=disabled --overwrite

    (repeat for all namespaces in which the injection webhook should be invoked for new pods)

    $ kubectl label namespace default istio-injection=enabled --overwrite

1. Check default policy

    Check the default injection policy in the `istio-sidecar-injector configmap`.

    $ kubectl -n istio-system get configmap istio-sidecar-injector -o jsonpath='{.data.config}' | grep policy:
    policy: enabled

    Allowed policy values are `disabled` and `enabled`. The default policy
    only applies if the webhook’s `namespaceSelector` matches the target
    namespace. Unrecognized policy causes injection to be disabled completely.

1. Check the per-pod override annotation

    The default policy can be overridden with the
    `sidecar.istio.io/inject` label in the _pod template spec’s metadata_.
    The deployment’s metadata is ignored. Label value
    of `true` forces the sidecar to be injected while a value of
    `false` forces the sidecar to _not_ be injected.

    The following label overrides whatever the default `policy` was
    to force the sidecar to be injected:

    $ kubectl get deployment curl -o yaml | grep "sidecar.istio.io/inject:" -B4
    template:
      metadata:
        labels:
          app: curl
          sidecar.istio.io/inject: "true"

## Pods cannot be created at all

Run `kubectl describe -n namespace deployment name` on the failing
pod's deployment. Failure to invoke the injection webhook will
typically be captured in the event log.

### x509 certificate related errors

Warning  FailedCreate  3m (x17 over 8m)  replicaset-controller  Error creating: Internal error occurred: \
    failed calling admission webhook "sidecar-injector.istio.io": Post https://istiod.istio-system.svc:443/inject: \
    x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying \
    to verify candidate authority certificate "Kubernetes.cluster.local")

`x509: certificate signed by unknown authority` errors are typically
caused by an empty `caBundle` in the webhook configuration.

Verify the `caBundle` in the `mutatingwebhookconfiguration` matches the
   root certificate mounted in the `istiod` pod.

$ kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | md5sum
4b95d2ba22ce8971c7c92084da31faf0  -
$ kubectl -n istio-system get configmap istio-ca-root-cert -o jsonpath='{.data.root-cert\.pem}' | base64 -w 0 | md5sum
4b95d2ba22ce8971c7c92084da31faf0  -

The CA certificate should match. If they do not, restart the
istiod pods.

$ kubectl -n istio-system patch deployment istiod \
    -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"date\":\"`date +'%s'`\"}}}}}"
deployment.extensions "istiod" patched

### Errors in deployment status

When automatic sidecar injection is enabled for a pod, and the injection fails for any reason, the pod creation
will also fail. In such cases, you can check the deployment status of the pod to identify the error. The errors
will also appear in the events of the namespace associated with the deployment.

For example, if the `istiod` control plane pod was not running when you tried to deploy your pod, the events would show the following error:

$ kubectl get events -n curl
...
23m Normal   SuccessfulCreate replicaset/curl-9454cc476   Created pod: curl-9454cc476-khp45
22m Warning  FailedCreate     replicaset/curl-9454cc476   Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.96.44.51:443: connect: connection refused

$ kubectl -n istio-system get pod -lapp=istiod
NAME                            READY     STATUS    RESTARTS   AGE
istiod-7d46d8d9db-jz2mh         1/1       Running     0         2d

$ kubectl -n istio-system get endpoints istiod
NAME           ENDPOINTS                                                  AGE
istiod   10.244.2.8:15012,10.244.2.8:15010,10.244.2.8:15017 + 1 more...   3h18m

If the istiod pod or endpoints aren't ready, check the pod logs and status
for any indication about why the webhook pod is failing to start and
serve traffic.

$ for pod in $(kubectl -n istio-system get pod -lapp=istiod -o jsonpath='{.items[*].metadata.name}'); do \
    kubectl -n istio-system logs ${pod} \
done

$ for pod in $(kubectl -n istio-system get pod -l app=istiod -o name); do \
kubectl -n istio-system describe ${pod}; \
done
$

## Automatic sidecar injection fails if the Kubernetes API server has proxy settings

When the Kubernetes API server includes proxy settings such as:

env:
  - name: http_proxy
    value: http://proxy-wsa.esl.foo.com:80
  - name: https_proxy
    value: http://proxy-wsa.esl.foo.com:80
  - name: no_proxy
    value: 127.0.0.1,localhost,dockerhub.foo.com,devhub-docker.foo.com,10.84.100.125,10.84.100.126,10.84.100.127

With these settings, Sidecar injection fails. The only related failure log can be found in `kube-apiserver` log:

W0227 21:51:03.156818       1 admission.go:257] Failed calling webhook, failing open sidecar-injector.istio.io: failed calling admission webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject: Service Unavailable

Make sure both pod and service CIDRs are not proxied according to `*_proxy` variables.  Check the `kube-apiserver` files and logs to verify the configuration and whether any requests are being proxied.

One workaround is to remove the proxy settings from the `kube-apiserver` manifest, another workaround is to include `istio-sidecar-injector.istio-system.svc` or `.svc` in the `no_proxy` value. Make sure that `kube-apiserver` is restarted after each workaround.

An [issue](https://github.com/kubernetes/kubeadm/issues/666) was filed with Kubernetes related to this and has since been closed.
[https://github.com/kubernetes/kubernetes/pull/58698#discussion_r163879443](https://github.com/kubernetes/kubernetes/pull/58698#discussion_r163879443)

## Limitations for using Tcpdump in pods

Tcpdump doesn't work in the sidecar pod - the container doesn't run as root. However any other container in the same pod will see all the packets, since the
network namespace is shared. `iptables` will also see the pod-wide configuration.

Communication between Envoy and the app happens on 127.0.0.1, and is not encrypted.

## Cluster is not scaled down automatically

Due to the fact that the sidecar container mounts a local storage volume, the
node autoscaler is unable to evict nodes with the injected pods. This is
a [known issue](https://github.com/kubernetes/autoscaler/issues/3947). The workaround is
to add a pod annotation `"cluster-autoscaler.kubernetes.io/safe-to-evict":
"true"` to the injected pods.

## Pod or containers start with network issues if istio-proxy is not ready

Many applications execute commands or checks during startup, which require network connectivity. This can cause application containers to hang or restart if the `istio-proxy` sidecar container is not ready.

To avoid this, set `holdApplicationUntilProxyStarts` to `true`. This causes the sidecar injector to inject the sidecar at the start of the pod’s container list, and configures it to block the start of all other containers until the proxy is ready.

This can be added as a global config option:

values.global.proxy.holdApplicationUntilProxyStarts: true

or as a pod annotation:

proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'

title  Sidecar Injection Problems description  Resolve common problems with Istio s use of Kubernetes webhooks for automatic sidecar injection  force inline toc  true weight  40 aliases       docs ops troubleshooting injection owner  istio wg user experience maintainers test  n a         The result of sidecar injection was not what I expected  This includes an injected sidecar when it wasn t expected and a lack of injected sidecar when it was   1  Ensure your pod is not in the  kube system  or  kube public  namespace     Automatic sidecar injection will be ignored for pods in these namespaces   1  Ensure your pod does not have  hostNetwork  true  in its pod spec     Automatic sidecar injection will be ignored for pods that are on the host network       The sidecar model assumes that the iptables changes required for Envoy to intercept     traffic are within the pod  For pods on the host network this assumption is violated      and this can lead to routing failures at the host level   1  Check the webhook s  namespaceSelector  to determine whether the    webhook is scoped to opt in or opt out for the target namespace       The  namespaceSelector  for opt in will look like the following              kubectl get mutatingwebhookconfiguration istio sidecar injector  o yaml   grep  namespaceSelector    A5       namespaceSelector          matchLabels            istio injection  enabled       rules          apiGroups                         The injection webhook will be invoked for pods created     in namespaces with the  istio injection enabled  label              kubectl get namespace  L istio injection     NAME           STATUS    AGE       ISTIO INJECTION     default        Active    18d       enabled     istio system   Active    3d     kube public    Active    18d     kube system    Active    18d           The  namespaceSelector  for opt out will look like the following              kubectl get mutatingwebhookconfiguration istio sidecar injector  o yaml   grep  namespaceSelector    A5       namespaceSelector          matchExpressions            key  istio injection           operator  NotIn           values              disabled       rules          apiGroups                         The injection webhook will be invoked for pods created in namespaces     without the  istio injection disabled  label              kubectl get namespace  L istio injection     NAME           STATUS    AGE       ISTIO INJECTION     default        Active    18d     istio system   Active    3d        disabled     kube public    Active    18d       disabled     kube system    Active    18d       disabled           Verify the application pod s namespace is labeled properly and  re  label accordingly  e g              kubectl label namespace istio system istio injection disabled   overwrite            repeat for all namespaces in which the injection webhook should be invoked for new pods              kubectl label namespace default istio injection enabled   overwrite       1  Check default policy      Check the default injection policy in the  istio sidecar injector configmap               kubectl  n istio system get configmap istio sidecar injector  o jsonpath    data config     grep policy      policy  enabled           Allowed policy values are  disabled  and  enabled   The default policy     only applies if the webhook s  namespaceSelector  matches the target     namespace  Unrecognized policy causes injection to be disabled completely   1  Check the per pod override annotation      The default policy can be overridden with the      sidecar istio io inject  label in the  pod template spec s metadata       The deployment s metadata is ignored  Label value     of  true  forces the sidecar to be injected while a value of      false  forces the sidecar to  not  be injected       The following label overrides whatever the default  policy  was     to force the sidecar to be injected              kubectl get deployment curl  o yaml   grep  sidecar istio io inject    B4     template        metadata          labels            app  curl           sidecar istio io inject   true           Pods cannot be created at all  Run  kubectl describe  n namespace deployment name  on the failing pod s deployment  Failure to invoke the injection webhook will typically be captured in the event log       x509 certificate related errors   Warning  FailedCreate  3m  x17 over 8m   replicaset controller  Error creating  Internal error occurred        failed calling admission webhook  sidecar injector istio io   Post https   istiod istio system svc 443 inject        x509  certificate signed by unknown authority  possibly because of  crypto rsa  verification error  while trying       to verify candidate authority certificate  Kubernetes cluster local      x509  certificate signed by unknown authority  errors are typically caused by an empty  caBundle  in the webhook configuration   Verify the  caBundle  in the  mutatingwebhookconfiguration  matches the    root certificate mounted in the  istiod  pod      kubectl get mutatingwebhookconfiguration istio sidecar injector  o yaml  o jsonpath    webhooks 0  clientConfig caBundle     md5sum 4b95d2ba22ce8971c7c92084da31faf0      kubectl  n istio system get configmap istio ca root cert  o jsonpath    data root cert  pem     base64  w 0   md5sum 4b95d2ba22ce8971c7c92084da31faf0      The CA certificate should match  If they do not  restart the istiod pods      kubectl  n istio system patch deployment istiod        p     spec      template      metadata      labels      date      date    s           deployment extensions  istiod  patched       Errors in deployment status  When automatic sidecar injection is enabled for a pod  and the injection fails for any reason  the pod creation will also fail  In such cases  you can check the deployment status of the pod to identify the error  The errors will also appear in the events of the namespace associated with the deployment   For example  if the  istiod  control plane pod was not running when you tried to deploy your pod  the events would show the following error      kubectl get events  n curl     23m Normal   SuccessfulCreate replicaset curl 9454cc476   Created pod  curl 9454cc476 khp45 22m Warning  FailedCreate     replicaset curl 9454cc476   Error creating  Internal error occurred  failed calling webhook  namespace sidecar injector istio io   failed to call webhook  Post  https   istiod istio system svc 443 inject timeout 10s   dial tcp 10 96 44 51 443  connect  connection refused      kubectl  n istio system get pod  lapp istiod NAME                            READY     STATUS    RESTARTS   AGE istiod 7d46d8d9db jz2mh         1 1       Running     0         2d      kubectl  n istio system get endpoints istiod NAME           ENDPOINTS                                                  AGE istiod   10 244 2 8 15012 10 244 2 8 15010 10 244 2 8 15017   1 more      3h18m   If the istiod pod or endpoints aren t ready  check the pod logs and status for any indication about why the webhook pod is failing to start and serve traffic      for pod in   kubectl  n istio system get pod  lapp istiod  o jsonpath    items    metadata name     do       kubectl  n istio system logs   pod    done     for pod in   kubectl  n istio system get pod  l app istiod  o name   do   kubectl  n istio system describe   pod     done        Automatic sidecar injection fails if the Kubernetes API server has proxy settings  When the Kubernetes API server includes proxy settings such as    env      name  http proxy     value  http   proxy wsa esl foo com 80     name  https proxy     value  http   proxy wsa esl foo com 80     name  no proxy     value  127 0 0 1 localhost dockerhub foo com devhub docker foo com 10 84 100 125 10 84 100 126 10 84 100 127   With these settings  Sidecar injection fails  The only related failure log can be found in  kube apiserver  log    W0227 21 51 03 156818       1 admission go 257  Failed calling webhook  failing open sidecar injector istio io  failed calling admission webhook  sidecar injector istio io   Post https   istio sidecar injector istio system svc 443 inject  Service Unavailable   Make sure both pod and service CIDRs are not proxied according to    proxy  variables   Check the  kube apiserver  files and logs to verify the configuration and whether any requests are being proxied   One workaround is to remove the proxy settings from the  kube apiserver  manifest  another workaround is to include  istio sidecar injector istio system svc  or   svc  in the  no proxy  value  Make sure that  kube apiserver  is restarted after each workaround   An  issue  https   github com kubernetes kubeadm issues 666  was filed with Kubernetes related to this and has since been closed   https   github com kubernetes kubernetes pull 58698 discussion r163879443  https   github com kubernetes kubernetes pull 58698 discussion r163879443      Limitations for using Tcpdump in pods  Tcpdump doesn t work in the sidecar pod   the container doesn t run as root  However any other container in the same pod will see all the packets  since the network namespace is shared   iptables  will also see the pod wide configuration   Communication between Envoy and the app happens on 127 0 0 1  and is not encrypted      Cluster is not scaled down automatically  Due to the fact that the sidecar container mounts a local storage volume  the node autoscaler is unable to evict nodes with the injected pods  This is a  known issue  https   github com kubernetes autoscaler issues 3947   The workaround is to add a pod annotation   cluster autoscaler kubernetes io safe to evict    true   to the injected pods      Pod or containers start with network issues if istio proxy is not ready  Many applications execute commands or checks during startup  which require network connectivity  This can cause application containers to hang or restart if the  istio proxy  sidecar container is not ready   To avoid this  set  holdApplicationUntilProxyStarts  to  true   This causes the sidecar injector to inject the sidecar at the start of the pod s container list  and configures it to block the start of all other containers until the proxy is ready   This can be added as a global config option    values global proxy holdApplicationUntilProxyStarts  true   or as a pod annotation    proxy istio io config      holdApplicationUntilProxyStarts   true