ingress nginx with specific addons e g for or There are multiple ways to install the Ingress Nginx Controller get started as fast as possible you can check the instructions However in many Installation Guide On most Kubernetes clusters the ingress controller will work without requiring any extra configuration If you want to with using the project repository chart with using YAML manifests

Answer:
# Installation Guide

There are multiple ways to install the Ingress-Nginx Controller:

- with [Helm](https://helm.sh), using the project repository chart;
- with `kubectl apply`, using YAML manifests;
- with specific addons (e.g. for [minikube](#minikube) or [MicroK8s](#microk8s)).

On most Kubernetes clusters, the ingress controller will work without requiring any extra configuration. If you want to
get started as fast as possible, you can check the [quick start](#quick-start) instructions. However, in many
environments, you can improve the performance or get better logs by enabling extra features. We recommend that you
check the [environment-specific instructions](#environment-specific-instructions) for details about optimizing the
ingress controller for your particular environment or cloud provider.

## Contents

- [Quick start](#quick-start)

- [Environment-specific instructions](#environment-specific-instructions)
  - ... [Docker Desktop](#docker-desktop)
  - ... [Rancher Desktop](#rancher-desktop)
  - ... [minikube](#minikube)
  - ... [MicroK8s](#microk8s)
  - ... [AWS](#aws)
  - ... [GCE - GKE](#gce-gke)
  - ... [Azure](#azure)
  - ... [Digital Ocean](#digital-ocean)
  - ... [Scaleway](#scaleway)
  - ... [Exoscale](#exoscale)
  - ... [Oracle Cloud Infrastructure](#oracle-cloud-infrastructure)
  - ... [OVHcloud](#ovhcloud)
  - ... [Bare-metal](#bare-metal-clusters)
- [Miscellaneous](#miscellaneous)

## Quick start

**If you have Helm,** you can deploy the ingress controller with the following command:

```console
helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx --create-namespace
```

It will install the controller in the `ingress-nginx` namespace, creating that namespace if it doesn't already exist.

!!! info
    This command is *idempotent*:

    - if the ingress controller is not installed, it will install it,
    - if the ingress controller is already installed, it will upgrade it.

**If you want a full list of values that you can set, while installing with Helm,** then run:

```console
helm show values ingress-nginx --repo https://kubernetes.github.io/ingress-nginx
```

!!! attention "Helm install on AWS/GCP/Azure/Other providers"
    The *ingress-nginx-controller helm-chart is a generic install out of the box*. The default set of helm values is **not** configured for installation on any infra provider. The annotations that are applicable to the cloud provider must be customized by the users.

    See [AWS LB Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/service/annotations/).

    Examples of some annotations recommended (healthecheck ones are required for target-type IP) for the service resource of `--type LoadBalancer` on AWS are below:
    ```yaml
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: deregistration_delay.timeout_seconds=270
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: /healthz
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "10254"
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: http
        service.beta.kubernetes.io/aws-load-balancer-healthcheck-success-codes: 200-299
        service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
        service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
        service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: "true"
        service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: "true"
        service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-something1 sg-something2"
        service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: "somebucket"
        service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: "ingress-nginx"
        service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval: "5"
    ```

**If you don't have Helm** or if you prefer to use a YAML manifest, you can run the following command instead:

```console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.0-beta.0/deploy/static/provider/cloud/deploy.yaml
```

!!! info
    The YAML manifest in the command above was generated with `helm template`, so you will end up with almost the same
    resources as if you had used Helm to install the controller.

!!! attention
    If you are running an old version of Kubernetes (1.18 or earlier), please read [this paragraph](#running-on-Kubernetes-versions-older-than-1.19) for specific instructions.
    Because of api deprecations, the default manifest may not work on your cluster.
    Specific manifests for supported Kubernetes versions are available within a sub-folder of each provider.

### Firewall configuration

To check which ports are used by your installation of ingress-nginx, look at the output of `kubectl -n ingress-nginx get pod -o yaml`. In general, you need:

- Port 8443 open between all hosts on which the kubernetes nodes are running. This is used for the ingress-nginx [admission controller](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/).
- Port 80 (for HTTP) and/or 443 (for HTTPS) open to the public on the kubernetes nodes to which the DNS of your apps are pointing.

### Pre-flight check

A few pods should start in the `ingress-nginx` namespace:

```console
kubectl get pods --namespace=ingress-nginx
```

After a while, they should all be running. The following command will wait for the ingress controller pod to be up,
running, and ready:

```console
kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=120s
```

### Local testing

Let's create a simple web server and the associated service:

```console
kubectl create deployment demo --image=httpd --port=80
kubectl expose deployment demo
```

Then create an ingress resource. The following example uses a host that maps to `localhost`:

```console
kubectl create ingress demo-localhost --class=nginx \
  --rule="demo.localdev.me/*=demo:80"
```

Now, forward a local port to the ingress controller:

```console
kubectl port-forward --namespace=ingress-nginx service/ingress-nginx-controller 8080:80
```

!!! info
    A note on DNS & network-connection.
    This documentation assumes that a user has awareness of the DNS and the network routing aspects involved in using ingress.
    The port-forwarding mentioned above, is the easiest way to demo the working of ingress. The "kubectl port-forward..." command above has forwarded the port number 8080, on the localhost's tcp/ip stack, where the command was typed, to the port  number 80, of the service created by the installation of ingress-nginx controller. So now, the traffic sent to port number 8080 on localhost will reach the port number 80, of the ingress-controller's service.
    Port-forwarding is not for a production environment use-case. But here we use port-forwarding, to simulate a HTTP request, originating from outside the cluster, to reach the service of the ingress-nginx controller, that is exposed to receive traffic from outside the cluster.
  [This issue](https://github.com/kubernetes/ingress-nginx/issues/10014#issuecomment-1567791549described) shows a typical DNS problem and its solution.

At this point, you can access your deployment using curl ;

```console
curl --resolve demo.localdev.me:8080:127.0.0.1 http://demo.localdev.me:8080
```

You should see a HTML response containing text like **"It works!"**.

### Online testing

If your Kubernetes cluster is a "real" cluster that supports services of type `LoadBalancer`, it will have allocated an
external IP address or FQDN to the ingress controller.

You can see that IP address or FQDN with the following command:

```console
kubectl get service ingress-nginx-controller --namespace=ingress-nginx
```

It will be the `EXTERNAL-IP` field. If that field shows `<pending>`, this means that your Kubernetes cluster wasn't
able to provision the load balancer (generally, this is because it doesn't support services of type `LoadBalancer`).

Once you have the external IP address (or FQDN), set up a DNS record pointing to it. Then you can create an ingress
resource. The following example assumes that you have set up a DNS record for `www.demo.io`:

```console
kubectl create ingress demo --class=nginx \
  --rule="www.demo.io/*=demo:80"
```

Alternatively, the above command can be rewritten as follows for the ```--rule``` command and below.

```console
kubectl create ingress demo --class=nginx \
  --rule www.demo.io/=demo:80
```

You should then be able to see the "It works!" page when you connect to <http://www.demo.io/>. Congratulations,
you are serving a public website hosted on a Kubernetes cluster! 🎉

## Environment-specific instructions

### Local development clusters

#### minikube

The ingress controller can be installed through minikube's addons system:

```console
minikube addons enable ingress
```

#### MicroK8s

The ingress controller can be installed through MicroK8s's addons system:

```console
microk8s enable ingress
```

Please check the MicroK8s [documentation page](https://microk8s.io/docs/addon-ingress) for details.

#### Docker Desktop

Kubernetes is available in Docker Desktop:

- Mac, from [version 18.06.0-ce](https://docs.docker.com/docker-for-mac/release-notes/#stable-releases-of-2018)
- Windows, from [version 18.06.0-ce](https://docs.docker.com/docker-for-windows/release-notes/#docker-community-edition-18060-ce-win70-2018-07-25)

First, make sure that Kubernetes is enabled in the Docker settings. The command `kubectl get nodes` should show a
single node called `docker-desktop`.

The ingress controller can be installed on Docker Desktop using the default [quick start](#quick-start) instructions.

On most systems, if you don't have any other service of type `LoadBalancer` bound to port 80, the ingress controller
will be assigned the `EXTERNAL-IP` of `localhost`, which means that it will be reachable on localhost:80. If that
doesn't work, you might have to fall back to the `kubectl port-forward` method described in the
[local testing section](#local-testing).

#### Rancher Desktop

Rancher Desktop provides Kubernetes and Container Management on the desktop. Kubernetes is enabled by default in Rancher Desktop.

Rancher Desktop uses K3s under the hood, which in turn uses Traefik as the default ingress controller for the Kubernetes cluster. To use Ingress-Nginx Controller in place of the default Traefik, disable Traefik from Preference > Kubernetes menu.

Once traefik is disabled, the Ingress-Nginx Controller can be installed on Rancher Desktop using the default [quick start](#quick-start) instructions. Follow the instructions described in the [local testing section](#local-testing) to try a sample.

### Cloud deployments

If the load balancers of your cloud provider do active healthchecks on their backends (most do), you can change the
`externalTrafficPolicy` of the ingress controller Service to `Local` (instead of the default `Cluster`) to save an
extra hop in some cases. If you're installing with Helm, this can be done by adding
`--set controller.service.externalTrafficPolicy=Local` to the `helm install` or `helm upgrade` command.

Furthermore, if the load balancers of your cloud provider support the PROXY protocol, you can enable it, and it will
let the ingress controller see the real IP address of the clients. Otherwise, it will generally see the IP address of
the upstream load balancer. This must be done both in the ingress controller
(with e.g. `--set controller.config.use-proxy-protocol=true`) and in the cloud provider's load balancer configuration
to function correctly.

In the following sections, we provide YAML manifests that enable these options when possible, using the specific
options of various cloud providers.

#### AWS

In AWS, we use a Network load balancer (NLB) to expose the Ingress-Nginx Controller behind a Service of `Type=LoadBalancer`.

!!! info
    The provided templates illustrate the setup for legacy in-tree service load balancer for AWS NLB.
    AWS provides the documentation on how to use
    [Network load balancing on Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/network-load-balancing.html)
    with [AWS Load Balancer Controller](https://github.com/kubernetes-sigs/aws-load-balancer-controller).

##### Network Load Balancer (NLB)

```console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.0-beta.0/deploy/static/provider/aws/deploy.yaml
```

##### TLS termination in AWS Load Balancer (NLB)

By default, TLS is terminated in the ingress controller. But it is also possible to terminate TLS in the Load Balancer.
This section explains how to do that on AWS using an NLB.

1. Download the [deploy.yaml](https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.0-beta.0/deploy/static/provider/aws/nlb-with-tls-termination/deploy.yaml) template

  ```console
  wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.0-beta.0/deploy/static/provider/aws/nlb-with-tls-termination/deploy.yaml
  ```

2. Edit the file and change the VPC CIDR in use for the Kubernetes cluster:

   ```
   proxy-real-ip-cidr: XXX.XXX.XXX/XX
   ```

3. Change the AWS Certificate Manager (ACM) ID as well:

   ```
   arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX
   ```

4. Deploy the manifest:

   ```console
   kubectl apply -f deploy.yaml
   ```

##### NLB Idle Timeouts

Idle timeout value for TCP flows is 350 seconds and
[cannot be modified](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancers.html#connection-idle-timeout).

For this reason, you need to ensure the
[keepalive_timeout](https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout)
value is configured less than 350 seconds to work as expected.

By default, NGINX `keepalive_timeout` is set to `75s`.

More information with regard to timeouts can be found in the
[official AWS documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancers.html#connection-idle-timeout)

#### GCE-GKE

First, your user needs to have `cluster-admin` permissions on the cluster. This can be done with the following command:

```console
kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole cluster-admin \
  --user $(gcloud config get-value account)
```

Then, the ingress controller can be installed like this:

```console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.0-beta.0/deploy/static/provider/cloud/deploy.yaml
```

!!! warning
    For private clusters, you will need to either add a firewall rule that allows master nodes access to
    port `8443/tcp` on worker nodes, or change the existing rule that allows access to port `80/tcp`, `443/tcp` and
    `10254/tcp` to also allow access to port `8443/tcp`. More information can be found in the
    [Official GCP Documentation](https://cloud.google.com/load-balancing/docs/tcp/setting-up-tcp#config-hc-firewall).

    See the [GKE documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#add_firewall_rules)
    on adding rules and the [Kubernetes issue](https://github.com/kubernetes/kubernetes/issues/79739) for more detail.

Proxy-protocol is supported in GCE check the [Official Documentations on how to enable.](https://cloud.google.com/load-balancing/docs/tcp/setting-up-tcp#proxy-protocol)

#### Azure

```console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.0-beta.0/deploy/static/provider/cloud/deploy.yaml
```

More information with regard to Azure annotations for ingress controller can be found in the [official AKS documentation](https://docs.microsoft.com/en-us/azure/aks/ingress-internal-ip#create-an-ingress-controller).

#### Digital Ocean

```console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.0-beta.0/deploy/static/provider/do/deploy.yaml
```

- By default the service object of the ingress-nginx-controller for Digital-Ocean, only configures one annotation. Its this one `service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"`. While this makes the service functional, it was reported that the Digital-Ocean LoadBalancer graphs shows `no data`, unless a few other annotations are also configured. Some of these other annotations require values that can not be generic and hence not forced in a out-of-the-box installation. These annotations and a discussion on them is well documented in [this issue](https://github.com/kubernetes/ingress-nginx/issues/8965). Please refer to the issue to add annotations, with values specific to user, to get graphs of the DO-LB populated with data.

#### Scaleway

```console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.0-beta.0/deploy/static/provider/scw/deploy.yaml
```

Refer to the [dedicated tutorial](https://www.scaleway.com/en/docs/tutorials/proxy-protocol-v2-load-balancer/#configuring-proxy-protocol-for-ingress-nginx) in the Scaleway documentation for configuring the proxy protocol for ingress-nginx with the Scaleway load balancer.

#### Exoscale

```console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/exoscale/deploy.yaml
```

The full list of annotations supported by Exoscale is available in the Exoscale Cloud Controller Manager
[documentation](https://github.com/exoscale/exoscale-cloud-controller-manager/blob/master/docs/service-loadbalancer.md).

#### Oracle Cloud Infrastructure

```console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.0-beta.0/deploy/static/provider/cloud/deploy.yaml
```

A
[complete list of available annotations for Oracle Cloud Infrastructure](https://github.com/oracle/oci-cloud-controller-manager/blob/master/docs/load-balancer-annotations.md)
can be found in the [OCI Cloud Controller Manager](https://github.com/oracle/oci-cloud-controller-manager) documentation.

#### OVHcloud

```console
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm -n ingress-nginx install ingress-nginx ingress-nginx/ingress-nginx --create-namespace
```

You can find the [complete tutorial](https://docs.ovh.com/gb/en/kubernetes/installing-nginx-ingress/).

### Bare metal clusters

This section is applicable to Kubernetes clusters deployed on bare metal servers, as well as "raw" VMs where Kubernetes
was installed manually, using generic Linux distros (like CentOS, Ubuntu...)

For quick testing, you can use a
[NodePort](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport).
This should work on almost every cluster, but it will typically use a port in the range 30000-32767.

```console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.0-beta.0/deploy/static/provider/baremetal/deploy.yaml
```

For more information about bare metal deployments (and how to use port 80 instead of a random port in the 30000-32767 range),
see [bare-metal considerations](./baremetal.md).

## Miscellaneous

### Checking ingress controller version

Run `/nginx-ingress-controller --version` within the pod, for instance with `kubectl exec`:

```console
POD_NAMESPACE=ingress-nginx
POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/name=ingress-nginx --field-selector=status.phase=Running -o name)
kubectl exec $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version
```

### Scope

By default, the controller watches Ingress objects from all namespaces. If you want to change this behavior,
use the flag `--watch-namespace` or check the Helm chart value `controller.scope` to limit the controller to a single
namespace. Although the use of this flag is not popular, one important fact to note is that the secret containing the default-ssl-certificate needs to also be present in the watched namespace(s).

See also
[“How to easily install multiple instances of the Ingress NGINX controller in the same cluster”](https://kubernetes.github.io/ingress-nginx/#how-to-easily-install-multiple-instances-of-the-ingress-nginx-controller-in-the-same-cluster)
for more details.

### Webhook network access

!!! warning
    The controller uses an [admission webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/)
    to validate Ingress definitions. Make sure that you don't have
    [Network policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/)
    or additional firewalls preventing connections from the API server to the `ingress-nginx-controller-admission` service.

### Certificate generation

!!! attention
    The first time the ingress controller starts, two [Jobs](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/) create the SSL Certificate used by the admission webhook.

This can cause an initial delay of up to two minutes until it is possible to create and validate Ingress definitions.

You can wait until it is ready to run the next command:

```yaml
 kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=120s
```

### Running on Kubernetes versions older than 1.19

Ingress resources evolved over time. They started with `apiVersion: extensions/v1beta1`,
then moved to `apiVersion: networking.k8s.io/v1beta1` and more recently to `apiVersion: networking.k8s.io/v1`.

Here is how these Ingress versions are supported in Kubernetes:

- before Kubernetes 1.19, only `v1beta1` Ingress resources are supported
- from Kubernetes 1.19 to 1.21, both `v1beta1` and `v1` Ingress resources are supported
- in Kubernetes 1.22 and above, only `v1` Ingress resources are supported

And here is how these Ingress versions are supported in Ingress-Nginx Controller:

- before version 1.0, only `v1beta1` Ingress resources are supported
- in version 1.0 and above, only `v1` Ingress resources are

As a result, if you're running Kubernetes 1.19 or later, you should be able to use the latest version of the NGINX
Ingress Controller; but if you're using an old version of Kubernetes (1.18 or earlier) you will have to use version 0.X
of the Ingress-Nginx Controller (e.g. version 0.49).

The Helm chart of the Ingress-Nginx Controller switched to version 1 in version 4 of the chart. In other words, if
you're running Kubernetes 1.19 or earlier, you should use version 3.X of the chart (this can be done by adding
`--version='<4'` to the `helm install` command ).

Installation Guide  There are multiple ways to install the Ingress Nginx Controller     with  Helm  https   helm sh   using the project repository chart    with  kubectl apply   using YAML manifests    with specific addons  e g  for  minikube   minikube  or  MicroK8s   microk8s     On most Kubernetes clusters  the ingress controller will work without requiring any extra configuration  If you want to get started as fast as possible  you can check the  quick start   quick start  instructions  However  in many environments  you can improve the performance or get better logs by enabling extra features  We recommend that you check the  environment specific instructions   environment specific instructions  for details about optimizing the ingress controller for your particular environment or cloud provider      Contents       Quick tip  run  grep       index md  to check that the table of contents is up to date          Quick start   quick start      Environment specific instructions   environment specific instructions           Docker Desktop   docker desktop           Rancher Desktop   rancher desktop           minikube   minikube           MicroK8s   microk8s           AWS   aws           GCE   GKE   gce gke           Azure   azure           Digital Ocean   digital ocean           Scaleway   scaleway           Exoscale   exoscale           Oracle Cloud Infrastructure   oracle cloud infrastructure           OVHcloud   ovhcloud           Bare metal   bare metal clusters     Miscellaneous   miscellaneous        TODO  We have subdirectories for kubernetes versions now because of a PR https   github com kubernetes ingress nginx pull 8162   You can see this here https   github com kubernetes ingress nginx tree main deploy static provider cloud   We need to add documentation here that is clear and unambiguous in guiding users to pick the deployment manifest under a subdirectory  based on the K8S version being used  But until the explicit clear docs land here  users are free to use those subdirectories and get the manifest s  related to their K8S version          Quick start    If you have Helm    you can deploy the ingress controller with the following command      console helm upgrade   install ingress nginx ingress nginx       repo https   kubernetes github io ingress nginx       namespace ingress nginx   create namespace      It will install the controller in the  ingress nginx  namespace  creating that namespace if it doesn t already exist       info     This command is  idempotent          if the ingress controller is not installed  it will install it        if the ingress controller is already installed  it will upgrade it     If you want a full list of values that you can set  while installing with Helm    then run      console helm show values ingress nginx   repo https   kubernetes github io ingress nginx          attention  Helm install on AWS GCP Azure Other providers      The  ingress nginx controller helm chart is a generic install out of the box   The default set of helm values is   not   configured for installation on any infra provider  The annotations that are applicable to the cloud provider must be customized by the users  br       See  AWS LB Controller  https   kubernetes sigs github io aws load balancer controller v2 2 guide service annotations    br       Examples of some annotations recommended  healthecheck ones are required for target type IP  for the service resource of    type LoadBalancer  on AWS are below         yaml       annotations          service beta kubernetes io aws load balancer target group attributes  deregistration delay timeout seconds 270         service beta kubernetes io aws load balancer nlb target type  ip         service beta kubernetes io aws load balancer healthcheck path   healthz         service beta kubernetes io aws load balancer healthcheck port   10254          service beta kubernetes io aws load balancer healthcheck protocol  http         service beta kubernetes io aws load balancer healthcheck success codes  200 299         service beta kubernetes io aws load balancer scheme   internet facing          service beta kubernetes io aws load balancer backend protocol  tcp         service beta kubernetes io aws load balancer cross zone load balancing enabled   true          service beta kubernetes io aws load balancer type  nlb         service beta kubernetes io aws load balancer manage backend security group rules   true          service beta kubernetes io aws load balancer access log enabled   true          service beta kubernetes io aws load balancer security groups   sg something1 sg something2          service beta kubernetes io aws load balancer access log s3 bucket name   somebucket          service beta kubernetes io aws load balancer access log s3 bucket prefix   ingress nginx          service beta kubernetes io aws load balancer access log emit interval   5             If you don t have Helm   or if you prefer to use a YAML manifest  you can run the following command instead      console kubectl apply  f https   raw githubusercontent com kubernetes ingress nginx controller v1 12 0 beta 0 deploy static provider cloud deploy yaml          info     The YAML manifest in the command above was generated with  helm template   so you will end up with almost the same     resources as if you had used Helm to install the controller       attention     If you are running an old version of Kubernetes  1 18 or earlier   please read  this paragraph   running on Kubernetes versions older than 1 19  for specific instructions      Because of api deprecations  the default manifest may not work on your cluster      Specific manifests for supported Kubernetes versions are available within a sub folder of each provider       Firewall configuration  To check which ports are used by your installation of ingress nginx  look at the output of  kubectl  n ingress nginx get pod  o yaml   In general  you need     Port 8443 open between all hosts on which the kubernetes nodes are running  This is used for the ingress nginx  admission controller  https   kubernetes io docs reference access authn authz admission controllers      Port 80  for HTTP  and or 443  for HTTPS  open to the public on the kubernetes nodes to which the DNS of your apps are pointing       Pre flight check  A few pods should start in the  ingress nginx  namespace      console kubectl get pods   namespace ingress nginx      After a while  they should all be running  The following command will wait for the ingress controller pod to be up  running  and ready      console kubectl wait   namespace ingress nginx       for condition ready pod       selector app kubernetes io component controller       timeout 120s          Local testing  Let s create a simple web server and the associated service      console kubectl create deployment demo   image httpd   port 80 kubectl expose deployment demo      Then create an ingress resource  The following example uses a host that maps to  localhost       console kubectl create ingress demo localhost   class nginx       rule  demo localdev me   demo 80       Now  forward a local port to the ingress controller      console kubectl port forward   namespace ingress nginx service ingress nginx controller 8080 80          info     A note on DNS   network connection      This documentation assumes that a user has awareness of the DNS and the network routing aspects involved in using ingress      The port forwarding mentioned above  is the easiest way to demo the working of ingress  The  kubectl port forward     command above has forwarded the port number 8080  on the localhost s tcp ip stack  where the command was typed  to the port  number 80  of the service created by the installation of ingress nginx controller  So now  the traffic sent to port number 8080 on localhost will reach the port number 80  of the ingress controller s service      Port forwarding is not for a production environment use case  But here we use port forwarding  to simulate a HTTP request  originating from outside the cluster  to reach the service of the ingress nginx controller  that is exposed to receive traffic from outside the cluster     This issue  https   github com kubernetes ingress nginx issues 10014 issuecomment 1567791549described  shows a typical DNS problem and its solution   At this point  you can access your deployment using curl       console curl   resolve demo localdev me 8080 127 0 0 1 http   demo localdev me 8080      You should see a HTML response containing text like    It works           Online testing  If your Kubernetes cluster is a  real  cluster that supports services of type  LoadBalancer   it will have allocated an external IP address or FQDN to the ingress controller   You can see that IP address or FQDN with the following command      console kubectl get service ingress nginx controller   namespace ingress nginx      It will be the  EXTERNAL IP  field  If that field shows   pending    this means that your Kubernetes cluster wasn t able to provision the load balancer  generally  this is because it doesn t support services of type  LoadBalancer     Once you have the external IP address  or FQDN   set up a DNS record pointing to it  Then you can create an ingress resource  The following example assumes that you have set up a DNS record for  www demo io       console kubectl create ingress demo   class nginx       rule  www demo io   demo 80       Alternatively  the above command can be rewritten as follows for the      rule    command and below      console kubectl create ingress demo   class nginx       rule www demo io  demo 80      You should then be able to see the  It works   page when you connect to  http   www demo io    Congratulations  you are serving a public website hosted on a Kubernetes cluster        Environment specific instructions      Local development clusters       minikube  The ingress controller can be installed through minikube s addons system      console minikube addons enable ingress           MicroK8s  The ingress controller can be installed through MicroK8s s addons system      console microk8s enable ingress      Please check the MicroK8s  documentation page  https   microk8s io docs addon ingress  for details        Docker Desktop  Kubernetes is available in Docker Desktop     Mac  from  version 18 06 0 ce  https   docs docker com docker for mac release notes  stable releases of 2018    Windows  from  version 18 06 0 ce  https   docs docker com docker for windows release notes  docker community edition 18060 ce win70 2018 07 25   First  make sure that Kubernetes is enabled in the Docker settings  The command  kubectl get nodes  should show a single node called  docker desktop    The ingress controller can be installed on Docker Desktop using the default  quick start   quick start  instructions   On most systems  if you don t have any other service of type  LoadBalancer  bound to port 80  the ingress controller will be assigned the  EXTERNAL IP  of  localhost   which means that it will be reachable on localhost 80  If that doesn t work  you might have to fall back to the  kubectl port forward  method described in the  local testing section   local testing         Rancher Desktop  Rancher Desktop provides Kubernetes and Container Management on the desktop  Kubernetes is enabled by default in Rancher Desktop   Rancher Desktop uses K3s under the hood  which in turn uses Traefik as the default ingress controller for the Kubernetes cluster  To use Ingress Nginx Controller in place of the default Traefik  disable Traefik from Preference   Kubernetes menu   Once traefik is disabled  the Ingress Nginx Controller can be installed on Rancher Desktop using the default  quick start   quick start  instructions  Follow the instructions described in the  local testing section   local testing  to try a sample       Cloud deployments  If the load balancers of your cloud provider do active healthchecks on their backends  most do   you can change the  externalTrafficPolicy  of the ingress controller Service to  Local   instead of the default  Cluster   to save an extra hop in some cases  If you re installing with Helm  this can be done by adding    set controller service externalTrafficPolicy Local  to the  helm install  or  helm upgrade  command   Furthermore  if the load balancers of your cloud provider support the PROXY protocol  you can enable it  and it will let the ingress controller see the real IP address of the clients  Otherwise  it will generally see the IP address of the upstream load balancer  This must be done both in the ingress controller  with e g     set controller config use proxy protocol true   and in the cloud provider s load balancer configuration to function correctly   In the following sections  we provide YAML manifests that enable these options when possible  using the specific options of various cloud providers        AWS  In AWS  we use a Network load balancer  NLB  to expose the Ingress Nginx Controller behind a Service of  Type LoadBalancer        info     The provided templates illustrate the setup for legacy in tree service load balancer for AWS NLB      AWS provides the documentation on how to use      Network load balancing on Amazon EKS  https   docs aws amazon com eks latest userguide network load balancing html      with  AWS Load Balancer Controller  https   github com kubernetes sigs aws load balancer controller          Network Load Balancer  NLB      console kubectl apply  f https   raw githubusercontent com kubernetes ingress nginx controller v1 12 0 beta 0 deploy static provider aws deploy yaml            TLS termination in AWS Load Balancer  NLB   By default  TLS is terminated in the ingress controller  But it is also possible to terminate TLS in the Load Balancer  This section explains how to do that on AWS using an NLB   1  Download the  deploy yaml  https   raw githubusercontent com kubernetes ingress nginx controller v1 12 0 beta 0 deploy static provider aws nlb with tls termination deploy yaml  template       console   wget https   raw githubusercontent com kubernetes ingress nginx controller v1 12 0 beta 0 deploy static provider aws nlb with tls termination deploy yaml        2  Edit the file and change the VPC CIDR in use for the Kubernetes cluster             proxy real ip cidr  XXX XXX XXX XX         3  Change the AWS Certificate Manager  ACM  ID as well             arn aws acm us west 2 XXXXXXXX certificate XXXXXX XXXXXXX XXXXXXX XXXXXXXX         4  Deploy the manifest         console    kubectl apply  f deploy yaml               NLB Idle Timeouts  Idle timeout value for TCP flows is 350 seconds and  cannot be modified  https   docs aws amazon com elasticloadbalancing latest network network load balancers html connection idle timeout    For this reason  you need to ensure the  keepalive timeout  https   nginx org en docs http ngx http core module html keepalive timeout  value is configured less than 350 seconds to work as expected   By default  NGINX  keepalive timeout  is set to  75s    More information with regard to timeouts can be found in the  official AWS documentation  https   docs aws amazon com elasticloadbalancing latest network network load balancers html connection idle timeout        GCE GKE  First  your user needs to have  cluster admin  permissions on the cluster  This can be done with the following command      console kubectl create clusterrolebinding cluster admin binding       clusterrole cluster admin       user   gcloud config get value account       Then  the ingress controller can be installed like this      console kubectl apply  f https   raw githubusercontent com kubernetes ingress nginx controller v1 12 0 beta 0 deploy static provider cloud deploy yaml          warning     For private clusters  you will need to either add a firewall rule that allows master nodes access to     port  8443 tcp  on worker nodes  or change the existing rule that allows access to port  80 tcp    443 tcp  and      10254 tcp  to also allow access to port  8443 tcp   More information can be found in the      Official GCP Documentation  https   cloud google com load balancing docs tcp setting up tcp config hc firewall        See the  GKE documentation  https   cloud google com kubernetes engine docs how to private clusters add firewall rules      on adding rules and the  Kubernetes issue  https   github com kubernetes kubernetes issues 79739  for more detail   Proxy protocol is supported in GCE check the  Official Documentations on how to enable   https   cloud google com load balancing docs tcp setting up tcp proxy protocol        Azure     console kubectl apply  f https   raw githubusercontent com kubernetes ingress nginx controller v1 12 0 beta 0 deploy static provider cloud deploy yaml      More information with regard to Azure annotations for ingress controller can be found in the  official AKS documentation  https   docs microsoft com en us azure aks ingress internal ip create an ingress controller         Digital Ocean     console kubectl apply  f https   raw githubusercontent com kubernetes ingress nginx controller v1 12 0 beta 0 deploy static provider do deploy yaml        By default the service object of the ingress nginx controller for Digital Ocean  only configures one annotation  Its this one  service beta kubernetes io do loadbalancer enable proxy protocol   true    While this makes the service functional  it was reported that the Digital Ocean LoadBalancer graphs shows  no data   unless a few other annotations are also configured  Some of these other annotations require values that can not be generic and hence not forced in a out of the box installation  These annotations and a discussion on them is well documented in  this issue  https   github com kubernetes ingress nginx issues 8965   Please refer to the issue to add annotations  with values specific to user  to get graphs of the DO LB populated with data        Scaleway     console kubectl apply  f https   raw githubusercontent com kubernetes ingress nginx controller v1 12 0 beta 0 deploy static provider scw deploy yaml      Refer to the  dedicated tutorial  https   www scaleway com en docs tutorials proxy protocol v2 load balancer  configuring proxy protocol for ingress nginx  in the Scaleway documentation for configuring the proxy protocol for ingress nginx with the Scaleway load balancer        Exoscale     console kubectl apply  f https   raw githubusercontent com kubernetes ingress nginx main deploy static provider exoscale deploy yaml      The full list of annotations supported by Exoscale is available in the Exoscale Cloud Controller Manager  documentation  https   github com exoscale exoscale cloud controller manager blob master docs service loadbalancer md         Oracle Cloud Infrastructure     console kubectl apply  f https   raw githubusercontent com kubernetes ingress nginx controller v1 12 0 beta 0 deploy static provider cloud deploy yaml      A  complete list of available annotations for Oracle Cloud Infrastructure  https   github com oracle oci cloud controller manager blob master docs load balancer annotations md  can be found in the  OCI Cloud Controller Manager  https   github com oracle oci cloud controller manager  documentation        OVHcloud     console helm repo add ingress nginx https   kubernetes github io ingress nginx helm repo update helm  n ingress nginx install ingress nginx ingress nginx ingress nginx   create namespace      You can find the  complete tutorial  https   docs ovh com gb en kubernetes installing nginx ingress         Bare metal clusters  This section is applicable to Kubernetes clusters deployed on bare metal servers  as well as  raw  VMs where Kubernetes was installed manually  using generic Linux distros  like CentOS  Ubuntu      For quick testing  you can use a  NodePort  https   kubernetes io docs concepts services networking service  type nodeport   This should work on almost every cluster  but it will typically use a port in the range 30000 32767      console kubectl apply  f https   raw githubusercontent com kubernetes ingress nginx controller v1 12 0 beta 0 deploy static provider baremetal deploy yaml      For more information about bare metal deployments  and how to use port 80 instead of a random port in the 30000 32767 range   see  bare metal considerations    baremetal md       Miscellaneous      Checking ingress controller version  Run   nginx ingress controller   version  within the pod  for instance with  kubectl exec       console POD NAMESPACE ingress nginx POD NAME   kubectl get pods  n  POD NAMESPACE  l app kubernetes io name ingress nginx   field selector status phase Running  o name  kubectl exec  POD NAME  n  POD NAMESPACE     nginx ingress controller   version          Scope  By default  the controller watches Ingress objects from all namespaces  If you want to change this behavior  use the flag    watch namespace  or check the Helm chart value  controller scope  to limit the controller to a single namespace  Although the use of this flag is not popular  one important fact to note is that the secret containing the default ssl certificate needs to also be present in the watched namespace s    See also   How to easily install multiple instances of the Ingress NGINX controller in the same cluster   https   kubernetes github io ingress nginx  how to easily install multiple instances of the ingress nginx controller in the same cluster  for more details       Webhook network access      warning     The controller uses an  admission webhook  https   kubernetes io docs reference access authn authz extensible admission controllers       to validate Ingress definitions  Make sure that you don t have      Network policies  https   kubernetes io docs concepts services networking network policies       or additional firewalls preventing connections from the API server to the  ingress nginx controller admission  service       Certificate generation      attention     The first time the ingress controller starts  two  Jobs  https   kubernetes io docs concepts workloads controllers jobs run to completion   create the SSL Certificate used by the admission webhook   This can cause an initial delay of up to two minutes until it is possible to create and validate Ingress definitions   You can wait until it is ready to run the next command      yaml  kubectl wait   namespace ingress nginx       for condition ready pod       selector app kubernetes io component controller       timeout 120s          Running on Kubernetes versions older than 1 19  Ingress resources evolved over time  They started with  apiVersion  extensions v1beta1   then moved to  apiVersion  networking k8s io v1beta1  and more recently to  apiVersion  networking k8s io v1    Here is how these Ingress versions are supported in Kubernetes     before Kubernetes 1 19  only  v1beta1  Ingress resources are supported   from Kubernetes 1 19 to 1 21  both  v1beta1  and  v1  Ingress resources are supported   in Kubernetes 1 22 and above  only  v1  Ingress resources are supported  And here is how these Ingress versions are supported in Ingress Nginx Controller     before version 1 0  only  v1beta1  Ingress resources are supported   in version 1 0 and above  only  v1  Ingress resources are  As a result  if you re running Kubernetes 1 19 or later  you should be able to use the latest version of the NGINX Ingress Controller  but if you re using an old version of Kubernetes  1 18 or earlier  you will have to use version 0 X of the Ingress Nginx Controller  e g  version 0 49    The Helm chart of the Ingress Nginx Controller switched to version 1 in version 4 of the chart  In other words  if you re running Kubernetes 1 19 or earlier  you should use version 3 X of the chart  this can be done by adding    version   4   to the  helm install  command


ingress nginx suffices to provide a single point of contact to the Ingress Nginx Controller to external clients and indirectly to different setup to offer the same kind of access to external consumers Bare metal considerations In traditional cloud environments where network load balancers are available on demand a single Kubernetes manifest any application running inside the cluster Bare metal environments lack this commodity requiring a slightly

Answer:
# Bare-metal considerations

In traditional *cloud* environments, where network load balancers are available on-demand, a single Kubernetes manifest
suffices to provide a single point of contact to the Ingress-Nginx Controller to external clients and, indirectly, to
any application running inside the cluster. *Bare-metal* environments lack this commodity, requiring a slightly
different setup to offer the same kind of access to external consumers.

![Cloud environment](../images/baremetal/cloud_overview.jpg)
![Bare-metal environment](../images/baremetal/baremetal_overview.jpg)

The rest of this document describes a few recommended approaches to deploying the Ingress-Nginx Controller inside a
Kubernetes cluster running on bare-metal.

## A pure software solution: MetalLB

[MetalLB][metallb] provides a network load-balancer implementation for Kubernetes clusters that do not run on a
supported cloud provider, effectively allowing the usage of LoadBalancer Services within any cluster.

This section demonstrates how to use the [Layer 2 configuration mode][metallb-l2] of MetalLB together with the NGINX
Ingress controller in a Kubernetes cluster that has **publicly accessible nodes**. In this mode, one node attracts all
the traffic for the `ingress-nginx` Service IP. See [Traffic policies][metallb-trafficpolicies] for more details.

![MetalLB in L2 mode](../images/baremetal/metallb.jpg)

!!! note
    The description of other supported configuration modes is off-scope for this document.

!!! warning
    MetalLB is currently in *beta*. Read about the [Project maturity][metallb-maturity] and make sure you inform
    yourself by reading the official documentation thoroughly.

MetalLB can be deployed either with a simple Kubernetes manifest or with Helm. The rest of this example assumes MetalLB
was deployed following the [Installation][metallb-install] instructions, and that the Ingress-Nginx Controller was installed
using the steps described in the [quickstart section of the installation guide][install-quickstart].

MetalLB requires a pool of IP addresses in order to be able to take ownership of the `ingress-nginx` Service. This pool
can be defined through `IPAddressPool` objects in the same namespace as the MetalLB controller. This pool of IPs **must** be dedicated to MetalLB's use, you can't reuse the Kubernetes node IPs or IPs handed out by a DHCP server.

!!! example
    Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal
    environments this value is <None\>)

    ```console
    $ kubectl get node
    NAME     STATUS   ROLES    EXTERNAL-IP
    host-1   Ready    master   203.0.113.1
    host-2   Ready    node     203.0.113.2
    host-3   Ready    node     203.0.113.3
    ```

    After creating the following objects, MetalLB takes ownership of one of the IP addresses in the pool and updates
    the *loadBalancer* IP field of the `ingress-nginx` Service accordingly.

    ```yaml
    ---
    apiVersion: metallb.io/v1beta1
    kind: IPAddressPool
    metadata:
      name: default
      namespace: metallb-system
    spec:
      addresses:
      - 203.0.113.10-203.0.113.15
      autoAssign: true
    ---
    apiVersion: metallb.io/v1beta1
    kind: L2Advertisement
    metadata:
      name: default
      namespace: metallb-system
    spec:
      ipAddressPools:
      - default
    ```

    ```console
    $ kubectl -n ingress-nginx get svc
    NAME                   TYPE          CLUSTER-IP     EXTERNAL-IP  PORT(S)
    default-http-backend   ClusterIP     10.0.64.249    <none>       80/TCP
    ingress-nginx          LoadBalancer  10.0.220.217   203.0.113.10  80:30100/TCP,443:30101/TCP
    ```

As soon as MetalLB sets the external IP address of the `ingress-nginx` LoadBalancer Service, the corresponding entries
are created in the iptables NAT table and the node with the selected IP address starts responding to HTTP requests on
the ports configured in the LoadBalancer Service:

```console
$ curl -D- http://203.0.113.10 -H 'Host: myapp.example.com'
HTTP/1.1 200 OK
Server: nginx/1.15.2
```

!!! tip
    In order to preserve the source IP address in HTTP requests sent to NGINX, it is necessary to use the `Local`
    traffic policy. Traffic policies are described in more details in [Traffic policies][metallb-trafficpolicies] as
    well as in the next section.

[metallb]: https://metallb.universe.tf/
[metallb-maturity]: https://metallb.universe.tf/concepts/maturity/
[metallb-l2]: https://metallb.universe.tf/concepts/layer2/
[metallb-install]: https://metallb.universe.tf/installation/
[metallb-trafficpolicies]: https://metallb.universe.tf/usage/#traffic-policies

## Over a NodePort Service

Due to its simplicity, this is the setup a user will deploy by default when following the steps described in the
[installation guide][install-baremetal].

!!! info
    A Service of type `NodePort` exposes, via the `kube-proxy` component, the **same unprivileged** port (default:
    30000-32767) on every Kubernetes node, masters included. For more information, see [Services][nodeport-def].

In this configuration, the NGINX container remains isolated from the host network. As a result, it can safely bind to
any port, including the standard HTTP ports 80 and 443. However, due to the container namespace isolation, a client
located outside the cluster network (e.g. on the public internet) is not able to access Ingress hosts directly on ports
80 and 443. Instead, the external client must append the NodePort allocated to the `ingress-nginx` Service to HTTP
requests.

![NodePort request flow](../images/baremetal/nodeport.jpg)

!!! example
    Given the NodePort `30100` allocated to the `ingress-nginx` Service

    ```console
    $ kubectl -n ingress-nginx get svc
    NAME                   TYPE        CLUSTER-IP     PORT(S)
    default-http-backend   ClusterIP   10.0.64.249    80/TCP
    ingress-nginx          NodePort    10.0.220.217   80:30100/TCP,443:30101/TCP
    ```

    and a Kubernetes node with the public IP address `203.0.113.2` (the external IP is added as an example, in most
    bare-metal environments this value is <None\>)

    ```console
    $ kubectl get node
    NAME     STATUS   ROLES    EXTERNAL-IP
    host-1   Ready    master   203.0.113.1
    host-2   Ready    node     203.0.113.2
    host-3   Ready    node     203.0.113.3
    ```

    a client would reach an Ingress with `host: myapp.example.com` at `http://myapp.example.com:30100`, where the
    myapp.example.com subdomain resolves to the 203.0.113.2 IP address.

!!! danger "Impact on the host system"
    While it may sound tempting to reconfigure the NodePort range using the `--service-node-port-range` API server flag
    to include unprivileged ports and be able to expose ports 80 and 443, doing so may result in unexpected issues
    including (but not limited to) the use of ports otherwise reserved to system daemons and the necessity to grant
    `kube-proxy` privileges it may otherwise not require.

    This practice is therefore **discouraged**. See the other approaches proposed in this page for alternatives.

This approach has a few other limitations one ought to be aware of:

* **Source IP address**

Services of type NodePort perform [source address translation][nodeport-nat] by default. This means the source IP of a
HTTP request is always **the IP address of the Kubernetes node that received the request** from the perspective of
NGINX.

The recommended way to preserve the source IP in a NodePort setup is to set the value of the `externalTrafficPolicy`
field of the `ingress-nginx` Service spec to `Local` ([example][preserve-ip]).

!!! warning
    This setting effectively **drops packets** sent to Kubernetes nodes which are not running any instance of the NGINX
    Ingress controller. Consider [assigning NGINX Pods to specific nodes][pod-assign] in order to control on what nodes
    the Ingress-Nginx Controller should be scheduled or not scheduled.

!!! example
    In a Kubernetes cluster composed of 3 nodes (the external IP is added as an example, in most bare-metal environments
    this value is <None\>)

    ```console
    $ kubectl get node
    NAME     STATUS   ROLES    EXTERNAL-IP
    host-1   Ready    master   203.0.113.1
    host-2   Ready    node     203.0.113.2
    host-3   Ready    node     203.0.113.3
    ```

    with a `ingress-nginx-controller` Deployment composed of 2 replicas

    ```console
    $ kubectl -n ingress-nginx get pod -o wide
    NAME                                       READY   STATUS    IP           NODE
    default-http-backend-7c5bc89cc9-p86md      1/1     Running   172.17.1.1   host-2
    ingress-nginx-controller-cf9ff8c96-8vvf8   1/1     Running   172.17.0.3   host-3
    ingress-nginx-controller-cf9ff8c96-pxsds   1/1     Running   172.17.1.4   host-2
    ```

    Requests sent to `host-2` and `host-3` would be forwarded to NGINX and original client's IP would be preserved,
    while requests to `host-1` would get dropped because there is no NGINX replica running on that node.

* **Ingress status**

Because NodePort Services do not get a LoadBalancerIP assigned by definition, the Ingress-Nginx Controller **does not
update the status of Ingress objects it manages**.

```console
$ kubectl get ingress
NAME           HOSTS               ADDRESS   PORTS
test-ingress   myapp.example.com             80
```

Despite the fact there is no load balancer providing a public IP address to the Ingress-Nginx Controller, it is possible
to force the status update of all managed Ingress objects by setting the `externalIPs` field of the `ingress-nginx`
Service.

!!! warning
    There is more to setting `externalIPs` than just enabling the Ingress-Nginx Controller to update the status of
    Ingress objects. Please read about this option in the [Services][external-ips] page of official Kubernetes
    documentation as well as the section about [External IPs](#external-ips) in this document for more information.

!!! example
    Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal
    environments this value is <None\>)

    ```console
    $ kubectl get node
    NAME     STATUS   ROLES    EXTERNAL-IP
    host-1   Ready    master   203.0.113.1
    host-2   Ready    node     203.0.113.2
    host-3   Ready    node     203.0.113.3
    ```

    one could edit the `ingress-nginx` Service and add the following field to the object spec

    ```yaml
    spec:
      externalIPs:
      - 203.0.113.1
      - 203.0.113.2
      - 203.0.113.3
    ```

    which would in turn be reflected on Ingress objects as follows:

    ```console
    $ kubectl get ingress -o wide
    NAME           HOSTS               ADDRESS                               PORTS
    test-ingress   myapp.example.com   203.0.113.1,203.0.113.2,203.0.113.3   80
    ```

* **Redirects**

As NGINX is **not aware of the port translation operated by the NodePort Service**, backend applications are responsible
for generating redirect URLs that take into account the URL used by external clients, including the NodePort.

!!! example
    Redirects generated by NGINX, for instance HTTP to HTTPS or `domain` to `www.domain`, are generated without
    NodePort:

    ```console
    $ curl -D- http://myapp.example.com:30100`
    HTTP/1.1 308 Permanent Redirect
    Server: nginx/1.15.2
    Location: https://myapp.example.com/  #-> missing NodePort in HTTPS redirect
    ```

[install-baremetal]: ./index.md#bare-metal
[install-quickstart]: ./index.md#quick-start
[nodeport-def]: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport
[nodeport-nat]: https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-nodeport
[pod-assign]: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
[preserve-ip]: https://github.com/kubernetes/ingress-nginx/blob/nginx-0.19.0/deploy/provider/aws/service-nlb.yaml#L12-L14

## Via the host network

In a setup where there is no external load balancer available but using NodePorts is not an option, one can configure
`ingress-nginx` Pods to use the network of the host they run on instead of a dedicated network namespace. The benefit of
this approach is that the Ingress-Nginx Controller can bind ports 80 and 443 directly to Kubernetes nodes' network
interfaces, without the extra network translation imposed by NodePort Services.

!!! note
    This approach does not leverage any Service object to expose the Ingress-Nginx Controller. If the `ingress-nginx`
    Service exists in the target cluster, it is **recommended to delete it**.

This can be achieved by enabling the `hostNetwork` option in the Pods' spec.

```yaml
template:
  spec:
    hostNetwork: true
```

!!! danger "Security considerations"
    Enabling this option **exposes every system daemon to the Ingress-Nginx Controller** on any network interface,
    including the host's loopback. Please evaluate the impact this may have on the security of your system carefully.

!!! example
    Consider this `ingress-nginx-controller` Deployment composed of 2 replicas, NGINX Pods inherit from the IP address
    of their host instead of an internal Pod IP.

    ```console
    $ kubectl -n ingress-nginx get pod -o wide
    NAME                                       READY   STATUS    IP            NODE
    default-http-backend-7c5bc89cc9-p86md      1/1     Running   172.17.1.1    host-2
    ingress-nginx-controller-5b4cf5fc6-7lg6c   1/1     Running   203.0.113.3   host-3
    ingress-nginx-controller-5b4cf5fc6-lzrls   1/1     Running   203.0.113.2   host-2
    ```

One major limitation of this deployment approach is that only **a single Ingress-Nginx Controller Pod** may be scheduled
on each cluster node, because binding the same port multiple times on the same network interface is technically
impossible. Pods that are unschedulable due to such situation fail with the following event:

```console
$ kubectl -n ingress-nginx describe pod <unschedulable-ingress-nginx-controller-pod>
...
Events:
  Type     Reason            From               Message
  ----     ------            ----               -------
  Warning  FailedScheduling  default-scheduler  0/3 nodes are available: 3 node(s) didn't have free ports for the requested pod ports.
```

One way to ensure only schedulable Pods are created is to deploy the Ingress-Nginx Controller as a *DaemonSet* instead
of a traditional Deployment.

!!! info
    A DaemonSet schedules exactly one type of Pod per cluster node, masters included, unless a node is configured to
    [repel those Pods][taints]. For more information, see [DaemonSet][daemonset].

Because most properties of DaemonSet objects are identical to Deployment objects, this documentation page leaves the
configuration of the corresponding manifest at the user's discretion.

![DaemonSet with hostNetwork flow](../images/baremetal/hostnetwork.jpg)

Like with NodePorts, this approach has a few quirks it is important to be aware of.

* **DNS resolution**

Pods configured with `hostNetwork: true` do not use the internal DNS resolver (i.e. *kube-dns* or *CoreDNS*), unless
their `dnsPolicy` spec field is set to [`ClusterFirstWithHostNet`][dnspolicy]. Consider using this setting if NGINX is
expected to resolve internal names for any reason.

* **Ingress status**

Because there is no Service exposing the Ingress-Nginx Controller in a configuration using the host network, the default
`--publish-service` flag used in standard cloud setups **does not apply** and the status of all Ingress objects remains
blank.

```console
$ kubectl get ingress
NAME           HOSTS               ADDRESS   PORTS
test-ingress   myapp.example.com             80
```

Instead, and because bare-metal nodes usually don't have an ExternalIP, one has to enable the
[`--report-node-internal-ip-address`][cli-args] flag, which sets the status of all Ingress objects to the internal IP
address of all nodes running the Ingress-Nginx Controller.

!!! example
    Given a `ingress-nginx-controller` DaemonSet composed of 2 replicas

    ```console
    $ kubectl -n ingress-nginx get pod -o wide
    NAME                                       READY   STATUS    IP            NODE
    default-http-backend-7c5bc89cc9-p86md      1/1     Running   172.17.1.1    host-2
    ingress-nginx-controller-5b4cf5fc6-7lg6c   1/1     Running   203.0.113.3   host-3
    ingress-nginx-controller-5b4cf5fc6-lzrls   1/1     Running   203.0.113.2   host-2
    ```

    the controller sets the status of all Ingress objects it manages to the following value:

    ```console
    $ kubectl get ingress -o wide
    NAME           HOSTS               ADDRESS                   PORTS
    test-ingress   myapp.example.com   203.0.113.2,203.0.113.3   80
    ```

!!! note
    Alternatively, it is possible to override the address written to Ingress objects using the
    `--publish-status-address` flag. See [Command line arguments][cli-args].

[taints]: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
[daemonset]: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
[dnspolicy]: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy
[cli-args]: ../user-guide/cli-arguments.md

## Using a self-provisioned edge

Similarly to cloud environments, this deployment approach requires an edge network component providing a public
entrypoint to the Kubernetes cluster. This edge component can be either hardware (e.g. vendor appliance) or software
(e.g. _HAproxy_) and is usually managed outside of the Kubernetes landscape by operations teams.

Such deployment builds upon the NodePort Service described above in [Over a NodePort Service](#over-a-nodeport-service),
with one significant difference: external clients do not access cluster nodes directly, only the edge component does.
This is particularly suitable for private Kubernetes clusters where none of the nodes has a public IP address.

On the edge side, the only prerequisite is to dedicate a public IP address that forwards all HTTP traffic to Kubernetes
nodes and/or masters. Incoming traffic on TCP ports 80 and 443 is forwarded to the corresponding HTTP and HTTPS NodePort
on the target nodes as shown in the diagram below:

![User edge](../images/baremetal/user_edge.jpg)

## External IPs

!!! danger "Source IP address"
    This method does not allow preserving the source IP of HTTP requests in any manner, it is therefore **not
    recommended** to use it despite its apparent simplicity.

The `externalIPs` Service option was previously mentioned in the [NodePort](#over-a-nodeport-service) section.

As per the [Services][external-ips] page of the official Kubernetes documentation, the `externalIPs` option causes
`kube-proxy` to route traffic sent to arbitrary IP addresses **and on the Service ports** to the endpoints of that
Service. These IP addresses **must belong to the target node**.

!!! example
    Given the following 3-node Kubernetes cluster (the external IP is added as an example, in most bare-metal
    environments this value is <None\>)

    ```console
    $ kubectl get node
    NAME     STATUS   ROLES    EXTERNAL-IP
    host-1   Ready    master   203.0.113.1
    host-2   Ready    node     203.0.113.2
    host-3   Ready    node     203.0.113.3
    ```

    and the following `ingress-nginx` NodePort Service

    ```console
    $ kubectl -n ingress-nginx get svc
    NAME                   TYPE        CLUSTER-IP     PORT(S)
    ingress-nginx          NodePort    10.0.220.217   80:30100/TCP,443:30101/TCP
    ```

    One could set the following external IPs in the Service spec, and NGINX would become available on both the NodePort
    and the Service port:

    ```yaml
    spec:
      externalIPs:
      - 203.0.113.2
      - 203.0.113.3
    ```

    ```console
    $ curl -D- http://myapp.example.com:30100
    HTTP/1.1 200 OK
    Server: nginx/1.15.2

    $ curl -D- http://myapp.example.com
    HTTP/1.1 200 OK
    Server: nginx/1.15.2
    ```

    We assume the myapp.example.com subdomain above resolves to both 203.0.113.2 and 203.0.113.3 IP addresses.

[external-ips]: https://kubernetes.io/docs/concepts/services-networking/service/#external-ips

Bare metal considerations  In traditional  cloud  environments  where network load balancers are available on demand  a single Kubernetes manifest suffices to provide a single point of contact to the Ingress Nginx Controller to external clients and  indirectly  to any application running inside the cluster   Bare metal  environments lack this commodity  requiring a slightly different setup to offer the same kind of access to external consumers     Cloud environment     images baremetal cloud overview jpg    Bare metal environment     images baremetal baremetal overview jpg   The rest of this document describes a few recommended approaches to deploying the Ingress Nginx Controller inside a Kubernetes cluster running on bare metal      A pure software solution  MetalLB   MetalLB  metallb  provides a network load balancer implementation for Kubernetes clusters that do not run on a supported cloud provider  effectively allowing the usage of LoadBalancer Services within any cluster   This section demonstrates how to use the  Layer 2 configuration mode  metallb l2  of MetalLB together with the NGINX Ingress controller in a Kubernetes cluster that has   publicly accessible nodes    In this mode  one node attracts all the traffic for the  ingress nginx  Service IP  See  Traffic policies  metallb trafficpolicies  for more details     MetalLB in L2 mode     images baremetal metallb jpg       note     The description of other supported configuration modes is off scope for this document       warning     MetalLB is currently in  beta   Read about the  Project maturity  metallb maturity  and make sure you inform     yourself by reading the official documentation thoroughly   MetalLB can be deployed either with a simple Kubernetes manifest or with Helm  The rest of this example assumes MetalLB was deployed following the  Installation  metallb install  instructions  and that the Ingress Nginx Controller was installed  using the steps described in the  quickstart section of the installation guide  install quickstart    MetalLB requires a pool of IP addresses in order to be able to take ownership of the  ingress nginx  Service  This pool can be defined through  IPAddressPool  objects in the same namespace as the MetalLB controller  This pool of IPs   must   be dedicated to MetalLB s use  you can t reuse the Kubernetes node IPs or IPs handed out by a DHCP server       example     Given the following 3 node Kubernetes cluster  the external IP is added as an example  in most bare metal     environments this value is  None            console       kubectl get node     NAME     STATUS   ROLES    EXTERNAL IP     host 1   Ready    master   203 0 113 1     host 2   Ready    node     203 0 113 2     host 3   Ready    node     203 0 113 3              After creating the following objects  MetalLB takes ownership of one of the IP addresses in the pool and updates     the  loadBalancer  IP field of the  ingress nginx  Service accordingly          yaml             apiVersion  metallb io v1beta1     kind  IPAddressPool     metadata        name  default       namespace  metallb system     spec        addresses          203 0 113 10 203 0 113 15       autoAssign  true             apiVersion  metallb io v1beta1     kind  L2Advertisement     metadata        name  default       namespace  metallb system     spec        ipAddressPools          default                 console       kubectl  n ingress nginx get svc     NAME                   TYPE          CLUSTER IP     EXTERNAL IP  PORT S      default http backend   ClusterIP     10 0 64 249     none        80 TCP     ingress nginx          LoadBalancer  10 0 220 217   203 0 113 10  80 30100 TCP 443 30101 TCP          As soon as MetalLB sets the external IP address of the  ingress nginx  LoadBalancer Service  the corresponding entries are created in the iptables NAT table and the node with the selected IP address starts responding to HTTP requests on the ports configured in the LoadBalancer Service      console   curl  D  http   203 0 113 10  H  Host  myapp example com  HTTP 1 1 200 OK Server  nginx 1 15 2          tip     In order to preserve the source IP address in HTTP requests sent to NGINX  it is necessary to use the  Local      traffic policy  Traffic policies are described in more details in  Traffic policies  metallb trafficpolicies  as     well as in the next section    metallb   https   metallb universe tf   metallb maturity   https   metallb universe tf concepts maturity   metallb l2   https   metallb universe tf concepts layer2   metallb install   https   metallb universe tf installation   metallb trafficpolicies   https   metallb universe tf usage  traffic policies     Over a NodePort Service  Due to its simplicity  this is the setup a user will deploy by default when following the steps described in the  installation guide  install baremetal        info     A Service of type  NodePort  exposes  via the  kube proxy  component  the   same unprivileged   port  default      30000 32767  on every Kubernetes node  masters included  For more information  see  Services  nodeport def    In this configuration  the NGINX container remains isolated from the host network  As a result  it can safely bind to any port  including the standard HTTP ports 80 and 443  However  due to the container namespace isolation  a client located outside the cluster network  e g  on the public internet  is not able to access Ingress hosts directly on ports 80 and 443  Instead  the external client must append the NodePort allocated to the  ingress nginx  Service to HTTP requests     NodePort request flow     images baremetal nodeport jpg       example     Given the NodePort  30100  allocated to the  ingress nginx  Service         console       kubectl  n ingress nginx get svc     NAME                   TYPE        CLUSTER IP     PORT S      default http backend   ClusterIP   10 0 64 249    80 TCP     ingress nginx          NodePort    10 0 220 217   80 30100 TCP 443 30101 TCP              and a Kubernetes node with the public IP address  203 0 113 2   the external IP is added as an example  in most     bare metal environments this value is  None            console       kubectl get node     NAME     STATUS   ROLES    EXTERNAL IP     host 1   Ready    master   203 0 113 1     host 2   Ready    node     203 0 113 2     host 3   Ready    node     203 0 113 3              a client would reach an Ingress with  host  myapp example com  at  http   myapp example com 30100   where the     myapp example com subdomain resolves to the 203 0 113 2 IP address       danger  Impact on the host system      While it may sound tempting to reconfigure the NodePort range using the    service node port range  API server flag     to include unprivileged ports and be able to expose ports 80 and 443  doing so may result in unexpected issues     including  but not limited to  the use of ports otherwise reserved to system daemons and the necessity to grant      kube proxy  privileges it may otherwise not require       This practice is therefore   discouraged    See the other approaches proposed in this page for alternatives   This approach has a few other limitations one ought to be aware of       Source IP address    Services of type NodePort perform  source address translation  nodeport nat  by default  This means the source IP of a HTTP request is always   the IP address of the Kubernetes node that received the request   from the perspective of NGINX   The recommended way to preserve the source IP in a NodePort setup is to set the value of the  externalTrafficPolicy  field of the  ingress nginx  Service spec to  Local    example  preserve ip         warning     This setting effectively   drops packets   sent to Kubernetes nodes which are not running any instance of the NGINX     Ingress controller  Consider  assigning NGINX Pods to specific nodes  pod assign  in order to control on what nodes     the Ingress Nginx Controller should be scheduled or not scheduled       example     In a Kubernetes cluster composed of 3 nodes  the external IP is added as an example  in most bare metal environments     this value is  None            console       kubectl get node     NAME     STATUS   ROLES    EXTERNAL IP     host 1   Ready    master   203 0 113 1     host 2   Ready    node     203 0 113 2     host 3   Ready    node     203 0 113 3              with a  ingress nginx controller  Deployment composed of 2 replicas         console       kubectl  n ingress nginx get pod  o wide     NAME                                       READY   STATUS    IP           NODE     default http backend 7c5bc89cc9 p86md      1 1     Running   172 17 1 1   host 2     ingress nginx controller cf9ff8c96 8vvf8   1 1     Running   172 17 0 3   host 3     ingress nginx controller cf9ff8c96 pxsds   1 1     Running   172 17 1 4   host 2              Requests sent to  host 2  and  host 3  would be forwarded to NGINX and original client s IP would be preserved      while requests to  host 1  would get dropped because there is no NGINX replica running on that node       Ingress status    Because NodePort Services do not get a LoadBalancerIP assigned by definition  the Ingress Nginx Controller   does not update the status of Ingress objects it manages        console   kubectl get ingress NAME           HOSTS               ADDRESS   PORTS test ingress   myapp example com             80      Despite the fact there is no load balancer providing a public IP address to the Ingress Nginx Controller  it is possible to force the status update of all managed Ingress objects by setting the  externalIPs  field of the  ingress nginx  Service       warning     There is more to setting  externalIPs  than just enabling the Ingress Nginx Controller to update the status of     Ingress objects  Please read about this option in the  Services  external ips  page of official Kubernetes     documentation as well as the section about  External IPs   external ips  in this document for more information       example     Given the following 3 node Kubernetes cluster  the external IP is added as an example  in most bare metal     environments this value is  None            console       kubectl get node     NAME     STATUS   ROLES    EXTERNAL IP     host 1   Ready    master   203 0 113 1     host 2   Ready    node     203 0 113 2     host 3   Ready    node     203 0 113 3              one could edit the  ingress nginx  Service and add the following field to the object spec         yaml     spec        externalIPs          203 0 113 1         203 0 113 2         203 0 113 3              which would in turn be reflected on Ingress objects as follows          console       kubectl get ingress  o wide     NAME           HOSTS               ADDRESS                               PORTS     test ingress   myapp example com   203 0 113 1 203 0 113 2 203 0 113 3   80              Redirects    As NGINX is   not aware of the port translation operated by the NodePort Service    backend applications are responsible for generating redirect URLs that take into account the URL used by external clients  including the NodePort       example     Redirects generated by NGINX  for instance HTTP to HTTPS or  domain  to  www domain   are generated without     NodePort          console       curl  D  http   myapp example com 30100      HTTP 1 1 308 Permanent Redirect     Server  nginx 1 15 2     Location  https   myapp example com       missing NodePort in HTTPS redirect           install baremetal     index md bare metal  install quickstart     index md quick start  nodeport def   https   kubernetes io docs concepts services networking service  type nodeport  nodeport nat   https   kubernetes io docs tutorials services source ip  source ip for services with type nodeport  pod assign   https   kubernetes io docs concepts configuration assign pod node   preserve ip   https   github com kubernetes ingress nginx blob nginx 0 19 0 deploy provider aws service nlb yaml L12 L14     Via the host network  In a setup where there is no external load balancer available but using NodePorts is not an option  one can configure  ingress nginx  Pods to use the network of the host they run on instead of a dedicated network namespace  The benefit of this approach is that the Ingress Nginx Controller can bind ports 80 and 443 directly to Kubernetes nodes  network interfaces  without the extra network translation imposed by NodePort Services       note     This approach does not leverage any Service object to expose the Ingress Nginx Controller  If the  ingress nginx      Service exists in the target cluster  it is   recommended to delete it     This can be achieved by enabling the  hostNetwork  option in the Pods  spec      yaml template    spec      hostNetwork  true          danger  Security considerations      Enabling this option   exposes every system daemon to the Ingress Nginx Controller   on any network interface      including the host s loopback  Please evaluate the impact this may have on the security of your system carefully       example     Consider this  ingress nginx controller  Deployment composed of 2 replicas  NGINX Pods inherit from the IP address     of their host instead of an internal Pod IP          console       kubectl  n ingress nginx get pod  o wide     NAME                                       READY   STATUS    IP            NODE     default http backend 7c5bc89cc9 p86md      1 1     Running   172 17 1 1    host 2     ingress nginx controller 5b4cf5fc6 7lg6c   1 1     Running   203 0 113 3   host 3     ingress nginx controller 5b4cf5fc6 lzrls   1 1     Running   203 0 113 2   host 2          One major limitation of this deployment approach is that only   a single Ingress Nginx Controller Pod   may be scheduled on each cluster node  because binding the same port multiple times on the same network interface is technically impossible  Pods that are unschedulable due to such situation fail with the following event      console   kubectl  n ingress nginx describe pod  unschedulable ingress nginx controller pod      Events    Type     Reason            From               Message                                                           Warning  FailedScheduling  default scheduler  0 3 nodes are available  3 node s  didn t have free ports for the requested pod ports       One way to ensure only schedulable Pods are created is to deploy the Ingress Nginx Controller as a  DaemonSet  instead of a traditional Deployment       info     A DaemonSet schedules exactly one type of Pod per cluster node  masters included  unless a node is configured to      repel those Pods  taints   For more information  see  DaemonSet  daemonset    Because most properties of DaemonSet objects are identical to Deployment objects  this documentation page leaves the configuration of the corresponding manifest at the user s discretion     DaemonSet with hostNetwork flow     images baremetal hostnetwork jpg   Like with NodePorts  this approach has a few quirks it is important to be aware of       DNS resolution    Pods configured with  hostNetwork  true  do not use the internal DNS resolver  i e   kube dns  or  CoreDNS    unless their  dnsPolicy  spec field is set to   ClusterFirstWithHostNet   dnspolicy   Consider using this setting if NGINX is expected to resolve internal names for any reason       Ingress status    Because there is no Service exposing the Ingress Nginx Controller in a configuration using the host network  the default    publish service  flag used in standard cloud setups   does not apply   and the status of all Ingress objects remains blank      console   kubectl get ingress NAME           HOSTS               ADDRESS   PORTS test ingress   myapp example com             80      Instead  and because bare metal nodes usually don t have an ExternalIP  one has to enable the     report node internal ip address   cli args  flag  which sets the status of all Ingress objects to the internal IP address of all nodes running the Ingress Nginx Controller       example     Given a  ingress nginx controller  DaemonSet composed of 2 replicas         console       kubectl  n ingress nginx get pod  o wide     NAME                                       READY   STATUS    IP            NODE     default http backend 7c5bc89cc9 p86md      1 1     Running   172 17 1 1    host 2     ingress nginx controller 5b4cf5fc6 7lg6c   1 1     Running   203 0 113 3   host 3     ingress nginx controller 5b4cf5fc6 lzrls   1 1     Running   203 0 113 2   host 2              the controller sets the status of all Ingress objects it manages to the following value          console       kubectl get ingress  o wide     NAME           HOSTS               ADDRESS                   PORTS     test ingress   myapp example com   203 0 113 2 203 0 113 3   80              note     Alternatively  it is possible to override the address written to Ingress objects using the        publish status address  flag  See  Command line arguments  cli args     taints   https   kubernetes io docs concepts configuration taint and toleration   daemonset   https   kubernetes io docs concepts workloads controllers daemonset   dnspolicy   https   kubernetes io docs concepts services networking dns pod service  pod s dns policy  cli args      user guide cli arguments md     Using a self provisioned edge  Similarly to cloud environments  this deployment approach requires an edge network component providing a public entrypoint to the Kubernetes cluster  This edge component can be either hardware  e g  vendor appliance  or software  e g   HAproxy   and is usually managed outside of the Kubernetes landscape by operations teams   Such deployment builds upon the NodePort Service described above in  Over a NodePort Service   over a nodeport service   with one significant difference  external clients do not access cluster nodes directly  only the edge component does  This is particularly suitable for private Kubernetes clusters where none of the nodes has a public IP address   On the edge side  the only prerequisite is to dedicate a public IP address that forwards all HTTP traffic to Kubernetes nodes and or masters  Incoming traffic on TCP ports 80 and 443 is forwarded to the corresponding HTTP and HTTPS NodePort on the target nodes as shown in the diagram below     User edge     images baremetal user edge jpg      External IPs      danger  Source IP address      This method does not allow preserving the source IP of HTTP requests in any manner  it is therefore   not     recommended   to use it despite its apparent simplicity   The  externalIPs  Service option was previously mentioned in the  NodePort   over a nodeport service  section   As per the  Services  external ips  page of the official Kubernetes documentation  the  externalIPs  option causes  kube proxy  to route traffic sent to arbitrary IP addresses   and on the Service ports   to the endpoints of that Service  These IP addresses   must belong to the target node         example     Given the following 3 node Kubernetes cluster  the external IP is added as an example  in most bare metal     environments this value is  None            console       kubectl get node     NAME     STATUS   ROLES    EXTERNAL IP     host 1   Ready    master   203 0 113 1     host 2   Ready    node     203 0 113 2     host 3   Ready    node     203 0 113 3              and the following  ingress nginx  NodePort Service         console       kubectl  n ingress nginx get svc     NAME                   TYPE        CLUSTER IP     PORT S      ingress nginx          NodePort    10 0 220 217   80 30100 TCP 443 30101 TCP              One could set the following external IPs in the Service spec  and NGINX would become available on both the NodePort     and the Service port          yaml     spec        externalIPs          203 0 113 2         203 0 113 3                 console       curl  D  http   myapp example com 30100     HTTP 1 1 200 OK     Server  nginx 1 15 2        curl  D  http   myapp example com     HTTP 1 1 200 OK     Server  nginx 1 15 2              We assume the myapp example com subdomain above resolves to both 203 0 113 2 and 203 0 113 3 IP addresses    external ips   https   kubernetes io docs concepts services networking service  external ips


ingress nginx key cert pair with an arbitrarily chosen hostname created as follows Unless otherwise mentioned the TLS secret used in examples is a 2048 bit RSA console Many of the examples in this directory have common prerequisites Prerequisites TLS certificates

Answer:
# Prerequisites

Many of the examples in this directory have common prerequisites.

## TLS certificates

Unless otherwise mentioned, the TLS secret used in examples is a 2048 bit RSA
key/cert pair with an arbitrarily chosen hostname, created as follows

```console
$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=nginxsvc/O=nginxsvc"
Generating a 2048 bit RSA private key
................+++
................+++
writing new private key to 'tls.key'
-----

$ kubectl create secret tls tls-secret --key tls.key --cert tls.crt
secret "tls-secret" created
```

Note: If using CA Authentication, described below, you will need to sign the server certificate with the CA.

## Client Certificate Authentication

CA Authentication also known as Mutual Authentication allows both the server and client to verify each others
identity via a common CA.

We have a CA Certificate which we usually obtain from a Certificate Authority and use that to sign
both our server certificate and client certificate. Then every time we want to access our backend, we must
pass the client certificate.

These instructions are based on the following [blog](https://medium.com/@awkwardferny/configuring-certificate-based-mutual-authentication-with-kubernetes-ingress-nginx-20e7e38fdfca)

**Generate the CA Key and Certificate:**

```console
openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=My Cert Authority'
```

**Generate the Server Key, and Certificate and Sign with the CA Certificate:**

```console
openssl req -new -newkey rsa:4096 -keyout server.key -out server.csr -nodes -subj '/CN=mydomain.com'
openssl x509 -req -sha256 -days 365 -in server.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out server.crt
```

**Generate the Client Key, and Certificate and Sign with the CA Certificate:**

```console
openssl req -new -newkey rsa:4096 -keyout client.key -out client.csr -nodes -subj '/CN=My Client'
openssl x509 -req -sha256 -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 02 -out client.crt
```

Once this is complete you can continue to follow the instructions [here](./auth/client-certs/README.md#creating-certificate-secrets)

## Test HTTP Service

All examples that require a test HTTP Service use the standard http-svc pod,
which you can deploy as follows

```console
$ kubectl create -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/http-svc.yaml
service "http-svc" created
replicationcontroller "http-svc" created

$ kubectl get po
NAME             READY     STATUS    RESTARTS   AGE
http-svc-p1t3t   1/1       Running   0          1d

$ kubectl get svc
NAME             CLUSTER-IP     EXTERNAL-IP   PORT(S)            AGE
http-svc         10.0.122.116   <pending>     80:30301/TCP       1d
```

You can test that the HTTP Service works by exposing it temporarily

```console
$ kubectl patch svc http-svc -p '{"spec":{"type": "LoadBalancer"}}'
"http-svc" patched

$ kubectl get svc http-svc
NAME             CLUSTER-IP     EXTERNAL-IP   PORT(S)            AGE
http-svc         10.0.122.116   <pending>     80:30301/TCP       1d

$ kubectl describe svc http-svc
Name:				    http-svc
Namespace:			    default
Labels:			        app=http-svc
Selector:		        app=http-svc
Type:			        LoadBalancer
IP:			            10.0.122.116
LoadBalancer Ingress:	108.59.87.136
Port:			        http	80/TCP
NodePort:		        http	30301/TCP
Endpoints:		        10.180.1.6:8080
Session Affinity:	    None
Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----			-------------	--------	------			-------
  1m		1m		1	{service-controller }			Normal		Type			ClusterIP -> LoadBalancer
  1m		1m		1	{service-controller }			Normal		CreatingLoadBalancer	Creating load balancer
  16s		16s		1	{service-controller }			Normal		CreatedLoadBalancer	Created load balancer

$ curl 108.59.87.136
CLIENT VALUES:
client_address=10.240.0.3
command=GET
real path=/
query=nil
request_version=1.1
request_uri=http://108.59.87.136:8080/

SERVER VALUES:
server_version=nginx: 1.9.11 - lua: 10001

HEADERS RECEIVED:
accept=*/*
host=108.59.87.136
user-agent=curl/7.46.0
BODY:
-no body in request-

$ kubectl patch svc http-svc -p '{"spec":{"type": "NodePort"}}'
"http-svc" patched
```

Prerequisites  Many of the examples in this directory have common prerequisites      TLS certificates  Unless otherwise mentioned  the TLS secret used in examples is a 2048 bit RSA key cert pair with an arbitrarily chosen hostname  created as follows     console   openssl req  x509  sha256  nodes  days 365  newkey rsa 2048  keyout tls key  out tls crt  subj   CN nginxsvc O nginxsvc  Generating a 2048 bit RSA private key                                         writing new private key to  tls key           kubectl create secret tls tls secret   key tls key   cert tls crt secret  tls secret  created      Note  If using CA Authentication  described below  you will need to sign the server certificate with the CA      Client Certificate Authentication  CA Authentication also known as Mutual Authentication allows both the server and client to verify each others identity via a common CA   We have a CA Certificate which we usually obtain from a Certificate Authority and use that to sign both our server certificate and client certificate  Then every time we want to access our backend  we must pass the client certificate   These instructions are based on the following  blog  https   medium com  awkwardferny configuring certificate based mutual authentication with kubernetes ingress nginx 20e7e38fdfca     Generate the CA Key and Certificate        console openssl req  x509  sha256  newkey rsa 4096  keyout ca key  out ca crt  days 356  nodes  subj   CN My Cert Authority         Generate the Server Key  and Certificate and Sign with the CA Certificate        console openssl req  new  newkey rsa 4096  keyout server key  out server csr  nodes  subj   CN mydomain com  openssl x509  req  sha256  days 365  in server csr  CA ca crt  CAkey ca key  set serial 01  out server crt        Generate the Client Key  and Certificate and Sign with the CA Certificate        console openssl req  new  newkey rsa 4096  keyout client key  out client csr  nodes  subj   CN My Client  openssl x509  req  sha256  days 365  in client csr  CA ca crt  CAkey ca key  set serial 02  out client crt      Once this is complete you can continue to follow the instructions  here    auth client certs README md creating certificate secrets        Test HTTP Service  All examples that require a test HTTP Service use the standard http svc pod  which you can deploy as follows     console   kubectl create  f https   raw githubusercontent com kubernetes ingress nginx main docs examples http svc yaml service  http svc  created replicationcontroller  http svc  created    kubectl get po NAME             READY     STATUS    RESTARTS   AGE http svc p1t3t   1 1       Running   0          1d    kubectl get svc NAME             CLUSTER IP     EXTERNAL IP   PORT S             AGE http svc         10 0 122 116    pending      80 30301 TCP       1d      You can test that the HTTP Service works by exposing it temporarily     console   kubectl patch svc http svc  p    spec    type    LoadBalancer      http svc  patched    kubectl get svc http svc NAME             CLUSTER IP     EXTERNAL IP   PORT S             AGE http svc         10 0 122 116    pending      80 30301 TCP       1d    kubectl describe svc http svc Name         http svc Namespace        default Labels            app http svc Selector           app http svc Type            LoadBalancer IP                10 0 122 116 LoadBalancer Ingress  108 59 87 136 Port            http 80 TCP NodePort           http 30301 TCP Endpoints           10 180 1 6 8080 Session Affinity      None Events    FirstSeen LastSeen Count From   SubObjectPath Type  Reason   Message                                                                             1m  1m  1  service controller     Normal  Type   ClusterIP    LoadBalancer   1m  1m  1  service controller     Normal  CreatingLoadBalancer Creating load balancer   16s  16s  1  service controller     Normal  CreatedLoadBalancer Created load balancer    curl 108 59 87 136 CLIENT VALUES  client address 10 240 0 3 command GET real path   query nil request version 1 1 request uri http   108 59 87 136 8080   SERVER VALUES  server version nginx  1 9 11   lua  10001  HEADERS RECEIVED  accept     host 108 59 87 136 user agent curl 7 46 0 BODY   no body in request     kubectl patch svc http svc  p    spec    type    NodePort      http svc  patched