kubernetes setup title Installing kubeadm title Install the kubeadm setup tool name setup contenttype task card weight 10 weight 40

Answer:
---
title: Installing kubeadm
content_type: task
weight: 10
card:
  name: setup
  weight: 40
  title: Install the kubeadm setup tool
---

This page shows how to install the `kubeadm` toolbox.
For information on how to create a cluster with kubeadm once you have performed this installation process,
see the [Creating a cluster with kubeadm](/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/) page.

##

* A compatible Linux host. The Kubernetes project provides generic instructions for Linux distributions
  based on Debian and Red Hat, and those distributions without a package manager.
* 2 GB or more of RAM per machine (any less will leave little room for your apps).
* 2 CPUs or more for control plane machines.
* Full network connectivity between all machines in the cluster (public or private network is fine).
* Unique hostname, MAC address, and product_uuid for every node. See [here](#verify-mac-address) for more details.
* Certain ports are open on your machines. See [here](#check-required-ports) for more details.

The `kubeadm` installation is done via binaries that use dynamic linking and assumes that your target system provides `glibc`.
This is a reasonable assumption on many Linux distributions (including Debian, Ubuntu, Fedora, CentOS, etc.)
but it is not always the case with custom and lightweight distributions which don't include `glibc` by default, such as Alpine Linux.
The expectation is that the distribution either includes `glibc` or a
[compatibility layer](https://wiki.alpinelinux.org/wiki/Running_glibc_programs)
that provides the expected symbols.

## Verify the MAC address and product_uuid are unique for every node {#verify-mac-address}

* You can get the MAC address of the network interfaces using the command `ip link` or `ifconfig -a`
* The product_uuid can be checked by using the command `sudo cat /sys/class/dmi/id/product_uuid`

It is very likely that hardware devices will have unique addresses, although some virtual machines may have
identical values. Kubernetes uses these values to uniquely identify the nodes in the cluster.
If these values are not unique to each node, the installation process
may [fail](https://github.com/kubernetes/kubeadm/issues/31).

## Check network adapters

If you have more than one network adapter, and your Kubernetes components are not reachable on the default
route, we recommend you add IP route(s) so Kubernetes cluster addresses go via the appropriate adapter.

## Check required ports {#check-required-ports}

These [required ports](/docs/reference/networking/ports-and-protocols/)
need to be open in order for Kubernetes components to communicate with each other.
You can use tools like [netcat](https://netcat.sourceforge.net) to check if a port is open. For example:

```shell
nc 127.0.0.1 6443 -v
```

The pod network plugin you use may also require certain ports to be
open. Since this differs with each pod network plugin, please see the
documentation for the plugins about what port(s) those need.

## Swap configuration {#swap-configuration}

The default behavior of a kubelet is to fail to start if swap memory is detected on a node.
This means that swap should either be disabled or tolerated by kubelet.

* To tolerate swap, add `failSwapOn: false` to kubelet configuration or as a command line argument.
  Note: even if `failSwapOn: false` is provided, workloads wouldn't have swap access by default.
  This can be changed by setting a `swapBehavior`, again in the kubelet configuration file. To use swap,
  set a `swapBehavior` other than the default `NoSwap` setting.
  See [Swap memory management](/docs/concepts/architecture/nodes/#swap-memory) for more details.
* To disable swap, `sudo swapoff -a` can be used to disable swapping temporarily.
  To make this change persistent across reboots, make sure swap is disabled in
  config files like `/etc/fstab`, `systemd.swap`, depending how it was configured on your system.

## Installing a container runtime {#installing-runtime}

To run containers in Pods, Kubernetes uses a
.

By default, Kubernetes uses the
 (CRI)
to interface with your chosen container runtime.

If you don't specify a runtime, kubeadm automatically tries to detect an installed
container runtime by scanning through a list of known endpoints.

If multiple or no container runtimes are detected kubeadm will throw an error
and will request that you specify which one you want to use.

See [container runtimes](/docs/setup/production-environment/container-runtimes/)
for more information.

Docker Engine does not implement the [CRI](/docs/concepts/architecture/cri/)
which is a requirement for a container runtime to work with Kubernetes.
For that reason, an additional service [cri-dockerd](https://mirantis.github.io/cri-dockerd/)
has to be installed. cri-dockerd is a project based on the legacy built-in
Docker Engine support that was [removed](/dockershim) from the kubelet in version 1.24.

The tables below include the known endpoints for supported operating systems:

| Runtime                            | Path to Unix domain socket                   |
|------------------------------------|----------------------------------------------|
| containerd                         | `unix:///var/run/containerd/containerd.sock` |
| CRI-O                              | `unix:///var/run/crio/crio.sock`             |
| Docker Engine (using cri-dockerd)  | `unix:///var/run/cri-dockerd.sock`           |

| Runtime                            | Path to Windows named pipe                   |
|------------------------------------|----------------------------------------------|
| containerd                         | `npipe:////./pipe/containerd-containerd`     |
| Docker Engine (using cri-dockerd)  | `npipe:////./pipe/cri-dockerd`               |

## Installing kubeadm, kubelet and kubectl

You will install these packages on all of your machines:

* `kubeadm`: the command to bootstrap the cluster.

* `kubelet`: the component that runs on all of the machines in your cluster
  and does things like starting pods and containers.

* `kubectl`: the command line util to talk to your cluster.

kubeadm **will not** install or manage `kubelet` or `kubectl` for you, so you will
need to ensure they match the version of the Kubernetes control plane you want
kubeadm to install for you. If you do not, there is a risk of a version skew occurring that
can lead to unexpected, buggy behaviour. However, _one_ minor version skew between the
kubelet and the control plane is supported, but the kubelet version may never exceed the API
server version. For example, the kubelet running 1.7.0 should be fully compatible with a 1.8.0 API server,
but not vice versa.

For information about installing `kubectl`, see [Install and set up kubectl](/docs/tasks/tools/).

These instructions exclude all Kubernetes packages from any system upgrades.
This is because kubeadm and Kubernetes require
[special attention to upgrade](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/).

For more information on version skews, see:

* Kubernetes [version and version-skew policy](/docs/setup/release/version-skew-policy/)
* Kubeadm-specific [version skew policy](/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#version-skew-policy)

There's a dedicated package repository for each Kubernetes minor version. If you want to install
a minor version other than v, please see the installation guide for
your desired minor version.

These instructions are for Kubernetes v.

1. Update the `apt` package index and install packages needed to use the Kubernetes `apt` repository:

   ```shell
   sudo apt-get update
   # apt-transport-https may be a dummy package; if so, you can skip that package
   sudo apt-get install -y apt-transport-https ca-certificates curl gpg
   ```

2. Download the public signing key for the Kubernetes package repositories.
   The same signing key is used for all repositories so you can disregard the version in the URL:

   ```shell
   # If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
   # sudo mkdir -p -m 755 /etc/apt/keyrings
   curl -fsSL https://pkgs.k8s.io/core:/stable://deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
   ```

In releases older than Debian 12 and Ubuntu 22.04, directory `/etc/apt/keyrings` does not
exist by default, and it should be created before the curl command.

3. Add the appropriate Kubernetes `apt` repository. Please note that this repository have packages
   only for Kubernetes ; for other Kubernetes minor versions, you need to
   change the Kubernetes minor version in the URL to match your desired minor version
   (you should also check that you are reading the documentation for the version of Kubernetes
   that you plan to install).

   ```shell
   # This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
   echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable://deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
   ```

4. Update the `apt` package index, install kubelet, kubeadm and kubectl, and pin their version:

   ```shell
   sudo apt-get update
   sudo apt-get install -y kubelet kubeadm kubectl
   sudo apt-mark hold kubelet kubeadm kubectl
   ```

5. (Optional) Enable the kubelet service before running kubeadm:

   ```shell
   sudo systemctl enable --now kubelet
   ```

1. Set SELinux to `permissive` mode:

   These instructions are for Kubernetes .

   ```shell
   # Set SELinux in permissive mode (effectively disabling it)
   sudo setenforce 0
   sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
   ```

- Setting SELinux in permissive mode by running `setenforce 0` and `sed ...`
  effectively disables it. This is required to allow containers to access the host
  filesystem; for example, some cluster network plugins require that. You have to
  do this until SELinux support is improved in the kubelet.
- You can leave SELinux enabled if you know how to configure it but it may require
  settings that are not supported by kubeadm.

2. Add the Kubernetes `yum` repository. The `exclude` parameter in the
   repository definition ensures that the packages related to Kubernetes are
   not upgraded upon running `yum update` as there's a special procedure that
   must be followed for upgrading Kubernetes. Please note that this repository
   have packages only for Kubernetes ; for other
   Kubernetes minor versions, you need to change the Kubernetes minor version
   in the URL to match your desired minor version (you should also check that
   you are reading the documentation for the version of Kubernetes that you
   plan to install).

   ```shell
   # This overwrites any existing configuration in /etc/yum.repos.d/kubernetes.repo
   cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
   [kubernetes]
   name=Kubernetes
   baseurl=https://pkgs.k8s.io/core:/stable://rpm/
   enabled=1
   gpgcheck=1
   gpgkey=https://pkgs.k8s.io/core:/stable://rpm/repodata/repomd.xml.key
   exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
   EOF
   ```

3. Install kubelet, kubeadm and kubectl:

   ```shell
   sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
   ```

4. (Optional) Enable the kubelet service before running kubeadm:

   ```shell
   sudo systemctl enable --now kubelet
   ```

Install CNI plugins (required for most pod network):

```bash
CNI_PLUGINS_VERSION="v1.3.0"
ARCH="amd64"
DEST="/opt/cni/bin"
sudo mkdir -p "$DEST"
curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_PLUGINS_VERSION}/cni-plugins-linux-${ARCH}-${CNI_PLUGINS_VERSION}.tgz" | sudo tar -C "$DEST" -xz
```

Define the directory to download command files:

The `DOWNLOAD_DIR` variable must be set to a writable directory.
If you are running Flatcar Container Linux, set `DOWNLOAD_DIR="/opt/bin"`.

```bash
DOWNLOAD_DIR="/usr/local/bin"
sudo mkdir -p "$DOWNLOAD_DIR"
```

Optionally install crictl (required for interaction with the Container Runtime Interface (CRI), optional for kubeadm):

```bash
CRICTL_VERSION="v1.31.0"
ARCH="amd64"
curl -L "https://github.com/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-${ARCH}.tar.gz" | sudo tar -C $DOWNLOAD_DIR -xz
```

Install `kubeadm`, `kubelet` and add a `kubelet` systemd service:

```bash
RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)"
ARCH="amd64"
cd $DOWNLOAD_DIR
sudo curl -L --remote-name-all https://dl.k8s.io/release/${RELEASE}/bin/linux/${ARCH}/{kubeadm,kubelet}
sudo chmod +x {kubeadm,kubelet}

RELEASE_VERSION="v0.16.2"
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/krel/templates/latest/kubelet/kubelet.service" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /usr/lib/systemd/system/kubelet.service
sudo mkdir -p /usr/lib/systemd/system/kubelet.service.d
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/krel/templates/latest/kubeadm/10-kubeadm.conf" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
```

Please refer to the note in the [Before you begin](#before-you-begin) section for Linux distributions
that do not include `glibc` by default.

Install `kubectl` by following the instructions on [Install Tools page](/docs/tasks/tools/#kubectl).

Optionally, enable the kubelet service before running kubeadm:

```bash
sudo systemctl enable --now kubelet
```

The Flatcar Container Linux distribution mounts the `/usr` directory as a read-only filesystem.
Before bootstrapping your cluster, you need to take additional steps to configure a writable directory.
See the [Kubeadm Troubleshooting guide](/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/#usr-mounted-read-only)
to learn how to set up a writable directory.

The kubelet is now restarting every few seconds, as it waits in a crashloop for
kubeadm to tell it what to do.

## Configuring a cgroup driver

Both the container runtime and the kubelet have a property called
["cgroup driver"](/docs/setup/production-environment/container-runtimes/#cgroup-drivers), which is important
for the management of cgroups on Linux machines.

Matching the container runtime and kubelet cgroup drivers is required or otherwise the kubelet process will fail.

See [Configuring a cgroup driver](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/) for more details.

## Troubleshooting

If you are running into difficulties with kubeadm, please consult our
[troubleshooting docs](/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/).

##

* [Using kubeadm to Create a Cluster](/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/)

title  Installing kubeadm content type  task weight  10 card    name  setup   weight  40   title  Install the kubeadm setup tool           overview       img src   images kubeadm stacked color png  align  right  width  150px    img  This page shows how to install the  kubeadm  toolbox  For information on how to create a cluster with kubeadm once you have performed this installation process  see the  Creating a cluster with kubeadm   docs setup production environment tools kubeadm create cluster kubeadm   page            A compatible Linux host  The Kubernetes project provides generic instructions for Linux distributions   based on Debian and Red Hat  and those distributions without a package manager    2 GB or more of RAM per machine  any less will leave little room for your apps     2 CPUs or more for control plane machines    Full network connectivity between all machines in the cluster  public or private network is fine     Unique hostname  MAC address  and product uuid for every node  See  here   verify mac address  for more details    Certain ports are open on your machines  See  here   check required ports  for more details    The  kubeadm  installation is done via binaries that use dynamic linking and assumes that your target system provides  glibc   This is a reasonable assumption on many Linux distributions  including Debian  Ubuntu  Fedora  CentOS  etc   but it is not always the case with custom and lightweight distributions which don t include  glibc  by default  such as Alpine Linux  The expectation is that the distribution either includes  glibc  or a  compatibility layer  https   wiki alpinelinux org wiki Running glibc programs  that provides the expected symbols         steps         Verify the MAC address and product uuid are unique for every node   verify mac address     You can get the MAC address of the network interfaces using the command  ip link  or  ifconfig  a    The product uuid can be checked by using the command  sudo cat  sys class dmi id product uuid   It is very likely that hardware devices will have unique addresses  although some virtual machines may have identical values  Kubernetes uses these values to uniquely identify the nodes in the cluster  If these values are not unique to each node  the installation process may  fail  https   github com kubernetes kubeadm issues 31       Check network adapters  If you have more than one network adapter  and your Kubernetes components are not reachable on the default route  we recommend you add IP route s  so Kubernetes cluster addresses go via the appropriate adapter      Check required ports   check required ports   These  required ports   docs reference networking ports and protocols   need to be open in order for Kubernetes components to communicate with each other  You can use tools like  netcat  https   netcat sourceforge net  to check if a port is open  For example      shell nc 127 0 0 1 6443  v      The pod network plugin you use may also require certain ports to be open  Since this differs with each pod network plugin  please see the documentation for the plugins about what port s  those need      Swap configuration   swap configuration   The default behavior of a kubelet is to fail to start if swap memory is detected on a node  This means that swap should either be disabled or tolerated by kubelet     To tolerate swap  add  failSwapOn  false  to kubelet configuration or as a command line argument    Note  even if  failSwapOn  false  is provided  workloads wouldn t have swap access by default    This can be changed by setting a  swapBehavior   again in the kubelet configuration file  To use swap    set a  swapBehavior  other than the default  NoSwap  setting    See  Swap memory management   docs concepts architecture nodes  swap memory  for more details    To disable swap   sudo swapoff  a  can be used to disable swapping temporarily    To make this change persistent across reboots  make sure swap is disabled in   config files like   etc fstab    systemd swap   depending how it was configured on your system       Installing a container runtime   installing runtime   To run containers in Pods  Kubernetes uses a    By default  Kubernetes uses the   CRI  to interface with your chosen container runtime   If you don t specify a runtime  kubeadm automatically tries to detect an installed container runtime by scanning through a list of known endpoints   If multiple or no container runtimes are detected kubeadm will throw an error and will request that you specify which one you want to use   See  container runtimes   docs setup production environment container runtimes   for more information    Docker Engine does not implement the  CRI   docs concepts architecture cri   which is a requirement for a container runtime to work with Kubernetes  For that reason  an additional service  cri dockerd  https   mirantis github io cri dockerd   has to be installed  cri dockerd is a project based on the legacy built in Docker Engine support that was  removed   dockershim  from the kubelet in version 1 24    The tables below include the known endpoints for supported operating systems         Runtime                              Path to Unix domain socket                                                                                containerd                            unix    var run containerd containerd sock      CRI O                                 unix    var run crio crio sock                  Docker Engine  using cri dockerd      unix    var run cri dockerd sock                       Runtime                              Path to Windows named pipe                                                                                containerd                            npipe       pipe containerd containerd          Docker Engine  using cri dockerd      npipe       pipe cri dockerd                          Installing kubeadm  kubelet and kubectl  You will install these packages on all of your machines      kubeadm   the command to bootstrap the cluster      kubelet   the component that runs on all of the machines in your cluster   and does things like starting pods and containers      kubectl   the command line util to talk to your cluster   kubeadm   will not   install or manage  kubelet  or  kubectl  for you  so you will need to ensure they match the version of the Kubernetes control plane you want kubeadm to install for you  If you do not  there is a risk of a version skew occurring that can lead to unexpected  buggy behaviour  However   one  minor version skew between the kubelet and the control plane is supported  but the kubelet version may never exceed the API server version  For example  the kubelet running 1 7 0 should be fully compatible with a 1 8 0 API server  but not vice versa   For information about installing  kubectl   see  Install and set up kubectl   docs tasks tools      These instructions exclude all Kubernetes packages from any system upgrades  This is because kubeadm and Kubernetes require  special attention to upgrade   docs tasks administer cluster kubeadm kubeadm upgrade      For more information on version skews  see     Kubernetes  version and version skew policy   docs setup release version skew policy     Kubeadm specific  version skew policy   docs setup production environment tools kubeadm create cluster kubeadm  version skew policy      There s a dedicated package repository for each Kubernetes minor version  If you want to install a minor version other than v  please see the installation guide for your desired minor version       These instructions are for Kubernetes v   1  Update the  apt  package index and install packages needed to use the Kubernetes  apt  repository         shell    sudo apt get update      apt transport https may be a dummy package  if so  you can skip that package    sudo apt get install  y apt transport https ca certificates curl gpg         2  Download the public signing key for the Kubernetes package repositories     The same signing key is used for all repositories so you can disregard the version in the URL         shell      If the directory   etc apt keyrings  does not exist  it should be created before the curl command  read the note below       sudo mkdir  p  m 755  etc apt keyrings    curl  fsSL https   pkgs k8s io core  stable   deb Release key   sudo gpg   dearmor  o  etc apt keyrings kubernetes apt keyring gpg          In releases older than Debian 12 and Ubuntu 22 04  directory   etc apt keyrings  does not exist by default  and it should be created before the curl command    3  Add the appropriate Kubernetes  apt  repository  Please note that this repository have packages    only for Kubernetes   for other Kubernetes minor versions  you need to    change the Kubernetes minor version in the URL to match your desired minor version     you should also check that you are reading the documentation for the version of Kubernetes    that you plan to install          shell      This overwrites any existing configuration in  etc apt sources list d kubernetes list    echo  deb  signed by  etc apt keyrings kubernetes apt keyring gpg  https   pkgs k8s io core  stable   deb       sudo tee  etc apt sources list d kubernetes list         4  Update the  apt  package index  install kubelet  kubeadm and kubectl  and pin their version         shell    sudo apt get update    sudo apt get install  y kubelet kubeadm kubectl    sudo apt mark hold kubelet kubeadm kubectl         5   Optional  Enable the kubelet service before running kubeadm         shell    sudo systemctl enable   now kubelet            1  Set SELinux to  permissive  mode      These instructions are for Kubernetes          shell      Set SELinux in permissive mode  effectively disabling it     sudo setenforce 0    sudo sed  i  s  SELINUX enforcing  SELINUX permissive    etc selinux config            Setting SELinux in permissive mode by running  setenforce 0  and  sed        effectively disables it  This is required to allow containers to access the host   filesystem  for example  some cluster network plugins require that  You have to   do this until SELinux support is improved in the kubelet    You can leave SELinux enabled if you know how to configure it but it may require   settings that are not supported by kubeadm    2  Add the Kubernetes  yum  repository  The  exclude  parameter in the    repository definition ensures that the packages related to Kubernetes are    not upgraded upon running  yum update  as there s a special procedure that    must be followed for upgrading Kubernetes  Please note that this repository    have packages only for Kubernetes   for other    Kubernetes minor versions  you need to change the Kubernetes minor version    in the URL to match your desired minor version  you should also check that    you are reading the documentation for the version of Kubernetes that you    plan to install          shell      This overwrites any existing configuration in  etc yum repos d kubernetes repo    cat   EOF   sudo tee  etc yum repos d kubernetes repo     kubernetes     name Kubernetes    baseurl https   pkgs k8s io core  stable   rpm     enabled 1    gpgcheck 1    gpgkey https   pkgs k8s io core  stable   rpm repodata repomd xml key    exclude kubelet kubeadm kubectl cri tools kubernetes cni    EOF         3  Install kubelet  kubeadm and kubectl         shell    sudo yum install  y kubelet kubeadm kubectl   disableexcludes kubernetes         4   Optional  Enable the kubelet service before running kubeadm         shell    sudo systemctl enable   now kubelet           Install CNI plugins  required for most pod network       bash CNI PLUGINS VERSION  v1 3 0  ARCH  amd64  DEST   opt cni bin  sudo mkdir  p   DEST  curl  L  https   github com containernetworking plugins releases download   CNI PLUGINS VERSION  cni plugins linux   ARCH    CNI PLUGINS VERSION  tgz    sudo tar  C   DEST   xz      Define the directory to download command files    The  DOWNLOAD DIR  variable must be set to a writable directory  If you are running Flatcar Container Linux  set  DOWNLOAD DIR   opt bin         bash DOWNLOAD DIR   usr local bin  sudo mkdir  p   DOWNLOAD DIR       Optionally install crictl  required for interaction with the Container Runtime Interface  CRI   optional for kubeadm       bash CRICTL VERSION  v1 31 0  ARCH  amd64  curl  L  https   github com kubernetes sigs cri tools releases download   CRICTL VERSION  crictl   CRICTL VERSION  linux   ARCH  tar gz    sudo tar  C  DOWNLOAD DIR  xz      Install  kubeadm    kubelet  and add a  kubelet  systemd service      bash RELEASE    curl  sSL https   dl k8s io release stable txt   ARCH  amd64  cd  DOWNLOAD DIR sudo curl  L   remote name all https   dl k8s io release   RELEASE  bin linux   ARCH   kubeadm kubelet  sudo chmod  x  kubeadm kubelet   RELEASE VERSION  v0 16 2  curl  sSL  https   raw githubusercontent com kubernetes release   RELEASE VERSION  cmd krel templates latest kubelet kubelet service    sed  s  usr bin   DOWNLOAD DIR  g    sudo tee  usr lib systemd system kubelet service sudo mkdir  p  usr lib systemd system kubelet service d curl  sSL  https   raw githubusercontent com kubernetes release   RELEASE VERSION  cmd krel templates latest kubeadm 10 kubeadm conf    sed  s  usr bin   DOWNLOAD DIR  g    sudo tee  usr lib systemd system kubelet service d 10 kubeadm conf       Please refer to the note in the  Before you begin   before you begin  section for Linux distributions that do not include  glibc  by default    Install  kubectl  by following the instructions on  Install Tools page   docs tasks tools  kubectl    Optionally  enable the kubelet service before running kubeadm      bash sudo systemctl enable   now kubelet       The Flatcar Container Linux distribution mounts the   usr  directory as a read only filesystem  Before bootstrapping your cluster  you need to take additional steps to configure a writable directory  See the  Kubeadm Troubleshooting guide   docs setup production environment tools kubeadm troubleshooting kubeadm  usr mounted read only  to learn how to set up a writable directory      The kubelet is now restarting every few seconds  as it waits in a crashloop for kubeadm to tell it what to do      Configuring a cgroup driver  Both the container runtime and the kubelet have a property called   cgroup driver    docs setup production environment container runtimes  cgroup drivers   which is important for the management of cgroups on Linux machines    Matching the container runtime and kubelet cgroup drivers is required or otherwise the kubelet process will fail   See  Configuring a cgroup driver   docs tasks administer cluster kubeadm configure cgroup driver   for more details       Troubleshooting  If you are running into difficulties with kubeadm  please consult our  troubleshooting docs   docs setup production environment tools kubeadm troubleshooting kubeadm             Using kubeadm to Create a Cluster   docs setup production environment tools kubeadm create cluster kubeadm


kubernetes setup title Customizing components with the kubeadm API contenttype concept weight 40 sig cluster lifecycle reviewers overview

Answer:
---
reviewers:
- sig-cluster-lifecycle
title: Customizing components with the kubeadm API
content_type: concept
weight: 40
---

<!-- overview -->

This page covers how to customize the components that kubeadm deploys. For control plane components
you can use flags in the `ClusterConfiguration` structure or patches per-node. For the kubelet
and kube-proxy you can use `KubeletConfiguration` and `KubeProxyConfiguration`, accordingly.

All of these options are possible via the kubeadm configuration API.
For more details on each field in the configuration you can navigate to our
[API reference pages](/docs/reference/config-api/kubeadm-config.v1beta4/).

Customizing the CoreDNS deployment of kubeadm is currently not supported. You must manually
patch the `kube-system/coredns`
and recreate the CoreDNS  after that. Alternatively,
you can skip the default CoreDNS deployment and deploy your own variant.
For more details on that see [Using init phases with kubeadm](/docs/reference/setup-tools/kubeadm/kubeadm-init/#init-phases).

To reconfigure a cluster that has already been created see
[Reconfiguring a kubeadm cluster](/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure).

<!-- body -->

## Customizing the control plane with flags in `ClusterConfiguration`

The kubeadm `ClusterConfiguration` object exposes a way for users to override the default
flags passed to control plane components such as the APIServer, ControllerManager, Scheduler and Etcd.
The components are defined using the following structures:

- `apiServer`
- `controllerManager`
- `scheduler`
- `etcd`

These structures contain a common `extraArgs` field, that consists of `name` / `value` pairs.
To override a flag for a control plane component:

1.  Add the appropriate `extraArgs` to your configuration.
2.  Add flags to the `extraArgs` field.
3.  Run `kubeadm init` with `--config <YOUR CONFIG YAML>`.

You can generate a `ClusterConfiguration` object with default values by running `kubeadm config print init-defaults`
and saving the output to a file of your choice.

The `ClusterConfiguration` object is currently global in kubeadm clusters. This means that any flags that you add,
will apply to all instances of the same component on different nodes. To apply individual configuration per component
on different nodes you can use [patches](#patches).

Duplicate flags (keys), or passing the same flag `--foo` multiple times, is currently not supported.
To workaround that you must use [patches](#patches).

### APIServer flags

For details, see the [reference documentation for kube-apiserver](/docs/reference/command-line-tools-reference/kube-apiserver/).

Example usage:

```yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.16.0
apiServer:
  extraArgs:
  - name: "enable-admission-plugins"
    value: "AlwaysPullImages,DefaultStorageClass"
  - name: "audit-log-path"
    value: "/home/johndoe/audit.log"
```

### ControllerManager flags

For details, see the [reference documentation for kube-controller-manager](/docs/reference/command-line-tools-reference/kube-controller-manager/).

Example usage:

```yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.16.0
controllerManager:
  extraArgs:
  - name: "cluster-signing-key-file"
    value: "/home/johndoe/keys/ca.key"
  - name: "deployment-controller-sync-period"
    value: "50"
```

### Scheduler flags

For details, see the [reference documentation for kube-scheduler](/docs/reference/command-line-tools-reference/kube-scheduler/).

Example usage:

```yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.16.0
scheduler:
  extraArgs:
  - name: "config"
    value: "/etc/kubernetes/scheduler-config.yaml"
  extraVolumes:
    - name: schedulerconfig
      hostPath: /home/johndoe/schedconfig.yaml
      mountPath: /etc/kubernetes/scheduler-config.yaml
      readOnly: true
      pathType: "File"
```

### Etcd flags

For details, see the [etcd server documentation](https://etcd.io/docs/).

Example usage:

```yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
etcd:
  local:
    extraArgs:
    - name: "election-timeout"
      value: 1000
```

## Customizing with patches {#patches}

Kubeadm allows you to pass a directory with patch files to `InitConfiguration` and `JoinConfiguration`
on individual nodes. These patches can be used as the last customization step before component configuration
is written to disk.

You can pass this file to `kubeadm init` with `--config <YOUR CONFIG YAML>`:

```yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
patches:
  directory: /home/user/somedir
```

For `kubeadm init` you can pass a file containing both a `ClusterConfiguration` and `InitConfiguration`
separated by `---`.

You can pass this file to `kubeadm join` with `--config <YOUR CONFIG YAML>`:

```yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: JoinConfiguration
patches:
  directory: /home/user/somedir
```

The directory must contain files named `target[suffix][+patchtype].extension`.
For example, `kube-apiserver0+merge.yaml` or just `etcd.json`.

- `target` can be one of `kube-apiserver`, `kube-controller-manager`, `kube-scheduler`, `etcd`
and `kubeletconfiguration`.
- `suffix` is an optional string that can be used to determine which patches are applied first
alpha-numerically.
- `patchtype` can be one of `strategic`, `merge` or `json` and these must match the patching formats
[supported by kubectl](/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch).
The default `patchtype` is `strategic`.
- `extension` must be either `json` or `yaml`.

If you are using `kubeadm upgrade` to upgrade your kubeadm nodes you must again provide the same
patches, so that the customization is preserved after upgrade. To do that you can use the `--patches`
flag, which must point to the same directory. `kubeadm upgrade` currently does not support a configuration
API structure that can be used for the same purpose.

## Customizing the kubelet {#kubelet}

To customize the kubelet you can add a [`KubeletConfiguration`](/docs/reference/config-api/kubelet-config.v1beta1/)
next to the `ClusterConfiguration` or `InitConfiguration` separated by `---` within the same configuration file.
This file can then be passed to `kubeadm init` and kubeadm will apply the same base `KubeletConfiguration`
to all nodes in the cluster.

For applying instance-specific configuration over the base `KubeletConfiguration` you can use the
[`kubeletconfiguration` patch target](#patches).

Alternatively, you can use kubelet flags as overrides by passing them in the
`nodeRegistration.kubeletExtraArgs` field supported by both `InitConfiguration` and `JoinConfiguration`.
Some kubelet flags are deprecated, so check their status in the
[kubelet reference documentation](/docs/reference/command-line-tools-reference/kubelet) before using them.

For additional details see [Configuring each kubelet in your cluster using kubeadm](/docs/setup/production-environment/tools/kubeadm/kubelet-integration)

## Customizing kube-proxy

To customize kube-proxy you can pass a `KubeProxyConfiguration` next your `ClusterConfiguration` or
`InitConfiguration` to `kubeadm init` separated by `---`.

For more details you can navigate to our [API reference pages](/docs/reference/config-api/kubeadm-config.v1beta4/).

kubeadm deploys kube-proxy as a , which means
that the `KubeProxyConfiguration` would apply to all instances of kube-proxy in the cluster.

reviewers    sig cluster lifecycle title  Customizing components with the kubeadm API content type  concept weight  40           overview      This page covers how to customize the components that kubeadm deploys  For control plane components you can use flags in the  ClusterConfiguration  structure or patches per node  For the kubelet and kube proxy you can use  KubeletConfiguration  and  KubeProxyConfiguration   accordingly   All of these options are possible via the kubeadm configuration API  For more details on each field in the configuration you can navigate to our  API reference pages   docs reference config api kubeadm config v1beta4      Customizing the CoreDNS deployment of kubeadm is currently not supported  You must manually patch the  kube system coredns   and recreate the CoreDNS  after that  Alternatively  you can skip the default CoreDNS deployment and deploy your own variant  For more details on that see  Using init phases with kubeadm   docs reference setup tools kubeadm kubeadm init  init phases      To reconfigure a cluster that has already been created see  Reconfiguring a kubeadm cluster   docs tasks administer cluster kubeadm kubeadm reconfigure          body         Customizing the control plane with flags in  ClusterConfiguration   The kubeadm  ClusterConfiguration  object exposes a way for users to override the default flags passed to control plane components such as the APIServer  ControllerManager  Scheduler and Etcd  The components are defined using the following structures      apiServer     controllerManager     scheduler     etcd   These structures contain a common  extraArgs  field  that consists of  name     value  pairs  To override a flag for a control plane component   1   Add the appropriate  extraArgs  to your configuration  2   Add flags to the  extraArgs  field  3   Run  kubeadm init  with    config  YOUR CONFIG YAML      You can generate a  ClusterConfiguration  object with default values by running  kubeadm config print init defaults  and saving the output to a file of your choice     The  ClusterConfiguration  object is currently global in kubeadm clusters  This means that any flags that you add  will apply to all instances of the same component on different nodes  To apply individual configuration per component on different nodes you can use  patches   patches      Duplicate flags  keys   or passing the same flag    foo  multiple times  is currently not supported  To workaround that you must use  patches   patches         APIServer flags  For details  see the  reference documentation for kube apiserver   docs reference command line tools reference kube apiserver     Example usage      yaml apiVersion  kubeadm k8s io v1beta4 kind  ClusterConfiguration kubernetesVersion  v1 16 0 apiServer    extraArgs      name   enable admission plugins      value   AlwaysPullImages DefaultStorageClass      name   audit log path      value    home johndoe audit log           ControllerManager flags  For details  see the  reference documentation for kube controller manager   docs reference command line tools reference kube controller manager     Example usage      yaml apiVersion  kubeadm k8s io v1beta4 kind  ClusterConfiguration kubernetesVersion  v1 16 0 controllerManager    extraArgs      name   cluster signing key file      value    home johndoe keys ca key      name   deployment controller sync period      value   50           Scheduler flags  For details  see the  reference documentation for kube scheduler   docs reference command line tools reference kube scheduler     Example usage      yaml apiVersion  kubeadm k8s io v1beta4 kind  ClusterConfiguration kubernetesVersion  v1 16 0 scheduler    extraArgs      name   config      value    etc kubernetes scheduler config yaml    extraVolumes        name  schedulerconfig       hostPath   home johndoe schedconfig yaml       mountPath   etc kubernetes scheduler config yaml       readOnly  true       pathType   File           Etcd flags  For details  see the  etcd server documentation  https   etcd io docs     Example usage      yaml apiVersion  kubeadm k8s io v1beta4 kind  ClusterConfiguration etcd    local      extraArgs        name   election timeout        value  1000         Customizing with patches   patches     Kubeadm allows you to pass a directory with patch files to  InitConfiguration  and  JoinConfiguration  on individual nodes  These patches can be used as the last customization step before component configuration is written to disk   You can pass this file to  kubeadm init  with    config  YOUR CONFIG YAML        yaml apiVersion  kubeadm k8s io v1beta4 kind  InitConfiguration patches    directory   home user somedir       For  kubeadm init  you can pass a file containing both a  ClusterConfiguration  and  InitConfiguration  separated by          You can pass this file to  kubeadm join  with    config  YOUR CONFIG YAML        yaml apiVersion  kubeadm k8s io v1beta4 kind  JoinConfiguration patches    directory   home user somedir      The directory must contain files named  target suffix   patchtype  extension   For example   kube apiserver0 merge yaml  or just  etcd json       target  can be one of  kube apiserver    kube controller manager    kube scheduler    etcd  and  kubeletconfiguration      suffix  is an optional string that can be used to determine which patches are applied first alpha numerically     patchtype  can be one of  strategic    merge  or  json  and these must match the patching formats  supported by kubectl   docs tasks manage kubernetes objects update api object kubectl patch   The default  patchtype  is  strategic      extension  must be either  json  or  yaml     If you are using  kubeadm upgrade  to upgrade your kubeadm nodes you must again provide the same patches  so that the customization is preserved after upgrade  To do that you can use the    patches  flag  which must point to the same directory   kubeadm upgrade  currently does not support a configuration API structure that can be used for the same purpose       Customizing the kubelet   kubelet   To customize the kubelet you can add a   KubeletConfiguration    docs reference config api kubelet config v1beta1   next to the  ClusterConfiguration  or  InitConfiguration  separated by       within the same configuration file  This file can then be passed to  kubeadm init  and kubeadm will apply the same base  KubeletConfiguration  to all nodes in the cluster   For applying instance specific configuration over the base  KubeletConfiguration  you can use the   kubeletconfiguration  patch target   patches    Alternatively  you can use kubelet flags as overrides by passing them in the  nodeRegistration kubeletExtraArgs  field supported by both  InitConfiguration  and  JoinConfiguration   Some kubelet flags are deprecated  so check their status in the  kubelet reference documentation   docs reference command line tools reference kubelet  before using them   For additional details see  Configuring each kubelet in your cluster using kubeadm   docs setup production environment tools kubeadm kubelet integration      Customizing kube proxy  To customize kube proxy you can pass a  KubeProxyConfiguration  next your  ClusterConfiguration  or  InitConfiguration  to  kubeadm init  separated by         For more details you can navigate to our  API reference pages   docs reference config api kubeadm config v1beta4      kubeadm deploys kube proxy as a   which means that the  KubeProxyConfiguration  would apply to all instances of kube proxy in the cluster


kubernetes setup weight 70 title Set up a High Availability etcd Cluster with kubeadm contenttype task sig cluster lifecycle reviewers overview

Answer:
---
reviewers:
- sig-cluster-lifecycle
title: Set up a High Availability etcd Cluster with kubeadm
content_type: task
weight: 70
---

<!-- overview -->

By default, kubeadm runs a local etcd instance on each control plane node.
It is also possible to treat the etcd cluster as external and provision
etcd instances on separate hosts. The differences between the two approaches are covered in the
[Options for Highly Available topology](/docs/setup/production-environment/tools/kubeadm/ha-topology) page.

This task walks through the process of creating a high availability external
etcd cluster of three members that can be used by kubeadm during cluster creation.

##

- Three hosts that can talk to each other over TCP ports 2379 and 2380. This
  document assumes these default ports. However, they are configurable through
  the kubeadm config file.
- Each host must have systemd and a bash compatible shell installed.
- Each host must [have a container runtime, kubelet, and kubeadm installed](/docs/setup/production-environment/tools/kubeadm/install-kubeadm/).
- Each host should have access to the Kubernetes container image registry (`registry.k8s.io`) or list/pull the required etcd image using
  `kubeadm config images list/pull`. This guide will set up etcd instances as
  [static pods](/docs/tasks/configure-pod-container/static-pod/) managed by a kubelet.
- Some infrastructure to copy files between hosts. For example `ssh` and `scp`
  can satisfy this requirement.

<!-- steps -->

## Setting up the cluster

The general approach is to generate all certs on one node and only distribute
the _necessary_ files to the other nodes.

kubeadm contains all the necessary cryptographic machinery to generate
the certificates described below; no other cryptographic tooling is required for
this example.

The examples below use IPv4 addresses but you can also configure kubeadm, the kubelet and etcd
to use IPv6 addresses. Dual-stack is supported by some Kubernetes options, but not by etcd. For more details
on Kubernetes dual-stack support see [Dual-stack support with kubeadm](/docs/setup/production-environment/tools/kubeadm/dual-stack-support/).

1. Configure the kubelet to be a service manager for etcd.

   You must do this on every host where etcd should be running.
   Since etcd was created first, you must override the service priority by creating a new unit file
   that has higher precedence than the kubeadm-provided kubelet unit file.

   ```sh
   cat << EOF > /etc/systemd/system/kubelet.service.d/kubelet.conf
   # Replace "systemd" with the cgroup driver of your container runtime. The default value in the kubelet is "cgroupfs".
   # Replace the value of "containerRuntimeEndpoint" for a different container runtime if needed.
   #
   apiVersion: kubelet.config.k8s.io/v1beta1
   kind: KubeletConfiguration
   authentication:
     anonymous:
       enabled: false
     webhook:
       enabled: false
   authorization:
     mode: AlwaysAllow
   cgroupDriver: systemd
   address: 127.0.0.1
   containerRuntimeEndpoint: unix:///var/run/containerd/containerd.sock
   staticPodPath: /etc/kubernetes/manifests
   EOF

   cat << EOF > /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf
   [Service]
   ExecStart=
   ExecStart=/usr/bin/kubelet --config=/etc/systemd/system/kubelet.service.d/kubelet.conf
   Restart=always
   EOF

   systemctl daemon-reload
   systemctl restart kubelet
   ```

   Check the kubelet status to ensure it is running.

   ```sh
   systemctl status kubelet
   ```

1. Create configuration files for kubeadm.

   Generate one kubeadm configuration file for each host that will have an etcd
   member running on it using the following script.

   ```sh
   # Update HOST0, HOST1 and HOST2 with the IPs of your hosts
   export HOST0=10.0.0.6
   export HOST1=10.0.0.7
   export HOST2=10.0.0.8

   # Update NAME0, NAME1 and NAME2 with the hostnames of your hosts
   export NAME0="infra0"
   export NAME1="infra1"
   export NAME2="infra2"

   # Create temp directories to store files that will end up on other hosts
   mkdir -p /tmp/${HOST0}/ /tmp/${HOST1}/ /tmp/${HOST2}/

   HOSTS=(${HOST0} ${HOST1} ${HOST2})
   NAMES=(${NAME0} ${NAME1} ${NAME2})

   for i in "${!HOSTS[@]}"; do
   HOST=${HOSTS[$i]}
   NAME=${NAMES[$i]}
   cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml
   ---
   apiVersion: "kubeadm.k8s.io/v1beta4"
   kind: InitConfiguration
   nodeRegistration:
       name: ${NAME}
   localAPIEndpoint:
       advertiseAddress: ${HOST}
   ---
   apiVersion: "kubeadm.k8s.io/v1beta4"
   kind: ClusterConfiguration
   etcd:
       local:
           serverCertSANs:
           - "${HOST}"
           peerCertSANs:
           - "${HOST}"
           extraArgs:
           - name: initial-cluster
             value: ${NAMES[0]}=https://${HOSTS[0]}:2380,${NAMES[1]}=https://${HOSTS[1]}:2380,${NAMES[2]}=https://${HOSTS[2]}:2380
           - name: initial-cluster-state
             value: new
           - name: name
             value: ${NAME}
           - name: listen-peer-urls
             value: https://${HOST}:2380
           - name: listen-client-urls
             value: https://${HOST}:2379
           - name: advertise-client-urls
             value: https://${HOST}:2379
           - name: initial-advertise-peer-urls
             value: https://${HOST}:2380
   EOF
   done
   ```

1. Generate the certificate authority.

   If you already have a CA then the only action that is copying the CA's `crt` and
   `key` file to `/etc/kubernetes/pki/etcd/ca.crt` and
   `/etc/kubernetes/pki/etcd/ca.key`. After those files have been copied,
   proceed to the next step, "Create certificates for each member".

   If you do not already have a CA then run this command on `$HOST0` (where you
   generated the configuration files for kubeadm).

   ```
   kubeadm init phase certs etcd-ca
   ```

   This creates two files:

   - `/etc/kubernetes/pki/etcd/ca.crt`
   - `/etc/kubernetes/pki/etcd/ca.key`

1. Create certificates for each member.

   ```sh
   kubeadm init phase certs etcd-server --config=/tmp/${HOST2}/kubeadmcfg.yaml
   kubeadm init phase certs etcd-peer --config=/tmp/${HOST2}/kubeadmcfg.yaml
   kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
   kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST2}/kubeadmcfg.yaml
   cp -R /etc/kubernetes/pki /tmp/${HOST2}/
   # cleanup non-reusable certificates
   find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete

   kubeadm init phase certs etcd-server --config=/tmp/${HOST1}/kubeadmcfg.yaml
   kubeadm init phase certs etcd-peer --config=/tmp/${HOST1}/kubeadmcfg.yaml
   kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
   kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST1}/kubeadmcfg.yaml
   cp -R /etc/kubernetes/pki /tmp/${HOST1}/
   find /etc/kubernetes/pki -not -name ca.crt -not -name ca.key -type f -delete

   kubeadm init phase certs etcd-server --config=/tmp/${HOST0}/kubeadmcfg.yaml
   kubeadm init phase certs etcd-peer --config=/tmp/${HOST0}/kubeadmcfg.yaml
   kubeadm init phase certs etcd-healthcheck-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
   kubeadm init phase certs apiserver-etcd-client --config=/tmp/${HOST0}/kubeadmcfg.yaml
   # No need to move the certs because they are for HOST0

   # clean up certs that should not be copied off this host
   find /tmp/${HOST2} -name ca.key -type f -delete
   find /tmp/${HOST1} -name ca.key -type f -delete
   ```

1. Copy certificates and kubeadm configs.

   The certificates have been generated and now they must be moved to their
   respective hosts.

   ```sh
   USER=ubuntu
   HOST=${HOST1}
   scp -r /tmp/${HOST}/* ${USER}@${HOST}:
   ssh ${USER}@${HOST}
   USER@HOST $ sudo -Es
   root@HOST $ chown -R root:root pki
   root@HOST $ mv pki /etc/kubernetes/
   ```

1. Ensure all expected files exist.

   The complete list of required files on `$HOST0` is:

   ```
   /tmp/${HOST0}
   └── kubeadmcfg.yaml
   ---
   /etc/kubernetes/pki
   ├── apiserver-etcd-client.crt
   ├── apiserver-etcd-client.key
   └── etcd
       ├── ca.crt
       ├── ca.key
       ├── healthcheck-client.crt
       ├── healthcheck-client.key
       ├── peer.crt
       ├── peer.key
       ├── server.crt
       └── server.key
   ```

   On `$HOST1`:

   ```
   $HOME
   └── kubeadmcfg.yaml
   ---
   /etc/kubernetes/pki
   ├── apiserver-etcd-client.crt
   ├── apiserver-etcd-client.key
   └── etcd
       ├── ca.crt
       ├── healthcheck-client.crt
       ├── healthcheck-client.key
       ├── peer.crt
       ├── peer.key
       ├── server.crt
       └── server.key
   ```

   On `$HOST2`:

   ```
   $HOME
   └── kubeadmcfg.yaml
   ---
   /etc/kubernetes/pki
   ├── apiserver-etcd-client.crt
   ├── apiserver-etcd-client.key
   └── etcd
       ├── ca.crt
       ├── healthcheck-client.crt
       ├── healthcheck-client.key
       ├── peer.crt
       ├── peer.key
       ├── server.crt
       └── server.key
   ```

1. Create the static pod manifests.

   Now that the certificates and configs are in place it's time to create the
   manifests. On each host run the `kubeadm` command to generate a static manifest
   for etcd.

   ```sh
   root@HOST0 $ kubeadm init phase etcd local --config=/tmp/${HOST0}/kubeadmcfg.yaml
   root@HOST1 $ kubeadm init phase etcd local --config=$HOME/kubeadmcfg.yaml
   root@HOST2 $ kubeadm init phase etcd local --config=$HOME/kubeadmcfg.yaml
   ```

1. Optional: Check the cluster health.

    If `etcdctl` isn't available, you can run this tool inside a container image.
    You would do that directly with your container runtime using a tool such as
    `crictl run` and not through Kubernetes

    ```sh
    ETCDCTL_API=3 etcdctl \
    --cert /etc/kubernetes/pki/etcd/peer.crt \
    --key /etc/kubernetes/pki/etcd/peer.key \
    --cacert /etc/kubernetes/pki/etcd/ca.crt \
    --endpoints https://${HOST0}:2379 endpoint health
    ...
    https://[HOST0 IP]:2379 is healthy: successfully committed proposal: took = 16.283339ms
    https://[HOST1 IP]:2379 is healthy: successfully committed proposal: took = 19.44402ms
    https://[HOST2 IP]:2379 is healthy: successfully committed proposal: took = 35.926451ms
    ```

    - Set `${HOST0}`to the IP address of the host you are testing.

##

Once you have an etcd cluster with 3 working members, you can continue setting up a
highly available control plane using the
[external etcd method with kubeadm](/docs/setup/production-environment/tools/kubeadm/high-availability/).

reviewers    sig cluster lifecycle title  Set up a High Availability etcd Cluster with kubeadm content type  task weight  70           overview       By default  kubeadm runs a local etcd instance on each control plane node  It is also possible to treat the etcd cluster as external and provision etcd instances on separate hosts  The differences between the two approaches are covered in the  Options for Highly Available topology   docs setup production environment tools kubeadm ha topology  page   This task walks through the process of creating a high availability external etcd cluster of three members that can be used by kubeadm during cluster creation          Three hosts that can talk to each other over TCP ports 2379 and 2380  This   document assumes these default ports  However  they are configurable through   the kubeadm config file    Each host must have systemd and a bash compatible shell installed    Each host must  have a container runtime  kubelet  and kubeadm installed   docs setup production environment tools kubeadm install kubeadm      Each host should have access to the Kubernetes container image registry   registry k8s io   or list pull the required etcd image using    kubeadm config images list pull   This guide will set up etcd instances as    static pods   docs tasks configure pod container static pod   managed by a kubelet    Some infrastructure to copy files between hosts  For example  ssh  and  scp    can satisfy this requirement        steps         Setting up the cluster  The general approach is to generate all certs on one node and only distribute the  necessary  files to the other nodes    kubeadm contains all the necessary cryptographic machinery to generate the certificates described below  no other cryptographic tooling is required for this example     The examples below use IPv4 addresses but you can also configure kubeadm  the kubelet and etcd to use IPv6 addresses  Dual stack is supported by some Kubernetes options  but not by etcd  For more details on Kubernetes dual stack support see  Dual stack support with kubeadm   docs setup production environment tools kubeadm dual stack support      1  Configure the kubelet to be a service manager for etcd      You must do this on every host where etcd should be running     Since etcd was created first  you must override the service priority by creating a new unit file    that has higher precedence than the kubeadm provided kubelet unit file         sh    cat    EOF    etc systemd system kubelet service d kubelet conf      Replace  systemd  with the cgroup driver of your container runtime  The default value in the kubelet is  cgroupfs        Replace the value of  containerRuntimeEndpoint  for a different container runtime if needed          apiVersion  kubelet config k8s io v1beta1    kind  KubeletConfiguration    authentication       anonymous         enabled  false      webhook         enabled  false    authorization       mode  AlwaysAllow    cgroupDriver  systemd    address  127 0 0 1    containerRuntimeEndpoint  unix    var run containerd containerd sock    staticPodPath   etc kubernetes manifests    EOF     cat    EOF    etc systemd system kubelet service d 20 etcd service manager conf     Service     ExecStart     ExecStart  usr bin kubelet   config  etc systemd system kubelet service d kubelet conf    Restart always    EOF     systemctl daemon reload    systemctl restart kubelet            Check the kubelet status to ensure it is running         sh    systemctl status kubelet         1  Create configuration files for kubeadm      Generate one kubeadm configuration file for each host that will have an etcd    member running on it using the following script         sh      Update HOST0  HOST1 and HOST2 with the IPs of your hosts    export HOST0 10 0 0 6    export HOST1 10 0 0 7    export HOST2 10 0 0 8       Update NAME0  NAME1 and NAME2 with the hostnames of your hosts    export NAME0  infra0     export NAME1  infra1     export NAME2  infra2        Create temp directories to store files that will end up on other hosts    mkdir  p  tmp   HOST0    tmp   HOST1    tmp   HOST2       HOSTS    HOST0    HOST1    HOST2      NAMES    NAME0    NAME1    NAME2       for i in     HOSTS       do    HOST   HOSTS  i      NAME   NAMES  i      cat    EOF    tmp   HOST  kubeadmcfg yaml           apiVersion   kubeadm k8s io v1beta4     kind  InitConfiguration    nodeRegistration         name    NAME     localAPIEndpoint         advertiseAddress    HOST            apiVersion   kubeadm k8s io v1beta4     kind  ClusterConfiguration    etcd         local             serverCertSANs                  HOST              peerCertSANs                  HOST              extraArgs               name  initial cluster              value    NAMES 0   https     HOSTS 0   2380   NAMES 1   https     HOSTS 1   2380   NAMES 2   https     HOSTS 2   2380              name  initial cluster state              value  new              name  name              value    NAME               name  listen peer urls              value  https     HOST  2380              name  listen client urls              value  https     HOST  2379              name  advertise client urls              value  https     HOST  2379              name  initial advertise peer urls              value  https     HOST  2380    EOF    done         1  Generate the certificate authority      If you already have a CA then the only action that is copying the CA s  crt  and     key  file to   etc kubernetes pki etcd ca crt  and      etc kubernetes pki etcd ca key   After those files have been copied     proceed to the next step   Create certificates for each member       If you do not already have a CA then run this command on   HOST0   where you    generated the configuration files for kubeadm              kubeadm init phase certs etcd ca            This creates two files          etc kubernetes pki etcd ca crt         etc kubernetes pki etcd ca key   1  Create certificates for each member         sh    kubeadm init phase certs etcd server   config  tmp   HOST2  kubeadmcfg yaml    kubeadm init phase certs etcd peer   config  tmp   HOST2  kubeadmcfg yaml    kubeadm init phase certs etcd healthcheck client   config  tmp   HOST2  kubeadmcfg yaml    kubeadm init phase certs apiserver etcd client   config  tmp   HOST2  kubeadmcfg yaml    cp  R  etc kubernetes pki  tmp   HOST2        cleanup non reusable certificates    find  etc kubernetes pki  not  name ca crt  not  name ca key  type f  delete     kubeadm init phase certs etcd server   config  tmp   HOST1  kubeadmcfg yaml    kubeadm init phase certs etcd peer   config  tmp   HOST1  kubeadmcfg yaml    kubeadm init phase certs etcd healthcheck client   config  tmp   HOST1  kubeadmcfg yaml    kubeadm init phase certs apiserver etcd client   config  tmp   HOST1  kubeadmcfg yaml    cp  R  etc kubernetes pki  tmp   HOST1      find  etc kubernetes pki  not  name ca crt  not  name ca key  type f  delete     kubeadm init phase certs etcd server   config  tmp   HOST0  kubeadmcfg yaml    kubeadm init phase certs etcd peer   config  tmp   HOST0  kubeadmcfg yaml    kubeadm init phase certs etcd healthcheck client   config  tmp   HOST0  kubeadmcfg yaml    kubeadm init phase certs apiserver etcd client   config  tmp   HOST0  kubeadmcfg yaml      No need to move the certs because they are for HOST0       clean up certs that should not be copied off this host    find  tmp   HOST2   name ca key  type f  delete    find  tmp   HOST1   name ca key  type f  delete         1  Copy certificates and kubeadm configs      The certificates have been generated and now they must be moved to their    respective hosts         sh    USER ubuntu    HOST   HOST1     scp  r  tmp   HOST      USER    HOST      ssh   USER    HOST     USER HOST   sudo  Es    root HOST   chown  R root root pki    root HOST   mv pki  etc kubernetes          1  Ensure all expected files exist      The complete list of required files on   HOST0  is              tmp   HOST0         kubeadmcfg yaml            etc kubernetes pki        apiserver etcd client crt        apiserver etcd client key        etcd            ca crt            ca key            healthcheck client crt            healthcheck client key            peer crt            peer key            server crt            server key            On   HOST1               HOME        kubeadmcfg yaml            etc kubernetes pki        apiserver etcd client crt        apiserver etcd client key        etcd            ca crt            healthcheck client crt            healthcheck client key            peer crt            peer key            server crt            server key            On   HOST2               HOME        kubeadmcfg yaml            etc kubernetes pki        apiserver etcd client crt        apiserver etcd client key        etcd            ca crt            healthcheck client crt            healthcheck client key            peer crt            peer key            server crt            server key         1  Create the static pod manifests      Now that the certificates and configs are in place it s time to create the    manifests  On each host run the  kubeadm  command to generate a static manifest    for etcd         sh    root HOST0   kubeadm init phase etcd local   config  tmp   HOST0  kubeadmcfg yaml    root HOST1   kubeadm init phase etcd local   config  HOME kubeadmcfg yaml    root HOST2   kubeadm init phase etcd local   config  HOME kubeadmcfg yaml         1  Optional  Check the cluster health       If  etcdctl  isn t available  you can run this tool inside a container image      You would do that directly with your container runtime using a tool such as      crictl run  and not through Kubernetes         sh     ETCDCTL API 3 etcdctl         cert  etc kubernetes pki etcd peer crt         key  etc kubernetes pki etcd peer key         cacert  etc kubernetes pki etcd ca crt         endpoints https     HOST0  2379 endpoint health             https    HOST0 IP  2379 is healthy  successfully committed proposal  took   16 283339ms     https    HOST1 IP  2379 is healthy  successfully committed proposal  took   19 44402ms     https    HOST2 IP  2379 is healthy  successfully committed proposal  took   35 926451ms                Set    HOST0  to the IP address of the host you are testing         Once you have an etcd cluster with 3 working members  you can continue setting up a highly available control plane using the  external etcd method with kubeadm   docs setup production environment tools kubeadm high availability