kubernetes setup A production quality Kubernetes cluster requires planning and preparation title Production environment weight 30 If your Kubernetes cluster is to run critical workloads it must be configured to be resilient nolist true overview Create a production quality Kubernetes cluster

Answer:
---
title: "Production environment"
description: Create a production-quality Kubernetes cluster
weight: 30
no_list: true
---
<!-- overview -->

A production-quality Kubernetes cluster requires planning and preparation.
If your Kubernetes cluster is to run critical workloads, it must be configured to be resilient.
This page explains steps you can take to set up a production-ready cluster,
or to promote an existing cluster for production use.
If you're already familiar with production setup and want the links, skip to
[What's next](#what-s-next).

<!-- body -->

## Production considerations

Typically, a production Kubernetes cluster environment has more requirements than a
personal learning, development, or test environment Kubernetes. A production environment may require
secure access by many users, consistent availability, and the resources to adapt
to changing demands.

As you decide where you want your production Kubernetes environment to live
(on premises or in a cloud) and the amount of management you want to take
on or hand to others, consider how your requirements for a Kubernetes cluster
are influenced by the following issues:

- *Availability*: A single-machine Kubernetes [learning environment](/docs/setup/#learning-environment)
  has a single point of failure. Creating a highly available cluster means considering:
  - Separating the control plane from the worker nodes.
  - Replicating the control plane components on multiple nodes.
  - Load balancing traffic to the cluster’s .
  - Having enough worker nodes available, or able to quickly become available, as changing workloads warrant it.

- *Scale*: If you expect your production Kubernetes environment to receive a stable amount of
  demand, you might be able to set up for the capacity you need and be done. However,
  if you expect demand to grow over time or change dramatically based on things like
  season or special events, you need to plan how to scale to relieve increased
  pressure from more requests to the control plane and worker nodes or scale down to reduce unused
  resources.

- *Security and access management*: You have full admin privileges on your own
  Kubernetes learning cluster. But shared clusters with important workloads, and
  more than one or two users, require a more refined approach to who and what can
  access cluster resources. You can use role-based access control
  ([RBAC](/docs/reference/access-authn-authz/rbac/)) and other
  security mechanisms to make sure that users and workloads can get access to the
  resources they need, while keeping workloads, and the cluster itself, secure.
  You can set limits on the resources that users and workloads can access
  by managing [policies](/docs/concepts/policy/) and
  [container resources](/docs/concepts/configuration/manage-resources-containers/).

Before building a Kubernetes production environment on your own, consider
handing off some or all of this job to
[Turnkey Cloud Solutions](/docs/setup/production-environment/turnkey-solutions/)
providers or other [Kubernetes Partners](/partners/).
Options include:

- *Serverless*: Just run workloads on third-party equipment without managing
  a cluster at all. You will be charged for things like CPU usage, memory, and
  disk requests.
- *Managed control plane*: Let the provider manage the scale and availability
  of the cluster's control plane, as well as handle patches and upgrades.
- *Managed worker nodes*: Configure pools of nodes to meet your needs,
  then the provider makes sure those nodes are available and ready to implement
  upgrades when needed.
- *Integration*: There are providers that integrate Kubernetes with other
  services you may need, such as storage, container registries, authentication
  methods, and development tools.

Whether you build a production Kubernetes cluster yourself or work with
partners, review the following sections to evaluate your needs as they relate
to your cluster’s *control plane*, *worker nodes*, *user access*, and
*workload resources*.

## Production cluster setup

In a production-quality Kubernetes cluster, the control plane manages the
cluster from services that can be spread across multiple computers
in different ways. Each worker node, however, represents a single entity that
is configured to run Kubernetes pods.

### Production control plane

The simplest Kubernetes cluster has the entire control plane and worker node
services running on the same machine. You can grow that environment by adding
worker nodes, as reflected in the diagram illustrated in
[Kubernetes Components](/docs/concepts/overview/components/).
If the cluster is meant to be available for a short period of time, or can be
discarded if something goes seriously wrong, this might meet your needs.

If you need a more permanent, highly available cluster, however, you should
consider ways of extending the control plane. By design, one-machine control
plane services running on a single machine are not highly available.
If keeping the cluster up and running
and ensuring that it can be repaired if something goes wrong is important,
consider these steps:

- *Choose deployment tools*: You can deploy a control plane using tools such
  as kubeadm, kops, and kubespray. See
  [Installing Kubernetes with deployment tools](/docs/setup/production-environment/tools/)
  to learn tips for production-quality deployments using each of those deployment
  methods. Different [Container Runtimes](/docs/setup/production-environment/container-runtimes/)
  are available to use with your deployments.
- *Manage certificates*: Secure communications between control plane services
  are implemented using certificates. Certificates are automatically generated
  during deployment or you can generate them using your own certificate authority.
  See [PKI certificates and requirements](/docs/setup/best-practices/certificates/) for details.
- *Configure load balancer for apiserver*: Configure a load balancer
  to distribute external API requests to the apiserver service instances running on different nodes. See
  [Create an External Load Balancer](/docs/tasks/access-application-cluster/create-external-load-balancer/)
  for details.
- *Separate and backup etcd service*: The etcd services can either run on the
  same machines as other control plane services or run on separate machines, for
  extra security and availability. Because etcd stores cluster configuration data,
  backing up the etcd database should be done regularly to ensure that you can
  repair that database if needed.
  See the [etcd FAQ](https://etcd.io/docs/v3.5/faq/) for details on configuring and using etcd.
  See [Operating etcd clusters for Kubernetes](/docs/tasks/administer-cluster/configure-upgrade-etcd/)
  and [Set up a High Availability etcd cluster with kubeadm](/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/)
  for details.
- *Create multiple control plane systems*: For high availability, the
  control plane should not be limited to a single machine. If the control plane
  services are run by an init service (such as systemd), each service should run on at
  least three machines. However, running control plane services as pods in
  Kubernetes ensures that the replicated number of services that you request
  will always be available.
  The scheduler should be fault tolerant,
  but not highly available. Some deployment tools set up [Raft](https://raft.github.io/)
  consensus algorithm to do leader election of Kubernetes services. If the
  primary goes away, another service elects itself and take over.
- *Span multiple zones*: If keeping your cluster available at all times is
  critical, consider creating a cluster that runs across multiple data centers,
  referred to as zones in cloud environments. Groups of zones are referred to as regions.
  By spreading a cluster across
  multiple zones in the same region, it can improve the chances that your
  cluster will continue to function even if one zone becomes unavailable.
  See [Running in multiple zones](/docs/setup/best-practices/multiple-zones/) for details.
- *Manage on-going features*: If you plan to keep your cluster over time,
  there are tasks you need to do to maintain its health and security. For example,
  if you installed with kubeadm, there are instructions to help you with
  [Certificate Management](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/)
  and [Upgrading kubeadm clusters](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/).
  See [Administer a Cluster](/docs/tasks/administer-cluster/)
  for a longer list of Kubernetes administrative tasks.

To learn about available options when you run control plane services, see
[kube-apiserver](/docs/reference/command-line-tools-reference/kube-apiserver/),
[kube-controller-manager](/docs/reference/command-line-tools-reference/kube-controller-manager/),
and [kube-scheduler](/docs/reference/command-line-tools-reference/kube-scheduler/)
component pages. For highly available control plane examples, see
[Options for Highly Available topology](/docs/setup/production-environment/tools/kubeadm/ha-topology/),
[Creating Highly Available clusters with kubeadm](/docs/setup/production-environment/tools/kubeadm/high-availability/),
and [Operating etcd clusters for Kubernetes](/docs/tasks/administer-cluster/configure-upgrade-etcd/).
See [Backing up an etcd cluster](/docs/tasks/administer-cluster/configure-upgrade-etcd/#backing-up-an-etcd-cluster)
for information on making an etcd backup plan.

### Production worker nodes

Production-quality workloads need to be resilient and anything they rely
on needs to be resilient (such as CoreDNS). Whether you manage your own
control plane or have a cloud provider do it for you, you still need to
consider how you want to manage your worker nodes (also referred to
simply as *nodes*).

- *Configure nodes*: Nodes can be physical or virtual machines. If you want to
  create and manage your own nodes, you can install a supported operating system,
  then add and run the appropriate
  [Node services](/docs/concepts/architecture/#node-components). Consider:
  - The demands of your workloads when you set up nodes by having appropriate memory, CPU, and disk speed and storage capacity available.
  - Whether generic computer systems will do or you have workloads that need GPU processors, Windows nodes, or VM isolation.
- *Validate nodes*: See [Valid node setup](/docs/setup/best-practices/node-conformance/)
  for information on how to ensure that a node meets the requirements to join
  a Kubernetes cluster.
- *Add nodes to the cluster*: If you are managing your own cluster you can
  add nodes by setting up your own machines and either adding them manually or
  having them register themselves to the cluster’s apiserver. See the
  [Nodes](/docs/concepts/architecture/nodes/) section for information on how to set up Kubernetes to add nodes in these ways.
- *Scale nodes*: Have a plan for expanding the capacity your cluster will
  eventually need. See [Considerations for large clusters](/docs/setup/best-practices/cluster-large/)
  to help determine how many nodes you need, based on the number of pods and
  containers you need to run. If you are managing nodes yourself, this can mean
  purchasing and installing your own physical equipment.
- *Autoscale nodes*: Read [Cluster Autoscaling](/docs/concepts/cluster-administration/cluster-autoscaling) to learn about the
  tools available to automatically manage your nodes and the capacity they
  provide.
- *Set up node health checks*: For important workloads, you want to make sure
  that the nodes and pods running on those nodes are healthy. Using the
  [Node Problem Detector](/docs/tasks/debug/debug-cluster/monitor-node-health/)
  daemon, you can ensure your nodes are healthy.

## Production user management

In production, you may be moving from a model where you or a small group of
people are accessing the cluster to where there may potentially be dozens or
hundreds of people. In a learning environment or platform prototype, you might have a single
administrative account for everything you do. In production, you will want
more accounts with different levels of access to different namespaces.

Taking on a production-quality cluster means deciding how you
want to selectively allow access by other users. In particular, you need to
select strategies for validating the identities of those who try to access your
cluster (authentication) and deciding if they have permissions to do what they
are asking (authorization):

- *Authentication*: The apiserver can authenticate users using client
  certificates, bearer tokens, an authenticating proxy, or HTTP basic auth.
  You can choose which authentication methods you want to use.
  Using plugins, the apiserver can leverage your organization’s existing
  authentication methods, such as LDAP or Kerberos. See
  [Authentication](/docs/reference/access-authn-authz/authentication/)
  for a description of these different methods of authenticating Kubernetes users.
- *Authorization*: When you set out to authorize your regular users, you will probably choose
  between RBAC and ABAC authorization. See [Authorization Overview](/docs/reference/access-authn-authz/authorization/)
  to review different modes for authorizing user accounts (as well as service account access to
  your cluster):
  - *Role-based access control* ([RBAC](/docs/reference/access-authn-authz/rbac/)): Lets you
    assign access to your cluster by allowing specific sets of permissions to authenticated users.
    Permissions can be assigned for a specific namespace (Role) or across the entire cluster
    (ClusterRole). Then using RoleBindings and ClusterRoleBindings, those permissions can be attached
    to particular users.
  - *Attribute-based access control* ([ABAC](/docs/reference/access-authn-authz/abac/)): Lets you
    create policies based on resource attributes in the cluster and will allow or deny access
    based on those attributes. Each line of a policy file identifies versioning properties (apiVersion
    and kind) and a map of spec properties to match the subject (user or group), resource property,
    non-resource property (/version or /apis), and readonly. See
    [Examples](/docs/reference/access-authn-authz/abac/#examples) for details.

As someone setting up authentication and authorization on your production Kubernetes cluster, here are some things to consider:

- *Set the authorization mode*: When the Kubernetes API server
  ([kube-apiserver](/docs/reference/command-line-tools-reference/kube-apiserver/))
  starts, the supported authentication modes must be set using the *--authorization-mode*
  flag. For example, that flag in the *kube-adminserver.yaml* file (in */etc/kubernetes/manifests*)
  could be set to Node,RBAC. This would allow Node and RBAC authorization for authenticated requests.
- *Create user certificates and role bindings (RBAC)*: If you are using RBAC
  authorization, users can create a CertificateSigningRequest (CSR) that can be
  signed by the cluster CA. Then you can bind Roles and ClusterRoles to each user.
  See [Certificate Signing Requests](/docs/reference/access-authn-authz/certificate-signing-requests/)
  for details.
- *Create policies that combine attributes (ABAC)*: If you are using ABAC
  authorization, you can assign combinations of attributes to form policies to
  authorize selected users or groups to access particular resources (such as a
  pod), namespace, or apiGroup. For more information, see
  [Examples](/docs/reference/access-authn-authz/abac/#examples).
- *Consider Admission Controllers*: Additional forms of authorization for
  requests that can come in through the API server include
  [Webhook Token Authentication](/docs/reference/access-authn-authz/authentication/#webhook-token-authentication).
  Webhooks and other special authorization types need to be enabled by adding
  [Admission Controllers](/docs/reference/access-authn-authz/admission-controllers/)
  to the API server.

## Set limits on workload resources

Demands from production workloads can cause pressure both inside and outside
of the Kubernetes control plane. Consider these items when setting up for the
needs of your cluster's workloads:

- *Set namespace limits*: Set per-namespace quotas on things like memory and CPU. See
  [Manage Memory, CPU, and API Resources](/docs/tasks/administer-cluster/manage-resources/)
  for details. You can also set
  [Hierarchical Namespaces](/blog/2020/08/14/introducing-hierarchical-namespaces/)
  for inheriting limits.
- *Prepare for DNS demand*: If you expect workloads to massively scale up,
  your DNS service must be ready to scale up as well. See
  [Autoscale the DNS service in a Cluster](/docs/tasks/administer-cluster/dns-horizontal-autoscaling/).
- *Create additional service accounts*: User accounts determine what users can
  do on a cluster, while a service account defines pod access within a particular
  namespace. By default, a pod takes on the default service account from its namespace.
  See [Managing Service Accounts](/docs/reference/access-authn-authz/service-accounts-admin/)
  for information on creating a new service account. For example, you might want to:
  - Add secrets that a pod could use to pull images from a particular container registry. See
    [Configure Service Accounts for Pods](/docs/tasks/configure-pod-container/configure-service-account/)
    for an example.
  - Assign RBAC permissions to a service account. See
    [ServiceAccount permissions](/docs/reference/access-authn-authz/rbac/#service-account-permissions)
    for details.

##

- Decide if you want to build your own production Kubernetes or obtain one from
  available [Turnkey Cloud Solutions](/docs/setup/production-environment/turnkey-solutions/)
  or [Kubernetes Partners](/partners/).
- If you choose to build your own cluster, plan how you want to
  handle [certificates](/docs/setup/best-practices/certificates/)
  and set up high availability for features such as
  [etcd](/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/)
  and the
  [API server](/docs/setup/production-environment/tools/kubeadm/ha-topology/).
- Choose from [kubeadm](/docs/setup/production-environment/tools/kubeadm/),
  [kops](https://kops.sigs.k8s.io/) or
  [Kubespray](https://kubespray.io/) deployment methods.
- Configure user management by determining your
  [Authentication](/docs/reference/access-authn-authz/authentication/) and
  [Authorization](/docs/reference/access-authn-authz/authorization/) methods.
- Prepare for application workloads by setting up
  [resource limits](/docs/tasks/administer-cluster/manage-resources/),
  [DNS autoscaling](/docs/tasks/administer-cluster/dns-horizontal-autoscaling/)
  and [service accounts](/docs/reference/access-authn-authz/service-accounts-admin/).

title   Production environment  description  Create a production quality Kubernetes cluster weight  30 no list  true          overview      A production quality Kubernetes cluster requires planning and preparation  If your Kubernetes cluster is to run critical workloads  it must be configured to be resilient  This page explains steps you can take to set up a production ready cluster  or to promote an existing cluster for production use  If you re already familiar with production setup and want the links  skip to  What s next   what s next         body         Production considerations  Typically  a production Kubernetes cluster environment has more requirements than a personal learning  development  or test environment Kubernetes  A production environment may require secure access by many users  consistent availability  and the resources to adapt to changing demands   As you decide where you want your production Kubernetes environment to live  on premises or in a cloud  and the amount of management you want to take on or hand to others  consider how your requirements for a Kubernetes cluster are influenced by the following issues      Availability   A single machine Kubernetes  learning environment   docs setup  learning environment    has a single point of failure  Creating a highly available cluster means considering      Separating the control plane from the worker nodes      Replicating the control plane components on multiple nodes      Load balancing traffic to the cluster s       Having enough worker nodes available  or able to quickly become available  as changing workloads warrant it      Scale   If you expect your production Kubernetes environment to receive a stable amount of   demand  you might be able to set up for the capacity you need and be done  However    if you expect demand to grow over time or change dramatically based on things like   season or special events  you need to plan how to scale to relieve increased   pressure from more requests to the control plane and worker nodes or scale down to reduce unused   resources      Security and access management   You have full admin privileges on your own   Kubernetes learning cluster  But shared clusters with important workloads  and   more than one or two users  require a more refined approach to who and what can   access cluster resources  You can use role based access control     RBAC   docs reference access authn authz rbac    and other   security mechanisms to make sure that users and workloads can get access to the   resources they need  while keeping workloads  and the cluster itself  secure    You can set limits on the resources that users and workloads can access   by managing  policies   docs concepts policy   and    container resources   docs concepts configuration manage resources containers     Before building a Kubernetes production environment on your own  consider handing off some or all of this job to   Turnkey Cloud Solutions   docs setup production environment turnkey solutions    providers or other  Kubernetes Partners   partners    Options include      Serverless   Just run workloads on third party equipment without managing   a cluster at all  You will be charged for things like CPU usage  memory  and   disk requests     Managed control plane   Let the provider manage the scale and availability   of the cluster s control plane  as well as handle patches and upgrades     Managed worker nodes   Configure pools of nodes to meet your needs    then the provider makes sure those nodes are available and ready to implement   upgrades when needed     Integration   There are providers that integrate Kubernetes with other   services you may need  such as storage  container registries  authentication   methods  and development tools   Whether you build a production Kubernetes cluster yourself or work with partners  review the following sections to evaluate your needs as they relate to your cluster s  control plane    worker nodes    user access   and  workload resources       Production cluster setup  In a production quality Kubernetes cluster  the control plane manages the cluster from services that can be spread across multiple computers in different ways  Each worker node  however  represents a single entity that is configured to run Kubernetes pods       Production control plane  The simplest Kubernetes cluster has the entire control plane and worker node services running on the same machine  You can grow that environment by adding worker nodes  as reflected in the diagram illustrated in  Kubernetes Components   docs concepts overview components    If the cluster is meant to be available for a short period of time  or can be discarded if something goes seriously wrong  this might meet your needs   If you need a more permanent  highly available cluster  however  you should consider ways of extending the control plane  By design  one machine control plane services running on a single machine are not highly available  If keeping the cluster up and running and ensuring that it can be repaired if something goes wrong is important  consider these steps      Choose deployment tools   You can deploy a control plane using tools such   as kubeadm  kops  and kubespray  See    Installing Kubernetes with deployment tools   docs setup production environment tools     to learn tips for production quality deployments using each of those deployment   methods  Different  Container Runtimes   docs setup production environment container runtimes     are available to use with your deployments     Manage certificates   Secure communications between control plane services   are implemented using certificates  Certificates are automatically generated   during deployment or you can generate them using your own certificate authority    See  PKI certificates and requirements   docs setup best practices certificates   for details     Configure load balancer for apiserver   Configure a load balancer   to distribute external API requests to the apiserver service instances running on different nodes  See     Create an External Load Balancer   docs tasks access application cluster create external load balancer     for details     Separate and backup etcd service   The etcd services can either run on the   same machines as other control plane services or run on separate machines  for   extra security and availability  Because etcd stores cluster configuration data    backing up the etcd database should be done regularly to ensure that you can   repair that database if needed    See the  etcd FAQ  https   etcd io docs v3 5 faq   for details on configuring and using etcd    See  Operating etcd clusters for Kubernetes   docs tasks administer cluster configure upgrade etcd     and  Set up a High Availability etcd cluster with kubeadm   docs setup production environment tools kubeadm setup ha etcd with kubeadm     for details     Create multiple control plane systems   For high availability  the   control plane should not be limited to a single machine  If the control plane   services are run by an init service  such as systemd   each service should run on at   least three machines  However  running control plane services as pods in   Kubernetes ensures that the replicated number of services that you request   will always be available    The scheduler should be fault tolerant    but not highly available  Some deployment tools set up  Raft  https   raft github io     consensus algorithm to do leader election of Kubernetes services  If the   primary goes away  another service elects itself and take over      Span multiple zones   If keeping your cluster available at all times is   critical  consider creating a cluster that runs across multiple data centers    referred to as zones in cloud environments  Groups of zones are referred to as regions    By spreading a cluster across   multiple zones in the same region  it can improve the chances that your   cluster will continue to function even if one zone becomes unavailable    See  Running in multiple zones   docs setup best practices multiple zones   for details     Manage on going features   If you plan to keep your cluster over time    there are tasks you need to do to maintain its health and security  For example    if you installed with kubeadm  there are instructions to help you with    Certificate Management   docs tasks administer cluster kubeadm kubeadm certs     and  Upgrading kubeadm clusters   docs tasks administer cluster kubeadm kubeadm upgrade      See  Administer a Cluster   docs tasks administer cluster     for a longer list of Kubernetes administrative tasks   To learn about available options when you run control plane services  see  kube apiserver   docs reference command line tools reference kube apiserver     kube controller manager   docs reference command line tools reference kube controller manager    and  kube scheduler   docs reference command line tools reference kube scheduler   component pages  For highly available control plane examples  see  Options for Highly Available topology   docs setup production environment tools kubeadm ha topology     Creating Highly Available clusters with kubeadm   docs setup production environment tools kubeadm high availability    and  Operating etcd clusters for Kubernetes   docs tasks administer cluster configure upgrade etcd    See  Backing up an etcd cluster   docs tasks administer cluster configure upgrade etcd  backing up an etcd cluster  for information on making an etcd backup plan       Production worker nodes  Production quality workloads need to be resilient and anything they rely on needs to be resilient  such as CoreDNS   Whether you manage your own control plane or have a cloud provider do it for you  you still need to consider how you want to manage your worker nodes  also referred to simply as  nodes          Configure nodes   Nodes can be physical or virtual machines  If you want to   create and manage your own nodes  you can install a supported operating system    then add and run the appropriate    Node services   docs concepts architecture  node components   Consider      The demands of your workloads when you set up nodes by having appropriate memory  CPU  and disk speed and storage capacity available      Whether generic computer systems will do or you have workloads that need GPU processors  Windows nodes  or VM isolation     Validate nodes   See  Valid node setup   docs setup best practices node conformance     for information on how to ensure that a node meets the requirements to join   a Kubernetes cluster     Add nodes to the cluster   If you are managing your own cluster you can   add nodes by setting up your own machines and either adding them manually or   having them register themselves to the cluster s apiserver  See the    Nodes   docs concepts architecture nodes   section for information on how to set up Kubernetes to add nodes in these ways     Scale nodes   Have a plan for expanding the capacity your cluster will   eventually need  See  Considerations for large clusters   docs setup best practices cluster large     to help determine how many nodes you need  based on the number of pods and   containers you need to run  If you are managing nodes yourself  this can mean   purchasing and installing your own physical equipment     Autoscale nodes   Read  Cluster Autoscaling   docs concepts cluster administration cluster autoscaling  to learn about the   tools available to automatically manage your nodes and the capacity they   provide     Set up node health checks   For important workloads  you want to make sure   that the nodes and pods running on those nodes are healthy  Using the    Node Problem Detector   docs tasks debug debug cluster monitor node health     daemon  you can ensure your nodes are healthy      Production user management  In production  you may be moving from a model where you or a small group of people are accessing the cluster to where there may potentially be dozens or hundreds of people  In a learning environment or platform prototype  you might have a single administrative account for everything you do  In production  you will want more accounts with different levels of access to different namespaces   Taking on a production quality cluster means deciding how you want to selectively allow access by other users  In particular  you need to select strategies for validating the identities of those who try to access your cluster  authentication  and deciding if they have permissions to do what they are asking  authorization       Authentication   The apiserver can authenticate users using client   certificates  bearer tokens  an authenticating proxy  or HTTP basic auth    You can choose which authentication methods you want to use    Using plugins  the apiserver can leverage your organization s existing   authentication methods  such as LDAP or Kerberos  See    Authentication   docs reference access authn authz authentication     for a description of these different methods of authenticating Kubernetes users     Authorization   When you set out to authorize your regular users  you will probably choose   between RBAC and ABAC authorization  See  Authorization Overview   docs reference access authn authz authorization     to review different modes for authorizing user accounts  as well as service account access to   your cluster        Role based access control    RBAC   docs reference access authn authz rbac     Lets you     assign access to your cluster by allowing specific sets of permissions to authenticated users      Permissions can be assigned for a specific namespace  Role  or across the entire cluster      ClusterRole   Then using RoleBindings and ClusterRoleBindings  those permissions can be attached     to particular users       Attribute based access control    ABAC   docs reference access authn authz abac     Lets you     create policies based on resource attributes in the cluster and will allow or deny access     based on those attributes  Each line of a policy file identifies versioning properties  apiVersion     and kind  and a map of spec properties to match the subject  user or group   resource property      non resource property   version or  apis   and readonly  See      Examples   docs reference access authn authz abac  examples  for details   As someone setting up authentication and authorization on your production Kubernetes cluster  here are some things to consider      Set the authorization mode   When the Kubernetes API server     kube apiserver   docs reference command line tools reference kube apiserver      starts  the supported authentication modes must be set using the    authorization mode    flag  For example  that flag in the  kube adminserver yaml  file  in   etc kubernetes manifests     could be set to Node RBAC  This would allow Node and RBAC authorization for authenticated requests     Create user certificates and role bindings  RBAC    If you are using RBAC   authorization  users can create a CertificateSigningRequest  CSR  that can be   signed by the cluster CA  Then you can bind Roles and ClusterRoles to each user    See  Certificate Signing Requests   docs reference access authn authz certificate signing requests     for details     Create policies that combine attributes  ABAC    If you are using ABAC   authorization  you can assign combinations of attributes to form policies to   authorize selected users or groups to access particular resources  such as a   pod   namespace  or apiGroup  For more information  see    Examples   docs reference access authn authz abac  examples      Consider Admission Controllers   Additional forms of authorization for   requests that can come in through the API server include    Webhook Token Authentication   docs reference access authn authz authentication  webhook token authentication     Webhooks and other special authorization types need to be enabled by adding    Admission Controllers   docs reference access authn authz admission controllers     to the API server      Set limits on workload resources  Demands from production workloads can cause pressure both inside and outside of the Kubernetes control plane  Consider these items when setting up for the needs of your cluster s workloads      Set namespace limits   Set per namespace quotas on things like memory and CPU  See    Manage Memory  CPU  and API Resources   docs tasks administer cluster manage resources     for details  You can also set    Hierarchical Namespaces   blog 2020 08 14 introducing hierarchical namespaces     for inheriting limits     Prepare for DNS demand   If you expect workloads to massively scale up    your DNS service must be ready to scale up as well  See    Autoscale the DNS service in a Cluster   docs tasks administer cluster dns horizontal autoscaling       Create additional service accounts   User accounts determine what users can   do on a cluster  while a service account defines pod access within a particular   namespace  By default  a pod takes on the default service account from its namespace    See  Managing Service Accounts   docs reference access authn authz service accounts admin     for information on creating a new service account  For example  you might want to      Add secrets that a pod could use to pull images from a particular container registry  See      Configure Service Accounts for Pods   docs tasks configure pod container configure service account       for an example      Assign RBAC permissions to a service account  See      ServiceAccount permissions   docs reference access authn authz rbac  service account permissions      for details          Decide if you want to build your own production Kubernetes or obtain one from   available  Turnkey Cloud Solutions   docs setup production environment turnkey solutions     or  Kubernetes Partners   partners      If you choose to build your own cluster  plan how you want to   handle  certificates   docs setup best practices certificates     and set up high availability for features such as    etcd   docs setup production environment tools kubeadm setup ha etcd with kubeadm     and the    API server   docs setup production environment tools kubeadm ha topology      Choose from  kubeadm   docs setup production environment tools kubeadm       kops  https   kops sigs k8s io   or    Kubespray  https   kubespray io   deployment methods    Configure user management by determining your    Authentication   docs reference access authn authz authentication   and    Authorization   docs reference access authn authz authorization   methods    Prepare for application workloads by setting up    resource limits   docs tasks administer cluster manage resources       DNS autoscaling   docs tasks administer cluster dns horizontal autoscaling     and  service accounts   docs reference access authn authz service accounts admin


kubernetes setup weight 20 title Troubleshooting kubeadm contenttype concept As with any program you might run into an error installing or running kubeadm This page lists some common failure scenarios and have provided steps that can help you understand and fix the problem overview

Answer:
---
title: Troubleshooting kubeadm
content_type: concept
weight: 20
---

<!-- overview -->

As with any program, you might run into an error installing or running kubeadm.
This page lists some common failure scenarios and have provided steps that can help you understand and fix the problem.

If your problem is not listed below, please follow the following steps:

- If you think your problem is a bug with kubeadm:
  - Go to [github.com/kubernetes/kubeadm](https://github.com/kubernetes/kubeadm/issues) and search for existing issues.
  - If no issue exists, please [open one](https://github.com/kubernetes/kubeadm/issues/new) and follow the issue template.

- If you are unsure about how kubeadm works, you can ask on [Slack](https://slack.k8s.io/) in `#kubeadm`,
  or open a question on [StackOverflow](https://stackoverflow.com/questions/tagged/kubernetes). Please include
  relevant tags like `#kubernetes` and `#kubeadm` so folks can help you.

<!-- body -->

## Not possible to join a v1.18 Node to a v1.17 cluster due to missing RBAC

In v1.18 kubeadm added prevention for joining a Node in the cluster if a Node with the same name already exists.
This required adding RBAC for the bootstrap-token user to be able to GET a Node object.

However this causes an issue where `kubeadm join` from v1.18 cannot join a cluster created by kubeadm v1.17.

To workaround the issue you have two options:

Execute `kubeadm init phase bootstrap-token` on a control-plane node using kubeadm v1.18.
Note that this enables the rest of the bootstrap-token permissions as well.

or

Apply the following RBAC manually using `kubectl apply -f ...`:

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubeadm:get-nodes
rules:
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadm:get-nodes
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubeadm:get-nodes
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: Group
    name: system:bootstrappers:kubeadm:default-node-token
```

## `ebtables` or some similar executable not found during installation

If you see the following warnings while running `kubeadm init`

```console
[preflight] WARNING: ebtables not found in system path
[preflight] WARNING: ethtool not found in system path
```

Then you may be missing `ebtables`, `ethtool` or a similar executable on your node.
You can install them with the following commands:

- For Ubuntu/Debian users, run `apt install ebtables ethtool`.
- For CentOS/Fedora users, run `yum install ebtables ethtool`.

## kubeadm blocks waiting for control plane during installation

If you notice that `kubeadm init` hangs after printing out the following line:

```console
[apiclient] Created API client, waiting for the control plane to become ready
```

This may be caused by a number of problems. The most common are:

- network connection problems. Check that your machine has full network connectivity before continuing.
- the cgroup driver of the container runtime differs from that of the kubelet. To understand how to
  configure it properly, see [Configuring a cgroup driver](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/).
- control plane containers are crashlooping or hanging. You can check this by running `docker ps`
  and investigating each container by running `docker logs`. For other container runtime, see
  [Debugging Kubernetes nodes with crictl](/docs/tasks/debug/debug-cluster/crictl/).

## kubeadm blocks when removing managed containers

The following could happen if the container runtime halts and does not remove
any Kubernetes-managed containers:

```shell
sudo kubeadm reset
```

```console
[preflight] Running pre-flight checks
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Removing kubernetes-managed containers
(block)
```

A possible solution is to restart the container runtime and then re-run `kubeadm reset`.
You can also use `crictl` to debug the state of the container runtime. See
[Debugging Kubernetes nodes with crictl](/docs/tasks/debug/debug-cluster/crictl/).

## Pods in `RunContainerError`, `CrashLoopBackOff` or `Error` state

Right after `kubeadm init` there should not be any pods in these states.

- If there are pods in one of these states _right after_ `kubeadm init`, please open an
  issue in the kubeadm repo. `coredns` (or `kube-dns`) should be in the `Pending` state
  until you have deployed the network add-on.
- If you see Pods in the `RunContainerError`, `CrashLoopBackOff` or `Error` state
  after deploying the network add-on and nothing happens to `coredns` (or `kube-dns`),
  it's very likely that the Pod Network add-on that you installed is somehow broken.
  You might have to grant it more RBAC privileges or use a newer version. Please file
  an issue in the Pod Network providers' issue tracker and get the issue triaged there.

## `coredns` is stuck in the `Pending` state

This is **expected** and part of the design. kubeadm is network provider-agnostic, so the admin
should [install the pod network add-on](/docs/concepts/cluster-administration/addons/)
of choice. You have to install a Pod Network
before CoreDNS may be deployed fully. Hence the `Pending` state before the network is set up.

## `HostPort` services do not work

The `HostPort` and `HostIP` functionality is available depending on your Pod Network
provider. Please contact the author of the Pod Network add-on to find out whether
`HostPort` and `HostIP` functionality are available.

Calico, Canal, and Flannel CNI providers are verified to support HostPort.

For more information, see the
[CNI portmap documentation](https://github.com/containernetworking/plugins/blob/master/plugins/meta/portmap/README.md).

If your network provider does not support the portmap CNI plugin, you may need to use the
[NodePort feature of services](/docs/concepts/services-networking/service/#type-nodeport)
or use `HostNetwork=true`.

## Pods are not accessible via their Service IP

- Many network add-ons do not yet enable [hairpin mode](/docs/tasks/debug/debug-application/debug-service/#a-pod-fails-to-reach-itself-via-the-service-ip)
  which allows pods to access themselves via their Service IP. This is an issue related to
  [CNI](https://github.com/containernetworking/cni/issues/476). Please contact the network
  add-on provider to get the latest status of their support for hairpin mode.

- If you are using VirtualBox (directly or via Vagrant), you will need to
  ensure that `hostname -i` returns a routable IP address. By default, the first
  interface is connected to a non-routable host-only network. A work around
  is to modify `/etc/hosts`, see this
  [Vagrantfile](https://github.com/errordeveloper/k8s-playground/blob/22dd39dfc06111235620e6c4404a96ae146f26fd/Vagrantfile#L11)
  for an example.

## TLS certificate errors

The following error indicates a possible certificate mismatch.

```none
# kubectl get pods
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
```

- Verify that the `$HOME/.kube/config` file contains a valid certificate, and
  regenerate a certificate if necessary. The certificates in a kubeconfig file
  are base64 encoded. The `base64 --decode` command can be used to decode the certificate
  and `openssl x509 -text -noout` can be used for viewing the certificate information.

- Unset the `KUBECONFIG` environment variable using:

  ```sh
  unset KUBECONFIG
  ```

  Or set it to the default `KUBECONFIG` location:

  ```sh
  export KUBECONFIG=/etc/kubernetes/admin.conf
  ```

- Another workaround is to overwrite the existing `kubeconfig` for the "admin" user:

  ```sh
  mv $HOME/.kube $HOME/.kube.bak
  mkdir $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
  ```

## Kubelet client certificate rotation fails {#kubelet-client-cert}

By default, kubeadm configures a kubelet with automatic rotation of client certificates by using the
`/var/lib/kubelet/pki/kubelet-client-current.pem` symlink specified in `/etc/kubernetes/kubelet.conf`.
If this rotation process fails you might see errors such as `x509: certificate has expired or is not yet valid`
in kube-apiserver logs. To fix the issue you must follow these steps:

1. Backup and delete `/etc/kubernetes/kubelet.conf` and `/var/lib/kubelet/pki/kubelet-client*` from the failed node.
1. From a working control plane node in the cluster that has `/etc/kubernetes/pki/ca.key` execute
   `kubeadm kubeconfig user --org system:nodes --client-name system:node:$NODE > kubelet.conf`.
   `$NODE` must be set to the name of the existing failed node in the cluster.
   Modify the resulted `kubelet.conf` manually to adjust the cluster name and server endpoint,
   or pass `kubeconfig user --config` (see [Generating kubeconfig files for additional users](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#kubeconfig-additional-users)). If your cluster does not have
   the `ca.key` you must sign the embedded certificates in the `kubelet.conf` externally.
1. Copy this resulted `kubelet.conf` to `/etc/kubernetes/kubelet.conf` on the failed node.
1. Restart the kubelet (`systemctl restart kubelet`) on the failed node and wait for
   `/var/lib/kubelet/pki/kubelet-client-current.pem` to be recreated.
1. Manually edit the `kubelet.conf` to point to the rotated kubelet client certificates, by replacing
   `client-certificate-data` and `client-key-data` with:

   ```yaml
   client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
   client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
   ```

1. Restart the kubelet.
1. Make sure the node becomes `Ready`.

## Default NIC When using flannel as the pod network in Vagrant

The following error might indicate that something was wrong in the pod network:

```sh
Error from server (NotFound): the server could not find the requested resource
```

- If you're using flannel as the pod network inside Vagrant, then you will have to
  specify the default interface name for flannel.

  Vagrant typically assigns two interfaces to all VMs. The first, for which all hosts
  are assigned the IP address `10.0.2.15`, is for external traffic that gets NATed.

  This may lead to problems with flannel, which defaults to the first interface on a host.
  This leads to all hosts thinking they have the same public IP address. To prevent this,
  pass the `--iface eth1` flag to flannel so that the second interface is chosen.

## Non-public IP used for containers

In some situations `kubectl logs` and `kubectl run` commands may return with the
following errors in an otherwise functional cluster:

```console
Error from server: Get https://10.19.0.41:10250/containerLogs/default/mysql-ddc65b868-glc5m/mysql: dial tcp 10.19.0.41:10250: getsockopt: no route to host
```

- This may be due to Kubernetes using an IP that can not communicate with other IPs on
  the seemingly same subnet, possibly by policy of the machine provider.
- DigitalOcean assigns a public IP to `eth0` as well as a private one to be used internally
  as anchor for their floating IP feature, yet `kubelet` will pick the latter as the node's
  `InternalIP` instead of the public one.

  Use `ip addr show` to check for this scenario instead of `ifconfig` because `ifconfig` will
  not display the offending alias IP address. Alternatively an API endpoint specific to
  DigitalOcean allows to query for the anchor IP from the droplet:

  ```sh
  curl http://169.254.169.254/metadata/v1/interfaces/public/0/anchor_ipv4/address
  ```

  The workaround is to tell `kubelet` which IP to use using `--node-ip`.
  When using DigitalOcean, it can be the public one (assigned to `eth0`) or
  the private one (assigned to `eth1`) should you want to use the optional
  private network. The `kubeletExtraArgs` section of the kubeadm
  [`NodeRegistrationOptions` structure](/docs/reference/config-api/kubeadm-config.v1beta4/#kubeadm-k8s-io-v1beta4-NodeRegistrationOptions)
  can be used for this.

  Then restart `kubelet`:

  ```sh
  systemctl daemon-reload
  systemctl restart kubelet
  ```

## `coredns` pods have `CrashLoopBackOff` or `Error` state

If you have nodes that are running SELinux with an older version of Docker, you might experience a scenario
where the `coredns` pods are not starting. To solve that, you can try one of the following options:

- Upgrade to a [newer version of Docker](/docs/setup/production-environment/container-runtimes/#docker).

- [Disable SELinux](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security-enhanced_linux/sect-security-enhanced_linux-enabling_and_disabling_selinux-disabling_selinux).

- Modify the `coredns` deployment to set `allowPrivilegeEscalation` to `true`:

```bash
kubectl -n kube-system get deployment coredns -o yaml | \
  sed 's/allowPrivilegeEscalation: false/allowPrivilegeEscalation: true/g' | \
  kubectl apply -f -
```

Another cause for CoreDNS to have `CrashLoopBackOff` is when a CoreDNS Pod deployed in Kubernetes detects a loop.
[A number of workarounds](https://github.com/coredns/coredns/tree/master/plugin/loop#troubleshooting-loops-in-kubernetes-clusters)
are available to avoid Kubernetes trying to restart the CoreDNS Pod every time CoreDNS detects the loop and exits.

Disabling SELinux or setting `allowPrivilegeEscalation` to `true` can compromise
the security of your cluster.

## etcd pods restart continually

If you encounter the following error:

```
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:110: decoding init error from pipe caused \"read parent: connection reset by peer\""
```

This issue appears if you run CentOS 7 with Docker 1.13.1.84.
This version of Docker can prevent the kubelet from executing into the etcd container.

To work around the issue, choose one of these options:

- Roll back to an earlier version of Docker, such as 1.13.1-75

  ```
  yum downgrade docker-1.13.1-75.git8633870.el7.centos.x86_64 docker-client-1.13.1-75.git8633870.el7.centos.x86_64 docker-common-1.13.1-75.git8633870.el7.centos.x86_64
  ```

- Install one of the more recent recommended versions, such as 18.06:

  ```bash
  sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
  yum install docker-ce-18.06.1.ce-3.el7.x86_64
  ```

## Not possible to pass a comma separated list of values to arguments inside a `--component-extra-args` flag

`kubeadm init` flags such as `--component-extra-args` allow you to pass custom arguments to a control-plane
component like the kube-apiserver. However, this mechanism is limited due to the underlying type used for parsing
the values (`mapStringString`).

If you decide to pass an argument that supports multiple, comma-separated values such as
`--apiserver-extra-args "enable-admission-plugins=LimitRanger,NamespaceExists"` this flag will fail with
`flag: malformed pair, expect string=string`. This happens because the list of arguments for
`--apiserver-extra-args` expects `key=value` pairs and in this case `NamespacesExists` is considered
as a key that is missing a value.

Alternatively, you can try separating the `key=value` pairs like so:
`--apiserver-extra-args "enable-admission-plugins=LimitRanger,enable-admission-plugins=NamespaceExists"`
but this will result in the key `enable-admission-plugins` only having the value of `NamespaceExists`.

A known workaround is to use the kubeadm [configuration file](/docs/reference/config-api/kubeadm-config.v1beta4/).

## kube-proxy scheduled before node is initialized by cloud-controller-manager

In cloud provider scenarios, kube-proxy can end up being scheduled on new worker nodes before
the cloud-controller-manager has initialized the node addresses. This causes kube-proxy to fail
to pick up the node's IP address properly and has knock-on effects to the proxy function managing
load balancers.

The following error can be seen in kube-proxy Pods:

```
server.go:610] Failed to retrieve node IP: host IP unknown; known addresses: []
proxier.go:340] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
```

A known solution is to patch the kube-proxy DaemonSet to allow scheduling it on control-plane
nodes regardless of their conditions, keeping it off of other nodes until their initial guarding
conditions abate:

```
kubectl -n kube-system patch ds kube-proxy -p='{
  "spec": {
    "template": {
      "spec": {
        "tolerations": [
          {
            "key": "CriticalAddonsOnly",
            "operator": "Exists"
          },
          {
            "effect": "NoSchedule",
            "key": "node-role.kubernetes.io/control-plane"
          }
        ]
      }
    }
  }
}'
```

The tracking issue for this problem is [here](https://github.com/kubernetes/kubeadm/issues/1027).

## `/usr` is mounted read-only on nodes {#usr-mounted-read-only}

On Linux distributions such as Fedora CoreOS or Flatcar Container Linux, the directory `/usr` is mounted as a read-only filesystem.
For [flex-volume support](https://github.com/kubernetes/community/blob/ab55d85/contributors/devel/sig-storage/flexvolume.md),
Kubernetes components like the kubelet and kube-controller-manager use the default path of
`/usr/libexec/kubernetes/kubelet-plugins/volume/exec/`, yet the flex-volume directory _must be writeable_
for the feature to work.

FlexVolume was deprecated in the Kubernetes v1.23 release.

To workaround this issue, you can configure the flex-volume directory using the kubeadm
[configuration file](/docs/reference/config-api/kubeadm-config.v1beta4/).

On the primary control-plane Node (created using `kubeadm init`), pass the following
file using `--config`:

```yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
nodeRegistration:
  kubeletExtraArgs:
  - name: "volume-plugin-dir"
    value: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
controllerManager:
  extraArgs:
  - name: "flex-volume-plugin-dir"
    value: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
```

On joining Nodes:

```yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: JoinConfiguration
nodeRegistration:
  kubeletExtraArgs:
  - name: "volume-plugin-dir"
    value: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"
```

Alternatively, you can modify `/etc/fstab` to make the `/usr` mount writeable, but please
be advised that this is modifying a design principle of the Linux distribution.

## `kubeadm upgrade plan` prints out `context deadline exceeded` error message

This error message is shown when upgrading a Kubernetes cluster with `kubeadm` in
the case of running an external etcd. This is not a critical bug and happens because
older versions of kubeadm perform a version check on the external etcd cluster.
You can proceed with `kubeadm upgrade apply ...`.

This issue is fixed as of version 1.19.

## `kubeadm reset` unmounts `/var/lib/kubelet`

If `/var/lib/kubelet` is being mounted, performing a `kubeadm reset` will effectively unmount it.

To workaround the issue, re-mount the `/var/lib/kubelet` directory after performing the `kubeadm reset` operation.

This is a regression introduced in kubeadm 1.15. The issue is fixed in 1.20.

## Cannot use the metrics-server securely in a kubeadm cluster

In a kubeadm cluster, the [metrics-server](https://github.com/kubernetes-sigs/metrics-server)
can be used insecurely by passing the `--kubelet-insecure-tls` to it. This is not recommended for production clusters.

If you want to use TLS between the metrics-server and the kubelet there is a problem,
since kubeadm deploys a self-signed serving certificate for the kubelet. This can cause the following errors
on the side of the metrics-server:

```
x509: certificate signed by unknown authority
x509: certificate is valid for IP-foo not IP-bar
```

See [Enabling signed kubelet serving certificates](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#kubelet-serving-certs)
to understand how to configure the kubelets in a kubeadm cluster to have properly signed serving certificates.

Also see [How to run the metrics-server securely](https://github.com/kubernetes-sigs/metrics-server/blob/master/FAQ.md#how-to-run-metrics-server-securely).

## Upgrade fails due to etcd hash not changing

Only applicable to upgrading a control plane node with a kubeadm binary v1.28.3 or later,
where the node is currently managed by kubeadm versions v1.28.0, v1.28.1 or v1.28.2.

Here is the error message you may encounter:

```
[upgrade/etcd] Failed to upgrade etcd: couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced: static Pod hash for component etcd on Node kinder-upgrade-control-plane-1 did not change after 5m0s: timed out waiting for the condition
[upgrade/etcd] Waiting for previous etcd to become available
I0907 10:10:09.109104    3704 etcd.go:588] [etcd] attempting to see if all cluster endpoints ([https://172.17.0.6:2379/ https://172.17.0.4:2379/ https://172.17.0.3:2379/]) are available 1/10
[upgrade/etcd] Etcd was rolled back and is now available
static Pod hash for component etcd on Node kinder-upgrade-control-plane-1 did not change after 5m0s: timed out waiting for the condition
couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced
k8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade.rollbackOldManifests
	cmd/kubeadm/app/phases/upgrade/staticpods.go:525
k8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade.upgradeComponent
	cmd/kubeadm/app/phases/upgrade/staticpods.go:254
k8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade.performEtcdStaticPodUpgrade
	cmd/kubeadm/app/phases/upgrade/staticpods.go:338
...
```

The reason for this failure is that the affected versions generate an etcd manifest file with
unwanted defaults in the PodSpec. This will result in a diff from the manifest comparison,
and kubeadm will expect a change in the Pod hash, but the kubelet will never update the hash.

There are two way to workaround this issue if you see it in your cluster:

- The etcd upgrade can be skipped between the affected versions and v1.28.3 (or later) by using:

  ```shell
  kubeadm upgrade {apply|node} [version] --etcd-upgrade=false
  ```

  This is not recommended in case a new etcd version was introduced by a later v1.28 patch version.

- Before upgrade, patch the manifest for the etcd static pod, to remove the problematic defaulted attributes:

  ```patch
  diff --git a/etc/kubernetes/manifests/etcd_defaults.yaml b/etc/kubernetes/manifests/etcd_origin.yaml
  index d807ccbe0aa..46b35f00e15 100644
  --- a/etc/kubernetes/manifests/etcd_defaults.yaml
  +++ b/etc/kubernetes/manifests/etcd_origin.yaml
  @@ -43,7 +43,6 @@ spec:
          scheme: HTTP
        initialDelaySeconds: 10
        periodSeconds: 10
  -      successThreshold: 1
        timeoutSeconds: 15
      name: etcd
      resources:
  @@ -59,26 +58,18 @@ spec:
          scheme: HTTP
        initialDelaySeconds: 10
        periodSeconds: 10
  -      successThreshold: 1
        timeoutSeconds: 15
  -    terminationMessagePath: /dev/termination-log
  -    terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /var/lib/etcd
        name: etcd-data
      - mountPath: /etc/kubernetes/pki/etcd
        name: etcd-certs
  -  dnsPolicy: ClusterFirst
  -  enableServiceLinks: true
    hostNetwork: true
    priority: 2000001000
    priorityClassName: system-node-critical
  -  restartPolicy: Always
  -  schedulerName: default-scheduler
    securityContext:
      seccompProfile:
        type: RuntimeDefault
  -  terminationGracePeriodSeconds: 30
    volumes:
    - hostPath:
        path: /etc/kubernetes/pki/etcd
  ```

More information can be found in the
[tracking issue](https://github.com/kubernetes/kubeadm/issues/2927) for this bug.

title  Troubleshooting kubeadm content type  concept weight  20           overview      As with any program  you might run into an error installing or running kubeadm  This page lists some common failure scenarios and have provided steps that can help you understand and fix the problem   If your problem is not listed below  please follow the following steps     If you think your problem is a bug with kubeadm      Go to  github com kubernetes kubeadm  https   github com kubernetes kubeadm issues  and search for existing issues      If no issue exists  please  open one  https   github com kubernetes kubeadm issues new  and follow the issue template     If you are unsure about how kubeadm works  you can ask on  Slack  https   slack k8s io   in   kubeadm     or open a question on  StackOverflow  https   stackoverflow com questions tagged kubernetes   Please include   relevant tags like   kubernetes  and   kubeadm  so folks can help you        body         Not possible to join a v1 18 Node to a v1 17 cluster due to missing RBAC  In v1 18 kubeadm added prevention for joining a Node in the cluster if a Node with the same name already exists  This required adding RBAC for the bootstrap token user to be able to GET a Node object   However this causes an issue where  kubeadm join  from v1 18 cannot join a cluster created by kubeadm v1 17   To workaround the issue you have two options   Execute  kubeadm init phase bootstrap token  on a control plane node using kubeadm v1 18  Note that this enables the rest of the bootstrap token permissions as well   or  Apply the following RBAC manually using  kubectl apply  f           yaml apiVersion  rbac authorization k8s io v1 kind  ClusterRole metadata    name  kubeadm get nodes rules      apiGroups                 resources          nodes     verbs          get     apiVersion  rbac authorization k8s io v1 kind  ClusterRoleBinding metadata    name  kubeadm get nodes roleRef    apiGroup  rbac authorization k8s io   kind  ClusterRole   name  kubeadm get nodes subjects      apiGroup  rbac authorization k8s io     kind  Group     name  system bootstrappers kubeadm default node token          ebtables  or some similar executable not found during installation  If you see the following warnings while running  kubeadm init      console  preflight  WARNING  ebtables not found in system path  preflight  WARNING  ethtool not found in system path      Then you may be missing  ebtables    ethtool  or a similar executable on your node  You can install them with the following commands     For Ubuntu Debian users  run  apt install ebtables ethtool     For CentOS Fedora users  run  yum install ebtables ethtool       kubeadm blocks waiting for control plane during installation  If you notice that  kubeadm init  hangs after printing out the following line      console  apiclient  Created API client  waiting for the control plane to become ready      This may be caused by a number of problems  The most common are     network connection problems  Check that your machine has full network connectivity before continuing    the cgroup driver of the container runtime differs from that of the kubelet  To understand how to   configure it properly  see  Configuring a cgroup driver   docs tasks administer cluster kubeadm configure cgroup driver      control plane containers are crashlooping or hanging  You can check this by running  docker ps    and investigating each container by running  docker logs   For other container runtime  see    Debugging Kubernetes nodes with crictl   docs tasks debug debug cluster crictl        kubeadm blocks when removing managed containers  The following could happen if the container runtime halts and does not remove any Kubernetes managed containers      shell sudo kubeadm reset         console  preflight  Running pre flight checks  reset  Stopping the kubelet service  reset  Unmounting mounted directories in   var lib kubelet   reset  Removing kubernetes managed containers  block       A possible solution is to restart the container runtime and then re run  kubeadm reset   You can also use  crictl  to debug the state of the container runtime  See  Debugging Kubernetes nodes with crictl   docs tasks debug debug cluster crictl        Pods in  RunContainerError    CrashLoopBackOff  or  Error  state  Right after  kubeadm init  there should not be any pods in these states     If there are pods in one of these states  right after   kubeadm init   please open an   issue in the kubeadm repo   coredns   or  kube dns   should be in the  Pending  state   until you have deployed the network add on    If you see Pods in the  RunContainerError    CrashLoopBackOff  or  Error  state   after deploying the network add on and nothing happens to  coredns   or  kube dns      it s very likely that the Pod Network add on that you installed is somehow broken    You might have to grant it more RBAC privileges or use a newer version  Please file   an issue in the Pod Network providers  issue tracker and get the issue triaged there       coredns  is stuck in the  Pending  state  This is   expected   and part of the design  kubeadm is network provider agnostic  so the admin should  install the pod network add on   docs concepts cluster administration addons   of choice  You have to install a Pod Network before CoreDNS may be deployed fully  Hence the  Pending  state before the network is set up       HostPort  services do not work  The  HostPort  and  HostIP  functionality is available depending on your Pod Network provider  Please contact the author of the Pod Network add on to find out whether  HostPort  and  HostIP  functionality are available   Calico  Canal  and Flannel CNI providers are verified to support HostPort   For more information  see the  CNI portmap documentation  https   github com containernetworking plugins blob master plugins meta portmap README md    If your network provider does not support the portmap CNI plugin  you may need to use the  NodePort feature of services   docs concepts services networking service  type nodeport  or use  HostNetwork true       Pods are not accessible via their Service IP    Many network add ons do not yet enable  hairpin mode   docs tasks debug debug application debug service  a pod fails to reach itself via the service ip    which allows pods to access themselves via their Service IP  This is an issue related to    CNI  https   github com containernetworking cni issues 476   Please contact the network   add on provider to get the latest status of their support for hairpin mode     If you are using VirtualBox  directly or via Vagrant   you will need to   ensure that  hostname  i  returns a routable IP address  By default  the first   interface is connected to a non routable host only network  A work around   is to modify   etc hosts   see this    Vagrantfile  https   github com errordeveloper k8s playground blob 22dd39dfc06111235620e6c4404a96ae146f26fd Vagrantfile L11    for an example      TLS certificate errors  The following error indicates a possible certificate mismatch      none   kubectl get pods Unable to connect to the server  x509  certificate signed by unknown authority  possibly because of  crypto rsa  verification error  while trying to verify candidate authority certificate  kubernetes          Verify that the   HOME  kube config  file contains a valid certificate  and   regenerate a certificate if necessary  The certificates in a kubeconfig file   are base64 encoded  The  base64   decode  command can be used to decode the certificate   and  openssl x509  text  noout  can be used for viewing the certificate information     Unset the  KUBECONFIG  environment variable using        sh   unset KUBECONFIG          Or set it to the default  KUBECONFIG  location        sh   export KUBECONFIG  etc kubernetes admin conf          Another workaround is to overwrite the existing  kubeconfig  for the  admin  user        sh   mv  HOME  kube  HOME  kube bak   mkdir  HOME  kube   sudo cp  i  etc kubernetes admin conf  HOME  kube config   sudo chown   id  u    id  g   HOME  kube config           Kubelet client certificate rotation fails   kubelet client cert   By default  kubeadm configures a kubelet with automatic rotation of client certificates by using the   var lib kubelet pki kubelet client current pem  symlink specified in   etc kubernetes kubelet conf   If this rotation process fails you might see errors such as  x509  certificate has expired or is not yet valid  in kube apiserver logs  To fix the issue you must follow these steps   1  Backup and delete   etc kubernetes kubelet conf  and   var lib kubelet pki kubelet client   from the failed node  1  From a working control plane node in the cluster that has   etc kubernetes pki ca key  execute     kubeadm kubeconfig user   org system nodes   client name system node  NODE   kubelet conf        NODE  must be set to the name of the existing failed node in the cluster     Modify the resulted  kubelet conf  manually to adjust the cluster name and server endpoint     or pass  kubeconfig user   config   see  Generating kubeconfig files for additional users   docs tasks administer cluster kubeadm kubeadm certs  kubeconfig additional users    If your cluster does not have    the  ca key  you must sign the embedded certificates in the  kubelet conf  externally  1  Copy this resulted  kubelet conf  to   etc kubernetes kubelet conf  on the failed node  1  Restart the kubelet   systemctl restart kubelet   on the failed node and wait for      var lib kubelet pki kubelet client current pem  to be recreated  1  Manually edit the  kubelet conf  to point to the rotated kubelet client certificates  by replacing     client certificate data  and  client key data  with         yaml    client certificate   var lib kubelet pki kubelet client current pem    client key   var lib kubelet pki kubelet client current pem         1  Restart the kubelet  1  Make sure the node becomes  Ready       Default NIC When using flannel as the pod network in Vagrant  The following error might indicate that something was wrong in the pod network      sh Error from server  NotFound   the server could not find the requested resource        If you re using flannel as the pod network inside Vagrant  then you will have to   specify the default interface name for flannel     Vagrant typically assigns two interfaces to all VMs  The first  for which all hosts   are assigned the IP address  10 0 2 15   is for external traffic that gets NATed     This may lead to problems with flannel  which defaults to the first interface on a host    This leads to all hosts thinking they have the same public IP address  To prevent this    pass the    iface eth1  flag to flannel so that the second interface is chosen      Non public IP used for containers  In some situations  kubectl logs  and  kubectl run  commands may return with the following errors in an otherwise functional cluster      console Error from server  Get https   10 19 0 41 10250 containerLogs default mysql ddc65b868 glc5m mysql  dial tcp 10 19 0 41 10250  getsockopt  no route to host        This may be due to Kubernetes using an IP that can not communicate with other IPs on   the seemingly same subnet  possibly by policy of the machine provider    DigitalOcean assigns a public IP to  eth0  as well as a private one to be used internally   as anchor for their floating IP feature  yet  kubelet  will pick the latter as the node s    InternalIP  instead of the public one     Use  ip addr show  to check for this scenario instead of  ifconfig  because  ifconfig  will   not display the offending alias IP address  Alternatively an API endpoint specific to   DigitalOcean allows to query for the anchor IP from the droplet        sh   curl http   169 254 169 254 metadata v1 interfaces public 0 anchor ipv4 address          The workaround is to tell  kubelet  which IP to use using    node ip     When using DigitalOcean  it can be the public one  assigned to  eth0   or   the private one  assigned to  eth1   should you want to use the optional   private network  The  kubeletExtraArgs  section of the kubeadm     NodeRegistrationOptions  structure   docs reference config api kubeadm config v1beta4  kubeadm k8s io v1beta4 NodeRegistrationOptions    can be used for this     Then restart  kubelet         sh   systemctl daemon reload   systemctl restart kubelet            coredns  pods have  CrashLoopBackOff  or  Error  state  If you have nodes that are running SELinux with an older version of Docker  you might experience a scenario where the  coredns  pods are not starting  To solve that  you can try one of the following options     Upgrade to a  newer version of Docker   docs setup production environment container runtimes  docker       Disable SELinux  https   access redhat com documentation en us red hat enterprise linux 6 html security enhanced linux sect security enhanced linux enabling and disabling selinux disabling selinux      Modify the  coredns  deployment to set  allowPrivilegeEscalation  to  true       bash kubectl  n kube system get deployment coredns  o yaml       sed  s allowPrivilegeEscalation  false allowPrivilegeEscalation  true g        kubectl apply  f        Another cause for CoreDNS to have  CrashLoopBackOff  is when a CoreDNS Pod deployed in Kubernetes detects a loop   A number of workarounds  https   github com coredns coredns tree master plugin loop troubleshooting loops in kubernetes clusters  are available to avoid Kubernetes trying to restart the CoreDNS Pod every time CoreDNS detects the loop and exits    Disabling SELinux or setting  allowPrivilegeEscalation  to  true  can compromise the security of your cluster       etcd pods restart continually  If you encounter the following error       rpc error  code   2 desc   oci runtime error  exec failed  container linux go 247  starting container process caused  process linux go 110  decoding init error from pipe caused   read parent  connection reset by peer         This issue appears if you run CentOS 7 with Docker 1 13 1 84  This version of Docker can prevent the kubelet from executing into the etcd container   To work around the issue  choose one of these options     Roll back to an earlier version of Docker  such as 1 13 1 75          yum downgrade docker 1 13 1 75 git8633870 el7 centos x86 64 docker client 1 13 1 75 git8633870 el7 centos x86 64 docker common 1 13 1 75 git8633870 el7 centos x86 64          Install one of the more recent recommended versions  such as 18 06        bash   sudo yum config manager   add repo https   download docker com linux centos docker ce repo   yum install docker ce 18 06 1 ce 3 el7 x86 64           Not possible to pass a comma separated list of values to arguments inside a    component extra args  flag   kubeadm init  flags such as    component extra args  allow you to pass custom arguments to a control plane component like the kube apiserver  However  this mechanism is limited due to the underlying type used for parsing the values   mapStringString     If you decide to pass an argument that supports multiple  comma separated values such as    apiserver extra args  enable admission plugins LimitRanger NamespaceExists   this flag will fail with  flag  malformed pair  expect string string   This happens because the list of arguments for    apiserver extra args  expects  key value  pairs and in this case  NamespacesExists  is considered as a key that is missing a value   Alternatively  you can try separating the  key value  pairs like so     apiserver extra args  enable admission plugins LimitRanger enable admission plugins NamespaceExists   but this will result in the key  enable admission plugins  only having the value of  NamespaceExists    A known workaround is to use the kubeadm  configuration file   docs reference config api kubeadm config v1beta4        kube proxy scheduled before node is initialized by cloud controller manager  In cloud provider scenarios  kube proxy can end up being scheduled on new worker nodes before the cloud controller manager has initialized the node addresses  This causes kube proxy to fail to pick up the node s IP address properly and has knock on effects to the proxy function managing load balancers   The following error can be seen in kube proxy Pods       server go 610  Failed to retrieve node IP  host IP unknown  known addresses     proxier go 340  invalid nodeIP  initializing kube proxy with 127 0 0 1 as nodeIP      A known solution is to patch the kube proxy DaemonSet to allow scheduling it on control plane nodes regardless of their conditions  keeping it off of other nodes until their initial guarding conditions abate       kubectl  n kube system patch ds kube proxy  p       spec          template            spec              tolerations                              key    CriticalAddonsOnly                operator    Exists                                        effect    NoSchedule                key    node role kubernetes io control plane                                                  The tracking issue for this problem is  here  https   github com kubernetes kubeadm issues 1027         usr  is mounted read only on nodes   usr mounted read only   On Linux distributions such as Fedora CoreOS or Flatcar Container Linux  the directory   usr  is mounted as a read only filesystem  For  flex volume support  https   github com kubernetes community blob ab55d85 contributors devel sig storage flexvolume md   Kubernetes components like the kubelet and kube controller manager use the default path of   usr libexec kubernetes kubelet plugins volume exec    yet the flex volume directory  must be writeable  for the feature to work    FlexVolume was deprecated in the Kubernetes v1 23 release    To workaround this issue  you can configure the flex volume directory using the kubeadm  configuration file   docs reference config api kubeadm config v1beta4     On the primary control plane Node  created using  kubeadm init    pass the following file using    config       yaml apiVersion  kubeadm k8s io v1beta4 kind  InitConfiguration nodeRegistration    kubeletExtraArgs      name   volume plugin dir      value    opt libexec kubernetes kubelet plugins volume exec       apiVersion  kubeadm k8s io v1beta4 kind  ClusterConfiguration controllerManager    extraArgs      name   flex volume plugin dir      value    opt libexec kubernetes kubelet plugins volume exec        On joining Nodes      yaml apiVersion  kubeadm k8s io v1beta4 kind  JoinConfiguration nodeRegistration    kubeletExtraArgs      name   volume plugin dir      value    opt libexec kubernetes kubelet plugins volume exec        Alternatively  you can modify   etc fstab  to make the   usr  mount writeable  but please be advised that this is modifying a design principle of the Linux distribution       kubeadm upgrade plan  prints out  context deadline exceeded  error message  This error message is shown when upgrading a Kubernetes cluster with  kubeadm  in the case of running an external etcd  This is not a critical bug and happens because older versions of kubeadm perform a version check on the external etcd cluster  You can proceed with  kubeadm upgrade apply        This issue is fixed as of version 1 19       kubeadm reset  unmounts   var lib kubelet   If   var lib kubelet  is being mounted  performing a  kubeadm reset  will effectively unmount it   To workaround the issue  re mount the   var lib kubelet  directory after performing the  kubeadm reset  operation   This is a regression introduced in kubeadm 1 15  The issue is fixed in 1 20      Cannot use the metrics server securely in a kubeadm cluster  In a kubeadm cluster  the  metrics server  https   github com kubernetes sigs metrics server  can be used insecurely by passing the    kubelet insecure tls  to it  This is not recommended for production clusters   If you want to use TLS between the metrics server and the kubelet there is a problem  since kubeadm deploys a self signed serving certificate for the kubelet  This can cause the following errors on the side of the metrics server       x509  certificate signed by unknown authority x509  certificate is valid for IP foo not IP bar      See  Enabling signed kubelet serving certificates   docs tasks administer cluster kubeadm kubeadm certs  kubelet serving certs  to understand how to configure the kubelets in a kubeadm cluster to have properly signed serving certificates   Also see  How to run the metrics server securely  https   github com kubernetes sigs metrics server blob master FAQ md how to run metrics server securely       Upgrade fails due to etcd hash not changing  Only applicable to upgrading a control plane node with a kubeadm binary v1 28 3 or later  where the node is currently managed by kubeadm versions v1 28 0  v1 28 1 or v1 28 2   Here is the error message you may encounter        upgrade etcd  Failed to upgrade etcd  couldn t upgrade control plane  kubeadm has tried to recover everything into the earlier state  Errors faced  static Pod hash for component etcd on Node kinder upgrade control plane 1 did not change after 5m0s  timed out waiting for the condition  upgrade etcd  Waiting for previous etcd to become available I0907 10 10 09 109104    3704 etcd go 588   etcd  attempting to see if all cluster endpoints   https   172 17 0 6 2379  https   172 17 0 4 2379  https   172 17 0 3 2379    are available 1 10  upgrade etcd  Etcd was rolled back and is now available static Pod hash for component etcd on Node kinder upgrade control plane 1 did not change after 5m0s  timed out waiting for the condition couldn t upgrade control plane  kubeadm has tried to recover everything into the earlier state  Errors faced k8s io kubernetes cmd kubeadm app phases upgrade rollbackOldManifests  cmd kubeadm app phases upgrade staticpods go 525 k8s io kubernetes cmd kubeadm app phases upgrade upgradeComponent  cmd kubeadm app phases upgrade staticpods go 254 k8s io kubernetes cmd kubeadm app phases upgrade performEtcdStaticPodUpgrade  cmd kubeadm app phases upgrade staticpods go 338          The reason for this failure is that the affected versions generate an etcd manifest file with unwanted defaults in the PodSpec  This will result in a diff from the manifest comparison  and kubeadm will expect a change in the Pod hash  but the kubelet will never update the hash   There are two way to workaround this issue if you see it in your cluster     The etcd upgrade can be skipped between the affected versions and v1 28 3  or later  by using        shell   kubeadm upgrade  apply node   version    etcd upgrade false          This is not recommended in case a new etcd version was introduced by a later v1 28 patch version     Before upgrade  patch the manifest for the etcd static pod  to remove the problematic defaulted attributes        patch   diff   git a etc kubernetes manifests etcd defaults yaml b etc kubernetes manifests etcd origin yaml   index d807ccbe0aa  46b35f00e15 100644       a etc kubernetes manifests etcd defaults yaml       b etc kubernetes manifests etcd origin yaml       43 7  43 6    spec            scheme  HTTP         initialDelaySeconds  10         periodSeconds  10          successThreshold  1         timeoutSeconds  15       name  etcd       resources        59 26  58 18    spec            scheme  HTTP         initialDelaySeconds  10         periodSeconds  10          successThreshold  1         timeoutSeconds  15        terminationMessagePath   dev termination log        terminationMessagePolicy  File       volumeMounts          mountPath   var lib etcd         name  etcd data         mountPath   etc kubernetes pki etcd         name  etcd certs      dnsPolicy  ClusterFirst      enableServiceLinks  true     hostNetwork  true     priority  2000001000     priorityClassName  system node critical      restartPolicy  Always      schedulerName  default scheduler     securityContext        seccompProfile          type  RuntimeDefault      terminationGracePeriodSeconds  30     volumes        hostPath          path   etc kubernetes pki etcd        More information can be found in the  tracking issue  https   github com kubernetes kubeadm issues 2927  for this bug