cilium Upgrade Guide docs cilium io adminupgrade upgradegeneral

Answer:
.. only:: not (epub or latex or html)

    WARNING: You are looking at unreleased Cilium documentation.
    Please use the official rendered version released here:
    https://docs.cilium.io

.. _admin_upgrade:

*************
Upgrade Guide
*************

.. _upgrade_general:

This upgrade guide is intended for Cilium running on Kubernetes. If you have
questions, feel free to ping us on `Cilium Slack`_.

.. include:: upgrade-warning.rst

.. _pre_flight:

Running pre-flight check (Required)
===================================

When rolling out an upgrade with Kubernetes, Kubernetes will first terminate the
pod followed by pulling the new image version and then finally spin up the new
image. In order to reduce the downtime of the agent and to prevent ``ErrImagePull``
errors during upgrade, the pre-flight check pre-pulls the new image version.
If you are running in :ref:`kubeproxy-free`
mode you must also pass on the Kubernetes API Server IP and /
or the Kubernetes API Server Port when generating the ``cilium-preflight.yaml``
file.

.. tabs::
  .. group-tab:: kubectl

    .. parsed-literal::

      helm template |CHART_RELEASE| \\
        --namespace=kube-system \\
        --set preflight.enabled=true \\
        --set agent=false \\
        --set operator.enabled=false \\
        > cilium-preflight.yaml
      kubectl create -f cilium-preflight.yaml

  .. group-tab:: Helm

    .. parsed-literal::

      helm install cilium-preflight |CHART_RELEASE| \\
        --namespace=kube-system \\
        --set preflight.enabled=true \\
        --set agent=false \\
        --set operator.enabled=false

  .. group-tab:: kubectl (kubeproxy-free)

    .. parsed-literal::

      helm template |CHART_RELEASE| \\
        --namespace=kube-system \\
        --set preflight.enabled=true \\
        --set agent=false \\
        --set operator.enabled=false \\
        --set k8sServiceHost=API_SERVER_IP \\
        --set k8sServicePort=API_SERVER_PORT \\
        > cilium-preflight.yaml
      kubectl create -f cilium-preflight.yaml

  .. group-tab:: Helm (kubeproxy-free)

    .. parsed-literal::

      helm install cilium-preflight |CHART_RELEASE| \\
        --namespace=kube-system \\
        --set preflight.enabled=true \\
        --set agent=false \\
        --set operator.enabled=false \\
        --set k8sServiceHost=API_SERVER_IP \\
        --set k8sServicePort=API_SERVER_PORT

After applying the ``cilium-preflight.yaml``, ensure that the number of READY
pods is the same number of Cilium pods running.

.. code-block:: shell-session

    $ kubectl get daemonset -n kube-system | sed -n '1p;/cilium/p'
    NAME                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
    cilium                    2         2         2       2            2           <none>          1h20m
    cilium-pre-flight-check   2         2         2       2            2           <none>          7m15s

Once the number of READY pods are equal, make sure the Cilium pre-flight
deployment is also marked as READY 1/1. If it shows READY 0/1, consult the
:ref:`cnp_validation` section and resolve issues with the deployment before
continuing with the upgrade.

.. code-block:: shell-session

    $ kubectl get deployment -n kube-system cilium-pre-flight-check -w
    NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
    cilium-pre-flight-check   1/1     1            0           12s

.. _cleanup_preflight_check:

Clean up pre-flight check
-------------------------

Once the number of READY for the preflight :term:`DaemonSet` is the same as the number
of cilium pods running and the preflight ``Deployment`` is marked as READY ``1/1``
you can delete the cilium-preflight and proceed with the upgrade.

.. tabs::
  .. group-tab:: kubectl

    .. code-block:: shell-session

      kubectl delete -f cilium-preflight.yaml

  .. group-tab:: Helm

    .. code-block:: shell-session

      helm delete cilium-preflight --namespace=kube-system

.. _upgrade_minor:

Upgrading Cilium
================

During normal cluster operations, all Cilium components should run the same
version. Upgrading just one of them (e.g., upgrading the agent without
upgrading the operator) could result in unexpected cluster behavior.
The following steps will describe how to upgrade all of the components from
one stable release to a later stable release.

.. include:: upgrade-warning.rst

Step 1: Upgrade to latest patch version
---------------------------------------

When upgrading from one minor release to another minor release, for example
1.x to 1.y, it is recommended to upgrade to the `latest patch release
<https://github.com/cilium/cilium#stable-releases>`__ for a Cilium release series first.
Upgrading to the latest patch release ensures the most seamless experience if a
rollback is required following the minor release upgrade. The upgrade guides
for previous versions can be found for each minor version at the bottom left
corner.

Step 2: Use Helm to Upgrade your Cilium deployment
--------------------------------------------------------------------------------------

:term:`Helm` can be used to either upgrade Cilium directly or to generate a new set of
YAML files that can be used to upgrade an existing deployment via ``kubectl``.
By default, Helm will generate the new templates using the default values files
packaged with each new release. You still need to ensure that you are
specifying the equivalent options as used for the initial deployment, either by
specifying a them at the command line or by committing the values to a YAML
file.

.. include:: ../installation/k8s-install-download-release.rst

To minimize datapath disruption during the upgrade, the
``upgradeCompatibility`` option should be set to the initial Cilium
version which was installed in this cluster.

.. tabs::
  .. group-tab:: kubectl

    Generate the required YAML file and deploy it:

    .. parsed-literal::

      helm template |CHART_RELEASE| \\
        --set upgradeCompatibility=1.X \\
        --namespace kube-system \\
        > cilium.yaml
      kubectl apply -f cilium.yaml

  .. group-tab:: Helm

    Deploy Cilium release via Helm:

    .. parsed-literal::

      helm upgrade cilium |CHART_RELEASE| \\
        --namespace=kube-system \\
        --set upgradeCompatibility=1.X

.. note::

   Instead of using ``--set``, you can also save the values relative to your
   deployment in a YAML file and use it to regenerate the YAML for the latest
   Cilium version. Running any of the previous commands will overwrite
   the existing cluster's :term:`ConfigMap` so it is critical to preserve any existing
   options, either by setting them at the command line or storing them in a
   YAML file, similar to:

   .. code-block:: yaml

      agent: true
      upgradeCompatibility: "1.8"
      ipam:
        mode: "kubernetes"
      k8sServiceHost: "API_SERVER_IP"
      k8sServicePort: "API_SERVER_PORT"
      kubeProxyReplacement: "true"

   You can then upgrade using this values file by running:

   .. parsed-literal::

      helm upgrade cilium |CHART_RELEASE| \\
        --namespace=kube-system \\
        -f my-values.yaml

When upgrading from one minor release to another minor release using
``helm upgrade``, do *not* use Helm's ``--reuse-values`` flag.
The ``--reuse-values`` flag ignores any newly introduced values present in
the new release and thus may cause the Helm template to render incorrectly.
Instead, if you want to reuse the values from your existing installation,
save the old values in a values file, check the file for any renamed or
deprecated values, and then pass it to the ``helm upgrade`` command as
described above. You can retrieve and save the values from an existing
installation with the following command:

.. code-block:: shell-session

  helm get values cilium --namespace=kube-system -o yaml > old-values.yaml

The ``--reuse-values`` flag may only be safely used if the Cilium chart version
remains unchanged, for example when ``helm upgrade`` is used to apply
configuration changes without upgrading Cilium.

Step 3: Rolling Back
--------------------

Occasionally, it may be necessary to undo the rollout because a step was missed
or something went wrong during upgrade. To undo the rollout run:

.. tabs::
  .. group-tab:: kubectl

    .. code-block:: shell-session

      kubectl rollout undo daemonset/cilium -n kube-system

  .. group-tab:: Helm

    .. code-block:: shell-session

      helm history cilium --namespace=kube-system
      helm rollback cilium [REVISION] --namespace=kube-system

This will revert the latest changes to the Cilium ``DaemonSet`` and return
Cilium to the state it was in prior to the upgrade.

.. note::

    When rolling back after new features of the new minor version have already
    been consumed, consult the :ref:`version_notes` to check and prepare for
    incompatible feature use before downgrading/rolling back. This step is only
    required after new functionality introduced in the new minor version has
    already been explicitly used by creating new resources or by opting into
    new features via the :term:`ConfigMap`.

.. _version_notes:
.. _upgrade_version_specifics:

Version Specific Notes
======================

This section details the upgrade notes specific to |CURRENT_RELEASE|. Read them
carefully and take the suggested actions before upgrading Cilium to |CURRENT_RELEASE|.
For upgrades to earlier releases, see the
:prev-docs:`upgrade notes to the previous version <operations/upgrade/#upgrade-notes>`.

The only tested upgrade and rollback path is between consecutive minor releases.
Always perform upgrades and rollbacks between one minor release at a time.
Additionally, always update to the latest patch release of your current version
before attempting an upgrade.

Tested upgrades are expected to have minimal to no impact on new and existing
connections matched by either no Network Policies, or L3/L4 Network Policies only.
Any traffic flowing via user space proxies (for example, because an L7 policy is
in place, or using Ingress/Gateway API) will be disrupted during upgrade. Endpoints
communicating via the proxy must reconnect to re-establish connections.

.. _current_release_required_changes:

.. _1.17_upgrade_notes:

1.17 Upgrade Notes
------------------

* Operating Cilium in ``--datapath-mode=lb-only`` for plain Docker mode now requires to
  add an additional ``--enable-k8s=false`` to the command line, otherwise it is assumed
  that Kubernetes is present.
* The Kubernetes clients used by Cilium Agent and Cilium Operator now have separately configurable
  rate limits. The default rate limit for Cilium Operator K8s clients has been increased to
  100 QPS/200 Burst. To configure the rate limit for Cilium Operator, use the
  ``--operator-k8s-client-qps`` and ``--operator-k8s-client-burst`` flags or the corresponding
  Helm values.
* Support for Consul, deprecated since v1.12, has been removed.
* Cilium now supports services protocol differentiation, which allows the agent to distinguish two
  services on the same port with different protocols (e.g. TCP and UDP).
  This feature, enabled by default, can be controlled with the ``--bpf-lb-proto-diff`` flag.
  After the upgrade, existing services without a protocol set will be preserved as such, to avoid any
  connection disruptions, and will need to be deleted and recreated in order for their protocol to
  be taken into account by the agent.
  In case of downgrades to a version that doesn't support services protocol differentiation,
  existing services with the protocol set will be deleted and recreated, without the protocol, by
  the agent, causing connection disruptions for such services.
* MTU auto-detection is now continuous during agent lifetime, changing device MTU no longer requires
  restarting the agent to pick up the new MTU.
* MTU auto-detection will now use the lowest MTU of all external interfaces. Before, only the primary
  interface was considered. One exception to this is in ENI mode where the secondary interfaces are not
  considered for MTU auto-detection. MTU can still be configured manually via the ``MTU`` helm option,
  ``--mtu`` agent flag or ``mtu`` option in CNI configuration.
* Support for L7 protocol visibility using Pod annotations (``policy.cilium.io/proxy-visibility``),
  deprecated since v1.15, has been removed.
* The Cilium cluster name validation cannot be bypassed anymore, both for the local and
  remote clusters. The cluster name is strictly enforced to consist of at most 32 lower
  case alphanumeric characters and '-', start and end with an alphanumeric character.
* Cilium could previously be run in a configuration where the Etcd instances
  that distribute Cilium state between nodes would be managed in pod network by
  Cilium itself. This support, which had been previously deprecated as complicated
  and error prone, has now been removed. Refer to :ref:`k8s_install_etcd` for
  alternatives for running Cilium with Etcd.
* For IPsec, support for a single key has been removed. Per-tunnel keys will
  now be used regardless of the presence of the ``+`` sign in the secret.
* The option to run a synchronous probe using ``cilium-health status --probe`` is no longer supported,
  and is now a hidden option that returns the results of the most recent cached probe. It will be
  removed in a future release.
* The Cilium status API now reports the KVStore subsystem with ``Disabled`` state when disabled,
  instead of ``OK`` state and ``Disabled`` message.
* Support for ``metallb-bgp``, deprecated since 1.14, has been removed.

Removed Options
~~~~~~~~~~~~~~~

* The previously deprecated ``clustermesh-ip-identities-sync-timeout`` flag has
  been removed in favor of ``clustermesh-sync-timeout``.
* The previously deprecated built-in WireGuard userspace-mode fallback (Helm ``wireguard.userspaceFallback``)
  has been removed. Users of WireGuard transparent encryption are required to use a Linux kernel with
  WireGuard support.
* The previously deprecated ``metallb-bgp`` flags ``bgp-config-path``, ``bgp-announce-lb-ip``
  and ``bgp-announce-pod-cidr`` have been removed. Users are now required to use Cilium BGP
  control plane for BGP advertisements.

Deprecated Options
~~~~~~~~~~~~~~~~~~

* The high-scale mode for ipcache has been deprecated and will be removed in v1.18.
* The hubble-relay flag ``--dial-timeout`` has been deprecated (now a no-op)
  and will be removed in Cilium 1.18.

Helm Options
~~~~~~~~~~~~

* The Helm options ``hubble.tls.server.cert``, ``hubble.tls.server.key``,
  ``hubble.relay.tls.client.cert``, ``hubble.relay.tls.client.key``,
  ``hubble.relay.tls.server.cert``, ``hubble.relay.tls.server.key``,
  ``hubble.ui.tls.client.cert``, and ``hubble.ui.tls.client.key`` have been
  deprecated in favor of the associated ``existingSecret`` options and will be
  removed in a future release.
* The default value of ``hubble.tls.auto.certValidityDuration`` has been
  lowered from 1095 days to 365 days because recent versions of MacOS will fail
  to validate certificates with expirations longer than 825 days.
* The Helm option ``hubble.relay.dialTimeout`` has been deprecated (now a no-op)
  and will be removed in Cilium 1.18.
* The ``metallb-bgp`` integration Helm options ``bgp.enabled``, ``bgp.announce.podCIDR``, and
  ``bgp.announce.loadbalancerIP`` have been removed. Users are now required to use Cilium BGP
  control plane options available under ``bgpControlPlane`` for BGP announcements.
* The default value of ``dnsProxy.endpointMaxIpPerHostname`` and its
  corresponding agent option has been increased from 50 to 1000 to reflect
  improved scaling of toFQDNs policies and to better handle domains which return
  a large number of IPs with short TTLs.

Agent Options
~~~~~~~~~~~~~

* The ``CONNTRACK_LOCAL`` option has been deprecated and will be removed in a
  future release.

Bugtool Options
~~~~~~~~~~~~~~~

* The flag ``k8s-mode`` (and related flags ``cilium-agent-container-name``, ``k8s-namespace`` & ``k8s-label``)
  have been deprecated and will be removed in a Cilium 1.18. Cilium CLI should be used to gather a sysdump from a K8s cluster.

Added Metrics
~~~~~~~~~~~~~
* ``cilium_node_health_connectivity_status``
* ``cilium_node_health_connectivity_latency_seconds``
* ``cilium_operator_unmanaged_pods``
* ``cilium_policy_selector_match_count_max``
* ``cilium_identity_cache_timer_duration``
* ``cilium_identity_cache_timer_trigger_latency``
* ``cilium_identity_cache_timer_trigger_folds``

Removed Metrics
~~~~~~~~~~~~~~~
* ``cilium_cidrgroup_translation_time_stats_seconds`` has been removed, as the measured code path no longer exists.
* ``cilium_triggers_policy_update_total`` has been removed.
* ``cilium_triggers_policy_update_folds`` has been removed.
* ``cilium_triggers_policy_update_call_duration`` has been removed.

Changed Metrics
~~~~~~~~~~~~~~~

Deprecated Metrics
~~~~~~~~~~~~~~~~~~
* ``cilium_node_connectivity_status`` is now deprecated. Please use ``cilium_node_health_connectivity_status`` instead.
* ``cilium_node_connectivity_latency_seconds`` is now deprecated. Please use ``cilium_node_health_connectivity_latency_seconds`` instead.

Hubble CLI
~~~~~~~~~~

* the ``--cluster`` behavior changed to show flows emitted from nodes outside of
  the provided cluster name (either coming from or going to the target cluster).
  This change brings consistency between the ``--cluster`` and ``--namespace``
  flags and removed the incompatibility between the ``--cluster`` and
  ``--node-name`` flags. The previous behavior of ``--cluster foo`` can be
  reproduced with ``--node-name foo/`` (shows all flows emitted from a node in
  cluster ``foo``).

Advanced
========

Upgrade Impact
--------------

Upgrades are designed to have minimal impact on your running deployment.
Networking connectivity, policy enforcement and load balancing will remain
functional in general. The following is a list of operations that will not be
available during the upgrade:

* API-aware policy rules are enforced in user space proxies and are
  running as part of the Cilium pod. Upgrading Cilium causes the proxy to
  restart, which results in a connectivity outage and causes the connection to reset.

* Existing policy will remain effective but implementation of new policy rules
  will be postponed to after the upgrade has been completed on a particular
  node.

* Monitoring components such as ``cilium-dbg monitor`` will experience a brief
  outage while the Cilium pod is restarting. Events are queued up and read
  after the upgrade. If the number of events exceeds the event buffer size,
  events will be lost.

.. _upgrade_configmap:

Rebasing a ConfigMap
--------------------

This section describes the procedure to rebase an existing :term:`ConfigMap` to the
template of another version.

Export the current ConfigMap
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

        $ kubectl get configmap -n kube-system cilium-config -o yaml --export > cilium-cm-old.yaml
        $ cat ./cilium-cm-old.yaml
        apiVersion: v1
        data:
          clean-cilium-state: "false"
          debug: "true"
          disable-ipv4: "false"
          etcd-config: |-
            ---
            endpoints:
            - https://192.168.60.11:2379
            #
            # In case you want to use TLS in etcd, uncomment the 'trusted-ca-file' line
            # and create a kubernetes secret by following the tutorial in
            # https://cilium.link/etcd-config
            trusted-ca-file: '/var/lib/etcd-secrets/etcd-client-ca.crt'
            #
            # In case you want client to server authentication, uncomment the following
            # lines and add the certificate and key in cilium-etcd-secrets below
            key-file: '/var/lib/etcd-secrets/etcd-client.key'
            cert-file: '/var/lib/etcd-secrets/etcd-client.crt'
        kind: ConfigMap
        metadata:
          creationTimestamp: null
          name: cilium-config
          selfLink: /api/v1/namespaces/kube-system/configmaps/cilium-config

In the :term:`ConfigMap` above, we can verify that Cilium is using ``debug`` with
``true``, it has a etcd endpoint running with `TLS <https://etcd.io/docs/latest/op-guide/security/>`_,
and the etcd is set up to have `client to server authentication <https://etcd.io/docs/latest/op-guide/security/#example-2-client-to-server-authentication-with-https-client-certificates>`_.

Generate the latest ConfigMap
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: shell-session

    helm template cilium \
      --namespace=kube-system \
      --set agent=false \
      --set config.enabled=true \
      --set operator.enabled=false \
      > cilium-configmap.yaml

Add new options
~~~~~~~~~~~~~~~

Add the new options manually to your old :term:`ConfigMap`, and make the necessary
changes.

In this example, the ``debug`` option is meant to be kept with ``true``, the
``etcd-config`` is kept unchanged, and ``monitor-aggregation`` is a new
option, but after reading the :ref:`version_notes` the value was kept unchanged
from the default value.

After making the necessary changes, the old :term:`ConfigMap` was migrated with the
new options while keeping the configuration that we wanted:

::

        $ cat ./cilium-cm-old.yaml
        apiVersion: v1
        data:
          debug: "true"
          disable-ipv4: "false"
          # If you want to clean cilium state; change this value to true
          clean-cilium-state: "false"
          monitor-aggregation: "medium"
          etcd-config: |-
            ---
            endpoints:
            - https://192.168.60.11:2379
            #
            # In case you want to use TLS in etcd, uncomment the 'trusted-ca-file' line
            # and create a kubernetes secret by following the tutorial in
            # https://cilium.link/etcd-config
            trusted-ca-file: '/var/lib/etcd-secrets/etcd-client-ca.crt'
            #
            # In case you want client to server authentication, uncomment the following
            # lines and add the certificate and key in cilium-etcd-secrets below
            key-file: '/var/lib/etcd-secrets/etcd-client.key'
            cert-file: '/var/lib/etcd-secrets/etcd-client.crt'
        kind: ConfigMap
        metadata:
          creationTimestamp: null
          name: cilium-config
          selfLink: /api/v1/namespaces/kube-system/configmaps/cilium-config

Apply new ConfigMap
~~~~~~~~~~~~~~~~~~~

After adding the options, manually save the file with your changes and install
the :term:`ConfigMap` in the ``kube-system`` namespace of your cluster.

.. code-block:: shell-session

        $ kubectl apply -n kube-system -f ./cilium-cm-old.yaml

As the :term:`ConfigMap` is successfully upgraded we can start upgrading Cilium
``DaemonSet`` and ``RBAC`` which will pick up the latest configuration from the
:term:`ConfigMap`.

Migrating from kvstore-backed identities to Kubernetes CRD-backed identities
----------------------------------------------------------------------------

Beginning with Cilium 1.6, Kubernetes CRD-backed security identities can be
used for smaller clusters. Along with other changes in 1.6, this allows
kvstore-free operation if desired. It is possible to migrate identities from an
existing kvstore deployment to CRD-backed identities. This minimizes
disruptions to traffic as the update rolls out through the cluster.

Migration
~~~~~~~~~

When identities change, existing connections can be disrupted while Cilium
initializes and synchronizes with the shared identity store. The disruption
occurs when new numeric identities are used for existing pods on some instances
and others are used on others. When converting to CRD-backed identities, it is
possible to pre-allocate CRD identities so that the numeric identities match
those in the kvstore. This allows new and old Cilium instances in the rollout
to agree.

There are two ways to achieve this: you can either run a one-off ``cilium preflight migrate-identity`` script
which will perform a point-in-time copy of all identities from the kvstore to CRDs (added in Cilium 1.6), or use the "Double Write" identity
allocation mode which will have Cilium manage identities in both the kvstore and CRD at the same time for a seamless migration (added in Cilium 1.17).

Migration with the ``cilium preflight migrate-identity`` script
###############################################################

The ``cilium preflight migrate-identity`` script is a one-off tool that can be used to copy identities from the kvstore into CRDs.
It has a couple of limitations:

* If an identity is created in the kvstore after the one-off migration has been completed, it will not be copied into a CRD.
  This means that you need to perform the migration on a cluster with no identity churn.
* There is no easy way to revert back to ``--identity-allocation-mode=kvstore`` if something goes wrong after
  Cilium has been migrated to ``--identity-allocation-mode=crd``

If these limitations are not acceptable, it is recommended to use the ":ref:`Double Write <double_write_migration>`" identity allocation mode instead.

The following steps show an example of performing the migration using the ``cilium preflight migrate-identity`` script.
It is safe to re-run the command if desired. It will identify already allocated identities or ones that
cannot be migrated. Note that identity ``34815`` is migrated, ``17003`` is
already migrated, and ``11730`` has a conflict and a new ID allocated for those
labels.

The steps below assume a stable cluster with no new identities created during
the rollout. Once Cilium using CRD-backed identities is running, it may begin
allocating identities in a way that conflicts with older ones in the kvstore.

The cilium preflight manifest requires etcd support and can be built with:

.. code-block:: shell-session

    helm template cilium \
      --namespace=kube-system \
      --set preflight.enabled=true \
      --set agent=false \
      --set config.enabled=false \
      --set operator.enabled=false \
      --set etcd.enabled=true \
      --set etcd.ssl=true \
      > cilium-preflight.yaml
    kubectl create -f cilium-preflight.yaml

Example migration
~~~~~~~~~~~~~~~~~

.. code-block:: shell-session

      $ kubectl exec -n kube-system cilium-pre-flight-check-1234 -- cilium-dbg preflight migrate-identity
      INFO[0000] Setting up kvstore client
      INFO[0000] Connecting to etcd server...                  config=/var/lib/cilium/etcd-config.yml endpoints="[https://192.168.60.11:2379]" subsys=kvstore
      INFO[0000] Setting up kubernetes client
      INFO[0000] Establishing connection to apiserver          host="https://192.168.60.11:6443" subsys=k8s
      INFO[0000] Connected to apiserver                        subsys=k8s
      INFO[0000] Got lease ID 29c66c67db8870c8                 subsys=kvstore
      INFO[0000] Got lock lease ID 29c66c67db8870ca            subsys=kvstore
      INFO[0000] Successfully verified version of etcd endpoint  config=/var/lib/cilium/etcd-config.yml endpoints="[https://192.168.60.11:2379]" etcdEndpoint="https://192.168.60.11:2379" subsys=kvstore version=3.3.13
      INFO[0000] CRD (CustomResourceDefinition) is installed and up-to-date  name=CiliumNetworkPolicy/v2 subsys=k8s
      INFO[0000] Updating CRD (CustomResourceDefinition)...    name=v2.CiliumEndpoint subsys=k8s
      INFO[0001] CRD (CustomResourceDefinition) is installed and up-to-date  name=v2.CiliumEndpoint subsys=k8s
      INFO[0001] Updating CRD (CustomResourceDefinition)...    name=v2.CiliumNode subsys=k8s
      INFO[0002] CRD (CustomResourceDefinition) is installed and up-to-date  name=v2.CiliumNode subsys=k8s
      INFO[0002] Updating CRD (CustomResourceDefinition)...    name=v2.CiliumIdentity subsys=k8s
      INFO[0003] CRD (CustomResourceDefinition) is installed and up-to-date  name=v2.CiliumIdentity subsys=k8s
      INFO[0003] Listing identities in kvstore
      INFO[0003] Migrating identities to CRD
      INFO[0003] Skipped non-kubernetes labels when labelling ciliumidentity. All labels will still be used in identity determination  labels="map[]" subsys=crd-allocator
      INFO[0003] Skipped non-kubernetes labels when labelling ciliumidentity. All labels will still be used in identity determination  labels="map[]" subsys=crd-allocator
      INFO[0003] Skipped non-kubernetes labels when labelling ciliumidentity. All labels will still be used in identity determination  labels="map[]" subsys=crd-allocator
      INFO[0003] Migrated identity                             identity=34815 identityLabels="k8s:class=tiefighter;k8s:io.cilium.k8s.policy.cluster=default;k8s:io.cilium.k8s.policy.serviceaccount=default;k8s:io.kubernetes.pod.namespace=default;k8s:org=empire;"
      WARN[0003] ID is allocated to a different key in CRD. A new ID will be allocated for the this key  identityLabels="k8s:class=deathstar;k8s:io.cilium.k8s.policy.cluster=default;k8s:io.cilium.k8s.policy.serviceaccount=default;k8s:io.kubernetes.pod.namespace=default;k8s:org=empire;" oldIdentity=11730
      INFO[0003] Reusing existing global key                   key="k8s:class=deathstar;k8s:io.cilium.k8s.policy.cluster=default;k8s:io.cilium.k8s.policy.serviceaccount=default;k8s:io.kubernetes.pod.namespace=default;k8s:org=empire;" subsys=allocator
      INFO[0003] New ID allocated for key in CRD               identity=17281 identityLabels="k8s:class=deathstar;k8s:io.cilium.k8s.policy.cluster=default;k8s:io.cilium.k8s.policy.serviceaccount=default;k8s:io.kubernetes.pod.namespace=default;k8s:org=empire;" oldIdentity=11730
      INFO[0003] ID was already allocated to this key. It is already migrated  identity=17003 identityLabels="k8s:class=xwing;k8s:io.cilium.k8s.policy.cluster=default;k8s:io.cilium.k8s.policy.serviceaccount=default;k8s:io.kubernetes.pod.namespace=default;k8s:org=alliance;"

.. note::

    It is also possible to use the ``--k8s-kubeconfig-path``  and ``--kvstore-opt``
    ``cilium`` CLI options with the preflight command. The default is to derive the
    configuration as cilium-agent does.

  .. code-block:: shell-session

        cilium preflight migrate-identity --k8s-kubeconfig-path /var/lib/cilium/cilium.kubeconfig --kvstore etcd --kvstore-opt etcd.config=/var/lib/cilium/etcd-config.yml

Once the migration is complete, confirm the endpoint identities match by listing the endpoints stored in CRDs and in etcd:

.. code-block:: shell-session

      $ kubectl get ciliumendpoints -A # new CRD-backed endpoints
      $ kubectl exec -n kube-system cilium-1234 -- cilium-dbg endpoint list # existing etcd-backed endpoints

Clearing CRD identities
~~~~~~~~~~~~~~~~~~~~~~~

If a migration has gone wrong, it possible to start with a clean slate. Ensure that no Cilium instances are running with ``--identity-allocation-mode=crd`` and execute:

.. code-block:: shell-session

      $ kubectl delete ciliumid --all

.. _double_write_migration:

Migration with the "Double Write" identity allocation mode
##########################################################

.. include:: ../beta.rst

The "Double Write" Identity Allocation Mode allows Cilium to allocate identities as KVStore values *and* as CRDs at the
same time. This mode also has two versions: one where the source of truth comes from the kvstore (``--identity-allocation-mode=doublewrite-readkvstore``),
and one where the source of truth comes from CRDs (``--identity-allocation-mode=doublewrite-readcrd``).

.. note::

    "Double Write" mode is not compatible with Consul as the KVStore

The high-level migration plan looks as follows:

#. Starting state: Cilium is running in KVStore mode.
#. Switch Cilium to “Double Write” mode with all reads happening from the KVStore. This is almost the same as the
   pure KVStore mode with the only difference being that all identities are duplicated as CRDs but are not used.
#. Switch Cilium to “Double Write” mode with all reads happening from CRDs. This is equivalent to Cilium running in
   pure CRD mode but identities will still be updated in the KVStore to allow for the possibility of a fast rollback.
#. Switch Cilium to CRD mode. The KVStore will no longer be used and will be ready for decommission.

This will allow you to perform a gradual and seamless migration with the possibility of a fast rollback at steps two or three.

Furthermore, when the "Double Write" mode is enabled, the Operator will emit additional metrics to help monitor the
migration progress. These metrics can be used for alerting about identity inconsistencies between the KVStore and CRDs.

Note that you can also use this to migrate from CRD to KVStore mode. All operations simply need to be repeated in reverse order.

Rollout Instructions
~~~~~~~~~~~~~~~~~~~~

#. Re-deploy first the Operator and then the Agents with ``--identity-allocation-mode=doublewrite-readkvstore``.
#. Monitor the Operator metrics and logs to ensure that all identities have converged between the KVStore and CRDs. The relevant metrics emitted by the Operator are:

   * ``cilium_operator_identity_crd_total_count`` and ``cilium_operator_identity_kvstore_total_count`` report the total number of identities in CRDs and KVStore respectively.
   * ``cilium_operator_identity_crd_only_count`` and ``cilium_operator_identity_kvstore_only_count`` report the number of
     identities that are only in CRDs or only in the KVStore respectively, to help detect inconsistencies.

   In case further investigation is needed, the Operator logs will contain detailed information about the discrepancies between KVStore and CRD identities.
   Note that Garbage Collection for KVStore identities and CRD identities happens at slightly different times, so it is possible to see discrepancies in the metrics
   for certain periods of time, depending on ``--identity-gc-interval`` and ``--identity-heartbeat-timeout`` settings.
#. Once all identities have converged, re-deploy the Operator and the Agents with ``--identity-allocation-mode=doublewrite-readcrd``.
   This will cause Cilium to read identities only from CRDs, but continue to write them to the KVStore.
#. Once you are ready to decommission the KVStore, re-deploy first the Agents and then the Operator with ``--identity-allocation-mode=crd``.
   This will make Cilium read and write identities only to CRDs.
#. You can now decommission the KVStore.

.. _cnp_validation:

CNP Validation
--------------

Running the CNP Validator will make sure the policies deployed in the cluster
are valid. It is important to run this validation before an upgrade so it will
make sure Cilium has a correct behavior after upgrade. Avoiding doing this
validation might cause Cilium from updating its ``NodeStatus`` in those invalid
Network Policies as well as in the worst case scenario it might give a false
sense of security to the user if a policy is badly formatted and Cilium is not
enforcing that policy due a bad validation schema. This CNP Validator is
automatically executed as part of the pre-flight check :ref:`pre_flight`.

Start by deployment the ``cilium-pre-flight-check`` and check if the
``Deployment`` shows READY 1/1, if it does not check the pod logs.

.. code-block:: shell-session

      $ kubectl get deployment -n kube-system cilium-pre-flight-check -w
      NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
      cilium-pre-flight-check   0/1     1            0           12s

      $ kubectl logs -n kube-system deployment/cilium-pre-flight-check -c cnp-validator --previous
      level=info msg="Setting up kubernetes client"
      level=info msg="Establishing connection to apiserver" host="https://172.20.0.1:443" subsys=k8s
      level=info msg="Connected to apiserver" subsys=k8s
      level=info msg="Validating CiliumNetworkPolicy 'default/cidr-rule': OK!
      level=error msg="Validating CiliumNetworkPolicy 'default/cnp-update': unexpected validation error: spec.labels: Invalid value: \"string\": spec.labels in body must be of type object: \"string\""
      level=error msg="Found invalid CiliumNetworkPolicy"

In this example, we can see the ``CiliumNetworkPolicy`` in the ``default``
namespace with the name ``cnp-update`` is not valid for the Cilium version we
are trying to upgrade. In order to fix this policy we need to edit it, we can
do this by saving the policy locally and modify it. For this example it seems
the ``.spec.labels`` has set an array of strings which is not correct as per
the official schema.

.. code-block:: shell-session

      $ kubectl get cnp -n default cnp-update -o yaml > cnp-bad.yaml
      $ cat cnp-bad.yaml
        apiVersion: cilium.io/v2
        kind: CiliumNetworkPolicy
        [...]
        spec:
          endpointSelector:
            matchLabels:
              id: app1
          ingress:
          - fromEndpoints:
            - matchLabels:
                id: app2
            toPorts:
            - ports:
              - port: "80"
                protocol: TCP
          labels:
          - custom=true
        [...]

To fix this policy we need to set the ``.spec.labels`` with the right format and
commit these changes into Kubernetes.

.. code-block:: shell-session

      $ cat cnp-bad.yaml
        apiVersion: cilium.io/v2
        kind: CiliumNetworkPolicy
        [...]
        spec:
          endpointSelector:
            matchLabels:
              id: app1
          ingress:
          - fromEndpoints:
            - matchLabels:
                id: app2
            toPorts:
            - ports:
              - port: "80"
                protocol: TCP
          labels:
          - key: "custom"
            value: "true"
        [...]
      $
      $ kubectl apply -f ./cnp-bad.yaml

After applying the fixed policy we can delete the pod that was validating the
policies so that Kubernetes creates a new pod immediately to verify if the fixed
policies are now valid.

.. code-block:: shell-session

      $ kubectl delete pod -n kube-system -l k8s-app=cilium-pre-flight-check-deployment
      pod "cilium-pre-flight-check-86dfb69668-ngbql" deleted
      $ kubectl get deployment -n kube-system cilium-pre-flight-check
      NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
      cilium-pre-flight-check   1/1     1            1           55m
      $ kubectl logs -n kube-system deployment/cilium-pre-flight-check -c cnp-validator
      level=info msg="Setting up kubernetes client"
      level=info msg="Establishing connection to apiserver" host="https://172.20.0.1:443" subsys=k8s
      level=info msg="Connected to apiserver" subsys=k8s
      level=info msg="Validating CiliumNetworkPolicy 'default/cidr-rule': OK!
      level=info msg="Validating CiliumNetworkPolicy 'default/cnp-update': OK!
      level=info msg="All CCNPs and CNPs valid!"

Once they are valid you can continue with the upgrade process. :ref:`cleanup_preflight_check`

only   not  epub or latex or html       WARNING  You are looking at unreleased Cilium documentation      Please use the official rendered version released here      https   docs cilium io      admin upgrade                 Upgrade Guide                    upgrade general   This upgrade guide is intended for Cilium running on Kubernetes  If you have questions  feel free to ping us on  Cilium Slack        include   upgrade warning rst      pre flight   Running pre flight check  Required                                       When rolling out an upgrade with Kubernetes  Kubernetes will first terminate the pod followed by pulling the new image version and then finally spin up the new image  In order to reduce the downtime of the agent and to prevent   ErrImagePull   errors during upgrade  the pre flight check pre pulls the new image version  If you are running in  ref  kubeproxy free  mode you must also pass on the Kubernetes API Server IP and   or the Kubernetes API Server Port when generating the   cilium preflight yaml   file      tabs        group tab   kubectl         parsed literal          helm template  CHART RELEASE               namespace kube system              set preflight enabled true              set agent false              set operator enabled false              cilium preflight yaml       kubectl create  f cilium preflight yaml       group tab   Helm         parsed literal          helm install cilium preflight  CHART RELEASE               namespace kube system              set preflight enabled true              set agent false              set operator enabled false       group tab   kubectl  kubeproxy free          parsed literal          helm template  CHART RELEASE               namespace kube system              set preflight enabled true              set agent false              set operator enabled false              set k8sServiceHost API SERVER IP              set k8sServicePort API SERVER PORT              cilium preflight yaml       kubectl create  f cilium preflight yaml       group tab   Helm  kubeproxy free          parsed literal          helm install cilium preflight  CHART RELEASE               namespace kube system              set preflight enabled true              set agent false              set operator enabled false              set k8sServiceHost API SERVER IP              set k8sServicePort API SERVER PORT  After applying the   cilium preflight yaml    ensure that the number of READY pods is the same number of Cilium pods running      code block   shell session        kubectl get daemonset  n kube system   sed  n  1p  cilium p      NAME                      DESIRED   CURRENT   READY   UP TO DATE   AVAILABLE   NODE SELECTOR   AGE     cilium                    2         2         2       2            2            none           1h20m     cilium pre flight check   2         2         2       2            2            none           7m15s  Once the number of READY pods are equal  make sure the Cilium pre flight deployment is also marked as READY 1 1  If it shows READY 0 1  consult the  ref  cnp validation  section and resolve issues with the deployment before continuing with the upgrade      code block   shell session        kubectl get deployment  n kube system cilium pre flight check  w     NAME                      READY   UP TO DATE   AVAILABLE   AGE     cilium pre flight check   1 1     1            0           12s      cleanup preflight check   Clean up pre flight check                            Once the number of READY for the preflight  term  DaemonSet  is the same as the number of cilium pods running and the preflight   Deployment   is marked as READY   1 1   you can delete the cilium preflight and proceed with the upgrade      tabs        group tab   kubectl         code block   shell session        kubectl delete  f cilium preflight yaml       group tab   Helm         code block   shell session        helm delete cilium preflight   namespace kube system      upgrade minor   Upgrading Cilium                   During normal cluster operations  all Cilium components should run the same version  Upgrading just one of them  e g   upgrading the agent without upgrading the operator  could result in unexpected cluster behavior  The following steps will describe how to upgrade all of the components from one stable release to a later stable release      include   upgrade warning rst  Step 1  Upgrade to latest patch version                                          When upgrading from one minor release to another minor release  for example 1 x to 1 y  it is recommended to upgrade to the  latest patch release  https   github com cilium cilium stable releases     for a Cilium release series first  Upgrading to the latest patch release ensures the most seamless experience if a rollback is required following the minor release upgrade  The upgrade guides for previous versions can be found for each minor version at the bottom left corner   Step 2  Use Helm to Upgrade your Cilium deployment                                                                                term  Helm  can be used to either upgrade Cilium directly or to generate a new set of YAML files that can be used to upgrade an existing deployment via   kubectl    By default  Helm will generate the new templates using the default values files packaged with each new release  You still need to ensure that you are specifying the equivalent options as used for the initial deployment  either by specifying a them at the command line or by committing the values to a YAML file      include      installation k8s install download release rst  To minimize datapath disruption during the upgrade  the   upgradeCompatibility   option should be set to the initial Cilium version which was installed in this cluster      tabs        group tab   kubectl      Generate the required YAML file and deploy it          parsed literal          helm template  CHART RELEASE               set upgradeCompatibility 1 X              namespace kube system              cilium yaml       kubectl apply  f cilium yaml       group tab   Helm      Deploy Cilium release via Helm          parsed literal          helm upgrade cilium  CHART RELEASE               namespace kube system              set upgradeCompatibility 1 X     note       Instead of using     set    you can also save the values relative to your    deployment in a YAML file and use it to regenerate the YAML for the latest    Cilium version  Running any of the previous commands will overwrite    the existing cluster s  term  ConfigMap  so it is critical to preserve any existing    options  either by setting them at the command line or storing them in a    YAML file  similar to         code block   yaml        agent  true       upgradeCompatibility   1 8        ipam          mode   kubernetes        k8sServiceHost   API SERVER IP        k8sServicePort   API SERVER PORT        kubeProxyReplacement   true      You can then upgrade using this values file by running         parsed literal          helm upgrade cilium  CHART RELEASE               namespace kube system             f my values yaml  When upgrading from one minor release to another minor release using   helm upgrade    do  not  use Helm s     reuse values   flag  The     reuse values   flag ignores any newly introduced values present in the new release and thus may cause the Helm template to render incorrectly  Instead  if you want to reuse the values from your existing installation  save the old values in a values file  check the file for any renamed or deprecated values  and then pass it to the   helm upgrade   command as described above  You can retrieve and save the values from an existing installation with the following command      code block   shell session    helm get values cilium   namespace kube system  o yaml   old values yaml  The     reuse values   flag may only be safely used if the Cilium chart version remains unchanged  for example when   helm upgrade   is used to apply configuration changes without upgrading Cilium   Step 3  Rolling Back                       Occasionally  it may be necessary to undo the rollout because a step was missed or something went wrong during upgrade  To undo the rollout run      tabs        group tab   kubectl         code block   shell session        kubectl rollout undo daemonset cilium  n kube system       group tab   Helm         code block   shell session        helm history cilium   namespace kube system       helm rollback cilium  REVISION    namespace kube system  This will revert the latest changes to the Cilium   DaemonSet   and return Cilium to the state it was in prior to the upgrade      note        When rolling back after new features of the new minor version have already     been consumed  consult the  ref  version notes  to check and prepare for     incompatible feature use before downgrading rolling back  This step is only     required after new functionality introduced in the new minor version has     already been explicitly used by creating new resources or by opting into     new features via the  term  ConfigMap        version notes      upgrade version specifics   Version Specific Notes                         This section details the upgrade notes specific to  CURRENT RELEASE   Read them carefully and take the suggested actions before upgrading Cilium to  CURRENT RELEASE   For upgrades to earlier releases  see the  prev docs  upgrade notes to the previous version  operations upgrade  upgrade notes     The only tested upgrade and rollback path is between consecutive minor releases  Always perform upgrades and rollbacks between one minor release at a time  Additionally  always update to the latest patch release of your current version before attempting an upgrade   Tested upgrades are expected to have minimal to no impact on new and existing connections matched by either no Network Policies  or L3 L4 Network Policies only  Any traffic flowing via user space proxies  for example  because an L7 policy is in place  or using Ingress Gateway API  will be disrupted during upgrade  Endpoints communicating via the proxy must reconnect to re establish connections       current release required changes       1 17 upgrade notes   1 17 Upgrade Notes                       Operating Cilium in     datapath mode lb only   for plain Docker mode now requires to   add an additional     enable k8s false   to the command line  otherwise it is assumed   that Kubernetes is present    The Kubernetes clients used by Cilium Agent and Cilium Operator now have separately configurable   rate limits  The default rate limit for Cilium Operator K8s clients has been increased to   100 QPS 200 Burst  To configure the rate limit for Cilium Operator  use the       operator k8s client qps   and     operator k8s client burst   flags or the corresponding   Helm values    Support for Consul  deprecated since v1 12  has been removed    Cilium now supports services protocol differentiation  which allows the agent to distinguish two   services on the same port with different protocols  e g  TCP and UDP     This feature  enabled by default  can be controlled with the     bpf lb proto diff   flag    After the upgrade  existing services without a protocol set will be preserved as such  to avoid any   connection disruptions  and will need to be deleted and recreated in order for their protocol to   be taken into account by the agent    In case of downgrades to a version that doesn t support services protocol differentiation    existing services with the protocol set will be deleted and recreated  without the protocol  by   the agent  causing connection disruptions for such services    MTU auto detection is now continuous during agent lifetime  changing device MTU no longer requires   restarting the agent to pick up the new MTU    MTU auto detection will now use the lowest MTU of all external interfaces  Before  only the primary   interface was considered  One exception to this is in ENI mode where the secondary interfaces are not   considered for MTU auto detection  MTU can still be configured manually via the   MTU   helm option        mtu   agent flag or   mtu   option in CNI configuration    Support for L7 protocol visibility using Pod annotations    policy cilium io proxy visibility       deprecated since v1 15  has been removed    The Cilium cluster name validation cannot be bypassed anymore  both for the local and   remote clusters  The cluster name is strictly enforced to consist of at most 32 lower   case alphanumeric characters and      start and end with an alphanumeric character    Cilium could previously be run in a configuration where the Etcd instances   that distribute Cilium state between nodes would be managed in pod network by   Cilium itself  This support  which had been previously deprecated as complicated   and error prone  has now been removed  Refer to  ref  k8s install etcd  for   alternatives for running Cilium with Etcd    For IPsec  support for a single key has been removed  Per tunnel keys will   now be used regardless of the presence of the       sign in the secret    The option to run a synchronous probe using   cilium health status   probe   is no longer supported    and is now a hidden option that returns the results of the most recent cached probe  It will be    removed in a future release    The Cilium status API now reports the KVStore subsystem with   Disabled   state when disabled    instead of   OK   state and   Disabled   message    Support for   metallb bgp    deprecated since 1 14  has been removed   Removed Options                    The previously deprecated   clustermesh ip identities sync timeout   flag has   been removed in favor of   clustermesh sync timeout      The previously deprecated built in WireGuard userspace mode fallback  Helm   wireguard userspaceFallback      has been removed  Users of WireGuard transparent encryption are required to use a Linux kernel with   WireGuard support    The previously deprecated   metallb bgp   flags   bgp config path      bgp announce lb ip     and   bgp announce pod cidr   have been removed  Users are now required to use Cilium BGP   control plane for BGP advertisements   Deprecated Options                       The high scale mode for ipcache has been deprecated and will be removed in v1 18    The hubble relay flag     dial timeout   has been deprecated  now a no op    and will be removed in Cilium 1 18   Helm Options                 The Helm options   hubble tls server cert      hubble tls server key        hubble relay tls client cert      hubble relay tls client key        hubble relay tls server cert      hubble relay tls server key        hubble ui tls client cert    and   hubble ui tls client key   have been   deprecated in favor of the associated   existingSecret   options and will be   removed in a future release    The default value of   hubble tls auto certValidityDuration   has been   lowered from 1095 days to 365 days because recent versions of MacOS will fail   to validate certificates with expirations longer than 825 days    The Helm option   hubble relay dialTimeout   has been deprecated  now a no op    and will be removed in Cilium 1 18    The   metallb bgp   integration Helm options   bgp enabled      bgp announce podCIDR    and     bgp announce loadbalancerIP   have been removed  Users are now required to use Cilium BGP   control plane options available under   bgpControlPlane   for BGP announcements    The default value of   dnsProxy endpointMaxIpPerHostname   and its   corresponding agent option has been increased from 50 to 1000 to reflect   improved scaling of toFQDNs policies and to better handle domains which return   a large number of IPs with short TTLs   Agent Options                  The   CONNTRACK LOCAL   option has been deprecated and will be removed in a   future release   Bugtool Options                    The flag   k8s mode    and related flags   cilium agent container name      k8s namespace       k8s label      have been deprecated and will be removed in a Cilium 1 18  Cilium CLI should be used to gather a sysdump from a K8s cluster   Added Metrics                   cilium node health connectivity status       cilium node health connectivity latency seconds       cilium operator unmanaged pods       cilium policy selector match count max       cilium identity cache timer duration       cilium identity cache timer trigger latency       cilium identity cache timer trigger folds    Removed Metrics                     cilium cidrgroup translation time stats seconds   has been removed  as the measured code path no longer exists      cilium triggers policy update total   has been removed      cilium triggers policy update folds   has been removed      cilium triggers policy update call duration   has been removed   Changed Metrics                  Deprecated Metrics                        cilium node connectivity status   is now deprecated  Please use   cilium node health connectivity status   instead      cilium node connectivity latency seconds   is now deprecated  Please use   cilium node health connectivity latency seconds   instead   Hubble CLI               the     cluster   behavior changed to show flows emitted from nodes outside of   the provided cluster name  either coming from or going to the target cluster     This change brings consistency between the     cluster   and     namespace     flags and removed the incompatibility between the     cluster   and       node name   flags  The previous behavior of     cluster foo   can be   reproduced with     node name foo     shows all flows emitted from a node in   cluster   foo      Advanced           Upgrade Impact                 Upgrades are designed to have minimal impact on your running deployment  Networking connectivity  policy enforcement and load balancing will remain functional in general  The following is a list of operations that will not be available during the upgrade     API aware policy rules are enforced in user space proxies and are   running as part of the Cilium pod  Upgrading Cilium causes the proxy to   restart  which results in a connectivity outage and causes the connection to reset     Existing policy will remain effective but implementation of new policy rules   will be postponed to after the upgrade has been completed on a particular   node     Monitoring components such as   cilium dbg monitor   will experience a brief   outage while the Cilium pod is restarting  Events are queued up and read   after the upgrade  If the number of events exceeds the event buffer size    events will be lost        upgrade configmap   Rebasing a ConfigMap                       This section describes the procedure to rebase an existing  term  ConfigMap  to the template of another version   Export the current ConfigMap                                             kubectl get configmap  n kube system cilium config  o yaml   export   cilium cm old yaml           cat   cilium cm old yaml         apiVersion  v1         data            clean cilium state   false            debug   true            disable ipv4   false            etcd config                                 endpoints                https   192 168 60 11 2379                             In case you want to use TLS in etcd  uncomment the  trusted ca file  line               and create a kubernetes secret by following the tutorial in               https   cilium link etcd config             trusted ca file    var lib etcd secrets etcd client ca crt                              In case you want client to server authentication  uncomment the following               lines and add the certificate and key in cilium etcd secrets below             key file    var lib etcd secrets etcd client key              cert file    var lib etcd secrets etcd client crt          kind  ConfigMap         metadata            creationTimestamp  null           name  cilium config           selfLink   api v1 namespaces kube system configmaps cilium config   In the  term  ConfigMap  above  we can verify that Cilium is using   debug   with   true    it has a etcd endpoint running with  TLS  https   etcd io docs latest op guide security      and the etcd is set up to have  client to server authentication  https   etcd io docs latest op guide security  example 2 client to server authentication with https client certificates      Generate the latest ConfigMap                                   code block   shell session      helm template cilium           namespace kube system           set agent false           set config enabled true           set operator enabled false           cilium configmap yaml  Add new options                  Add the new options manually to your old  term  ConfigMap   and make the necessary changes   In this example  the   debug   option is meant to be kept with   true    the   etcd config   is kept unchanged  and   monitor aggregation   is a new option  but after reading the  ref  version notes  the value was kept unchanged from the default value   After making the necessary changes  the old  term  ConfigMap  was migrated with the new options while keeping the configuration that we wanted                 cat   cilium cm old yaml         apiVersion  v1         data            debug   true            disable ipv4   false              If you want to clean cilium state  change this value to true           clean cilium state   false            monitor aggregation   medium            etcd config                                 endpoints                https   192 168 60 11 2379                             In case you want to use TLS in etcd  uncomment the  trusted ca file  line               and create a kubernetes secret by following the tutorial in               https   cilium link etcd config             trusted ca file    var lib etcd secrets etcd client ca crt                              In case you want client to server authentication  uncomment the following               lines and add the certificate and key in cilium etcd secrets below             key file    var lib etcd secrets etcd client key              cert file    var lib etcd secrets etcd client crt          kind  ConfigMap         metadata            creationTimestamp  null           name  cilium config           selfLink   api v1 namespaces kube system configmaps cilium config  Apply new ConfigMap                      After adding the options  manually save the file with your changes and install the  term  ConfigMap  in the   kube system   namespace of your cluster      code block   shell session            kubectl apply  n kube system  f   cilium cm old yaml  As the  term  ConfigMap  is successfully upgraded we can start upgrading Cilium   DaemonSet   and   RBAC   which will pick up the latest configuration from the  term  ConfigMap     Migrating from kvstore backed identities to Kubernetes CRD backed identities                                                                               Beginning with Cilium 1 6  Kubernetes CRD backed security identities can be used for smaller clusters  Along with other changes in 1 6  this allows kvstore free operation if desired  It is possible to migrate identities from an existing kvstore deployment to CRD backed identities  This minimizes disruptions to traffic as the update rolls out through the cluster   Migration            When identities change  existing connections can be disrupted while Cilium initializes and synchronizes with the shared identity store  The disruption occurs when new numeric identities are used for existing pods on some instances and others are used on others  When converting to CRD backed identities  it is possible to pre allocate CRD identities so that the numeric identities match those in the kvstore  This allows new and old Cilium instances in the rollout to agree   There are two ways to achieve this  you can either run a one off   cilium preflight migrate identity   script which will perform a point in time copy of all identities from the kvstore to CRDs  added in Cilium 1 6   or use the  Double Write  identity allocation mode which will have Cilium manage identities in both the kvstore and CRD at the same time for a seamless migration  added in Cilium 1 17    Migration with the   cilium preflight migrate identity   script                                                                  The   cilium preflight migrate identity   script is a one off tool that can be used to copy identities from the kvstore into CRDs  It has a couple of limitations     If an identity is created in the kvstore after the one off migration has been completed  it will not be copied into a CRD    This means that you need to perform the migration on a cluster with no identity churn    There is no easy way to revert back to     identity allocation mode kvstore   if something goes wrong after   Cilium has been migrated to     identity allocation mode crd    If these limitations are not acceptable  it is recommended to use the   ref  Double Write  double write migration    identity allocation mode instead   The following steps show an example of performing the migration using the   cilium preflight migrate identity   script  It is safe to re run the command if desired  It will identify already allocated identities or ones that cannot be migrated  Note that identity   34815   is migrated    17003   is already migrated  and   11730   has a conflict and a new ID allocated for those labels   The steps below assume a stable cluster with no new identities created during the rollout  Once Cilium using CRD backed identities is running  it may begin allocating identities in a way that conflicts with older ones in the kvstore   The cilium preflight manifest requires etcd support and can be built with      code block   shell session      helm template cilium           namespace kube system           set preflight enabled true           set agent false           set config enabled false           set operator enabled false           set etcd enabled true           set etcd ssl true           cilium preflight yaml     kubectl create  f cilium preflight yaml   Example migration                       code block   shell session          kubectl exec  n kube system cilium pre flight check 1234    cilium dbg preflight migrate identity       INFO 0000  Setting up kvstore client       INFO 0000  Connecting to etcd server                     config  var lib cilium etcd config yml endpoints   https   192 168 60 11 2379   subsys kvstore       INFO 0000  Setting up kubernetes client       INFO 0000  Establishing connection to apiserver          host  https   192 168 60 11 6443  subsys k8s       INFO 0000  Connected to apiserver                        subsys k8s       INFO 0000  Got lease ID 29c66c67db8870c8                 subsys kvstore       INFO 0000  Got lock lease ID 29c66c67db8870ca            subsys kvstore       INFO 0000  Successfully verified version of etcd endpoint  config  var lib cilium etcd config yml endpoints   https   192 168 60 11 2379   etcdEndpoint  https   192 168 60 11 2379  subsys kvstore version 3 3 13       INFO 0000  CRD  CustomResourceDefinition  is installed and up to date  name CiliumNetworkPolicy v2 subsys k8s       INFO 0000  Updating CRD  CustomResourceDefinition        name v2 CiliumEndpoint subsys k8s       INFO 0001  CRD  CustomResourceDefinition  is installed and up to date  name v2 CiliumEndpoint subsys k8s       INFO 0001  Updating CRD  CustomResourceDefinition        name v2 CiliumNode subsys k8s       INFO 0002  CRD  CustomResourceDefinition  is installed and up to date  name v2 CiliumNode subsys k8s       INFO 0002  Updating CRD  CustomResourceDefinition        name v2 CiliumIdentity subsys k8s       INFO 0003  CRD  CustomResourceDefinition  is installed and up to date  name v2 CiliumIdentity subsys k8s       INFO 0003  Listing identities in kvstore       INFO 0003  Migrating identities to CRD       INFO 0003  Skipped non kubernetes labels when labelling ciliumidentity  All labels will still be used in identity determination  labels  map    subsys crd allocator       INFO 0003  Skipped non kubernetes labels when labelling ciliumidentity  All labels will still be used in identity determination  labels  map    subsys crd allocator       INFO 0003  Skipped non kubernetes labels when labelling ciliumidentity  All labels will still be used in identity determination  labels  map    subsys crd allocator       INFO 0003  Migrated identity                             identity 34815 identityLabels  k8s class tiefighter k8s io cilium k8s policy cluster default k8s io cilium k8s policy serviceaccount default k8s io kubernetes pod namespace default k8s org empire         WARN 0003  ID is allocated to a different key in CRD  A new ID will be allocated for the this key  identityLabels  k8s class deathstar k8s io cilium k8s policy cluster default k8s io cilium k8s policy serviceaccount default k8s io kubernetes pod namespace default k8s org empire   oldIdentity 11730       INFO 0003  Reusing existing global key                   key  k8s class deathstar k8s io cilium k8s policy cluster default k8s io cilium k8s policy serviceaccount default k8s io kubernetes pod namespace default k8s org empire   subsys allocator       INFO 0003  New ID allocated for key in CRD               identity 17281 identityLabels  k8s class deathstar k8s io cilium k8s policy cluster default k8s io cilium k8s policy serviceaccount default k8s io kubernetes pod namespace default k8s org empire   oldIdentity 11730       INFO 0003  ID was already allocated to this key  It is already migrated  identity 17003 identityLabels  k8s class xwing k8s io cilium k8s policy cluster default k8s io cilium k8s policy serviceaccount default k8s io kubernetes pod namespace default k8s org alliance       note        It is also possible to use the     k8s kubeconfig path    and     kvstore opt         cilium   CLI options with the preflight command  The default is to derive the     configuration as cilium agent does        code block   shell session          cilium preflight migrate identity   k8s kubeconfig path  var lib cilium cilium kubeconfig   kvstore etcd   kvstore opt etcd config  var lib cilium etcd config yml  Once the migration is complete  confirm the endpoint identities match by listing the endpoints stored in CRDs and in etcd      code block   shell session          kubectl get ciliumendpoints  A   new CRD backed endpoints         kubectl exec  n kube system cilium 1234    cilium dbg endpoint list   existing etcd backed endpoints  Clearing CRD identities                          If a migration has gone wrong  it possible to start with a clean slate  Ensure that no Cilium instances are running with     identity allocation mode crd   and execute      code block   shell session          kubectl delete ciliumid   all      double write migration   Migration with the  Double Write  identity allocation mode                                                                include      beta rst  The  Double Write  Identity Allocation Mode allows Cilium to allocate identities as KVStore values  and  as CRDs at the same time  This mode also has two versions  one where the source of truth comes from the kvstore      identity allocation mode doublewrite readkvstore     and one where the source of truth comes from CRDs      identity allocation mode doublewrite readcrd         note         Double Write  mode is not compatible with Consul as the KVStore  The high level migration plan looks as follows      Starting state  Cilium is running in KVStore mode     Switch Cilium to  Double Write  mode with all reads happening from the KVStore  This is almost the same as the    pure KVStore mode with the only difference being that all identities are duplicated as CRDs but are not used     Switch Cilium to  Double Write  mode with all reads happening from CRDs  This is equivalent to Cilium running in    pure CRD mode but identities will still be updated in the KVStore to allow for the possibility of a fast rollback     Switch Cilium to CRD mode  The KVStore will no longer be used and will be ready for decommission   This will allow you to perform a gradual and seamless migration with the possibility of a fast rollback at steps two or three   Furthermore  when the  Double Write  mode is enabled  the Operator will emit additional metrics to help monitor the migration progress  These metrics can be used for alerting about identity inconsistencies between the KVStore and CRDs   Note that you can also use this to migrate from CRD to KVStore mode  All operations simply need to be repeated in reverse order   Rollout Instructions                          Re deploy first the Operator and then the Agents with     identity allocation mode doublewrite readkvstore       Monitor the Operator metrics and logs to ensure that all identities have converged between the KVStore and CRDs  The relevant metrics emitted by the Operator are          cilium operator identity crd total count   and   cilium operator identity kvstore total count   report the total number of identities in CRDs and KVStore respectively         cilium operator identity crd only count   and   cilium operator identity kvstore only count   report the number of      identities that are only in CRDs or only in the KVStore respectively  to help detect inconsistencies      In case further investigation is needed  the Operator logs will contain detailed information about the discrepancies between KVStore and CRD identities     Note that Garbage Collection for KVStore identities and CRD identities happens at slightly different times  so it is possible to see discrepancies in the metrics    for certain periods of time  depending on     identity gc interval   and     identity heartbeat timeout   settings     Once all identities have converged  re deploy the Operator and the Agents with     identity allocation mode doublewrite readcrd       This will cause Cilium to read identities only from CRDs  but continue to write them to the KVStore     Once you are ready to decommission the KVStore  re deploy first the Agents and then the Operator with     identity allocation mode crd       This will make Cilium read and write identities only to CRDs     You can now decommission the KVStore       cnp validation   CNP Validation                 Running the CNP Validator will make sure the policies deployed in the cluster are valid  It is important to run this validation before an upgrade so it will make sure Cilium has a correct behavior after upgrade  Avoiding doing this validation might cause Cilium from updating its   NodeStatus   in those invalid Network Policies as well as in the worst case scenario it might give a false sense of security to the user if a policy is badly formatted and Cilium is not enforcing that policy due a bad validation schema  This CNP Validator is automatically executed as part of the pre flight check  ref  pre flight    Start by deployment the   cilium pre flight check   and check if the   Deployment   shows READY 1 1  if it does not check the pod logs      code block   shell session          kubectl get deployment  n kube system cilium pre flight check  w       NAME                      READY   UP TO DATE   AVAILABLE   AGE       cilium pre flight check   0 1     1            0           12s          kubectl logs  n kube system deployment cilium pre flight check  c cnp validator   previous       level info msg  Setting up kubernetes client        level info msg  Establishing connection to apiserver  host  https   172 20 0 1 443  subsys k8s       level info msg  Connected to apiserver  subsys k8s       level info msg  Validating CiliumNetworkPolicy  default cidr rule   OK        level error msg  Validating CiliumNetworkPolicy  default cnp update   unexpected validation error  spec labels  Invalid value    string    spec labels in body must be of type object    string          level error msg  Found invalid CiliumNetworkPolicy   In this example  we can see the   CiliumNetworkPolicy   in the   default   namespace with the name   cnp update   is not valid for the Cilium version we are trying to upgrade  In order to fix this policy we need to edit it  we can do this by saving the policy locally and modify it  For this example it seems the    spec labels   has set an array of strings which is not correct as per the official schema      code block   shell session          kubectl get cnp  n default cnp update  o yaml   cnp bad yaml         cat cnp bad yaml         apiVersion  cilium io v2         kind  CiliumNetworkPolicy                       spec            endpointSelector              matchLabels                id  app1           ingress              fromEndpoints                matchLabels                  id  app2             toPorts                ports                  port   80                  protocol  TCP           labels              custom true                To fix this policy we need to set the    spec labels   with the right format and commit these changes into Kubernetes      code block   shell session          cat cnp bad yaml         apiVersion  cilium io v2         kind  CiliumNetworkPolicy                       spec            endpointSelector              matchLabels                id  app1           ingress              fromEndpoints                matchLabels                  id  app2             toPorts                ports                  port   80                  protocol  TCP           labels              key   custom              value   true                                kubectl apply  f   cnp bad yaml  After applying the fixed policy we can delete the pod that was validating the policies so that Kubernetes creates a new pod immediately to verify if the fixed policies are now valid      code block   shell session          kubectl delete pod  n kube system  l k8s app cilium pre flight check deployment       pod  cilium pre flight check 86dfb69668 ngbql  deleted         kubectl get deployment  n kube system cilium pre flight check       NAME                      READY   UP TO DATE   AVAILABLE   AGE       cilium pre flight check   1 1     1            1           55m         kubectl logs  n kube system deployment cilium pre flight check  c cnp validator       level info msg  Setting up kubernetes client        level info msg  Establishing connection to apiserver  host  https   172 20 0 1 443  subsys k8s       level info msg  Connected to apiserver  subsys k8s       level info msg  Validating CiliumNetworkPolicy  default cidr rule   OK        level info msg  Validating CiliumNetworkPolicy  default cnp update   OK        level info msg  All CCNPs and CNPs valid    Once they are valid you can continue with the upgrade process   ref  cleanup preflight check