cilium AWS VPC CNI plugin docs cilium io plugin In this hybrid mode the AWS VPC CNI plugin is responsible for setting This guide explains how to set up Cilium in combination with the AWS VPC CNI chainingawscni

Answer:
.. only:: not (epub or latex or html)

    WARNING: You are looking at unreleased Cilium documentation.
    Please use the official rendered version released here:
    https://docs.cilium.io

.. _chaining_aws_cni:

******************
AWS VPC CNI plugin
******************

This guide explains how to set up Cilium in combination with the AWS VPC CNI
plugin. In this hybrid mode, the AWS VPC CNI plugin is responsible for setting
up the virtual network devices as well as for IP address management (IPAM) via
ENIs. After the initial networking is setup for a given pod, the Cilium CNI
plugin is called to attach eBPF programs to the network devices set up by the
AWS VPC CNI plugin in order to enforce network policies, perform load-balancing
and provide encryption.

.. image:: aws-cilium-architecture.png

.. include:: cni-chaining-limitations.rst

.. admonition:: Video
  :class: attention

  If you require advanced features of Cilium, consider migrating fully to Cilium.
  To help you with the process, you can watch two Principal Engineers at Meltwater talk about `how they migrated
  Meltwater's production Kubernetes clusters - from the AWS VPC CNI plugin to Cilium <https://www.youtube.com/watch?v=w6S6baRHHu8&list=PLDg_GiBbAx-kDXqDYimwytMLh2kAHyMPd&t=182s>`__.

.. important::

   Please ensure that you are running version `1.11.2 <https://github.com/aws/amazon-vpc-cni-k8s/releases/tag/v1.11.2>`_
   or newer of the AWS VPC CNI plugin to guarantee compatibility with Cilium.

   .. code-block:: shell-session

      $ kubectl -n kube-system get ds/aws-node -o json | jq -r '.spec.template.spec.containers[0].image'
      602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.11.2

   If you are running an older version, as in the above example, you can upgrade it with:

   .. code-block:: shell-session

      $ kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.11/config/master/aws-k8s-cni.yaml

.. image:: aws-cni-architecture.png

Setting up a cluster on AWS
===========================

Follow the instructions in the :ref:`k8s_install_quick` guide to set up an EKS
cluster, or use any other method of your preference to set up a Kubernetes
cluster on AWS.

Ensure that the `aws-vpc-cni-k8s <https://github.com/aws/amazon-vpc-cni-k8s>`_
plugin is installed — which will already be the case if you have created an EKS
cluster. Also, ensure the version of the plugin is up-to-date as per the above.

.. include:: k8s-install-download-release.rst

Deploy Cilium via Helm:

.. parsed-literal::

   helm install cilium |CHART_RELEASE| \\
     --namespace kube-system \\
     --set cni.chainingMode=aws-cni \\
     --set cni.exclusive=false \\
     --set enableIPv4Masquerade=false \\
     --set routingMode=native \\
     --set endpointRoutes.enabled=true

This will enable chaining with the AWS VPC CNI plugin. It will also disable
tunneling, as it's not required since ENI IP addresses can be directly routed
in the VPC. For the same reason, masquerading can be disabled as well.

Restart existing pods
=====================

The new CNI chaining configuration *will not* apply to any pod that is already
running in the cluster. Existing pods will be reachable, and Cilium will
load-balance *to* them, but not *from* them. Policy enforcement will also not
be applied. For these reasons, you must restart these pods so that the chaining
configuration can be applied to them.

The following command can be used to check which pods need to be restarted:

.. code-block:: bash

   for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
        ceps=$(kubectl -n "${ns}" get cep \
            -o jsonpath='{.items[*].metadata.name}')
        pods=$(kubectl -n "${ns}" get pod \
            -o custom-columns=NAME:.metadata.name,NETWORK:.spec.hostNetwork \
            | grep -E '\s(<none>|false)' | awk '{print $1}' | tr '\n' ' ')
        ncep=$(echo "${pods} ${ceps}" | tr ' ' '\n' | sort | uniq -u | paste -s -d ' ' -)
        for pod in $(echo $ncep); do
          echo "${ns}/${pod}";
        done
   done

.. include:: k8s-install-validate.rst

Advanced
========

Enabling security groups for pods (EKS)
---------------------------------------

Cilium can be used alongside the `security groups for pods <https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html>`_
feature of EKS in supported clusters when running in chaining mode. Follow the
instructions below to enable this feature:

.. important::

   The following guide requires `jq <https://stedolan.github.io/jq/>`_ and the
   `AWS CLI <https://aws.amazon.com/cli/>`_ to be installed and configured.

Make sure that the ``AmazonEKSVPCResourceController`` managed policy is attached
to the IAM role associated with the EKS cluster:

.. code-block:: shell-session

   export EKS_CLUSTER_NAME="my-eks-cluster" # Change accordingly
   export EKS_CLUSTER_ROLE_NAME=$(aws eks describe-cluster \
        --name "${EKS_CLUSTER_NAME}" \
        | jq -r '.cluster.roleArn' | awk -F/ '{print $NF}')
   aws iam attach-role-policy \
        --policy-arn arn:aws:iam::aws:policy/AmazonEKSVPCResourceController \
        --role-name "${EKS_CLUSTER_ROLE_NAME}"

Then, as mentioned above, make sure that the version of the AWS VPC CNI
plugin running in the cluster is up-to-date:

.. code-block:: shell-session

   kubectl -n kube-system get ds/aws-node \
     -o jsonpath='{.spec.template.spec.containers[0].image}'
   602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.7.10

Next, patch the ``kube-system/aws-node`` DaemonSet in order to enable security
groups for pods:

.. code-block:: shell-session

   kubectl -n kube-system patch ds aws-node \
     -p '{"spec":{"template":{"spec":{"initContainers":[{"env":[{"name":"DISABLE_TCP_EARLY_DEMUX","value":"true"}],"name":"aws-vpc-cni-init"}],"containers":[{"env":[{"name":"ENABLE_POD_ENI","value":"true"}],"name":"aws-node"}]}}}}'
   kubectl -n kube-system rollout status ds aws-node

After the rollout is complete, all nodes in the cluster should have the ``vps.amazonaws.com/has-trunk-attached`` label set to ``true``:

.. code-block:: shell-session

   kubectl get nodes -L vpc.amazonaws.com/has-trunk-attached
   NAME                                            STATUS   ROLES    AGE   VERSION              HAS-TRUNK-ATTACHED
   ip-192-168-111-169.eu-west-2.compute.internal   Ready    <none>   22m   v1.19.6-eks-49a6c0   true
   ip-192-168-129-175.eu-west-2.compute.internal   Ready    <none>   22m   v1.19.6-eks-49a6c0   true

From this moment everything should be in place. For details on how to actually
associate security groups to pods, please refer to the `official documentation <https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html>`_.

.. include:: next-steps.rst

only   not  epub or latex or html       WARNING  You are looking at unreleased Cilium documentation      Please use the official rendered version released here      https   docs cilium io      chaining aws cni                      AWS VPC CNI plugin                     This guide explains how to set up Cilium in combination with the AWS VPC CNI plugin  In this hybrid mode  the AWS VPC CNI plugin is responsible for setting up the virtual network devices as well as for IP address management  IPAM  via ENIs  After the initial networking is setup for a given pod  the Cilium CNI plugin is called to attach eBPF programs to the network devices set up by the AWS VPC CNI plugin in order to enforce network policies  perform load balancing and provide encryption      image   aws cilium architecture png     include   cni chaining limitations rst     admonition   Video    class  attention    If you require advanced features of Cilium  consider migrating fully to Cilium    To help you with the process  you can watch two Principal Engineers at Meltwater talk about  how they migrated   Meltwater s production Kubernetes clusters   from the AWS VPC CNI plugin to Cilium  https   www youtube com watch v w6S6baRHHu8 list PLDg GiBbAx kDXqDYimwytMLh2kAHyMPd t 182s          important       Please ensure that you are running version  1 11 2  https   github com aws amazon vpc cni k8s releases tag v1 11 2       or newer of the AWS VPC CNI plugin to guarantee compatibility with Cilium         code block   shell session          kubectl  n kube system get ds aws node  o json   jq  r   spec template spec containers 0  image        602401143452 dkr ecr us west 2 amazonaws com amazon k8s cni v1 11 2     If you are running an older version  as in the above example  you can upgrade it with         code block   shell session          kubectl apply  f https   raw githubusercontent com aws amazon vpc cni k8s release 1 11 config master aws k8s cni yaml     image   aws cni architecture png   Setting up a cluster on AWS                              Follow the instructions in the  ref  k8s install quick  guide to set up an EKS cluster  or use any other method of your preference to set up a Kubernetes cluster on AWS   Ensure that the  aws vpc cni k8s  https   github com aws amazon vpc cni k8s    plugin is installed   which will already be the case if you have created an EKS cluster  Also  ensure the version of the plugin is up to date as per the above      include   k8s install download release rst  Deploy Cilium via Helm      parsed literal       helm install cilium  CHART RELEASE            namespace kube system           set cni chainingMode aws cni           set cni exclusive false           set enableIPv4Masquerade false           set routingMode native           set endpointRoutes enabled true  This will enable chaining with the AWS VPC CNI plugin  It will also disable tunneling  as it s not required since ENI IP addresses can be directly routed in the VPC  For the same reason  masquerading can be disabled as well   Restart existing pods                        The new CNI chaining configuration  will not  apply to any pod that is already running in the cluster  Existing pods will be reachable  and Cilium will load balance  to  them  but not  from  them  Policy enforcement will also not be applied  For these reasons  you must restart these pods so that the chaining configuration can be applied to them   The following command can be used to check which pods need to be restarted      code block   bash     for ns in   kubectl get ns  o jsonpath    items    metadata name     do         ceps   kubectl  n    ns   get cep                o jsonpath    items    metadata name            pods   kubectl  n    ns   get pod                o custom columns NAME  metadata name NETWORK  spec hostNetwork                 grep  E   s  none  false     awk   print  1     tr   n               ncep   echo    pods    ceps     tr       n    sort   uniq  u   paste  s  d                for pod in   echo  ncep   do           echo    ns    pod            done    done     include   k8s install validate rst  Advanced           Enabling security groups for pods  EKS                                           Cilium can be used alongside the  security groups for pods  https   docs aws amazon com eks latest userguide security groups for pods html    feature of EKS in supported clusters when running in chaining mode  Follow the instructions below to enable this feature      important       The following guide requires  jq  https   stedolan github io jq     and the     AWS CLI  https   aws amazon com cli     to be installed and configured   Make sure that the   AmazonEKSVPCResourceController   managed policy is attached to the IAM role associated with the EKS cluster      code block   shell session     export EKS CLUSTER NAME  my eks cluster    Change accordingly    export EKS CLUSTER ROLE NAME   aws eks describe cluster             name    EKS CLUSTER NAME               jq  r   cluster roleArn    awk  F    print  NF       aws iam attach role policy             policy arn arn aws iam  aws policy AmazonEKSVPCResourceController             role name    EKS CLUSTER ROLE NAME    Then  as mentioned above  make sure that the version of the AWS VPC CNI plugin running in the cluster is up to date      code block   shell session     kubectl  n kube system get ds aws node         o jsonpath    spec template spec containers 0  image      602401143452 dkr ecr us west 2 amazonaws com amazon k8s cni v1 7 10  Next  patch the   kube system aws node   DaemonSet in order to enable security groups for pods      code block   shell session     kubectl  n kube system patch ds aws node         p    spec    template    spec    initContainers     env     name   DISABLE TCP EARLY DEMUX   value   true     name   aws vpc cni init     containers     env     name   ENABLE POD ENI   value   true     name   aws node            kubectl  n kube system rollout status ds aws node  After the rollout is complete  all nodes in the cluster should have the   vps amazonaws com has trunk attached   label set to   true        code block   shell session     kubectl get nodes  L vpc amazonaws com has trunk attached    NAME                                            STATUS   ROLES    AGE   VERSION              HAS TRUNK ATTACHED    ip 192 168 111 169 eu west 2 compute internal   Ready     none    22m   v1 19 6 eks 49a6c0   true    ip 192 168 129 175 eu west 2 compute internal   Ready     none    22m   v1 19 6 eks 49a6c0   true  From this moment everything should be in place  For details on how to actually associate security groups to pods  please refer to the  official documentation  https   docs aws amazon com eks latest userguide security groups for pods html         include   next steps rst


cilium Migrating a cluster to Cilium docs cilium io be migrated on a node by node basis without disrupting existing traffic cnimigration Cilium can be used to migrate from another cni Running clusters can

Answer:
.. only:: not (epub or latex or html)

    WARNING: You are looking at unreleased Cilium documentation.
    Please use the official rendered version released here:
    https://docs.cilium.io

.. _cni_migration:

*************************************
Migrating a cluster to Cilium
*************************************

Cilium can be used to migrate from another cni. Running clusters can
be migrated on a node-by-node basis, without disrupting existing traffic
or requiring a complete cluster outage or rebuild depending on the complexity of the migration case.

This document outlines how migrations with Cilium work. You will have a good
understanding of the basic requirements, as well as see an example migration
which you can practice using :ref:`Kind <gs_kind>`.

Background
==========

When the kubelet creates a Pod's Sandbox, the installed CNI, as configured in ``/etc/cni/net.d/``,
is called. The cni will handle the networking for a pod - including allocating
an ip address, creating & configuring a network interface, and (potentially)
establishing an overlay network. The Pod's network configuration shares the
same life cycle as the PodSandbox.

In the case of migration, we typically reconfigure ``/etc/cni/net.d/`` to point
to Cilium. However, any existing pods will still have been configured by the old
network plugin and any new pods will be configured by the newer CNI. To complete
the migration all Pods on the cluster that are configured by the old cni must be
recycled in order to be a member of the new CNI.

A naive approach to migrating a CNI would be to reconfigure all nodes with a new
CNI and then gradually restart each node in the cluster, thus replacing the CNI
when the node is brought back up and ensuring that all pods are part of the new CNI.

This simple migration, while effective, comes at the cost of disrupting cluster
connectivity during the rollout. Unmigrated and migrated nodes would be split in
to two "islands" of connectivity, and pods would be randomly unable to reach one-another
until the migration is complete.

Migration via dual overlays
---------------------------

Instead, Cilium supports a *hybrid* mode, where two separate overlays are established
across the cluster. While pods on a given node can only be attached to one network,
they have access to both Cilium and non-Cilium pods while the migration is
taking place. As long as Cilium and the existing networking provider use a separate
IP range, the Linux routing table takes care of separating traffic.

In this document we will discuss a model for live migrating between two deployed
CNI implementations. This will have the benefit of reducing downtime of nodes
and workloads and ensuring that workloads on both configured CNIs can communicate
during migration.

For live migration to work, Cilium will be installed with a separate
CIDR range and encapsulation port than that of the currently installed CNI. As
long as Cilium and the existing CNI use a separate IP range, the Linux
routing table takes care of separating traffic.

Requirements
============

Live migration requires the following:

- A new, distinct Cluster CIDR for Cilium to use
- Use of the :ref:`Cluster Pool IPAM mode<ipam_crd_cluster_pool>`
- A distinct overlay, either protocol or port
- An existing network plugin that uses the Linux routing stack, such as Flannel, Calico, or AWS-CNI

Limitations
===========

Currently, Cilium migration has not been tested with:

- BGP-based routing
- Changing IP families (e.g. from IPv4 to IPv6)
- Migrating from Cilium in chained mode
- An existing NetworkPolicy provider

During migration, Cilium's  NetworkPolicy and CiliumNetworkPolicy enforcement
will be disabled. Otherwise, traffic from non-Cilium pods may be incorrectly
dropped. Once the migration process is complete, policy enforcement can
be re-enabled. If there is an existing NetworkPolicy provider, you may wish to
temporarily delete all NetworkPolicies before proceeding.

It is strongly recommended to install Cilium using the :ref:`cluster-pool <ipam_crd_cluster_pool>`
IPAM allocator. This provides the strongest assurance that there will
be no IP collisions.

.. warning::
  Migration is highly dependent on the exact configuration of existing
  clusters. It is, thus, strongly recommended to perform a trial migration
  on a test or lab cluster.

Overview
========

The migration process utilizes the :ref:`per-node configuration<per-node-configuration>`
feature to selectively enable Cilium CNI. This allows for a controlled rollout
of Cilium without disrupting existing workloads.

Cilium will be installed, first, in a mode where it establishes an overlay
but does not provide CNI networking for any pods. Then, individual nodes will
be migrated.

In summary, the process looks like:

1. Install cilium in "secondary" mode
2. Cordon, drain, migrate, and reboot each node
3. Remove the existing network provider
4. (Optional) Reboot each node again

Migration procedure
===================

Preparation
-----------

- Optional: Create a :ref:`Kind <gs_kind>` cluster and install `Flannel <https://github.com/flannel-io/flannel>`_ on it.

    .. parsed-literal::

      $ cat <<EOF > kind-config.yaml
      apiVersion: kind.x-k8s.io/v1alpha4
      kind: Cluster
      nodes:
      - role: control-plane
      - role: worker
      - role: worker
      networking:
        disableDefaultCNI: true
      EOF
      $ kind create cluster --config=kind-config.yaml
      $ kubectl apply -n kube-system --server-side -f \ |SCM_WEB|\/examples/misc/migration/install-reference-cni-plugins.yaml
      $ kubectl apply --server-side -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
      $ kubectl wait --for=condition=Ready nodes --all

- Optional: Monitor connectivity.

  You may wish to install a tool such as `goldpinger <https://github.com/bloomberg/goldpinger>`_
  to detect any possible connectivity issues.

1.  Select a **new** CIDR for pods. It must be distinct from all other CIDRs in use.

    For Kind clusters, the default is ``10.244.0.0/16``. So, for this example, we will
    use ``10.245.0.0/16``.

2.  Select a **distinct** encapsulation port. For example, if the existing cluster
    is using VXLAN, then you should either use GENEVE or configure Cilium to use VXLAN
    with a different port.

    For this example, we will use VXLAN with a non-default port of 8473.

3.  Create a helm ``values-migration.yaml`` file based on the following example. Be sure to fill
    in the CIDR you selected in step 1.

    .. code-block:: yaml

        operator:
          unmanagedPodWatcher:
            restart: false # Migration: Don't restart unmigrated pods
        routingMode: tunnel # Migration: Optional: default is tunneling, configure as needed
        tunnelProtocol: vxlan # Migration: Optional: default is VXLAN, configure as needed
        tunnelPort: 8473 # Migration: Optional, change only if both networks use the same port by default
        cni:
          customConf: true # Migration: Don't install a CNI configuration file
          uninstall: false # Migration: Don't remove CNI configuration on shutdown
        ipam:
          mode: "cluster-pool"
          operator:
            clusterPoolIPv4PodCIDRList: ["10.245.0.0/16"] # Migration: Ensure this is distinct and unused
        policyEnforcementMode: "never" # Migration: Disable policy enforcement
        bpf:
          hostLegacyRouting: true # Migration: Allow for routing between Cilium and the existing overlay

4.  Configure any additional Cilium Helm values.

    Cilium supports a number of :ref:`Helm configuration options<helm_reference>`. You may choose to
    auto-detect typical ones using the :ref:`cilium-cli <install_cilium_cli>`.
    This will consume the template and auto-detect any other relevant Helm values.
    Review these values for your particular installation.

    .. parsed-literal::

        $ cilium install |CHART_VERSION| --values values-migration.yaml --dry-run-helm-values > values-initial.yaml
        $ cat values-initial.yaml

5.  Install cilium using :ref:`helm <k8s_install_helm>`.

    .. code-block:: shell-session

      $ helm repo add cilium https://helm.cilium.io/
      $ helm install cilium cilium/cilium --namespace kube-system --values values-initial.yaml

    At this point, you should have a cluster with Cilium installed and an overlay established, but no
    pods managed by Cilium itself. You can verify this with the ``cilium`` command.

    .. code-block:: shell-session

      $ cilium status --wait
      ...
      Cluster Pods:     0/3 managed by Cilium

6.  Create a :ref:`per-node config<per-node-configuration>` that will instruct Cilium to "take over" CNI networking
    on the node. Initially, this will apply to no nodes; you will roll it out gradually via
    the migration process.

    .. code-block:: shell-session

        cat <<EOF | kubectl apply --server-side -f -
        apiVersion: cilium.io/v2
        kind: CiliumNodeConfig
        metadata:
          namespace: kube-system
          name: cilium-default
        spec:
          nodeSelector:
            matchLabels:
              io.cilium.migration/cilium-default: "true"
          defaults:
            write-cni-conf-when-ready: /host/etc/cni/net.d/05-cilium.conflist
            custom-cni-conf: "false"
            cni-chaining-mode: "none"
            cni-exclusive: "true"
        EOF

Migration
---------

At this point, you are ready to begin the migration process. The basic flow is:

Select a node to be migrated. It is not recommended to start with a control-plane node.

.. code-block:: shell-session

  $ NODE="kind-worker" # for the Kind example

1.  Cordon and, optionally, drain the node in question.

    .. code-block:: shell-session

      $ kubectl cordon $NODE
      $ kubectl drain --ignore-daemonsets $NODE

    Draining is not strictly required, but it is recommended. Otherwise pods will encounter
    a brief interruption while the node is rebooted.

2.  Label the node. This causes the ``CiliumNodeConfig`` to apply to this node.

    .. code-block:: shell-session

      $ kubectl label node $NODE --overwrite "io.cilium.migration/cilium-default=true"

3.  Restart Cilium. This will cause it to write its CNI configuration file.

    .. code-block:: shell-session

      $ kubectl -n kube-system delete pod --field-selector spec.nodeName=$NODE -l k8s-app=cilium
      $ kubectl -n kube-system rollout status ds/cilium -w

4.  Reboot the node.

    If using kind, do so with docker:

    .. code-block:: shell-session

      docker restart $NODE

5.  Validate that the node has been successfully migrated.

    .. code-block:: shell-session

      $ cilium status --wait
      $ kubectl get -o wide node $NODE
      $ kubectl -n kube-system run --attach --rm --restart=Never verify-network \
        --overrides='{"spec": {"nodeName": "'$NODE'", "tolerations": [{"operator": "Exists"}]}}' \
        --image ghcr.io/nicolaka/netshoot:v0.8 -- /bin/bash -c 'ip -br addr && curl -s -k https://$KUBERNETES_SERVICE_HOST/healthz && echo'

    Ensure the IP address of the pod is in the Cilium CIDR(s) supplied above and that the apiserver
    is reachable.

6.  Uncordon the node.

    .. code-block:: shell-session

      $ kubectl uncordon $NODE

Once you are satisfied everything has been migrated successfully, select another unmigrated node in the cluster
and repeat these steps.

Post-migration
--------------

Perform these steps once the cluster is fully migrated.

1.  Ensure Cilium is healthy and that all pods have been migrated:

    .. code-block:: shell-session

      $ cilium status

2.  Update the Cilium configuration:

    - Cilium should be the primary CNI
    - NetworkPolicy should be enforced
    - The Operator can restart unmanaged pods
    - **Optional**: use :ref:`eBPF_Host_Routing`. Enabling this will cause a short connectivity
      interruption on each node as the daemon restarts, but improves networking performance.

    You can do this manually, or via the ``cilium`` tool (this will not apply changes to the cluster):

    .. parsed-literal::

      $ cilium install |CHART_VERSION| --values values-initial.yaml --dry-run-helm-values \
        --set operator.unmanagedPodWatcher.restart=true --set cni.customConf=false \
        --set policyEnforcementMode=default \
        --set bpf.hostLegacyRouting=false > values-final.yaml # optional, can cause brief interruptions
      $ diff values-initial.yaml values-final.yaml

    Then, apply the changes to the cluster:

    .. code-block:: shell-session

      $ helm upgrade --namespace kube-system cilium cilium/cilium --values values-final.yaml
      $ kubectl -n kube-system rollout restart daemonset cilium
      $ cilium status --wait

3.  Delete the per-node configuration:

    .. code-block:: shell-session

      $ kubectl delete -n kube-system ciliumnodeconfig cilium-default

4.  Delete the previous network plugin.

    At this point, all pods should be using Cilium for networking. You can easily verify this with ``cilium status``.
    It is now safe to delete the previous network plugin from the cluster.

    Most network plugins leave behind some resources, e.g. iptables rules and interfaces. These will be
    cleaned up when the node next reboots. If desired, you may perform a rolling reboot again.

only   not  epub or latex or html       WARNING  You are looking at unreleased Cilium documentation      Please use the official rendered version released here      https   docs cilium io      cni migration                                         Migrating a cluster to Cilium                                        Cilium can be used to migrate from another cni  Running clusters can be migrated on a node by node basis  without disrupting existing traffic or requiring a complete cluster outage or rebuild depending on the complexity of the migration case   This document outlines how migrations with Cilium work  You will have a good understanding of the basic requirements  as well as see an example migration which you can practice using  ref  Kind  gs kind      Background             When the kubelet creates a Pod s Sandbox  the installed CNI  as configured in    etc cni net d     is called  The cni will handle the networking for a pod   including allocating  an ip address  creating   configuring a network interface  and  potentially  establishing an overlay network  The Pod s network configuration shares the same life cycle as the PodSandbox   In the case of migration  we typically reconfigure    etc cni net d    to point to Cilium  However  any existing pods will still have been configured by the old network plugin and any new pods will be configured by the newer CNI  To complete the migration all Pods on the cluster that are configured by the old cni must be recycled in order to be a member of the new CNI   A naive approach to migrating a CNI would be to reconfigure all nodes with a new CNI and then gradually restart each node in the cluster  thus replacing the CNI when the node is brought back up and ensuring that all pods are part of the new CNI   This simple migration  while effective  comes at the cost of disrupting cluster connectivity during the rollout  Unmigrated and migrated nodes would be split in to two  islands  of connectivity  and pods would be randomly unable to reach one another until the migration is complete   Migration via dual overlays                              Instead  Cilium supports a  hybrid  mode  where two separate overlays are established across the cluster  While pods on a given node can only be attached to one network  they have access to both Cilium and non Cilium pods while the migration is taking place  As long as Cilium and the existing networking provider use a separate IP range  the Linux routing table takes care of separating traffic   In this document we will discuss a model for live migrating between two deployed CNI implementations  This will have the benefit of reducing downtime of nodes and workloads and ensuring that workloads on both configured CNIs can communicate during migration   For live migration to work  Cilium will be installed with a separate CIDR range and encapsulation port than that of the currently installed CNI  As long as Cilium and the existing CNI use a separate IP range  the Linux  routing table takes care of separating traffic     Requirements               Live migration requires the following     A new  distinct Cluster CIDR for Cilium to use   Use of the  ref  Cluster Pool IPAM mode ipam crd cluster pool     A distinct overlay  either protocol or port   An existing network plugin that uses the Linux routing stack  such as Flannel  Calico  or AWS CNI  Limitations              Currently  Cilium migration has not been tested with     BGP based routing   Changing IP families  e g  from IPv4 to IPv6    Migrating from Cilium in chained mode   An existing NetworkPolicy provider  During migration  Cilium s  NetworkPolicy and CiliumNetworkPolicy enforcement  will be disabled  Otherwise  traffic from non Cilium pods may be incorrectly dropped  Once the migration process is complete  policy enforcement can be re enabled  If there is an existing NetworkPolicy provider  you may wish to temporarily delete all NetworkPolicies before proceeding   It is strongly recommended to install Cilium using the  ref  cluster pool  ipam crd cluster pool   IPAM allocator  This provides the strongest assurance that there will be no IP collisions      warning     Migration is highly dependent on the exact configuration of existing   clusters  It is  thus  strongly recommended to perform a trial migration   on a test or lab cluster   Overview           The migration process utilizes the  ref  per node configuration per node configuration   feature to selectively enable Cilium CNI  This allows for a controlled rollout of Cilium without disrupting existing workloads   Cilium will be installed  first  in a mode where it establishes an overlay but does not provide CNI networking for any pods  Then  individual nodes will be migrated   In summary  the process looks like   1  Install cilium in  secondary  mode 2  Cordon  drain  migrate  and reboot each node 3  Remove the existing network provider 4   Optional  Reboot each node again   Migration procedure                      Preparation                Optional  Create a  ref  Kind  gs kind   cluster and install  Flannel  https   github com flannel io flannel    on it          parsed literal            cat   EOF   kind config yaml       apiVersion  kind x k8s io v1alpha4       kind  Cluster       nodes          role  control plane         role  worker         role  worker       networking          disableDefaultCNI  true       EOF         kind create cluster   config kind config yaml         kubectl apply  n kube system   server side  f    SCM WEB   examples misc migration install reference cni plugins yaml         kubectl apply   server side  f https   github com flannel io flannel releases latest download kube flannel yml         kubectl wait   for condition Ready nodes   all    Optional  Monitor connectivity     You may wish to install a tool such as  goldpinger  https   github com bloomberg goldpinger      to detect any possible connectivity issues   1   Select a   new   CIDR for pods  It must be distinct from all other CIDRs in use       For Kind clusters  the default is   10 244 0 0 16    So  for this example  we will     use   10 245 0 0 16     2   Select a   distinct   encapsulation port  For example  if the existing cluster     is using VXLAN  then you should either use GENEVE or configure Cilium to use VXLAN     with a different port       For this example  we will use VXLAN with a non default port of 8473   3   Create a helm   values migration yaml   file based on the following example  Be sure to fill     in the CIDR you selected in step 1          code block   yaml          operator            unmanagedPodWatcher              restart  false   Migration  Don t restart unmigrated pods         routingMode  tunnel   Migration  Optional  default is tunneling  configure as needed         tunnelProtocol  vxlan   Migration  Optional  default is VXLAN  configure as needed         tunnelPort  8473   Migration  Optional  change only if both networks use the same port by default         cni            customConf  true   Migration  Don t install a CNI configuration file           uninstall  false   Migration  Don t remove CNI configuration on shutdown         ipam            mode   cluster pool            operator              clusterPoolIPv4PodCIDRList    10 245 0 0 16     Migration  Ensure this is distinct and unused         policyEnforcementMode   never    Migration  Disable policy enforcement         bpf            hostLegacyRouting  true   Migration  Allow for routing between Cilium and the existing overlay  4   Configure any additional Cilium Helm values       Cilium supports a number of  ref  Helm configuration options helm reference    You may choose to     auto detect typical ones using the  ref  cilium cli  install cilium cli        This will consume the template and auto detect any other relevant Helm values      Review these values for your particular installation          parsed literal              cilium install  CHART VERSION    values values migration yaml   dry run helm values   values initial yaml           cat values initial yaml   5   Install cilium using  ref  helm  k8s install helm            code block   shell session          helm repo add cilium https   helm cilium io          helm install cilium cilium cilium   namespace kube system   values values initial yaml       At this point  you should have a cluster with Cilium installed and an overlay established  but no     pods managed by Cilium itself  You can verify this with the   cilium   command          code block   shell session          cilium status   wait                 Cluster Pods      0 3 managed by Cilium   6   Create a  ref  per node config per node configuration   that will instruct Cilium to  take over  CNI networking     on the node  Initially  this will apply to no nodes  you will roll it out gradually via     the migration process          code block   shell session          cat   EOF   kubectl apply   server side  f           apiVersion  cilium io v2         kind  CiliumNodeConfig         metadata            namespace  kube system           name  cilium default         spec            nodeSelector              matchLabels                io cilium migration cilium default   true            defaults              write cni conf when ready   host etc cni net d 05 cilium conflist             custom cni conf   false              cni chaining mode   none              cni exclusive   true          EOF  Migration            At this point  you are ready to begin the migration process  The basic flow is   Select a node to be migrated  It is not recommended to start with a control plane node      code block   shell session      NODE  kind worker    for the Kind example  1   Cordon and  optionally  drain the node in question          code block   shell session          kubectl cordon  NODE         kubectl drain   ignore daemonsets  NODE      Draining is not strictly required  but it is recommended  Otherwise pods will encounter     a brief interruption while the node is rebooted   2   Label the node  This causes the   CiliumNodeConfig   to apply to this node          code block   shell session          kubectl label node  NODE   overwrite  io cilium migration cilium default true   3   Restart Cilium  This will cause it to write its CNI configuration file          code block   shell session          kubectl  n kube system delete pod   field selector spec nodeName  NODE  l k8s app cilium         kubectl  n kube system rollout status ds cilium  w  4   Reboot the node       If using kind  do so with docker          code block   shell session            docker restart  NODE  5   Validate that the node has been successfully migrated          code block   shell session          cilium status   wait         kubectl get  o wide node  NODE         kubectl  n kube system run   attach   rm   restart Never verify network             overrides    spec     nodeName      NODE     tolerations      operator    Exists                   image ghcr io nicolaka netshoot v0 8     bin bash  c  ip  br addr    curl  s  k https    KUBERNETES SERVICE HOST healthz    echo       Ensure the IP address of the pod is in the Cilium CIDR s  supplied above and that the apiserver     is reachable   6   Uncordon the node          code block   shell session          kubectl uncordon  NODE   Once you are satisfied everything has been migrated successfully  select another unmigrated node in the cluster and repeat these steps   Post migration                 Perform these steps once the cluster is fully migrated   1   Ensure Cilium is healthy and that all pods have been migrated          code block   shell session          cilium status  2   Update the Cilium configuration         Cilium should be the primary CNI       NetworkPolicy should be enforced       The Operator can restart unmanaged pods         Optional    use  ref  eBPF Host Routing   Enabling this will cause a short connectivity        interruption on each node as the daemon restarts  but improves networking performance       You can do this manually  or via the   cilium   tool  this will not apply changes to the cluster           parsed literal            cilium install  CHART VERSION    values values initial yaml   dry run helm values             set operator unmanagedPodWatcher restart true   set cni customConf false             set policyEnforcementMode default             set bpf hostLegacyRouting false   values final yaml   optional  can cause brief interruptions         diff values initial yaml values final yaml      Then  apply the changes to the cluster          code block   shell session          helm upgrade   namespace kube system cilium cilium cilium   values values final yaml         kubectl  n kube system rollout restart daemonset cilium         cilium status   wait  3   Delete the per node configuration          code block   shell session          kubectl delete  n kube system ciliumnodeconfig cilium default  4   Delete the previous network plugin       At this point  all pods should be using Cilium for networking  You can easily verify this with   cilium status        It is now safe to delete the previous network plugin from the cluster        Most network plugins leave behind some resources  e g  iptables rules and interfaces  These will be     cleaned up when the node next reboots  If desired  you may perform a rolling reboot again


cilium docs cilium io ranchermanagedrkeclusters Introduction Installation using Rancher

Answer:
.. only:: not (epub or latex or html)

    WARNING: You are looking at unreleased Cilium documentation.
    Please use the official rendered version released here:
    https://docs.cilium.io

.. _rancher_managed_rke_clusters:

**************************
Installation using Rancher
**************************

Introduction
============

If you're not using the Rancher Management Console/UI to install your clusters, head
over to the :ref:`installation guides for standalone RKE clusters <rke_install>`.

Rancher comes with `official support for Cilium <https://ranchermanager.docs.rancher.com/faq/container-network-interface-providers>`__.
For most Rancher users, that's the recommended way to use Cilium on Rancher-managed
clusters.

However, as Rancher is using a custom
``rke2-cilium`` `Helm chart <https://github.com/rancher/rke2-charts/tree/main-source/packages/rke2-cilium>`__
with independent release cycles, Cilium power-users might want to use an
out-of-band Cilium installation instead, based on the official
`Cilium Helm chart <https://github.com/cilium/charts>`__,
on top of their Rancher-managed RKE1/RKE2 downstream clusters.
This guide explains how to achieve this.

.. note::

    This guide only shows a step-by-step guide for Rancher-managed (**non-standalone**)
    **RKE2** clusters.

    However, for a legacy RKE1 cluster, it's even easier. You also need to edit
    the cluster YAML and change ``network.cni`` to ``none`` as described in the
    :ref:`RKE 1 standalone guide<rke1_cni_none>`, but there's no need to copy over
    a Control Plane node local KubeConfig manually. Luckily, Rancher allows access
    to RKE1 clusters in ``Updating`` state, which are not ready yet. Hence, there's
    no chicken-egg issue to resolve.

Prerequisites
=============

* Fully functioning `Rancher Version 2.x <https://ranchermanager.docs.rancher.com/>`__ instance
* At least one empty Linux VM, to be used as initial downstream "Custom Cluster" (Control Plane) node
* DNS record pointing to the Kubernetes API of the downstream "Custom Cluster" Control Plane node(s) or L4 load-balancer

Create a New Cluster
====================

In Rancher UI, navigate to the Cluster Management page. In the top right, click on the
``Create`` button to create a new cluster.

.. image:: images/rancher_add_cluster.png

On the Cluster creation page select to create a new ``Custom`` cluster:

.. image:: images/rancher_existing_nodes.png

When the ``Create Custom`` page opens, provide at least a name for the cluster.
Go through the other configuration options and configure the ones that are
relevant for your setup.

Next to the ``Cluster Options`` section click the box to ``Edit as YAML``.
The configuration for the cluster will open up in an editor in the window.

.. image:: images/rancher_edit_as_yaml.png

Within the ``Cluster`` CustomResource (``provisioning.cattle.io/v1``), the relevant
parts to change are ``spec.rkeConfig.machineGlobalConfig.cni``,
``spec.rkeConfig.machineGlobalConfig.tls-san``, and optionally
``spec.rkeConfig.chartValues.rke2-calico`` and
``spec.rkeConfig.machineGlobalConfig.disable-kube-proxy``:

.. image:: images/rancher_delete_network_plugin.png

It's required to add a DNS record, pointing to the Control Plane node IP(s)
or an L4 load-balancer in front of them, under
``spec.rkeConfig.machineGlobalConfig.tls-san``, as that's required to resolve
a chicken-egg issue further down the line.

Ensure that ``spec.rkeConfig.machineGlobalConfig.cni`` is set to ``none`` and
``spec.rkeConfig.machineGlobalConfig.tls-san`` lists the mentioned DNS record:

.. image:: images/rancher_network_plugin_none.png

Optionally, if ``spec.rkeConfig.chartValues.rke2-calico`` is not empty, remove the
full object as you won't deploy Rancher's default CNI. At the same time, change
``spec.rkeConfig.machineGlobalConfig.disable-kube-proxy`` to ``true`` in case you
want to run :ref:`Cilium without Kube-Proxy<kubeproxy-free>`.

Make any additional changes to the configuration that are appropriate for your
environment. When you are ready, click ``Create`` and Rancher will create the
cluster.

.. image:: images/rancher_cluster_state_provisioning.png

The cluster will stay in ``Updating`` state until you add nodes. Click on the cluster.
In the ``Registration`` tab you should see the generated ``Registation command`` you
need to run on the downstream cluster nodes.

Do not forget to select the correct node roles. Rancher comes with the default to
deploy all three roles (``etcd``, ``Control Plane``, and ``Worker``), which is often
not what you want for multi-node clusters.

.. image:: images/rancher_add_nodes.png

A few seconds after you added at least a single node, you should see the new node(s)
in the ``Machines`` tab. The machine will be stuck in ``Reconciling`` state and
won't become ``Active``:

.. image:: images/rancher_node_not_ready.png

That's expected as there's no CNI running on this cluster yet. Unfortunately, this also
means critical pods like ``rke2-coredns-rke2-coredns-*`` and ``cattle-cluster-agent-*``
are stuck in ``PENDING`` state. Hence, the downstream cluster is not yet able
to register itself on Rancher.

As a next step, you need to resolve this chicken-egg issue by directly accessing
the downstream cluster's Kubernetes API, without going via Rancher. Rancher will not allow
access to this downstream cluster, as it's still in ``Updating`` state. That's why you
can't use the downstream cluster's KubeConfig provided by the Rancher management console/UI.

Copy ``/etc/rancher/rke2/rke2.yaml`` from the first downstream cluster Control Plane
node to your jump/bastion host where you have ``helm`` installed and can access the
Cilium Helm charts.

.. code-block:: shell-session

    scp root@<cp-node-1-ip>:/etc/rancher/rke2/rke2.yaml .

Search and replace ``127.0.0.1`` (``clusters[0].cluster.server``) with the
already mentioned DNS record pointing to the Control Plane / L4 load-balancer IP(s).

.. code-block:: yaml

    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data: LS0...S0K
        server: https://127.0.0.1:6443
    name: default
    contexts: {}

Check if you can access the Kubernetes API:

.. code-block:: shell-session

    export KUBECONFIG=$(pwd)/my-cluster-kubeconfig.yaml
    kubectl get nodes
    NAME                    STATUS     ROLES                       AGE   VERSION
    rancher-demo-node       NotReady   control-plane,etcd,master   44m   v1.27.8+rke2r1

If successful, you can now install Cilium via Helm CLI:

.. parsed-literal::

    helm install cilium |CHART_RELEASE| \\
      --namespace kube-system \\
      -f my-cluster-cilium-values.yaml

After a few minutes, you should see that the node changed to the ``Ready`` status:

.. code-block:: shell-session

    kubectl get nodes
    NAME                    STATUS   ROLES                       AGE   VERSION
    rancher-demo-node       Ready    control-plane,etcd,master   48m   v1.27.8+rke2r1

Back in the Rancher UI, you should see that the cluster changed to the healthy
``Active`` status:

.. image:: images/rancher_my_cluster_active.png

That's it. You can now normally work with this cluster as if you
installed the CNI the default Rancher way. Additional nodes can now be added
straightaway and the "local Control Plane RKE2 KubeConfig" workaround
is not required anymore.

Optional: Add Cilium to Rancher Registries
==========================================

One small, optional convenience item would be to add the Cilium Helm repository
to Rancher so that, in the future, Cilium can easily be upgraded via Rancher UI.

You have two options available:

**Option 1**: Navigate to ``Cluster Management`` -> ``Advanced`` -> ``Repositories`` and
click the ``Create`` button:

.. image:: images/rancher_add_repository.png

**Option 2**: Alternatively, you can also just add the Cilium Helm repository
on a single cluster by navigating to ``<your-cluster>`` -> ``Apps`` -> ``Repositories``:

.. image:: images/rancher_add_repository_cluster.png

For either option, in the window that opens, add the official Cilium Helm chart
repository (``https://helm.cilium.io``) to the Rancher repository list:

.. image:: images/rancher_add_cilium_repository.png

Once added, you should see the Cilium repository in the repositories list:

.. image:: images/rancher_repositories_list_success.png

If you now head to ``<your-cluster>`` -> ``Apps`` -> ``Installed Apps``, you
should see the ``cilium`` app. Ensure ``All Namespaces`` or
``Project: System -> kube-system`` is selected at the top of the page.

.. image:: images/rancher_cluster_cilium_app.png

Since you added the Cilium repository, you will now see a small hint on this app entry
when there's a new Cilium version released. You can then upgrade directly via Rancher UI.

.. image:: images/rancher_cluster_cilium_app_upgrade.png

.. image:: images/rancher_cluster_cilium_app_upgrade_version.pn

only   not  epub or latex or html       WARNING  You are looking at unreleased Cilium documentation      Please use the official rendered version released here      https   docs cilium io      rancher managed rke clusters                              Installation using Rancher                             Introduction               If you re not using the Rancher Management Console UI to install your clusters  head over to the  ref  installation guides for standalone RKE clusters  rke install     Rancher comes with  official support for Cilium  https   ranchermanager docs rancher com faq container network interface providers      For most Rancher users  that s the recommended way to use Cilium on Rancher managed clusters   However  as Rancher is using a custom   rke2 cilium    Helm chart  https   github com rancher rke2 charts tree main source packages rke2 cilium     with independent release cycles  Cilium power users might want to use an out of band Cilium installation instead  based on the official  Cilium Helm chart  https   github com cilium charts      on top of their Rancher managed RKE1 RKE2 downstream clusters  This guide explains how to achieve this      note        This guide only shows a step by step guide for Rancher managed    non standalone          RKE2   clusters       However  for a legacy RKE1 cluster  it s even easier  You also need to edit     the cluster YAML and change   network cni   to   none   as described in the      ref  RKE 1 standalone guide rke1 cni none    but there s no need to copy over     a Control Plane node local KubeConfig manually  Luckily  Rancher allows access     to RKE1 clusters in   Updating   state  which are not ready yet  Hence  there s     no chicken egg issue to resolve   Prerequisites                  Fully functioning  Rancher Version 2 x  https   ranchermanager docs rancher com      instance   At least one empty Linux VM  to be used as initial downstream  Custom Cluster   Control Plane  node   DNS record pointing to the Kubernetes API of the downstream  Custom Cluster  Control Plane node s  or L4 load balancer  Create a New Cluster                       In Rancher UI  navigate to the Cluster Management page  In the top right  click on the   Create   button to create a new cluster      image   images rancher add cluster png  On the Cluster creation page select to create a new   Custom   cluster      image   images rancher existing nodes png  When the   Create Custom   page opens  provide at least a name for the cluster  Go through the other configuration options and configure the ones that are relevant for your setup   Next to the   Cluster Options   section click the box to   Edit as YAML    The configuration for the cluster will open up in an editor in the window      image   images rancher edit as yaml png  Within the   Cluster   CustomResource    provisioning cattle io v1     the relevant parts to change are   spec rkeConfig machineGlobalConfig cni      spec rkeConfig machineGlobalConfig tls san    and optionally   spec rkeConfig chartValues rke2 calico   and   spec rkeConfig machineGlobalConfig disable kube proxy        image   images rancher delete network plugin png  It s required to add a DNS record  pointing to the Control Plane node IP s  or an L4 load balancer in front of them  under   spec rkeConfig machineGlobalConfig tls san    as that s required to resolve a chicken egg issue further down the line   Ensure that   spec rkeConfig machineGlobalConfig cni   is set to   none   and   spec rkeConfig machineGlobalConfig tls san   lists the mentioned DNS record      image   images rancher network plugin none png  Optionally  if   spec rkeConfig chartValues rke2 calico   is not empty  remove the full object as you won t deploy Rancher s default CNI  At the same time  change   spec rkeConfig machineGlobalConfig disable kube proxy   to   true   in case you want to run  ref  Cilium without Kube Proxy kubeproxy free     Make any additional changes to the configuration that are appropriate for your environment  When you are ready  click   Create   and Rancher will create the cluster      image   images rancher cluster state provisioning png  The cluster will stay in   Updating   state until you add nodes  Click on the cluster  In the   Registration   tab you should see the generated   Registation command   you need to run on the downstream cluster nodes   Do not forget to select the correct node roles  Rancher comes with the default to deploy all three roles    etcd      Control Plane    and   Worker     which is often not what you want for multi node clusters      image   images rancher add nodes png  A few seconds after you added at least a single node  you should see the new node s  in the   Machines   tab  The machine will be stuck in   Reconciling   state and won t become   Active        image   images rancher node not ready png  That s expected as there s no CNI running on this cluster yet  Unfortunately  this also means critical pods like   rke2 coredns rke2 coredns     and   cattle cluster agent      are stuck in   PENDING   state  Hence  the downstream cluster is not yet able to register itself on Rancher   As a next step  you need to resolve this chicken egg issue by directly accessing the downstream cluster s Kubernetes API  without going via Rancher  Rancher will not allow access to this downstream cluster  as it s still in   Updating   state  That s why you can t use the downstream cluster s KubeConfig provided by the Rancher management console UI   Copy    etc rancher rke2 rke2 yaml   from the first downstream cluster Control Plane node to your jump bastion host where you have   helm   installed and can access the Cilium Helm charts      code block   shell session      scp root  cp node 1 ip   etc rancher rke2 rke2 yaml    Search and replace   127 0 0 1      clusters 0  cluster server    with the already mentioned DNS record pointing to the Control Plane   L4 load balancer IP s       code block   yaml      apiVersion  v1     clusters        cluster          certificate authority data  LS0   S0K         server  https   127 0 0 1 6443     name  default     contexts      Check if you can access the Kubernetes API      code block   shell session      export KUBECONFIG   pwd  my cluster kubeconfig yaml     kubectl get nodes     NAME                    STATUS     ROLES                       AGE   VERSION     rancher demo node       NotReady   control plane etcd master   44m   v1 27 8 rke2r1  If successful  you can now install Cilium via Helm CLI      parsed literal        helm install cilium  CHART RELEASE             namespace kube system           f my cluster cilium values yaml  After a few minutes  you should see that the node changed to the   Ready   status      code block   shell session      kubectl get nodes     NAME                    STATUS   ROLES                       AGE   VERSION     rancher demo node       Ready    control plane etcd master   48m   v1 27 8 rke2r1  Back in the Rancher UI  you should see that the cluster changed to the healthy   Active   status      image   images rancher my cluster active png  That s it  You can now normally work with this cluster as if you installed the CNI the default Rancher way  Additional nodes can now be added straightaway and the  local Control Plane RKE2 KubeConfig  workaround is not required anymore   Optional  Add Cilium to Rancher Registries                                             One small  optional convenience item would be to add the Cilium Helm repository to Rancher so that  in the future  Cilium can easily be upgraded via Rancher UI   You have two options available     Option 1    Navigate to   Cluster Management        Advanced        Repositories   and click the   Create   button      image   images rancher add repository png    Option 2    Alternatively  you can also just add the Cilium Helm repository on a single cluster by navigating to    your cluster         Apps        Repositories        image   images rancher add repository cluster png  For either option  in the window that opens  add the official Cilium Helm chart repository    https   helm cilium io    to the Rancher repository list      image   images rancher add cilium repository png  Once added  you should see the Cilium repository in the repositories list      image   images rancher repositories list success png  If you now head to    your cluster         Apps        Installed Apps    you should see the   cilium   app  Ensure   All Namespaces   or   Project  System    kube system   is selected at the top of the page      image   images rancher cluster cilium app png  Since you added the Cilium repository  you will now see a small hint on this app entry when there s a new Cilium version released  You can then upgrade directly via Rancher UI      image   images rancher cluster cilium app upgrade png     image   images rancher cluster cilium app upgrade version pn


cilium Hubble internals docs cilium io hubbleinternals This documentation section is targeted at developers who are interested in contributing to Hubble For this purpose it describes

Answer:
.. only:: not (epub or latex or html)

    WARNING: You are looking at unreleased Cilium documentation.
    Please use the official rendered version released here:
    https://docs.cilium.io

.. _hubble_internals:

****************
Hubble internals
****************

.. note:: This documentation section is targeted at developers who are
          interested in contributing to Hubble. For this purpose, it describes
          Hubble internals.

.. note:: This documentation covers the Hubble server (sometimes referred as
          "Hubble embedded") and Hubble Relay components but does not cover the
          Hubble UI and CLI.

Hubble builds on top of Cilium and eBPF to enable deep visibility into the
communication and behavior of services as well as the networking infrastructure
in a completely transparent manner. One of the design goals of Hubble is to
achieve all of this at large scale.

Hubble's server component is embedded into the Cilium agent in order to achieve
high performance with low-overhead. The gRPC services offered by Hubble server
may be consumed locally via a Unix domain socket or, more typically, through
Hubble Relay. Hubble Relay is a standalone component which is aware of all
Hubble instances and offers full cluster visibility by connecting to their
respective gRPC APIs. This capability is usually referred to as multi-node.
Hubble Relay's main goal is to offer a rich API that can be safely exposed and
consumed by the Hubble UI and CLI.

Hubble Architecture
===================

Hubble exposes gRPC services from the Cilium process that allows clients to
receive flows and other type of data.

Hubble server
-------------

The Hubble server component implements two gRPC services. The **Observer
service** which may optionally be exposed via a TCP socket in addition to a
local Unix domain socket and the **Peer service**, which is served on both
as well as being exposed as a Kubernetes Service when enabled via TCP.

The Observer service
^^^^^^^^^^^^^^^^^^^^

The Observer service is the principal service. It provides four RPC endpoints:
``GetFlows``, ``GetNodes``, ``GetNamespaces``  and ``ServerStatus``.

* ``GetNodes`` returns a list of metrics and other information related to each Hubble instance
* ``ServerStatus`` returns a summary the information in ``GetNodes``
* ``GetNamespaces`` returns a list of namespaces that had network flows within the last one hour
* ``GetFlows`` returns a stream of flow related events

Using ``GetFlows``, callers get a stream of payloads. Request parameters allow
callers to specify filters in the form of allow lists and deny lists to allow
for fine-grained filtering of data.

In order to answer ``GetFlows`` requests, Hubble stores monitoring events from
Cilium's event monitor into a user-space ring buffer structure. Monitoring
events are obtained by registering a new listener on Cilium monitor. The
ring buffer is capable of storing a configurable amount of events in memory.
Events are continuously consumed, overriding older ones once the ring buffer is
full.

Additionally, the Observer service also provides the ``GetAgentEvents`` and
``GetDebugEvents`` RPC endpoints to expose data about the Cilium agent events
and Cilium datapath debug events, respectively. Both are similar to ``GetFlows``
except they do not implement filtering capabilities.

.. image:: ./../images/hubble_getflows.png

For efficiency, the internal buffer length is a bit mask of ones + 1. The most
significant bit of this bit mask is the same position of the most significant
bit position of 'n'. In other terms, the internal buffer size is always a power
of 2 with 1 slot reserved for the writer. In effect, from a user perspective,
the ring buffer capacity is one less than a power of 2. As the ring buffer is a
hot code path, it has been designed to not employ any locking mechanisms and
uses atomic operations instead. While this approach has performance benefits,
it also has the downsides of being a complex component.

Due to its complex nature, the ring buffer is typically accessed via a ring
reader that abstracts the complexity of this data structure for reading. The
ring reader allows reading one event at the time with 'previous' and 'next'
methods but also implements a follow mode where events are continuously read as
they are written to the ring buffer.

The Peer service
^^^^^^^^^^^^^^^^

The Peer service sends information about Hubble peers in the cluster in a
stream. When the ``Notify`` method is called, it reports information about all
the peers in the cluster and subsequently sends information about peers that
are updated, added, or removed from the cluster. Thus, it allows the caller to
keep track of all Hubble instances and query their respective gRPC services.

This service is exposed as a Kubernetes Service and is primarily used by Hubble
Relay in order to have a cluster-wide view of all Hubble instances.

The Peer service obtains peer change notifications by subscribing to Cilium's
node manager. To this end, it internally defines a handler that implements
Cilium's datapath node handler interface.

.. _hubble_relay:

Hubble Relay
------------

Hubble Relay is the Hubble component that brings multi-node support. It
leverages the Peer service to obtain information about Hubble instances and
consume their gRPC API in order to provide a more rich API that covers events
from across the entire cluster (or even multiple clusters in a ClusterMesh
scenario).

Hubble Relay was first introduced as a technology preview with the release of
Cilium v1.8 and was declared stable with the release of Cilium v1.9.

Hubble Relay implements the Observer service for multi-node. To that end, it
maintains a persistent connection with every Hubble peer in a cluster with a
peer manager. This component provides callers with the list of peers. Callers
may report when a peer is unreachable, in which case the peer manager will
attempt to reconnect.

As Hubble Relay connects to every node in a cluster, the Hubble server
instances must make their API available (by default on port 4244). By default,
Hubble server endpoints are secured using mutual TLS (mTLS) when exposed on a
TCP port in order to limit access to Hubble Relay only.

only   not  epub or latex or html       WARNING  You are looking at unreleased Cilium documentation      Please use the official rendered version released here      https   docs cilium io      hubble internals                    Hubble internals                      note   This documentation section is targeted at developers who are           interested in contributing to Hubble  For this purpose  it describes           Hubble internals      note   This documentation covers the Hubble server  sometimes referred as            Hubble embedded   and Hubble Relay components but does not cover the           Hubble UI and CLI   Hubble builds on top of Cilium and eBPF to enable deep visibility into the communication and behavior of services as well as the networking infrastructure in a completely transparent manner  One of the design goals of Hubble is to achieve all of this at large scale   Hubble s server component is embedded into the Cilium agent in order to achieve high performance with low overhead  The gRPC services offered by Hubble server may be consumed locally via a Unix domain socket or  more typically  through Hubble Relay  Hubble Relay is a standalone component which is aware of all Hubble instances and offers full cluster visibility by connecting to their respective gRPC APIs  This capability is usually referred to as multi node  Hubble Relay s main goal is to offer a rich API that can be safely exposed and consumed by the Hubble UI and CLI   Hubble Architecture                      Hubble exposes gRPC services from the Cilium process that allows clients to receive flows and other type of data   Hubble server                The Hubble server component implements two gRPC services  The   Observer service   which may optionally be exposed via a TCP socket in addition to a local Unix domain socket and the   Peer service    which is served on both as well as being exposed as a Kubernetes Service when enabled via TCP   The Observer service                       The Observer service is the principal service  It provides four RPC endpoints    GetFlows      GetNodes      GetNamespaces    and   ServerStatus         GetNodes   returns a list of metrics and other information related to each Hubble instance     ServerStatus   returns a summary the information in   GetNodes       GetNamespaces   returns a list of namespaces that had network flows within the last one hour     GetFlows   returns a stream of flow related events  Using   GetFlows    callers get a stream of payloads  Request parameters allow callers to specify filters in the form of allow lists and deny lists to allow for fine grained filtering of data   In order to answer   GetFlows   requests  Hubble stores monitoring events from Cilium s event monitor into a user space ring buffer structure  Monitoring events are obtained by registering a new listener on Cilium monitor  The ring buffer is capable of storing a configurable amount of events in memory  Events are continuously consumed  overriding older ones once the ring buffer is full   Additionally  the Observer service also provides the   GetAgentEvents   and   GetDebugEvents   RPC endpoints to expose data about the Cilium agent events and Cilium datapath debug events  respectively  Both are similar to   GetFlows   except they do not implement filtering capabilities      image        images hubble getflows png  For efficiency  the internal buffer length is a bit mask of ones   1  The most significant bit of this bit mask is the same position of the most significant bit position of  n   In other terms  the internal buffer size is always a power of 2 with 1 slot reserved for the writer  In effect  from a user perspective  the ring buffer capacity is one less than a power of 2  As the ring buffer is a hot code path  it has been designed to not employ any locking mechanisms and uses atomic operations instead  While this approach has performance benefits  it also has the downsides of being a complex component   Due to its complex nature  the ring buffer is typically accessed via a ring reader that abstracts the complexity of this data structure for reading  The ring reader allows reading one event at the time with  previous  and  next  methods but also implements a follow mode where events are continuously read as they are written to the ring buffer   The Peer service                   The Peer service sends information about Hubble peers in the cluster in a stream  When the   Notify   method is called  it reports information about all the peers in the cluster and subsequently sends information about peers that are updated  added  or removed from the cluster  Thus  it allows the caller to keep track of all Hubble instances and query their respective gRPC services   This service is exposed as a Kubernetes Service and is primarily used by Hubble Relay in order to have a cluster wide view of all Hubble instances   The Peer service obtains peer change notifications by subscribing to Cilium s node manager  To this end  it internally defines a handler that implements Cilium s datapath node handler interface       hubble relay   Hubble Relay               Hubble Relay is the Hubble component that brings multi node support  It leverages the Peer service to obtain information about Hubble instances and consume their gRPC API in order to provide a more rich API that covers events from across the entire cluster  or even multiple clusters in a ClusterMesh scenario    Hubble Relay was first introduced as a technology preview with the release of Cilium v1 8 and was declared stable with the release of Cilium v1 9   Hubble Relay implements the Observer service for multi node  To that end  it maintains a persistent connection with every Hubble peer in a cluster with a peer manager  This component provides callers with the list of peers  Callers may report when a peer is unreachable  in which case the peer manager will attempt to reconnect   As Hubble Relay connects to every node in a cluster  the Hubble server instances must make their API available  by default on port 4244   By default  Hubble server endpoints are secured using mutual TLS  mTLS  when exposed on a TCP port in order to limit access to Hubble Relay only


cilium ciliumoperatorinternals This document provides a technical overview of the Cilium Operator and describes Cilium Operator docs cilium io the cluster wide operations it is responsible for

Answer:
.. only:: not (epub or latex or html)

    WARNING: You are looking at unreleased Cilium documentation.
    Please use the official rendered version released here:
    https://docs.cilium.io

.. _cilium_operator_internals:

Cilium Operator
===============

This document provides a technical overview of the Cilium Operator and describes
the cluster-wide operations it is responsible for.

Highly Available Cilium Operator
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The Cilium Operator uses Kubernetes leader election library in conjunction with
lease locks to provide HA functionality. The capability is supported on Kubernetes
versions 1.14 and above. It is Cilium's default behavior since the 1.9 release.

The number of replicas for the HA deployment can be configured using
Helm option ``operator.replicas``.

.. parsed-literal::

    helm install cilium |CHART_RELEASE| \\
      --namespace kube-system \\
      --set operator.replicas=3

.. code-block:: shell-session

    $ kubectl get deployment cilium-operator -n kube-system
    NAME              READY   UP-TO-DATE   AVAILABLE   AGE
    cilium-operator   3/3     3            3           46s

The operator is an integral part of Cilium installations in Kubernetes
environments and is tasked to perform the following operations:

CRD Registration
~~~~~~~~~~~~~~~~

The default behavior of the Cilium Operator is to register the CRDs used by
Cilium. The following custom resources are registered by the Cilium Operator:

.. include:: ../crdlist.rst

IPAM
~~~~

Cilium Operator is responsible for IP address management when running in
the following modes:

-  :ref:`ipam_azure`
-  :ref:`ipam_eni`
-  :ref:`ipam_crd_cluster_pool`

When running in IPAM mode :ref:`k8s_hostscope`, the allocation CIDRs used by
``cilium-agent`` is derived from the fields ``podCIDR`` and ``podCIDRs``
populated by Kubernetes in the Kubernetes ``Node`` resource.

For :ref:`concepts_ipam_crd` IPAM allocation mode, it is the job of Cloud-specific
operator to populate the required information about CIDRs in the
``CiliumNode`` resource.

Cilium currently has native support for the following Cloud providers in CRD IPAM
mode:

- Azure - ``cilium-operator-azure``
- AWS - ``cilium-operator-aws``

For more information on IPAM visit :ref:`address_management`.

Load Balancer IP Address Management
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When :ref:`lb_ipam` is used, Cilium Operator manages IP address
for ``type: LoadBalancer`` services.

KVStore operations
~~~~~~~~~~~~~~~~~~

These operations are performed only when KVStore is enabled for the
Cilium Operator. In addition, KVStore operations are only required when
``cilium-operator`` is running with any of the below options:

-  ``--synchronize-k8s-services``
-  ``--synchronize-k8s-nodes``
-  ``--identity-allocation-mode=kvstore``

K8s Services synchronization
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Cilium Operator performs the job of synchronizing Kubernetes services to
external KVStore configured for the Cilium Operator if running with
``--synchronize-k8s-services`` flag.

The Cilium Operator performs this operation only for shared services (services
that have ``service.cilium.io/shared`` annotation set to true). This is
meaningful when running Cilium to setup a ClusterMesh.

K8s Nodes synchronization
^^^^^^^^^^^^^^^^^^^^^^^^^

Similar to K8s services, Cilium Operator also synchronizes Kubernetes nodes
information to the shared KVStore.

When a ``Node`` object is deleted it is not possible to reliably cleanup
the corresponding ``CiliumNode`` object from the Agent itself. The Cilium Operator
holds the responsibility to garbage collect orphaned ``CiliumNodes``.

Heartbeat update
^^^^^^^^^^^^^^^^

The Cilium Operator periodically updates the Cilium's heartbeat path key
with the current time. The default key for this heartbeat is
``cilium/.heartbeat`` in the KVStore. It is used by Cilium Agents to validate
that KVStore updates can be received.

Identity garbage collection
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Each workload in Kubernetes is assigned a security identity that is used
for policy decision making. This identity is based on common workload
markers like labels. Cilium supports two identity allocation mechanisms:

-  CRD Identity allocation
-  KVStore Identity allocation

Both the mechanisms of identity allocation require the Cilium
Operator to perform the garbage collection of stale
identities. This garbage collection is necessary because a 16-bit
unsigned integer represents the security identity, and thus we can only
have a maximum of 65536 identities in the cluster.

CRD Identity garbage collection
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

CRD identity allocation uses Kubernetes custom resource
``CiliumIdentity`` to represent a security identity. This is the default
behavior of Cilium and works out of the box in any K8s environment
without any external dependency.

The Cilium Operator maintains a local cache for CiliumIdentities with
the last time they were seen active. A controller runs in the background
periodically which scans this local cache and deletes identities that
have not had their heartbeat life sign updated since
``identity-heartbeat-timeout``.

One thing to note here is that an Identity is always assumed to be live
if it has an endpoint associated with it.

KVStore Identity garbage collection
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

While the CRD allocation mode for identities is more common, it is
limited in terms of scale. When running in a very large environment, a
saner choice is to use the KVStore allocation mode. This mode stores
the identities in an external store like etcd.

For more information on Cilium's scalability visit :ref:`scalability_guide`.

The garbage collection mechanism involves scanning the KVStore of all
the identities. For each identity, the Cilium Operator search in the KVStore
if there are any active users of that identity. The entry is deleted from the
KVStore if there are no active users.

CiliumEndpoint garbage collection
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

CiliumEndpoint object is created by the ``cilium-agent`` for each ``Pod``
in the cluster. The Cilium Operator manages a controller to handle the
garbage collection of orphaned ``CiliumEndpoint`` objects. An orphaned
``CiliumEndpoint`` object means that the owner of the endpoint object is
not active anymore in the cluster. CiliumEndpoints are also considered
orphaned if the owner is an existing Pod in ``PodFailed`` or ``PodSucceeded``
state.
This controller is run periodically if the ``endpoint-gc-interval`` option
is specified and only once during startup if the option is unspecified.

Derivative network policy creation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When using Cloud-provider-specific constructs like ``toGroups`` in the
network policy spec, the Cilium Operator performs the job of converting these
constructs to derivative CNP/CCNP objects without these fields.

For more information, see how Cilium network policies incorporate the
use of ``toGroups`` to :ref:`lock down external access using AWS security groups<aws_metadata_with_policy>`.

Ingress and Gateway API Support
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When Ingress or Gateway API support is enabled, the Cilium Operator performs the
task of parsing Ingress or Gateway API objects and converting them into
``CiliumEnvoyConfig`` objects used for configuring the per-node Envoy proxy.

Additionally, Secrets used by Ingress or Gateway API objects will be synced to
a Cilium-managed namespace that the Cilium Agent is then granted access to. This
reduces the permissions required of the Cilium Agent.

Mutual Authentication Support
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When Cilium's Mutual Authentication Support is enabled, the Cilium Operator is
responsible for ensuring that each Cilium Identity has an associated identity
in the certificate management system. It will create and delete identity
registrations in the configured certificate management section as required.
The Cilium Operator does not, however have any to the key material in the
identities.

That information is only shared with the Cilium Agent via other channels.

only   not  epub or latex or html       WARNING  You are looking at unreleased Cilium documentation      Please use the official rendered version released here      https   docs cilium io      cilium operator internals   Cilium Operator                  This document provides a technical overview of the Cilium Operator and describes the cluster wide operations it is responsible for   Highly Available Cilium Operator                                   The Cilium Operator uses Kubernetes leader election library in conjunction with lease locks to provide HA functionality  The capability is supported on Kubernetes versions 1 14 and above  It is Cilium s default behavior since the 1 9 release   The number of replicas for the HA deployment can be configured using Helm option   operator replicas        parsed literal        helm install cilium  CHART RELEASE             namespace kube system            set operator replicas 3     code block   shell session        kubectl get deployment cilium operator  n kube system     NAME              READY   UP TO DATE   AVAILABLE   AGE     cilium operator   3 3     3            3           46s  The operator is an integral part of Cilium installations in Kubernetes environments and is tasked to perform the following operations   CRD Registration                   The default behavior of the Cilium Operator is to register the CRDs used by Cilium  The following custom resources are registered by the Cilium Operator      include      crdlist rst  IPAM       Cilium Operator is responsible for IP address management when running in the following modes       ref  ipam azure      ref  ipam eni      ref  ipam crd cluster pool   When running in IPAM mode  ref  k8s hostscope   the allocation CIDRs used by   cilium agent   is derived from the fields   podCIDR   and   podCIDRs   populated by Kubernetes in the Kubernetes   Node   resource   For  ref  concepts ipam crd  IPAM allocation mode  it is the job of Cloud specific operator to populate the required information about CIDRs in the   CiliumNode   resource   Cilium currently has native support for the following Cloud providers in CRD IPAM mode     Azure     cilium operator azure     AWS     cilium operator aws    For more information on IPAM visit  ref  address management    Load Balancer IP Address Management                                      When  ref  lb ipam  is used  Cilium Operator manages IP address for   type  LoadBalancer   services   KVStore operations                     These operations are performed only when KVStore is enabled for the Cilium Operator  In addition  KVStore operations are only required when   cilium operator   is running with any of the below options          synchronize k8s services          synchronize k8s nodes          identity allocation mode kvstore    K8s Services synchronization                               Cilium Operator performs the job of synchronizing Kubernetes services to external KVStore configured for the Cilium Operator if running with     synchronize k8s services   flag   The Cilium Operator performs this operation only for shared services  services that have   service cilium io shared   annotation set to true   This is meaningful when running Cilium to setup a ClusterMesh   K8s Nodes synchronization                            Similar to K8s services  Cilium Operator also synchronizes Kubernetes nodes information to the shared KVStore   When a   Node   object is deleted it is not possible to reliably cleanup the corresponding   CiliumNode   object from the Agent itself  The Cilium Operator holds the responsibility to garbage collect orphaned   CiliumNodes     Heartbeat update                   The Cilium Operator periodically updates the Cilium s heartbeat path key with the current time  The default key for this heartbeat is   cilium  heartbeat   in the KVStore  It is used by Cilium Agents to validate that KVStore updates can be received   Identity garbage collection                              Each workload in Kubernetes is assigned a security identity that is used for policy decision making  This identity is based on common workload markers like labels  Cilium supports two identity allocation mechanisms      CRD Identity allocation    KVStore Identity allocation  Both the mechanisms of identity allocation require the Cilium Operator to perform the garbage collection of stale identities  This garbage collection is necessary because a 16 bit unsigned integer represents the security identity  and thus we can only have a maximum of 65536 identities in the cluster   CRD Identity garbage collection                                  CRD identity allocation uses Kubernetes custom resource   CiliumIdentity   to represent a security identity  This is the default behavior of Cilium and works out of the box in any K8s environment without any external dependency   The Cilium Operator maintains a local cache for CiliumIdentities with the last time they were seen active  A controller runs in the background periodically which scans this local cache and deletes identities that have not had their heartbeat life sign updated since   identity heartbeat timeout     One thing to note here is that an Identity is always assumed to be live if it has an endpoint associated with it   KVStore Identity garbage collection                                      While the CRD allocation mode for identities is more common  it is limited in terms of scale  When running in a very large environment  a saner choice is to use the KVStore allocation mode  This mode stores the identities in an external store like etcd   For more information on Cilium s scalability visit  ref  scalability guide    The garbage collection mechanism involves scanning the KVStore of all the identities  For each identity  the Cilium Operator search in the KVStore if there are any active users of that identity  The entry is deleted from the KVStore if there are no active users   CiliumEndpoint garbage collection                                    CiliumEndpoint object is created by the   cilium agent   for each   Pod   in the cluster  The Cilium Operator manages a controller to handle the garbage collection of orphaned   CiliumEndpoint   objects  An orphaned   CiliumEndpoint   object means that the owner of the endpoint object is not active anymore in the cluster  CiliumEndpoints are also considered orphaned if the owner is an existing Pod in   PodFailed   or   PodSucceeded   state  This controller is run periodically if the   endpoint gc interval   option is specified and only once during startup if the option is unspecified   Derivative network policy creation                                     When using Cloud provider specific constructs like   toGroups   in the network policy spec  the Cilium Operator performs the job of converting these constructs to derivative CNP CCNP objects without these fields   For more information  see how Cilium network policies incorporate the use of   toGroups   to  ref  lock down external access using AWS security groups aws metadata with policy     Ingress and Gateway API Support                                  When Ingress or Gateway API support is enabled  the Cilium Operator performs the task of parsing Ingress or Gateway API objects and converting them into   CiliumEnvoyConfig   objects used for configuring the per node Envoy proxy   Additionally  Secrets used by Ingress or Gateway API objects will be synced to a Cilium managed namespace that the Cilium Agent is then granted access to  This reduces the permissions required of the Cilium Agent   Mutual Authentication Support                                When Cilium s Mutual Authentication Support is enabled  the Cilium Operator is responsible for ensuring that each Cilium Identity has an associated identity in the certificate management system  It will create and delete identity registrations in the configured certificate management section as required  The Cilium Operator does not  however have any to the key material in the identities   That information is only shared with the Cilium Agent via other channels