cilium policyverdicts docs cilium io Policy Audit Mode configures Cilium to allow all traffic while logging all Creating Policies from Verdicts connections that would otherwise be dropped by network policies Policy Audit

Answer:
.. only:: not (epub or latex or html)

    WARNING: You are looking at unreleased Cilium documentation.
    Please use the official rendered version released here:
    https://docs.cilium.io

.. _policy_verdicts:

*******************************
Creating Policies from Verdicts
*******************************

Policy Audit Mode configures Cilium to allow all traffic while logging all
connections that would otherwise be dropped by network policies. Policy Audit
Mode may be configured for the entire daemon using ``--policy-audit-mode=true``
or for individual Cilium Endpoints. When Policy Audit Mode is enabled, no
network policy is enforced so this setting is **not recommended for production
deployment**. Policy Audit Mode supports auditing network policies implemented
at networks layers 3 and 4. This guide walks through the process of creating
policies using Policy Audit Mode.

.. include:: gsg_requirements.rst
.. include:: gsg_sw_demo.rst

Scale down the deathstar Deployment
===================================

In this guide we're going to scale down the deathstar Deployment in order to
simplify the next steps:

.. code-block:: shell-session

   $ kubectl scale --replicas=1 deployment deathstar
   deployment.apps/deathstar scaled

Enable Policy Audit Mode (Entire Daemon)
========================================

To observe policy audit messages for all endpoints managed by this Daemonset,
modify the Cilium ConfigMap and restart all daemons:

   .. tabs::

      .. group-tab:: Configure via kubectl

         .. code-block:: shell-session

            $ kubectl patch -n $CILIUM_NAMESPACE configmap cilium-config --type merge --patch '{"data":{"policy-audit-mode":"true"}}'
            configmap/cilium-config patched
            $ kubectl -n $CILIUM_NAMESPACE rollout restart ds/cilium
            daemonset.apps/cilium restarted
            $ kubectl -n $CILIUM_NAMESPACE rollout status ds/cilium
            Waiting for daemon set "cilium" rollout to finish: 0 of 1 updated pods are available...
            daemon set "cilium" successfully rolled out

      .. group-tab:: Helm Upgrade

         If you installed Cilium via ``helm install``, then you can use ``helm
         upgrade`` to enable Policy Audit Mode:

         .. parsed-literal::

            $ helm upgrade cilium |CHART_RELEASE| \\
                --namespace $CILIUM_NAMESPACE \\
                --reuse-values \\
                --set policyAuditMode=true

Enable Policy Audit Mode (Specific Endpoint)
============================================

Cilium can enable Policy Audit Mode for a specific endpoint. This may be helpful when enabling
Policy Audit Mode for the entire daemon is too broad. Enabling per endpoint will ensure other
endpoints managed by the same daemon are not impacted.

This approach is meant to be temporary.  **Restarting Cilium pod will reset the Policy Audit
Mode to match the daemon's configuration.**

Policy Audit Mode is enabled for a given endpoint by modifying the endpoint configuration via
the ``cilium`` tool on the endpoint's Kubernetes node. The steps include:

#. Determine the endpoint id on which Policy Audit Mode will be enabled.
#. Identify the Cilium pod running on the same Kubernetes node corresponding to the endpoint.
#. Using the Cilium pod above, modify the endpoint configuration by setting ``PolicyAuditMode=Enabled``.

The following shell commands perform these steps:

.. code-block:: shell-session

   $ PODNAME=$(kubectl get pods -l app.kubernetes.io/name=deathstar -o jsonpath='{.items[*].metadata.name}')
   $ NODENAME=$(kubectl get pod -o jsonpath="{.items[?(@.metadata.name=='$PODNAME')].spec.nodeName}")
   $ ENDPOINT=$(kubectl get cep -o jsonpath="{.items[?(@.metadata.name=='$PODNAME')].status.id}")
   $ CILIUM_POD=$(kubectl -n "$CILIUM_NAMESPACE" get pod --all-namespaces --field-selector spec.nodeName="$NODENAME" -lk8s-app=cilium -o jsonpath='{.items[*].metadata.name}')
   $ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
       cilium-dbg endpoint config "$ENDPOINT" PolicyAuditMode=Enabled
    Endpoint 232 configuration updated successfully

We can check that Policy Audit Mode is enabled for this endpoint with

.. code-block:: shell-session

   $ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
       cilium-dbg endpoint get "$ENDPOINT" -o jsonpath='{[*].spec.options.PolicyAuditMode}'
   Enabled

.. _observe_policy_verdicts:

Observe policy verdicts
=======================

In this example, we are tasked with applying security policy for the deathstar.
First, from the Cilium pod we need to monitor the notifications for policy
verdicts using the Hubble CLI. We'll be monitoring for inbound traffic towards
the deathstar to identify it and determine whether to extend the network policy
to allow that traffic.

Apply a default-deny policy:

.. literalinclude:: ../../examples/minikube/sw_deny_policy.yaml

CiliumNetworkPolicies match on pod labels using an ``endpointSelector`` to identify
the sources and destinations to which the policy applies. The above policy denies
traffic sent to any pods with label (``org=empire``). Due to the Policy Audit Mode
enabled above (either for the entire daemon, or for just the ``deathstar`` endpoint),
the traffic will not actually be denied but will instead trigger policy verdict
notifications.

To apply this policy, run:

.. parsed-literal::

    $ kubectl create -f \ |SCM_WEB|\/examples/minikube/sw_deny_policy.yaml
    ciliumnetworkpolicy.cilium.io/empire-default-deny created

With the above policy, we will enable a default-deny posture on ingress to pods
with the label ``org=empire`` and enable the policy verdict notifications for
those pods. The same principle applies on egress as well.

Now let's send some traffic from the tiefighter to the deathstar:

.. code-block:: shell-session

    $ kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
    Ship landed

We can check the policy verdict from the Cilium Pod:

.. code-block:: shell-session

   $ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
       hubble observe flows -t policy-verdict --last 1
   Feb  7 12:53:39.168: default/tiefighter:54134 (ID:31028) -> default/deathstar-6fb5694d48-5hmds:80 (ID:16530) policy-verdict:none AUDITED (TCP Flags: SYN)

In the above example, we can see that the Pod ``deathstar-6fb5694d48-5hmds`` has
received traffic from the ``tiefighter`` Pod which doesn't match the policy
(``policy-verdict:none AUDITED``).

.. _create_network_policy:

Create the Network Policy
=========================

We can get more information about the flow with

.. code-block:: shell-session

   $ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
       hubble observe flows -t policy-verdict -o json --last 1

Given the above information, we now know the labels of the source and
destination Pods, the traffic direction, and the destination port. In this case,
we can see clearly that the source (i.e. the tiefighter Pod) is an empire
aircraft (as it has the ``org=empire`` label) so once we've determined that we
expect this traffic to arrive at the deathstar, we can form a policy to match
the traffic:

.. literalinclude:: ../../examples/minikube/sw_l3_l4_policy.yaml

To apply this L3/L4 policy, run:

.. parsed-literal::

    $ kubectl create -f \ |SCM_WEB|\/examples/minikube/sw_l3_l4_policy.yaml
    ciliumnetworkpolicy.cilium.io/rule1 created

Now if we run the landing requests again,

.. code-block:: shell-session

    $ kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
    Ship landed

we can then observe that the traffic which was previously audited to be dropped
by the policy is reported as allowed:

.. code-block:: shell-session

   $ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
       hubble observe flows -t policy-verdict --last 1
   ...
   Feb  7 13:06:45.130: default/tiefighter:59824 (ID:31028) -> default/deathstar-6fb5694d48-5hmds:80 (ID:16530) policy-verdict:L3-L4 ALLOWED (TCP Flags: SYN)

Now the policy verdict states that the traffic would be allowed:
``policy-verdict:L3-L4 ALLOWED``. Success!

Disable Policy Audit Mode (Entire Daemon)
=========================================

These steps should be repeated for each connection in the cluster to ensure
that the network policy allows all of the expected traffic. The final step
after deploying the policy is to disable Policy Audit Mode again:

   .. tabs::

      .. group-tab:: Configure via kubectl

         .. code-block:: shell-session

            $ kubectl patch -n $CILIUM_NAMESPACE configmap cilium-config --type merge --patch '{"data":{"policy-audit-mode":"false"}}'
            configmap/cilium-config patched
            $ kubectl -n $CILIUM_NAMESPACE rollout restart ds/cilium
            daemonset.apps/cilium restarted
            $ kubectl -n kube-system rollout status ds/cilium
            Waiting for daemon set "cilium" rollout to finish: 0 of 1 updated pods are available...
            daemon set "cilium" successfully rolled out

      .. group-tab:: Helm Upgrade

         .. parsed-literal::

            $ helm upgrade cilium |CHART_RELEASE| \\
                --namespace $CILIUM_NAMESPACE \\
                --reuse-values \\
                --set policyAuditMode=false

Disable Policy Audit Mode (Specific Endpoint)
=============================================

These steps are nearly identical to enabling Policy Audit Mode.

.. code-block:: shell-session

   $ PODNAME=$(kubectl get pods -l app.kubernetes.io/name=deathstar -o jsonpath='{.items[*].metadata.name}')
   $ NODENAME=$(kubectl get pod -o jsonpath="{.items[?(@.metadata.name=='$PODNAME')].spec.nodeName}")
   $ ENDPOINT=$(kubectl get cep -o jsonpath="{.items[?(@.metadata.name=='$PODNAME')].status.id}")
   $ CILIUM_POD=$(kubectl -n "$CILIUM_NAMESPACE" get pod --all-namespaces --field-selector spec.nodeName="$NODENAME" -lk8s-app=cilium -o jsonpath='{.items[*].metadata.name}')
   $ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
       cilium-dbg endpoint config "$ENDPOINT" PolicyAuditMode=Disabled
    Endpoint 232 configuration updated successfully

Alternatively, **restarting the Cilium pod** will set the endpoint Policy Audit Mode to the daemon set configuration.

Verify Policy Audit Mode is Disabled
====================================

.. code-block:: shell-session

   $ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
       cilium-dbg endpoint get "$ENDPOINT" -o jsonpath='{[*].spec.options.PolicyAuditMode}'
   Disabled

Now if we run the landing requests again, only the *tiefighter* pods with the
label ``org=empire`` should succeed:

.. code-block:: shell-session

    $ kubectl exec tiefighter -- curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
    Ship landed

And we can observe that the traffic was allowed by the policy:

.. code-block:: shell-session

    $ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
        hubble observe flows -t policy-verdict --from-pod tiefighter --last 1
    Feb  7 13:34:26.112: default/tiefighter:37314 (ID:31028) -> default/deathstar-6fb5694d48-5hmds:80 (ID:16530) policy-verdict:L3-L4 ALLOWED (TCP Flags: SYN)

This works as expected. Now the same request from an *xwing* Pod should fail:

.. code-block:: shell-session

    $ kubectl exec xwing -- curl --connect-timeout 3 -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
    command terminated with exit code 28

This curl request should timeout after three seconds, we can observe the policy
verdict with:

.. code-block:: shell-session

    $ kubectl -n "$CILIUM_NAMESPACE" exec "$CILIUM_POD" -c cilium-agent -- \
        hubble observe flows -t policy-verdict --from-pod xwing --last 1
    Feb  7 13:43:46.791: default/xwing:54842 (ID:22654) <> default/deathstar-6fb5694d48-5hmds:80 (ID:16530) policy-verdict:none DENIED (TCP Flags: SYN)

We hope you enjoyed the tutorial.  Feel free to play more with the setup,
follow the `gs_http` guide, and reach out to us on `Cilium Slack`_ with any
questions!

Clean-up
========

.. parsed-literal::

   $ kubectl delete -f \ |SCM_WEB|\/examples/minikube/http-sw-app.yaml
   $ kubectl delete cnp empire-default-deny rule1

only   not  epub or latex or html       WARNING  You are looking at unreleased Cilium documentation      Please use the official rendered version released here      https   docs cilium io      policy verdicts                                   Creating Policies from Verdicts                                  Policy Audit Mode configures Cilium to allow all traffic while logging all connections that would otherwise be dropped by network policies  Policy Audit Mode may be configured for the entire daemon using     policy audit mode true   or for individual Cilium Endpoints  When Policy Audit Mode is enabled  no network policy is enforced so this setting is   not recommended for production deployment    Policy Audit Mode supports auditing network policies implemented at networks layers 3 and 4  This guide walks through the process of creating policies using Policy Audit Mode      include   gsg requirements rst    include   gsg sw demo rst  Scale down the deathstar Deployment                                      In this guide we re going to scale down the deathstar Deployment in order to simplify the next steps      code block   shell session       kubectl scale   replicas 1 deployment deathstar    deployment apps deathstar scaled  Enable Policy Audit Mode  Entire Daemon                                            To observe policy audit messages for all endpoints managed by this Daemonset  modify the Cilium ConfigMap and restart all daemons         tabs             group tab   Configure via kubectl              code block   shell session                kubectl patch  n  CILIUM NAMESPACE configmap cilium config   type merge   patch    data    policy audit mode   true                 configmap cilium config patched               kubectl  n  CILIUM NAMESPACE rollout restart ds cilium             daemonset apps cilium restarted               kubectl  n  CILIUM NAMESPACE rollout status ds cilium             Waiting for daemon set  cilium  rollout to finish  0 of 1 updated pods are available                daemon set  cilium  successfully rolled out           group tab   Helm Upgrade           If you installed Cilium via   helm install    then you can use   helm          upgrade   to enable Policy Audit Mode               parsed literal                  helm upgrade cilium  CHART RELEASE                       namespace  CILIUM NAMESPACE                      reuse values                      set policyAuditMode true   Enable Policy Audit Mode  Specific Endpoint                                                Cilium can enable Policy Audit Mode for a specific endpoint  This may be helpful when enabling Policy Audit Mode for the entire daemon is too broad  Enabling per endpoint will ensure other endpoints managed by the same daemon are not impacted   This approach is meant to be temporary     Restarting Cilium pod will reset the Policy Audit Mode to match the daemon s configuration     Policy Audit Mode is enabled for a given endpoint by modifying the endpoint configuration via the   cilium   tool on the endpoint s Kubernetes node  The steps include      Determine the endpoint id on which Policy Audit Mode will be enabled     Identify the Cilium pod running on the same Kubernetes node corresponding to the endpoint     Using the Cilium pod above  modify the endpoint configuration by setting   PolicyAuditMode Enabled     The following shell commands perform these steps      code block   shell session       PODNAME   kubectl get pods  l app kubernetes io name deathstar  o jsonpath    items    metadata name         NODENAME   kubectl get pod  o jsonpath    items     metadata name    PODNAME    spec nodeName         ENDPOINT   kubectl get cep  o jsonpath    items     metadata name    PODNAME    status id         CILIUM POD   kubectl  n   CILIUM NAMESPACE  get pod   all namespaces   field selector spec nodeName   NODENAME   lk8s app cilium  o jsonpath    items    metadata name         kubectl  n   CILIUM NAMESPACE  exec   CILIUM POD   c cilium agent             cilium dbg endpoint config   ENDPOINT  PolicyAuditMode Enabled     Endpoint 232 configuration updated successfully  We can check that Policy Audit Mode is enabled for this endpoint with     code block   shell session       kubectl  n   CILIUM NAMESPACE  exec   CILIUM POD   c cilium agent             cilium dbg endpoint get   ENDPOINT   o jsonpath       spec options PolicyAuditMode      Enabled      observe policy verdicts   Observe policy verdicts                          In this example  we are tasked with applying security policy for the deathstar  First  from the Cilium pod we need to monitor the notifications for policy verdicts using the Hubble CLI  We ll be monitoring for inbound traffic towards the deathstar to identify it and determine whether to extend the network policy to allow that traffic   Apply a default deny policy      literalinclude         examples minikube sw deny policy yaml  CiliumNetworkPolicies match on pod labels using an   endpointSelector   to identify the sources and destinations to which the policy applies  The above policy denies traffic sent to any pods with label    org empire     Due to the Policy Audit Mode enabled above  either for the entire daemon  or for just the   deathstar   endpoint   the traffic will not actually be denied but will instead trigger policy verdict notifications   To apply this policy  run      parsed literal          kubectl create  f    SCM WEB   examples minikube sw deny policy yaml     ciliumnetworkpolicy cilium io empire default deny created  With the above policy  we will enable a default deny posture on ingress to pods with the label   org empire   and enable the policy verdict notifications for those pods  The same principle applies on egress as well   Now let s send some traffic from the tiefighter to the deathstar      code block   shell session        kubectl exec tiefighter    curl  s  XPOST deathstar default svc cluster local v1 request landing     Ship landed  We can check the policy verdict from the Cilium Pod      code block   shell session       kubectl  n   CILIUM NAMESPACE  exec   CILIUM POD   c cilium agent             hubble observe flows  t policy verdict   last 1    Feb  7 12 53 39 168  default tiefighter 54134  ID 31028     default deathstar 6fb5694d48 5hmds 80  ID 16530  policy verdict none AUDITED  TCP Flags  SYN   In the above example  we can see that the Pod   deathstar 6fb5694d48 5hmds   has received traffic from the   tiefighter   Pod which doesn t match the policy    policy verdict none AUDITED          create network policy   Create the Network Policy                            We can get more information about the flow with     code block   shell session       kubectl  n   CILIUM NAMESPACE  exec   CILIUM POD   c cilium agent             hubble observe flows  t policy verdict  o json   last 1  Given the above information  we now know the labels of the source and destination Pods  the traffic direction  and the destination port  In this case  we can see clearly that the source  i e  the tiefighter Pod  is an empire aircraft  as it has the   org empire   label  so once we ve determined that we expect this traffic to arrive at the deathstar  we can form a policy to match the traffic      literalinclude         examples minikube sw l3 l4 policy yaml  To apply this L3 L4 policy  run      parsed literal          kubectl create  f    SCM WEB   examples minikube sw l3 l4 policy yaml     ciliumnetworkpolicy cilium io rule1 created  Now if we run the landing requests again      code block   shell session        kubectl exec tiefighter    curl  s  XPOST deathstar default svc cluster local v1 request landing     Ship landed  we can then observe that the traffic which was previously audited to be dropped by the policy is reported as allowed      code block   shell session       kubectl  n   CILIUM NAMESPACE  exec   CILIUM POD   c cilium agent             hubble observe flows  t policy verdict   last 1           Feb  7 13 06 45 130  default tiefighter 59824  ID 31028     default deathstar 6fb5694d48 5hmds 80  ID 16530  policy verdict L3 L4 ALLOWED  TCP Flags  SYN   Now the policy verdict states that the traffic would be allowed    policy verdict L3 L4 ALLOWED    Success   Disable Policy Audit Mode  Entire Daemon                                             These steps should be repeated for each connection in the cluster to ensure that the network policy allows all of the expected traffic  The final step after deploying the policy is to disable Policy Audit Mode again         tabs             group tab   Configure via kubectl              code block   shell session                kubectl patch  n  CILIUM NAMESPACE configmap cilium config   type merge   patch    data    policy audit mode   false                 configmap cilium config patched               kubectl  n  CILIUM NAMESPACE rollout restart ds cilium             daemonset apps cilium restarted               kubectl  n kube system rollout status ds cilium             Waiting for daemon set  cilium  rollout to finish  0 of 1 updated pods are available                daemon set  cilium  successfully rolled out           group tab   Helm Upgrade              parsed literal                  helm upgrade cilium  CHART RELEASE                       namespace  CILIUM NAMESPACE                      reuse values                      set policyAuditMode false   Disable Policy Audit Mode  Specific Endpoint                                                 These steps are nearly identical to enabling Policy Audit Mode      code block   shell session       PODNAME   kubectl get pods  l app kubernetes io name deathstar  o jsonpath    items    metadata name         NODENAME   kubectl get pod  o jsonpath    items     metadata name    PODNAME    spec nodeName         ENDPOINT   kubectl get cep  o jsonpath    items     metadata name    PODNAME    status id         CILIUM POD   kubectl  n   CILIUM NAMESPACE  get pod   all namespaces   field selector spec nodeName   NODENAME   lk8s app cilium  o jsonpath    items    metadata name         kubectl  n   CILIUM NAMESPACE  exec   CILIUM POD   c cilium agent             cilium dbg endpoint config   ENDPOINT  PolicyAuditMode Disabled     Endpoint 232 configuration updated successfully  Alternatively    restarting the Cilium pod   will set the endpoint Policy Audit Mode to the daemon set configuration    Verify Policy Audit Mode is Disabled                                          code block   shell session       kubectl  n   CILIUM NAMESPACE  exec   CILIUM POD   c cilium agent             cilium dbg endpoint get   ENDPOINT   o jsonpath       spec options PolicyAuditMode      Disabled  Now if we run the landing requests again  only the  tiefighter  pods with the label   org empire   should succeed      code block   shell session        kubectl exec tiefighter    curl  s  XPOST deathstar default svc cluster local v1 request landing     Ship landed  And we can observe that the traffic was allowed by the policy      code block   shell session        kubectl  n   CILIUM NAMESPACE  exec   CILIUM POD   c cilium agent              hubble observe flows  t policy verdict   from pod tiefighter   last 1     Feb  7 13 34 26 112  default tiefighter 37314  ID 31028     default deathstar 6fb5694d48 5hmds 80  ID 16530  policy verdict L3 L4 ALLOWED  TCP Flags  SYN    This works as expected  Now the same request from an  xwing  Pod should fail      code block   shell session        kubectl exec xwing    curl   connect timeout 3  s  XPOST deathstar default svc cluster local v1 request landing     command terminated with exit code 28  This curl request should timeout after three seconds  we can observe the policy verdict with      code block   shell session        kubectl  n   CILIUM NAMESPACE  exec   CILIUM POD   c cilium agent              hubble observe flows  t policy verdict   from pod xwing   last 1     Feb  7 13 43 46 791  default xwing 54842  ID 22654     default deathstar 6fb5694d48 5hmds 80  ID 16530  policy verdict none DENIED  TCP Flags  SYN    We hope you enjoyed the tutorial   Feel free to play more with the setup  follow the  gs http  guide  and reach out to us on  Cilium Slack   with any questions   Clean up              parsed literal         kubectl delete  f    SCM WEB   examples minikube http sw app yaml      kubectl delete cnp empire default deny rule1


cilium Cilium environment running on your machine It is designed to take 15 30 docs cilium io This document serves as an introduction to using Cilium to enforce memcached aware security policies It walks through a single node Securing Memcached minutes

Answer:
.. only:: not (epub or latex or html)

    WARNING: You are looking at unreleased Cilium documentation.
    Please use the official rendered version released here:
    https://docs.cilium.io

******************
Securing Memcached
******************

This document serves as an introduction to using Cilium to enforce memcached-aware
security policies. It walks through a single-node
Cilium environment running on your machine. It is designed to take 15-30
minutes.

**NOTE:** memcached-aware policy support is still in beta.  It is not yet ready for
production use. Additionally, the memcached-specific policy language is highly likely to
change in a future Cilium version.

`Memcached <https://memcached.org/>`_ is a high performance, distributed memory object caching system. It's simple yet powerful, and used by dynamic web applications to alleviate database load. Memcached is designed to work efficiently for a very large number of open connections. Thus, clients are encouraged to cache their connections rather
than the overhead of reopening TCP connections every time they need to store or retrieve data. Multiple clients can benefit from this distributed cache's performance benefits.

There are two kinds of data sent in the memcache protocol: text lines
and unstructured (binary) data.  We will demonstrate clients using both types of protocols to communicate with a memcached server.

.. include:: gsg_requirements.rst

Step 2: Deploy the Demo Application
===================================

Now that we have Cilium deployed and ``kube-dns`` operating correctly we can
deploy our demo memcached application.  Since our first
`HTTP-aware Cilium demo <https://cilium.io/blog/2017/5/4/demo-may-the-force-be-with-you/>`_ was based on Star Wars, we continue with the theme for the memcached demo as well.

Ever wonder how the Alliance Fleet manages the changing positions of their ships? The Alliance Fleet uses memcached to store the coordinates of their ships. The Alliance Fleet leverages the memcached-svc service implemented as a memcached server. Each ship in the fleet constantly updates its coordinates and has the ability to get the coordinates of other ships in the Alliance Fleet.

In this simple example, the Alliance Fleet uses a memcached server for their starfighters to store their own supergalatic coordinates and get those of other starfighters.

In order to avoid collisions and protect against compromised starfighters, memcached commands are limited to gets for any starfighter coordinates and sets only to a key specific to the starfighter. Thus the following operations are allowed:

- **A-wing**: can set coordinates to key "awing-coord" and get the key coordinates.
- **X-wing**: can set coordinates to key "xwing-coord" and get the key coordinates.
- **Alliance-Tracker**: can get any coordinates but not set any.

To keep the setup small, we will launch a small number of pods to represent a larger environment:

- **memcached-server** : A Kubernetes service represented by a single pod running a memcached server (label app=memcd-server).
- **a-wing** memcached binary client : A pod representing an A-wing starfighter, which can update its coordinates and read it via the binary memcached protocol (label app=a-wing).
- **x-wing** memcached text-based client : A pod representing an X-wing starfighter, which can update its coordinates and read it via the text-based memcached protocol (label app=x-wing).
- **alliance-tracker** memcached binary client : A pod representing the Alliance Fleet Tracker, able to read the coordinates of all starfighters (label name=fleet-tracker).

Memcached clients access the *memcached-server* on TCP port 11211 and send memcached protocol messages to it.

.. image:: images/cilium_memcd_gsg_topology.png

The file ``memcd-sw-app.yaml`` contains a Kubernetes Deployment for each of the pods described
above, as well as a Kubernetes Service *memcached-server* for the Memcached server.

.. parsed-literal::

    $ kubectl create -f \ |SCM_WEB|\/examples/kubernetes-memcached/memcd-sw-app.yaml
    deployment.apps/memcached-server created
    service/memcached-server created
    deployment.apps/a-wing created
    deployment.apps/x-wing created
    deployment.apps/alliance-tracker created

Kubernetes will deploy the pods and service in the background.
Running ``kubectl get svc,pods`` will inform you about the progress of the operation.
Each pod will go through several states until it reaches ``Running`` at which
point the setup is ready.

.. code-block:: shell-session

    $ kubectl get svc,pods
    NAME                       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)     AGE
    service/kubernetes         ClusterIP   10.96.0.1    <none>        443/TCP     31m
    service/memcached-server   ClusterIP   None         <none>        11211/TCP   14m

    NAME                                    READY   STATUS    RESTARTS   AGE
    pod/a-wing-67db8d5fcc-dpwl4             1/1     Running   0          14m
    pod/alliance-tracker-6b6447bd69-sz5hz   1/1     Running   0          14m
    pod/memcached-server-bdbfb87cd-8tdh7    1/1     Running   0          14m
    pod/x-wing-fd5dfb9d9-wrtwn              1/1     Running   0          14m

We suggest having a main terminal window to execute *kubectl* commands and two additional terminal windows dedicated to accessing the **A-Wing** and **Alliance-Tracker**, which use a python library to communicate to the memcached server using the binary protocol.

In **all three** terminal windows, set some handy environment variables for the demo with the following script:

.. parsed-literal::

    $ curl -s \ |SCM_WEB|\/examples/kubernetes-memcached/memcd-env.sh > memcd-env.sh
    $ source memcd-env.sh

In the terminal window dedicated for the A-wing pod, exec in, use python to import the binary memcached library and set the client connection to the memcached server:

.. code-block:: shell-session

    $ kubectl exec -ti $AWING_POD -- sh
    # python
    Python 3.7.0 (default, Sep  5 2018, 03:25:31)
    [GCC 6.3.0 20170516] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import bmemcached
    >>> client = bmemcached.Client(("memcached-server:11211", ))

In the terminal window dedicated for the Alliance-Tracker, exec in, use python to import the binary memcached library and set the client connection to the memcached server:

.. code-block:: shell-session

    $ kubectl exec -ti $TRACKER_POD -- sh
    # python
    Python 3.7.0 (default, Sep  5 2018, 03:25:31)
    [GCC 6.3.0 20170516] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import bmemcached
    >>> client = bmemcached.Client(("memcached-server:11211", ))

Step 3: Test Basic Memcached Access
===================================

Let's show that each client is able to access the memcached server. Execute the following to have the A-wing and X-wing starfighters update the Alliance Fleet memcached-server with their respective supergalatic coordinates:

A-wing will access the memcached-server using the *binary protocol*. In your terminal window for A-Wing, set A-wing's coordinates:

.. code-block:: python

    >>> client.set("awing-coord","4309.432,918.980",time=2400)
    True
    >>> client.get("awing-coord")
    '4309.432,918.980'

In your main terminal window, have X-wing starfighter set their coordinates using the text-based protocol to the memcached server.

.. code-block:: shell-session

    $ kubectl exec $XWING_POD -- sh -c "echo -en \\"$SETXC\\" | nc memcached-server 11211"
    STORED
    $ kubectl exec $XWING_POD -- sh -c "echo -en \\"$GETXC\\" | nc memcached-server 11211"
    VALUE xwing-coord 0 16
    8893.34,234.3290
    END

Check that the Alliance Fleet Tracker is able to get all starfighters' coordinates in your terminal window for the Alliance-Tracker:

.. code-block:: python

    >>> client.get("awing-coord")
    '4309.432,918.980'
    >>> client.get("xwing-coord")
    '8893.34,234.3290'

Step 4:  The Danger of a Compromised Memcached Client
=====================================================

Imagine if a starfighter ship is captured. Should the starfighter be able to set the coordinates of other ships, or get the coordinates of all other ships? Or if the Alliance-Tracker is compromised, can it modify the coordinates of any starfighter ship?
If every client has access to the Memcached API on port 11211, all have over-privileged access until further locked down.

With L4 port access to the memcached server, all starfighters could write to any key/ship and read all ship coordinates. In your main terminal, execute:

.. code-block:: shell-session

    $ kubectl exec $XWING_POD sh -- -c "echo -en \\"$GETAC\\" | nc memcached-server 11211"
    VALUE awing-coord 0 16
    4309.432,918.980
    END

In your A-Wing terminal window, confirm the over-privileged access:

.. code-block:: python

    >>> client.get("xwing-coord")
    '8893.34,234.3290'
    >>> client.set("xwing-coord","0.0,0.0",time=2400)
    True
    >>> client.get("xwing-coord")
    '0.0,0.0'

From A-Wing, set the X-Wing coordinates back to their proper position:

.. code-block:: python

    >>> client.set("xwing-coord","8893.34,234.3290",time=2400)
    True

Thus, the Alliance Fleet Tracking System could be made weak if a single starfighter ship is compromised.

Step 5: Securing Access to Memcached with Cilium
================================================

Cilium helps lock down Memcached servers to ensure clients have secure access to it. Beyond just providing access to port 11211,
Cilium can enforce specific key value access by understanding both the text-based and the unstructured (binary) memcached protocol.

We'll create a policy that limits the scope of what a starfighter can access and write. Thus, only the intended memcached protocol calls to the memcached-server can be made.

In this example, we'll only allow A-Wing to get and set the key "awing-coord", only allow X-Wing to get and set key "xwing-coord", and allow Alliance-Tracker to only get coordinates.

.. image:: images/cilium_memcd_gsg_attack.png

Here is the *CiliumNetworkPolicy* rule that limits the access of starfighters to their own key and allows Alliance Tracker to get any coordinate:

.. literalinclude:: ../../examples/kubernetes-memcached/memcd-sw-security-policy.yaml

A *CiliumNetworkPolicy* contains a list of rules that define allowed memcached commands, and requests
that do not match any rules are denied. The rules explicitly match connections destined to the Memcached Service on TCP 11211.

The rules apply to inbound (i.e., "ingress") connections bound for memcached-server pods (as indicated by ``app:memcached-server``
in the "endpointSelector" section).  The rules apply differently depending on the
client pod: ``app:a-wing``, ``app:x-wing``, or ``name:fleet-tracker`` as indicated by the "fromEndpoints" section.

With the policy in place, A-wings can only get and set the key "awing-coord"; similarly the X-Wing can only get and set "xwing-coord". The Alliance Tracker can only get coordinates - not set.

Apply this Memcached-aware network security policy using ``kubectl`` in your main terminal window:

.. parsed-literal::

    $ kubectl create -f \ |SCM_WEB|\/examples/kubernetes-memcached/memcd-sw-security-policy.yaml

If we then try to perform the attacks from the *X-wing* pod from the main terminal window, we'll see that they are denied:

.. code-block:: shell-session

    $ kubectl exec $XWING_POD -- sh -c "echo -en \\"$GETAC\\" | nc memcached-server 11211"
    CLIENT_ERROR access denied

From the A-Wing terminal window, we can confirm that if *A-wing* goes outside of the bounds of its allowed calls. You may need to run the ``client.get`` command twice for the python call:

.. code-block:: python

    >>> client.get("awing-coord")
    '4309.432,918.980'
    >>> client.get("xwing-coord")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/local/lib/python3.7/site-packages/bmemcached/client/replicating.py", line 42, in get
        value, cas = server.get(key)
      File "/usr/local/lib/python3.7/site-packages/bmemcached/protocol.py", line 440, in get
        raise MemcachedException('Code: %d Message: %s' % (status, extra_content), status)
    bmemcached.exceptions.MemcachedException: ("Code: 8 Message: b'access denied'", 8)

Similarly, the Alliance-Tracker cannot set any coordinates, which you can attempt from the Alliance-Tracker terminal window:

.. code-block:: python

    >>> client.get("xwing-coord")
    '8893.34,234.3290'
    >>> client.set("awing-coord","0.0,0.0",time=1200)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/local/lib/python3.7/site-packages/bmemcached/client/replicating.py", line 112, in set
        returns.append(server.set(key, value, time, compress_level=compress_level))
      File "/usr/local/lib/python3.7/site-packages/bmemcached/protocol.py", line 604, in set
        return self._set_add_replace('set', key, value, time, compress_level=compress_level)
      File "/usr/local/lib/python3.7/site-packages/bmemcached/protocol.py", line 583, in _set_add_replace
        raise MemcachedException('Code: %d Message: %s' % (status, extra_content), status)
    bmemcached.exceptions.MemcachedException: ("Code: 8 Message: b'access denied'", 8)

The policy is working as expected.

With the CiliumNetworkPolicy in place, the allowed Memcached calls are still allowed from the respective pods.

In the main terminal window, execute:

.. code-block:: shell-session

  $ kubectl exec $XWING_POD -- sh -c "echo -en \\"$GETXC\\" | nc memcached-server 11211"
  VALUE xwing-coord 0 16
  8893.34,234.3290
  END
  $ SETXC="set xwing-coord 0 1200 16\\r\\n9854.34,926.9187\\r\\nquit\\r\\n"
  $ kubectl exec $XWING_POD -- sh -c "echo -en \\"$SETXC\\" | nc memcached-server 11211"
  STORED
  $ kubectl exec $XWING_POD -- sh -c "echo -en \\"$GETXC\\" | nc memcached-server 11211"
  VALUE xwing-coord 0 16
  9854.34,926.9187
  END

In the A-Wing terminal window, execute:

.. code-block:: python

  >>> client.set("awing-coord","9852.542,892.1318",time=1200)
  True
  >>> client.get("awing-coord")
  '9852.542,892.1318'
  >>> exit()
  # exit

In the Alliance-Tracker terminal window, execute:

.. code-block:: python

  >>> client.get("awing-coord")
  '9852.542,892.1318'
  >>> client.get("xwing-coord")
  '9854.34,926.9187'
  >>> exit()
  # exit

Step 6: Clean Up
================

You have now installed Cilium, deployed a demo app, and tested
L7 memcached-aware network security policies.  To clean up, in your main terminal window, run:

.. parsed-literal::

   $ kubectl delete -f \ |SCM_WEB|\/examples/kubernetes-memcached/memcd-sw-app.yaml
   $ kubectl delete cnp secure-fleet

For some handy memcached references, see below:

* https://memcached.org/
* https://github.com/memcached/memcached/blob/master/doc/protocol.txt
* https://python-binary-memcached.readthedocs.io/en/latest/intro/

only   not  epub or latex or html       WARNING  You are looking at unreleased Cilium documentation      Please use the official rendered version released here      https   docs cilium io                     Securing Memcached                     This document serves as an introduction to using Cilium to enforce memcached aware security policies  It walks through a single node Cilium environment running on your machine  It is designed to take 15 30 minutes     NOTE    memcached aware policy support is still in beta   It is not yet ready for production use  Additionally  the memcached specific policy language is highly likely to change in a future Cilium version    Memcached  https   memcached org     is a high performance  distributed memory object caching system  It s simple yet powerful  and used by dynamic web applications to alleviate database load  Memcached is designed to work efficiently for a very large number of open connections  Thus  clients are encouraged to cache their connections rather than the overhead of reopening TCP connections every time they need to store or retrieve data  Multiple clients can benefit from this distributed cache s performance benefits   There are two kinds of data sent in the memcache protocol  text lines and unstructured  binary  data   We will demonstrate clients using both types of protocols to communicate with a memcached server      include   gsg requirements rst  Step 2  Deploy the Demo Application                                      Now that we have Cilium deployed and   kube dns   operating correctly we can deploy our demo memcached application   Since our first  HTTP aware Cilium demo  https   cilium io blog 2017 5 4 demo may the force be with you     was based on Star Wars  we continue with the theme for the memcached demo as well   Ever wonder how the Alliance Fleet manages the changing positions of their ships  The Alliance Fleet uses memcached to store the coordinates of their ships  The Alliance Fleet leverages the memcached svc service implemented as a memcached server  Each ship in the fleet constantly updates its coordinates and has the ability to get the coordinates of other ships in the Alliance Fleet   In this simple example  the Alliance Fleet uses a memcached server for their starfighters to store their own supergalatic coordinates and get those of other starfighters   In order to avoid collisions and protect against compromised starfighters  memcached commands are limited to gets for any starfighter coordinates and sets only to a key specific to the starfighter  Thus the following operations are allowed       A wing    can set coordinates to key  awing coord  and get the key coordinates      X wing    can set coordinates to key  xwing coord  and get the key coordinates      Alliance Tracker    can get any coordinates but not set any   To keep the setup small  we will launch a small number of pods to represent a larger environment       memcached server     A Kubernetes service represented by a single pod running a memcached server  label app memcd server       a wing   memcached binary client   A pod representing an A wing starfighter  which can update its coordinates and read it via the binary memcached protocol  label app a wing       x wing   memcached text based client   A pod representing an X wing starfighter  which can update its coordinates and read it via the text based memcached protocol  label app x wing       alliance tracker   memcached binary client   A pod representing the Alliance Fleet Tracker  able to read the coordinates of all starfighters  label name fleet tracker     Memcached clients access the  memcached server  on TCP port 11211 and send memcached protocol messages to it      image   images cilium memcd gsg topology png   The file   memcd sw app yaml   contains a Kubernetes Deployment for each of the pods described above  as well as a Kubernetes Service  memcached server  for the Memcached server      parsed literal          kubectl create  f    SCM WEB   examples kubernetes memcached memcd sw app yaml     deployment apps memcached server created     service memcached server created     deployment apps a wing created     deployment apps x wing created     deployment apps alliance tracker created  Kubernetes will deploy the pods and service in the background  Running   kubectl get svc pods   will inform you about the progress of the operation  Each pod will go through several states until it reaches   Running   at which point the setup is ready      code block   shell session        kubectl get svc pods     NAME                       TYPE        CLUSTER IP   EXTERNAL IP   PORT S      AGE     service kubernetes         ClusterIP   10 96 0 1     none         443 TCP     31m     service memcached server   ClusterIP   None          none         11211 TCP   14m      NAME                                    READY   STATUS    RESTARTS   AGE     pod a wing 67db8d5fcc dpwl4             1 1     Running   0          14m     pod alliance tracker 6b6447bd69 sz5hz   1 1     Running   0          14m     pod memcached server bdbfb87cd 8tdh7    1 1     Running   0          14m     pod x wing fd5dfb9d9 wrtwn              1 1     Running   0          14m  We suggest having a main terminal window to execute  kubectl  commands and two additional terminal windows dedicated to accessing the   A Wing   and   Alliance Tracker    which use a python library to communicate to the memcached server using the binary protocol   In   all three   terminal windows  set some handy environment variables for the demo with the following script      parsed literal          curl  s    SCM WEB   examples kubernetes memcached memcd env sh   memcd env sh       source memcd env sh   In the terminal window dedicated for the A wing pod  exec in  use python to import the binary memcached library and set the client connection to the memcached server      code block   shell session        kubectl exec  ti  AWING POD    sh       python     Python 3 7 0  default  Sep  5 2018  03 25 31       GCC 6 3 0 20170516  on linux     Type  help    copyright    credits  or  license  for more information          import bmemcached         client   bmemcached Client   memcached server 11211       In the terminal window dedicated for the Alliance Tracker  exec in  use python to import the binary memcached library and set the client connection to the memcached server      code block   shell session        kubectl exec  ti  TRACKER POD    sh       python     Python 3 7 0  default  Sep  5 2018  03 25 31       GCC 6 3 0 20170516  on linux     Type  help    copyright    credits  or  license  for more information          import bmemcached         client   bmemcached Client   memcached server 11211         Step 3  Test Basic Memcached Access                                      Let s show that each client is able to access the memcached server  Execute the following to have the A wing and X wing starfighters update the Alliance Fleet memcached server with their respective supergalatic coordinates   A wing will access the memcached server using the  binary protocol   In your terminal window for A Wing  set A wing s coordinates      code block   python          client set  awing coord   4309 432 918 980  time 2400      True         client get  awing coord        4309 432 918 980    In your main terminal window  have X wing starfighter set their coordinates using the text based protocol to the memcached server      code block   shell session        kubectl exec  XWING POD    sh  c  echo  en     SETXC      nc memcached server 11211      STORED       kubectl exec  XWING POD    sh  c  echo  en     GETXC      nc memcached server 11211      VALUE xwing coord 0 16     8893 34 234 3290     END  Check that the Alliance Fleet Tracker is able to get all starfighters  coordinates in your terminal window for the Alliance Tracker      code block   python          client get  awing coord        4309 432 918 980          client get  xwing coord        8893 34 234 3290    Step 4   The Danger of a Compromised Memcached Client                                                        Imagine if a starfighter ship is captured  Should the starfighter be able to set the coordinates of other ships  or get the coordinates of all other ships  Or if the Alliance Tracker is compromised  can it modify the coordinates of any starfighter ship  If every client has access to the Memcached API on port 11211  all have over privileged access until further locked down   With L4 port access to the memcached server  all starfighters could write to any key ship and read all ship coordinates  In your main terminal  execute      code block   shell session        kubectl exec  XWING POD sh     c  echo  en     GETAC      nc memcached server 11211      VALUE awing coord 0 16     4309 432 918 980     END  In your A Wing terminal window  confirm the over privileged access      code block   python          client get  xwing coord        8893 34 234 3290          client set  xwing coord   0 0 0 0  time 2400      True         client get  xwing coord        0 0 0 0   From A Wing  set the X Wing coordinates back to their proper position      code block   python          client set  xwing coord   8893 34 234 3290  time 2400      True  Thus  the Alliance Fleet Tracking System could be made weak if a single starfighter ship is compromised   Step 5  Securing Access to Memcached with Cilium                                                   Cilium helps lock down Memcached servers to ensure clients have secure access to it  Beyond just providing access to port 11211  Cilium can enforce specific key value access by understanding both the text based and the unstructured  binary  memcached protocol   We ll create a policy that limits the scope of what a starfighter can access and write  Thus  only the intended memcached protocol calls to the memcached server can be made   In this example  we ll only allow A Wing to get and set the key  awing coord   only allow X Wing to get and set key  xwing coord   and allow Alliance Tracker to only get coordinates       image   images cilium memcd gsg attack png  Here is the  CiliumNetworkPolicy  rule that limits the access of starfighters to their own key and allows Alliance Tracker to get any coordinate      literalinclude         examples kubernetes memcached memcd sw security policy yaml  A  CiliumNetworkPolicy  contains a list of rules that define allowed memcached commands  and requests that do not match any rules are denied  The rules explicitly match connections destined to the Memcached Service on TCP 11211   The rules apply to inbound  i e    ingress   connections bound for memcached server pods  as indicated by   app memcached server   in the  endpointSelector  section    The rules apply differently depending on the client pod    app a wing      app x wing    or   name fleet tracker   as indicated by the  fromEndpoints  section   With the policy in place  A wings can only get and set the key  awing coord   similarly the X Wing can only get and set  xwing coord   The Alliance Tracker can only get coordinates   not set   Apply this Memcached aware network security policy using   kubectl   in your main terminal window      parsed literal          kubectl create  f    SCM WEB   examples kubernetes memcached memcd sw security policy yaml  If we then try to perform the attacks from the  X wing  pod from the main terminal window  we ll see that they are denied      code block   shell session        kubectl exec  XWING POD    sh  c  echo  en     GETAC      nc memcached server 11211      CLIENT ERROR access denied  From the A Wing terminal window  we can confirm that if  A wing  goes outside of the bounds of its allowed calls  You may need to run the   client get   command twice for the python call      code block   python          client get  awing coord        4309 432 918 980          client get  xwing coord       Traceback  most recent call last         File   stdin    line 1  in  module        File   usr local lib python3 7 site packages bmemcached client replicating py   line 42  in get         value  cas   server get key        File   usr local lib python3 7 site packages bmemcached protocol py   line 440  in get         raise MemcachedException  Code   d Message   s     status  extra content   status      bmemcached exceptions MemcachedException    Code  8 Message  b access denied    8   Similarly  the Alliance Tracker cannot set any coordinates  which you can attempt from the Alliance Tracker terminal window      code block   python          client get  xwing coord        8893 34 234 3290          client set  awing coord   0 0 0 0  time 1200      Traceback  most recent call last         File   stdin    line 1  in  module        File   usr local lib python3 7 site packages bmemcached client replicating py   line 112  in set         returns append server set key  value  time  compress level compress level         File   usr local lib python3 7 site packages bmemcached protocol py   line 604  in set         return self  set add replace  set   key  value  time  compress level compress level        File   usr local lib python3 7 site packages bmemcached protocol py   line 583  in  set add replace         raise MemcachedException  Code   d Message   s     status  extra content   status      bmemcached exceptions MemcachedException    Code  8 Message  b access denied    8   The policy is working as expected   With the CiliumNetworkPolicy in place  the allowed Memcached calls are still allowed from the respective pods   In the main terminal window  execute      code block   shell session      kubectl exec  XWING POD    sh  c  echo  en     GETXC      nc memcached server 11211    VALUE xwing coord 0 16   8893 34 234 3290   END     SETXC  set xwing coord 0 1200 16  r  n9854 34 926 9187  r  nquit  r  n      kubectl exec  XWING POD    sh  c  echo  en     SETXC      nc memcached server 11211    STORED     kubectl exec  XWING POD    sh  c  echo  en     GETXC      nc memcached server 11211    VALUE xwing coord 0 16   9854 34 926 9187   END  In the A Wing terminal window  execute      code block   python        client set  awing coord   9852 542 892 1318  time 1200    True       client get  awing coord      9852 542 892 1318        exit       exit  In the Alliance Tracker terminal window  execute      code block   python        client get  awing coord      9852 542 892 1318        client get  xwing coord      9854 34 926 9187        exit       exit   Step 6  Clean Up                   You have now installed Cilium  deployed a demo app  and tested L7 memcached aware network security policies   To clean up  in your main terminal window  run      parsed literal         kubectl delete  f    SCM WEB   examples kubernetes memcached memcd sw app yaml      kubectl delete cnp secure fleet  For some handy memcached references  see below     https   memcached org    https   github com memcached memcached blob master doc protocol txt   https   python binary memcached readthedocs io en latest intro


cilium Securing a Cassandra Database Cilium environment running on your machine It is designed to take 15 30 docs cilium io security policies It is a detailed walk through of getting a single node This document serves as an introduction to using Cilium to enforce Cassandra aware minutes

Answer:
.. only:: not (epub or latex or html)

    WARNING: You are looking at unreleased Cilium documentation.
    Please use the official rendered version released here:
    https://docs.cilium.io

*****************************
Securing a Cassandra Database
*****************************

This document serves as an introduction to using Cilium to enforce Cassandra-aware
security policies.  It is a detailed walk-through of getting a single-node
Cilium environment running on your machine. It is designed to take 15-30
minutes.

**NOTE:** Cassandra-aware policy support is still in beta phase.  It is not yet ready for
production use.   Additionally, the Cassandra-specific policy language is highly likely to
change in a future Cilium version.

.. include:: gsg_requirements.rst

Deploy the Demo Application
===========================

Now that we have Cilium deployed and ``kube-dns`` operating correctly we can
deploy our demo Cassandra application.  Since our first
`HTTP-aware Cilium  Star Wars demo <https://cilium.io/blog/2017/5/4/demo-may-the-force-be-with-you/>`_
showed how the Galactic Empire used HTTP-aware security policies to protect the Death Star from the
Rebel Alliance, this Cassandra demo is Star Wars-themed as well.

`Apache Cassanadra <http://cassandra.apache.org>`_ is a popular NOSQL database focused on
delivering high-performance transactions (especially on writes) without sacrificing on availability or scale.
Cassandra operates as a cluster of servers, and Cassandra clients query these services via a
the `native Cassandra protocol <https://github.com/apache/cassandra/blob/trunk/doc/native_protocol_v4.spec>`_ .
Cilium understands the Cassandra protocol, and thus is able to provide deep visibility and control over
which clients are able to access particular tables inside a Cassandra cluster, and which actions
(e.g., "select", "insert", "update", "delete") can be performed on tables.

With Cassandra, each table belongs to a "keyspace", allowing multiple groups to use a single cluster without conflicting.
Cassandra queries specify the full table name qualified by the keyspace using the syntax "<keyspace>.<table>".

In our simple example, the Empire uses a Cassandra cluster to store two different types of information:

- **Employee Attendance Records** : Use to store daily attendance data (attendance.daily_records).
- **Deathstar Scrum Reports** : Daily scrum reports from the teams working on the Deathstar (deathstar.scrum_reports).

To keep the setup small, we will just launch a small number of pods to represent this setup:

- **cass-server** : A single pod running the Cassandra service, representing a Cassandra cluster
  (label app=cass-server).
- **empire-hq** : A pod representing the Empire's Headquarters, which is the only pod that should
  be able to read all attendance data, or read/write the Deathstar scrum notes (label app=empire-hq).
- **empire-outpost** : A random outpost in the empire.  It should be able to insert employee attendance
  records, but not read records for other empire facilities.   It also should not have any access to the
  deathstar keyspace (label app=empire-outpost).

All pods other than *cass-server* are Cassandra clients, which need access to the *cass-server*
container on TCP port 9042 in order to send Cassandra protocol messages.

.. image:: images/cilium_cass_gsg_topology.png

The file ``cass-sw-app.yaml`` contains a Kubernetes Deployment for each of the pods described
above, as well as a Kubernetes Service *cassandra-svc* for the Cassandra cluster.

.. parsed-literal::

    $ kubectl create -f \ |SCM_WEB|\/examples/kubernetes-cassandra/cass-sw-app.yaml
    deployment.apps/cass-server created
    service/cassandra-svc created
    deployment.apps/empire-hq created
    deployment.apps/empire-outpost created

Kubernetes will deploy the pods and service in the background.
Running ``kubectl get svc,pods`` will inform you about the progress of the operation.
Each pod will go through several states until it reaches ``Running`` at which
point the setup is ready.

.. code-block:: shell-session

    $ kubectl get svc,pods
    NAME                    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
    service/cassandra-svc   ClusterIP   None         <none>        9042/TCP   1m
    service/kubernetes      ClusterIP   10.96.0.1    <none>        443/TCP    15h

    NAME                                  READY     STATUS    RESTARTS   AGE
    pod/cass-server-5674d5b946-x8v4j      1/1       Running   0          1m
    pod/empire-hq-c494c664d-xmvdl         1/1       Running   0          1m
    pod/empire-outpost-68bf76858d-flczn   1/1       Running   0          1m

Step 3: Test Basic Cassandra Access
===================================

First, we'll create the keyspaces and tables mentioned above, and populate them with some initial data:

.. parsed-literal::

   $  curl -s \ |SCM_WEB|\/examples/kubernetes-cassandra/cass-populate-tables.sh | bash

Next, create two environment variables that refer to the *empire-hq* and *empire-outpost* pods:

.. code-block:: shell-session

   $ HQ_POD=$(kubectl get pods -l app=empire-hq -o jsonpath='{.items[0].metadata.name}')
   $ OUTPOST_POD=$(kubectl get pods -l app=empire-outpost -o jsonpath='{.items[0].metadata.name}')

Now we will run the 'cqlsh' Cassandra client in the *empire-outpost* pod, telling it to access
the Cassandra cluster identified by the 'cassandra-svc' DNS name:

.. code-block:: shell-session

    $ kubectl exec -it $OUTPOST_POD -- cqlsh cassandra-svc
    Connected to Test Cluster at cassandra-svc:9042.
    [cqlsh 5.0.1 | Cassandra 3.11.3 | CQL spec 3.4.4 | Native protocol v4]
    Use HELP for help.
    cqlsh>

Next, using the cqlsh prompt, we'll show that the outpost can add records to the "daily_records" table
in the "attendance" keyspace:

.. code-block:: shell-session

    cqlsh> INSERT INTO attendance.daily_records (creation, loc_id, present, empire_member_id) values (now(), 074AD3B9-A47D-4EBC-83D3-CAD75B1911CE, true, 6AD3139F-EBFC-4E0C-9F79-8F997BA01D90);

We have confirmed that outposts are able to report daily attendance records as intended. We're off to a good start!

The Danger of a Compromised Cassandra Client
============================================

But what if a rebel spy gains access to any of the remote outposts that act as a Cassandra client?
Since every client has access to the Cassandra API on port 9042, it can do some bad stuff.
For starters, the outpost container can not only add entries to the attendance.daily_reports table,
but it could read all entries as well.

To see this, we can run the following command:

.. code-block:: shell-session

  $ cqlsh> SELECT * FROM attendance.daily_records;

    loc_id                               | creation                             | empire_member_id                     | present
   --------------------------------------+--------------------------------------+--------------------------------------+---------
   a855e745-69d8-4159-b8b6-e2bafed8387a | c692ce90-bf57-11e8-98e6-f1a9f45fc4d8 | cee6d956-dbeb-4b09-ad21-1dd93290fa6c |    True
   5b9a7990-657e-442d-a3f7-94484f06696e | c8493120-bf57-11e8-98e6-f1a9f45fc4d8 | e74a0300-94f3-4b3d-aee4-fea85eca5af7 |    True
   53ed94d0-ddac-4b14-8c2f-ba6f83a8218c | c641a150-bf57-11e8-98e6-f1a9f45fc4d8 | 104ddbb6-f2f7-4cd0-8683-cc18cccc1326 |    True
   074ad3b9-a47d-4ebc-83d3-cad75b1911ce | 9674ed40-bf59-11e8-98e6-f1a9f45fc4d8 | 6ad3139f-ebfc-4e0c-9f79-8f997ba01d90 |    True
   fe72cc39-dffb-45dc-8e5f-86c674a58951 | c5e79a70-bf57-11e8-98e6-f1a9f45fc4d8 | 6782689c-0488-4ecb-b582-a2ccd282405e |    True
   461f4176-eb4c-4bcc-a08a-46787ca01af3 | c6fefde0-bf57-11e8-98e6-f1a9f45fc4d8 | 01009199-3d6b-4041-9c43-b1ca9aef021c |    True
   64dbf608-6947-4a23-98e9-63339c413136 | c8096900-bf57-11e8-98e6-f1a9f45fc4d8 | 6ffe024e-beff-4370-a1b5-dcf6330ec82b |    True
   13cefcac-5652-4c69-a3c2-1484671f2467 | c53f4c80-bf57-11e8-98e6-f1a9f45fc4d8 | 55218adc-2f3d-4f84-a693-87a2c238bb26 |    True
   eabf5185-376b-4d4a-a5b5-99f912d98279 | c593fc30-bf57-11e8-98e6-f1a9f45fc4d8 | 5e22159b-f3a9-4f8a-9944-97375df570e9 |    True
   3c0ae2d1-c836-4aa4-8fe2-5db6cc1f92fc | c7af1400-bf57-11e8-98e6-f1a9f45fc4d8 | 0ccb3df7-78d0-4434-8a7f-4bfa8d714275 |    True
   31a292e0-2e28-4a7d-8c84-8d4cf0c57483 | c4e0d8d0-bf57-11e8-98e6-f1a9f45fc4d8 | 8fe7625c-f482-4eb6-b33e-271440777403 |    True

  (11 rows)

Uh oh!  The rebels now has strategic information about empire troop strengths at each location in the galaxy.

But even more nasty from a security perspective is that the outpost container can also access information in any keyspace,
including the deathstar keyspace.  For example, run:

.. code-block:: shell-session

 $ cqlsh> SELECT * FROM deathstar.scrum_notes;

  empire_member_id                     | content                                                                                | creation
 --------------------------------------+----------------------------------------------------------------------------------------------------------------+--------------------------------------
 34e564c2-781b-477e-acd0-b357d67f94f2 | Designed protective shield for deathstar.  Could be based on nearby moon.  Feature punted to v2.  Not blocked. | c3c8b210-bf57-11e8-98e6-f1a9f45fc4d8
 dfa974ea-88cd-4e9b-85e3-542b9d00e2df |   I think the exhaust port could be vulnerable to a direct hit.  Hope no one finds out about it.  Not blocked. | c37f4d00-bf57-11e8-98e6-f1a9f45fc4d8
 ee12306a-7b44-46a4-ad68-42e86f0f111e |        Trying to figure out if we should paint it medium grey, light grey, or medium-light grey.  Not blocked. | c32daa90-bf57-11e8-98e6-f1a9f45fc4d8

 (3 rows)

We see that any outpost can actually access the deathstar scrum notes, which mentions a pretty serious issue with the exhaust port.

Securing Access to Cassandra with Cilium
========================================

Obviously, it would be much more secure to limit each pod's access to the Cassandra server to be
least privilege (i.e., only what is needed for the app to operate correctly and nothing more).

We can do that with the following Cilium security policy.   As with Cilium HTTP policies, we can write
policies that identify pods by labels, and then limit the traffic in/out of this pod.  In
this case, we'll create a policy that identifies the tables that each client should be able to access,
the actions that are allowed on those tables, and deny the rest.

As an example, a policy could limit containers with label *app=empire-outpost* to only be able to
insert entries into the table "attendance.daily_reports", but would block any attempt by a compromised outpost
to read all attendance information or access other keyspaces.

.. image:: images/cilium_cass_gsg_attack.png

Here is the *CiliumNetworkPolicy* rule that limits access of pods with label *app=empire-outpost* to
only insert records into "attendance.daily_reports":

.. literalinclude:: ../../examples/kubernetes-cassandra/cass-sw-security-policy.yaml

A *CiliumNetworkPolicy* contains a list of rules that define allowed requests, meaning that requests
that do not match any rules are denied as invalid.

The rule explicitly matches Cassandra connections destined to TCP 9042 on cass-server pods, and allows
query actions like select/insert/update/delete only on a specified set of tables.
The above rule applies to inbound (i.e., "ingress") connections to cass-server pods (as indicated by "app:cass-server"
in the "endpointSelector" section).  The rule applies different rules based on whether the
client pod has labels "app: empire-outpost" or "app: empire-hq" as indicated by the "fromEndpoints" section.

The policy limits the *empire-outpost* pod to performing "select" queries on the "system" and "system_schema"
keyspaces (required by cqlsh on startup) and "insert" queries to the "attendance.daily_records" table.

The full policy adds another rule that allows all queries from the *empire-hq* pod.

Apply this Cassandra-aware network security policy using ``kubectl`` in a new window:

.. parsed-literal::

    $ kubectl create -f \ |SCM_WEB|\/examples/kubernetes-cassandra/cass-sw-security-policy.yaml

If we then again try to perform the attacks from the *empire-outpost* pod, we'll see that they are denied:

.. code-block:: shell-session

  $ cqlsh> SELECT * FROM attendance.daily_records;
  Unauthorized: Error from server: code=2100 [Unauthorized] message="Request Unauthorized"

This is because the policy only permits pods with labels app: empire-outpost to insert into attendance.daily_records, it does
not permit select on that table, or any action on other tables (with the exception of the system.* and system_schema.*
keyspaces).  Its worth noting that we don't simply drop the message (which
could easily be confused with a network error), but rather we respond with the Cassandra Unauthorized error message.
(similar to how HTTP would return an error code of 403 unauthorized).

Likewise, if the outpost pod ever tries to access a table in another keyspace, like deathstar, this request will also be
denied:

.. code-block:: shell-session

  $ cqlsh> SELECT * FROM deathstar.scrum_notes;
  Unauthorized: Error from server: code=2100 [Unauthorized] message="Request Unauthorized"

This is blocked as well, thanks to the Cilium network policy.

Use another window to confirm that the *empire-hq* pod still has full access to the cassandra cluster:

.. code-block:: shell-session

    $ kubectl exec -it $HQ_POD -- cqlsh cassandra-svc
    Connected to Test Cluster at cassandra-svc:9042.
    [cqlsh 5.0.1 | Cassandra 3.11.3 | CQL spec 3.4.4 | Native protocol v4]
    Use HELP for help.
    cqlsh>

The power of Cilium's identity-based security allows *empire-hq* to still have full access
to both tables:

.. code-block:: shell-session

  $ cqlsh> SELECT * FROM attendance.daily_records;
   loc_id                               | creation                             | empire_member_id                     | present
  --------------------------------------+--------------------------------------+--------------------------------------+---------
  a855e745-69d8-4159-b8b6-e2bafed8387a | c692ce90-bf57-11e8-98e6-f1a9f45fc4d8 | cee6d956-dbeb-4b09-ad21-1dd93290fa6c |    True

  <snip>

  (12 rows)

Similarly, the deathstar can still access the scrum notes:

.. code-block:: shell-session

  $ cqlsh> SELECT * FROM deathstar.scrum_notes;

    <snip>

  (3 rows)

Cassandra-Aware Visibility (Bonus)
==================================

As a bonus, you can re-run the above queries with policy enforced and view how Cilium provides Cassandra-aware visibility, including
whether requests are forwarded or denied.   First, use "kubectl exec" to access the cilium pod.

.. code-block:: shell-session

  $ CILIUM_POD=$(kubectl get pods -n kube-system -l k8s-app=cilium -o jsonpath='{.items[0].metadata.name}')
  $ kubectl exec -it -n kube-system $CILIUM_POD -- /bin/bash
  root@minikube:~#

Next, start Cilium monitor, and limit the output to only "l7" type messages using the "-t" flag:

::

  root@minikube:~# cilium-dbg monitor -t l7
  Listening for events on 2 CPUs with 64x4096 of shared memory
  Press Ctrl-C to quit

In the other windows, re-run the above queries, and you will see that Cilium provides full visibility at the level of
each Cassandra request, indicating:

- The Kubernetes label-based identity of both the sending and receiving pod.
- The details of the Cassandra request, including the 'query_action' (e.g., 'select', 'insert')
  and 'query_table' (e.g., 'system.local', 'attendance.daily_records')
- The 'verdict' indicating whether the request was allowed by policy ('Forwarded' or 'Denied').

Example output is below.   All requests are from *empire-outpost* to *cass-server*.   The first two requests are
allowed, a 'select' into 'system.local' and an 'insert' into 'attendance.daily_records'.
The second two requests are denied, a 'select' into 'attendance.daily_records' and a select into 'deathstar.scrum_notes' :

::

  <- Request cassandra from 0 ([k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.kubernetes.pod.namespace=default k8s:app=empire-outpost]) to 64503 ([k8s:app=cass-server k8s:io.kubernetes.pod.namespace=default k8s:io.cilium.k8s.policy.serviceaccount=default]), identity 12443->16168, verdict Forwarded query_table:system.local query_action:selec
  <- Request cassandra from 0 ([k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.kubernetes.pod.namespace=default k8s:app=empire-outpost]) to 64503 ([k8s:app=cass-server k8s:io.kubernetes.pod.namespace=default k8s:io.cilium.k8s.policy.serviceaccount=default]), identity 12443->16168, verdict Forwarded query_action:insert query_table:attendance.daily_records
  <- Request cassandra from 0 ([k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.kubernetes.pod.namespace=default k8s:app=empire-outpost]) to 64503 ([k8s:app=cass-server k8s:io.kubernetes.pod.namespace=default k8s:io.cilium.k8s.policy.serviceaccount=default]), identity 12443->16168, verdict Denied query_action:select query_table:attendance.daily_records
  <- Request cassandra from 0 ([k8s:io.cilium.k8s.policy.serviceaccount=default k8s:io.kubernetes.pod.namespace=default k8s:app=empire-outpost]) to 64503 ([k8s:app=cass-server k8s:io.kubernetes.pod.namespace=default k8s:io.cilium.k8s.policy.serviceaccount=default]), identity 12443->16168, verdict Denied query_table:deathstar.scrum_notes query_action:select

Clean Up
========

You have now installed Cilium, deployed a demo app, and tested
L7 Cassandra-aware network security policies.  To clean up, run:

.. parsed-literal::

   $ kubectl delete -f \ |SCM_WEB|\/examples/kubernetes-cassandra/cass-sw-app.yaml
   $ kubectl delete cnp secure-empire-cassandra

After this, you can re-run the tutorial from Step 1.

only   not  epub or latex or html       WARNING  You are looking at unreleased Cilium documentation      Please use the official rendered version released here      https   docs cilium io                                Securing a Cassandra Database                                This document serves as an introduction to using Cilium to enforce Cassandra aware security policies   It is a detailed walk through of getting a single node Cilium environment running on your machine  It is designed to take 15 30 minutes     NOTE    Cassandra aware policy support is still in beta phase   It is not yet ready for production use    Additionally  the Cassandra specific policy language is highly likely to change in a future Cilium version      include   gsg requirements rst  Deploy the Demo Application                              Now that we have Cilium deployed and   kube dns   operating correctly we can deploy our demo Cassandra application   Since our first  HTTP aware Cilium  Star Wars demo  https   cilium io blog 2017 5 4 demo may the force be with you     showed how the Galactic Empire used HTTP aware security policies to protect the Death Star from the Rebel Alliance  this Cassandra demo is Star Wars themed as well    Apache Cassanadra  http   cassandra apache org    is a popular NOSQL database focused on delivering high performance transactions  especially on writes  without sacrificing on availability or scale  Cassandra operates as a cluster of servers  and Cassandra clients query these services via a the  native Cassandra protocol  https   github com apache cassandra blob trunk doc native protocol v4 spec      Cilium understands the Cassandra protocol  and thus is able to provide deep visibility and control over which clients are able to access particular tables inside a Cassandra cluster  and which actions  e g    select    insert    update    delete   can be performed on tables   With Cassandra  each table belongs to a  keyspace   allowing multiple groups to use a single cluster without conflicting  Cassandra queries specify the full table name qualified by the keyspace using the syntax   keyspace   table     In our simple example  the Empire uses a Cassandra cluster to store two different types of information       Employee Attendance Records     Use to store daily attendance data  attendance daily records       Deathstar Scrum Reports     Daily scrum reports from the teams working on the Deathstar  deathstar scrum reports    To keep the setup small  we will just launch a small number of pods to represent this setup       cass server     A single pod running the Cassandra service  representing a Cassandra cluster    label app cass server       empire hq     A pod representing the Empire s Headquarters  which is the only pod that should   be able to read all attendance data  or read write the Deathstar scrum notes  label app empire hq       empire outpost     A random outpost in the empire   It should be able to insert employee attendance   records  but not read records for other empire facilities    It also should not have any access to the   deathstar keyspace  label app empire outpost    All pods other than  cass server  are Cassandra clients  which need access to the  cass server  container on TCP port 9042 in order to send Cassandra protocol messages      image   images cilium cass gsg topology png  The file   cass sw app yaml   contains a Kubernetes Deployment for each of the pods described above  as well as a Kubernetes Service  cassandra svc  for the Cassandra cluster      parsed literal          kubectl create  f    SCM WEB   examples kubernetes cassandra cass sw app yaml     deployment apps cass server created     service cassandra svc created     deployment apps empire hq created     deployment apps empire outpost created  Kubernetes will deploy the pods and service in the background  Running   kubectl get svc pods   will inform you about the progress of the operation  Each pod will go through several states until it reaches   Running   at which point the setup is ready      code block   shell session        kubectl get svc pods     NAME                    TYPE        CLUSTER IP   EXTERNAL IP   PORT S     AGE     service cassandra svc   ClusterIP   None          none         9042 TCP   1m     service kubernetes      ClusterIP   10 96 0 1     none         443 TCP    15h      NAME                                  READY     STATUS    RESTARTS   AGE     pod cass server 5674d5b946 x8v4j      1 1       Running   0          1m     pod empire hq c494c664d xmvdl         1 1       Running   0          1m     pod empire outpost 68bf76858d flczn   1 1       Running   0          1m   Step 3  Test Basic Cassandra Access                                      First  we ll create the keyspaces and tables mentioned above  and populate them with some initial data      parsed literal          curl  s    SCM WEB   examples kubernetes cassandra cass populate tables sh   bash  Next  create two environment variables that refer to the  empire hq  and  empire outpost  pods      code block   shell session       HQ POD   kubectl get pods  l app empire hq  o jsonpath    items 0  metadata name         OUTPOST POD   kubectl get pods  l app empire outpost  o jsonpath    items 0  metadata name      Now we will run the  cqlsh  Cassandra client in the  empire outpost  pod  telling it to access the Cassandra cluster identified by the  cassandra svc  DNS name      code block   shell session        kubectl exec  it  OUTPOST POD    cqlsh cassandra svc     Connected to Test Cluster at cassandra svc 9042       cqlsh 5 0 1   Cassandra 3 11 3   CQL spec 3 4 4   Native protocol v4      Use HELP for help      cqlsh   Next  using the cqlsh prompt  we ll show that the outpost can add records to the  daily records  table in the  attendance  keyspace      code block   shell session      cqlsh  INSERT INTO attendance daily records  creation  loc id  present  empire member id  values  now    074AD3B9 A47D 4EBC 83D3 CAD75B1911CE  true  6AD3139F EBFC 4E0C 9F79 8F997BA01D90    We have confirmed that outposts are able to report daily attendance records as intended  We re off to a good start   The Danger of a Compromised Cassandra Client                                               But what if a rebel spy gains access to any of the remote outposts that act as a Cassandra client  Since every client has access to the Cassandra API on port 9042  it can do some bad stuff  For starters  the outpost container can not only add entries to the attendance daily reports table  but it could read all entries as well   To see this  we can run the following command      code block   shell session      cqlsh  SELECT   FROM attendance daily records       loc id                                 creation                               empire member id                       present                                                                                a855e745 69d8 4159 b8b6 e2bafed8387a   c692ce90 bf57 11e8 98e6 f1a9f45fc4d8   cee6d956 dbeb 4b09 ad21 1dd93290fa6c      True    5b9a7990 657e 442d a3f7 94484f06696e   c8493120 bf57 11e8 98e6 f1a9f45fc4d8   e74a0300 94f3 4b3d aee4 fea85eca5af7      True    53ed94d0 ddac 4b14 8c2f ba6f83a8218c   c641a150 bf57 11e8 98e6 f1a9f45fc4d8   104ddbb6 f2f7 4cd0 8683 cc18cccc1326      True    074ad3b9 a47d 4ebc 83d3 cad75b1911ce   9674ed40 bf59 11e8 98e6 f1a9f45fc4d8   6ad3139f ebfc 4e0c 9f79 8f997ba01d90      True    fe72cc39 dffb 45dc 8e5f 86c674a58951   c5e79a70 bf57 11e8 98e6 f1a9f45fc4d8   6782689c 0488 4ecb b582 a2ccd282405e      True    461f4176 eb4c 4bcc a08a 46787ca01af3   c6fefde0 bf57 11e8 98e6 f1a9f45fc4d8   01009199 3d6b 4041 9c43 b1ca9aef021c      True    64dbf608 6947 4a23 98e9 63339c413136   c8096900 bf57 11e8 98e6 f1a9f45fc4d8   6ffe024e beff 4370 a1b5 dcf6330ec82b      True    13cefcac 5652 4c69 a3c2 1484671f2467   c53f4c80 bf57 11e8 98e6 f1a9f45fc4d8   55218adc 2f3d 4f84 a693 87a2c238bb26      True    eabf5185 376b 4d4a a5b5 99f912d98279   c593fc30 bf57 11e8 98e6 f1a9f45fc4d8   5e22159b f3a9 4f8a 9944 97375df570e9      True    3c0ae2d1 c836 4aa4 8fe2 5db6cc1f92fc   c7af1400 bf57 11e8 98e6 f1a9f45fc4d8   0ccb3df7 78d0 4434 8a7f 4bfa8d714275      True    31a292e0 2e28 4a7d 8c84 8d4cf0c57483   c4e0d8d0 bf57 11e8 98e6 f1a9f45fc4d8   8fe7625c f482 4eb6 b33e 271440777403      True     11 rows    Uh oh   The rebels now has strategic information about empire troop strengths at each location in the galaxy   But even more nasty from a security perspective is that the outpost container can also access information in any keyspace  including the deathstar keyspace   For example  run      code block   shell session     cqlsh  SELECT   FROM deathstar scrum notes     empire member id                       content                                                                                creation                                                                                34e564c2 781b 477e acd0 b357d67f94f2   Designed protective shield for deathstar   Could be based on nearby moon   Feature punted to v2   Not blocked    c3c8b210 bf57 11e8 98e6 f1a9f45fc4d8  dfa974ea 88cd 4e9b 85e3 542b9d00e2df     I think the exhaust port could be vulnerable to a direct hit   Hope no one finds out about it   Not blocked    c37f4d00 bf57 11e8 98e6 f1a9f45fc4d8  ee12306a 7b44 46a4 ad68 42e86f0f111e          Trying to figure out if we should paint it medium grey  light grey  or medium light grey   Not blocked    c32daa90 bf57 11e8 98e6 f1a9f45fc4d8    3 rows   We see that any outpost can actually access the deathstar scrum notes  which mentions a pretty serious issue with the exhaust port   Securing Access to Cassandra with Cilium                                           Obviously  it would be much more secure to limit each pod s access to the Cassandra server to be least privilege  i e   only what is needed for the app to operate correctly and nothing more    We can do that with the following Cilium security policy    As with Cilium HTTP policies  we can write policies that identify pods by labels  and then limit the traffic in out of this pod   In this case  we ll create a policy that identifies the tables that each client should be able to access  the actions that are allowed on those tables  and deny the rest   As an example  a policy could limit containers with label  app empire outpost  to only be able to insert entries into the table  attendance daily reports   but would block any attempt by a compromised outpost to read all attendance information or access other keyspaces      image   images cilium cass gsg attack png  Here is the  CiliumNetworkPolicy  rule that limits access of pods with label  app empire outpost  to only insert records into  attendance daily reports       literalinclude         examples kubernetes cassandra cass sw security policy yaml  A  CiliumNetworkPolicy  contains a list of rules that define allowed requests  meaning that requests that do not match any rules are denied as invalid   The rule explicitly matches Cassandra connections destined to TCP 9042 on cass server pods  and allows query actions like select insert update delete only on a specified set of tables  The above rule applies to inbound  i e    ingress   connections to cass server pods  as indicated by  app cass server  in the  endpointSelector  section    The rule applies different rules based on whether the client pod has labels  app  empire outpost  or  app  empire hq  as indicated by the  fromEndpoints  section   The policy limits the  empire outpost  pod to performing  select  queries on the  system  and  system schema  keyspaces  required by cqlsh on startup  and  insert  queries to the  attendance daily records  table   The full policy adds another rule that allows all queries from the  empire hq  pod   Apply this Cassandra aware network security policy using   kubectl   in a new window      parsed literal          kubectl create  f    SCM WEB   examples kubernetes cassandra cass sw security policy yaml  If we then again try to perform the attacks from the  empire outpost  pod  we ll see that they are denied      code block   shell session      cqlsh  SELECT   FROM attendance daily records    Unauthorized  Error from server  code 2100  Unauthorized  message  Request Unauthorized   This is because the policy only permits pods with labels app  empire outpost to insert into attendance daily records  it does not permit select on that table  or any action on other tables  with the exception of the system   and system schema   keyspaces    Its worth noting that we don t simply drop the message  which could easily be confused with a network error   but rather we respond with the Cassandra Unauthorized error message   similar to how HTTP would return an error code of 403 unauthorized    Likewise  if the outpost pod ever tries to access a table in another keyspace  like deathstar  this request will also be denied      code block   shell session      cqlsh  SELECT   FROM deathstar scrum notes    Unauthorized  Error from server  code 2100  Unauthorized  message  Request Unauthorized   This is blocked as well  thanks to the Cilium network policy   Use another window to confirm that the  empire hq  pod still has full access to the cassandra cluster      code block   shell session        kubectl exec  it  HQ POD    cqlsh cassandra svc     Connected to Test Cluster at cassandra svc 9042       cqlsh 5 0 1   Cassandra 3 11 3   CQL spec 3 4 4   Native protocol v4      Use HELP for help      cqlsh   The power of Cilium s identity based security allows  empire hq  to still have full access to both tables      code block   shell session       cqlsh  SELECT   FROM attendance daily records     loc id                                 creation                               empire member id                       present                                                                                a855e745 69d8 4159 b8b6 e2bafed8387a   c692ce90 bf57 11e8 98e6 f1a9f45fc4d8   cee6d956 dbeb 4b09 ad21 1dd93290fa6c      True     snip      12 rows    Similarly  the deathstar can still access the scrum notes      code block   shell session      cqlsh  SELECT   FROM deathstar scrum notes        snip      3 rows   Cassandra Aware Visibility  Bonus                                      As a bonus  you can re run the above queries with policy enforced and view how Cilium provides Cassandra aware visibility  including whether requests are forwarded or denied    First  use  kubectl exec  to access the cilium pod      code block   shell session      CILIUM POD   kubectl get pods  n kube system  l k8s app cilium  o jsonpath    items 0  metadata name        kubectl exec  it  n kube system  CILIUM POD     bin bash   root minikube     Next  start Cilium monitor  and limit the output to only  l7  type messages using the   t  flag         root minikube    cilium dbg monitor  t l7   Listening for events on 2 CPUs with 64x4096 of shared memory   Press Ctrl C to quit  In the other windows  re run the above queries  and you will see that Cilium provides full visibility at the level of each Cassandra request  indicating     The Kubernetes label based identity of both the sending and receiving pod    The details of the Cassandra request  including the  query action   e g    select    insert     and  query table   e g    system local    attendance daily records     The  verdict  indicating whether the request was allowed by policy   Forwarded  or  Denied     Example output is below    All requests are from  empire outpost  to  cass server     The first two requests are allowed  a  select  into  system local  and an  insert  into  attendance daily records   The second two requests are denied  a  select  into  attendance daily records  and a select into  deathstar scrum notes              Request cassandra from 0   k8s io cilium k8s policy serviceaccount default k8s io kubernetes pod namespace default k8s app empire outpost   to 64503   k8s app cass server k8s io kubernetes pod namespace default k8s io cilium k8s policy serviceaccount default    identity 12443  16168  verdict Forwarded query table system local query action selec      Request cassandra from 0   k8s io cilium k8s policy serviceaccount default k8s io kubernetes pod namespace default k8s app empire outpost   to 64503   k8s app cass server k8s io kubernetes pod namespace default k8s io cilium k8s policy serviceaccount default    identity 12443  16168  verdict Forwarded query action insert query table attendance daily records      Request cassandra from 0   k8s io cilium k8s policy serviceaccount default k8s io kubernetes pod namespace default k8s app empire outpost   to 64503   k8s app cass server k8s io kubernetes pod namespace default k8s io cilium k8s policy serviceaccount default    identity 12443  16168  verdict Denied query action select query table attendance daily records      Request cassandra from 0   k8s io cilium k8s policy serviceaccount default k8s io kubernetes pod namespace default k8s app empire outpost   to 64503   k8s app cass server k8s io kubernetes pod namespace default k8s io cilium k8s policy serviceaccount default    identity 12443  16168  verdict Denied query table deathstar scrum notes query action select  Clean Up           You have now installed Cilium  deployed a demo app  and tested L7 Cassandra aware network security policies   To clean up  run      parsed literal         kubectl delete  f    SCM WEB   examples kubernetes cassandra cass sw app yaml      kubectl delete cnp secure empire cassandra  After this  you can re run the tutorial from Step 1