Cilium has become a prominent player in the Container Network Interface space and notably has been in acquisition of Cisco announced this week of Isovalent. It’s no surprise the leverage of eBPF for Runtime Enforcement and Security Observability also has enhanced visibility of the kernel system calls and activity that can deviate from known behavior, the introduction of the tool Tetragon. What makes Tetragon powerful is the capability of kubernetes-aware its able to identify namespaces, pods, and other aspects of kubernetes so event detection can also be targeted for individual workloads.
Architecture view of Tetragon included for breaking this concepts down in a digestible format.
Leveraging visibility at the kernel level allows for deep visibility in tracing system calls that are refined, its noted in the documentation the knowledge base behind Tetragon are kernel developers. This is able to leverage that knowledge base of underlying system components that gives a substantial advantage on how to monitor events and integration.
Getting Started
Leveraging a sandbox installation I’ve used KinD for the installation following along with the documentation provided.
## Creating the KinD cluster - tetragon depends on access to the host/proc file system.
cat <<EOF > kind-config.yaml
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
- role: control-plane
extraMounts:
- hostPath: /proc
containerPath: /procHost
EOF
kind create cluster --config kind-config.yaml
EXTRA_HELM_FLAGS="--set tetragon.hostProcPath=/procHost" # flags for helm install
helm repo add cilium https://helm.cilium.io
helm repo update
helm install tetragon cilium/tetragon -n kube-system
You should see in our kube-system (tetragon-operator, tetragon container) once the rollout is finished from the daemonset.
Following along using the demo application located here https://docs.cilium.io/en/v1.11/gettingstarted/http/#deploy-the-demo-application
Once this is deployed your cluster should have the following pods running along with services shown in this image.
This also assumes you are running Cilium and the status is reporting ok as shown in the image below if you haven’t installed cilium as the CNI follow the demo application install this covers that.
We can now run the following that should produce some output.
kubectl exec -ti -n kube-system ds/tetragon -c tetragon -- tetra getevents -o compact --pods deathstar
If this command hangs you can revert the use of –pods flag to just -o compact and the output should appear for events tetragon captures.
We can see the process along with when a process exit along with context of the event in a color-coded format. Additionally, we can see SIGTERM (Signal Terminated) attributed to one of the exit events.
Moving to examine File Access Monitoring we will deploy the following YAML.
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/file_monitoring.yaml
When we run the following command we can see the access of writing is happening from our events.
kubectl exec -ti -n kube-system ds/tetragon -c tetragon — tetra getevents -o compact
This can be drilled down to in who has access to what files in our cluster along with what operations are they performing (early indicators of comprise or malicious uses).
For instance deploying a pod and writing to to file or reading /etc/shadow for password lets generate this use.
This should of generate a log which it was catching the entire writing of the cluster but appears to have a issue if I run the command getevents -o compact –pod nginxtest it hangs on that command likely a timeout. Additionally, this should be monitored and not occurring on your container you should confine operations and access in these sensitive file areas.
Capturing events of where files are being written to should also confine and be visible signs of unusual activity as shown below I ran in another terminal the command to access the pod and write to a specific folder.
Summary
Tetragon provides granular visibility into cluster operations that can be leveraged as early signs of malicious activity that should be monitored and configured for remediation. Additionally, like any tool this is one piece to our defense-in-depth strategy of container security as we should be scanning images, validating integrity of the images components, understanding dependencies and much more. Additionally while this is catching activity we purposely crafted from the monitoring standpoint as a detective control we should also implement preventative controls such as using Kyverno or other policy enforcement tools to restrict various scenarios we should limit. I could see this tool growing in popularity with the integration of Cilium and likely using Hubble while we didn’t cover every use-case of tetragon wanted to introduce some of the key concepts in a security view point.
Sources
https://tetragon.io/docs/getting-started/execution/
For more on the use of Cilium in AKS I created this video to show deployment via Terraform for this use.