RodrigTech

Simulating Kubernetes Attacks with Detection of Falco + Tetragon

—

by

rodrigtech
in

Microsoft recently released an open-source repository designed to simulate attacks on Kubernetes clusters, showcasing the detection capabilities of Defender for Cloud sensors. However, this tool can also be extended to work with other Intrusion Detection Systems (IDS) for monitoring Kubernetes environments. As with any detection solution, it is crucial to rigorously test various attack scenarios to ensure robust alerting mechanisms. The repository currently includes several predefined attack scenarios (illustrated in the image below), which can be triggered by running a simulation script and selecting specific options to monitor their effects. For this blog post, I’ll demonstrate the use of Tetragon, an open-source software leveraging eBPF-based security observability and runtime enforcement, in combination with Falco by Sysdig for enhanced attack detection.

Getting started

For you to work along with this simulation the main repository is listed below.

  • Python 3.7
  • Admin Permissions over the target Kubernetes Cluster
  • Helm client installed on local machine
  • If running on AKS (point your kubeconfig file to your target cluster with the following code below.)
az aks get-credentials --name [cluster-name] --resource-group [resource-group]

After completing the requirements above we can now grab our tool you can clone the repository or use the curl command.

https://github.com/microsoft/Defender-for-Cloud-Attack-Simulation

# Run the download tool
curl -O https://raw.githubusercontent.com/microsoft/Defender-for-Cloud-Attack-Simulation/refs/heads/main/simulation.py

For the cluster I’m using I’ve installed Tetragon using the following commands mind you I’m running Kubeadm.

helm repo add cilium https://helm.cilium.io
helm repo update
helm install tetragon ${EXTRA_HELM_FLAGS[@]} cilium/tetragon -n kube-system
kubectl rollout status -n kube-system ds/tetragon -w

Additionally, I’m also installing the Tetragon CLI for simplicity this allows me to use the synatx tetragon getevents -o compact.

GOOS=$(go env GOOS)
GOARCH=$(go env GOARCH)
curl -L --remote-name-all https://github.com/cilium/tetragon/releases/latest/download/tetra-${GOOS}-${GOARCH}.tar.gz{,.sha256sum}
sha256sum --check tetra-${GOOS}-${GOARCH}.tar.gz.sha256sum
sudo tar -C /usr/local/bin -xzvf tetra-${GOOS}-${GOARCH}.tar.gz
rm tetra-${GOOS}-${GOARCH}.tar.gz{,.sha256sum}

This is optional if you want to observe from this perspective. The output should look much more user-friendly in the terminal as shown now.

Now switching over to show you how bare-bones the cluster is with pods deployed here is what is shown on kubectl get pods -A.

I’ve already cloned the repository’s simulation.py file and will run the scenarios.

python simulation.py

Now for this experiment I’m running all scenarios to observe what is going to be caught in our other terminal that is running tetragon get events. Select 6 if you want to do this or select your desired scenario.

Switching over to the console with Tetragon running we can pick up some activity.

We can also see the results of running another scenario the ‘Web Recon’.

Another way to filter out the events given this is targeting a specific namespace to launch the scenarios is running the following using the CLI.

tetra getevents -o compact --namespace mdc-simulation-attacker

For some reason the namespace itself had a hard time returning and timed out for monitoring on Falco you can use the following helm chart for the install.

# Step 1
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

# Step 2
helm install --replace falco --namespace falco --create-namespace --set tty=true falcosecurity/falco

# Step 3
kubectl wait pods --for=condition=Ready --all -n falco

If you choose to run just as the helm chart install you’ll have to run the following to get the logs from Falco.

kubectl logs -l app.kubernetes.io/name=falco -n falco -c falco

So now in one tab you have the Falco logs monitored and in another tab we launch our attack scenarios simultaneously and see what is picked up.

We can see we’ve caught the Notice Shell spawned by untrusted binary in near-real time along with the other attacks on the Kubernetes API Server from a container.

After you’ve completed the scenarios you can hit Control + C to kill the process then delete your resources.

Summary

The scenarios outlined are used for testing the detection mechanisms of the solutions protecting your cluster. It is important to note that these scenarios should be executed on non-production clusters. As with any detection strategy, adopting defense-in-depth approach is critical particularly given the challenges of logging at the speed of Kubernetes operations. I am an avid fan of real-time detection systems, such as Falco and eBPF-based solutions, which provide a comprehensive view of cluster activity. I am currently using these scenarios to enhance the detection mechanisms in my open-source tool, Paranoia, which aims to improve Kubernetes security. By leveraging these simulations, you can validate the effectiveness of your cluster’s defenses and identify potential gaps.