How to Enable Audit Logging on a Kubernetes Cluster
In this blog post, we’ll show you how to enable audit logging on a Kubernetes cluster. Audit logging is a valuable tool for understanding what’s happening on your cluster, and can be used to troubleshoot issues or investigate potential security problems. We’ll cover what audit logging is, why you might want to enable it, and how to configure it.
Photo by Philippa Jean-Mather Brown on Pexels
Enable Audit Logging on a Kubernetes Cluster.
Audit logging is the process of tracking and logging events that occur on a computer system. These events can include user login information, file access times, and system configuration changes. Audit logs can be used to track down security breaches, troubleshoot system issues, and compliance purposes.
Why Enable Audit Logging.
Enabling audit logging on your Kubernetes cluster has several benefits. First, it allows you to track and monitor all activity that occurs within your cluster. This can be helpful in troubleshooting issues or investigating suspicious activity. Second, it provides a record of all activity that can be used for compliance purposes. Finally, it can help improve the security of your cluster by providing a detailed audit trail that can be used to detect and investigate potential security threats.
How to Enable Audit Logging
There are two main ways to enable audit logging on a Kubernetes cluster: through the use of an auditing webhook or by configuring the kube-apiserver to write log files to disk.
The first method is to configure an auditing webhook. An auditing webhook is a piece of code that runs inside the Kubernetes cluster and sends information about events that occur within the cluster to an external logging service. There are many different auditing webhooks available, so you will need to choose one that best suits your needs. Once you have chosen an auditing webhook, you will need to deploy it into your Kubernetes cluster and configure it according to your specific requirements.
The second method for enabling audit logging is to configure the kube-apiserver to write log files to disk. The kube-apiserver is the central component of a Kubernetes cluster responsible for handling all API requests. By default, the kube-apiserver does not generate any log files but it can be configured to do so by modifying its configuration file (kubeconfig).
Configure Audit Logging.
The first step in configuring audit logging is deciding what information should be logged. This will vary depending on the needs of your organization, but some common items to consider logging include:
-API calls made to the Kubernetes API server
-Changes made to Kubernetes resources (e.g., Deployments, Pods, Services)
-Users and service accounts accessing the cluster
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
rules:
- level: Metadata
verbs: ["delete", "get"]
resources:
- group: "" # core API group
resources: ["secrets", "configmaps"]Reference to this yaml is listed below https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/
For starters we can follow by the YAML policy taking a look at where the indention “level” is noted we are searching for the metadata in regards to the resources listed below so for this out of the box you can tailor for what you’re logging for.
We should consider monitoring secrets for delete operations as this can disrupt a service in our cluster if its pulling a secret that is deleted so as an example using the verbs clause allows us to put actions that we are searching for.
Next run the command below to move the file to somewhere we will reference later on.
mv audit-policy.yaml /etc/kubernetesNow we have to tell our kube-apiserver of this file and along with some other parameters so we can start capturing audit logs
Navigate to the old trusty /etc/kubernetes/manifests with the below command
cd /etc/kubernetes/manifests
So we are interested in the kube-apiserver.yaml file we will use the following command to open the file
vim kube-apiserver.yamlYour file will add the following lines shown in the image below and I’ll break those down
We are telling the kube-apiserver first where is the audit policy to enforce or follow in the –audit-policy-file (where we moved that file previously). Second we are designating where that log path will write the logs to so I used the /var/log area following docs and as far as the next lines defining age by days at 10 and max files stored to 3.
We are not out of the woods yet we are going to add the volume mounts and volumes the image below shows the volume mounts
Then we add the volumes as shown in the image below
Remember after this modification the kube-apiserver will reboot and it will take time to start up to watch this operation you can run the following command
crictl ps --latest
After we can communicate with the kube-apiserver we will run the following command
So this is a generic secret remember secrets (are base64 encoded) not necessarily safe in this regard we will cover encryption in another post on how to protect etcd.
And now we are starting to collect logs in the background I realized I made a typo in the kube-apiserver I’ll have this remediated on the next post but hopefully this gives you a idea of what is needed for audit logging as natively available.
Where to Log.
Once you have decided what information to log, you need to choose where to send the logs. This will usually be a central logging system such as Splunk, Elasticsearch, or Fluentd. There are many other options available, so consult your organization’s logging requirements before making a decision.
Conclusion
Audit logging is a critical security feature for any Kubernetes cluster. By enabling audit logging, you can track and monitor all actions taken on the cluster, which can help prevent and detect malicious activity. While audit logging can be enabled by default in most Kubernetes distributions, it is important to understand how to configure it to ensure that only the desired information is being logged. In this blog post, we covered how to enable and configure audit logging on a Kubernetes cluster. We hope you found this information helpful and encourage you to enable audit logging on your own clusters today.
References in this tutorial as always