AKS and Prometheus

What is Prometheus?

Prometheus is an open source monitoring system which was originally built by SoundCloud. It is now a part of the Cloud Native Computing Foundation. Prometheus scrapes metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is met.

Azure Kubernetes Service (AKS) is a managed container orchestration service based on the open-source Kubernetes system. AKS reduces the complexity and operational overhead of managing Kubernetes by offloading much of that responsibility to Azure. It also simplifies the process of deploying and managing containerized applications on Azure.

Running AKS with Prometheus in Open-Source for observability and metrics is a monitoring solution that has gained popularity as it’s a graduated project from CNCF.

to use Prometheus with Azure Kubernetes Service.

Installing Prometheus is a simple process that can be completed in just a few steps. The first step is to create a new resource group in Azure with the following command:

az group create --name aks-prom --location eastus

Once the resource group has been created, you can then proceed to create the actual Azure Kubernetes Service cluster with the following command:

az aks create --resource-group aks-prom --name aks-prom-test --node-count 3 --enable-addons monitoring --generate-ssh-keys

After the cluster has been provisioned, you can then install Prometheus using the Helm package manager. Helm is a tool that makes it easy to install and manage Kubernetes applications. You can install Helm with the following command:

git clone https://github.com/prometheus-operator/kube-prometheus.git

cd kube-prometheus

# Create the namespace and CRDs, and then wait for them to be available before creating the remaining resources
# Note that due to some CRD size we are using kubectl server-side apply feature which is generally available since kubernetes 1.22.
# If you are using previous kubernetes versions this feature may not be available and you would need to use kubectl create instead.
kubectl apply --server-side -f manifests/setup
kubectl wait \
	--for condition=Established \
	--all CustomResourceDefinition \
	--namespace=monitoring
kubectl apply -f manifests/

After this we will run the command below to check out our pods in monitoring namespace

kubectl get pods -n monitoring

So as we can see we have some that are still Pending we should allow these to finish deploying and then check when its state is Running

Next run the following command

kubectl --namespace monitoring port-forward svc/grafana 3000

Important note if your doing this on your cloud shell you’ll want to do this on your own terminal as I had issues with setting up the preview via gateway

This is grafana dashboard that is populated with a login and password by default this is set to admin:admin (important to change this……!)

After we reset our password to our dash we should have a view that looks like this

As we can see we have a few options we can use the + Import to bring in data connectors for monitoring

So that’s a small snapshot of Grafana for visualization now lets explore what Prometheus looks like in our browser

For some reason after this install I was having issues with the pod moving from pending to running after running the script installed a few times as recommended by the github. So I went with artifact hub and used helm to redeploy this following below

https://artifacthub.io/packages/helm/prometheus-community/prometheus

Once Helm has been installed, you can then add the official Prometheus charts repository with the following command:

helm repo add prometheus https://prometheus-community.github.io/helm-charts

helm install my-prometheus prometheus-community/prometheus --version 19.3.1

After this since I’m in Windows I ran the command a little different so if your in Windows the command will look like this

kubectl --namespace default port-forward prometheus-server-f5ffc8766-dp87c 9090

Navigate to your local browser and type in the following

http://localhost:9090

So now we run a PromQL Query to get info luckily (auto-completion) helps us with this

So now I can see what type of allocated ips are to the kube-apiserver along with any context relevant.

Prometheus provides a web UI that you can use to run PromQL queries and visualize the results.

To query for a metric, select the metric from the drop-down menu and then enter your PromQL query in the text box. For example, to see a list of all pods in our Azure Kubernetes Service cluster, we can use the following query:

kube_pod_info{}

This will return a table with information on each pod in our cluster, including its name, namespace, status, and so on.

We can also use PromQL to plot graphs of our metrics data. To do this, select the “Graph” tab and then enter your PromQL query in the text box. For example, to plot the CPU usage of all pods in our cluster over time, we can use the following query:

sum(rate(container_cpu_usage_seconds_total{}[10m])) by (pod)

So we can now have visualization of the CPU utilization across our cluster from the pods separated by colors you can get this more granular as well this is just a simple example.

Prometheus is a powerful monitoring tool that can be used to monitor Azure Kubernetes Service clusters. In this blog post, we have seen how to install and configure Prometheus, and how to use PromQL to query metrics and visualize the results.

Why use Prometheus with Azure Kubernetes Service?

There are many benefits to using Prometheus with Azure Kubernetes Service, including:

• Improved performance monitoring and logging: By default, AKS only provides basic performance monitoring data. However, by installing and configuring Prometheus, you can gain access to much more detailed information about your cluster’s performance. This can be extremely helpful when trying to diagnose and fix performance issues.

• Better visibility into resource usage: With Prometheus, you can not only monitor overall resource usage for your AKS cluster but also drill down to see how individual pods or containers are consuming resources. This can be very useful for identifying and addressing potential bottlenecks.

• Increased flexibility: Prometheus is highly customizable and can be configured to collect exactly the data you need. It also supports a wide range of data visualization tools, so you can choose the best way to view and analyze your data.

Overall, using Prometheus with Azure Kubernetes Service can provide a significant boost to your cluster’s performance and stability.

Conclusion

If you’re looking for a powerful monitoring solution for your Azure Kubernetes Service, Prometheus is a great option. Not only is it open source, but it also has a wide range of features that make it ideal for monitoring large and complex systems like AKS. Plus, thanks to its integration with Grafana, you can easily visualize all your data in one place.

So if you’re ready to get started with Prometheus on AKS, check out our guide on how to install and configure it. And if you need help getting started with Grafana, we’ve got you covered there too.