AKS Running Istio and Prometheus

Terraform by popularity and selection is by far the premier Infrastructure as Code go to for most Infrastructure due to the ability to universally move from one CSP to another. In this blog I will use a few modules such as kubectl and helm that will be used to install Istio on AKS. I’ve been experimenting with more advanced features of using more of the functionality outside of the native resources in Azure. If you’d like to follow along the pre-requisites are listed below

Pre-requisites

Kubectl has to be installed locally for this function properly
Azure CLI
Terraform installed on the client (shell) if your running on Windows OS use WSL
Azure Subscription (this will incur some charges so use at your discretion)

Since we are doing this from the CLI we first start with the basics of authenticating

az login

This will pop open a browser and you’ll authenticate after this is completed we are now ready to get started

I’m working out of this repo I’ve forked and modified

https://github.com/sn0rlaxlife/istio-aks-example

git clone https://github.com/sn0rlaxlife/istio-aks-example.git

Additionally if you want to utilize the istioctl you’ll have to run the following commands to get this on your local machine

If you’d like to do this you’ll have to run the following

curl -L https://istio.io/downloadIstio | sh -

If you want a specific version you can run the following below

curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.17.0 TARGET_ARCH=x86_64 sh -

Then we move into the folder

cd istio-1.17.0

Now we add the istioctl client to your path (Linux or MacOS)

export PATH=$PWD/bin:$PATH

Okay if I lost you no worries again remember this is to get up and running with a configuration after all so it might be a little cumbersome for this set up but well worth the effort.

Navigating the Repo

The folder structure is as follows

-istio-aks-example

—istio-on-aks

—multicluster-istio-on-aks

cd istio-aks-example
cd istio-on-aks

If we then run a ls command we should see this output

If you want to change the node machine image you’ll edit the tfvars file

By default this will run the following image Standard_DC4s_v2

cp tfvars .tfvars
terraform init -upgrade
terraform apply -var-file=.tfvars

If you get this message everything has been initialized and we are ready to proceed to the apply portion mind you – run terraform plan so you can catch anything for these purposes I’m skipping this.

terraform plan

AKS Details

You can also run a terraform validate if you’d like to however I will apply this as the output matches my preferences

terraform apply -var-file=.tfvars

I did get a prompt on a argument being deprecated I will address this at a later time after this post.

Depending on how your subscription is set up you might have a quota limit on specific nodes ability if this is the case you have request a quota increase.

If you run into this issue check on the Quotas page for what the limitations are set on your subscription I had to modify my size of the tfvars for this

After quite sometime for installation of the prometheus helm chart we are now up and running lets take a peak at a our cluster

Let’s grab our credentials then authenticate to our cluster

az aks get-credentials --resource-group istio-aks --name istio-aks

kubectl get pods -A

Injecting Sidecar

For us to inject the side car we will run the following command to annotate the label on our default namespace

kubectl label namespace default istio-injection=enabled

I took a quick view at the Azure Portal to see the resources in our resource group

We can see the sidecar status running the following command

istioctl proxy-status

We will run the following command for a example

kubectl apply -f - <<EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      run: echoserver
  template:
    metadata:
      labels:
        run: echoserver
    spec:
      containers:
      - name: echoserver
        image: gcr.io/google_containers/echoserver:1.10
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: echoserver
  namespace: default
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    run: echoserver
EOF

Now let’s rerun the following command

istioctl proxy-status

We can see our addition of the sidecar

Configure Gateway

Our terraform provisioned a istio-ingress the Gateway is an envoy pod at the border of the service mesh.

We can view this running the following command

istioctl proxy-config listener -n istio-ingress $(kubectl get pod -n istio-ingress -oname| head -n 1)

The gateway is exposed with a Kubernetes service LoadBalancer, to grab those details run the following command.

kubectl get service -n istio-ingress istio-ingress -o=jsonpath='{.status.loadBalancer.ingress[0].ip}'

Without any binding the connection will be refused we will configure the gateway resource

kubectl apply -f - <<EOF
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: istio-ingressgateway
  namespace: istio-ingress
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - '*'
EOF

Your output should look similar to the image above.

Now we will run the following command to show the gateway listening on port 80 this will still serve a HTTP 404.

istioctl proxy-config listener -n istio-ingress $(kubectl get pod -n istio-ingress -oname| head -n 1)
curl -v $(kubectl get service -n istio-ingress istio-ingress -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')

Configuring the Virtual Service

For the next iteration of this post we will configure the virtual service this apiversion is referenced as virtualservices.networking.istio.io. We are describing how a request is routed to our service.

Since we deployed the echoserver to route the request ClusterIP service created in the default namespace we will use the following code

kubectl apply -f - <<EOF
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: echoserver
  namespace: default
spec:
  hosts:
    - "*"
  gateways:
    - istio-ingress/istio-ingressgateway
  http:
  - match:
    - uri:
        prefix: "/"
    route:
    - destination:
        host: "echoserver.default.svc.cluster.local"
        port:
          number: 8080
EOF

Then we will check if we can reach it running a curl command to echoserver pod.

curl -v $(kubectl get service -n istio-ingress istio-ingress -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')

We are now in business with the status 200!!

Ok back to business lets run a health check probe on port 15021 with the following code

kubectl apply -f - <<EOF
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: healthcheck
  namespace: istio-ingress
spec:
  hosts:
    - "*"
  gateways:
    - istio-ingress/istio-ingressgateway
  http:
  - match:
    - uri:
        prefix: "/probe"
    rewrite:
        uri: "/healthz/ready"
    route:
    - destination:
        host: "istio-ingress.istio-ingress.svc.cluster.local"
        port:
          number: 15021
EOF

PeerAuthentications to enforce mTLS

We can enforce mTLS referencing the API peerauthentications.security.istio.io

kubectl apply -f - <<EOF
---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT
EOF

So two things to consider this is enforcing mTLS at the receiver of the TLS connection. We now have to work with Destination Rules to enforce at the client level.

Destination Rules

By applying the following DestinationRule the mTLS protocol is enforced when a connection is started by any client. Note: Istio ingress gateway will not be able to connect to a backend without a sidecar…..

Further documentation for working with this are located at the link below

https://istio.io/latest/docs/reference/config/networking/destination-rule/

kubectl apply -f - <<EOF
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: force-client-mtls
  namespace: istio-system
spec:
  host: "*"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
EOF

Verifying Encryption

Let’s start the verification process of our mTLS we will tcpdump from an AKS node this will use https://github.com/alexei-led/nsenter

wget https://raw.githubusercontent.com/alexei-led/nsenter/master/nsenter-node.sh
bash nsenter-node.sh $(kubectl get pod -l run=echoserver -o jsonpath='{.items[0].spec.nodeName}')
tcpdump -i eth0 -n -c 15 -X port 8080

This will launch into a privileged shell then run the tcpdump command

This gargled numbers and letters is the output of interception this is not clear-text this is why encryption is important running in kubernetes.

Once your done with analyzing the output ensure you run exit

You’ll see the following output on your terminal then we will move into Observability

Observability in Prometheus

In the files we’ve cloned the prometheus.tf installs the Helm chart for Prometheus and Grafana, and creates the configuration to scrape the istio sidecars

Let’s access the dashboard

kubectl -n prometheus port-forward svc/prometheus-grafana 3000:80

Now lets navigate to localhost

http://127.0.0.1:3000

If all goes well your screen should look like this

Default username and default password is prom-operator. (Ensure this is changed…….!)

Navigate to the side menu that looks like the image below we are moving to the Imports page

If you have trouble finding this no worries the url will be http://127.0.0.1:3000/dashboard/import

Importing the dashboards for our Istio Visibility

https://grafana.com/grafana/dashboards/7645-istio-control-plane-dashboard/
https://grafana.com/grafana/dashboards/7639-istio-mesh-dashboard/
https://grafana.com/grafana/dashboards/7636-istio-service-dashboard/
https://grafana.com/grafana/dashboards/7630-istio-workload-dashboard/
https://grafana.com/grafana/dashboards/13277-istio-wasm-extension-dashboard/

Then select Import this is to show you what to expect

After those are imported we can now run the Authorization Policies

To see what the Istio dashboard looks like I used this as a Istio Service Dashboard

Authorization Policies

The Authorization Policies allow or deny requests depending on conditions if you want to explore them further https://istio.io/latest/docs/reference/config/security/conditions/

We are going to use the example that runs the nip.io simple wilcard DNS service for IP Address

kubectl apply -f - <<EOF
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: probe
  namespace: istio-ingress
spec:
  action: ALLOW
  rules:
  - to:
    - operation:
        hosts: ["*.nip.io"]
  selector:
    matchLabels:
      istio: ingressgateway
EOF

Now let’s run the following curl command

curl -v $(kubectl get service -n istio-ingress istio-ingress -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')/

No let’s run the following command appending the nip.io

curl -v $(kubectl get service -n istio-ingress istio-ingress -o=jsonpath='{.status.loadBalancer.ingress[0].ip}').nip.io/

Now lets run the next command to navigate back to our dashboard see if our request are populated

kubectl -n prometheus port-forward svc/prometheus-grafana 3000:80

Okay now we’ve done quite a lot in this if you’ve stuck with me hopefully you’ll takeaway that you can quickly get up and running with terraform and extending capabilities of kubernetes.

Ensure we destroy our resources running a terraform destroy