RodrigTech

Mutability of FIPS on AKS

—

by

rodrigtech
in

Introduction

Your in compliance and tasked with identifying which microservice supported supports Federal Information Processing standards. Operations are dynamic and can change from supporting a business unit that might have this requirement, so what are you options if you have to revert and keep the cluster? Currently in Azure Kubernetes Service this has been capable of supporting FIPS 140-2 (Level – 2) however if operations shift or a specific node you’ve created with this has to repurposed this wasn’t achievable. This new preview feature that has been released supports mutability to change the existing node pool to either enable or disable this capability, this will update the node in place to the FIPS 140-2 Image. This blog post is going to demonstrate the use of a existing cluster that isn’t enabled with FIPS 140-2 and the configuration changes when we enable this on the cluster.

Setup

  • Azure Subscription
  • Azure Kubernetes Service
  • Azure CLI Preview Updated/Feature Flags

Getting started

For the features to work on our existing subscription this service uses the “Microsoft.ContainerService” feature.

az login --use-device-code
# Register the aks-preview extension
az extension add --name aks-preview

# Update the aks-preview extension
az extension update --name aks-preview

Once these are updated to the latest preview version this will allow us to retrieve the next service as registered.

az feature register --namespace "Microsoft.ContainerService" --name "MutableFipsPreview"

This will return the following in the CLI screen as “Registering” this takes a few minutes so allow this to register then the next command will review the status as “Registered”

az feature show --namespace "Microsoft.ContainerService" --name "MutableFipsPreview"

This will now reflect Registered and we can move on to running the next command.

az provider register --namespace Microsoft.ContainerService

Once this command runs and is completed we can now use the following to spin up a cluster I’ve created a script for this but you can keep this straight-forward.

#!/bin/bash

echo "Creating the resource group"
az group create --name aks-fips-enabled --location eastus

echo "Creating a AKS Cluster with FIPS Enabled"

az aks create \
  --resource-group aks-fips-enabled \
  --name aks-fips-enabled \
  --node-vm-size Standard_D2s_v3 \
  --enable-fips-image \
  --api-server-authorized-ip-ranges <x.x.x.x/x>

You can save this as aks.sh if you are using vim/nano then chmod +x for executable format then run as follows.

./aks.sh

Once we have our cluster credentials and access it we can run the following to check if FIPS is enabled.

az aks show --resource-group aks-fips-enabled --name aks-fips-enabled --query="agentPoolProfiles[].{Name:name enableFips:enableFips}" -o table

Since we’ve enabled FIPS at cluster creation if we run the following on our node we can see what image changes with the node.

kubectl describe node aks-nodepool-<insert-your-info>

After we run the code of the AKS Show we can see the following in our output desired as a table.

When we run the following we can gather all nodes under our nodepool1 resource.

kubectl get nodes -o wide

So if we also want to take a step further on validating the FIPS-enabled node pool we can access our node directly using a debug command.

kubectl debug node/aks-nodepool1-84504712-vms000000 -it --image=mcr.microsoft.com/dotnet/runtime-deps:6.0

You’ll want to change the aks-nodepool1 if this references something different when you run the kubectl get nodes -o wide.

cat /proc/sys/crypto/fips_enabled

This returning a 1 as true in this regard remember that node pools will have the annotation of kubernetes.azure.com/fips_enabled=true as a label this can be the target of workloads requiring this standard.

Now as far as disabling FIPS we will run the following on our existing node pool.

az aks nodepool update --resource-group aks-fips-enabled --cluster-name aks-fips-enabled --name nodepool1 --disable-fips-image

This will run the command and will take some time you’ll get a banner as shown below as this is in preview.

We can run the output table command against the cluster to validate the changes.

az aks show --resource-group aks-fips-enabled --name aks-fips-enabled --query="agentPoolProfiles[].{Name:name enableFips:enableFips}" -o table

You’ll see that now our FIPS Image has been updated on the node pool representing nodepool1 to now false if we run a kubectl get nodes -o wide to see the image upgrade you’ll notice each Virtual Machine Scale Set has changed from the upgrade.

This changes from the previous node version that annotates the FIPS in the image as vanilla Ubuntu 22.04.

To delete resources associated we can run the following

az group delete --name aks-fips-enabled

Summary

Mutability opens up the option to previously move existing cluster or add node pools that are fips enabled and allow for disablement as needed. Ideally this opens up less recreation and can use existing clusters with segmented node pools with this offering. As always evaluate your business and technical requirements for this specific feature but its important to node when updating the node pool whether this is enabling the feature or disable this will take that nodepool in my case only with one nodepool downtime while this happens so if you are managing a smaller cluster ensure some movement of workloads to another node if possible to avoid downtime.