RodrigTech

AKS Isitio Bring Your Own Certificate

—

by

rodrigtech
in

Azure Kubernetes Service added the Istio-add-on to provide native function to the user without having to install an additional helm chart while this is managed by Microsoft a request for some of the areas from customers encompass areas that have more control over data protections. Natively the use of self-signed root certificates are generated and the key uses them to sign workload certificates, if you want to bring your own certificate for keys for the Istio Certificate Authority this leverages more control over the certificate used to sign workloads. This natively integrates with Azure Key Vault and requires a few enablement’s prior to deploying.

Source Image: https://learn.microsoft.com/en-us/azure/aks/istio-plugin-ca

Requirements

  • Azure CLI Version 2.49.0 or later installed
  • AKS-preview – Azure CLII extension of version 0.5.163 or later installed
  • Azure Kubernetes Service Cluster
  • Azure Key Vault (this is used to hold our own certificate)

Getting started

For starting off lets validate we have the required extension in our terminal.

Think about what you are going to use for areas associated components such as this requires a AKS cluster I typically deploy this with terraform for quick demonstrations but also ensure your not exposing your cluster to the open internet. For Istio on the AKS module you can use the following code to ensure Istio is deployed upon AKS creation.

  service_mesh_profile = {
    enabled = true    mode    = "Istio"     internal_ingress_gateway_enabled = true    external_ingress_gateway_enabled = true  }

To assist you with creation I’m working off my repository here to deploy AKS in a authorized access for testing I use and added Istio into the configuration.

#Assuming you’re logged into your cloud shell/or Linux machine/Windows
az login
az extension add —-name AKS-preview

#Update this with the following to ensure your on the latest
az extension update —-name AKS-preview

Once this finishes, we will continue to register the following features in our subscription that enable this capability.

az feature register —-namespace “Microsoft.ContainerService” —-name “AzureServiceMeshPreview”

#Validate this with the following command
az feature show —-namespace “Microsoft.ContainerService” —-name “AzureServiceMeshPreview”

Once this output reflect “Registered”
az provider register —-namespace Microsoft.ContainerService

You’ll want to use a production-ready CA and storage such as Azure Key Vault if your cloud agnostic a huge proponent in this space would be Hashicorp Vault. However following the next steps we have to generate certificates.

Assuming you are going to use the istio package natively you can run the following so you don’t clone the repo.

We will start with making our directories

mkdir -p certs
pushd certs

Wget https://github.com/istio/blob/master/tools/certs/Makefile.selfsigned.mk

# Generate the root certificate and key
make -f Makefile.selfsigned.mk root-ca

This will populate the following using openssl

  • Root-cert.pem – this is generated the root certificate
  • root-key.pem – generated root key
  • root-ca.conf – the configuration for OpenSSL to generate the root certificate
  • root-cert.csr – the generated CSR for the root certificate

Azure Key Vault

Adding the secrets into the Azure Key Vault are using the following assuming you have a key vault created if not I’ll provide the commands to assist you.

# creates the resource group
az group create —-name “Istio-KV” —-location “EastUS”

#Create our key vault in our resource group
az keyvault create —-name “IstioKV100” —-resource-group “Istio-Ops” —-location “EastUS”

$AKV_NAME=“IstioKV100”
#Set secrets
az keyvault secret set —-vault-name $AKV_NAME —-name root-cert —-file certs/root.pem
az keyvault secret set —-vault-name $AKV_NAME —-name ca-cert —-file certs/ca-cert.pem
az keyvault secret set —-vault-name $AKV_NAME —-name ca-key —-file certs/ca-cert.pem
az keyvault secret set —-vault-name $AKV_NAME —-name cert-chain —-file certs/cert-chain.pem

Enable AKS Keyvault Provider for Secret Store CSI Driver for your cluster
az AKS enable-Addons —-addons azure-keyvault-secrets-provider —-resource-group “istio-ops” —-name “AKS-chaos-mesh-aks”

If you deployed Istio like I did by code you’ll have to disable it to provide some the certificates from your key vault to the cluster.

We have to authorize the user-assigned managed identity of the add-on to have access to the Azure Key Vault resources.

# This assumes you have declared environment 
RESOURCE_GROUP=aks-chaos-mesh-rg
CLUSTER=aks-chaos-mesh-aks
OBJECT_ID=$(az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER --query 'addonProfiles.azureKeyvaultSecretsProvider.identity.objectId' -o tsv)

az keyvault set-policy --name $AKV_NAME --object-id $OBJECT_ID --secret-permissions get list

We have to ensure that we also use the following if you have Istio enabled change this parameter to disable similar to this.

az aks mesh disable --resource-group $RESOURCE_GROUP --name $CLUSTER
az aks mesh enable --resource-group $RESOURCE_GROUP --name $CLUSTER \
--root-cert-object-name root-cert \
--ca-cert-object-name ca-cert \
--ca-key-object-name ca-key \
--cert-chain-object-name cert-chain \
--key-vault-id /subscriptions/$SUBSCRIPTION/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.KeyVault/vaults/$AKV_NAME

This should reflect the output once enablement finishes notice our cacerts is added if we inspect this secret we can see our values further.

The issue I had was my root-cert.pem was generated at a earlier time frame and the purge protection after I deleted this had the certificate stored so I ran into issues but showing what that output should look like are below.

Running the following commands this is my inputs to show you my error encountered.

kubectl logs deploy/istiod-asm-1-17 -c discovery -n aks-istio-system | grep -v validationController | grep x509

Our expected output should look like the example provided by the docs here.

2023-11-06T15:49:15.493732Z     info    x509 cert - Issuer: "CN=Intermediate CA - A1,O=Istio,L=cluster-A1", Subject: "", SN: e191d220af347c7e164ec418d75ed19e, NotBefore: "2023-11-06T15:47:15Z", NotAfter: "2033-11-03T15:49:15Z"
2023-11-06T15:49:15.493764Z     info    x509 cert - Issuer: "CN=Root A,O=Istio", Subject: "CN=Intermediate CA - A1,O=Istio,L=cluster-A1", SN: 885034cba2894f61036f2956fd9d0ed337dc636, NotBefore: "2023-11-04T01:40:02Z", NotAfter: "2033-11-01T01:40:02Z"
2023-11-06T15:49:15.493795Z     info    x509 cert - Issuer: "CN=Root A,O=Istio", Subject: "CN=Root A,O=Istio", SN: 18e2ee4089c5a7363ec306627d21d9bb212bed3e, NotBefore: "2023-11-04T01:38:27Z", NotAfter: "2033-11-01T01:38:27Z"

Since I had to rename root-ca to root-ca1 this is likely the choke point that caused the issue, the best part of this integration is your not out of luck you can also rotate with restarting the deployment which I plan to do again at a later time.

Summary

Azure Kubernetes Service continues to push out more features to natively support requests related to Istio and the movement forward of adoption is increasing across organization I could see this also being a design decision on more control to the end user of certificates. It should be noted if you are generating certificates this should be done in a air-gapped offline system due to the sensitivity of the certificates and stored in a production grade KMS. If you got this far you’re likely in the advanced iterations of cluster operations of service-mesh stay tuned for more on this.

Sources: https://learn.microsoft.com/en-us/azure/aks/istio-plugin-ca