Multi-tenancy in Kubernetes with Kiosk

Multi-tenancy shouldn’t be unfamiliar to those who’ve already adopted some type of cloud model regardless if it’s hybrid and not full cloud-native. This also has been a area of adoption in the realm of kubernetes as likely every organization is facing financial optimization decisions. Today’s blog post explores the tool known as Kiosk that is by a organization known as loft-sh that splits up your cluster in for multiple teams to utilize with isolation.

Important to note multi-tennacy can be achieved with logical isolation such as separation of organizations in a cluster with namespaces, or physically isolated this is essentially a cluster for different departments.

To understand this further is simply put you have options like anything in cloud but especially with kubernetes the sprawl of many clusters can create a issue with administration if multiple departments are utilizing services it can help centralize this with its own namespace with defined parameters. Kiosk provides the image below as a depiction of the isolation and how the tool works in operations.

https://github.com/loft-sh/kiosk/blob/master/docs/website/static/img/kiosk-workflow-kubernetes-multi-tenancy-extension.png

Getting Started

Some pre-requisites if you’d like to follow along I’m using KinD (locally) for quick operations.

Requirements

Kubernetes version 1.14 and higher
Kubectl cli
Helm
Admin Context (if you can run the following command you’re likely a admin) kubectl auth can-i “*” “*” –all-namespaces

Installation starts with the following commands

kubectl create namespace kiosk
helm install kiosk --repo https://charts.devspace.sh/ kiosk --namespace kiosk --atomic

To verify the installation

kubectl get pod -n kiosk

After the installation is completed we are going to create a account to use throughout our demo, for this we are going to use the account john

kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/account.yaml

What this YAML has states we are calling the api tenancy.kiosk.sh/v1alpha1 with the request of a account – known as john the kind indicates the grouping of the object.

We now can run the following command to see our account and spaces ‘kubectl get accounts –as=john’

Spaces – What are these?

Spaces are essentially what kiosk classifies as representation of namespaces. Each space represents one namespace, this will allow us to split the resource to allow resources to be segmented on what can be seen by those authorized and hide other spaces from others.

Starting to allow users to create spaces

Mind you this command allows All account users to create spaces for their own accounts.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kiosk-creator
subjects:
- kind: Group
  name: system:authenticated
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: kiosk-edit
  apiGroup: rbac.authorization.k8s.io

Kiosk documentation notes you can adjust clusterrolebinding in a way that only certain subjects/users can create Spaces for their own account we’d have to modify the subjects.

# Run this as cluster admin:
kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/rbac-creator.yaml

After this is applied to our cluster we can now use the user john via impersonation to create a space.

kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/space.yaml --as=john

So we can also note that we are referring to creation of a new space and also specifying who it belongs to.

So remember when we add –as=john in our command we are requesting essentially what the user john sees when he runs this command he is assigned johns-space as his area and only allowed to see that.

Let’s see how operations work now for john as he will request to create resources inside his space

kubectl apply -n johns-space --as=john -f https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/application/deployment.yaml

We are authorized to create a deployment to the space under the alias john seamlessly.

Okay but where is the power in this?

Well this is just the logical isolation next let’s enforce some limits to ensure john and his team of developers aren’t over utilizing our cluster resources. The way we do that is creating limit number of spaces for accounts, and set account quotas known as maximum values for aggregated resources across all spaces of a accounts (purview).

Run a vi (editor) name the yaml file as you’d like and enter the following

apiVersion: tenancy.kiosk.sh/v1alpha1
kind: Account
metadata:
  name: johns-account
spec:
  space:
    limits: 2
  subjects:
  - kind: User
    name: john
    apiGroup: rbac.authorization.k8s.io

So now we’ve instructed johns assigned account a limit – we run the similar command to list the following spaces under john with this depicted now we try to deploy more spaces.

kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/space-2.yaml --as=john

#for space 3 this should populate as shown below
kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/space-3.yaml --as=john

Account Quotas

You’ll likely be familiar at this point with using OPA Gatekeeper or native integration if your running hosted k8s such as GKE/AKS/EKS with policies/constraints that can be implemented to kubernetes. For this area we can define at the granular level on the user what are the quotas ability for the user to deploy

apiVersion: config.kiosk.sh/v1alpha1
kind: AccountQuota
metadata:
  name: default-user-limits
spec:
  account: johns-account
  quota:
    hard:
      pods: "2" 
      limits.cpu: "4"

kubectl apply -f https://raw.githubusercontent.com/kiosk-sh/kiosk/master/examples/accountquota.yaml

Feel free to edit this as you’d like to test out the functionality further, after the deployment we’ve stated now that we can allow 2 pods and 4 cpus as limit for our friend john.

I didn’t specify any cpu-limits in the cli command and got rejected based on the quota enforcement we can see this hits the admission webhook.

To demonstrate further remember we’ve deployed a nginx deployment earlier? That is a part of johns quota which is also demonstrated further.

Summary

This is just scratching the surface to a ongoing movement in kubernetes of shifting to multiple clusters as isolation with extending namespaces as virtual restrictions and quotas. Kiosk provides really good documentation outlining use cases further such as mandorty or optional templates along with using Template Sync to ensure that template instances are updated this will tell kiosk to keep the template instance in sync with underlying template with a 3-way merge.

You can read more on this project and contribute at the following github repo.

https://github.com/loft-sh/kiosk