Some applications need a lot of operational knowledge - not just the complexity of modelling the app in Kubernetes, but ongoing maintenance tasks. Think of a stateful application where you want to create backups of data. The application deployment is modelled in Kubernetes resources, and it would be great to model the operations in Kubernetes too.
That's what the operator pattern does. It's a loose definition for an approach where you install an application which extends Kubernetes. So in that stateful application your operator would deploy the app, and it would also create a custom DataBackup resource in the cluster. Any time you want to take a backup, you deploy a backup object, the operator sees the object and performs all the backup tasks.
One of the problems with public operators is that you take a dependency on a third-party to maintain and update their deployments. The operators we use in this lab are good examples - they don't support ARM64 processors, so if you're using Apple Silicon then you won't be able to run all the exercises.
Operators typically work by adding CustomResourceDefinitions (CRDs) to the cluster.
CRDs are pretty simple in themselved, you deploy an object which describes the schema of your new resource type:
You deploy CRDs in the usual way:
kubectl apply -f labs/operators/specs/crd
📋 List all the custom resource in the cluster, and print the details of the new Student CRD.
kubectl get crd
kubectl describe crd students
The CRD is just the object schema. Kubernetes only stores objects if it understands the type, which is what the CRD describes.
Now your Kubernetes cluster understands Student resources, you can define them with YAML:
📋 Create all the Students in the labs/operators/specs/students folder, list them and print the details for Priti.
These objects are standard YAML:
kubectl apply -f labs/operators/specs/students
And the resources can be accesses using the CRD name:
kubectl get students
kubectl describe student priti
If you try to apply this YAML in a cluster which doesn't have the Student CRD installed, you'll get an error.
The standard Kubectl verbs (get, delete, describe) work for all objects, including custom resources.
A CRD itself doesn't do anything, it just lets you store resources in the cluster. Operators typically install CRDs and also run controllers in the cluster. A controller is just an app running in a Pod which connects to the Kubernetes API and watches for CRDs being created or updated.
When you create a new custom resource, the controller see that and takes action - which could mean creating Deployment, Services and ConfigMaps, or running any custom code you need.
NATS is a high-performance message queue which is very popular for asynchronous messaging in distributed apps. The NATS operator runs as a Deployment:
Install the operator and look carefully at the output:
kubectl apply -f labs/operators/specs/nats/operator
The operator installs some RBAC objects and the Deployment
📋 List the custom resource types in your cluster now. You'll see some NATS types - how did these get created?
kubectl get crd
Shows your custom Student resource, and also NatsCluster and NatsServiceRole resources.
There's no YAML for these CRDs, so the NATS controller running in the Pod must have created them by using the Kubernetes API in code.
You can confirm the RBAC setup gives the controller ServiceAccount permission to do that:
kubectl auth can-i create crds --as system:serviceaccount:default:nats-operator
We can use a NatsCluster object to create a clustered, highly-available message queue for applications to use:
Create the cluster resource:
kubectl apply -f labs/operators/specs/nats/cluster
A single object gets created.
📋 Print the details of your new message queue, and look at the other objects running in the default namespace. The operator logs will show how the Pods were created.
The output from the CRD doesn't show much:
kubectl get natscluster -o wide
But the operator has created Pods and Services:
kubectl get all --show-labels
Check the logs and you'll see the operator is managing the Pods - there's no Deployment or ReplicaSet for the message queue Pods:
kubectl logs -l name=nats-operator
The NATS operator is unusual because it acts as a Pod controller. Typically operators build on top of Kubernetes resources, so they would use Deployments to manage Pods.
Print the details of one of the NATS Pods and you'll see it's managed by the operator:
kubectl describe po msgq-1
You'll see Controlled By: NatsCluster/msgq
📋 The NATS operator still provides high availability. Delete one of the message queue Pods and confirm it gets recreated.
kubectl delete po msgq-2
kubectl get po -l app=nats
kubectl logs -l name=nats-operator
You'll see a new Pod called msgq-2 gets created, and the operator logs show it coming online.
There's not much more you can do with the NATS operator, so we'll try one which has some more features.
There's a Helm chart for the Presslabs MySql operator in this repository:
Install the operator (you'll need the Helm CLI installed):
helm install mysql-operator labs/operators/specs/mysql/operator/
# the operator pod might restart and take a few minutes to be ready:
kubectl get po -l app.kubernetes.io/name=mysql-operator -w
📋 What resources do you need to create to deploy a MySql database cluster using the operator?
The Helm output gives you an example of what you need:
a MysqlCluster object - this is a CRD installed by the operator
a Secret containing the admin user password for the database
You can create a replicated database cluster using these specs:
mysql/database/01-secret.yaml - the database password
mysql/database/db.yaml - the cluster set to use two MySql servers
Create the database:
kubectl apply -f labs/operators/specs/mysql/database
📋 The database Pods take a while to start up. What controller does the operator use, and what's the container configuration in the Pods?
List the Pods and you'll see db-mysql-0. That name should suggest that it's managed by a StatefulSet:
kubectl get statefulset
Print the Pod details and you'll see multiple containers:
kubectl describe po db-mysql-0
The container setup is pretty complext:
Check the logs of the primary database server in Pod 0:
kubectl logs db-mysql-0 -c mysql
You'll see mysqld: ready for connections showing the database server is running successfully
And the logs of the secondary database server in Pod 1:
kubectl logs db-mysql-1 -c mysql
You'll see 'replication@db-mysql-0.mysql.default:3306',replication started showing the secondary is replicating data from the primary.
The operator provides a production-grade deployment of MySql, and it also sets up a CRD for creating database backups and sending them to cloud storage.
We'll make use of the operators to install infrastructure components for a demo app.
Start by deleting the existing message queue and database clusters:
kubectl delete natscluster,mysqlcluster --all
The operators are watching for resources to be deleted, and will remove all the objects they created
Now deploy a simple to-do list application:
kubectl apply -f labs/operators/specs/todo-list
The app has a website listening on http://localhost:30028 which posts messages to a queue when you create a new to-do item. A message handler listens on the same queue and creates items in the database.
Browse to the app now and you'll see an error - the components it needs don't exist yet
You'll need to create NatsCluster and MysqlCluster objects matching the config in the app to make everything work correctly.
Delete the basic objects and CRDs:
kubectl delete crd natsclusters.nats.io natsserviceroles.nats.io
kubectl delete all,cm,secret,crd -l kubernetes.courselabs.co=operators
The order is important, deleting CRDs deletes custom resources - make sure the controller still exists to tidy up
Delete the NATS operator:
kubectl delete -f labs/operators/specs/nats/operator
Delete the MySql CRD and operator:
kubectl delete crd -l app.kubernetes.io/name=mysql-operator
helm uninstall mysql-operator