By Richard Lander, CTO | May 3, 2024
A Kubernetes operator is a custom extension of the Kubernetes control plane.
There are many open source Kubernetes operators available for you to use for various purposes. This blog post will help you understand how they work. Additionally, operators can be very helpful in managing the workloads you develop and deliver to your end users. We will also outline how you can tackle developing them.
To understand Kubernetes operators, it is critical to understand how the Kubernetes control plane works in general. For this, see our blog post: What is the Kubernetes Control Plane?
This category of operators provides an abstraction for deploying and managing particular applications. They are most helpful with sophisticated stateful apps that are relatively involved to operate and for those apps that have multiple workload components. With this kind of operator a user often creates, updates, and deletes a single custom resource instance that triggers a Kubernetes controller to create, update, and delete all the Kubernetes resources that constitute that app. The app often consists of dozens of different resources so this kind of operator is extremely helpful in reducing operational toil and improving consistency and reliability. Popular examples include the Prometheus Operator and various database operators. These are a very common category of Kubernetes operator.
This kind of operator uses custom resources to define systems external to Kubernetes such as cloud provider resources. The AWS Controllers for Kubernetes are a good example of this.
Some operators don't directly manage entire app stacks or external resources, but instead provide configuration support services for applications. They often watch resources created by other systems and take config actions to support different workloads. A good example of this is cert-manager which is commonly used to manage SSL/TLS certificates based on Ingress resource configurations.
There are two main components to an operator.
A custom resource definition (CRD) is a core resource in Kubernetes. It allows you to use the OpenAPI spec to define the fields for a custom resource and some basic validation rules. When you create a CRD, you instantly extend the Kubernetes API to support a custom resource kind.
The CRD encapsulates the data model for your operator. When designing a CRD, focus on what pieces of information are required by your controller to make decisions about configuration for your use case. Start with a bare minimum of fields on your first iterations and add them as you find the need. And always be on the lookout for high level configuration values that can be evaluated by your controller to abstract away details of configuration.
Over-parameterization impacts the usability of your operator. If your custom resource accumulates dozens of required fields that necessitate deep configuration from the user, see if sensible defaults can be used to make them optional. Or find higher level configuration that can allow your operator to make the right decisions for the user.
Let's use the AppStack example from above. The following manifest is an example of requiring a user to set several granular configurations for:
/apiVersion: example.com/v1beta1
kind: AppStack
metadata:
name: dev-app-stack
spec:
replicas: 2
enableHorizontalScaling: false
resources:
requests:
memory: "64Mi"
cpu: "250m"
Instead, you could allow users to define a single high level config that lets the controller set all the granular values for them. In this example the replicas, the HPA and resource requests would all default to sensible values for each of the three tiers. You are unlikely to need the HPA for dev instances, but likely will in prod, for example.
apiVersion: example.com/v1beta1
kind: AppStack
metadata:
name: dev-app-stack
spec:
tier: dev # dev | staging | prod
When providing high level options like this, recognize that sometimes granular config is required to test certain features or solve problems. The best abstraction is one that allows high level config when useful, but retains the ability to provide granular overrides when needed. That would mean the more granular fields are optional and will override the defaults used when the spec.tier field is set.
The custom controller contains the logic for configuring, deploying, updating, and deleting components of the software under management by the operator. In the app stack example, that would include all the resources that comprise the app stack.
The controller connects to the Kubernetes API and watches the custom resource/s that it is responsible for. In the diagrammed example above, the controller watches AppStack resources. When one of those resource is created, the controller deploys all the Kubernetes resources to spin up an instance of the app. When the AppStack resource is updated, the controller makes the necessary changes to the underlying Kubernetes resources. When the AppStack resource is deleted, all the underlying Kubernetes resources will be removed.
This encapsulates all the CRUD (Create, Read, Update, and Delete) operations that make up the essential functionality an operator must handle. This functionality will provide a vastly improved experience to users and significantly reduce complexity in automated delivery systems, e.g. GitOps pipelines, but this is just the beginning. You can now add advanced functionality that provides tremendous benefit for critical operations such as:
The Kubernetes API is a RESTful API and client libraries exist for many different languages. However, Kubernetes is written in Go and, if you write your operator in Go, you can use the core Kubernetes libraries and all of the features available in them. Furthermore, Go is a good language for the purpose. Most developers can pick up Go pretty quickly and use one of the SDKs to get them started.
Kubebuilder is the most common tool used for building operators. The project has excellent documentation and provides all the scaffolding and boilerplate for almost any operator project.
Operator Builder extends Kubebuilder and provides the opportunity to supply tested Kubernetes manifests to inform the code generation for your operator. Operator Builder provides significant time-saving when building an App Stack Management operator. It takes the Kubernetes manifests provided and generates all the Go code to define those resources. If you've ever had to code all the resource definitions in an operator project, you'll appreciate how much time can be saved. Operator Builder will generate all the code necessary for the CRUD operations in your controller.
The Operator SDK offers the ability to develop an operator with Go, Ansible or Helm. I have not seen a useful implementation using Ansible or Helm and would advise against it. Config tools do not have the capabilities of the Go programming language and, while they may be useful for a quick POC in some cases, you'll likely find the limitations will require a re-write in Go later anyway, especially when tackling advanced functionality beyond the CRUD operations.
Threeport is not a Kubernetes operator. Rather it is an application orchestrator that integrates very well with the operator pattern. We use operators as a part of Threeport and strongly advocate for their use on high value workloads. If you're managing cloud native applications, try out Threeport by downloading the CLI at GitHub and see our docs instructions to get started.
If you like what you see in Threeport and want the features without the overhead of managing Threeport control planes yourself, check out Qleet. We provide managed Threeport control planes which are supported by the developers of Threeport.