A Kubernetes operator is a controller that watches custom resources you define and takes action to make the cluster match their declared state. The idea is straightforward; the implementation has sharp edges if you do not know the patterns. Every production operator uses the same skeleton: a CRD that defines the resource shape, a controller with a reconciliation loop that observes and acts, finalizers for cleanup on deletion, status conditions for communicating state back to users, and RBAC rules that give the controller exactly the permissions it needs and nothing more.
This post builds a real operator end-to-end. The example — a DatabaseCluster operator that manages a stateful database deployment — is simple enough to fit on screen but complex enough to exercise every pattern you will need in a production operator: multi-resource ownership, status reporting, graceful deletion with finalizers, and the idempotency requirements that make reconciliation loops correct.
What an Operator Is, Mechanically
Kubernetes is built around a control loop: observe desired state, observe actual state, act to converge them. Built-in controllers (Deployment controller, StatefulSet controller) implement this loop for core resource types. An operator extends this model to custom resource types using the same machinery.
The components:
┌──────────────────────────────────────────────────────────────────┐
│ Kubernetes API Server │
│ │
│ Custom Resource Definitions (CRDs) │
│ DatabaseCluster, DatabaseBackup, ... │
│ │
│ Custom Resources (instances of CRDs) │
│ my-postgres (DatabaseCluster) │
└────────────────────────┬─────────────────────────────────────────┘
│ watch / list
│ create / update / patch
▼
┌──────────────────────────────────────────────────────────────────┐
│ Your Operator │
│ │
│ Informer: watches API server for CR changes │
│ Work Queue: deduplicates and buffers reconcile requests │
│ Reconciler: your code — observe → compare → act │
│ │
│ Manages owned resources: │
│ StatefulSet, Service, ConfigMap, PVC, Secret │
└──────────────────────────────────────────────────────────────────┘
controller-runtime (the library underlying both kubebuilder and Operator SDK) provides the informer, work queue, and manager infrastructure. You write the Reconcile() function.
Project Setup with Kubebuilder
Kubebuilder scaffolds the project structure, CRD Go types, and controller skeleton. It is the fastest correct starting point.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
# Install kubebuilder
curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"
chmod +x kubebuilder && sudo mv kubebuilder /usr/local/bin/
# Scaffold a new project
mkdir database-operator && cd database-operator
kubebuilder init --domain example.com --repo github.com/example/database-operator
# Add a new API (creates CRD type + controller skeleton)
kubebuilder create api \
--group db \
--version v1alpha1 \
--kind DatabaseCluster \
--resource --controller
|
This creates:
database-operator/
├── api/
│ └── v1alpha1/
│ ├── databasecluster_types.go ← CRD Go types — you edit this
│ └── groupversion_info.go
├── internal/
│ └── controller/
│ └── databasecluster_controller.go ← reconciler — you edit this
├── config/
│ ├── crd/ ← generated CRD YAML
│ ├── rbac/ ← generated RBAC YAML
│ └── manager/ ← operator Deployment YAML
├── main.go ← manager setup
└── Makefile
Defining the CRD Types
The Go struct becomes the CRD schema. Everything in Spec is user-controlled desired state. Everything in Status is controller-written observed state. They must be separate — the API server enforces this through the status subresource.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
|
// api/v1alpha1/databasecluster_types.go
package v1alpha1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/api/resource"
)
// DatabaseClusterSpec defines what the user wants
type DatabaseClusterSpec struct {
// +kubebuilder:validation:Enum=postgres;mysql
// +kubebuilder:default=postgres
Engine string `json:"engine"`
// +kubebuilder:validation:Minimum=1
// +kubebuilder:validation:Maximum=7
Replicas int32 `json:"replicas"`
// +kubebuilder:validation:Pattern=`^\d+\.\d+$`
Version string `json:"version"`
Storage StorageSpec `json:"storage"`
// +optional
Resources ResourceRequirements `json:"resources,omitempty"`
// +optional
// +kubebuilder:default=false
BackupsEnabled bool `json:"backupsEnabled,omitempty"`
}
type StorageSpec struct {
// +kubebuilder:validation:Pattern=`^\d+(Ki|Mi|Gi|Ti)$`
Size string `json:"size"`
// +optional
StorageClassName *string `json:"storageClassName,omitempty"`
}
type ResourceRequirements struct {
// +optional
CPU string `json:"cpu,omitempty"`
// +optional
Memory string `json:"memory,omitempty"`
}
// DatabaseClusterStatus is written by the controller, not the user
type DatabaseClusterStatus struct {
// Standard Kubernetes condition list
// +optional
// +listType=map
// +listMapKey=type
Conditions []metav1.Condition `json:"conditions,omitempty"`
// Human-readable summary phase
// +optional
// +kubebuilder:default=Pending
Phase string `json:"phase,omitempty"`
// ReadyReplicas tracks how many replicas have passed readiness
// +optional
ReadyReplicas int32 `json:"readyReplicas,omitempty"`
// The generation of the spec this status reflects
// +optional
ObservedGeneration int64 `json:"observedGeneration,omitempty"`
// Connection string users can use to connect (injected into a Secret)
// +optional
ConnectionSecretName string `json:"connectionSecretName,omitempty"`
}
// Condition type constants — avoid magic strings
const (
ConditionTypeReady = "Ready"
ConditionTypeProgressing = "Progressing"
ConditionTypeDegraded = "Degraded"
ReasonReconciling = "Reconciling"
ReasonStatefulSetReady = "StatefulSetReady"
ReasonStorageFailed = "StorageFailed"
)
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Engine",type=string,JSONPath=`.spec.engine`
// +kubebuilder:printcolumn:name="Replicas",type=integer,JSONPath=`.spec.replicas`
// +kubebuilder:printcolumn:name="Ready",type=integer,JSONPath=`.status.readyReplicas`
// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
type DatabaseCluster struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec DatabaseClusterSpec `json:"spec,omitempty"`
Status DatabaseClusterStatus `json:"status,omitempty"`
}
// +kubebuilder:object:root=true
type DatabaseClusterList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []DatabaseCluster `json:"items"`
}
func init() {
SchemeBuilder.Register(&DatabaseCluster{}, &DatabaseClusterList{})
}
|
The // +kubebuilder: comments are markers — kubebuilder reads them and generates the CRD YAML, RBAC rules, and validation schema from them. They are not optional decoration; they drive code generation.
1
2
3
|
# Generate CRD YAML and deepcopy methods from markers
make generate
make manifests
|
The Reconciler
The reconciler is the heart of the operator. It is called whenever any watched resource changes. It must be idempotent: calling it ten times on the same resource should produce the same result as calling it once. The API server may call it multiple times for the same event, and it will be called at startup for every existing resource.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
|
// internal/controller/databasecluster_controller.go
package controller
import (
"context"
"fmt"
"time"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/api/meta"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/types"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"sigs.k8s.io/controller-runtime/pkg/log"
dbv1alpha1 "github.com/example/database-operator/api/v1alpha1"
)
const (
finalizerName = "db.example.com/finalizer"
)
type DatabaseClusterReconciler struct {
client.Client
Scheme *runtime.Scheme
}
// RBAC markers — kubebuilder generates ClusterRole from these
// +kubebuilder:rbac:groups=db.example.com,resources=databaseclusters,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=db.example.com,resources=databaseclusters/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=db.example.com,resources=databaseclusters/finalizers,verbs=update
// +kubebuilder:rbac:groups=apps,resources=statefulsets,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=services;configmaps;secrets;persistentvolumeclaims,verbs=get;list;watch;create;update;patch;delete
func (r *DatabaseClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
logger := log.FromContext(ctx)
// 1. Fetch the DatabaseCluster instance
cluster := &dbv1alpha1.DatabaseCluster{}
if err := r.Get(ctx, req.NamespacedName, cluster); err != nil {
if errors.IsNotFound(err) {
// Object deleted before we got to it — nothing to do
return ctrl.Result{}, nil
}
return ctrl.Result{}, fmt.Errorf("fetching DatabaseCluster: %w", err)
}
// 2. Handle deletion: check for finalizer
if !cluster.DeletionTimestamp.IsZero() {
return r.handleDeletion(ctx, cluster)
}
// 3. Ensure finalizer is present
if !controllerutil.ContainsFinalizer(cluster, finalizerName) {
controllerutil.AddFinalizer(cluster, finalizerName)
if err := r.Update(ctx, cluster); err != nil {
return ctrl.Result{}, fmt.Errorf("adding finalizer: %w", err)
}
// Re-queue after adding finalizer — a new reconcile will run
return ctrl.Result{}, nil
}
// 4. Mark as Progressing
if err := r.setCondition(ctx, cluster, dbv1alpha1.ConditionTypeProgressing,
metav1.ConditionTrue, dbv1alpha1.ReasonReconciling, "Reconciling cluster"); err != nil {
return ctrl.Result{}, err
}
// 5. Reconcile owned resources
if err := r.reconcileConfigMap(ctx, cluster); err != nil {
return ctrl.Result{}, err
}
if err := r.reconcileStatefulSet(ctx, cluster); err != nil {
return ctrl.Result{}, err
}
if err := r.reconcileService(ctx, cluster); err != nil {
return ctrl.Result{}, err
}
// 6. Observe actual state and update status
return r.updateStatus(ctx, cluster)
}
// handleDeletion runs cleanup before allowing the resource to be deleted
func (r *DatabaseClusterReconciler) handleDeletion(ctx context.Context, cluster *dbv1alpha1.DatabaseCluster) (ctrl.Result, error) {
logger := log.FromContext(ctx)
if !controllerutil.ContainsFinalizer(cluster, finalizerName) {
return ctrl.Result{}, nil
}
logger.Info("Running cleanup before deletion", "cluster", cluster.Name)
// Perform external cleanup (e.g., delete cloud snapshots, DNS records)
if err := r.cleanupExternalResources(ctx, cluster); err != nil {
return ctrl.Result{}, fmt.Errorf("cleanup failed: %w", err)
}
// Remove the finalizer — this allows the object to be deleted
controllerutil.RemoveFinalizer(cluster, finalizerName)
if err := r.Update(ctx, cluster); err != nil {
return ctrl.Result{}, fmt.Errorf("removing finalizer: %w", err)
}
return ctrl.Result{}, nil
}
func (r *DatabaseClusterReconciler) cleanupExternalResources(ctx context.Context, cluster *dbv1alpha1.DatabaseCluster) error {
// Add external cleanup logic here: cloud snapshots, Route53 records, etc.
return nil
}
|
Reconciling Owned Resources
The reconciler creates, updates, and owns child resources (StatefulSet, Service, ConfigMap). Owner references tell the garbage collector to delete child resources when the parent CR is deleted. controllerutil.CreateOrUpdate is the idiomatic pattern for ensuring a resource exists with the right spec:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
|
func (r *DatabaseClusterReconciler) reconcileStatefulSet(ctx context.Context, cluster *dbv1alpha1.DatabaseCluster) error {
sts := &appsv1.StatefulSet{
ObjectMeta: metav1.ObjectMeta{
Name: cluster.Name,
Namespace: cluster.Namespace,
},
}
// CreateOrUpdate: if sts exists, call mutate; if not, create it
op, err := controllerutil.CreateOrUpdate(ctx, r.Client, sts, func() error {
// Set owner reference — child is GC'd when parent is deleted
if err := controllerutil.SetControllerReference(cluster, sts, r.Scheme); err != nil {
return err
}
// Always set labels for watching
sts.Labels = map[string]string{
"app.kubernetes.io/name": "database-cluster",
"app.kubernetes.io/instance": cluster.Name,
"app.kubernetes.io/managed-by": "database-operator",
}
replicas := cluster.Spec.Replicas
sts.Spec.Replicas = &replicas
sts.Spec.Selector = &metav1.LabelSelector{
MatchLabels: map[string]string{"app": cluster.Name},
}
sts.Spec.ServiceName = cluster.Name + "-headless"
sts.Spec.Template = corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: map[string]string{"app": cluster.Name},
},
Spec: corev1.PodSpec{
Containers: []corev1.Container{
{
Name: "database",
Image: fmt.Sprintf("%s:%s", cluster.Spec.Engine, cluster.Spec.Version),
Ports: []corev1.ContainerPort{{ContainerPort: 5432}},
EnvFrom: []corev1.EnvFromSource{
{ConfigMapRef: &corev1.ConfigMapEnvSource{
LocalObjectReference: corev1.LocalObjectReference{
Name: cluster.Name + "-config",
},
}},
},
Resources: corev1.ResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse(defaultOr(cluster.Spec.Resources.CPU, "250m")),
corev1.ResourceMemory: resource.MustParse(defaultOr(cluster.Spec.Resources.Memory, "512Mi")),
},
},
},
},
},
}
// Volume claim templates for persistent storage
if len(sts.Spec.VolumeClaimTemplates) == 0 {
sts.Spec.VolumeClaimTemplates = []corev1.PersistentVolumeClaim{
{
ObjectMeta: metav1.ObjectMeta{Name: "data"},
Spec: corev1.PersistentVolumeClaimSpec{
AccessModes: []corev1.PersistentVolumeAccessMode{corev1.ReadWriteOnce},
StorageClassName: cluster.Spec.Storage.StorageClassName,
Resources: corev1.ResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceStorage: resource.MustParse(cluster.Spec.Storage.Size),
},
},
},
},
}
}
return nil
})
if err != nil {
return fmt.Errorf("reconciling StatefulSet: %w", err)
}
log.FromContext(ctx).Info("StatefulSet reconciled", "operation", op)
return nil
}
func (r *DatabaseClusterReconciler) reconcileService(ctx context.Context, cluster *dbv1alpha1.DatabaseCluster) error {
// Headless service for stable DNS within the StatefulSet
headless := &corev1.Service{
ObjectMeta: metav1.ObjectMeta{
Name: cluster.Name + "-headless",
Namespace: cluster.Namespace,
},
}
_, err := controllerutil.CreateOrUpdate(ctx, r.Client, headless, func() error {
controllerutil.SetControllerReference(cluster, headless, r.Scheme)
headless.Spec.ClusterIP = "None"
headless.Spec.Selector = map[string]string{"app": cluster.Name}
headless.Spec.Ports = []corev1.ServicePort{{Port: 5432}}
return nil
})
return err
}
func (r *DatabaseClusterReconciler) reconcileConfigMap(ctx context.Context, cluster *dbv1alpha1.DatabaseCluster) error {
cm := &corev1.ConfigMap{
ObjectMeta: metav1.ObjectMeta{
Name: cluster.Name + "-config",
Namespace: cluster.Namespace,
},
}
_, err := controllerutil.CreateOrUpdate(ctx, r.Client, cm, func() error {
controllerutil.SetControllerReference(cluster, cm, r.Scheme)
cm.Data = map[string]string{
"PGDATA": "/var/lib/postgresql/data/pgdata",
"POSTGRES_DB": cluster.Name,
}
return nil
})
return err
}
func defaultOr(val, fallback string) string {
if val == "" {
return fallback
}
return val
}
|
Status Conditions and Observability
Status conditions are the Kubernetes-idiomatic way to communicate multi-dimensional state. They follow the metav1.Condition pattern: a list of conditions, each with a Type, Status (True/False/Unknown), Reason (machine-readable CamelCase), Message (human-readable), and ObservedGeneration.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
|
func (r *DatabaseClusterReconciler) updateStatus(ctx context.Context, cluster *dbv1alpha1.DatabaseCluster) (ctrl.Result, error) {
// Fetch the current StatefulSet to check its state
sts := &appsv1.StatefulSet{}
if err := r.Get(ctx, types.NamespacedName{
Name: cluster.Name,
Namespace: cluster.Namespace,
}, sts); err != nil {
return ctrl.Result{}, err
}
// Update ready replica count
cluster.Status.ReadyReplicas = sts.Status.ReadyReplicas
cluster.Status.ObservedGeneration = cluster.Generation
desiredReplicas := cluster.Spec.Replicas
readyReplicas := sts.Status.ReadyReplicas
if readyReplicas >= desiredReplicas {
cluster.Status.Phase = "Ready"
r.setCondition(ctx, cluster, dbv1alpha1.ConditionTypeReady,
metav1.ConditionTrue,
dbv1alpha1.ReasonStatefulSetReady,
fmt.Sprintf("All %d replicas ready", desiredReplicas))
r.setCondition(ctx, cluster, dbv1alpha1.ConditionTypeProgressing,
metav1.ConditionFalse,
"ReconcileComplete",
"Reconciliation complete")
} else {
cluster.Status.Phase = "Progressing"
r.setCondition(ctx, cluster, dbv1alpha1.ConditionTypeReady,
metav1.ConditionFalse,
dbv1alpha1.ReasonReconciling,
fmt.Sprintf("%d/%d replicas ready", readyReplicas, desiredReplicas))
// Re-queue to check again shortly
return ctrl.Result{RequeueAfter: 10 * time.Second}, r.Status().Update(ctx, cluster)
}
return ctrl.Result{}, r.Status().Update(ctx, cluster)
}
func (r *DatabaseClusterReconciler) setCondition(ctx context.Context, cluster *dbv1alpha1.DatabaseCluster,
condType string, status metav1.ConditionStatus, reason, message string) error {
meta.SetStatusCondition(&cluster.Status.Conditions, metav1.Condition{
Type: condType,
Status: status,
Reason: reason,
Message: message,
ObservedGeneration: cluster.Generation,
LastTransitionTime: metav1.Now(),
})
return r.Status().Update(ctx, cluster)
}
|
With this in place, kubectl get databaseclusters shows the custom print columns, and kubectl describe databasecluster my-postgres shows the condition list — identical to how built-in resources communicate state.
Watching Owned Resources
By default the reconciler only triggers when the DatabaseCluster CR changes. You also want it to trigger when an owned StatefulSet changes — for example, when a pod crashes and the StatefulSet’s ready count drops.
1
2
3
4
5
6
7
8
9
10
11
12
|
// In SetupWithManager, add watches for owned resource types
func (r *DatabaseClusterReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&dbv1alpha1.DatabaseCluster{}).
// Trigger reconcile when an owned StatefulSet changes
Owns(&appsv1.StatefulSet{}).
// Trigger reconcile when an owned Service changes
Owns(&corev1.Service{}).
// Trigger reconcile when an owned ConfigMap changes
Owns(&corev1.ConfigMap{}).
Complete(r)
}
|
Owns() sets up a watch with an owner reference filter — only StatefulSets owned by a DatabaseCluster trigger a reconcile for that specific cluster. Unrelated StatefulSets do not cause spurious reconciles.
For watching resources you do not own (for example, a Secret managed by cert-manager that your CR references), use Watches() with a custom EnqueueRequestsFromMapFunc:
1
2
3
4
5
6
7
8
9
|
Watches(
&corev1.Secret{},
handler.EnqueueRequestsFromMapFunc(func(ctx context.Context, obj client.Object) []reconcile.Request {
// Map the changed Secret to the DatabaseCluster that references it
// Return a reconcile request for each affected cluster
return r.findClustersForSecret(ctx, obj)
}),
builder.WithPredicates(predicate.ResourceVersionChangedPredicate{}),
).
|
Predicates: Filtering Spurious Reconciles
Without predicates, the reconciler triggers on every update to a watched resource — including status-only updates, which would cause a reconcile loop if your reconciler writes status. Use predicates to filter:
1
2
3
4
5
6
|
// Only trigger on spec changes, not status updates
ctrl.NewControllerManagedBy(mgr).
For(&dbv1alpha1.DatabaseCluster{},
builder.WithPredicates(predicate.GenerationChangedPredicate{})).
Owns(&appsv1.StatefulSet{}).
Complete(r)
|
GenerationChangedPredicate only fires when the resource’s .metadata.generation increments. Generation only increments on spec changes — not status updates. This prevents the reconciler from looping on its own status writes.
The Finalizer Pattern in Full
Finalizers are the mechanism for running cleanup logic before a resource is deleted. Without a finalizer, kubectl delete databasecluster my-postgres immediately removes the object and Kubernetes garbage-collects owned resources — but external resources (cloud storage snapshots, DNS entries, load balancers you created outside Kubernetes) are not cleaned up.
kubectl delete databasecluster my-postgres
│
▼
API server sets DeletionTimestamp (does NOT delete yet)
│
▼
Reconciler sees DeletionTimestamp != nil → runs handleDeletion()
│
├─ Performs external cleanup
│ - delete cloud snapshots
│ - de-register from service discovery
│ - notify downstream systems
│
├─ RemoveFinalizer() → Update() → finalizer list is now empty
│
▼
API server sees no finalizers → deletes the object
Kubernetes GC deletes owned resources (StatefulSet, Service, ConfigMap)
The failure mode to avoid: if cleanupExternalResources returns an error and you return early without removing the finalizer, the object stays in a terminating state until you fix the underlying problem. This is intentional — the finalizer blocks deletion until cleanup succeeds. Always provide a documented emergency procedure for force-removing a stuck finalizer:
1
2
3
4
|
# Emergency removal if cleanup is impossible (e.g., cloud account deleted)
kubectl patch databasecluster my-postgres \
-p '{"metadata":{"finalizers":[]}}' \
--type=merge
|
Running and Testing
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
|
# Install CRDs into the current cluster
make install
# Run the controller locally (against current kubeconfig context)
make run
# Apply a DatabaseCluster resource
cat <<EOF | kubectl apply -f -
apiVersion: db.example.com/v1alpha1
kind: DatabaseCluster
metadata:
name: my-postgres
namespace: default
spec:
engine: postgres
version: "16.2"
replicas: 3
storage:
size: 10Gi
backupsEnabled: true
EOF
# Watch what happens
kubectl get databaseclusters -w
# See full status including conditions
kubectl describe databasecluster my-postgres
# Check owned StatefulSet
kubectl get statefulset my-postgres
# Test deletion (runs finalizer cleanup)
kubectl delete databasecluster my-postgres
|
For integration testing, controller-runtime provides envtest — a real API server and etcd running in your test process, with no need for a real cluster:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
|
// suite_test.go
var (
cfg *rest.Config
k8sClient client.Client
testEnv *envtest.Environment
)
func TestAPIs(t *testing.T) {
RegisterFailHandler(Fail)
RunSpecs(t, "Controller Suite")
}
var _ = BeforeSuite(func() {
testEnv = &envtest.Environment{
CRDDirectoryPaths: []string{filepath.Join("..", "..", "config", "crd", "bases")},
}
var err error
cfg, err = testEnv.Start()
Expect(err).NotTo(HaveOccurred())
err = dbv1alpha1.AddToScheme(scheme.Scheme)
Expect(err).NotTo(HaveOccurred())
k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme})
Expect(err).NotTo(HaveOccurred())
})
// reconciler_test.go
var _ = Describe("DatabaseCluster controller", func() {
It("should create a StatefulSet for a new DatabaseCluster", func() {
cluster := &dbv1alpha1.DatabaseCluster{
ObjectMeta: metav1.ObjectMeta{Name: "test-cluster", Namespace: "default"},
Spec: dbv1alpha1.DatabaseClusterSpec{
Engine: "postgres",
Version: "16.2",
Replicas: 1,
Storage: dbv1alpha1.StorageSpec{Size: "1Gi"},
},
}
Expect(k8sClient.Create(ctx, cluster)).To(Succeed())
sts := &appsv1.StatefulSet{}
Eventually(func() error {
return k8sClient.Get(ctx, types.NamespacedName{
Name: "test-cluster",
Namespace: "default",
}, sts)
}, timeout, interval).Should(Succeed())
Expect(*sts.Spec.Replicas).To(Equal(int32(1)))
})
})
|
Deploying the Operator
1
2
3
4
5
6
7
8
9
|
# Build and push the operator image
make docker-build docker-push IMG=your-registry/database-operator:v0.1.0
# Deploy to cluster (installs CRDs + controller Deployment + RBAC)
make deploy IMG=your-registry/database-operator:v0.1.0
# Verify
kubectl get pods -n database-operator-system
kubectl get crds | grep example.com
|
The generated Deployment runs as a single replica by default. For HA, the controller-runtime manager supports leader election — only the elected leader actively reconciles while others stand by:
1
2
3
4
5
6
|
// main.go — enable leader election
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
LeaderElection: true,
LeaderElectionID: "database-operator-leader-election",
LeaderElectionNamespace: "database-operator-system",
})
|
The Patterns That Matter Most
Idempotency is not optional. The reconciler may be called many times for one logical event — at startup for every existing resource, on any update, after errors. Every write must be a no-op if the desired state is already achieved. CreateOrUpdate handles this for resource mutations. Status updates should check ObservedGeneration before writing.
Never read-modify-write. Fetching a resource, modifying a field, and updating it races with other writers. Use controllerutil.CreateOrUpdate with a mutate function that sets only the fields you own. Use Server-Side Apply for complex resources.
Return errors to re-queue. When reconciliation fails partway through, return the error. controller-runtime re-queues with exponential backoff. Do not swallow errors — a silent failure leaves the cluster in a partially converged state with no indication of the problem.
Own what you create. Set controller owner references on every resource you create. If you do not, orphaned resources accumulate when CRs are deleted, and users have no way to know which operator created them.
Keep controllers single-purpose. One controller per CRD. If you have DatabaseCluster and DatabaseBackup, they get separate controllers even if they interact. Cross-resource logic goes in the reconcile flow, not in shared mutable state.
Writing an operator the first time is more code than expected. Writing it correctly — idempotent, properly owned, with real status conditions, clean deletion — requires internalizing these patterns. The kubebuilder scaffold gets you to a working skeleton in an hour. The patterns above are what separate a working operator from one that is safe to run in production.
Comments